While cleaning up old images/containers, I accidentally removed an image that other images/containers depended on. After that, my trusty LEMP container broke. This isn’t supposed to be possible in the latest version of Docker, but if you find yourself in the same boat, here is what I did to recover some important files:
Note: I tried to undelete the image using extundelete, but it didn’t recover anything worthwhile for me.
The Problem
aleckz@kronos:~$ docker restart 860e297944ca Error: Error restarting container 860e297944ca: No such id: 3262d0ef973b7207b2cef95407d1a11aa564417378fb1fee630d0a9bf8bbcb84 aleckz@kronos:~$ docker run -t -i BugvoteLempClean:latest /bin/bash 2013/08/07 18:08:02 Error: Error starting container f04a69e0afbc: Error while getting parent image: No such id: 3262d0ef973b7207b2cef95407d1a11aa564417378fb1fee630d0a9bf8bbcb84
Oh boy!
Containers are transparent
Luckily, Docker images/containers are easier to work with than VMs. No crazy mounting tools, no image conversion (.vdi to .raw, then mounting over loopback anyone?)
Lets find a good container that we can rummage through:
aleckz@kronos:~$ docker ps -a ID IMAGE COMMAND CREATED STATUS PORTS f04a69e0afbc BugvoteLempClean:latest /bin/bash 3 minutes ago Exit 0 f8fa77e12392 ubuntu:12.04 /bin/bash 32 minutes ago Exit 0 e0ed3a9254aa BugvoteLempClean:latest /bin/bash 35 minutes ago Exit 0 12fe1461d525 BugvoteLempClean:latest /bin/bash 42 minutes ago Exit 0 a62a53caa0e2 centos:latest /bin/bash 42 minutes ago Exit 0 6bf57354a39b centos:latest /bin/bash 44 minutes ago Exit 0 be5e8a6d75fd centos:latest /bin/bash 45 minutes ago Exit 0 fddd52749abe BugvoteLempClean:latest /bin/bash 48 minutes ago Exit 0 6088041d14a6 PreciseLempSelfContained:latest -v /var/www/bugvote: 2 hours ago Exit 127 72dcf23220d0 BugvoteLempClean:latest /bin/bash 2 hours ago Exit 0 8e0d8d8ef60f BugvoteLempClean:latest /bin/bash 2 hours ago Exit 0 d88ecd0ef300 cdd5136d7139 /bin/bash 2 hours ago Exit 1 0531469182aa e1fb8edf1e6d /bin/bash 4 days ago Exit 255 860e297944ca 3262d0ef973b /bin/bash 5 days ago Exit 0 ab9ee134b5c0 3262d0ef973b /bin/bash 5 days ago Exit 0 13fb5aa4ed42 3262d0ef973b /bin/bash 5 days ago Exit 0 f54ac1b77e0b 3262d0ef973b /bin/bash 5 days ago Exit 0 75db43743f5a 2eb422efc0da /bin/bash 7 days ago Exit 0 069aa6a7ac56 PreciseLempSelfContained:latest /bin/bash 7 days ago Exit 0 e89982362d3c ubuntu:12.04 /bin/bash 7 days ago Exit 127
The last hour of activity is me starting to rebuilding from scratch. Skip that. The highlighted group is where I think my good container is.
Code and SQL data is hosted outside of Docker, I mount them into the container, and use the container to host nginx, php-fpm, mysql, postgresql, redis, sphinxsearch, etc. The server binaries can be reinstalled easily. The only thing I really want is those painstakingly tweaked config files!
Specifically, I want to recover this one nginx config file full of sexy image-resizing rewrite rules for etags/304/caching.
Let’s peak inside container 860e297944ca. It’s 5 days ago and from an unnamed image (one of many that I deleted). Here are the the last few lines from its life:
aleckz@kronos:~$ docker logs 860e297944ca | tail -n 40 -rw-rw-r-- 1 1000 www-data 640 Aug 1 22:48 deviceseeks.stp -rw-rw-r-- 1 1000 www-data 699 Aug 1 22:48 ipc-listener.php -rw-rw-r-- 1 1000 www-data 3.9K Aug 1 22:48 listener-renderer.php -rw-rw-r-- 1 1000 www-data 881 Aug 1 22:48 listener.php -rw-rw-r-- 1 1000 www-data 347 Aug 1 22:48 msg-listener.php -rw-rw-r-- 1 1000 www-data 821 Aug 1 22:48 udp-listener.php drwxrwxr-x 2 1000 www-data 4.0K Aug 1 22:48 workers/ -rw-rw-r-- 1 1000 www-data 448 Aug 1 22:48 zeromq-listener.php root@prometheus:/var/www/www.bugvote.com/system/tools# ps faux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 18236 1884 ? S Aug05 0:00 /bin/bash root 50 0.0 0.0 17984 1456 ? S Aug05 0:00 /bin/bash /usr/bin/mysqld_safe mysql 765 0.0 0.6 1462788 80880 ? Sl Aug05 1:31 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql -- root 766 0.0 0.0 4300 572 ? S Aug05 0:00 \_ logger -t mysqld -p daemon.error root 1053 0.0 0.0 62952 1360 ? Ss Aug05 0:00 nginx: master process /usr/sbin/nginx www-data 1054 0.0 0.0 63556 2808 ? S Aug05 0:04 \_ nginx: worker process www-data 1056 0.0 0.0 63684 2812 ? S Aug05 0:00 \_ nginx: worker process dnsmasq 1204 0.0 0.0 25964 844 ? S Aug05 0:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new root 8455 0.0 0.0 139208 4652 ? Ss Aug06 0:01 php-fpm: master process (/etc/php5/fpm/php-fpm.conf) www-data 8456 0.0 0.1 143780 18504 ? S Aug06 0:02 \_ php-fpm: pool www www-data 8457 0.0 0.1 143556 18420 ? S Aug06 0:02 \_ php-fpm: pool www root 8472 0.0 0.0 15272 1080 ? R+ 19:09 0:00 ps faux root@prometheus:/var/www/www.bugvote.com/system/tools# service mysql stop root@prometheus:/var/www/www.bugvote.com/system/tools# service php5-fpm stop root@prometheus:/var/www/www.bugvote.com/system/tools# service nginx stop ...
I remember manually turning the services off before stopping the container (hello OCD), so this may be our guy! But I can’t start it due to missing parent, so now what?
Because I erased the parent image (3262d0ef973b), the contents baked into the image are gone. But any modifications I made SINCE the image creation (ie: any tweaked config files) will be within the container’s layer! Cross your fingers and sudo up, then find a matching container in /var/lib/docker/ (the default docker install path on Ubuntu)
root@kronos:/var/lib/docker/containers# ls -halF total 88K drwx------ 22 root root 4.0K Aug 7 18:08 ./ drwx------ 5 root root 4.0K Jul 30 23:30 ../ drwx------ 3 root root 4.0K Aug 7 17:29 0531469182aa7a396f5b75369d16f4f5f5bd8bdaee9fc58c223b4d6a8cf45fe4/ drwx------ 3 root root 4.0K Jul 30 23:43 069aa6a7ac562ad3f4a16803dd26dc0ea131daeb16c4aef8b568d7f52fd89538/ drwx------ 2 root root 4.0K Aug 7 17:29 12fe1461d52596c4d1a5ad21cc37407a2a50db76953ad21c406555b037ac6557/ drwx------ 3 root root 4.0K Aug 1 18:14 13fb5aa4ed42f9af2b8b789510659fa5d4744b6c1de20d75c2c95b7a764ea095/ drwx------ 3 root root 4.0K Aug 7 16:02 6088041d14a67ec68bbb41682a1c8c1399ab51d701acfc73e85d1386770ca118/ drwx------ 4 root root 4.0K Aug 7 17:26 6bf57354a39b03a1a94631bb12d8cb77b3c94f89cb89244aee25ae08e4e8848f/ drwx------ 2 root root 4.0K Aug 7 15:41 72dcf23220d067ee54a273a4fbb20c723446b8d5790f2875d798868395bb7c9d/ drwx------ 3 root root 4.0K Aug 1 18:01 75db43743f5aa9932de522f6c6a36fda96957782c74943c9fa66dfdc739a282a/ drwx------ 3 root root 4.0K Aug 7 15:10 860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/ drwx------ 2 root root 4.0K Aug 7 15:21 8e0d8d8ef60f5da90dac058a19b90d611124c2497d34325bf25135d7aac02f9c/ drwx------ 3 root root 4.0K Aug 7 17:28 a62a53caa0e2f7a1d583eeb110c377c3ded0a97b98de46c37dd80cc9204ad3eb/ drwx------ 3 root root 4.0K Aug 1 18:16 ab9ee134b5c0974c69d384af644a5670cbadd13d705f275746747134c578e89d/ drwx------ 4 root root 4.0K Aug 7 17:25 be5e8a6d75fde19118f087fa1f2538f3e5cf0202ae7d82c25b840f9f90e037e6/ drwx------ 3 root root 4.0K Aug 7 15:20 d88ecd0ef30027ea083b00f43365fd3f0af9189667b6102ca2e0e6338ba78e49/ drwx------ 2 root root 4.0K Aug 7 17:35 e0ed3a9254aaebb62b7da13fbf36a4a82da8c28adbaeb462ca9bb99cc8d9808f/ drwx------ 3 root root 4.0K Jul 30 23:07 e89982362d3c0d27f31de77bda23f89be854acfb5999db9f27e42291cdc7543d/ drwx------ 2 root root 4.0K Aug 7 18:08 f04a69e0afbcfde340dc460b1e8a405d8b90e37477f6ff4f37f4361f2557d075/ drwx------ 3 root root 4.0K Aug 1 18:13 f54ac1b77e0bee766c264e68c1b9deaf293e1d31264ebc9ac76ff2df77b03c13/ drwx------ 3 root root 4.0K Aug 7 18:06 f8fa77e1239235fde5706fa767ddcadd2a808285619e901836bacb61b3a45606/ drwx------ 2 root root 4.0K Aug 7 17:22 fddd52749abef5ff7de5127799c42cf3f3c98b2123ccb35dca95d880f42afa99/
Let’s see if the container has anything good:
root@kronos:/var/lib/docker/containers# ls -R 860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/rw/etc/nginx/ 860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/rw/etc/nginx/: nginx.conf sites-available 860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/rw/etc/nginx/sites-available: static.bugvote.com www.bugvote.com
Oh yeah! The container has the /etc/nginx folder and the most recently modified nginx configs! I even found an php-xdebug config (debugging in PhpStorm mmm yeah!)
Copy and restore
To grab files out of a container, just copy them! I copied the config files, and used them to rebuild my LEMP container. Hooray!
Recovering files from within Docker is possible. Containers are not encrypted or obscured. It’s all there for easy pickings. Assuming you get lucky :)
Worst case scenario you could always fgrep/find your way through the entire /var/lib/docker/containers and /var/lib/docker/graph folders, digging for treasure:
root@kronos:/var/lib/docker/graph# find . -iname nginx.conf ./6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/nginx.conf ./b6b4d496e1bcc1a2e12d60287aa85da6af5e6a4c1ab923a643e6bb795c8c62c2/layer/etc/nginx/nginx.conf root@kronos:/var/lib/docker/graph# ls -R -halF 6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/ 6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/: total 16K drwxr-xr-x 3 root root 4.0K Jul 30 23:56 ./ drwxr-xr-x 37 root root 4.0K Aug 5 23:55 ../ -rw-r--r-- 1 root root 1.5K Aug 1 18:22 nginx.conf drwxr-xr-x 2 root root 4.0K Jul 30 23:55 sites-available/ 6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/sites-available: total 16K drwxr-xr-x 2 root root 4.0K Jul 30 23:55 ./ drwxr-xr-x 3 root root 4.0K Jul 30 23:56 ../ -rw-r--r-- 1 root root 3.0K Aug 1 18:23 static.bugvote.com -rw-r--r-- 1 root root 1.4K Aug 1 18:21 www.bugvote.com
Yarr, treasure! :)