Recovering files from a fudged docker container

While cleaning up old images/containers, I accidentally removed an image that other images/containers depended on. After that, my trusty LEMP container broke. This isn’t supposed to be possible in the latest version of Docker, but if you find yourself in the same boat, here is what I did to recover some important files:

Note: I tried to undelete the image using extundelete, but it didn’t recover anything worthwhile for me.

The Problem

aleckz@kronos:~$ docker restart 860e297944ca
Error: Error restarting container 860e297944ca: No such id: 3262d0ef973b7207b2cef95407d1a11aa564417378fb1fee630d0a9bf8bbcb84

aleckz@kronos:~$ docker run -t -i BugvoteLempClean:latest /bin/bash
2013/08/07 18:08:02 Error: Error starting container f04a69e0afbc: Error while getting parent image: No such id: 3262d0ef973b7207b2cef95407d1a11aa564417378fb1fee630d0a9bf8bbcb84

Oh boy!

Containers are transparent

Luckily, Docker images/containers are easier to work with than VMs. No crazy mounting tools, no image conversion (.vdi to .raw, then mounting over loopback anyone?)

Lets find a good container that we can rummage through:

aleckz@kronos:~$ docker ps -a
ID                  IMAGE                             COMMAND                CREATED             STATUS              PORTS
f04a69e0afbc        BugvoteLempClean:latest           /bin/bash              3 minutes ago       Exit 0                                  
f8fa77e12392        ubuntu:12.04                      /bin/bash              32 minutes ago      Exit 0                                  
e0ed3a9254aa        BugvoteLempClean:latest           /bin/bash              35 minutes ago      Exit 0                                  
12fe1461d525        BugvoteLempClean:latest           /bin/bash              42 minutes ago      Exit 0                                  
a62a53caa0e2        centos:latest                     /bin/bash              42 minutes ago      Exit 0                                  
6bf57354a39b        centos:latest                     /bin/bash              44 minutes ago      Exit 0                                  
be5e8a6d75fd        centos:latest                     /bin/bash              45 minutes ago      Exit 0                                  
fddd52749abe        BugvoteLempClean:latest           /bin/bash              48 minutes ago      Exit 0                                  
6088041d14a6        PreciseLempSelfContained:latest   -v /var/www/bugvote:   2 hours ago         Exit 127                                
72dcf23220d0        BugvoteLempClean:latest           /bin/bash              2 hours ago         Exit 0                                  
8e0d8d8ef60f        BugvoteLempClean:latest           /bin/bash              2 hours ago         Exit 0                                  
d88ecd0ef300        cdd5136d7139                      /bin/bash              2 hours ago         Exit 1                                  
0531469182aa        e1fb8edf1e6d                      /bin/bash              4 days ago          Exit 255                                
860e297944ca        3262d0ef973b                      /bin/bash              5 days ago          Exit 0                                  
ab9ee134b5c0        3262d0ef973b                      /bin/bash              5 days ago          Exit 0                                  
13fb5aa4ed42        3262d0ef973b                      /bin/bash              5 days ago          Exit 0                                  
f54ac1b77e0b        3262d0ef973b                      /bin/bash              5 days ago          Exit 0                                 
75db43743f5a        2eb422efc0da                      /bin/bash              7 days ago          Exit 0                                  
069aa6a7ac56        PreciseLempSelfContained:latest   /bin/bash              7 days ago          Exit 0                                  
e89982362d3c        ubuntu:12.04                      /bin/bash              7 days ago          Exit 127

The last hour of activity is me starting to rebuilding from scratch. Skip that. The highlighted group is where I think my good container is.

Code and SQL data is hosted outside of Docker, I mount them into the container, and use the container to host nginx, php-fpm, mysql, postgresql, redis, sphinxsearch, etc. The server  binaries can be reinstalled easily. The only thing I really want is those painstakingly tweaked config files!

Specifically, I want to recover this one nginx config file full of sexy image-resizing rewrite rules for etags/304/caching.

Let’s peak inside container 860e297944ca. It’s 5 days ago and from an unnamed image (one of many that I deleted). Here are the the last few lines from its life:

aleckz@kronos:~$ docker logs 860e297944ca | tail -n 40
-rw-rw-r--  1 1000 www-data  640 Aug  1 22:48 deviceseeks.stp
-rw-rw-r--  1 1000 www-data  699 Aug  1 22:48 ipc-listener.php
-rw-rw-r--  1 1000 www-data 3.9K Aug  1 22:48 listener-renderer.php
-rw-rw-r--  1 1000 www-data  881 Aug  1 22:48 listener.php
-rw-rw-r--  1 1000 www-data  347 Aug  1 22:48 msg-listener.php
-rw-rw-r--  1 1000 www-data  821 Aug  1 22:48 udp-listener.php
drwxrwxr-x  2 1000 www-data 4.0K Aug  1 22:48 workers/
-rw-rw-r--  1 1000 www-data  448 Aug  1 22:48 zeromq-listener.php

root@prometheus:/var/www/www.bugvote.com/system/tools# ps faux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  18236  1884 ?        S    Aug05   0:00 /bin/bash
root        50  0.0  0.0  17984  1456 ?        S    Aug05   0:00 /bin/bash /usr/bin/mysqld_safe
mysql      765  0.0  0.6 1462788 80880 ?       Sl   Aug05   1:31  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --
root       766  0.0  0.0   4300   572 ?        S    Aug05   0:00  \_ logger -t mysqld -p daemon.error
root      1053  0.0  0.0  62952  1360 ?        Ss   Aug05   0:00 nginx: master process /usr/sbin/nginx
www-data  1054  0.0  0.0  63556  2808 ?        S    Aug05   0:04  \_ nginx: worker process
www-data  1056  0.0  0.0  63684  2812 ?        S    Aug05   0:00  \_ nginx: worker process
dnsmasq   1204  0.0  0.0  25964   844 ?        S    Aug05   0:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
root      8455  0.0  0.0 139208  4652 ?        Ss   Aug06   0:01 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)      
www-data  8456  0.0  0.1 143780 18504 ?        S    Aug06   0:02  \_ php-fpm: pool www                                         
www-data  8457  0.0  0.1 143556 18420 ?        S    Aug06   0:02  \_ php-fpm: pool www                                         
root      8472  0.0  0.0  15272  1080 ?        R+   19:09   0:00 ps faux

root@prometheus:/var/www/www.bugvote.com/system/tools# service mysql stop
root@prometheus:/var/www/www.bugvote.com/system/tools# service php5-fpm stop
root@prometheus:/var/www/www.bugvote.com/system/tools# service nginx stop
...

I remember manually turning the services off before stopping the container (hello OCD), so this may be our guy! But I can’t start it due to missing parent, so now what?

Because I erased the parent image (3262d0ef973b), the contents baked into the image are gone. But any modifications I made SINCE the image creation (ie: any tweaked config files) will be within the container’s layer! Cross your fingers and sudo up, then find a matching container in /var/lib/docker/ (the default docker install path on Ubuntu)

root@kronos:/var/lib/docker/containers# ls -halF
total 88K
drwx------ 22 root root 4.0K Aug  7 18:08 ./
drwx------  5 root root 4.0K Jul 30 23:30 ../
drwx------  3 root root 4.0K Aug  7 17:29 0531469182aa7a396f5b75369d16f4f5f5bd8bdaee9fc58c223b4d6a8cf45fe4/
drwx------  3 root root 4.0K Jul 30 23:43 069aa6a7ac562ad3f4a16803dd26dc0ea131daeb16c4aef8b568d7f52fd89538/
drwx------  2 root root 4.0K Aug  7 17:29 12fe1461d52596c4d1a5ad21cc37407a2a50db76953ad21c406555b037ac6557/
drwx------  3 root root 4.0K Aug  1 18:14 13fb5aa4ed42f9af2b8b789510659fa5d4744b6c1de20d75c2c95b7a764ea095/
drwx------  3 root root 4.0K Aug  7 16:02 6088041d14a67ec68bbb41682a1c8c1399ab51d701acfc73e85d1386770ca118/
drwx------  4 root root 4.0K Aug  7 17:26 6bf57354a39b03a1a94631bb12d8cb77b3c94f89cb89244aee25ae08e4e8848f/
drwx------  2 root root 4.0K Aug  7 15:41 72dcf23220d067ee54a273a4fbb20c723446b8d5790f2875d798868395bb7c9d/
drwx------  3 root root 4.0K Aug  1 18:01 75db43743f5aa9932de522f6c6a36fda96957782c74943c9fa66dfdc739a282a/
drwx------  3 root root 4.0K Aug  7 15:10 860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/
drwx------  2 root root 4.0K Aug  7 15:21 8e0d8d8ef60f5da90dac058a19b90d611124c2497d34325bf25135d7aac02f9c/
drwx------  3 root root 4.0K Aug  7 17:28 a62a53caa0e2f7a1d583eeb110c377c3ded0a97b98de46c37dd80cc9204ad3eb/
drwx------  3 root root 4.0K Aug  1 18:16 ab9ee134b5c0974c69d384af644a5670cbadd13d705f275746747134c578e89d/
drwx------  4 root root 4.0K Aug  7 17:25 be5e8a6d75fde19118f087fa1f2538f3e5cf0202ae7d82c25b840f9f90e037e6/
drwx------  3 root root 4.0K Aug  7 15:20 d88ecd0ef30027ea083b00f43365fd3f0af9189667b6102ca2e0e6338ba78e49/
drwx------  2 root root 4.0K Aug  7 17:35 e0ed3a9254aaebb62b7da13fbf36a4a82da8c28adbaeb462ca9bb99cc8d9808f/
drwx------  3 root root 4.0K Jul 30 23:07 e89982362d3c0d27f31de77bda23f89be854acfb5999db9f27e42291cdc7543d/
drwx------  2 root root 4.0K Aug  7 18:08 f04a69e0afbcfde340dc460b1e8a405d8b90e37477f6ff4f37f4361f2557d075/
drwx------  3 root root 4.0K Aug  1 18:13 f54ac1b77e0bee766c264e68c1b9deaf293e1d31264ebc9ac76ff2df77b03c13/
drwx------  3 root root 4.0K Aug  7 18:06 f8fa77e1239235fde5706fa767ddcadd2a808285619e901836bacb61b3a45606/
drwx------  2 root root 4.0K Aug  7 17:22 fddd52749abef5ff7de5127799c42cf3f3c98b2123ccb35dca95d880f42afa99/

Let’s see if the container has anything good:

root@kronos:/var/lib/docker/containers# ls -R 860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/rw/etc/nginx/
860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/rw/etc/nginx/:
nginx.conf  sites-available

860e297944caba577b1ba8074bfd665ccaa106607a55a4f48bcd0be102c709db/rw/etc/nginx/sites-available:
static.bugvote.com  www.bugvote.com

Oh yeah! The container has the /etc/nginx folder and the most recently modified nginx configs! I even found an php-xdebug config (debugging in PhpStorm mmm yeah!)

Copy and restore

To grab files out of a container, just copy them! I copied the config files, and used them to rebuild my LEMP container. Hooray!

Recovering files from within Docker is possible. Containers are not encrypted or obscured. It’s all there for easy pickings. Assuming you get lucky :)

Worst case scenario you could always fgrep/find your way through the entire /var/lib/docker/containers and /var/lib/docker/graph folders, digging for treasure:

root@kronos:/var/lib/docker/graph# find . -iname nginx.conf
./6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/nginx.conf
./b6b4d496e1bcc1a2e12d60287aa85da6af5e6a4c1ab923a643e6bb795c8c62c2/layer/etc/nginx/nginx.conf

root@kronos:/var/lib/docker/graph# ls -R -halF 6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/
6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/:
total 16K
drwxr-xr-x  3 root root 4.0K Jul 30 23:56 ./
drwxr-xr-x 37 root root 4.0K Aug  5 23:55 ../
-rw-r--r--  1 root root 1.5K Aug  1 18:22 nginx.conf
drwxr-xr-x  2 root root 4.0K Jul 30 23:55 sites-available/

6b262906313b0c4bc4e7e475ca0d322e4a6bdc55f6dfa0711136aeaf3e6aa74b/layer/etc/nginx/sites-available:
total 16K
drwxr-xr-x 2 root root 4.0K Jul 30 23:55 ./
drwxr-xr-x 3 root root 4.0K Jul 30 23:56 ../
-rw-r--r-- 1 root root 3.0K Aug  1 18:23 static.bugvote.com
-rw-r--r-- 1 root root 1.4K Aug  1 18:21 www.bugvote.com

Yarr, treasure! :)

Leave a Reply

Your email address will not be published.