Recovering files from a fudged docker container

While cleaning up old images/containers, I accidentally removed an image that other images/containers depended on. After that, my trusty LEMP container broke. This isn’t supposed to be possible in the latest version of Docker, but if you find yourself in the same boat, here is what I did to recover some important files:

Note: I tried to undelete the image using extundelete, but it didn’t recover anything worthwhile for me.

The Problem

Oh boy!

Containers are transparent

Luckily, Docker images/containers are easier to work with than VMs. No crazy mounting tools, no image conversion (.vdi to .raw, then mounting over loopback anyone?)

Lets find a good container that we can rummage through:

The last hour of activity is me starting to rebuilding from scratch. Skip that. The highlighted group is where I think my good container is.

Code and SQL data is hosted outside of Docker, I mount them into the container, and use the container to host nginx, php-fpm, mysql, postgresql, redis, sphinxsearch, etc. The server  binaries can be reinstalled easily. The only thing I really want is those painstakingly tweaked config files!

Specifically, I want to recover this one nginx config file full of sexy image-resizing rewrite rules for etags/304/caching.

Let’s peak inside container 860e297944ca. It’s 5 days ago and from an unnamed image (one of many that I deleted). Here are the the last few lines from its life:

I remember manually turning the services off before stopping the container (hello OCD), so this may be our guy! But I can’t start it due to missing parent, so now what?

Because I erased the parent image (3262d0ef973b), the contents baked into the image are gone. But any modifications I made SINCE the image creation (ie: any tweaked config files) will be within the container’s layer! Cross your fingers and sudo up, then find a matching container in /var/lib/docker/ (the default docker install path on Ubuntu)

Let’s see if the container has anything good:

Oh yeah! The container has the /etc/nginx folder and the most recently modified nginx configs! I even found an php-xdebug config (debugging in PhpStorm mmm yeah!)

Copy and restore

To grab files out of a container, just copy them! I copied the config files, and used them to rebuild my LEMP container. Hooray!

Recovering files from within Docker is possible. Containers are not encrypted or obscured. It’s all there for easy pickings. Assuming you get lucky :)

Worst case scenario you could always fgrep/find your way through the entire /var/lib/docker/containers and /var/lib/docker/graph folders, digging for treasure:

Yarr, treasure! :)

Leave a Reply

Your email address will not be published. Required fields are marked *