How to recover an orphaned Docker volume for a data container

Posted by Matt Bryson on 18-Jan-2016 20:58:05
Find me on:

So, last week I was writing a post about cleaning up after docker, and I got a bit carried away and ended up deleting a data container. The container was no longer running, but it's docker volume was still in use by other running containers.

Check out these articles for more info on Images and Containers, Volumes and Data Containers and why you would use them.

Thankfully, I didn't use the -v flag (which removes the containers volumes as well - although volumes in use wont get removed), so I still had all the data, I just had lost the container that created it.

The problem is, docker references the volume via the container when mounting it as a shared volume. As I had deleted the container, I could no longer reference the volume.

Thankfully there is a very simple solution to get the volume mapped back to the data container.

To jump to the solution, click here.

The Problem

When creating a data container, you first run a container that creates a volume, this doesn't need to run an application, it simply creates a volume, and then stops the container. But crucially it allows you to target that volume from other containers.

$ docker create -v /path/to/volume/on/container --name my_data_container busybox

You can see where the volume lives on the host by inspecting the container...

$ docker inspect my_data_container

You can see that it has one mount, at the path we set above.

"Mounts": [
 { 
   "Name": "f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f",
   "Source": "/var/lib/docker/volumes/f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f/_data",
   "Destination": "/path/to/volume/on/container",
   "Driver": "local",
   "Mode": "",
   "RW": true
  }
 ],
 

Now, lets mount this volume to another container...

$ docker run -d --name my_app --volumes-from my_data_container busybox

If we now inspect the my_app container we can see 2 volumes...
[bash]
$ docker inspect my_app
[/bash]

"Mounts": [
{
  "Name": "f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f",
  "Source": "/var/lib/docker/volumes/f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f/_data",
  "Destination": "/path/to/volume/on/container",
  "Driver": "local",
  "Mode": "",
  "RW": true
},
{
  "Name": "26eb04ec6137130f93b5345b2307d6654d0f65ac5974e36d3c572ca6eb44966a",
  "Source": "/var/lib/docker/volumes/26eb04ec6137130f93b5345b2307d6654d0f65ac5974e36d3c572ca6eb44966a/_data",
  "Destination": "/data",
  "Driver": "local",
  "Mode": "",
  "RW": true
 }
],

You can see that the first one here is the volume from my_data_container.

So my_app now has read / write access to this shared volume, as can any number of other containers.

So, what happens when you delete my_data_container?
When you try to run my_app you will get an error along the lines of

$ docker rm my_data_container
$ docker run -d --name my_app --volumes-from my_data_container busybox
Error response from daemon: no such id: my_data_container

Even though the volume is still on disk, the container has gone, so you can no longer mount it to another container.

How to remount the docker volume

To fix the issue, we need to do two things.

  1. Recreate our data container with a new volume
  2. Move/Copy all the files from the existing volume over to the new one

Firstly recreate your image, and then note down where its new data container is..

$ docker create -v /path/to/volume/on/container --name my_data_container busybox
$ docker inspect --format='' my_data_container
/var/lib/docker/volumes/09903445d22dc60120de9f5a826ff149c8a039413412915eb5174650742fc5e2/_data

Now, find where the OLD data container is, by inspecting a container that uses it.

$ docker inspect --format='' my_data_container

You will have to look at the list of mounts (there may be many) and try to work out which is the one you want by the Destination property.

[
{   "Name": "f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f",
    "Source": "/var/lib/docker/volumes/f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f/_data",
    "Destination": "/path/to/volume/on/container",
    "Driver": "local",
    "Mode": "",
    "RW": true
 },
 {
    "Name": "26eb04ec6137130f93b5345b2307d6654d0f65ac5974e36d3c572ca6eb44966a",
    "Source": "/var/lib/docker/volumes/26eb04ec6137130f93b5345b2307d6654d0f65ac5974e36d3c572ca6eb44966a/_data",
    "Destination": "/data",
    "Driver": "local",
    "Mode": "",
    "RW": true
 }
]

Here we can see that the one we want is the /path/to/volume/on/container volume.

So, now we simply copy all the files and folders from the Source of this old volume, into the Source of the new volume.

$ cp /var/lib/docker/volumes/f149c8a039413412915eb5174650742fc5e209903445d22dc60120de9f5a826f/_data/. /var/lib/docker/volumes/09903445d22dc60120de9f5a826ff149c8a039413412915eb5174650742fc5e2/_data/ -R

Now you can launch your containers as before with the --volumes-from my_data_container and all will be back to normal!

m

Topics: Deployment, Docker, Technology

About the author

Matt Bryson is an experienced systems engineer with over 15 years Development and Systems Operations experience.

NEW - Free eBook

Transforming your eDetail process. Click below to download the eBook.

New Call-to-action

Subscribe to Email Updates