Symlinking in Docker

Intro (and brief ‘what is a Symlink?’)

#

While using some self-hosted services in docker containers I have come across some situations where I’d like to symlink my files only to find they didn’t show up inside of the docker container

Symbolic Links (aka Symlinks or soft links) are a way to link a file or directory to another location. An example would be using ln with the -s option for a soft link:

ln -s <target> <link_name>

• Target: the file or directory you wish to link
• link_name: the new filepath where the file/directory will be accessible from.

Using symlinks you are able to have a file in two different locations but it is still only one copy on your drive so you aren’t doubling the amount of space it takes up. These symlinks are a special type of file that just points to the location you specify.

The Problem

#

When I was creating symlinks I was creating them based on the filepath of the device they’re stored at, but not the docker container that needs to see them. For example, lets say my audiobooks are organized in two different places:

/hdd/
├─── storage/audiobooks/
│   ├── librivox/
│   │   ├── frankenstein/frankenstein.m4b
│   │   └── picture-dorian-gray/gray.m4b
│   ├── purchased/
│   │   ├── audiobook-1/audiobook-1.m4b
│   │   └── audiobook-2/audiobook-2.m4b
│   └── self-recordings/  <-- desired new location ('link_name')
│
└── user-files/user1/audacity/my-audiobooks/ <-- symlink target
    ├── my-audiobook-1/
    │   └── my-audiobook-1.m4a
    └── my-audiobook-2/
        └── my-audiobook-2.m4a

Here I keep librivox (public domain) audiobooks and purchased audiobooks in a storage filepath, while I also have some self-recorded audiobooks. For practical purposes I want to keep these in my user files so they are accessible on my main PC but I also want them accessible in audiobookshelf. Using the following symlink:

ln -s /hdd/user-files/user1/audacity/my-audiobooks /hdd/storage/audiobooks/self-recordings

Would seem like it would work when viewing the files on my server, however, when mounting these files within the audiobookshelf docker container, the container wouldn’t be able to read the file reference. For example, if I had my storage mounted in docker as:

services:
    service-name:
        volumes:
        - /hdd/storage/audiobooks:/audiobooks

This means from the docker perspective the directory structure looks like:

/audiobooks/
├── librivox/
│   ├── frankenstein/frankenstein.m4b
│   └── picture-dorian-gray/gray.m4b
├── purchased/
│   ├── audiobook-1/audiobook-1.m4b
│   └── audiobook-2/audiobook-2.m4b
└── self-recordings/ ---> symlink target /hdd/user-files/user1/audacity/my-audiobooks

The container would see the /audiobooks/self-recordings symlink pointing to hdd/user-files/user1/audacity/my-audiobooks/ but this filepath wouldn’t exist within the docker container. My docker container would just have the path(s) I mounted. This means there are two problems with symlinks in docker containers:

1. The container needs access to the target file/directory.
2. The symlink needs to be for the filepath within the docker container.

Solution

#

1. Make the files you are trying to symlink are accessible by the docker container. In my above example this would mean:

   services:
       service-name:
           volumes:
               - /hdd/storage/audiobooks:/audiobooks
   
               # adding the target directory
               - /hdd/user-files/user1/audacity/my-audiobooks:/hdd/user-files/user1/audacity/my-audiobooks

   Tip

   Since these files will be referenced via symlinks then their exact path is only relevant for symlinks. This means we can simply things by ensuring the target directory filepath is mounted so the filepath is the same both in and out of the container.

2. Symlink using the filepath from within the docker container. If you followed the above tip then this would be just like symlinking normally:

   ln -s /hdd/user-files/user1/audacity/my-audiobooks /hdd/storage/audiobooks/self-recordings

   Now, The docker container will see /audiobooks/self-recordings symlink pointing to hdd/user-files/user1/audacity/my-audiobooks/, and since the docker container has that filepath it can resolve, then it now correctly points to your files.

   Note

   If you mounted the target directory under a different filepath, like /hdd/user-files/user1/audacity/my-audiobooks:/my-audiobooks then your symlink would need to be done from the container’s perspective. Meaning you would need to symlink via ln -s /my-audiobooks /hdd/storage/audiobooks/self-recordings in this example since the target within the container is /my-audiobooks.

Alternative solution

#

I generally recommend the first solution, but include this as it can be helpful in some cases.

An alternative is to use what’s called a hard link. Hard links don’t point to a filepath but instead point directly to the actual data on the disk. To use them though, you need to link the specific individual files (or write a simple script to iterate over the contents of a directory), and you also need to create the file structure if it doesn’t already exist. The syntax is similar to symlinks, just without the -s option.

# create directory if it doesn't already exist
mkdir -p /hdd/storage/audiobooks/self-recordings/my-audiobook-1

# hard link individual file
ln /hdd/user-files/user1/audacity/my-audiobooks/my-audiobook-1/my-audiobook-1.m4a /hdd/storage/audiobooks/self-recordings/my-audiobook-1/my-audiobook-1.m4a

Advantages:

• Does not require mounting an additional volume in docker

Caveats:

• Cannot hard link a directory like soft links (need to do each file individually). This can be fine for directory contents that don’t change but if files are being added frequently then the soft link solution would be better.
• Need to create directory structure before linking