Investigate on docker overlay volume size (+Solution)

[ignazio-bovo]

I have observed that on my server the /var/lib/docker used as mount point for the overlay volume is occupying 850GB that is all the available space in the vg0 partition for my partition schema.

Long term solution

Investigate on the docker overlay volume size culprits

Quick solution

Reconfigure docker so that the data-root folder points to a directory in a bigger volume (like the one used to store asset, usually several TBs large)

Moving the Docker overlay2 volume to another directory within the same filesystem involves a process that ensures all your Docker data is safely transferred to the new location without data loss. Here's how you can do it:

Step 1: Stop Docker Service

First, you need to stop Docker to ensure that no containers are running and no changes are being made to the data during the move. You can stop Docker using the following command:

sudo systemctl stop docker

Step 2: Copy the Docker Data Directory

Next, copy the entire Docker data directory (/var/lib/docker by default) to the new location. You can use the rsync command for this purpose as it's efficient and can resume interrupted transfers. Replace /new/docker/data/location with the path to where you want to move the Docker data.

sudo rsync -avzh /var/lib/docker/ /new/docker/data/location/

Step 3: Update Docker's Storage Location

After copying the data, you need to tell Docker to use the new location for its data. This is done by editing or creating the Docker daemon configuration file, usually found at /etc/docker/daemon.json. If this file does not exist, you will create it; if it does, add or modify the data-root attribute.

{
  "data-root": "/new/docker/data/location"
}

Make sure to replace /new/docker/data/location with the actual new path of your Docker data.

Step 4: Restart Docker Service

Now, restart Docker to apply the changes. Docker will now use the new directory for all its data, including overlay2.

sudo systemctl start docker

Step 5: Verify and Clean Up

After Docker starts, verify that your containers and images are correctly recognized and that everything operates as expected. If all is well, you can remove the old Docker data directory to free up space on your system. Ensure that everything is working correctly before doing this, as this step is irreversible.

sudo rm -rf /var/lib/docker/

Important Considerations

• Backups: Before proceeding with the move, it's highly recommended to backup your Docker data to avoid any loss.
• Docker Version: Ensure the Docker version is the same before and after the move to avoid compatibility issues.
• Disk Space: Ensure the destination directory has enough space to accommodate the Docker data.
• Permissions: Keep an eye on permissions and ownership of the new directory to match the original. Docker needs specific permissions to manage its data correctly.

By following these steps, you should be able to move the Docker overlay2 volume to a new directory on the same filesystem without losing your Docker containers and images.

[ignazio-bovo]

A quick inspection to the rsync logs for copying the data :

overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315019
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315121
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/131515
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315601
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315604
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315799
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315800
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315840
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315842
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315844
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315845
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315846
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316003
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316005
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316007
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316008
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316009
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316011
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316013
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316015
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316025
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316026
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316030
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316032
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316037
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/131617

And so on, probably revealing that the approach in the dwg-production repo needs to make the operator conscious about the distributor-data folder