Docker Issue : devmapper: Thin Pool is less than minimum required, use dm.min_free_space option to change behavior

Sometime while we build docker images or do any docker operation we might encounter thinpool space issue. like this:

devmapper: Thin Pool has 132480 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

If you do not want to use the device-tool utility, you can resize a loop-lvm thin pool manually using the following procedure.

In loop-lvm mode, a loopback device is used to store the data, and another to store the metadata. loop-lvm mode is only supported for testing, because it has significant performance and stability drawbacks.

If you are using loop-lvm mode, the output of docker info shows file paths for Data loop file and Metadata loop file:

[root@rvm-c431e558 proj_odi11g]# docker info |grep ‘loop file’
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `–storage-opt dm.thinpooldev` to specify a custom block storage device.
Data loop file: /data/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /data/lib/docker/devicemapper/devicemapper/metadata
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabledFollow these steps to increase the size of the thin pool.

In this example, the thin pool is 100 GB, and is increased to 200 GB.

List the sizes of the devices.

[root@rvm-c431e558 proj_odi11g]# ls -lh /data/lib/docker/devicemapper/devicemapper/
total 89G
-rw——- 1 root root 200G Mar 19 08:45 data
-rw——- 1 root root 2.0G Mar 19 08:45 metadata

Increase the size of the data file to 200 G using the truncate command, which is used to increase or decrease the size of a file. Note that decreasing the size is a destructive operation.

# truncate -s 200G /data/lib/docker/devicemapper/devicemapper/data
Verify the file size changed.

#  ls -lh /var/lib/docker/devicemapper/

total 1.2G
-rw——- 1 root root 100G Apr 14 08:47 data
-rw——- 1 root root 2.0G Apr 19 13:27 metadata

The loopback file has changed on disk but not in memory. List the size of the loopback device in memory, in GB. Reload it, then list the size again. After the reload, the size is 200 GB.

# echo $[ $(sudo blockdev –getsize64 /dev/loop0) / 1024 / 1024 / 1024 ]

100

# losetup -c /dev/loop0

# echo $[ $(sudo blockdev –getsize64 /dev/loop0) / 1024 / 1024 / 1024 ]

200

Reload the devicemapper thin pool.

a. Get the pool name first. The pool name is the first field, delimited by ` :`. This command extracts it.

#  dmsetup status | grep ‘ thin-pool ‘ | awk -F ‘: ‘ {‘print $1’}

docker-0:39-1566-pool

b. Dump the device mapper table for the thin pool.

#  dmsetup table docker-0:39-1566-pool

0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing

c. Calculate the total sectors of the thin pool using the second field of the output. The number is expressed in 512-k sectors. A 100G file has 209715200 512-k sectors. If you double this number to 200G, you get 419430400 512-k sectors.

d. Reload the thin pool with the new sector number, using the following three dmsetup commands.

# dmsetup suspend docker-0:39-1566-pool

#  dmsetup reload docker-0:39-1566-pool –table ‘0 419430400 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing’

# dmsetup resume docker-0:39-1566-pool

#dmsetup suspend docker-0:39-1566-pool

#dmsetup reload docker-0:39-1566-pool –table ‘0 419430400 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing’

#dmsetup resume docker-0:39-1566-pool

Now space is increased to 200GB.

4 thoughts on “Docker Issue : devmapper: Thin Pool is less than minimum required, use dm.min_free_space option to change behavior

  1. I found a few typos in that post and copy paste issues

    FYI here is a bash script that I’ve used to fix some of my hosts:
    “`
    #!/bin/bash -ex
    RAMDRIVE_SIZE=$(df /ephemeral -t tmpfs –output=size |tail -1)
    [ -z $RAMDRIVE_SIZE ] && echo “no ramdrive here” && exit 0
    [[ $RAMDRIVE_SIZE -lt 104857600]] && echo “ramdrive < thin pool: will not resize" && exit 1
    truncate -s ${RAMDRIVE_SIZE}k /ephemeral/docker/devicemapper/devicemapper/data
    TABLE_NAME=$(dmsetup status | grep ' thin-pool ' | awk -F ': ' {'print $1'})
    TABLE_CONTENT=( $(dmsetup table $TABLE_NAME) )
    TABLE_CONTENT[1]=$(( $RAMDRIVE_SIZE * 2))
    dmsetup suspend $TABLE_NAME
    dmsetup reload $TABLE_NAME –table "${TABLE_CONTENT[*]}"
    dmsetup resume $TABLE_NAME
    “`

  2. Graeme Frear says:

    Just a sidenote here: I had this error come up, the issue was space on the host, an ec2 instance, I cleaned all the dangling images and containers and it was fine again.

Leave a Reply