So today we're gonna benchmark between docker-volume (bind to docker-managed volume), bind/mount-fs (binding to host filesystem), and tmpfs. Which one can be the fastest? here's the docker compose:
version: "3.7"
services:
web:
image: ubuntu
command: "sleep 3600"
volumes:
- ./temp1:/temp1 # mountfs
- temp2:/temp2 # dockvol
- temp3:/temp3 # tmpfs
volumes:
temp2:
temp3:
driver_opts:
type: tmpfs
device: tmpfs
The docker compose file is on the sibling directory as data-root of docker to ensure using the same SSD. First benchmark we're gonna clone from this repository, then run copy, create 100 small files, then do 2 sequential write (small and large), here's the result of those (some steps not pasted below, eg. removing file when running benchmark twice for example):
apt install git g++ make time
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
cd /temp3 # tmpfs
git clone https://github.com/nikolausmayer/file-IO-benchmark.git
### copy small files
time cp -R /temp3/file-IO-benchmark /temp2 # dockvol
CPU: 0.00s Real: 1.02s RAM: 2048KB
time cp -R /temp3/file-IO-benchmark /temp1 # bindfs
CPU: 0.00s Real: 1.00s RAM: 2048KB
### create 100 x 10MB files
cd /temp3/file*
time make data # tmpfs
CPU: 0.41s Real: 0.91s RAM: 3072KB
cd /temp2/file*
time make data # dockvol
CPU: 0.44s Real: 1.94s RAM: 2816KB
cd /temp1/file*
time make data # mountfs
CPU: 0.51s Real: 1.83s RAM: 2816KB
### compile
cd /temp3/file*
time make # tmpfs
CPU: 2.93s Real: 3.23s RAM: 236640KB
cd /temp2/file*
time make # dockvol
CPU: 2.94s Real: 3.22s RAM: 236584KB
cd /temp1/file*
time make # mountfs
CPU: 2.89s Real: 3.13s RAM: 236300KB
### sequential small
cd /temp3 # tmpfs
time dd if=/dev/zero of=./test.img count=10 bs=200M
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 0.910784 s, 2.3 GB/s
cd /temp2 # dockvol
time dd if=/dev/zero of=./test.img count=10 bs=200M
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 2.26261 s, 927 MB/s
cd /temp1 # mountfs
time dd if=/dev/zero of=./test.img count=10 bs=200M
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 2.46954 s, 849 MB/s
### sequential large
cd /temp3 # tmpfs
time dd if=/dev/zero of=./test.img count=10 bs=1G
10737418240 bytes (11 GB, 10 GiB) copied, 4.95956 s, 2.2 GB/s
cd /temp2 # dockvol
time dd if=/dev/zero of=./test.img count=10 bs=1G
10737418240 bytes (11 GB, 10 GiB) copied, 81.8511 s, 131 MB/s
10737418240 bytes (11 GB, 10 GiB) copied, 44.2367 s, 243 MB/s
# ^ running twice because I'm not sure why it's so slow
cd /temp1 # mountfs
time dd if=/dev/zero of=./test.img count=10 bs=1G
10737418240 bytes (11 GB, 10 GiB) copied, 12.7516 s, 842 MB/s
The conclusion is, docker volume is a bit faster (+10%) for sequential small, but significantly slower (-72% to -84%) for large sequential files compared to bind/mount-fs, for the other cases seems there's no noticeable difference. I always prefer bind/mount-fs over docker volume because of safety, for example if you accidentally run docker volume rm $(docker volume ls -q) this would delete all your docker volume (I did this multiple times on my own dev PC), also you can easily backup/rsync/copy/manage files if using bind/mount-fs. For other cases, that you don't care whether losing files or not and need high performance (as long as your ram is enough), just use tmpfs.
version: "3.7"
services:
web:
image: ubuntu
command: "sleep 3600"
volumes:
- ./temp1:/temp1 # mountfs
- temp2:/temp2 # dockvol
- temp3:/temp3 # tmpfs
volumes:
temp2:
temp3:
driver_opts:
type: tmpfs
device: tmpfs
The docker compose file is on the sibling directory as data-root of docker to ensure using the same SSD. First benchmark we're gonna clone from this repository, then run copy, create 100 small files, then do 2 sequential write (small and large), here's the result of those (some steps not pasted below, eg. removing file when running benchmark twice for example):
apt install git g++ make time
alias time='/usr/bin/time -f "\nCPU: %Us\tReal: %es\tRAM: %MKB"'
cd /temp3 # tmpfs
git clone https://github.com/nikolausmayer/file-IO-benchmark.git
### copy small files
time cp -R /temp3/file-IO-benchmark /temp2 # dockvol
CPU: 0.00s Real: 1.02s RAM: 2048KB
time cp -R /temp3/file-IO-benchmark /temp1 # bindfs
CPU: 0.00s Real: 1.00s RAM: 2048KB
### create 100 x 10MB files
cd /temp3/file*
time make data # tmpfs
CPU: 0.41s Real: 0.91s RAM: 3072KB
cd /temp2/file*
time make data # dockvol
CPU: 0.44s Real: 1.94s RAM: 2816KB
cd /temp1/file*
time make data # mountfs
CPU: 0.51s Real: 1.83s RAM: 2816KB
### compile
cd /temp3/file*
time make # tmpfs
CPU: 2.93s Real: 3.23s RAM: 236640KB
cd /temp2/file*
time make # dockvol
CPU: 2.94s Real: 3.22s RAM: 236584KB
cd /temp1/file*
time make # mountfs
CPU: 2.89s Real: 3.13s RAM: 236300KB
### sequential small
cd /temp3 # tmpfs
time dd if=/dev/zero of=./test.img count=10 bs=200M
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 0.910784 s, 2.3 GB/s
cd /temp2 # dockvol
time dd if=/dev/zero of=./test.img count=10 bs=200M
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 2.26261 s, 927 MB/s
cd /temp1 # mountfs
time dd if=/dev/zero of=./test.img count=10 bs=200M
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 2.46954 s, 849 MB/s
### sequential large
cd /temp3 # tmpfs
time dd if=/dev/zero of=./test.img count=10 bs=1G
10737418240 bytes (11 GB, 10 GiB) copied, 4.95956 s, 2.2 GB/s
cd /temp2 # dockvol
time dd if=/dev/zero of=./test.img count=10 bs=1G
10737418240 bytes (11 GB, 10 GiB) copied, 81.8511 s, 131 MB/s
10737418240 bytes (11 GB, 10 GiB) copied, 44.2367 s, 243 MB/s
# ^ running twice because I'm not sure why it's so slow
cd /temp1 # mountfs
time dd if=/dev/zero of=./test.img count=10 bs=1G
10737418240 bytes (11 GB, 10 GiB) copied, 12.7516 s, 842 MB/s
The conclusion is, docker volume is a bit faster (+10%) for sequential small, but significantly slower (-72% to -84%) for large sequential files compared to bind/mount-fs, for the other cases seems there's no noticeable difference. I always prefer bind/mount-fs over docker volume because of safety, for example if you accidentally run docker volume rm $(docker volume ls -q) this would delete all your docker volume (I did this multiple times on my own dev PC), also you can easily backup/rsync/copy/manage files if using bind/mount-fs. For other cases, that you don't care whether losing files or not and need high performance (as long as your ram is enough), just use tmpfs.