Backup into permanent_storage fails with `no space left on device`

[christoph-maurer]

cloud@database:/app/production$ ./database.mk backup DIR=/app/data/permanent_storage/26070101_database_backup
mkdir --parents "/app/data/permanent_storage/26070101_database_backup"
mkdir --parents "./.content-addressable-storage"
docker compose up \
	--no-build \
	--no-recreate \
	--wait \
	database
[+] up 7/7
 ✔ Image pgautoupgrade/pgautoupgrade:18-trixie      Pulled                                                                                          26.8s
 ✔ Image postgres:18.4-trixie                       Pulled                                                                                          26.8s
 ✔ Container database_production-reverse_proxy-1    Running                                                                                          0.0s
 ✔ Container database_production-frontend-1         Running                                                                                          0.0s
 ✔ Container database_production-backend-1          Running                                                                                          0.0s
 ✔ Container database_production-database-1         Healthy                                                                                          1.1s
 ✔ Container database_production-upgrade_postgres-1 Exited                                                                                           1.0s
docker compose exec \
	--no-tty \
	database \
	pg_dump \
		--clean \
		--if-exists \
		--username="postgres" \
		--dbname="xbase" \
	| gzip \
	> "/app/data/permanent_storage/26070101_database_backup/postgresql_dumpall.gz"
docker compose run \
	--rm \
	--no-deps \
	--no-TTY \
	--volume "/app/data/permanent_storage/26070101_database_backup":/backup \
	--volume "./.content-addressable-storage":/.content-addressable-storage \
	backend \
	bash -o errexit -o errtrace -o nounset -o pipefail -c ' \
		if [[ "production" == "development" ]]; then \
			cd /app/src/files ; \
		else \
			cd /app/files ; \
		fi ; \
		mkdir --parents /backup/files ; \
		find . -type f -printf "%f\n" | while read -r original_file_name; do \
			echo "Backing up file: ${original_file_name}" ; \
			hash=$(sha256sum ./"${original_file_name}" | cut --delimiter=" " --fields=1) ; \
			content_addressed_file_path="/.content-addressable-storage/${hash}" ; \
			if [[ ! -f "${content_addressed_file_path}" ]]; then \
					cp "${original_file_name}" "${content_addressed_file_path}" ; \
					echo "Stored new content-addressed file: ${content_addressed_file_path}" ; \
			else \
					echo "File already exists in content-addressable storage: ${content_addressed_file_path}" ; \
			fi ; \
			ln --symbolic "${content_addressed_file_path}" /backup/files/"${original_file_name}" ; \
		done \
	'
[+]  4/4te 4/4
 ✔ Container database_production-reverse_proxy-1 Running                                                                                             0.0s
 ✔ Container database_production-frontend-1      Running                                                                                             0.0s
 ✔ Container database_production-backend-1       Running                                                                                             0.0s
 ✔ Container database_production-database-1      Running                                                                                             0.0s
Container database_production-backend-run-a224777edda1 Creating 
Container database_production-backend-run-a224777edda1 Created 
Backing up file: 9bc379c3-87ab-41ef-b3e0-fa8641e2136d.json
Stored new content-addressed file: /.content-addressable-storage/16510593788ce966eeaa0d2d0c8e0e4286fe5468e56ff5d2b0fa51833dff817b
Backing up file: 019ef4f8-1648-773d-bd42-ebf9ea0fd2b8.xml
Stored new content-addressed file: /.content-addressable-storage/b625f2b208157c8bc5bed25f0b8d603fde146e041a26370ce7a16145a305c97a
Backing up file: 9e2089ba-9a24-4d3f-a3c3-b4a63a4e5ae7.json
Stored new content-addressed file: /.content-addressable-storage/f4751a998391440ef91295aae15b069415a8e4db4bc6f6d1c9eed85b161bc07b
Backing up file: 41fe2367-f109-44ff-af45-83948b8145b7
Stored new content-addressed file: /.content-addressable-storage/7c4054dcbe120c719074f886aeb103714264b573c1239c137ce68aa20ef285ea
Backing up file: 51df026e-3019-4d5c-8568-f5490ece61c9.json
Stored new content-addressed file: /.content-addressable-storage/218f878fc537f0952d3cad2f950b0c6113402e90793925d55e4b59216e2b466e
Backing up file: 18fb7aa6-d345-4c51-845a-f1a42a204a01
Stored new content-addressed file: /.content-addressable-storage/086652ba9d5e85b362caa94d5a4efc2b63cedde9fbfe6539d50b4ccdf72704eb
Backing up file: 4d903c4f-682f-4c1a-90c2-a3464f731806
Stored new content-addressed file: /.content-addressable-storage/af2e592bbca600319e9a857a857c12fd75ba951b91ecbdb5260b35b31c915b67
Backing up file: b91fc007-bfc2-4a1b-9f24-851b0553fc31
Stored new content-addressed file: /.content-addressable-storage/dfabdbcc3224a4ff801987051eac75b6fbf04db860fcbc25337a6d54511b5836
Backing up file: 019ef505-0b97-7fe7-8637-43e8dda0b3d2.xml
Stored new content-addressed file: /.content-addressable-storage/e58ac7e527c9a17fad7ca26d3caef1c39294f76fd1462eac2317f0178826583f
Backing up file: 9cc31145-0fa9-4087-83ba-f990a1df2b00
Stored new content-addressed file: /.content-addressable-storage/11d97bdff02e4a057c278ab21cea49f012f134630a9e149f1381273c55a49ced
Backing up file: 8f22407c-792a-4eb2-b748-1d0d5e22a8c9
Stored new content-addressed file: /.content-addressable-storage/3ea38dd5622279d36df11b661b7d4cec6f9ff9dd5c0a52c9045c8807ee91ae31
Backing up file: 019ef505-bccc-7bdc-b663-d65eaa42e8bb.xml
Stored new content-addressed file: /.content-addressable-storage/b474170bc9ee7fc0842bdb8bd9e1d8ac0249fbc8acf2c13ce34c8812ce0974ca
Backing up file: 8c3acdcd-dc41-4df8-a5fb-7465619790a7
Stored new content-addressed file: /.content-addressable-storage/5101500effc712b2f56aeed5ebf8118507f8d426a101859cce330dbd6926bd55
Backing up file: f831a60f-b278-4a22-b02f-e150fb337fe9.json
Stored new content-addressed file: /.content-addressable-storage/8b1f7ad05e676e581fcff632c7b037c52e67e10c99f6177534e319aa6e821e98
Backing up file: e45b4ac8-4264-42d1-9624-09bea2954cb2
Stored new content-addressed file: /.content-addressable-storage/386ec2e120a168adb18bc336d8047adb125d57dce5bd7a970277e40c8414bf33
Backing up file: 48840e82-7e6b-4915-9620-96c364506e24
Stored new content-addressed file: /.content-addressable-storage/0158f667f8b8ded9bb34afef1723882eca8d2d85b2dd6b1f7c9594068987f40a
Backing up file: 92ba8c51-5846-436d-ac09-972eea2d32de
Stored new content-addressed file: /.content-addressable-storage/d0618d7a083394f5eb5f4b0b62690024f0b890e06a3dc0e00769209a2983459c
Backing up file: 6f8c8185-eb68-4c0e-89b4-6a2677d3f6e6.json
Stored new content-addressed file: /.content-addressable-storage/5cf99df9319007041da629397f676298e43bef6e08568c4b2f01fe3fddfd1322
Backing up file: 2d40b285-79d4-4ec6-8d46-767bd9b0f249
Stored new content-addressed file: /.content-addressable-storage/3de8b46624db0f442d6a70f299148352f754196bf9fafcb71282463d73472187
Backing up file: 019ef4ff-1873-7d75-8dc3-72c6eeea5c14.xml
Stored new content-addressed file: /.content-addressable-storage/aaad2a9789963ca9484fcf9ae22e7f36dae1338cce0d765290c0c7aca33b8f56
Backing up file: ab3db916-be28-45f1-a03b-7a5e749e8e84
Stored new content-addressed file: /.content-addressable-storage/828c855a964f9efb7f93a7824631c819fcad25090db7e97b6dcf35946310ea07
Backing up file: dd841d18-1e13-49bb-a216-5c93be6fe5be
Stored new content-addressed file: /.content-addressable-storage/55acd2679232a1e16662f184877f4936d1545000f75eacf9c48b4eaffd7699f0
Backing up file: 9a9abb01-ec93-4a8b-a602-93b908d6dbd5
Stored new content-addressed file: /.content-addressable-storage/969e830357f6472363136dd35435508d3f798fc41e5b4c092cc99db30bc8f0e7
Backing up file: a3051f99-766a-4b6b-b137-92e0e51901f6
Stored new content-addressed file: /.content-addressable-storage/a4445107f0627ae6cdb7f28ea7ab8011c4cf992b91de52e16d5d7f636ae362d6
Backing up file: 3ae1f4d1-689f-4bcd-b9d3-35e8c88d8ab1
File already exists in content-addressable storage: /.content-addressable-storage/0158f667f8b8ded9bb34afef1723882eca8d2d85b2dd6b1f7c9594068987f40a
Backing up file: 019ef4e9-a08b-729d-b07d-5ee5ef0d6953.xml
Stored new content-addressed file: /.content-addressable-storage/60dd247af7c03c1469ebc42531fffe9d34e8ae89e516a142ea9228478d5ebe7a
Backing up file: 9942026b-b9a2-4b4e-b9a0-5ba64f5a2eef
Stored new content-addressed file: /.content-addressable-storage/de98f927e37a481d4595d7621b8e49655e8ecec1f077c10be9e0b59043c7c77e
Backing up file: 019ed5b8-dbe9-7a69-97bc-47415af86624.xml
File already exists in content-addressable storage: /.content-addressable-storage/aaad2a9789963ca9484fcf9ae22e7f36dae1338cce0d765290c0c7aca33b8f56
Backing up file: 019ef4c7-923f-727d-a087-d85ad378e6ee.xml
Stored new content-addressed file: /.content-addressable-storage/509d070c6ff8239bacc281c6d9d6fccd3a500ff9ace04fac9e840e9b1e389c55
Backing up file: 007c9a32-377f-4393-bc68-8cfad4bafb09
Stored new content-addressed file: /.content-addressable-storage/180c955193ac359206d1c415f35a5d8fcf81b920763ddf5ff96844930386b170
Backing up file: 6b853c7a-f4ba-47ae-978c-ec1064f57609
Stored new content-addressed file: /.content-addressable-storage/18cab7f0f365168be42a6b4bf5d68aa386bb15ac9f17db1b5a7c2b3c794c764b
Backing up file: 019ef4fa-5427-720c-9c20-ddb7188dd874.xml
cp: error writing '/.content-addressable-storage/3031af06878ca731c8545b4efdf8338fd500ced53ad50238d2cee326e2edcd26': No space left on device

make: *** [database.mk:108: backup] Error 1

[christoph-maurer]

root is full:

cloud@database:/app/production$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           389M  1.2M  388M   1% /run
/dev/sdc1       9.8G  9.8G     0 100% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.9G     0  1.9G   0% /tmp
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
/dev/sda1       492G   60G  407G  13% /home/cloud/data
tmpfs           1.0M     0  1.0M   0% /run/credentials/getty@tty1.service

[christoph-maurer]

• Deleting the files in permanent_storage and deleting unused docker images does not help, because they are stored on DATADISK, not ROOT.
• 6.6 of 10 GB are in /home/cloud/production/.content-addressable-storage.
• The folder /home/cloud/production/.content-addressable-storage was created by ./database.mk backup.

cloud@database:~/production/.content-addressable-storage$ ls -alh
total 6.6G
drwxr-xr-x  2 cloud cloud 4.0K Jul  1 09:51 .
drwxr-xr-x 10 cloud cloud 4.0K Jul  1 09:49 ..
-rw-r--r--  1 root  root  2.2K Jul  1 09:50 0158f667f8b8ded9bb34afef1723882eca8d2d85b2dd6b1f7c9594068987f40a
-rw-r--r--  1 root  root   226 Jul  1 09:49 086652ba9d5e85b362caa94d5a4efc2b63cedde9fbfe6539d50b4ccdf72704eb
-rw-r--r--  1 root  root   216 Jul  1 09:50 11d97bdff02e4a057c278ab21cea49f012f134630a9e149f1381273c55a49ced
-rw-r--r--  1 root  root  1.1M Jul  1 09:49 16510593788ce966eeaa0d2d0c8e0e4286fe5468e56ff5d2b0fa51833dff817b
-rw-r--r--  1 root  root  1.1K Jul  1 09:50 180c955193ac359206d1c415f35a5d8fcf81b920763ddf5ff96844930386b170
-rw-r--r--  1 root  root   719 Jul  1 09:50 18cab7f0f365168be42a6b4bf5d68aa386bb15ac9f17db1b5a7c2b3c794c764b
-rw-r--r--  1 root  root  1.4K Jul  1 09:49 218f878fc537f0952d3cad2f950b0c6113402e90793925d55e4b59216e2b466e
-rw-r--r--  1 root  root  4.1G Jul  1 09:52 3031af06878ca731c8545b4efdf8338fd500ced53ad50238d2cee326e2edcd26
-rw-r--r--  1 root  root  5.6M Jul  1 09:50 386ec2e120a168adb18bc336d8047adb125d57dce5bd7a970277e40c8414bf33
-rw-r--r--  1 root  root  566K Jul  1 09:50 3de8b46624db0f442d6a70f299148352f754196bf9fafcb71282463d73472187
-rw-r--r--  1 root  root  1.7M Jul  1 09:50 3ea38dd5622279d36df11b661b7d4cec6f9ff9dd5c0a52c9045c8807ee91ae31
-rw-r--r--  1 root  root  2.3G Jul  1 09:50 509d070c6ff8239bacc281c6d9d6fccd3a500ff9ace04fac9e840e9b1e389c55
-rw-r--r--  1 root  root  2.3K Jul  1 09:50 5101500effc712b2f56aeed5ebf8118507f8d426a101859cce330dbd6926bd55
-rw-r--r--  1 root  root  1.7M Jul  1 09:50 55acd2679232a1e16662f184877f4936d1545000f75eacf9c48b4eaffd7699f0
-rw-r--r--  1 root  root  1.1M Jul  1 09:50 5cf99df9319007041da629397f676298e43bef6e08568c4b2f01fe3fddfd1322
-rw-r--r--  1 root  root   30M Jul  1 09:50 60dd247af7c03c1469ebc42531fffe9d34e8ae89e516a142ea9228478d5ebe7a
-rw-r--r--  1 root  root   719 Jul  1 09:49 7c4054dcbe120c719074f886aeb103714264b573c1239c137ce68aa20ef285ea
-rw-r--r--  1 root  root   718 Jul  1 09:50 828c855a964f9efb7f93a7824631c819fcad25090db7e97b6dcf35946310ea07
-rw-r--r--  1 root  root  1.1M Jul  1 09:50 8b1f7ad05e676e581fcff632c7b037c52e67e10c99f6177534e319aa6e821e98
-rw-r--r--  1 root  root  827K Jul  1 09:50 969e830357f6472363136dd35435508d3f798fc41e5b4c092cc99db30bc8f0e7
-rw-r--r--  1 root  root   282 Jul  1 09:50 a4445107f0627ae6cdb7f28ea7ab8011c4cf992b91de52e16d5d7f636ae362d6
-rw-r--r--  1 root  root   38M Jul  1 09:50 aaad2a9789963ca9484fcf9ae22e7f36dae1338cce0d765290c0c7aca33b8f56
-rw-r--r--  1 root  root  2.4K Jul  1 09:49 af2e592bbca600319e9a857a857c12fd75ba951b91ecbdb5260b35b31c915b67
-rw-r--r--  1 root  root   32M Jul  1 09:50 b474170bc9ee7fc0842bdb8bd9e1d8ac0249fbc8acf2c13ce34c8812ce0974ca
-rw-r--r--  1 root  root   31M Jul  1 09:49 b625f2b208157c8bc5bed25f0b8d603fde146e041a26370ce7a16145a305c97a
-rw-r--r--  1 root  root  6.0M Jul  1 09:50 d0618d7a083394f5eb5f4b0b62690024f0b890e06a3dc0e00769209a2983459c
-rw-r--r--  1 root  root   718 Jul  1 09:50 de98f927e37a481d4595d7621b8e49655e8ecec1f077c10be9e0b59043c7c77e
-rw-r--r--  1 root  root  1.8K Jul  1 09:49 dfabdbcc3224a4ff801987051eac75b6fbf04db860fcbc25337a6d54511b5836
-rw-r--r--  1 root  root   36M Jul  1 09:50 e58ac7e527c9a17fad7ca26d3caef1c39294f76fd1462eac2317f0178826583f
-rw-r--r--  1 root  root  1.1M Jul  1 09:49 f4751a998391440ef91295aae15b069415a8e4db4bc6f6d1c9eed85b161bc07b

[christoph-maurer]

Options:

• Understand the targets database./database.mk backup and restore in relation to machine./maintenance.mk backup, restore and prune-backups.
• Deploy the current version of develop.
• Test backup and restore in development.
• Set up a new ROOT for database with more space.
• Move the temporary folder /home/cloud/production/.content-addressable-storage of ./database.mk backup to /app/data.

[christoph-maurer]

There are 46 files in /app/data/backups/.content-addressable-storage content-addressable-storage-hashes.txt and 55 symlinks in /app/data/backups/2026-07-01_03_00_02/files symlink_target_hashes.txt because some symlinks point to the same file.

[christoph-maurer]

The changes of yesterday require that the new parameter CONTENT_ADDRESSABLE_STORAGE is set to a path on /app/data on the DATADISK volume. Without that, it defaults to /app/production and fills quickly the 10 GB of the ROOT volume. Such a folder /app/production/.content-addressable-storage can be deleted safely, because the original files live in the backend container in /app/files and are backuped into /app/data/backups/.content-addressable-storage. ./database.mk backup DIR=/app/data/backups/$(date +"%Y-%m-%d_%H_%M_%S") CONTENT_ADDRESSABLE_STORAGE=/app/data/backups/.content-addressable-storage works fine.