Introduction
I woke up one morning to a full disk with no space left and a sticky error that I could not diagnose without the help of my restless friend Claude. I had forgotten to set the docker log limits which snowballed into the file system refusing to apply any changes despite being properly mounted read/write. I managed to fix this issue without restarting and this article will be my sticky note so that I can come back to it in case this ever happens again.
My setup
So i’m running this VM at a hyperscaler on an external block device mounted as a CIFS container. That container holds a luks.img file which needs to be mounted with a loopback device so that the kernel can use it as if it were a raw block device (loop0). Dm-crypt then decrypts the data as luks.img since it’s encrypted at rest with LUKS. The kernel sets up another virtual block device — /dev/mapper/storbackup — that transparently encrypts writes and decrypts reads as they pass through, using the key I entered manually on boot. The kernel then takes over by reading and writing to the resulting — /dev/mapper/storbackup — ext4 partition
1
2
3
4
5
6
7
8
9
10
11
12
13
Storage Box / CIFS
↓
luks.img
↓
loop0
↓
LUKS / dm-crypt
↓
/dev/mapper/storbackup
↓
ext4
↓
decrypted data
The incident
No matter what I did, I could not edit any of the files on my directory /mnt/storage-box-1/decrypted/garage
1
touch: cannot touch '/mnt/storage-box-1/decrypted/garage/test': Input/output error
Ext4 doesn’t write metadata changes straight to their final location on disk. Instead, it first writes a description of the change to a reserved journal area (managed by jbd2), and only afterward commits the change to its real location.
Clearing the ring buffer used by the kernel to report its errors and triggering the error strangely did not trigger any errors at all.
1
2
3
4
5
6
7
8
root@vm# dmesg -c > /tmp/before.log # clears the buffer so the next error is easy to spot
touch /mnt/storage-box-1/decrypted/garage/test
dmesg -T | tail -30
touch: cannot touch '/mnt/storage-box-1/decrypted/garage/test': Input/output error
root@vm# dmesg -T | grep -iE "ext4|i/o error|EXT4-fs|remount-ro"
root@vm# dmesg -T | tail -30
root@vm# echo "exit code: $?"
exit code: 0
Stat’ing the mount and listing the contents showed no obvious clue
1
2
3
4
5
6
7
8
9
root@vm: garage# stat /mnt/storage-box-1/encrypted/luks.img
File: /mnt/storage-box-1/decrypted/garage
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fc00h/64512d Inode: 55050241 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2026-09-05 10:29:57.245832360 +0200
Modify: 2026-08-05 10:31:43.038652755 +0200
Change: 2026-08-05 10:31:43.038652755 +0200
Birth: 2026-05-31 09:37:31.532005566 +0200
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
root@vm: garage# ls -la /mnt/storage-box-1/decrypted/garage
stat /mnt/storage-box-1/decrypted/garage
ls: /mnt/storage-box-1/decrypted/garage: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/garage-local.toml: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/unlock.sh: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/data: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/garage.toml: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/readme: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/readme-luks.md: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/docker-compose.yml: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/garage: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/.: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/cluster_layout: Input/output error
ls: /mnt/storage-box-1/decrypted/garage/..: Input/output error
total 25992
drwxr-xr-x 3 root root 4096 Aug 5 10:31 .
drwxr-xr-x 3 root root 4096 May 31 09:37 ..
-rw-r--r-- 1 root root 1299 Jul 9 20:31 cluster_layout
drwxr-xr-x 4 root root 4096 Oct 23 2025 data
-rw-r--r-- 1 root root 897 Jul 9 20:00 docker-compose.yml
-rwxr--r-- 1 root root 26572704 Sep 15 2025 garage
-rw-r--r-- 1 root root 650 Apr 16 22:20 garage-local.toml
-rw-r--r-- 1 root root 714 Jul 9 21:32 garage.toml
-rw-r--r-- 1 root root 2219 Aug 5 10:31 readme
-rw-r--r-- 1 root root 422 May 31 10:31 readme-luks.md
-rwxr--r-- 1 root root 422 Jul 15 07:41 unlock.sh
1
2
3
4
5
6
root@vm# df -h /mnt/storage-box-1/decrypted
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/storbackup 984G 491G 443G 53% /mnt/storage-box-1/decrypted
root@vm# df -i /mnt/storage-box-1/decrypted
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/storbackup 65536000 585347 64950653 1% /mnt/storage-box-1/decrypted
Finding a clue
The clue finally arrived when running the command tune2fs
1
2
3
4
5
6
7
8
9
10
11
12
13
root@vm# tune2fs -l /dev/tune2fs -l /dev/mapper/storbackup | grep -iE "error|mount count|state"
Filesystem state: clean with errors
Errors behavior: Continue
Mount count: 3
Maximum mount count: -1
FS Error count: 2
First error time: Thu Jun 18 13:07:26 2026
First error function: ext4_journal_check_start
First error line #: 84
Last error time: Mon Aug 24 11:30:07 2026
Last error function: ext4_journal_check_start
Last error line #: 84
Last error err: EIO
I had run the command multiple times on september the 5th and gotten no error message on the “dmesg -T” output and the block device’s meta data “tune2fs” recorded the “Last error time” as “Mon Aug 24 11:30:07 2026” which implies that the jbd2 journal aborted and stopped. I would need to unmount it, running e2fsck to check/fix the journal in case it got corrupted and remount it
The fix
Unmounting was not working because I was sitting on the directory being unmounted and because some processes were using files on that mount
1
2
3
4
5
6
root@vm# fuser -vm /mnt/storage-box-1/decrypted
USER PID ACCESS COMMAND
/mnt/storage-box-1/decrypted:
root kernel mount /mnt/storage-box-1/decrypted
root 3067709 ..c.. bash
fedora-backup 3067928 ..c.. bash
I tried to run a SIGINT on those processes in vain and the SIGKILL sealed the deal for me.
1
2
3
4
root@vm# fuser -vm /mnt/storage-box-1/decrypted
USER PID ACCESS COMMAND
/mnt/storage-box-1/decrypted:
root kernel mount /mnt/storage-box-1/decrypted
And now the unmount, e2fsck ran successfully and the error is gone
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
root@vm# umount /mnt/storage-box-1/decrypted
root@vm# e2fsck -f /dev/mapper/storbackup
e2fsck 1.46.5 (30-Dec-2021)
/dev/mapper/storbackup: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found. Create<y>? yes
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/storbackup: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/storbackup: 585348/65536000 files (0.0% non-contiguous), 132904921/262139904 blocks
root@vm# mount /dev/mapper/storbackup /mnt/storage-box-1/decrypted
root@vm# touch /mnt/storage-box-1/decrypted/garage/test
root@vm# echo "exit code: $?"
exit code: 0
Now let’s prevent this error from every happening again by rotating the logs, restarting the systemd service and recreating the containers so that they pick up on this change:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@vm# docker ps -q | while read id; do docker inspect "$id" --format '{{.Name}} {{json .HostConfig.LogConfig}}'; done
/hbbs {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/hbbr {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/minio-minio-1 {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/cadvisor {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/node_exporter {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/traefik-proxy-1 {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/traefik-whoami-1 {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/garage {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/ecom_frontend {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/ecom_backend {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/ecom_dockerized-postgres-1 {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/guacamole_compose {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/postgres_guacamole_compose {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}}
/guacd_compose {"Type":"json-file","Config":{"max-file":"5","max-size":"10m"}
root@vm# cat /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}