"No space left on device" even though external drive is connected

[johnpc]

I am not sure why this is happening, but whenever I try to do anything while ssh'd into my umbrel, I get the error No space left on device. I can cd into my /mnt/data directory and write things to that, but I can't even edit my .bashrc file because there is no space left.

df shows that 100% disk space is used on the main filesystem:

umbrel@umbrel:/ $ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/root       30362124  30308112         0 100% /
devtmpfs         1777356         0   1777356   0% /dev
tmpfs            1943052         0   1943052   0% /dev/shm
tmpfs            1943052    188380   1754672  10% /run
tmpfs               5120         8      5112   1% /run/lock
tmpfs            1943052         0   1943052   0% /sys/fs/cgroup
/dev/mmcblk0p1    258095     31254    226842  13% /boot
/dev/sda1      960379920 731622256 179903240  81% /mnt/data
tmpfs             388608         0    388608   0% /run/user/1000

but it looks like this is because everything in /mnt is duplicated to /home

from ./ncdu:

. 690.0 GiB             [##################] /mnt
. 686.2 GiB             [################# ] /home
.  27.6 GiB S  52.4 MiB [                  ] /sd-root
.  24.6 GiB             [                  ] /var
    4.0 GiB             [                  ] /swap
    2.6 GiB S  52.4 MiB [                  ] /usr
  203.8 MiB             [                  ] /lib
. 182.9 MiB             [                  ] /run
   30.5 MiB             [                  ] /boot
   11.1 MiB S  40.0 KiB [                  ] /bin
   10.6 MiB             [                  ] /sbin
.   5.5 MiB             [                  ] /etc
   92.0 KiB             [                  ] /status-server
.  24.0 KiB             [                  ] /tmp
    8.0 KiB             [                  ]  umbrel-status
e   4.0 KiB             [                  ] /srv
.   4.0 KiB             [                  ] /opt
e   4.0 KiB             [                  ] /media
.   0.0   B             [                  ] /sys
.   0.0   B             [                  ] /proc
    0.0   B             [                  ] /dev
!   0.0   B             [                  ]  root
!   0.0   B             [                  ]  lost+found

htop doesn't show anything interesting and docker ps shows all my images as running - but I cannot visit any of my apps (including my bitcoin node via http://umbrel.local:2100) and the Umbrel dashboard shows all apps as "starting..."

I've tried rebooting the device multiple times but it boots back up into this same state every time.

[johnpc]

I determined the cause. It's a 32GB SD card and 7.7GB were taken up by each of /sd-root/var/log/syslog, /sd-root/var/log/messages and /sd-root/var/log/user.log for a total of 23.1GB in log files. The rest of the SD storage contains various caches and the OS itself. Is it safe to remove those three log files? Why did they get so bloated?

[banneord-puzzle]

Following advice in: https://askubuntu.com/questions/746535/var-log-syslog-growing-indefinitely-in-size it might be a good idea to hunt down the offending service/daemon that is constantly filling up the logs with errors.

Start by looking at the log entries themselves.

Does any error repeat?
Is it related to the Umbrel itself?

[lukechilds]

@johnpc thanks for reporting this. Did you find anything in the logfiles that pointed to what the problem was?

Sounds like you are but just to clarify, you're running Umbrel OS on a Raspberry Pi, right?

Also when did this problem start? Was it immediately after doing an update or seemingly randomly? If after an update do you remember which version you were update from/to?

Thanks!

[IanMichaelHarper]

I have just experienced this exact same issue. My syslog is 10G and has the following message written 1382210 times:

Dec  2 00:41:31 umbrel dockerd[856]: time="2023-12-02T00:41:31.842107093Z" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/1170aa1573e586fc60e782aaed295e4c66af8372910a40ea368429541397ab72/1170aa1573e586fc60e782aaed295e4c66af8372910a40ea368429541397ab72-json.log: no space left on device"

My daemon.log.1 is 13G and has the same message written over and over.

I was not using my umbrel around the 2nd December when this log is dated, but it was not an issue until a few days ago when I began installing more apps and transferring other data files to my umbrel. Then suddenly my Umbrel stopped working and I began getting this error on the command line. Before that it seemed to work fine via the web interface.

I did not update the Umbrel OS at all in this time but I did update a few apps.

Removing these log files fixed the issue.