I’ve been running, on and off, a bunch of Freenet nodes for a while, and I’ve never had issues with them. But a couple of months ago, a server where I installed it pretty much as usual started behaving strangely. Its available disk space got consumed regularly, at a fairly large rate of 4-5 GiB/day, as I could see from the ever decreasing available space shown by Webmin and df -h.
I tried looking for what could be causing it, learning a few interesting commands in the process, for instance this one:
find / -mmin -2 -ls
to see all written files in the last 2 minutes, and this one:
du -h / | grep -P '^[0-9\.]+[GT]'
to list all folders with a size measured in TiB or GiB.
The latter returned just the /usr folder, with a bit more than 1 GiB, and Freenet’s folder (actually a subfolders chain to the Freenet datastore), with the expected (considering my setup), non-growing size of 1.6 TiB. This, while my 2TB disk was almost 100% full, because I didn’t have time to investigate it sooner. All-in-all, I had about 250 GiB unaccounted for.
I also tried a tool called ncdu, which didn’t give interesting results either.
Oh and by the way, if you’re wondering what happens when your disk is full, it’s not really an enjoyable experience: Apache HTTPd goes down (Webmin too as far as I remember), the computer is globally slow (for instance, running du -h / | grep -P '^[0-9\.]+[GT]' took ages, while it took seconds when I ran it earlier while there was some space left), and some basic feature like auto-completion in the console via TAB don’t work (you suddenly realize that for some reason they require a bit of disk space, even for what could be assumed to be read-only, namely fetching the file/folder names in the current folder)
Anyhow, I was pretty puzzled and, since asking for help didn’t work, I decided I would just free some space, do an up-to-date backup of the single service I was running there, and reinstall, updating Ubuntu in the process. How to free some space? Well, Freenet appeared to be the obvious choice, as deleting just one file from the datastore would give me more than a month of time, assuming the disk would keep filling up at 5 GiB a day.
But I wanted to do it clean, so first I tried shutting down Freenet the clean way, using ./run.sh stop. To my surprise, it worked without a scratch. I assumed that shutting down would require writing a few things that were in RAM, so I expected at least a slow down, but no: no error, not even an abnormally large delay.
Then I had to choose what to delete. I listed all files and I picked CHK-cache.hd, because 1) it was big and 2) I thought maybe I’d want to restart the node later and having a flushed cache sounded better than having a flushed store or other things. ls -la said CHK-cache.hd was 730 GiB.
I ran rm CHK-cache.hd. Something else about having a full drive: it makes deleting a file slow as hell. I could follow, via df -h, first the MiBs and then the GiBs slowly getting freed, which became faster and faster the more space was already freed up. And then, maybe half an hour later, the file was finally fully deleted. The whole 730 GiB file. And I now had 971 GiB of free space. Which, obviously, was 241 GiB too much. So this is the mystery, where did those 241 GiB vanish? How come ls -la returned a size of 730 GiB for a file which was actually taking 971 GiB? Not sure I’ll ever know. Was I lucky I picked this very file, or would another big Freenet datastore file have freed mystery extra space too? (actually, only CHK-store.hd was as big, the other big files were all smaller than 40 GiB)
This was the first time I experienced that, after running Freenet on maybe a dozen of other setups with never a single disk space issue… I hope it won’t happen against, but at least now if it happens I’ll know where to look, what to stop, and what to delete.
There are different ways to count file size on GNU/Linux. If you use sparse files (files where zero-values don’t consume extra space), these ways can give different answers.
Another thing which complicates file sizes is the file system. What kind of file system do you use? For example btrfs and zfs should never be filled beyond 90% capacity.
As a hint for these investigations: There is filelight to list folder sizes, and you can sort du -h with sort -h to make it consider the units.
du -h / | sort -h
Oh hi ^^
I guess my wall of text was a bit unclear. Freenet was configured to use about 1.6TiB (I don’t remember the exact config, but with the cache and store size I aimed to use about that), and appeared to be using this.
I used
du -h / | grep -P '^[0-9\.]+[GT]'and it showed something like:1.6T /home/fnode1.6T /home/fnode/freenet
1.6T /home/fnode/freenet/datastore
1.6T /home/
1.6T /
which I assume is pretty much the same as I would have gotten from
du -h / | sort -hls -lain the datastore folder showed exactly:-rw-rw-r-- 1 fnode fnode 784377786368 Jun 17 19:54 CHK-cache.hdbut deleting this file resulted in 971 GiB freed
I’m not quite sure about sparse files, but I’ve configured other nodes similarly before, and they never ended up eating up all the disk space.
For instance I have a long-running (> 1 year) node with a datastore + client cache configured at 700GiB, and
du -handdf -hreturn consistent results: the former sees the datastore using “726G” and” “/ using “737G”, the latter sees 727G used space.Which just brought me to notice that on my “broken” node,
du -hsees “/” using “875G” whiledf -hsees “/” using “884G”, so there’s still a little leak in a file somewhere. If I had to bet, I’d go for CHK-store.hd ^^As for the filesystem, it is btrfs indeed, but I never intended to fill up the hard drive so much: I aimed for 1.6TiB out of 1.9 (the stuff beside Freenet was negligible in size), it’s just that things got out of hands when this file started invisibly growing…
I gave this some more thought and, it’s a long shot / kind of unlikely but, could it be a bug/issue in btrfs? I believe this is the first node I ran on btrfs, after all… (or at least the first one with such a big cache & store)