Moving /var on debian vm to a SMB share (hint… don’t do it)

February 4, 2024

Now that the drive has been shared to this VM, I had this idea that I’d move /var to this drive. The theory being that it will just work and var will have this huge space to work with (2 or 4 TB depending on the drive I’ve connected). Looks like there’s a challenge with this idea and it has been abandoned. BUT I thought I’d try.

First before moving /var I need to stop any processes using var. I discovered that this isn’t trivial on a working system that is doing jounrald logging. Logging processes are dumping stuff here but stopping them isn’t difficult. Stopping journald is not smart, but I can mask the service which maps the service files (systemd-journald.service and systemd-journald.socket) to /dev/null. Rebooting with the systemd-journald.service masked DESTROYS these files and systemd-journald will be DEAD on your machine after this. This is BAD. Don’t do this. 

Find the processes using /var to start with.

lsof -n | grep /var

everything but systemd-journald could be easily killed or stopped nicely. This one required masking which maps this to /dev/null… while it’s still running (before a reboot) you can find the logs at /run/log

systemctl mask systemd-journald.service

if you ever want to unravel this change the following will do the trick. NOTE you can NOT do this after a reboot as the .service and .socket files have been destroyed. 

systemctl unmask systemd-journald.service

NOTE: there are much smarter ways to do this. I haven’t bothered figuring them out as I can only find complicated ways to do that and this is a homelab and it’s not worth figuring out. 

So now, if you reboot while this is masked DO NOT DO THIS… you’ll have to do something else first. I strongly recommend first making a back up of your systemd-journald.service and systemd.journald.socket files… they turned out to be in my /lib/systemd/ directory and NOT in /etc/systemd/system/ where I expected them to be based on online homework. 

So what I did next was a wholesale move of /var to the mounted drive

mv /var /mnt/newdisk/var

then mount the new directory as /var

mount -o bind /mnt/newdisk/var /var

next I updated fstab

nano /etc/fstab

with

 /mnt/newdisk/var /var        none    bind

then I tried a reboot and the system threw a bunch of errors and booted in safe mode and required a root password to get up and running. I also discovered that I’d blown away the systemd-journald service which is not good (journalctl -xb produced NO entries so no way to actually know what broke on boot). This post helped me recover from the stupidity above. I happened to have another debian VM working on the proxmox host and found the systemd-journald.service files there and duplicated them back after deleting the symlink /etc/systemd/system/systemd-journald.service to /dev-null (this is what happened with the masking). putting everything back and then enabling ‘systemctl enable systemd-journald.service’. A reboot brought the system back. 

At this point I don’t have a good answer for this if you REALLY want to do this. Note, journald can not be redirected to another directory for logging, however recommendations seem to suggest using rsyslog to write the logs to disk… journald can be persuaded to dump log data into rsyslog as I understand it.