So back in 2013 I was posting about getting VirtualBox running on BackTrack 5 R3, fighting the 64-bit hardware acceleration error on my T420, and getting VirtualBox mostly working on an Acer C7 (mostly being the key word there). One laptop, one hypervisor, and a whole lot of Googling.
Fast forward a few years… and part of the “lab” is now a Proxmox setup with three nodes, a NAS, and a growing pile of drives I may or may not have told anyone about. The Googling has not stopped. If anything, it got worse.
This isn’t a how-to install guide (the Proxmox docs do that way better than I would). It’s the stuff I learned by doing, the commands that got me out of trouble, and a few moments where I stared at a screen and said “um.. what?” Hopefully it saves someone time. If not – that’s okay too.
The hardware (aka whatever I had laying around)
While I don’t have a full server room to play with – I did have a spare Desktop PC and a few dual NIC NUC’s that I figured why not, right?
- The old desktop: an i7-7700K on a Z270 board with 4x16GB of DDR4. Only 8 threads, but 64GB of RAM, which in a home lab is what actually runs out first.
- Two mini PCs: a Beelink and an i9-12900HK box with 20 threads. Quiet, low power, and… not much room for drives.
Lesson learned: match the workload to the box. The desktop is my big-RAM, not-so-much-CPU node. The CPU-hungry stuff goes on the 20-thread mini PC. When one of the mini PCs started running HOT with a heavy VM on it, I moved the VM to a different node instead of buying anything new. My lab budget thanked me.
1 socket, and size the cores to the job. I used to hand out cores like candy. Now most VMs get 1 socket and only the cores they need (Pi-hole is happy with 2, a Linux desktop VM gets 2 with 4GB of RAM). Nothing gets the whole host.
Double-check NIC’s: know what your NIC actually is. I was “pretty sure” one of my boxes had a 2.5Gb NIC. I was pretty wrong. One command and the dream was over:
# ethtool enp1s0 | grep -iE 'supported link|speed'
No 2500baseT/Full in the list = it’s gig. Everything in my lab is 1GbE right now.
Welcome to 2026 – SSDs are NOT cheap
At one point I looked at the price of a new 2TB SSD (about $500 at the time) and said “dude… SSD is not cheap.” So I went the used route and grabbed a used 2TB SATA SSD (99-100% health) plus a used 6TB surveillance drive for local backups and ISOs. Both for less than one new drive. Waiting to receive those and maybe post about it – migrating from my 8 year old drives to these.
New to me (SSD): check SMART before putting anything on them.
# smartctl -a /dev/sdX | grep -iE 'power_on|realloc|pending'
Power_On_Hours is the one I care about most. My personal rule is if it’s over about 40,000 hours, it goes back. For comparison, one of the drives I was ALREADY running VMs on had about 73,600 power-on hours. That’s roughly 8.4 years of spinning, nonstop. That drive has seen things. It’s on borrowed time.
A few other things I picked up along the way:
- I skip SMR drives for backups (big sequential writes + SMR = sadness).
- For VM disks I want an SSD with DRAM. For ISOs and backups, a DRAM-less SSD or plain old spinning rust is fine.
- I passed on a cheap refurbished “enterprise” drive that turned out to be from 2015.
Backups: where the real fun happened
Fun moment #1: backing up a mini PC… to itself. One of my mini PCs kept failing its backups with this lovely message:
zstd: error 70 : Write error : cannot write block : No space left on device
Turns out a small mini PC drive cannot hold a big VM AND a backup of that big VM. Backups belong somewhere else, which is how the NAS got involved.
A copy is not always a good copy. Before I had the NAS sorted out, I SCP’d some backups from one node to another. File sizes matched to the byte. Great, done, right?
Nope. Out of habit I ran sha256sum on both ends, and the two biggest files did NOT match. Same size, different hash. So I downloaded the same files straight to my PC, and those matched the source just fine. Disks on the receiving box checked out clean on SMART, so the culprit pointed to bad RAM on that box… and that’s a story for another day.
The takeaway for me: a file being the right size means nothing. Now I check every backup I move. Rebuilding in a LAB isn’t Prod – but still a PITA:
# sha256sum vzdump-qemu-*.vma.zst
# zstd -t vzdump-qemu-*.vma.zst; echo $?
I run sha256sum on both ends and compare. zstd -t tests the archive itself (it’s quiet when it’s happy, so the echo $? tells you, and 0 means good). And the real test is restoring it to an unused VM ID and booting it. A backup I’ve never restored is just a rumor.
The NAS that time forgot. My NAS is… let’s say “seasoned.” When I tried to add it as storage in the Proxmox GUI, I got a super helpful mount error, and in dmesg:
cifs_mount failed w/return code = -22
After some digging (I mean Googling) I found the NAS was configured for SMB1 only. Proxmox wants SMB2 at minimum, so I went into the NAS settings and switched it over to SMB2 to match. Then I added the storage from the command line so I could pin the version:
# pvesm add cifs nas-backups --server X.X.X.X --share lab-backups --subdir /pve01 --smbversion 2.0 --username youruser --content backup
Note the subdir needs a leading slash. Ask me how I know. :/
Over 1GbE I get roughly 120 MiB/s to the NAS, so a 300GB backup takes around 40-45 minutes. Not fast, but it’s off the box, and that’s the point. I also gave each node its own folder on the NAS so I can tell whose “VM 102” is whose (that one’s foreshadowing).
What I’m running now: a fast local backup disk on the desktop, the NAS as the “trusted” copy, and Proxmox Backup Server is next on the list for the dedupe/incremental goodness.
Bonus fright: the gray question marks. One day I opened the Proxmox GUI and every VM had a gray “?” on it. Heart rate: elevated. Everything was actually still running fine. The stats service had crashed after about 24 days up:
# systemctl restart pvestatd
Icons came back, heart rate came back down.
What I’d do differently
- Buy RAM and SATA disks before an unknown AI boom happens.
- Get backups off the box from day one, and verify them (hash, zstd -t, test restore).
- Document steps – what you did / what you are doing – this way you can refer back or figure out what you did.
- Invest in 2.5Gbps NIC’s if possible – most networks have this capability (i have a 10Gig switch available) but if you’re doing this on a home lab transferring takes time :(.
Next up: I clustered my nodes
So with backups verified, the NAS speaking a language from this decade, and three nodes humming along… I did the obvious thing and clustered them. It certainly wasn’t horrible, but a few things where I found CLI commands to be a bit easier for troubleshooting.
Hope this helps someone searching Google for the same problems. Happy labbing!





