Troubleshooting Bad RAM in a Proxmox Home Lab (memtest86+, One Stick at a Time)
So in my Proxmox home lab post I mentioned that some backups I SCP’d between nodes came out with the right size but the wrong hash, that the disks checked out clean, and that the culprit pointed to bad RAM… “and that’s a story for another day.”
Well. It’s another day.
Clues –
Looking back, some activity actually gave me a heads up about this problem:
- Checksums didn’t match. The two biggest backup files I copied to the desktop had the same size and a different sha256. The same files downloaded to my PC matched the source just fine. SMART on the receiving disks was clean.
- “I ran memtest before.” I had. A while back, and it passed (and it takes a long time to run, so I wasn’t in a hurry to do it again). I routed my backups to the NAS and moved on. The test takes a while – and this is a lab… :/
- The gray question marks. This was the kicker – when I clustered my Proxmox devices – I got grey question marks on my PC Proxmox node – and the pvestatd service stopped. This was new… why did it do this – and stop the pvestatd stop?
Then it did it again. Same day, less than an hour after the first crash (about 12 minutes after I restarted it):
Well crud…
# journalctl -u pvestatd --since today | tail -30
The first crash was a segfault (status=11/SEGV). The second was an abort (status=6/ABRT). One random crash can be a software bug. The same service falling over twice in under an hour with different errors… that smells like memory getting corrupted.
While I was waiting to test, I gave pvestatd a safety net so the GUI would fix itself if it happened again:
# mkdir -p /etc/systemd/system/pvestatd.service.d
# printf "[Service]\nRestart=on-failure\nRestartSec=10\n" > /etc/systemd/system/pvestatd.service.d/restart.conf
# systemctl daemon-reload && systemctl restart pvestatd
That’s a band-aid, not a fix.
Let’s double-check what I do have?
Before buying anything, I wanted to know exactly what RAM was in the box without opening it (Cuz you don’t have to):
# dmidecode -t memory | grep -E "Size|Speed|Part Number|Manufacturer|Locator" | grep -v "No Module"
4 x 16GB G.Skill Ripjaws V DDR4-3200, the max for that older Z270 board, all running at 2133. XMP is off, so it’s running at stock speed. At stock speed, it’s harder to blame the settings.
While you’re in there, check the logs for anything the kernel noticed:
# dmesg -T | grep -iE "segfault|mce|machine check|edac|error" | tail -20
# journalctl --since today | grep -iE "segfault|general protection|traps:" | tail -20
Random segfaults across unrelated programs are a hint. MCE (machine check) or EDAC lines are a bigger one – that’s the hardware itself reporting errors.
Proxmox has memtest86+
Proxmox ships memtest86+ and puts it right in the boot menu. No USB stick required. To check on a shell you can do this:
# dpkg -l | grep memtest
# ls /boot | grep -i memtest
If it’s not there, apt install memtest86+ and then update-grub. Reboot, hold Shift or tap Esc after the logo to get the GRUB menu, and pick the memory test entry (the x64.efi one if you boot UEFI).
The plan was to let it run overnight with at least 2 full passes, which is what you want when you’re chasing something intermittent. A single pass that finishes clean doesn’t prove much.
It didn’t need overnight. With all four sticks in, it failed test #3 in about 90 seconds – 290 errors in the first 92MB, all the same single bit. Starting mem test it isn’t looking good :(.
Shortcut to finding the bad stick: one at a time
Since the error happened pretty quickly – the thought was I could probably narrow this down one stick at a time. Since memtest tells you the memory address, not the stick. With dual channel the memory is interleaved across the sticks, so you can’t map an address to a slot. The only way I could think of was testing 1 stick at a time.
Slow.. but worked 🙂
- Power off, unplug it, pull all but one stick.
- Put that one stick in the same slot every time. On my board the slots go A1, A2, B1, B2 counting from the CPU, and a single stick goes in A2 (check your manual – and I’ll admit I had to draw this one out on a picture of the board before it clicked).
- Run memtest. Label the stick with a piece of tape. Next.
Results:
- Stick 1: about 10 minutes, made it to test #6, 0 errors.
- Sticks 2 and 3: about 5 minutes each, clean.
- Stick 4: near instant failure. 17 errors in 9 seconds.
Found it. If none of them fail alone but they fail together, then I’d start looking at the slot, the board, or the memory settings instead of the sticks.
Living on three sticks
I put the 3 good sticks back in (A2 and B2 first, then A1) and Proxmox sees about 47GiB. That node was using around 40GB, so headroom is… tight. I’m keeping an eye on it with free -h and watching for pvestatd to crash again.
Replacement: a matched 4 x 16GB DDR4 set was running about $300 when I looked. For DDR4. In 2026. Buy RAM before an unknown AI boom happens, I said that last time and I meant it :/
Before you buy, check the warranty. A lot of RAM has a lifetime warranty, mine included. One gotcha: my 64GB turned out to be two 2 x 16GB kits, and the RMA wants the whole kit back, so I’ll be down to 32GB while it’s out. Working on the RMA now.
ECC vs. non-ECC (the thing I’d think about next time)
This is a consumer desktop, so no ECC. ECC memory can catch and correct single-bit errors, and with EDAC the kernel logs them so you’d see a stick starting to go bad before it eats a backup. My failure was literally one bit flipping over and over – exactly what ECC is built for. For a lab, non-ECC is fine and cheaper. For the box holding your backups or ZFS pool, ECC is worth a look. ZFS checksums are great, but they can’t protect data that was already wrong in RAM before it was written.
Nuggets for the next person
- Checksums are your early warning. Same size, different hash, clean disks = suspect RAM. I’m glad I hashed out of habit.
- Weird crashes with different errors in the same service point at memory, not the software.
- A memtest that passed a few months ago doesn’t clear the RAM today. Sticks go bad.
- Run multiple passes for intermittent problems. Overnight if you can. – this is a Do as I say – not as I do kind of idea… since I found a shortcut.
- Turn XMP/EXPO off before testing so you know you’re testing the RAM and not the overclock. – I typically don’t overclock because I don’t want to.
- One stick at a time, same slot, label them. It’s slow, but it’s the only way to know.
- Check the warranty before you buy a replacement. – This is certainly a good idea – if there is a warranty (especially now with prices they way they are) you should use it. Which also means buy from a reputable vendor.
Hope this helps someone searching Google for the same problems. Happy labbing!
Comments are closed.