After "apt upgrade" every other boot fails

[ViRb3]

Describe the bug

I ran apt update && apt upgrade today. Since then, approximately every other reboot fails (50% chance) with the following error:

Started systemd-udevd.service - Rule-based Manager for Device Events and Files.
[ TIME ] Timed out waiting for device dev-disk-by\2xdpartuuid-4e639091\x2d01.device - /dev/disk/by-partuuid/4e639091-01.
[DEPEND] Dependency failed for failed for boot-firmware.mount - /boot/firmware.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.

I was unfortunate to update 3 of my Raspberry Pi 5s, and every single one experiences this issue.

Steps to reproduce the behaviour

1. Run apt update && apt upgrade
2. Repeat reboot until reproduced

Device (s)

Raspberry Pi 5

System

$ cat /etc/rpi-issue
Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2

$ vcgencmd version
2024/07/30 15:25:46
Copyright (c) 2012 Broadcom
version 790da7ef (release) (embedded)

$ uname -a
Linux server 6.6.47+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.47-1+rpt1 (2024-09-02) aarch64 GNU/Linux

Logs

No response

Additional context

No response

[pelwell]

Which medium are these Pis booting from?

[ViRb3]

@pelwell Micro SD card.

[P33M]

What sort of micro sd card?

[ViRb3]

It's a SanDisk Extreme Pro 128GB on all three of the devices. For what is worth, I also have a bunch of external HDDs attached to each Pi, however those have their own power supply and are connected via a powered hub.

I also use an initramfs with dropbear for rootfs decryption according to https://github.com/ViRb3/pi-encrypted-boot-ssh.

This setup hasn't ever caused issues until yesterday.

[pelwell]

Is your debug output above from a serial port? Or rather, do you have a Pi 5-compatible serial cable you can use to capture debug output? If so, try with ignore_loglevel in cmdline.txt (keeping it to a singe line) - there should be some messages about the SD card, but whatever you get, post it all.

[ViRb3]

These logs were taken from the HDMI output, I just wrote down the important lines by hand.

I do actually have a debug probe, I hooked it up for the past 2 days, but unfortunately even after 30+ restarts at various times of the day and intervals, I could not reproduce the issue.

I thought maybe it was a fluke, so I restarted my other two Raspberry Pis to test, and surprisingly both of then failed to boot exactly like before. Sadly, those two devices are remote, so I can't attach a probe to them.

I would like to mention, when this issue occurred before, I could enter the initramfs/dropbear environment, but after decrypting the rootfs, the mounting timeout would happen.

In my last attempt on the remote Raspberry Pis, I did apt update && apt upgrade && rpi-eeprom-update -a, and when I rebooted, the Raspberry Pi never came online - the green light was on, but even initramfs/dropbear was not reachable. A force power cycle fixed it for now.

Sorry I can't provide any concrete details, but since the day the Raspberry Pi 5s have been released, I have been running these 3 in the exact same setup with no issues. So while I can't tell what's wrong, something definitely seems off. Thanks!

[timg236]

Is it possible that you need to regenerate the initramfs following the kernel update?

[ViRb3]

It auto-regenerates during every update, and I explicitly looked for those lines to make sure it didn't go wrong as it was my first suspect as well. I also did manually regenerate, just to double-check.

[ViRb3]

I would also like to add, I have updated all 3 devices in this exact way at least 10 times since the Raspberry Pi 5 was released and never experienced an issue.

Here's ls of initrd file:

# ls -alt /boot/initrd*
-rw------- 1 root root 13890585 Sep 13 18:36 /boot/initrd.img-6.6.31+rpt-rpi-2712
-rw------- 1 root root 13888365 Sep 13 18:35 /boot/initrd.img-6.6.31+rpt-rpi-v8
-rw------- 1 root root 14076413 Sep 13 18:35 /boot/initrd.img-6.6.47+rpt-rpi-2712
-rw------- 1 root root 14070518 Sep 13 18:35 /boot/initrd.img-6.6.47+rpt-rpi-v8

Actually, one more thing I forgot to mention, I am running the kernel in 4KB addressing mode instead of 16KB for compatibility reasons with my external disks, which are formatted with BTRFS and 4KB sectors.

[ViRb3]

One of my remote Raspberry Pi 5s just hung up out of the blue. It did not reboot, the green light was still on, but it was completely unresponsive - could not SSH, no disk activity, etc. After force restarting it, I checked the logs, and there was absolutely nothing since it crashed ~8 hours ago. Last logs were just some routine networking related cron jobs I have running.

I have now downgraded all my devices to the previous kernel, 6.6.31, will report back how it goes. For reference, this is how I did it:

set -e
cd /boot/firmware
mv kernel8.img kernel8.img.bak
mv kernel_2712.img kernel_2712.img.bak
mv initramfs_2712 initramfs_2712.bak
mv initramfs8 initramfs8.bak
cp -a ../vmlinuz-6.6.31+rpt-rpi-2712 kernel_2712.img
cp -a ../vmlinuz-6.6.31+rpt-rpi-v8 kernel8.img
cp -a ../initrd.img-6.6.31+rpt-rpi-2712 initramfs_2712
cp -a ../initrd.img-6.6.31+rpt-rpi-v8 initramfs8

Reference of kernel versions: https://archive.raspberrypi.com/debian/pool/main/l/linux/

[ViRb3]

Since downgrading, have restarted the servers multiple times and had no issues. No random crashes either. I am therefore very confident that this was due to the kernel upgrade. Sadly, not sure how I can help more.

[pelwell]

The major change to the SD card handling has been the addition of support for command queueing. I'm going to list a number of kernel builds in order of incrementing time. For each build, run sudo rpi-update <hash>, where <hash> is the 7-digit hexadecimal number on the line. I suggest you start at the top and work your way down until you start to see the problem again.

The builds:
f17defc - "kernel: Bump to 6.6.22" // before command queueing was enabled
f637152 - "kernel: drivers: mmc: add SD support for Command Queueing"
8ec753c - "kernel: SD command queueing take 2"
b0bbbab - "kernel: SD fixes round 3"

[fshimizu]

I think I have the same issue on my RPi 5 on 6.6.47. OS and all data are on a SanDisk Extreme 256 GB SD card. Sometimes it hangs at boot, but for me it's less than 50% of the time.

Additionally, sometimes during normal system runtime, the whole system suddenly freezes. Because I have a system monitor open at all times, I always see disk I/O of my SD card completely stop when that happens. From one moment to the next there is not a single byte disk I/O to and from the SD card any longer. And from here on the system completely freezes and needs a hard power cycle. For me this happens much more frequently when there is a lot of I/O going on, such as pulling large container images. For example, there is one particular image that's almost 2 GB which I tried to pull dozens of times last week and each time the system froze eventually. It can also happen during normal desktop use, e.g. while browsing with Firefox or using vscode, though less frequently.

I can reasonably rule out a faulty SD card because I bought a new SanDisk Extreme 128 GB card, cloned the OS to it (yes, I did properly downsize partitions) and had the exact same issue with that.

Just now I downgraded to 6.6.22 using rpi-update f17defc and was able to pull that large image for the first time without freeze. So I have no long-term experience yet, but it looks very promising. Thank you so much for the research. I almost thought my Pi was faulty until I found this issue.

[ViRb3]

Thanks for chiming in @fshimizu! Happy to hear this is not an isolated case. I have been meaning to troubleshoot more, but sadly have a busy week ahead. I hope I can help more after that.

Can I ask if you have any external disks connected? When you mention that I/O causes the crash, do you mean load on the SD card or any external disk?