Pre- and post-reboot actions/checklist for remote Linux servers

These instructions apply primarily to Owl systems. Many other modern Linux distributions use GRUB instead of LILO by default, but we intentionally keep using LILO in Owl.

• Take note of the running services and OpenVZ containers (if applicable), optionally make sure that all of them and no others are configured to start upon bootup
• Sanity-check and save the NTP-synchronized system time to the RTC/NVRAM with ”/sbin/clock -uw” ¹⁾
• Make sure that an fsck won't be forced for ext2/3/4 filesystems on bootup because of them possibly having reached or exceeded the “maximum mount count” or “maximum time since last check”: check each filesystem with tune2fs -l /dev/… and, if needed, disable the regular checks with tune2fs -c0 -i0 /dev/… (we normally do this right after creating filesystems, but it does not hurt to check it pre-reboot)
• Check the status of any software RAID devices with cat /proc/mdstat (if any of them are degraded, then at least take this into consideration)
• If switching to a different kernel and/or userland install, then prepare a “safety net”:
  • Configure the new boot target for the very next reboot only, with automatic fallback to the current/tested boot target (you'll use lilo -R … once you're done editing /etc/lilo.conf and after you've run lilo)
  • In the new boot target, specify the panic=10 kernel parameter (to ensure that the system is automatically rebooted into the previous boot target should the kernel panic, e.g. upon failing to mount the root filesystem)
  • Consider configuring netconsole (another kernel parameter) and running nc -vul UDP-PORT-NUMBER under screen on a nearby system to capture the kernel messages (but then you will likely need a second reboot to disable netconsole)
  • When making changes to /etc/lilo.conf, be careful about your use of append vs. addappend (if in doubt about these, review the lilo.conf(5) man page first)
  • Place /sbin/shutdown -r 5 & into /etc/rc.d/rc.local (this is needed in case the new system boots up mostly fine, but its networking setup fails for whatever reason)
• Right before the reboot, consider shutting some of the services down while you still have control and see the shutdown messages (e.g., run service vz stop to shutdown OpenVZ containers) - this may serve to reduce the risk of the system getting stuck on shutdown, as well as provide more info to us

• If the delayed reboot from rc.local trick was used, then run shutdown -c ²⁾, then comment out the command from rc.local
• Sanity-check the system - what kernel booted up, any potentially unexpected messages from it, software RAIDs status, system time, started services and containers, whatever else may be relevant
• If netconsole was activated and you don't intend to keep using it (such as running nc on the other end), disable it and prepare for another reboot
• Make the new configuration the default (e.g., swap the previous/new boot targets in /etc/lilo.conf and run lilo)