Issues with loader leading to unbootable systems

A bug was found in the illumos bootloader that resulted in the inability to boot after updating Openindiana Hipster on MBR disk installs. The problem with booting is solved by updating to OI Hipster osnet-incorporation 0.5.11-2017.0.0.16205 (2017-02-25_1202) with system/boot/loader 1.1-2017.0.0.16205 (2017-02-25_1228) and upwards and reinstalling boot blocks. Here are some additional notes and technical information on the issue:

  • The boot programs written into disk boot block areas are read into memory based on recorded boot program sizes, since the MBR boot record is not updated by “pkg update” in case of MBR setups, the boot process will not read the gptzfsboot fully into the memory. This is problem with installboot command.
  • The loader partition reading code was built using optimized data sharing, assuming only one partition is “open” at the time. Unfortunately in case of ZFS this assumption is not true and as an result, the disk read validation checks did invalidate the IO requests.

Both issues are now identified, proper updates are available and the package updates are available in Openindiana Hipster package repository. Instructions below are provided, to perform the update, while avoiding the issues. The second issue is about the implementation specific details inside the loader and updated binaries which implement the fix. Once updated binaries are installed, no other special activities are needed. The installboot problem is more complicated. The installboot command update implements MBR update, to make MBR boot code able to read and load the partition boot record, and only record gptzfsboot location and size in partition boot record. As “pkg update” will always cause partition boot record to be updated, this change means that gptzfsboot will always be read using correct size. The complication is about the fact that we can not enforce MBR update automatically, and the MBR update has to be performed by the operator. The second complication that the patched installboot command is available only in the updated BE, meaning that bootblock update has to be performed twice. Who is affected:

  • Fresh installs with 20161030 OI Hipster snapshot usb/ISO using MBR partition/slice install, using illumos loader.

Who is not affected:

  • Older 20160421 usb/ISO and earlier installs still using GRUB1.
  • Full-disk installs and GPT installs for rpool.

How problem appears: Problem appears by issuing regular ‘pkg update ‘ procedure, which results in having unbootable system after update and restart. Workaround 1 is done right after update, before reboot, so you don’t experience boot failures. Nothing happens if you reinstall loader upon update and BEFORE restart. Workaround 2 is there if you already restarted after update and you have an unbootable system. Workaround 1: Bootblock update has to be performed twice, after regular pkg update and before reboot and after reboot again. Find the name of your new active updated BE:

$ beadm list
oi-hipster-87                    R     /          36.8G static 2017-02-25 19:07

Mount new BE into /mnt dir, so we can install new loader into MBR: (assume root privileges by su, sudo or pfexec)

$ pfexec bash
# beadm mount oi-hipster-87 /mnt

Install new illumos loader from new BE into MBR to be able to boot from HD again:

# bootadm install-bootloader -MfvR  /mnt

The bootadm command above will reinstall boot code, using boot programs from /mnt/boot, using verbose mode, so you can see the MBR code is also updated. -unmount new BE from /mnt dir:

# beadm unmount oi-hipster-87
# shutdown -y -g 0 -i 6

After reboot, again re-install boot code as priviledged user: (assume root privileges by su, sudo or pfexec)

$ pfexec bash
# bootadm install-bootloader -Mfv

Because this bootadm command is run from updated BE and the patched installboot command is used, MBR is updated to read partition boot record, and future “pkg update” command does not need any special workarounds. Workaround 2: If you already experienced the boot failure after update, you need to boot from live USB/DVD media into your new updated BE and reinstall loader: Use bootable USB/ISO disk to boot from media other then HD (by selecting it to be first bootable media in motherboard settings). Hit ‘ESC’ key to get loader “ok” prompt and list Boot Environments(BE) (rpool is name of boot pool):

ok beadm list zfs:rpool
BE                                 bootfs
oi-hipster-87                      rpool/ROOT/oi-hipster-87

Activate new BE to boot from (where beadm_name is the new BE created after update):

ok beadm activate oi-hipster-87 zfs:rpool

Boot into new BE:

ok boot

After booting into updated BE, you would need to issue this command to reinstall loader on HD MBR (so that problem is resolved for the next reboot): Install the new illumos loader from new BE into MBR to be able to boot from HD again (as the privileged user):

$ pfexec bash
# bootadm install-bootloader -Mfv

After that you can safely update and restart. Thank you for your patience understanding. The loader is a part of the illumos which is still a work in progress. MBR booting problem slipped in testing, but is now safely overcomed. Send your appreciation to Toomas Soome (tsoome) for both creating and fixing the illumos loader and for providing larger a large part of this article.