[OpenIndiana-discuss] [HEADS UP] Issues with loader leading to unbootable systems

[Alexander Pyhalov]

Hello, guys.

Please take the notice that fixes on illumos loader did reveal 
additional issues, resulted in inability to boot after updating 
Openindiana Hipster on MBR disk installs.
Problem with booting is solved updating to  OI hipster 
osnet-incorporation 0.5.11-2017.0.0.16205 (2017-02-25_1202) with 
system/boot/loader 1.1-2017.0.0.16205 (2017-02-25_1228) and onwards and 
reinstalling boot blocks.

Here are some additional technical information and explanations of what 
additional issues were:
  -  The boot programs written into disk boot block areas are read into 
memory based on recorded boot program sizes, since the MBR boot record 
is not updated by "pkg update" in case of MBR setups, the boot process 
will not read the  gptzfsboot fully into the memory. This is problem 
with installboot command.
  - The loader partition reading code was built using optimized data 
sharing, assuming only one partition is "open" at the time. 
Unfortunately in case of ZFS this assumption is not true and as an 
result, the disk read validation checks did invalidate the IO requests.

Both issues are now identified, proper updates are available and the 
package updates are available in Openindiana Hipster package repository. 
Instructions below are provided, to perform the update, while avoiding 
the issues.

The second issue is about the implementation specific details inside the 
loader and updated binaries which implement the fix, once updated 
binaries are installed, so no other special activities are needed.

The installboot problem is more complicated. The installboot command 
update implements MBR update, to make MBR boot code able to read and 
load the partition boot record, and only record gptzfsboot location and 
size in partition boot record.
As "pkg update" will always cause partition boot record to be updated, 
this change means that gptzfsboot will always be read using correct 
size. The complication is about the fact that we can not enforce MBR 
update automatically, and the MBR update has to be performed by the 
operator. The secondary complication is about the fact that patched 
installboot command is available only in updated BE, meaning that 
bootblock update has to be performed twice.

Who is affected:
Fresh installs with 20161030 OI hipster snapshot usb/ISO using MBR 
partition/slice install, using illumos loader
Who is not affected:
  Older 20160421 usb/ISO and earlier installs still using GRUB1
  Full-disk installs and GPT installs for rpool.

How problem appears:
Problem appears by issuing regular 'pkg update ' procedure, with affect 
of having unbootable system after update and restart.

Workaround1 is done right after update, before reboot, so you don't 
experience any inability of boot after update, so that nothing happens 
if you reinstall loader upon update and BEFORE restart.
Workaround2 is there if you already restarted after update and you have 
unbootable system.

Workaround1:
Bootblock update has to be performed twice, after regular pkg update and 
before reboot and after reboot again.

  -find the name of your new active updated BE:
$ beadm list
--
oi-hipster-87                    R     /          36.8G static 
2017-02-25 19:07

  -mount new BE into /mnt dir, so we can install new loader into MBR: 
(assume root privileges by su, sudo or pfexec)
$ pfexec bash
# beadm mount oi-hipster-87 /mnt


  -Install new illumos loader from new BE into MBR to be able to boot 
from HD again:
# bootadm install-bootloader -MfvR  /mnt

The bootadm command above will reinstall boot code, using boot programs 
from /mnt/boot, using verbose mode, so you can see the MBR code is also 
updated.

  -unmount new BE from /mnt dir:
# beadm unmount oi-hipster-87
# shutdown -y -g 0 -i 6

After reboot, again re-install boot code as priviledged user: (assume 
root privileges by su, sudo or pfexec)
$ pfexec bash
# bootadm install-bootloader -Mfv

Because this bootadm command is run from updated BE and the patched 
installboot command is used, MBR is updated to read partition boot 
record, and future "pkg update" command does not need any special 
workaround.


Workaround2:
If you already experienced inability to boot after update, you need to 
boot from live USB/DVD media into your new updated BE and reintall loader:

  -Use bootable USB/ISO disk to boot from media other then HD (by 
selecting it to be first bootable media in motherboard settings)
  -hit 'ESC' key to get loader "ok" prompt and list Boot 
Environments(BE) (rpool is name of boot pool):
ok beadm list zfs:rpool
--
oi-hipster-87                    NR     /          36.8G static 
2017-02-25 19:07
-Activate new BE to boot from (where beadm_name is the new BE created 
after update):
ok beadm activate oi-hipster-87 zfs:rpool
-Boot into new BE:
ok boot
After booting into updated BE, you would need to issue this command to 
reinstall loader on HD MBR (so that problem is resolved for the next 
reboot):

  -Install new illumos loader from new BE into MBR to be able to boot 
from HD again (as the priviledged user):
$ pfexec bash
# bootadm install-bootloader -Mfv

After that you can safely update and restart.

Thank you for your understanding, since loader is part of the illumos 
but still work in progress, and MBR booting problem slipped in testing, 
but is now safely overcomed.