One of NixOS’s big advantage is the ability to rollback changes/upgrades.
Even if the machine does not boot up, you can boot to older generation in the bootloader menu…
Provided that you can access it, which is not generally the case on servers.
We can still avoid to lock a server and take advantage of this.
The key idea is to reboot only once on the new generation, but keep the default boot choice on the safe generation for further boots.
This as been tested using NixOS 2018.2 with Grub.
I did not find a solution using systemd-boot.
change configuration/upgrade and optionally test it with:
build new system and add it to boot menu (will be the default!):
rollback default boot to current safe generation (but keep the new one in the menu):
nixos-rebuild boot --rollback
boot (only once) to the new generation (should be entry #1 (most recent entry, except that #0 is a copy of the default entry), maybe not if using profiles):
At this point, the first reboot will boot on new generation, but subsequent boots will boot on safe generation.
If something went wrong, just reboot from your server management console; if everything was fine, you can change default boot to the new generation, with either of the following commands:
Oops, I was too confident
If boot is broken, you can always boot in a rescue-like mode, mount the /boot partition and edit the bootloader configuration to reboot on a safe generation.