The CPU crisis continues. After Windows and Ubuntu patches for the Meltdown and Spectre flaws caused problems for some users, Intel is now investigating reports that its CPU firmware updates are triggering system crashes and reboots.
“We have received reports from a few customers of higher system reboots after applying firmware updates,” Navin Shenoy, the executive vice president and general manager of the Data Center Group at Intel, said in an announcement. “Specifically, these systems are running Intel Broadwell and Haswell CPUs for both client and data center.”
Mitigating Spectre variant 2 requires CPU microcode changes so Intel has shared firmware patches for a number of CPUs with computer manufacturers which have already started releasing BIOS/UEFI updates that incorporate those fixes.
The new problems identified with Intel’s patches put those OEMs in the unusual position of asking customers who already updated their BIOS to now perform a downgrade. And that’s not exactly a straightforward operation on many computers.
Some computer vendors have already started withdrawing BIOS updates from their support websites until Intel corrects its CPU microcode so they can create and test new BIOS versions. Lenovo has updated its advisory to alert users about the impact that some of the updates delivered so far could have on their systems:
“Intel recently notified Lenovo of quality issues in two of these microcode updates, and concerns about one more. These are marked in the product tables with ‘Earlier update X withdrawn by Intel’ and a footnote reference to one of the following:
1 – (Kaby Lake U/Y, U23e, H/S/X) Symptom: Intermittent system hang during system sleep (S3) cycling. If you have already applied the firmware update and experience hangs during sleep/wake, please flash back to the previous BIOS/UEFI level, or disable sleep (S3) mode on your system; and then apply the improved update when it becomes available. If you have not already applied the update, please wait until the improved firmware level is available.
2 – (Broadwell E) Symptom: Intermittent blue screen during system restart. If you have already applied the update, Intel suggests continuing to use the firmware level until an improved one is available. If you have not applied the update, please wait until the improved firmware level is available.
3 – (Broadwell E, H, U/Y; Haswell standard, Core Extreme, ULT) Symptom: Intel has received reports of unexpected page faults, which they are currently investigating. Out of an abundance of caution, Intel requested Lenovo to stop distributing this firmware.”
CPUs don’t have non-volatile memory, so their code cannot be permanently updated. Because of this, the CPU microcode patches have to be reapplied at every system reboot. This is typically done by the BIOS/UEFI early in the boot process, but can also be done by the OS kernel during its initialization.
Linux has a built-in mechanism for applying CPU microcodes and Intel provides a firmware update package for the OS. This is good because computer manufacturers generally stop delivering BIOS/UEFI updates to devices after a couple of years.
By having the ability to apply microcode updates directly, Linux can ensure that even old computers will benefit from important bug fixes for their CPUs. It also makes microcode rollbacks easier in case they introduce problems, like in this case.
Windows also has the same capability and Microsoft has delivered Intel CPU microcode patches through Windows Update in the past. However, for this major flaw, the company has opted to let Intel and computer manufacturers deal with CPU updates.
Bundling microcode patches with BIOS is the preferred method, but delivering them through Windows Update would have made it much easier for users to apply them and would have arguably protected computers faster. It’s not clear why Microsoft took the decision to stay out of this process. Maybe the nature of the flaw required a low-level approach, or maybe Microsoft foresaw the potential problems and didn’t want to shoulder the blame.
Intel is quickly working with customers to “understand, diagnose and address this reboot issue,” Intel’s Shenoy said. “If this requires a revised firmware update from Intel, we will distribute that update through the normal channels. We are also working directly with data center customers to discuss the issue.”