On the schedule for the Blackhat conference is a presentation by Christopher Domas titled The Memory Sinkhole – Unleashing an x86 Design Flaw Allowing Universal Privilege Escalation. There’s a discussion going on Hacker News today.
I’m studying the AGI containment problem – that is, how to test and experiment with artificial general intelligences without letting them bypass your safeguards and take over the world. While the development of working AGI is far enough in the future that this particular bug will certainly be fixed by then, this is one the categories of things that keeps me up at night.
The containment system architectures I’ve focused on so far involve layering and defense in depth. You don’t just put your AGI in a VM and call it a day; nor do you airgap it and call it a day. No, you write it in a language with strict semantics, then you put it in an operating system sandbox, that you put that in a virtual machine, and you airgap that. This is necessary both because of occasional security vulnerabilities like this one, but also to enable tripwires and other safeguards; you need the safeguards to be on a sort of high-ground, relative to the AGI, so that they can inspect it and shut it down but the AGI can’t expect them or turn them off.
Only a few details on this exploit are available so far; I’ve only read the one-paragraph synopsis, and I don’t expect more details to come out until August, so the remainder of this post is speculation and prediction on my part.
If I’m understanding the synopsis correctly, this particular exploit is an escalation from VM guest kernel space to host kernel space or system management mode. In the architecture I described, there are two layers of defense (language semantics and process sandboxing) in between the AGI and the point where it could make use of this exploit, and one more layer of defense left (airgapping) after it does. However, the layer it breaks is important, because it’s the last layer protecting tripwires, the integrity of log files, and preventing the installation of BIOS modifications to subvert the hardware after a reset.