Category Existential Risk

Powers That Must Not Be

Speed Read This
Posted by on June 4, 2015

Death Note is a manga by Tsugumi Obha and Takeshi Obata, which centers around a magical book called a Death Note. This book confers a power: by following its instructions, the possessor of the Death Note can kill anyone by writing down their name, and choose the manner of death. The story centers on Light Yagami, who comes into possession of the Note and uses it to kill criminals.

While the story the Death Note is a supernatural artifact, there are things in real life that are somewhat analogous. Assassins have existed for at least as long as recorded history; in a sense, having a Death Note is like commanding a master assassin. What makes the Note different and scary, however, is its anonymity. Real-world assassination is inherently risky and difficult, and it’s risky and difficult every time. This is good; in a world with too many Death Notes, some would fall into the hands of villains, and be used to terrible effect. One reason the world today is a better, safer place than the world of a thousand years ago is because forensic science has made it virtually impossible to get away with murder.

A straightforward extrapolation of currently-existing technologies suggests that this power does exist, and is possessed by a small number of spymasters and heads of state. This is unfortunate, but it’s no worse than it was in humanity’s past.

But there are analogous powers, similar to the Death Note power but not quite the same, which humanity did not used to have, and which we would prefer it didn’t.

One such power is the power of atomic weapons. Rather than killing individuals, these kill regions. Their existence is a large and obvious threat to everyone, but fortunately, they are hard enough to make that only governments are able to acquire them, and their terribleness is imprecise enough that no government is willing to use them; the ensuing disaster would affect them as well.

There are other powers, which are worse, which technology hasn’t brought into the world yet, but might. Consider what happens in a world where internet-connected drones are common. Drones are dangerous; they can kill people by crashing into them. What happens if they’re hackable?

The combination of bad computer security, face recognition, and hazardous internet-connected devices could create a Reference Class Note: an artifact like Tsugumi’s Death Note which, instead of targeting a person, targets an arbitrary category of people. And the nature of computer software is unfortunately not like the nature of fissile elements; hazardous software can be constructed by skilled individuals.

There are other ways this could happen besides drones, which I will not name. What all of them have in common, though, is that they depend on the continued failure of computer security. In the 90s and early 2000s, cracking computer security was within easy reach for bored teenagers; they used this power for pranks, and were the first canary in the coal mine. Today, the bored teenagers mostly can’t manage it, but skilled individuals can; they use this power for theft, and are the second canary. We need to drive the bar up higher, to the point where defeating computer security is limited to large organized groups. We need to do this soon, before the “Internet of Things” takes hold.

If we can’t? Bam, sci-fi dystopia.