Category Uncategorized

Sanity Check the News

Speed Read This
Posted by on August 15, 2017

Regarding what happened recently in Charlottesville –

Actually, first, I’m going to put this filler paragraph here, so that people looking for quick emotional engagement can go read something else. This post is about subtleties, and I would prefer that anyone who wants to eschew subtlety in favor of emotion unsubscribe from my wall.

That out of the way, Alex Fields, the driver of the car that ran into a crowd, is by all accounts a racist and a scumbag. I fully expect that he will be hung to dry, and I have no problem with this. However, given his prominence in the news cycle, I believe it’s important to be truthful and precise about exactly what he did. The story running in the news is that he committed premeditated mass-murder, by running into a crowd of counter-protesters with his car.

I make a habit of fact-checking news stories, both political and non-political, and calling out any errors I see. I decided to check this one. Was it really murder, or was it merely vehicular manslaughter? This is important, because falsely believing that other people have escalated the level of violence in a conflict can *cause* the level of violence to escalate. So, how strong is the case against Alex Fields?

I’ve looked at it from as many angles as I could find, and reconstructed the sequence of events as best I could. Using a stopwatch for time and Google Maps for distance, I (imprecisely) estimated the car’s speed. I sorted through YouTube’s terrible search to get non-duplicate videos. I watched the important moments of them frame-by-frame. Here’s the sequence of events:

  1. Alex Fields is driving south towards a protest. There are pedestrians about 120ft ahead of him. His brake lights are on. The camera moves away from his car. and returns a second later.
  2. In 3.1s the car moves 105ft (about 24mph).
  3. Alex’s car is struck hard on the rear bumper by a stick or pole. Prior to this point he has not hit any pedestrians.
  4. About 50ft ahead of Alex’s car is a white convertible; in between, there are pedestrians.
  5. Alex’s car accelerates. Two pedestrians go over his hood, one is sandwiched between his car and the next, and several more on the sides are knocked over without being seriously hurt.
  6. Alex’s car hits the convertible, and the convertible hits the car in front of it, a red minivan. The minivan goes into the intersection, hitting about ten pedestrians
  7. Alex Fields shifts into reverse and starts backing up
  8. About seven people charge the car, striking it with sticks, smashing the rear windshield and the front windshield on the passenger side
  9. The car backs into two of the people who were attacking it, one of whom seems to end up seriously hurt

I think the most likely explanation is that, after the first strike on his car, he panicked. I am fairly certain that this will be his defense in court, and, unless evidence is found which is not yet public, I don’t think his guilt can be proven even to a preponderance of evidence standard.

From behind:
From front:
People hitting the car after it crashes:
Drone view:

(Cross-posted on Facebook)

Election Fraud and a Roll to Disbelieve

Speed Read This
Posted by on November 8, 2016

In 2000, the US presidential election between George Bush and Al Gore was swung by hacked voting machines. This fact can be verified by anyone, beyond a shadow of a doubt. In the close race between Bush and Gore, Volusia County reported a total of negative-16,022 votes for Al Gore[1]. Upon investigation, the negative vote count disappeared and was replaced with a normal-looking one, but not before the number reached the press. At the time, this was unexplainable. It was not until 2005 that security researcher Harri Hursti demonstrated a technique for hacking voting machines that involves sneaking memory cards with negative vote totals on them into the counting process[2]. The idea is that, by inserting a memory card with positive votes for one candidate and negative votes for another candidate, one can change the vote totals without messing up the turnout numbers, which would reveal the fraud. But if one is performing the Hursti hack, and messes up by putting the fake memory card into the process *in the wrong place*, then a county may accidentally report a negative vote total – because a memory card that was supposed to be used in a large precinct was used in a small precinct, without enough votes to steal. The machines used in Volusia County were in fact vulnerable to this technique, and this is what happened.

Because the margin in Florida was small, a “recount” was triggered; in reality, ballots were being viewed by humans for the first time. Bush and Gore argued before the Supreme Court, and the Supreme Court ruled 5-4 (along party lines) to stop the counting. Gore then conceded. It all happened too quickly for little things like election fraud to be noticed; the media narrative, rather than being about fraud, was about “hanging chads” and other innocent things.

Since then, forces within the Republican party have worked to promote a false narrative of vote fraud as something symmetric, that both parties do, by manufacturing false evidence of Democratic voter fraud. Like voting machine hacking, this becomes most obvious when there’s a mistake and the mistake becomes a scandal. In 2006, seven attorneys were fired mid-term from the department of justice[3]. In 2008, the Inspector General determined[4] that these firings were motivated by refusal to prosecute voter fraud cases against Democrats.
Over the past few weeks, Donald Trump has been loudly warning that democrats would engage in election fraud to give the election to Hillary Clinton. This is a possible thing, but so is the reverse. Fortunately, there’s a standard strategy for determining whether election fraud took place, and which direct it went in: exit polls.
I don’t have access to exit poll data for today’s election. Neither do you, and neither do most of the news outlets that’re reporting the results. But Nate Silver has this to say about them on his blog:
“One reason people find Trump’s competitive margins across a wide range of swing states so surprising is because exit polls showed Clinton beating her pre-election polls in most states, instead of underperforming them.”[5]

At the time I’m writing this, FiveThirtyEight says (on its sidebar) that Clinton and Trump are equally likely to win, based on states that have been officially called; other sources are strongly favoring Trump’s chances, based on preliminary counts.

I roll to disbelieve. Maybe Trump really did get more votes than expected, and more votes than indicated by the exit polls Silver was referring to. Or maybe he didn’t. One thing’s for certain, though: the American people will hear a definitive result before any of this is investigated.

(Cross-posted on Facebook)

National Psyches, Expressed As Sports

Speed Read This
Posted by on July 26, 2016

It’s often said that sports are a substitute for warfare, an alternative place for people to channel their most destructive instincts. As someone with only benevolent instincts I’ve never felt the appeal, but the thought occurred to me that one might learn quite a bit about cultures’ psyches by paying attention to their choice of sports.

For example, consider football. Each team has a leader, who controls the gaol and is allowed to break the rules. The rest of the players coordinate in a decentralized manner to set up shots at the other team’s leader, whose job is to survive the assassination attempts. Football is the most popular sport in Venezuela, Colombia, Nigeria, Brazil and Turkey. In the United States, however, the name refers to something else entirely.

In American football, each team’s leader is a general who sits on the sidelines and gives orders to a group of drug-enhanced supersoldiers, who were selected by the draft. One team’s goal is to take the “political football” and deliver it into the other team’s territory; the other team’s goal is to find the player with the football and detain him, so that the referee can see who’s really responsible. Recent studies have shown players suffering brain damage from repeated concussions, but no one knows what to do about this so things pretty much continue on as usual.

Not all sports represent warfare, though! Consider basketball, also popular in the United States. In basketball, two teams of mostly black people compete to see which is taller, thereby demonstrating their superior genes.

Um. I mean, um. Baseball! Look at baseball. In baseball, players approach one by one, and everyone works together to stop them from scoring home runs. A home run is a well-standardized euphemism for having sex. Other key players are the pitcher and catcher, which are well-standardized euphemisms for participants in gay sex. In fact, pretty much every baseball-related term is a euphemism for something sexual, at least according to UrbanDictionary. Whereas in Britain, they play a game that’s superficially the same except none of its terminology is euphemistic. This is because the British are uncultured barbarians.

I wish I could say that the historical shift in peoples’ choice of sports reflected the advancing enlightenment and universal culture. Instead, I will observe that in the de facto national sport of South Korea, which is Starcraft, one third of players are literally giant insects and the most exciting moments for spectators involve nuclear weapons.

What Google’s TPUs Mean for AI Timing and Safety

Speed Read This
Posted by on May 21, 2016

Last Wednesday, Google announced that AlphaGo was not powered by GPUs as everyone thought, but by Google’s own custom ASICs (application-specific integrated circuits), which they are calling “tensor processing units” (TPUs).

We’ve been running TPUs inside our data centers for more than a year, and have found them to deliver an order of magnitude better-optimized performance per watt for machine learning.

TPU is tailored to machine learning applications, allowing the chip to be more tolerant of reduced computational precision, which means it requires fewer transistors per operation.

So, what does this mean for AGI timelines, and how does the existence of TPUs affect the outcome when AGI does come into existence?

The development of TPUs accelerated the timeline of AGI development. This is fairly straightforward; researchers can do more experiments with more computing power, and algorithms that stretched past the limits of available computing power before became possible.

If your estimate of when there will be human-comparable or superintelligent AGI was based on the very high rate of progress in the past year, then this should make you expect AGI to arrive later, because it explains some of that progress with a one-time gain that can’t be repeated. If your timeline estimate was based on extrapolating Moore’s Law or the rate of progress excluding the past year, then this should make you expect AGI to arrive sooner.

Some people model AGI development as a race between capability and control, and want us to know more about how to control AGIs before they’re created. Under a model of differential technological development, the creation of TPUs could be bad if it accelerates progress in AI capability more than it accelerates progress in AI safety. I have mixed feelings about differential technological development as applied to AGI; while the safety/control research has a long way to go, humanity faces a lot of serious problems which AGI could solve. In this particular case, however, I think the differential technological advancement model is wrong in an interesting way.

Take the perspective of a few years ago, before Google invested in developing ASICs. Switching from GPUs to more specialized processors looks pretty inevitable; it’s a question of when it will happen, not whether it will. Whenever the transition happens, it creates a discontinuous jump in capability; Google’s announcement calls it “roughly equivalent to fast-forwarding technology about seven years into the future”. This is slight hyperbole, but if you take it at face value, it raises an interesting question: which seven years do you want to fast-forward over? Suppose the transition were delayed for a very long time, until AGI of near-human or greater-than-human intelligence was created or was about to be created. Under those circumstances, introducing specialized processors into the mix would be much riskier than it is now. A discontinuous increase in computational power could mean that AGI capability skips discontinuously over the region that contains the best opportunities to study an AGI and work on its safety.

In diagram form:


I don’t know whether this is what Google was thinking when they decided to invest in TPUs. (It probably wasn’t; gaining a competitive advantage is reason enough). But it does seem extremely important.

There are a few smaller strategic considerations that also point in the direction of TPUs being a good idea. GPU drivers are extremely complicated, and rumor has it that the code bases of both major GPU
manufacturers are quite messy; starting from scratch in a context that doesn’t have to deal with games and legacy code can greatly improve reliability. When AGIs first come into existence, if they run on specialized hardware then the developers won’t be able to increase its power as rapidly by renting more computers because availability of the specialized hardware will be more limited. Similarly, an AGI acting autonomously won’t be able to increase its power that way either. Datacenters full of AI-specific chips make monitoring easier by concentrating AI development into predictable locations.

Overall, I’d say Google’s TPUs are a very positive development from a safety standpoint.

Of course, there’s still the question of how the heck they actually are, beyond the fact that they’re specialized processors that train neural nets quickly. In all likelihood, many of the gains come from tricks they haven’t talked about publicly, but we can make some reasonable inferences from what they have said.

Training a neural net involves doing a lot of arithmetic with a very regular structure, like multiplying large matrices and tensors together. Algorithms for training neural nets parallelize extremely well; if you double the amount of processors working on a neural net, you can finish the same task in half the time, or make your neural net bigger. Prior to 2008 or so, machine learning was mostly done on general-purpose CPUs — ie, Intel and AMD’s x86 and x86_64 chips. Around 2008, GPUs started becoming less specific to graphics and more general purpose, and today nearly all machine learning is done with “general-purpose GPU” (GPGPU). GPUs can perform operations like tensor multiplication more than an order of magnitude faster. Why’s that? Here’s a picture of an AMD Bulldozer CPU which illustrates the problem CPUs have. This is a four-core x86_64 CPU from late 2011.


(Image source)

Highlighted in red, I’ve marked the floating point unit, which is the only part of the CPU that’s doing actual arithmetic when you use it to train a neural net. It is very small. This is typical of modern CPU architectures; the vast majority of the silicon and the power is spent dealing with control flow, instruction decoding and scheduling, and the memory hierarchy. If we could somehow get rid of that overhead, we could fill the whole chip with floating-point units.

This is exactly what a GPU is. GPUs only work on computations with highly regular structure; they can’t handle branches or other control flow, they have comparatively simple instruction sets (and hide that instruction set behind a driver so it doesn’t have to be backwards compatibility), and they have predictable memory-access patterns to reduce the need for cache. They spend most of their energy and chip-area on arithmetic units that take in very wide vectors of numbers, and operate on all of them at once.

But GPUs still retain a lot of computational flexibility that training a neural net doesn’t need. In particular, they work on numbers with varying numbers of digits, which requires duplicating a lot of the arithmetic circuitry. While Google has published very little about their TPUs, one thing they did mention is reduced computational precision.

As a point of comparison, take Nvidia’s most recent GPU architecture, Pascal.

Each SM [streaming multiprocessor] in GP100 features 32 double precision (FP64) CUDA Cores, which is one-half the number of FP32 single precision CUDA Cores.

Using FP16 computation improves performance up to 2x compared to FP32 arithmetic, and similarly FP16 data transfers take less time than FP32 or FP64 transfers.

Format Bits of
Bits of
FP16 5 10
FP32 9 23
FP64 11 52

So a significant fraction of Nvidia’s GPUs are FP64 cores, which are useless for deep learning. When it does FP16 operations, it uses an FP32 core in a special mode, which is almost certainly less efficient than using two purpose-built FP16 cores. A TPU can also omit hardware for unused operations like trigonometric functions and probably, for that matter, division. Does this add up to a full order of magnitude? I’m not really sure. But I’d love to see Google publish more details of their TPUs, so that the whole AI research community can make the same switch.

Suspicious Memes and Past Battles

Speed Read This
Posted by on May 17, 2016

When people organize to commit a crime, the legal term for this is a conspiracy. For example, if five people get together and plan a bank robbery with the intent to execute their plan, then each of them has committed the crime of “conspiracy to commit robbery”.

If you suspect that a crime has taken place, and it’s the sort of crime that would’ve involved multiple people, then this is a “conspiracy theory”, particularly if those people would’ve had to be powerful and well-connected. Until recently, everyone seemed to agree that conspiracy theories were exclusively the creations of idiots and schizophrenics. One of the more common, well known conspiracy theories is literally that “the CIA is causing my command hallucinations”. So if someone were to claim, for example, that the 2000 election was swung by hacked voting machines in Volusia County, or that Michael Hastings was murdered by one of the top-ranking military officials he singled out for criticism in Rolling Stone? Then the anticipation is that this would be followed by some incoherent rambling about aliens, the social expectation is that this person is to be shunned, and the practical effect is that those things are hard to say, hard to hear, and unlikely to be taken seriously.

Somehow, somewhere along the line, accusing governments and the powerful of crimes became associated with mental illness, disrespectability and raging idiocy. And when you pause and notice that, it is *incredibly creepy*. It’s certainly possible to imagine how such a meme could arise naturally, but it’s also a little too perfect — like it’s the radioactive waste of a propaganda war fought generations back.

As it turns out, the last few generations did have propaganda wars, and we know what they were: the counterculture, the civil rights movement, McCarthyism. Each side of each memetic conflict left its mark on our culture, and some of those marks are things we’d better scrub away. So now I’m wondering: what were they?

(Cross-posted to Facebook)

Introducing conservative-pad 0.4

Speed Read This
Posted by on March 24, 2016


There’s been some controversy lately in the Javascript world about micro-packages, version pinning, and the NPM package repository’s regulatory regime. For a serious take, I wrote a comment on the HN thread. There is a looming spectre threatening our entire software ecosystem, of which the deplorable decline of our dependency hygiene is but one example. The left has gone too far. That’s why I’m writing conservative-pad, a string-padding function free from the constraints of political correctness.

Here’s the original:

module.exports = leftpad;
function leftpad (str, len, ch) {
str = String(str);
var i = -1;
if (!ch && ch !== 0) ch = ' ';
len = len - str.length;
while (++i < len) { str = ch + str; } return str; }

This is terrible. You can expect that my replacement version will:

  • Be O(n). The original version uses Shlemiel the Painter's algorithm, which is mumble slur mumble.
  • Be written in LISP, to protect purity. These immigrant Javascript programmers are driving down wages and destroying our precious modularity.
  • Make web development great again.
  • Remove the unnecessary 'ch' parameter, a ridiculous piece of political correctness. It isn't our fault that string-padding is dominated by ' '; it won fair and square, and we shouldn't make exceptions to meritocracy just because someone wants a string-padding character that isn't white.
  • Compile with -Wall, because otherwise the bugs will keep coming back over the border.

Thank you.

Truth Coming Out of Her Well

Speed Read This
Posted by on February 28, 2016

There is a semi-famous picture used as a test image in computer graphics, the Lenna picture. It has an interesting history:


Alexander Sawchuk estimates that it was in June or July of 1973 when he, then an assistant professor of electrical engineering at the USC Signal and Image Processing Institute (SIPI), along with a graduate student and the SIPI lab manager, was hurriedly searching the lab for a good image to scan for a colleague’s conference paper. They had tired of their stock of usual test images, dull stuff dating back to television standards work in the early 1960s. They wanted something glossy to ensure good output dynamic range, and they wanted a human face. Just then, somebody happened to walk in with a recent issue of Playboy.

The engineers tore away the top third of the centerfold so they could wrap it around the drum of their Muirhead wirephoto scanner, which they had outfitted with analog-to-digital converters (one each for the red, green, and blue channels) and a Hewlett Packard 2100 minicomputer. The Muirhead had a fixed resolution of 100 lines per inch and the engineers wanted a 512 x 512 image, so they limited the scan to the top 5.12 inches of the picture, effectively cropping it at the subject’s shoulders.

Yes, for strictly technical engineering reasons, they tastefully cropped Lenna to the shoulders. This reminded me of a more recently semi-famous picture, which suffers from similar aspect-ratio and other technical problems. And so, ladies, gentlemen and other persons, I present: Truth Coming Out of Her Well to Politely Clarify for Mankind.


For use as a meme, this version has several significant advantages over the 1896 Jean-Léon Gérôme original. In particular:

  • It is square
  • The woman’s face is properly centered
  • There is a uniform area at the top to overlay text on

Here are some examples of how you might use it:

Nice headline/The actual study says the oppositeThey didn't do a Bonferroni correctionThat sounds pretty damning/Unless you read the original context

I claim no copyright over this image, and the original painting is old enough to be expired. So feel free to use this meme to shame politely clarify for mankind wherever you see fit!

EDIT: Rob Bensinger suggests cropping slightly differently. This version gets a closer view, but adds graininess. Whether graininess is a technical failing reminiscent of 1970s image processing or an appropriate signal of meme-iness is up for debate.


A Series of Writing Exercises

Speed Read This
Posted by on January 27, 2016

Last Monday, my house hosted a Meta Monday on the subject of how to write about aversive things, particularly self-promotion. This spawned a discussion of how to avoid getting stuck on writing generally, and an interesting exercise. I’ve modified the description of the exercise slightly to refine it, so consider it untested, but I think this is worth doing at least once, especially if you struggle with writer’s block or want to write more.

The goal is to reliably summon and observe the feeling of writing, while definitely unblocked, on a series of subjects that starts out easy to write about and gets successively harder. It involves a facilitator, so it’s best done with a group or with a friend. Several people in the group thought of themselves as not being capable of prose-writing in general, but everyone who tried the exercise succeeded at it.

The facilitator explains the rules of the exercise, then picks a physical object in the room and sets a five-minute timer. For five minutes, everyone writes descriptions of the object. When the timer rounds out, the facilitator asks a question about the object. When we did it, the object was a projector. Everyone’s goal is for their 5-minute description to contain the answer the question. Since no one knows what the question’s going to be, the only way to have a high chance of answering the question is to mention everything. After five minutes were up, I pointed out the vent on the front of the projector, and asked: what’s this? Everyone’s description mentioned it, and everyone was able to write more or less continuously for the whole time without stopping. Success!

In round two of the exercise, everyone picked a project they had worked on. Same rules: after five minutes of writing, someone asks a question about your project – something basic and factual – and your goal is to have written the answer. Most people’s projects were software, so my question was “what programming language was it in”? Some other questions might have been “when did you start?” or “did you work with anyone?” This is very close to resume-writing, but because of the way it was presented, people in the room who were complaining about being stuck procrastinating and unable to do resume-writing were able to do it without any difficulty.

In round three, everyone picked a topic they’re interested in, and wrote about how they relate to that topic. This is similar to what one might write in a cover letter, but, continuing the format of the previous exercises, everyone optimized for answering all the basic factual questions. This one was harder; everyone was able to write something, but not everyone was able to go nonstop for the whole five minutes. Afterwards I asked “when did you first become interested in your topic?” and two-thirds of us had answered.

Several people observed that these exercises felt easy because they weren’t bullshit-y. So, for the fourth and final exercise, we ramped up the challenge level: Pick a villain from Game of Thrones (or some other fiction) and argue why they should rule. Someone else will independently pick a reason, and the goal is to have included that one. This one did, in fact, turn out to be more difficult; everyone managed to write something, but some of our pages were short.

My main takeaway from the exercise was the idea and mental motion of anticipating and trying to preemptively answer easy questions, but to find and answer all of them. This seems to work better for me than trying to answer a particular hard question, or to write without a mental representation of a questioner.

The Malthusian Trap Song

Speed Read This
Posted by on December 17, 2015

The Malthusian Trap is what happens to societies that reach the carrying capacity of their environment. For most of human history, humans have lived a Malthusian existence, constantly threatened by famine and disease and warfare. I wrote this for the 2015 MIT Winter Solstice. To the tune of Garden Song.

Dawn to dusk, row by row, better make this garden grow
Move your feet with a plow in tow, on a piece of fertile ground
Inch by inch, blow by blow, gonna take these crops you sow,
From the peasants down below, ‘till the wall comes tumbling down

Pulling weeds and picking stones, chasing dreams and breaking bones
Got no choice but to grow my own ‘cause the winter’s close at hand
Grain for grain, drought and pain, find my way through nature’s chain
Tune my body and my brain to competition’s demand.

Plant those rows straight and long, thicker than with prayer and song.
Emperors will make you strong if your skillset’s very rare.
Old crow watching hungrily, from his perch in yonder tree.
In my garden I’m as free as that feathered thief up there.

OpenAI Should Hold Off On Choosing Tactics

Speed Read This
Posted by on December 14, 2015

OpenAI is a non-profit artificial intelligence research company founded this lead by Ilya Sutskever as research director with $1B of funding from Sam Altman, Greg Brockman, Elon Musk, Reid Hoffman, Jessica Livingston, Peter Thiel, Amazon Web Services, Infosys, and YC Research. The launch was announced in this blog post on December 11, saying:.

Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.

Since our research is free from financial obligations, we can better focus on a positive human impact. We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as is possible safely.

This is a noble and worthy mission. Right now, most AI research is done inside a few large for-profit corporations, and there is a considerable risk that profit motive could lead them to pursue paths that are not good for humanity as a whole. For the sorts of AI research going on now, leading towards goals like medical advances and self-driving cars, the best way to do that is to be very open. Letting research accumulate in secret silos would delay these advances and cost lives. In the future, there are going to be decisions about which paths to follow, where some paths may be safer and some paths may be more profitable, and I personally am much more comfortable with a non-profit organization making those decisions than I would be with a for-profit corporation or a government doing the same.

There are predictions that in the medium- to long-term future, powerful AGIs could be very dangerous. There are two main dangers that have been tentatively identified, and there’s a lot of uncertainty around both. The first concern is that a powerful enough AI in the hands of bad actors would let them do much more damage than they could otherwise. For example, extremists could download an AI and ask it to help them design weapons or plan attacks. Others might use the same AI help them design defenses and to find the extremists, but there’s no guarantee that these would cancel out; it could end up like computer security, where the balance of power strongly favors offense over defense. To give one particularly extreme example, suppose someone created a general-purpose question-answering system that was smart enough that, if asked, it could invent a nuclear bomb made without exotic ingredients and provide simple instructions to make one. Letting everyone in the world download that AI and run it privately on their desktop computer would be predictably disastrous, and couldn’t be allowed. On the other hand, the balance could end up favoring defenders; in that case, widespread distribution would be less of a problem. The second concern is the possibility of an AGI undergoing recursive self-improvement; if someone developed and trained an AI all the way to a point where it could do further AI research by itself, then by repeatedly upgrading its ability to upgrade itself it could quickly become very very powerful. This scenario is frightening because if the seed AI was a little bit flawed, then theory suggests that the process of recursive self-improvement might greatly magnify the effects of the flaw, resulting in something that destroys humanity. Dealing with this is going to be really tricky, because on one hand we’ll want the entire research community to be able to hunt for those flaws, but on the other hand we don’t want anyone to take an AI and tell it or let it start a recursive self-improvement process before everyone’s sure it’s safe.

At this point, no one really knows whether recursive self-improvement is possible, nor what the interaction will be between AI-empowered bad actors and AI-empowered defenders. We’ll probably know more in a decade, and more research will certainly help. OpenAI’s founding statement seemed to strike a good balance: “as broadly and evenly distributed as is possible safely”, acknowledging both the importance of sharing the benefits of AI and also the possibility that safety concerns might force them to close up in the future. As OpenAI themselves put it:

AI systems today have impressive but narrow capabilities. It seems that we’ll keep whittling away at their constraints, and in the extreme case they will reach human performance on virtually every intellectual task. It’s hard to fathom how much human-level AI could benefit society, and it’s equally hard to imagine how much it could damage society if built or used incorrectly.

Yesterday, two days after that founding statement was published, it was edited. The new version reads:

OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.

Since our research is free from financial obligations, we can better focus on a positive human impact. We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible.

The word “safely” has been removed. There is no announcement or acknowledgement of the change, within that blog post or anywhere else, and no indication who made it or who knew about it.

I sort of understand why they’d do this. There’s a problem right now with ignorant press articles fearmongering over research that very clearly doesn’t pose any risk, and seizing on out-of-context discussions of long-term future issues to reinforce that. But those words where there for an important reason. When an organization states its mission in a founding statement, that has effects – both cultural and legal – lasting far into the future, and there’s going to come a time, probably in this century, when some OpenAI’s researcher is going to wonder whether their latest creation might be unsafe to publish. The modified statement says: publish no matter what. If there’s a really clear-cut danger, they might refuse to publish anyways, but this will be hard to defend in the face of ambiguity.

OpenAI is less than a week old, so it’s too early to criticize them or to praise them for anything they’ve done. Still, the direction they’ve indicated worries me – both because I doubt whether openness is going to be a safe strategy a decade from now, and because they don’t seem to have waited for the data to come in before baking it into their organizational identity. OpenAI should be working with the AI safety community to figure out what strategies to pursue in the short and long term. They have a lot of flexibility by virtue of being a non-profit, and they shouldn’t throw that flexibility away. They’re going to need it.