Lunch was supposed to be casual. Mikey Dickerson ’01 was in Chicago catching up with Dan Wagner, a friend who’d been in the trenches with him on Barack Obama’s campaign for the presidency in 2012. Wagner had since gone on to found a company, Civis Analytics; Dickerson was a site reliability engineer at Google, one of the people who make sure that the search engine never, ever breaks down.
This was October of 2013, no time for the President’s geekiest loyalists to have a little fun. Healthcare.gov, the sign-up website that was the signature element of President Obama’s signature initiative, was a technological disaster. People couldn’t sign up even if they wanted to—the site would break, or fail. Delays were interminable. Information got lost. Customer service was about as good as you’d expect from a cable TV company. The Department of Health and Human Services, responsible for the new health care system, couldn’t seem to get it working.
“So, we got this phone call yesterday,” Wagner told Dickerson. “HHS is looking for help with healthcare.gov. Can I list you as an advisor or consultant?”
“Yeah, sure. If it’s any value to you, list me,” Dickerson replied. It seemed innocuous enough. Today, he smiles at his own naïveté. “I had no idea what I was getting into,” he says. About a week later, Dickerson found himself on a 5 a.m. conference call with a van full of technologists in Washington D.C., headed over to HHS. With him in the White House motor-pool car was Todd Park, the U.S. chief technology officer. And Park, whom Dickerson didn’t know, was selling the group as a team of experts who could solve any tech problem. Dickerson realized: They’re saying I can fix healthcare.gov.
Without really meaning to, Dickerson had become an anchor of the Obama administration’s “tech surge,” a Silicon Valley-powered push to fix the bugs in the healthcare.gov system. But the system was more than just software. In D.C., Dickerson and his new team found an organization in bureaucratic and technological meltdown, unable to execute what any e-commerce start-up would consider basic prerequisites for being in business.
The crazy part is, they fixed it.
To a Connecticut native like Dickerson, good at math and computers but with no desire to attend a big university, Pomona shows itself off pretty well—especially on a campus visit in May, when Dartmouth might still have slush on the ground. It’s not that he was so avid about computer science—in those days, as a major, CS really ran out of Harvey Mudd anyway—it’s just that Dickerson was an ace. He felt like he was cheating just a little. “It seemed dumb to be spending all that money on something I was already good at,” he says. In fact, Dickerson was already coding for various companies while in school. After graduation, he ended up working in Pomona’s computer lab.
Then the 2000 presidential election came around, with its photo finish in favor of George W. Bush. “It was a trauma for me,” Dickerson says. “That razor’s edge. All that was intensely painful. Almost anything would have moved those last 200 votes.” So in 2004 Dickerson volunteered with a poll-watching group … and caught the politics bug. Four years later he was working at Google, where CEO Eric Schmidt was (and remains) a multimillion-dollar Obama supporter. During campaign season an email went to a mass-distribution list that Dickerson was on, looking for people who could manage big databases for the Obama campaign.
Hey, Dickerson thought. I manage a group that runs large databases. And that was it. He worked as a volunteer in Chicago, one of a small group of techies who, during their long nights, idly wondered if maybe they could do something useful for the campaign with better records of people’s voting history. When the 2012 campaign came around, he was still on the campaign organizers’ list. This time, though, he was no newbie—though still technically a volunteer, his experience made him a trusted veteran. Those vague ideas about leveraging voter lists went into practice, and Dickerson’s group became the analytics team, credited by some political analysts as having been the key to Obama’s re-election. Once the campaign was over, Dickerson went back to managing a site reliability engineering team at Google, but he stayed in touch with his friends—which is why Dickerson was at lunch with Wagner on October 11.
The tech team’s first stop, in Virginia on October 17, was PowerPoint Hell. Technically, it was a large IT firm working as a government contractor. “They scheduled a three-hour meeting and sent a VP with, I shit you not, a 130-slide PowerPoint presentation,” Dickerson says. Over beers in a bar on San Francisco’s Embarcadero, about a block from Google’s offices, Dickerson wears the uniform of the coder—hoodie, Google ID badge, Google T-shirt, close-cropped hair and unshaven chin. In San Francisco, that’s stealth armor. In Washington’s blue-sports-coated, khaki-pantsed hallways, he was an alien.
The group fought its way out of the meeting and took over the office of someone who was on vacation. Then they went wandering, finding teams huddled in cubicles and asking them what they were working on, which bugs they were trying to fix. But they weren’t—mostly they were waiting for instructions. In their defense, it was hard to figure out what needed fixing. Engineers weren’t really allowed to talk to clients or users, and the people who created the healthcare.gov website hadn’t even built a dashboard, a way to monitor the health and status of their own system. If you wanted to know whether healthcare.gov was functioning, the only way to find out was to try to log on. “We thought this would be a targeted assessment and we’d spend a few days there,” says Paul Smith, another member of the team. “When we realized how bad things were, we just independently decided, we’re not going home. This is what we’re doing now, for an indefinite period of time, until it gets better.”
After a couple of days, Park asked them whether it could be fixed. “Todd, they have made all the mistakes that can be made,” Dickerson told him. “We can barely find a case where, when two decisions could be made, they made the right one. But low-hanging fruit isn’t the right metaphor. We’re stepping on the fruit.” The point was, some very simple fixes would yield some very big gains. Any improvement would be a massive improvement. Google site reliability engineers have a saying—they tell each other, if we have an outage that big it’ll be on the front page of The New York Times. Is that what you want? “But here’s the thing,” says Dickerson. “Healthcare.gov had been on the front page of The New York Times for four weeks. That was the silver lining. How much more could I screw it up?”
The group of coders decided that if no one was telling anyone what to do, they would. That’s when they started getting called “the Ad Hoc Team.” The name stuck. “We had a big stick, because we were the magical guys from the White House,” Dickerson says. “After a couple of days, we instituted a war room.” Every morning at 10 a.m., every team had to send a representative to a big meeting to explain what was going right, or wrong, and why. “It was an incredibly expensive thing to do—60 people in a room while we arbitrate disputes between two of them. But we made so much progress we stopped worrying,” Dickerson says. “Having a giant studio audience is better sometimes. It’s harder to say, ‘I didn’t do that because it wasn’t on my task order.’”
In other words, Dickerson had built into the system something no one had thought of: accountability. “What Mikey really excelled at was, if there’s a priority issue that needs to be addressed, how can people address it? What do they know? What do they need to know? What’s blocking them?” says Smith. “That’s just his demeanor and the way he operates.” The meetings were so productive and making so much of a difference in site performance that the Ad Hoc Team instituted a second one, making them twice a day, seven days a week.
When they weren’t in the war room, they coded. Problems started getting solved. A stupid little flaw that required the same kind of wait to connect to the database every time went away with the change of a couple of configuration settings, and poof! An eight-second response delay dropped to a two-second delay. “And that’s still terrible,” Dickerson says. The site stopped crashing. People actually started signing up for health care.
The work took a toll, though. Except for a quick trip back to California to pick up some clothes—Dickerson had come to the East Coast with a carry-on bag and a Google computer, expecting a short visit—he was in the greater D.C. area from mid-October through Christmas. Dickerson estimated he ran 150 war-room meetings in a row.
After a couple of moves to accommodate bureaucracy, Dickerson ended up working remotely, alone, from an operations center in Columbia, Md.—three hours from D.C. in what locals sometimes call “spook valley” for its preponderance of government contractors. Since healthcare.gov’s original creators hadn’t built a ship-in-a-bottle version of the software to test updates and fixes, everything the Ad Hoc Team fixed had to get changed on the live site, and the primary maintenance window was when traffic was lightest, between 1 and 5 a.m. “It was literally 20-hour days a lot of time. ” Dickerson says. “I was hallucinating by the end, hearing things.”
With 12 days left before the deadline, Dickerson was ready to go home. He gave a speech listing the five mission-critical things remaining, and attempted to flee back to California. But the bosses panicked. The Ad Hoc guys can’t go home, they said. They gave him the service-to-your-country pitch. They begged. So Dickerson agreed to stay through to the end—with some conditions. He got to set the specific technical goals for what his team and the rest of the government coders would do. And he got to hire whomever he wanted, without arguing the point. He wanted to be able to trust the new team members, so he chose them himself. Eventually a rotating team of Google site reliability engineers started coming through to keep the project on track.
Dickerson got to dictate those terms because he was getting results. He had become indispensable. “Mikey is an incredible talent who was seemingly built in a lab to help fix healthcare.gov,” Park says. “It’s not just the fact that he’s got a sky-high tech IQ, honed over years as a star site reliability engineering leader. He’s also got tremendous EQ, enabling him to step into a tough situation, mesh well with others, and help rally them to the job at hand.”
The real bummer, of course, is that healthcare.gov, while an unprecedented attempt to link government services, private insurers and identity verification, shouldn’t have been that hard to build. “It’s basically a distributed, transactional, retail-type website, and we’ve been building those for years,” says Smith. “In the private sector, we know how to do that. We’re not forging new computer science ground here, right?”
By April of 2014, just a few days after Dickerson and I spoke, the Obama administration announced that over 7 million people had signed up for private health care through federal and state exchanges, and 3 million had signed up for Medicare. The program had made its numbers—barely, to be sure—because people, in the end, could actually use the website.
Dickerson is back at Google, but as he says, “you can never unsee the things you see in the federal government.” He has become an outspoken advocate for reform in the ways government builds technology, concentrating especially on trying to convince young technologists to go work for government. “You’re gonna eat free food and drink free soda in micro-kitchens and work on another version of what we’ll say, for argument’s sake, lets people share pictures of what they ate for breakfast, and tens of thousands of people will die of leukemia because we couldn’t get a website to work,” Dickerson says. “These are real people’s lives that will end in 2014, and you’re going to sit at your desk working on picture sharing.”
The problem isn’t competence. People who work on websites for the government are every bit as competent as the ones who work at Google or Facebook. “The mechanisms by which you do a contract with the federal government are so complex that it requires expertise in and of itself,” says Jennifer Pahlka, founder and executive director of Code for America, a group that connects software developers with local governments. “Fundamentally the process in government has evolved to meet government needs. A federal project has dozens of stakeholders, none of whom represent the user.”
That’s why Code for America focuses on local governments, Pahlka says. The feds are too hard to crack, and anyway, most people’s interactions with government are at the state and city level—think DMV, local parks, or trash pick-up. So Dickerson has started stumping for Code for America, giving speeches at their events. And he is lobbying Eric Schmidt and his other bosses at Google to develop programs that would allow—maybe even encourage—software developers there to take time to work on government projects. Consider: The feds paid $700 million for healthcare.gov, and it didn’t work. Imagine being able to bid for that contract at a tenth the price. “I don’t have to appeal to your altruism or desire to serve your country,” Dickerson says. “I can just say, ‘Do you want to make a ton of money?”
Pahlka thinks the pitch might actually work—and not just because of capitalism. “The consumer internet has influenced the way a generation feels about doing things together,” she says. “You have a generation of people who value collective intelligence and collective will—not necessarily collective political will, but the ability to actually do things together.” Software designers and engineers are already political, Pahlka and Dickerson are saying; it’s just that the web generation is ignoring the greater good. Going to work at Twitter is a political choice just as much as going to work for the Department of Veterans Affairs.
“I give the worst sales pitch,” Dickerson says. “I tell people, ‘This is what your world is going to be like: It’s a website that is a Lovecraft horror. They made every possible mistake at every possible layer. But if you succeed, you will save the lives of thousands of people.’”
The weird part: Almost everyone says yes.
————-
EDITOR’S NOTE: Shortly before this magazine went to press, Dickerson announced that he’s going to practice what he preaches, full time. He is leaving Google to join the Obama administration as administrator of the U.S. Digital Service, a newly created office overseeing government spending on information technology. And after signing on, he discovered that the lead designer on the initial staff for U.S.D.S. is another Pomona grad, Mollie Ruskin ’08.