Hacked!

I waited until my wife got up a few hours later to rant and rave and yell. A company presents itself as the world’s leader in handling big data; it attracts users to its services (albeit free, but indispensable to the company as advertising vehicles) with the idea that trusting cloud services is moderne; and then it exposes them to something few sane users would accept on their own—the risk of “single-point failure” that could in a few minutes eliminate many years’ worth of crucial data. This is the same company famed for making every bit of data part of the world’s “permanent record.” That embarrassing picture of you in a nightclub, that subversive definition of “santorum”—they and other ephemera are eternal, but all your e‑mail can disappear before noon?

At this point I thought: To hell with journalistic detachment. Over the years I’d come to know lots of people at Google, and I decided to forward the “Unfortunately …” message to one of them. This friend was not Eric Schmidt, the company’s longtime CEO and now executive chairman, whose family my wife and I had gotten to know long before his Google era. (Embarrassingly enough for us, and possibly for him, he had received one of the “Mugged in Madrid” notes, which he passed on to me with a terse “Deb’s e‑mail has been hacked” subject line.) But the person I sent it to, Michael Jones, Google’s “Chief Technology Advocate,” was in a position to direct extra attention to the problem. My message was: You (Google) cannot be serious about this. You cannot entice people into relying on your services, and be so cavalier about the risks they are exposed to. Can you?

A little more than a week later, after several more warnings that “unfortunately” nothing might be recoverable, my wife did in fact get her messages back, all 4+ gigabytes. The first thing I did was to back them all up onto her hard disk, with Thunderbird—and then back up those archives elsewhere, just in case. But one of the next things I did was to arrange a trip to Mountain View to try to understand what had happened. My main discoveries exposed the gulf between the way information professionals understand the realities, vulnerabilities, and responsibilities of the cloud era and the way the rest of us do.

After interviews at Google with staff members ranging from the senior officials who set security policy to the young engineers who had eventually figured out how to recover maliciously deleted e‑mail (including my wife’s), plus follow-up interviews elsewhere, I had three “key takeaways,” as they say in the tech world. They involved the scale of the hacking problem, not just for individual users like us but for organizations; the nature of the arms race between people trying to steal or alter data and those trying to protect it; and the expectations of what citizens need to do to protect themselves.

At Google I asked Byrant Gehring, of Gmail’s consumer-operations team, how often attacks occur. “Probably in the low thousands,” he said. “Per month?,” I asked. “No, per day,” followed by the reassurance that most were short-lived “hijackings,” used to send spam and phishing messages, and caused little or no damage, unlike our full-out attack. My wife and I, having heard from half a dozen friends who’d recently had similar problems, had innocently imagined that we all were part of some general upsurge in Gmail attacks. In our grandiosity, we thought it was perhaps even aimed at journalists. But according to the experts, while there are more e‑mail attacks worldwide than a year ago, it was mere coincidence that people we knew had been hit around the same time. On average, half a dozen accounts are taken over every two or three minutes, round the clock, including now.

Why are so many accounts so vulnerable? Again we were naive in the assumptions we’d made about our own case. We’d just returned from China, where everything about Internet use in general and Google services in particular is insecure and fraught. Had some malware made its way onto our computers? What about that time my wife entered her Gmail username and password on a public terminal at the airport in Zhuhai? Might it have had a keystroke logger, recording names and passwords for later misuse?

Perhaps it did, but that didn’t seem to be our problem. As in the great majority of hacking cases, my wife had been using the same password for her Gmail account as for some other, less secure sites, where her username was her Gmail address. (Who hasn’t done this?) And one way or another, a list of e‑mail addresses and associated passwords from one of those sites had made its way to hackers. A possible source was the notorious Gawker hack of 2010, when more than a million e‑mail addresses and matching passwords for people who had registered on Gawker sites were stolen. “If you have ever used the same password in more than one place, you have reduced your overall safety record to whichever site had the lowest amount of protection,” Jones told me during my visit. Yet this is an overwhelmingly common practice. An analysis of posted username/password combinations after several recent hacks showed that in two-thirds of the cases, if you knew a user’s password for one site, you knew it for another.

It’s possible, too, that my wife’s password was simply “guessed,” though in a different way from what laymen might assume. Guessing less often involves social engineering—trying your birthday or your hometown or your relatives’ names—than “brute-force attacks,” in which a hacker’s computer tries every word or combination of words in existence, in a variety of languages, to see if it finds a match. From most officials, I heard reminders that if a password can be found in a dictionary, that password is not safe. Andrew Kovacs, communications manager for the Google security staff, added: “And those tricks about changing E’s to threes and O’s to zeros? Sorry to tell you, but the hackers have thought of those too.” Several of the people I spoke with pointed out that brute-force attacks have recently become much more effective, as hackers have taken advantage of the powers of new computer-graphics chips, which can handle certain kinds of computations even more quickly, and with more parallel processes running simultaneously, than a computer’s central processing chip can. These turn out to be the computations necessary for producing password hacks.

My wife’s password was judged as “strong” when she first chose it for use with Gmail. But it was a combination of two short English words followed by numbers, so if it didn’t leak from some other site, it might just have been guessed in a brute-force attack. For reasons too complex to explain here, even some systems, like Gmail’s, that don’t allow intruders to make millions of random guesses at a password can still be vulnerable to brute-force attacks.

Once her password got out, it ended up, according to later sleuthing by officials at Google, in the hands of someone possibly in Cote d’Ivoire but probably in Nigeria. My wife and I could again flatter ourselves that we were the objects of Chinese state security, but the probable truth is more mundane—“it’s like being the victims of pickpockets,” as Jay Nancarrow, of the Gmail team, put it. Why Nigeria? All of the people I spoke with had thought about this question—“I think about it every day,” Bryant Gehring said—and no one had a fully convincing answer. Their hypotheses involved some combination of talent, opportunity, and lack of legal controls. “They have the Internet, they can get rich this way, there’s not really a lot of government enforcement,” Gehring said. “For a lot of crafty people, it can be the best way to make a living.” That could apply to a lot of places, but it seems to be most relevant to West Africa.

In “The Chilling Story of Genius in a Land of Chronic Unemployment,” this past May in TechCrunch, Sarah Lacy portrayed a number of the hackers she had met in Lagos. In other circumstances, she said, the best of them might have been like Sergey Brin or Max Levchin, the immigrants who co-founded Google and PayPal, respectively. They were that clever and technically gifted. Or, more modestly, they could have been like the engineers and managers I’ve met over the years at Google, Microsoft, Intel, and other companies, at least half of whom were born and raised overseas. But these hacking entrepreneurs couldn’t get out of Nigeria, and so they dealt with the outside world via “Mugged in Madrid” messages. Shreyas Doshi, a Google senior project manager, said that the company had run analyses to see how much money the scams might produce. “With a variety of assumptions, we believe they could easily make about $500 a day, if not more,” he said; that many people fall for these scams.

The greatest practical fear for my wife and me was that, even if she eventually managed to retrieve her records, so much of our personal and financial data would be in someone else’s presumably hostile hands that we would spend our remaining years looking over our shoulders, wondering how and when something would be put to damaging use. At some point over the past six years, our correspondence would certainly have included every number or code that was important to us—credit-card numbers, bank-account information, medical info, and any other sensitive data you can imagine.

The long siege of identity theft could still happen, and I have put all the credit-card companies on alert. But a few months into our post-hacking life, we’ve seen no indication of trouble, and according to the Google officials, the hacker would depart from past patterns if he began using the data in the way we most feared. “We haven’t had a major incident with credit-card [numbers obtained from scanning Gmail] yet,” Bryant Gehring said. The economics of hacking are constantly changing, so we are still on guard. But security officials at Google and elsewhere said that for talented hackers in Lagos or Abidjan, it is generally safer and more rewarding to collect money at Western Union, which can be done virtually untraceably, as opposed to poring through e‑mail archives in search of data to create identity-theft schemes. Why, then, did the hacker bother to erase all the archives? “Because he—or she—was a good hacker,” Gehring said. “He wanted to make it harder for you to get in touch with all your contacts and warn them not to send money to Madrid. You remember their names but probably not all their e‑mail addresses.” Many e-mail hackings, whether on Gmail or another major system, involve deletion of mail that arrives after the hijacking; only a destructive minority involve a complete zeroing-out of the archives like what happened to my wife.

Against this assault, the Google security team, like its counterparts at other companies, is constantly monitoring activity across its systems, toward the end of detecting break-ins and hijacks before damage has been done, and even before the owners know that something has gone wrong. The signs they’re looking for have endlessly evolved: this year’s “Mugged in Madrid” message was last year’s “Mugged in London,” with enough minor changes to avoid many filters. “Google is a very data-centric place,” Eric Grosse, Google’s head of security, said to me. “We log lots of things and constantly have computers crunching over this data, trying to look for subtle patterns and keep up with the continual changes. That lets us do a better job of spam filtering than you could do on your own machine, since we see so much of it. We try to use the same techniques to look for other signs of abuse.” Jay Nancarrow, of Google’s communications team, added: “A lot of the signals we analyze are evident only because we’re offering the service to so many people. This is the kind of machine-learning the cloud enables that is very hard otherwise.”

I kept asking for examples of such tells; perhaps reassuringly, the Google officials kept saying they would rather not get into details. “But suppose we see that all your friends start marking mail from your account as spam,” Shreyas Doshi said. Obviously he didn’t mean that any human being at Google is monitoring your correspondence. Rather, its systems, like those of other companies that handle large numbers of online accounts, are constantly searching for anomalies. “Or if suddenly the recovery phone number in your account is changed to a number in South Africa. Or if you suddenly delete 20,000 e‑mails at once.” Any of these would be a sign that something might be going wrong, which Google would use to automatically immobilize the account. Indeed, the deletion of tens of thousands of e‑mails at once is what happened to my wife—and according to Gehring, the monitoring system had already noted an anomaly and was beginning to freeze the account by the time we filed our first report.

Perhaps the most startling thing I learned at Google about my wife’s case was how “lucky” we had been. Lucky not in having friends we could turn to in the otherwise automated and unapproachable Google edifice—though, of course, we were—but simply in the timing of the attack. If this had happened six months or a year earlier, or if it happened even today at most other e-mail services, the archives would likely have been gone forever. It was only because of the Undeletion Project that recovery, although slow, was feasible at all.

How could this be? How could big tech companies offer cloud services to hundreds of millions of people without better guarding their data against catastrophic loss? On Google’s side, one explanation involved complexities of the law. My wife and I might think that Google had a “duty” to be able to find her messages after some hacker had erased them. But according to Google’s legal department, its higher and more stringent duty is to ensure that messages are erased, if whoever is in charge of an account wants them gone. Political activists in repressive countries, people who for whatever reason (@RepWeiner) want parts of their electronic correspondence to disappear—they are the ones Google, like other e‑mail providers, had in mind in designing a system optimized for deletion rather than recovery. In exceptional cases, mainly in response to government orders in criminal or anti-terrorism investigations, Google could laboriously piece together already deleted records from its tape backups. But such recovery efforts were slow, hand-crafted undertakings, impractical for responding to thousands of hacking episodes a day (only some of which lead to total erasures).

“Every year, we do an immersion session with people involved in consumer operations,” I was told by Birendro Roy, one of the two young engineers at Google who developed the software for the Undeletion Project. He and his colleague Jaishankar Sundararaman were introduced to me as “the people who actually restored your wife’s mail.” (Roy, whose father is from India and mother is from Finland, grew up in Baton Rouge and went to MIT. Sundararaman grew up and went to college in Tamil Nadu, in southern India, and then did graduate work at Virginia Tech. I mention this as part of my ongoing chronicle of America’s strength through attracting foreign talent.) “The consumer reps tell us what are the common user ‘pain points,’ and restoring mail had definitely been something users were requesting.”

What a surprise, that people would want to recover from catastrophe! But from Google’s engineering perspective, the deleted-mail problem, while dire for those confronting it, affected only a tiny fraction of their users, and also was more complicated to solve than some other mainstream usability issues. In companies like Google, relatively few innovations are the result of top-down orders from executives. More evolve bottom-up, as engineers and product managers become sold on the need to add a new product or feature to the company’s offerings. Sometime last year, the Gmail engineers became sold on the value of a recovery system for maliciously deleted e‑mail. My wife’s luck was that her hack occurred this year rather than last—“Before, users were pretty much out of luck,” Bryant Gehring said—and that it hit the debugging cycle just when it did. To streamline a long story, the Undeletion Project software still had an important bug at the time of our first complaint. When the recovery software reached a message of a certain sort (“it was a ‘corner case’ involving saved drafts,” Sundararaman said), it mistakenly concluded that it had gotten to the end of the e‑mail list and could find nothing more. The engineers discovered the bug at just the time I was launching my “You cannot be serious!” screeds from outside. So now, if you lose your e-mail and start the recovery in time (deleted e-mail stays in a theoretically recoverable limbo for as long as 30 days, after which it’s likely gone for good), you should be able to get it back. In September, Google began offering a live help line for e-mail recovery, at a small cost (to deter abuse) of a few dollars.

“Of course, the very best way to recover data is not to lose it in the first place,” Shreyas Doshi said, in outlining what he described as a three-tier philosophy of data protection that was shared by Google, Yahoo, and other “cloud” companies. But that won’t help people like Deb when they make their horrifying discoveries. So Doshi outlined a tiered strategy I endorse for anyone who runs or uses e‑mail or Internet services—which is to say, everybody.

The first tier is for companies to give users a more realistic view of where and why they might be vulnerable, so they can do more to protect themselves. The second is for companies to reduce the incentive toward Internet crime by making it less profitable for the perpetrators—for instance, with better, faster ways of detecting hijackings, so that accounts will automatically be closed, even before the first “Mugged in Madrid” spam goes out. (And it seems that someone might usefully have a word with Western Union too.) The third is for companies to create better recovery systems, like the Undeletion Project, for when steps one and two fail.

Discussion of the cyber-threat menace has become a boom industry on its own. My latest immersion in the topic has left me more consoled than I might have expected—especially when I was imagining, early on, what the intrusion on my own household might mean to everyone else on the Internet who might innocently have exposed un‑backed‑up data to similar risk.

True, there is a lot to be worried about. “We have not examined a [corporate or governmental] network yet in which we have not discovered some level of intrusion,” Ron Ritchey, a longtime security expert now working at Booz Allen, told me. And “these things have been going on for years and years,” his colleague Ken Silva, the NSA veteran, said. “But companies are only now beginning to talk about them. After the Google report”—not about low-level cases like ours but about the massive Chinese intrusion on Google’s internal systems that the company revealed early last year—“the landscape for reporting these events really changed.” Both experts said the potential threat that concerned them most was not data theft or data loss, especially as it affected individual users, but data alteration that could affect institutions. “Imagine what could happen to the financial system, if trading data were altered,” Silva said. (Imagine: people might not trust financiers!) Or the medical data in a hospital. Or the formulas and data in spreadsheets on which companies base their bids and negotiations.

But on the other side, between us and data chaos are potentially protective factors of scale, technology, and strategy. Scale, in that big companies can invest big resources in monitoring and protecting their and their customers’ data. My money should be harder to steal if it’s in the bank than if it’s in my pocket, and something similar should be true of data. Technology, in that every person I spoke with emphasized that we’re living through an awkward stage in our current reliance on passwords. “Good passwords are bad for people, and bad passwords are good for criminals,” Michael Jones of Google said. Sooner or later, a better, easier, more reliable system of verifying identity will become widespread. It could be “biometric”—thumbprint recognition, iris scans, even a match of your face’s image, as you sit in front of the computer’s camera, with a stored picture of you. It could be purely digital, like Google’s recently introduced “two-step verification” system for Gmail identity, in which users enter both their password and a special code sent by Google to their cell phone or mobile device. None of these is perfect, but each one is a big advance over security now.

And strategy, in the form of a corporate and governmental emphasis on “data hardening” to limit the resulting loss when some attack inevitably succeeds. No free society can completely eliminate loss from crime, accidental death or destruction, even terrorism. The sane strategy is to prepare to contain the damage if and when things go wrong. Airbags in cars, watertight compartments in ships, hardened cockpit doors in airliners—these all illustrate the approach. The scary part in my wife’s case was that failure in one place meant potential failure of the system as a whole. Because she had used her password in one vulnerable place, all of the electronic data that mattered to her was at risk. If the data-hardening principle got through to me in the course of interviews, it has certainly occurred to security professionals at the institutions that collectively make up our new cloud info-system.

What about the rest of us, who are not security professionals? I asked that of every person I interviewed. Many of their recommendations boiled down to the hope that people would think more about their life online. “We’d like people to view their information life the way they view other parts of their life,” Andrew Kovacs of Google said. “It’s a good practice to review your financial situation every so often, and it’s a good practice to review your passwords and online-account information too.” Another official compared “cloud hygiene” to personal hygiene: you feel bad if you don’t brush your teeth or take a shower, and you should learn to feel bad if you’re taking risks online.

In practical terms, I have three action points I preach to anyone who will listen. Really, only two and a half.

The half step is extremely important, but I count it only as a half because it applies specifically to users of Gmail rather than other online systems. Here it is: if you use Gmail, please use Google’s new “two-step verification” system. In practice this means that to log into your account from any place other than your own computer, you have to enter an additional code, from Google, shown on your mobile phone. On your own computer, you enter a code only once every 30 days. This is not an airtight solution, but it can thwart nearly all of the remote attacks that affect Gmail thousands of times a day. Even though the hacker in Lagos has your password, if he doesn’t have your cell phone, he can’t get in.

In case you’ve missed the point: if you use Gmail, use this system. Also, make sure the recovery information for your account—a backup e-mail address or cell phone where you can receive password-reset information—is current. Google uses these to verify that you are the real owner.

Next we have password selection, that seemingly impossible task. The science, psychology, and sociology of creating strong passwords is a surprisingly well-chronicled and fascinating field. On The Atlantic’s Web site, we will describe some of the main strategies and the reasoning behind them. Even security professionals recognize the contradiction: the stronger the password, the less likely you are to remember it. Thus the Post-it notes with passwords, on monitor screens or in desk drawers.

But there is a middle ground, of passwords strong enough to create problems for hackers and still simple enough to be manageable. There are more details on our site, but strategies include:

• Choose a long, familiar-to-you sequence of ordinary words, with spaces between them as in an ordinary sentence, which more and more sites now allow. “Lake Winnebago is deep and chilly,” for instance. Or “my favorite packer is not brett favre.” You could remember a phrase like that, but a hacker’s computer, which couldn’t tell spaces from characters, would see only one forbiddingly long password sequence.

• Choose a shorter sequence of words that are not “real” English words. I once lived in a Ghanaian village called Assin Fosu. I can remember its name easily, but it would be hard to guess. Even harder if I added numbers or characters.

• Choose a truly obscure, gibberish password—“V*!amYEg5M5!3R” is one I generated just now with the LastPass system, and you’re welcome to it—and then find a way to store it. Having it written down in your wallet is one, though the paper it’s on shouldn’t say “Passwords” at the top. The approach I prefer, and use for some passwords, is to entrust them to online managers like LastPass or RoboForm. Even if their corporate sites were hacked, that wouldn’t reveal all your passwords, since the programs work by storing part of the encoding information in the cloud and part on your own machine.

At a minimum, any step up from “password,” “123456,” or your own birthday is worthwhile.

Finally, use different passwords. Not hundreds of different ones, for the hundreds of different places that require logins of some kind. The guide should be: any site that matters needs its own password—one you don’t currently use for any other site, and that you have never used anywhere else.

“Using an important password anywhere else is just like mailing your house key to anyone who might be making a delivery,” Michael Jones of Google said. “If you use your password in two places, it is not a valid password.”

I asked my experts how many passwords they personally used. The highest I heard was “about a dozen.” The lowest was four, and the norm was five or six. They all stressed that they managed their passwords and sites in different categories. In my own case, there are five sites whose security really matters to me: my main e‑mail account, two credit-card sites, a banking account, and an investment firm. Each has its own, good password, never used anywhere else. Next are the sites I’d just as soon not have compromised: airline-mileage accounts, Amazon and Barnes & Noble, various message boards and memberships. I have two or three semi-strong passwords I use among all of them. If you hacked one of them you might hack the others, but I don’t really care. Then there is everything else, the thicket of annoying little logins we all deal with. I have one or two passwords for them too. By making it easy to deal with unimportant accounts, I can concentrate on protecting the ones that matter.

At the end of my visit to Google, I went by to see Michael Jones, the friend to whom I’d first turned in data-loss despair. I told him what I’d learned, and how I would try to spread the message of shared responsibility, individual and organizational, for security in the cloud age.

“I see that you’ve got it!” he said. “The zeal of the convert. People in the business think about the risks all the time, but normal people don’t, until they’ve gotten a taste of the consequences of failure.”

I have now had that taste and am here to share the experience. As with so many other challenges in modern life, responding with panic or zealotry doesn’t get us anywhere. But a few simple self-protective steps can save a lot of heartache later on.