Learning the Lessons from Google’s Failure in Los Angeles

By Jeff Gould, CEO & Director of Research, Peerstone Research

Summary

Is Google serious about its push to sell Google Apps to big companies and government agencies? Is the world’s leading online advertising firm really committed to becoming a major player in enterprise software? Based on the evidence so far, the answer is unclear. In this article we examine the pros and cons of Google Apps in the enterprise. After defining what Google Apps is, we review recent research by Gartner about the rise of cloud-based solutions in the enterprise email market and Google’s early success in that market. We then look at the reasons for the stalled deployment of Gmail at one of Google’s largest public sector accounts to date, the city of Los Angeles. We conclude that while Gmail is a first-rate consumer email product, Google does not presently have the right products, the corporate culture or the sales volume required to be a significant player in enterprise software.

What is Google Apps, exactly?

First of all, let’s clarify what we mean by Google Apps. Google Apps began life about five years ago as a bundle of the company’s free consumer Gmail service with Google Calendar and a few other features. Today the product has evolved into a rich portfolio of web-based apps, including Google Docs (word processing, spreadsheets, presentation), Google Sites (simple web page creation), Google Video, and others. These offerings are distilled into several editions, including Google Apps for Business, Government, Education, and Nonprofit, in addition to the basic edition. The latter is now free only up to 10 users, considerably less generous than the 200 free accounts originally offered back in 2006. Pricing on the vertical editions ranges from $50 per user per year for Business and Government to free for most Education and Nonprofit users[1].

So much for the theory. In practice, most large Google Apps deployments – the major account wins that Google hypes on its web page – are limited to Gmail and Calendar. Why this should be so is an interesting topic that we don’t have room to consider here. The most important reason is probably that while Gmail is a functionally very rich and mature consumer email service, the productivity apps in Google Docs still fall far short of their Microsoft Office equivalents in features, performance and file compatibility. Interestingly, Gmail was developed from scratch at Google, while the other apps were patched together from a string of acquisitions of tiny Silicon Valley startups, none of which were originally designed to work with the others.

How big is Google Apps?

It’s hard to say how much revenue Google earns from Apps. A recent and generally quite favorable Gartner report on Gmail in the enterprise[2] notes that Google Apps is “no more than a hobby for Google”, accounting in Gartner’s estimate for less than 1% of Google’s revenue, and an even smaller share of the firm’s profit. Still, considering that Google is on track to earn revenue somewhere around $35 billion in 2011, “less than 1%” is not necessarily a small number. It’s safe to say that the Apps business certainly brings in more than $100 million per year, and perhaps as much as twice that. Profits, as Gartner points out, are another matter. It would be very surprising if the Google Apps business is profitable today or even on track to achieve profitability in the near future (if it were, Google would have said so by now). But it’s probably not losing hundreds of millions of dollars per year either.

While Google Apps is not a very big business revenue-wise, it has made a big splash in the media and has notched up an impressive number of wins in the enterprise market. Gartner puts Gmail’s share of the enterprise email market at roughly 1%. So depending on how you look at it, the enterprise Gmail glass is 1% full or 99% empty.  But things look better for Google when we consider just the cloud segment of enterprise email. Here Gartner estimates Gmail’s share at close to half the market, with an equal portion going to Microsoft’s competing cloud offering, Office 365[3].

Enterprise email as a whole is a fairly mature market. Most analysts agree that Microsoft Exchange and IBM Lotus Notes continue to dominate the on-premises portion of the enterprise email market, with these two vendors sharing around 90% of the installed base[4]. The cloud segment is still quite small – no more than 3% to 4% of the total, according to Gartner[5] – but it is growing at a much faster clip than the rest of the market, perhaps three times as fast. The reason is clear – moving email from an organization’s own data center to a vendor’s cloud relieves that organization of the capital, labor and recurring operating costs of all those servers and the software that goes with them. While it turns out that migration is often a bigger and more costly hurdle than most people think, the long term return on investment from cloud email is nevertheless compelling. This is why Google Apps and Gmail in particular have made such a big impression so fast.

But despite this early success, enterprise cloud email appears to be growing more slowly than expected. Gartner has just revised downward its very optimistic forecast that cloud would account for 10% of enterprise email by the end of 2012. It has now pushed that anticipated benchmark back to the end of 2014. Why? Partly because many organizations, especially in the private sector, have existing email systems that are quite modern and functional, thus reducing the incentive to replace them immediately. And partly because the “practical realities” of the cloud email vendor offerings are, as Gartner delicately puts it, “sometimes less compelling than the glossy stories they tell”.

A third factor is that replacing one email system with another is not always high on the CIO’s list of strategic priorities. The truth is that while cloud email does have attractive features, the migration is not likely to have a major strategic impact in most large organizations. It’s not going to help a firm develop new products, enter new markets, drive top-line growth or improve profits. Its real justification is simply the promise of long-term cost savings. But it happens that there is one sector of the economy where saving money right now is the biggest strategic priority of all – government. And so it comes as no surprise that this is the market where Google Apps and Gmail have scored their biggest wins to date. In the next section we look at one particularly prominent example of Gmail in government, the city of Los Angeles.

Google Apps in LA: how an ambitious project went wrong

Probably no other government cloud email deployment has received more attention than the city of Los Angeles’ ongoing attempt to migrate some 30,000 users of an antiquated on-premises email system (based on Novell GroupWise) to Gmail. What makes this case exceptionally interesting is the rich paper trail of official documents that detail almost every step of the project’s progress – or lack thereof[6].

When Los Angeles CTO Randi Levin pitched Google’s three-year $7.2 million proposal to the City Council back in 2009, it was greeted with enthusiasm. And it’s easy to see why. Going from GroupWise to Google Apps would allow the city to repurpose (or retire) 92 servers in the city’s data centers and cancel hundreds of thousands of dollars worth of GroupWise licenses. Longer term, the city also hoped to cut back drastically on its Microsoft Office licenses. In addition, the move would save nearly $150,000 per year in power and cooling costs, and would free up the 16 full-time positions devoted to supporting the old system. While the shift from the GroupWise Windows client software to Gmail’s web interface did entail some loss of functionality for the city’s email users, the latter would benefit from a huge gain in email storage space, which would soar from one to 25 gigabytes per user. All this would be delivered for less than $50 per user per year, not counting one-time implementation, training and migration costs. What was not to like?

The city’s original plan was to move all users from GroupWise to Gmail by December 31, 2009. This date was important, because on January 1 the city would have to pay another half year of GroupWise license fees. But this plan proved to be wildly overoptimistic. Today, nearly two years later, 13,000 of the city’s 30,000 users are still on GroupWise. The original plan also called for 80% of the city’s users to migrate from Microsoft Office to Google Docs for basic tasks such as word processing, presentations and spreadsheets. However, this migration, which was to have taken place over two years, now appears to be on indefinite hold, and there is real doubt as to whether the basic email migration itself will ever be completed. Both the rapid deployment of Gmail (allowing the early termination of the city’s GroupWise licenses) and the gradual shift away from Microsoft Office were key elements in the cost savings the city expected to achieve from the project. Without these elements, much of the city’s business case for the migration falls apart.

The main stumbling block for Gmail in Los Angeles has been security. On the eve of the scheduled migration of its 13,000 employees in the spring of 2010, the Los Angeles Police Department announced that a long list of promised improvements to Gmail’s security had not been completed and that as a consequence the department could not move forward with the migration[7]. This decision was the first in a series of increasingly serious setbacks that have turned the LA deployment from a shining example of public sector cloud computing (one that is still touted on Google’s web site) into an embarrassing debacle.

What were the LAPD requirements that Google couldn’t meet? Some involved missing features such as email receipt confirmations (which have subsequently been added) or the integration of Gmail with other IT systems, such as the city’s electronic system for distributing subpoenas to police officers. But the most important requirement concerned the security of the back-end servers used to implement Gmail in Google’s data centers. During initial negotiations, the city told Google it would require these servers to be physically separated from servers used by non-government customers, and in 2010 Google duly rolled out its Google Apps for Government edition, which did indeed isolate all servers used to deliver Gmail and Calendar services to government customers in the United States. However, Google did not isolate the servers used for Google Docs or its Postini eDiscovery service. LAPD  told the City Council it didn’t mind dropping Google Docs, but considered that eDiscovery (i.e. archival and search functions for the department’s mandatory email retention policies) was indispensable. Without the assurance that these servers would also be physically segregated, the department decided to put its migration on hold.

But there was more. LAPD, like all police departments, is an intensive user of State and Federal criminal records databases. Access to these systems is tightly controlled by a set of Department of Justice protocols supervised by the FBI. In addition to restrictions on system access and extensive training and audit requirements, employees of government contractors involved in the administration of IT systems through which such sensitive data transits must pass stringent criminal background checks (including the submission of fingerprints). The prime systems integrator on the contract, CSC, assured the city that meeting these requirements would not be a problem.

But in October 2010, just a week before the promised deadline, Google unexpectedly told CSC it would not be able to meet the requirements after all. Just what the problem was has not been publicly disclosed. But according to Gartner, one possible issue is the fact that some of Google’s support personnel with access to the Apps for Government servers are based in Europe. Gartner reports that Google hopes to meet the FBI’s requirements by the first quarter of 2012 – that is, more than two years after the contract with the city was first signed[8]. But it now appears increasingly unlikely that LAPD will agree to restart its long delayed migration to Gmail.

“These failures are wholly unacceptable…”

Reading the project documents published on the city’s web site or leaked to the press, we witness a progressive breakdown in communication between the city and its vendors. Consider the following excerpts:

• In July 2009, the City Administrative Officer, in a report evaluating the proposed contract with Google, optimistically wrote that: “The shift in control over the City's e-mail and office applications is mitigated by service level agreements guaranteeing a high level of service from Google.”
• In September 2009, Los Angeles CTO Randi Levin, in a notably upbeat report, informed the City Council that the issues concerning the police department’s security needs had been resolved: “This Security offering [i.e. Google’s promised Gov Cloud] is a significant step above the security given a tentative approval from CA DOJ… I am very excited to offer this to the LAPD...”
• By the following spring, however, signs of trouble began to appear. In April 2010 the City Administrative Officer informed the City Council that: “[M]any of the departments [have] expressed concerns about both the performance and the functionality of the new system.”
• Again in July 2010, the City Administrative Officer reported to the Council that: “There continue to be performance problems with Google where e-mails are not delivered in a timely fashion.”
• Also in July 2010, LAPD informed the City Administrative Officer that: “At this time, a definitive timeline for LAPD migration cannot be provided.”
• Then, in November 2010, the delays erupted into a full-fledged crisis of confidence between the city and CSC/Google, leading CTO Levin to dispatch a formal Notice of Deficiencies to the contractor that bluntly expressed the city’s disappointment: “These failures are wholly unacceptable to the City of Los Angeles. CSC and Google have repeatedly committed to meet particular deliverables on specific dates, only to reveal, at the last minute, that the set deliverables/dates will not be met. CSC and Google’s behavior goes beyond a mere failure to communicate in a timely manner, and instead, on several occasions, has risen to the level of misrepresentation.”
• In April 2011, the Los Angeles Times wrote that: “The city's controller has launched an inquiry [into Google’s inability to meet security requirements], and City Council members have held closed-door meetings to confer with attorneys, raising the possibility of legal action.”
• In May 2011, CSC sent a letter to CTO Levin in which, according to Levin, CSC acknowledged that it and Google would be “unable to complete and comply with all of the City’s security requirements.”
• Finally, in August 2011, Levin sent a proposed contract amendment to CSC stipulating that “without completion and compliance with all City security requirements, the LAPD and other law enforcement employees will be unable to migrate to Google Apps”. Since the city’s 13,000 law enforcement employees have been forced to continue using the legacy GroupWise email system far beyond the expected migration date, the contract amendment also requires that Google “shall pay the City for the GroupWise System Costs for the period of July 1, 2011 through November 20, 2012”[9].

In recent months little new information regarding the status of the contract or the city’s deliberations on the matter has been disclosed. Whatever investigations have been launched, whatever legal actions are being contemplated, whatever negotiations may be occurring between the city and Google to save the contract, it’s all happening behind closed doors. What is certain though is that, even in the absence of litigation or an embarrassing cancellation of the contract, the Los Angeles Gmail contract has turned into a money-losing disaster for Google. The city’s demand that Google pay the cost of keeping GroupWise in service beyond its expected retirement will likely cost the firm several million dollars, and will certainly wipe out any profit Google may have anticipated from the contract.

There is something about the LA Gmail project which is depressingly familiar to students of large IT projects, especially those involving huge enterprise software implementations such as ERP. It is a shockingly frequent occurrence for such projects to experience cost overruns, functionality shortfalls and even catastrophic “pull-the-plug-and-walk-away” failures. Something about the nature of these huge projects apparently creates skewed incentives for the teams that manage them, causing the leaders to systematically overpromise and underdeliver. The Los Angeles CTO in particular can be faulted for her failure to obtain buy-in from LAPD – the city agency with the largest number of users and the most sensitive security requirements – before steamrolling the project through the City Council.

Consumer vs. Enterprise

Nevertheless, the blame for the LA cloud email debacle must ultimately lie with Google, its product and its corporate culture. There is no doubt that Gmail is an excellent product. Since its launch in 2004, it has consistently been the most innovative and talked-about of the free consumer email services. I have personally used Gmail for the past five years, both in its native web interface and with an Outlook front-end. I can testify that on most days it is a joy to use. At the urging of my Microsoft friends, I’ve also dabbled with that vendor’s competing Office 365 email service, but I find that product much less polished in its current state than Gmail.

Despite Gmail’s well-deserved popularity, it has a fundamental flaw that is at the heart of the LA debacle. Gmail was conceived as a no-frills, low-cost, one-size-fits-all mass consumer product. It was designed to be extremely cheap to run on a huge scale, so that it could be given away free to millions of consumers, whose numerous eyeballs would drive a torrent of keyword-triggered advertising revenue into Google’s coffers. It was never intended as a product that could meet the endlessly expanding feature requirements and even more daunting security needs of large enterprises or government agencies[10].

Google’s developers lavish users with a continual stream of new bells and whistles that are folded into the service on an almost daily basis[11]. But the purchasers of the Google Apps for Business and Government editions are not consumers, and Google’s business model for these markets is not sustainable. The main and, in truth, the only real selling point of Google Apps for large customers is its unbeatably low price – just $50 per user per year (or even less with volume discounts). This is why cash-strapped government agencies have been the most eager to embrace these apps. But there is a reason why enterprise software is complex and expensive. Large users have very complex requirements that are constantly evolving, and the development effort to meet those requirements is both never-ending and very expensive. An enterprise software vendor that charges big fees for its software is obliged to listen when a major customer says that it needs a new feature. Yet when Los Angeles asked Google to make Google Docs compatible with legacy WordPerfect files, it was informed that this feature was “incompatible with the Google philosophy”[12]. This is not the response of a company that has a deep commitment to serving the enterprise market.

Enterprise email products like Microsoft Exchange and IBM Lotus Notes offer a vast and sometimes bewildering array of features and options, some delivered by the vendors themselves, but many provided by third party partners. For example, numerous products allow Exchange and Notes customers to exercise extremely fine-grained control over who can do what with the information contained in corporate email. Add-ons exist that prevent unauthorized users from forwarding, printing, transferring to a USB key or even opening sensitive email attachments without proper credentials. Similar products are beginning to appear for Gmail, but to date they are less numerous and more rudimentary. To be fair, the cloud versions of Exchange and Notes (Office 365 and LotusLive Notes respectively) suffer from many of the same functionality gaps as Google Apps. But these products benefit from enterprise software development cultures at Microsoft and IBM that are far more mature – and more richly funded – than Google’s. It is a safe bet that they will acquire the enterprise-grade features they need sooner than Google Apps.

Even more importantly, Exchange and Notes do not stand on their own within their parent organizations. Rather, they are part of vast, multi-billion dollar enterprise software businesses. It is no accident that the global leaders of enterprise software – not just Microsoft and IBM, but also Oracle, SAP, VMware, Computer Associates and Infor – all possess broad portfolios with dozens or even hundreds of products. Even the champion of software as a service (SaaS), Salesforce.com, is rapidly expanding beyond its initial base in salesforce automation. Enterprise software is a portfolio business because this is the only way to amortize the extremely high costs of developing and selling such complex products to the world’s very diverse population of large and mid-sized organizations. In enterprise software, one-trick ponies don’t stay that way for long. They either get big or get acquired. While Google has been assiduously building its portfolio of apps, a business unit that only generates a few hundred million dollars per year in revenue and probably no profit doesn’t have the scale to thrive alongside the giants. Cheap cloud email and some limited-function collaboration and productivity apps don’t have enough growth potential to make this a serious stand-alone business for Google. It is very telling that in the company’s last quarterly conference call, CEO Larry Page identified his priorities as search, mobile and social, but conspicuously failed to mention enterprise apps.

Finally, there is the issue of Google’s insistence that all cloud applications must be delivered solely from its own data centers. The economic logic behind the rise of cloud computing is compelling: it’s more efficient to concentrate expensive IT resources in a few big data centers managed by specialists and shared by many users, rather than distribute them over lots of redundant smaller facilities run by the users themselves. But not every case fits this centralized template. Some user needs are so specialized or so sensitive – for example, LAPD’s security requirements – that the added cost of a private cloud is justified. The leading enterprise cloud players – IBM, Microsoft, Oracle, VMware – all propose a mix of private and shared cloud solutions. All of them, that is, except Google, whose rejection of hybrid solutions appears motivated more by ideology than by a pragmatic desire to solve customer problems.

The bottom line

The reality is that Google Apps is little more than an afterthought in Google’s overall strategy. It’s probably never going to be big enough or important enough to merit its own line item in the company’s profit and loss statement. But it probably isn’t going to disappear anytime soon either. It’s an inexpensive way of putting pressure on two of competitor Microsoft’s most successful products, Exchange and Office. More important still, it is an expression of the Google founders’ strongly held conviction that end-user computing should be conducted in vast centralized data centers rather than on untethered local devices.

Despite the negatives, Google’s disruptive entry into the market has turned out to be a very good thing for enterprise email, because it has educated users about the benefits of cloud computing and is forcing the established leaders to hustle. It has prodded market leader Microsoft to respond with Office 365 and a multi-tenant cloud version of Exchange. It has even awakened slumbering giant IBM, which in recent years appeared to lose interest in an email market it once dominated, but has now launched a cloud email service of its own (LotusLive Notes).

In short, users of all sizes benefit by having Google hold the leaders’ feet to the fire. But enterprise customers with the most demanding needs might want to take a careful look at the Los Angeles case before they sign on for Google’s cloud. And investors should not expect this particular product to make any measurable contribution to Google’s bottom line.

---

[1] In an odd fit of ideological pique, Google recently decided to exclude all churches from its definition of “nonprofit”, so these institutions now have to pay rack rate just like any Fortune 500 corporation – an act of discrimination which earned Google a stinging denunciation from church-going novelist and Yale Law prof Stephen Carter.

[2] Gartner, August 2011, “Google Gmail Emerges as a Significant Threat to Microsoft in the Enterprise”.

[3] Office 365 was until recently known by the unlovely name of Business Productivity Online Suite or BPOS – it’s not hard to see why Redmond’s marketing department decided to kill this moniker.

[4] See www.computerworld.com/s/article/9126118/Au_contraire_Exchange_s_lead_over_Notes_actually_getting_bigger_and_bigger_says_Gartner.

[5] Gartner, August 2011, “The Cloud Email and Collaboration Services Market, 2011 Update”.

[6] Most official city documents cited in this article are available from the LA City Clerk’s web site: http://cityclerk.lacity.org/lacityclerkconnect/.

[7] See LAPD Commanding Officer, Information Technology Bureau, “Supplemental Report to the City Administrative Officer Second Status Report on the Implementation of the Google E-Mail and Collaboration System”, July 2010.

[8] Gartner, July 2011, “Case Study: City of Los Angeles Migrates to Google Gmail”.

[9] Randi Levin, CTO, City of Los Angeles, “Contract No. C-116359 Proposed Second Amendment”, August 17, 2011. In the May “Notice of Deficiencies” Levin had already demanded that Google pay all costs related to maintaining GroupWise beyond its originally planned retirement date.

[10] The appendix to the Los Angeles Google contract (available on the city’s web site) contains a revealing reminder of Gmail’s fundamentally consumer-oriented nature: although the contract specifies that ads in city emails are to be turned off “by default”, it nevertheless stipulates that “Customer may change this setting in the Admin Console, which constitutes Customer's authorization for Google to serve Ads”. One wonders why Los Angeles would want to allow its IT administrators even the option of turning on ad serving in city emails – in a product architected from the ground up for the enterprise, such a feature would not exist. For a fascinating account of the creation of Gmail, see Steven Levy’s In The Plex: How Google Thinks, Works, and Shapes Our Lives. Levy makes clear that the decision to scan the text of consumers’ email messages in order to place relevant ads nearby was the key breakthrough that made Gmail financially viable.

[11] See http://www.google.com/apps/intl/en/business/latest_info.html.

[12] See the Gartner case study.