May 18th, 2007

Gmail And The Challenge Of Web Services Scaling

by Scott Karp

  •  View Comments

Google has more engineers and infrastructure than God, but that hasn’t prevented me from being frozen out of my Gmail account for the last hour with this error message:

gmail-server-error.jpg

“don’t worry, your account data and messages are safe”

OK, well that makes me feel MUCH better. Now can I please get into my email!

Post to Twitter Tweet This Post  Post to Facebook Share on Facebook

  • Mike D: I would love to know how you came to the conclusion that gmail gets less traffic than msnbc. According to Alexa, gmail is the 3rd most visited website (although it appears it's stats are bundled with googles as a whole for some reason, but MSNBC is at 7,780. Google is just a *tad* bit more international than MSNBC.

    As far as everyone going down... I'd argue. If done properly, with enough fail over, there are ways of making sure not everyone is down. I would be curious if during the gmail outage that was experienced in this post, if someone else 100 miles away from you could have gotten into their account. Often times it's a particular data center that is having a problem, and because the failover for that location hasn't kicked it for your browser, your still seeing the error. It's the one downside to the DNS caching that is out there.... There is no way to instant fail over except to reclaim the IP address on another machine, which sometimes just isn't possible.

    I would think though, with over 100,000 computers power Google that they could get the failover to work 100% of the time... The only thing then that could take them out would be a very very nasty virus... But there are ways to prevent that as well. Not saying it's impossible, but you can get 'accidental' downtime to 0... Malicious downtime would be all that's left.

    What scares me about it is this: IF they can't keep the web portion running, how do I know they are keeping the smtp servers running? I understand the failover both from a DNS multiple MX record position, as well as the delays put into peoples mail servers to hold them until they deliver... but still... if googles smtp server failed... how would we ever know?
  • 1. The answer to that is "nobody". Everybody goes down from time to time. And I mean everybody. There is no such thing as a bulletproof website, given the fact that there are so many points along the chain where something can go wrong. MSNBC.com was down for about an hour yesterday as well and a *lot* more people use that site than Gmail.

    2. Nope, because I don't use it. :) Mail, to me, is something I want stored on my own server and not archived by a company who puts its own value on my information.

    3. Yep, that is true. Here's the thing: any online business has an uptime equation. It goes something like this. To stay up 80% of the time, it will cost X. To stay up 90% of the time, it will cost 2X. To stay up 99% of the time, it will cost 10X. To stay up 99.99% of the time, it will cost 500X. And to stay up 100% of the time just isn't possible. As you approach 100%, your returns (goodwill, mainly) diminish down to zero while your costs don't. I'd bet that if you asked most people at Google who actually know how these systems work if one or two periods of hour-long downtime a year was perfectly ok, they'd say yes. And if I was a customer, I'd also say yes.

    It's just the nature of the beast for sites to go down from time-to-time. If you told me this was the 3rd time this has happened to you lately, then ok, that sounds like a problem, but downtime is nothing like data-loss *at all*. There *is* an acceptable non-zero level.
  • Sam
    Scott-

    The data message is disturbing ("Don't worry"). It reminds you how much power they're acccumulating over everything we do online. Kind of scary.

    -Sam
  • Mike,

    1. It's Google -- if they can't keep their service up, who can? That's really my point.

    2. Ever see the ads in Gmail?

    3. Shit does happen, but to say an hour email outage is no big deal just demonstrates the low state of expectations and how much farther this all has to go.
  • Two things:

    1. Shit happens.

    2. Gmail is free.

    I know you're probably just venting, but surely a random one-hour outage in Gmail isn't actually enough to make you feel ill about the service.
blog comments powered by Disqus

Subscribe

Receive a free daily email newsletter with new Publishing 2.0 posts


Recent Posts

Clicky Web Analytics