Obviously, we would have preferred not to have been down at all, but at least this article helps explain what happened. Sorry if you were hit by a “Connectivity Required” message last night. That message in our app either means “You don’t have any internet connectivity right now” or “we can’t talk to our service from your phone.”
We actually did a bunch of work in Wordament to fail gracefully in a number of error conditions—but we never expected the case that AppEngine would be 100% down. We’ve only ever seen one or two components go down at a time. Last night we couldn’t even get our “hello” message back from the server—which was why we couldn’t do anything accept tell you that “Connectivity was required.”
For all of you that hit this, we are sorry—it hit all of us—and it sucked for all of us. As of now, our service appears back up and running smoothly. The issue was not our newest update or your phone—it was Google. We are reconsidering our options for cloud redundancy now.