Google Cloud explains how it accidentally deleted a customer account
Google Cloud explains how it accidentally deleted a customer account
UniSuper's 647,000 users faced two weeks of downtime because of a Google Cloud bug.
Earlier this month, Google Cloud experienced one of its biggest blunders ever when UniSuper, a $135 billion Australian pension fund, had its Google Cloud account wiped out due to some kind of mistake on Google's end. At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members. There were joint statements from the Google Cloud CEO and UniSuper CEO on the matter, a lot of apologies, and presumably a lot of worried customers who wondered if their retirement fund had disappeared.
UniSuper's mistake was relying too much on Google Cloud even for critical backups.
I think about how many times things like this happen to individuals that don’t have the weight to warrant an article where their entire collection of online data just gets wiped out and Google responds with: ¯\_(ツ)_/¯
Don’t trust your data to the cloud, folks.
My thought exactly. I read the headline and thought that this probably happens to people fairly often, I can't believe Google admitted to this... Oh wait there it is - that's why. I have an old Google apps for your domain account. They switched it to an "apps for work account" then they switched it to "apps for business". Then switched it to "G suite". They then switched it to "Google workspace". Each time they made changes in the infrastructure and that account is all whacked up. There's lots of things including Google made things that just don't work with it and features are late to arrive or never do when standard Google accounts are changed. I'm amazed they haven't just accidentally deleted all my stuff in one of these change overs.
I also have one of these old Google apps accounts. I wish they would accidentally delete it so I could get the last couple stragglers off and take my domain elsewhere.
During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
Your data is safe in the cloud with multiple redundant backups, unless your account is marked as delinquent which will be deleted immediately and irrevocably.
That's not what I got out of that at all. It looks more like an errant setting made the account expire automatically after some amount of time, triggering the wipe.
Errant settings that marked the account as delinquent/unpaid at the end of the month, triggering immediate and irrecoverable account deletion. Basically, the scariest part of the google cloud is if they think you can't pay anymore, even if it's a mistake, your account will be wiped along with the backups. They did say they'll have more safeguard after this, but finger crossed.
What I got out of that is that gcp is not a stable product and should be avoided.
Don't trust the Cloud, people.
Abe Simpson was right all along.
This is the best summary I could come up with:
At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members.
Two weeks later, Google Cloud's internal review of the problem is finished, and the company has a blog post up detailing what happened.
During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank.
This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period.
During its many downtime updates, UniSuper indicated it did not have access to Google Cloud backups and had to dig into a third-party (presumably less up-to-date) store to get back up and running.
These backups have minimized data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration."
The original article contains 651 words, the summary contains 170 words. Saved 74%. I'm a bot and I'm open source!