Who among us has deleted a production database?

I’d hope it’s very few of you that have done this in your career. I’m sure a few of you have deleted (or truncated or updated all rows for) a table in production. I’ve done that a few times, but fortunately, I’ve been able to recover the data quickly. I had this happen in SQL 6.5 and was grateful I could start a single-table restore before my phone rang.

Here’s another question: which of you has had a storage admin delete or remove some remote storage and cause you database problems? Has anyone had that happen in their environment? I haven’t had this in production, but I have had this happen to test systems, and I was very irate with the storage people when it did. After that, I’m sure they were very cautious about changing any configuration for database servers. I’m also sure that also contributed to my struggles in getting more space promptly as well, so I’m not sure I came out ahead in that situation.

A hospital system had issues after this happened to them: engineers deleted critical storage that connected to a database system. Fortunately, no services have stopped, and no patients are missing services, at least as far as we know. Everything has to be moving slower, and that might mean that staff is spending more time on “downtime procedures”, i.e. paper, than focusing on patients. Knowing a few medical professionals, this means they’re more stressed and working harder to be sure patients aren’t affected. Sucks for them, and I doubt the hospital compensates them for an engineer’s mistake.

I haven’t heard about this happening in a long time, and I’m surprised by that. Almost all storage these days for server systems is remote, especially in the cloud, and it would be easy to click the wrong button or select the wrong disk when re-configuring a system and remove critical storage. Maybe we’ve gotten better at popping up warnings that slow people down and prevent mistakes.

Or maybe we don’t delete disks and only add them to database systems

In any case, I hope they can recover things quickly and easily. If you’ve seen this, let me know. If you haven’t, here’s a reminder that this could happen. You should be sure your backups are running AND you can perform a test restore.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.