A malfunction that shut down all of Toyota Motor's assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.
I haven't read the article because documentation is overhead but I'm guessing the real reason is because the guy who kept saying they needed to add more storage was repeatedly told to calm down and stop overreacting.
Sysadmin pro tip: Keep a 1-10GB file of random data named DELETEME on your data drives. Then if this happens you can get some quick breathing room to fix things.
This happens. Recently we had a problem in production where our database grew by a factor of 10 in just a few minutes due to a replication glitch. Of course it took down the whole application as we ran out of space.
Some things just happen and all head room and monitoring cannot save you if things go seriously wrong. You cannot prepare for everything in life and IT I guess. It is part of the job.
I blame lean philosophy. Keeping spare parts and redundancy is expensive so definitely don't do it...which is just rolling the dice until it comes up snake eyes and your plant shuts down.
It's the "save 5% yearly and stop trying to avoid a daily 5% chance of disaster"
There's some irony to every tech company modeling their pipeline off Toyota's Kanban system...
Only for Toyota to completely fuck up their tech by running out of disk space for their system to exist on. Looks like someone should have put "Buy more hard drives" to the board.
TOKYO, Sept 6 (Reuters) - A malfunction that shut down all of Toyota Motor's (7203.T) assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.
The system halt followed an error due to insufficient disk space on some of the servers and was not caused by a cyberattack, the world's largest automaker by sales said in a statement on Wednesday.
"The system was restored after the data was transferred to a server with a larger capacity," Toyota said.
The issue occurred following regular maintenance work on the servers, the company said, adding that it would review its maintenance procedures.
Two people with knowledge of the matter had told Reuters the malfunction occurred during an update of the automaker's parts ordering system.
Toyota restarted operations at its assembly plants in its home market on Wednesday last week, a day after the malfunction occurred.
The original article contains 159 words, the summary contains 159 words. Saved 0%. I'm a bot and I'm open source!
And that's why I have a weekly cronjob in my server to call BleachBit, remove cycled logs, and compress the images on my storage. Having to make do with a rather limited VPS for years taught me to be resourceful with what I had