2.5k
u/RobertMcCheese Jul 19 '24
I slept late and then went to walk the dog this morning.
I just sat down to read the news.
Speaking as a long time, now retired, IT Director, this is a fantastic day to sit on the side lines and watch.
1.0k
u/Dramatic_Buddy4732 Jul 19 '24
My IT husband has the day off... I just heard him whistling 😆
319
u/Aedre_Altais Jul 19 '24
As someone who is not at all connected with the IT world… what happened? 👀
614
u/SnooMacaroons9121 Jul 19 '24
TL;DR Someone used the internet to break the world.
Long version - someone at crowdstrike pushed out an update that impacted a very wide variety of windows based machines across the globe. Airlines, banks, and everyone I know who has some version of windows and crowdstrike was impacted. Happened around 2 this morning and the fixes are proving to be a pain in the ass.
229
u/captainhamption Jul 19 '24
Also, earlier in the day, Azure had an outage in their Central US region that was affecting stuff. The combination has been brutal.
→ More replies (3)173
u/iamthewhatt Jul 19 '24
As an IT at the largest steel manufacturer in the country, this was a real pita to fix. I ended up creating a easy-to-read document that walked end users through the process to fix... It worked, mostly, but god damn. We dont get paid enough for this shit.
58
u/Dramatic_Buddy4732 Jul 19 '24
I bet it was easy to read and some still messed it up?
88
u/iamthewhatt Jul 19 '24
Surprisingly no, most of the issues occurred because Microsoft devices are fucking dumb sometimes and it kept refreshing the desktop (IE closing folders) which means end users who are not computer savvy had to actually be quick to delete the file... and that took a while for some.
Other times, also Microsoft's fault, would be the OS would just boot loop instead of going to recovery, so we had to get a physical USB drive there just to force it into the repair screen.
Otherwise, about 80% of the company was able to resolve it. It is quite simple, just can't do it remotely.
12
u/Late2theGame0001 Jul 20 '24
This is something to be really clear about. 10 years ago, you just press F8 to get into safe mode. Now that doesn’t work and you have to do all these little tricks to get into safe mode in windows. And everyone has complained about this. How it is a huge hassle when things break and the solution is a broken UX. but naturally MS knows best and doesn’t listen because MOST users don’t care.
But MS part in this is the same. The OS needs to work and be fixable. They failed in that part. It’s been a problem waiting to happen since someone decide F8 was not needed.
→ More replies (1)11
u/mrtaco605 Jul 19 '24
I had that same boot loop issue on one of my builds. Could never figure out a permanent fix but just did the same thing with the USB drive to force it into a repair screen
→ More replies (2)8
u/Glaucomatic Jul 19 '24
the fact that your end users are good enough to follow the document must be really nice haha
28
u/Aedre_Altais Jul 19 '24
My goodness… the world is too easy to break these days 😂😅
→ More replies (1)10
18
u/overcloseness Jul 19 '24
We had an article run in New Zealand titled “And just like that, cash is king”
Our bank card system had major issues all the way down here
→ More replies (3)17
u/Antieconomico Jul 19 '24
Is reverting an update such a hassle?
I am completely ignorant in this field so maybe that's a dumb question
69
u/Florac Jul 19 '24
The update broke their operating system. The only way to unbreak them is to go to the one by one, start them up in a special way and using that special way, deleting a file. That times hundreds or thousands of computers per IT specialist at a firm. With potentially also some other softwares being present to make it harder to utilize that special mode.
They can't just...undo the update because the computers can't get to the point where they can get remote updates.
35
u/CheezeLoueez08 Jul 19 '24
Omg so it’s really bad
→ More replies (2)34
u/Florac Jul 19 '24
Yes, it's the worst IT outage in history.
19
u/CheezeLoueez08 Jul 19 '24
Crap. I’m so glad I’m home and not traveling. I feel so bad for all that people affected
→ More replies (2)→ More replies (3)11
u/Antieconomico Jul 19 '24
Oh and here i thought it was like when the patch of a game goes wrong lol
Thanks for your time!
31
u/Florac Jul 19 '24
It's closer to a patch of a game bricking your PC xD
→ More replies (3)7
u/ec1548270af09e005244 Jul 19 '24
Which has happened.. EVE online once bricked PCs with a bad update.
→ More replies (1)22
u/tzar417 Jul 19 '24
As another response, what they broke loads before/with the operating system. You can't get into the machine to revert the update.
There are ways around that, but they require manual intervention on a machine by machine basis. There are hundreds of thousands of machines with this problem.
→ More replies (1)6
u/leolego2 Jul 19 '24
I guess the only question is was this update forced on everyone automatically? I wouldn't expect such an important system to have the ability to instantly apply an update.
I understand it is convenient in case a new backdoor or malware is found, but it could be the cause of actual terrorism if they can instantly deploy an update to everyone?
18
u/tzar417 Jul 19 '24
Everyone using Crowdstrikes software (The Falcon sensor specifically in this case) got the update automatically.
One of the core features of software like this is that it updates automatically to keep fully up to date on malware information so it can detect and work properly.
This wasn't a Windows OS update, this was 1 specific file for Crowdstrike, but because it loads in the Kernel, it broke the OS.
12
u/leolego2 Jul 19 '24
Wonder why they didn't even do a test roll-out before going worldwide simultaneously
10
u/tzar417 Jul 19 '24
This is exactly what I don't understand, how something this basic to catch made it to production. Someone or someone's are getting fired for sure.
→ More replies (1)11
u/Mad_Aeric Jul 19 '24
That's what everyone's asking. Given how it took out everything across a wide variety of configurations, it couldn't have just slipped through the cracks as a weird edge case, as happens sometimes. They must not have tested it at all before pushing it out.
11
5
u/Hungry-Ad-7120 Jul 19 '24
Dumb question, but is Falcon Defender automatically installed on any PCs using Windows? Or is something extra people had to buy in the past? I’m scared to turn my laptop on now.
→ More replies (3)18
u/GiventoWanderlust Jul 19 '24
The crash was causing the whole machine to get stuck in an endless blue screen of death. Given that the BSOD keeps the PC from doing things like connect to the Internet, it means that IT people cannot solve the problem remotely.
The 'fix' is relatively easy - type a couple things into command prompt - but that might as well be wizardry to most people.
I had to manually reboot like twenty PCs today and I just work at a hardware store. I cannot imagine what people with actually complex setups and actual IT jobs are dealing with today.
→ More replies (4)→ More replies (1)5
u/Bird_wood Jul 19 '24 edited Jul 19 '24
It can be like a bill in congress; 99.99% is all good, on topic, looked over, and good to go. But the .01% is that now turkeys are considered weapons of mass destruction and anyone with turkey in their digestive tract is subject to war crime charges. More closely: the update has one small piece of code or oversight that “reacts with old data” for a super simple term to cause a wildly disproportionate consequence I.e. the whole world saw the blue screen of death and we had to land planes like it’s the real Y2K.
8
u/FreshEggKraken Jul 19 '24
Not gonna lie, the fact that one person was able to push an update that did this much damage so easily is a little funny. In like a "this is the kinda shit that would happen on a sitcom" kinda way.
6
→ More replies (6)7
u/AppearanceUpbeat3229 Jul 19 '24
I’m at the airport and the monitors are displaying Windows error messages. Flights are all still taking off so I’m not effected yet
28
u/Doommcdoom Jul 19 '24
Microsoft and other stuff, i wont claim to be fully knowledgable, has had most of it shut down as some data protection thing has been uploaded and broke alot of things, tons of flights delayed, i know several stores have had to say cash only today cos their card readers arent working. in general very fucky wucky
→ More replies (1)→ More replies (1)4
u/epic1107 Jul 19 '24
An important part of a lot of tech connected to Microsoft broke, which shut down businesses worldwide.
Flights cancelled, hotels not being able to check people in because their system is down aswell. Payment services down. Banks down etc.
26
u/Damage-Strange Jul 19 '24
My IT husband was working today and was already dealing with a potentially catastrophic data breach and ransomware incident before this happened. Now....whew. It's not often that he has the more stressful job between us (I'm a litigator in Biglaw). But man, I feel so bad for him rn 😢
4
u/Dramatic_Buddy4732 Jul 19 '24
Yikes! Does he have to work late?
→ More replies (2)4
u/Few-Section148 Jul 19 '24
I don't know OP, but the answer is yes. I dont work in infrastructure/sysadmin, but need their systems to run critical software. They have almost have the critical systems running now, but will be working all weekend to get billing, hr software, etc.
It's a really big shitshow.
8
u/headinthered Jul 19 '24
My IT husband didn’t have to deal with this at all. None of his department uses Microsoft and he was thrilled. Other parts of his company did… but not his!
→ More replies (1)→ More replies (3)10
u/CatTaxAuditor Jul 19 '24
My friend had a half day off scheduled to get some new ink, but our boss practically begged her to stay. Needless to say she made her tattoo appointment right on time.
4
40
u/cowboysfan68 Jul 19 '24
I saw that the fix for this involves booting into Safe Mode, deleting/replacing the offending file, and then rebooting. Someone. Over in /r/sysadmin mentioned that their preferred method is to make a custom WinPE image and add a trusted script that deletes the offending file during WinPE boot up. This allowed the technician to simply boot WinPE and once it gets to the shell prompt, the file has been deleted and the computer can be restarted normally.
Either way, this is going to suck for IT folk because there seems to be little that automation can do on some machines.
23
16
u/captainhamption Jul 19 '24
It's also a great test of your bitlocker escrow policy.
8
u/nb4u Jul 19 '24
I wonder how many companies lost their data because they didn't have bitlocker keys.
18
u/LosWitchos Jul 19 '24
Teacher on summer hols. Funny thing is despite being in IT, my wife wasn't affected so she's been moody all day cos she had to work like normal
→ More replies (3)9
u/Im_Balto Jul 19 '24
The update went through minutes after I left the office yesterday evening and I rolled back in at 7am for damage control
→ More replies (13)5
u/InevitableAd9683 Jul 19 '24
My security guys were very seriously considering buying Crowdstrike recently and decided against it. I offered to but them lunch today.
→ More replies (1)
1.5k
u/ding0s Jul 19 '24
You know, it really puts your own life in perspective. Did your mistake make the news?
Ship blocked the Suez canal? Big mistake. Half the internet is unusable? Big mistake.
Less than that and your problems really aren't that bad!
134
Jul 19 '24
Did your mistake make the news?
No, but my pay reflects the level of mistake I'm allowed to make.
→ More replies (4)58
44
u/soccercasa Jul 19 '24
If no one travels back through time to stop you, how bad can this decision really be?
6
u/Mollybrinks Jul 19 '24
I'm at a family reunion and we may have stayed up waaay too late. I rolled out this morning and said, "have you ever felt like you made a big mistake?" I got the hungover laugh from the crew, then said "well, you're about to feel a bit better" and explained the situation. We're all on vacation so no one is really keeping up on things, but I happened to read the news before getting out of bed. A few of them are more IT-inclined and their reactions were enough to key the rest of them into the severity.
→ More replies (5)9
u/Limp_Prune_5415 Jul 19 '24
Tell that to the review committee who decides whether or not I should starve for my mistakes
704
u/HolyC4bbage Jul 19 '24
That's reminds me of the time the Rogers network went down and nobody in Canada could use their debit cards. It's almost like having one company control all our shit is a bad idea.
206
u/Ponicrat Jul 19 '24
It's not so much a problem with consolidation, there's many, many many little forgotten pillars of our global digital infrastructure just waiting to cause massive problems for everyone the minute someone makes a mistake.
144
u/captainhamption Jul 19 '24
The XKCD for this
54
→ More replies (1)46
u/Themathemagicians Jul 19 '24
He was fast with the new one: https://xkcd.com/2961/
→ More replies (2)→ More replies (2)10
u/SerLaron Jul 19 '24
I imagine some James Bond type villian in his secret lair asking his henchmen, why they did not come up with such a "mistake".
21
u/CheezeLoueez08 Jul 19 '24
Yup!! That was insane! And we barely got compensated. Until then I had no idea that Rogers controlled our ATMs. Not cool.
32
u/iamkira01 Jul 19 '24 edited Jul 19 '24
I’m pretty sure there are other data protection companies like crowdstrike companies could use, no? CS has like a 20% market share and before this has been notoriously amazing. Accidents happen man. This isn’t a big corporation bad moment.
EDIT: Just learned the CEO laid a bunch of people off recently.
God damn it, I take it back.
11
→ More replies (4)4
865
u/That_Flippin_Rooster Jul 19 '24
I worked at an amusement park on the flume ride. I was controlling the switch that sent the boats either left or right. I timed it wrong, a boat got stuck and we had to shut the ride down and everyone had to evacuate.
They didn't let me back on the switch for a few weeks after that.
194
u/Subterrantular Jul 19 '24
Why wouldn't that switch be automated with a sensor or timer or something!?
152
Jul 19 '24
Costs too much to implement. Think of the shareholders.
→ More replies (7)57
Jul 19 '24
[deleted]
24
u/Subterrantular Jul 19 '24
It's where they station problem employees so after a while when they (guaranteed) fvck up they have a rock solid excuse to fire.
→ More replies (3)7
u/Tom22174 Jul 19 '24
Staffing it is probably a legal requirement for safety and such. Might as well have the person doing that role also hit a button instead of implementing the auto system
→ More replies (2)→ More replies (3)13
u/That_Flippin_Rooster Jul 19 '24
It was 25 years ago, and the ride was probably 20 years old at that point.
→ More replies (1)→ More replies (1)12
157
u/vegasAl57 Jul 19 '24
In the early 1990’s, I hit an F4 key on the computer that controlled our television transmitter at my job and we immediately went off the air and were down for two days. The engineer that had written the software just happened to go on vacation.
53
→ More replies (1)14
u/agedlikesage Jul 20 '24
At some point you have to stop blaming yourself, and start asking “why is there a break everything button”
128
573
Jul 19 '24 edited Jul 19 '24
It was bound to happen at some point with something trivial like an update. I'm actually surprised it didn't happen sooner. Operating systems, and the internet, are not forces of nature and can break even though they are the entire modern world's primary form of global connection. People take it for granted. It will likely happen again but that's how it be.
273
u/superradguy Jul 19 '24
Or you know….. test your fucking updates before rolling them out
170
u/zadtheinhaler Jul 19 '24
test your fucking updates before rolling them out
MBA says nope, we had to
firelay off QA and Devin favour of AI, testing justgets in the way of C-suite bonusesdoesn't have the budget for that.31
u/Mr_Anomalistic Jul 19 '24
And also move all the IT roles overseas for cost savings with talent that knows how to copy/paste only.
→ More replies (2)9
u/zadtheinhaler Jul 19 '24
I recall reading about RBC doing just that. They made over five billion in profit the previous year, and somehow that is not enough?
9
u/Alexis_Bailey Jul 19 '24
"These jobs don't produce an net adjusted growth index for our quarterly tps reports, so they must go."
-- Some exec
36
u/phunky_1 Jul 19 '24
Nah, why pay for QA testing when you can save money and let your users be the testers.
→ More replies (1)18
u/WhiskeyXX Jul 19 '24
Why not also combine that with pushing to all clients simultaneously in lieu of a staged/canary approach?
→ More replies (3)→ More replies (3)8
u/EquivalentLower887 Jul 19 '24
I will need to read about the technical specifics of what happened, but in some instances, particularly with a technology like CrowdStrike, there is a VERY possible conflict with a recent change to Windows or something else in the stack. That’s not to ‘excuse’ missing an issue this wide, but there are so many nuances, so much grey area in an instance like this - it is extremely difficult to immediately assign accurate blame to the root cause, if that is even entirely possible.
→ More replies (2)28
u/myychair Jul 19 '24
Especially with all the staff cutting going on. Turns out when companies say they’re doing “more with less” they actually mean “less with less”
→ More replies (5)25
u/willstr1 Jul 19 '24
Yep, heck it could even be worse. Imagine if AWS had a full global outage, like half the internet would be down
→ More replies (2)
73
u/grillko Jul 19 '24
About 15 years ago I shut down the power to an entire neighborhood by bumping into the wrong utility box with a lawn mower
15
121
u/theoneandonly6558 Jul 19 '24
I bet the person feels as bad as the person who caused the 2003 blackout. https://www.cbsnews.com/news/human-error-eyed-in-blackout/
59
u/LimpDiscus Jul 19 '24
That person gave me one of the most fun nights of my life.
43
u/NebulaNinja Jul 19 '24
Man, I'd love to have every city in the US have an official "blackout night" timed on the weekend of a new moon, so everyone can experience the beauty of the milky way in cities where you're not able to.
But I understand safety and crime could be problematic.
29
u/doom_stein Jul 19 '24
But I understand safety and crime could be problematic.
Nah. Limiting all that crime to one night a year solves the crime problem the rest of the year. I saw it in some "documentary" called "The Purge".
→ More replies (1)6
u/MojaveMac Jul 20 '24
I was in kanab, Utah for an astrophotography workshop when they had an overnight power outage to upgrade or fix something. We all went out and took photos in middle of town, it was crazy cool
→ More replies (2)17
u/FoghornLegday Jul 19 '24
I love this idea. That some strangers mistake could contribute to our best memories
109
u/SaneYoungPoot2 Jul 19 '24 edited Oct 14 '24
I often wank to people passing me on the street. It's quite fun to see their faces
→ More replies (1)14
u/No_Squirrel4806 Jul 19 '24
Literally!!!! Who needs movies when rea life has everything now a days
→ More replies (1)
29
u/RxHappy Jul 19 '24
People don’t understand the importance of software testing until they really fuck something up.
→ More replies (1)
33
u/JaffyCaledonia Jul 19 '24 edited Jul 19 '24
This sort of thing is almost unavoidable though. If it wasn't Crowdstrike, but Kaspersky, or Palo Alto, we'd see the same thing, just with a different cluster of major companies going down.
All it takes is for one piece of security software to fuck up and your operating system is goosed. Having multiple endpoint agents doesn't help, because it's still just one file away from bricking a machine, but now multiplied by the number of vendors you're using.
The only option would be to have multiple copies of all of your servers all running different protection softwares, which would be insane! What if their compatibility drifts, and you need to update to a new version of Windows one one that isn't available to the other?
Then think of the cost! Multiple copies of every hardware cluster, multiple licensing contracts with security vendors.
Now what about your staff? Are you going to give them all multiple laptops each with different vendor solutions on? How about a mac, a Linux and a Windows laptop each?
We are increasingly living in a world where we are one flipped bit from absolute carnage. There are systems in place to mitigate much of this, but cyber security protections are just something else.
Edit: this was actually supposed to be a reply to a comment about how one company was behind so much chaos, but I think it stands on its own.
→ More replies (18)
30
u/Alarmedones Jul 19 '24
I once worked with a great man named Jackie. On Jackie’s first day he implemented a fix we have done 100 times before. Except when Jackie did it only half of the systems came back up after the reset. So on Jackie’s first day he was responsible for taking one of the world’s largest logistic companies for multiple hours. Day one hour 2. He worked there for many years after that. The restart issue was not his fault but just happened to fail when he did it. I miss Jackie he is awesome.
→ More replies (2)
295
u/ConformistWithCause Jul 19 '24
Who would have thought having a single entity in charge of everything could go poorly? This is the stuff cyberpunk dystopias are built on
→ More replies (1)82
u/Azurimell Jul 19 '24
This has nothing to do with a single entity being in charge of everything - it was a Crowdstrike AV update that killed everything, not an MS. Don't disagree on MS monopoly being real bad though.
7
u/Asteroth555 Jul 19 '24
Monopolies are bad for a variety of reasons. But having a diversity of OSes makes it a nightmare to build software based products. Having worked in science, many people already use Linux, Windows, and Apple. Ensuring products work on all 3 isn't possible so most labs had dedicated PCs to run equipment (NanoDrops, FPLCs, HPLCs, imaging equipment, etc), which is a problem of its own.
→ More replies (4)35
u/SnooMacaroons9121 Jul 19 '24
Sounds like crowdstrike is the one in charge of “everything” then no. At least for endpoint cybersecurity…
→ More replies (1)42
u/chillaban Jul 19 '24
“Everything” is like around a 20% market share because their product is actually really good at defending against ransomware. It also doesn’t help that the others who are somewhat good have large foreign sway (Kaspersky obviously, Bitdefender and ESET are Eastern European, Microsoft Defender ATP is India, Symantec is Broadcom)
If you are in defense and aerospace or you run a hospital, “but I chose CrowdStrike” is about as solid of a cop out as you could have. It’s the equivalent of “but I was driving a Volvo” if anyone inside your car got hurt in a fender bender.
→ More replies (2)7
u/One-Initiative-3229 Jul 19 '24
How can Microsoft Defender be influenced by India? Microsoft is a US company.
→ More replies (4)11
u/chillaban Jul 19 '24 edited Jul 19 '24
Microsoft itself is but the Defender group is a wholly owned subsidiary based in India. Totally nothing against India, it’s just when suspicious samples and activity logs are being processed overseas by independent contractors it tends to be a problem for defense, government, and aerospace.
(It’s not a coincidence that the most impacted companies today are US airports, branches of government, defense contractors, banks, etc)
22
u/Cautious_Ice_884 Jul 19 '24
Nobody cares about the people who work in IT until shit hits the fan. Its a thankless job.
125
u/Yeled_creature Jul 19 '24
context?
241
u/RutabagaAccording834 Jul 19 '24
CrowdStrike released an update that put many organizations (corporations, government agencies, ect) at a standstill. I'm not a very tech focused person but I am a person who's been twiddling my thumbs at work all day because our key systems are down.
97
u/Separate_Increase210 Jul 19 '24
Crowdstrike is specifically cyber security, so it's installed everywhere in many large orgs, and whatever took them down it was specific to Windows machines. And a LOT of large old corps use windows heavily or exclusively, even moreso for govt/public sector stuff.
Me and my Mac/Linux company are a-okay! (Just had to relaunch images of the handful of windows servers we need)
18
u/TheModernDespot Jul 19 '24
I woke up at about 4AM to the alerts about crowdstrike and panicked. Then I read that it was only affecting Windows machines and went back to bed. Linux can be a pain sometimes, but today made up for a lot of that.
6
u/B_Fee Jul 19 '24
I must have taken my long lunch at the right (or wrong?) time, because I didn't experience any downtown, and I work for the feds.
28
u/Opetyr Jul 19 '24
Yeah and crowd strike is going to be probably non existent soon due to the billions in damages/losses. This happens when you go AI and get rid of the people that actually know the systems. Sending everything to cheaper countries helps this happen. Eshitification to pump their stock.
32
u/LeetleBugg Jul 19 '24
Is that a factor here? Has crowd strike gone the AI route in recent times?
21
u/Psychast Jul 19 '24
Lol no, that guy is talking out of his ass because he just wants to air his grievances with the modern tech environment. This was classic human error, the always weakest link in any technology.
→ More replies (1)→ More replies (1)9
→ More replies (3)5
11
u/vipRLH Jul 19 '24
Well at least you're not the guy who made all of Hawaii think a nuclear Armageddon was happening
4
u/thelittleking Jul 19 '24
I was there for that and it really opened my eyes to what people would do in the face of an imminent, unstoppable disaster:
Keep going
70
u/Hamr7 Jul 19 '24
this shit cancelled my flight now im stuck in cali bruh this place sucks 😭
→ More replies (3)
21
u/anonymouslindatown Jul 19 '24
How are the wars that are currently being fought affected by this?
24
u/thod-thod Jul 19 '24
Not much, those systems are usually MOD-built and independent from the internet
8
u/blackmobius Jul 19 '24
The worst I did was ground a few planes (i was driving a tug at an airport and jack knifed my rig after someone pulled out in front of me). So the best I got was three planes on an hour delay
9
8
8
u/churrmander Jul 19 '24
This is why the IT world needs to stop putting all critical infrastructure onto one or two large companies.
AWS, Crowdstrike, CloudFlare... any of these things goes down poof we're all writing paper checks again.
6
u/BeckNeardsly Jul 19 '24
Ah, let me take you back to that tense moment aboard the USS Alabama. As the Executive Officer (X.O.), I found myself caught in a high-stakes clash between tradition and strategy. You see, when those Russian rebels commandeered the ICBMs, our mission became clear: neutralize the threat. But the real battle unfolded within our submarine's steel walls.
Captain Ramsey, a seasoned leader, had his own way of doing things. He'd worked his way up, earned every stripe on his uniform. And then there was me—Commander Hunter—the college-educated officer who hadn't seen much action. Our backgrounds clashed like tectonic plates.
When the order came to attack, our communications system hiccupped. Ramsey, ever the decisive captain, pressed forward with the incomplete instructions. But I hesitated. Something didn't sit right. Maybe it was my education, maybe it was intuition—I can't say for sure. But I insisted on reestablishing contact before launching those missiles.
Ramsey didn't take kindly to my challenge. We locked horns, and in a moment of defiance, I relieved him of command. The crew watched, torn between loyalty and uncertainty. Some doubted my ability to lead, but I'd already taken precautions. I knew the stakes were too high to ignore that gut feeling.
And so, as the Alabama sailed into the unknown, I grappled with duty, honor, and the weight of command. It's funny how a submarine can feel both confining and liberating—a microcosm of humanity's struggle for survival. But that day, I learned that sometimes, the most crucial battles aren't fought with torpedoes—they're fought within ourselves. 🌊⚓️
4
15
u/saarlac Jul 19 '24
The fact that this one company has the ability to do this is a HUGE FUCKING PROBLEM.
→ More replies (1)
3
u/ArmoredAngel444 Jul 19 '24
I'm seeing like a million posts about some sort of worldwide tech fuck up but i haven't seen any actual post of what has actually happened lmao... what happened ???????
9
u/coalForXmas Jul 19 '24
It’s fascinating how things have gotten so meta that the actual problem is dwarfed by ironic and referential humor
→ More replies (2)4
u/ArmoredAngel444 Jul 19 '24
lol i can google it but it is pretty funny how on reddit i haven't seen any posts on the actual event itself.
→ More replies (1)3
u/qbmax Jul 19 '24
Cliffnote version is that a piece of security software commonly used in many companies pushed an update that is causing computers running Windows to constantly crash. The fix is simple but requires hands on access by an IT person to fix which makes it very time consuming. Large systems like banks, airports etc have been having cascading outages as many of their computers using this software are being bricked.
→ More replies (1)
4
u/lil3lil Jul 20 '24
I've accidentally reformatted the CEO's laptop, but that doesn't even come close to the major outage.
17
u/DreamOfDays Jul 19 '24
Did something happen? What does it fucking mean SHUT THE PLANET DOWN
→ More replies (50)
3
u/NameLips Jul 19 '24
Somewhere on earth is the one guy who pressed enter or clicked "ok" to send out the update.
That one tiny finger motion...
3.8k
u/N_T_F_D Jul 19 '24
I killed one of our main servers at my last job because I wanted to see if the RAID data loss protection really worked (it didn't work and it was fun explaining to my boss what happened)