As we all know, Delta faced an absolutely massive meltdown in the past couple of weeks, whereby nearly 7,000 flights were canceled, and even more flights were significantly delayed. While the issue arose due to IT outages with Microsoft and CrowdStrike, ultimately the scale of the meltdown came down to Delta’s crew scheduling software being unable to cope, and breaking down.
Along those lines, there’s an interesting update when it comes to action that Delta is taking to minimize its losses from this event…
In this post:
Delta seeking damages from CrowdStrike & Microsoft
CNBC reports that Delta has hired attorney David Boies, Chairman of Boies Schiller Flexner, to seek damages from both CrowdStrike and Microsoft. The outages reportedly cost Delta somewhere around $350 to $500 million, and the airline clearly wants to recoup as much of that as possible.
Delta isn’t just hiring any attorney, though. Boises is a prominent attorney who represented the US government in its antitrust case against Microsoft, and who helped with a decision that overturned California’s ban on gay marriage. He has also represented some questionable individuals, like Harvey Weinstein and Elizabeth Holmes.
As you’d expect, CrowdStrike’s stock has plummeted in the past couple of weeks, and in particular this week, when it was announced that Delta plans to pursue compensation from the company.
You can’t blame Delta for trying, but…
Ultimately Delta’s operation was sent into a spiral for something that was initially outside of its control. However, the extent of the meltdown came down to Delta’s lack of investment in its crew scheduling software, since other airlines weren’t impacted to the same extent.
This case should set an interesting precedent, and I’m curious to see how it plays out (to the extent that terms are disclosed). To what degree should a company like CrowdStrike be on the hook for the domino effect caused by a bad software update? Should the company owe compensation for the initial meltdown, or should it also be responsible for everything else that melted down because of the initial issue?
For that matter, so many companies had issues as a result of this, so to what extent can a cybersecurity company that’s so widely used be expected to make companies whole? It sort of upends the business model, when you consider the risk…
While I don’t in any way blame Delta for trying to pursue compensation (and even think that’s smart and fair), what I find frustrating is the extent to which the airline has tried to publicly shift blame, even for issues resulting from the company’s crew scheduling software.
Having a meltdown is one thing, but the lack of humility and accountability is what many Delta loyalists have found to be most frustrating. For example, the airline didn’t even want to provide refunds of tickets on other airlines until it was strong-armed into it, despite constantly claiming it can command a premium thanks to its operational reliability.
Bottom line
Delta is pursuing compensation from both CrowdStrike and Microsoft, following the recent meltdown that caused roughly 7,000 flight cancelations, and is expected to cost the airline up to $500 million. Delta didn’t just hire any lawyer, though, and I’d say that sends a pretty clear message as to the extent to which the airline is hoping to shift blame (and losses).
How do you see Delta seeking compensation from CrowdStrike and Microsoft playing out?
Good luck to Delta. I have a quicker and just as beneficial way to spend the millions they are going to pay David Boies. Withdraw them in cash, pile them up in front of the Delta HQ, pour lighter fluid on and throw a match.
What Delta should be asking themselves is why their IT department did not test the update before releasing it to the network, admittedly many companies failed on this one, but...
Good luck to Delta. I have a quicker and just as beneficial way to spend the millions they are going to pay David Boies. Withdraw them in cash, pile them up in front of the Delta HQ, pour lighter fluid on and throw a match.
What Delta should be asking themselves is why their IT department did not test the update before releasing it to the network, admittedly many companies failed on this one, but the update is not automatic, someone needs to accept to release the hounds of hell on the network.
Ridiculous comment. Delta's IT department isn't responsible for "testing" an update that is rolled out to them from the IT vendor Crowdstrike. It's Crowdstrike's responsibility to make sure their software works before putting it live. They aren't going to give the software update to Delta first and say, "Here's an update, but we aren't sure if it's going to work with Windoze so would you mind testing this for us on a few of your computers!"
Tell me you don't know anything about IT without telling me. This update was rolled out by Crowdstrike themselves, IT departments had no involvement with the issue it caused because this update was rolled out on Crowdstrike's backend. IT departments had no clue what was the cause of the issue until Crowdstrike announced what happened and what was the fix.
Of course they do, any qualified IT department should and does mostly test updates and what they do to the It environment before releasing them.
No networks are designed differently, hence unexpected effects may come.
Tell me you never looked It infrastructure in a global company without actually telling me :)
Microsoft was forced long ago by the regulators to expose the dangerous low-level interfaces that were used by Crowdstrike to crash these systems.
Another problem is software like Crowdstrike uses central control to "simplify" management creating a single point of failure. It is very reason I refuse to use software like this in my I.T. work.
And the worst problem is the fix required an operator approach each server, reboot it into "safe"...
Microsoft was forced long ago by the regulators to expose the dangerous low-level interfaces that were used by Crowdstrike to crash these systems.
Another problem is software like Crowdstrike uses central control to "simplify" management creating a single point of failure. It is very reason I refuse to use software like this in my I.T. work.
And the worst problem is the fix required an operator approach each server, reboot it into "safe" mode, and delete the Crowdstrike driver. As you may or may not know, lots of servers operate "headless" with no monitor and keyboard and even network booting won't get past the bad driver. I can only imagine the panic and personnel required to fix this mess.
Hey Tim, you work on reception at Delta HQ - how many IT Bods have left the building holding a cardboard box this week?
Really I think it's Crowdstrike's fault and, to a lesser extent perhaps, Microsoft's fault. Most tech companies in the US have been putting half-tested junk on the market for decades. They expect the "testing" to be done on the end-user so that they can get their products on the market ASAP and make as much money as possible. After the problems start rolling in, then they release a bunch of "patches" to fix their software...
Really I think it's Crowdstrike's fault and, to a lesser extent perhaps, Microsoft's fault. Most tech companies in the US have been putting half-tested junk on the market for decades. They expect the "testing" to be done on the end-user so that they can get their products on the market ASAP and make as much money as possible. After the problems start rolling in, then they release a bunch of "patches" to fix their software which was defective from the beginning. If these tech companies were run by ethical people who respected customer time and money, they would be testing their products thoroughly before putting it for sale. Instead, the customer ends up suffering with lost time, aggravation, lost data, etc. while the tech CEOs reap the profits. It's really the same with so many products these days, whether cars, refrigerators, airplanes, and so on. Those unethical people running such companies should be thrown in prison and the companies shut down. At least we have attorneys to punish them via civil lawsuits, since as we can see with Boeing, the US government doesn't have the decency to do what's right and thrown 'em in prison.
Maybe Microsoft / Crowdstrike should call Ben as an expert witness to explain why its mostly Delta's fault.
Almost certainly Microsoft has no responsibility here. From the descriptions I have seen Crowdstrike probably inserted a low level operating system driver which crashed Windows. The IT admistrators at Delta would have had to approve this or Microsoft would not allow it to be installed. Windows would not have crashed without Crowdstrike taking it down. If you submerge your computer in water it is not going to work very long. It doesn't mean the computer...
Almost certainly Microsoft has no responsibility here. From the descriptions I have seen Crowdstrike probably inserted a low level operating system driver which crashed Windows. The IT admistrators at Delta would have had to approve this or Microsoft would not allow it to be installed. Windows would not have crashed without Crowdstrike taking it down. If you submerge your computer in water it is not going to work very long. It doesn't mean the computer or the OS is bad, you just did something bad to it that caused it to fail.
Microsoft's responsibility is WHO they allow to have deep access to their operating system
Submerging in water is not the correct analogy. If the power company suddenly delivered 440 volt electricity to your house or business is more like the proper analogy.
Anyone can write anything to cause windows to crash. I don’t think CrowdStrike need Microsoft permission to do that. Microsoft is in competition with CrowdStrike with this kind of endpoint protection software. So I am pretty sure Microsoft will just say: Hey Delta, it is your own fault to use CrowdStrike and not Microsoft products.
I think it is not all or nothing. I think the two IT companies had some degree of fault. Perhaps not to all of Delta's problems but that's another issue.
From United’s reacconodation scandal, to Delta’s scheduling meltdown, I really think US airline CEOs are the most arrogant people in this world. They just can’t accept that the airline under their leadership is at fault. They will work tirelessly to find someone else to put the blame on. It’s no wonder their employees act like they’ve been wronged by their customers. They learn from the best. US airlines is an industry where service means disservice.
This is likely the first of many and I think they will be successful. Really inexcusable for having not tested. Companies may start rethinking putting all their assets in the Microsoft Azure cloud.
This is why all Apple apps need tested and approved by Apple.
This is incredible! A Delta story where @ConcordeLadyBoy posts MORE than @Dim Sum. Incredible!!
In tort law, you take your victim as you find them. In other words, if you accidentally bump into someone as a result of your own negligence, and that person happens to have a condition which makes their injuries a thousand times worse than the average person, then the tortfeasor is fully responsible for those damages suffered. In Delta's case, the entire ordeal was set in motion due to the IT provider's incompetence and error....
In tort law, you take your victim as you find them. In other words, if you accidentally bump into someone as a result of your own negligence, and that person happens to have a condition which makes their injuries a thousand times worse than the average person, then the tortfeasor is fully responsible for those damages suffered. In Delta's case, the entire ordeal was set in motion due to the IT provider's incompetence and error. Just because Delta had not acquired the latest and greatest crew scheduling program that might have mitigated the damages should make no difference, especially since even with that system, other airlines were severely impacted. To think otherwise would be like saying that a victim of a red-light-runner is partially at fault for driving a 20 year old car that doesn't have the newest model of airbags. The IT company (and certainly Microsoft) should have liability insurance, and quite frankly, if they can't test their software updates before release, then they shouldn't be in business.
thank you.
that is EXACTLY the point.
DL's systems were running perfectly fine but were destabilized by CRWD which was allowed in by MSFT.
Trying to find fault w/ how DL runs its IT and citing how quickly other airlines got their systems is the OPPOSITE of the point.
Despite the narrative that so many have pushed, DL HAS invested hundreds of millions in IT and had a more advanced IT platform...
thank you.
that is EXACTLY the point.
DL's systems were running perfectly fine but were destabilized by CRWD which was allowed in by MSFT.
Trying to find fault w/ how DL runs its IT and citing how quickly other airlines got their systems is the OPPOSITE of the point.
Despite the narrative that so many have pushed, DL HAS invested hundreds of millions in IT and had a more advanced IT platform than many of other airlines - remember that DL runs its own reservation system and has customized enormous amounts of software to that res system and all of the associated support structures.
If someone comes along and blows that all up by sending fatally flawed code which should have never been released to the world in the way it was released, then the case is no longer just about the typical warranties but sheer negligence.
MSFT's responsibility is who they allowed to alter their operating system and in creating a system that could not be rolled back if a vendor inserted fatally flawed code.
The amount of damages that DL is asking for is really quite small compared to the total resources of CRWD and MSFT. DL will keep the issue in the forefront as long as it takes with negative impact esp to CRWD.
It is far easier - as is true with most lawsuits - just to settle, divide the bill, add insurance from multiple parties, and move on.
@Tim
They are asking for your credentials.
I am an attorney and work in IT. I negotiated these kinds of contracts many times with large Fortune 500 companies, states, federal government etc. when my company sold bolt on ERP products that could take mission critical systems down. Typically, you have a clause limiting liability and another clause excluding consequential damage.
Further, the court will examine what Delta's disaster recovery plan was. Mission critical systems not have a failover that's tested regularly...
I am an attorney and work in IT. I negotiated these kinds of contracts many times with large Fortune 500 companies, states, federal government etc. when my company sold bolt on ERP products that could take mission critical systems down. Typically, you have a clause limiting liability and another clause excluding consequential damage.
Further, the court will examine what Delta's disaster recovery plan was. Mission critical systems not have a failover that's tested regularly is a big no no. Court will engage in fact finding about Delta's systems and compare them to industry standard.
I think Delta is engaging in a PR move. It should have humility to apologize to its customers.
Exactly this.
They are doubling down on blaming the someone else.
This might be a smart move because by the time they lose the case, all this would have been forgotten too.
It's consumers like us who needs to remind each other where the fault really lies and who should be accountable.
If you're paying a premium for something, make sure you get premium in return.
While AA or UA are also denying responsibility,...
Exactly this.
They are doubling down on blaming the someone else.
This might be a smart move because by the time they lose the case, all this would have been forgotten too.
It's consumers like us who needs to remind each other where the fault really lies and who should be accountable.
If you're paying a premium for something, make sure you get premium in return.
While AA or UA are also denying responsibility, they at least fixed the problem much faster.
Lucky ones get taken care by AA.
And at least those didn't, they still left the AAdmirals Club open all night, unlike SkyClosed.
Everything you state is correct. However, Boies was hired because of his deep political connections, and if these connections decide that Crowdsource is no longer a team member in good standing, they have the people in position to get the court to make a political decision holding them accountable instead of one guided by the rule of law.
Helpful context. I work at a major US hospital and over 70% of our workstations went down. No problem, we continued caring for patients and were fully back online within 24 hours. Why? Because we always have work-around procedures for downtime. Mission critical keeps on going.
It's their own fault. You absolutely should not run mission critical PCs with Windows Update or any other vendor specific update mechanisms enabled. You test updates to ensure no deleterious effects, and batch roll them out through periodically re-installing (weekly or monthly) previously tested and validated corporate OS builds. The systems should also be on properly firewalled networks, USB ports locked down, etc.
Do you also grow your own food?
Build you own house from the tree you chopped?
Travel by horses you breed?
You're blaming the right cause for the wrong reasons.
The crash was the result of Crowdstrike getting access to the Windows Kernel in Microsoft Azure and pushing out the quick flawed update. Able to replicate fast within the Azure cloud.
Companies need to go Hybrid and keep a backup system on Prem.
What happened to "Rule 240" which basically endorsed a passengers ticket over to another airline that could accommodate them with a new reservation? When I worked for the airlines (many years ago I admit) it was easy endorse the ticket and rebook someone on another carrier we interlined with, regardless of a mechanical or weather delay/cancellation. Many times I would even rebook a coach passenger in first (when no other seats were available in coach)...
What happened to "Rule 240" which basically endorsed a passengers ticket over to another airline that could accommodate them with a new reservation? When I worked for the airlines (many years ago I admit) it was easy endorse the ticket and rebook someone on another carrier we interlined with, regardless of a mechanical or weather delay/cancellation. Many times I would even rebook a coach passenger in first (when no other seats were available in coach) under the Rule240 guidelines. We had more happy campers than irate passengers, that was for sure!
Rule 240 went away with airline deregulation and CAB.
You have taken care of pax at the discretion of the airline not because of Rule 240.
As clueless most airline employees are now as then.
What you did was exactly what you said but only that, you endorsed the ticket.
I only wish I had you as my agent when things go wrong.
IATA 735d for international flights. The problem was most airlines were already fully booked anyhow being peak season. Airlines can also reject a sale from another carrier. Officially you should phone them which is impractical and more or less impossible.
Time to sue Delta using the same arguments they made against their own IT vendors?
What still amazes me the most about all of this is that American was able to recover the quickest of the big 3. American - the airline whose entire operation falls to pieces at the drop of a hat.
American uses a different platform for most ops and that was not impacted. It was a much smaller set of computers that were impacted at AA. That’s why
Hasn’t really been true about aa for a few years now
They may be crap financially but their operation has been quite resilient and recovers well even with storms
So for those who think this is going to be some cakewalk dismissal in favor of Crowdstrike, I'd suggest you take a step back.
Boies isn't the type of attorney who's motivated by upfront payment anymore.... the dude's made millions (upon millions!) of dollars, and has no need to ever really work again.
However, namesake and brand-equity are more their thing, because THAT brings in long-term residual funds for the firm's founding/equity partners, even if...
So for those who think this is going to be some cakewalk dismissal in favor of Crowdstrike, I'd suggest you take a step back.
Boies isn't the type of attorney who's motivated by upfront payment anymore.... the dude's made millions (upon millions!) of dollars, and has no need to ever really work again.
However, namesake and brand-equity are more their thing, because THAT brings in long-term residual funds for the firm's founding/equity partners, even if they personally never try another case.
Thus, he's not the type of lawyer who'd take a case that's going to get him laughed out of court, no matter how high the upfront billables are. He'd lose more money in the longrun, from doing that.
So if he's willing to take this case, I'm betting there's some issues in play here, that might not be clear to we outsiders, but that could bode in Delta's favor...... at least enough to get some kind of settlement.
You sound like Tim Dunn for Boies.
He also defended Harvey Weinstein and Theranos.
So much for "namesake and brand-equity are more their thing," because "the dude's made millions (upon millions!) of dollars, and has no need to ever really work again"
OUCH!
Maybe Paul Weiss can chime in, if his busy schedule allows but...
You give big-name attorneys too much credit.
There's no such thing as bad publicity.
Signed, spouse of a long-suffering but highly paid litigator.
Fairly certain that Michael Avenatti, Rudy Giuliani, Tom Girardi, Rachael Rollins, Fani Willis, etc would disagree....
Fani Willis is on the level of Rudy Giuliani now?
Disqualified from engaging me on anything from now on.
Both facing significant ethics probes, the latter having already been disbarred, the former facing a renewed possibility of such. What differences do you see?
Exactly what have you ever accomplished, such that that would be a detriment?
What have you done, except comment on a blog? I know my net worth.
LOL.... sure ya do, champ.
I asked you a question.
As has been noted by a number of firms that specialize in IP law, Delta has a very tough road to get any compensation from either Crowdstrike or Microsoft. First of all the stand contract for both (I was a CIO and CTO who negotiated and managed hundreds of similar such contract) caps damages are fees paid so that is a major issue. Also, there is an exclusion for consequential damages (and most of what...
As has been noted by a number of firms that specialize in IP law, Delta has a very tough road to get any compensation from either Crowdstrike or Microsoft. First of all the stand contract for both (I was a CIO and CTO who negotiated and managed hundreds of similar such contract) caps damages are fees paid so that is a major issue. Also, there is an exclusion for consequential damages (and most of what DL incurred fall into this category). Finally you have the fact that much of DL's damage was ultimately related to how they have set up their IT environment, along with how it is staffed and maintained. The fact other airlines didn't have nearly the problem DL did undermines a lot of their case.
One attorney summed it up by saying the rest of the summer (and likely fall) will be "hell on earth" for the CRWD attorneys but ultimately expect them to pay little if anything to DL.
Part of this is a PR move by DL as evidenced by the fact they went public with it and that they haven't actually filed a lawsuit.
Colour me surprised...not! It is the usual modus operandi for major US businesses. Sue the service provider, regardless if it has any merit, or not. Expect the countersuit for a defamation. LOL. Would it include requests for various disclosures? It would be interesting to understand the way Delta runs its internal systems.
I don't fly US airlines much, but if I do, it is mostly on Delta, and quite happily.
The meltdown was...
Colour me surprised...not! It is the usual modus operandi for major US businesses. Sue the service provider, regardless if it has any merit, or not. Expect the countersuit for a defamation. LOL. Would it include requests for various disclosures? It would be interesting to understand the way Delta runs its internal systems.
I don't fly US airlines much, but if I do, it is mostly on Delta, and quite happily.
The meltdown was a sorry spectacle. Let's hope for an operational rethink at Delta.
The idea that Microsoft is responsible for software that delta chose to install is so stupid that it makes me doubt delta’s exec team even more.
Dang. Pre-Covid, Boies already earned $1K/ hour. Other global airlines CEOs do not resort to lawsuits like their US counterparts because their cultures are wholly opposite. Did Lion Air and Ethiopian Airlines sue or collect damages from Boeing for the fatal crashes? American CEOs' priorities are quarterly profits and deflection of blames. Instead of engaging in costly lawsuit that may not yield favorable verdict, they should focus on the roots of the problem to improve...
Dang. Pre-Covid, Boies already earned $1K/ hour. Other global airlines CEOs do not resort to lawsuits like their US counterparts because their cultures are wholly opposite. Did Lion Air and Ethiopian Airlines sue or collect damages from Boeing for the fatal crashes? American CEOs' priorities are quarterly profits and deflection of blames. Instead of engaging in costly lawsuit that may not yield favorable verdict, they should focus on the roots of the problem to improve the company's operations. Likewise, the main responsibilities of the colleges' presidents are fundraising and costs cut, not academic improvements. Take Jack Welch as an example. He was a ruthless SOB: aggressively snapped up financially struggling companies,
viciously trimming labor costs resulting employees' loss of livelihood and
pensions, greedily sold off valuable assets of bought companies,
maliciously saddling them with heavy debts, and finally sold them off to squeeze the last juice. Disgracefully, Welch continues to be a protege of countless CEOs after him. Fortunately for Boeing, it just named a new competent CEO who has background in engineering and aerospace industry. Hopefully, he will replace all other executives under Calhoun to effectively turn around Boeing.
That's....... not true at all.
Of course they did.
Ethiopian in particular wasn't playing, as they were happy to meet Boeing in court after rejecting multiple settlement proposals from Boeing.
Boeing ended...
That's....... not true at all.
Of course they did.
Ethiopian in particular wasn't playing, as they were happy to meet Boeing in court after rejecting multiple settlement proposals from Boeing.
Boeing ended up making a last-minute settlement in September 2021, prior to legal proceedings beginning in earnest. Can't imagine how much they had to cough up to make that go away.
Yup, Crowdstrike & Microsoft should totally offer a settlement in the form of a goodwill gesture service credit to be used 1x that expires in 1 year.
To be clear, they all share some responsibility and Ed Bastian should be held personally liable.
The $10 gift card wasn’t enough?
Now that I finally understand what happened on a technical level I don't see how Microsoft should be held responsible. It's almost like if your car was parked on the street and a guy who does landscaping drove by and scratched your car. So you sue the landscaper but you also sued the guy who owns the house who hired the landscaper.
@Bob
Not the right analogy but still proves the point.
In the end, only lawyers win.
I feel pity for TD. HIs misguided defense of DL with whataboutism is just pathetic. Also, he has no freaking idea on how IT works. When every freaking large business in the world recovered within a day or two but DL continued to struggle for more days, it was very obvious what caused the meltdown.
This is a nuisance lawsuit DL filed to clearly shift the blame. Premium companies own up their mistakes rather than...
I feel pity for TD. HIs misguided defense of DL with whataboutism is just pathetic. Also, he has no freaking idea on how IT works. When every freaking large business in the world recovered within a day or two but DL continued to struggle for more days, it was very obvious what caused the meltdown.
This is a nuisance lawsuit DL filed to clearly shift the blame. Premium companies own up their mistakes rather than blaming it on others. Michael Scott once said I do want the credit without any of the blame.
Let’s not forget that it is the EU that requires Microsoft provide Kernel level access to other companies. This was as a result of the Anti trust lawsuit many moons ago.
MacOS and Linux are not required to provide this low level access so we’re not affected. I’m not swimming in the weeds deep technically on this in the windows world to know if Crowdstrike could design there windows software in a similar fashion
Exactly. This outage was 100% a Crowdstrike outage and 0% a Microsoft outage. Microsoft is required per antitrust to provide a destructive level of access to companies like Crowdstrike. It was Crowdstrike's responsibility to use that level of access responsibly. Remember that Crowdstrike is a competitor to Microsoft's Security products
I suspect some of the confusion is also that there was a 6 hour period of an Azure outage directly before the cloud strike outage that has a lot of people connecting unrelated issues
It's boot level kernel access so it's os independent. But in this particular instance it's the windows driver that crowdstrike updated and I don't know what the heck happened to regression testing. Seems like a company of that size would have things in place to prevent a oops like that.
Good, sue them into the ground.
These arrogant, out of touch tech companies deserve to be taken down.
And "the cloud" as a concept needs to die a most expeditious death.
“And "the cloud" as a concept needs to die a most expeditious death.”
Why do you say/think that?
Unix geek probably
I think the exact opposite. Completely clueless on IT ops, but sharing an opinion anyway.
Tell me you don't understand tech without telling me you don't understand tech.....
@Robin Sound like a very boomerific ludite. The moment we allowed 2 computers to talk were already going to the cloud. I guess you want to go back to the days of travel agents booking your plane tickets by calling the airlines themselves. The out of touch arrogant person to me is not the tech company...
Crowdstrike: We would like to offer Delta 100,000 CS points (blackout restrictions apply) for the delay you've endured.
Thank you for being a loyal Crowdstike customer, we now consider this matter closed.
@ben, OMAT got a major shout out from Jim Cramer Squawk on the Street yesterday on this topic!
You mean Tim Dunn of the stock trading?
There is almost certainly a clause in the contract capping limitations of liability for gross negligence for both direct and indirect damages
I'll repeat an analogy I made in another post:
If you drive over a nail and get a flat, you can blame the nail. If you keep driving around on the flat and end up needing new rims too, that's not the nail's fault.
CrowdStrike's liability can only go so far. At some point it's on DL for failing to do what ever other airline did.
Analogies never work.
Except that you’re wrong. Delta failed to stay on an obsolete windows platform like SW. they failed to stay on a mainframe like AA. Delta *happened* to be on the most modern version of windows when others weren’t. What, exactly, do you think they should have done?
The version of Windows is irrelevant. It's on the boot kernel and that's os independent. Their beef is with crowdstrike because that's the vendor delta selected.
Tim, did you major in Deltanomics at Delta U in the Mississippi Delta region? I just cannot fathom the reason why you must side with Delta every time even when it is so wrong or at fault. No company is perfect.
If I recall correctly, you were quite silent when Delta tried very unsuccessfully to overhaul its SkyMiles program which led to a lot of backtracking due to massive backlash from even its most...
Tim, did you major in Deltanomics at Delta U in the Mississippi Delta region? I just cannot fathom the reason why you must side with Delta every time even when it is so wrong or at fault. No company is perfect.
If I recall correctly, you were quite silent when Delta tried very unsuccessfully to overhaul its SkyMiles program which led to a lot of backtracking due to massive backlash from even its most frequent flyers and no, I do not mean most loyal as we all know that is you. In this situation, just let the court make its decision. We may even find out if it truly is the so-called most premium airline.
Nah, Tim is the uneducated one. But he had a unique skill and became a gunman for his made up unit, Delta Farce.
He now shoots down any comments against Delta.
Typical of what I see in the world of tech today. Penny pinching until the penny pinching causes a major meltdown.
Now talk…AGAIN…about how Pete Buttigieg wants to be an airline CEO. :rolleyes:
Evidence? Tech is the least penny pinching sector out there.
If it's up to engineers they would spend more money for testing and QA. From experience in the 7 companies I have worked for it is ALWAYS the management level who understands ZERO about testing and why it's important that causes these issues. I have had managers tell me point blank let's skip testing we don't have time. We need to deliver something. We can deal with any aftermath later. And I'm like we only...
If it's up to engineers they would spend more money for testing and QA. From experience in the 7 companies I have worked for it is ALWAYS the management level who understands ZERO about testing and why it's important that causes these issues. I have had managers tell me point blank let's skip testing we don't have time. We need to deliver something. We can deal with any aftermath later. And I'm like we only did 2 days of testing and I had estimated for the complete cycle we should do at least 7 weeks soooooo. Every good forsaken company where the manager is some old boomer who is constantly ornery and hate that they can't be a part of the IT conversation because they don't know what's going on but they have to prove they're in charge.
As others have mentioned, Crowdstrike should offer DL a credit voucher to be used within the next 12 months
@Tom, very few insurance policies would protect against something like this, especially to compensate at these levels.
I do think there may be some interesting items that come out of discovery here. Did Crowdstrike or Microsoft mislead DL in thinking through applying a fix? Did either company provide support resources that made the issue worse? Was DL given improper info on how to reboot their crew management system? There could be a lot more here than we know of.
They are not the only ones. Airlines or any other companies,have the right to recoup damages from parties causing damage. You can bet all European carriers will claim refund for all the costs related to paying for hotels etc.
As much as I dislike Delta because of their arrogance, but I'd love to see they sue the crap out of Microsoft and CrowdStrike lol
Either way, it's a win-win for those who dislike all those 3 companies.
There's more than one way to implement MS Cloud Computing into your business infrastructure. Some do it well and are able to recover themselves quickly from outages, others do it less-well and suffer. Delta did it worse than any other firm on the planet and should be ashamed.
please post your credentials.
And if you are really that good, don't you think DL would pay you about $500 million to get it right as you think it should be?
@UncleRonnie is not incorrect. It's a simplified assertion, but it is still fundamentally correct. My credentials? I'm a cloud architect for a CSP, specializing in zero trust, with an M.Sc. in Cybersecurity.
What are YOUR credentials?
BOOM! Headshot!
This will give Tim something to think about
IT exec here...I agree that it's likely that DL's infrastructure was not configured properly to bring the servers back up successfully while managing a massive incoming workload. It's also true that the other airlines weren't hit as hard because DL has a more modern infrastructure.
I think Tim's credentials are
I'm Tim Dunn, b**ch
But he seems to have left this entire post to say it.
Delta should be suing every OMAAT commenter who has ever doubted them!
Do I write to settlement check to Tim or Delta?
once again, you prove that you are not only incapable of having a real discussion but even showing your own face and name, coward.
honestly i thought this latest one was a parody but it does say Diamond next to his name...he's gonna put me out of business if he keeps posting "comments" like this!
So amusing when Tim talks about face and name given the MANY fake names he's used over the years (most banned by other websites). To say nothing of his fake name on Seeking Alpha (Tim Dunn), a 2-bit website that makes it quite clear in their own policies they don't require writers to use their actual names. I'm sure you'll be happy to post your linkedin on here for others to view your "true name"...
So amusing when Tim talks about face and name given the MANY fake names he's used over the years (most banned by other websites). To say nothing of his fake name on Seeking Alpha (Tim Dunn), a 2-bit website that makes it quite clear in their own policies they don't require writers to use their actual names. I'm sure you'll be happy to post your linkedin on here for others to view your "true name" since you love telling others they're fake.
Keep on trying to throw stones, Tim. It only makes it funnier to call you out.
It’s clear from here and his seeking alpha articles Tim is so biased when it comes DL it’s not even funny. You would know someone is Intellectually honest by admitting they screwed up the recovery and DL didn’t change its tune until it heard from the government. Many companies were impacted and recovered with grace not DL. Whenever Tim talks about DL you have to realize DL can never do wrong. Rest assured that type of thinking always brings down good companies.
Delta should be sued for falsely advertising itself as a premium airline.
FU, Ocean!
Any IT service provider worth its salt will have a Limitation of Liability clause in its contract or terms of service along the lines of "in no event shall XYZ company be held liable for any indirect, special, incidental, consequential, punitive, or exemplary damages arising under this contract, blah, blah, blah....."
It seems the great, all-knowing, can-do-no-wrong Delta maybe doesn't understand the contracts and terms of service under which it operates.
Maybe less time and...
Any IT service provider worth its salt will have a Limitation of Liability clause in its contract or terms of service along the lines of "in no event shall XYZ company be held liable for any indirect, special, incidental, consequential, punitive, or exemplary damages arising under this contract, blah, blah, blah....."
It seems the great, all-knowing, can-do-no-wrong Delta maybe doesn't understand the contracts and terms of service under which it operates.
Maybe less time and money in the courtroom and more on ditching Windows and upgrading its business-critical applications.
Hope Crowdstrike doesn't give in to what sure looks like a nuisance suit.
That's not much of a deterrent, as indemnity clauses (standard though they may be) can/will be tossed by a court upon...
That's not much of a deterrent, as indemnity clauses (standard though they may be) can/will be tossed by a court upon a successful showing of negligence, malpractice, deviation from industry standards, etc, etc.
"Questionable individuals" have a right to effective legal representation, too.
Delta - 0
CrowdStrike - 0
Microsoft - 0
Billable hours - 1
The reduction in CRWD's market cap since DAL announced it intended to seek compensation - $4.5 billion.
Oh so I guess in that case the outage was good for Delta then
you clearly aren't the sharpest tack in the desk drawer.
Delta's announcement that it intends to seek compensation from MSFT and CRWD has cost the latter over $4 billion in market cap alone.
Delta hasn't gained anything.
Yet
Oh so I guess in that case the outage is going to turn into a net positive for Delta then
CS stock falling off a cliff doesn't help Delta. If anything it's more of a problem. If they start failing as a company who's going to pay out DL?
More importantly, as others have stated, CS almost certainly has a limited liability clause in place. This is how all these things go. One side sues for the max amount for publicity. Then that amount get's reduced to the actual amount they can claim, then they...
CS stock falling off a cliff doesn't help Delta. If anything it's more of a problem. If they start failing as a company who's going to pay out DL?
More importantly, as others have stated, CS almost certainly has a limited liability clause in place. This is how all these things go. One side sues for the max amount for publicity. Then that amount get's reduced to the actual amount they can claim, then they usually just settle before it goes to court.
Microsoft also holds no significant blame here so that side of the lawsuit is meaningless. CS Falcon runs on MS machines but it isn't an MS product.
In the end DL will probably get little to nothing in compensation, but no one will care because that will happen 1 year + down the line when everyone has forgotten about this. They got what they wanted short term, the blame shifted off of them.
first, Ben, tell us how much American, Delta and United actually spend on IT per year and specifically on their crew tracking systems.
You have no clue what any of these airlines have spent on IT and you also don't even know what systems any of these airlines use.
Delta was running most of its systems on cloud computing on the Microsoft operating system with CrowdStrike as their software to protect against cyber threats.
...
first, Ben, tell us how much American, Delta and United actually spend on IT per year and specifically on their crew tracking systems.
You have no clue what any of these airlines have spent on IT and you also don't even know what systems any of these airlines use.
Delta was running most of its systems on cloud computing on the Microsoft operating system with CrowdStrike as their software to protect against cyber threats.
DL's previous IT systems were mainframe based. Remember the power outage not that many years ago that caused a massive IT and operational meltdown? DL has invested more than $1 billion on IT in the past 5 years.
The only mistake Delta made is that they used CrowdStrike on the Microsoft platform on all their equipment and assumed both companies would not do something that would turn computers into blue screens of death.
CrowdStrike pushed failed code to the world - a complete "no no" in IT - rather than releasing it in batches - and the Microsoft operating system can be shut down by poor quality 3rd vendor software.
It is very UNLIKELY that other airlines used CrowdStrike across their entire enterprises. AA was the LEAST likely to use it because their systems were up very quickly.
There is no evidence that UA's crew tracking systems used the same architecture as DL's.
In case you missed it, Ben, airlines do not run the same programs on the same types of computers with the same architecture.
Quit making a fool of yourself by drawing conclusions when you have no idea about the technical aspects of what was involved.
This would also be a good time to remind you and your readers that there was no basis for compensation from WN's IT failure because they simply didn't invest in modern technology.
There was no basis for UA to seek compensation for its OWN errors in overscheduling EWR which led to their operational meltdown.
Both of those meltdowns took longer for those airlines to stabilize their operations than Delta.
Delta didn't seek compensation for its other IT meltdowns because they paid the bill themselves.
The topic is clearly over your head, Ben, and yet you repeat the same popular mantra with no evidence to support your statements.
And more of CRWD's market cap has been erased just since news of DL's intention to seek compensation than the actual compensation DL could possibly seek.
MSFT's market cap is over $3 TRILLION - more than the revenues of the entire global airline industry - and also has fallen more than half of what DL is likely to ask from MSFT.
There are HUGE financial incentives for CRWD and MSFT to settle not just with DL but also other companies that suffered huge financial impacts.
And if this case goes to a judge or jury, let alone if DL manages to get some legislative people behind this, the impact to CRWD and MSFT will be far, far larger than if they simply each throw in $200 million, let DL's insurance pay another couple hundred million - and then CRWD and MSFT will figure out how to never due to any other company, let alone the entire world, what was allowed to happen almost 2 weeks ago.
It isn't hard for anyone with a modicum of intelligence and perspective to see that a couple of massive IT companies created huge damage to companies around the world with Delta, because of the architecture of its IT, the most hard hit.
As for the accusation of how DL handled this, there are hundreds of news articles about the UA and WN IT meltdowns as well as from other airlines and NONE of them come close to the pristine, customer focused nature that you think occurred.
The National Guard was called out to multiple airports for the WN meltdown and there are equal amounts of stories about how poorly UA's customers were treated.
I almost feel sorry for you.
In fairness it’s true. I doubt Ben is an expert or has any knowledge on Altea DCS and Deltaterm amongst the dozens of other systems airlines use.
All these companies made infrastructure decisions and there were pros and cons to everything. It just turned out Delta made a decision with some serious cons. Surely you know as well that the "only mistake" of relying on a single point of failure turned out to be a pretty massive mistake
Creating the worst IT meltdown in global history is not just a bad vendor decision. When a company that has access to billions of computers pushes fatally flawed code, it is not just an issue of vendor choice.
When Microsoft is incapable of stopping a failure of a vendor code update from irreparably taking down MSFT's operating system, there is a problem far beyond vendor choices.
And, no, I don't get paid anything to think....
Creating the worst IT meltdown in global history is not just a bad vendor decision. When a company that has access to billions of computers pushes fatally flawed code, it is not just an issue of vendor choice.
When Microsoft is incapable of stopping a failure of a vendor code update from irreparably taking down MSFT's operating system, there is a problem far beyond vendor choices.
And, no, I don't get paid anything to think. There are a whole lot of people on here that should have been paid to stay in school.
Airbus vs. Boeing is vendor choice. Southwest and United have paid huge prices for bad sourcing decisions.
Antitrust law requires Microsoft to provide that level of access to companies like Crowdstrike. Microsoft has no control over what 3rd party/competitor companies do with that access. It is up to a company's IT team to weigh the risks of using software like Crowdstrike with that level of access and take steps to mitigate them.
Source: I'm CISSP certified (top cybersecurity certification) and have worked in cybersecurity for over 20 years. I am a consultant...
Antitrust law requires Microsoft to provide that level of access to companies like Crowdstrike. Microsoft has no control over what 3rd party/competitor companies do with that access. It is up to a company's IT team to weigh the risks of using software like Crowdstrike with that level of access and take steps to mitigate them.
Source: I'm CISSP certified (top cybersecurity certification) and have worked in cybersecurity for over 20 years. I am a consultant for companies like Delta - they hire me to plan for these types of events.
@Tim Dunn .... "And, no, I don't get paid anything to think."
Finally something you have said that everyone can agree with.
Agree, in airline terms, Southwest was lauded for their single fleet efficiency until the fleet had issues. The Big 3 were frowned upon financially for all the complexity that comes from large fleet varieties, until they were lauded for "insuring" their diversification.
We all wish we had tomorrow's newspaper today, wouldn't that make life so much easier? I'd be happy getting one of tomorrow's Wall Street Journals today, just once in my life..... No...
Agree, in airline terms, Southwest was lauded for their single fleet efficiency until the fleet had issues. The Big 3 were frowned upon financially for all the complexity that comes from large fleet varieties, until they were lauded for "insuring" their diversification.
We all wish we had tomorrow's newspaper today, wouldn't that make life so much easier? I'd be happy getting one of tomorrow's Wall Street Journals today, just once in my life..... No need to get greedy and ask for two!
Does Ed give you Delta 360 status for writing this?
too long, didn't read
Well, Ben along with the rest of the known universe can all see that Delta is using Windows because Linux and Chrome devices were unaffected, so we all know that part of Delta's environment
We've all had over a week to see that one or more of Delta's mission-critical applications are not up to the task so who cares about the underlying database, development language, middleware, etc. IT JUST CAN'T GET THE JOB DONE.
...Well, Ben along with the rest of the known universe can all see that Delta is using Windows because Linux and Chrome devices were unaffected, so we all know that part of Delta's environment
We've all had over a week to see that one or more of Delta's mission-critical applications are not up to the task so who cares about the underlying database, development language, middleware, etc. IT JUST CAN'T GET THE JOB DONE.
To Crowdstrike's credit, they isolated the problem very quickly along with an incredibly simple fix (as these things go), at least as far as removing the update file that caused the problem. For IT departments running Windows, the Crowdstrike fix was child's play.
The silence is deafening regarding headlines of any of the other very large companies using Crowdstrike having anything remotely close to Delta's meltdown - nada, roaring silence, crickets. Just Delta.
Not defending Tim... However, I have to take issue with TD's CIO, who I think is not a CIO. This was NOT child's play. We had to reboot systems up to 15 times and if bitlocker was present (as it should), then the bitlocker key - or a workaround - was necessary. This meant an IT professional had to touch every system. While most enterprises (likely Delta included) got all their infrastructure up on Friday....
Not defending Tim... However, I have to take issue with TD's CIO, who I think is not a CIO. This was NOT child's play. We had to reboot systems up to 15 times and if bitlocker was present (as it should), then the bitlocker key - or a workaround - was necessary. This meant an IT professional had to touch every system. While most enterprises (likely Delta included) got all their infrastructure up on Friday. Most were working on endpoints for days. While it was not the worst to recover, it was NOT child's play.
There were likely other companies hit WAY harder. However, few organizations have the kind of knock-on effects as a transportation company -- assets in all the wrong places.
There were hospitals with their ERs on diversion, 911 systems down and more. But when those systems come up, they basically start up from where they are. They don't have to reposition all their assets.
The airlines unarguably had a harder problem than others.
@AD the solution for end points were remarkably simple. We had a communicated solution out during the 19th regarding how end users with bitlocker could manage the issue themselves, and I know a lot of people did. One issue was of course how to communicate to users that can't access their laptop. But we have contingencies for that as well.
What I find particularly surprising is that OUR security teams did not test the...
@AD the solution for end points were remarkably simple. We had a communicated solution out during the 19th regarding how end users with bitlocker could manage the issue themselves, and I know a lot of people did. One issue was of course how to communicate to users that can't access their laptop. But we have contingencies for that as well.
What I find particularly surprising is that OUR security teams did not test the update before releasing it. But I guess that is a process we have to revisit.
While my company uses a Crowdstrike competitor and was unaffected, I did take it upon myself to review the Crowdstrike remediation steps published by Microsoft and, if one has been around Windows long enough, following steps similar to the Crowdstrike remediation steps becomes almost second nature.
BSODs can be caused for many reasons and I've seen a lot of different ones in 30+ years and, unfortunately, most of the time it does require touching each...
While my company uses a Crowdstrike competitor and was unaffected, I did take it upon myself to review the Crowdstrike remediation steps published by Microsoft and, if one has been around Windows long enough, following steps similar to the Crowdstrike remediation steps becomes almost second nature.
BSODs can be caused for many reasons and I've seen a lot of different ones in 30+ years and, unfortunately, most of the time it does require touching each affected device; that goes with the territory in the Microsoft world.
From the news reports, it doesn't sound like Delta had an inordinately difficult time recovering the Windows devices, it appears that their critical software applications were not designed to recover from a situation where most or all of the networked devices are removed from the network (practical effect of a BSOD) and are recovered piecemeal.
I can literally tell you my company iPad at AA has Crowdstrike on it. We are dependent on it. throughout the company. We literally had IT at our data centers in the middle of the night with flashdrives to to bring the operation back to life with a priority on the systems for operational control and crew tracking, in that order
So... You don't know either.
"It is very UNLIKELY that other airlines used CrowdStrike across their entire enterprises. AA was the LEAST likely to use it because their systems were up very quickly.
There is no evidence that UA's crew tracking systems used the same architecture as DL's."
Timmy, you really, really need a proper lesson in concise writing style. Plugging in buzz words and fillers in an effort to deflect the facts of what occurred here does not endear you any higher to readers than the abysmal reputation you already possess.
It's all intentional, designed to blur a weak argument, and most of all feed his weapons-grade addiction to arguing with strangers on the internet.
Good lord take it to a publisher
This just seems incredibly unlikely to be fruitful for several reasons. First, if Delta was successful in its claim, which seems very unlikely, then two things will likely happen next:
1) Delta will have to separate the initial damage from its ongoing damage (that is, the crowdstrike issue from the issue that followed with its scheduling software) and that's going to be a HUGE undertaking to figure out what was caused by which issue,...
This just seems incredibly unlikely to be fruitful for several reasons. First, if Delta was successful in its claim, which seems very unlikely, then two things will likely happen next:
1) Delta will have to separate the initial damage from its ongoing damage (that is, the crowdstrike issue from the issue that followed with its scheduling software) and that's going to be a HUGE undertaking to figure out what was caused by which issue, and from my understanding it sounds like the majority of the issues were from Delta's scheduling software, not the Crowdstrike issue, so the huge chunk of their damages still aren't going to be recovered, which brings me to....
2) If Delta is successful then every company is going to sue Crowdstrike, Crowdstrike will file for bankruptcy, and no one will get paid anything.
So there may be a narrow path to a legal victory but I just don't see how there is a path to any sort of financial recovery.
This will go to the insurance companies.
That's it.
By this logic, Delta (or any other airline) should also be accountable to pay for hotel/car rental etc. cancellations that may result due to a flight cancellation
Good luck, Delta. You still suck.
Not sure why they're suing Microsoft seeing as this mess was entirely Crowdstrike's fault (if your Windows computer didn't have Crowdstrike on it, then it wasn't affected), but in any case, my understanding is the the Crowdstrike terms of service limits their liability to what you paid them.
I don't see this going very far to be honest.
I think the MSFT suit is DOA. I don't see how any judge looks at this piece specifically and asks how the rest of the world managed to correct the issues in a matter of *less than a week*
I'm curious to see how the Crowdstrike piece plays out, although I doubt this gets settled in court.
Delta might get a tiny amount and will pay a massive amount in legal fees.
They were impacted 3x as much as united and American because they failed to be prepared. Sucks to suck
In other news, Delta's legal team will also be suing Toyota after someone put diesel in their Corollas and they no longer run.
I sincerely hope that Delta gets a crazy high legal bill, completely loses the case & appeal, and then is forced to pay really high legal bills of CrowdStrike & Microsofts as well. And that this all get heavily reported in the Press/Media. #Deltageddon
This is exactly what insurance is for ...
Yup! I think we're really seeing how Delta makes it's true profits, not from being Premium, but from not investing in it's Tech/Software, viable contingency plans, and insurance for events like this. I mean, lowly AA was able to get back up and fully running by 5am THE SAME DAY. Gimme a break Delta!
On one hand I feel sorry for Delta because their intent, in theory, was to have a robust cybersecurity posture - for which I applaud them. However, they put all their eggs in the CrowdStrike basket (or so it seems) and simply didn't plan for it to ever go tits-up. Which it did. And Delta was caught with its pants down, unable to even get things in Excel spreadsheets or a whiteboard or ANYTHING resembling...
On one hand I feel sorry for Delta because their intent, in theory, was to have a robust cybersecurity posture - for which I applaud them. However, they put all their eggs in the CrowdStrike basket (or so it seems) and simply didn't plan for it to ever go tits-up. Which it did. And Delta was caught with its pants down, unable to even get things in Excel spreadsheets or a whiteboard or ANYTHING resembling an old timey scheduling sheet. And I think you're right: All this will do is reinforce the C-suite - which is hardly full of tech-savvy folks - to keep thinking that older tech is "better" and accruing a form of tech debt. I fully expect this will happen again. Not with CrowdStrike, but with something else.
For delta ? Insurance is there cover catastrophic events not pay for bad business operations, the cost of premiums will more the cover the gain of such a claim making this a bad idea
No, that is EXACTLY what Business Interruption Insurance is for - but I hope Delta doesn't get a penny, because their behavior was abominable.