IAG CEO Denies That British Airways Had An “IT Failure”

It's an IT failure. I say that, currently working in IT...and I'm sure my entire IT budget is less than what BA probably pays a single DBA. At this office, IT is responsible for everything which might potentially affect IT, so power, generators, air conditioning, even roofing, falls in our lap. We've met with management and threshed out what is acceptable / non-acceptable downtime vs. expense and came to an agreement on what both sides feel is reasonable. Even when the power grid blew up last year, setting an adjacent building on fire and damaging switchgear, our racks kept on running. We've lost a few servers, but usually are back up to full capacity within 10 minutes, which is the design. Then again, we are running 20 year old servers (no joke) in some cases. In a previous life, I did broadcast engineering. It was pounded into me at my first broadcast job that "everything MUST run 24/7, even during the middle of a Category 5 hurricane." That was our edict, and we were given the appropriate funds to make it happen. I say appropriate, as in just enough. We really didn't go hog-wild with this. About the most extravagant thing we ordered were stainless-steel flood barriers for the doors & windows on the first floor. We never got to test it to a Cat 5, but the facilities withstood multiple Cat 3 / Cat 4 hurricanes without missing a beat, and the flood barriers proved to be a wise expenditure. I say this to prove that it *can* be done and done on the cheap. BUT you have to design it this way from the ground-up. I also want to say that no outsourced firm will take care of you as well as your own in-house staff will. The outsourced firms may indeed have more experienced people and more resources at-hand, but in an emergency, you're just another customer. The outsourced helpdesk operator has no skin in the game. It's not THEIR business and THEIR data on the line.

C-Level idiots! If you have ever worked on servers like Dell, you know that there is a configuration setting that will automatically restart the server in the event that electrical power becomes available. This is not the default setting. The default setting is the "on/off" button on the server has to be pushed. So, someone in British Airways IT department deliberately configured their servers to restart automatically when power was restored. They did this configuration as a policy and over years. I am not blaming BA IT for that one. It is standard practice in the "computer-biz". The servers were probably not damaged. Some databases likely took a hit. What probably did not come back correctly when power was restored was the network routing. The switching / routing certainly became very confused about was and was not online as power was being restored. Now, here is the fundamental cause. To test system failover, you have to invest a lot of money in duplicate systems and personal redundancy. After 30 years running IT systems, I have yet to see C-Level people spend that kind of money in any major company. Couple that with IT's tendency test for "success" instead of "failure", you end up with major system failures and C-Level idiots running their mouths off about why it was not their fault.

How BA characterizes the event may have more to do with indemnity claims they've asserted against the power company or their business interruption insurers than claims asserted by passengers against BA. If that's the case, you'd certainly want the CEO to be consistent with his public statements.

It's worth noting that an airlines systems are slightly more complex than wacking a $50 UPS on your desktop at home. From what i've read on various tech sites it would appear that the cause was a data centre employee trying to over ride an automated UPS / Generator cut over. The result being double the standard voltage was likely sent down the line to the customer (BA) equipment, causing physical hardware damage. The real issue is the lack of redunduncy. BA supposedly have 2 additional data centres, and you would expect at least one of them to be a live DR site ready to spin up. One of the other interesting facts was some comments around the complexities of BA's systems. Rather than a somewhat simplified unified system, BA reportedly had upwards of 250 different systems integrating together to perform different functions.

I find it difficult to believe that it was a simple power failure. They have back up power in place. Just about every aspect of their operation does. Accidentally pulling the plug from it's socket is pretty much the most basic simple disaster recovery scenario that is nearly impossible to overlook. So no, I don't buy the ceo's response. I'm pretty sure however that with his track record of saving money and outsourcing he cannot continue to advocate outsourcing as a brilliant business move if these sorts of catastrophe's occur. So naturally, do some alt-facts and say no it wasn't outsourcing, someone just pulled the plug and none of our contingency systems kicked in....yeah right.