In the wee small hours of July 19th, Crowdstrike issued an update to their security software. Updates are something we are all used to and it´s a pretty normal thing for us to all have to reboot the computer from time to time because of updates and security implementations. The update that failed was a configuration update for CrowdStrike’s Falcon sensor software, specifically related to Channel File 291. This update was part of CrowdStrike’s Rapid Response Content updates, which are designed to enhance the Falcon platform’s ability to detect and respond to new cybersecurity threats. The update was intended to modify a configuration file responsible for screening named pipes. However, it contained a logic error that caused an out-of-bounds memory read in the Windows sensor client, resulting in an invalid page fault. This led to affected Windows machines either entering a boot loop or booting into recovery mode.
According to Wikipedia, around 85 million computers and servers running the Windows software were affected. Just stop to think about that for a moment – the sheer volume of businesses that are using the software provided by Crowdstrike – this is big stuff!
What strikes me about the situation…
There are two things that came to my mind about this, firstly lessons to be learned and secondly the image of Crowdstrike and how they responded to the situation. Let´s start with the latter. When businesses get things wrong they usually have three main courses of action:
- Try and wiggle their way out of it to limit damage
- Quietly speak to the affected client and come to some sort of arrangement to keep them happy
- Hold their hands up and take full responsibility
Given the scale of this problem, it would be difficult for the 2nd option. The outage made headlines around the world almost instantly. But what impressed me was their choice to hold their hands up and face the music. They actually responded pretty fast, identifying the problem, releasing a repair and from what I understand, working with their clients to implement it as fast as possible. But, perhaps much more profound in my opinion was the way in which they handled the media and public. It´s not unusual for companies of this scale to immediately call in their PR department or pay a high profile PR company to release a carefully contracted statement that loosely addresses the issue and hints at external blame but in this case, the CEO George Kurtz was fast to face the media, holding his hands up, acknowledging the problem and offering reassurance and updates. Frankly it takes guts for any CEO to make the decision to hold their hands up from the outset rather than being forced to later down the line and I take my hat off to him and the company in how they have handled the situation from this point of view.
My personal style in business is one of transparency and I believe that the right level of transparency and ownership can pay dividends later on in terms of the trust that clients and the public build in the brand and my personal view (from the information I have seen as a mere spectator) is that this unfortunate situation could end up cementing a solid reputation for Crowdstrike going forward. I know that personally if I was voting in a board meeting over a choice of cyber security partner, Crowdstrike have set a very favourable impression in my mind as things stand.
Don´t get me wrong, the fallout from this error was massive and ironic – businesses unable to function because of a problem with a service that they trust to prevent them from very such events! It is, however important to remember that this was not a fundamental failure of their platform – that`s to say that if thousands of companies were knocked offline because of a cyber attack then it would cast a very dark shadow over the company but this was not that, it was just something went wrong in the updating process and let´s be honest, mistakes do happen and every business will at some point get it wrong – whoever they are. But, when it´s a business of scale, the effects are huge.
We live in a world that is evolving faster than ever as we are only just beginning this new chapter of AI and just as the good guys are making advances, so are the bad guys. The more we put our trust in technology, the more we have to anticipate possible problems and this brings me nicely into my next point – where does the buck stop? With the suppliers of services such as Crowdstrike, or with us all as businesses owners or leaders?
Contingency Planning – A fundamental that´s more important now than ever before.
I am not going to comment on individual companies and how they were affected by the Crowdstrike outage but I think there are a few things that in general every company needs to address. Firstly, contingency planning. Any serious business will have some type of backup plan – what to do if something goes wrong so that they can continue to operate. If tomorrow there was a problem with the water supply in your town, we would all expect the water providers to swing into action and setup water collection points, distribute bottled drinking water and keep us all hydrated one way or another and over the past I´ve seen these types of things happen.
But when it comes to IT, are we just assuming, perhaps very wrongly, that out outsourced services will take care of everything? Of course, if we contract a company to take care of our IT systems then we expect them to do just that but things can go wrong, and they do and as business owners we must have plans in place on how to cope.
One news story I saw during the Crowdstrike outage was covering delays in an airport and a lady commented that they had hand-written her boarding card at the check-in desk. Now, I don´t know which airline that was or even which airport but it is a sign of a good contingency plan. Yes there were delays, yes there were cancelations but they had a way to somehow resort to plan B and use good old pen and paper. The moral of the story is that whether your business is small or multinational, we should all be including IT disasters in our contingency planning to at least mitigate losses and problems whilst the experts are addressing the underlying problem. I don´t think a business has the responsibility to become cyber security experts – that´s why specialist companies exist but we all do have a responsibility to know what we will do and how we will put in place at least a minimum service when things go wrong. And I use the word “when” because in general we have to accept that the more we rely on technology, the more we will experience bumps in the road at some point.
Hiding behind the small print… terms and conditions!
Again, I am not referring to Crowdstrike in particular or indeed any of their clients in this opinion because I dimply don´t have the facts but having negotiated supplier and outsourcing contracts from both sides of the table in multi-million euro agreements over many years, it´s important to keep in mind that businesses don´t necessarily (country and specifics dependant) have automatic protections in the same way that consumers often do. That´s to say, if we are a B2C company out conditions of business must respect certain laws and in general consumers have protections because consumers are not lawyers so whilst everything we buy these days has some type of contract attached to it, if a company is deemed to be unfair to their customers there are often laws to protect the consumer.
In the business world however, we must be much more aware of what we are signing before we sign it and certainly when we are talking corporate/big scale businesses, certainly in all organisations that I have been involved in, contracts, particularly for outsourced services have to be negotiated by the board and then passed through compliance and legal departments and part of that is to look at what we are agreeing to in terms of damages when things go wrong and what we agree to in the contract.
So if we are talking about a contract for something like a Crowdstrike service (and again, I have never dealt with Crowdstrike nor have I seen their contracts so this is a generalised opinion and what I would be doing as a Director), careful attention should be paid to the “what if this happens” part. Usually there is give and take in this type of thing. It would be very hard for a supplier to say “it it breaks we will pay you all of your losses”, because there are so many variables but I would personally want to see some type of tolerance and service standards established and as part of the contract there would usually be levels, timescales and so on established as well as what the company must do to facilitate any assistance offered by the company.
So, let´s say I am signing my company up to “XYZ” security (fictional) to keep my services online. I would want to see a commitment to service in terms of response times, actions and what level of compensation (if any) they have to pay me. Perhaps it may say that the company must respond within X hours and if they restore service in this time then they are not responsible but if it takes between Y & Z hours then they will pay X per hour, or whatever calculation is used. I would expect there to be cooperation clauses, for example “this person or their team must respond to all reasonable requests by the supplier to allow them access to their property or systems to fix it”. So, I think the lesson to learn here in addition to the contingency planning is that when we use companies like this we establish the contract clearly and build in as many contingencies as possible.
What does the future hold for Crowdstike?
This of course is just my opinion, if you are reading this with a view to investing, you should do your own research as I am not a financial advisor, nor do I give advice! But we obviously saw a huge dip when this happened in terms of their share prices. That´s normal – news like this is about as bad as it gets, certainly in terms of knee jerk reactions but we soon saw it turn and begin to crawl up again. I´m not going to get into valuations of the stock in this article, I´ve written enough and if you are reading this on your coffee break, you ate late back to work! But, although I had heard the name, Crowstrike was not particularly on my radar until this happened when it very much got me looking into them and for disclosure purposes I have added a small amount of this stock to my portfolio because they are clearly market leaders and I think they have a good future.
More than anything, the positive way in which the CEO came out and took it like a man was beyond impressive for me and so I feel confident in the management style of this business and whilst you never know – long term, this whole bump in the road could have been the best bit of marketing and brand awareness that they needed!
Now, I must get back to work – the computer needs to reboot after an upgrade to the operating system. So all being well I will live to write another article!
If you liked this article then thanks for reading. This is a new site setup to give a holistic view to all things business and investing related. Signup for free news and updates.
