Anything that can go wrong…
“Anything that can go wrong, will go wrong.” – Murphy’s law.
It’s been a couple weeks. The work has caught up to me, as it often does. Unplanned work is usually just that, unplanned work, and so it gets in the way of planned work – which derails everything in its path. These confessions are planned work – and so thus had been put on hold.
Most people are familiar with the adage, “Murphy’s Law” – which states that “Anything that can go wrong, will go wrong.” There are lots of articles about it, and you can read what Wikipedia has on it, here.
You’ll get varying opinions on Mr. Murphy and his law, depending on who you speak to. I think most people in the executive/leadership space use it as motivation as to why one must be prepared… because if you’re prepared, there’s nothing for Murphy to make go wrong, because you’re prepared to handle it.
Interestingly, the wikipedia article referenced above has this line:
“According to Richard Dawkins, so-called laws like Murphy’s law and Sod’s law are nonsense because they require inanimate objects to have desires of their own, or else to react according to one’s own desires. Dawkins points out that a certain class of events may occur all the time, but are only noticed when they become a nuisance. He gives as an example aircraft noise interfering with filming. Aircraft are in the sky all the time, but are only taken note of when they cause a problem. This is a form of confirmation bias whereby the investigator seeks out evidence to confirm his already formed ideas, but does not look for evidence that contradicts them”
(taken from https://en.wikipedia.org/wiki/Murphy%27s_law)
Dawkins is right, and I don’t believe that inanimate objects have desires of their own. I also agree with the leadership teachings that you can avoid Murphy’s law by being prepared. That being said, how does that relate to IT? How can you adequately be prepared for anything, when the organization is usually focused on REDUCING IT costs, and planning in preparedness increases cost?
I once told one of my newest employers that I wasn’t going to make any changes, do anything out of the ordinary, or make any recommendations until I had done at least a 30 day assessment of the environment in which I found myself.
The company I had landed in had never brought IT in-house, and had found their IT costs ballooning utilizing external consultants. The company had been purchased by private equity a few years prior, and as most PE firms are, they were mostly concerned with EBITA. As the PE firm revamped the organization, brought in upper echelon talent, they began to look more closely at their expenditures, and IT was no small expenditure. The decision was made to bring IT in-house, to 1) lower costs, and 2) determine the state that the organization was in. There was no SME in house to be able to decipher whether or not the organization was realizing the value provided by the consultant or not. I digress.
Upon my arrival very little direction was given. Lower costs. Just lower them. There wasn’t even a true “IT” budget… more of an IT bucket in which all IT-related spending came from. I had my work cut out for me.
A trusted mentor recommended a book called, “The First 90 Days: Proven Strategies for Getting Up to Speed Faster and Smarter” by Michael Watkins. (Available HERE) and one of the things the book suggests is a 30 day assessment. Being in IT, having zero documentation, and not having anyone on staff to give me the lay of the land, as it were, a 30 day assessment (if not more) was just what the doctor ordered.
What I found was pretty shocking. Infrastructure was mostly, if not all EOSL (End of service life). There were (and still are) single points of failure in multiple places, and core infrastructure had reached its limit in terms of pure bandwidth. Frustration was high as the middle of the day rolled around and most people grinded to a halt as the core infrastructure struggled to keep up. Applications were fragmented, and operationally, end users had begun creating their own work-arounds to circumvent the system’s shortcomings.
My assessment was pretty clear – it all had to go – all of it. You can imagine the organization’s surprised faces when I told them the condition of their system. The board was even less pleased, becoming concerned mainly about the state of backups, and how this would affect the PE firm’s ability to sell the company. But most of all – I was supposed to CUT costs, not bring more to the table!
”A strong wind can take down our system” – is what I said. They took it as hyperbole.
I was being serious.
So the question remains – how does one prepare for Murphy, if you aren’t given the resources to prepare? Proposals are all rejected, and the only approvals were for cost cutting?
Something has to break.
It isn’t pretty.
You purchase insurance not because you intend on getting into a car accident, but because the cost of you getting into one without insurance far outweighs the cost of insurance. (Plus it’s mandatory, but that doesn’t work for the purposes of my illustration)
Organizations have to realize that the cost of stretching the dollar and not having any sort of contingency planning in IT far outweighs the cost of the resources to prevent an outage. I used to work for a company that (reportedly) lost $40,000 an hour when the system is down. All it took was the system to go down for a couple hours (or 36 hours, once) for executive management to invest in IT so that it doesn’t happen again.
But now we’re in a conundrum. As an IT professional, you’re tasked with the health of the system, no matter what the situation you inherited is like. You can’t ever advocate for something to break… because it’s your head when it does.I had an executive tell me once, “This is why you’re well compensated, you have battle pay because we know that the system is less than ideal”
As if somehow my level of compensation (it’s not THAT well compensated) determined the propensity of the system to go down or not. In a lot of ways, you get what you pay for… but paying someone more or less doesn’t change whether or not a system is in good or bad shape.
Fast forward – the organization I work for experienced an outage – one that stopped production for nearly a full day a couple weeks back. Thankfully I had purchased a last-ditch contingency, but the damage was done. A full day of working to get the organization back up and running, and executive management comes back with these types of questions…
And I quote:
“Was something [sic] that was to be done after- hours? In my experience, this is what is usually done to make sure all issues are taken care of and business efficiencies are not compromised.”
And
“I need to be told what is going on on a real time basis when the environment goes down so that we can explain it. I don’t like coming to you with a lot of questions because I don’t know what is going on”
Murphy had struck. And all I could do was sit there and say, “but I told you!”
In this blog I keep coming back to having to do a better job of bridging that gap between IT and the organization. Man is it difficult – I’ll tell you. Just based on the fact that when a system outage happens and the first response from executive management is (paraphrasing), “what were you doing when this happened?” “Could what you were doing have been done after hours?”
I posted this on LinkedIn a little while ago – Correlation does not imply causation.
All to say – is that sometimes something has to break before the company realizes that an investment has to be made in that area. It doesn’t preempt Murphy, but if it pushes the company towards making investments so that when things go wrong, they’re more prepared… then who am I to complain?
That is – unless they fire the IT guy. : |
So you balance, you hedge. You keep the system running as best as possible, while creating as much documentation as possible to prove that your environment is a shift away from coming down. You do your best to CYA, and hope that when things go down, they don’t come hunting for you.
It’d be so much easier if the organization would just listen – Investing in the immediate provides much more savings in the future.
—
A couple weeks back I got a call from an HR person looking for a reference for an old employee of mine. I, of course, gave a great reference and he got the job. (Why would you put someone on your reference list that wouldn’t give you a good review?) He sent a text thread to myself and some of our old teammates (we’ve all gone our separate ways now) informing us that he’s leaving the old stomping grounds, making him one of the last to leave.
We all left under not the most auspicious of circumstances, but in retrospect you don’t realize how good things are until they’re done. My teammates and I from that job still converse pretty regularly, and we all remark how when things were good, they were really good, and we all were enjoying some of the most satisfying, most productive times of our lives. It’s amazing how a great environment makes you want to work.
Alas – all good things must come to an end.
Confessions of an IT Director 