Confessions of an IT Director

The “Right” Fit.

IT Executives tend to wear out their welcome after a little while. I’ve explored this before, but the premise is that with a new IT exec, there tend to be massive changes that accompany the arrival of the person. A new system, an implementation, organization, practices, etc… but as time moves on you get a regression toward the mean — or more simply put (in the article) “extreme performance tends to get less extreme the next time” Why is this? SImply put – you run out of projects to do, and eventually you go into maintenance mode. Right or not, the organization judges IT Executives (Directors) on a “What have you done for me lately” style – and so when the changes become less extreme, the (wrong) thought process is that the IT Executive is doing less, and so therefore the organization begins the process of replacing that executive, which starts the whole process over again. The new person will bring with them a stash of projects and ideas that the company will implement out of excitement, and then eventually that person will wear out as well. 

I don’t have any real stats to prove this, but I had a previous IT Director that I reported to once tell me that the lifespan of an IT Director is 3-5 years. I believe it.

Thus – companies are always on this cycle of looking for new IT executives, depending on where on the “bell curve” the current IT executive is in their lifespan. 

In my career I’ve changed jobs a few times for one reason or another. Better opportunities, better pay, closer to home, increase in title, etc. As I reflect on my career journey to this point, I notice that there have been some similarities as to how IT executives, and even standard IT technical people are recruited, but I’m not sure that’s the best way to do it…. And it has to do with the right fit.

Recruiters are usually given a technical “checklist” of attributes that an organization is looking for in a potential new person. If you’re looking for someone who makes a widget, you usually start by listing a request for someone who has that particular “widget-making” experience. The recruiter very often has no idea what it takes to make said widget, only that the candidate has to meet the “widget-making” requirement on the checklist before the candidate would even be admitted to a hiring manager. 

In the IT world, I find that this backfires quite often. I had a recruiter once tell me, “it sounds like you really know your stuff, but this client is looking for someone with RDP experience, and you don’t have that on your resume.” After assuring the recruiter that I was quite proficient in RDP, he then asked that I revamp my resume to mention it a few times so that the hiring manager can see it. So I did… uneasily, but I did.

This began a snowball effect in my mind about how to go about looking for a job. Does one make a specific resume that hits specific listed points so that you can get through the door? I suppose that’s the way to play the game, but I can’t help but think that organizations are short-changing themselves out of perfectly viable and perhaps better candidates by insisting they have arbitrary attributes that some job description has listed, versus gauging a technical candidate on their intangibles? Is too much importance put on specific acumen as opposed to how a particular person reacts, handles, or adapts to a situation? Can a weighted system where a technical baseline is pitted against a candidates intangibles make for a better choice in candidate?

I had a job interview for an organization once where the “wrong” answer on a technical question cost me the ability to procure the job. I had gotten along swimmingly in my initial video interview with the panel, and all looked very good for a second interview. The premise of this particular job was that this IT Executive was to bring management of IT resources in-house. Things looked good. It was for an exciting, well funded company, and the benefits were great. There was no current executive, but the firm had been using a consultancy to manage their IT needs.

Therein lay the problem. During the second interview, the firm asked the consultant to ask technical questions. Besides the fact that it was a conflict of interest (why would the consultant sign off on a candidate that would potentially lower the consultant’s footprint in the organization), I couldn’t remember an oddly specific question about a feature of Microsoft Exchange. I answered the question, with the caveat that I’d have to look it up (trying to prove that I have the resources necessary to adapt and find the answers that I didn’t know off the top of my head) but it was too late. The nail was in the coffin, and they didn’t hire me, despite the recruiter telling me that I was their first choice. 

Conversely, I worked for another organization that had me interview with a technical person, but then with a non-technical person for “mission fit.” I thought this was a pretty smart approach. There were no “gotchas” in the technical interview, only an establishing of a baseline to prove that I could handle the level of technology that I’d be managing at that organization, and then I was passed on to another executive to determine if I was a “mission fit” (also referred to as “cultural fit”) to the organization. 

The interview went well, and at the end when I was asked if I had any questions, I inquired about the “mission/cultural fit” interview as I hadn’t been part of that to that point in my career. The person I interviewed with told me, “the tangible, we can teach, but we want to make sure you have the intangible.”  

Somewhat recently, I had met with another organization who was looking for an IT executive. After a couple minutes, I could tell that they didn’t really know what they were looking for. There was someone in that role before, they had left, and for 8 months the role was vacant. They quizzed me incessantly about their particular ERP, the different applications they used, everything from their hardware purchases to data consolidation to worldwide collaboration between different ROBOs (remote office branch offices). I flew through everything, not hiccuping once, answering all their questions. I explained that someone to keep up the status-quo wasn’t exactly what the organization needed, and explained my spiel about bridging the gap between the organization and IT, aligning the business priorities with IT, and helping drive top-line revenue. Same story, the recruiter told me I was far and away their first choice. 

I didn’t get the position. “The other person was a better fit,” the recruiter told me. 

Now, giving that organization the benefit of the doubt, maybe the other candidate did blow them away. Maybe they were charismatic, knowledgeable, and motivated. More likely, they probably fit within their budget better, but it doesn’t change my point. There’s a vicious cycle where a request for an IT executive is put out by an organization that really isn’t sure what they need, but rather, are choosing an IT executive based on a checklist that someone, somewhere put together. A few years later frustration grows (due to regression towards the mean), the IT department is looked at as a cost center that needs to be reduced (instead of helping drive top-line growth), and the IT executive is replaced, with the company looking again for someone with (likely) the same checklist for “skills” that will get the next person through the door. Wash, rinse, repeat.

This is why it’s so important for IT leaders, executives, and directors to change the conversation. To help organizations understand that the intangible is much more important than the tangible. It doesn’t matter how proficient your candidate is in your particular ERP system, if they can’t align themselves with the overall direction of the organization and adapt to help the organization grow. “We all wear different hats” is a common phrase (especially where I work). Organizations need to look for the person who can change hats quickly and wear lots of them, not the person who has a specific set of hats. 

When you’re looking for “the right fit”, you need to be looking for someone who can buy in and further the goals of the company. Someone who is hungry and eager to prove their worth through their intangible skills is much more valuable than that person who has worked with all the technologies you’ve put on your candidate checklist. 

Anything that can go wrong…

“Anything that can go wrong, will go wrong.” – Murphy’s law.

It’s been a couple weeks. The work has caught up to me, as it often does. Unplanned work is usually just that, unplanned work, and so it gets in the way of planned work – which derails everything in its path. These confessions are planned work – and so thus had been put on hold.

Most people are familiar with the adage, “Murphy’s Law” – which states that “Anything that can go wrong, will go wrong.” There are lots of articles about it, and you can read what Wikipedia has on it, here

You’ll get varying opinions on Mr. Murphy and his law, depending on who you speak to. I think most people in the executive/leadership space use it as motivation as to why one must be prepared… because if you’re prepared, there’s nothing for Murphy to make go wrong, because you’re prepared to handle it. 

Interestingly, the wikipedia article referenced above has this line:

According to Richard Dawkins, so-called laws like Murphy’s law and Sod’s law are nonsense because they require inanimate objects to have desires of their own, or else to react according to one’s own desires. Dawkins points out that a certain class of events may occur all the time, but are only noticed when they become a nuisance. He gives as an example aircraft noise interfering with filming. Aircraft are in the sky all the time, but are only taken note of when they cause a problem. This is a form of confirmation bias whereby the investigator seeks out evidence to confirm his already formed ideas, but does not look for evidence that contradicts them”

(taken from

Dawkins is right, and I don’t believe that inanimate objects have desires of their own. I also agree with the leadership teachings that you can avoid Murphy’s law by being prepared. That being said, how does that relate to IT? How can you adequately be prepared for anything, when the organization is usually focused on REDUCING IT costs, and planning in preparedness increases cost? 

I once told one of my newest employers that I wasn’t going to make any changes, do anything out of the ordinary, or make any recommendations until I had done at least a 30 day assessment of the environment in which I found myself. 

The company I had landed in had never brought IT in-house, and had found their IT costs ballooning utilizing external consultants. The company had been purchased by private equity a few years prior, and as most PE firms are, they were mostly concerned with EBITA. As the PE firm revamped the organization, brought in upper echelon talent, they began to look more closely at their expenditures, and IT was no small expenditure. The decision was made to bring IT in-house, to 1) lower costs, and 2) determine the state that the organization was in. There was no SME in house to be able to decipher whether or not the organization was realizing the value provided by the consultant or not. I digress.

Upon my arrival very little direction was given. Lower costs. Just lower them. There wasn’t even a true “IT” budget… more of an IT bucket in which all IT-related spending came from. I had my work cut out for me.

A trusted mentor recommended a book called, “The First 90 Days: Proven Strategies for Getting Up to Speed Faster and Smarter” by Michael Watkins. (Available HERE) and one of the things the book suggests is a 30 day assessment. Being in IT, having zero documentation, and not having anyone on staff to give me the lay of the land, as it were, a 30 day assessment (if not more) was just what the doctor ordered.

What I found was pretty shocking. Infrastructure was mostly, if not all EOSL (End of service life). There were (and still are) single points of failure in multiple places, and core infrastructure had reached its limit in terms of pure bandwidth. Frustration was high as the middle of the day rolled around and most people grinded to a halt as the core infrastructure struggled to keep up. Applications were fragmented, and operationally, end users had begun creating their own work-arounds to circumvent the system’s shortcomings.

My assessment was pretty clear – it all had to go – all of it. You can imagine the organization’s surprised faces when I told them the condition of their system. The board was even less pleased, becoming concerned mainly about the state of backups, and how this would affect the PE firm’s ability to sell the company. But most of all – I was supposed to CUT costs, not bring more to the table!

”A strong wind can take down our system” – is what I said. They took it as hyperbole.

I was being serious. 

So the question remains – how does one prepare for Murphy, if you aren’t given the resources to prepare? Proposals are all rejected, and the only approvals were for cost cutting?

Something has to break. 

It isn’t pretty.

You purchase insurance not because you intend on getting into a car accident, but because the cost of you getting into one without insurance far outweighs the cost of insurance. (Plus it’s mandatory, but that doesn’t work for the purposes of my illustration) 

Organizations have to realize that the cost of stretching the dollar and not having any sort of contingency planning in IT far outweighs the cost of the resources to prevent an outage. I used to work for a company that (reportedly) lost $40,000 an hour when the system is down. All it took was the system to go down for a couple hours (or 36 hours, once) for executive management to invest in IT so that it doesn’t happen again.

But now we’re in a conundrum. As an IT professional, you’re tasked with the health of the system, no matter what the situation you inherited is like. You can’t ever advocate for something to break… because it’s your head when it does.I had an executive tell me once, “This is why you’re well compensated, you have battle pay because we know that the system is less than ideal” 

I am not that well compensated, comparatively speaking.

As if somehow my level of compensation (it’s not THAT well compensated) determined the propensity of the system to go down or not. In a lot of ways, you get what you pay for… but paying someone more or less doesn’t change whether or not a system is in good or bad shape.

Fast forward – the organization I work for experienced an outage – one that stopped production for nearly a full day a couple weeks back. Thankfully I had purchased a last-ditch contingency, but the damage was done. A full day of working to get the organization back up and running, and executive management comes back with these types of questions…

And I quote:

“Was something [sic] that was to be done after- hours? In my experience, this is what is usually done to make sure all issues are taken care of and business efficiencies are not compromised.”


“I need to be told what is going on on a real time basis when the environment goes down so that we can explain it. I don’t like coming to you with a lot of questions because I don’t know what is going on”

Murphy had struck. And all I could do was sit there and say, “but I told you!”

In this blog I keep coming back to having to do a better job of bridging that gap between IT and the organization. Man is it difficult – I’ll tell you. Just based on the fact that when a system outage happens and the first response from executive management is (paraphrasing), “what were you doing when this happened?” “Could what you were doing have been done after hours?”

I posted this on LinkedIn a little while ago – Correlation does not imply causation.

All to say – is that sometimes something has to break before the company realizes that an investment has to be made in that area. It doesn’t preempt Murphy, but if it pushes the company towards making investments so that when things go wrong, they’re more prepared… then who am I to complain?

That is – unless they fire the IT guy. : |

So you balance, you hedge. You keep the system running as best as possible, while creating as much documentation as possible to prove that your environment is a shift away from coming down. You do your best to CYA, and hope that when things go down, they don’t come hunting for you.

It’d be so much easier if the organization would just listen – Investing in the immediate provides much more savings in the future.

A couple weeks back I got a call from an HR person looking for a reference for an old employee of mine. I, of course, gave a great reference and he got the job. (Why would you put someone on your reference list that wouldn’t give you a good review?) He sent a text thread to myself and some of our old teammates (we’ve all gone our separate ways now) informing us that he’s leaving the old stomping grounds, making him one of the last to leave. 

We all left under not the most auspicious of circumstances, but in retrospect you don’t realize how good things are until they’re done. My teammates and I from that job still converse pretty regularly, and we all remark how when things were good, they were really good, and we all were enjoying some of the most satisfying, most productive times of our lives. It’s amazing how a great environment makes you want to work.

Alas – all good things must come to an end. 

my old employee rides off into the sunset.

Strategic AND Tactical AND Operational AND Executional

Confession: I try to write on Wednesdays, but I didn’t yesterday. Truth be told, my professional life lately has been a lot like the photo above… no way that dude is going to catch all of those. I also can’t tell, but they look like billiard balls, which means that if any hit him on the way down, they’re going to hurt. What an apt pic. (I stole it from here)

I have my MBA, not because I was a glutton for punishment and wanted more schooling, but I believe (and had been told) that it was the gold star for positions I want (and continue to want) to attain in my career… and throughout my academic schooling (and my professional schooling, ie, the things I learn on the job every day) you hear the terms “strategic” and “tactical”. 

  • Strategy defines your long-term goals and how you’re planning to achieve them. In other words, your strategy gives you the path you need toward achieving your organization’s mission.
  • Tactics are much more concrete and are often oriented toward smaller steps and a shorter time frame along the way. They involve best practices, specific plans, resources, etc. They’re also called “initiatives.”

I was able to locate that explanation (and countless others) in a 5 second search… So there’s plenty of the “business school” explanations out there. Strangely enough, however, while there’s lots of info out there about being strategic versus tactical, there isn’t much (if anything… still searching) about being both strategic AND tactical. 

I don’t want to be so egotistical to think that the only sector that has to be both strategic and tactical simultaneously is IT, but I’m hard pressed to think of another function that has the same burden. You see, if you do your research, everything is about “drawing the line between strategic and tactical” or “the importance of being strategic vs. tactical”. It’s always “or”, never “and”.

This is one of my chief disappointments with academics. I worked for an academic institution for a while, and I observed a large disconnect between what academics taught things should be like, and what things were actually like. In George Bernard Shaw’s Maxims for Revolutionists, there is a quote that has become a social idiom, “He who can does; he who cannot, teaches.” It was a nice throwaway witticism that has become a way to denigrate teachers. I don’t think the quote is exceptionally fair, because I’ve had lots of brilliant-minded people who had “done”, but couldn’t teach for beans. However, Shaw isn’t wrong, either, because I’ve also had and met many academics who know how something is done by the book, but can’t put it into practice. I digress.

I posted on my LinkedIn a while back that it feels like the IT space is becoming more generalized again. When I started in this career, it was largely a generalist space. The IT guy did your audio/visual, your printers, your computers, your troubleshooting, your website, security, servers, networking, development, wrote code, and programmed your phones. Then it got super specialized… you had a security officer, dedicated helpdesk techs, a network admin, a storage admin, printer guy, web development guy, programmers, coders, telecom admins, etc… the IT department blew up in size. Now, perhaps with the adoption of the cloud (it’s just someone else’s computer), it’s getting more generalized again. You have less staff that can handle more as platforms are consolidated, data centers are virtualized and offloaded, and tools become single-panes of glass for management.  

I love this photo

But, it seems like while the expectation from IT is that less will be able to handle more, the business still expects results as if there are specialized assets in the field. 

So the question remains (and the entire point of this post)… how is one supposed to be Strategic AND Tactical? Expanding that conundrum… with little to no assets and no bandwidth, how are we to be Strategic and Tactical and Operational and Executional simultaneously? It’s very difficult, and I find myself navigating the latter two categories more often than not, perusing the Operational and Executional aspects of my day-to-day than in the former (Strategic and Tactical). I’ve heard it described so many ways, firefighting, being too busy fighting alligators to drain the swamp, etc… but it’s all true. 

This week I got chastised for neglecting my duties to be strategic and tactical, and I had let management reports, project plans, updates, and status reports slide to not-great levels. At first my response was a defensive one, thinking, “How on earth am I supposed to do all of this by myself?” My wife (and primary support) keeps telling me, “You can only do what you can do”, and I’ve tried very hard to take that to heart. It isn’t a platitude, but an accurate representation of the situation. If I’m to change the culture about IT, and IT folks in general, I need to start by level-setting expectations and creating boundaries. How can you be simultaneously strategic, tactical, operational, and executional? By doing them all in very small doses. I’m beginning to experiment with removing a majority of the billiard balls that I’m juggling. Last post I talked about denying the call (you can read it HERE), and surprisingly, it didn’t come back to haunt me. So I’m implementing changes, making moves. Here are three things that I’m trying to do daily:

  1. List. List. List. I come into work and I list out a few tasks that I need to accomplish that day. Make sure that there are tasks that encompass all the categories talked about above. A strategic task, a tactical task, operational tasks, and executional tasks.
  2. Transparency: In listing the “tasks” – I publish it so that people can see what it is I’m doing. Much of the disconnect comes from the business not understanding what I do, so there’s apprehension that I’m not accomplishing what it is I’m supposed to be accomplishing. Publishing the tasks where people can see them usually has gets the response of, “Oh…” when people look at it.
  3. Saying “No.” This is arguably the most difficult part, because it goes against everything I’m supposed to be doing, but it’s okay to say “No” to people. People adapt and can find their own solutions to things, which ends up being okay, at least in the interim. 

Being more organized and making sure I prioritize something from all of the buckets (strategic, tactical, operational, and executional) daily is checking the boxes. Is it sustainable? Probably not… there will be days where the fires are too great and I can’t do anything else. But the idea is, if I can spend more time on the first couple buckets, there will be less of the last two to have to deal with. Maybe then the fires can go from a 5-alarm forest fire to some dude who put too much lighter fluid on their barbeque. Or a candle I can blow out. One can dream.

Random Musings:

If there’s a picture to sum up what my experience has been lately, this is it. This was left (unopened, shrink-wrapped, new-in-box) on my desk this week. I really need to do a better job of explaining and showing the organization what it is I do for it. Meanwhile, I’ll be looking for something to do with all these 3.5” floppy disks.

My wonderful wife has started an amazon business – so if you’re in the market for what she has to sell, check it out! Her new brand (Mop Knot Hair) is taking off! See that link for more details! 

Another meeting that could have been an email…

“If you had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be ‘meetings.” 

Dave Barry

Meetings – a necessary tool or something that clogs up the productivity of the system? 

Honestly? I can’t tell you how many super productive meetings I’ve been a part of. I know I have been, and I know there has been some exciting stuff decided upon and definitely some necessary items that were resolved in meetings, I just can’t sit here and wax poetic on them. 

Why? Because for me in my career in IT, meetings have done nothing but suck the life and productivity out of a day. Early in my career (when I was a System Admin) I worked for an enterprise where the Director of IT held weekly staff meetings on Monday… from 8-11am. 3 painstakingly long hours where everyone around the room (the staff wasn’t that big, maybe 7 people) would detail out every aspect of every issue they were working on, from difficulties with end-users to waiting for parts, to whatever else they wanted to talk about. 

I worked in an enterprise where part of the staff was remote (across the country) so meetings were set to get on the same page. I didn’t mind those so much, as the IT manager at the time (then when I took over), kept them short and sweet. It built a camaraderie between the teams and was useful in towing the company line. 

I worked at a non-profit where everything was a meeting. Meetings to determine when the next meeting was. Meeting to create sub-committees to make more meetings. Meetings where the meeting attendees are all the same, but called different things. Meetings. Meetings. Meetings.

Today is no exception. I’m in the middle of a project and I’ve run into some issues. When asked for an update, I say something to the effect of, “We’ve run into some issues, we’re remediating and should be back on track shortly. Still on track for hitting my milestones”… And the response is, “let’s have a meeting this afternoon to discuss this”

Um. Wut?

Don’t get me wrong, it isn’t like the people asking have technical experience they can lend to the project to get it up and running. There’s no secret that I’m not thinking of that they might know. It’s a meeting for meeting’s sake. It’s code for, “I want more information so I can grill you on things that I think you may not have been doing”

Maybe it’s borne out of genuine care and want to help. Maybe they want to be productive. Maybe they feel they want to see where they can help. Or maybe…


Just maybe…

We’re conditioned as leaders to use meetings as a tool to control our subordinates. Maybe it’s just a forum to get others to listen to what we have to say. Maybe meetings should be more collaborative and informative, instead of impeding productivity. Maybe clear and concise guidelines should be established for meetings – instead of open ended meetings because that’s how we were all taught to administer. This was up in an old meeting room from one of the companies I worked for… and I thought it was relevant. 

1. Only hold a meeting if necessary. 

2. All meetings must have clear objectives. 

3. Invite a neutral facilitator to sensitive meetings. 

4. All meetings must have an agenda which includes:  

  • topics for discussion
  • presenter or discussion leader for each topic  
  • time allotment for each topic 

5. Meeting information needs to be circulated to everyone prior to the meeting. Make sure to include:  

  • meeting objectives  
  • meeting agenda  
  • location/date/time  
  • background information  
  • assigned items for preparation 

6. Meetings must start precisely on time so as not to punish those who are punctual. This also sets the stage for how serious you are about making the meeting effective. 

7. Meeting participants must:  

  • arrive on time  
  • be well-prepared  
  • be concise and to the point  
  • participate in a constructive manner 

8. Meeting notes must be recorded and made part of the company’s meeting information archives. 

9. The decisions made by the group must be documented. 

10. Assigned action items must be documented, and the host, or an appropriate participant, must be appointed to follow-up on the completion of all action items. 

11. Meeting effectiveness must be reviewed at the end of each meeting and suggested improvements applied to the next meeting.

Admittedly, I’m not great at meetings. I just don’t do them if I don’t find them necessary – so I’m always criticized for not meeting enough. I think it’s because I work best autonomously, and I understand that not everyone does that. I’m learning to find that balance, developing my leadership skills in the meeting space, but also making sure I’m not wasting anyone’s time. I’m still learning. Gotta go… I’ve got a meeting.

Knowing what you don’t know… (I TOLD YOU!)

I told you so!

I tend to be a competitive guy. BUT, as I’ve gotten older I realize you don’t know what you don’t know, and normally, people are okay with that. As a matter of fact, I consider it one of my strengths, the fact that I’m aware of my areas of weakness and have cultivated relationships and resources to help me in those areas. I can’t do something for you? Don’t fret, I’ve got a guy. However, a large part of the IT director gig is being a generalist.. A person who knows just enough about everything to orchestrate and execute on strategy and vision, someone who can be both strategic and tactical, dealing with today and 5 years from now. I have to have the ability to deal with IT items on a company P&L sheet, but also be able to roll my sleeves up and troubleshoot why a monitor won’t turn on. 

Thus, being an “IT guy” (or normal guy who’s proficient in IT) inherently develops a “know it all” attitude that isn’t always positively received by…. Well…. Anyone. In all my posts thus far, I’ve referenced Jimmy Fallon’s character on Saturday Night Live, “Nick Burns, your company’s computer guy”  As funny as the bits are (because they’re rooted in somewhat exaggerated truth), you hear a consistent theme, that the end users Nick Burns is servicing don’t like him. “I don’t like that guy,” one of them says. Even the theme song talks about how first (Nick) will fix your problem, and then he’s going to make fun of you. 

A “C” level person for a company I used to work for had a talk with me once, when I was younger in my career, and called me a “bull in a china shop.” He was attempting to coach me because I had rubbed people the wrong way. “You walk around here acting like you know everything, because you probably do know everything,” is what he told me. I was frustrated because my new colleagues (I was new to the company) just operated so much slower than I was accustomed to. I had just left a company where everything was urgent. They wanted it done, and they wanted it done yesterday. So this new company, where everything was a quarter speed (“Oh you can give me that budget proposal a week from Friday”) was jarring, in all the wrong ways. I tried to change the culture by example, kicking off and accomplishing so many “wins” (in my eyes) but all that did was cause my colleagues angst, because they didn’t operate my way. Looking back, it put me on the outside with those I worked with, and I never fully recovered from that position in that organization.

But what happens when we (the “IT guy” ) DO know something? When we warn and warn that if the company does not do something to remedy a problem, that the company is at risk? And then when the company ultimately decides not to go through with our recommendation (because IT is just too expensive) and pays a huge price for it when the thing we said was going to fail, does… who’s fault is it?

There was this article in the New York Times last week, about a former IT Director for a city in Florida who was fired after a ransomware attack caused the city to have to pay out almost half a million dollars ($460,000 or 42 bitcoin) in order to retrieve the key to decrypt the city’s files. It took the city weeks to recover from the attack, and shortly after the IT Director was fired. However, he claims he warned the city two years prior about this vulnerability, and proposed a system that would have kept the city from having to pay for the key. Basically, he identified a problem, proposed a solution, and the organization said no. There’s a lawsuit in progress… you can read about it here:

When Ransomware Cripples a City, Who’s to Blame? This I.T. Chief Is Fighting Back”

I’ve engaged in a couple discussions on this topic with a few colleagues. People definitely end up on different sides. “IT people” take the side of the Director, the “I told you so” approach. I tend to agree, to an extent. The problem was identified early on and a solution was already proposed. How is it the IT Director’s fault if the organization decides not to go with the proposal? And then, there is the business side of me that disagrees, because in any other walk of life, someone pays the price for a massive problem like that. Is the IT Director personally responsible for the ransomware attack? No. However, he’s responsible for the health of the system, and I can’t help but wonder what his actions were when the organization turned his proposal down. Did he throw his hands up in resignation? Did he refuse to research alternative methods? Did he provide other, possibly less expensive proposals that maybe could have mitigated some of the vulnerability? Obviously, the article doesn’t stipulate… but I know how I’ve reacted in the past, and I’m not super proud of it. We get “IT tunnel vision” because we “know what’s best.” 

I’ve been there, more times than I’d like to count. I do the research, come up with the charts, graphs, ROI, cost justification, risk analysis… and I’m lucky to get 6 minutes of hard pressed attention from executives. In the end, they deem the cost too great, don’t understand the risk, or don’t believe it could happen to them, or by the time I’m explaining the risk, their attention span has moved to another topic. 

It’s certainly easy to fall into the trap of throwing my hands up and saying, “oh well, not my circus, not my monkeys”… but that’s the catch. It IS my circus. When something goes wrong, no executive is going to say, “oh man, IT warned us about this two years ago”… heads are going to roll!

It’s also easy to say that the IT Director needs to do a better job pushing for the solutions. Someone told me, “ If you don’t want to be the IT director who ends up the scapegoat, you must insist that your employer deploy technology and training you know will protect your organization.” I laugh at that statement. What am I supposed to do? Stage a sit in? Lay in front of the CEO’s car until they approve my solution? I recently read, “The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win”  (it’s a good, unique look at the world of IT through a fictional story).. And at one point, the main character resigns over constant disagreements and unrealistic expectations from the CEO. They have it out, he resigns.. Then (SPOILER ALERT) the IT department suffers, the CEO sees the error in his ways, asks the IT Director (VP, in this story) to return, the VP gets his way, the day is eventually saved, and the VP of IT (main character) is eventually put on the track to be the next COO of the organization, due to his unique understanding of the workings of the enterprise. 

While this makes for a great book, most of us don’t have the luxury of resigning when we don’t get what we’ve asked for. Most of us have people who depend on our staying employed to live… and more realistic still, most CEOs won’t “see the error of their ways” and ask you to come back. If there’s anything I’ve learned in my days as a professional musician (yes, before IT I wanted to be a pro musician… another blog post, perhaps), it’s that if you cannot or will not do something, there’s always someone just as good as you or even better than you waiting in the wings who will do what you can’t or won’t do, for probably cheaper. 

So which side is right on the above example? The city, who fired their IT Director due to the massive breach and subsequent massive payout? Or the IT Director, who warned of the vulnerability two years prior, but got shut down? 

In my opinion? Both. And both are wrong too. It’s important to look at situations like the city IT Director, and the one I posted about my experience where I was a “bull in a china shop” and learn from them. Yes, there will always be problems (thank God for that, because that’s why I find myself employed), and I will likely still see many of my future proposals shut down. I believe IT has to do a better job of changing the “know it all” culture, and integrating themselves as normal operating pieces of the business, and businesses have to do a better job of raising the awareness, training, and competency of their integration with IT. 

The next time I’m asked to do a task that I find easy, I will refrain from calling it an ID-10-T problem (or that the issue resides between the spacebar and the back of the chair), and take the time to sit, kneecap to kneecap with someone, educate them, and in return, maybe be educated things I don’t know.. Because as much as I want to say, “I told you so…” I’m not doing my mission any favors that way. The more I do this, the more I find that this job is a small percentage technical, but a large percentage relational. A smile, nod, and genuine attention go a long way… maybe then I can start to change the culture, one “IT request” at a time… and in the end? I’m finding there’s just so much I don’t know…. And that’s okay.. Because it’ll make me better at my job.