Confessions of an IT Director

Fighting Alligators.

I’m back.  Happy New Year!

I didn’t really go anywhere, to be sure, but I was sort of on hiatus from this blog. Part being “too busy fighting alligators to drain the swamp” and partly a case of writer’s block, I’ve taken the last six weeks (mostly unintentionally) away from writing.

My writer’s block notwithstanding, the former reason (being too busy fighting alligators to drain the swamp) tends to be a common reasoning as to why people in this profession get stuck. IT, if not carefully cultivated, cared for, and organized can devolve into a constant and vicious cycle of firefighting. Get to work, figure out what’s “wrong” today – and then get to fixing it. Then go home, wash, rinse, repeat.

On one hand, I’m always grateful for technical issues. Being that I chose to be in a technical field, the mere existence of these issues ensures my gainful employment. Having a wife and three kids and a mortgage, gainful employment means even more. But it’s a double-edged sword, for several reasons. Of which I’ll detail:

The “What have you done for me lately” conundrum.

It’s been my experience that troubleshooting, technical resolution, crisis remediation, and generalized “firefighting” aren’t usually considered when senior leaders (especially “Non-IT” executives) review the job performance of an IT executive. It’s a little more applicable when an IT director reviews the job function of the IT service members, but even then it can be flawed. I used to (way back when) be a system administrator for a city municipal government, and every week the IT manager would print out and display on a bulletin board the ticket metrics of the IT team. 

The colored bars of the graph would indicate how many tickets were opened that week, how many were assigned to a specific technician, and how many tickets that technician closed. It became this competition, as members of the team would compare how many tickets each person closed that week. It’s been a while, but I want to say that the IT director even had incentives for the person who resolved the most amount of tickets.

Thing is – ticket counts are a horrible metric. Plug in a keyboard? Ticket. Reset a password? Ticket. Reimage 80 machines? Ticket. Some tickets could take seconds to resolve, while some could take days, months, etc. There were no guidelines as to what did and didn’t constitute a ticket, what constituted a project, etc. 

As time went on, members of the service team would make everything a ticket. Move a few boxes of computers. Ticket. All that mattered was the count – that’s how senior leadership was judging.

Sidebar – years later when I had moved into the managed service space, the problem as to what is a “ticket” versus what isn’t reared its ugly head again. The company I was with sold managed service contracts to their clients, but what was covered under those agreements was ambiguous, at best. Even I, who worked for the company, didn’t always understand it. A client would want something done, but it would constitute a project and thus be subject to a different price structure and not covered under the contract. Needless to say, that company lost a lot of business.

All to say, tickets are never weighted the same, so they make for a bad metric.

I digress.

As an IT Director, or any other part of the IT leadership team the onus isn’t placed on remediation, for whatever reason. It’s just sort of expected that it’s part of the job, but it doesn’t really count as your accomplishments for the period. When asked what I’m up to, exeuctive leadership wants to know progress on tasks and projects that take the company forward, save the company money, or improve something that needed to be improved. Even though remediation is paramount, it isn’t looked upon favorably. You’re supposed to be taking us into the future, not keeping alive where we’ve been.

What have you done for me lately?

The “Perception is reality” problem:

Despite all the trendy talk at every conference or gathering of like-minded individuals about the mobile workforce and the ability to do work from anywhere, the truth of the matter is that many executives can only gauge the work by what they see. Meaning, if they don’t see you, the automatic assumption is that you aren’t working. 

Most IT people in my position can probably relate to this. IT isn’t a 9-5 job – especially at this level. Computers don’t sleep, the systems that run on these computers don’t magically go home at 5pm and come back in the morning, so despite what your hours are, IT is 24/7/365. This is a particularly difficult point for my wife, though she’s learned to accept it better in recent years. I’ve been contacted for (seemingly) stupid reasons in the middle of the night, or on vacation, or during a movie, show, event, church service, or what have you, more times than I can probably count. 

So the conundrum here is that there’s an expectation that as an IT Executive, you’ll be ready at a moment’s notice to execute on some major issue, but subsequently are judged on the number of hours you’re in the office. Recently I witnessed what can only be described as “a competition between rivals to determine superiority, predominance, or leadership” between two VPs at an organization. One arrives at the office prior to 6am, and leaves near 6pm, and the other arrives near 10am, but doesn’t leave the office until nearly 10pm. The first was criticizing the second for the amount of time he had to wait for the second to come in, while the second was saying things like, “You may get here early, but I close this place up by myself every day.” I, for one, couldn’t imagine what it is that they had to do in the office that required 12 hour days, every day, despite their 4 hour gap between arrivals.

Later the VP that my department reports up to made a comment about my (the IT director) schedule and how the leadership was “old school”. It wasn’t worth the chest-beating, so I just said nothing and nodded, but the truth of the matter is, is that if I, as the IT director, had to spend 12 hours in the office daily, then something was seriously wrong, and I would NOT be doing my job.

Just because you don’t see me doesn’t mean I’m not working. 

This is especially frustrating, because the expectation (even if not stated) is that time be spent in the office. But also the expectation is that IT will be available any time someone else is working (whether it be a Sunday evening, 2:00am, or any other “inconvenient time”) But like the earlier example about ticket counts as a bad metric, amount of time spent in the office is a bad KPI as well. I can guarantee that I am several times more effective, efficient, and productive working remotely outside the office than I am sitting at my desk. Why? For the entire reason of this post, because when you’re in the office you tend to do more “fighting alligators” than “draining the swamp.” 

But… many times, unfortunately, perception is reality. Even if it’s wrong.

Truth be told, IT more than anything else has become more about “that laptop life”, and organizations need to realize that chaining your IT department to their desks is less effective than letting them work remotely. 

The IT organization I ran for 4 years a few years back called them “drive bys”, but I’m sure every IT department experiences something of the same phenomenon. As members of the organization see the IT team (any one of them) walking around, they’re quickly reminded of an issue, no matter how great or small, that they forgot to report through proper channels, and so they grab the IT person to tell them about it.

I had a former IT manager tell me he took his lunch in his car every day because if he took his lunch in the employee lunchroom, he’d be inundated with questions revolving the system, a ticket status, a technical problem someone was having, or some sort of glitch. I quickly realized all the IT team members did the same thing, or attempted to take refuge offsite, for the same reason.

The notion that perception is reality can be beneficial to IT in some cases – for instance, I always told my employees that when they walked to address a user’s issue, that they walk with speed and urgency, because lackadaisical walking gave the perception that the IT employee was not doing anything. However, using time spent in the office to make a judgement about the effectiveness of an IT executive is not a good perception. Like Obi-Wan says – “Your eyes can deceive you, don’t trust them”

The Law of Diminishing Returns

Issue remediation can buy you some short-term capital, at the cost of long-term investment. While some senior leadership can excuse a period of review filled with remediation, repeated periods of work where all that was done is issue management and remediation eventually works against the IT executive, and IT team in general.

Eventually, the organization will look at the constant problems and make the correlation that the IT executive (and by proxy, the rest of the IT team) isn’t cutting it, because the company’s problems persist and don’t seem to be getting better. 

I don’t have any statistics to back this fact, but this is why IT executives tend to only be in place a few years. Like a coach or GM of a sports team, without consistent forward progress, the underlying thought is that the IT executive has lost their effectiveness, and “new blood” is brought in. Occupational hazard. 

I think that’s why, at the end of the day, I decided to start this blog. I made the comment the other day at my work that everyone speaks English but nobody is speaking the same language. (I think I stole that from a “Calvin and Hobbes” comic strip) The same can be true for organizations and IT. I keep calling it the “chasm” between these two entities (that really should be the same), but it’s the same no matter where I go. I want to serve to bridge that gap, to help organizations understand the world of IT, and help IT relate more relevantly to the organization. Right now, I’m doing it through this blog – who knows what form this mission will take going forward.

I do know one thing, in order to keep my dream of making the “IT vs. Organization” situation a better, more productive, more seamless relationship, I need to stop being so busy fighting the alligators and start draining the swamp.

What do you do here?

What Do you do?

I posed the question on LinkedIn the other day and for my 16 twitter followers (side note: I have GOT to get more followers)… “What is valuable (IT) information to bring to a monthly MOR (Management by Objectives and Results) meeting? What kind of metrics/KPIs do you think the senior management wants to see?”

That was my sort of “LinkedIn” version of wondering aloud what sort of metrics executive management wants to see. 

Let me explain

Anytime I can throw a “Princess Bride” reference in, I’m going to.

My current place of employment is owned by private equity. So every month there is a (monthly) internal, and external MOR meeting (Management by Objectives and Results). The internal one is with the CEO and Leadership, the VPs and Directors of their respective departments. The meeting itself is a good 2 to 3 hours (depending on the month) long. Each department is expected to contribute a few slides to a PowerPoint deck that sum up objectives and results for their department. After the internal MOR, the top end leadership (CEO, VPs) take the deck, make whatever changes, and deliver a similar deck to the Board of Directors for the PE company. Usually the operational stuff is considered superfluous, and the BOD is really only interested in the financials… sales, forecast info, EBITA, etc…. Their primary focus is how profitable the company is, because when they decide to sell the company, that’s the marker. “We’re flipping a house” is a common phrase said around here.

The presentation deck for the internal MOR is usually 130 or so slides long, with the last 15 or so slides being appendix slides. The IT department slides (my slides) come in around slide 100. The final department prior to the appendix slides showing all the financial datasets, etc. 

So, a recap. 15 to 20 people have been sitting in a small-ish office for nearly 3 hours, and it’s my turn to present. When I was first briefed on this particular exercise, I was told to prepare 4 or 5 slides. That was pretty much it… no real direction on what they wanted to see, because I honestly don’t know that they knew what they wanted to see. Everyone is a little stir crazy due to being cooped up in the office listening to data going on and on and on… and then there’s IT, a department people don’t really understand to begin with. What DO you do?

Therein lies my problem. Get too technical, and people don’t understand and tune out. Nobody wants to know the issues you had in switching the routing from the old firewall to the new firewall, or the details in the open case with Cisco because one of the switches wasn’t passing traffic, or how this old version of one of the ERP modules isn’t compatible with the new version of the ERP, and so you’re in negotiation with the vendor to provide an updated file. It’s the teacher from Charlie Brown.

The other way isn’t great either. Don’t give any real info, and people start asking, “What about my issue from…. _______?” and, better yet, people start reporting IT issues to me that I wasn’t aware of (which invariably precedes the question, “Why didn’t you know about this” Cause it wasn’t reported.) Then I’m talking about a purchasing clerk’s computer and when it’s up for refresh, or why the credit card processor was slow at 11am, or why the new sales guy doesn’t need 64gb of RAM.

I gave less information once, and one of the bosses called me into his office afterwards and told me I missed my opportunity to “showcase my wins”. “Wins” are subjective. What is it he considered wins?

Worse yet are the PowerPoint slides – best “data” I can give is ticket counts… opened, closed, categories… etc. Not a great metric, because if you didn’t know… a ticket can constitute 15 minutes or 15 hours, depending on what it is. Some tickets are closed immediately after opening them, and some stay open for weeks with a variety of variables, priority, type, resources needed… etc. 

But… it’s the best I have to offer right now. There are multitudes of tools (for $$$) out there that can deliver automated reports on system health, bandwidth usage, top talkers, and potential trouble spots… but… “We’re flipping a house”. Granite countertops, but galvanized piping. 

So I strive to stay on that narrow road. Ticket counts shows the stakeholders that IT is busy, but they don’t really show what IT is doing. Don’t give too much attention so I don’t put everyone to sleep, but give enough so that people feel that IT is doing something. 

But, at the end of the day – I don’t have an answer. If someone out there has found a winning formula, a KPI that energizes the organization and gets executive management engaged into IT direction, then please forward that my way. Until then, I’ll keep throwing stuff on the screen showing that IT is doing something, even if they don’t understand WHAT it is.

Be excellent to each other

It’s my birthday today.

Coincidentally, it’s my son’s birthday too. He was born 31 years exactly after me. Everyone says that one day we’re going to find that super cool. I’m sure as we get older and enjoy the same things, that will be the case. Right now, he’s more about how many treats he can have… me, I’m sorta about that too, but more because I’m trying to figure out what will fit in my macros, not how many I can have without getting in trouble. I joke – it’s great. He’s into LEGOs and Star Wars and video games… and lets be honest, I’m into those things too.

I had a medical thing this week, so I was out of work for a few days. I wish I could tell you that things marched right along without me, but that isn’t the case. I’m not surprised, though. IT doesn’t sleep.. Not in the organizational sense. If computers, servers, services, websites, security all would shut down at 5pm when the business shut down, and then was able to be turned on each morning, that’d make my life easier… but systems don’t take a break. I guess that means the IT department doesn’t take a break either. 

I can never tell if my absence (or IT’s absence) from the organization results in a, “oh my gosh where is he we need him” response, or a “oh my gosh this is so annoying he’s never around when we need him” response. Probably a little of both. As expected, in my first day back to work, the work was piled deep and wide. It’s a tenuous relationship, the one IT has with the business… I guess that’s the entire point of this writing journey, to explore and help define that relationship. 

I admit I’m not good at the interpersonal stuff that comes along with working in an office. I don’t do watercooler talk, don’t really do idle chit-chat, and due to being burned (severely) by friendships that crossed the work/personal divide, I keep my coworkers as just that, coworkers. That being said, I’m told if you want things to be different, you have to try different things…. So today I’m shooting the breeze with people I know. Joking about people breaking their stuff, joking (sorta) about being jealous that they miss their old IT guy (a consultant), giving people grief that their current ask is in danger of lowering in the queue if they give me grief – office banter. It’s difficult, because my personality is one that I want to get in, do my work, and get out. I don’t want to wax poetic for 48 minutes about comic-con or your fantasy football picks, at least not at work. But (there’s always a but)… I’ve learned my lesson (harshly) several times about how important it is to the people we (IT) service to be personal and friendly with them. IT is a service industry, after all. Users want to feel comfortable, so relating to them on some level helps the IT/organization relationship. It actually improves it, because there’s such a stigma surrounding what it is we do, working on the relationship first makes you appear… dare I say it… human.

I actually have those “Ctrl Alt Del” cups

It’s this humanizing of the “IT” guy (IT Director, in my case, thank you very much) that surprisingly allows for a lot of leeway from the business. That friendliness paves the way for understanding, patience, and even statements like, “I know you’re just so busy, I don’t want to bother you with…” which is a far cry from, “It’s broken, you need to fix it” 

So why is it so difficult to find that balance? That crossroads between professionalism and interpersonal relationships? I’m not the first IT guy to be labeled as anti-social… I’ve already given multiple examples of how in popular culture the IT guy is tagged as a condescending know-it-all who enjoys making people feel stupid. Someone who has little patience for people who don’t have their technical knowledge, and makes sure that they tell people they have no time for their crap. 

I wish I could tell you. Maybe it’s part of the gig. Maybe it’s a deeply rooted psychological trait that is attributed to IT people. Maybe we are inundated with so much, we feel we have no time for the relational. Whatever the case may be, IT people are much to blame why there is such a chasm between the business and IT… and that aint it, chief. 

I had lots of anxiety driving into work today, because after being out for 4 days I dreaded the amount of work that was about to be heaved my way. My shields were up and I geared up for a fight. I dreaded amount of people lining up at my cubicle to explain to me their issue (one user used their cell phone to video how slow their processes were running on their machine, as if I didn’t believe them)… but I can tell you, the Proverb is true. A gentle answer turns away wrath, but a harsh word stirs up anger. It’s amazing what a smile, some small talk, some laughs, and a little bit of self-deprecation does. It goes a long way towards disarming your would-be work assailants and putting them in a position where not only do they give you understanding, but they become advocates on your behalf.

I write this primarily for me… but on the off chance that there’s another IT dude or dudette out there googling “IT confessions” because you’re frustrated with your environment and you want to see who else out there can relate to your predicament. You’re overworked and underpaid, under appreciated and under resourced… I don’t have the answers… but I do know that we’re all in the same boat. One thing I need to remember myself is the same advice I give here.. Be nice. Or rather – listen to Abraham Lincoln:

Abraham Lincoln: Fourscore and… 7 minutes ago, we, your forefathers, were brought forth upon a most excellent adventure, conceived by our new friends: Bill and Ted. These two great gentlemen are dedicated to a proposition, which was true in my time, just as it’s true today. Be excellent to each other… and… PARTY ON, DUDES!

Knowing what you don’t know… (I TOLD YOU!)

I told you so!

I tend to be a competitive guy. BUT, as I’ve gotten older I realize you don’t know what you don’t know, and normally, people are okay with that. As a matter of fact, I consider it one of my strengths, the fact that I’m aware of my areas of weakness and have cultivated relationships and resources to help me in those areas. I can’t do something for you? Don’t fret, I’ve got a guy. However, a large part of the IT director gig is being a generalist.. A person who knows just enough about everything to orchestrate and execute on strategy and vision, someone who can be both strategic and tactical, dealing with today and 5 years from now. I have to have the ability to deal with IT items on a company P&L sheet, but also be able to roll my sleeves up and troubleshoot why a monitor won’t turn on. 

Thus, being an “IT guy” (or normal guy who’s proficient in IT) inherently develops a “know it all” attitude that isn’t always positively received by…. Well…. Anyone. In all my posts thus far, I’ve referenced Jimmy Fallon’s character on Saturday Night Live, “Nick Burns, your company’s computer guy”  As funny as the bits are (because they’re rooted in somewhat exaggerated truth), you hear a consistent theme, that the end users Nick Burns is servicing don’t like him. “I don’t like that guy,” one of them says. Even the theme song talks about how first (Nick) will fix your problem, and then he’s going to make fun of you. 

A “C” level person for a company I used to work for had a talk with me once, when I was younger in my career, and called me a “bull in a china shop.” He was attempting to coach me because I had rubbed people the wrong way. “You walk around here acting like you know everything, because you probably do know everything,” is what he told me. I was frustrated because my new colleagues (I was new to the company) just operated so much slower than I was accustomed to. I had just left a company where everything was urgent. They wanted it done, and they wanted it done yesterday. So this new company, where everything was a quarter speed (“Oh you can give me that budget proposal a week from Friday”) was jarring, in all the wrong ways. I tried to change the culture by example, kicking off and accomplishing so many “wins” (in my eyes) but all that did was cause my colleagues angst, because they didn’t operate my way. Looking back, it put me on the outside with those I worked with, and I never fully recovered from that position in that organization.

But what happens when we (the “IT guy” ) DO know something? When we warn and warn that if the company does not do something to remedy a problem, that the company is at risk? And then when the company ultimately decides not to go through with our recommendation (because IT is just too expensive) and pays a huge price for it when the thing we said was going to fail, does… who’s fault is it?

There was this article in the New York Times last week, about a former IT Director for a city in Florida who was fired after a ransomware attack caused the city to have to pay out almost half a million dollars ($460,000 or 42 bitcoin) in order to retrieve the key to decrypt the city’s files. It took the city weeks to recover from the attack, and shortly after the IT Director was fired. However, he claims he warned the city two years prior about this vulnerability, and proposed a system that would have kept the city from having to pay for the key. Basically, he identified a problem, proposed a solution, and the organization said no. There’s a lawsuit in progress… you can read about it here:

When Ransomware Cripples a City, Who’s to Blame? This I.T. Chief Is Fighting Back”

I’ve engaged in a couple discussions on this topic with a few colleagues. People definitely end up on different sides. “IT people” take the side of the Director, the “I told you so” approach. I tend to agree, to an extent. The problem was identified early on and a solution was already proposed. How is it the IT Director’s fault if the organization decides not to go with the proposal? And then, there is the business side of me that disagrees, because in any other walk of life, someone pays the price for a massive problem like that. Is the IT Director personally responsible for the ransomware attack? No. However, he’s responsible for the health of the system, and I can’t help but wonder what his actions were when the organization turned his proposal down. Did he throw his hands up in resignation? Did he refuse to research alternative methods? Did he provide other, possibly less expensive proposals that maybe could have mitigated some of the vulnerability? Obviously, the article doesn’t stipulate… but I know how I’ve reacted in the past, and I’m not super proud of it. We get “IT tunnel vision” because we “know what’s best.” 

I’ve been there, more times than I’d like to count. I do the research, come up with the charts, graphs, ROI, cost justification, risk analysis… and I’m lucky to get 6 minutes of hard pressed attention from executives. In the end, they deem the cost too great, don’t understand the risk, or don’t believe it could happen to them, or by the time I’m explaining the risk, their attention span has moved to another topic. 

It’s certainly easy to fall into the trap of throwing my hands up and saying, “oh well, not my circus, not my monkeys”… but that’s the catch. It IS my circus. When something goes wrong, no executive is going to say, “oh man, IT warned us about this two years ago”… heads are going to roll!

It’s also easy to say that the IT Director needs to do a better job pushing for the solutions. Someone told me, “ If you don’t want to be the IT director who ends up the scapegoat, you must insist that your employer deploy technology and training you know will protect your organization.” I laugh at that statement. What am I supposed to do? Stage a sit in? Lay in front of the CEO’s car until they approve my solution? I recently read, “The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win”  (it’s a good, unique look at the world of IT through a fictional story).. And at one point, the main character resigns over constant disagreements and unrealistic expectations from the CEO. They have it out, he resigns.. Then (SPOILER ALERT) the IT department suffers, the CEO sees the error in his ways, asks the IT Director (VP, in this story) to return, the VP gets his way, the day is eventually saved, and the VP of IT (main character) is eventually put on the track to be the next COO of the organization, due to his unique understanding of the workings of the enterprise. 

While this makes for a great book, most of us don’t have the luxury of resigning when we don’t get what we’ve asked for. Most of us have people who depend on our staying employed to live… and more realistic still, most CEOs won’t “see the error of their ways” and ask you to come back. If there’s anything I’ve learned in my days as a professional musician (yes, before IT I wanted to be a pro musician… another blog post, perhaps), it’s that if you cannot or will not do something, there’s always someone just as good as you or even better than you waiting in the wings who will do what you can’t or won’t do, for probably cheaper. 

So which side is right on the above example? The city, who fired their IT Director due to the massive breach and subsequent massive payout? Or the IT Director, who warned of the vulnerability two years prior, but got shut down? 

In my opinion? Both. And both are wrong too. It’s important to look at situations like the city IT Director, and the one I posted about my experience where I was a “bull in a china shop” and learn from them. Yes, there will always be problems (thank God for that, because that’s why I find myself employed), and I will likely still see many of my future proposals shut down. I believe IT has to do a better job of changing the “know it all” culture, and integrating themselves as normal operating pieces of the business, and businesses have to do a better job of raising the awareness, training, and competency of their integration with IT. 

The next time I’m asked to do a task that I find easy, I will refrain from calling it an ID-10-T problem (or that the issue resides between the spacebar and the back of the chair), and take the time to sit, kneecap to kneecap with someone, educate them, and in return, maybe be educated things I don’t know.. Because as much as I want to say, “I told you so…” I’m not doing my mission any favors that way. The more I do this, the more I find that this job is a small percentage technical, but a large percentage relational. A smile, nod, and genuine attention go a long way… maybe then I can start to change the culture, one “IT request” at a time… and in the end? I’m finding there’s just so much I don’t know…. And that’s okay.. Because it’ll make me better at my job.