I was pulled down a rabbit hole recently in trying to solve a technical issue. I was working in an unfamiliar environment with limited access to the administrators of the environment. I found some misconfigurations on the network that solved some minor issues. I suspected that there were additional networks issues, so I pulled in some more senior and experienced people to assist me with troubleshooting the problem. We all were troubleshooting the network, and an additional not-so-minor technical misconfiguration was again found on the network.
Yet the network was not the cause for the problem that I was troubleshooting. Despite network misconfigurations having been found I was troubleshooting the wrong components. I made a foolish mistake, because I was troubleshooting based on what I knew how to do. I should have taken a step back and looked at the bigger picture. I was doing tasks that I knew were possible, but unfortunately they were tasks that were not needed.
Finding minor misconfigurations on the network that I could resolve just reinforced my faulty logic. I kept chasing that ghost of a network issue, and I developed tunnel vision. The worst part of all was that once I realized that the network was not where the problem resided (with the help of a colleague) I then learned that the real problem was one that I was perfectly capable of solving on my own.
I knew how to deploy, configure, and test the needed solution. What I lacked was the knowledge and experience to know when I was supposed to apply the needed solution.
What I have described is actually very common in the IT profession. I have made this same mistake before, and I will make this same mistake again (although less often as I gain in experience). I have also helped others realize when they were making this same mistake when troubleshooting other matters, just as others have helped me do the same. Based upon my experience there are only three ways to avoid making this mistake:
- Deep technical training. Unfortunately this is what many organizations fail to provide for various reasons. Sometimes it is a financial issue, and other times it is a lack of good training resources. Yet nothing is better for avoiding troubleshooting rabbit holes than teaching someone how to spot a rabbit hole in the first place.
- Detailed troubleshooting tasks tailored to the technology. If you cannot provide deep technical training on a technology, then you should have someone who truly understands how the technology works develop a solid troubleshooting methodology. There should be some form of documentation that says “Do these X number of tasks exactly as shown. If the results are A, then it is unlikely that the problem resides with B.” In other words, if you cannot give your IT staff the book (deep technical training) you better provide them with the CliffsNotes version (troubleshooting tasks). Furthermore, everyone on your IT staff has to know where to find the documented tasks and that there is an expectation that the tasks are completed with every issue before troubleshooting is escalated.
- Experience. Nothing can beat real world experience! Experience is the ultimate teacher, but for a business you must be ready to incur the high expense of people having to learn through experience. Remember that without deep technical training and/or detailed troubleshooting tasks you have made it a requirement of your IT staff to make mistakes. That is what IT experience is made up of – a collection of real world mistakes and the lessons learned as a result of making those mistakes. If you blame your IT staff for having made mistakes when troubleshooting, but you did not provide your IT staff with deep technical training and/or detailed troubleshooting tasks then the real problem lies outside of IT.
So the next time you find yourself digging deep into a technical issue but not being able to solve the problem, take a step back and ask yourself if you are going down a rabbit hole chasing a ghost. Have you had the deep technical training needed to work with the technology? Do you have a set of detailed troubleshooting tasks that you can follow in order to eliminate elements of the infrastructure from the troubleshooting process? Do you have enough experience with the technology in order to troubleshoot the issue?
Taking that step back may be what keeps you out of a rabbit hole in the first place. The willingness to re-evaluate the situation combined with an openness for receiving help from others will make you a better IT professional regardless of the technology that you are working with. The best part of all is that developing these habits requires nothing more than personal commitment to the practice of these habits, so start applying them right now and you will avoid the rabbit hole of a bad IT career.