My Jeep

Up until recently, I drove an old jeep. Like most aging vehicles, it had multiple problems and nuances, so every trip to the mechanic produced a long list of problems that needed be solved. In general, I only go to the mechanic when I know there is a problem, and usually I have a good idea of what the problem is.

Fortunately I have a good mechanic (shout out to the folks at J&S Automotive) who is able to identify the actual problem, what is causing the problem, issues related to the problem, as well as identify less urgent points that will eventually need repair. My mechanic is very good about explaining each problem and the need, urgency and cost of fixing the problem. This helps me spend money fixing whatever is actually wrong, as well as budget for future repairs.

Of course, I want to fix the urgent issues (such as a cracked fuel pump that spurts fuel onto the engine – true story!), but I tend to be more hesitant to fix things that I didn’t know were wrong. His expertise helps me ultimately save money and prepare for future expenses, rather than just hopping from one problem to the next without much forethought.

Debugging Software

In the world of software development support, we often deal with problems in much the same way: hopping from one problem to the next without much exploration or investigation that might reveal more systematic problems. This may be a product of low budgets and insufficient maintenance, but it may be just a symptom of inexperience by those maintaining our systems.

So in honor of my mechanic, I’m writing some high-level tips for debugging software.

1. Never Assume You Have Just One Problem

Software bugs can appear in a variety of forms and with a variety of symptoms. Unfortunately, we often get caught in our specialties or tend to be somewhat myopic in how we view a complex system.

For example, while working with one particular website we were finding that certain pages were taking up to a minute to display. After reworking some of the code we were able to bring a database query used by the page down to about 10 seconds, however the page was still taking about 30 seconds because we also had a problem with a piece of code that did image manipulation, which was taking 30 seconds to complete. Since the image manipulation worked simultaneously with the query we hadn’t noticed that it was also a problem, but it was now preventing us from reducing the entire query time to what we had originally hoped.

It wasn’t a big issue to fix both pieces, but for developers that work on fixed-bid projects or under tight budget restrictions, it is especially important double check that your one problem doesn’t turn out to be a whole bunch of problems once you lift the hood.

2. Sometimes You Can’t Find One Problem Until You Fix Another

In many cases, some problems are not revealed until an upstream problem or bottleneck is revealed (as somewhat discussed above). However, in this case it important to emphasize that for each "fix" that is put in place, you should check all critical paths related to that fix in case it reveals new issues.

For example, while working with one e-commerce system we found that it was "biased against the Irish" because the sales form rejected names with an apostrophe, such as O’Malley.

This was identified and fixed fairly easily, however before launching the new contact form; we made sure to check that it did not cause any other problems downstream. In doing so, we found a similar problem in the payment processing system, which prevented the billing system from processing names containing an apostrophe, even though orders would still ship.

By having the discipline and experience to follow our solution through all critical paths, we were able to identify and solve other potential problems before they went live, and provided a full solution to the client without accidentally creating an even bigger problem (or more disgruntled Irishmen).

3. Work in Teams

Teams are not possible in all situations, but it is very helpful to have someone to at least help brainstorm possible ideas and potential problems. I often find myself pulling in other experienced developers from around the office, and quickly explaining a system to them so that they can suggest ideas of approaches that I may not have considered.

Teams are also beneficial because each person has their own background and specialties. For example, as a SQL database developer, I notice that I tend to want to look at how things interact with the database, whereas others have other specialties which helps explore multiple options very quickly.

4. Gather Information and Follow Evidence

One of the benefits to software is that it can be broken down and bugs can be tracked. However, in many cases it can be fairly time consuming or challenging to figure out what the exact problem is, and so we make generic assumptions or have certain preconceptions about what may be causing the actual problem.

It is important to identify and track exactly what is happening because the assumed problem may only be a symptom of a larger issue. Break down problems as much as possible and gather as much information as you need in order to correctly identify the issue. Nothing is worse than spending a week rebuilding problem code only to find out that the problem still exists once you are done.

For example, while working with a wide-scale kiosk system we thought we had identified the issue that was causing some trouble, related to a software decoding issue. However as we investigated further and found the main issue was how information packets were encoded and decoded by the cell phone telecom that handled the data communication for the kiosks. This helped us identify the main issue that was causing the bulk of the problems, rather than spending time troubleshooting the related bugs, and in turn not actually solving the main problem.

Rather than assuming you know what the problem is, dig into the issue and see what else may be related so that you can provide a real solution instead of just a patch worked fix.

At Amadeus Consulting, we work on a lot of client systems across a lot of technology platforms. I know these are not the most tech filled solutions, but they are the basics that we often forget because of a lack of experience, or in some cases, too much experience and we assume we already know what the problem is.

In any case, if you have more specific questions, I’d be happy to try to answer them in the comments below, or if you have a larger support crisis we do offer software support and maintenance as part of our custom software development services.