Apr 3
Debugging and Testing
Debugging and testing are critical skills every developer should have. But how do you learn them?
· Can you learn to debug?
· Different Levels of Debugging
∘ Debugging 101: Local debugging
∘ Debugging 201: Remote Debugging
∘ Debugging 301: Debugging in Prod
· Debugging and Testing
· Conclusion
· References
Debugging is “the process of finding and resolving bugs (defects or problems that prevent correct operation) within computer programs, software, or systems.”[from the wiki definition] A recent study reports that a typical developer spends about 15–60% of their in debugging.[1] Other than coding itself, debugging is one of the most important skills one should learn to be a better developer. The purpose of this article is to give you an overview of the debugging process so that you are better equipped to find your way in your debug learning journey.
Can you learn to debug?
Considering how much time developers spend debugging, it is strange that there is no school or class that teaches you how to debug. I can think of two reasons for the lack of formal education in debugging.
First, debugging is a necessary process of coding. If you write a program, you are going to introduce bugs and you need to debug them to make any progress. However, unlike algorithms or data structures that we can learn by studying, debugging is a process that is difficult to formalize and teach.
Second, the process of debugging is hard to formalize because debugging is mostly about problem-solving that comes at all levels of programming. From the moment that you write your first line of the ‘Hello world!’ code, you have to investigate why things do not work as expected. Bugs can be present in every aspect of coding and so is the need to find where the problems are.
The difficulty with formalization does not mean that debugging is an innate skill that cannot be learned later, however. No child is born with great tennis skills. With practice and training, one can learn the sport of tennis. Similarly, I believe that debugging can be trained and improved with proper methodology and practice.
Different Levels of Debugging
Debugging can be divided into three different levels in terms of difficulty.
Debugging 101: Local debugging
The easiest way to debug is debugging in your local computer environment. Typical symptoms are wrong outputs, logic problems, crashes, infinite loops, etc. All modern IDE (Integrated Development Environment) comes with easy ways to start a program in debug mode, set breakpoints, step through code, etc.
In a lot of cases, you do not even need to put the program in a debugger. In some cases, you cannot even use the debugger. Syntax mistakes, compile errors, and merge conflicts are all errors that you have to debug outside the debugger with logical approaches and good problem-solving skills.
The environment is your own development machine which means that you have full control. In the vast majority of cases, bugs can be reproed (meaning ‘reproduced’) almost 100% so debugging is easier.
Debugging 201: Remote Debugging
The next level of debugging is what I call ‘remote debugging.’ You as a developer compiled and tested your program successfully and pushed the code so that other people can start testing it within the company. Soon, bugs are reported and they are reproed fairly consistently in other people’s machines. However, when you want to debug them on your own machine, you just cannot repro it. You just hit ‘it works on my machine’ bugs.
Typical causes of ‘remote’ bugs are due to integration errors. Such errors come in different forms: e.g., wrong data, wrong versions, wrong environment, or wrong timing.
- Wrong data: input and output formats might mismatch or data are truncated or missing.
- Wrong versions: dependencies come as libraries (static or runtime ) or APIs. Static dependencies are easy to spot and debug. API version mismatches are rare and any breaking changes are announced early. The most difficult to debug are runtime dependencies. In older Microsoft Windows systems, the term ‘DLL Hell’ sent shivers down the spine of any Windows developer. DLLs are dynamically linked libraries that are shared runtime libraries specific to Windows.
- Wrong timing: Asynchronous calls or multithreaded programs create nasty bugs that have to do with timing. Bugs occur very infrequently and are very hard to repro and debug.
- Wrong environment: The environment problem is probably the most frequent issue you will encounter in a remote environment. Some bugs can be reproed only in a certain environment: e.g., a specific browser, certain hardware types, OS versions, etc.
Other than remote debugging through IDE, good problem-solving skills and dogged persistence and patience are required for successful debugging in a remote debugging environment. The best way to minimize integration issues is ‘containment.’ Container solutions like Docker or Kubernetes or Virtual Machines come with some overhead but you trade in hours of debugging and agony and anxiety in return.
Debugging 301: Debugging in Prod
The graduate-level of debugging is debugging in Prod: i.e., in actual Production. When the product is released ‘in the wild,’ all kinds of unexpected and weird issues are reported back.
Typical symptoms include crash reports, runtime errors, reduced performance issues, etc. It used to be every Windows user’s nightmare to encounter a blue screen of death.
Common causes include environmental issues, user errors, and unexpected sequences or data.
Environment issues can be more serious depending on the OS type. Windows OS, for instance, tends to have more crashes and blue screens not because the OS code is bad but because the hardware environment is more diverse than Mac OS. Due to the differences in business models, Apple has full control of the hardware ecosystem because it makes both the software (Mac OS) and the hardware. Microsoft, on the other hand, opted to focus only on the OS and let hardware vendors come up with their own hardware design and specs. The diverse hardware ecosystem might be good for business but a jungle for testing and quality control.
Every developer has biases one way or the other. Even when the spec was written to the teeth, the implementation might be different from an actual user’s expectations. The issues might not be reported as crashes but they might be more severe than functional bugs.
No matter how many unit and integration tests were written, there is simply no way that you can exhaust all user scenarios or types of data in tests. For this reason, it is important to test and monitor in production and quickly iterate debugging and fixing issues as they come in. The agile methodology is invented to accommodate the fast iteration cycle of the service-oriented software development life cycle.
Debugging in “the wild” is the most difficult level of debugging. Here are some best practices that can help to debug in Prod easier.
- Logging: Since remote debugging is next to impossible, logging is the next best tool for debugging. Sprinkle logging statements throughout your program but judiciously so that you can trace the program easily when bugs are reported. The recommendation is to keep at least a few months of log data. Raw log files are also extracted and transformed using an ELK Stack tool for better searching and debugging.
- Crash reports: Crash reports allow for collecting more debug information useful to the developers. Windows and Mac both have a crash reporting functionality and ask for user consent before sending it. The crash report should include the hardware specs, date and time, OS type and version info as well as the call stack.
- Telemetry: Telemetry refers to collected data from remote machines. User actions like clickstreams can reveal valuable usage patterns that can guide the feature team for the next iteration of development and testing.
Debugging and Testing
An observant reader might have noticed that different levels of debugging roughly correspond to different levels of testing.
Tests are usually explained in terms of three different groups in a pyramid: unit, integration, and end-to-end testing.
They form a pyramid because there should be more tests as you move down the pyramid: i.e., there should be more integration tests than end-to-end tests and far more unit tests than integration tests. The idea is to cover as many of testing areas as possible in a more reliable environment. The most reliable environment is the development or the build machine and thus you want to write a lot of unit tests so that you spend less time on integration and end-to-end testing.
In an ideal world, your unit tests cover all logic problems in your program before it is integrated with other components. In that case, integration tests can just focus on testing integration issues and thus do not need to be extensive. Even less should be your end-to-end tests which should be testing only end-user scenarios.
When tests are written and automated this way, debugging is much easier and faster because the difficulty level of debugging (remote or production debugging) is significantly reduced. Most of your debugging can be done in a local environment where you have full access and control.
The anti-pattern of the testing pyramid is an upside-down pyramid in a cone shape.
The flakiness and the cost of testing increase as well as the time you spend in debugging.
Conclusion
We started by asking for ways to debug better. Ironically, the conclusion I want to draw is that the best way to debug is actually not to debug at all. What that means is to remove the need to debug by adding more tests. “The best defense is a good offense” is a famous military adage. For software development, we can now change it to, “The best debugging is testing.”
References
I listed some standard definitions of debugging and testing. The intention is to show how similar and overlapping these two concepts are and thus they should be approached with a similar attitude and rigor.
[1] Alaboudi, A. & LaToza, T. (2021) An Exploratory Study of Debugging Episodes. shows that developers spend 15–60% of their time debugging.
[2] Hetzel, William C., The Complete Guide to Software Testing, 2nd ed. Publication info: Wellesley, Mass.: QED Information Sciences, 1988.
Herzel defined testing to be “ the process of establishing confidence that a program or system does what it is supposed to.” (1972 ed.)
Later, Hertzel redefined it to be “any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results.” (1983 ed.)
[3] Myers, The Art of Software Testing (2012 ed.)
Myers defined testing to be “the process of executing a program or system with the intent of finding errors.”
[4] ANSI/IEEE 1059 standard defines testing to be “the process of analysing a software item to detect the differences between existing and required conditions (that is defects/errors/bugs) and to evaluate the features of the software item.”
[5] Wikipedia defines debugging to be “the process of finding and resolving bugs (defects or problems that prevent correct operation) within computer programs, software, or systems.”
