Cowboy Programming Game Development and General Hacking by the Old West

January 4, 2007

Debug and Release

Filed under: Game Development,Inner Product — Mick West @ 6:05 pm

This article originally appeared in the “Inner Product” column, Game Developer Magazine, October 2005


Game projects usually have a number of “build configurations”, which control how the game code is compiled and linked into the game executable. Each build configuration, or build mode, builds the game with varying amounts of additional debugging code, and with the optimization options modified to aid debugging. Microsoft’s Visual Studio has by default just two: “Debug” and “Release”. The idea is that you will develop your game in Debug mode, making it easier to find bugs, and then you ship it in Release mode, with debugging code removed and optimizations switched on to make your code small and fast.

This division usually starts out fine. At the start of a game development cycle there are not too many assets in the game, level maps are small, not much of the logic is implemented and the CPU is rarely taxed. But after a few months of development, things start to get into the game. As the quantity of game assets, entities and logic approaches the level of the final game, debug mode becomes unusably slow. This leads the development team to switch to using release mode for day-to-day development work.

This situation can make bugs a lot harder to track down. The traditional release mode lacks debugging code and assertions. When the programmer attempts to reproduce the bugs in Debug mode, they first have to rebuild in Debug mode (which can take a very long time) then the code runs at a really slow frame rate, making it very difficult to reproduce the bug, and sometimes the bugs cannot be reproduced at all, due to the different configuration and initialization of memory.

I argue that the traditional division into Debug and Release is inappropriate for game development. Debug mode and release mode as traditionally envisaged should no longer be considered useful options, and should be replaced by a single build configuration “develop mode” that should be used at all stages of development.

Debug and Release mode

When you set up a project in Microsoft Visual Studio, you get two “build configurations”: Debug and Release. The distinction between these seems obvious: You use “Debug” to debug the project, and when it’s ready to ship you switch over to “Release”

This distinction is usually found in any game development project, and sometimes it’s extended to some additional configurations, like “ExtraDebug”, “Final” or “Submit”.

The intent of debug mode is to make the code easier to debug. Now we’ve become so used to having the distinction between debug and release over the years that it’s useful to take a step back and see exactly why we needed a debug mode in the first place.

A common misconception that I’d like to clear up right away is that you can’t use the debugger unless you build your code in debug mode. This is simply not true. The debugger will work just fine in release mode, but some aspects will be a bit harder to debug.

The idea behind debugging mode is to make it as easy as possible to track down bugs in your code. This results in two major differences:
1) The code is compiled in such a way that the debugger works as well as possible.
2) Extra code is compiled in, usually in the form of assertions, to trap bugs as early as possible, and to provide additional debugging information.

Optimization: In debug mode, optimization is disabled, so the code runs a lot slower. Why is this? Well, optimization involves re-ordering the sequences of assembly instructions so there is not always a direct correlation between a group of assembly instructions and a line of C++ code. So when you step through your code in the debugger, it’s not always clear exactly where you are in the code, and the PC might jump oddly between lines.

Also, optimizing involves keeping values in registers instead of memory as frequently as possible. So, local variables that are not used very much are not stored in the local stack frame. This means that in the debugger you often can’t examine the contents of a local variable in optimized code, as one you step past the code that uses it, the value no longer exists. In debug mode the local variables always have local storage for each local variable, so you can always examine their values in the debugger.

Inline expansion: Another form of optimization is the expansion of inline functions. This is done to clarify the flow of control in the debugger. If an inline function is expanded inline then you can’t step “into” it, and you can’t step “over” it. It almost becomes invisible to the debugger.

Debug Code: Besides the optimizations, you usually add additional “debugging code” to your program to help track down bugs. To this end you generally have some symbols defined that tell you at compile time which build configuration you are using, so you can conditionally compile code in for debugging.

In Microsoft visual studio these symbols are “DEBUG” which if defined when you are in debug mode, and “NDEBUG” which is defined when you are not in debug mode (or in release mode)

Assertions: Everyone should be familiar with assertions. Assertions are additional line of code that are sprinkled through the program to ensure that everything is as it should be. Basically it’s a macro that checks (asserts) that a condition that you know should be true actually is true. If the assert fails, then the program execution halts, and you can see what went wrong. Asserts should be your most useful tool in debugging your code.

Let’s have a quick look at Microsoft’s implementation of the assert macro:

#ifdef NDEBUG
#define assert(exp) ((void)0)
#define assert(exp) (void)( (exp) || (_assert(#exp, __FILE__,__LINE__), 0) )
#endif /* NDEBUG */

As you can see, the assert is compiled away to nothing when you build in release mode. This makes sense, as when you release the game, you don’t want asserts going off. Plus they take up space, and all those extra checks slow down the game.

Debug and Browse information: For the debugger to work it needs to know how to link the code it is executing to the original source. This information is generated during compilation and linking and is referred to as “debug info”. In many “release” configurations the debug information is switched off. This is generally to make the project build faster, and to make the executable smaller. An executable .ELF file built with GNU C++ can be anything use to 100MB in size when built with full debug info, compared with a 3-5MB file without debug info.

Memory allocation and initialization: Memory allocation is the source of many bugs. So it makes sense to have extra debugging code in your memory allocator. Typically memory will be initialized to a known value when it is allocated. The pattern of allocations will also differ, as blocks have extra debug info added. Since most games use their own custom allocators, the differences vary. But nearly all games will have some difference in memory allocation between debug and release modes.

Uninitialized variables are a common source of error. Generally you should be catching such problems at compile time. But if not, then you need to realize that their behavior will differ in release and debug modes. In debug modes, local variables are more likely to have actual storage, and depending on your precise build configuration and compiler, they will probably be initialized to zero in debug mode. In release mode the uninitialized variable will be some random value – whatever happened to be in memory or the register used.

Game are real time applications. They have to run at a reasonable speed in order for you to say that they are working. So if the debug mode is very slow, then it’s not going to be practical to use the debug mode as the development build – the build used by artists and level designers implementing assets, and by programmers implementing new features in code or script.

However games are also incredibly complex applications, where the game engine is often still in development while the game content is being created and implemented using that game engine. If everyone is using the release mode for development then it makes it very difficult to track down the bugs that inevitably arise in a development environment.

Clearly then neither of the traditional build modes is suitable for game development. I’d like to make the case that a single hybrid mode should be used for 99% of all development. This hybrid mode (which I’ll call “Develop Mode”) should be fast enough so that gameplay is not affected, and yet contain enough debugging features so that bugs are caught early, and easily tracked down. I’ll also make the case that develop mode should essentially be the configuration mode your game ships with.

Develop Mode

Assertions switched on. Developing without assertions is like driving with your eyes closed. You’ll know when you crashed, but you won’t know why you crashed, and your crashes will be much worse. Having assertions on all the time during development will greatly improve the rate at which you find and fix bugs.

Optimization switched on. Your code needs to be fast if develop mode is going to be used by artists and level designers. Sure, you can’t tell exactly what is going on in the debugger, you won’t be able the see the contents of local variables. But you will be still be able to identify the place where in the code the crash occurs, see the call stack and roughly follow the logic flow. If you need more information then often the solution is to add more assertions. You can add logging calls to track the contents of variables, and if all else fails you can temporarily switch on optimization.

Inline Expansion switched on. Similar to optimization, but with games the inline expansion being off is often a far greater source of slow debug code than other aspects of optimization. Most games will have some kind of custom 3D vector class, usually with accessor functions, or overloaded [] operators that use inline functions. Having these functions be explicit adds a vast overhead to code execution. The benefit you get is simpler flow of execution in the debugger, something you rarely need. So, switch inlining back on.

Link without debug information – This one can be a vast time saver. The link stage of the edit-compile-link-run cycle can take over a minute, or even several minutes, depending on the type of project. The vast majority of that time is spent in generating debug information, when really all you want is the executable. Remember the compilation units are still being complied with debug information, so if the code crashes, and you want to go into the debugger, then you can just switch debug information back on and re-link, and you’ll have the debug information, but only when you actually need it.

Make your assertions fast. Assertions should never need more than 5% of your total CPU time. If you turn assertions on and your framerate plummets, then there may be a problem in the implementation of your assertion macro (many game developers implement their own version of assert). The majority of assertions are simply comparisons of two values, and usually one of these values is something the compiler will have in a register, and the other if frequently a constant. So your assertion should compile to two or three instructions that perform the test and then skip over the code that arranges the parameters and calls the assert handler. You should verify this by looking at the compiled code in the debugger.

Use assertions appropriately. Much as I love assertions, there is such a thing as too many assertions. Assertions in very low level functions are often testing the same thing over and over again. For example, collision detection code might use unit normals stored in the mesh. Since collision code runs a lot each frame, then adding an assertion to verify that the normals were of unit length might have a serious impact on framerate. Plus you’d be repeatedly checking the same normals over and over. In this case it might be better to verify the input just once when the mesh is initially loaded, or even when it’s originally generated.

In addition, putting assertions in at such a low level of the code is often hit and miss. The conditions that might cause the code to fail could occur very infrequently. You might have to play the game for hours in order to hit the triangle with a malformed normal. Here you want to use automated tests that ensure full coverage of the code and data. These tests can be run constantly, and need not be part of the develop build.

Ideally your develop mode should also be your release mode. This automatically eliminates that bane of game development: the bugs that only show up when the game is in release mode. The problem here is that you’ve got to devote up to 5% of your CPU time, and a chunk of memory, if you want to leave in all your assertions. The solution is obviously to budget for that from the start.

This may seem like a lot of ask. The harsh reality is that game developers are often scrambling for a few thousand extra bytes, and CPU time is never adequate. But consider the benefits of having just one build configuration up to and including the version you ship. There is no risk of obscure bugs popping up due to changes in the code. If you ship for console you can get back more useful information from publisher tests. If you ship on PC, you can add a facility for users to report assertions, which will allow you to get that patch out quicker.

If you ARE going to ship without assertions, for whatever reason, then make sure you budget enough time to test that version. The majority of your testing should be done on a version with assertions in, as you’ll track down ordinary bugs much quicker. But at some point you’ll need to test your final version. Just don’t switch off the assertions the day before you submit. It’s actually might be a good idea to occasionally test with assertions removed, as the different code configuration might bring more obscure bugs to the surface.

Reference and further reading

Rabin, S. Squeezing more out of Assert. Game Programming Gems 1.
Etherton, D. Designing and Maintaining Large Cross-Platform Libraries.. Game Programming Gems 4.
Hunt, A & Thomas, D. Leave Assertions turned on. The Pragmatic Programmer. p123.

No Comments »

No comments yet.

RSS feed for comments on this post. TrackBack URL

Leave a comment

You must be logged in to post a comment.

Powered by WordPress