Setting up a Symbol Server

July 6, 2006 @ 2:27 | In Programming | 7 Comments | twitter rss digg

Debug symbols allow you to have better debugging sessions. They have information about the functions contained in executables and dynamic libraries and provide you with information to get clean call stacks. A Symbol Server allows the debugger to load the correct symbols, binaries and sources automatically.

In this article, I describe how to set up a Symbol Server for your programming team to store symbols (from the operating system, third party libraries and from your own libraries), binaries and source code to improve the productivity of the team and the service given to your clients.

1. Configuring the Symbol Server

You will need the latest version of Debugging Tools for Windows. Download it and install (install the source server support under SDK/Source Server; it is not installed by default and we will need it later). After installing add the installation directory (C:\Program Files\Debugging Tools for Windows) and source server directory (C:\Program Files\Debugging Tools for Windows\sdk\srcsrv) to the PATH environment variable.

Now, you have to decide where you will store the symbols. For a team, having a shared repository is a good idea. Here we will store the symbols in \\BuildServer\Symbols. The only service needed in BuildServer machine is sharing the Symbols folder. Reserve enough space for this purpose (I recommend a minimum of 30Gb).

We will start using symbols downloaded from Microsoft Symbol Server (http://msdl.microsoft.com/download/symbols). This server stores all the symbols from Microsoft products (released operating systems, service packs, directx …)

If you are using Visual Studio 2005, options for symbols are under Tools/Options/Debugging/Symbols

Symbols Option in Visual Studio

If you are not using Visual Studio 2005, you must set the _NT_SYMBOL_PATH environment variable as follows:

_NT_SYMBOL_PATH = SRV*\\BuildServer\Symbols*http://msdl.microsoft.com/download/symbols

The next time you start debugging, you will start to populate the symbol server with symbols through HTTP from Microsoft. It will only download the symbols needed and cache them for your next debugging session (and for the debugging session of your teammates if you are using a shared directory).

I show you here the stack trace from a session without a Symbol Server and from a session with a Symbol Server installed.

Call Stack with symbols loaded
Call Stack with symbols loaded

Great difference, isn’t it?

This is only the beginning of the history. Continue reading to discover the amazing things you can do with your new Symbol Server.

2. Populating the server with your own symbols and binaries

You can store your own symbols and binaries into the symbol store using symtore.exe. For a detailed description of the command-line options for symstore consult the documentation given in the SDK or in MSDN.

A good idea is to have your daily build integrated with your Symbol Server, adding all the exes, dlls and pdbs resulting from your builds (daily, stable, beta, etc). For example, the following command stores all pdbs and binary files found under c:\build:

symstore add /r /f c:\build\*.* /s \\BuildServer\Symbols /t “Prueba” /v “Build 0.5.730″ /c “05/05/06 Daily Build”

Inside the directory 000Admin of your shared symbol server directory you will find a file called history.txt with all the transactions that have ocurred in the Symbol Server. You will need this information when you want to delete symbols from the database. For more information consult the documentation.

The next thing you must do is set up your MODPATH environment variable. It says where to get binaries (exes and dlls) from the symbols. It must point to your symbol server cache directory:

MODPATH=\\BuildServer\Symbols

You don’t have to do this if you are using Visual Studio 2005. It will automatically retrieve binaries from your Symbol Server.

And now, for the funny part: generate a dump of your program (you can create one while debugging in Debug/Save Dump As…) and copy it to another computer. Now, open that dump (with Visual Studio) and press F5 to start debugging. Visual Studio will automatically download the symbols and the binaries associated to that dump. Magic isn’t it?

But we can do it better. We can retrieve proper source codes too, continue reading.

3. Retrieving source files

To use this feature, you need Visual Studio 2005 (it is not supported by previous versions) and activate the option under Tool/Options/Debugging/General/Enable source server support. This is a new option that has not been publicized by Microsoft and I don’t understand why.

To store sources in your Symbol Server you have to source index your symbols files (pdbs). Inside the symbols you have to store a reference to the source that was used to generate each symbol. The source code is not saved inside the symbols, only a reference to the file in your source server control. Read the srcsrv.doc document provided by the SDK to understand how this feature exactly works.

Inside the SDK of Debugging Tools for Windows you have scripts for Perforce, Visual SourceSafe and SubVersion. They give you enough documentation to extend the support to other source control servers. Perforce will be used here. The scripts are written in perl (yes, Microsoft giving perl scripts) so you will need to install a perl interpreter (you can find one here).

Before storing your symbols in the server (and generally you will be doing this as a phase of your build process) you source index them using p4index (for perforce) located in C:\Program Files\Debugging Tools for Windows\sdk\srcsrv.

You can check that your symbols have been properly indexed using the srctool command.

srctool Prueba2\Debug\Prueba.pdb

Prueba2\Debug\Prueba.pdb: 3 source files are indexed – 241 are not

And now you are ready to repeat the experiment, open the dump in a machine with no source, no binaries and no symbols. As soon as you start debugging, the source will be automatically downloaded from your source server (perforce in this example)

Visual Studio Security Warning Dialog

Don’t you find this exciting?

4. How to get maximum productivity

We have seen how Visual Studio connect with the Symbol Server to get the needed information. Your program can do the same to get extra information. These are some guidelines:

  • Store all your builds in the Symbol Server (daily and stable at least). Clients should be using the binaries produced by your build system.
  • Provide stack traces in asserts, error dialogs, loggers. Get those traces using your symbol server.
  • Generate minidumps in critical errors. Those minidumps should be sent to your team for analyzing.
  • Use all this information (traces, dumps) when publishing reports in your bugs database.

I’ll write more articles about these points and how to access the Symbol Server from your code in future posts. For now, this is enough. I hope you enjoyed this.




  1. Nice post.

    It is also worth mentioning that if you want to jump start the microsoft symbols on your hard drive or share, you can download the packages for the links above, then add them using symstore.exe (if you want to maintain the same organization). This is a good idea if you anticipate having to debug something remotely with minimal (or no) internet access.



    Comment by josh
    October 30, 2007 @ 21:09 #

  2. Yeah, thanks for clarifying. Those symbols are a good start for the symbol server.

    One day, I will update this article with that kind of information, how to set up the source control server with subversion and other interesting tricks. :)



    Comment by ent
    October 31, 2007 @ 0:34 #

  3. Thanks for this entry. I managed to save a minidump.dmp file without problem. And I can see that the symbol store is correct using windbg/cdb.
    However I cannot seem to load this file meaninfully into Visual Studio 2005 (I start that by hand as 2003 is installed as well on my machine). When opening this file I get into some sort of hexeditor mode. There doesn’t seem to be a ‘Load Crashdump’ option or similar in VS2005 (like there is in WinDbg).

    Am I missing something? I would realy like to finish this setup and use
    this to checkout files from our revision control system.



    Comment by Anthon
    November 29, 2007 @ 16:02 #

  4. Sorry, I fired the question to early.
    You have to load this *file* not with File->Open->File but with File->Open->Project/Solution %-)



    Comment by Anthon
    November 29, 2007 @ 16:16 #

  5. Yes, you gave the answer to yourself. :)
    In my machine, .dmp are registered to Visual Studio, so double clicking is enough to open the MiniDump. Then you need to hit F5 to start debugging it.



    Comment by ent
    November 29, 2007 @ 16:55 #

  6. [BLD-339] Setup MS symbol server for plugin symbols…

    Just like we can load symbols for IE, Windows, and FF from symbol servers it would be incredible useful if we could do the same for the plugin. Could you look into what we need to do? It’s probably just a question of transferring files somewhere and t…



    Trackback by JIRA: Build Farm
    August 22, 2008 @ 12:53 #

  7. [...] Crashes are stored as minidumps in that same folder. And, as we distribute pdb with our releases, this means that everybody in the team can open any dump from any release and reproduce the exact crashing conditions everywhere. For more information about symbols, read my previous article about Setting up a Symbol Server [...]



    Pingback by Tangential Software Usage | EntBlog
    September 30, 2008 @ 9:33 #


Tue, 02 Sep 2014 23:38:08 +0000 / 27 queries. 1.253 seconds / 2 Users Online

gentoo link wordpress link apache link PHP link

Theme modified from Pool theme. Valid XHTML and CSS