EntBlog » CodeGems

Compile-Time Strings

ent — Tue, 28 Apr 2009 22:19:40 +0000

It would be nice if we had such a feature in the C language, wouldn’t it? The term ‘compile-time string’ is referred here as strings that are converted to unique integer identifiers at compile time. At run-time those identifiers are simple integers that can be compared and hashed very fast. In other languages, like for example Smalltalk, the concept of Symbol implements a similar idea. The following post describes a possible implementation of this feature in C/C++.

Imagine, for example, a generic object factory where object instances are created using unique identifiers. The classical solution here is having a shared-by-all-code header where all the identifiers are declared in a C enumeration. This solution, apart from creating a serious physical dependency where adding a new identifier to the enumeration forces a recompilation for all the project, is unfeasible in modular architectures where modules are isolated. In those architectures having a global header is not an option.

One viable solution may be using strings as identifiers. But strings are heavy objects, hard to compare and prone to typing errors because miswritten symbols would be detected at run-time instead of compile-time. Other equally insufficient solutions to this problem include FourCC and esoteric template tricks for generating a hash at compile time (desist from it, it is not possible to solve this 100% with templates because strings cannot be used as template parameters and anyway hashing a string is not collision-free. More information in this usenet thread). Mick West proposes more solutions in his Practical Hash IDs article.

What follows is an implementation that has been working nicely for me and that satisfactorily fits the requirements for simulating compile-time strings. First let me show you two examples of the usage:

namespace
{
DECLARE_SYMBOL(CubeMesh)
DECLARE_SYMBOL(SphereMesh)
DECLARE_SYMBOL(DuckMesh)
}
 
void CollectNodes(Ptr<Node>& node)
{
    Ptr<Mesh> mesh0 = CreateObject(S(CubeMesh));
    node->Add(mesh0);
 
    Ptr<Mesh> mesh1 = CreateObject(S(SphereMesh));
    node->Add(mesh1);
 
    Ptr<Mesh> mesh2 = CreateObject(S(DuckMesh));
    node->Add(mesh2);
}

namespace
{
DECLARE_SYMBOL(FirstMessage)
DECLARE_SYMBOL(SecondMessage)
DECLARE_SYMBOL(ThirdMessage)
}
 
void ProcessMessage(const Message& msg)
{
    if (msg.id == S(FirstMessage))
    {
       /// ...
    }
    else if (msg.id == S(SecondMessage))
    {
       /// ...
    }
    else if (msg.id== S(ThirdMessage))
    {
       /// ...
    }
}

A symbol represents a compile-time string. They must be declared before being used. The macro for declaring a symbol is hiding an inline function with a static inside itself:

#define DECLARE_SYMBOL(id)\
    inline Symbol __GetSymbol##id() throw()\
    {\
        static size_t sym;\
        if (sym == 0)\
        {\
            sym = GetIdFromString(#id);\
        }\
        return Symbol(sym);\
    }

The function GetIdFromString() hashes the string, stores it in an internal table and returns the table position for that string (the Symbol class is a simple wrapper around the identifier). This is done only the first time the symbol is requested. For future requests the static ID is returned. This adds a little overhead against using simple integers as symbols. Beware of local static initializations: they are not thread-safe. That is the reason of the manual comparison against 0. GetIdFromString() must be thread-safe for this code to work.

The S macro simply invokes the local function previously generated:

#define S(id) __GetSymbol##id()

And there you have it. Compile-time strings with negligible (in case you are doing anything more that simply comparing symbols) overhead. In case you need 100% efficient code you could pre-generate a table with the symbols being used by your project (searching for all DECLARE_SYMBOL blocks) and substitute each S() with a really unique identifier generated at compile-time. And that would be so easy if the preprocessor could be extended in a standard way…

Hope this makes sense. Thank you for reading.

Stripping comments from Shader bytecodes

ent — Thu, 15 Jan 2009 20:37:39 +0000

In DirectX, when compiling a shader with D3DXCompileShader() a buffer containing the shader bytecodes is received. Apart from the bytecodes, extra content like debug and symbol table information is embedded. That extra information is added in form of comments that probably can be eliminated because you are already processing it at compile-time and it is not needed at run-time when loading the shader.

If you can do without that information the following code will help you to save a few bytes, even halving the size of the byte-code in the best cases.

Although not documented in the DirectX SDK, this CodeGem is not an undocumented hack. The Direct3D shader code format is documented in the MSDN, so it probaly won’t change in future revision of DirectX v9.0 (if there is going to be any more…)

D3DPtr<ID3DXBuffer> StripComments(const D3DPtr<ID3DXBuffer>& code)
{
    // Calculates the new size (without comments)
    int* codeData = static_cast<int*>(code->GetBufferPointer());
    unsigned int sizeInWords = code->GetBufferSize() / 4;
    unsigned int strippedSizeInWords = sizeInWords;
 
    for (unsigned int i = 0; i < sizeInWords; i++)
    {
        if ((codeData[i] & 0xffff) == D3DSIO_COMMENT)
        {
            int commentSize = codeData[i] >> 16;
            strippedSizeInWords -= 1 + commentSize;
            i += commentSize;
        }
    }
 
    // Creates a new buffer with the original code but omitting the comments
    D3DPtr<ID3DXBuffer> strippedCode;
    V(D3DXCreateBuffer(strippedSizeInWords * 4, strippedCode.GetPtrForInit()));
 
    int* strippedCodeData = static_cast<int*>(strippedCode->GetBufferPointer());
    size_t offset = 0;
 
    for (unsigned int i = 0; i < sizeInWords; i++)
    {
        if ((codeData[i] & 0xffff) == D3DSIO_COMMENT)
        {
            int commentSize = codeData[i] >> 16;
            i += commentSize;
        }
        else
        {
            strippedCodeData[offset++] = codeData[i];
        }
    }
 
    return strippedCode;
}

Capturing OutputDebugString

ent — Fri, 09 Nov 2007 01:20:51 +0000

Using OutputDebugString for tracing your programs is probably not a good idea, at least if you have lots of traces being generated (OutputDebugString raises an exception and causes a kernel mode transition). So you will probably end up implementing your own tracing/logging system. Parts of code that are not under your control may be still using OutputDebugString (like Debugging Tools for Windows, DirectX, etc). The code listed below allows capturing OutputDebugString calls generated inside your own process (in fact, it is capturing all the OSD calls generated by all the active processes).

You will need to put this piece of code in a separate thread, but those details are omitted for better clarity.

NOTE: If the process is being debugged, the ODS calls will be intercepted by debugger.

struct DbWinBuffer
{
    DWORD dwProcessId;
    char data[4096 - sizeof(DWORD)];
};
 
DbWinBuffer* dbBuffer;
 
HANDLE hAckEvent;
HANDLE hEvent;
HANDLE hSharedFile;
 
SECURITY_DESCRIPTOR sd;
SECURITY_ATTRIBUTES sa;
/////////////////////////////////////////////////////////////////////////////
 
sa.nLength = sizeof(SECURITY_ATTRIBUTES);
sa.bInheritHandle = true;
sa.lpSecurityDescriptor = &sd;
 
if (!InitializeSecurityDescriptor(&sd, SECURITY_DESCRIPTOR_REVISION))
{
    printf(”ERROR: InitializeSecurityDescriptor\n”);
    return 1;
}
 
if (!SetSecurityDescriptorDacl(&sd, true, 0, false))
{
    printf(”ERROR: SetSecurityDescriptorDacl\n”);
    return 1;
}
 
hAckEvent = CreateEvent(&sa, false, false, L”DBWIN_BUFFER_READY”);
if (!hAckEvent)
{
    printf(”ERROR: CreateEvent(\”DBWIN_BUFFER_READY\”)\n”);
    return 1;
}
 
hEvent = CreateEvent(&sa, false, false, L”DBWIN_DATA_READY”);
if (!hEvent)
{
    printf(”ERROR: CreateEvent(\”DBWIN_DATA_READY\”)\n”);
    return 1;
}
 
hSharedFile = CreateFileMapping((HANDLE)-1, &sa, PAGE_READWRITE, 0, 
    L”DBWIN_BUFFER”);
if (!hSharedFile)
{
    printf(”ERROR: CreateFileMapping(\”DBWIN_BUFFER\”)\n”);
    return 1;
}
 
dbBuffer = static_cast<DbWinBuffer*>(MapViewOfFile(hSharedFile, FILE_MAP_READ, 0,
    0, 4096));
if (!dbBuffer)
{
    printf(”ERROR: MapViewOfFile\n”);
    return 1;
}
 
SetEvent(hAckEvent);
 
DWORD pid = GetCurrentProcessId();
printf(”Tracing PID: %d\n\n”, pid);
 
for (;;)
{
    DWORD ret = WaitForSingleObject(hEvent, INFINITE);
    if (ret == WAIT_FAILED)
    {
        printf(”ERROR: WaitForSingleObject\n”);
        return 1;
    }
 
    if (dbBuffer->dwProcessId == pid)
    {
        printf(”%s”, dbBuffer->data);
    }
 
    SetEvent(hAckEvent);
}

Efficient update of stl containers

ent — Wed, 17 May 2006 21:08:20 +0000

Update: Fixed some bugs in the sample code.
Update: Note added in the third option about the hint.

I have seen dozens of times code like this to update an entry in a map:

typedef std::map<std::string, int> Items;
Items items;
 
Items::iterator it = items.find("counter");
 
if (it == items.end())
{
   items.insert(std::make_pair("counter", 1));
}
else
{
   it->second++;
}

The problem with this code is that you are traversing the map two times if the item is not found in the map, one for the first find and one more for the insert.

You can improve the code with something like this:

std::pair<items::iterator, bool> res = items.insert(std::make_pair("counter", 1));
 
if(!res.second)
{
   (*res.first).second++;
}

This is notably better. You only traverse the map one time. But there is still a problem with this code: you are creating the pair even when you do not need it. In this case the pair is simple and its creation is not a problem, but imagine a pair with more complex objects hard to create. You can avoid the creation with a code like this:

Items::iterator lowerBound = items.lower_bound("counter");
 
if(lowerBound != items.end()
   && !(items.key_comp()("counter", lowerBound->first)))
{
   lowerBound->second++;
}
else
{
   items.insert(lowerBound, std::make_pair("counter", 1));
}

Although being more efficient, this code is more obscure, so in case you do not need those nanoseconds you can live with the more clear second option. Apart from being more obscure, the hint is only that, a hint. The implementation can ignore it. As noted by Steven in the comments below, Visual Studio ignores it for the hash_map and hast_set containers.