01 September 2011

Working with registers

I had a nasty bug recently, which was causing stack smashing. To solve it, I wanted to to read the values stored in the Base Pointer and Stack pointer registers (BP and SP). This is how I did it:
void ***ebp=0,*esp=0;
asm("mov %%rbp, %0" : "=m"(ebp));
asm("mov %%rsp, %0" : "=m"(esp));
printf("[%016X][%016X][%016X][%016X]\n", esp, ebp, *ebp, **ebp);
Now this code reads the two registers, and then prints the current stack pointer, followed by the base pointers of the following 3 frames. This can be done to see what went wrong with those values. The bug I had was caused by an overflow of a string which was allocated on the stack. A few months ago I had another nasty stack smashing bug, caused by a faulty pointer allocated in heap, which was pointing to a structure allocated on the stack. So when the function was done, the pointer was pointing to a random spot on the stack basically (sometimes to a base pointer, smashing the stack). This is done for a 64 bit architecture. On a 32 bit machine, things are not very different.