How To Avoid Crashes When Debugging System-wide Hooks

MORE INFORMATION

First, your hook procedure needs to check to see if the current process is the debugger. If it is not, proceed to your code. If it is, simply call the CallNextHookEx function. Because this is a system-wide hook, the procedure for determining if you are in the debugger should be kept to a minimum, or else it could slow the process down tremendously.

Sample Code

Use the following sample code to debug a keyboard hook:

   #ifdef _DEBUG
   #define PATHSIZE 256

   BOOL  IsCurProcDebugger();
   #endif

   // Shared DATA
   #pragma data_seg("HOOKVARS")
   // Other variables should be in here (for example, hhookKeyboard).
   // Add this variable to the shared data segment.
   #ifdef _DEBUG
   DWORD dwDebuggerProcId = 0;
   #endif
   #pragma data_seg()

   // Filter function for the WH_KEYBOARD.

   LRESULT CALLBACK KeyboardFunc (int nCode, WPARAM wParam, LPARAM lParam )
   {
      if ( nCode >= 0 )
      {

         // Add this if-block to your code.
         //*** You should put breakpoints ONLY inside this if-block ***
   #ifdef _DEBUG
         if ( !IsCurProcDebugger() ) // If you are not in the debugger...
         {
   #endif

            // Your hook code goes here.

   #ifdef _DEBUG
         }
   #endif
      }

      return( CallNextHookEx(hhookKeyboard, nCode, wParam, lParam));
   }


   #ifdef _DEBUG
   // *** DO NOT PUT ANY BREAKPOINTS IN THIS CODE!!!
   BOOL IsCurProcDebugger()
   {
      // In Visual C++, declare these three variables after the
      // following if-block to speed things up.
      char szPathName[PATHSIZE];
      char *szFileName = szPathName;
      char *tcp;

      // Do this first for speed.
      // If there is only one debugger present
      // and you have already found it.
      if ( dwDebuggerProcId )
      {
         return ( GetCurrentProcessId() == dwDebuggerProcId );
      }

      // If only one debugger is running, then the rest of this code
      // should be entered only until the debugger is first hooked.
      // After that, the preceding code should catch it every time.

      GetModuleFileName( NULL, szPathName, PATHSIZE );

      // Only check the file name, not the full path.
      // A co-worker's path may be different.

      for ( tcp = szPathName; *tcp; tcp++ )
      {
         if ( *tcp == '/' || *tcp == '\\' )
            szFileName = tcp + 1;
      }

      // Use "MSDEV.EXE" for the Visual C++ debugger, or
      // else use YOUR debugger's name.

      if ( !strcmp( _strupr(szFileName), "MSDEV.EXE") )
      {
         // It's the debugger!
         dwDebuggerProcId = GetCurrentProcessId();
         return TRUE;
      }

      return FALSE;
   }
   #endif  // #ifdef _DEBUG
				

Important Caveats

1. The usual rules apply to the global variables (especially the DebuggerProcId and the hhookKeyboard). They need to be in a shared data segment or a memory mapped file.
2. Certain hooks will still have problems with this method (most notably the journal hooks).
3. Certain types of hooks will occasionally exhibit strange behavior. For example, the above code was tested this with a keyboard hook. Without this solution, pressing the F10 key to step to the next line of code or the F5 key to continue would be impossible. With this solution, it works. However, if you press F5 and release too slowly, a problem may occur. The "keydown" event on F5 causes the debugger to "continue." By the time you release the key, you are out of the debugger and the hook captures the F5's "keyup" event. If you press F5 and release it quickly enough, the "keyup" event occurs before the debugger continues the process.
4. This code has not been tested with more than one debugger running. With minor modifications, it should theoretically work. Also, you have to add extra code if the debuggers have different names.
5. Because of pre-emptive multi-tasking, shared data may be overwritten. If this is a problem for you, synchronization objects (such as mutual exclusions and semaphores) can help, but they should be used with caution because synchronization objects put the hooked thread to sleep. You may also need to watch out for deadlocks. Critical sections should not be used because they are valid only within a single process space.