How To Use VDMDBG Functions on Windows NT, Windows 2000, and Windows XP

MORE INFORMATION

When you use any of the VDMDBG functions, you need to link VDMDBG.lib with your project.

The VDMDBG functions mentioned below are only a subset of the available functions. However, the functions that are not mentioned are only relevant to a debugger.

Enumerating VDMs

The VDMEnumProcessWOW() function provides a simple way to enumerate all VDMs running 16-bit Windows tasks. These VDMs contain the WowExec.exe task. DOS VDMs are not enumerated.

The declaration for this function is as follows:

   INT WINAPI VDMEnumProcessWOW( PROCESSENUMPROC fp, LPARAM lparam );
				

The return value for this function is the number of VDMs currently running, or the number enumerated before enumeration was terminated. fp is a pointer to a callback function. The function is called for each VDM that is enumerated. lParam is a user-defined value that is passed to the callback function.

PROCESSENUMPROC is declared as follows:

   typedef BOOL ( WINAPI *PROCESSENUMPROC )(
      DWORD dwProcessId,
      DWORD dwAttributes,
      LPARAM lpUserDefined
   );
				

The function should return TRUE to stop enumeration or FALSE to continue enumeration. dwProcessId is the process ID of the NTVDM.exe process. You will need this ID when calling the other VDM functions mentioned below.

Enumerating 16-bit Windows Tasks

You can use VDMEnumTaskWOW() and VDMEnumTaskWOWEx() to enumerate tasks within a particular VDM. The difference between the two is that VDMEnumTaskWOWEx() provides more information to the callback function. You should only use VDMs returned by VDMEnumProcessWOW() with these task enumeration functions. Using DOS VDMs does not make sense because every DOS application runs in its own VDM.

The declarations are:

   INT WINAPI VDMEnumTaskWOW( DWORD dwProcessId, TASKENUMPROC fp,
                              LPARAM lparam );
				

   INT WINAPI VDMEnumTaskWOWEx( DWORD dwProcessId, TASKENUMPROCEX fp,
                                LPARAM lparam );
				

The return value for each of these functions is the number of tasks currently running within the indicated VDM, or the number enumerated before enumeration was terminated. dwProcessId is the process ID of the VDM. fp is a pointer to a callback function. The function is called for each task that is enumerated. lparam is a user-defined value that is passed to the callback function.

TASKENUMPROC and TASKENUMPROCEX are defined as follows:

   typedef BOOL ( WINAPI *TASKENUMPROC )(
      DWORD dwThreadId,
      WORD hMod16,
      WORD hTask16,
      LPARAM lpUserDefined
   );
				

   typedef BOOL ( WINAPI *TASKENUMPROCEX )(
      DWORD dwThreadId,
      WORD hMod16,
      WORD hTask16,
      PSZ pszModName,
      PSZ pszFileName,
      LPARAM lpUserDefined
   );
				

These functions should return TRUE to stop enumeration or FALSE to continue enumeration. You can use hTask16 in a call to terminate the task.

Example of Enumeration

   // Enumerate all 16-bit tasks on the system.
   #include <windows.h>
   #include <stdio.h>
   #include <vdmdbg.h>

   BOOL WINAPI ProcessEnumProc( DWORD, DWORD, LPARAM );
   BOOL WINAPI TaskEnumProcEx( DWORD, WORD, WORD, PSZ, PSZ, LPARAM );

   void main()
   {
      // Enumerate VDMs.
      VDMEnumProcessWOW(
         (PROCESSENUMPROC)ProcessEnumProc,
         (LPARAM)NULL
      );

   }

   BOOL WINAPI ProcessEnumProc( DWORD dwProcessId, DWORD dwAttrib,
      LPARAM t )
   {
      printf("\nProcess ID: %d\n", dwProcessId);

      // Use process ID of VDM to enumerate through its tasks.
      VDMEnumTaskWOWEx(
         dwProcessId,
         (TASKENUMPROCEX)TaskEnumProcEx,
         (LPARAM)NULL
      );

      // Keep enumerating.
      return FALSE;
   }

   BOOL WINAPI TaskEnumProcEx( DWORD dwThreadId, WORD hMod16, WORD hTask16,
      PSZ pszModName, PSZ pszFileName, LPARAM lParam )
   {
      //print task's information
      printf("Thread ID: %d\n", dwThreadId);
      printf("Module handle: %d\n", hMod16);
      printf("Task handle: %d\n", hTask16);
      printf("Module Name: %s\n", pszModName);
      printf("File Name: %s\n", pszFileName);

      // Keep enumerating.
      return FALSE;
   }
				

Creating 16-bit Tasks

VDMStartTaskInWOW() creates a task within a VDM. The declaration is as follows:

   BOOL WINAPI VDMStartTaskInWOW( DWORD dwProcessId, LPSTR lpCommandLine,
                                  WORD wShow );
				

The return value of this function is TRUE if the task is successfully started, otherwise it is FALSE. dwProcessId is the VDM process ID. lpCommandLine is a string indicating the filename of the 16-bit application along with any command-line parameters. wShow indicates how the window will be shown. wShow can be any value that is valid for the 16-bit ShowWindow() function.

Terminating 16-bit Tasks

To terminate a task within a VDM call VDMTerminateTaskWOW(). The declaration is as follows:

   BOOL WINAPI VDMTerminateTaskWOW( DWORD dwProcessId, WORD htask );
				

The return value of this function is TRUE if the task is successfully terminated, otherwise it is FALSE. dwProcessId is the VDM process ID. hTask is the handle to the task. This task handle can be obtained through VDMEnumTaskWOW() or VDMEnumTaskWOWEx().

This method is a rough equivalent to TerminateProcess() in Win32. It should be avoided, if possible. It does not give the task a chance to cleanly exit, so data may be lost. Unlike Win32, the WowExec is not guaranteed to clean up after a terminated task. This can leave the VDM corrupt and unusable. To terminate the task cleanly, send a WM_CLOSE to its top-level window.

Note Regarding 16-bit DOS Applications

None of the VDMDBG functions work with 16-bit DOS applications. To enumerate DOS VDMs, you need to use another method. First, you could use VDMEnumProcessWOW() to make a list of all Win16 VDMs, and then enumerate all instances of NTVDM.exe using some other scheme (such as PSAPI). Any NTVDM.exe from the full enumeration that was not in the Win16 list is a DOS VDM. You can create and terminate 16-bit DOS applications with CreateProcess() and TerminateProcess().