Trapper V1.3
============

Written by Manfred Lippert
Email: mani@mani.de
Web:   http://www.mani.de/


What is Trapper?
================

Trapper is a program that greatly simplifies the development of system 
extensions by programmers.

With the help of Trapper other programs can hook themselves simply and
effectively before and after system calls, even completely replace 
them, or add new system calls to the system.

Assembler programming is no longer required for this. One can hook  
any desired C-routine into the system immediately. Other programming 
languages should also be possible.


Installation
============

Trapper belongs in the AUTO folder. For test purposes it can also be 
started subsequently from the desktop.

One should note that Trapper naturally has to be launched before any 
of the system extensions that require Trapper.

Under MagiC one should use the file AUTOEXEC.BAT to establish the 
order of the AUTO folder programs (see the MagiC docs). The order can 
also be shown and changed in some boot managers.


Freeware
========

Trapper may be distributed freely - but only with this text, and both 
unaltered.

Trapper may only be used by freeware programs. If Trapper is to be 
used by commercial programs then this has to be agreed with me first:
Mail to mani@mani.de.


I would like a few more details
===============================

For many clever tools it is necessary to hook into certain system 
calls at various sytem levels (Bios, Xbios, Gemdos, VDI or AES). 
Up to now this always required knowledge of Assembler programming. 
The greatest problem is due to the fact that the newly hooked-in 
functions have to be completely reentrant.

As long as one is only hooking in _before_ a system function, this 
is not particularly difficult: One bends the vector of the system 
function to a function of one's own, and when this finishes one jumps 
back again to the original function. (This is best done with the XBRA 
procedure).

But one encounters enormous problems if one wants to hook in _after_ 
system functions, say to manipulate return values, or to be able to 
execute actions that one may only perform in user mode: One has to 
bend the return-jump pointer (which is on the stack) to a routine of 
one's own and then let the system call continue normally. When the 
system call returns one has control again, and naturally you have 
to jump back subsequently to the original return address.

The last bit conceals the problem:

Where does one store the old return address so that the whole thing 
remains reentrant? The only solution would be on the currently active 
stack. Unfortunately, for most system functions this already holds the 
parameters... In order to solve this problem cleanly, one has to 
resort to a trick. Trapper solves the problem by storing the old 
return address on the stack, and after that the parameters are packed 
on top of it (again, as a copy).

Every programmer who wants to write a cool tool that hooks into system 
calls has to fight with this problem. As a side-effect the system will 
be slowed down slightly by each such tool and the stack consumptiom 
rises - possibly so much that it can lead to system crashes...

Here Trapper attempts to provide a remedy in several respects:

Instead of one having to hook into system calls oneself, one simply 
leaves this to Trapper and installs the functions to be hooked in 
with it.

This has the following advantages:

- One does not have to muck around with complex Assembler programming.
With quite normal C functions one can hook in before and also after (!)
any of the system calls. (Other programming languages should also be 
possible). This greatly simplifies development of system extensions.

- There is only a single hooking into the system calls, due to which 
the speed and the stack consumption remain constant.

- One can notify Trapper which function opcodes the hooked-in function 
should be responsible for. This does away with the dispatch of opcodes 
for each individually hooked-in program and the system will not be 
slowed down. Trapper finds the hooked-in functions extremely quickly 
via special algorithms (trees and lists). The overhead will not be 
noticeable even with an enormous number of hooked-in functions.

- A program can unhook itself again from the system calls and 
terminate without problems. If the program itself hooks directly into 
a system call and bends the return addresses, then it has the enormous 
problem that it can only terminate after all functions called with the 
bent return address have returned once more. But many system calls 
take a long time to return in some circumstances, for instance Pexec()
or the evnt_ functions of the AES. As one is hanging "indirectly" in 
the calls via Trapper, unhooking can be done without problems.


For programmers
===============

Trapper hooks into Bios, Xbios, Gemdos, VDI and AES. As an interface 
to Trapper the Xbios function number 555 with the following calling 
conventions was chosen:

int32 Trapper(int16 layer, int16 install, int16 opcode, void *function);

It is imperative to test the return value from Trapper. If the 
hooking-in has worked then E_OK will be returned, otherwise another 
value (e.g. ENSMEM if there is insufficient memory available).

The parameters:

layer: Specifies the system layer which the function should hook into.
The following values are possible:

#define TRAPPER_CHECK  -1
#define TRAPPER_BIOS    0
#define TRAPPER_XBIOS   1
#define TRAPPER_GEMDOS  2
#define TRAPPER_VDI     3
#define TRAPPER_AES     4

TRAPPER_CHECK: Special function for testing the presence of Trapper.
"install", "opcode" should be set to 0, "function" is either NULL or
points to the following structure:

typedef struct {
	int16 version;     /* BCD coded, e.g. 0x110 for Version 1.10 */
	int16 reserved[15];
} TrapperInfo;

If Trapper is installed then Trapper(TRAPPER_CHECK) returns the value 
E_OK (0) and fills out the above structure. Otherwise a value not 
equal to E_OK is returned and one can not use Trapper.

TRAPPER_BIOS, TRAPPER_XBIOS, TRAPPER_GEMDOS, TRAPPER_VDI and
TRAPPER_AES specify the corresponding system layer into which the 
function "function" should be hooked. Depending on the layer, function 
has a (slightly) differing calling convention (see below).

install: Determines what is to happen with "function":

#define TRAPPER_INSTALL_CALL    0
#define TRAPPER_REMOVE_CALL     1
#define TRAPPER_INSTALL_RETURN  2
#define TRAPPER_REMOVE_RETURN   3

TRAPPER_INSTALL_CALL: Installs the function _before_ the system call.

TRAPPER_INSTALL_RETURN: Installs the function _after_ the system call.

TRAPPER_REMOVE_CALL and TRAPPER_INSTALL_RETURN remove the relevant 
functions from Trapper again.

opcode: Determines the function opcode for which the function should 
be called. If one passes -1 then the function will be called for 
_every_ function of the system layer.

function: Points to the function to be hooked in or out.
(Exception: TRAPPER_CHECK, see above.)

Depending on the system layer (layer) and installation (install) the 
following function-types are valid (calling conventions):

typedef int32 CDECL (*BiosCallFunc)(int16 *para, int16 *call_original, int16 super_called);
typedef int32 CDECL (*BiosReturnFunc)(int32 ret, int16 *para, int16 is_super);

typedef int32 CDECL (*XbiosCallFunc)(int16 *para, int16 *call_original, int16 super_called);
typedef int32 CDECL (*XbiosReturnFunc)(int32 ret, int16 *para, int16 is_super);

typedef int32 CDECL (*GemdosCallFunc)(int16 *para, int16 *call_original, int16 super_called);
typedef int32 CDECL (*GemdosReturnFunc)(int32 ret, int16 *para, int16 is_super);

typedef void CDECL (*VDICallFunc)(VDIPB *para, int16 *call_original, int16 super_called);
typedef void CDECL (*VDIReturnFunc)(VDIPB *para, int16 is_super);

typedef void CDECL (*AESCallFunc)(AESPB *para, int16 *call_original, int16 super_called);
typedef void CDECL (*AESReturnFunc)(AESPB *para, int16 is_super);

CDECL means that the function will, according to C-convention, have 
the parameters passed on the stack (in Pure-C one writes "cdecl" 
instead for this).
int16 is a 16-bit integer (in Pure-C "int" or "short"), int32 is a 
32-bit integer (in Pure-C "long").

The return value will be returned via register D0, as usual in C.

The functions may - after C-convention - only alter registers
D0,D1,D2,A0,A1. All other registers must be restored again after the 
function. All C-compilers I know keep to this convention.

"para" points for Bios, Xbios and Gemdos to the stack area with the 
opcode and the parameters. para[0] is, for instance, the opcode of 
the function. After this follow the actual parameters, which naturally 
depend on the opcode.

"call_original" plays a special role: This variable points to a flag 
signalling whether the original system function is to be called after 
the execution of the function. *call_original always has the value 
1 (true) during the function call, so normally does not need to be 
altered. If one wants to replace the original system function, then 
one sets *call_original to 0 (false).

The return value of the "Call" functions normally only plays a role 
when *call_original was set to 0, or when one adds a completely new 
function to the system (so that there is no original function). In 
all other cases one should simply return E_OK (0).

"super_called" (for "Call" functions) contains 0 (false) if the 
system function was called from the user mode, and 1 (true), if it 
was called from the supervisor mode. One should note that during the 
execution of the function one is naturally always in the supervisor 
mode! One is, so to speak, "in" the trap.

"is_super" (for "Return" functions) contains 0 (false) if one is 
currently in the user mode and 1 (true) if one is currently in 
supervisor mode.

One should take great case that the functions use as little of the 
stack as possible.

In "Call" functions, i.e. if one is hooking in before functions, one 
will always be in supervisor mode and therefore use the supervisor 
stack (which is some circumstances can correspond to the user stack of 
the program in question if this has used Super() to switch to the 
supervisor mode).

In "Return" functions, i.e. if one is hooking in after functions, one 
will normally use the stack of the program that has called the system 
function. Hence one should always keep in mind that one is using 
"foreign" stacks where one has no knowledge about their size, and 
therefore use as little of the stack as possible.


Important: A program that uses Trapper should (as long as it is not a 
TSR) definitely hook into etv_term, and in case of an unforeseen 
program termination use it to unhook again all functions hooked into 
Trapper.


Useful bindings
===============

One should create one's own "low level" binding to Trapper (Xbios 
function number 555) for the system one is using. The Assembler binding 
could look something like this, if the Trapper function gets its 
parameters passed on the stack following C-conventions:

int32 CDECL Trapper(int16 layer, int16 install, int16 opcode, void *function);

Trapper:
	pea (a2)              // TOS does not save A2 in traps
	move.l 14(sp),-(sp)   // function
	move.l 14(sp),-(sp)   // install/opcode
	move.w 16(sp),-(sp)   // layer
	move.w #555,-(sp)     // Opcode 555
	trap #14              // Xbios-trap
	lea 12(sp),sp         // correct stack
	move.l (sp)+,a2       // restore A2
	rts

With register passing the whole thing looks slightly different, of
course.


To make the hooking-in more comfortable, it is useful to define the 
following "higher level" bindings, for instance, that in turn call the 
"low level" Trapper binding. This has the added advantage that a type 
check takes place for the passed functions.

int32 TrapperCheck(TrapperInfo *info)
{
	return Trapper(TRAPPER_CHECK, 0, 0, info);
}


int32 TrapperInstallBiosCall(int16 opcode, BiosCallFunc func)
{
	return Trapper(TRAPPER_BIOS, TRAPPER_INSTALL_CALL, opcode, func);
}

int32 TrapperRemoveBiosCall(int16 opcode, BiosCallFunc func)
{
	return Trapper(TRAPPER_BIOS, TRAPPER_REMOVE_CALL, opcode, func);
}

int32 TrapperInstallBiosReturn(int16 opcode, BiosReturnFunc func)
{
	return Trapper(TRAPPER_BIOS, TRAPPER_INSTALL_RETURN, opcode, func);
}

int32 TrapperRemoveBiosReturn(int16 opcode, BiosReturnFunc func)
{
	return Trapper(TRAPPER_BIOS, TRAPPER_REMOVE_RETURN, opcode, func);
}


int32 TrapperInstallXbiosCall(int16 opcode, XbiosCallFunc func)
{
	return Trapper(TRAPPER_XBIOS, TRAPPER_INSTALL_CALL, opcode, func);
}

int32 TrapperRemoveXbiosCall(int16 opcode, XbiosCallFunc func)
{
	return Trapper(TRAPPER_XBIOS, TRAPPER_REMOVE_CALL, opcode, func);
}

int32 TrapperInstallXbiosReturn(int16 opcode, XbiosReturnFunc func)
{
	return Trapper(TRAPPER_XBIOS, TRAPPER_INSTALL_RETURN, opcode, func);
}

int32 TrapperRemoveXbiosReturn(int16 opcode, XbiosReturnFunc func)
{
	return Trapper(TRAPPER_XBIOS, TRAPPER_REMOVE_RETURN, opcode, func);
}


int32 TrapperInstallGemdosCall(int16 opcode, GemdosCallFunc func)
{
	return Trapper(TRAPPER_GEMDOS, TRAPPER_INSTALL_CALL, opcode, func);
}

int32 TrapperRemoveGemdosCall(int16 opcode, GemdosCallFunc func)
{
	return Trapper(TRAPPER_GEMDOS, TRAPPER_REMOVE_CALL, opcode, func);
}

int32 TrapperInstallGemdosReturn(int16 opcode, GemdosReturnFunc func)
{
	return Trapper(TRAPPER_GEMDOS, TRAPPER_INSTALL_RETURN, opcode, func);
}

int32 TrapperRemoveGemdosReturn(int16 opcode, GemdosReturnFunc func)
{
	return Trapper(TRAPPER_GEMDOS, TRAPPER_REMOVE_RETURN, opcode, func);
}


int32 TrapperInstallVDICall(int16 opcode, VDICallFunc func)
{
	return Trapper(TRAPPER_VDI, TRAPPER_INSTALL_CALL, opcode, func);
}

int32 TrapperRemoveVDICall(int16 opcode, VDICallFunc func)
{
	return Trapper(TRAPPER_VDI, TRAPPER_REMOVE_CALL, opcode, func);
}

int32 TrapperInstallVDIReturn(int16 opcode, VDIReturnFunc func)
{
	return Trapper(TRAPPER_VDI, TRAPPER_INSTALL_RETURN, opcode, func);
}

int32 TrapperRemoveVDIReturn(int16 opcode, VDIReturnFunc func)
{
	return Trapper(TRAPPER_VDI, TRAPPER_REMOVE_RETURN, opcode, func);
}


int32 TrapperInstallAESCall(int16 opcode, AESCallFunc func)
{
	return Trapper(TRAPPER_AES, TRAPPER_INSTALL_CALL, opcode, func);
}

int32 TrapperRemoveAESCall(int16 opcode, AESCallFunc func)
{
	return Trapper(TRAPPER_AES, TRAPPER_REMOVE_CALL, opcode, func);
}

int32 TrapperInstallAESReturn(int16 opcode, AESReturnFunc func)
{
	return Trapper(TRAPPER_AES, TRAPPER_INSTALL_RETURN, opcode, func);
}

int32 TrapperRemoveAESReturn(int16 opcode, AESReturnFunc func)
{
	return Trapper(TRAPPER_AES, TRAPPER_REMOVE_RETURN, opcode, func);
}


Questions and answers
=====================

Q: What happes if several functions hook into the same system call 
(same system level and same opcode)?

A: If several functions are hooked into one system call, then the 
"Call" functions (hooked in before the original call) will be called 
one after the other in reverse order and "Return" functions (after the 
system call) in the normal order of their installation.

For "Calls" however the general functions (with Opcode -1) installed 
for opcodes will be called first, after this the functions installed 
specially for the opcodes in question. For "Returns" it is the other 
way round.

For "Calls" the following applies as well: As soon as one of the 
functions sets the call_original flag to 0, the original function will 
be replaced and no longer called afterwards. The return value will 
then be that of the last function that has set call_original to 0.
All further functions hooked into Trapper will still be called. That 
applies both for the remaining hooked-in "Call" functions, as well as 
the "Return" functions. Shortly there will be an alternative 
hooking-in option that behaves in a slightly different way. See 
"Outlook".

Q: How do you guarantee the speed with very many hooked-in functions?

A: For each system layer an automatic optimal self-balancing binary 
tree (AVL tree) with all hooked-in functions is created. This way all 
the functions hooked into a functions opcode can be found very quickly
(O(log n)). Hence it makes little sense to hook into a system level 
with opcode -1 and then test for the correct opcodes oneself, as 
Trapper can accomplish this far more effectively.

Q: I see. And what do I do if I want to use the same function in two 
or more functions (opcodes)?

A: Even then one should not hook in with opcode -1, but simply hook 
the function several times into the desired opcodes. The interrogation 
for the opcodes will then still have to be built into one's function, 
but Trapper will not call the function at all with other opcodes.


Outlook
=======

Shortly one should be able to hook in "Calls" and "Returns" in pairs. 
Programs that hook into the system via Trapper in this way will then 
behave more like "manually" hooked in programs (without Trapper). If 
for one of these "Call-Return" pairs the Call function sets the 
*call_original flag to 0 (false), then all other "Call-Return" pairs 
called earlier will not be called - just like in real life. ;-)

The previous hooking-in possibilities in Trapper will however be 
retained for compatibility reasons. Just the reverse - they still have 
their useful right to existence then: Functions hooked-in in the 
present manner will be called in any case. Functions linked in as 
"Pairs" can be replaced by other pairs. Both can be useful.
To completely replace functions, the coming pair option is more 
useful.


History
=======

V2.00, 18.3.2000
----------------

- Mechanismfor linking "after" the traps was rewritten completely.
The amount of the stack used is now appreciably lower, the code 
simpler, cleaner and faster. Trapper should now run everywhere.
laufen.

V1.42, 1.2.2000
----------------

- Nullpointer bug fixed that copuld lead to crashes on original 
Atari hardware.

V1.41, 27.7.1999
----------------

- If Trapper is already installed, then it now returns a return value
0 instead of -1, so that MagiC no longer displays a "Fatal error" alert.

V1.4, 29.4.1999
---------------

- Bug fixed that could cause Trapper to crash under some circumstances 
on original Ataris.

- Code optimised somewhat, Trapper now some 4 KB smaller.

V1.3, 2.3.1999
--------------

- Debug session under SingleTOS performed and during this two bugs 
  were forced out. Trapper should now run cleanly everywhere.

V1.21, 1.3.1999
---------------

- A seldom occurring problem during hooking-in via XBRA removed.

V1.2, 25.2.1999
---------------

- Workaround for faulty programs whose startup code does not create 
  a stack (mostly TSR programs).

V1.1, 19.2.1999
---------------

- Stupid bug removed. Thanks to this the Pure-Debugger now runs with
  Trapper insalled, for instance.

- Traps can be traced in the AUTO folder already (apart from AES and
  VDI).

- The calls are now called in the reverse order of their installation.
  (Return as before.)


V1.0, 29.1.1999 bis 6.2.1999
----------------------------

- First version


Have fun with Trapper,
Manfred Lippert

------------------------------------
English translation: Peter West, DDP
