Today we continue our little journey through my newest project and have a look at libraries and structures in the “virtual” Amiga RAM of vamos.
Make sure to read Part 1 of the series first…
Todays topics:
- Amiga Libraries
- Amiga Structures
1. Amiga Libraries
We already know from our first expirments that an Amiga library has a large jump table for all functions which we already use to trap the calls and redirect them to our Python vamos code.
The open question is now what functions does a library have and what registers are filled by the application before calling this function. Furthermore, we need to know what we have to return in the registers before returning to the caller. The last question is answered quickly: data registrer D0 is the return value of all lib calls (if the lib call isn’t void then no register contains a return value). So mapping the python trapped call return value to D0 was quickly done.
For the function description in a library Commodore invented a textual programming language neutral description called the FD files describing each function call and the arguments required along with the CPU registers where the argument will be found.
To simplify my work with vamos I first wrote a parser for the FD files (that can be found in the Amiga NDKs). This little tool is called fdtool and also available in my amitools source tree. You simply call it with a single FD file and it shows all functions and the relative jump offset in the table:
> ./fdtool NDK_3.1/Includes\&Libs/fd/exec_lib.fd NDK_3.1/Includes&Libs/fd/exec_lib.fd base: _SysBase #0001 30 0x001e Supervisor [userFunction,a5] #0002 72 0x0048 InitCode [startClass,d0][version,d1] #0003 78 0x004e InitStruct [initTable,a1][memory,a2][size,d0] #0004 84 0x0054 MakeLibrary [funcInit,a0][structInit,a1][libInit,a2][dataSize,d0][segList,d1] #0005 90 0x005a MakeFunctions [target,a0][functionArray,a1][funcDispBase,a2] #0006 96 0x0060 FindResident [name,a1] #0007 102 0x0066 InitResident [resident,a1][segList,d1] #0008 108 0x006c Alert [alertNum,d7] #0009 114 0x0072 Debug [flags,d0] #0010 120 0x0078 Disable #0011 126 0x007e Enable #0012 132 0x0084 Forbid #0013 138 0x008a Permit #0014 144 0x0090 SetSR [newSR,d0][mask,d1] #0015 150 0x0096 SuperState #0016 156 0x009c UserState [sysStack,d0 ..
Its already very useful to decode jumps into the lib (Note: the jump uses a negative offset while the tools displays positive ones).
The next step was to create a code generator that writes stubs for Python that describe all functions:
./fdtool -g NDK_3.1/Includes\&Libs/fd/exec_lib.fd
(30, 'Supervisor', (('userFunction', 'a5'),)),
(36, 'execPrivate1', None),
(42, 'execPrivate2', None),
(48, 'execPrivate3', None),
(54, 'execPrivate4', None),
(60, 'execPrivate5', None),
(66, 'execPrivate6', None),
(72, 'InitCode', (('startClass', 'd0'), ('version', 'd1'))),
(78, 'InitStruct', (('initTable', 'a1'), ('memory', 'a2'), ('size', 'd0'))),
(84, 'MakeLibrary', (('funcInit', 'a0'), ('structInit', 'a1'), ('libInit', 'a2'), ('dataSize', 'd0'), ('segList', 'd1'))),
(90, 'MakeFunctions', (('target', 'a0'), ('functionArray', 'a1'), ('funcDispBase', 'a2'))),
(96, 'FindResident', (('name', 'a1'),)),
(102, 'InitResident', (('resident', 'a1'), ('segList', 'd1'))),
(108, 'Alert', (('alertNum', 'd7'),)),
(114, 'Debug', (('flags', 'd0'),)),
(120, 'Disable', None),
(126, 'Enable', None),
(132, 'Forbid', None),
(138, 'Permit', None),
(144, 'SetSR', (('newSR', 'd0'), ('mask', 'd1'))),
(150, 'SuperState', None),
I simply copy & paste this output in my ExecLibrary class and this table now describes all functions available and also their parameters. This information is not used for trapping in the first place but for logging: Now I can decode each jump into the Lib and display the named function call…
19:39:29.159 lib: INFO: [ exec.library] { CALL: 198 AllocMem( byteSize[d0]=def4, requirements[d1]=10001 ) from PC=00205a
19:39:29.159 lib: INFO: [ exec.library] } END CALL: d0=000108ac
19:39:29.159 lib: INFO: [ exec.library] { CALL: 732 StackSwap( newStack[a0]=113e8 ) from PC=0020e8
19:39:29.159 lib: INFO: [ exec.library] } END CALL: d0=0000000
With this information in place I could add trapped functions by adding a new table to my Library object: this one maps the function offset (again positive) to an actual Python method inside the class. I can do this for only the functions I need – all others have a default handler that returns D0=0 and issues a warning trace that this function is still unimplemented.
An excerpt of the Python ExecLibrary class looks like this:
def __init__(self): exec_funcs = ( (408, self.OldOpenLibrary), (414, self.CloseLibrary), ... ) self.set_funcs(exec_funcs) def OpenLibrary(self, lib, ctx): name_ptr = ctx.cpu.r_reg(REG_A1) ... return lib_addr
You see the mapping of Python functions in this example. Note that each Python call gets the same parameters and not the Amiga function calls directlly: It gets a context object and via this context the function can access the virtual CPU and read out the registers it needs…
The return value of the Python method will be directly written to D0 automatically. Returning None will not alter D0. This is useful for void function calls.
2. Amiga Structures
The library jump tables are not the only kind of data an Amiga application directly accesses outside its own memory segments: A lot of public data structures reside in memory and function calls often pass in pointers to them and also private ones. Some functions even require to create new structures and we then need to return their pointers. Furthermore, each library itself has (beside the jump table) a so called “pos size range” that contains data structures of “public” accessible information. E.g. exec.library has a large pos size with lots of system constants and also information like your own task structure…
Our task in vamos is now to correctly fill these structures as needed and provide a convinient mechanism in Python to access those data members. Unfortunately, those data structures are only defined in the language headers of Commodore’s NDKs and there is no generic description like there is the FDs for the calls 🙁
Without having a tool to decode some generic data structure definitions I started to convert the C Amiga headers manually into a Python structure definition. Those definition are not structures in Python itself but rather meta objects that describe a structure in the “virtual” Amiga Memory of vamos. This tedious process was only performed on demand, i.e. onyl the structures currently needed were transcribed to Pyhon.
Similar to fdtool I have written a typetool to provide a small command line utility that uses the structures defined in Python and displays them (This is again useful for disassembly reading as you can quickly look up and index and find the structure entry):
./typetool Library
@0000 Library {
@0000 Node {
0000 @0000/0000 +0004 Node* ln_Succ (ptr=True, sub=False)
0001 @0004/0004 +0004 Node* ln_Pred (ptr=True, sub=False)
0002 @0008/0008 +0001 UBYTE ln_Type (ptr=False, sub=False)
0003 @0009/0009 +0001 BYTE ln_Pri (ptr=False, sub=False)
0004 @0010/000a +0004 char* ln_Name (ptr=True, sub=False)
@0014 =0014 } lib_Node
0005 @0014/000e +0001 UBYTE lib_Flags (ptr=False, sub=False)
0006 @0015/000f +0001 UBYTE lib_pad (ptr=False, sub=False)
0007 @0016/0010 +0002 UWORD lib_NegSize (ptr=False, sub=False)
0008 @0018/0012 +0002 UWORD lib_PosSize (ptr=False, sub=False)
0009 @0020/0014 +0002 UWORD lib_Version (ptr=False, sub=False)
0010 @0022/0016 +0002 UWORD lib_Revision (ptr=False, sub=False)
0011 @0024/0018 +0004 APTR lib_IdString (ptr=False, sub=False)
0012 @0028/001c +0004 ULONG lib_Sum (ptr=False, sub=False)
0013 @0032/0020 +0002 UWORD lib_OpenCnt (ptr=False, sub=False)
@0034 =0034 }
This query for the Library structure of Exec displays all entries including nested structures, their byte offset from the beginning, the C type and the name to access each entry.
This structure information is now used for two purposes in vamos: First it provides a convinient way to access structures inside memory for the Python lib calls when they have to access data that was provided by giving structure pointers. The second use of structures is to provide so called memory labels for logging. Every time vamos allocates a structure (e.g. ExecLib pos space) it also registers a struct memory label for the same address. A label manager keeps all registered labels and if memory tracing is enabled then you can look up an arbitrary address if it can be labelled. A structure label looks like this:
19:22:23.891 mem: INFO: R(4): 0102a8: 00000000 Struct [@010210 +000098 ThisTask] Process+152 = pr_CurrentDir(BPTR)+0 19:22:23.892 mem: INFO: R(4): 0102bc: 00004070 Struct [@010210 +0000ac ThisTask] Process+172 = pr_CLI(BPTR)+0 19:22:23.892 mem: INFO: R(4): 0102bc: 00004070 Struct [@010210 +0000ac ThisTask] Process+172 = pr_CLI(BPTR)+0 19:22:23.892 mem: INFO: R(4): 0101d0: 00004080 Struct [@0101c0 +000010 CLI] CLI+16 = cli_CommandName(BSTR)+0
So a read access to memory address 0x0102a8 is detected as structure access to ThisTask.pr_CurrentDir. That’s what I call comfortable debugging 😉 Note that the labelling look ups cost quite a lot of performance and that’s the reason why you have to enable memory tracing with a command line switch (-t/-T).
The most common use of Structures is to decode lib call arguments. In Python I provide a so called AccessStruct object that assist reading/writing structures:
def StackSwap(self, lib, ctx):
stsw_ptr = ctx.cpu.r_reg(REG_A0)
stsw = AccessStruct(ctx.mem, StackSwapDef, struct_addr=stsw_ptr)
# get new stack values
new_lower = stsw.r_s('stk_Lower')
new_upper = stsw.r_s('stk_Upper')
new_pointer = stsw.r_s('stk_Pointer')
This is an excerpt of the Python implemenation of the Exec StackSwap call: In A0 a pointer to a StackSwap structure is passed in. We have already declared the function in Python as a defintion (hence the trailing Def):
class StackSwapStruct(AmigaStruct):
_name = "StackSwap"
_format = [
('APTR', 'stk_Lower'),
('ULONG', 'stk_Upper'),
('APTR', 'stk_Pointer')
]
StackSwapDef = StackSwapStruct()
With a memory pointer and a structure definition we can create an access object. Now the reads and writes to the entries of the structure a simple calls can be performed with the entry name of the structure… Ok its slower than direct offset handling but far more confortable and less error prone…
Beside the structure access vamos also provides a more generic memory access object that allows to read and write blocks of data, reads c-type strings and bcpl strings with a single call:
def OpenLibrary(self, lib, ctx): ver = ctx.cpu.r_reg(REG_D0) name_ptr = ctx.cpu.r_reg(REG_A1) name = ctx.mem.access.r_cstr(name_ptr) ..
That’s it for today… I hope you got more insight on the Library and Struct handling and the confortable features available in vamos for working with them.
Leave a Reply