Today we continue our little journey through my newest project and have a look at libraries and structures in the “virtual” Amiga RAM of vamos.

Make sure to read Part 1 of the series first…

Todays topics:

• Amiga Libraries
• Amiga Structures

1. Amiga Libraries

We already know from our first expirments that an Amiga library has a large jump table for all functions which we already use to trap the calls and redirect them to our Python vamos code.

The open question is now what functions does a library have and what registers are filled by the application before calling this function. Furthermore, we need to know what we have to return in the registers before returning to the caller. The last question is answered quickly: data registrer D0 is the return value of all lib calls (if the lib call isn’t void then no register contains a return value). So mapping the python trapped call return value to D0 was quickly done.

For the function description in a library Commodore invented a textual programming language neutral description called the FD files describing each function call and the arguments required along with the CPU registers where the argument will be found.

To simplify my work with vamos I first wrote a parser for the FD files (that can be found in the Amiga NDKs). This little tool is called fdtool and also available in my amitools source tree. You simply call it with a single FD file and it shows all functions and the relative jump offset in the table:

> ./fdtool NDK_3.1/Includes\&Libs/fd/exec_lib.fd 
NDK_3.1/Includes&Libs/fd/exec_lib.fd
 base: _SysBase
 #0001     30  0x001e                Supervisor [userFunction,a5]
 #0002     72  0x0048                  InitCode [startClass,d0][version,d1]
 #0003     78  0x004e                InitStruct [initTable,a1][memory,a2][size,d0]
 #0004     84  0x0054               MakeLibrary [funcInit,a0][structInit,a1][libInit,a2][dataSize,d0][segList,d1]
 #0005     90  0x005a             MakeFunctions [target,a0][functionArray,a1][funcDispBase,a2]
 #0006     96  0x0060              FindResident [name,a1]
 #0007    102  0x0066              InitResident [resident,a1][segList,d1]
 #0008    108  0x006c                     Alert [alertNum,d7]
 #0009    114  0x0072                     Debug [flags,d0]
 #0010    120  0x0078                   Disable 
 #0011    126  0x007e                    Enable 
 #0012    132  0x0084                    Forbid 
 #0013    138  0x008a                    Permit 
 #0014    144  0x0090                     SetSR [newSR,d0][mask,d1]
 #0015    150  0x0096                SuperState 
 #0016    156  0x009c                 UserState [sysStack,d0
..

Its already very useful to decode jumps into the lib (Note: the jump uses a negative offset while the tools displays positive ones).

The next step was to create a code generator that writes stubs for Python that describe all functions:

./fdtool -g NDK_3.1/Includes\&Libs/fd/exec_lib.fd 
 (30, 'Supervisor', (('userFunction', 'a5'),)),
 (36, 'execPrivate1', None),
 (42, 'execPrivate2', None),
 (48, 'execPrivate3', None),
 (54, 'execPrivate4', None),
 (60, 'execPrivate5', None),
 (66, 'execPrivate6', None),
 (72, 'InitCode', (('startClass', 'd0'), ('version', 'd1'))),
 (78, 'InitStruct', (('initTable', 'a1'), ('memory', 'a2'), ('size', 'd0'))),
 (84, 'MakeLibrary', (('funcInit', 'a0'), ('structInit', 'a1'), ('libInit', 'a2'), ('dataSize', 'd0'), ('segList', 'd1'))),
 (90, 'MakeFunctions', (('target', 'a0'), ('functionArray', 'a1'), ('funcDispBase', 'a2'))),
 (96, 'FindResident', (('name', 'a1'),)),
 (102, 'InitResident', (('resident', 'a1'), ('segList', 'd1'))),
 (108, 'Alert', (('alertNum', 'd7'),)),
 (114, 'Debug', (('flags', 'd0'),)),
 (120, 'Disable', None),
 (126, 'Enable', None),
 (132, 'Forbid', None),
 (138, 'Permit', None),
 (144, 'SetSR', (('newSR', 'd0'), ('mask', 'd1'))),
 (150, 'SuperState', None),

I simply copy & paste this output in my ExecLibrary class and this table now describes all functions available and also their parameters. This information is not used for trapping in the first place but for logging: Now I can decode each jump into the Lib and display the named function call…

19:39:29.159        lib:   INFO:  [    exec.library]  { CALL:  198 AllocMem( byteSize[d0]=def4, requirements[d1]=10001 ) from PC=00205a
19:39:29.159        lib:   INFO:  [    exec.library]  } END CALL: d0=000108ac
19:39:29.159        lib:   INFO:  [    exec.library]  { CALL:  732 StackSwap( newStack[a0]=113e8 ) from PC=0020e8
19:39:29.159        lib:   INFO:  [    exec.library]  } END CALL: d0=0000000

With this information in place I could add trapped functions by adding a new table to my Library object: this one maps the function offset (again positive) to an actual Python method inside the class. I can do this for only the functions I need – all others have a default handler that returns D0=0 and issues a warning trace that this function is still unimplemented.

An excerpt of the Python ExecLibrary class looks like this:

def __init__(self):
  exec_funcs = (
     (408, self.OldOpenLibrary),
     (414, self.CloseLibrary),
     ...
  )
  self.set_funcs(exec_funcs)

def OpenLibrary(self, lib, ctx):
  name_ptr = ctx.cpu.r_reg(REG_A1)
  ...
  return lib_addr

You see the mapping of Python functions in this example. Note that each Python call gets the same parameters and not the Amiga function calls directlly: It gets a context object and via this context the function can access the virtual CPU and read out the registers it needs…

The return value of the Python method will be directly written to D0 automatically. Returning None will not alter D0. This is useful for void function calls.

2. Amiga Structures

The library jump tables are not the only kind of data an Amiga application directly accesses outside its own memory segments: A lot of public data structures reside in memory and function calls often pass in pointers to them and also private ones. Some functions even require to create new structures and we then need to return their pointers. Furthermore, each library itself has (beside the jump table) a so called “pos size range” that contains data structures of “public” accessible information. E.g. exec.library has a large pos size with lots of system constants and also information like your own task structure…

Our task in vamos is now to correctly fill these structures as needed and provide a convinient mechanism in Python to access those data members. Unfortunately, those data structures are only defined in the language headers of Commodore’s NDKs and there is no generic description like there is the FDs for the calls 🙁

Without having a tool to decode some generic data structure definitions I started to convert the C Amiga headers manually into a Python structure definition. Those definition are not structures in Python itself but rather meta objects that describe a structure in the “virtual” Amiga Memory of vamos. This tedious process was only performed on demand, i.e. onyl the structures currently needed were transcribed to Pyhon.

Similar to fdtool I have written a typetool to provide a small command line utility that uses the structures defined in Python and displays them (This is again useful for disassembly reading as you can quickly look up and index and find the structure entry):

./typetool Library
 @0000        Library {
 @0000          Node {
0000 @0000/0000 +0004      Node*      ln_Succ               (ptr=True, sub=False)
0001 @0004/0004 +0004      Node*      ln_Pred               (ptr=True, sub=False)
0002 @0008/0008 +0001      UBYTE      ln_Type               (ptr=False, sub=False)
0003 @0009/0009 +0001      BYTE       ln_Pri                (ptr=False, sub=False)
0004 @0010/000a +0004      char*      ln_Name               (ptr=True, sub=False)
 @0014 =0014    } lib_Node
0005 @0014/000e +0001    UBYTE      lib_Flags             (ptr=False, sub=False)
0006 @0015/000f +0001    UBYTE      lib_pad               (ptr=False, sub=False)
0007 @0016/0010 +0002    UWORD      lib_NegSize           (ptr=False, sub=False)
0008 @0018/0012 +0002    UWORD      lib_PosSize           (ptr=False, sub=False)
0009 @0020/0014 +0002    UWORD      lib_Version           (ptr=False, sub=False)
0010 @0022/0016 +0002    UWORD      lib_Revision          (ptr=False, sub=False)
0011 @0024/0018 +0004    APTR       lib_IdString          (ptr=False, sub=False)
0012 @0028/001c +0004    ULONG      lib_Sum               (ptr=False, sub=False)
0013 @0032/0020 +0002    UWORD      lib_OpenCnt           (ptr=False, sub=False)
 @0034 =0034  } 

This query for the Library structure of Exec displays all entries including nested structures, their byte offset from the beginning, the C type and the name to access each entry.

This structure information is now used for two purposes in vamos: First it provides a convinient way to access structures inside memory for the Python lib calls when they have to access data that was provided by giving structure pointers. The second use of structures is to provide so called memory labels for logging. Every time vamos allocates a structure (e.g. ExecLib pos space) it also registers a struct memory label for the same address. A label manager keeps all registered labels and if memory tracing is enabled then you can look up an arbitrary address if it can be labelled. A structure label looks like this:

19:22:23.891        mem:   INFO:  R(4): 0102a8: 00000000  Struct  [@010210 +000098 ThisTask] Process+152 = pr_CurrentDir(BPTR)+0
19:22:23.892        mem:   INFO:  R(4): 0102bc: 00004070  Struct  [@010210 +0000ac ThisTask] Process+172 = pr_CLI(BPTR)+0
19:22:23.892        mem:   INFO:  R(4): 0102bc: 00004070  Struct  [@010210 +0000ac ThisTask] Process+172 = pr_CLI(BPTR)+0
19:22:23.892        mem:   INFO:  R(4): 0101d0: 00004080  Struct  [@0101c0 +000010 CLI] CLI+16 = cli_CommandName(BSTR)+0

So a read access to memory address 0x0102a8 is detected as structure access to ThisTask.pr_CurrentDir. That’s what I call comfortable debugging ;)  Note that the labelling look ups cost quite a lot of performance and that’s the reason why you have to enable memory tracing with a command line switch (-t/-T).

The most common use of Structures is to decode lib call arguments. In Python I provide a so called AccessStruct object that assist reading/writing structures:

def StackSwap(self, lib, ctx):
 stsw_ptr = ctx.cpu.r_reg(REG_A0)
 stsw = AccessStruct(ctx.mem, StackSwapDef, struct_addr=stsw_ptr)
 # get new stack values
 new_lower = stsw.r_s('stk_Lower')
 new_upper = stsw.r_s('stk_Upper')
 new_pointer = stsw.r_s('stk_Pointer')

This is an excerpt of the Python implemenation of the Exec StackSwap call: In A0 a pointer to a StackSwap structure is passed in. We have already declared the function in Python as a defintion (hence the trailing Def):

class StackSwapStruct(AmigaStruct):
 _name = "StackSwap"
 _format = [
 ('APTR', 'stk_Lower'),
 ('ULONG', 'stk_Upper'),
 ('APTR', 'stk_Pointer')
 ]
StackSwapDef = StackSwapStruct()

With a memory pointer and a structure definition we can create an access object. Now the reads and writes to the entries of the structure a simple calls can be performed with the entry name of the structure… Ok its slower than direct offset handling but far more confortable and less error prone…

Beside the structure access vamos also provides a more generic memory access object that allows to read and write blocks of data, reads c-type strings and bcpl strings with a single call:

def OpenLibrary(self, lib, ctx):
 ver = ctx.cpu.r_reg(REG_D0)
 name_ptr = ctx.cpu.r_reg(REG_A1)
 name = ctx.mem.access.r_cstr(name_ptr)
 ..

That’s it for today… I hope you got more insight on the Library and Struct handling and the confortable features available in vamos for working with them.