I had the idea for this tool while changing the CF flash card of my A1200. I removed the old card, had look at the files found there and wanted to retrieve files from there and then set up a new shiny card and build up partitions there. The current way to accomplish this, is to dump the card’s raw data from the block device and use this image as a RDB hard disk image in UAE to retrieve the files from there. Same thing with partitioning the new card: mount the block device or an empty image in UAE, run HDToolBox there, format partitions and copy files around in the virtual Amige environment… This works, but its a roundabout way. I wanted to have a nifty tool the works directly on my Mac’s Terminal…

Read on to see how this task (and lots more!) can be achieved with rdbtool…

Inspecting an old CF Card

Ok, here we go: I unmount the CF card from the A1200 and insert it into my CF card USB reader on my Mac. Mac OS X tells me that it can’t find a valid file system on it (and that’s true for osx ) – we ignore this and cancel the request to initalize (= format) it. First thing is now to find out the block device for this disk. On OS X I call a:

> diskutil list

and it gives me:

/dev/disk2
 #:                       TYPE NAME                    SIZE       IDENTIFIER
 0:                                                   *4.0 GB     disk2

The empty Type Name gives you the hint that the partioning format is not recognized by OS X! So, it’s /dev/disk2… On Linux or other Posix systems the naming of raw block devices is slighty different (e.g. /dev/sd?). Just have a look at your system manuals (or dmesg) for details…

Now I use rdbtool to list the partition setup directly on the card:

> rdbtool /dev/disk2 info
PhysicalDisk:               0     7817     7880544  3.8Gi  heads=16 sectors=63
LogicalDisk:                2     7817     7878528  3.8Gi  rdb_blks=[0:2015,#2016] used=[hi=60,#61] cyl_blks=1008
Partition: #0 'CDH0'        2      103      102816   50Mi    1.31%  DOS3 bootable pri=0
Partition: #1 'DH0'       104      205      102816   50Mi    1.31%  DOS3
Partition: #2 'DH1'       206     2035     1844640  900Mi   23.41%  DOS3
Partition: #3 'DH2'      2036     3763     1741824  850Mi   22.11%  DOS3
Partition: #4 'DH3'      3764     3909      147168   71Mi    1.87%  DOS3
Partition: #5 'CDH1'     3910     3971       62496   30Mi    0.79%  DOS3
Partition: #6 'DH4'      3972     4124      154224   75Mi    1.96%  DOS3
Partition: #7 'DH5'      4125     5953     1843632  900Mi   23.40%  DOS3
Partition: #8 'DH6'      5954     7817     1878912  917Mi   23.85%  DOS3
FileSystem #0 DOS1 version=40.1 size=24588 seg_list_blk=0xb global_vec=0xffffffff

Wow! Got a lot of partitions there… With the drive names (DH0, …) at hand you use xdftool to access the file system on each partition (I use -r option here to enable read-only mode – just for safety reasons):

> xdftool -r /dev/disk2 open part=DH1 + list
Games 1                                          VOLUME  --------  29.03.1993 11:35:33 t46 
  Disk.info                                         364  ----rw-d  29.03.1993 11:35:33 t47
...

With the unpack command of xdftool I was able to easily recover all files of a partition into a directory structure on my Mac.

The info output also showed that a file system driver was also embedded in the RDB structure. You can use the fsget command to retrieve the Amiga loadSeg()able binary:

> rdbtool /dev/disk2 fsget 0 ffs

0 is the number of the first and only file system driver on this disk and “ffs” is the local Mac file name for the binary:

> ls -la ffs
-rw-r--r--  1 chris  staff  24588  6 Apr 13:48 ffs

You can also have a closer look of the internals of the RDB structure with the map and show commands. Map shows the contents of the RDB area and describes the contents of each block (see rdbtool page for details) while show dumps all blocks describing the RDB:

> rdbtool /dev/disk2 map
000000:  RD P0 P1 P2 P3 P4 P5 P6 P7 P8 F0 F0 F0 F0 F0 F0
000016:  F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0
000032:  F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0
000048:  F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 F0 -- -- --
...

> rdbtool /dev/disk2 show
RigidDiskBlock(0):
 types:     5244534b/0 (valid: 5244534b/0)
 chksum:    0x7471bb06 (got) 0x7471bb06 (calc)
 valid:     True
 size:           64
 host_id:        7
 block_size:     512
 flags:          0x00000017
 badblk_list:    none
 part_list:      1
 fs_list:        10
 init_code:      none
...

While working on a real disk you do not want to alter, it is advisable to either use the -r switch to enable read-only mode in rdbtool or simply copy the whole disk into a disk image and work on the image (if you have the disk space available, of course!):

> dd if=/dev/disk2 of=disk.rdb bs=512
> rdbtool disk.rdb list

That’s it for inspecting an existing RDB disk. Now let’s create an own, new one…

Partitioning a new Disk

Insert a shiny new CF card into your card reader. Typically some FAT file system already resides on a partition of the card. First thing is to remove the existing partioning layout. On OSX I do (for /dev/disk1):

> diskutil partitionDisk disk1 free none 100%

Now rdbtool can access the disk. The first command we will issue is an init, it will create an empty RDB block on the disk. The info will show an empty layout:

> rdbtool /dev/disk1 init + info
PhysicalDisk:               0    61999     1984000  968Mi  heads=1 sectors=32
LogicalDisk:                1    61999     1983968  968Mi  rdb_blks=[0:31,#32] used=[hi=0,#1] cyl_blks=32

A logical RDB disk is created automatically but no partition exist yet. So let’s add some! The add command adds a partition and only requires a size given either in percent (20%), bytes (2Mb), KiBytes (2Mib), or cylinders (20). If no start is given then the next free region on the logical disk is chosen:

> rdbtool /dev/disk1 add size=50Mib + add size=100Mib + fill + info
PhysicalDisk:               0    61999     1984000  968Mi  heads=1 sectors=32
LogicalDisk:                1    61999     1983968  968Mi  rdb_blks=[0:31,#32] used=[hi=3,#4] cyl_blks=32
Partition: #0 'DH0'         1     3200      102400   50Mi    5.16%  DOS3
Partition: #1 'DH1'      3201     9600      204800  100Mi   10.32%  DOS3
Partition: #2 'DH2'      9601    61999     1676768  818Mi   84.52%  DOS3

Note the fill command: It will create a partition to fill the remaining space on the logical disk.

You can alter settings of a partition with the change command: I use it to make the first partition bootable and set its priority:

> rdbtool /dev/disk1 change DH0 bootable pri=10 + info
...
Partition: #0 'DH0'         1     3200      102400   50Mi    5.16%  DOS3 bootable pri=10
...

(See rdbtool manual page for more options). You can also pass these options to the add command in the previous step for a more compact usage of the tool.

The partition layout is already finished, the only thing I’d like to add to my RDB is the FFS file system driver restored from the old card (see above):

> rdbtool /dev/disk1 fsadd ffs + info
ERROR adding filesystem! (no space in RDB left)

Oops! There is not enough space left for the file system. I have to recreate the RDB with more space reserved for the filesystem (see rdb_cyls = reserve a number of cylinders for RDB usage):

> rdbtool /dev/disk1 init rdb_cyls=2 + add size=50Mib bootable pri=10 + add size=100Mib + fill + fsadd ffs + info
PhysicalDisk:               0    61999     1984000  968Mi  heads=1 sectors=32
LogicalDisk:                2    61999     1983936  968Mi  rdb_blks=[0:63,#64] used=[hi=54,#55] cyl_blks=32
Partition: #0 'DH0'         2     3201      102400   50Mi    5.16%  DOS3 bootable pri=10
Partition: #1 'DH1'      3202     9601      204800  100Mi   10.32%  DOS3
Partition: #2 'DH2'      9602    61999     1676736  818Mi   84.52%  DOS3
FileSystem #0 DOS1 version=40.1 size=24588

Voila! Everything set up in a single command

RDB work is done now! (See rdbtool page for lots of more commands available in this tool). Now let’s head over to xdftool and format the partitions:

> xdftool /dev/disk1 open part=dh0 + format System
> xdftool /dev/disk1 open part=dh1 + format Work
> xdftool /dev/disk1 open part=dh2 + format Data

Now everything is partitioned and formatted for AmigaDOS use. Typically, you want to boot from the first partition, so its advisable to fill it now. In my case I used the pack command of xdftool to restore the Workbench files I recovered with unpack from my old card…

With this approach, setting up an Amiga disk drive on your Mac is done in seconds with rdbtool and xdftool. No need to boot up an emulator, just plain native tools I hope you enjoyed this little introduction and find the tools as useful as I do…

• added support for RBD/RDSK hdf images
• changed repack command to be more flexible
• added open/create commands for better control of disk image geometry

In the following I give you a short round up of all new features by giving you some examples…

1. RDB Support

One very nasty thing when working with HDF image files is the fact that the extension .hdf is actually used for two types of different disk images in the context of Amiga emulation.

The first type is a disk image like a big floppy disk dump, i.e. it only contains a number of blocks and on these blocks an Amiga file system (like OFS or FFS) is located. Thats the way a floppy handles data. But in comparison with a floppy disk, a HDF has no pre-defined or fixed size. You can use every size for a HDF if you like. Now the drawback with these images is that you do not know the disk geometry (given in cylinders, heads, and sectors) directly but only the total size in bytes or blocks. You have to apply some magic (here: algorithms) to deduce the correct geometry. So this type of disk image is only fully useable if you know the disk geometry as external information. I call this type a partition HDF image because this type of image can be created by extracting the contents of a disk partition into an image file.

The other type I call a whole disk HDF image because it stores the whole contents of the disk. This includes some blocks describing the paritions found on this disk. This information starts with a RDISK/RDB (Rigid Disk Block) and thus images containing paritioning informations on an Amiga called like this first block RDB. The RDB references information on all partitions and knowns how large they are, what disk geometry the use locally and what file systems reside on them. Additionally, each partition has a AmigaOS device name associated so you can address them on your Amiga (e.g. dh0, dh1, …). An RDB is typically set up on a blank disk with the HDToolBox program on AmigaOS (soon you can use amitools’ rdbtool for that – but thats another story/blog post ).

I distuingish these two types by naming my images differently: partition HDFs with *.hdf and disk images with *.rdsk, but thats my personal approach and unfortonately not widely used.

I added support of these RDB images to xdftool fully transparently, i.e. you simply specify a *.hdf file of any of the two types and xdftool will do the right thing. If its a partition HDF then there is only one file system and you can work with this one immediately:

> xdftool wb31.hdf blkdev
cylinders:   1280
heads:       1
sectors:     32
block_bytes: 512
reserved:    2
bootblocks:  2

This command prints the disk geometry detected from the byte size of the disk image. If the detection went wrong you can specify the new open command and give the missing parameters:

> xdftool wb31.hdf open chs=640,2,32 + blkdev
cylinders:   640
heads:       2
sectors:     32
block_bytes: 512
reserved:    2
bootblocks:  2

This forces a disk geometry of 10 cylinders, 2 heads and 32 sectors. Another approach is to still use auto-detection but guide it by giving the heads or/and sectors count:

> xdftool wb31.hdf open h=2 + blkdev
cylinders:   640
heads:       2
sectors:     32
block_bytes: 512
reserved:    2
bootblocks:  2

With a RDB/RDSK whole disk image you do not need to specify any geometry as this information is stored in the RDB structure. The only thing you have to specify is what partition shall xdftool work on. As xdftool only works on file system level you have to choose the partition either by giving the device name of AmigaOS (e.g. dh0, dh1) or the index number (starting with 0 for the first partition):

> xdftool wb31.rdsk open part=dh1 + list

This list the contents of the second partition on my wb31 whole disk image (being dh0 the first partition). If you do not specify a partition then xdftool automatically takes the first partition:

> xdftool wb31.rdsk list

2. New Create Command

With the new open command in place I added a new create command to create empty disk images of a given size. For ADFs the size is pre-defined so you need no extra arguments:

> xdftool empty.adf create   # create empty (and unformatted) floppy disk image

but for a (partition) HDF you have to give the size. You can either give a size in bytes (with short cuts for M)ega G)iga… and ‘i’ for KiBi units) and use the internal algorithm for disk geometry or directly specify “chs” (cylinder, heads, and sectors):

> xdftool empty.hdf create size=10Mi
> xdftool empty.hdf create chs=10,1,32
> xdftool empty.hdf create size=10Mi h=2

In the last example you see that you can also hint the disk geometry algorithm and specify the heads count (similar to the open command above).

Note: you will need to format this image before usage

2nd Note: you cannot create a full RDB/RDSK image with xdftool. Use HDToolBox on your Amiga (or soon: rdbtool) for that!

Typically You use a create command followed by a format command to create a file system you can work with:

> xdftool new.adf create + format MyDisk
> xdftool work.hdf create size=10Mi + format Work ffs

As this is used very often the format can do an implicit create if you simply omit the create command. Then you have to specify the options for create with the format command:

> xdftool new.adf format MyDisk
> xdftool work.hdf format Work size=10Mi ffs

3. Repack Command Changed

With the rework done handling rdisk and hdf I had to change the existing repack command. In older xdftool version you specify a new image as argument and the given image on the xdftool command line was the source. This is now flipped:

> xdftool old.hdf repack new.hdf   # OLD syntax
> xdftool new.hdf repack old.hdf  # NEW syntax

I changed it to make its usage more versatile: You can give open like options (see above) in the repack command to specify the source image:

> xdftool new.hdf repack old.rdsk part=dh0  # repack first partition into new partition image
> xdftool new.hdf repack old.hdf chs=10,2,32   # repack old image with given geometry

(Note: if you get an error that the new image can’t created then either remove the existing file first or specify the -f force option). Now you can also create a new image with different size or layout and repack all data into it:

> xdftool new.hdf create size=10Mi + repack wb31.hdf  # move my wb disk into a larger image
> xdftool disk.rdsk open part=dh0 + repack wb31.hdf # move my wb disk into a partition

That’s it! Have Fun!

While building the code is done automatically in a Makefile, the disk image creation still involves manual steps including launching an UAE emulator to create the disk image. Hmm, I thought, an ADF file mastering tool (like mkisofs is used for CDs) would be a great tool!! Adding this to my Makefile would fully automates my build cycle… I was aware of ADFlib as being the portable library for ADF manipulation and I had a look there, but I did not found a mastering tool. Some more googling didn’t show me a similar tool – so again – I was left on my own and had to create the tool myself

As a result xdftool was born and added to my amitools tool set! I started writing a small tool using the Python binding of ADFlib to create an ADF image and copy files there. This worked really quickly but soon I found the limitations of this library: the mastering part is only partially supported and essential things like setting all meta infos of a file were missing (or I didn’t find them . In the end I dropped the ADFlib approach and started to build a FS library for Amiga OFS/FFS from scratch… Thanks to the excellent ADF Disk Format FAQ I soon had all necessary information available and some hours later the first code files written.

The lib is done now and included in amitools as “fs” module. In its first incarnation it already has an impressive feature set:

• full object-oriented API in Python
• supports ADF and HDF containers
• supports all OFS/FFS modes including international and dircache
• read/write files/dir trees from/to host file system
• multi-layer support: work on block level or on FS level
• query and modify all parameters found in the file system including comments, protection flags and all time stamps in tick resolution
• supports unpacking/packing/repacking of full images into host file system with meta db files to store information not available on host file system

xdftool now supports all features of the library and while it started as a mastering tool for ADFs it is now a full featured command line tool to work with all kinds of ADF and HDF images… The remainder of this post gives you a short tutorial what you can do with xdftool:

First of all install xdftool. It is a part of amitools so have a look at my amitools page for installation details. It is handy to pack the top-level dir of amitools into your PATH so you have all tools available in your shell at any place.

In the following I will use a Workbench 3.1 disk image called wb31.adf which was taken from Cloanto’s Amiga Forever Pack. You can use any other disk image as well. Simply adjust the paths if necessary. Host files are taken from the top level directory of amitools.

This tutorial only shows a sub set of functions available in xdftool. Have a look at the xdftool Homepage for a full reference of commands.

Look at a Disk

> xdftool wb31.adf list
Workbench3.1                                    VOLUME  --------  06.07.1986 14:32:19 t08 
 C                                                 DIR  ----rwed  22.06.1991 02:13:25 t18 
 AddBuffers                                        444  --p-rwed  06.07.1986 14:39:06 t31 
 AddDataTypes                                     5880  --p-rwed  06.07.1986 14:39:06 t37 
 Assign                                           3220  --p-rwed  06.07.1986 14:39:06 t43 
 Avail                                             736  --p-rwed  06.07.1986 14:39:06 t49 
 BindDrivers                                      1420  ----rwed  06.07.1986 14:39:07 t04 
 Break                                             432  --p-rwed  06.07.1986 14:39:07 t16 
 ChangeTaskPri                                     460  --p-rwed  06.07.1986 14:39:07 t21 
 ConClip                                          2432  --p-rwed  06.07.1986 14:39:07 t29 
...
T                                                  DIR  ----rwed  22.06.1991 02:06:08 t25 
 Utilities                                         DIR  ----rwed  06.07.1986 14:39:05 t07 
 Clock                                           13856  ----rwed  06.07.1986 14:39:04 t31 
 Clock.info                                        590  ----rw-d  06.07.1986 14:39:04 t37 
 More                                            12752  --p-rwed  06.07.1986 14:39:04 t43 
 MultiView                                       28836  --p-rwed  06.07.1986 14:39:05 t01 
 MultiView.info                                    861  ----rw-d  06.07.1986 14:39:05 t11 
 Utilities.info                                    632  ----rw-d  06.07.1986 14:39:06 t14 
 WBStartup                                         DIR  ----rwed  06.07.1986 14:39:05 t19 
 WBStartup.info                                    632  ----rw-d  06.07.1986 14:39:06 t21 
Blocks:    data:     1496, fs:      199, sum:     1695, ratio=88.26, disk total:     1760
Bytes:     data:   765952, fs:   101888, sum:   867840, ratio=88.26, disk total:   901120
File:      data:   726742, ratio=94.88

Ok, first command shows you whats on a disk! Note the calling convention in xdftool: always pass the image file name first then the command with parameters. You see the complete directory tree of the volume inside the disk image. Each entry shows you the size, protection flags, modification time stamps (with ticks) and if available the file comment. At the end a summary on block usage is given.

Create a disk and write to it

> xdftool mydisk.adf format MyDisk + boot install + list
MyDisk                                         VOLUME  --------  14.01.2012 21:49:13 t00 
Blocks:    data:        0, fs:        1, sum:        1, ratio=0.00, disk total:     1760
Bytes:     data:        0, fs:      512, sum:      512, ratio=0.00, disk total:   901120
File:      data:        0, ratio=0.00

With this command creating an empty formatted image is a no-brainer… Note that you can combine multiple commands in a single call to xdftool by separating them with a plus sign. Creating a hard disk image is also very easy:

> xdftool myhd.hdf format Work 10M

Here a size (or a disk geometry) needs to be specified.

You can put some files from your host on it with the ‘write’ command:

> xdftool mydisk.adf write README + list
test                                            VOLUME  --------  14.01.2012 21:51:42 t00 
 README                                           3728  ----rwed  14.01.2012 21:51:42 t00 
Blocks:    data:        8, fs:        2, sum:       10, ratio=80.00, disk total:     1760
Bytes:     data:     4096, fs:     1024, sum:     5120, ratio=80.00, disk total:   901120
File:      data:     3728, ratio=91.02

This will write the file README from the host’s current directory to the volume directory of the image. You can also write a full directory tree to the image:

> xdftool mydisk.adf write doc + list
test                                            VOLUME  --------  14.01.2012 21:53:20 t00 
 doc                                               DIR  ----rwed  14.01.2012 21:53:20 t00 
   xdftool.txt                                   18798  ----rwed  14.01.2012 21:53:20 t00 
Blocks:    data:       39, fs:        3, sum:       42, ratio=92.86, disk total:     1760
Bytes:     data:    19968, fs:     1536, sum:    21504, ratio=92.86, disk total:   901120
File:      data:    18798, ratio=94.14

Now you can alter the protect flags, the modification time or set a comment:

> xdftool mydisk.adf protect doc rwe + time doc "12.01.1991 12:00:00 t11" + comment doc "my doc comment" + list
test                                            VOLUME  --------  14.01.2012 21:53:20 t00 
 doc                                               DIR  ----rwe-  12.01.1991 12:00:00 t11  my doc comment
   xdftool.txt                                   18798  ----rwed  14.01.2012 21:53:20 t00 
Blocks:    data:       39, fs:        3, sum:       42, ratio=92.86, disk total:     1760
Bytes:     data:    19968, fs:     1536, sum:    21504, ratio=92.86, disk total:   901120
File:      data:    18798, ratio=94.14

With these commands at hand you can now easily write a Makefile entry that creates an ADF image and places all your files on it.

But xdftool can more:

Unpacking and Packing Images

The unpack command extracts all files and directories from a disk image and creates a corresponding directory structure on your host’s file system. Additionally a MetaDB file will be created with all flags of the Amiga file system that cannot be represented in the host file system, namely protect flags, timestamps with ticks and comments:

> xdftool wb31.adf unpack .
> ls Workbench3.1*
Workbench3.1.blkdev    Workbench3.1.bootcode  Workbench3.1.xdfmeta

Workbench3.1:
C/              Devs.info       Expansion.info  Prefs/          S/              T/              WBStartup/
Classes/        Disk.info       L/              Prefs.info      System/         Utilities/      WBStartup.info
Devs/           Expansion/      Libs/           Rexxc/          System.info     Utilities.info

You see a directory named after the volume inside the image was created with all files and dirs of the image. Additionally the MetaDB file *.xdfmeta, the boot code in *.bootcode and the description of the disk’s block device geometry *.blkdev was written.

With these files in place you can always recreate a new disk image that is equivalent to the original image (from the FS perspective):

> xdftool newimg.adf pack Workbench3.1 + list
Workbench3.1                                    VOLUME  --------  06.07.1986 14:32:19 t08 
 C                                                 DIR  ----rwed  22.06.1991 02:13:25 t18 
 AddBuffers                                        444  --p-rwed  06.07.1986 14:39:06 t31 
 AddDataTypes                                     5880  --p-rwed  06.07.1986 14:39:06 t37 
 Assign                                           3220  --p-rwed  06.07.1986 14:39:06 t43 
...
   MultiView.info                                  861  ----rw-d  06.07.1986 14:39:05 t11 
 Utilities.info                                    632  ----rw-d  06.07.1986 14:39:06 t14 
 WBStartup                                         DIR  ----rwed  06.07.1986 14:39:05 t19 
 WBStartup.info                                    632  ----rw-d  06.07.1986 14:39:06 t21 
Blocks:    data:     1496, fs:      199, sum:     1695, ratio=88.26, disk total:     1760
Bytes:     data:   765952, fs:   101888, sum:   867840, ratio=88.26, disk total:   901120
File:      data:   726742, ratio=94.88

Note the correct time stamps and protect flags after packing…

With this approach you can quickly unpack an image, modify some files on your host file system and pack the image again.

Also mastering full images is now easy: create a Volume directory from scratch, place all your files in it and if you need special flags then write a *.xdfmeta file. Call xdftool pack to create your image. Done!

Repacking Images

Another nice operation in xdftool is repack: This essentially combines a unpack command on an existing image with a pack command on a new image. So you can rebuild the contents of the whole volume of the old image and write a fresh new FS structure. This will recreate the FS and re-aligns all file data blocks to be in one place. Originally this command helps me to test the integrity of amitools fs library but it may be useful for you, too:

> xdftool wb31.adf repack new.adf

In combination with a hard disk image repack also allows you to create a larger target disk, so this command essentially allows you to “grow” your Amiga hard disk by repacking its image to a larger one:

> xdftool old.hdf repack new.hdf 10M

You can even repack a disk image into a hard disk image:

> xdftool wb31.adf repack wb31.hdf 10M

That’s it… If you want to play more with xdftool have a look at the xdftool page!

Have Fun! And if you find bugs in the rather fresh code then please don’t hesitate to report them… Thanks!

In this post I will show you how you can build the magPLIP network driver I used in my plip2slip project directly on your Mac without using an Amiga machine emulator like P-UAE…

1. Setup Build Environment

• I assume that you have installed the SAS C compiler ready to use for vamos as described in my last post
• The netword driver we will compile needs the Amiga Network Driver Includes (i.e. SANA-II headers)
  • Download the AmiTCP-SDK-4.3.lha from aminet
  • You can unpack it on your Mac with the lha command available in MacPorts (call: sudo port install lha if the tool is still missing on your machine):

> cd ~/amiga/src
> lha x ~/Downloads/AmiTCP-SDK-4.3.lha

• Now we need to setup assigns in vamos so the build can reach the new network includes. My ~/.vamosrc config file looks like this:

[volumes]
wb310=~/amiga/wb310
sc=~/amiga/shared/sc
net=~/amiga/src/AmiTCP-SDK-4.3

[assigns]
include=sc:include
lib=sc:lib
t=root:tmp
c=wb310:c
netinclude=net:netinclude
netlib=net:netlib
includeasm=sc:include

[path]
path=sc:c,wb310:c

[vamos]
quiet=True
ram_size=2048

What’s new here?

• First I added a new volume in vamos called net: where the SANA-II includes reside. It points to the native path we just unpacked.
• Some new assigns were added: netinclude and netlib point inside our new net: volume
• Another assign called includeasm: was necessary as the following build requires it. It simply points directly to the include directory in the SAS C installation.
• Finally, a new section called vamos was introduced. This section holds defaults for common option that were currently only available via command line:
  • The quiet switch is similar to the -q command line option and suppresses information about un-implemented Amiga OS calls in vamos
  • The ram_size sets the RAM for the AmigaOS emulation of vamos to 2 MiB. Similar to the -m command line option. This ensures that we do not run out of memory during compilation.

2. Grab the Source

• Next we download the source for magPLIP from aminet
• I unpacked the source in a build directory:

> mkdir ~/build
> cd build
> lha x ~/magPLIP38.1.lha
> ls
magplip/      magplip.info

• I also downloaded the plip2slip-0.1 source archive because it contains a patch for the driver we are going to build
• Unpack this, too:

> unzip ~/Downloads/plip2slip-0.1.zip
> ls
magplip/       magplip.info   plip2slip-0.1/

• Next, we patch the source:

> patch -p0 < plip2slip-0.1/contrib/magplip/magplip.patch
patching file magplip/source/magplip_rev.h
patching file magplip/source/magplip_rev.i
patching file magplip/source/magport.asm
patching file magplip/source/server.c
patching file magplip/source/smakefile

• That’s it, now the source is ready to be compiled!

3. Perform the Build

• The build itself is now rather unspectacular… It simply works

> cd magplip/source
> vamos smake all_opt
SAS/C_SMAKE 6.58 (27.12.96)
Copyright (c) 1988-1995 SAS Institute, Inc.
 sc:c/sc INCDIR="INCLUDEASM:" INCDIR="include:" INCDIR="netinclude:" VERBOSE DEFINE MAGPLIP=1 OBJECTNAME MAGrt.o rt.asm
No default scoptions file found
SAS/C Amiga Compiler 6.58
Copyright (c) 1988-1995 SAS Institute Inc.

asm "-dMAGPLIP=1" -iINCLUDEASM: -iinclude: -inetinclude: -oMAGrt.o rt.asm
...

SLINK Complete - Maximum code size = 8824 ($00002278) bytes

Final output file size = 8940 ($000022ec) bytes

• If everything went fine then the resulting binary resides in the magplip/devs/networks folder:

> ls ../devs/networks/magplip.device.000
-rwxr-xr-x  1 chris  staff  8940 18 Dez 18:16 ../devs/networks/magplip.device.000*

• Ah! A fresh build of the driver… Now let’s create a 68020 version, too:

> vamos smake CPUSUFFIX=020 all_opt
...
> ls ../devs/networks/magplip.device.020
-rwxr-xr-x  1 chris  staff  8872 18 Dez 18:21 ../devs/networks/magplip.device.020*

• Finally, you can compare your new builds with the reference compiles I did on P-UAE and that are available in the plip2slip release:

> cmp ../devs/networks/magplip.device.000 ../../plip2slip-0.1/contrib/magplip/magplip.device.000
> cmp ../devs/networks/magplip.device.020 ../../plip2slip-0.1/contrib/magplip/magplip.device.020

• It really works! Same output generated on vamos!!
• Now its time to read the plip2slip docs on how to actually use the driver on your real machine…

4. Some additional Notes

• While vamos now supports the typical builds on SAS C even with smakefiles, there are still things missing. Most notably, the internal Shell commands like Echo are not emulated yet. So the default/help rule in our project’s smakefile does not work. But the internal Shell commands and a more complete support for the typical CLI commands will be the next on my TODO list…
• Compared to last vamos versions I optimized the loading of native libraries (most notably sc1.library and sc2.library of sc) and that gave a real speed boost when calling the compiler.
• To better judge where the bottlenecks in the vamos emulation are, I added profiling support for the library calls. You can use the new -P switch and give the libraries you want to get detailed information for:

> vamos smake clean # first clean our build again
...
> vamos -v -P exec,dos smake all_opt # now build with profiling on
...
18:34:09.103       prof:   INFO:  'dos.library' Function Call Profile
18:34:09.103       prof:   INFO:                   Close: #     251  total=    0.0115  per call=    0.0000
18:34:09.103       prof:   INFO:              CurrentDir: #      45  total=    0.0009  per call=    0.0000
18:34:09.103       prof:   INFO:               DateStamp: #       9  total=    0.0005  per call=    0.0001
18:34:09.103       prof:   INFO:              DeleteFile: #      12  total=    0.0033  per call=    0.0003
18:34:09.103       prof:   INFO:                 DupLock: #       3  total=    0.0003  per call=    0.0001
18:34:09.103       prof:   INFO:                 Examine: #      85  total=    0.0158  per call=    0.0002
18:34:09.104       prof:   INFO:                FilePart: #       3  total=    0.0002  per call=    0.0001
18:34:09.104       prof:   INFO:                FreeArgs: #       6  total=    0.0017  per call=    0.0003
18:34:09.104       prof:   INFO:          FreeDeviceProc: #       3  total=    0.0005  per call=    0.0002
18:34:09.104       prof:   INFO:           GetDeviceProc: #       3  total=    0.0008  per call=    0.0003
18:34:09.104       prof:   INFO:                   Input: #      20  total=    0.0001  per call=    0.0000
18:34:09.104       prof:   INFO:                   IoErr: #      94  total=    0.0006  per call=    0.0000
18:34:09.104       prof:   INFO:                    Lock: #      77  total=    0.0163  per call=    0.0002
18:34:09.104       prof:   INFO:                MatchEnd: #       6  total=    0.0019  per call=    0.0003
18:34:09.104       prof:   INFO:              MatchFirst: #       6  total=    0.0055  per call=    0.0009
18:34:09.104       prof:   INFO:               MatchNext: #       6  total=    0.0002  per call=    0.0000
18:34:09.104       prof:   INFO:                    Open: #     352  total=    0.0850  per call=    0.0002
18:34:09.104       prof:   INFO:                  Output: #      37  total=    0.0003  per call=    0.0000
18:34:09.104       prof:   INFO:               ParentDir: #      62  total=    0.0061  per call=    0.0001
18:34:09.104       prof:   INFO:            ParsePattern: #       3  total=    0.0006  per call=    0.0002
18:34:09.104       prof:   INFO:                  PutStr: #       9  total=    0.0005  per call=    0.0001
18:34:09.104       prof:   INFO:                    Read: #     617  total=    0.0273  per call=    0.0000
18:34:09.105       prof:   INFO:                ReadArgs: #       6  total=    0.0038  per call=    0.0006
18:34:09.105       prof:   INFO:                  Rename: #       3  total=    0.0013  per call=    0.0004
18:34:09.105       prof:   INFO:                    Seek: #     155  total=    0.0048  per call=    0.0000
18:34:09.105       prof:   INFO:                SetIoErr: #       3  total=    0.0000  per call=    0.0000
18:34:09.105       prof:   INFO:           SetProtection: #       4  total=    0.0009  per call=    0.0002
18:34:09.105       prof:   INFO:           SystemTagList: #      16  total=    0.0486  per call=    0.0030
18:34:09.105       prof:   INFO:                  UnLock: #     108  total=    0.0026  per call=    0.0000
18:34:09.105       prof:   INFO:                   Write: #     277  total=    0.0152  per call=    0.0001
18:34:09.105       prof:   INFO:  sum total=0.2570
18:34:09.105       main:   INFO:  done 3218549052 cycles in host time 14.9556s -> 174.60 MHz m68k CPU
18:34:09.106       main:   INFO:  code time 14.9556s (81.13 %), trap time 3.4781s (18.87 %) -> total time 18.4337s
18:34:09.106       main:   INFO:  exit code=0
18:34:09.107       prof:   INFO:  'exec.library' Function Call Profile
18:34:09.107       prof:   INFO:                AllocMem: #    1266  total=    0.1440  per call=    0.0001
18:34:09.107       prof:   INFO:                AllocVec: #      39  total=    0.0024  per call=    0.0001
18:34:09.107       prof:   INFO:            CloseLibrary: #      85  total=    0.0129  per call=    0.0002
18:34:09.107       prof:   INFO:                FindTask: #     137  total=    0.0024  per call=    0.0000
18:34:09.107       prof:   INFO:                  Forbid: #     137  total=    0.0004  per call=    0.0000
18:34:09.107       prof:   INFO:                 FreeMem: #    1318  total=    0.4038  per call=    0.0003
18:34:09.107       prof:   INFO:                 FreeVec: #      39  total=    0.0059  per call=    0.0002
18:34:09.107       prof:   INFO:                  GetMsg: #     650  total=    0.0082  per call=    0.0000
18:34:09.107       prof:   INFO:             OpenLibrary: #      85  total=    0.1304  per call=    0.0015
18:34:09.107       prof:   INFO:                  Permit: #     137  total=    0.0004  per call=    0.0000
18:34:09.107       prof:   INFO:                  PutMsg: #     650  total=    0.0794  per call=    0.0001
18:34:09.108       prof:   INFO:                RawDoFmt: #      11  total=    0.0006  per call=    0.0001
18:34:09.108       prof:   INFO:               SetSignal: #  103942  total=    0.9651  per call=    0.0000
18:34:09.108       prof:   INFO:               StackSwap: #      34  total=    0.0030  per call=    0.0001
18:34:09.108       prof:   INFO:  sum total=1.7587

• That’s it for today! Enjoy vamos and if you already use it for doing your own “cross”-compiles then I’d really like to hear from your projects…

That was the beginning of my newes project: vamos – The Virtual AMiga OS emulator.

Read on to learn more about the birth of vamos and its first major milestone: run the SAS C compiler in my new “Developer’s Diary” series

Mission Statement

If you know Wine then the goal is fairly clear: write a native Mac OS X program that “executes” the Amiga binary, trap all native library calls (to Exec or DOS libary) and replace them with native versions, i.e. an Open() call to Amiga’s dos.library will directly open a file on my Mac.

I want to focus on Exec and Dos library as those provide the functions required for typical command line tools including compilers. I will start with all trapped Exec/Dos libs with empty functions and then step-by-step fill in the functions called by my first target tool: the SAS C compiler…

The thing in vamos that’s more difficult than in Wine is the machine architecture that differs: the Amiga binary is written in m68k code and my Mac runs an Intel x86 based CPU… So the first thing vamos will need is a m68k CPU emulation. I searched the web and found the Musashi CPU emulator.

First Experiment

I wanted to write (the first prototype of) vamos in Python 2.x as it allows to me to quickly get started and let’s me play around with different approaches fairly fast. If everything works out and some portions are slow due to Python scripting I can gradually move them to C later on…

So the first thing was to attach the native m68k CPU emulator to Python. With the help of Python’s ctype module binding the native code was very easy. Now the only thing required to run this virtual CPU is memory. In a first attempt I routed all memory accesss from Musashi to Python and managed memory there, but that was too slow (a 0.2 MHz CPU ) and so I moved to a C-based memory implementation. Now a single memory “RAM block” is allocated (e.g. 1 MiB) and accessed by the virtual CPU (200 MHz CPU ).

The native Python interface now allows to create the memory, read/write data to the block and control the CPU: read/write its registers and let it run…

Now I was ready for a first experiment: I used my Python HunkReader from my amitools library to load a Amiga LoadSeg()able binary and relocate it into my still empty memory location. Then point the virtual CPU to the first instruction of the first code segment and let the CPU run and boom! To see how far we got I early added extensive memory debugging to have a close look what is fetched from memory… I got as far as the first read to exec base at location 4… so we need some environment to run our binary in vamos!

Setting up the Environment

Ok. We need the exec lib structure in our virtual memory filled with the essential parts, i.e. mainly the jump vectors to the lib functions. And then I need a method to trap the calls so that I know that the virtual CPU will call a library function and can redirect it to my Python code.

In a new experiment I crafted a faked Exec Lib structure in memory and for the call trapping I tried the RESET opcode. Musashi provides a callback whenever this opcode occurrs and I wanted to use this for the lib traps. The idea is to use the RESET callback to enter my Python code then look at the CPU’s PC to find out which entry of the jump vector was selected. In my python code I will emulate the effect of the lib call (or do nothing) and then the next opecode will be executed. I placed an RTS there so the jsr libcall will return to the caller.

So my lib jump table with a trap looks like this (compared to a original Amiga jump):

vamos lib entry: <RESET opcode.w> <RTS opcode.w> <lib_id.w>
amiga lib entry: <JMP opcode.w> <func_addr.l>

Both entries have 6 bytes so they are binary compatible… Good. The lib_id word will be used later to identify which lib was trapped…

In my Python code I fetch the PC and then retrieve the library base (here of EXEC) to call the offset of the trapped function. Then I know the function to be called and can emulate it. First thing here was to setup the full exec structure and add traces for all trapped lib calls…

My experiment was done with the Type command of Workbench 3.1… And voila! The code ran until a first call to Exec… Its OpenLibrary!

Stack and Terminate

I almost forgot the stack.. Both PC and initial stack have to be paced at address 0 and 4 before powering up the virtual CPU. PC is the first instruction of my loaded binary. For the stack I simply reserved some static memory region in my RAM block and set the stack pointer…

I can now imagine how running the code would work. But leaving vamos was still an open question: The Amiga binary is called by jsr()ing into the code and it will end with an RTS. So at the end of the code the last value is fetched from the stack and jumped there. So a simple solution for the termination problem was to initialise the stack with a jump to address 0 and there I placed another RESET opcode to trigger my callback. The callback was then extended to watch out for a PC of 0. In this case no lib trap occurred but the program wants to exit. I quit vamos then and termination was solved…

A very simply program only consisting of a empty main() code block was now able to run in my Mac based vamos enviroment! A more complex Amiga program was already able to fetch exec base and to do jumps into exec. Those jumps are trapped but the python code only does logging for now. The exchange of parameters in the trapped calls is done by readingw/writing the “virtual” CPU registers…

All in all I was very happy with these first results and really looked forward to get more Amiga code running on vamos…

To be continued…

That’s it for today… Next time we will have a look on more features of vamos.

If you can’t wait then just head over to the amitools page and download the vamos code there and play around with it… enjoy!

Following  the same approach as found in my dtv2ser project, I just attached an AVR ATmega microcontroller to the parallel lines of my Amiga and implemented the PLIP counterpart there. The ATmega on the popular Arduino 2009 boards has a fast serial connection that maps to a USB port via a FTDI 232. This is the ideal data channel for transporting the PLIP packets to your host Mac or PC. With baudrates up to 500 kBaud (~50 KiB/s) and hardware handshaking with RTS/CTS you can easily transfer the data packets very fast to your PC and encapsulate them again in SLIP.

With this idea I started to implement the plip2slip firmware. Fortunately, the magPLIP driver for the Amiga comes with source and so I could port the code to the AVR. With slight modifications on the Amiga part (patch included) I was able to receive the first packets from the Amiga on the Arduino. Then I implemented a simple Ping mode that receives ICMP Ping Requests transforms them to Replies and returns them to the Amiga: plip2slip soon was a ping machine I repeated the same thing for the SLIP side and with a patched slattach tool on Ubuntu Linux I was soon able to ping the AVR from this side, too.

With the basic parts of plip2slip in place and working I finally added the transport/bridging mode that transfers all received PLIP packets to SLIP and vice versa. With this mode running I could ping the Ubuntu host from my Amiga! After some IP forward configuration on Ubuntu I was able to reach my home network and also the Internet with the A500.

And its real fast! An FTP download on my Amiga reaches 25 KiB/s when transferring 10k from Linux! So compared to SLIP with 9600 Baud its a real break through and worth the little hardware effort needed to build the Arduino device…

There is still lots of potential for tuning and optimizing, but I wanted to share this little project as soon as possible with you. So I crammed up everything you need into a little 0.1 release that is available on my shiny new plip2slip homepage. I hope you enjoy it and bring lots of your classic machines back to the net with decent speed…