Mastering ADF/HDF Images with xdftool

My Amiga cross development environment on my Mac is getting really useful now: with vamos running the SAS C compiler I can create Amiga binaries with ease. With the binaries in place I want to try them on the real machine, too. For my trusty old Amiga 500, I still use disks to transfer the data. So I create an ADF image with my files on it and either use my kryoflux setup to write a real disk or write a virtual HFE disk image on an SD card to be used with the HXC2001 floppy emulator.

While building the code is done automatically in a Makefile, the disk image creation still involves manual steps including launching an UAE emulator to create the disk image. Hmm, I thought, an ADF file mastering tool (like mkisofs is used for CDs) would be a great tool!! Adding this to my Makefile would fully automates my build cycle… I was aware of ADFlib as being the portable library for ADF manipulation and I had a look there, but I did not found a mastering tool. Some more googling didn’t show me a similar tool – so again – I was left on my own and had to create the tool myself 😉

As a result xdftool was born and added to my amitools tool set! I started writing a small tool using the Python binding of ADFlib to create an ADF image and copy files there. This worked really quickly but soon I found the limitations of this library: the mastering part is only partially supported and essential things like setting all meta infos of a file were missing (or I didn’t find them ;). In the end I dropped the ADFlib approach and started to build a FS library for Amiga OFS/FFS from scratch… Thanks to the excellent ADF Disk Format FAQ I soon had all necessary information available and some hours later the first code files written.

The lib is done now and included in amitools as “fs” module. In its first incarnation it already has an impressive feature set:

full object-oriented API in Python
supports ADF and HDF containers
supports all OFS/FFS modes including international and dircache
read/write files/dir trees from/to host file system
multi-layer support: work on block level or on FS level
query and modify all parameters found in the file system including comments, protection flags and all time stamps in tick resolution
supports unpacking/packing/repacking of full images into host file system with meta db files to store information not available on host file system

xdftool now supports all features of the library and while it started as a mastering tool for ADFs it is now a full featured command line tool to work with all kinds of ADF and HDF images… The remainder of this post gives you a short tutorial what you can do with xdftool:

First of all install xdftool. It is a part of amitools so have a look at my amitools page for installation details. It is handy to pack the top-level dir of amitools into your PATH so you have all tools available in your shell at any place.

In the following I will use a Workbench 3.1 disk image called wb31.adf which was taken from Cloanto’s Amiga Forever Pack. You can use any other disk image as well. Simply adjust the paths if necessary. Host files are taken from the top level directory of amitools.

This tutorial only shows a sub set of functions available in xdftool. Have a look at the xdftool Homepage for a full reference of commands.

Look at a Disk

> xdftool wb31.adf list
Workbench3.1Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  VOLUMEÂ  --------Â  06.07.1986 14:32:19 t08Â 
 CÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  22.06.1991 02:13:25 t18Â 
 AddBuffersÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  444Â  --p-rwedÂ  06.07.1986 14:39:06 t31Â 
 AddDataTypesÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  5880Â  --p-rwedÂ  06.07.1986 14:39:06 t37Â 
 AssignÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  3220Â  --p-rwedÂ  06.07.1986 14:39:06 t43Â 
 AvailÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  736Â  --p-rwedÂ  06.07.1986 14:39:06 t49Â 
 BindDriversÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â Â  1420Â  ----rwedÂ  06.07.1986 14:39:07 t04Â 
 BreakÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â Â  432Â  --p-rwedÂ  06.07.1986 14:39:07 t16Â 
 ChangeTaskPriÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â Â  460Â  --p-rwedÂ  06.07.1986 14:39:07 t21Â 
 ConClipÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â  2432Â  --p-rwedÂ  06.07.1986 14:39:07 t29Â 
...
TÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  22.06.1991 02:06:08 t25Â 
 UtilitiesÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  06.07.1986 14:39:05 t07Â 
 ClockÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â  13856Â  ----rwedÂ  06.07.1986 14:39:04 t31Â 
 Clock.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â  590Â  ----rw-dÂ  06.07.1986 14:39:04 t37Â 
 MoreÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â  12752Â  --p-rwedÂ  06.07.1986 14:39:04 t43Â 
 MultiViewÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â  28836Â  --p-rwedÂ  06.07.1986 14:39:05 t01Â 
 MultiView.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â  861Â  ----rw-dÂ  06.07.1986 14:39:05 t11Â 
 Utilities.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  632Â  ----rw-dÂ  06.07.1986 14:39:06 t14Â 
 WBStartupÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  06.07.1986 14:39:05 t19Â 
 WBStartup.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  632Â  ----rw-dÂ  06.07.1986 14:39:06 t21Â 
Blocks:Â Â Â  data:Â Â Â Â  1496, fs:Â Â Â Â Â  199, sum:Â Â Â Â  1695, ratio=88.26, disk total:Â Â Â Â  1760
Bytes:Â Â Â Â  data:Â Â  765952, fs:Â Â  101888, sum:Â Â  867840, ratio=88.26, disk total:Â Â  901120
File:Â Â Â Â Â  data:Â Â  726742, ratio=94.88

Ok, first command shows you whats on a disk! Note the calling convention in xdftool: always pass the image file name first then the command with parameters. You see the complete directory tree of the volume inside the disk image. Each entry shows you the size, protection flags, modification time stamps (with ticks) and if available the file comment. At the end a summary on block usage is given.

Create a disk and write to it

> xdftool mydisk.adf format MyDisk + boot install + list
MyDiskÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  VOLUMEÂ  --------Â  14.01.2012 21:49:13 t00Â 
Blocks:Â Â Â  data:Â Â Â Â Â Â Â  0, fs:Â Â Â Â Â Â Â  1, sum:Â Â Â Â Â Â Â  1, ratio=0.00, disk total:Â Â Â Â  1760
Bytes:Â Â Â Â  data:Â Â Â Â Â Â Â  0, fs:Â Â Â Â Â  512, sum:Â Â Â Â Â  512, ratio=0.00, disk total:Â Â  901120
File:Â Â Â Â Â  data:Â Â Â Â Â Â Â  0, ratio=0.00

With this command creating an empty formatted image is a no-brainer… Note that you can combine multiple commands in a single call to xdftool by separating them with a plus sign. Creating a hard disk image is also very easy:

> xdftool myhd.hdf format Work 10M

Here a size (or a disk geometry) needs to be specified.

You can put some files from your host on it with the ‘write’ command:

> xdftool mydisk.adf write README + list
testÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  VOLUMEÂ  --------Â  14.01.2012 21:51:42 t00Â 
 READMEÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  3728Â  ----rwedÂ  14.01.2012 21:51:42 t00Â 
Blocks:Â Â Â  data:Â Â Â Â Â Â Â  8, fs:Â Â Â Â Â Â Â  2, sum:Â Â Â Â Â Â  10, ratio=80.00, disk total:Â Â Â Â  1760
Bytes:Â Â Â Â  data:Â Â Â Â  4096, fs:Â Â Â Â  1024, sum:Â Â Â Â  5120, ratio=80.00, disk total:Â Â  901120
File:Â Â Â Â Â  data:Â Â Â Â  3728, ratio=91.02

This will write the file README from the host’s current directory to the volume directory of the image. You can also write a full directory tree to the image:

> xdftool mydisk.adf write doc + list
testÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  VOLUMEÂ  --------Â  14.01.2012 21:53:20 t00Â 
 docÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  14.01.2012 21:53:20 t00Â 
   xdftool.txtÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â  18798Â  ----rwedÂ  14.01.2012 21:53:20 t00Â 
Blocks:Â Â Â  data:Â Â Â Â Â Â  39, fs:Â Â Â Â Â Â Â  3, sum:Â Â Â Â Â Â  42, ratio=92.86, disk total:Â Â Â Â  1760
Bytes:Â Â Â Â  data:Â Â Â  19968, fs:Â Â Â Â  1536, sum:Â Â Â  21504, ratio=92.86, disk total:Â Â  901120
File:Â Â Â Â Â  data:Â Â Â  18798, ratio=94.14

Now you can alter the protect flags, the modification time or set a comment:

> xdftool mydisk.adf protect doc rwe + time doc "12.01.1991 12:00:00 t11" + comment doc "my doc comment" + list
testÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  VOLUMEÂ  --------Â  14.01.2012 21:53:20 t00Â 
 docÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwe-Â  12.01.1991 12:00:00 t11Â  my doc comment
   xdftool.txtÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  18798Â  ----rwedÂ  14.01.2012 21:53:20 t00Â 
Blocks:Â Â Â  data:Â Â Â Â Â Â  39, fs:Â Â Â Â Â Â Â  3, sum:Â Â Â Â Â Â  42, ratio=92.86, disk total:Â Â Â Â  1760
Bytes:Â Â Â Â  data:Â Â Â  19968, fs:Â Â Â Â  1536, sum:Â Â Â  21504, ratio=92.86, disk total:Â Â  901120
File:Â Â Â Â Â  data:Â Â Â  18798, ratio=94.14

With these commands at hand you can now easily write a Makefile entry that creates an ADF image and places all your files on it.

But xdftool can more:

Unpacking and Packing Images

The unpack command extracts all files and directories from a disk image and creates a corresponding directory structure on your host’s file system. Additionally a MetaDB file will be created with all flags of the Amiga file system that cannot be represented in the host file system, namely protect flags, timestamps with ticks and comments:

> xdftool wb31.adf unpack .
> ls Workbench3.1*
Workbench3.1.blkdevÂ Â Â  Workbench3.1.bootcodeÂ  Workbench3.1.xdfmeta

Workbench3.1:
C/Â Â Â Â Â Â Â Â Â Â Â Â Â  Devs.infoÂ Â Â Â Â Â  Expansion.infoÂ  Prefs/Â Â Â Â Â Â Â Â Â  S/Â Â Â Â Â Â Â Â Â Â Â Â Â  T/Â Â Â Â Â Â Â Â Â Â Â Â Â  WBStartup/
Classes/Â Â Â Â Â Â Â  Disk.infoÂ Â Â Â Â Â  L/Â Â Â Â Â Â Â Â Â Â Â Â Â  Prefs.infoÂ Â Â Â Â  System/Â Â Â Â Â Â Â Â  Utilities/Â Â Â Â Â  WBStartup.info
Devs/Â Â Â Â Â Â Â Â Â Â  Expansion/Â Â Â Â Â  Libs/Â Â Â Â Â Â Â Â Â Â  Rexxc/Â Â Â Â Â Â Â Â Â  System.infoÂ Â Â Â  Utilities.info

You see a directory named after the volume inside the image was created with all files and dirs of the image. Additionally the MetaDB file *.xdfmeta, the boot code in *.bootcode and the description of the disk’s block device geometry *.blkdev was written.

With these files in place you can always recreate a new disk image that is equivalent to the original image (from the FS perspective):

> xdftool newimg.adf pack Workbench3.1 + list
Workbench3.1Â Â Â Â Â Â Â  Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  Â Â Â Â Â Â Â Â  VOLUMEÂ  --------Â  06.07.1986 14:32:19 t08Â  
 CÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  22.06.1991 02:13:25 t18Â  
 AddBuffersÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â Â  444Â  --p-rwedÂ  06.07.1986 14:39:06 t31Â  
 AddDataTypesÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â Â  5880Â  --p-rwedÂ  06.07.1986 14:39:06 t37Â  
 AssignÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â   Â Â Â Â Â Â Â Â  3220Â  --p-rwedÂ  06.07.1986 14:39:06 t43Â  
...
   MultiView.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  861Â  ----rw-dÂ  06.07.1986 14:39:05 t11Â  
 Utilities.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  632Â  ----rw-dÂ  06.07.1986 14:39:06 t14Â  
 WBStartupÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  DIRÂ  ----rwedÂ  06.07.1986 14:39:05 t19Â  
 WBStartup.infoÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â  632Â  ----rw-dÂ  06.07.1986 14:39:06 t21Â  
Blocks:Â Â Â  data:Â Â Â Â  1496, fs:Â Â Â Â Â  199, sum:Â Â Â Â  1695, ratio=88.26, disk total:Â Â Â Â  1760
Bytes:Â Â Â Â  data:Â Â  765952, fs:Â Â  101888, sum:Â Â  867840, ratio=88.26, disk total:Â Â  901120
File:Â Â Â Â Â  data:Â Â  726742, ratio=94.88

Note the correct time stamps and protect flags after packing…

With this approach you can quickly unpack an image, modify some files on your host file system and pack the image again.

Also mastering full images is now easy: create a Volume directory from scratch, place all your files in it and if you need special flags then write a *.xdfmeta file. Call xdftool pack to create your image. Done!

Repacking Images

Another nice operation in xdftool is repack: This essentially combines a unpack command on an existing image with a pack command on a new image. So you can rebuild the contents of the whole volume of the old image and write a fresh new FS structure. This will recreate the FS and re-aligns all file data blocks to be in one place. Originally this command helps me to test the integrity of amitools fs library but it may be useful for you, too:

> xdftool wb31.adf repack new.adf

In combination with a hard disk image repack also allows you to create a larger target disk, so this command essentially allows you to “grow” your Amiga hard disk by repacking its image to a larger one:

> xdftool old.hdf repack new.hdf 10M

You can even repack a disk image into a hard disk image:

> xdftool wb31.adf repack wb31.hdf 10M

That’s it… If you want to play more with xdftool have a look at the xdftool page!

Have Fun! And if you find bugs in the rather fresh code then please don’t hesitate to report them… Thanks!

Lallafa's Blog

Personal Musings about the C64, Amiga, Macs and my other Hobby Projects

Mastering ADF/HDF Images with xdftool

Look at a Disk

Create a disk and write to it

Unpacking and Packing Images

Repacking Images

Like this:

Related

Leave a ReplyCancel reply

Look at a Disk

Create a disk and write to it

Unpacking and Packing Images

Repacking Images

Share this:

Like this:

Related

Leave a ReplyCancel reply