Let's develop an open-source media archive standard

From: Vintage Computer Festival <vcf_at_siconic.com>
Date: Thu Aug 12 14:18:23 2004

On Wed, 11 Aug 2004, Fred Cisin wrote:

> And for regeneration, there is such a thing as TOO MUCH
> data. If you try to replicate a disk, where every single
> bit/flux transition is specified, it is not feasable.
> There is a reason for "GAP" fields! If you try to
> replicate the disk, you will not be able to recreate
> some of the mangled bytes in the write splice fields.
> Unless you are trying to replicate sopy-protection,
> you would be WAY better off storing JUST the data,
> and the specs of the format, and recreating the format
> instead of trying to replicate it!

I totally agree with this. However, there are cases where the only
practical method for imaging a disk will be at a level below what I refer
to as "logical" format, meaning what the sector data translates to (i.e.
if a sector stores 256 bytes, the "logical" data is the actual 256 bytes).

Case 1: A disk with copy protection so tortuous that the only way to image
it is at the bit level.

Case 2: One of the copy protections that was employed on the Apple ][ was
an 18 sector format that Roland Gustafson invented. He managed to
optimize the storing of bytes on a track and managed to squeeze 18 sectors
worth of 256-byte sectors on a single track (where normally 16 would
reside). So the question is, how do you image this? I would image it at
what I'm now referring to as the "raw" level. Since the Apple ][ uses
GCR, and since with GCR you can't have more than two zero bits in a row,
the actual bytes written to disk are encoded into what's called 6&2
encoding: 6 bits of an 8-bit byte are used to look up an 8-bit byte in a
table containing bytes that have no more than 2 consecutive zero bits; the
other 2 bits are combined with 2-bit sets from two other bytes and used to
look up a valid 8-bit value. There's also another encoding called 4&4
(one byte is stored as two valid bytes, with all the odd bits being a 1 to
ensure against two consecutive zeroes) and the Commodore 64 used 5&4. So
all valid bytes on a GCR track will always have their two highest bits
set. This is what I called "raw". So for an 18-sector disk, which is not
a defined Apple ][ disk format, I'd image it in raw format.

Raw format would have to entail the capability of being able to specify
sync bytes, which are 10 bit bytes: eight 1's and two 0's. With something
like 4 sync bytes in a row, you're always guaranteed that the read head
will be in sync with the proper byte boundaries.

So there is a need for bit, "raw", and "logical" image types. Going
further up the logical chain, we have "filesystem" image types (an entire
filesystem is imaged, i.e. we are above tracks and sectors) and then file
images.

> Instead, a 360K disk should have 360K of data, whether
> as 360K of binary bytes, or as 720K+ of ASCII hex dump,
> PLUS just the information that it is 9 512 byte sectors
> per track. Any exceptions to "standard" formatting
> could be documented in the header, OR within the data
> as necessary.

This can certainly be included in the spec and is a good idea. There's no
point in filling up an image file with useless tags if a disk has a
standard geometry that can be easily described like this.

> But OTHER people, such as the CAPS project feel a need to
> replicate every flux transition.

With certain disks (i.e. copy protection) I share the feeling of this
need, and they explain that. If they are doing it for every disk,
regardless of whether it is copy protected, then that's overkill.

However, the spec will allow one to create an image of whatever type they
deem necessary. We can make recommendations with white papers, but people
will do what they want.

> Therefore, since this is an extensible spec,
> I propose that it have a multi-layer structure:
> (I realize that this will grievously offend those
> who are insisting on a single invariable structure!)

Not I. I've been advocating a multiple layer structure for a few messages
now.

-- 
Sellam Ismail                                        Vintage Computer Festival
------------------------------------------------------------------------------
International Man of Intrigue and Danger                http://www.vintage.org
[ Old computing resources for business || Buy/Sell/Trade Vintage Computers   ]
[ and academia at www.VintageTech.com  || at http://marketplace.vintage.org  ]
Received on Thu Aug 12 2004 - 14:18:23 BST

This archive was generated by hypermail 2.3.0 : Fri Oct 10 2014 - 23:36:34 BST