Let's develop an open-source media archive standard

From: Jason McBrien <jbmcb_at_hotmail.com>
Date: Wed Aug 11 10:56:21 2004

----- Original Message -----
From: "Jules Richardson" <julesrichardsonuk_at_yahoo.co.uk>
To: "General Discussion: On-Topic and Off-Topic Posts"
<cctalk_at_classiccmp.org>
Sent: Wednesday, August 11, 2004 10:50 AM
Subject: Re: Let's develop an open-source media archive standard

> "could be kept" in zip files, yes - but then that's no use in 50 years
> time if someone stumbles across a compressed file and has no idea how to
> decompress it in order to read it and see what it is :-)
> Hence keeping the archive small would seem sensible so that it can be
> left as-is without any compression. My wild guesstimate on archive size
> would be to aim for 110 - 120% of the raw data size if possible...

I'd say ZIP is so ubiquitous, if the format itself is dead and gone there
should be a pile of documentation on how to extract it, especially if you
keep to the basic "deflate" algorithm, used in everything from PNG to gzip
to Jar.

> My background (in XML terms) is with Java - but I've not come across
> software that mixes a human-readable format in with a large amount of
> binary data though (whether encoded or not). Typically the metadata's
> kept seperate from the binary data itself, either in parallel files (not
> suitable in our case) or as seperate sections within the same file.

Fork it. Make it a binary file, but keep the metadata in the header
ASCII/Unicode/XML/Whatever.
That way a simple cat or less or opening in notepad will show you what's in
it.

> See my posting the other week when I was trying to convert ASCII-based
> hex data back into binary on a Unix platform :-) There's no *standard*
> utility to do it (which means there certainly isn't on Windows). If the
> data within the file is raw binary, then it's just a case of using dd to
> extract it even if there's no high-level utility available to do it.

On the macs there's Binhex which has been the standard since file transfers
have begun. Also, there are dozens of *nix utilties to deal with Base64
(MIME) encoded data.

> But the data describing all aspects of disk image would be readable by a
> human; it's only the raw data itself that wouldn't be - for both
> efficiency and for ease of use. The driving force for having
> human-readable data in the archive is so that it can be reconstructed at
> a later date, possibly without any reference to any spec, is it not? If
> it was guaranteed that a spec was *always* going to be available, having
> human-readable data at all wouldn't make much sense as it just
> introduces bloat; a pure binary format would be better.

IMHO if you have a file laying around and you have no idea what it is, it's
still nice to be able to
read it in as text and see what's going on, the Unix "file" command
notwithstanding.

>
> I'm not quite sure what having binary data represented as hex for the
> original disk data gives you over having the raw binary data itself -
> all it seems to do is make the resultant file bigger and add an extra
> conversion step into the decode process.

Usually text moves around multiple machines/oses a lot easier than binary
data. If you can dump a text file to a machine somehow, you can transfer a
media archive and a .c program to compile and decode it.

>
> Yep, I'm with you there. CRC's are a nice idea. Question: does it make
> sense to make CRC info a compulsory section in the archive file? Does it
> make sense to have it always present, given that it's *likely* that
> these archive files will only ever be transferred from place to place
> using modern hardware? I'm not sure. If you're spitting data across a
> buggy serial link, then the CRC info is nice to have - but maybe it
> should be an optional inclusion rather than mandatory, so that in a lot
> of cases archive size can be kept down? (and the assumption made that
> there exists source code / spec for a utility to add CRC info to an
> existing archive file if desired)

I think CRC's should be compulsorary, with an external hash optional.
Corruption can happen on new hardware as well as old :)

>
> cheers
>
> Jules
>
>
Received on Wed Aug 11 2004 - 10:56:21 BST