On Feb 5, 12:22, Lawrence LeMay wrote:
> >
> > See above. CD burners are wonderful.
> >
>
> Somewhat out of question, but does anyone know of any software that would
> download a web site, or a portion, to a local disk? IE, I would want to
be
> able to archive part of a web site onto a CDrom, without having to
manually
> save every image and web page, and manually edit the html links, etc.
>
> Ideally the software would understand that I was planning to burn
> 650 meg, or 700 meg, etc images, and would break the download into
> separate directories/images.
wget on a unix or Linux system. You'd want to pick your root point and
start wget with the -r (--recursive) and -k (or --convert-links) options
(to make any absolute links into relative ones, wherever possible) and the
-np (or --no-parent) so it doesn't follow links that go *up* the directory
tree rather than down (a good way to get lots more than you intended!).
There's also a -m (--mirror) option. You can refine things with other
options (such as those to avoid certain files or types, useful to exclude
those extra ?D=A indexes from directory listings) but I usually start with
-r -k -np. On a Windows system? Telnet to a Unix machine. There is a
Windows port, but I'm not sure how good it is.
wget can limit the total size of a download by setting a quota, but it's
not as useful as you might think. Better for you to decide yourself where
to split a collection over two or more CDs. Of course some sites make this
easy (like PUPS/TUHS, for which I run one of the official mirrors).
Another polite option is --wait, which waits between fetches to reduce the
bandwidth. Some sites might not like you hogging their connection. In
fact, recently I came across another site that had banned certain addresses
altogether for attempting to download a whole site without asking first.
The owner took the view that no-one could possibly want the whole site,
unless for a mirror, in which case it's best to ask.
Some sites have a megabytes-per-month limit imposed by their ISP, so again,
ask.
There are also programs for FTP (only) mirroring: the imaginatively-named
'mirror' and 'ftpmirror' come to mind, and pavuk too, but wget is a pretty
good all-rounder.
http://www.wget.org/
http://www.gnu.org/software/wget/wget.html
http://www.idata.sk/~ondrej/pavuk/
http://noc.intec.co.jp/ftpmirror.html
http://sunsite.org.uk/packages/mirror/
--
Pete Peter Turnbull
Network Manager
University of York
Received on Tue Feb 05 2002 - 16:12:01 GMT