> > Actually, FTP is even harder to control IMHO because you can't block the
> > obvious copy program's user agent strings, and there are a lot of robo-FTP
> > leeches too. Even if you required a username and password, they can still
> > misbehave when they get on, and they only need to suck you dry once (so
> > kicking them off accomplishes nothing). Bandwidth rapists are a scourge.
>
> Can't one put monitoring software in place that cuts off any one IP
> address after, say, more than 100MB (or whatever amount makes the most
> sense) has been downloaded?
I don't know if ipfilters is that smart. Plus, you run the risk of cutting
off proxies, which might have legitimate reasons to download a lot of data
(like if a lot of people are using them). Not a big deal for FTP since there
aren't many FTP proxies, but it's a possible problem for HTTP.
The leech detection tools I wrote look at the pattern of access and mean time
between accesses. Short and/or constant interval access times, plus
actions with a high index of suspicion for being non-human access (such as
going in alphabetical or numerical order through the pages on a site rather
than following their link network, or jumping to files that aren't linked
from the last one they viewed) wind up in my real-time ban list. I also keep
a list of offending user agents and ban them on site, and I'll ban IP blocks
or networks if I get persistent jerks who try to morph their browser string
or jump around from IP to IP within a subnet. I'm not game enough to allow it
to autoban, but I review top offenders every few hours and add to the ban list
as appropriate (a cron job does the heavy lifting and mails it to me).
Obviously, they have the bandwidth to suck my servers dry, so why can't they
look at them online like everyone else instead of bottling up my uplink by
trying to slurp all the pages down in parallel? I don't mind offering stuff
up for view. What I mind is people taking advantage of it and not considering
the impact it has when everyone and their monkey wants a local copy of
freaking everything.
Since I implemented this (admittedly draconian) policy, I haven't lost many
users except the ones I wanted to, and my site availability is much better.
--
---------------------------------- personal: http://www.armory.com/~spectre/ --
Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser_at_floodgap.com
-- Magic armour is not all it's cracked up to be. -- Terry Pratchett ----------
Received on Tue Feb 08 2005 - 19:17:17 GMT