Distributed Referrer Blocking

Over the past few months referrer logs
have been removed from many of the blogs out there due to the amount of
spam that shows up in them.  Thanks to Gerco
Wolfswinkel we were
in the process of adding the MT-Blacklist function to BlogSphere however
the maintainer of the master blacklist decided to close his servers due
to the large amount of downloads of the master list ( which was pretty
big ).

Obviously a centralised list does not
work, it is too much work to maintain and would cause a lot of bandwidth
usage when people check for updates or need to download the entire list.
 What’s needed is a decentralised method to distribute new blocked
referrers and maybe the answer can be found by looking at how RSS feeds
work.

Lets imagine that you have just added
a new referrer to your blacklist. Your blog then adds this referrers information
to an XML file that lists the latest 10 blacklisted referrers.  Other
blogs can then connect to your list and download the latest additions,
it will add the new ones to it’s own blacklist ( skipping the ones it already
has ) and then it’s own ‘latest 10’ list gets updated.  Other sites
will pull from that list also and so on.  If each blog pulls blacklist
updates from 3 – 4 other blogs then pretty soon all interconnected blogs
will have a complete list.

The work that Gerco was doing also contained
WhiteLists, these are referrers that are automatically approved.  These
whitelists could also be distributed the same way or even within the same
file. If you get a referrer from Ed Brill your certainly not going to throw
it away.

The new referrer checker in your blog
will work as follows, it looks at all the unchecked referrers, automatically
approves the whitelisted ones and then automatically removes the blacklisted
ones.  Then if the referrer is not on either list it will put it on
the pending list.  You can manually check each one and whitelist/blacklist
as required. Your blacklist/whitelist will be distributed to others and
everybody benefits.

What if you want to blacklist somebody
but don’t want it distributed? well your blog software could have a ‘keep
private’ option that would not put it on your latest updates list.

What if somebody adds something into
the blacklist and it gets distributed to other sites? Well the XML file
could contain commands like ‘add’ or ‘remove’ so that you can send around
commands to remove referrers from the blacklist if required.

What if somebody keeps adding valid
domains to the blacklist? Well stop pulling updates from them.  Or,
maybe the XML file could contain an ‘submitter’ tag and your updater could
reject updates from a certain submitter.

How do I know what’s being added to
my blacklists/whitelists? Well the task that pulls in the updates could
email you details of the changes so you can keep an eye on it.

There are lots of questions that need
to be answered before a scheme like this can work so I’d love to hear your
comments on this idea.  This idea is not just for BlogSphere, I’m
sure it could be used in any blog software, and not just Domino based.

Would you be willing to use something
like this so you could put your referrers back on screen.