Anti-Scraper Defences Update

You might have noticed a change to how we’re dealing with bot-detection. Anubis, the JavaScript-driven proof-of-work system that’s been in place for about a year, has been replaced with a button for you to press.

There’s nothing fancy here: you press a the button, submit an empty form using HTTP POST, and you get a cookie which grants access for a week. Very similar to the approach our friends at FreshPorts have taken.

This has – for now at least – been almost completely effective. In the course of about a week the prototype implementation (a vibe-coded Go proxy) issued 2.3 million challenges, of which a grand total of 135 were completed. The effect is most stark on an outgoing network traffic graph – see if you can spot the deployment:

Network transmission graph, showing growth up to 800 kB/s and a cliff-edge down to 30 kB/s

With the concept proven, the prototype has now been replaced with a hand-reworked Vinyl VCL, which it probably should have been from the start. Here’s the basic outline if you wanted to try it yourself, some assembly required:

vcl 4.1;
import cookie;

sub vcl_recv {
  ...
  if (req.url !~ "^/(favicon\.ico|images|stylesheets|...)" && ...) {
    cookie.parse(req.http.Cookie);
    
    if (!cookie.isset("botless")) {
      if (req.method == "POST") {
        # Prevent scheme-relative redirects, //foo and /\foo
        if (req.url ~ "^/[/\\]") {
          return (synth(400, "Bad URL"));
        }
        return (synth(10303)); # Redirect: solved challenge
      }
      return (synth(10403)); # Forbidden: challenge page
    }
  }
  ...
}

sub vcl_synth {
  # Forbidden: challenge page
  if (resp.status == 10403) {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Cache-Control = "no-store";
    set resp.body = {"<!doctype html> ...
      <form method="post"><button type="submit" autofocus>I'm made out of meat</button></form>
    ..."};
    return (deliver);
  }

  # Redirect: solved challenge
  if (resp.status == 10303) {
    set resp.http.Location = req.url;
    set resp.http.Set-Cookie = "botless=1; Path=/; Max-Age=604800; HttpOnly; Secure; SameSite=Lax";
    set resp.http.Cache-Control = "no-store";
    return (deliver);
  }
}

Obviously this isn’t going to stop anyone determined to scrape, and it could lose effectiveness if bots adapt, but for the time being I’m enjoying actually being able to see that I do, in fact, still have some human users. Hi!

Watchgoose Supports FreshBSD

FreshBSD’s cronjobs and alerting systems are now monitored with Watchgoose, who were kind enough to offer a free paid account – no strings attached – as part of their Open Source program.

Previously I was sending a notification via Pushover when and if a backup or other cronjob explicitly failed. Switching to Watchgoose means I now also get a notification if a job doesn’t complete on its expected schedule – whether it takes too long, or simply doesn’t run at all.

I also have it integrated into my local monitoring, with an always-firing alert triggering a webhook on a regular schedule, so if my local Victoria Metrics stack goes down, I’ll actually notice.

We got a little shout-out, too, alongside some lesser-known projects:

Watchgoose — "Open Source projects we support: FreeBSD, rclone, FreshBSD"
Screenshot: watchgoose.com/open-source

Cheers to founder Bartek Rutkowski, who is also a member of the FreeBSD project.

Happy Birthday to Us!

FreshBSD.org is 20 years old today:

❯ whois freshbsd.org | grep Creation
Creation Date: 2006-03-06T18:30:19Z

I suppose I should do something to mark it, so let’s take a little stroll through time with the help of the Internet Archive.

Here’s how FreshBSD looked back in 2006, with its simple drop-down filtering menu and lacking even commit message search:

FreshBSD.org version 1 on April 26th 2006 - indexing 220k commits, with a simple HTML drop-down filtering form, and a tagline of '{DragonFly, Free, Net, Open}BSD Commit Logs Every 3 Minutes'.

It worked by parsing emails from source code commit mailing lists into a MySQL database. The frontend ran Ruby on Rails, and notably this is roughly the last time I thought that might be a good idea.

By 2009, a rewrite that replaced MySQL with Apache Solr was running alongside, which enabled more efficient search and a first go at supporting facets:

FreshBSD beta site, version 2, March 17th 2009 - indexing 201k commits, with a search text input on the left and a set of dual column tables on the right showing 'Project', 'Branch' and 'Committer' facet counts. The tagline of this early version promises 'Commit Logs Every Day Or So'.

By the start of 2012, another rewrite had the search indexer working directly off source code repositories, and Solr replaced with Elasticsearch. Not relying on email parsers allowed for the expansion to a lot more projects:

FreshBSD.org version 3 on the January 27th 2012, '*BSD Commit Log Search', indexing 1.6 million commits, with a long search text input and two-column tables down the right side showing search facets. The FreshBSD text 'logo' glows slightly for some reason.

2016 saw the first signs of the site you’re using now - another rewrite which added PostgreSQL to the stack and switched the frontend over to Roda.

I also finally got around to making a logo, albeit a rather bland one:

FreshBSD.org version 4 beta on January 12th 2016, 'BSD Commit Log Search', indexing 5.1 million commits. The plain text 'FreshBSD' logo has a stylised 'E' constructed out of plus and minus characters reminiscent of a unified diff. The header is a sort of blue-white gradient that I guess I thought looked good at the time. The search form including drop-down facets is in a bar down the right side. The pagination display helpfully invites you to page 102,841.

I did a big refinement pass in 2021 which brought the site up to the current standard. Behold, in case you forgot what that looks like:

FreshBSD.org version 4 on March 5th 2021, indexing 9.2 million commits. The 'FreshBSD' logo now has red and green on the stylised 'E'. The compact pagination display invites you to reverse the sort order instead of going to the last page.

I guess we’re kind of overdue another big update. The current trend is to make everything worse, right? I’m sure I’ll think of something…