BSD Commit Log Search

You might have noticed a change to how we’re dealing with bot-detection. Anubis, the JavaScript-driven proof-of-work system that’s been in place for about a year, has been replaced with a button for you to press.

There’s nothing fancy here: you press a the button, submit an empty form using HTTP POST, and you get a cookie which grants access for a week. Very similar to the approach our friends at FreshPorts have taken.

This has – for now at least – been almost completely effective. In the course of about a week the prototype implementation (a vibe-coded Go proxy) issued 2.3 million challenges, of which a grand total of 135 were completed. The effect is most stark on an outgoing network traffic graph – see if you can spot the deployment:

With the concept proven, the prototype has now been replaced with a hand-reworked Vinyl VCL, which it probably should have been from the start. Here’s the basic outline if you wanted to try it yourself, some assembly required:

vcl 4.1;
import cookie;

sub vcl_recv {
  ...
  if (req.url ~ "^/(favicon\.ico|images|stylesheets|...)" && ...) {
    cookie.parse(req.http.Cookie);
    
    if (!cookie.isset("botless")) {
      if (req.method == "POST") {
        # Prevent scheme-relative redirects, //foo and /\foo
        if (req.url ~ "^/[/\\]") {
          return (synth(400, "Bad URL"));
        }
        return (synth(10303)); # Redirect: solved challenge
      }
      return (synth(10403)); # Forbidden: challenge page
    }
  }
  ...
}

sub vcl_synth {
  # Forbidden: challenge page
  if (resp.status == 10403) {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Cache-Control = "no-store";
    set resp.body = {"<!doctype html> ...
      <form method="post"><button type="submit" autofocus>I'm made out of meat</button></form>
    ..."};
    return (deliver);
  }

  # Redirect: solved challenge
  if (resp.status == 10303) {
    set resp.http.Location = req.url;
    set resp.http.Set-Cookie = "botless=1; Path=/; Max-Age=604800; HttpOnly; Secure; SameSite=Lax";
    set resp.http.Cache-Control = "no-store";
    return (deliver);
  }
}

Obviously this isn’t going to stop anyone determined to scrape, and it could lose effectiveness if bots adapt, but for the time being I’m enjoying actually being able to see that I do, in fact, still have some human users. Hi!