There is an on-going distributed attack against the SQLite website, and specifically against URLs of the form:

https://sqlite.org/src/file?....

where the "...." is various query parameters which are always different and also always valid. I clicked the box to require anonymous login to access that URL, but the attacks keep on coming, which suggests that the robots are also sending valid anonymous-login cookies. (The web server does not log cookies so I don't know this for certain.) The user-agent strings masquerade as legitimate web browsers, of course.

Each request comes from a different IP address; the average number of requests per IP address is 1.2. I know that a robot is involved, though, since the CSS is not being requested. The IP addresses are from all over the world. So I cannot mitigate the attack by null-routing selected IP addresses.

The SQLite.org server is keeping up for now, but the attack rate is growing. If this continues, I'll probably need to turn off anonymous access to the affected URL. And in the meantime, you might notice that https://sqlite.org/ might be less responsive than normal.

The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things....

That sounds like the current MO of so-called "AI" web scrapers. If you can think of a low-resource method of serving poison text in response to these requests, please implement it.

I don't (yet) have a way to distinguishing these robot requests from legitimate requests.

That's the CON of being fully in charge of your stack, Web Server and SCM included.
But I know you wouldn't have it any other way. Still, that's time not on SQLite itself,
which is a shame for all the rest of us.

Would be nice to be able to dependent on a giant like CloudFare for DDoS protection.
And as a CDN too. Although with something as dynamic as Fossil, not sure how that would work.

Each request comes from a different IP address; the average number of requests per IP address is 1.2. I know that a robot is involved, though, since the CSS is not being requested. The IP addresses are from all over the world. So I cannot mitigate the attack by null-routing selected IP addresses.

That's particularly challenging. Someone with a lot of bots or a massive ability to spoof the source of the connections is clearly toying with you.

Are these Https connections? Or just http? I seem to remember SQLite.org redirects all http to HTTPS, but if not and this traffic is only port 80, you could restrict anonymous access solely on that port.

If it's HTTPS traffic then they must actually have control of the endpoint, not just spoofing it.

That's going to be a tough crowd to weed out.

Althttpd isn't going to let you directly block access to ..../src/file only is it? It's only routing based on the .../src level, as I understand it.

Years ago, anonymous login cookies were tied to a particular IP address. But some users complained that they were behind a firewall that caused their IP address to constantly shift, meaning they couldn't stay logged in as anonymous. So I disabled that feature and now the same anonymous login cookie will work from any IP address. Maybe I need to bring back the feature of tying anonymous to a specific IP address....

Just to reassure you, I've been reading about this happening to 40 other sites in the last 24 hours. According to the admins who have reported it, they range all over the map, from business-to-business webstores to hobby sites to … erm, how do I put it … the most-used sites on the internet. Cloudflare has also reported a peak on attacks on many of the sites it runs.

So whatever it is, it's not somebody attacking SQLite specifically.

Nobody has come up with a good specific defence yet. If I see one, I'll post. The best I've seen is a treacle defence: introduce a 3 second delay in all responses to the affected URLs. Humans accept it, but bots won't send more requests until old requests have been serviced.

bots won't send more requests until old requests have been serviced.

Remember, we are getting 1.2 requests per IP address. So the individual slave robots are sending no more than 1 request on average anyhow. Convincing them to give up after one delay does not improve the situation.

Maybe I need to bring back the feature of tying anonymous to a specific IP address....

The bigger concern in that specific case was mobile usage was on the increase with the introduction of the forum, and my IP was changing several times on each train commute to/from work. If we go back to IP-based login cookies, mobile use will suffer.

IP-based anonymous cookies only - not real-user cookies.

IP-based anonymous cookies only - not real-user cookies.

Oh, of course those are different. The only cases which would be broken by active roaming that way are anonymous posters, but that's not a limitation i'd lose any sleep over.

Would something like anubis work? https://anubis.techaro.lol/

Would something like anubis work?

As ddevienne alluded to earlier, this project has a long history of "rolling its own" and not depending on third-party services. It will be a sad day in history if it's eventually pressured into hiding behind Anubis, Cloudflare, or the like.

I could be wrong but I think that anubis is not a third party service, you have to host your own instance.

Fossil (which is the system that provides the services being attacked) already does use heuristics to detect and deflect robots. Been doing that for many years. The problem here is that this new robot is impersonating a human with sufficient accuracy to defeat all the heuristics.

I could be wrong but I think that anubis is not a third party service, you have to host your own instance.

Even so, it's a third-party dependency. This project's web presence currently has no third-party dependencies beyond the OS upon which the web server¹ sits, libz, libssl, and common CLI-based software development tools.

1. ^{^} Written by drh

The distributed robot attack continues to increase. Therefore, access to https://sqlite.org/src/.. is now mostly restricted unless you first login as anonymous. Sorry for the inconvenience.

Great. A captcha that is unreadable in Brave. At least I can hear and type it.

And now a fossil pull no longer works: "Error: not authorized to read"

Might as well pack up for the day - or the week.

A captcha that is unreadable in Brave.

Works for me in Brave. Send me a screenshot to private email: drh at sqlite dot org.

fossil pull no longer works

I'm working on something better. Please try again in 24 hours, and repeatedly there after until it starts working again.

It appears to be an Android thing...

Brave on Android = unreadble

Chrome on Android = unreadable

Firefox ESR on Windows = ok

Firefox on Linux = ok

Firefox ESR on Linux = ok

Brave on Linux = ok

I don't think that is worth bothering with - having the captcha read out works.

As for Fossil - obviously nothing to do but wait.

Have you tried matching the offending IPs with those used by Perplexity?

According to this week's "Security Now" podcast Perplexity has been structurally naughty lately.

Have you tried matching the offending IPs with those used by Perplexity?

That likely won't help, they appear to be using different ASNs to avoid such blocks.

I didn't mean as a way to block them, but as a possible way to identify them if there is an overlap.

No. There is no way (yet known) to block the attack.

One theory (which is unproven but makes sense) is that the rogue HTTP requests are coming from Perplexity's Chrome plugin. So normal users on the internet install Perplexity's plugin and go about their business. Meanwhile, in the background, the plugin is surreptitiously harvesting training data, under the guise of being the innocent user who installed the plugin, and forwarding the data on to Perplexity. So the rogue requests are coming from a normal users browser, hence there is no good way for us to distinguish them from legitimate requests.

Well, I think you did a fine job distinguishing them in your first post: they are all essentially one-off deep links. Your reporting the incident made me look closer at my own logs, and I'm finding similar requests (only to a less punishing extent, partly because I don't host anything as major as SQLite and partly because I already have extensive firewall listings of networks that have already been abusing my servers). Even if Fossil has an easy way to detect that sort of traffic, what an effective "block" would look like is the real question, given the one-and-done nature of the DDoS attack.

What about a low-resource "prove you're human" page: For a GET deep link, present something like "Your page is now ready:" followed by a "Load Page" button that does a POST submit instead, which is accepted.

The current antibot-defenses also seem to break the SQLite source repository RSS feed (when loaded from scripts). Is there any chance this can be enabled, again?

Works fine for me when an anonymous login cookie is presented.

Presumably this is exactly the page type that is vulnerable to the current issue, so it must need either protecting behind some sort of cookie style gate, or hiving out as a static page to minimise load.

Incidentally, all my login cookies seem to have recently been invalidated - possibly fluke or perhaps the user session table has recently been flushed.

So if you were previously presenting an anonymous login cookie for the RSS feed, you may need to regenerate it.

I see the server load has now dropped significantly from recent numbers.

Is there any sign of abatement?

I'm loading the RSS feed from a script.

Noted, but depending on exactly what you're willing to do, you can capture an anonymous login cookie and provide that with eg. curl -c cookiefile

I'm not suggesting that's an optimal solution, but it's a workaround that seems fine here.

You're obviously at the mercy of cookie I validation and there is some manual input required.

But while SQLite.org is trying to stay afloat, it might be easier to just temporise rather than waiting expectantly, if this really is an issue for your workflow.

The bot-net is still sending in about 1 million "deep" requests per day. By "deep request", I mean some HTTP request that requires following long delta chains, generating complex diffs, and otherwise doing a lot of computation. A "deep" request can take 100 milliseconds or more to answer.

I currently have https://sqlite.org/src configured so that all deep requests are behind a captcha. Deep requests redirect to the captcha, and then the robot gives up. This configuration actually doubles the number of requests coming from the bot-net—there is the initial request itself which now redirects to the captcha page plus the request to the captcha page. But none of the requests are "deep". Each is satisfied with just a few milliseconds of CPU time. And so the server is able to keep up easily.

So while the immediate problem has been temporarily resolved, the rise of bot-nets is a worrying sign. It means that sites like https://sqlite.org/src which provide a huge amounts of information have difficulty existing on the open internet. They now have to be hidden behind captchas or similar to ward off predatory robots. This make it more difficult for people to write scripts to extract the information they need; now they have to interact with the captcha to get through. It complicates the management of websites like https://sqlite.org/src since they now require constant security monitoring. To my mind, this incident is a harbinger of a dystopian future for the internet.

I'm more hopeful - I suspect it will herald some sort of reputation based client certificate equivalent to server provided Https certificates, signed by some central trusted authority on the basis of some initial arduous criterion.

But it's a tech solution to a tech problem.

In the same way that http is no longer tenable, truly anonymous in the sense of uncorrelated internet communication is probably becoming untenable.

A basic Auth option for scripts is pretty easy to implement though.

some sort of reputation based client certificate

The bot-net is recruiting zombie clients to work on its behalf, and the zombies are the ones that own the (presumably good) client certificate.

I don't think you understand what is happening here. The flood of expensive HTTP requests is not coming directly from the bad actor. The requests are being laundered through innocent third parties. That's what makes this so diabolical. The troublesome HTTP requests are coming from ordinary people who are just trying to read their email or order some groceries on-line. Their computer/tablet/phone just happens to have been hijacked by a virus or chrome-extension that exploits their good reputation for nefarious purposes.

What's the solution to update the sqlite repo now?

$ fossil up trunk
Pull from https://sqlite.org/src
Round-trips: 1   Artifacts sent: 0  received: 0
Error: not authorized to read
Round-trips: 1   Artifacts sent: 0  received: 0
Pull done, wire bytes sent: 304  received: 212  remote: 2600:3c02::f03c:95ff:fe07:695
Autosync failed.

Right this moment, you can still sync against the back-up repositories at https://www2.sqlite.org/src and https://www3.sqlite.org/src. But that could change if the bot-net discovers them.

We are actively working on enhancements to Fossil that will permanently resolve your problem, but those enhancements are not ready yet.

You may have already done this.

At the level of a million requests a day, I would contact my hosting service and ask them whether they can help, or want to help, or even just want to take logs for later analysis. It may be that, although you can't see how traffic is carried from your servers, their network infrastructure may show things that this bot traffic has in common. Blocking the unwanted traffic themselves, if it is possible, may save them significant traffic, power, or money in cooling.

You have the advantage that the pages being requested are public. No harm will be done if you let the host's admins access everything about them.

Error: not authorized to read

We've put a temporary measure in place to keep the sync protocol working (the bot doesn't use it). Thank you for the report!

hijacked by a virus or chrome-extension that exploits their good reputation

It's a bit off topic, and not the best time to have tangential conversations of course. However, I do understand the nature of this current attack, and I agree about how and why this is especially pernicious and challenging.

The idea of reputation is that the innocent party would lose it, but be advised as to why, and therefore motivated to control their browser extension / virus / whatever is using network bandwidth on their behalf.

Where malware is presenting itself as an innocent user, the only way to differentiate between the malware and the user ultimately is for the user themselves to do so, and somehow be motivated and facilitated to correct the situation.

A new type of malware that impacts not your resources or security but your reputation, such that non-experts are incentivised to keep a clean ship. System checking tools that highlight "reputational risks" to computer users.

But that's the bigger philosophical issue, not the solution to this particular current manifestation of the problem.

To my mind, this incident is a harbinger of a dystopian future for the internet.

Some of us would say that "future" started 20+ years ago, when the Internet became primarily about the web, and the web became primarily about ads. As you said earlier, there's no need to spider your sites at all, or any site that is essentially a rendered repo that could be cloned and used locally. I would fully support you not enabling a web interface by default for any of the high-cost-but-normally-low-usage features that Fossil provides.

Maybe a stupid suggestion, but can the user nobody not have access to any links available on the timeline, and have it load only a few commits?
Maybe even have it load to a preexisting branch with a single commit when you detect it's from a bot farm.

Maybe a stupid suggestion, but can the user nobody not have access to any links available on the timeline,

Fossil has the ability to disable many links for non-logged-in users, but blanketly disallowing links for such users means that when we post links in the forum, there's a good chance that the folks following the links can't do much with them because link generation on the resulting page will be off for them. The usability (for those not logged in) really suffers.

To quote fossil's docs on the topic:

But requiring a login, even an anonymous login, can be annoying.

Unfortunately this "breaks" build scripts that (e.g.) use curl to download a tarball to build a given branch (my case bedrock).

Not complaining, completely understandable.

Since I use GitHub to host my project, I now use use your mirror there and their source code archive URLs instead:

Mostly commenting so if anyone else needs this they know they have this option.

There is a work-around for this. If you have an account on the Fossil repository, you can visit the /tokens page (ex: https://fossil-scm.org/forum/tokens) and there you can create an "access token" associated with your account. Suppose your token is "0123456789abcdef". Then on your curl URL, you add a query parameter "token=0123456789abcdef" and that will disable the robot screening.

Cool, that's a good option to have, thank you.

Thanks, that's a good option to have.

Since I'll be posting the URL to an open repository, I risk either exposing the token, or having to use secrets (which makes things harder for downstream users).

But noted for future use!

I notice that /src is not excluded in https://sqlite.org/robots.txt -- the paths mentioned here are mostly /cvstrac/ which I think is outdated?

Of course robots.txt is not going to make a difference for the kind of nefarious distributed bot-net that appears to be active at the moment, which appears designed to circumvent the usual restrictions against automated web requests. However in the long term it might be worth keeping this up to date as it represents a "standard" internet convention and as noted elsewhere in this thread¹, there is kickback against agents which do not honour it

¹ https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/

It is difficult to tell whether a particular HTTP request was coming from the bot-net versus some guy who just copy/pasted a link to SQLite from Reddit. But as best as I can discern, it was about 1 million distinct IPs per day.

It wasn't that big of an "attack". In fact, I would guess that the bot-net took steps to help ensure that it didn't overwhelm the target and hence call too much attention to itself. The problem is that SQLite.org is a small target, with just a single 4-CPU machine. And even that is plenty of power to handle millions bot request per day for static pages. It is just when they start doing dynamic pages that compute big diffs and source code annotations that take multiple hundreds of milliseconds to compute that it begins to cause problems. The server at sqlite.org has about 345,600,000 milliseconds of compute available per day. So a million requests that take about 250 milliseconds each to process gets us uncomfortably close to saturation.

The defensive measures that were put in place strive to prevent the bot-net from accessing the slow-to-compute pages. So each request ends up using just 3 or 4 milliseconds of CPU time. The existing server can handle that level of traffic without any trouble.

Aside: At the bottom of this page, in the light grey text at the very bottom, you can see how many milliseconds of CPU time were needed to render this forum thread.

Why not to put the actual web server behind Cloudflare that will handle efficiently such abuse free of charge?

Why not to put the actual web server behind Cloudflare that will handle efficiently such abuse free of charge?

That's answered up-thread in /forumpost/add443de2a933a3e and its follow-ups.

We were NOT getting 1 million queries to the same URL. We are getting 1 million queries to 1 million distinct URLs. Every request is unique (well, mostly, certainly to a good approximation). I don't see how a cache helps in that scenario.

There are currently more than 34,000 distinct check-ins in the SQLite source tree. We are getting requests for differences between every distinct pair of check-ins. That's already more than a billion combinations. But there's more. Each request might look something like this:

`https://sqlite.org/src/vdiff?from=0fe77341a0f1e869&to=172f4e4772d90f47

Where the from= and to= query parameters specify the specific check-ins to diff between. The values on those query parameters can be any unique prefix of the 64-character SHA3-256 hash that identifies the check-in. Or they can be any tag associated with the check-in. So even for the same diff, there are many different ways to specify the same diff. And there are other optional parameters that can alter the diff output in various ways (ex: side-by-side versus unified)

Add all this together and we are easily into billions and billions of distinct pages. And that's just for the /vdiff page. There are other pages on the SQLite.org/src website with just as many combinations. In the bot-net problem we were having a few days ago, most of the requests were for a different one of those billions and billions of distinct pages.

How does cloudflare help with that?

How does cloudflare help with that?

There still is benefit using CDN.

• ClodFlare supports: Edge Cache by query rules, that can be adjusted for specific patterns that shouldn't be aggregated by spiders. Specific query, like vdiff?from=/to= probably should be exposed to authenticated users only. Those links doesn't bring popularity to sqlite from SEO point of view and those who really need such queries most likely will download repository and use fossil-scm to query such links locally.
• Bot Mitigation: that will eat up on CF side most of bad bot traffic since they have statistic regarding bad/good. Also "Bot Fight Mode"/"Super Bot Fight Mode" can challenge or block scrapers on CDN side.
• Rate limiting: CF already has dynamic statistic regarding offensive IP and even redistributed attack from millions of IP might help to slow down traffic to a target
• JavaScript/Captcha challenge: will stop simple scrapers that don’t handle JS on CF side without touching a target

So, Cloudflare’s WAF/bot protection still does work on dynamic pages.

What is feeding the bot the billion combinations?

Fossil is. Visit the timeline page of any repo and you will find an inexhaustable supply of links.

I'm constantly told that my "we have detected that your device has engaged in suspicious activity"

It's a clear case of "hate the game, not the player." Bot swarms like the recent one are an existential threat for small-scale web hosting. The people behind these bots are the proverbial reason we can't have nice things (like captcha-free access to websites).

Absolutely agree!

Having said that, it still might be sensible to add such high resource use pages to the robots.txt exclusion file, which currently contains none of them...

I guess I'm still confused how /timeline can provide such a huge number of links to crawl without some kind of extra effort on the part of the crawler and knowledge of how to interpret commit hashes as inputs to from and to parameters for /vdiff to generate billions of combinations.

While an interesting question, this would appear to be rather besides the point, would it not?

This isn't a botnet dynamic website scraping how-to discussion forum, after all.

Which pages accessible from /timeline provide generated links for /vdiff?from=&to= that are exposed to crawlers?

Not only do the direct links lead to incalculable numbers of other links, but tapping on any two of the small dots in the timeline will diff against those specific two versions. With a 50-entry timeline (the default on SQLite's site), that's another huge number of combinations of links, each of which tends to get more expensive to calculate as the distance (in check-ins) between each pair of versions grows.

I guess I'm still confused how /timeline can provide ... for /vdiff to generate billions of combinations

Billions is likely a conservative estimate but, frankly, my brain's not big enough to do even a rough estimate of the math.

Try manually browsing from page to page in any moderately-sized fossil repository and in your natural lifetime you won't be able to follow all of the distinct links.

Which part of Fossil will generate crawlable links for every distinct pair of check-ins?

The timeline does. Clicking any two of the versioned dots on the timeline will diff between those versions, and the display length of the timeline can be controlled via a documented URL parameter (not linked to here to avoid throwing oil on the fire).

Not all /vdiff links are directly reachable by a crawler. But many are.

Here is an experiment you should run:

• Write a crawler (or maybe one already exists that does this—I dunno) that does not save page text, but just remembers all hyperlinks it find on the page and save the hyperlinks in a database, crawling each hyperlink just once. Take care to remove duplicates hyperlinks. Ignore all links that go "off site".
• Clone a copy of the SQLite Fossil repository to your local machine.
• Run "fossil ui" on the clone so that it presents a full website on http://localhost:8080/
• Set your crawler loose on http://localhost:8080/timeline
• Let us know how many links it find and how much real-time and CPU time it took to find them all.

Something like

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://localhost:8080/timeline

should work just fine.

Could you not identity a crawler by adding some link on each page to an internal url that is not really visible to a normal user but when followed you know it is a crawler? Then you can deny further access. A fair crawler would not check the site because of the robots.txt.

In a bot-net, each HTTP request comes from a different IP address and a different browser. Millions of machines, all over the world, each send a single HTTP request and get back a single reply, and we never hear from them again.

How do you deny further access when no further access is ever even attempted?

Links on generated pages should then only be allowed to be followed by the original requestor of the generated page. For example by encoding the requestor ip in or other technical means. Then botnets cannot access those links.

Links on generated pages should then only be allowed to be followed by the original requestor

So you are no longer allowed to copy/paste a link to Reddit or HN or Twitter/X? or to an email? Seems kinda harsh.

Don't forget that whatever generates the links is open source software.

Security through obscurity merely delays the inevitable. In the case of Fossil, they would be links to its own source code containing the procedure for generating the links.

And if you are suggesting that, rather than some algorithmic deterministic link creation there is instead a single use random coding system used then you have to save all of that state in a database somewhere to determine if a link is valid.

Your solution is too clunky and expensive.

And also renders makes the utility of sharing links worthless.

I cobbled together one with Tcl ... So far it has discovered 376,489 unique URLs...

Keep in mind that yours is not processing JavaScript, but industrial-grade bots do (and have been since at least 2010 or 2012). That is: they can interact with content which a non-JS-aware browser can't, like the clickable dots on the timeline. JS code can detect the presence of those and programmatically click them. They're also not hindered by UI elements which are invisible because they don't fully lay out the page, they just crawl the resulting DOM tree using either JS or C or C++ or Rust. (At least, that's how a sophisticated crawler would work, as opposed to one which regexes links out of the HTML (the humble beginnings of all crawlers, of course).)

As you say:

One would think that such a tool already exists, but it's hard to find...

My strong suspicion is that they can be purchased in "gray markets" if one knows where to look.

Related article on The Register published yesterday: AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders.

It's worth pointing out that I was getting enormous amounts of AI crawling over the brotli test website I setup a while ago, until I added a robots.txt exclusion file to the root.

Following which it all stopped.

I'm sure some crawlers are evil and clever and malign.

But if there is no indication that they should avoid crawling /src (which there currently isn't) then ongoing attention is probably to be expected.

It would appear quite straightforward to combine the current /src/robots.txt file (which is not used by crawlers) with the /robots.txt file (which is used, but doesn't contain any relevant instructions).

Could you generate all links dynamically such that the IP-address of the client is encoded in the URL itself. And then decode all URLs - and check whether the encoded URL and the incoming IP match up - if they don't - redirect to a 'entry' point URL, which would then allocate the URL with IP-encoding in it?

The idea being - if 'browser-A' learns of 'link-x-ip-n.n.n.n.', it can follow it - and so can 'bot-A' - but 'bot-B' (different IP presumably) wouldn't be able to ?

If 'bot-A' is still a nuisance, you could stop it or rate-limit it based on the IP.

Edit: I see this is probably the same idea as one of the anonymous ones above - aside from adding the redirect - to make the shared original link work.

Edit 2: The botnet would just have drop part of the URL to get a freshly valid link though....easily defeated...

Dunno: add a learning model to see if it can learn the shape of the botnet and the path it is following maybe...then automatically rate-limit (incrementally escalating time, like sshguard does - albeit for a single IP) any IPs it thought are participating in the botnet?

... such that the IP-address of the client ...

That doesn't work. Many users are behind large ISPs using NAT so requests from the same end user system will appear to come from different IP addresses.

In any event what you are describing can be implemented using signed cookies, which is roughly what the anonymous login does.

The approaches taken so far have been to fingerprint the client (eg user agent, language, crypto suites in TLS etc), and combine that with proof of work: make the client computer spend time and energy calculating something pointless, or make the human spend time and energy calculating something pointless (captcha). This makes for a miserable experience for humans, and keep being overcome by the crawlers leading to a constant arms race.

Many folks have given up and outsourced the problem to Cloudflare who are doing that work, but they have false positives which is really annoying to the affected people. There are no easy solutions.

It would seem to me that if Fossil instead limited the ability to generate /vdiff URLs dynamically from it's own website, then it wouldn't matter what crawlers did (unless we think crawlers are being adapted to crawl Fossil hosted repositories).

That's precisely what's happened. The first cases of bots logging in to fossil go back to 2010 or so, where they were seen polluting wikis and tickets on repos which were set up to allow anonymous users to append comments to those¹. Nowadays bots run full headless browsers and can do anything with the site a human can do (even more, because they're not limited by the visibility of DOM elements, and catching them interacting with such elements is a strong hint that they're a bot).

I have a hard time believing that crawlers know enough about Fossil internals to know that they can obtain a list of commit hashes from /timeline and then to call /vdiff?from=&to= using 2 of those commit hashes.

They don't need to know about internals for that. They can either use trial-and-error (which is what machine learning is all about) or they can read it in the the docs which accompany every single fossil repo. When one bot learns it, all of its compatriots learn it.

Bots running their own JS engine (which need not play by the same rules as the one in an end-user's browser) can either programmatically detect which elements are clickable or they can just click all elements and see what happens. The next time around, if they're sophisticated enough, they will have learned what can be clicked and what can't.

1. ^{^} Recall that "anonymous" requires a login in fossil, as distinct from "nobody", which is the not-logged-in user.

I wonder what possibly could be an intent for such attack/misbehavior? Is it to just mine for whatever, DDoS out of malice, trying to crash and "take over"?

What's the expected end-game for the perpetrators? Trying to guess this could help in setting up some honey-pot approach. For example, some links may be always different, yet set up to be circular in nature, thus tying in an attacker infinitely.

It's probably the same reason you get robo-calls trying to sell you stuff. I seldom get crank-calls, but two days ago I received an AI call. I think, it was my first time.