Trust layer
DeadRiskBot
Our crawler. What it looks for, how slowly it moves, and the one line that switches it off.
DeadRiskBot/1.0 (+https://www.deadrisk.com/bot; greg@deadrisk.com)
What it reads
DeadRiskBot looks for news signals to understand what drives markets and the AI industry.
The premise is that the mechanisms of capitalism are legible. Read enough of the record, in order, and you can see who moved first, what was announced the week before the price answered, and which story an industry told itself before it knew better. That is work for truth-seeking machines — models pointed at what happened rather than at what is popular to say about it.
A model is only as honest as the record underneath it. This is how that record gets built: slowly, from public pages, with a name on it.
It does not touch anything behind a login or a paywall, does not submit forms, and does not follow links into search, checkout, or session URLs. It issues GET requests and nothing else.
How slowly
Slowly enough that you would have to go looking to notice.
| Target | Rate | Concurrency |
|---|---|---|
| Any single site | 1 request / 14 min — about 100 a day | 1 |
| Internet Archive | 1 request / sec | 1 |
Where the Internet Archive already holds a copy of a page, we read that instead of asking you for it. Most of what we need comes from there, which is why the rate above is as low as it is.
What happens when you say no
A 429 or a 403 stops the run. It does not back off and retry, and it does not rotate anything to get around you; a refusal trips a breaker and the crawl waits for a human. That is the whole retry policy.
To block it outright, add this to your robots.txt:
User-agent: DeadRiskBot Disallow: /
Honoured within 24 hours. A Crawl-delay
is honoured too, if you would rather slow it down than shut it out. And you can
always just email — that reaches a person,
usually the same day, and it is a faster way to get whatever you actually want.
If it misbehaves
Send the log lines to greg@deadrisk.com and it gets switched off while we look. A crawler that cannot be reached is a crawler that cannot be corrected, which is the reason this page exists at all.