DocsStart here
Bot filtering
How Datacenter keeps bots, crawlers and automated browsers out of your numbers, why hosting traffic still counts as people, and why they are never billed.
On this page
A lot of web traffic is not people. Datacenter filters it in a few layers, so your numbers stay close to real visits.
What is dropped
These events are never written:
- A user agent that names a crawler, a link preview, an uptime monitor, a headless browser or an HTTP library such as
curl. An empty user agent and a bareMozilla/5.0count as bots too. - Requests from verified bots, such as search engine crawlers.
- Browsers that report
navigator.webdriver, which most automation tools do. The script sends nothing from them. - Browsers that cannot send a beacon. Real browsers today all can.
What is marked
What gets through is written with a set of flags, so it can be judged again later.
| Flag | Means |
|---|---|
| Unknown user agent | The user agent names no browser and no operating system |
| Bot score | The network's bot score says it is likely automated |
| Hosting network | The request came from a known cloud or hosting network |
The dashboard leaves out events with either of the first two flags.
Why hosting traffic still shows
Many real people browse through a VPN or a privacy relay. Those often run on cloud and hosting networks. If we dropped every visit from one, we would undercount real people. So a hosting network flag on its own does not remove a visit from your stats.
It does keep it off your bill. Pageviews and clicks from a hosting network never count toward your monthly allowance or extra use, so a script on a cloud server cannot run up what you pay.
Pageviews need a visible tab
The script waits until the tab has been visible for 1.5 seconds before it counts a pageview. Tabs that open in the background and close unseen are not counted. That also stops many simple bots that load a page and leave at once.
Something missing or wrong? Email hello@datacenter.so.