How much of my website traffic is actually bots?
More than your analytics shows, because analytics is the wrong instrument for this particular question. A JavaScript tag only counts visitors that run JavaScript, so plain crawlers and scrapers never appear in it at all, while the ones that do render a page get filed as Direct and read as people. Server or CDN logs count every request instead, and that is where the real ratio lives — in one three-hour sample of our own origin logs, roughly three quarters of requests came from datacentre clients with spoofed or empty user agents. Measure it from the logs once before you read another traffic chart.
Your analytics tag and your server log are counting different things
The question is hard because the two places you could look disagree by design, and neither of them is lying. A page-view tag fires from the browser after JavaScript has run, so it counts rendering clients — real people, plus the minority of bots that drive a full browser. A server or CDN log records every request that reached your origin, so it counts fetches — including the crawler that took your sitemap and pulled four hundred pages in a minute without executing a single line of your code. The same site can honestly report two hundred sessions in analytics and twelve thousand requests at the edge on the same day.
Which number is right depends on what you are deciding. If you want to know whether your writing is reaching humans, the tag is closer to the truth and the log will flatter you. If you want to know why your hosting bill moved, why your rankings feel disconnected from your traffic graph, or whether last Tuesday's spike was real, the log is the only witness. The mistake is not choosing wrongly between them — it is reading a number off the instrument that cannot see the thing you are asking about.
- Visible only in the log — plain crawlers. Search crawlers, AI crawlers, feed readers, uptime checkers, security scanners and the scrapers republishing your content all fetch HTML and stop. None of them execute your analytics tag, so none of them exist as far as your dashboard is concerned.
- Visible only in the log — everything that is not a page. Images, fonts, JSON endpoints, RSS, robots.txt, and every 404 somebody probed for. This is usually where an unusual pattern shows itself first, because a bot hunting for admin panels has no reason to hide among your articles.
- Visible in both, and wrong in both — headless browsers. A bot driving a real browser runs your tag and arrives as a normal session with a device, a location and a referrer attached. These are the ones that quietly move your averages, because nothing about the row looks unusual on its own.
- Visible only in analytics — what happened after the page loaded. Scrolling, clicking, filling a form, signing up. A log can tell you a page was served; it can never tell you the visit was worth anything, which is why the answer is to read both rather than to switch.
The tells that a session is not a person
You almost never get a row labelled bot. What you get is a set of implausibilities that a human visitor could not produce, and they are easier to spot than they sound because automated traffic is bad at being inconsistent in the ways people are. Take any suspicious segment and ask whether one human body could have generated it.
| What you see | What it usually is | Why it looks like that |
|---|---|---|
| One visitor recorded in several cities on several devices within seconds | A single automated client, split into separate sessions by a rotating proxy pool | Scrapers that buy residential proxies change IP address per request, and analytics infers the location and device independently from each one. |
| A cluster of sessions with no city or region, all sharing one user agent | One script that never varied its fingerprint | Real browsers differ from each other; a default HTTP client sends an identical header every time, and geolocation databases often hold nothing useful for datacentre address ranges. |
| A viewport width that never changes, reporting from a small town you have never sold to | A headless browser at its default window size, running in a cloud region | Headless Chrome opens at a fixed size unless told otherwise, and cloud regions sit in a handful of small towns, so the location is a hosting fact rather than an audience fact. |
| Every page bounced in under two seconds, visited in your sitemap's order | A crawl walking your links | It is reading, not browsing. There is no dwell time and no scroll, and the sequence matches your internal linking rather than anybody's curiosity. |
What we found when we audited our own traffic
This part is from operating agentceres.com rather than from documentation. We pulled three hours of raw origin logs and counted 1,438 requests spread across 558 distinct URLs — 2.58 requests per URL — and roughly three quarters of that came from datacentre clients with spoofed or empty user agents. The shape mattered more than the share: a long tail of pages each fetched two or three times is what a crawl looks like, and an audience looks nothing like it. It also had a price we could see, because those were mostly cache misses on pages no person had asked for, and they showed up as memory and bandwidth on the hosting bill rather than as readers.
A second audit, of our analytics rather than our logs, found two distortions and only one of them was bots. Our own team browsing the internal operations console accounted for about 35% of every pageview on the site — we were comfortably our own largest audience, and nobody had noticed because internal pages look like pages. Then, inside the traffic filed as Direct, roughly half did not survive inspection: one visitor recorded in four cities on four devices inside 1.7 seconds, a group of sessions with an empty city field all sharing a single user agent, and an 800-pixel-wide viewport reporting from a small Oregon town better known for hosting a large datacentre than for buying software. Of five referral sources listed that week, exactly one was a real referral.
The correction did not change what we should do next; it changed what we believed we already knew, which is worse. Two things fixed most of it and neither cost anything. Exclude your own team first, because it is the largest error and the easiest to remove. Then stop treating Direct as a channel and start treating it as a bucket of visits you have not identified yet — some of it is automated, and much of the rest is dark social, which is real word-of-mouth arriving without a referrer. Both live in the same row of the same report, and separating them is the entire job.
FAQ
- Is bot traffic bad for SEO?
- Mostly no. Search crawlers are bots and you want them; being crawled is how you get indexed at all. What can hurt is volume without value — a very large site where crawlers spend their time on parameter URLs, duplicates and dead ends instead of your real pages, which is the crawl budget problem. For a small site, bot traffic is a cost and a measurement problem long before it is a ranking problem.
- Doesn't Google Analytics filter bots automatically?
- It filters known bots and spiders against a published list, and that list is by definition made of clients that identify themselves honestly. Anything sending an ordinary browser user agent from a cloud server is not on it and never will be, because there is nothing to match. Automatic filtering removes the polite half of the problem.
- Should I block AI crawlers?
- That is a strategy decision, not a hygiene one, and it is worth making deliberately rather than by default. Blocking reduces load and stops your text feeding a model; it also removes you from the systems that increasingly answer questions your customers ask. The trade-offs, and which crawler controls which behaviour, are in what an AI crawler is.
- Why is my Direct traffic suddenly so high?
- Direct is not a source, it is the absence of one, so it grows for two unrelated reasons at once. Automated clients arrive with no referrer, and so do real people clicking a link in a message, an email client or a private community. If Direct is climbing, split it before you celebrate it or dismiss it — see dark social for the half that is genuinely good news.
- Does this mean my traffic numbers are fake?
- No. It means one of your numbers is answering a different question from the one you are asking it. Sessions from a JavaScript tag remain a decent proxy for humans reading your pages; requests in a log remain the truth about load and crawling. Trouble only starts when a number from one is used to make a decision that belongs to the other, which is also the argument in how do I measure if my marketing is working.
Want this done for you?
AgentCeres is a managed AI marketing team — specialists draft the work, you approve what ships. 14-day free trial, from $39/month.