Full disclosure: I run DevTune, which builds AI search tracking for dev tools. Read what follows with that in mind.
If you're a marketer running content programmes driven by AI search results, do you know how trustworthy the data behind those programmes is?
With insufficient data or infrequent sampling, a lot of the AI search numbers you're seeing from your tool of choice may feel good on paper. But they carry a wide range of potential variance. So when you see these numbers shift, it's very likely the shift is down to chance rather than any of the work you or your team have been doing.
Help your AI tools know your business.
Join HubSpot’s live webinar to see how Claude, ChatGPT, and other assistants can connect to your CRM context, so you can ask questions, take action, and update records without leaving the conversation. Then, learn how HubSpot’s governance keeps you in control.
Good news, or just noise?
Step into the shoes of a marketer at a dev tools company. You're working on a specific set of content, maybe in a competitive comparison area where you're seemingly falling behind.
Your AI search tracking tool tells you that you were mentioned in 40% of ChatGPT answers last month. This month it's 60%.
You'll immediately conclude the comparison pages worked. Some of that may be true.
But say that's 30 answers for a given prompt. At 40%, the range the real rate plausibly sits in runs from 27% to 55%. At 60%, it runs from 45% to 73%. That's significant overlap. The improvement feels good and looks good on paper. In reality it could just be chance.
Run 100 answers instead and the same two months come out at 32–48% and 52–68%. Now they don't even touch.
So the frequency of the sampling, and the volume of data you have over a given period, is critical to interpreting the results and drawing any conclusions at all.
The de facto value
I'm a bit of a statistics nerd, mainly from my time in growth running experimentation programmes. And it's always bothered me that I can ask an LLM the same question several times and get different answers, different recommendations, each time.
Yet most AI search platforms don't cater for that. In fact they gloss over it. They hand you a mean as the de facto value, the number you should adhere to and base all of your work on.
In reality there's a range: a confidence interval. At a 50% rate, 10 answers gives you a range of 27% to 73%. A hundred answers gives you 42% to 58%. That range needs taking into account whenever you compare across LLM platforms, across prompts, or between yourself and competitors.
To be generous to the platforms, they're probably not doing this intentionally. They just aren't thinking about that.
We've made this mistake before
It's always been a particular bugbear of mine. Folks confidently running experiments where the process is flawed in some key way that makes the results, or the interpretation of them, invalid.
Peeking at results early. Not setting the exposure or assignment event at the right time, and diluting the numbers.
And the more common issue: not having sufficient sample size, and not understanding how sample size affects the variance you may see. Whether you can draw meaningful conclusions, or whether the result could just as equally be down to chance.
We're seeing that creep into AI search programmes. Folks simply aren't shown the information about the variance in their results, so they don't know how much weight that data can bear when they make decisions on it.
Three more ways the number misleads you
Leading the witness
Include a brand in the prompt and you're effectively leading the witness. Expect the result to be biased.
Ask "Is Acme good for X?" and you'll get a very different set of data than asking "What's good for X?"
So when branded prompts sit inside your overall prompt set, expect the whole set to be biased, and make your analysis take that into account. At the very least you need segmentation, so you can analyse the branded subset separately from the non-branded set.
A citation isn't a visit
A citation isn't a visit. Neither is a source.
A source in an LLM response says this link was used in creating the response. A citation is something explicitly linked within it. Either could be your page, a competitor's or a third party's.
And the key thing is that these responses come from prompts you've defined. They're not live buyer queries, and until the LLM platforms open some of that data up, they can't be.
What they really are is simulated probes.
I see a lot of platforms claiming these are your real buyer questions, the prompts buyers actually use. That's just marketing speak. They're a set of simulated probes that you define, or that the platform suggests to you. Some may be based on research or broader aggregate data. They're still somewhat far from actual buyer questions.
Then think about what a citation actually is: one source link in a single response to one of these probes. ChatGPT Search averaged 3.42 of them per answer in our State of AI Search for Dev Tools report [LINK needed], across 54,747 answers from April to June 2026. So 400 citations might come from only about 120 answers. In many cases, 400 sounds bigger than it is.
Cited doesn't necessarily mean recommended, either. You can be cited heavily while the response recommends a competitor. I'm pretty sure most folks reading this have seen it happen.
Then there's where you sit in the answer. Are you the fourth or fifth link in a footer of sources, or are you named in the first sentence? Does your tool treat those as equivalent, or show you the difference?
For dev tools specifically, docs are incredibly important. You need citation rate, presence rate and mention rate for your docs, tracked independently of your marketing pages. That's how you get a full picture of how each content effort contributes.
Causation vs correlation
Comparing simulated probes in any before-and-after window can never tell you what caused business impact.
What it can show is that a page is being picked up by LLM platforms when people ask questions. That's the point of the probes. What it can't tell you is that these are the questions real people are asking in the wild. That's where customer research, support calls and sales call data come in, building a prompt set closer to the language your users and buyers use with these answer engines.
But it's still a little disconnected from business impact. That's only measurable on the AI traffic side.
So look beyond the probes. What actual AI traffic is reaching the site, and how does it change as you work on your content? How does it break down across training crawls and live answer fetches? Across each answer engine's bot? How is Perplexity finding, indexing and referencing your content, versus ChatGPT, versus Claude?
Then: how are those answer fetches translating into clicks within the answers, and referrals to your site? And a step beyond that: are those referrals converting into sign-ups, new users, demo requests? Are they high quality, high intent?
Those are the numbers that truly demonstrate impact. Measure them over different time windows. You want short-term reads for direction, but you need to keep monitoring over a longer period too. Your tool should cater for that.
It's also a much more compelling story to take to your peers and the exec team. Not only did responses to the probes shift, but AI traffic, AI referrals and conversions from those referrals are all increasing.
What I'd do
Question whether your tool gives you the data you need to make effective decisions on your content strategy and execution.
If it doesn't, these tools are misleading you. To be generous, perhaps not intentionally. But actively misleading you.
The result? You'll be spraying content at the wall and just seeing what sticks. Nowhere near as directed or efficient as it could be with better data behind your decisions.
So look for a tool that's transparent about what it measures and the insights it derives, in terms of ranges and confidence, and that helps you interpret them in a way that drives the work you and your teams are doing.
And when you report upwards on where you stand with AI search, or on the improvements you've made, do it in the context of the statistics behind those numbers. Then the interpretation is accurate, not misleading.
An untrodden path
AI search is a relatively new discipline. It's still very much a black box, and will be until the LLM platforms open some of their doors the way Google has for SEO over the years.
Marketers focusing on it now are at the forefront, treading an untrodden path. They're leading the way and finding things out as they go.
But I think it's up to those of us working in this space daily, at scale, to raise the bar for the discipline, and help marketers make sense of all this in truly meaningful ways.
That's what we're trying to do at DevTune. If you want the detail, here's how we measure AI search visibility, ranges and all, and the State of AI Search for Dev Tools report.
Until next time!
Book a free 1:1 consultation call with me - I keep a handful of slots open each week for founders and product growth leaders to explore working together and get some free advice along the way. Book a call.
View the State of AI Search for Dev Tools 2026 benchmark report - 126,814 citation records across 1,075 representative tool-evaluation scenarios, 43 developer-tool categories, and five AI search platforms
View your free public Dev Tool AI Market Presence Report - 500+ dev tools across 47+ verticals and growing, all in the Dev Tool AI Search Landscape.
Sponsor this newsletter - Reach over 11,000 founders, leaders and operators working in product and growth at some of the world’s best tech companies including Paypal, Adobe, Canva, Miro, Amplitude, Google, Meta, Tailscale, Twilio and Salesforce.
Thanks again to our sponsor: HubSpot, whose live AMA shows how to connect Claude, ChatGPT and other assistants to your CRM context.
