Discussion about this post

User's avatar
Selim Yoruk's avatar

I’d keep a small set of untouched pages in the report too. If their visibility moves with the pages you changed, the content team may be getting credit for a model update.

AISO Tools's avatar

Agree on intervals, and the same applies to zeros. We published a study last month that came out at 0 hits: 1,339 newly launched AI tools, 6,710 blind "best AI tool for..." questions to gpt-4o-mini, and none of the tools was named. The honest way to report that is an upper bound rather than "never": exact 95% CI 0 to 0.28%. When we re-ran a sample of 125 tools with web search required, we only had 625 answers, so the same zero widens to 0 to 2.91%. Same headline, very different claim.

On "a citation isn't a visit", the source/citation gap was the biggest thing in that search run. The model retrieved pages from 2,703 domains but cited 687. A Reddit thread was retrieved for 595 of the 625 answers and cited in 8. So a tracker that reports only citations and one that reports retrievals will tell you two different stories about the same answers.

Method, intervals and CSVs (disclosure: it's our report, and we sell monitoring for this): https://aisotools.com/research/chatgpt-ai-tool-visibility-2026?utm_source=substack&utm_medium=comment&utm_campaign=dist-20261103-substack-plgeek&ref=dist-20261103-substack-plgeek

2 more comments...

Ready for more?