Fixing Gaps in AI Visibility Tracking
Most businesses check their AI visibility by typing a question into ChatGPT now and then and seeing whether their name comes up. That produces a feeling, not a measurement.
Answers change from one run to the next, differ between assistants, and shift with location, account settings and model updates, so a single check cannot tell you whether anything you did made a difference. This guide shows how to turn those spot checks into a steady record: what to count, how to hold the test conditions still, how many runs you need, and how to read a change without fooling yourself.
- Signs your tracking cannot be trusted
- Why one check tells you so little
- What to count, and what each figure tells you
- The fix, step by step
- A record layout that works
- Connecting answers to visits
- Reading a change without fooling yourself
- Edge cases
- Keeping the measurement honest over time
- Questions and answers
Signs your tracking cannot be trusted
A tracking gap rarely looks like a gap. It looks like a set of numbers that nobody quite believes. If any of these sound familiar, the problem is the measuring, not (yet) the marketing:
- The result depends on who checked. One colleague says ChatGPT recommends you; another, asking a slightly different question from a different account, says it never does. Both are telling the truth about what they saw.
- The prompts keep changing. Each month someone adds a few questions and drops a few others, so this month's figure is not comparable with last month's.
- You only have ticks. A spreadsheet says "yes" or "no" against each question, but nobody kept the wording of the answer, the sources shown or which competitor was named instead.
- A single run decides the verdict. One good answer on a Tuesday is reported as a win; one bad answer the following week is reported as a drop.
- There is no record of what changed on the site. Visibility went up in June, but no one can say whether that followed the new comparison page, the press coverage or nothing at all.
- Assistants are lumped together. A combined "AI visibility" figure hides the fact that Perplexity names you often while Gemini does not mention you at all.
Each of these is fixable with a little discipline, and none needs expensive software. What matters is that every run is done the same way and that the raw evidence is kept.
Why one check tells you so little
Traditional rank tracking works because a search results page is fairly stable for a given query, place and device. AI answers are not like that, for several reasons that are worth understanding before you design a measurement.
The same question gets different answers
Language models generate text with an element of randomness, so asking an identical question twice can produce two different lists of suggestions. Assistants that search the web add a second source of variation: the pages they retrieve can differ between runs, and each run may cite a different handful of sources. A brand that appears in three out of five runs has a real but partial presence, and a single check has a good chance of showing either extreme.
Context changes the answer
Signed-in accounts can carry memory, saved instructions and past conversations that shape later replies. Approximate location affects local questions. Some assistants search the web for one question and answer another from what the model already knows, and the choice can differ between runs. Free and paid tiers may use different models. If these conditions drift between checks, you are measuring the drift as much as your brand.
The assistants themselves change
Providers update models, add or remove search behaviour and change how sources are shown, often with little notice. A sudden step change across every prompt in one assistant, on the same day, is more likely to be a product change than something you did. You only notice that pattern if you track each assistant separately and keep dated records.
What to count, and what each figure tells you
Decide the outcome categories before the first run so that everyone scores answers the same way. The AI Visibility Tracker records brand presence as cited, mentioned, recommended or invisible; if you keep your own records, use the same four so the two can be compared.
| Outcome | What it means in the answer | Why it matters |
|---|---|---|
| Cited | One of your pages is listed or linked as a source. | The assistant used your site as evidence. This is the outcome most likely to send a visit. |
| Mentioned | Your brand name appears in the text, with no link to your site. | You are known in the topic, but the evidence came from somewhere else, often a review site, forum or directory. |
| Recommended | The answer suggests you as an option or ranks you favourably. | The closest thing to winning a recommendation question. Record your position in any list. |
| Invisible | Neither your brand nor your site appears. | Note who did appear, because that is the gap you need to close. |
From those outcomes you can derive a small set of figures. Keep the definitions written down, because small differences in counting produce large differences in the result.
- Mention rate: runs where the brand is named, divided by all runs, per assistant and per prompt group.
- Citation rate: runs where one of your URLs is a source, divided by all runs.
- Share of voice: your mentions divided by the total mentions of you and a fixed list of competitors across the same runs. Keep the competitor list stable, or the figure moves when the list does.
- Accuracy: runs where what is said about you is correct. A mention that gets your prices or location wrong is a problem to fix, not a win to count. The tracker surfaces wrong-attribution incidents for exactly this reason.
The fix, step by step
Step 1
Write down what counts as a result
Before running anything, agree the outcome labels and the rules for applying them. Use cited, mentioned, recommended and invisible, and decide the awkward cases in advance: does a mention of your founder count as a brand mention, does a link to your social profile count as a citation, and how do you score an answer that names you but gets a fact wrong? Put the rules in a short note that sits beside the data so that anyone scoring later reaches the same verdict.
Step 2
Freeze the prompt list and give it an owner
Choose a working set of questions, typically a few dozen, covering category questions, comparisons, problem questions and questions about your brand by name. Record each prompt word for word with an ID, a group and the page you would want cited. Then freeze it. New prompts go into a separate, dated batch rather than replacing old ones, and one named person approves any change so that month-on-month figures stay comparable instead of quietly shifting underneath you.
Step 3
Hold the test conditions still
Write down the exact conditions for each assistant and use them every time: which product and plan, signed in or signed out, whether memory and personalisation are switched off, the country or region, and whether web search is on. Where a clean session is possible, such as a temporary or private chat, use it. Record the model name shown in the interface with each run, because a model change is one of the commonest reasons a whole assistant moves at once.
Step 4
Run every prompt several times
Because answers vary, one run per prompt tells you very little. Run each prompt several times per assistant in the same session window, for example three to five times, and record each run separately. Report the share of runs in which you were named or cited, not whether the latest run happened to include you. The same approach applies whether you use the AI Visibility Tracker or check by hand; the point is the repetition, not the tool.
Step 5
Keep the whole answer, not a tick box
For every run, store the date, assistant, model, prompt ID, the full answer text, the list of sources with their URLs, every brand named and the position of yours. A tick loses the reasons behind a result; the full text lets you see later which competitor replaced you, which third-party page was cited instead of yours, and whether a statement about you was wrong. Storage is cheap, and old answers cannot be recreated once the assistant has changed.
Step 6
Log every change you make, with dates
Keep a simple change log alongside the results: the date, what changed, which pages or prompts it should affect and who made it. Include site changes such as a new comparison page or a robots.txt edit, off-site work such as a review campaign or press coverage, and outside events such as a competitor launch or an assistant update. Without dates, any later rise or fall is a guess; with them, you can at least check whether movement followed a change.
Step 7
Add AI referral traffic from analytics
Answers
tell you whether you appear; analytics tells you whether anyone clicks. In Google Analytics 4,
create a custom channel group with a rule matching AI assistant sources such as chatgpt.com,
perplexity.ai, gemini.google.com, copilot.microsoft.com and claude.ai,
placed above the Referral channel. Treat the result as a minimum, because some apps pass no referrer,
and compare it month by month against the mention and citation rates you record.
Step 8
Review on a fixed monthly rhythm
Collect data weekly or fortnightly, but make decisions monthly, looking at the rate per assistant and per prompt group rather than individual runs. Flag prompts that moved from invisible to cited, and investigate any that went the other way. Check the change log against each movement and note the assistant updates you know about. Once a quarter, ask whether the prompt set still reflects what customers ask, and add a new dated batch if it does not.
A record layout that works
Whether you keep results in a spreadsheet, a database or an export from the tracker, one row per run is the layout that keeps every later question answerable. A row per prompt, with the latest result overwritten each week, is the most common mistake: it destroys the history you need.
run_date,assistant,model_shown,search_used,prompt_id,run_no,brand_named,position,our_url_cited,competitors_named,sources,accuracy_note
2026-09-07,ChatGPT,shown in UI,yes,CAT-03,1,yes,2,no,"Oakridge Joinery; Millfield Stairs","example-reviews.co.uk/...; example.co.uk/stairs",ok
2026-09-07,ChatGPT,shown in UI,yes,CAT-03,2,no,,no,"Oakridge Joinery","example-directory.co.uk/...",
2026-09-07,Perplexity,shown in UI,yes,CAT-03,1,yes,1,yes,"Millfield Stairs","example.co.uk/stairs; example-forum.co.uk/...",wrong opening hoursKeep the full answer text in a separate column or linked file keyed on the same row. With this layout a pivot table gives you mention rate by assistant, citation rate by prompt group, share of voice against each competitor and a list of every inaccurate statement, without re-running anything.
Grouping prompts so the numbers mean something
Report by group as well as in total. Useful groups are:
- Brand: questions that name you. Low scores here usually point to access or identity problems rather than competition.
- Category: "who offers X in Y" questions. This is where recommendations are won or lost.
- Comparison: you against a named competitor. Watch accuracy closely here.
- Problem: questions describing a need without naming a product. These show whether you are associated with the problem you solve.
Connecting answers to visits
Mentions and citations are leading signals; visits and enquiries are what the business feels. Linking the two takes a little set-up.
A channel group for AI assistants in GA4
In Google Analytics 4, go to Admin, then Data display, then Channel groups, and create a new group by copying the default one. Add a channel called AI assistants with a condition on session source matching a regular expression such as the one below, and move it above Referral so those sessions are not swallowed by it. Channel groups apply to reports retrospectively, so you can see past data once it is set up.
(^|\.)(chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai)$Check your own source list before relying on the pattern, because assistants change domains and apps, and add any others you find in the Traffic acquisition report.
What the traffic figure can and cannot show
- It undercounts. Mobile apps, copy-and-paste and some privacy settings remove the referrer, and those visits appear as direct.
- It says nothing about unlinked mentions. A recommendation without a link can still lead someone to search your name later, which shows up as branded search or direct traffic instead.
- It is most useful as a trend beside your answer data. If citation rate rises for a group of prompts and AI referrals to the matching pages rise in the same months, you have two independent signals pointing the same way.
Reading a change without fooling yourself
Once you have a few months of consistent data, the temptation is to credit every rise to the last thing you did. A few checks keep the reading honest:
- Is the change bigger than normal variation? Look at how much each prompt group moved in months when you changed nothing. A movement no larger than that is not evidence of anything.
- Is it in the prompts you expected? A new comparison page should move comparison prompts. If category prompts rose instead, something else is probably at work.
- Is it in one assistant or all of them? A rise only in the assistant that just updated its model is more likely to be the update. A rise across several assistants that search the web is more consistent with a change on your side or in third-party coverage.
- Did the cited sources change? If your new page is now the cited source, the link is direct. If the answer now names you but cites a review site, the review site did the work.
- Does it repeat? One apparent cause and effect is an anecdote. When similar changes are followed by similar movements more than once, it becomes something you can plan around.
There is no published figure for how long a change takes to show up. It depends on how quickly the relevant crawlers revisit your pages, whether the assistant searches the web for that question at all, and, for answers drawn from a model's built-in knowledge, on training cycles you cannot see. Judge timing from your own records rather than from rules of thumb.
Edge cases
Very low numbers
For a small business with a narrow set of prompts, a mention rate can jump from zero to a third on the strength of one extra run. Pool results over longer periods, or group prompts more broadly, rather than reacting to single-run swings.
Local questions
Answers to "near me" or town-specific questions depend on location. If you serve several towns, track each as a separate prompt with the place named in the text, rather than relying on the assistant to infer where you are.
Answers with no sources
When an assistant answers from its built-in knowledge without searching, there is nothing to cite. Record these runs as "no sources" rather than counting them as citation failures, or your citation rate will fall every time an assistant decides not to search.
Brand names that are common words
If your brand shares a name with something else, automated matching will over-count. Score mentions by hand for a sample of runs to check that "named" really means your business.
Keeping the measurement honest over time
- Write the method down once. A one-page note covering the prompt set, conditions, number of runs, outcome rules and metric definitions outlives any one person's memory.
- Version the prompt set. Name batches by date, and never edit the text of a prompt that already has history; retire it and add a new one.
- Keep raw answers for at least a year. Year-on-year comparisons and questions you have not thought of yet both depend on them.
- Record assistant changes. Note model and feature changes in the change log as soon as you notice them in the interface.
- Re-check the competitor list. Review it quarterly, but keep an unchanged version running alongside any new one for continuity.
For turning the measurement into more citations once it is reliable, see fixing low citation frequency. For choosing prompts that match real customer questions, see fixing AI query match.
1. How many prompts should I track for AI visibility?
Enough to cover the main ways customers look for what you sell, and few enough that you can run each one several times and still review the results. For many small businesses that means a few dozen prompts split across brand, category, comparison and problem questions. A larger set is only useful if you keep it stable and run it consistently. Twenty prompts run properly every fortnight tell you more than two hundred run once and then abandoned.
2. Why does ChatGPT give a different answer every time I ask?
Language models include an element of randomness when they generate text, so the same question can produce different wording and a different list of suggestions. When the assistant searches the web, the pages it retrieves and cites can also differ between runs. Personal settings such as memory or past chats add further variation for signed-in users. This is why a single check is unreliable and why tracking should count the share of repeated runs in which you appear.
3. Should I track signed in or signed out?
Pick one set of conditions per assistant and keep to it, because consistency matters more than which you choose. A signed-out or temporary chat avoids memory and past conversations shaping the answer, which makes results easier to compare over time. Some features are only available when signed in, however, so if your customers mostly use a paid plan you may prefer to track that. Record the choice in your method note so nobody changes it by accident.
4. What is AI share of voice and how do I calculate it?
Share of voice is your share of all brand mentions in a set of answers, measured against a fixed list of competitors. Count how many times you are named across all runs, count the mentions of each competitor across the same runs, and divide your figure by the total. It is more informative than a raw mention count because it shows whether you are gaining ground on rivals, but it only stays meaningful if the competitor list and prompt set stay the same.
5. Can Google Analytics show visits from ChatGPT and Perplexity?
Partly. Visits that arrive with a referrer or source such as chatgpt.com or perplexity.ai show up in the Traffic acquisition report, and a custom channel group can collect them into one AI assistants channel. Some visits lose their referrer on the way, for example from mobile apps or when people copy a link, and appear as direct traffic instead. Treat the figure as a minimum and use it as a trend beside your answer tracking rather than a complete count.
6. How long before a website change shows up in AI answers?
There is no fixed or published timescale. For assistants that search the web, it depends on when their crawler or search provider revisits the changed page and whether the question triggers a search at all. For answers drawn from what a model learned in training, changes may not appear until a later model is released. Keep a dated change log and watch the specific prompts the change should affect, and let your own records tell you what is typical.
7. Is a brand mention without a link worth tracking?
Yes. Many answers name businesses without linking to them, especially in recommendation questions, and a named recommendation can still lead a customer to search for you. Mentions also show which topics an assistant associates you with. Track mentions and citations separately, though, because they have different causes: a citation means your own page was used as evidence, while a mention often comes from reviews, directories or articles written by other people about your business.
The AI Visibility Tracker queries ChatGPT, Claude, Perplexity, Gemini, Copilot and Brave Leo with your target prompts, records whether your brand is cited, mentioned, recommended or invisible, tracks how often over time, and identifies citation gaps against competitors.
Open AI Visibility Fixes →