Austen, Eliot and the Messy Business of AI Visibility & Prompt tracking.
Prompt tracking can be useful, but only if you know what the data can, and can’t, tell you.
Over the past year there’s been a lot of articles published to help you dip your toe into the world of AI prompt tracking. But there’s a problem.
Well, two problems, actually.
The first is this: trying to explain how AI actually works is not particularly easy, especially with a technology that’s evolving so quickly.
The second is what I’m going to call Austen’s Law of Predictability, which goes a bit like this:
“It is a truth universally acknowledged, that a single man or woman in possession of AI visibility data, must be in want of a LinkedIn post declaring they have cracked the code once and for all.”
And that’s because, when it comes to AI visibility, precision is tricky.
The biggest mistake I see businesses making right now is treating prompt-tracking tools like SEO ranking tools. In other words, they assume you can simply pop in the prompts you want to track and et voilà! - you can “track” your exact visibility in neat daily or weekly charts. If the tool tells you that you’re winning, then you must be winning.
The problem with this is what I’ll call Eliot’s Law which goes something like this:
‘“Humankind cannot bear very much reality. Faced with uncertainty, people will always choose a nice, shiny AI visibility dashboard.”
Now, I like a nice, shiny AI visibility dashboard as much as the next guy, but if you don’t have a strategy in place before generating your prompts, or understand how an LLM generates an answer through variance, then the value of that dashboard becomes redundant pretty quickly.
And that’s because:
AI answers are probabilistic, not deterministic.
MEANING. The same or similar prompt can produce different answers depending on the model, phrasing, location, timing, user context, retrieval sources, and even random variation in generation.
WHICH MEANS A dashboard may tell you what happened for a sample of prompts at a particular moment. It does not tell you exactly what every user is seeing.
This is why I think when you’re reporting back on the main trends in your AI visibility report, you also need to flag the limitations behind the data. It should make clear whether any changes during the time period selected are statistically meaningful, directionally interesting, or just noise.
You should also caveat any presentations by explaining your prompt methodology: which prompts were tracked, why they were chosen, how often they were tested, and across which models or platforms.
Also, better reporting should always combine dashboard data with qualitative analysis: what answers say, which sources are cited, how brands are framed, and whether the responses are actually useful or accurate.
Sounds messy? Well, here are some tactics I’ve tried that have helped with reporting back on performance.
Start With the Business Objective
When starting out with AI prompt tracking it’s very easy to start generating a LOAD of prompts that COULD be related to your business but are not CORE to the business.
Many tools will offer the ability to AI-generate your prompts for you at a click of a button which is all very empowering, but do they reflect the CORE questions that your business needs to answer?
An example of that is using a prompt to compare yourself with a competitor. Insights like these can often be used to create a gap analysis to flag content that your competitor is doing and what you’re not.
But before you plunge into the task of writing 50 articles on topic A or B, stop. Is this content actually CORE to your business or are you just producing content for content’s sake? Is this the right competitor to track performance against? Is this even the right prompt?
You are ALWAYS in the stronger position if your prompts map to a specific and realistic business objective - which is why you need to establish those before you go anywhere near a prompt tracking tool.
Do you want to generate more sales? Are you looking to address negative brand reputation? Are you trying to promote a new product which is being misrepresented?
Nailing those objectives will really help you avoid generating hundreds of vaguely related prompts and being lost in the response data.
Make Every Prompt Actionable
If you’ve achieved step one, then step two should be a breeze. A prompt should not just be relevant. It should create a decision.
For example, if the answer changes, what would we actually do differently?
That question matters because when the connection between prompt and business goal is self-evident, stakeholder sign-off becomes much easier when you come to deliver your action plan.
So, have a strategy for your prompt generation and THEN you can get into things like entity consistency, schema, precision writing, what third parties are saying about you, topic clusters etc.
Map Prompts to the Buyer Journey
Your prompts need to reflect where the buyer is in relation to your brand which is typically:
Awareness
Consideration
Decision
You need to look at the prompts that follow the audience all the way from discoverability through to final evaluation. I will often spend time here looking at the actual language that’s used by the potential customers - in surveys, review sites, subreddits etc. You really can’t overvalue this kind of research and it’s something you should be very careful about delegating to AI.
Once mapped, and depending on your weapon of choice, you can tag your prompts for easier reporting.
Very much connected with step 3 is to profile your actual consumers. However, as someone who has seen many beautiful persona slides in my life, I would always seek to ground this in actual data and not fanciful imaginings. Want to test this method quickly? Kevin Indig has a neat tool here.
Ok, so we are coming to the end of this ‘stack’ so let’s round off with a nice modest potential LinkedIn post title…
HERE’S THE ULTIMATE ANSWER TO ALL YOUR AI VISIBILITY WOES
Here goes. Want to excel at any type of visibility? Then get better at telling the story about your company. Who you are, what you do, the impact you are making, the audience you are serving. You’d be surprised at how poor we are about telling our own brand stories.
And that’s really it. All hail the brand marketing gods, you really were onto something.
Lastly, get someone with strong brand sensibilities who also understands the technical framework needed to make your brand legible to AI systems.
I may know a guy.