Before a machine decides whether your page is relevant, it decides whether your page is worth waiting for. A lot of sites lose that first decision without ever knowing it.
In our latest episode of Get Discovered, host Joe Walsh sat down with Michael King, founder and CEO of iPullRank and creator of Relevance Engineering. They cover why ChatGPT drops slow pages, why most AI crawlers don’t render JavaScript, what to hide from real-time crawlers, and how to measure AI search when traffic isn’t the whole story.
What is relevance engineering?
Michael calls it the evolution of SEO: the point where AI, information retrieval, content strategy, UX, and digital PR meet. He leaves SEO out of that list on purpose, arguing that you can be a great SEO and still not know how anything works.
What’s changed is that open source versions of ChatGPT and AI Overviews exist. Teams can test content changes in a lab, watch how the pipeline responds, then push them live. For once, you can call your shot.
Why AI crawlers give up on slow pages
Michael’s team found that ChatGPT returns a 499 response code when a page takes too long to respond. The request is dropped because 499 sits outside the standard spec, most SEO teams never looked for it.
Rendering widens the gap. Googlebot renders on a budget and decides by page template. If the first 100 product pages it renders hold nothing extra, it may skip the 100,000 that do. In addition to this, the likes of ChatGPTbot and Perplexitybot fetch pages in real time and don’t render at all.
Michael advises starting with your log files and counting the 499s from AI crawlers. Zero 499s doesn’t mean you’re clear as a page can respond quickly and still arrive almost empty, or sit behind a CDN rule nobody has revisited.
What to hide from AI crawlers, and when to stop
Not everything should be open. Michael’s advice is to hide your proprietary material from the real-time crawlers that fetch pages on a user’s behalf. Otherwise, anyone can point an agent at your best page and republish it as their own.
That holds until the interface changes. Michael calls what’s coming machine media: apps that pull your content through MCPs and APIs and remix it into whatever format the user wants, from a podcast to a daily video briefing. Websites become the Yellow Pages, still there but rarely opened. When that happens, you open everything up. The signs are already showing, from mortgage calculators inside Google’s results to research suggesting machine traffic now outweighs human traffic.
How to measure AI search visibility
Michael sorts metrics into three buckets:
- Performance metrics: referral traffic, conversions, and time on site, the same as always.
- Channel metrics: share of voice, citation rate, and citation accuracy. Are you showing up, and showing up right?
- Input metrics: crawler activity, rankings for the synthetic queries AI platforms run in the background, entity salience, and response speed. These are the ones you can change.
Improve the inputs and visibility follows, then performance. The links can’t be proven yet, so some faith is required.
Treat visibility tools with care. LLMs are probabilistic at every stage, so accuracy doesn’t exist, only precision. A tool that checks once a day tells you little. Prerender’s own testing found API responses matched what real users saw less than 10% of the time.
The takeaway
Relevance starts with eligibility. If ChatGPT gives up waiting, or Googlebot skips your template, your content never enters the race. Michael advises checking your logs for 499s. Compare what each crawler receives and then build focused content, measure with patience, and guard your moat for as long as it’s worth guarding.
Tune in to the full conversation
Listen to the full episode of Get Discovered and subscribe across YouTube, Spotify, and Apple Podcasts so you don’t miss any conversation on the fundamentals of online discovery and machine experience.
Michael King is founder and CEO of iPullRank and the creator of Relevance Engineering. Try Pareto, his free tool for comparing what crawlers see, on the iPullRank website.