AI search visibility
How assistants find, fetch and cite pages — separating the mechanisms you can verify from the tactics that are only being sold.
Search stopped being one thing. A page can now be reached by a classical crawler that builds an index, by a training crawler that will never send a visitor, and by a fetcher acting on one person’s question in a chat window a second after they asked it. Those three want different things from your server and leave different traces in your logs.
Almost everything written about this subject is sold rather than checked. That is the gap these pages fill. Where a mechanism is written down in a specification or in an operator’s documentation, they explain it precisely and quote the syntax. Where you can observe it on your own server in one command, they give the command. And where a claim is about the inside of a ranking system nobody outside the lab can see, they say that plainly instead of dressing a guess as a technique.
The practical consequence is that this cluster asks you to measure rather than to believe. Most of the pages end in something you run.
How AI Assistants Choose Which Pages to Cite
The pipeline between a question and a citation list, stage by stage, marking which stages you can observe and which nobody outside the lab can.
9 min read
Generative Engine Optimisation: What Is Real and What Is Sold
A three-tier test for every claim in the GEO field, applied to the common tactics one by one, with the evidence each would need to be believed.
10 min read
llms.txt: The Proposal, the Adoption, the Honest Status
What the llms.txt format actually specifies, why publishing it is not the same as anything reading it, and the one-line log query that tells you which is happening to yours.
8 min read
robots.txt for AI Crawlers
The agent tokens, the exact directive syntax, the group-matching rule that silently breaks most AI robots.txt files, and why none of it is enforcement.
10 min read
Finding Out Which AI Bots Visit You
A log parser that identifies AI crawlers and verifies them properly — reverse DNS for the search engines that support it, published address ranges for the ones that do not.
11 min read
Structured Data That Machines Read
Which schema.org types are worth the effort, the rule that keeps structured data honest, and the reason JSON-LD is invisible to a plain text extractor.
10 min read
Writing Pages a Model Can Quote Accurately
Why a retrieved passage arrives without its page, and the sentence-level rewrites that make a paragraph survive being read alone.
9 min read
Zero-Click Answers and What Happened to Referral Traffic
Why the published figures on AI search and referral traffic disagree so violently, what is genuinely measurable from your own logs, and what is not measurable by anyone.
9 min read
Content Licensing Deals Between Publishers and AI Labs
The five structures these agreements take, which terms are disclosed and which never are, and how litigation and infrastructure changed the negotiating position.
9 min read
Paywalls, Crawlers and Selective Access
The documented way to let a crawler see paywalled content without cloaking, the access models publishers are testing, and where the line actually sits.
9 min read
Measuring Whether AI Assistants Mention You
A repeatable sampling protocol with the binomial arithmetic worked out, showing why a thirty-run study cannot detect the improvements this field advertises.
10 min read
When an AI Says Something False About Your Company
How to tell which of four mechanisms produced the false statement, and which corrections have any chance of working for each one.
10 min read
Retrieval-Friendly Site Architecture
How chunk boundaries, heading structure and the one-answer-per-URL rule decide whether your site can be retrieved as passages rather than pages.
9 min read
Sitemaps, IndexNow and Getting Crawled Sooner
A working IndexNow implementation with the status codes, the state file that stops it resubmitting everything, and a clear account of what it does not do.
10 min read
JavaScript Rendering and What Crawlers See
Fetch your page as a bot, diff the raw HTML against the rendered DOM, and close whatever gap you find — with the measurement script included.
10 min read
Programmatic Pages Without Becoming a Content Farm
The test that separates a useful template from scaled abuse, with both shown, plus a similarity script that tells you which side yours is on.
9 min read
E-E-A-T for Technical Content, Concretely
What E-E-A-T is and is not, and how to turn each of its four adjectives into a page element somebody could point at.
9 min read
What Makes a Page Worth Quoting
Four properties derived from how retrieval and citation actually work, each with a test you can apply to a page in under a minute.
8 min read
Answer Engine Optimisation: What Changed in the Results Page
The dated product changes that reshaped the results page, and a harness that measures how far down the first organic link now sits on your own queries.
9 min read
An Audit for a Site That Wants to Be Read by Machines
Twenty checks with a command and a pass condition each, covering access, delivery, structure and evidence — runnable in an afternoon.
12 min read
Other topics
- LLM fundamentals & architecture
- Tokens, tokenization & context windows
- Prompt engineering
- Reasoning models & test-time compute
- Multimodal AI: vision, audio, video
- RAG & retrieval
- Embeddings & vector search
- AI agents & tool use
- Structured output & function calling
- Fine-tuning & post-training
- Local inference errors, string by string
- Running local models day to day
- Testing code that calls an LLM
- Snapshot and property testing for model output
- Regression suites for prompts
- Eval gates in CI
- Flaky tests against a model
- Determinism and the cost of testing
- Contract and streaming tests
- Testing tool calls and retrieval
- Inference, serving & latency
- Rolling out a prompt change
- Testing AI systems in practice
- Forecasting a time series
- Machine learning on tabular data
- Geospatial data and models
- Understanding audio that is not speech
- Understanding video
- Core computer vision tasks
- Machine learning on graphs
- Point clouds and 3D
- Evaluation, benchmarks & LLM-as-judge
- Sensor and IoT data
- Logs and event streams
- Models over biological sequences
- Machine learning on molecules
- Embedding and searching code
- Extracting invoices and purchase orders
- Receipts, statements and tax forms
- Insurance policies and contracts
- Deeds, court filings and patents
- Extracting from medical records
- Observability & LLMOps
- CVs, certificates and identity documents
- Shipping, customs and technical documents
- Meetings, email, chat and filled-in forms
- Building an extraction pipeline
- Business, property and inspection documents
- Contract clauses and insurance claims
- Regulated and compliance documents
- Consumer, travel and closing documents
- Mapping one chat API onto another
- SDK and framework migrations
- Hallucination & failure modes
- Re-embedding and model deprecation
- Cutting over between providers
- Parity gaps, shims and legacy endpoints
- Moving between model versions
- Migrating vector stores and caches
- Mapping capabilities and parameters
- Migrating pipelines and agents
- Contracts, runbooks and rollback
- Auditing a codebase before a cutover
- Compliance and fine-tune migration
- LLM cost engineering
- Routing, cost tracking and multi-tenancy
- What a migration does to your prompts
- AI security & prompt injection
- Privacy, compliance & data residency
- AI governance, policy & society
- Building reliable AI applications
- AI hardware, GPUs & compute
- Open-weight models & local inference
- AI for developers & coding agents
- AI in industry: vertical playbooks
- AGI, superintelligence, alignment & the long future
- Machine learning foundations
- NLP fundamentals & classical tasks
- Data engineering for AI
- Synthetic data & dataset curation
- AI product design & UX
- Search, ranking & recommendation
- Enterprise adoption & change management
- AI careers, skills & teams
- Reading AI research
- AI in science & discovery
- Robotics & embodied AI
- AI economics, markets & business models
- AI myths, hype & media literacy
- Context engineering
- Shipping AI features: patterns & anti-patterns
- Build it: end-to-end AI tutorials
- Python for AI: hands-on recipes
- TypeScript, React and the web
- Frameworks and SDKs
- Errors and troubleshooting
- AI facts, numbers and statistics
- The history of AI
- The maths behind AI
- Architectures beyond the transformer
- Reinforcement learning
- Diffusion and generative media
- Speech, audio and voice engineering
- Benchmarks, one at a time
- Infrastructure and operations
- Databases and storage for AI
- Knowledge graphs and structured knowledge
- Classical ML in production
- Regulation, jurisdiction by jurisdiction
- Prompt recipes and pattern library
- AI for people who do not write code
- Writing, media and creative work
- Edge and on-device AI
- Interpretability and model internals
- Field notes
- OpenAI model behaviour
- Claude model behaviour
- Gemini model behaviour
- Llama model behaviour
- Mistral model behaviour
- Qwen model behaviour
- DeepSeek model behaviour
- Cohere model behaviour
- Grok model behaviour
- Small model behaviour
- Hybrid model architectures
- Token cost by language and script
- Transliteration, romanization and script handling
- Locale-correct output
- Multilingual generation quality
- Multilingual pipelines
- The EU AI Act, article by article
- AI under the GDPR and EU data law
- US AI regulation, state and sector
- International AI governance and standards
- AI litigation and enforcement
- Running AI workloads on AWS
- Running AI workloads on Google Cloud
- Running AI workloads on Azure
- AI at the edge: Workers, Vercel and Netlify
- Serving models on Kubernetes
- Operating AI infrastructure
- Quantization formats and what they cost
- llama.cpp, flag by flag
- Ollama and the desktop local-model runtimes
- Local models on Apple Silicon
- Hardware for local inference
- Running speech and embedding models locally
- Model files, adapters and conversion
- VRAM arithmetic for local models