When Google has already indexed nearly the entire open web, why would anyone spend $26 million to crawl it all over again? Because the most important user of search is shifting from humans to AI agents. Keenable's seed round, Exa's $2.2 billion valuation, and Tencent and Alibaba's search API overhauls all point to the same shift: agent search is becoming a standalone infrastructure layer. Whoever becomes the default interface for agents connecting to the web could become the next Google. This article unpacks three structural changes behind the rebuild: what search optimizes for, how it is priced, and how value is exchanged.
Humans Search Once, Agents Search a Hundred Times
The human loop is: Query → results page → click → read, and a session produces a handful of queries. The agent loop is a task cycle: Search → Fetch → Read → spot gaps → rewrite query → cross-verify → Act. A research agent runs dozens of searches to answer one complex question; a coding agent constantly hunts for fresh docs, issues and changelogs; a sales agent may check company websites, funding news, hiring pages and executive moves at the same time.
Search is no longer a deliberate click by a user; it is a step inside a task loop. Two consequences follow. First, agents do not need ten blue links; they need context that is complete, fresh, trustworthy and compact enough for further reasoning. Second, search volume scales by orders of magnitude: Exa predicts agent-initiated web searches will surpass human ones this year, and that LLM-generated search volume could reach a thousand times today's Google.
The Google Flywheel Does Not Turn for Agents
The moat of Google is a feedback flywheel built on human clicks: rank → click → satisfaction → optimize. Agents do not click; they read entire pages and act. Click-through rate and dwell time, metrics tuned for human behavior, mean almost nothing to a machine user. The new optimization targets are recall, freshness, latency, QPS, machine readability, token efficiency, provenance, and above all: what a completed task actually costs.
On top of that sits a classic innovator's dilemma. Google and Microsoft are both shrinking public search APIs to protect their own products, leaving a gap exactly where agent infrastructure needs to live. As our earlier piece on Keenable noted, Styskin is blunt: making humans leave Google is nearly impossible, but on agent queries, the giant is beatable.
From Cost per Search to Search Cost per Completed Task
Agents search a lot, so the cost structure changes. One task with 100 web searches: at Keenable's $4 per 1,000 requests for regular developers, that is $0.40 in search calls; at Exa's $7 base rate, $0.70. In production, you add fetch, page processing, model context and reasoning tokens on top.
The industry's unit of competition is shifting from cost per search to search cost per completed task. Alibaba Cloud OpenSearch's billing structure shows why: one web search can trigger an LLM query rewrite, result filtering consumes tokens, and the API itemizes input and output tokens for the rewrite and filter models. A single "search" is now a pipeline: query understanding → rewrite → retrieval → filtering → context assembly.
Hence the token-efficiency arms race. Agents usually need a few passages from a 5,000-word page, not the whole document. If the search layer strips ads, navigation and recommendation modules before sending context in, it saves not just search fees but downstream inference cost. A new cost chain forms: search quality → context quality → token consumption → agent cost. Search infrastructure and model infrastructure now appear on the same invoice. The MiniMax pivot to API token sales follows the same logic: as agent calls surge, whoever makes each call cheaper owns the entry point.
From Finding Documents to Querying the Web
Cost is only the first layer. The deeper shift is that agents may not want "search" at all. Ask one to "find European AI companies that raised Series A in the last six months, whose founders came from Google, and who are hiring inference engineers." No single page answers that. Funding lives in news, founder backgrounds on company sites and LinkedIn, hiring status on careers pages. The agent must Search → Fetch → Join → Filter → Verify.
Keenable's in-development WebQueryLanguage targets exactly this: composing multiple web sources to answer questions no single page can. Search systems are trying to turn the open web from a collection of pages into a machine-queryable dataset. If that holds, the endgame of agent search is not a smarter Google API but something like SQL for the open web: pages still exist, but the page may no longer be the primary unit of machine knowledge.
Three Fights Happening Right Now
Fight one: the broken value exchange. For two decades, a deal held: sites let Googlebot crawl, and Google sent clicks and traffic back. Agents may read ten sites, synthesize an answer and act, while the user never opens the original page. Publishers supply knowledge but get no visits, no subscriptions, no ad revenue. Cloudflare's Pay Per Crawl made the conflict explicit: sites can open to AI crawlers for free, block them entirely, or charge. The paradox: the more successful agent search is, the less traffic sites get; the more sites restrict AI crawlers, the more expensive independent web indexes become.
Fight two: security becomes perception. Traditional SEO shaped what humans see; agent search shapes what machines believe and do. A malicious page that enters agent context can use indirect prompt injection against an agent with GitHub, email, enterprise or payment access — a retrieval mistake now has operational consequences. Provenance, trust and security are becoming core search metrics. In a sense, agent search is the perception layer of an agent toward the real world.
Fight three: the closed Chinese web. Bocha and Xinliu are racing for agent search API positions; Tencent's connected-search API (minute-level freshness, around 300ms latency, and an agent-facing MCP launched on August 24) and Alibaba Cloud OpenSearch are retrofitting incumbent search capabilities. But almost no Chinese startup treats "crawl the entire web plus a 100B-grade independent index" as its core moat. Much of China's valuable real-time information lives inside WeChat, Xiaohongshu, Douyin, Zhihu and apps — not in a uniformly crawlable open web. A 100B index is not the same as seeing the Chinese internet. This is the other side of the coin we explored in our analysis of the zero-eyeballs ad dilemma: attention is shifting to new battlegrounds.
Who Wins: Three Endgames
Three routes are competing for the same position — the default interface for agents reaching the web:
- Independent indexers (Exa, Keenable): betting that a self-built index can become cheap and fast. The purest thesis, the most expensive to run, and exposed to publisher blocking.
- Incumbent retrofitters (Google, Microsoft, Tencent, Alibaba): they own indexes, content ecosystems and cloud entry points, but the innovator's dilemma makes them hesitate to cannibalize themselves.
- Closed-ecosystem connectors (China's Bocha, Xinliu): competing on authorized access to walled content, where licensing and platform relationships matter more than algorithms.
For developers and enterprises: ① put search cost per completed task into agent cost models instead of raw API price; ② prefer vendors with token-efficient, passage-level content extraction; ③ treat provenance as a production requirement — traceable citations are how you debug wrong agent decisions; ④ expect MCP to become the standard integration surface for search. Near term, independent indexers will capture the growth from AI labs and inference platforms. Medium term, whoever cuts completed-task cost by an order of magnitude defines this infrastructure layer.
FAQ
Q1: How is Keenable different from an AI search box like Perplexity?
A1: Perplexity is a consumer-facing AI search UI, while Keenable and Exa are search infrastructure for agents. They rebuild crawl, index and retrieval from scratch and expose it through REST API, MCP Server and CLI — no result pages, just machine-readable context.
Q2: Can agent search really replace Google?
A2: Not for humans, at least soon. But it will absorb agent-generated volume: Exa predicts agent searches will exceed human ones this year and could reach a thousand times today's Google. Even if Google keeps serving humans, agent traffic alone can fund an independent layer.
Q3: Why is the Chinese internet a special problem for agent search?
A3: Most real-time Chinese information lives in closed ecosystems like WeChat, Xiaohongshu and Douyin that cannot be uniformly crawled. In China, connecting closed ecosystems matters more than index size; licensing and platform relationships are harder moats than algorithms.
