Should an ecommerce site block AI agents?

Blocking all of them costs visibility with shoppers who ask an AI assistant for recommendations. Allowing all of them lets scrapers in under an agent's name. Sort automated traffic by identity and purpose (verified search and shopping agents, training crawlers, unverified automation) and set a policy for each group.

AI traffic is several different things arriving under one label. Training crawlers collect pages in bulk to build models. Search crawlers index pages so that an assistant can cite them in answers. Fetchers retrieve a specific page at the moment a user asks a question about it. Browsing agents go further and act on a site for a user, comparing products or completing a purchase. A fifth group is ordinary automation that claims to be one of the others.

Each group brings a different trade. Search crawlers and user-triggered fetchers are how your products appear when someone asks an assistant what to buy. Blocking them can limit their direct access to your content; it does not guarantee that your products disappear from answers. Training crawlers take content and return little directly, so the decision depends on how you view your content being used that way. Shopping agents may bring orders. Impersonators bring nothing and often take the most.

A robots.txt file lets you state a preference for each named crawler, and established attackers say they follow it. It is a request. It does nothing against software that ignores it or lies about its name, so it cannot be your only control.

Identity comes before policy. A user-agent string is a label anyone can type. Operators of established agents publish the network ranges their traffic comes from, and some sign their requests so a site can check the signature. A request that cannot be verified should remain unverified rather than receiving an automatic trusted exception.

With identity settled, write the policy as a small table: each group against each area of the site. A common starting point is to allow verified search crawlers and user-triggered fetchers on product and content pages, decide deliberately about training crawlers, allow verified shopping agents through to checkout with your normal fraud controls, and limit or challenge everything unverified.

Measure before you decide. Look at how much of each group you receive, which pages they request and whether assistant referrals convert. Many sites find the volume smaller and the mix different from what they assumed.

Review the policy regularly. New agents appear, existing ones change how they identify themselves, and the commercial value of assistant traffic is still being established.

How Kairal helps

Agent Traffic Intelligence, currently in early access, checks agent identity and purpose to support your access policies. A recognised agent is not proof of a customer’s authorisation; keep account, purchase and payment checks on the operations it can request.

Agent Traffic Intelligence