Price and catalogue scraping in Retail & e-commerce

How the attack works

1. A competitor or aggregator maps your category, search and product-detail routes to find where prices and stock availability are returned. The operator wants a reusable view of your catalogue, including changes that can inform pricing or resale decisions.
2. Automated clients fetch those routes repeatedly, working through products and variants. A single product request resembles ordinary browsing; the value comes from collecting many records and refreshing them as prices or availability change.
3. The operator spreads collection across proxies or changes the client presentation when requests are limited. A familiar crawler name may also be supplied in the user-agent string, even though that name alone does not establish the client's identity or purpose.
4. The collected data feeds comparison, repricing or arbitrage workflows outside your business. Your infrastructure handles the repeated reads, while product-view totals and acquisition reporting may include visits that were never likely to become customers.

Why it gets through

Public product information must remain reachable by shoppers and useful discovery services. Blocking every automated visitor would also interfere with search and shopping engines you want to support. The decision therefore depends on the client and the access policy, not simply whether a request is automated.

IP-based limits see individual sources rather than the whole collection effort. Residential proxies can distribute a catalogue crawl across many addresses. Raising the threshold preserves access but leaves more repeated collection untouched; lowering it can disrupt people on shared networks.

A user-agent label is supplied by the requesting client. Treating a familiar name as permission leaves room for impersonation. Static signatures also age as collection tools change, while a rule attached only to the visible product page can miss the routes that return its underlying catalogue data.

Infrastructure spend is the operational measure. Repeated product, search and availability reads consume capacity even when they do not produce an order. During a commercial campaign, that additional work makes it harder to understand how much demand is coming from customers.

Conversion rate and product-view reporting can be distorted by collection traffic. A frequently fetched product can look popular while the visitors are automated clients refreshing a dataset. Merchandising and acquisition decisions then rest on activity with a different purpose from shopping.

Price and availability exposure is a commercial concern even when the pages are public. Repeated collection gives another operator a current picture of your assortment. Your team needs to decide which access supports discovery and which primarily supplies an outside business at your expense.