Does robots.txt prevent scraping?

No. robots.txt publishes instructions that participating crawlers choose to follow. It does not prevent a client from requesting a public URL. The Robots Exclusion Protocol is not an authorisation mechanism, and the file itself is public.

Use authentication and server-side access controls for information that must remain private. Do not publish sensitive paths in robots.txt expecting the file to hide them. For public catalogue pages, combine a documented crawler policy with controls on the requests you actually receive.

Kairal Bot Protection can evaluate requests to the pages connected to your integration. Keep permitted search access working when you change rules, and verify declared crawler identities where documented methods exist. A robots.txt rule and a technical block serve different purposes; neither guarantees that already copied material disappears.

How Kairal helps

Kairal Bot Protection checks requests to connected content and catalogue pages. Events and Sessions let you investigate repeated requests using recorded devices and IPs. Define which access to permit and check the effect on authorised crawlers before tightening controls.

Kairal Bot Protection