Pro@mander.xyz to Technology@programming.devEnglish · edit-25 months agoCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comexternal-linkmessage-square4linkfedilinkarrow-up136file-text
arrow-up136external-linkCloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocksblog.cloudflare.comPro@mander.xyz to Technology@programming.devEnglish · edit-25 months agomessage-square4linkfedilinkfile-text
minus-squareCarbonatedPastaSauce@lemmy.worldlinkfedilinkEnglisharrow-up2·5 months agoThe article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.
The article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.