Andy Reid to Technology@lemmy.worldEnglish • 8 months agoAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comexternal-linkmessage-square173fedilinkarrow-up11.05Karrow-down114cross-posted to: technology@beehaw.orgwolnyinternet
arrow-up11.04Karrow-down1external-linkAI companies are violating a basic social contract of the web and and ignoring robots.txtwww.theverge.comAndy Reid to Technology@lemmy.worldEnglish • 8 months agomessage-square173fedilinkcross-posted to: technology@beehaw.orgwolnyinternet
minus-square@ShitpostCentral@lemmy.worldlinkfedilinkEnglish15•8 months agoYou’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.
minus-square@GenderNeutralBro@lemmy.sdf.orglinkfedilinkEnglish10•8 months agoYou’d probably have to go out of your way to avoid logging this. I’ve always seen such logs enabled by default when setting up web servers.
You’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.
You’d probably have to go out of your way to avoid logging this. I’ve always seen such logs enabled by default when setting up web servers.