• ShitpostCentral@lemmy.world
    link
    fedilink
    English
    arrow-up
    15
    ·
    10 months ago

    You’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.

    • GenderNeutralBro@lemmy.sdf.org
      link
      fedilink
      English
      arrow-up
      10
      ·
      10 months ago

      You’d probably have to go out of your way to avoid logging this. I’ve always seen such logs enabled by default when setting up web servers.