• unskilled5117@feddit.org
      link
      fedilink
      English
      arrow-up
      22
      ·
      edit-2
      5 months ago

      That seems highly unlikely to me. Could a reason be website scrapers for AI using different user agents to prevent being blocked? The recent reports of different projects plagued by scrapers fit the timeline

        • unskilled5117@feddit.org
          link
          fedilink
          English
          arrow-up
          2
          ·
          edit-2
          5 months ago

          Yeah i read that too. But how well is it working? I mean that’s what the news was all about the last few months. A lot off projects were having trouble blocking the bots because they were trying and succeeded to circumvent detection measures.

          • highball@lemmy.world
            link
            fedilink
            English
            arrow-up
            2
            ·
            5 months ago

            But how well is it working?

            Yeah, that’s a good question. I don’t remember reading any documentation from Statcounter speaking about how well that bot detection and removal algorithms work. The last company I worked for did data collection, as we recorded all kinds of metrics from users. We had a team that focused on bot detection. Mostly it involved discovering the bot net and then removing all of their metrics that were accounted for in the system. Then updating the bloom filters so that any further metrics from the known bots would be ignored. Companies were paying millions of dollars for our contracts and every click was getting itemized and charged for. So, trust me, it was very important for my old company to discover and eliminate bots aggressively.

            Statcounters sells their statistics and not the data from what I understand. So it does beg the question, how aggressive are they?!