• Programmer Belch@lemmy.dbzer0.comEnglish
      29·
      12 天前

      I use a tool that downloads a website to check for new chapters of series every day, then creates an RSS feed with the contents. Would this be considered a harmful scraper?

      The problem with AI scrapers and bots is their scale, thousands of requests to webpages that the internal server cannot handle, resulting in slow traffic.

      • ulterno@programming.devEnglish
        5·
        12 天前

        If the site is getting slowed at times (regardless of whether it is when you scrape), you might want to not scrape at all.

        Probably not a good idea to download the whole site, but then that depends upon the site.

        • If it is a static site, if you just setup your scraper to not download CSS/JS and images/videos, that should make a difference.
        • For a dynamically created site, there’s nothing I can say
        • Then again, if you try to reduce your download to what you are using, as much as possible, that might be good enough
        • Since sites are originally made for human consumption, you might have considered keeping the link traversal rates similar to that
        • The best would be if you could ask the website dev whether they have an API available.
          • Even better, ask them to provide an RSS feed.
        • Programmer Belch@lemmy.dbzer0.comEnglish
          2·
          12 天前

          As far as I know, the website doesn’t have an API but I just download the HTML and format the result with a simple Python script, it makes around 10 to 20 requests, one for each series I’m following at each time.

          • ulterno@programming.devEnglish
            1·
            12 天前

            That might/might not be much.
            Depends upon the site, I’d say.

            e.g. If it’s something like Netflix, I wouldn’t think much, because they have the means to serve the requests.
            But for some PeerTube instance, even a single request seems to be too heavy for them. So if that server does not respond to my request, I usually wait for an hour or so before refreshing the page.

      • Flax@feddit.ukEnglish
        4·
        12 天前

        The problem is these are constant army hordes / datacentres. You have one tool. Sending a few requests from your device wouldn’t even dent a raspberry pi, nevermind a beefier server

        I think the intention of traffic is also important. Your tool is so you can consume the content freely provided by the website. Their tool is so they can profit off of the work on the website.

  • chicken@lemmy.dbzer0.comEnglish
    29·
    12 天前

    Seems like such a massive waste of bandwidth since it’s the same work being repeated by many different actors to piece together the same dataset bit by bit.

  • 0_o7@lemmy.dbzer0.comEnglish
    27·
    11 天前

    I blocked almost all big players in hosting, China, Ruasia, Vietnam and now they’re now bombarding my site with residential IP address from all over the world. They must be using compromised smart home devices or phones with malware.

    Soon everything on the internet will be behind a wall.

      • aev_software@programming.devEnglish
        15·
        11 天前

        In the mean time, sites are getting DDOS-ed by scrapers. One way to stop your site from getting scraped is having it be inaccessible… which is what the scalpers are causing.

        Normally I would assume DDOS-ing is performed in order to take a site offline. But ai-scalpers require the opposite. They need their targets online and willing. One would think they’d be a bit more careful about the damage they cause.

        But they aren’t, because capitalism.

    • ExLisperAEnglish
      5·
      11 天前

      There are many commercial VPNs offering residential IPs. I doubt they use malware.

    • ILikeTraaaains@lemmy.worldEnglish
      2·
      10 天前

      Not necessarily compromised, I saw a VPN provider (don’t remember the name) that offered a free tier where the client accepts being used for this.

      And I suspect that in the future some VPN companies will be exposed doing the same but with their paid customers.

  • Blackmist@feddit.ukEnglish
    19·
    11 天前

    Business idea: AWS, but hosted entirely within the computing power of AI web crawlers.

  • sp3ctr4l@lemmy.dbzer0.comEnglish
    17·
    11 天前

    Do we all want the fucking Blackwall from Cyberpunk 2077?

    Fucking NetWatch?

    Because this is how we end up with them.

    …excuse me, I need to go buy a digital pack of cigarettes for the angry voice in my head.

  • rozodru@lemmy.worldEnglish
    10·
    11 天前

    I run my own gitea instance on my own server and within the past week or so I’ve noticed it just getting absolutely nailed. One repo in particular, a Wayland WM I built. Just keeps getting hammered over and over by IPs in China.

      • daniskarma@lemmy.dbzer0.comEnglish
        17·
        11 天前

        On the contrary. Open community based block lists can be very effective. Everyone can contribute to them and asphyxiate people with malicious intents.

        If you think something like, “if the blocklist is available then malicious agents simply won’t use that ips” I don’t think if that makes a lot of sense. As the malicious agent will know any of their IPs being blocked as soon as they use them.

  • Harbinger01173430@lemmy.worldEnglish
    4·
    11 天前

    A good solution would be to load with a virus, to the PCs connecting from the AI ips, that overloads the computer and makes it explode.