I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site’s servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn’t know why, exactly.

This isn’t the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

E: Many people have replied comparing this to the Reddit “hug of death” or “The Slashdot effect”. This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.

  • FundMECFS@piefed.zip
    link
    fedilink
    English
    arrow-up
    80
    ·
    1 month ago

    Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.

    • Derpgon@programming.dev
      link
      fedilink
      English
      arrow-up
      11
      ·
      30 days ago

      Either publishing server should serve the metadata - although it would be prone to bad actors, so a refetch in the next 0-180 seconds would be appropriate. Or just add a random 0-60 second delay before fetching the data.

  • Rimu@piefed.social
    link
    fedilink
    English
    arrow-up
    71
    ·
    1 month ago

    Lemmy does this to itself, too - when you create a post it federates out and then every Lemmy instance that receives it makes a GET request to retrieve the post from the original server. Even though it just received the exact same data. Some other fedi software is even worse - Akkoma does a GET of the id of every activity it receives, not just for posts but for Likes too.

    It’s insane.

    Anyway, proper caching at the reverse proxy / Cloudflare level solves it. The same kind of caching setup would do the same for sites serving HTML, like NPR too.

    • Ghoelian@piefed.social
      link
      fedilink
      English
      arrow-up
      7
      ·
      1 month ago

      That’s weird. Is that how it’s supposed to work, or did they just implement it in a bad way?

      • The_Decryptor@aussie.zone
        link
        fedilink
        English
        arrow-up
        12
        ·
        1 month ago

        It’s an access control method, called “authorized fetch”. Rather than broadcasting the data you just announce the source of it, and then any client that wants to show it needs to negotiate with the originating server to request the data (And then blocklists and such come into play)

        Say you’ve got 3 users, A, B, and C. A has blocked C, B follows A, and C follows B. Now A makes a post, B sees it in their feed and reboosts it, what does C see?

        If B’s server sends the full response along, then C can see the contents of A’s post even though they’ve been blocked by A. But if all it sends is the URL of the post, C’s server needs to make a request for it (providing authentication), and then A’s server can reject it, so C sees nothing.

        • Ghoelian@piefed.social
          link
          fedilink
          English
          arrow-up
          1
          ·
          26 days ago

          Okay, what you explained does make sense, but not in the context of a federation event (as far as I understand, which could very well be wrong).

          If B’s server sends the full response along, then C can see the contents of A’s post even though they’ve been blocked by A.

          Sure, but this would only be needed if a user needs permission or something to view a post. They don’t, a post is (always, afaik?) public.

          Besides, federation is not on a user level, but on an instance level. It doesn’t matter if user A has blocked user B. As long as just one person on user A’s instance does not have user B blocked, events and metadata will need to be federated. So, for public federated services, I don’t think it makes any sense to use this kind of access control.

      • Rimu@piefed.social
        link
        fedilink
        English
        arrow-up
        2
        ·
        30 days ago

        I’m not sure but my guess it’s an attempt to deal with the problem where sometimes federation activities arrive in the wrong order (especially after an instance has been offline for a while) - by making a GET to the origin server you can be sure you’ve got the latest version of the content. There are probably better ways to achieve that, though.

        PieFed prioritises efficiency over correctness so it just accepts whatever is sent to it. In 3 years no one has complained about seeing stale content although I’m sure it happens occasionally.

    • SorteKanin@feddit.dk
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 month ago

      How else can you be sure the content is valid though? Lemmy posts are sent as Announces from the community, not from the actor posting the post. The community could just fake a post if you just trust it.

  • GenosseFlosse@feddit.org
    link
    fedilink
    English
    arrow-up
    51
    ·
    1 month ago

    Is there a reason why the publishing instance is not downloading the preview image, and then submits it with the post to the other instances? This way they would not have to fetch it themselves.

      • Gibibit@lemmy.world
        link
        fedilink
        English
        arrow-up
        29
        ·
        1 month ago

        Seems abusable though

        If one instance doesn’t trust another, they shouldn’t federate in the first place.

      • _haha_oh_wow_@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        23
        ·
        30 days ago

        It is, but that’s what defederation is for: If there are instances that are being sketchy, they are supposed to get cut off from the rest of the Fediverse.

      • a4ng3l@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 month ago

        Add metadata with a checksum of the og one with signature and let each client check that up.

        • inzen@lemmy.world
          link
          fedilink
          English
          arrow-up
          5
          ·
          1 month ago

          If it’s a signed checksum then maybe. But if the og instance is doing the abusing then it won’t help.

          • a4ng3l@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 month ago

            Yeah there must be some trust indeed of the og instance. A scoring system could help with that.

            • prenatal_confusion@feddit.org
              link
              fedilink
              English
              arrow-up
              1
              ·
              30 days ago

              Or … Or we could stagger the requests of the different instances so that they don’t overload the linked site. Why reinvent the wheel?

              • a4ng3l@lemmy.world
                link
                fedilink
                English
                arrow-up
                1
                ·
                30 days ago

                Eventually that leaves the load to the source event if it’s differed. And the load would still grow along the growth of the fediverse. So this would likely be pushing the issue to tomorrow. Signing content checksum isn’t exactly like reinventing a wheel, it’s something done in many contexts. And it removes the load from the source entirely.

  • cybervegan@lemmy.world
    link
    fedilink
    English
    arrow-up
    44
    ·
    1 month ago

    I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It’s no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.

    Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don’t honour robots.txt (or aren’t yet listed).

    • 0x0@lemmy.zip
      link
      fedilink
      English
      arrow-up
      12
      ·
      30 days ago

      that don’t honour robots.txt (or aren’t yet listed).

      Throw’em in the tar pit.

      • cybervegan@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        29 days ago

        Limited mental capacity these days, so I don’t have the spoons to properly research and implement a solution at the moment. I installed Nepenthes last year, but I’ve never managed to get round to integrating it into my site. It’s low priority at the moment.

    • iglou@programming.dev
      link
      fedilink
      English
      arrow-up
      2
      ·
      29 days ago

      To be honest, if a website has no caching of their server-rendered parts, they are also doing something wrong! Very little sites really need to re-render a public page on every view.

      • cybervegan@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        ·
        29 days ago

        Around 2010, I used to work at an MSP, and once had a customer that ran a custom LAMP stack website for local radio UK stations web presence. It was a bit slow, running on a dedicated physical server, but every time it snowed, the website would 500 out - because of too many requests to the “School Closures” page. The site had no caching other than Apache’s internal cache, but PHP pages could not be cached because “reasons”, and the original developer was “not available”. I optimised their MySQL indexes, increased the number of Apache workers, and it improved the render time per page and number of concurrent connections. Stress testing looked good.

        Anyway, some months later, 2AM one snowy morning, I was on call, and it went down again. This was when I discovered the school closures problem. After diagnosing, and then liaising with the customer to discuss the options, I was told “just make it work”. I ended up modifying the PHP script to add some “on spec” caching that saved the HTML to a temp file in ramdisk, and only re-ran the query-and-render code if the cached copy was more than 5 minutes old. It was hacky, but it did the trick, and I was finished by about 4 AM. This hack was later applied at their request to a number of trouble-points in the source tree, as and when things broke.

        We recommended adding a Varnish cache, and found a more generalised caching library but they wouldn’t/couldn’t authorise engineering work for this - they were running on a shoestring - and they already had the tech break-fix contract with us, and decided it was sufficient to just spot fix things as they broke rather than fix the root cause.

        I left that MSP in 2012, so I don’t know what happened after that, but as far as I know, the site could still be running… unlikely, but possible.

  • ByteJunk@lemmy.world
    link
    fedilink
    English
    arrow-up
    42
    ·
    1 month ago

    I mean, just how many instances are gonna ping the URL? Several hundred, low thousands at worst?

    Unless it happens to be some very large media content or a dynamic page with no caching, I’d think the automated federation hits shouldn’t be too bad.

    The influx of users that might come after a viral post though, that’s rough…

      • 1984@lemmy.today
        link
        fedilink
        English
        arrow-up
        8
        ·
        1 month ago

        I liked writing efficient code but every place I worked didn’t care at all about it. All they wanted was a finished feature and a kanban note moved to finished.

        • mcv@lemmy.zip
          link
          fedilink
          English
          arrow-up
          1
          ·
          29 days ago

          The many pointless presentations I’ve given about tech debt and code quality. Argued for redesigning poorly designed systems. They don’t care. Just churn out more features to cross off the list, and why do those features take so long?

          • 1984@lemmy.today
            link
            fedilink
            English
            arrow-up
            1
            ·
            29 days ago

            Yeah I think a lot of us share this experience. All the stuff we are tought in our educations about clean code and architecture… Once we come out and work, management don’t want that stuff if it takes extra time. They want features shipped as soon as possible, and every meeting is about “how fast can you do this feature”.

            And now of course embracing AI since it’s so fast. When things break, engineers will have to figure out the problem when the entire system is down, not management. :)

            And then they can have a post mortem where they say that engineering must have better routines…lol.

            I no longer work in tech but I quit at the top I think, before Ai was a requirement. And when consulting salaries was still good. Now they are awful.

              • 1984@lemmy.today
                link
                fedilink
                English
                arrow-up
                2
                ·
                29 days ago

                Sure, but there are a lot of bad places. I bet the majority of engineers work in bad places like this.

                • KeenFlame@feddit.nu
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  29 days ago

                  Yeah, but it’s also dependent entirely on if you chase money or actually want to become a good engineer because it is fun.

                • mcv@lemmy.zip
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  29 days ago

                  I think so. I’ve worked at bad, average and good places, but the bad places tend to be larger and far less efficient, so they have to hire more engineers to make up for the inefficiency.

                  And they don’t necessarily pay well either. They see engineers as powerless cogs who have to do what they’re told, so they get paid like that too. And then they complain they can’t find seniors.

    • Doom@lemmy.world
      link
      fedilink
      English
      arrow-up
      10
      ·
      1 month ago

      Yeah but we can’t say we want to encourage people to have/build/share independent smaller websites then beat them up when they are linked. That’s a huge mismatch of morality. Either we want to encourage people off the big platforms and should make it easier for the little guys or we are blowing magic smoke.

  • dropdrip@lemmy.ml
    link
    fedilink
    English
    arrow-up
    36
    ·
    1 month ago

    It’s almost like the software that powers the internet was designed for a different time and is only still used because there’s monied interest in keeping it around. We need GNUnet.

    • milbyte@lemmy.ml
      link
      fedilink
      English
      arrow-up
      23
      ·
      1 month ago

      is there something that makes this different to 17 other proposals for a new internet or is it just a case of https://xkcd.com/927/

    • sem@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      15
      ·
      1 month ago

      GNUnet helps building a new Internet

      GNUnet is a network protocol stack for building secure, distributed, and privacy-preserving applications.

      With strong roots in academic research, our goal is to replace the old insecure Internet protocol stack.

      GNUnet provides privacy-by design

      It provides [improving addressing, routing, naming and content distribution](https://www.w3.org/2014/strint/papers/65.pdf) in a technically robust manner - as opposed to ad-hoc designs in place today.

  • Blackmist@feddit.uk
    link
    fedilink
    English
    arrow-up
    17
    ·
    29 days ago

    Sounds like every instance is doing the preview generation. Maybe it could grab the preview data it needs on the original instance, and pass that around with the post…

    • boonhet@sopuli.xyz
      link
      fedilink
      English
      arrow-up
      2
      ·
      29 days ago

      Well you can’t really trust the original instance to be truthful with its preview.

      • axx@slrpnk.net
        link
        fedilink
        English
        arrow-up
        1
        ·
        28 days ago

        There are ways around that. See my other comment replying to the same parent as your comment.

  • AdrianTheFrog@lemmy.world
    link
    fedilink
    English
    arrow-up
    14
    ·
    30 days ago

    it seems like ideally it shouldn’t attempt to preview the link before a user tries to view the content. i guess that would increase latency for users though

    • artyom@piefed.socialOP
      link
      fedilink
      English
      arrow-up
      8
      ·
      1 month ago

      If they’re all delayed the same it would have the same effect. In order to prevent that it would require some form of coordination amongst them. And even then would have the same complaint about missing link previews. Better than nothing but not really a “solution”, per se.