At the time of writing, Lemmyworld has the second highest number of active users (compared to all lemmy instances)

Also at the time of writing, Lemmyworld has >99% uptime.

By comparison, other lemmy instances with as many users as Lemmyworld keep going down.

What optimizations has Lemmyworld made to their hosting configuration that has made it more resilient than other instances’ hosting configurations?

See also Does Lemmy cache the frontpage by default (read-only)? on !lemmy_support@lemmy.ml

  • Ruud@lemmy.worldM
    6·
    2 years ago

    Thank you for all the compliments.

    This ride reminds me of Mastodon.world in November. Details on that are here: https://blog.mastodon.world/and-then-november-happened

    So I started lemmy.world on a 2CPU/4GB VPS. Keeping an eye on the performance. Soon I decided to double that. And after the first few thousand of users joined, doubled it again to 8CPU/16GB. That also was the max I could for that VPS type.

    But, already I saw some donations come in, without really asking. That reminded me of the willingness to donate on Mastodon, which allowed me to easily pay for a very powerful server for mastodon.world, one of the reasons it grew so fast. Other (large) servers crashed and closed registrations, I (mainly) didn’t.

    So, I decided to buy the same large server (32cpu/64threads with 128GB RAM) as for masto (but that masto one has double the RAM). With the post announcing that, I also mentioned the donation possibilities. That brought a lot of donations immediately, already funding this server for at least 2 months. (To the anonymous person donating $100 : wow!).

    Now next: to solve the issue with post slowness. That’s probably a database issue.

    And again: migration took 4 minutes downtime, and that could have been less if I wasn’t eating pizza at the same time. So if any server wants to migrate: please do! If you have the userbase, you’ll get the donations for it. Contact me if you have questions.

    • Xaphanos@lemmy.world
      2·
      1 year ago

      As someone “in the business”, but not nearly as technical as you… How far can a single instance scale? Can a load balancer spread it over mulitple front-ends to handle user load? Can the back-end be re-worked to handle hundreds of millions of user operations per second? Can it work with a CDN? Can a single “Lemmy.World” site exist as a distributed site - with hundreds of servers spread across dozens of sites across the globe?

      I expect this entire line of thought is antithetical to the entire Lemmy philosophy of distributed operation. I expect that the “correct” way is to spin off “NA.Lemmy.World”, EMEA. Lemmy.World", APAC.Lemmy.World", etc. as separate servers. Is that correct?

      Thanks.

      • Ruud@lemmy.worldM
        2·
        1 year ago

        Best would be to have thousands of servers. I wouldn’t make them subdomains of lemmy.world, but people could create separate instances for large communities, for example. https://lemmy-selfhosted.com Scaling: if the Lemmy software gets tuned a bit better, I think the current server could host at least a million users. But let’s hope for more servers first…

    • million@lemmy.world
      2·
      2 years ago

      Nice job, thanks very much for the write up.

      Out of curiosity are you cloud hosting or do you own a server on a rack somewhere? Scaling with Kubernetes or VMs or just running bare-metal?

    • Balthazar@sopuli.xyz
      1·
      2 years ago

      And this kindness and willingness to help is why I’ve already fallen in love with Lemmy. Thank you good sir, thank you dearly for helping the next generation of internet denisens :D

    • Xanvial@beehaw.org
      1·
      2 years ago

      Interesting, I’m new on Lemmy (and fediverse itself), but when you said server does it means the backend that handles frontend traffic or database that stores all the data? Seems the next optimization step is distributing the traffic to multiple servers.

      Also (again I don’t know about the lemmy system itself), maybe you can get away with just upgrading CPU cores only or RAM only (depends on what bottlenecking the system). From my experience, the RAM requirement is scaling slower compared to CPU

    • Dream_state@lemmy.world
      0·
      2 years ago

      Am I able to use the same account to login to mastodon.world? Or do I need to make an account there too? Never used mastodon but vining the fediverse stuff

        • Balthazar@sopuli.xyz
          1·
          2 years ago

          You can, you just must log in to the server/website in which you made the account, then browse over to the server you wish to contribute to or use.

          It’s a bit weird, I know. If you’ve got any questions I’ll try my best to answer them :D

  • Skull giver@popplesburger.hilciferous.nl
    2·
    2 years ago

    Lemmy.world is using some Big Boy Hardware while most other servers are hosted on more hobbyist-grade hardware.

    Lemmy is quite efficient software. It has its challenges and weirdness, but I wouldn’t bat an eye if someone told me a server with 200 people was running on a Raspberry Pi attached so an SSD in someone’s basement. It barely consumes any CPU and for small communities the entire thing runs on less memory than a copy of Chrome loading the homepage.

    Needless to say, a network of servers that has been serving hundreds of people every day struggled to adapt to hundreds of thousands of people coming in at once.

    This isn’t unique to Lemmy either. We saw this happen to Mastodon when Twitter first started fucking up, then we saw it on Mastodon alternatives when the main Mastodon servers closed registration while they desperately scaled up.

    kbin, which is written in PHP, has also been seriously struggling. I’m pretty sure they still have their Cloudflare anti-DDoS-firewall enabled, meaning federating with kbin users is basically impossible until they can get that fixed.

    There are all sorts of things that could be improved (the frontend can use more aggressive caching, the backend might be written in a way that you can spawn more to take the load, the database can be sharded, you name it) but for Lemmy’s current size, throwing more cores and more RAM at the problem will quickly and easily fix the issue.

    If anything, this shows how inefficient some easier web application frameworks are. Good luck getting Mastodon to run well with less than half a GB reserved for Mastodon itself. Maybe there’s a point where Mastodon’s architecture starts to become more efficient than Lemmy’s, but from what I can tell that’s not all that likely.

      • Skull giver@popplesburger.hilciferous.nl
        1·
        1 year ago

        For super cheap, there’s Oracle’s free* servers. Restricted to about 50mbps in bandwidth but doesn’t cost you a penny. Get the ARM server if you can, those 4 cores and 24GB of RAM will serve you well.

        For cheap, there are hosting providers like Contabo and Hetzner that will offer you servers for a fair price. You’ll need to do your own backups though!

        OVH has some very cheap dedicated servers if you need more performance. Your uplink will be limited to 100mbps but you get dedicated CPU cores, dedicated hard drives/SSDs, dedicated RAM, everything. This comes with the risk that you’ll need to do your own backups because those dedicated components can die and there’s no failover (by default).

        It’s hard to say what hardware you’ll actually need, but I’ve read somewhere that lemmy.ml used to run on a machine with 8 CPU cores to serve thousands of users before everyone migrated over. The admin of lemmy.world posts regular updates on how the server is doing, I think the 21k users and all of its federated communities are running on a 32 core CPU with 128GB of RAM. Based on this, 6 cores and 8GB of RAM should be more than enough to serve ±100 users with the responsiveness of the lemmy.world instance (probably better because you’ll have comparatively more bandwidth).

        *=free forever but you do need to answer a call from a marketeer

  • andrew@radiation.party
    1·
    2 years ago

    Likely experience and knowledge improving the quality of deployment. Most instances are likely underspecced, are on hosts that aren’t easy to scale up with, or are maxed out in their current offering tier (lemmy.ml comes to mind there)

  • PriorProject@lemmy.world
    0·
    2 years ago

    I’m not an admin, but have followed the sizing discussions around the lemmyverse as closely as I can from my position of lacking first-hand knowledge:

    • lemmy.ml is the biggest instance by user count, but runs on incredibly modest 8-cpu hardware. Their cloud provider doesn’t provide any easy scale up options for them, so they can’t trivially restart on a bigger VM with their db and disk in place. I suspect this means that instance is going to suffer for a bit as they figure out what to do next.
    • lemmy.world on the other hand was running on a box at least twice as big as lemmy.ml at last count, and I believe they can go quite a bit bigger if they need to.
    • The lemmy.world admins also run mastodon.world and lived through the twitterpocalypse, seeing peak user registrations rates of 4k per hour. So this is not their first rodeo in terms of explosive growth, I’m sure that experience gives them some tricks up their sleeve.
    • The admin team is pretty clearly technically strong. If I recall correctly, ruud is a professional database admin. One of the spooky parts of Lemmy performance-wise is the db. If ruud or others on the admin team custom-tuned their pg setup based on their own analysis of how/why it’s slow, they may be getting more performance per CPU cycle than other instances running more stock configs or that are cargo-culting tweaks that aren’t optimal for their setup without understanding what makes them work.

    I’m surprised that sh.itjust.works isn’t growing faster. They also have a hefty hardware setup and seemingly the technical admins to handle big user counts. I wonder if it’s a branding problem, where lemmy.world sounds inviting and plausibly serious where sh.itjust.works sounds like clowntown even though it’s run by a capable and serious team.

    • Retro@lemmy.world
      2·
      2 years ago

      I originally signed up with sh.itjust.works, but I wanted to be on the instance with the majority of migrants.

      Also, it sounds dumb, but I think the sh.itjust.works domain is just kinda weird, technically has a “curse word” in it (not that I personally care), and they don’t support NSFW content (which isn’t just used for porn). So, it didn’t make sense to have that as my home instance. 🤷‍♂️

      Edit: Also, this is my first comment on here! Hello world! 👋

      • PriorProject@lemmy.world
        2·
        2 years ago

        Yeah, I get it. Naming optics aside, it seems an instance with a lot of headroom relative to others, with a capable team. Would be near the top of my word-of-mouth options in spite of the idiosyncratic name.

    • Master@lemmy.world
      21·
      2 years ago

      Can confirm… I didnt sign up for sh.itjust.works solely because of the name… I dont particularly want that attached to every post I make.

    • wheen@lemmy.world
      1·
      2 years ago

      Can none of this scale horizontally? Every mention of scaling has been just “throw a bigger computer at it”.

      We’re already running into issues with the bigger servers being unable to handle the load. Spinning up entirely new instances technically works, but is an awful user experience and seems like it could be exploited.

      • PriorProject@lemmy.world
        1·
        2 years ago

        It’s important to recall that last week the biggest lemmy server in the world ran on a 4-core VM. Anybody that says you can scale from this to reddit overnight with “horizontal scaling” is selling some snake oil. Scaling is hard work and there aren’t really any shortcuts. Lemmy is doing pretty well on the curve of how systems tend to handle major waves of adoption.

        But that’s not your question, you asked if Lemmy can horizontally scale. The answer is yes, but in a limited/finite way. The production docker-compose file that many lemmy installs are based on has 5 components. From the inside out, they are:

        • Postgres: The database, stores most of the data for the other components. Exposes a protocol to accept and return SQL queries and responses.
        • Lemmy: The application server, exposes websockets and http protocols for lemmy clients… also talks to the db.
        • Lemmy-ui: Talks to Lemmy over websockets (for now, they’re working to deprecate that soon) and does some fancy dynamic webpage construction.
        • Nginx: Acts as a web proxy. Does https encryption, compression over the wire, could potentially do some static asset caching of images but I didn’t see that configured in my skim of the config.
        • Pict-rs: Some kind of image-hosting server.

        So… first off… there’s 5 layers there that talk to each other over the docker network. So you can definitely use 5 computers to run a lemmy instance. That’s a non-zero amount of horizontal scaling. Of those layers, I’m told that lemmy and lemmy-ui are stateles and you can run an arbitrary number of them today. There are ways of scaling nginx using round-robin DNS and other load-balancing mechanisms. So 3 out of the 5 layers scale horizontally.

        Pict-rs does not. It can be backed by object storage like S3, and there are lots of object storage systems that scale horizontally. But pict-rs itself seems to still need to be a single instance. But still, that’s just one part of lemmy and you can throw it on a giant multicore box backed by scalable object storage. Should take you pretty far.

        Which leaves postgres. Right now I believe everyone is running a single postgres instance and scaling it bigger, which is common. But postgres has ways to scale across boxes as well. It supports “read-replicas”, where the “main” postgres copies data to the replicas and they serve reads so the leader can focus on handling just the writes. Lemmy doesn’t support this kind of advanced request routing today, but Postgres is ready when it can. In the far future, there’s also sharding writes across multiple leaders, which is complex and has its downsides but can scale writes quite a lot.

        All of which is to say… lemmy isn’t built on purely distributed primitives that can each scale horizontally to arbitrary numbers of machines. But there is quite a lot of opportunity to scale out in the current architecture. Why don’t people do it more? Because buying a bigger box is 10x-100x easier until it stops being possible, and we haven’t hit that point yet.

    • WaffleFriends@lemmy.world
      1·
      2 years ago

      I had a very similar thought process when choosing my instance. lemmy.world seemed like it would be more open to new users than an instance named sh.itjust.works. Idk why that was my thought process but I’m here now

    • Pspspspspsps@lemmy.world
      1·
      2 years ago

      I wonder if it’s a branding problem, where lemmy.world sounds inviting and plausibly serious where sh.itjust.works sounds like clowntown

      That was my thought process when choosing an instance tbh. I’m not a tech person, I looked at the list and lemmy.world was the first ‘safest feeling’ instance that had open sign up. I saw sh.itjust.works and didn’t even check their sign up process, there was too many periods in the strange name and it just looks weird to me as someone not used to these things. Edit: spelling

      • s4if@lemmy.world
        2·
        2 years ago

        nah, I’m bit regretting not signing up on their instance. sh.itjust.works is a cool name and can be a brag point. lol. lemmy world is a bit too generalist, but I won’t migrate there as ruud (the admin of lemmyworld) is doing a good job managing the instance. I appreciate that. :)

      • Z______@lemmy.world
        0·
        2 years ago

        I definitely second the motion on it being a branding problem. Stuff like sh.itjust.works seem to me like something that dark basement tech nerds would come up with that is “edgy” and really only used by them and other people like them.

        I’m not really into the ironic “edgy” aesthetic and part of the struggle with this transition for me has been orienting myself in the space because I don’t want to commit to some “sketchy” edgelord URL

        • DarkwingDuck@sh.itjust.works
          1·
          2 years ago

          something that dark basement tech nerds would come up with that is “edgy” and really only used by them and other people like them.

          That’s exactly what it is and why I love it. The whole thing about this federated networking is that it doesn’t matter where you signed up.

    • Druidgrove@lemmy.world
      1·
      2 years ago

      I’m now going to start incorporating “Sounds like clowntown” into my everyday conversations - that’s funny!

    • StrayPizza@lemmy.world
      1·
      2 years ago

      I hope lemmy.ml can upgrade at some point. A lot of the slowness I’m running into is trying to browse/discovery communities that happen to live on that instance.