Skip Navigation

Can the fediverse scale up indefinitely?

The tech giants make enough money that they could keep on growing forever, from my understanding.

But the fediverse? Sure the main instances that get enough funding are going to be okay, but what about the single-user instances 10 years from now on when there's a lot more content to download? Won't they go bankrupt just by trying to annex the big instances?

And I have the impression that the lemmy giants are going to change over time: does that mean that 50 years from now on, the posts I'm posting here today might get lost in time because the instances that annex it will have shut down by then?

I probably misunderstand how the fediverse works, but my worry is that the small instances won't be able to hold an ever-growing amount of data forever.

I spoke in absolutes for the sake of readability, but I'm as in-the-dark as can be.

42

You're viewing a single thread.

42 comments
  • Mostly serious answer: the current implementation is not going to scale effectively with growth. The software implementation is still rough around the edges, and the ActivityPub protocol probably needs more knobs to handle bulk data synchronization. Within the service, moderaton is a serious challenge with many unanswered questions.

    Likewise, the back end software implementation is monolithic, meaning it's one software stack that does everything from sign in to subscriptions to synchronization and scheduling. Housekeeping and garbage collection probably isn't that tight, either. This is mostly speculation as I've watched things over the last couple of weeks' growth.

    I believe the data store is based on Postgres RDBMS, which while being robust and scalable is fussy and needs tuning when turning over large amounts of highly unique data.

    None of this is an indictment on the devs! Rather the opposite, because the software IS chugging along while experiencing tremendous growth.

    I expect over time the back end will devolve into micro services that communicate over a highly scalable, or stream-based messaging bus. Larger instances could probably also benefit from static caching and CDN techniques to keep pages loading quickly even while the back end thrashes.

    The structure.if the ecosystem needs to strike a balance between fewer large instances and many-many small instances. In the first scenario, the scaling limit is in the monolithic stack, which introduces I/O bottlenecks and serialization delays (even if massively threaded). In the latter scenario, message state and synchronous distribution become challenging because a full mesh of federations could scale faster than network state tables have room to support. Some middle tier might be needed, and I have no idea what that might even look like.

    So to answer your question, can it scale indefinitely? Probably not because we hit scaling limits pretty quickly on a number of dimensions. Nevertheless, smart people.are starting to hang out here, and I expect will take an interest in how it all works. Improvement is inevitable, and I think the early roadblocks will be overcome easily enough

You've viewed 42 comments.