llostre
A self-hostable Bluesky client in Go, built to keep working if Bluesky's own infrastructure goes away. No AppView. Records are cached with an expiry rather than stored, and referenced records are fetched from their authors' servers as they come up.
The constraint
The atproto network is a stack of three different pieces. Your data lives in a personal data server (a PDS), which anyone can run. The Relays are aggregator processes, compiling data from several PDSs. And these in turn are consumed in turn by an AppView: a single large service, which consumes a specific set of records and hands back an assembled timeline.
The AppView is what the clients usually talk to. A post record contains a bare reference to its author and to whatever it replies to, so somebody has to fetch each of those, in order, before there is anything to display. Aggregate numbers (likes, reposts, reply counts) exist nowhere in the underlying data at all; they are counted by the service that indexes everything. That’s the kind of work an AppView does.
The AppView is infamously hard to host on your own, so the network currently relies mainly on a single monolith (owned by the Bluesky company), with only a couple alternatives (Blacksky, Eurosky, and soon Northsky.) A desire for easier methods of selfhosting has been expressed several times by the atproto community; that goes for clients as well, since most community-made active clients are forks of Bluesky’s social-app project. “Llostre” is an exploration of that. It’s also how I’m teaching myself Go and Typescript, as I’m a very hands-on learner.
Assembling a timeline without an index
Without an AppView (or Relays), the client does the assembly itself. It starts from the accounts you follow, reads their repositories directly, and walks outward: a post references an author, the author reference resolves to a profile, a reply references a parent, the parent has its own author. Each hop is another server, usually somebody else’s.
Fetching is cursor-based and paginated, with more pages pulled in the background once the visible list runs thin, so scrolling does not stall waiting on a chain of requests to strangers’ servers.
Records are normalised on the way in, into a shape the renderer understands regardless of which lexicon produced them. That separation is the part I would defend: the fetching layer knows about protocols, the rendering layer knows about display, and neither needs to learn the other’s job.
Cache but no database
The obvious approach is to store everything you fetch. I deliberately did not, because storing everything means becoming a partial mirror of the network, and the disk cost of that grows without limit, and is too much for a personal instance.
Instead, every record is cached with an expiry, and each read pushes its expiry back. Things you look at stay resident; things you only looked at once decay out on their own. There is no eviction policy to tune and no cleanup job to schedule, because the access pattern does the work.
That leaves the correctness problem: cached records can be edited or deleted at their source, and a cache that does not hear about it will serve something that no longer exists. So the client also listens to the network’s live event stream and drops anything it sees mutate. Time-based expiry handles the general case, the event stream handles the sharp case.
Two paths to the same record
Reading directly from thousands of personal servers is slow. There is a community aggregator service that will fetch a record for you far faster, so the client tries one first and falls back to querying the author’s own server when that fails. On paper, this doesn’t sound all that different to an AppView, but there is a roadmap goal for this service to allow for selfhosting, it’s lighter than an actual AppView, and coupled with the fact that it’s not tied to one specific company, it has some promise of higher resilience.
In any case, the fast path (the service) is just an optimisation; the slow path (direct fetching) is the guarantee. This way, if the aggregator disappears the client keeps working, just less quickly. A design where the fast path is the only path is the thing I was trying to avoid in the first place.
Starting cold on purpose
On startup the cache is flushed rather than backfilled. Warming it would mean crawling a large slice of the network before the client is usable, which is slow, rude to other people’s servers, and mostly wasted: the majority of what you would fetch is content you probably won’t scroll to.
Starting empty means the first minute is slower and everything after it is correct. That is the trade, and it is stated in the repository so nobody has to discover it by reading the code.
Where it is
Still a work in progress. I have looked at splitting the data layer out into its own service (creating an AppView-like with the behavior I wanted, and an agnostic client that could swap it for a normal AppView) but decided against it: the appeal of this client is that it is opinionated, and splitting it would double the work for an experience that would only be a shard of the original design half of the time.
- Language
- GoTypeScriptHTML/CSS
- Protocol
- atprotoXRPCOAuth 2.0DPoPJetstream
- Storage
- RedisTTL cachingcursor pagination
- Practice
- graceful degradationcache invalidationAPI design