A directory at directory scale

The forests directory is the directory of forests — the screen you open to find a place. The shape has to do two opposite things at once. With a dozen forests in the system, it shouldn’t feel like an empty room. With a thousand, it shouldn’t feel like a wall of text.

This week we filled the directory with a thousand procedural forests and rebuilt the experience around them.

What you see

The directory has a heading count — the real total, not the visible page. You scroll smoothly through whatever range you care about; rows below the viewport aren’t rendered until they’re about to slide into view.

The trick isn’t the count itself. It’s that “1,247 forests” reads as a healthy ecosystem you can explore, without being a performance burden the moment it’s true.

The directory shape · Yours / Discover / Search as tabs

Search, in four buckets

Search exists because nobody scrolls through a thousand things to find one. The four buckets are four answers to “what are you actually looking for”:

Forest names. Direct hits.
Inside the forests. Whatever people have written there — the forest is the address.
People. Folks whose handles match.
Personal forests. Sealed conversations you’re already in where something matches.

Idle, you see suggested forests. Searching, the buckets fill in as the query firms up. Empty, the screen tells you what to try next.

Just opened · recent queries and a few suggestions

Direct name hits as the query firms up

Matches inside what people have written, not just forest names

The plant-a-forest fallback when nothing matches

How a directory feels at twenty is barely a signal for how it feels at two thousand. Cheaper to build the right shape now than retrofit one later.

Tech notes

The directory is read from a Postgres database. Two queries back the landing’s Yours and Discover panes: one for forests the viewer is a member of, one for forests they aren’t. Both return a paginated slice ordered by the most-recent activity in each forest. Pagination is cursor-based — every page carries an opaque token pointing at the last row returned, and the next request asks for everything strictly older than it. Cursor pagination is the standard escape from offset pagination’s OFFSET 50000 problem: at large offsets, the database has to scan and discard everything before the cursor, whereas a cursor query jumps straight to the next page using an index seek. Cost stays roughly constant regardless of how deep into the list the reader has scrolled.

The directory’s row count — the “1,247 forests” the post opens with — is a separate query, a count(*) over the same filter set as the page but without the cursor window. This is a known v1 cost: count(*) walks the index every time, and it gets more expensive as the directory grows. A production-grade system either swaps in an estimated count (Postgres’ own statistics tables expose one, accurate to within a few percent) or maintains a running counter in a cache layer — Redis, or a denormalised counter table — invalidated on writes. We’ve done neither yet because at private-beta scale, the exact count query is still cheap.

Search uses Postgres’ built-in full-text indexing. Each forest’s content is indexed into a search vector, and queries against it return ranked matches plus short highlighted snippets. This scales comfortably into the millions of rows; beyond that, a dedicated search service (Elasticsearch, Meilisearch, Typesense) gives more control over ranking, typo tolerance, and faceting.

Ranking is the largest piece of v1 debt. Today the directory sorts by raw recency — the forest with the most recent activity goes first. There’s no scoring for how well a forest matches a reader’s interests, how active the people they know in it are, or how an unfamiliar topic should weight for a stranger. A real ranking pass — even something as simple as recency × engagement × interest-overlap — would change what the directory feels like at any non-trivial size. For private beta, where a fresh user has near-zero history to score against, recency is honest enough.

Horizontal scalability. The API is a stateless service; adding more instances behind a load balancer is the easy axis. The database is the harder one. A single primary handles every write today, and reads from Yours / Discover / Search all hit that same instance. The first scaling move is read replicas: duplicate the database onto a few read-only followers and route the directory’s read workload there, keeping the primary for writes. After that comes application-level sharding (by user, by forest, by region), which is more invasive — moving from one logical database to several. None of that is needed yet; everything to date sits comfortably on a single instance, well inside its working budget. Enough for a private beta, not enough forever.