Record and Query Layers: Which One(s) for this Working Group?

Working Groups WG Community Standards
1

Hello everyone! @brittanyellich.com and I met yesterday for an informal brainstorming/thinking session about Lexicon, and ended up with a question I’ve never really seen spelled out, but that may help make sure we’re aligned on what/why/how.

Record and Query Layers

When defining Lexicons there’s 2 possible layers at which we can operate:

  1. Record Layer: app.bsky.feed.post, defines how information is stored
  2. Query Layer: app.bsky.feed.getPosts, defines how information is transferred/requested

These may depend on each other (getPosts returns a view over stored post records), but can be tackled independently.

A standardization process like the one of this WG may span both layers. For example we could:

  • Agree on the record layer exclusively ⇒ “WG participants agree this is the way Communities store their data, which means anyone with direct PDS access can interpret that data for themselves”
  • Agree on the query layer exclusively ⇒ “WG participants agree this is the way “Arbiter/AppView-like” entities (whoever mediates the community) are asked for data and how they respond.”
  • Some in-between ⇒ We may agree on how some data is represented at the record layer for e.g. membership, and also provide a getMembers query layer, but for other features (e.g. getSubgroup) we agree we only standardize the query.

What Layer(s) do we work at?

As Brittany and I worked through a possible Lexicon proposal, I realized people here may have different ideas about which layer our output is supposed to target, or at least we have never specified enough to have a clear answer to “which one are we interested in, and for which operations?”

I’ve been thinking about the implications of focusing on each, and I’ve come to think that:

  • People building apps may be most interested in a standardized Query Layer, as it allows them to fetch information from different “Group AppViews” in a standardized way (or make sure their AppView is compatible with other people’s).
  • But it’s important to get the Record Layer right, because it’s what protects implementations from drift and allows real data portability across AppViews. For example, it allows two different AppViews to serve different views over the same community data. And also, it allows independent AppViews with access to the same data to verify assertions about e.g. membership.

The risk

What I see as “the risk” (and this is mostly my musing) is falling for the temptation of mostly standardizing the Query Layer. After all, that’s the one that unblocks building actual apps!

But without some underlying idea of what data an AppView can expect straight from the PDS, users risk finding themselves with a lot of really nice data that no AppView knows how to interpret. Or locked into whatever AppView they signed up with, because no other implementation can make sense of its data.

The question

I mostly wanted to bring this up as food for thought cause I had not quite seen it spelled out this way, and I realized I did not have an answer myself. But now I’m curious: @dholms.xyz and @bnewbold.net, is there any recommendation for this somewhere we can point people/ourselves at? Has this been explicitly discussed for Bluesky Lexicons? And also @zicklag.dev and @blaine.bsky.social et al., I’d love to hear if you’ve thought about this in your own implementations of groups.

7 Likes
2

I’m not in this working group, but I just wanted to share some thoughts because I think this conversation touches on the lack of guiding principles for atproto. I’ve always been fascinated by the design principles that emerged out of the early web, whether for platform design or building on top of the platform (e.g. progressive enhancement, rule of least power, browser as a user agent, etc.) and the ways that it shaped those working on the web. It makes me wonder what design principles could be for atproto and, for example, if it would help guide this working group! Off the top of my head:

  • The PDS as a server-user agent.
  • The PDS is an actor-agnostic ingestion point for an end-user. This was first spelled out to me by @nekomimi.pet.
  • Per @tyler.fun, lexicons should be designed as the minimal source of truth. Data not generated by the end-user should not be stored in their PDS, even if it might be convenient to do so.
  • End-user’s must always consent to someone writing to their PDS, and in the case of permissioned data, consent to who is reading from it.
  • Apps should link to the atproto underlying atproto data they are referencing.
  • There are many different social context’s an AT record could hydrated with.

These aren’t really that complete or fleshed out, but I think they do poke at a direction for some of the questions being asked.

The PDS as an actor-agnostic ingestion point would lead the working group to consider the lexicon layer to be the most important layer to reach consensus on. Especially if the rough definition of a community in the potentially atmosphere lives across many different apps. That’s why I think your risk assessment is spot on! I also wonder if there is an assumption that only appviews interact with lexicons data?

At the query layer, I wonder if the question to start with is what community data can’t be modeled as the minimal source of truth per account. That is the data that I would suggest this working group should prioritize standardizing. I would recommend against the urge of standardizing how records are hydrated with social context, as there could be many potential contexts that emerge across app views. I’d also love to hear more about what commodifying the appview for communities would enable.

2 Likes