Post-alpha plans

Working Groups WG Private Data
1

Hey folks :waving_hand:

We’re about a month into the spaces alpha! From what I can tell, it’s going along pretty nicely. We pushed out one breaking change about two weeks back, but it’s been pretty stable apart from that.

I wanted to share a list of some of the open questions we’re still noodling on. Many of these will likely involve breaking changes. Most of these are not truly “open”, we have opinions on them and need to just nail it down. However, if anything is important to you, please share your thoughts.

We’re hoping to hit all of these (and any other changes of similar scope) by end of October.

Any change we make will first be made as a PR on the proposal repo. Then it will be merged when the implementation is updated and deployed. This happens on Thursdays and will be recorded in the alpha updates Discourse thread.

If you missed it, we’ll be doing an office hours session on Thursday (sept 24) to chat about the alpha, answer questions, and collect feedback.

  • Open write spaces
  • Space credential expiration/revocation
  • Public spaces
    • Are public spaces in any way different from permissioned spaces?
    • Should space authorities be able to mint a credential when no delegation token is offered (ie unauthenticated)?
      • Of course this is always “possible”, but is it considered “offspec”?
    • Could/should public space repos be rebroadcastable (ie signed)?
  • DID docs
    • Do we definitely want new service/key identifiers?
    • What exactly are the semantics?
      • For instance, can space authority sign a space credential with their #atproto key (specified through kid claim) if they also have an #atproto_space key in their DID document?
  • Repo serialization
    • The currently-specced serialization format sucks.
    • Should this be a CAR file? Or should it be a new format? (for instance STAR-lite)
    • Do we need a “multi-repo” format used in account migration?
  • DPoP
    • Is this the right fit in the credential flow?
    • Technically specified for OAuth.
    • We could have our own construction using HTTP signatures or similar
  • getSpaceCredential
  • Commits and signatures
    • Should signatures be on the commit or should they be transferred as metadata (ie in HTTP headers)?
    • Should we offer a deniable signature on single-record requests? (ie via com.atproto.space.getRecord)
    • I think Bryan convinced me that the HMAC signing scheme needs a consumer-provided nonce. Expect some sort of tweak here
  • Account status
    • What is the account status of a space authority?
    • Is it the same as normal account status? Can #atproto_pds and #atproto_space_host vary?
  • Sync scaling/reliability
    • Can we make write notifications less lossy?
    • Does PDS (space repo) need to remember and retry notifications to space host?
    • Does space host need to provide better self-healing mechanisms?
  • Space deletion
    • Is the deletion flow robust enough? It is lossy & best-effort.
    • Do members need to receive a notification of space deletion? Or does just syncers suffice, and applications re in charge of informing members?
  • URI scheme
    • Should we try to resolve IETF syntax issue for space URIs (ie dropping slashes, ie just at:)
    • Should we support either blobs or versioned records in space URIs (things we’ve wanted in public URIs but just haven’t gotten around to)?
12 Likes
2

Thank you for laying out the questions for the post-alpha phase. I look forward to Spaces becoming a foundation for a wide range of apps. I support a design that provides shared mechanisms for authorization and sync while leaving apps and Space authorities to decide the policies appropriate to their use cases.

Below are my comments from an app implementation and operations perspective, including points I support, concrete use cases, and behavior I would like to clarify.

Open write spaces

I support open write spaces. They would be useful for receiving information, such as membership requests or inquiries, from people who have not already been registered as members. Independent read and write settings are particularly useful here: being able to submit something should not imply being able to read other people’s submissions.

Space credential expiration/revocation

I support shorter credential lifetimes combined with an explicit revocation mechanism. Using short lifetimes as the baseline, with revocation of individual credentials when access needs to be withdrawn sooner, seems like a good way to make revocation take effect promptly without requiring a call to the authority on every read.

Public spaces

For public spaces, I think the specification should allow a Space authority to issue read credentials without a delegation token. It seems natural that an app should not need to maintain someone’s user session just to retrieve public information. I support keeping the same data structures and sync APIs for public and private spaces while allowing the conditions for issuing credentials to differ.

I also support signatures that remain verifiable after redistribution, so that data from public spaces can be authenticated when retrieved from mirrors or archives.

DID docs

I support dedicated service and key identifiers for Spaces. The option to continue using #atproto_pds and #atproto also seems useful for migration from existing services.

Allowing Space Credentials to be signed with #atproto even when a dedicated key is present should be an option for compatibility, rather than a requirement for every Space authority.

Repo serialization

For serializing a single repo, I would prefer CAR if it meets the requirements, since it allows reuse of existing libraries and tools. I am also interested in formats such as STAR-lite, but I think the choice should be based on how much they improve transfer size, memory usage, and implementation complexity for the data structures used by Spaces.

A format that bundles multiple repos would be useful for account migration and backups. Reusing the serialization format for individual repos within that bundle would make it easier to share implementation between routine sync and migration.

DPoP

I support continuing to use DPoP. When the same Space Credential is presented to multiple repo hosts, binding it to the presenter’s key seems appropriate to prevent a host that receives the credential from reusing it to access another host.

There is also value in reusing implementations and experience from AT Protocol’s use of OAuth. As long as its application to Spaces can be clearly specified, I do not think the fact that DPoP was defined for OAuth is, by itself, a reason to choose a different mechanism.

getSpaceCredential

I support allowing getSpaceCredential to work with authorization methods that do not require a user’s delegation token. For example, a backup or search indexing service explicitly authorized by a Space administrator should not need to keep maintaining and refreshing a particular user’s OAuth session for ongoing sync.

It would be useful to establish the service’s identity through client attestation and issue a Space Credential based on explicit permission granted to that service by the Space authority. This would allow access for services to be managed independently of individual user sessions, while repo hosts continue to use the same Space Credential verification mechanism.

Commits and signatures

I support providing deniable signatures for individual record fetches as well. For example, a client displaying just one linked post, or a tool fetching only the records it needs, would benefit from being able to verify those records without syncing the entire repo.

For permissioned data, it would be good for these partial fetches to preserve deniability in the same way as regular repo sync.

Account status

If #atproto_pds and #atproto_space_host can be operated as separate services, I support treating their account statuses independently as well. For example, someone might deactivate their PDS account for public posts while continuing to act as the authority for a community Space hosted by a separate service.

Deactivating an account on one service should not necessarily imply deactivating it on another service associated with the same DID. I think the design should distinguish the availability of an author’s records from their active status as a Space authority.

Sync scaling/reliability

It would be good to have a mechanism for retrying write notifications from repo hosts to Space hosts after an outage. I am particularly concerned about a new writer’s first notification being lost, with no subsequent writes. If the Space host does not know about that writer, wouldn’t the writer remain undiscoverable even if a syncer periodically checks listRepos?

I would like a guarantee that the writer can be discovered and synced in this case without requiring another write. If notifications can be coalesced to the latest revision of each repo, there should be no need to retain and resend every write notification individually.

Space deletion

I support sending Space deletion notifications to syncers and leaving member notifications to apps. How users should be notified depends on the use case, and I do not think the Spaces protocol needs to require enumerating all readers or managing their notification destinations for this purpose.

I also support using SpaceDeleted during credential renewal to detect deletion when a notification has been missed. Distinguishing deletion from a temporary failure would let syncers proceed with deleting their local copies and derived data without depending solely on receiving a notification.

URI scheme

I support adopting a format that conforms to standard URI syntax, provided that Spaces will not end up using a mixture of the existing AT URI format and the new URI format. I see value in integration with existing link handling, such as treating references in documents as links in native apps and opening them in an app that handles the scheme.

I also support being able to reference a blob or a specific version of a record with a single URI. This would be useful for linking to attachments and making it explicit which version of a document or post is being cited.

3 Likes
3

I was about to make a separate topic but noticed this was here and this is niche-ish sync concern.

Just had a longer talk on the atproto touchers discord about recovering from situations where an appview looses it’s list of spaces it has access to for whatever reason. A bit upside of public atproto data is that, assuming your app is properly designed, you can more or less nuke your appview DB and fully or mostly fully recover your whole app just from backfill. Spaces are harder in this regard though since even the existence of a space is potentially permissioned knowledge. So if you end up “forgetting” you have access to a space you might never know when trying to backfill. For that use case it would probably be nice for space hosts to expose a “hey there space host do you have any spaces for me?” endpoint. An authenticated endpoint that services (or any identity really) can make calls to to get a list of all spaces that that space host is willing to offer that identity access to to some degree. Could be styled as a spaces version of listReposByCollection. This could feasibly also be part of space-management implementations instead of the core spaces protocol. But I think it’s worth having a generic option. Or at the very least not leaving simplespace out of it if we do decide this should be a space-management thing not a core protocol thing.

Other stuff here I have an opinion on:

and

I think we definetly want space hosts to have dedicated service and key entries. I don’t have a strong opinion on the currently proposed fallback behaviour, though I might argue that not having a fallback is simpler conceptually. Similarly allowing use of an arbitrary key entry with the kid claim follows the now established pattern of how atproto JWTs work so I’d argue it’s good to keep that going. For ease of use, implementation and understanding by everyone.

Most importantly I think that having a connected space host should not constitute that identity having an account. That resolves the account status question: space authorities have no account status unless they also have an account at a PDS, and if they do then it follows the same semantics as for all other identities with accounts. Space authorities are a lot more similar to service identities than user identities and we should treat them more as such in my opinion.

I think signature should be in the commit. Mainly makes the commit object self contained but also makes it better suited for potential future transports (or apps choosing specific custom transports).

Single-record deniable signature would be really good if you ask me.

I’d say public spaces are absolutely different enough to be worth considering special protocol support for. To me public spaces probably exist in two forms: spaces that have always been public and permissioned spaces that turned public. Both of these would benefit from public space commits being rebroadcastable and maybe even being allowed in the firehose/subscribeRepos semantics. I remember talking about public spaces with @zicklag.dev a whiiiiile back and we kept returning to how loosing out on the firehose is a big big downside to private data that later got made public, but also that permissioned repos later going public is a practically unavoidable usage pattern for many app modalities. I think the sync patterns change enough in these cases that seamlessly integrating with the firehose or something similar is absolutely worth it if you ask me. For the same reasons that not using MSTs and the other existing sync primitives is worth it for permissioned spaces. Not to say it should necessarily be exactly the same primitives as existing public data, but something close to it probably.

4

There is a bunch here!

An authenticated endpoint that services (or any identity really) can make calls to to get a list of all spaces that that space host is willing to offer that identity access to to some degree. Could be styled as a spaces version of listReposByCollection.

I understand the desire for something like this, but there are a few issues with it.

In the current design, space access is gated primarily by identity of an individual account, and then optionally by app client identity. This means that apps only get access to spaces through an active user session, like an OAuth session. If all app state has been lost, then all those sessions are gone, and the app can’t get access even if they knew to ask.

A counter-point is that an app service does register as a syncer, and gets notifyWrite pings when any space repo in the space is updated. But that doesn’t include the actual content, and the syncing services needs to authenticate (with a space credential, from a user session) to get the actual data.

For some applications I think this might be fine. The recovering app will gradually reindex as users log back in and grant access to their space repos, and the app backfills the entire space from the space host. If a user doesn’t re-authenticate, maybe it is good that the service can’t re-index the data?

I’m not entirely sure how spaces that a user only has read-only access to would be discovered for indexing by the app service. Maybe this becomes modality-specific?

One thing you can do is backfill all the discoverable “public” spaces (which have declaration records, and will hand out space credentials to anybody).

On DID document entries:

I was actually just thinking the opposite today: that we might be able to assume that the space host is always the account host. For individual accounts on a regular PDS with only simplespace, this will be true. And with the “group host” concept, it will also be true for group/community accounts.

But need to think it through further.

Public spaces:

I kind of think it should be possible to tell if a space is re-broadcastable or not should be a permanent status, and indicated in the URI (!).

A hard direction in particular is a space having been publicly re-broadcast, and then flipped back to non-public.

5

Some short updates on design progress (@dholms.xyz can you turn the original post in to a “wiki post”? I think that is just a config flag):

Public spaces: pretty confident we aren’t going to make any big changes here. No re-distributable cryptographic commitments, no space data on firehoses, repo hosts require space credentials for all space repos (even if “public”). As a small change, will probably update space credential endpoint so that both delegation token and client attestation are optional: that makes it clearer how to fetch “public” spaces w/o an atproto account (if the space authority allows it).

Repo serialization: expect to ship with STAR files not CAR files

DPoP: switched to HTTP Message Signatures last week

Sync scaling/reliability: changes shipped last week

Cryptographic commitment: think we might need to do proper Diffie-Helman (then KEM for post-quantum). That would mean publishing an addition key in all DID documents :pensive_face: but not the end of the world. Also want to iterate on which fields end up in the repo commit metadata object, and which end up as an HTTP header.

URI scheme: still furiously debating :sweat_smile:

Simplespace configuration: have a thread going over here: Updateable space policy by matthieusieben · Pull Request #119 · bluesky-social/proposals · GitHub , probably should have been a forum thread instead

Have not started on a couple of the other questions yet. But generally making progress and feels to me like we are on-track to have most of these resolved by end of October, and to be able to ship some time in November.

4 Likes
6

Yes have made it wiki editable. OP author should be able to do this too FYI.