Permissioned data proposal discussion

Working Groups WG Private Data
1

This is a forum thread to collect feedback on the permissioned data protocol proposal!

To clarify: this is still a proposal. I expect many changes along the way. Suggestions may also be submitted as issues/PRs.

I had hoped to have an alpha version of the PDS alongside this proposal, however I didn’t quite get there yet. Hoping to have that in the next couple weeks.

The previous thread was here: Early permissioned data proposal draft feedback

Changes since then:

  • Responded to a bunch of feedback from the PR. Tightening, clarifications, renames, etc
  • Added “Considerations” section at the bottom with some non-normative thoughts on moderation, scaling & account lifecycle
  • Added a permissioned repo serialization (CAR) format
  • Switched from ats:// → at:// (this one is still not quite decided - discussion over here)
  • DID-doc entries are optional with fallback to current #atproto/#atproto_pds values
  • client_id removed from space credential
  • Oauth scope parameter changed from did → authority & defaults to self (so bare space:<type> scope now grants only spaces under the user’s authority)
13 Likes
Permissioned Data Token Exchange
2

Sharing my official public feedback on this proposal:

Hot. dog.

7 Likes
3

Overall looks great.

The access control requiring a users delegation token leans heavily into user triggered access, which makes sense for most use cases. For clients that need to process data ahead of delivery to users (e.g, client moderation, notification of new content), or clients that are working on publicly readable permissioned spaces, is it intended that they use a bot DID of their own to request space credentials?

And a minor one: you mention that a space authority can “un-register if it no longer wishes to be notified”, but I don’t see where that can be done in the current write up.

4

I want to put this on record because the proposal lands pretty well for the book-clubs work I’ve been doing on Livtet ( What's Supposed to be Different about Livtet? - Jacky Alciné ) - and that’s a narrow enough use case to be a useful stress test.

The shape works. Federation of shared reading groups is now possible without me having to stand up a relay-equivalent; notifyWrite plus listRepos is the operational backbone, and the LtHash self-heals missed events. The per-reader ikm and HMAC means a leaked proposal doesn’t prove what the user said, which matters when a club is reading something a member would rather not be associated with later. That’s not a hypothetical concern -I think about it for clubs I’m in (and considering the current political landscape).

A few things gave me pause, though:

  • Member eviction has no protocol signal. @dholms.xyz named this in Diary 6 ( Permissioned Data Diary 6: Boring Auth - Daniel's Leaflets ): “revocation has to fan out to every writer in the space, not just the space owner. Tractable, but not trivial.” Shape I’d want: com.atproto.space.notifyMemberEvicted, sent by the authority to every endpoint registered via registerNotify. Payload would be something like
    { space, memberDid, evictedAt, reason? } where reason is a typed list of voluntary | removed | expired. Sender authenticated via service auth. Receivers treat it as a hint and confirm via credential-renewal failure before deleting - that’s what closes the door on spoofed-notification-triggered purges and it’s also the path missed notifications self-heal on. Without this, GDPR’s right-to-erasure story has no answer for pull-based syncers (and tbh, this makes it easy to allow folks to consentually leave and not be forcibly readded; gives clients a way to pull immediate removal from groups, etc etc).
  • URI unification. @bnewbold.net — I’m following your lean toward at:// + space even though you argued for ats://. I’ve isolated URI handling behind one abstraction, so a flip is a one-file change if the consensus swings the other way. But your visual-distinctiveness point landed for me; I won’t mourn it either way.

authority=self defaulting on the OAuth scope I think is right — but the revocation UX is where the protocol can’t help me. Once a member has granted space:…?authority=alice, how do they later see and revoke that grant without losing their data? Bluesky’s per-app-permission screen is the closest reference; the protocol doesn’t dictate the shape and the granularity here is more than most users are used to reading, I imagine.

1 Like
5

Hi Dan, discussing this in a stream Eli is doing following the most recent blob post.

One big question is what happens if the data in a space becomes public, I think there’s two versions of this.

  1. Data in a space becomes “accessible by anyone with an account” which is I think already accounted for in the proposal.
  2. Data in a space becomes “accessible by anyone” similar to the current public data

It’s a bit unclear from the current proposal but could you have a space host just hand out space credentials to any user if we wanted to move to a truly public access pattern?

1 Like
6

Hi there,

I’ve been making apps using the Solid Protocol for a while, and I’ve also been following ATProto’s evolution because there are many similarities. However, most of my work has been on Personal Apps with private data, so I couldn’t do anything with those in ATProto… until now :).

I’m saying that to add some context to the feedback I’m about to give. I know ATProto’s community is very much focused on “Social Apps”, but every time I’ve raised this issue I’ve been told that this would be solved by permissioned data. So here’s my feedback on that!

I have read through the Permissioned Data Diaries and the proposal, and there are two main things that I want to comment on:

1. Different protocol/schema

There are many references in the blog series about this being a different protocol, such as:

we are specifying a new data and sync protocol, not just an extension to the existing protocol

I was a bit surprised about this. Maybe I don’t understand how ATProto actually works (though I have read the whitepaper and a bunch of blog posts/talks), but this seems like a false dichotomy. Why do we need a different protocol?

This is briefly addressed when you talk about Public Spaces, but I still don’t understand why. If public data can also be “permissioned”, why not use the same thing everywhere? It may seem like I’m just saying that to be pedantic, but there are some practical implications.

In Solid, and the Web in general, the shape of the data doesn’t depend on its permissions. If you try to GET a resource, it doesn’t matter if that resource is public or not. If you can read it, you’ll get its contents; if you don’t, you’ll get a 401. But you don’t have to treat “public” or “permissioned” data any differently.

By the way, I’m glad you’ve decided to go back to at:// for now (instead of ats://). Tim Berners-Lee has talked about how https:// was a mistake for the Web. Mostly, because it breaks links that transition from one to another. And we’re still suffering the consequences of that, even in Solid. For example, schema.org is one of the most popular RDF Ontologies (the equivalent to Lexicons in ATProto), but some apps write it as https://schema.org whilst others use http://schema.org. Which keeps them from being interoperable :(.

2. Data coupled to permissions

Even if we accept that permissioned data will be different than “public” data, there are still some issues with Spaces as I understand them.

In particular, this seems to assume that the permission of one piece of data is not going to change. This seems a bit abstract, so let me share a specific example.

I have an app called Umai which is a recipes manager. Basically, I use it to store my cooking recipes, and by default everything is private. However, sometimes I do want to share a recipe with some other people. This can mean making a recipe completely public, or adding individual people or groups gradually.

If I was to implement Umai using ATProto’s permissioned data, how would that work?

As I mentioned, recipes are private by default. So I guess I would need to create a private Space for recipes in the user’s PDS. But then, how do I share a recipe with a specific person? Would I need to create a different Space that is shared only with that one person, and move the resource in there? Wouldn’t that break the URI, and all the references to it? And I’d need to go through all of that every time I wanted to change permissions?

3 Likes
Interoperable Serendipity
7

HI Noel! Welcome to the community

A lot of us have struggled with the linking of space to permissions. Where data is “stored” being a part of how it’s permissioned. The practical thing myself and others have landed on is “lots of spaces” so in your case probably one space per recipe. This is done using the “skey” or “space key” which should be reasonably low overhead as the spaces can all share a single “space host” and I believe actually get a single “space credential” to access all of them.

There is a general issue with this in that you need to over allocate spaces based on future sharing preferences, assuming you want stable urls. So in your case let’s assume you start out wanting to just “share a recipe book” with a friend, you may be tempted to have a recipe book be a space. But once you’ve decided that you want to share a single recipe then your data needs to either move or get duplicated.

2 Likes
8

Really the only bit of feedback I have is to keep the ats:// or heck even atp:// schema - if it goes into at:// then parsing an at:// link just got more complicated and now has an initial path fragment that isn’t a collection but the space indicator (or whatever you’d call it).

Perhaps trivial to moan about, but it’s faster to check at:// v.s. ats:// than it is to parse the path out of an at:// URI and then parse the path to see if it’s a space or not.

Besides, ats:// would give you a clean slate for potential later bolt-ons, whereas trying to jam everything into one schema seems… not right somehow.

My 2 cents, for what they’re worth :slight_smile:

9

4 posts were split to a new topic: Permissioned Data Adversarial Migration

10

Boy I did an awful job replying here… Thanks for all the comments!

Nope, the intention is that you do so with a long-standing OAuth credential, which shouldn’t require any user interaction (aside from the initial grant.

Good catch. Need an unregisterNotify as well

“Membership” is not a protocol notion. That’s an application notion. So if it’s important that downstream syncers understand that membership has been revoked, this should be represented as records within the space.

Though I am increasingly thinking that we need a facility for fanning out authorization revocations which are a protocol notion (rather than just relying on credentials expiring).

Yes it’s totally the space host’s prerogative! They may still require a delegation token before minting a credential (even if they’ll mint a credential to anyone who asks), or they may mint a credential to anyone, with no proof of account.

There’s still a difference between “public broadcast” (in the public square, archival, self-certifying, global reach from an account) vs “public spaces” (bounded, has a container (the space, non-archival, etc). In other words, I don’t view the difference between the protocols has just public vs permissioned.

The space itself is coupled to permissions. Access control is at the space level. So generally you should group together only data whose access will change at the same time. If each recipe has different access boundary, then they should each be stored in a different space. Spaces are cheap and very low overhead!

2 Likes
11

Can you share some specific examples of data that makes sense in a public space but doesn’t as public broadcast? I’m still not sure what’s the actual difference in practice.

The fact that some data is “bounded” by a container/space is only a semantic or organizational aspect of the data, I don’t think it’s related with permissions. And you mention archival, but I’m not sure how that’s related to permissions either (in a practical way). The wayback machine is archiving most of the internet, and they don’t have to ask permissions to anyone because they archive already public content.

I do understand there is some difference here, but I’m not sure why it’s worth making the trade-off to have two types of “public data that anyone can read”. Specially since the distinction results in using two completely different protocols.

The point I’m trying to make is that most of the time, we shouldn’t be making any assumptions about how the permissions of certain piece of data are going to evolve. It’s always possible that you want to start sharing some piece of data with a given individual or group. Or that you want to restrict some data that you published too broadly, etc. When data is coupled to applications, it may make sense to think like this. But now that we’re trying to decouple data from applications, I think trying to make assumptions about how the data is going to be used could lead to problems.

In practice, this means that I’ll always be creating a space for each piece of data. And even though you mention there isn’t too much overhead, I’m not sure what’s the point of spaces if we have to create a space per resource.

By the way, can spaces be nested? Like, could I have a space that contains other spaces and permissions are calculated by doing an overlapp of permissions? Maybe that would solve the problems I’m thinking about. But in any case, if moving pieces of data between spaces breaks their URIs, we’ll end up with the same issue.

1 Like
12

I’m building email on atproto, comail, which started as a sending relay (think SES/Postmark, not Gmail), because a relay never stores anyone’s mail and I don’t want to be responsible for storing and managing people’s emails. Basically, I want to avoid as many data custodian problems with email as possible, lest we end up in a wikipedia page like comail_email_controversy. Permissioned data lets me do personal inboxes without giving that up. The mailbox becomes a space on the user’s own PDS, and comail stays a gateway for sending mail.

Correct me if I’m wrong, but reading the Sync section makes me think comail will just work(™️) out of the box? Comail joins the user’s mail space as a write-member and at delivery time writes the message into its own repo in the space, with its own session (w/ the long-standing credential). The user doesn’t need to be online for any of this. Their PDS gets notifyWrite and their app pulls. Do I have that right?

The part that undoes my custodianship story: the authoring host is canonical for sync and backfill, so every message comail delivers lives on comail’s host, forever. I can work around it above the protocol (user’s app copies each record into their own repo, comail deletes its copy, comail’s repo becomes a short-lived delivery queue), but that re-authors every record, moves every blob twice, only runs while the user’s agent is online, and every app will invent its own version of it.

So I guess my question is, is protocol-level re-homing (written by one member, accepted by another, then served from the recipient’s host) on the table? Even a documented convention would be great. It’d also make @omg.jacky.wtf’s revocation point cleaner: removing a member-service like comail shouldn’t orphan your mail archive.

13

Loving this! I worked on personal data stores in the Unhosted project and did some work in the past on remoteStorage, Open Cloud Mesh, Solid Protocol, and some failed attempts to build auth into CRDTs, so a lot of the things I’ve been running into there have been solved very elegantly here.

I think the main bottleneck I’m seeing in this design is that the space authority has to make an access decision based on user identity (and optionally client attestation). It’s the model that Solid also uses and it’s limiting for use cases where for instance you want to host a company intranet. Attribute-based access control would be much easier there if the company has many employees.

There may also be use cases where revealing the reader’s identity is not necessary and not required - for instance if I paid to attend an event or to ride a train, there is no reason to link that transaction to my real identity.

There might even be EU regulation that prohibits the application and the space authority to exchange PII if this is not needed for the use case. AFAIK attribute-based access control is pretty standard practice nowadays for those reason of scalability and of pseudonymity in stacks like SAML, OIDC, EUDI, and basically any system that mentions wallets and Verifiable Credentials. I know AT has a strong focus on identity in its fundamental design (same as remoteStorage, OCM and Solid), so you may want to stick to that, but at least worth writing down the pros and cons of that design choice in comparison to attribute-based.

You may have already discussed this somewhere I didn’t see in which case please point me to that discussion and I’ll shut up! :slight_smile:

2 Likes
14

This could be applied either by a Labeler or have an organization signed verification in the accounts repo. I think this is compatible with what we have.

1 Like
15

You’re right! The fact that the space authority is free to choose how it makes its access decision makes all of these use cases and scalability optimisations possible with some extra steps, except that it still has the hard requirement of including proof of identity as a claim in the call to getSpaceCredential. So my main feedback then is that I think the protocol could be more generic there. I wrote that up in re: 0016 make getSpaceCredential more generic? · Issue #97 · bluesky-social/proposals · GitHub.

1 Like
16

I want to make a general comment about the idea of permissioned data spaces - not on the design and implementation details specifically.

I think that this proposal changes drastically the privacy threat model of ATProto PDSes and that will lead to problems. This is moving from a fairly simple model where everything is public to one where you now have to trust the PDS operator and be more confident in the implementation to not leak data.

The complexity of the system is much higher. This is a scary evolution and we can expect that many end users will not understand the subtleties of the model and will over-share, thinking that permissioned data is somewhat private. PDSes will also become juicier targets for attacks, especially given the lack of e2e encryption of the permissioned records.

3 Likes
17

We have some feedback.

4 Likes
18

A post was split to a new topic: Discoverability of spaces at the protocol level for permissioned spaces

19

WordPress Jetpack’s evaluation of atproto permissioned data is instructive: