Transitioning data from permissioned to public?

Working Groups WG Private Data
1

Dropping my post here to see if we can kick off discussion. What is an atproto repository? - The Book of Eli

But here’s real-world use case and I’d love to hear peoples’ takes on how we should handle this: we want to allow paid subscribers to see a creator’s VODs three days before they go live for the rest of the world. That includes likes and comments on this “unreleased” content.

The part that seems clearest to me: we can implement this by having a separate space for each video, owned by the creator:

at://did:web:creator.example.com/space/place.stream.videoSpace/3mqumd5c5nf22

The video itself would be at:

at://did:web:creator.example.com/space/place.stream.videoSpace/3mqumd5c5nf22/did:web:creator.example.com/place.stream.video/3mqumd5c5nf22

And then all the likes and comments can live there too:

at://did:web:creator.example.com/space/place.stream.videoSpace/3mqumd5c5nf22/did:web:fan.example.com/place.stream.like/3mqumeppy2z22

{
  "$type": "place.stream.like",
  "subject": "at://did:web:creator.example.com/space/place.stream.videoSpace/3mqumd5c5nf22/did:web:creator.example.com/place.stream.video/3mqumd5c5nf22"
}

at://did:web:creator.example.com/space/place.stream.videoSpace/3mqumd5c5nf22/did:web:fan.example.com/place.stream.vod.comment/3mqumfkzuv622

{
  "$type": "place.stream.vod.comment",
  "subject": "at://did:web:creator.example.com/space/place.stream.videoSpace/3mqumd5c5nf22/did:web:creator.example.com/place.stream.video/3mqumd5c5nf22",
  "text": "I love this video!"
}

So far so good, all of this makes sense to me. Now the video needs to become public. So do all of the likes and comments. AFAICT, the recommended way to do this: leave the video in the space, but admit anyone who asks; it’s now a “public space” of sorts. I don’t love it for a few reasons:

  1. There’s overhead associated with spaces; it seems a shame to incur that overhead for content that is private for 3 days and then public for 10,000 years.
  2. We now need a new mechanism to advertise this public data on the public firehose so it may be discovered by apps listing public Streamplace VODs. Doable, but more conceptual complexity for anyone implementing this stuff.
  3. Spaces are apparently designed to scale to “millions” of users, but not hundreds of millions. The most liked MrBeast video on YouTube has about 57,000,000 likes, so we’re right up against the scaling boundary there.
  4. It should be possible to create a dashboard of all public Streamplace content prior to a single user signing into that dashboard. Do we have a pattern for anonymous access to a space? I guess the syncer could just mint itself a did:web identity and use that to get access to everything.

Perhaps not insurmountable, but it feels really weird and counterintuitive to be on the AT Protocol forums talking about all these hacks we’re gonna need to do to make data public. If only someone had invented a way for us to do that!

In my article I propose allowing me to sync spaces using the public sync mechanism. (As it happens, the aturi-record pairs themselves don’t know or care whether they’re organized into a MST and signed or deniably LtHashed and HMAC’d.) I’m cognizant of the fact we wouldn’t want this to happen accidentally – I could even imagine for some cases you might want a protocol-level guarantee that a particular space will never be made public. That’s fine!

TLDR: I’ve got a bunch of spaces full of public data, may I please publish them on the public firehose?

16 Likes
2

Highly agree. This also honestly goes the other way, all things considered; what if I want to make public content private, without losing all of the information associated with it? I feel like permissioned data is, in a way, too restrictive for its own good. There’s too much “we have permission data” and not enough “the line between public and private data should be invisible to cross” which is frankly a massive shame.

1 Like
3

Spent some more time thinking through this.

I think the “big” overhead is in the space credential. Even if we had a space host that would give space credentials to unauthenticated requests (out of spec but possible) a getRecord request is now 2 did resolutions and 2 requests (get space credential and get record) we could mitigate this but putting a never expiring space credential publicly on the internet and just telling clients to hard code it. That seems wrong.

We could move data, and have a standard for redirects but that means that anything pointing at the permissioned URI now needs 3 requests.

A PDS could serve data in a space without a credential (public) fairly easily, you wouldn’t need to change how the data is stored or addressed, the PDS would just need to know that it’s not supposed to require a credential. The advantage of this is that you could also solve the “newest blog post is free, back catalog for subscribers” pattern where posts start public and become permissioned. A signal from the space host to basically say “turn off/on auth for this space/skey” could be best effort on a PDS.

The one problem with this approach is that to get the benefit of it you need to know if you should try a direct PDS request, I think in practice you could do this in parallel and do some caching. But it definitely means worse case has gone to 3 requests instead of 2. Even worse for PDS that didn’t get the “turn off auth” message you still need to go get a Space Credential.

Ultimately, where I’m landing is that we don’t need new machinery for this other than a space host that’s willing to give out long lives credentials for unauthenticated requests. The expiration time of the space credential is the period on which you can change permissions. So if it’s really public for 10,000 years then you know what your expiration time is.

On the 1M soft cap that seems like it’s a cap on writers not readers. After all if you have a multikb hash to compute / hold on to per writer it’s going to get expensive. So when Mr Beast gets on stream place I think most of those likes are going to have to be in public records. That means public records pointing at permissioned records which I think is fine.

4

@iame.li I gave this some more thought and I think this is the answer.
Permissioned data has the following scheme*:
at:/did/recordtype/rkey/

Internally the pds does the mapping of which records are in which spaces but it’s not part of the aturi.
When you do a getRecord request you get back a 401/403 with a header that has the did of the space host, and the space type and space key (at:/did/spacetype/skey). You then authenticate with the space host and then make the request again to the pds with a space credential.

Everything else about the protocol stays the same.

When data wants to “become public” your pds just stops sending the 401 and starts serving content.
This also has the nice benefit that I’ve been looking at where a single records (single aturi) could be in multiple spaces at once which generally solves an over-provisioning problem

2 Likes
5

Seems related to how @brittanyellich.com designed OpenSocial with “wrapper” records. That was all pre-spaces, but the approach might hold.

1 Like
6

Just dropping this good post in here for reference and perspective:

Before we get into it, the through-line on all of this is that public broadcast data is substantially different from permissioned data.

I fundamentally disagree with this. (I am not the only one.)

Private and public data are basically identical. A restaurant review is a restaurant review whether I am the only one who ever sees it or it gets shouted to the rooftops. A book review I share with my book club is identical to a book review I share with my wife or publish on my web site.

If I believe public and private data are fundamentally the same, just with different access rights, but the community believes they are and should be different… I am fighting the protocol, storage system, and community every step of the way. I’ve done this before, and it sucks.

I’ve not had the bandwidth to analyze possibilities of bridging the public and private data.

For Roomy I’ve looked at the perm data proposal, evaluated it as sufficient for Roomy’s case, and said that Roomy will mostly just selectively publish things meant explicitly for discovery to the public repo, and otherwise just use perm spaces anybody can get access to.

So at this point, for me so far, it goes into the “good enough, I think and we need to start making Roomy on top of it to make sure”.

But I do like the idea of streamlining the transfer between public and private data.

I feel like I agree with @dholms.xyz that private data is fundamentally different when it comes to the technical level of signatures and distribution, and I agree with Luke Kanies’s post that to a person this data is exactly the same.

It would be cool if there was a way to put less of a divide between them somehow.

3 Likes