[Proposal] A lexicon for user-defined collections

[chrisshank.com]

Hi! I’ve put together a proposal for a set of lexicons around user-defined, heterogeneous collections that includes an explainer and a first pass at proposed schemas. This is mostly to seed a conversation, nothing feels that solidified yet.

[tyler.fun]

Thanks for writing this up! Sharing a few notes and questions here:

1. I’d like to recommend including a concept of intent: is the collection for music, web bookmarks, research, etc? Is provenance meant to help here?
2. List ordering is probably a thing that needs standardization on its own. Keeping a second list of each item id’s order feels inefficient and subject to drift (a client adds an item in the middle of a collection and doesn’t update each other items’ order).

[disnetdev.com]

Really cool! Been also thinking about collections for Skyreader since we currently interoperate with the collection types of semble and margin.

Few initial thoughts on the proposal:

item.collection is a strongRef

A strongRef pins the collection’s CID, so renaming a collection or editing its description changes the CID that every existing item points at. Either every item goes stale, or readers ignore the CID, in which case it’s just a cost.

Maybe a plain at-uri, or say explicitly that the CID is advisory?

Ordering via a list record

Like Tyler mentioned using a list for ordering might be a problem.

• every reorder rewrites the whole record
• two apps editing at once clobber each other unless both use swapRecord
• the strongRefs go stale whenever an item is edited;
• an item another app adds isn’t in the list, and there’s no rule for where it goes.

What about an optional fractional-index position string on item (lexicographic, ties broken by at-uri)?

Set or bag?

The collection is described as “an unordered collection of URIs”, but nothing stops two items with the same uri, from one repo or several. Should there be an implied or stated handling of dups? I’d suggest a collection is a set by uri, and the earliest item wins.

[chrisshank.com]

@tyler.fun

I’d like to recommend including a concept of intent: is the collection for music, web bookmarks, research, etc? Is provenance meant to help here?

ya this is a good point, I thought about collections having a tags field to indicate this, not sure we can prescribe what those tags are in this proposal though?

@tyler.fun

List ordering is probably a thing that needs standardization on its own. Keeping a second list of each item id’s order feels inefficient and subject to drift (a client adds an item in the middle of a collection and doesn’t update each other items’ order).

ya this is the least polished part of the proposal, totally something that could be punted but I think it’s worth considering how it would all works together. Another option is for there to be an order field on items, that has it’s own downsides. Curious other people’s ideas on this!

[chrisshank.com]

thanks @disnetdev.com! I’m really curious to hear more about how you’d use a lexicon like this for SkyReader!

@disnetdev.com

Maybe a plain at-uri, or say explicitly that the CID is advisory?

good point, updated!

@disnetdev.com

What about an optional fractional-index position string on item (lexicographic, ties broken by at-uri)?

ya this is definitely the other decent option. A couple of tradeoffs for us to think about closer:

• multiple items can have the same position, how are indexes expected to sort in that case?
• an insertion could cause a propagation of record updates?
• for the use case of pinned items you would have to load all of items to know which ones are pinned.

@disnetdev.com

Set or bag?

The collection is described as “an unordered collection of URIs”, but nothing stops two items with the same uri, from one repo or several. Should there be an implied or stated handling of dups? I’d suggest a collection is a set by uri, and the earliest item wins.

good point, gonna think on this more! Currently the proposal leaves this up to the index to answer, since there could be cases where duplicates are desirable. But I wonder if the collection should declare that intent.

[byjp.me]

This is very exciting!

You may already have spotted this (apologies if so!), but I think you may have an issue with your collection lexicon as the article suggests it, as atproto doesn’t support heterogeneous lists without $type descriminators (for the benefit of statically typed languages), so I think:

  schemes": [
    "https",
    "doi",
    {
      "type": "at", 
      "lexicons": [
        "community.lexicon.collection", 
        "app.bsky.feed.post", 
        "site.standard.document", 
        "community.lexicon.calendar.event"
      ]
   }

Would need to be something like:

  schemes": [
    { "$type": "community.lexicon.collections.collection#schemeExternal", "kind": "https" },
    { "$type": "community.lexicon.collections.collection#schemeExternal", "kind": "doi" },
    {
      "$type": "community.lexicon.collections.collection#schemeAt", 
      "lexicons": [
        "community.lexicon.collections.collection", 
        "app.bsky.feed.post", 
        "site.standard.document", 
        "community.lexicon.calendar.event"
      ]
   }

You were likely just using shorthand though, so ignore me if this isn’t news to you!

[chrisshank.com]

@byjp.me

You may already have spotted this (apologies if so!), but I think you may have an issue with your collection lexicon as the article suggests it, as atproto doesn’t support heterogeneous lists without $type descriminators (for the benefit of statically typed languages),

Oh interesting the docs were ambiguous about this, but ya might need to polish how the scheme type works

In theory arrays have homogeneous types (meaning every element as the same type). However, with union types this restriction is meaningless, so implementations can not assume that all the elements have the same type.

(source)

[byjp.me]

Yeah — I think that’s a guide to defensive coding around reading records that could have been written without a $type, as there’s no way to prevent a PDS from storing them, but it’d be great if it was clearer

[byarielm.fyi]

I’ll wait to see the tangled repo for more thoughts, but regarding “interested in hearing about more use cases” one that comes up is being able to make collections/lists of atproto apps that reference the community app lexicon. This would be something both the Aster and Eurosky Portal would make use of since right now we’re just hard-coding apps to categories. The ability to add metadata like ‘Made in Netherlands’ or ‘Science’ is something I can see being of interest.

[disnetdev.com]

@chrisshank.com

I’m really curious to hear more about how you’d use a lexicon like this for SkyReader!

Currently we have collections to manage your saved articles, which can either be private (just stored on the Skyreader servers) or public using either the semble or margin lexicons. That’s communicated to the user as “store/link on semble/margin” (with ui pickers for which service to use) but a common collection lexicon could clean all that up (and skyreader’s semble/margin lexicon adapters) and tell a nicer story.

I’m also experimenting with “reading rooms”, shared collections of articles to read together with co-presence affordances, that might fit as well. For rooms I need some notion of membership, not sure if that belongs on the lexicon (accessType plus collaborators like what semble does?) or atproto space / atmospheric group.

@chrisshank.com

A couple of tradeoffs for us to think about closer:

• multiple items can have the same position, how are indexes expected to sort in that case?
• an insertion could cause a propagation of record updates?
• for the use case of pinned items you would have to load all of items to know which ones are pinned.

• should only only really happen when two clients insert between the same neighbors at once so could just specify position + createdAt + at-uri to break ties or similar?
• fractional indexing avoids update propagation right? that’s what makes it so cool, you only ever have to update the item you insert
• true, maybe have both the optional fraction-index position field for whole-collection ordering and list record for subsets like pinned?

[chrisshank.com]

I think I’ve incorporated almost everyone’s feedback/suggestions:

• added a tags field to collections
• remove strongRefs
• removed the list lexicon in favor of fractional indexes
  • this does mean that collections have one ordering across all apps
  • fixed the scheme type

Still not sure about @disnetdev.com’s point about deduplicating, currently leaning towards apps interpreting at their own discretion.

@byarielm.fyi

I’ll wait to see the tangled repo for more thoughts, but regarding “interested in hearing about more use cases” one that comes up is being able to make collections/lists of atproto apps that reference the community app lexicon. This would be something both the Aster and Eurosky Portal would make use of since right now we’re just hard-coding apps to categories. The ability to add metadata like ‘Made in Netherlands’ or ‘Science’ is something I can see being of interest.

ah yes, these are tuse cases I was imagining as well!

@disnetdev.com

Currently we have collections to manage your saved articles, which can either be private (just stored on the Skyreader servers) or public using either the semble or margin lexicons. That’s communicated to the user as “store/link on semble/margin” (with ui pickers for which service to use) but a common collection lexicon could clean all that up (and skyreader’s semble/margin lexicon adapters) and tell a nicer story.

I’m also experimenting with “reading rooms”, shared collections of articles to read together with co-presence affordances, that might fit as well. For rooms I need some notion of membership, not sure if that belongs on the lexicon (accessType plus collaborators like what semble does?) or atproto space / atmospheric group.

ah super interesting, particularly reading rooms. Collaboratively build a list and co-presence emerges from folks providing itemData for each item. I’ve been thinking a collaborators field too, this is a good example. Even in a atproto space, I can see it being useful to pick a subset of people to collaborate on certain collections.