Working group: service self-description

Lexicon Community
1

Working group: service describe

Objective: Define one XRPC query that any service in the ATmosphere can implement to say what it is: its roles, its capabilities, and the XRPC methods it supports. Use case: PDS implementations, feed services, permissioned data space hosts, AppViews, etc.

Expected deliverable: the community.lexicon.service.describe schema, validated example responses for the four service shapes above, and at least one reference implementation.

Timeline: iterate in this thread over the next few weeks, then a pull request to the lexicons repository once the design settles.

Looking for: people who run or build services (PDS, feed, AppView, relay, labeler, anything with an XRPC surface), and anyone with opinions about role and capability vocabularies.

Problem

AT Protocol applications and services have implied XRPC endpoints that are exposed, but as application updates occur and specs change, the version of those XRPC endpoints, their parameters, and their output can change without any notice.

Additionally, this becomes a real service-discovery problem when never-seen-before endpoints appear.

Design

The proposal is one unauthenticated query with no parameters that any service can implement. The community.lexicon.service.describe endpoint returns its roles, its capabilities, and its method surface. This grew out of conversations with Trezy and Nate, and it is better for them.

Solution

An XRPC query lexicon schema that returns an object with 2 fields. roles is an array of plain strings. methods is an array where each entry takes one of three shapes: an NSID, an AT-URI pointing at the lexicon schema record for a method, or a com.atproto.repo.strongRef that pins a schema record to an exact CID.

json

{
  "lexicon": 1,
  "id": "community.lexicon.service.describe",
  "defs": {
    "main": {
      "type": "query",
      "description": "Describes the roles, capabilities, and supported XRPC methods of a service endpoint. Allows any XRPC service to self-describe. Does not require authentication.",
      "output": {
        "encoding": "application/json",
        "schema": {
          "type": "object",
          "required": ["roles", "methods"],
          "properties": {
            "roles": {
              "type": "array",
              "description": "The role or roles this service fulfills in the network (e.g., 'pds', 'appview', 'relay', 'labeler', 'feed-generator').",
              "items": { "type": "string" }
            },
            "methods": {
              "type": "array",
              "description": "The XRPC methods this service supports, each referenced by NSID, by the AT-URI of its lexicon schema record, or by a strong (CID-pinned) reference to a lexicon schema record.",
              "items": {
                "type": "union",
                "refs": ["#nsid", "#atUri", "com.atproto.repo.strongRef"]
              }
            }
          }
        }
      }
    },
    "nsid": {
      "type": "object",
      "description": "A supported XRPC method, identified by its NSID.",
      "required": ["value"],
      "properties": {
        "value": {
          "type": "string",
          "format": "nsid",
          "description": "The NSID of the supported XRPC method."
        }
      }
    },
    "atUri": {
      "type": "object",
      "description": "A supported XRPC method, identified by the AT-URI of the lexicon schema record that defines it.",
      "required": ["value"],
      "properties": {
        "value": {
          "type": "string",
          "format": "at-uri",
          "description": "The AT-URI of the supported XRPC method's lexicon schema record."
        }
      }
    }
  }
}

The three reference forms answer different questions. An NSID is the compact form for well-known methods. An AT-URI says where the schema record lives, and because lexicon schema records use their NSID as the rkey, the NSID rides along inside the URI. A strongRef pins an exact revision: published lexicon records are mutable at their rkey, so the CID is the only way a service can say it implements this version, specifically. That last form matters most for methods that are not standardized anywhere yet, which is exactly the situation for newer service shapes.

Example request

There are no parameters and no authentication:

GET /xrpc/community.lexicon.service.describe
Host: pds.example.com

Or:

curl https://pds.example.com/xrpc/community.lexicon.service.describe

Example responses

A PDS

A PDS answers with its core surface plus, in this example, one nonstandard extension method. The extension is advertised with a strongRef because it is not standardized anywhere; the CID pins the exact schema revision the server implements.

{
  "roles": ["pds"],
  "methods": [
    { "$type": "community.lexicon.service.describe#nsid", "value": "community.lexicon.service.describe" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.server.describeServer" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.server.createAccount" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.server.createSession" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.identity.resolveHandle" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.repo.createRecord" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.repo.putRecord" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.repo.deleteRecord" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.repo.getRecord" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.repo.listRecords" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.repo.uploadBlob" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.sync.getRepo" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "com.atproto.sync.subscribeRepos" },
    {
      "$type": "com.atproto.repo.strongRef",
      "uri": "at://did:web:pds.example.com/com.atproto.lexicon.schema/com.example.pds.getAccountUsage",
      "cid": "bafyreidxl6igiec3hbdsrveibjq7wxzoupdvzzh2wgewibzfet5lii445q"
    }
  ]
}

A feed service

A feed generator has a small, well-known surface, so plain NSIDs cover it. An empty capabilities array is valid; not every service has something to declare there.

json

{
  "roles": ["feed-generator"],
  "methods": [
    { "$type": "community.lexicon.service.describe#nsid", "value": "community.lexicon.service.describe" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "app.bsky.feed.describeFeedGenerator" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "app.bsky.feed.getFeedSkeleton" }
  ]
}

The Smoke Signal AppView

Smoke Signal indexes community.lexicon.calendar.* records and serves its own query methods on top of them. The method NSIDs here are illustrative, but the shape is what an events AppView returns:

json

{
  "roles": ["appview"],
  "methods": [
    { "$type": "community.lexicon.service.describe#nsid", "value": "community.lexicon.service.describe" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "events.smokesignal.app.getEvent" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "events.smokesignal.app.listRsvps" },
    { "$type": "community.lexicon.service.describe#nsid", "value": "events.smokesignal.app.getActorEvents" }
  ]
}
4 Likes
2

This feels like draft-ietf-oauth-client-id-metadata-document-02 - OAuth Client ID Metadata Document but for ATProto; I’m into it. It could help with determining what aspects of an interface one should show on a per-PDS basis (for example, Blacksky’s community posts could be implemented using the same lexicon on a self-hosted PDS and then picked up by https://blacksky.community/ for support).

1 Like
3

Pinging @thisismissem.social then (CIMD co-author)

4

I strongly support more structured service discovery. It pairs with the self-descriptive nature of ATproto Lexicons, which is one of the protocol’s major strengths. IMO the ability to introspect services is fundamental to the scalability of the ecosystem, and, as @omg.jacky.wtf says…

…which is important for smaller, independent websites (and something I keep asking for whenever I get the chance to).

I have some thoughts on the design, but regarding this:

I didn’t see the concept of asking a generic service for its “endpoints” or declaring them in the CIMD spec, so let me know if I missed anything.

Overall, this seems complementary to the DID document surfacing the preferred/available services for each DID, and would have been helpful to have when I was figuring out how labelers worked.

Some feedback

On Capabilities

I didn’t see capabilities in the Lexicon itself, just roles and references to XRPC methods. Am I missing something?

On Versioning

IIRC when a record gets replaced, the one with the old CID is not available in the repo anymore (or, at least, not easily fetched), which means all a StrongRef really tells me is whether a services is implementing the latest version of the lexicon/the same version as “me”.

I know there may not be good alternatives right now* for API versioning, but I’d like to make sure any eventual spec is clear on expected usage and limits (or at least, suggest solutions).

(*The spec does mention “an optional in-repo mechanism for storing multiple versions of the same record may be implemented” so maybe it’s time to push for that? :P)

On Roles

Is "appview" supposed to be a fallback mode for anything that doesn’t fit in another protocol role? I went back to check the definition of AppView, and it’s a looser definition than I’d like for a field like this, and I know for a fact the term is confusing to people.

In general, I think the “roles” field should be more tightly specified. We should at least consider reusing the exact same names as the services in the DID documents, like atproto_pds and bsky_fg. A generic “feed-generator” role doesn’t state almost anything about the service that you cannot infer from the provided endpoints.

We could consider using an NSID to scope the role. For example, if we did want to give a precise definition of feed, we could do something like

{
  "roles": [
     "community.lexicon.service.describe#feed-generator",
     // Ideally we'd "lovingly bul—" I mean, *persuade* Bluesky 
     // to publish this somewhere where it can be described/referenced
     // e.g. app.bsky.feed.defs#bsky_fg
     "bsky_fg"
  ],
  // these are a superset of what both the bsky_fg and our 
  // feed-generator roles provide
  "methods": [
      { "$type": "community.lexicon.service.describe#nsid", "value": "community.lexicon.service.describe" },
      { "$type": "community.lexicon.service.describe#nsid", "value": "app.bsky.feed.describeFeedGenerator" },
      { "$type": "community.lexicon.service.describe#nsid", "value": "app.bsky.feed.getFeedSkeleton" }
    ]
}

Regardless, of how we specify them, there should be some agreement about what each role means/implies, and how the information is supposed to be used. I’d rather we skip the field and rely just on the methods returned, than we under-specify something like this, so we get a “gold rush” to claim cool names that everyone uses a different way.

(Relatedly, I’ve been trying to find where Bluesky defined/maintains the canonical spec of a “#bsky_fg” service, and I can’t find anything authoritative. So this may be the right time to also sketch the opposite direction, or intentionally work around it.)

3 Likes
5

This is good. You’re right that the self-description query needs more definition than I sketched out.

My thinking was that a single XRPC endpoint that returns declared service capabilities would be enough to start, but I’m open to expanding it if there are concrete use cases that need more structure. I left the idea of role, capabilities, behavior, and version intentionally vague. I’m not sure what the best way to represent all of that is.

6

Any issue you see with starting with just the “methods” array? I think that gets us quite far and it’s easier to tackle on its own.

I’m curious about capabilities vs methods too, because I’m not sure I get the difference.

1 Like
7

I don’t have any issue with that at all.

8

I’ve been using and integrating community.lexicon.service.describe in production with https://bulleted.app/ and the feedback has been useful. With @essentialrandom.bsky.social’s feedback, here’s the updated lexicon schema. I’m good with adding more fields once there’s a clearer shared vocabulary for them.

{
  "lexicon": 1,
  "id": "community.lexicon.service.describe",
  "defs": {
    "main": {
      "type": "query",
      "description": "Describes the XRPC methods supported by a service endpoint. Allows any XRPC service to self-describe. Does not require authentication.",
      "output": {
        "encoding": "application/json",
        "schema": {
          "type": "object",
          "required": ["methods"],
          "properties": {
            "methods": {
              "type": "array",
              "description": "The XRPC methods this service supports, each referenced by NSID.",
              "items": {
                "type": "union",
                "refs": ["#nsid"]
              }
            }
          }
        }
      }
    },
    "nsid": {
      "type": "object",
      "required": ["value"],
      "properties": {
        "value": {
          "type": "string",
          "format": "nsid",
          "description": "The NSID of the supported XRPC method."
        }
      }
    }
  }
}

I dropped the AT-URI and strongRef reference forms too, although the union is open. I like the idea that a server could “pin” to a specific lexicon schema version (via CID). Although it doesn’t advertise it, this would still be allowed by a service putting a strongRef in there anyway.

The endpoints are live now if you want to poke at them:

These implementations have been enabling cross-PDS permissioned data reads in production, with public announcements from Bulleted.app showing interop across zds, pds.js, and rsky whom all provide this endpoint for discovery:

(Ignore the “roles” field in their output.)

My next step is to open a pull request to the lexicons repository with this schema and open it up for public discussion. I’d love direct feedback from the implementers who’ve been running this, as well as from potential future adopters like Tranquil and other AppView maintainers who might want to advertise their custom method surfaces.

2 Likes
9

I’ve created a pull-request for this lexicon schema with the goal of directing feedback and comments to this discussion topic.

1 Like