A general purpose ratings lexicon

I took a stab at solving ratings for everyone forever. I’d love your thoughts.

4 Likes

This seems solid; it’s definitely powerful enough to support most rating systems.

However, I’m working with rating systems in games, like Chess’ Elo ratings. It’s arguably a distinct space, but this lexicon is VERY close to encompassing it already.

However, the rating subject union only includes URIs and strongRefs, but not DIDs. While my rating service could point at some profile record, the semantics of rating the player directly seems cleaner.

Additionally, we might want to know a given player’s rating over time. A service might want to dump a set of historical ratings on the protocol. My thought is that we need a field to allow identities to express WHEN the rating was given in cases where that doesn’t match createdAt. Perhaps “ratedAt”? I’d think we’d want it optional, since createdAt can stand in for it in the more common case where those values should match.

Thanks! I admit there is some ambiguity because words collide in meaning. Something like ELO rating would fall under an aggregation or analysis, rather than an individual instance of rating something. ELO is computed on your wins and losses. Writing a record would be like saying “I say your elo is 4”, which doesn’t mean much. The rating record is a statement of opinion in response to a prompt and subject. ELO and other stats are business logic.

On the note of subject including DIDs, good point! I was hesitant to include it at first due to the ickyness of rating an account as a whole (the facebook flashbacks :face_with_peeking_eye:) but I agree it does have utility.

For backdating, I believe you can edit the createdAt to indicate when the record is from. A lot of social archive importers do this (hence some atproto accounts have records from before 2022). I brought in my old Instagram posts and they are dated accurately.

1 Like

Indeed, Elo is computed on wins/losses. But by whom? With what inputs? People and organizations might differ on disqualifications and game results.

Modern chess ratings are the FIDE org answering the prompt “how good is {subject}” at {blitz|rapid|classical} chess" by publishing two new ratings after every game. The Elo system itself maps nicely to a scale. It’s a rating, and your proposal solves ratings forever (as long as it allows DID subjects).

But… yeah. Systems where people rate others could easily be creepy or harmful. Let’s discourage and criticize such things but not limit our language by excluding the category. The original Facebook should be socially unacceptable, not technically impossible.

And I see, createdAt can serve. We would be dropping the import time info, but I suppose it doesn’t matter.

There being multiple potential answers to ‘computed by whom on what?’ is exactly the point. putting the raw data on protocol allows for a marketplace of services on top of that information. Trust signals and service quality are differentiators. If you wanted to document an ELO at a particular time I don’t see why the current lex set would prevent that. If you wanted to also offer some proof of what went into that calculation, I think that would be left to a sidecar record.

I’ll add DID as an option for the subject.

1 Like

Agreed. I think this lexicon fits well for recording the results of such rating calculations. This would naturally be in addition to publishing the underlying game records the ratings are computed from. The raw data belongs on the protocol, as do the outputs of the marketplace of services built atop it.

Thanks, I’ll look forward to publishing my game’s ratings this way!

PS: just put some thought into the sidecar pattern here: it’s good, with one caveat. If you model a parent-child, N-1 relationship, make sure to prevent the invalid N-N cases structurally. Logic that prevents a second sidecar from being written can work, but defining the sidecar rkey as the child record’s tid or URI makes the invalid states unrepresentable.

Hey folks! Since this has reached some consensus, I’d like to find a wg co-chair so that we can move this into an official proposal pr on lexicon garden. Lots of clear application spaces for this lexicon set so anyone interested is welcome, especially if you make an app that uses ratings/reviews.

I volunteer as tribute :raised_back_of_hand:. (will be atprotofying my game beyond login soon, I swear :sweat_smile:).

What does being a wg co-chair involve?

1 Like

afaik there just have to be at least two names under a submission for lexicon community to accept a PR. I knight you co-chair of the community ratings proposal!

1 Like

Ok taking in some feedback, I’ve made a couple adjustments.

  • the subject of a rating can now be a DID
  • community.lexicon.rating.prompt now requires a strongRef to a rating scale. While one rating scale can be used with many different prompts, a prompt could change meaning if used with different scales, so it makes sense to pin a prompt to one particular scale.
  • scale has been removed as a field from community.lexicon.rating.rate since the prompt now carries the link to a scale.

With those changes, here are the latest lexicon definitions

community.lexicon.rating.rate
{
  "lexicon": 1,
  "id": "community.lexicon.rating.rate",
  "defs": {
    "main": {
      "type": "record",
      "description": "A rating of a subject in response to a prompt, answered on the scale the prompt is bound to.",
      "key": "tid",
      "record": {
        "type": "object",
        "required": [
          "subject",
          "prompt",
          "value",
          "createdAt"
        ],
        "properties": {
          "subject": {
            "type": "union",
            "description": "The thing being rated: an ATProto record (via strongRef), an ATProto identity (via DID), or an external URI (website, document, etc).",
            "refs": [
              "com.atproto.repo.strongRef",
              "#subjectDid",
              "#subjectUri"
            ]
          },
          "prompt": {
            "type": "ref",
            "ref": "com.atproto.repo.strongRef",
            "description": "Pointer to a community.lexicon.rating.prompt record. The prompt's scale defines the valid values for this rating."
          },
          "value": {
            "type": "union",
            "description": "The chosen value, typed to match the code of the scale referenced by the prompt.",
            "refs": [
              "community.lexicon.rating.scale#nominalValue",
              "community.lexicon.rating.scale#ordinalValue",
              "community.lexicon.rating.scale#intervalValue",
              "community.lexicon.rating.scale#ratioValue"
            ]
          },
          "createdAt": {
            "type": "string",
            "format": "datetime"
          }
        }
      }
    },
    "subjectDid": {
      "type": "object",
      "description": "An ATProto identity as a whole (account, labeler, feed generator service, etc.), not any particular public or permissioned repo belonging to it.",
      "required": [
        "did"
      ],
      "properties": {
        "did": {
          "type": "string",
          "format": "did",
          "description": "The DID of the identity being rated."
        }
      }
    },
    "subjectUri": {
      "type": "object",
      "required": [
        "uri"
      ],
      "properties": {
        "uri": {
          "type": "string",
          "format": "uri",
          "maxLength": 2048,
          "description": "A URI pointing to an external resource (website, document, etc)."
        }
      }
    }
  }
}
community.lexicon.rating.prompt
{
  "lexicon": 1,
  "id": "community.lexicon.rating.prompt",
  "defs": {
    "main": {
      "type": "record",
      "description": "A prompt or question that a rating answers, clarifying what aspect of a subject is being evaluated.",
      "key": "tid",
      "record": {
        "type": "object",
        "required": [
          "text",
          "scale",
          "createdAt"
        ],
        "properties": {
          "text": {
            "type": "string",
            "maxLength": 1024,
            "description": "The prompt or question text."
          },
          "scale": {
            "type": "ref",
            "ref": "com.atproto.repo.strongRef",
            "description": "Pointer to the community.lexicon.rating.scale record this prompt is answered on. A prompt is only meaningful with a fixed set of possible answers, so every prompt is bound to exactly one scale."
          },
          "langs": {
            "type": "array",
            "items": {
              "type": "string",
              "format": "language"
            },
            "maxItems": 3,
            "description": "BCP-47 tag(s) the prompt text is written in."
          },
          "createdAt": {
            "type": "string",
            "format": "datetime"
          }
        }
      }
    }
  }
}
community.lexicon.rating.scale
{
  "lexicon": 1,
  "id": "community.lexicon.rating.scale",
  "defs": {
    "main": {
      "type": "record",
      "description": "Defines what makes a valid rating value for a scale: the value's shape and measurement level.",
      "key": "tid",
      "record": {
        "type": "object",
        "required": ["name", "code", "createdAt"],
        "properties": {
          "name": { "type": "string", "maxLength": 256 },
          "description": { "type": "string", "maxLength": 1024 },
          "documentation": { "type": "string", "format": "uri", "maxLength": 1024, "description": "Optional URI linking to further documentation about this scale (e.g. methodology, citations)." },
          "langs": {
            "type": "array",
            "items": { "type": "string", "format": "language" },
            "maxItems": 3,
            "description": "BCP-47 tag(s) the name/description/option labels in this scale are written in."
          },
          "code": {
            "type": "union",
            "refs": ["#nominal", "#ordinal", "#interval", "#ratio"]
          },
          "createdAt": { "type": "string", "format": "datetime" }
        }
      }
    },
    "nominal": {
      "type": "object",
      "required": ["options"],
      "properties": {
        "options": { "type": "array", "maxItems": 100, "items": { "type": "ref", "ref": "#nominalOption" } }
      }
    },
    "nominalOption": {
      "type": "object",
      "required": ["label", "value"],
      "properties": {
        "label": { "type": "string", "maxLength": 128 },
        "value": { "type": "string", "maxLength": 64 },
        "description": { "type": "string", "maxLength": 256 },
        "labelImage": { "type": "blob", "accept": ["image/*"], "maxSize": 1000000, "description": "Optional image for this option." }
      }
    },
    "nominalValue": {
      "type": "object",
      "required": ["value"],
      "properties": { "value": { "type": "string", "maxLength": 64 } }
    },
    "ordinal": {
      "type": "object",
      "required": ["options"],
      "properties": {
        "options": { "type": "array", "maxItems": 100, "items": { "type": "ref", "ref": "#ordinalOption" } }
      }
    },
    "ordinalOption": {
      "type": "object",
      "required": ["label", "value"],
      "properties": {
        "label": { "type": "string", "maxLength": 128 },
        "value": { "type": "integer", "description": "Rank. Leave gaps for future insertions." },
        "description": { "type": "string", "maxLength": 256 },
        "labelImage": { "type": "blob", "accept": ["image/*"], "maxSize": 1000000, "description": "Optional image for this option." }
      }
    },
    "ordinalValue": {
      "type": "object",
      "required": ["value"],
      "properties": { "value": { "type": "integer" } }
    },
    "interval": {
      "type": "object",
      "required": ["min", "max", "unitsPerWhole"],
      "properties": {
        "min": { "type": "integer" },
        "max": { "type": "integer" },
        "unitsPerWhole": { "type": "integer", "minimum": 1, "description": "Divide the stored integer by this to get the value in `unit`s. Any positive integer, not just a power of 10. Choose min/max directly in the smallest (or coarsest) legal increment; every integer between min and max is then a valid value, by construction." },
        "unit": { "type": "string", "maxLength": 64 },
        "minLabel": { "type": "string", "maxLength": 128, "description": "Optional label for the minimum value." },
        "maxLabel": { "type": "string", "maxLength": 128, "description": "Optional label for the maximum value." },
        "minLabelImage": { "type": "blob", "accept": ["image/*"], "maxSize": 1000000, "description": "Optional image for the minimum endpoint." },
        "maxLabelImage": { "type": "blob", "accept": ["image/*"], "maxSize": 1000000, "description": "Optional image for the maximum endpoint." }
      }
    },
    "intervalValue": {
      "type": "object",
      "required": ["value"],
      "properties": { "value": { "type": "integer" } }
    },
    "ratio": {
      "type": "object",
      "required": ["min", "max", "unitsPerWhole"],
      "properties": {
        "min": { "type": "integer" },
        "max": { "type": "integer" },
        "unitsPerWhole": { "type": "integer", "minimum": 1, "description": "Divide the stored integer by this to get the value in `unit`s. Choose min/max directly in the smallest (or coarsest) legal increment so that every integer between min and max is then a valid value." },
        "unit": { "type": "string", "maxLength": 64 },
        "minLabel": { "type": "string", "maxLength": 128, "description": "Optional label for the minimum value (true absence)." },
        "maxLabel": { "type": "string", "maxLength": 128, "description": "Optional label for the maximum value." },
        "minLabelImage": { "type": "blob", "accept": ["image/*"], "maxSize": 1000000, "description": "Optional image for the minimum endpoint." },
        "maxLabelImage": { "type": "blob", "accept": ["image/*"], "maxSize": 1000000, "description": "Optional image for the maximum endpoint." }
      }
    },
    "ratioValue": {
      "type": "object",
      "required": ["value"],
      "properties": { "value": { "type": "integer" } }
    }
  }
}

Updated live lexicons for testing are available at garden.lexicon.avid-antelope-359377 - Browse - Lexicon Garden

3 Likes

This could definitely work. Not being able to express ratings on multiple scales for the same prompt is a minor loss, but those can just be separate prompts if I want to calculate both Elo and Glicko2 for the same set of games. The minor clarity increase and the savings on storage make it a reasonable choice.

We should also probably note somewhere that ratings should be interpreted only via the scale on the related prompt, and that any rating that’s not valid for its scale should be rejected.

good call. are you thinking in the lexicon itself or in the docs on lexicon.community?

The docs at a minimum, but we could also change

“The chosen value, typed to match the code of the scale referenced by the prompt.”

to

“The chosen value. The type and value must be valid for the scale referenced by the prompt.”

The docs could say something like “Ratings are only valid if the value’s type matches the prompt’s scale and value is with in allowed range. Clients should only write valid ratings and consumers should reject any invalid ratings.”.

1 Like

This covers a lot of what I’d want for rating books tbh! I’ll begin folding this into Livtet