Proposal: a community lexicon for standalone images

[ngerakines.me]

Checking in on a few things!

It looks like there is a good bit of interest and activity here, so I wanted to see who’s interested in taking the lead here to create an initial lexicon and start implementing use to demonstrate the need and satisfaction of the need.

Looking back at the draft in the first post, this is the shape I see:

{
  "lexicon": 1,
  "id": "community.lexicon.media.image",
  "defs": {
    "main": {
      "type": "record",
      "description": "A standalone image.",
      "key": "tid",
      "record": {
        "type": "object",
        "required": ["image", "dimensions", "createdAt"],
        "properties": {
          "image": {
            "type": "blob",
            "description": "The image content itself.",
            "accept": ["image/*"],
            "maxSize": 1000000
          },
          "dimensions": {
            "type": "ref",
            "ref": "#dimensions",
            "description": "The dimensions of the image."
          },
          "createdAt": {
            "type": "string",
            "format": "datetime",
            "description": "The date and time that the record was created."
          },
          "alt": {
            "type": "string",
            "description": "Text used to describe the image for screen readers and accessibility purposes.",
            "maxGraphemes": 1000,
            "maxLength": 10000
          }
        }
      }
    },
    "dimensions": {
      "type": "object",
      "description": "The pixel dimensions of the image. The aspect ratio may be derived from these values.",
      "required": ["width", "height"],
      "properties": {
        "width": {
          "type": "integer",
          "minimum": 1,
          "description": "The width of the image in pixels."
        },
        "height": {
          "type": "integer",
          "minimum": 1,
          "description": "The height of the image in pixels."
        }
      }
    }
  }
}

One change that I’d like to propose is that the image should represent dimensions because aspect ratio can be calculated from them.

[natespilman.at]

This lexicon proposal looks good to me, and I agree with the use of dimensions instead of aspect ratio.

Next steps - reach out to the publishers of the lexicons that @byarielm.fyi assembled and ask prettyplease to crosspost to this new lexicon?

What’s the governance around the creation of the community.lexicon.media.image lexicon in the first place?

Thanks!

[byarielm.fyi]

From what I saw in reviewing, your ‘dimensions’ is what ‘aspectRatio’ actually is under all the lexicons. Some have it defined in the same lexicon, others have it in another.

Probably easier to get but in if we use the same language even if it’s confusing lol

[essentialrandom.bsky.social]

Thank you @natespilman.at for starting this and @ngerakines.me for evolving the proposal!

The lexicon looks good, just a couple thoughts:

Alt Text as Array

As @al9000.com wisely said, alt text should be an array field with language information. This helps in multiple ways:

• Displaying the image across locales, of course
• Providing longer/shorter alt text according to where the image is displayed and what is supported by a specific platform. For example:
  • Images that are the focus of a page may get a longer description than e.g. images in a list of thumbnails
  • When cross-posting on different socials no one seems to be able to agree what length is appropriate and it should be easy for platforms to make different choices without needing to cut arbitrarily.

Taking Al’s alt text record as a base:

@al9000.com

"alt": {
    "type": "record",
    "record": {
      "required": [
        // Remove for now (see below)
        // "extendedText",
        // Language shouldn't be required. If unknown, it's better to
        // not force applications to guess.
        // "language",
        "text"
      ],
      "properties": {
        // I'd remove extendedText for now, although we could suggest this be added
        // as a namespaced field e.g. "iptc.extendedText" even without making it official.
        //
        // "extendedText": {
        //   "type": "string",
        //   "description": "A second text field to match the format of the IPTC metadata setup."
        // },
        "language": {
          "type": "string",
          "description": "The ISO 639 language code for the alt text. Unknown if not specified"
        },
        "text": {
          "type": "string",
          "description": "The text content of the alt text."
        }
      }
    }
  },

As Al mentions this array should be required but it can be empty. In the usual case of a single alt text field, this essentially boils down to the same content as a scalar vs a single item array.

dimensions vs aspectRatio

One “concern” about precise dimensions: can the PDS ever (for example) resize the image in the blob to save space? In that case, dimensions may end up out of sync, while aspect ratio will be always be accurate.

I do see the point of having width/height information without needing to download the blob, so my preference is:

• name: aspectRatio (never wrong)
• width/height descriptions should accurately describe what assumptions user should make. For example, if resizing is possible, then “the maximum height” and/or “the height of the image when saved”.

And, regarding pixel size: how does that work with SVGs or other vector-based image formats? I’m not sure what current best practices are, but if anyone knows do let us know, or I’ll try to research it later.

[essentialrandom.bsky.social]

Also, regarding additional information like copyright, this is a great use case for a sidecar lexicon. For example, platform-specific metadata may be done as (pseudo-coded):

Image record:

key: cool-image
$type: community.lexicon.media.image
data: 
  - the blob
  - the alt text
  - the aspect ratio

and then in the same PDS, the application-specific data can be:

key: cool-image
$type: cool.application.image.metadata
data: 
  - filters used
  - version of software
  - title/description

we could consider using this pattern for licensing information too, but we should first focus on getting the image lexicon itself adopted by the ecosystem.

[al9000.com]

@essentialrandom.bsky.social

One “concern” about precise dimensions: can the PDS ever (for example) resize the image in the blob to save space? In that case, dimensions may end up out of sync, while aspect ratio will be always be accurate.

If a PDS (or any other process) changes the size of the image it should update the pixel dimensions as well. I don’t know if there’s a MUST way to require that, but if the listed pixel dimensions don’t match those of the blob, the record is effectively invalid. Any app relying on the pixel info will be working off faulty data.

@essentialrandom.bsky.social

width/height descriptions should accurately describe what assumptions user should make. For example, if resizing is possible, then “the maximum height” and/or “the height of the image when saved”.

In the same vein, if a process changes the image size without updating the pixel dimensions the user can no longer safely use the values to make assumptions. (I might be missing something with the max height, but if an image is uploaded with a given height, but a process changes the blob, the value can no longer be trusted regardless.)

@essentialrandom.bsky.social

And, regarding pixel size: how does that work with SVGs or other vector-based image formats?

Ah, right. I was implicitly thinking about this as only raster based images. My instinct is to make that explicit and create a second lexicon for vector based art. There’s enough difference between the two of them that it feels weird to have them in the same spot. If they end up in the same lexicon the usage requirements go up non-trivially for end users in terms of having to handle both types (e.g. if I’m using it for an avatar on a website, the code I generate for the output has to change between raster/vector).

[byarielm.fyi]

Regarding size changes etc, some snippets from Blobs - AT Protocol :

It is effectively not supported to dynamically serve assets directly out of blob storage (the getBlob endpoint) directly to browsers and web applications. Applications must proxy blobs, files, and assets through an independent CDN, proxy, or other web service before serving to browsers and web agents, and such services are expected to implement security precautions.

and

Servers are strongly recommended against parsing media files (image, video, audio, or any other non-trivial formats) directly, without the use of strong sandboxing mechanisms. In particular, PDS instances themselves should not directly implement media resizing or transcoding.

Also, I believe changing dimensions would also change the CID and therefore the blob link.

[al9000.com]

I just made a post in the book lexicon thread about pondering if it makes sense to think about them like video file containers and payloads. The same question applies here. Instead of copy/pasting it (and having to change all the examples from books to images), I’m just gonna link it if you want to take a look:

In Which A Book Lexicon is Born

This literally came to me in a dream: Does it make sense to think of book lexicons like video file formats? For example, the .mkv format is “a container format that can hold an unlimited number of video, audio, picture, or subtitle tracks in one file.” The stuff can be of different types (e.g. encoded with different video and audio codecs). In a similar manner, would it make sense to have a top level lexicon that functions as a container that you put other stuff in? Like a minimal root level l…

[essentialrandom.bsky.social]

Hello folks!
At the last lexicon.community meeting I was (very gladly!) voluntold to help bring this lexicon to the finish line as the TSC representative.

We’ve had a lot of discussions here about possible bells and whistles to add! I know there’s a lot of interest for things like credits, copyright, tags, etc. (not to mention even more things to build on top, like galleries, comics, etc.).

That said, it’s generally best to have a simple base lexicon that can be used and reused in a lot of different applications, and then define “hook points” for further expansion. There also seems to be general consensus here over the base definition (multi-language alt text, dimensions, creation metadata, etc.).

Given this, here’s how I’m thinking of moving this forward:

• I’m going to open a PR with the simple definition we converged towards so we have a solid initial base to start from
• Once it’s open, I’ll post it here, open it up for comments, and publish it more widely to get everyone’s wisdom in
• Separately, given the interest, we can start thinking about other extensions we may want to push forward, and apps that may want to work with us to standardize some of what we’ll be doing. If there’s enough interest, that could turn into its own WG.
• @byarelm.fyi did a lot of great research on how image Lexicons are currently being used, so that seems a great starting point to let what’s out there lead development rather than building a priori

Do let me know if you want to be involved and have thoughts

[byarielm.fyi]

Happy to help

Also want to note that I think atproto self labels should be part of the base lexicon here; we should encourage self labeling.