In Which A Book Lexicon is Born

Lexicon Community WG Books
, ,
1

Hi everyone,

We recently held a sync to discuss creating a shared book lexicon for the Atmosphere Protocol. I’ve shared the full recording (thanks Ronen!) and notes from the meeting here: Fathom with some early notes here: Notion (also do we have a ATProto version of Notion yet?)

The book logging social scene is highly fragmented. Dozens of Goodreads alternatives are competing but not interoperating, leading to a lack of shared network effects and ultimately, app failures. This thread is meant to gather community feedback, surface adjacent integrations, and recruit members for a future formal working group (ONE OF US! ONE OF US!).

Holding “Data Unions” and “IndieSky” in Tandem

A core concept from our meeting was the idea of building a “Data Union”—a model where apps share data interoperability to prevent user lock-in and build a shared network effect.

As we think about this, it’s important to hold the concept of a Data Union in tandem with the ongoing infrastructure work of IndieSky. IndieSky is doing the heavy lifting of providing sovereign, user-controlled data rails; the Data Union concept could (maybe?) build on top of it. IndieSky provides the underlying infrastructure (the “how” and “where” of sovereign data), while a shared Book Lexicon provides the cooperative application layer (the “what” and “with whom”).

The question for the community: How do we ensure that a Data Union built on Atmosphere leverages IndieSky’s infrastructure to maximize user sovereignty while still providing the shared network effects apps need to survive?

The Proposed Technical Strategy: Copy-over-Reference

To ensure data resilience, the proposed MVP relies on a “copy-over-reference” model. When a user logs a book, their PDS gets:

  1. A strong reference to the main book record.

  2. A minimal metadata copy (title, author, identifiers) as a fallback.

This ensures user data remains functional even if the source book record disappears.

MVP Focus: We want to prioritize the lexicon for core social interactions first:

  • Read status (reading, read, to-read)

  • Reviews (rating, text)

The complex book indexing lexicon (handling edge cases like partial books, compilations, and manga) will be developed in parallel. For indexing, we discussed a “lazy initialization” model—creating book records on-protocol only when a user interacts with them—avoiding a massive, costly backfill. Deduplication remains a key challenge here, specifically regarding fallback keys and hash collision risks.

Spotlight on Adjacent Builders (Bookhive, Semble, Hardcover)

If you are currently building apps adjacent to this space—like Bookhive, Semble, or Hardcover—we want to hear from you!

  • How are you currently handling book metadata and user interactions?

  • Does the “copy-over-reference” model align with your app’s architecture?

  • Where do you see friction in adopting a shared lexicon?

Please drop a link to your project, highlight your current integrations, and let us know how a shared lexicon could benefit your ecosystem.

A Prototype

I also took some time this weekend to work on a prototypical AppView (think of https://public.api.bsky.app/ but for books) that lives up to some of these ideas. It’s available at https://biblio.livtet.olamaelcu.net/ with the source on Tangled and on GitHub at GitHub - olamaelcu/bibliograph: An AppView for interfacing with bibliographic information on ATProto · GitHub. I’ve primed it with a lot of books from Toni Morrison and Arthur C. Clarke but it should auto-import new books if they’re discoverable by Google Books (please be kind to my API key!) or OpenLibrary. Let me know what you think - happy to host it for the community!

8 Likes
2

Cross posting this to Hardcover’s roadmap discussion

1 Like
3

Do y’all have a plan for the format to use for the review text?

Text content storage in general feels like a potential, unintentional app lock-in vector across the atmosphere. At least, if the text is anything beyond paragraphs or maybe markdown.

I spent the last few years designing a markdown replacement format. The original goal was to make it easier to make website content without having to jump all the way to HTML. I think the same features that make it work there have the potential to act as an interoperability layer for text across the atmosphere too. I’m happy to share a link if you’re interested in taking a look.

4

I’m very interested to see what you got! I’m personally open to having some sort of content type hint and then letting clients support what they can (with an optional plain text value so clients that don’t support anything can still see/present something to end users).

I’m also of the mind of having reviews be referential, so it can point to a Standard Site publication or a Bluesky post (for example) so clients can more deeply represent information.

2 Likes
5

Yeah, I’m right there with you on letting the clients control what they see (and being able to fall back to plain-text). My design is basically a replacement for markdown with extra horsepower thrown in. I’m making a lexicon for it, but the idea is that it works primarily as a portable, stand-alone format. You can check out the intro page here: https://neodoc.style/

(I’m also writing a post to start a thread to discuss it. I should have that posted sometime tonight.)

1 Like
Bibliograph: A Prototypical AppView for Books
6

I’m liking what I’m seeing! It doesn’t look immediately backward compatible with Markdown so do you see folks using something like contentType: “text/x-neodoc somewhere in their records?

7

Hi there,

I talked a bit about BookHive is doing this in my atmosphereconf talk & write up: The Design Philosophy of BookHive - Nick The Sick - pckt

But, to summarize, BookHive is principled on data-sovereignty, so this means:

So, anyone can bootstrap bookhive.buzz from scratch & run the service on their own with the OSS code at: GitHub - nperez0111/bookhive: Track your books, share your shelves, see what others are reading · GitHub

This copy-over-reference idea is almost 1:1 what I’ve already implemented in BookHive, so I’m obviously in support of it.

I think that the real problem lies in multiple applications having different datasets for understanding what a book is, so making a review on one platform would simply be missed on another platform, since there is not a great way to understand that we are actually talking about the same book. Most platforms are using centralized databases where they can unilaterally make these moderation decisions, but in an open network like atproto I don’t think we really can have the same approach. I think there needs to be a sort of community linking & voting & submission system for applications to interoperate with one another.

Would be happy to be part of the working group, life has just been busy lately so I didn’t get around to this until now.

5 Likes
8

Thanks! And yeah, the overall format can’t be transformed directly with a Markdown parser. I played around with the idea early on but quick realized it would have added a bunch of complexity to the both the parsing and the mental model about the format when authoring content with it. So, it’s explicitly not directly compatible with Markdown.

But, I’m adding a -- markdown section type that’ll provide Markdown parsing inside a NeoDoc. If you’ve got a bunch of markdown already, the conversion process would be to just throw the -- markdown section header in front of it and it’ll do the thing.

You’ve also got me thinking about YAML. I know lots of folks use it for front matter, but I wasn’t originally planning on adding support. That decision was based off the fact that the reference parser is written and rust and the main yaml parsing crate is no longer maintained. But, I just checked and it still gets hundreds of thousands of downloads every 90 days. So I think it’s more that it’s basically feature complete. The only potential issue would be bugs/security problems, but it could always be forked if that happens. Probably best to add it to make it easier for folks to migrate old docs.

I hadn’t thought about the contentType. That sounds like a good idea, but I don’t know enough about how everything works to know what to do with it (e.g. if I’ve got a lexicon I’d expect that’s not required? but maybe if other folks want to use the parser inside their stuff they need it? I’ll have to look into that more).

1 Like
9

Loving the conversation here (and in person and in the email thread!). I’ve been staying up to date and trying to wrap my head around it all. I’ve been working on this problem for the last 5 years on the Hardcover side, but this is the first anything I’ve done using Atmosphere and PDSs and that side of things, so please feel free to correct anything I say that feels off, or lacks understanding of that side of this.

I wanted to break down a few concepts that I’ve seen touched on, with some thoughts on how we’re solving them today, and touching on some ideas on how to solve them long-term in a decentralized way.

Books/Works vs Editions

I think this is one of the biggest concepts that if we can get right, it powers a ton of things across different sites (ex: View the Spanish version? / View the Audiobook version? / Is there a DRM free ebook for this book? Which audiobook edition has the highest rating? Which has a narrator that I like the most?).

To go back to @nickthesick ‘s mention of including all the useful data needed, the Edition has some of the relevant useful data to the user reading that edition. Without edition information you could say “I’m 50% of the way done with the book”. With the edition information you could say “I’m on page 256 out of 512 on the Hardcover edition of Book”, or “I’m 12 hours and 42 seconds out 20 hours and 5 minutes of the audiobook edition”. Also, different editions have different cover images, and we’ve found users really want to see the cover of the edition they have.

In order to see stats like listening time or pages read, edition information is required, and progress updates need to be linked to the edition. It can be an annoying step, both for data and for the user, but for full accuracy it’s needed.

All of that data lives in the relationship between a Book and an Edition. OpenLibrary calls these Works, which I think is a more universal term for an abstract “representation of a book”.

Important things to note about a Work/Book vs an Edition:

  • An Edition AND a Book/Work can have identifiers back to other platforms. On Hardcover, we have edition_id and book_id. OpenLibrary uses keys like /works/OL82563W (work/book) and /books/OL1439870M (Edition).
  • Choosing a title for the “Book/Work” gets more difficult the broader it is (ex: different languages, or even HP and the [Sorcerers|Philosophers] Stone).
  • Not all Editions have an identifier. Some editions don’t have ISBNs or ASINs (I’m looking at you Manga :face_with_peeking_eye:.)
  • Series data is best mapped to the Book/Work, not the edition. That way * The Hobbit is considered Book 0 of The Lord of the Rings, rather than all 519+ editions needing that data.

I think the takeaway here is that an ISBN identifies an edition, not a work. So if ISBN is the dedup key, community.lexicon.book.book is edition-shaped. But .review and .status are going to be written at whatever level users think in, which is usually the work. If I listen to the audiobook and you read the paperback, we’re both reading the same book and we want to see each other’s reviews, but we don’t share an ISBN.

Deciding where to draw the line for a work/book is tough. Here are some fun edge cases, and what decisions we made:

  • If a work/book has editions in multiple languages, are those the same work? (Hardcover: all language editions are children of the same book/work, just in different languages)
  • What about abridged vs non-abridged? (Hardcover: These are children editions of the same book, however we don’t current abridged separately other than in text fields).
  • What about partial books (Hardcover: * The Way of Kings Part 1) (Hardcover: we call this a “partial edition”, which has a single parent book).
  • What about book collections, ex: * The Adventures of Tom Sawyer and Adventures of Huckleberry Finn contains two separate books. (Hardcover: We track this as a different book than Sawyer/Huckberry Fin, with a “Collection” boolean, then track the “books” this book contains as an ordered array of book ids).
  • How is non-english data about the Work/Book stored (title, description, etc)? (Hardcover: we do not do this today :sob:).
  • What about translations of an Edition? The translation of * The Odyssey by Emily Wilson was the first by a woman from the original source text, and has notably different meaning in some places to other translations. (Hardcover: we consider all translations to be editions of the same source, and all the same book).

One tricky thing I see is that different sites will decide what makes up a Book/Work is different. Some places might want to separate Work/Book by language to side step that last question. Or they might consider abridged a separate Work/Book because reviews would be notably different.

Here’s the visual representation of how this ends up working.

   ┌───────────────────────────┐
   │           Book            │   the work
   │  title, authors, series   │   "Project Hail Mary"
   │  canonical edition        │
   │  dto json     ◄───────────┼── cached json combining all edition dtos
   └─────────────┬─────────────┘
                 │ 1..n
   ┌─────────────▼─────────────┐
   │          Edition          │   the manifestation
   │  isbn10, isbn13, asin     │   format: physical | ebook | audio
   │  format, language         │   pages OR audio duration
   │  publisher, release date  │
   │  pages / audio seconds    │
   │  contributors (narrator,  │
   │    translator, editor)    │
   │  dto json     ◄───────────┼── cached json combining all book mapping dtos
   └─────────────┬─────────────┘
                 │ 1..n
   ┌─────────────▼─────────────┐
   │       BookMapping         │   external identity + snapshot
   │  source (OpenLibrary,     │
   │    Google Books, etc)     │
   │  external_id              │
   │  payload text ◄───────────┼── cached snapshot of source
   │  dto json     ◄───────────┼── cached json representation of source
   │  fetched_at               │    
   └───────────────────────────┘

This mirrors how we store data on Hardcover, which I wrote more about here. We have a service we call a “prioritizer” which knows which fields to grab from different mappings (ex: use the Title from Goodreads, use the data and description from Google Books, use the cover from Google, use the edition information from Open Library, etc). That rolls up N BookMappings into a single Edition. Then we repeat that process to roll up N Editions into a Book, favoring the Editions in each format that has the most readers (ex: This Book is 294 pages or 12 hours 9 minutes based on Edition X and Y).

I could see a case for all of us collaborating on the Edition Level, but then with different concepts of what the “book/work” level is in different platforms. We could have both entities, and then use a composite key / snapshot both of those dto (data transfer objects / snaptshot / whatever we call it) objects into the users PDS to represent the book + edition.

In that case edition wouldn’t “belong” to a book, but would belong to many “books”. I’d love to figure out a structure that works for both, but I suspect some voting/linking might be needed there.

Librarians / Curators / Non-Author Editors

Hardcover maintains a book database with a community of volunteer librarians. That’s a role people apply for, and a Hardcover team member (usually one of 3 librarians, but could be any of the 15 of us working part time for equity + me full-ish time) will grant that permission. Librarians have resources in our docs to understand standards, as well as a channel in the Hardcover Discord that only Librarians can access by linking Discord with Hardcover.

Librarians can merge have an “access level”, and can merge books together that impact users below that threshold. This can be disruptive if done wrong, since now all users past history is now wrong.

Progress is Two Different Records

This one seems small and I don’t think it is. “I’m on page 200” is three separate facts:

  1. A log entry. An event, at a timestamp, against a specific edition. Immutable, append-only, and it’s what reading history, stats, and streaks are built from.

  2. Current position. State. Mutable, one per user per book, and it’s what the UI reads to show you where you are.

  3. All reads. Log of the overall reads, edition and when. I started the audiobook edition on May 1 and finished it on May 8.

We store both. Collapsing them into one mutable record loses history, and I noticed BookHive ran into a version of this already (issue #41, where status changes on a mutated record all inherit the original createdAt).

It also matters for format: page 200 and minute 340 aren’t the same unit, and progress only makes sense against the edition you’re actually consuming. So the log entry needs an edition reference even if status lives at the work level.

(The above progress section was co-written with Claude, which did some research on my behalf).

Review Text Formatting

I think we should make plaintext field mandatory, and have content included in multiple formats. Maybe a plaintext version then an array of contentType versions provided by the platform?

Hardcover currently saves all reviews in plaintext (which is used to create the also cached length), and Slate in JSON custom format that supports spoiler tags, links to other entities in our system, etc). If we used something like @omg.jacky.wtf mentioned with contentType: “text/x-slate-hardcover-1 for this, each platform could own this format, while still having the plaintext. We’d save the HTML and Markdown versions in here as well for convenience, which are derived from the Slate document.

Claims and edit rights are two systems

Bibliograph’s claim record seems to do double duty: it establishes that a DID is the author or publisher, and it grants edit rights, with librarian status accruing from holding multiple verified claims.

Those probably want to come apart. Authors are subjects of the data. Librarians are curators of it. In our experience the overlap is small, and the incentives are different enough that merging them causes trouble.

We’re mid-design on author verification right now, and the thing that pushed us away from a single gate (ID upload, DNS record, that kind of thing) is that it excludes almost everyone who isn’t already well resourced. What we landed on instead is accumulating lightweight signals toward a two-tier badge: social cross-linking, publisher vouching, email domain, and ORCID, which is the one signal that’s fully automatable and which I haven’t seen mentioned here yet. Worth a look if the claim record is going to carry any weight.

Additional Review Metadata

I’m curious to learn more how about reviews in other platforms handle it when different services might add on additional fields as part of a review. We have a few other fields that we consider part of the “review”:

  • Sponsored Content?
  • Review contains spoilers?
  • Canonical URL for this review
  • Media URLs (posts to Instagram, TikTok, etc that contain the same content).

The last two are important for reshowing the review and linking back to the original. When we show a review that has a canonical tag, we add that canonical URL in the header so the original author URL gets the SEO credit. The media URLs allow for embedding other formats of the review alongside and for some other really neat future-features (ex: video feed of reviews by readers with similar taste to you for books we recommend for you).

Other sites might have other metadata they ask for too: tags, moods, genres, pace, character vs plot, spiciness, you name it.

Private and follower-only data is unaddressed

I’ve been following different people working on private and following only data on ATproto, but I think this is still something that’s being figured out. Is anyone familiar with those options and their status?

The MVP scope is read status and reviews, with no visibility model. Repos are world-readable and there aren’t private data primitives yet, so anything written on-protocol is public, full stop.

That’s fine for the MVP, but I’d like to see a visibility field in the record shape from day one even if the only supported value initially is public. It gives apps a defensible answer to “why isn’t all my data here,” and it means the lexicon doesn’t need a v2 the moment private primitives land. On our side, private entries just wouldn’t sync at all, driven by that field.

What Hardcover Can Bring

@nickthesick ‘s second takeaway is a that good book data is very hard to get and enrichment work benefits everyone, and I agree. Once we land on a shape of the models that works, we can help seed it. We currently don’t have ATProto integration on the site, but I’ve been looking into allowing users to bring their own PDS and we’ll save their data there. The private/following only side of Hardcover’s data has been the biggest blocker for me on getting started with any kind of integration there. I think if we can figure out how the entities map between systems and figure out private/following only data, then we’d be in a good spot to start something.

I suspect we’d create a bunch more entities in there specific to Hardcover as well (lists the user created, prompts they’ve answered, etc).

Count me in for the working group!

5 Likes
Handling Reviews for the Book Lexicon
10

It’s me, I’m such a user :grin:
There’s something about seeing the thing you have being represented to the highest fidelity that is affirming.

This is an important distinction and makes me wonder if we should introduce a community.lexicon.book.edition as well. So far, with Bibliograph, it’s been operating solely on the edition level - I can rework it to consider this.

Frankly, I am willing to follow your lead so far with the modeling of the data since it works and is durable.

I agree here. So https://constellation.microcosm.blue/ provides the ability to discover backlinks to data. If every version of a edition points back to its parent book that we agree on then this should work well. Especially if one was building their own version of Bibliograph (I call it an AppView but tbh, someone could make it as a cron that processes things from the firehose once in a while).

Regarding librarians, this is a little tricky. We could have a labeler service that would be appointing folks the title so then if they publish an edit of a book then these services would update in response. But then we’d need trusted labelers (think of this like the verification process on BlueSky). Tbh, if we could get OpenLibrary in the loop as well, that would give us their verifiers and data in one go versus doing monthly dumps.

I specifically wrote a progress unit for this reason because I plan to incorporate a reader and Readium’s approach uses CFI for pinpoint accuracy.

I’m very in favor of this. Graceful degradation and opt-in enrichment. Only things that wouldn’t be possible would probably be like spoilers, content warnings, etc but that might be a fact of the content medium (how would you implement a spoilers tag in the CLI - for example).

True! I didn’t consider this. I ran ahead with the idea that an author can also update the information about their own book but I guess that’s a semantic conflict of access.

I thiiiiiink https://sifa.id/ might be able to help with that part re: ORCID

I think it’s less of a field on the data and more about where the data is written. Permissioned spaces would enable this (so it’s like gated access to data that doesn’t live in one’s repo explicitly, from my understanding; tagging @dholms.xyz to confirm this; they’re the resident expert on that).

2 Likes
11

I agree. I’m in favor of stuffing identifiers to the brim if it helps with disambiguation. I enjoyed your talk and think I have some ideas for Bibliograph (namely using your dump as a weekly seeder).

12

I think an optional field in the review with a strong ref to the users current reading progress record would probably sufficiently cover spoiler gating. You might even create an XRPC method for normalizing progress across editions.

1 Like
Handling Reviews for the Book Lexicon
Handling Reviews for the Book Lexicon
13

@adamfortuna if you know any of the developers of other review apps like storygraph or fable or all the other ones, I’m not even aware of, I would love to invite them to this discussion as well. The idea is lots of apps adopt whatever we define here so I would love to include them in the schema shaping process.

1 Like
14

Hi all - not to overcomplicate everything, but I see this as part of a class of problems, and I wonder if it’s helpful to think of the wider class when designing.

I’ve been thinking of a collection of review apps that all share a couple of traits:

  • They’re, uh, reviews (usually; they might just be logs, of reading or eating or whatever)
  • They review objective things in the world (i.e., thinks you could have an objective list of, vs, say, experiences or activities which are essentially infinite)
  • There are apps for most/all of these, but they are problematic in multiple ways (in particular, users don’t own their data, and the list of objects is private and proprietary)

Examples are:

  • Restaurants (the big one, IMO, in terms of user base and utility)
  • Books (also very big)
  • Whiskey/Wine/Beer/etc

The data you need to store about a book is quite different from the data you need to store about wine. But both need a method for building a publicly maintained list that will increase in depth and quality over time and provides objective reference capabilities. And both seem simple but actually have a lot of complexity (editions for books, as discussed above, and things like year, vineyard, and grape mix for wine).

I wouldn’t want someone to slow down building in order to solve everything at once. But as people build, it’s worth considering whether lessons from these other areas can inform designs. I’m also interested in what a meta-review app might look like; e.g., I only drink wine once in a while, so I’m not interested in installing a wine app. But I’d like the ability to record a review when i happen across a notable one. It’d be nice to have a restaurant and book review app installed (because I care a lot about those), but not a wine review app, and be able to see and manage them all in that one app, or go into the focused app for the details the meta app doesn’t or can’t care about.

In particular, the concerns around authenticating an author resonate very strongly with restaurants: I am flummoxed that every restaurant in the world doesn’t at least have a basic web site with a menu, hours, and contact info, but… they don’t. Thus, we can’t really expect them to be able to authenticate themselves as an owner.

Handling Reviews for the Book Lexicon
15

Notes on a Plain Text Field

Definitely a Good Idea

First off, couldn’t agree more with the first part. A plain text field feels absolutely necessary. We should strive to make content in the Atmosphere as portable as possible. Plain text distribution can go a long way in that direction. The specific contents of the field are worthy of discussion, though.

From the surrounding context, I read adamfortuna’s perspective to mean no explicit formatting for the plain text (i.e. it’s not markdown, just un-formatted strings of text with blank lines representing paragraph breaks). I believe that’s the wrong direction. Defining a spec with effectively no formatting severally limits the practical portability of the content.

For example, say we have two apps:

  • App Alfa: A book review app. It lets folks write reviews that contain strikethrough text.

    The lexicon for the app’s content is set up to store the original review (including identifying which text is strikethrough) in a custom format associated with a specific MIME type (e.g. contentType: "text/x-app-alfa-book-review") that other apps can ingest.

    It also stores/provides a plain text version of the review with no formatting beyond empty lines for paragraph breaks.

  • App Bravo: A book review reader app. It gathers reviews from other apps (including App Alfa) and displays them in a single feed.

The developers of App Bravo have a few options when it comes to content from App Alfa. They can:

  1. Create a transformation process that converts text/x-app-alfa-book-review content into their desired output.

  2. Ingest and distribute the plain text version of the App Alfa content with a generic converter that turns empty lines into paragraph breaks.

  3. Not include the content from App Alfa.

Option 1 takes work, obviously. But, if App Bravo wants the content from App Alfa, it’s really the only option.

If App Bravo tried the second option to just output the plain text content then the reviews would either be incomplete (e.g. if App Alfa removed the strikethrough text when making the plain text copy), or they would be a mess (e.g. if the strikethrough text gets included, there would be no way to identify what should be struck and what shouldn’t). Either way, the view of the content that App Bravo provides diverges from the original every time strikethrough text is used.

So, if App Bravo wants the content from App Alfa in a presentable style, they have to do the work to ingest the text/x-app-alfa-book-review format. Doable, but it could end up being a lot of work depending on the complexity of the format. The work would also have to be done for every other app that App Bravo wants to add to their reader.

If the plain text content App Alfa distributes is in a portable format, App Bravo can use pre-existing packages to handle the transformations with minimal workload. Even better. They only have to do that work once to cover ingesting content from every other app that also uses the format.

That leads us to Markdown.

Markdown

It’s entirely possible I misunderstood the context and adamfortuna was talking about storing Markdown content in the mandatory plain text field. Things get a little better if that’s the case.

With Markdown, App Alfa can still store their data in their custom text/x-app-alfa-book-review format. But, the plain text Markdown version they ship has known formatting rules to apply to it. All App Bravo needs to do to render the content is put it through one of the many available Markdown processors and render the results.

Mark down has a native syntax built in for strikethrough. So, App Bravo can fully reproduce the content from App Alfa in their review reader. That’s an improvement, but it leads back to a new version of the original problem: Instead of limiting content to the least common features of plain text, content is now limited to the least common features of Markdown. Anything beyond basic blog post formatting is out of scope.

For example, say App Alfa provides a spoiler feature that let’s folks hide content behind a spoiler cover with a label on it. There’s no native way to represent that in Markdown. The feature needs a custom content type defined which is not something Markdown can do.

[aside: Embedding HTML in Markdown is possible, but that locks content into the specific tags used for the embed. I consider that a non-starter for portable content.]

App Bravo can always implement the text/x-app-alfa-book-review engine. But, we’re back to the same issue of effort. If the formats get complicated and/or the review reader wants to ingest content from lots of other apps, the workload becomes a limiting factor (and the effort would be better spent building social features rather than ETLs). Dealing with custom formats also add significant complexity to the system since both sides must stay in sync if there are any changes.

We also haven’t mentioned the increased burden on App Alfa. They must expend effort to generate and maintain the text/x-app-alfa-book-review spec and outputs. If apps aren’t in place to use the format, the appeal of doing that work is limited. Which leads to the a chicken/egg/cold-start problem because building an App Bravo reader app based on custom formats which might exist someday isn’t super appealing either.

I expect the practical reality of all this is that if the plaintext field is Markdown, then content from any app that does more than what Markdown can represent will effectively be locked into the original app. Trying to use it elsewhere will either be too much work, to broken, or both.

Given the creativity of the Atmosphere, that’ll happen a lot.

I believe instead of expanding the number of content formats (i.e. each app being responsible for defining their own format/content-type), we should focus down to standardize on a smaller number of formats that are better at what they do. Markdown will certainly play a role. And, it’s fine, as long as all you want to do is basic paragraph content. But, having to truncate all the content we want to be portable down to Markdown’s limited feature set would be a huge miss.

That leads us to NeoDoc.

NeoDoc

NeoDoc is a content format born out of frustrations with Markdown’s limitations. (And, full disclosure, it was me. I was the one who got so frustrated with Markdown’s limitations that I designed the format to use on my various sites.)

Here’s a quick example of how to use it to make spoiler content for a book review:

-- title

Dungeon Crawler Carl

A bunch of words about the book
go here.

-- spoiler/
-- label: Only read this if 
you're done with chapter 27.

If the doomsday device isn't a 
Chekhov's Gun, I don't know
what is.

-- /spoiler

And back to more writing
about the book that's not
under the spoiler.

NeoDoc produces a standardized AST designed for use in a templating system (which is also standardized). NeoDoc provides a default set of templates with the expectation that apps will customize them to their liking. App Alfa (where the content originated) would have title and spoiler templates responsible for rendering the -- title and -- spoiler sections of the content. The spoiler being set up to cover the text with an opaque box and the label.

App Bravo can make it’s own versions of those templates. Or, it can use the original ones from App Alfa. (The idea being that App Alfa stores its templates in a lexicon that App Bravo has access to. More details on that to come in future discussions.) Or, it can fall all the way back to the defaults built into NeoDoc.

This gives each app complete control over how it display content without having to interact with each other through custom formats. Other apps can use the content just as easily and just as independently. All without limiting features beyond basic paragraphs.

The net result is a portable content format that doesn’t sacrifice creativity to the limitations of Markdown’s format.

Outro

There’s a little more to NeoDoc than you see here, but not much. It’s designed to keep the mental model small so you can focus on making content. You can check out the NeoDoc site for more info. Or, ping me or the neodoc accounts on bluesky if you’ve got question or feedback beyond the scope of this thread.

-a

Postscript

  • NeoDoc has been around for three years. But, you’ve probably never heard of it given that it’s embedded in my personal static site generator. It puts in a lot of work there powering roughly three thousand pages of content, but hasn’t seen wider distribution yet. I’m extracting it to a library and independent app that can be used in the Atmosphere as we speak.

  • I’d probably stick an WYSIWYG editor in front of NeoDoc most of the time, but it’s entirely human readable/writable plain text without one. It’ll be even more so as syntax highlighters for the format come online.

  • I hadn’t thought about spoiler tags or content warnings until this thread. They’re perfect examples of the types of content I believe we should be working to ensure are portable. My general point being that if content isn’t fully portable, you’re effectively locked in to an app even if you can change the PDS that’s storing the data. For example, if I wrote a bunch of posts with spoilers in them any app that doesn’t support them will appear broken. I wouldn’t move to them.

  • Regarding:

    I don’t think we should use the limitations of our output environments to determine the the fidelity of the content distribution. We should make the raw content is rich as possible. That way, apps that are rending for specific applications (like the CLI) can have as many options as possible to come up with a solution for their environments.

    (Thinking about the CLI specifically. One option would be to transform the characters of spoiler content to █ blocks, then provide a hot key to toggle the actual content on and off. Of course, for this to happen, the app generating the CLI content would need something like NeoDocs formatting to know what to hide and what to show.)

  • I’m exited NeoDoc handles spoilers out of the box. Of course, it’s not the only solution. While I’d love to see NeoDoc become a part of the Atmosphere, I won’t be too disappointed if it doesn’t, so long as we put other solutions for fully portable content in place.

  • Oh, yeah, there’s an entire bit about how the named NeoDoc sections and their associated attributes act as ad hoc keys for database style querying. This post is already super long and I don’t have the spoons to get into that right now. There’s a overview on the NeoDoc site if you want to learn more.

  • I also started another thread talking about NeoDoc specifically if you’re interested in that.

1 Like
Handling Reviews for the Book Lexicon
16

I’m going to make a new thread soon for handling reviews in relation to this but I think the core idea for a book lexicon as described here seems to be something everyone’s in touch with! What I would like to do next is to start a working group around this (so we can have a dedicated space for review conversations, how to handle contributors to materials and verification thereof and what have you).

Update: the thread is here, Handling Reviews for the Book Lexicon . I’d like to know if anyone would like to co-chair the WG with me (I am opting myself in unless there’s two other more adamantly charged folks for this work).

3 Likes
Handling Reviews for the Book Lexicon
17

First off, huge credit to @omg.jacky.wtf for actually getting this moving.

A bunch of us have been talking about books, bibliographic data and shared lexicons for a while, but mostly with our heads down building our own projects. Jacky got people into a room and started turning that into something concrete. That’s important.

I think the next step is getting the builders together and treating this a little like what happened with Standard.site.

We already have working code and different approaches in the wild. Bibliograph is now doing some really interesting work on the bibliographic side. BookHive, @xardex.dev, has a working book and reader implementation (CAVU is working on a reader - @chris.pardy.family). Byline.pub, me, has been working on author and book records. There are other projects coming at this from reading, publishing, metadata and discovery as well.

None of that work should be thrown away. Quite the opposite. That’s the material we should be bringing to the table.

I also don’t think we should assume any existing book.*, *.book or project namespace automatically becomes the canonical book lexicon. We aren’t really at that point yet. The useful question is: what is common between the things we’re already building, and what should apps across the ATmosphere be able to rely on?

I’d suggest we establish a small Book/Bibliography Lexicon working group.

Not an IETF bureaucracy. Nobody needs another committee eating their evenings. :grinning_face_with_smiling_eyes:

A group of the people actually building this stuff.

Maybe:

  • gather the existing schemas and implementations

  • identify the common primitives: works, editions, identifiers, contributors, authorship, reading state, reviews, relationships, etc.

  • run a few workshops across different time zones

  • document the places where our implementations disagree

  • get something resembling a v0 proposal

  • put it out publicly for comment

  • implement it in several real applications

  • learn where it’s wrong

  • then publish the shared lexicon

Basically: rough consensus and running code.

I’d also really like to invite some of the folks involved with @standard.site into that discussion — @aka.dad, @brookie.blog, @awarm.space and anyone else I’ve missed. They’ve already gone through the exercise of getting several independent AT Protocol applications to converge on common lexicons, and there are probably lessons there that would save us some bruises.

Jacky’s thread is a damn good starting point. Bibliograph gives us something concrete to experiment against. The various book apps give us real-world requirements.

Now we need to get those people around the same virtual table.

Different projects. Different time zones. Different needs. But we all have a stake in getting this right.

I’d be very happy to help organise it.

1 Like
18

I’d love to see some of those folks pop in for pointers/suggestions but I imagine a lot of it would devolve into:

  • a committed team of builders
  • durable experience from folks using it
  • an agreement on an eventually consistent lexicon

FWIW Bibliograph, I’m using it as an example; I expect every large enough service to implement their own AppView that’ll have their own bits and quirks of information about a book. For example, Livtet is being set up to accept an AppView as a DID prefix, a bit like how Bluesky and Bluesky-centric/-lexicon consuming apps are set up. So, if you wanted to view book information from Bibliograph using Livtet, the URL would be something like http://livtet.olamaelcu.net/book/did:web:biblio.livtet.olamaelcu.net/lgwy6mnok6vk7 and it’d resolve accordingly. The client could do some post-fetch cleaning up or only use the AppView for identity discovery (I can see Hardcover doing this and relying on their metadata as collected and described by @adamfortuna already) whereas folks like me would be relying solely on the AppView for information like this.

I’m down for some of the research points you’ve described here, I can get started on that and provide a link to some sort of document back here in a bit. I’m happy to work with you with @jasonbu.online to get this going!

1 Like
19

Yes, I think we’re very much on the same page.

I’m definitely not imagining a canonical AppView or metadata source. Different apps should absolutely have their own views, enrichment and opinions.

The shared piece is the contract between them.

I also read Chris Shank’s post, and his distinction between content lexicons and publishing/discovery lexicons helped sharpen this for me. Done a lot of work with BookHive (@xardex.dev) around a shared lexicon.

“Eventually consistent lexicon” feels like the right approach. Enough common vocabulary that Byline, Livtet, Bibliograph, BookHive, Cavu and other ATProto book apps can understand each other without forcing everyone into the same implementation.

For the group itself, I’d keep the focus on AT Protocol, projects already building on it, consuming it, or seriously exploring support for it. But also include others like @adamfortuna perfect contribution above… (are you guys thinking of AT proto?)

Next step might simply be to identify a small core group of builders and ask who’s interested.

Then we can inventory what already exists, identify the common problems, run a few working sessions across time zones, pull in the Standard.site folks for lessons learned, and work toward a v0 proposal that multiple apps can actually implement.

  1. inventory the lexicons and approaches already in use
  2. identify the common problems we actually need to solve
  3. hold a few working sessions across time zones
  4. pull in the Standard.site folks for lessons learned
  5. publish notes and decisions openly
  6. work toward a v0 proposal that multiple apps can actually implement

Maybe we start nominating projects and people here?

Others that are interested or that should be recommended?

1 Like
20

I think you’ve hit the needle on the head with the folks for recommending for this. We can sync back in a few days after folks have seen this and decide to report back. This gives me a chance to refine what I’m doing with Bibliograph, tbh.