In Which A Book Lexicon is Born

Discussion
, ,
1

Hi everyone,

We recently held a sync to discuss creating a shared book lexicon for the Atmosphere Protocol. I’ve shared the full recording (thanks Ronen!) and notes from the meeting here: Fathom with some early notes here: https://app.notion.com/p/Book-Lexicon-Session-1-211b2b43aabb8007b098d063972ca58c (also do we have a ATProto version of Notion yet?)

The book logging social scene is highly fragmented. Dozens of Goodreads alternatives are competing but not interoperating, leading to a lack of shared network effects and ultimately, app failures. This thread is meant to gather community feedback, surface adjacent integrations, and recruit members for a future formal working group (ONE OF US! ONE OF US!).

Holding “Data Unions” and “IndieSky” in Tandem

A core concept from our meeting was the idea of building a “Data Union”—a model where apps share data interoperability to prevent user lock-in and build a shared network effect.

As we think about this, it’s important to hold the concept of a Data Union in tandem with the ongoing infrastructure work of IndieSky. IndieSky is doing the heavy lifting of providing sovereign, user-controlled data rails; the Data Union concept could (maybe?) build on top of it. IndieSky provides the underlying infrastructure (the “how” and “where” of sovereign data), while a shared Book Lexicon provides the cooperative application layer (the “what” and “with whom”).

The question for the community: How do we ensure that a Data Union built on Atmosphere leverages IndieSky’s infrastructure to maximize user sovereignty while still providing the shared network effects apps need to survive?

The Proposed Technical Strategy: Copy-over-Reference

To ensure data resilience, the proposed MVP relies on a “copy-over-reference” model. When a user logs a book, their PDS gets:

  1. A strong reference to the main book record.

  2. A minimal metadata copy (title, author, identifiers) as a fallback.

This ensures user data remains functional even if the source book record disappears.

MVP Focus: We want to prioritize the lexicon for core social interactions first:

  • Read status (reading, read, to-read)

  • Reviews (rating, text)

The complex book indexing lexicon (handling edge cases like partial books, compilations, and manga) will be developed in parallel. For indexing, we discussed a “lazy initialization” model—creating book records on-protocol only when a user interacts with them—avoiding a massive, costly backfill. Deduplication remains a key challenge here, specifically regarding fallback keys and hash collision risks.

Spotlight on Adjacent Builders (Bookhive, Semble, Hardcover)

If you are currently building apps adjacent to this space—like Bookhive, Semble, or Hardcover—we want to hear from you!

  • How are you currently handling book metadata and user interactions?

  • Does the “copy-over-reference” model align with your app’s architecture?

  • Where do you see friction in adopting a shared lexicon?

Please drop a link to your project, highlight your current integrations, and let us know how a shared lexicon could benefit your ecosystem.

A Prototype

I also took some time this weekend to work on a prototypical AppView (think of https://public.api.bsky.app/ but for books) that lives up to some of these ideas. It’s available at https://biblio.livtet.olamaelcu.net/ with the source on Tangled and on GitHub at GitHub - olamaelcu/bibliograph: An AppView for interfacing with bilbographic information on ATProto · GitHub. I’ve primed it with a lot of books from Toni Morrison and Arthur C. Clarke but it should auto-import new books if they’re discoverable by Google Books (please be kind to my API key!) or OpenLibrary. Let me know what you think - happy to host it for the community!

6 Likes
2

Cross posting this to Hardcover’s roadmap discussion

1 Like
3

Do y’all have a plan for the format to use for the review text?

Text content storage in general feels like a potential, unintentional app lock-in vector across the atmosphere. At least, if the text is anything beyond paragraphs or maybe markdown.

I spent the last few years designing a markdown replacement format. The original goal was to make it easier to make website content without having to jump all the way to HTML. I think the same features that make it work there have the potential to act as an interoperability layer for text across the atmosphere too. I’m happy to share a link if you’re interested in taking a look.

4

I’m very interested to see what you got! I’m personally open to having some sort of content type hint and then letting clients support what they can (with an optional plain text value so clients that don’t support anything can still see/present something to end users).

I’m also of the mind of having reviews be referential, so it can point to a Standard Site publication or a Bluesky post (for example) so clients can more deeply represent information.

5

Yeah, I’m right there with you on letting the clients control what they see (and being able to fall back to plain-text). My design is basically a replacement for markdown with extra horsepower thrown in. I’m making a lexicon for it, but the idea is that it works primarily as a portable, stand-alone format. You can check out the intro page here: https://neodoc.style/

(I’m also writing a post to start a thread to discuss it. I should have that posted sometime tonight.)

1 Like
Bibliograph: A Prototypical AppView for Books
6

I’m liking what I’m seeing! It doesn’t look immediately backward compatible with Markdown so do you see folks using something like contentType: “text/x-neodoc somewhere in their records?

7

Hi there,

I talked a bit about BookHive is doing this in my atmosphereconf talk & write up: The Design Philosophy of BookHive - Nick The Sick - pckt

But, to summarize, BookHive is principled on data-sovereignty, so this means:

So, anyone can bootstrap bookhive.buzz from scratch & run the service on their own with the OSS code at: GitHub - nperez0111/bookhive: Track your books, share your shelves, see what others are reading · GitHub

This copy-over-reference idea is almost 1:1 what I’ve already implemented in BookHive, so I’m obviously in support of it.

I think that the real problem lies in multiple applications having different datasets for understanding what a book is, so making a review on one platform would simply be missed on another platform, since there is not a great way to understand that we are actually talking about the same book. Most platforms are using centralized databases where they can unilaterally make these moderation decisions, but in an open network like atproto I don’t think we really can have the same approach. I think there needs to be a sort of community linking & voting & submission system for applications to interoperate with one another.

Would be happy to be part of the working group, life has just been busy lately so I didn’t get around to this until now.

3 Likes
8

Thanks! And yeah, the overall format can’t be transformed directly with a Markdown parser. I played around with the idea early on but quick realized it would have added a bunch of complexity to the both the parsing and the mental model about the format when authoring content with it. So, it’s explicitly not directly compatible with Markdown.

But, I’m adding a -- markdown section type that’ll provide Markdown parsing inside a NeoDoc. If you’ve got a bunch of markdown already, the conversion process would be to just throw the -- markdown section header in front of it and it’ll do the thing.

You’ve also got me thinking about YAML. I know lots of folks use it for front matter, but I wasn’t originally planning on adding support. That decision was based off the fact that the reference parser is written and rust and the main yaml parsing crate is no longer maintained. But, I just checked and it still gets hundreds of thousands of downloads every 90 days. So I think it’s more that it’s basically feature complete. The only potential issue would be bugs/security problems, but it could always be forked if that happens. Probably best to add it to make it easier for folks to migrate old docs.

I hadn’t thought about the contentType. That sounds like a good idea, but I don’t know enough about how everything works to know what to do with it (e.g. if I’ve got a lexicon I’d expect that’s not required? but maybe if other folks want to use the parser inside their stuff they need it? I’ll have to look into that more).

1 Like
9

Loving the conversation here (and in person and in the email thread!). I’ve been staying up to date and trying to wrap my head around it all. I’ve been working on this problem for the last 5 years on the Hardcover side, but this is the first anything I’ve done using Atmosphere and PDSs and that side of things, so please feel free to correct anything I say that feels off, or lacks understanding of that side of this.

I wanted to break down a few concepts that I’ve seen touched on, with some thoughts on how we’re solving them today, and touching on some ideas on how to solve them long-term in a decentralized way.

Books/Works vs Editions

I think this is one of the biggest concepts that if we can get right, it powers a ton of things across different sites (ex: View the Spanish version? / View the Audiobook version? / Is there a DRM free ebook for this book? Which audiobook edition has the highest rating? Which has a narrator that I like the most?).

To go back to @nickthesick ‘s mention of including all the useful data needed, the Edition has some of the relevant useful data to the user reading that edition. Without edition information you could say “I’m 50% of the way done with the book”. With the edition information you could say “I’m on page 256 out of 512 on the Hardcover edition of Book”, or “I’m 12 hours and 42 seconds out 20 hours and 5 minutes of the audiobook edition”. Also, different editions have different cover images, and we’ve found users really want to see the cover of the edition they have.

In order to see stats like listening time or pages read, edition information is required, and progress updates need to be linked to the edition. It can be an annoying step, both for data and for the user, but for full accuracy it’s needed.

All of that data lives in the relationship between a Book and an Edition. OpenLibrary calls these Works, which I think is a more universal term for an abstract “representation of a book”.

Important things to note about a Work/Book vs an Edition:

  • An Edition AND a Book/Work can have identifiers back to other platforms. On Hardcover, we have edition_id and book_id. OpenLibrary uses keys like /works/OL82563W (work/book) and /books/OL1439870M (Edition).
  • Choosing a title for the “Book/Work” gets more difficult the broader it is (ex: different languages, or even HP and the [Sorcerers|Philosophers] Stone).
  • Not all Editions have an identifier. Some editions don’t have ISBNs or ASINs (I’m looking at you Manga :face_with_peeking_eye:.)
  • Series data is best mapped to the Book/Work, not the edition. That way * The Hobbit is considered Book 0 of The Lord of the Rings, rather than all 519+ editions needing that data.

I think the takeaway here is that an ISBN identifies an edition, not a work. So if ISBN is the dedup key, community.lexicon.book.book is edition-shaped. But .review and .status are going to be written at whatever level users think in, which is usually the work. If I listen to the audiobook and you read the paperback, we’re both reading the same book and we want to see each other’s reviews, but we don’t share an ISBN.

Deciding where to draw the line for a work/book is tough. Here are some fun edge cases, and what decisions we made:

  • If a work/book has editions in multiple languages, are those the same work? (Hardcover: all language editions are children of the same book/work, just in different languages)
  • What about abridged vs non-abridged? (Hardcover: These are children editions of the same book, however we don’t current abridged separately other than in text fields).
  • What about partial books (Hardcover: * The Way of Kings Part 1) (Hardcover: we call this a “partial edition”, which has a single parent book).
  • What about book collections, ex: * The Adventures of Tom Sawyer and Adventures of Huckleberry Finn contains two separate books. (Hardcover: We track this as a different book than Sawyer/Huckberry Fin, with a “Collection” boolean, then track the “books” this book contains as an ordered array of book ids).
  • How is non-english data about the Work/Book stored (title, description, etc)? (Hardcover: we do not do this today :sob:).
  • What about translations of an Edition? The translation of * The Odyssey by Emily Wilson was the first by a woman from the original source text, and has notably different meaning in some places to other translations. (Hardcover: we consider all translations to be editions of the same source, and all the same book).

One tricky thing I see is that different sites will decide what makes up a Book/Work is different. Some places might want to separate Work/Book by language to side step that last question. Or they might consider abridged a separate Work/Book because reviews would be notably different.

Here’s the visual representation of how this ends up working.

   ┌───────────────────────────┐
   │           Book            │   the work
   │  title, authors, series   │   "Project Hail Mary"
   │  canonical edition        │
   │  dto json     ◄───────────┼── cached json combining all edition dtos
   └─────────────┬─────────────┘
                 │ 1..n
   ┌─────────────▼─────────────┐
   │          Edition          │   the manifestation
   │  isbn10, isbn13, asin     │   format: physical | ebook | audio
   │  format, language         │   pages OR audio duration
   │  publisher, release date  │
   │  pages / audio seconds    │
   │  contributors (narrator,  │
   │    translator, editor)    │
   │  dto json     ◄───────────┼── cached json combining all book mapping dtos
   └─────────────┬─────────────┘
                 │ 1..n
   ┌─────────────▼─────────────┐
   │       BookMapping         │   external identity + snapshot
   │  source (OpenLibrary,     │
   │    Google Books, etc)     │
   │  external_id              │
   │  payload text ◄───────────┼── cached snapshot of source
   │  dto json     ◄───────────┼── cached json representation of source
   │  fetched_at               │    
   └───────────────────────────┘

This mirrors how we store data on Hardcover, which I wrote more about here. We have a service we call a “prioritizer” which knows which fields to grab from different mappings (ex: use the Title from Goodreads, use the data and description from Google Books, use the cover from Google, use the edition information from Open Library, etc). That rolls up N BookMappings into a single Edition. Then we repeat that process to roll up N Editions into a Book, favoring the Editions in each format that has the most readers (ex: This Book is 294 pages or 12 hours 9 minutes based on Edition X and Y).

I could see a case for all of us collaborating on the Edition Level, but then with different concepts of what the “book/work” level is in different platforms. We could have both entities, and then use a composite key / snapshot both of those dto (data transfer objects / snaptshot / whatever we call it) objects into the users PDS to represent the book + edition.

In that case edition wouldn’t “belong” to a book, but would belong to many “books”. I’d love to figure out a structure that works for both, but I suspect some voting/linking might be needed there.

Librarians / Curators / Non-Author Editors

Hardcover maintains a book database with a community of volunteer librarians. That’s a role people apply for, and a Hardcover team member (usually one of 3 librarians, but could be any of the 15 of us working part time for equity + me full-ish time) will grant that permission. Librarians have resources in our docs to understand standards, as well as a channel in the Hardcover Discord that only Librarians can access by linking Discord with Hardcover.

Librarians can merge have an “access level”, and can merge books together that impact users below that threshold. This can be disruptive if done wrong, since now all users past history is now wrong.

Progress is Two Different Records

This one seems small and I don’t think it is. “I’m on page 200” is three separate facts:

  1. A log entry. An event, at a timestamp, against a specific edition. Immutable, append-only, and it’s what reading history, stats, and streaks are built from.

  2. Current position. State. Mutable, one per user per book, and it’s what the UI reads to show you where you are.

  3. All reads. Log of the overall reads, edition and when. I started the audiobook edition on May 1 and finished it on May 8.

We store both. Collapsing them into one mutable record loses history, and I noticed BookHive ran into a version of this already (issue #41, where status changes on a mutated record all inherit the original createdAt).

It also matters for format: page 200 and minute 340 aren’t the same unit, and progress only makes sense against the edition you’re actually consuming. So the log entry needs an edition reference even if status lives at the work level.

(The above progress section was co-written with Claude, which did some research on my behalf).

Review Text Formatting

I think we should make plaintext field mandatory, and have content included in multiple formats. Maybe a plaintext version then an array of contentType versions provided by the platform?

Hardcover currently saves all reviews in plaintext (which is used to create the also cached length), and Slate in JSON custom format that supports spoiler tags, links to other entities in our system, etc). If we used something like @omg.jacky.wtf mentioned with contentType: “text/x-slate-hardcover-1 for this, each platform could own this format, while still having the plaintext. We’d save the HTML and Markdown versions in here as well for convenience, which are derived from the Slate document.

Claims and edit rights are two systems

Bibliograph’s claim record seems to do double duty: it establishes that a DID is the author or publisher, and it grants edit rights, with librarian status accruing from holding multiple verified claims.

Those probably want to come apart. Authors are subjects of the data. Librarians are curators of it. In our experience the overlap is small, and the incentives are different enough that merging them causes trouble.

We’re mid-design on author verification right now, and the thing that pushed us away from a single gate (ID upload, DNS record, that kind of thing) is that it excludes almost everyone who isn’t already well resourced. What we landed on instead is accumulating lightweight signals toward a two-tier badge: social cross-linking, publisher vouching, email domain, and ORCID, which is the one signal that’s fully automatable and which I haven’t seen mentioned here yet. Worth a look if the claim record is going to carry any weight.

Additional Review Metadata

I’m curious to learn more how about reviews in other platforms handle it when different services might add on additional fields as part of a review. We have a few other fields that we consider part of the “review”:

  • Sponsored Content?
  • Review contains spoilers?
  • Canonical URL for this review
  • Media URLs (posts to Instagram, TikTok, etc that contain the same content).

The last two are important for reshowing the review and linking back to the original. When we show a review that has a canonical tag, we add that canonical URL in the header so the original author URL gets the SEO credit. The media URLs allow for embedding other formats of the review alongside and for some other really neat future-features (ex: video feed of reviews by readers with similar taste to you for books we recommend for you).

Other sites might have other metadata they ask for too: tags, moods, genres, pace, character vs plot, spiciness, you name it.

Private and follower-only data is unaddressed

I’ve been following different people working on private and following only data on ATproto, but I think this is still something that’s being figured out. Is anyone familiar with those options and their status?

The MVP scope is read status and reviews, with no visibility model. Repos are world-readable and there aren’t private data primitives yet, so anything written on-protocol is public, full stop.

That’s fine for the MVP, but I’d like to see a visibility field in the record shape from day one even if the only supported value initially is public. It gives apps a defensible answer to “why isn’t all my data here,” and it means the lexicon doesn’t need a v2 the moment private primitives land. On our side, private entries just wouldn’t sync at all, driven by that field.

What Hardcover Can Bring

@nickthesick ‘s second takeaway is a that good book data is very hard to get and enrichment work benefits everyone, and I agree. Once we land on a shape of the models that works, we can help seed it. We currently don’t have ATProto integration on the site, but I’ve been looking into allowing users to bring their own PDS and we’ll save their data there. The private/following only side of Hardcover’s data has been the biggest blocker for me on getting started with any kind of integration there. I think if we can figure out how the entities map between systems and figure out private/following only data, then we’d be in a good spot to start something.

I suspect we’d create a bunch more entities in there specific to Hardcover as well (lists the user created, prompts they’ve answered, etc).

Count me in for the working group!

2 Likes