Reams: Obsidian Automerge Plugin for Localfirst collab, ATProto is next

I had a great catch up with @hyl.st this afternoon talking about his Reams app. It has a full implementation of Automerge supporting many Obsidian features including Dataviews.

It also syncs files on disk in Markdown to Automerge and back, with a few edge cases that may or may not matter to you.

It’s in a private repo now, and ready for some alpha testers.

This part has no atproto crossover! You point it at an Automerge sync server or run your own (we have one as part of the commons computer stack). You can share a link to a page and collab with others running the same plugin.

Obsidian ATProto

@hyl.st has some early atproto stuff - like, sync your entire PDS locally, and reference things.

I suggested that this could be a great separate plugin, and also the chocolate to the Automerge peanut butter: fits great for those that want both, but also supports certain use cases that work alone.

We’ll come back to atproto.

Interop where?

Our general discussion was about how we interop and how we collab and why.

I’m having fun building features for my own website at https://bmannconsulting.com. It’s Obsidian editable markdown but goes into a build system and makes an Astro website with atmospheric features and integrations.

I’d love to collab with @hyl.st but I’m not the target for replacing my website custom build. I do want Automerge and some atproto features.

Automerge use cases for me are likely to be private and stay private. I can share meeting notes with others.

But I can also get edits before I publish something. Publish to atproto in what shape?

Digital Garden Note Shapes - maybe lichen wiki

There are a handful of folks thinking about standard site publishing out of their Obsidian. I have blog shaped content on my site, but I aggregate my writing of ATProto native tools with local blogs → https://bmannconsulting.com/blog/

For me, my notes pages are more like wikis. They are never “finished”, rather just some snapshot of a work in progress.

The thing I’ve seen that is the closest to the style and type of atmospheric content is https://lichen.wiki

I’ve kicked the tires on it a bit, there are groups, it uses Markdown, it’s really nice today already. No private data, but public wikis that are published and listed, and hidden ones that are on protocol but not publically online.

So I think I’d like to sync my Obsidian garden notes to Lichen. To optionally publish any note, or to sync any notes from the PDS like my blog example, to private spaces over time.

The diffs are the part where I have an issue: let’s say I edit a note that @hyl.st started. I don’t want just a diff in my PDS. I want the full content of the note after my edits, so I have a copy and don’t have to aggregate it myself.

Other use cases for this mode. So we have a folder aka a wiki like this Reams Testers one. Some of the content in here will be by @hyl.st, some by me, some by you all. As we edit and collab - I’d like to choose which version to pin. I’m fine if someone edits my note - but on my site, I want to pin the full content version of the note that I want.

Maybe we store diff AND optional full content? @juprodh.me I really have no idea here.

Multiplayer collab with Automerge, to published lichen.wiki page and back again.

Private Spaces with HappyView

@hyl.st is already using HappyView for his backend, for synching, and with an eye towards private data and especially groups.

Maybe folders in obsidian are different kinds of content collections, groups, spaces? The interface to lean into Obsidan may very well be a thing. We have to experiment.

Next

Leave a comment if you want alpha access to Obsidian Automerge.

Let’s all experiment with lichen.wiki? I’ll do it on my codebase, @hyl.st will do it in his, and we’ll see what it feels like and what collab through a lexicon works. Leave a comment if you’re coming along for the ride. We’ll use the Reams Testers Wiki as our test space.

5 Likes

I can help with testing.

2 Likes

Amazing job @bmann.ca !

I’m the dev of lichen.wiki.

I think it would be cool if we could have interop with @hyl.st new solution.

For lichen, currently users store a diff in their PDS and the app view replay the linked chain.

I’ve been thinking about storing snapshots in users PDS alongside the diffs. It would be like a last snapshot thing that updates when a user makes a new diff.

There are a few benefits:

  • if the chain is lost, we have a fallback
  • it’s easier for other collaborative apps to emerge in the atmosphere by just looking at snapshots
  • might be easier to export Obsidian-like notes
  • might enable more compatibility between apps like the one mentioned

I started to look at it a few weeks ago, but had no time recently. I may take a look at it again

3 Likes

This would be amazing @juprodh.me! I would certainly similarly have a “materialized” snapshot that is markdown / plain text through which we can interop :slight_smile:

I’m only just starting to get to atmospheric integration, so still might be a little while until I get around to it, but will implement through this interop contract.

Are you planning to use standard lichen.wiki lexicons, or should we denote these snapshots in some other fashion? Boris’ test leads me to believe there’s little to be gained from having a seperate lexicon, but happy to be convinced!

Hey @hyl.st !

I agree on the common lexicon, wiki.lichen.* should be fine. But I’d need to change a few things first: editing an existing note isn’t easy right now, so I’ll expose what’s needed more cleanly.

I thought about the lexicon updates I’d need for better interop, and here are some things I kinda like:

  • make a small XRPC read API: getWiki, getNote, listNotes, maybe more. getNote returns the note URI, the content and the current head, which is everything you need to build a revision.
  • accept full note content alongside diffs, with diffFormat: “full-text”: you send the whole body and the patch is computed by the appview. In this case, diff-match-patch stays optional for you.
  • add the noteSnapshot lexicon, but revisions stay the write path, that’s what Lichen consumes. A noteSnapshot is user-specific: with several contributors, each one holds that user’s last edit.

Either way you’d write a noteRevision, with parentRevision set to the current head, which getNote gives you.

I’ll try to implement this quite soon when I’ve time, but it should help other apps to interop with Lichen as well.

1 Like

hey @bmann.ca @hyl.st

update: the read queries and full-text revisions are live, and the lexicons are published. I haven’t done the snapshots yet.

  • wiki.lichen.getWiki
  • getNote
  • listNotes
  • listWikis

You can check here!

you can also try to curl, for example:
curl 'https://lichen.wiki/xrpc/wiki.lichen.listNotes?wiki=at://did:plc:32womkhqqvdofazze5zoi7mo/wiki.lichen.wiki/welcome-to-lichen'

1 Like

nice! so the getNote lexicon is the materialized view of the markdown? I noticed it says it shouldn’t include frontmatter in the note? I’ve been saving frontmatter to my records. For the at-uri property, it seems like I can put in slug, since the record itself is the note? Or should it reference the automerge backed bit? Maybe the latter now that I’m thinking about it.

What’s the difference between full-text revisions and the snapshot?

Appreciate you working on this! :slight_smile:

hi!

I just rewrote the lexicon descriptions for external implementers and republished them, so I think it should be more clear now.

Yes getNote is the materialized view, where the chain already replayed. Full-text revisions are the write path and the snapshot will be a readable copy, parentRevision must be the current head, otherwise the revision is dropped.

Lichen never adds frontmatter of its own for now, it is currently part of the note body, stored and returned verbatim in the revision chain and by getNote. The frontmatter block is hidden at render time in the app view. I haven’t decided on how to show some useful stuff to the user in a cool way.

The at-uri can’t be built from the slug today. Only wiki.lichen.wiki is slug-keyed; notes are TID-keyed, and the DID in a note URI is only whoever created the record (not the wiki owner, not the last editor).

You can:

  • read the URI from listNotes or getNote
  • address the note by wiki + slug, which getNote takes as a pair
1 Like

Hi! Quick update on my end, I finally got reams deployed, and it’s able to write notes into lichen wikis using the getNote lexicon :slight_smile: This note was published 261004 — Reams Testers · Lichen

Let me know if there’s something I should change in the way I publish these.

Right now, I only have one way publishing, using the getNote lexicon as a materialized view. However if someone makes changes to that note, reams doesn’t really handle it and just clobbers over the changes next time the user publishes the note. I’m not super sure how to handle ingesting notes from lichen either at this point. I’ll have to think about how to handle that, given the source of truth is an automerge document currently :thinking:

2 Likes

Awesome!

I think if the lichen note has changed but the AM doc hasn’t, you make the lichen note the newest version of the AM doc.

If the AM head has moved, you could make the note a branch (handwave over a merge interface)

I am painting in some of the features of Patchwork here, but you have to handle this for multi device and/or multi person syncs anyway?

1 Like