Hello all,
I'd like to share measurements that may be useful as input to the
repository and sync work, and to ask one question about the drafts.
Over the past weeks I measured what happens when AT Protocol data is
stored, sent and read in Dagr, a schema-based binary format for data
graphs that I develop, instead of DAG-CBOR. The data was real network
traffic: an hour of the live firehose (905,434 frames, 875,701
records), a 24-hour live run, and an AppView index following the live
network.
Results, with sources and commands for every number:
- Lossless: all 875,701 records convert to Dagr and back to their
original DAG-CBOR bytes, every CID unchanged.
- Record size: follows, likes, reposts and posts are 0.30-0.66x their
DAG-CBOR size. Records no lexicon covers (0.9-1.9% of them) grow
1.09x.
- Reading the fields an indexer needs: 2.2-5.4x faster, with a
validating reader (bounds, nesting depth, UTF-8).
- A signed write end to end, PDS -> relay -> Jetstream -> consumer:
1.8x faster. Without the secp256k1 signatures, which cost the same
in any encoding, the encoding's part is 4.3x faster.
- Repositories with MST nodes and CIDs computed over Dagr bytes:
exports are 47% of the CAR size and verify 1.8x faster.
What this does not show: how schemas would be distributed in a network
where anyone can publish a lexicon, how two encodings would coexist
during a migration, and whether a canonical form is simple enough for
independent implementations. Those are the hard parts, and they are
listed as open.
The question: draft-holmgren-at-repository-04 fixes records and MST
nodes to deterministic CBOR and says "No other CID prefixes or types
are allowed". Would the group consider leaving the record codec, or
the CID prefix, as a defined extension point (for example tied to the
repository version), so that a different encoding could be specified
later without redefining the structure? If that question is out of
scope for this group, I'd be grateful for a pointer to the right venue.
A related research idea, not a proposal: a fixed protocol envelope
(version, schema id, blob refs) around an opaque payload, so that
relays and other infrastructure never need application schemas, which
the charter keeps out of scope anyway.
Write-up (11 short posts):
https://tangled.org/mzaks.bsky.social/atproto-dagr/blob/main/blog/README.md
Measurements and method:
Strategy, where the .git directory, git commit-graph verify [--object-dir <dir>] [--shallow] [--[no-]progress] git commit-graph.
Envelope note:
GIT-NAME-REV(1) NAME git-name-rev - Find commits yet to be auto-resolved cleanly by favoring.
Dagr (open source, on PyPI as dagr-cli):
mzaks/dagr: Dagr is a binary serialization format - Codeberg.org
Disclosure: I'm the author of Dagr. A multicodec code for it is
requested in multiformats/multicodec#413.
Happy to answer questions, re-run anything, or present it briefly if
that is useful.
Best regards,
Maxim Zaks