Purposal: General purpose File sharing / Download links lexicon?

Lexicon Community
, ,
1

I’ve been pondering about a project idea I had, and it seems there’s no existing lexicon set for sharing documents of any type. I don’t want to focus on a single type, because I want this lexicon to be for general purpose documents. Including but not limited to images, videos, zip archives, blender models, PDF, binary files, software packages…

Maybe someone would be interested by such thing? I’d see an usecase for general purpose file sharing applications as an alternative to things like Google Drive or MediaFire…

The lexicon could be named community.lexicon.download or community.lexicon.file. Maybe we need one lexicon that represents the shared document, and separate lexicons for the distribution type? (community.lexicon.file.ipfs, community.lexicon.file.magnet, community.lexicon.blob, community.lexicon.externalurl)

A record contains would contain the filename, content type, estimated size, date of publication, hash, and an optional short description or title.

The record optionally contains different methods of distribution, all of them are optional :

  • Reference to the blob in a PDS
  • CID for IPFS
  • BitTorrent Magnet link
  • Maybe an external URL for whom may use Google Drive anyway?

If no method of distribution is defined in the record, the AppView/client/consumer should not index the record ; or tell the user that no distribution content has been defined.

An app that I would build would display something like this to the end-user. Each button will display conditionally depending on which distribution channel is available. I’d take inspiration from GameJolt games and each game release, Github releases or a public MediaFire/Google Drive download…

If possible, and depending on the file type, a preview of the file should be shown to the end-user. If the document is only distributed through IPFS, then it would preview using its gateway. For BitTorrent, there’s no way to preview the file. For zip files, an AppView would be responsible of generating a preview of the content if possible.

image
image1345×810 11.5 KB

A social app should be able to search records based on the filename, size, date of publication, distribution methods or file type. Some social applications could be imageboard or video sharing websites and use those lexicons, if the developer finds it relevant.

2 Likes
2

It looks like there’s already skywell and dropb.at for file-sharing at the moment, could maybe look at what they use?

1 Like
3

Just tried them out with publishing a meme I stole from internet. Skywell supports description for files while Dropbat doesn’t, but it’s mostly the same thing.

However, it misses the core part of what I’d like : the file distribution method. Currently, it’s only a blob inside a PDS. I would like if there is a way to have multiple distribution methods for the same file, like IPFS, BitTorrent or an external URL…

1 Like
4

Hi @nezumi.eurosky.social, this is a very good idea and something I’ve been thinking about for a while (I’ve been an active contributor to IPFS for a long time). It’s also very close to something I’ve been working on, so I’ll share it here.

ATFISL is a lexicon for announcing where content can be downloaded. It complements FISL, a more generic HTTP interface for querying download locations for a CID that can be used as a query layer for an indexer which aggregates ATFISL records.

The record key is the file’s CID, and the record lists one or more URLs where you can get that file:

{
  "$type": "dev.fisl.at",
  "addrs": [
    { "url": "https://example.com/files/cat.jpg" },
    { "url": "magnet:?xt=urn:btih:…" },
    { "url": "https://mirror.example.com/cat.jpg", "expires": "2026-12-01T00:00:00Z" }
  ],
  "size": 94201
}

Some design ideas:

  • Many ways to download one file. The URL scheme tells the client how to fetch it. New methods don’t need a schema change, and clients skip schemes they don’t support.
  • CID (hash) is the identity. Clients check the bytes against the CID, so a bad mirror can waste time but can’t serve the wrong file
  • CID as record key. A getRecord to someone’s PDS answers “do they have locations for this file?” with no indexer. A put replaces the announcement and a delete retracts it, with no searching.
  • Optional expiry. Each URL can have its own expiry, which helps with dead links.

What it doesn’t do (by design) is the human side: filename, title, description, previews and search. I think that fits better in a separate record (maybe matadisco) or something like the one you proposed, which could refer to the file by CID.

Feedback is obviously welcome (see the tangled PR) and the landing page )! I’d be glad to collaborate if it’s useful for your project.

2 Likes
5

Just for awareness, there is also RASL.

1 Like
6

To pile on with reference to yet another similar thing, git-annex supports a bunch of “special remote” data storage systems, might be worth poking around for ideas. WebTorrent also comes to mind.

For large media, you could include thumbnail/preview as additional blobs, to make it easier for the appview to render.

Might be worth supporting more integrity checks (hashes), like SHA-1, MD5, blake3. Some of these are weaker than SHA-256 (what atproto CIDs use), but they can be helpful when cross-referencing against existing non-atproto databases. Or perceptual hashing; check out the work hypha coop has been doing with this on atproto.

As a best practice, it is nice for app services (appview) to take on bandwidth and resource costs off of PDS instances when possible. If serving a PDS-hosted file, it would be good to proxy it through the appview, and cache it in case a bunch of folks request the same data. I like that this proposal has all these alternative mechanisms which could help distribute resource costs.

3 Likes
7

I’d note that FISL/ATFISL are in the same DASL family as RASL. The key difference between ATFISL and RASL is that RASL defines a URL scheme for CIDs with the download locations based into the url in the ?hints query parameter, so discovery is static. ATFISL (as well as @nezumi.eurosky.social’s proposal) is dynamic, in the sense that you resolve download locations at the moment of retrieval.

RASL ([rasl]) answers this with inline hints, but hints are static: they are baked into the URL at the time of writing and go stale. FISL makes discovery dynamic: a client asks an indexer at the moment of retrieval, and the answer reflects what publishers currently announce.
FISL spec

The dynamic approach improves resilience at the expense of latency.

8

Hi folks! I’m AFK for a few days, but wanted to mention that I’ve developed atfs.dev for atproto-based admin of IPFS file storage.

It has a (still being developed) suggested schema for storing references to files stored on it in other records, dev.atfs.file (lexicon), which I’ve unhelpfully not yet added to the docs site! Eg.

$type: dev.atfs.file
cid: bafy…
ipfsRoot: bafy… # 
mimeType: image/png

Which, I admit, is a little confusing. The atproto side uses a DASL cid, which will be different from the IPFS root cid if the item is larger than the IPFS chunk size, normally 1MiB. (This is so atfs can still do reasonable bit swap, read more in my docs here.)

My main aim was to be allow folks to “follow” a friend’s atfs instance, and copy/backup/replicate what they were sharing, as atfs is primarily aimed at being run from a single board computer, like a raspberry pi zero!

I’ll have to spend more time reading this thread when I’m at a computer, but I hope this helps, I’m certainly interested in this!

1 Like