Proposal: Atomizer, a library of data converters and services

IndieSky
1

@erlend.sh reminded me of glue today, so let’s get this idea out there.

Thanks to the good people in the EU, incumbent platforms are required to provide some kind of data export for their users. I was able to bring my entire Google Photos library into Immich because of Google Takeout. Facebook, Instagram, Twitter, all of the big platforms offer some version of this. As we grow the Atmosphere, people moving over their social media accounts will want to bring this data with them. Starting a new account with your full post archive is a much better experience than starting with a blank slate. How might we make this easy for users?

Atomizer, a rosetta stone of takeout zips and lexicon

I propose a community project to build tooling and services which convert between lexicon and takeout archives.

Data Import

Accept these data takeout archives and convert them into various appropriate lexicon. Atomizer would contain a series of libraries/functions for converting common compatible data, such as:

  • Instagram → Flashes | Bluesky | Spark
  • Twitter → Bluesky | Anisota
  • Substack → Leaflet | Whitewind | PiPup

Libraries would provide standard logic for interpreting these archives and writing them to the PDS.

Data Conversion

The second component is for converting between similar lexicon. ATproto facilitates competition, and we have seen many emerging lexicon for the same kinds of applications, each with different trade-offs and design decisions. Luckily, public JSON files of similar schemas are not hard to translate. I propose the second aspect of Atomizer provide tooling for conversion between similar lexicon to facilitate data import and compatibility between competing apps on the Atmosphere, such as:

  • Bluesky ↔ Anisota
  • Bluesky ↔ Flashes ↔ Spark
  • Leaflet ↔ Whitewind ↔ PiPup ↔ Pckt
  • Community Bookmarks ↔ Monomark ↔ Semble

Not all similar conversions will maintain full data integrity. Sufficient warning of data loss will be needed for conversions that will not maintain all data. Which brings me to the third component.

Services

These public libraries should enable the creation of data import and conversion services. The project itself can have a flagship service on a website, allowing users to submit their takeout file and upload the converted lexicon to their PDS. Or, choose a collection on your PDS to convert to another. Want to move from Whitewind to Leaflet? Go to this website, select that conversion, click go.

Beyond a community hosted service, the open libraries would allow any Atmosphere developers to integrate import and conversion services into their own apps. What if you could give your Instagram takeout to Flashes and they import all your posts for you? What if Pckt’s onboarding had an option to convert all your old Pipup posts? What if Blacksky let you import your Twitter archive? What if you want to set up your own online conversion service separate from the community instance, with different technical features. You might offer a paid service which syncs between your Substack and Leaflet accounts using a cloud service. The tooling is shared infrastructure for the whole community.

What do we need?

I would love feedback on this concept overall. I am not a technical person, so especially thoughts on how to architect such a collection of tooling would be appreciated. After something like this is built, would there be community funding to operate it? Can we fund it’s development? Can it be built in a way that minimizes costs, such as client side logic (thinking like PDSMoover)? Would such a community resource be useful to you as a developer? What user stories would this enable that you wish were easier?

4 Likes
2

probably the biggest problem you’ll encounter with this is the rate limits around writes, which is like 1666 per day or something.

3

Great idea! Sounds like https://granary.io/ and/or Project Cambria: Translate your data with lenses

2 Likes
4

Yup. I’m only doing following lists for ATlast and I’ve included my own rate limits (follow up to 1k users) so users aren’t surprised they can’t like or post after following a bunch of accounts.

Is there a way to know if an account has been rate limited? That would help devs include that info in their converters and apps. I know when I got rate limited there wasn’t anything shown to me in bsky.app - just couldn’t like or create posts.

5

Rate Limits | Bluesky ?

6

yes these seem like the right idea. Needs excellent UX and DX

1 Like
7

One mushroom-hosted alt account later and some testing, Granary is quite out of date and novice in ATproto integration. Maybe a service could be built on top of it. Also seems they are just a python library. I imagine ATdevs would want npm packages, rust crates, deno, etc. Perhaps it can provide a kickstart and we can make it more at home in the atmosphere for our needs.

8

The problem is that - given the architecture of ATProto - you may cause trouble downstream if you throw everything at once at the firehose. Hence the rate limits.
To run this as a service for other you will also run into IP-based rate limits afaik. So the solution I’m working on for Flashes is to run the import in batches on the device, carefully monitoring the rate limits in the headers.

1 Like
9

That should still be feasible tho? Even if it’s all clientside and in a background process, like syncing your icloud photos. Rate limits are fair, just a design constraint. Also, depending on the service, the zip archive could be large (most social archives aren’t, but what if I’m migrating from flickr to Grain.social?).

10

Or, hacky idea I have no idea if it has legs, what if you restored the whole repo at once? When you migrate to another PDS, you upload the whole CAR and blobs. What if you download the CAR, assemble your new content into lexicon, reassemble the CAR, then restore the account to the same PDS?

11

Sounds to me like a cli would be the best fit? These takeouts are likely on your main computer, the cli would be there, you’d authorize it, point it at the takeout dir, and then let it chug along at a rate that would be guaranteed to be within rate limits. Would that be enough? Or are you set on this having some sort of (web) UI?

12

Yes that’s a fairly elegant solution for a certain set of users. I’ve done it for uploading an Instagram archive as Bluesky data. They are usually piecemeal scripts that do only one kind of conversion. I want a website for normies. I also want a comprehensive library so that if cli is your jam you can convert all sorts of stuff in one package.

13

Hmm! granary has been the conversion logic for Bridgy Fed and a few other services I run, aomng others by other people, for years. I’m always open to bug reports and feature requests, feel free to let me know if you have anything specific!

14

My take on granary was likely unfair. I tried using the demo on the site and it refused my non mycosphere account and relied on app passwords. If y’all are using it for bridgy I’m sure it’s more than capable.

15

Ah, makes sense. No worries! The demo UI is definitely dated and unloved, agreed. I’d love to get it onto OAuth at some point. The library isn’t dependent on app passwords and fully supports all PDSes, largely based on https://lexrpc.readthedocs.io/ and https://arroba.readthedocs.io/.

16

Resurfacing this idea. I have a more detailed proposal now of how to implement this idea of shared importing tooling. Would love to hear y’all’s thoughts. Disclaimer that this document was a collaboration with claude haiku and opus, though it captures my idea and intent well.

2 Likes
17

Yeah it’s a bit of a slog to read.

I get what you’re getting at but can you in your own words answer “why”.

I shipped robot assisted Ghost to Offprint. Didn’t feel the need to put it in an intermediate lexicon.

I used Cyd to import my twitter archive, still might run it again in the future with the copy I saved. I don’t think this improved my experience or did much.

The format of exports might change… so the parser just changes.

This is also very much a product decision for an app. Would it pay off strategically for an app to implement Instagram import? Yeah probably but first you’d need to build an app that would appeal to Instagram shaped posts.

That’s also writing a lot, sorry!

I just don’t see the need or momentum for a bunch of people to get together to do it — just ship it as a product feature and see who actually wants it.

1 Like
18

Thanks for taking a look. The main why behind it is two things I believe are true:

  1. There are legacy platforms from which people will want to bring their existing content with them to the atmosphere
  2. The original content from the legacy platforms will not necessarily convert to new lexicons losslessly.

Therefore, there is value in creating lossless lexicons to store the legacy content on protocol. Sure, it is pretty low cost to roll an importer for any two formats. I would still say there is value to developers to not have to reroll importing logic for every takeout format which might be useful to them.

As for why the user might benefit, I agree twitter exports aren’t all that useful. There are many legacy apps that are valuable and I want to keep my stuff from there. Instapaper, Pinterest, tumblr, Flickr, SoundCloud, Spotify, Letterboxd, just to name a few. As a user if I only have to import my history once from each legacy app, and any new app can decide how to lens that into their schemas if they wish, creates value for me and saves everyone time and effort. Maybe a BeaconBits competitor comes along which uses more of the old Foursquare data than BB does. Better if I don’t have to import my FSQ data twice.

Many of these should certainly be imported as permissioned data. That’s why the initial scope is just tooling to convert whatever junk pile takeout the platform spits out into lossless lexicon (since permission spaces will use the same lexicons), leaving the product design for later. In an actual importing service, as @gui.do pointed out on the skyline, you need to be careful with consent and PII. One could even limit which takeouts to support in Atomizer to mitigate harmful use of the library.

My hope would be that developers would contribute lexicons and import parsers from lots of legacy apps that I haven’t even heard of. And on top of that create panproto lenses to make those archives useful in the atmospheric apps which make sense to include that content.

TLDR I do think lossless archive on protocol has value and I agree that Instagram and twitter are not likely high value archives for most people.

1 Like
19

Twitter might be valuable for some! I was telling my personal story. And I wrote that Instagram WOULD be valuable, just that it’s going to be an atproto app that is shaped like that which is needed.

You have a hypothesis. Now building a proof of concept, maybe surveying users about what apps they want supported, and soliciting other builders to work on it.

Next step, you’re looking for collaborators?

PS your short reply is more readable than the longer doc!

Hope that’s helpful!

1 Like
20

In addition to my comments on bsky:

A) I kinda doubt people will just import their stuff if there is no app reading it.

This might be a bit of a chicken-egg problem that needs solving. Like Why should I import all my instagram/twitter/linkedin/threads/whatever content if I can’t actually use it or point my connections to?

B) BigTech companies do not like exporting user data and the only reason most have it is because the EU forced them to do so. So they are trying to do this with minimum effort, barely compliant. There is 0 benefit to them to make the export process easy or complete PLUS when it comes to social apps: they obviously cannot give you data from other users (GDPR/PII and all). So the rich info you might have in the app, is not what will be in your export.

With the Linkedin export specifically:

  • some sections are just missing. Data you enter in their app, but that is not in the export
  • the export might consist of different parts and might not be consistent between users
  • the export might take a long time to generate, completely taking away your momentum when you finally convinced a user to do it. And this is completely out of your control.
  • you might actually not get the export. For Linkedin the export consists of 2 parts: the first one reliably lands in your inbox within 3 minutes. The second one may or may not arrive after a couple days. In over 50% of the requests, the 2nd one just never arrives.
  • “Connections” (obviously) don’t contain the users atproto DID:PLC or handle. So if you want to import those…. good luck with that.

To be clear: I do think this is a good idea and would be great to have this around. Just have some reservations about how to start it up and get broad adopton, mainly because how restricted BigTech exports are in my experience.

2 Likes