Using S3 for blob storage on PDS

Resources & Learning
,
1

Out of the box, the Bluesky PDS stores files (aka blobs) on disk.

It also supports storing them in S3-compatible cloud storage.

Where files are stored is in the env variable PDS_BLOBSTORE_DISK_LOCATION. This needs to get unset.

Then the following env variables need to be filled in with the info from your S3 host.

Aside from Amazon, there are a number of services that are S3 compatible.

PDS_BLOBSTORE_S3_BUCKET
PDS_BLOBSTORE_S3_REGION
PDS_BLOBSTORE_S3_ENDPOINT
PDS_BLOBSTORE_S3_ACCESS_KEY_ID
PDS_BLOBSTORE_S3_SECRET_ACCESS_KEY
PDS_BLOBSTORE_S3_FORCE_PATH_STYLE

Resources

https://baileytownsend.dev/articles/s3-blob-store

@pds.dad has details here for Upcloud and Digital Ocean spaces, and has also written up migrating files to S3 using rclone.

8 Likes
2

If you’re on Hetzner, you can use their object storage as well quite easily. Create a bucket, and get credentials for it, and alter the PDS configuration as follows:

	PDS_BLOBSTORE_S3_BUCKET="your-bucket-name"
	PDS_BLOBSTORE_S3_REGION="region-at-hetzner" # e.g. fsn1, nbg1, hel1, etc.
  	PDS_BLOBSTORE_S3_ENDPOINT="https://region-at-hetzner.your-objectstorage.com"
	PDS_BLOBSTORE_S3_FORCE_PATH_STYLE="true"

The “region-at-hetzner” bit is the DC location, such as nbg1, fsn1, hel1, and the others. You will also find the full S3 endpoint URL mentioned in the Console if you’re looking at your bucket.

The only thing left to do is restart your PDS. Although you probably want to stop it, use an appropriate tool to copy the existing on-disk blobs to the S3 bucket, and then start it.

4 Likes
3

this just helped me huge-time to get some PDS hosting test stuff up and running on Hetzner! there is now a test blob of a cute cat sitting in my test bucket — thanks y’all

2 Likes
4

Thanks for dropping by and saying it was helpful!

1 Like
5

of course! super happy to see so much knowledge and so many resources in this community, what a fun context to develop in.

and excited for the day I run into a problem that someone else hasn’t figured out yet, and get to contribute my own :grin: (hopefully :sweat_smile:)

1 Like
6

Using rclone is pretty easy to migrate files to S3 from disk, provided same directory layout. Not sure if the PDS implementation stores full URLs to blobs or just path in the database.

7

Don’t hold me to it (because it’s been ages) but I think the layout is the same, IIRC I just used s3cmd’s mirror command to get things into the bucket :slight_smile: I think the full URL is just done via getBlob with the CID and internally that’s converted to a path (IIRC did/cid) that the storage driver then figures out what to do with.