Full programmtic migration of LW communities.

submitted 6 months ago by

edited 6 months ago

After yet another encounter with a tankie roach, I am increasingly getting annoyed with Lemmy (the frontend/platform, not the community).

I mod/curate:
!tycoon@lemmy.world
!crpg@lemmy.world

Is there a way to fully move them to piefed.world?

I've moved lemm.ee community to piefed.social but not all posts/comments went through and most importantly, the subscribers didn't move over.

Since World controls both the LW and PW DBs, is there a way for a full programmatic migration?

Hot Top New Old

MrKaplan@piefed.world

6 months ago

Hello,

we currently do not use PieFed's "move community" feature. I've looked into it a bit, but I'm currently not satisfied with how it works to start using it.

What happens in PieFed community move:

Community gets marked as local in the database
A fresh RSA keypair is generated for the community
The community is renamed to the desired new local name
A community owner is set if provided
The previous list of changes is committed to the DB
A background task starts to migrate contents to the local instance
1. Posts and comments in the community are changed to be local, replacing instance id and AP ids to be local
2. Non-deleted image posts in community are identified, for each one of those posts
  1. The image is downloaded to a temporary path
  2. If the response code for the download was 200 and the content type of the response matched image/*
    1. The image is uploaded to S3
    2. The temporary file gets deleted
    3. The post image URL is updated to the new S3 image URL

Some consequences/limitations of this:

All history of the old community known to the instance performing the move will be migrated to the new local community. Content that may have been posted prior to the moving instance having a subscriber for the community is usually not migrated.
Post and comment creators no longer "own" the content they posted to the original community, so they will no longer be able to edit or delete their posts and comments, unless they're on the same instance as the new community.
This does not allow you to merge a remote community into an existing local one, so if the desired destination community already exists, it will prevent moving the community to that name.
This does not federate, other instances will see the moved community as an entirely separate new community. Other instances will not migrate followers (subscribers).
The old community can probably be federated again and continue to be actively used, which can lead to duplication of the content that was posted there previously. I haven't tested this yet, but reading the code it looks like this should work.

From the technical perspective, we would be able to migrate all community followers (subscribers) on Lemmy.World or PieFed.World to a different community by updating this in the database.

While at this point we are not ready to use PieFed's community move feature, we could probably move subscribers on both Lemmy.World and PieFed.World over to the new community. This will require some preparation, but I don't see a reason why we wouldn't want to do this.

FYI also @AbNormalHumanBeing@piefed.world, as you requested something similar recently, which was what originally prompted me to spend some time on investigating this.