Lemmy is still young. If you move your community now, pin an announcement on top with a link to the new one, and lock the old one, you'll probably have almost everyone following you. I did it for one of my communities, the subscribers number of the new one is almost identical to the old one.
It might be better to do it now than in a few weeks/months when your communities are much larger.
[email protected]