Imagine your example of an empty window frame, but in an empty void, with no points of reference other than the important objects. You could see this interaction in two ways. The frame moves towards the people, around the people, and then away from the people or the people move towards the frame, through the frame, and than away from the frame. This is the same interaction. It does not matter which is "moving", either one results in the same interaction.

In this portal case, you combine the two methods of observing the interaction. The portal moves towards the people. The people go through the portal. The interaction now says the portal moves away from the people, but this second portal is stationary. So you flip how you look at it. The portal now "moves" away from the people by the people moving away from the portal.

Hopefully this makes sense.