Defederation happens on an instance to instance level. There will always be instances which do not approve of those who conduct themselves differently. It is not worth it to make decisions based on threat of defederation, because you essentially cede community control to those who would make those threats.
I'm not saying the community in question should be free to be assholes, but if they are to be banned, it should be because they are breaking the rules of this instance, not anyone else's. In my opinion, a better way to frame the discussion is "should we allow this content/these users?"
It goes deeper than that, according to IMDb there are 97 people who appear in both Mad Men and LA Noire: