One of many reasons for this could be your choice of topics, the location of the argument, and your perception. Any Centrists in the argument may be on your side of that particular issue and therefore are appearing left-leaning to you at the moment. See Penn jillette, Richard Dawkins, Bill Maher and scads of psychologists authors for example of centrists who often appear to be one side or the other depending on the issue presented (and have been mislabeled as both sides by people not willing to understand nuance).

You not looking for them does not mean that they are not there. Them not coming out and identifying themselves continually does not mean they are not there.

Most of the Centrists I've spoken to also tend to argue the issue, and not the side. They see sides and at-all-costs group membership as a form of lunacy.

Just like anything in the world, your perception is not always reality, especially when it seems that you are looking to enforce an opinion, not find the truth.