I'm dumb as fucking bricks brother, I have no idea where you're getting this idea that people think they're fucking geniuses for recognizing the Vatican flag. Saying it's the Vatican coat of arms is also redundant, because I imagine the cross section of people who don't know the Vatican flag and the people who don't know that the Catholic Church has an official country is rather large.

Even if you didn't intend it, your target audience for your joke extension became people who know of Vatican City and not its flag, which likely isn't a small amount of people; however, including the line "that's the Vatican coat of arms" makes the delivery worse for people who know of Vatican City and its flag, which sours the joke, and still excludes people who don't know of Vatican City.