@ThisIsAManWhoKnowsHowToGling +
For tightly woven natural fibers, I’d go with a quilting cotton, cotton flannel shirting, or, if you can get a good price for it, midweight wool suiting that you’ve lightly fulled in the wash (wash warm, tumble dry a couple times, unfolding it all the way in between cycles to make sure it doesn’t stick to itself).
Prewash the cotton too (on hot) to make sure it’s done most of the shrinking it’s going to do before you sew.
@ThisIsAManWhoKnowsHowToGling for wind resistance you either want something tightly woven or something plastic.
The plastics are going to be a little noisy, though if you sandwich them like you describe you’ll end up with crinkle but no swish sounds so that may be enough depending on your needs. For that, use nylon lining fabric, woven, “ripstop” keyword optional.
+