I have a lot of indirect experience/knowledge with caves. I don't have to be one of the people who directly explored one to know one of the first things one learns about caves is how ill-suited for wandering around they are. Slopes can change on a dime, it's incredibly rocky inside, and they fill with whatever falls from the sky. Imagine hiding in one of these, as someone with less intelligence than us, and not expecting to even stub a toe, let alone fall or suffocate. I assume it must be a concern because all the cave hobbyists I know (even if I only know a couple) say they have to split up based on their physical skills (so one chooses a slope-free while the other a rain-free one as would be the case for me as someone who never learned), though I admit I'd be amused if cavepeople didn't ignore these slopey checkpoints and instead it caused them to make some inventions.
Another thing that sticks out to me is the rule against fire. When cave explorers (not me) venture in, the most important rule of all time they learn is that it's an absolute cardinal sin to light so much as a match in a cave, let alone a torch (in contrast to Indiana Jones movies where that's the first darn thing he does). The heat from fire is enough to interfere with the stone composition, which in turn threatens to collapse a cave. Imagine having just discovered fire, and you go running to show your family but everyone dies before you can say "hey Mario look what I made". I wouldn't expect a caveman to know about that rule, but I would expect them to feel tempted to find out the hard way.
Finally, there's the fact they're filled with disease. Most notably from the cave animals; while things like ticks and rabies are not common in cave animals, they do happen. If that wasn't enough to outright stigmatize dwelling in a cave, even the environment itself is viral. There are caves where the reservoirs are like 100% condensed bacteria. That's got to send awful mixed messages to seek refuge from an oasis and suddenly you have a fear of water like me.
How did caves become such a go-to and one where nobody is depicted as having any serious accidents in?
That was 7ft flames in a sandstone cave. Yes it can happen. No it wouldn’t happen often enough to have killed every single human living in a cave system.
Not every one in every part, no. It's just a hazard per section.