For mammals we are, sure, but there's loads of things that'd kill humans that other animals chow down on perfectly happily, especially when it comes to microorganisms, mushrooms and the rotting things they're often found in/around

I don't think scavenging is right also given that humans used to mainly pick fresh fruits and persistence hunt, both of which are very fresh food which is not overlooked or left by others... Given the fact we picked fresh fruits and hunted for fresh meat, being resistant to berry and fruit based poisons was more important than microorganism based ones, so it makes a lot of sense that so many of the non-intoxicating poisons we like are from fruits and berries