What level of involvement in producing the food makes it vegan or not vegan?

It's about A) exploitation and B) harming the animal.

Pollination is done by all kinds of insects, but they are part of our ecosystem and happen to be pollinating the plants that we eat. We don't breed them, we don't kill them (pesticides, sure), we simply coexist.

Honey isn't vegan because we breed the bees, take their food and often kill the entire hive because they get sick and cannot survive winter without their honey. It's also not sustainable, because honey bees are being bred en masse and are pushing out native pollinators that are highly specialized in certain kinds of plants, causing them to go extinct.