Testing does seem like the best solution.

This article indicates that they're harmful, but the explanation as to why is not obvious. They stress that asphalt is a petroleum product, but plastic is also a petroleum product, and while there is some concern about microwaving food in plastic containers, it's widely regarded as a negligible public health concern. If plastic wasn't largely inert and non-reactive, carbonated soda (which is acidic) wouldn't be sold in them.

Asphalt is linked to lung cancer in roofers and road workers - but it seems that's because they're exposed to it in a heated and aerosolized form. Any particulate that changed from liquid to solid in your lungs on a microscopic scale is likely to cause cancer regardless of its other physical properties or chemical origin. A minority of microscopic solid asphalt particulate in your food seems less likely to have serious health effects. Having microscopic petroleum products in your food is the status quo, has been since the invention of polyester thread, famous for shedding microplastic.

The other mechanisms for contamination - settled chemical particulate and bird poop on your roof getting washed into your garden - sound like problems not unique to asphalt roofing. This paper seems to indicate that while not drinkable, most roof runoff is usable for gardening. Rain water itself can contain small amounts of heavy metals depending on the industry in your area, but zinc is the only harmful substance some roofs are adding to the mix. If your runoff doesn't contain more zinc than non-roof rainfall, then you're probably fine.