I read both. I tend to forget the nonfiction and the low tier fiction pretty quickly. The good fiction stays with me.
There's also a lot of truly terrible nonfiction bestsellers. (Fans of the "If Books Could Kill" podcast know what I mean).
There's no reason to feel guilty, IMO, unless you're failing school or something.