RetroGaming

They were broke.

The Dreamcast "managed to get one of the best libraries of that era" because Sega cranked out every genre at once, themselves, after third parties shunned the platform. EA publicly abandoned them before the machine even launched.

Nintendo can afford to stick with dodgy consoles because they have a pile of money the size of god. They are old and they are huge and they are shockingly conservative. They're still a toy company that happens to do video games.

Sega was an arcade giant that had a home-console side gig at the right time and place. They lucked out with the Genesis arriving well before the SNES (and picking the right CPU, holy shit, Nintendo) and thereafter made an impressive variety of terrible decisions. Customers got burned or burned-out on wonky add-ons leading up to the Saturn, which then kinda sucked. Developers got tired of paying Sega a Nintendo-sized cut for pitiful sales figures. Arcades quietly died out while the PS1 was eating absolutely everyone's lunch, so they had little to fall back on. By the time the Dreamcast launched they were basically fucked.

Really, the Dreamcast was the absolute best console Sega could have made at that time, and it did not help. Sony had a PS2 ad that just said: "Wait."