Not sure what you're trying to do - Thunderbolt can be daisy chained so not sure why would need to put a TB card in the dock.
The enclosure is generally transparent to the OS. It just sees the devices plugged.
TechnoTim on youtube a video on rack mounting his Mac Mini (M1 model) and used an external chassis and tried different cards without issue.
You're probably going to limited by Apple's support (for example you can't put in a nVidia graphics card in).