This is no longer true.
If you use NV's TensorRT plugin with the A1111 development branch, TensorRT works very well with SDXL (it's actually much less painful to use than SD1.5 TensorRT was initially).
The big constraint is VRAM capacity. I can use it for 1024x1024 (and similar-total-pixel-count) SDXL generations on my 4090, but can't go much beyond that without tiling (though that is generally what you do anyway for larger resolutions).
Just like for SD1.5, TensorRT speeds up generation by almost a factor of 2 for SDXL (compared to an "optimized" baseline using SDP).
Specialized hardware can make sense for inference of known networks, or actually a bit more broadly, network structures. But for training and research, the structure of models still seems to be in too much flux for specialization much beyond the level of a modern GPU to make sense. For now at least.
The (research) ecosystem would have to settle down a it before you can get a really substantial improvement out o specialization, and be confident that your architecture can still run the latest and greatest in ~2 years after you designed it.