Your DE won't use that much cpu no matter which you use.
As long as you want to run out of ram you should be fine.
But imo it's a bad idea to mix a server os with a normal desktop use.
How about using proxmox and having a vm just for desktop use, like this you could also just pause all the resources easily when you're not using your desktop computer. Personally i would have a debian stable and a proxomx vm and just enable the proxmox vm whenever I need it.
If your not scared of scripting you could also unsuspend the vm via a keyboard listener. (for example trigger unsuspend when key f3 is pressed. This would be doable with evtest and qm resume )
Or if you don't want to use a vm you could also disable the autostart of the DE and start it via shell when you really need it.