Hardware

CPUs and GPUs are both Turing complete, so from a computation perspective, they can technically do the same things. However each is optimized for a different set of metrics, so it may be wise to target a particular workload to one over the other.

The CPU is optimized for minimum latency, aka maximum single threaded performance. This is really important for a lot of workloads.

GPUs are optimized for maximum throughput and are willing to suffer higher latency to achieve it. There are many architectural differences as a result of this goal, for example: GPUs don't have branch prediction, they also don't have out of order processing, and also have much smaller caches per computational unit. All of these saves a ton of transistors, which can be used to add more cores.