Hardware

The most important thing about GPU cores is that they are parallel in nature. A lot of GPUs out there use 1024-bit arithmetic units that can process 32 numbers at the same time. That is, if you do something like a + b, both a and b are "vectors" consisting of 32 numbers. Since a GPU is built to process large amount of data simultaneously, for example shading all pixels in a triangle, this is optimal design that has good balance between cost, performance, and power consumption.

But the parallel design of GPU units also means that they will have problems if you want more execution granularity. For example in common control logic like "if condition is true do x, otherwise do y", especially if both x and y are complex things. Remember that GPUs really want to do the same thing for 32 items at a time, if you don't have that many things to work with, their efficiency will suffer. So a lot of common problem solutions that are formulated with "one value at a time" approach in mind won't translate directly to a GPU. For example, sorting. On a CPU it's easy to compare numbers and put them in sorted order. On a GPU you want to compare and order hundreds or even thousands of numbers simultaneously to get good performance, and it's much more difficult to design a program that will do it.

If you are talking about math specifically, well, it depends on the GPU. Modern GPUs are very well optimised for many operations and have native instructions to compute trigonometric functions (sin, cos), exponential functions and logarithms, as well as do complex bit manipulation. They also natively support a range of data values such as 32- and 16-bit floating point values. But 64-bit floating point value (double) support is usually lacking (either low performance or missing entirely).