Hardware

I heard this same stuff in the 90s about GPUs. "GPUs are too specialized and don't have the flexibility of CPUs".

Startups failing doesn't prove anything. There are dozens of startups and there will only be 2-4 winners. Of course MOST are going to fail. Moving in too early before things have settled down or too late after your competitors are too established are both guaranteed ways to fail.

In any case, algorithms and languages have a symbiotic relationship with hardware.

C is considered fast, but did you know that it SUCKS for old CISC ISAs? They are too irregular and make a lot of assumptions that don't mesh well with the compute model of C. C pls x86 is where things changed. x86 could be adapted to run C code well. C compilers then adapted to be fast on x86 then x86 adapted to run that compiled C code better then the loop goes round and round.

This is true for GPUs too. Apple's M1/M2 GPU design isn't fundamentally bad, but it is different from AMD/Nvidia, so the programmer's hardware assumptions and normal optimizations aren't effective. Same applies to some extent for Intel Xe where they've been spending huge amounts to "optimize" various games (most likely literally writing new code to replace the original game code with versions optimized for their ISA).

The same will happen to AI.

Imagine that one of those startups gets compute-in-SSD working. Now you can do compute on models that would require terabytes of RAM on a GPU. You could do massive amounts of TOPS on massive working sets using just a few watts of power on a device costing just a few hundred dollars. This is in stark contrast to a GPU costing tens of thousands of dollars and costing a fortune in power to run that can't even work on a model that big because the memory hierarchy is too slow.

Such a technology would warp the algorithms around it. You'll simply be told to "make it work" and creative people will find a way to harness that compute power -- especially as it is already naturally tuned to AI needs. Once that loop gets started in earnest, the cost of switching algorithms and running them on a GPU will be far too high. Over time it will be not just cost, but also ecosystem lockin.

I'm not saying that compute-in-memory will be the winner, but I'm quite certain that GPU is not because literally ALL of the prominent algorithms get faster and lower power with their specific ASIC accelerators.

Even if we accept the worst-case scenario and 2-4 approaches rise to the top and each requires a separate ASIC, the situation STILL favors the ASIC approach. We can support dozens of ISAs for dozens of purposes. We can certainly support 2-4 ISAs with 1-3 competitors for each.

First-up, here's a Veritasium breakdown of why a lot of next-gen AI is leaning into analog computing to save space and power while increasing total numbers of computations per second.

https://www.youtube.com/watch?v=GVsUOuSjvcg

The unreliability of analog makes it unsuited for the deterministic algorithms we normally run on computers, but doesn't have large negative effects on AI algorithms because of their low fidelity nature (and for some algorithms, getting some free entropy is actually a feature rather than a bug).

Here's an Asianometry breakdown of silicon photonics

https://www.youtube.com/watch?v=t0yj4hBDUsc

Silicon Photonics is the use of light between transistors. It's been in research for decades and is already seeing limited use in some networking applications. IBM in particular has been researching this for a very long time in hopes of solving some chip communication issues, but there are a lot of technical issues to solve to put billions of these things in a CPU.

AI changed the equation because it allows analog compute. A multiply generally takes 4-5 cycles with each cycle doing a bunch of shift then add operations in series. With silicon photonics, this is as simple as turning on two emitters, merging the light, then recording the output. If you want to multiply 10 numbers together, you can do it in ONE cycle instead of 40-50 on a normal chip (not including all the setup instructions likely needed by that normal multiplier circuit).

Here's a quick IBM explainer on in-memory compute.

https://www.youtube.com/watch?v=BTnr8z-ePR4

Basically, it takes several times more energy to move two numbers into a CPU than it does to add them together. Ohm's law allows us to do analog multiplication by connecting various resistors and measuring the output.

You can use this to do calculations and the beauty is that your data hardly has to travel at all and you were already having to use energy to refresh it frequently anyway. The total clockspeed is far lower due to physics limitations of capacitors, but if you can be calculating every single cell of a multi-terabyte matrix at the same time, that really doesn't matter as your total compute power will be massively faster in aggregate AND use several times less power.

Of course, all these analog alternatives have absolutely nothing in common with modern GPUs, but simple operations are massively more power efficient with in-memory compute and complex operations are massively more power efficient with silicon photonics.