If you cut down a GPU enough, the CPU doesn’t become good, the GPU becomes bad.
Double precision fp (yes a different metric but the main one super computers are concerned about) performance of Nvidia 4080 is 761.52GFlops. Ryzen 9 7950x is about 256GFlops, so on that one metric, it’s not like the GPUs are orders of magnitude behind. At a certain point, using a cut down gaming card won’t make sense.
These gaming cards are often crippled at both the software (fixable) and hardware level when it comes to compute capabilities. If the gaming card is 1/4th or 1/32 or 1/256 as good as an actual compute card, how long before it’s better to use something home built? At a certain point, even normal amd and Intel cpus will be faster.
This applies more to double precision floating point (useful for nuclear simulations), but the ML hardware on gaming cards is crippled compared to the compute cards regardless.
There’s going to be a crossover point where the cut over from gaming chips used for AI accelerators will be worse than what china can build domestically.
Home built meaning domestically in China, not in a home.