And the latest Kimi is better than Fable 5. Another plane has indeed hit the Anthropic towers.
It does not seem to be better except in terms of cost
It does not seem to be better
except in terms of cost
Except cost. Except open source. Except the entire fucking global economic trillion dollar US AI model.
Those are very big exceptions.
(Also, I’m mostly stealing from Yσɠƚԋσʂ’s post.)
That’s one benchmark that they focused on, but having double checked, you’re right, I was thinking of this one where it lags behind gpt 5.6 https://deepswe.datacurve.ai/
I’m just waiting for the closed-source AI industry to have their Black Swan moment.
Something’s going to come out of left field from the open AI community and all that investment in proprietary models and mega-compute is going to be rendered useless.
You still need the mega-compute. Even for inference, 1.5 terabytes of RAM in modern servers isn’t cheap.
You can run Claude Sonnet equivalent quantised models locally on much less RAM.
Local LLMs are close to being viable. We’re almost at the point where they fit a Macbook.
You’re not thinking black-swan enough.
You’re typing your comments using a blob of goo with about a hundred million neurons in it that cycles under a hundred hertz and draws less than 20 watts.
I don’t think that we’ll be running packs of goo in our PCs any time soon. But I do think some entirely different way of looking at the problem will emerge that will reduce computational requirements by many orders of magnitude. And it won’t involve gigantic statistical engines trying to find the best average response to a question.
There are 86 billion neurons in the human brain and 16 billions in the cerebral cortex. More importantly, there are 100-1000 trillion synapses, and we know that parts the dendritic tree do independent computations as well. The computations are not synchronized with a global clock, but 1 ms events correspond to a kHz refresh rate and we know spikes can do temporal coding. Meanwhile, the best we can do is Cerebras CS-3 and with WSI at 5 nm there isn’t much more where that came from.
Try doing the math on how many CS-3 you’d need to represent and refresh the above.
It could be TurboQuant or something like it. The biggest detraction to local LLM models is being able to close the gulf between obscenely-expensive 512GB NPUs, to house the 230GB uncompressed models (+ context), and more common 24GB GPUs. Quantized 15-18GB models are already working pretty well, but context size is still a bit of a problem.
Of course, the whole industry need to ramp up memory production and wrestle duopolies from the few that can make the raw silicon. It was pretty fucking pathetic that parts of the PC industry decided to leave these silicon processing weaknesses in various places. Large corps could have easily jumped into the industry and made bank in the long-term, but that would require not funneling into short-term quarterly profit bullshit.
the Chinese would know they fucking copied it.
Cope
Not really. I’m glad that China is copying AI models and providing them for cheaper usage or open sourcing the training.
I don’t believe any US company should monopolize the entirety of human knowledge.
I gotta know what you mean by “copying” ai models. Same training data? Amount of training? How would you go about copying a closed-source model. I think china is just…. Making good models.
Distillation is a technique where an older “teacher” AI model is used to train a newer, “student,” model that replicates the capabilities of the earlier system — often at a much lower cost than producing an original model from scratch. Some forms of distillation are widely accepted and even encouraged by AI labs, such as when companies create smaller, more efficient versions of their own models, or allow outside developers to use distillation to build non-competitive technologies.
Basically one AI asks questions and records the answers to recreate training data without massive compute for training data.
I see and agree with you then. I’m glad it gets us better cheap models but more traditional innovation would definitely be preferred.
You can’t distill models that fast lmfao. People just keep parroting this without having any clue how any of this works.
It’s not that complicated. Recreating training data via distillation is basically asking structured questions and recording the responses and reformatting that to use as cleaned “good” training data. Much less energy and compute intensive than creating the training data on your own.
I think I remeber reading somewhere how Chinese research’s do this by basically using bots and spreading out the distillation to many source queries.
The process takes time because even when you’re distilling answers, you still need to actually do reinforcement training on the model. And given that Fable and GPT 5.6 just came out there simply hasn’t been much time to do that. However, models like Kimi also do better than Fable or GPT on a lot of tasks, which means it’s not just distillation but also difference in architecture. You can watch this talk from Kimi founder to see how Kimi was actually trained and why it performs well.
It’s also absolutely hilarious that people think only Chinese companies use distillation, as if Anthropic or OpenAI are above that or something. Not to mention that they basically ignored copyrights on all the data the siphoned and are now crying that people aren’t respecting their terms of use.
Yeah I know they’re not the only companies doing distillation, it’s just currently in the news and on peoples minds.
fuck AI
Sir, this is a
Wendy’stechnology forum.Indeed, and as Cory Doctorow says: “The most important thing about a gadget isn’t what it does; it’s who it does it for and who it does it to.”
Tap for spoiler
Fuck AI
so brave
