DeepSWE and FrontierCode are more realistic if you read up on what they actually measure. But the most realistic is to try it yourself. Benchmarks can only vaguely represent typical usage, and how you judge the result. Giving the same real task you have to a few models will make you understand them better than chasing benchmarks.
When the model is open, then anybody can download and trying doing things with it. Distill it, change its architecture, inspect how its layers work, and so on. A lot of research ends up being published as a result which the original authors of the model can integrate back to improve it. This is precisely why China now tops AI publications https://www.science.org/content/article/china-tops-world-art...
The models themselves aren't what's valuable. The goal is to get the models to become the global standard which everyone uses, and people are familiar with. This further ensures that standards and hardware will be developed around these models going forward. It ensure these models are what most people are familiar with using, and so on. This becomes common infrastructure like Linux that's effectively impossible for commercial offerings to compete with.
The money is going to come from providing custm integrations, customization, robotics applications, and operating cloud services like AWS. That's where Chinese companies are aiming. There's a good interview with Alibaba Cloud founder where he explains the strategy. https://www.youtube.com/watch?v=JPzevOpIzPg
While American companies are betting on the idea that if one model can pull away it's going to keep self improving and nobody will catch up, Chinese companies are betting there will be a plateau to this tech, and it's more important to focus on market dominance because they will catch up later.
And it's becoming clear that Chinese companies made the right bet because Chinese models are closing the gap now, which means there is no singularity effect being observed with frontier models. We are starting to get to the point of diminishing returns already where companies have to put ever more effort into squeezing just a bit more capability. The easy gains appear to be over.
1. If LLMs keep improving, burning models onto silicon becomes obsolete too fast and is not worth doing. Outcome: We keep getting better LLMs.
2. If LLM improvements slow down, they will be burned onto silicon. Outcome: We get faster, cheaper and energy-efficient LLMs.
Either way sounds great to me. It will certainly be a mix so we can even get both.
Looking at some benchmarks, the latest ~30B Gemma/Qwen score similar as Claude or GPT versions that were released just one year earlier. That's crazy progress. I can't imagine how it will be in a few years.
I think this is inevitable. Sooner or later, model-specific ASIC's will make economical sense. We're already seeing it happening with Taalas/Cerebras so I think it's sooner than 5 years. And inference is order of magnitude faster which is amazing.
This seems extremely inefficient considering data transfer between model layers if the model is distributed. I found this project called Petals that claim up to 4 tok/s for a 180B model although its repository hasn't been updated in two years.
I think their "code" ranking is biased towards visual aesthetics more than raw coding as the voters are just asked which generated website they prefer.
reply