Industry group invalidates 2,600 official Intel CPU benchmarks
https://www.tomshardware.com/pc-components/cpus/spec-invalidates-2600-intel-cpu-benchmarks-says-companys-compiler-used-unfair-optimizations-that-boosted-performance
8 replies (most recent on top)
This is more an issue with synthetic benchmarks, but a synthetic benchmark company can’t really come out and say that. An optimization that improves something and hurts nothing isn’t a cheat. This is a good reminder though, test with real workloads and not fake ones.
It doesn’t matter.
Hyperscalers run their own internal benchmarks and SPEC is irrelevant.
They’re overwhelmingly choosing AMD Epyc now and for good reason. Intel is in serious trouble in server land.
Intel keeps removing features and putting up pay walls for old features... Low to mid end cpu's no longer have hyperthreading and unless you buy a ridiculous high end board their chipsets don't even support standard bifurcation 4x4 like even the cheapest AMD ones... Losing both consumers and enterprise... The last few enterprise solution I've built were AMD. Dollar for dollar having many more cores was much more advantageous for horizontal scaling and container environments.
What is wrong with compiler optimizations? It is very normal for developers to use optimizations. They say “real world performance”. Whether performance is real world or not depends on whether workload is real world or not. Not whether the software is optimized or not.
The reason given by SPECInt doesn’t sound normal. It is not black and white. Some underhandedness by SPECInt here.
NEX is also notorious for releasing benchmarks obtained using highly modified and favorable conditions for their Ethernet HW.
Cheating on benchmarks is a time-honored Intel compiler tradition. No news here.
Not surprising. DSE is a notoriously crooked division that has been draining Intel funding for years with little value coming from their efforts. Will be interesting to see if any change in course from SATG, or if the OneAPI team gets away with the "I was just doing my job" excuse.
Cheaty cheaty bang bang!