Many often confuse the meaning of Moore's Law... it refers to the number of transistors on a chip, not performance.

A while back, it became apparent that the gains from increasing clock frequency on chips was not worth the expense and chip makers started adding extra cores to CPUs.

However, the increase in the number of cores on consumer chips has not matched the increase in transistors on each chip.

I surmise that a lot of these transistors have gone into features such as prediction logic ect, because it is difficult for some workloads to be parallelized, or many programmers find parallelizing their programs too time intensive, or CPUs are optimized for existing programs.

However, from my perspective, I would like to see transistors go into increasing core count and on-chip-cache as this would benefit my programs more than marginal increases in single threaded performance given that I have no trouble writing multi-threaded code for most of my particular goals.

If I use the extra transistors for a really large cache, I will not have to make as many trips to memory, which can also be a big performance booster.

Am I incorrect as to the reason core counts do not seem to be increasing at the same rate as the number of transistors? Or is there also some diminished return for increasing core count even for easily parallelized work loads such as memory bandwidth?

Why have core counts not increased at anywhere near the rate as the number of transistors on a chip?

Edit: Just because a workload can be run in parallel does not mean it is an appropriate task for a GPU ect which tend to deal with doing a lot of floating point calculations. CPUs have diverse general purpose capabilities which more specialized chips lack.

An example of this could be, let's say I have a set of 50 heuristic functions I need to run against a large set of data that is already in memory.

This is easy to multi thread, give each function its own thread, and you can multi thread it further by diving up subsets of the data for each function (if the data is not highly interdependent). You could easily satiturate all the cores of even a top end Xeon processor, but you won't be able to make much use of a GPU or SIMD.

Or, just a common web application serving many different requests that do not need to be coordinated.

Or, just several different applications running on the same server for political or administrative reasons.


closed as primarily opinion-based by Chris Stratton, Ale..chenski, old_timer, PeterJ, clabacchio Sep 21 '17 at 9:18

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 13
    \$\begingroup\$ I have no trouble writing multi-threaded code. Good for you, but most of the existing code and algorithms are sequential and not fitting well into the parallelism paradigm. So the benefit of having more cores is not linear. \$\endgroup\$ – Eugene Sh. Sep 20 '17 at 19:27
  • 3
    \$\begingroup\$ Have a look at the number of cores in a GPU. \$\endgroup\$ – Nick Alexeev Sep 20 '17 at 19:28
  • 4
    \$\begingroup\$ Memory bandwidth and latency are both nightmares. There exists a 1024 implementation (Adapteva) where each core only has 32K of "primary memory" and accessing global memory and other core's memory has a huge performance hit. The Taihu-light is a 256 core per chip and also suffer from memory bandwidth problems. \$\endgroup\$ – user3528438 Sep 20 '17 at 20:40
  • 3
    \$\begingroup\$ @TheCatWhisperer, there are dozens of engineers and architects working on the problem of performance optimizations and overall CPU architectures, mostly PhDs and former computer scientists/professors/academics, best of the best, with a firm grip on current and future fabrication technologies/capabilities, and with high alert of business side. Did they forgot to invite you to their meetings? \$\endgroup\$ – Ale..chenski Sep 20 '17 at 21:13
  • 3
    \$\begingroup\$ @TheCatWhisperer it seems like the real problem is your procurement process rather than anything electronic. Essentially you'd like more something and assume it ought to be available in your price bracket? \$\endgroup\$ – pjc50 Sep 21 '17 at 8:22

There's a number of technical and business reasons in no particular order:

  1. Memory bandwidth becomes an issue with scaling of cores. Memory contention can actually decrease your performance.
  2. Xeon Phi is the platform where core hungry (and cash loaded) customers can go.
  3. Most software has been designed to run well single threaded. This forms a chicken and egg problem. Why try and sell more cores when most customers can't use them? Most customers won't use them because hardware isn't built in a core-scaled fashion.
  4. Many customers are more interested in IIO bandwidth. In that case, you just need enough cores to service IIO.
  5. Intel Xeon's do have many more cores as well, but you'll pay a pretty penny for them in general. In that regard, it's simply supply and demand.
  6. Because transistor count continues to scale (although not really by Moore's law anymore either), single threaded applications still dominate, and core processing power isn't the bottleneck usually, it's more effective to put those transistors to use making the cache larger and more efficient. Basically, instead of parallelizing the workload by creating more cores, the cores are now getting fed better.
  7. Lack of competition in the highly parallel compute segment prevents consumer level pricing.
  8. Most mainstream programming languages are ill equipped to handle parallel code well. Even those that appear to haven't seemed to find a way to easily debug parallel code. Potentially a new programming paradigm is necessary to overcome this.
  9. Certain common OS's can actually suffer exponential performance loss the more forks you make so even if you have the cores, the OS handling of it ruins the usage of them. This is an extension of points 3 and 8.

Basically as discussed: most people buy Intel-compatible processors to run their existing programs, which have minimal parallelism or may be entirely dependent on single-threaded performance.

If you want many cores, buy a GPU. Or one of these 1024-core processors. The main limiting factor then becomes memory bandwidth.

given that I have no trouble writing multi-threaded code.

With which tools? What sort of algorithms?

  • 5
    \$\begingroup\$ I have no trouble too to write multi threaded code, if it has to be correct, race condition free and scalable though, I get headaches... \$\endgroup\$ – PlasmaHH Sep 20 '17 at 20:17
  • \$\begingroup\$ Most programs have minimal parallelism because they implement human logic, which is inherently sequential, not because the existing programs are bad or stupid. And computers are just accelerators of our [dis]abilities. \$\endgroup\$ – Ale..chenski Sep 20 '17 at 21:06
  • 1
    \$\begingroup\$ Even if a work load is not easily made parallel to speed it, if you have to do it for a hundred different people at a time on a server, you can often make use of more cores. Even if you are limited by IO, there is a certain cost to making a CPU core switch threads. \$\endgroup\$ – TheCatWhisperer Sep 20 '17 at 21:31
  • 2
    \$\begingroup\$ @AliChen Well, that is not completely true. If an algorithm can be parallelized, there is a parallel version of it, and it is eventually implemented if the algorithm is a bottleneck. This is why we have mathematicians and algorithmists - the guys who go beyond the average human logic and (dis)abilities. \$\endgroup\$ – Eugene Sh. Sep 20 '17 at 21:32

Additionally to what has been already said, there is also an additional cost in parallelism. And I am not talking about Amdahl's law. The more cores you have in parallel the more complex the hardware becomes that has to negotiate between the cores. Some of these scale with $O(n)$ others with $O(n^2)$. This imposes an upper limit before it becomes more economic to insert another layer and use more parallel CPU's instead if more cores.

You can see these trade-offs quite nicely with Intel: there are multi-core CPU's with higher number of cores, like the Xeons that go up to 28 cores. But the Xeons are limited to about 3GHz clock speed. Most of them just do around 2.5GHz. The consumer processors go higher up in clock speed, but are limited on the number of cores.

If you have an embarrassingly parallel problem you can get one of the 8 socket motherboards and eight of those 28 core Xeons to get a whooping 224 cores in a single computer. At this point you are deep in NUMA territory and your application has to be tailored to the computer architecture you have, in order to get the full performance.


Not the answer you're looking for? Browse other questions tagged or ask your own question.