Intel Haswell-EP Xeon 12 Core Review: E5-2650L V3 and E5-2690 V3

As part of our Haswell-EP coverage, the next two processors on our test beds are both 12 core variants. The E5-2650L V3 is a surprising monster, giving 12 Haswell cores at 1.8 GHz with 2.5 GHz turbo for only 65W, while the E5-2690 V3 extends the power budget to 135W for all 12 cores at a 2.6 GHz base frequency.

With Haswell-EP, the landscape of the server and workstation CPU has changed ever so slightly. In previous generations, Intel balanced core counts with frequency at the same power level. This makes a lot of sense, as you cannot add cores or frequency without drawing more power. What makes Haswell-EP feel different is a slight change in strategy as to how the new core designs are binned according to their characteristics.

As we explained in our Haswell-EP overview and the review of our 10c samples, all Intel Haswell-EP CPUs are derived from three main die designs. These die designs all look fairly similar, varying in how many columns or cores, but this allows Intel to keep the range of designs simple but also allows them to fuse off bad cores and keep yields high.

One of the other factors in high yields is binning - the process of separating the dies into those that require less voltage for the same frequency, those that need more voltage, or those with cores that are disabled. This gives CPU manufacturers room to adjust both voltage and frequency to hit a desired power point. With Haswell-EP, Intel is being very adventurous in its goals - the same die design underpins both the CPUs in today's review, but both dies have different voltage/frequency characteristics within their power target.

Statistically, the further away from the average yield point a die is, the more money Intel can charge for it because they will end up with less of them in that bin. It is not always that easy and depends also on demand and market positioning. As discussed in the 10c review, Intel's main competition is with itself, so it has to be able to convince old customers to upgrade. By being more aggressive with die binning, this allows customers to optimize and configure the design they want.

This is why I say that the general expectation of raising cores results in a lower frequency has shifted with Haswell-EP. By being aggressive with their binning strategy, Intel can approach customers and say 'how many cores / what frequency / what power window do you need? Choose two and we can tell you what we can do on the third'. This is why processors like the E5-2650L V3 exist. A low high core count (12 cores, 24 threads), reasonable frequency part (1.8 GHz base, 2.5 GHz turbo) that has a TDP of only 65W. It sounds rather crazy.

The Market

In the broadest sense, Intel tackles several markets with the entire Haswell-EP line. At the low end we get quad core parts with lots of cache and DDR4 support aimed at memory bandwidth contained applications that do not rely on core count. Moving on up we have workstation related SKUs, extended support units, off-roadmap designs for specific customers, a lot of SKUs that won't ever reach the shelves as individual parts, parts focused on 2P compute, and others for networking, infrastructure and compute-dependent storage.

Our two CPUs today, while both 12C/24T parts, are aimed at different markets as well.

The E5-2690 V3 sits in a stack of CPUs at a 2.3-2.6 GHz base frequency but vary in their core counts. The reason why a customer moves up and down this stack is more about cost vs. performance of the part itself, with a moderate rise in power consumption as cores are added. The E5-2690 V3 has a TDP of 135W, and this kind of performance stack has been very typical since Intel's Xeon class went multicore.

The E5-2650L V3, by virtue of its 'L' designation, sits in the low power segment. The combination of high performance and low power seems like an oxymoron, however it is targeted at code that can take advantage of more cores over more frequency. Some programs do not scale with additional frequency due to other bottlenecks in the system, such as memory accesses, but can take advantage of more cores. The E5-2650L V3 does this in 65W TDP, less than half of the E5-2690 V3.

In terms of upfront cost, the E5-2690 V3 follows the pattern of rising cost compared to the other 2.3-2.6 GHz parts and sits around the $2090 mark. The E5-2650L V3 however comes across as a bit expensive to begin with (~$1329), but the idea here is that for a 2P system limited to 135W, you can either have a single E5-2690 V3 with 12 cores at 2.6 GHz for $2090, or two E5-2650L V3 CPUs with 24 cores at 1.8 GHz for $2660. If you don't mind the ~30% price difference and have the software to take advantage, it is almost a no-brainer comparison.

There is a distinct rider on this however, as it comes down to the software in use. As we have discovered over the last couple of years, most off the shelf software does not scale to more sockets, or even worse, reduces in performance due to memory mismanagement across the two CPUs. So on paper, if the software used ticks all the boxes, a dual E5-2650L V3 system sounds like a good deal.

The frequency response of the 12-core lineup for E5 26xx v3 CPUs puts the two L CPUs at the same frequency but a 10W difference in power consumption. The E5-2690 v3 heads the top of the list, with the 2685 v3 and 2670 v3 taking the biggest drop between single core and multi-core performance. The 2670 v3 spreads the drop over more cores in use, finally reaching its multi-core frequency when half the CPU is loaded, rather than 5/12.

Test Setup

For our testing, it is worth noting that our CPU samples arrived at different times and due to the testing setup at the time, certain benchmarks were unable to be run due to updates required. We were also able to source a second E5-2650L V3 and a 2P motherboard, allowing the comparison between the two CPUs on their own and a 130W combination.

Many thanks to...

We must thank the following companies for kindly providing hardware for our test bed:

Thank you to OCZ for providing us with PSUs and SSDs.
Thank you to G.Skill and ADATA for providing us with memory.
Thank you to Corsair for providing us with memory, an AX1200i PSU and a Corsair H80i CLC.
Thank you to MSI for providing us with the NVIDIA GTX 770 Lightning GPUs.
Thank you to Rosewill for providing us with PSUs and RK-9100 keyboards.
Thank you to ASRock for providing us with some IO testing kit.
Thank you to Cooler Master for providing us with Nepton 140XL CLCs.

Load Delta Power Consumption

Power consumption was tested on the system with a wall meter connected to the power supply. This power supply is Gold rated, and as I am in the UK on a 230-240 V supply, leads to ~75% efficiency > 50W, and 90%+ efficiency at 250W, suitable for both idle and multi-GPU loading. This method of power reading allows us to compare the power management of the UEFI and the board to supply components with power under load, and includes typical PSU losses due to efficiency.

We take the power delta difference between idle and load as our tested value, giving an indication of the power increase from the CPU when placed under stress. Unfortunately due to the timing of our testing, we were unable to check the power difference of the E5-2690 v3.

Overclocking on a Xeon?

Similar to our last Xeon review, multiplier overclocking on the E5-2600 V3 series is disabled.  Nevertheless with the right motherboard users can adjust the BCLK to add a few percent more performance within the stock frequency. Our E5-2600 V2 testing yielded 110 MHz AVX stable for a +10% boost, but the 10-core E5-2600 V3 CPU we tested only gave 104 MHz.  Today we are testing the E5-2650L V3, which should arguably have at least the power headroom if the stock voltage is set a little high by Intel.

Similarly to the E5-2687W V3, the moment we hit a frequency the system does not like, it returns as a complete failed POST rather than a BSOD.  However, we did manage a 5% boost in base frequency, giving a 4.4% rise in POV-Ray scores. The load voltage remained constant, and despite the frequency raising the power margins were all within a 9W window from stock usage.