PCI Express 2.0: Scalable Interconnect Technology, TNG
by Kris Boughton on January 5, 2008 2:00 AM EST- Posted in
- CPUs
Intel Chipset PCI Express Resource Assignments
Intel desktop platforms make use of a pair of bus controllers, commonly referred to as the chipset. The Memory Controller Hub (MCH) or "Northbridge" (NB) is responsible for directly interfacing the processor with memory as well as all other system resources, and it communicates all information to the CPU over the Front Side Bus (FSB). The memory controller, which makes up a portion of the MCH, is responsible for arbitrating requests for bus time and translating requests for data located in system memory into the appropriate commands and addresses needed to retrieve and deliver that information to the CPU and other DMA (Direct Memory Access) devices. This discussion, although relevant to understanding the operation of the MCH, is better left for another day. Instead, we would like to concentrate on the second major function of the Northbridge, which is providing PCI Express ports for lane assignments and data transfers. Incorporating the PCI Express controller into the Northbridge ensures that installed graphics cards have easy access to system memory and CPU resources.
The Northbridge communicates with the I/O Controller Hub (ICH), or "Southbridge", using an interface that Intel calls Direct Media Interface (DMI) at a maximum transfer rate of 2GB/s. This connection is actually nothing more than a PCI Express x4 link with a dedicated transaction layer which handles routing of all input/output (I/O) data from attached peripherals such as hard disks, optical drives, Ethernet controllers, onboard audio, and Universal Serial Bus (USB) and PCI devices. The Intel 875/ICH5 chipset, the last to make use of then-aging AGP technology, made use of a PCI link in this location which provided only 266MB/s (133MB/s in each direction) total bandwidth. The 915/ICH6 "Express" chipset was the first to make use of this higher speed link - a change that improved the performance of technologies such as SATA (both 1.5Gbps and 3.0Gbps) and onboard 1Gbps Ethernet ports, giving them the headroom they need to function at their maximum potential.
Each PCI Express "Port" in the Northbridge is actually a pre-configured x4 lane (meaning that lane resources can only be assigned in chunks of four). The table below shows the PCI Express resources available from some of the more recent Intel chipsets. Reviewing this we see that the only real difference between the older 965P and 975X chipset, in term of graphics capabilities, was the ability for the 975X chipset to split the available x16 lane into two x8 lanes for CrossFire configurations. This difference was more than likely for market distinction and not a true limitation of the chipset. PCI Express is highly scalable and there is no obvious reason as to why 2x8 CrossFire on a P965 chipset should not work. Much like P965, P35 is limited to single x16 configurations.
The real change comes with the introduction of Intel's current X38 flagship chipset. The X38 chipset provides two true PCI Express 2.0 capable x16 lanes for use with the latest PCI-E 2.0 graphics cards. A pair of ATI/AMD graphics cards in CrossFire will receive a full 16 lanes of bandwidth each regardless of the PCI Express specification they use - the only difference is the maximum theoretical throughput (16GB/s with PCI-E 2.0 and 8GB/s with PCI-E 1.x). Future X48 chipsets, which appear to be speed-binned X38 chipsets, will provide exactly the same capabilities.
Some 975X users may have asked themselves if it was possible to "unlock" x16 link speeds for their two graphics cards running in CrossFire - if one card can run at x16 speed, why not two? (The slots are certainly big enough to house the cards and all the right traces are on the motherboard.) As we can see, the chipset is physically incapable of that configuration as the PCI Express resources needed to form two x16 links simply are not there - there is nothing to change and nothing to re-configure.
In the case of each "Express" chipset, a final x4 port is available for the purposes of linking the Southbridge to the Northbridge. Although the X38/X48 chipsets are true PCI Express 2.0 parts, they configure Port8 for PCI Express 1.x speeds, as the ICH9(R) Southbridge is only a PCI Express 1.x enabled part. As such, we expect that Intel's upcoming ICH10(R) will make full use of PCI-E 2.0 as well meaning that the associated DMI connection should see a nice boost in speed to 4GB/s (double that of today). Below is a simple physical representation of the X38/X48 PCI Express resources.
PCI Express 2.0 to 1.x backwards-compatibility allowed Intel the opportunity to release a PCI Express 2.0 MCH now with an ICH to follow, something that would not have been possible otherwise. Traditionally we see a new ICH at the same time as the MCH. Given the extra time Intel has for the refinement of ICH10(R), we will be satisfied with nothing less than perfection.
The ICH provides the system with an additional six PCI Express 1.x lanes (to be grouped as desired by the motherboard design company). However, as we have seen, the DMI link is capable of sustaining a maximum throughput of x4 speed to the Northbridge. This means that in the case of fully populated link states a lot of the bandwidth originating from downstream-attached peripherals is bottlenecked if outputting anything more than short bursts of data. Coincidentally, this is why pseudo-CrossFire configurations on P965 and P35 chipsets are limited to x4 speeds for the second card (additionally, x6 link states are not allowed). For the same reasons, the use of a third graphics card in X38/X48 configurations for physics acceleration purposes sees the same limitation. Previous testing indicates that there may be a slight performance penalty associated with x16/x4 CrossFire compared to x8/x8 or x16/x16 solutions, but despite having less than one fourth the bandwidth we do not see a massive loss of performance. In other words, we are definitely not at the point yet where we are using all of the available bandwidth for even PCI-E 1.x x16 slots.
Facebook
Twitter
RSS
21 Comments
View All Comments
kjboughton - Sunday, January 6, 2008 - link
Are you sure this isn't fiber or optical? Any supporting information you can provide would be great.Hulk - Saturday, January 5, 2008 - link
First of all great article. Great writing. You should be proud of that article.I see that currently the Southbridge can transmit data to the Northbridge at 2GB/sec max. In real world situations about how much bandwidth would the Southbridge require assuming a light, medium, and heavy loading situation?
kjboughton - Monday, January 7, 2008 - link
I can help you with some of the base information needed to calculate this yourself (since every system is different based on attached peripherals as well as their type) and we'll leave the rest to you as an exercise.For example, a 1Gbps Ethernet connection to the ICH would have a maximum theoretical sustained data transfer rate of 125MB/s (1Gbps x 1 byte/8 bits). A single SATA 3.0Gbps drive would be limited by the interface to three times this number, or about 375MB/s (although the disk to bus/cache transfer rate is much less, somewhere on the order of 120-140MB/s sustained) - but nevertheless, burst read speeds could easily saturate the bus in one direction (1Gbps). Then there's USB devices, possibly a sound card or other onboard solution...going through the numbers, adding up the maximum possible bandwidth for all your attached devices you should be able to get an idea for what would be "light, medium and heavy" loading for your system. Again, this is something that varies from system to system. Hope this helps.
LTG - Saturday, January 5, 2008 - link
Excellent article, good tech level.Would you believe "simple ecards" benefit from PCI-E 2.0 right now?
At most sites when you send an ecard it just e-mails a link to a flash animation to someone.
However when you send an ecard at the site below, it's rendering and compositing custom photos and messages into a 3d scene on the fly for each card sent.
Because this is a web site all of this runs on the server side for many users at once.
PCI-E 3.0 will be welcome :).
http://www.hdgreetings.com/preview.aspx?name=count...">http://www.hdgreetings.com/preview.aspx?name=count...
or
www.hdgreetings.com (sorry, link buttons not working)
JarredWalton - Saturday, January 5, 2008 - link
I don't know that e-cards would really benefit much - especially right now. The FSB and memory bandwidth aren't much more than what an x16 PCI-E 2.0 slot can provide in one direction (8GB/s). I would imagine memory capacity and the storage subsystem - not to mention network bandwidth - are larger factors than the PCI-E bus.Are you affiliated with that site at all? If so, I'd be very interested to see a performance comparison with a single 8800 GTX vs. an 8800 GT on a PCI-E 2.0 capable motherboard. The 8800 GTX even has a memory and performance advantage, but if as you say the bottleneck is the PCI-E bus, it should still see a performance increase from the 8800 GT.
LTG - Sunday, January 6, 2008 - link
Hi Jared, yes I'm a developer on the site - (pls don't think of my post as spam, i've been a reader at AT forever and it just seemed relevant :)You could be right, we are just now starting to test pci2.0 so the benchmark you mention will definitely shed some light.
The network is not a bottleneck because cards are rendered and compressed on a given server node.
The disk IO is a 6 drive RAID0 array (no data is at risk because the nodes just render jobs) with the Segate 7200.11 drives max out at 100MB/sec transfer rate each, which is less than 600MB/sec total, however I have "heard" that the effective PCI-E video card bandwidth is much less than the theoretical limit.
I wish there were a utility to easily measure PCI-E bandwidth but currently I only know of indirect experiments as you mention.
Thanks again for the nice article.
PizzaPops - Saturday, January 5, 2008 - link
I can't help but be amazed by the speed at which hardware is improving. I remember when we were stuck with just PCI and AGP for what seemed like forever. Now the speeds are getting ridiculous. Can't wait to see what the future has in store.Very informative article and not too difficult for the average person to understand either. Now I know why my X38 gets so hot.
Spoelie - Saturday, January 5, 2008 - link
Why does the 790FX stays so cool then?Besides, there hasn't been a review of that one yet on AT.
Gary Key - Saturday, January 5, 2008 - link
The 790FX does not have the memory controller on-board among other items, so the additional power required for PCI-E 2.0 is minimal at best as are resulting thermal increases. AMD also took a very elegant approach on the 790FX in regards to PCI-E 2.0 (they had time to ensure proper integration, Intel's is fine, just they had a lot to cram into the chipset this time around ;) ) that we will cover shortly.NVIDIA's current approach on the 780i is to use a bridge chipset that is creating a few problems for us right now when overclocking both the bus and video card. We will have a complete 790FX roundup the week of the 14th along with a "how to" guide on getting the most out of Phenom on these boards.
Comdrpopnfresh - Monday, January 7, 2008 - link
So by bridge chip, some intermediate chip slows things? Or creates asymmetric latencies leading to unbalanced clocks (like initial SATA drives implementing connections and features like NCQ natively on PATA with a cross-over to SATA)?If so, I read an article dealing with a similiar problem with raid-spanning of SSDs somewhere... have to dig up the link...
The problem began with one dive on the Intel ICH..9 (I understand there is a workaround now). So the tester switched to an add-in discrete raid-handling card. When they began adding more and more drives (believe they went 1-2-3-4), these mucho-expensive raid cards were zapping throughput one after another (with successively higher prices of course) because the companies shaved on the onboard processing power because before these SSDs, the throughput on a RAID spanning standard HDDs just wasn't nearly as great. When they got to four drives, and something like a > $900 add-in card they stopped- one hell of an expensive review!