Ethernet
Performance of ethernet networks varies widely. Factors include the switch and NIC manufacturer, firmware settings and system/software settings. Physical layer even plays a role: 10 GigE over RJ45 (10GBase-T) has a higher latency than SFP+ Direct-Attach copper.
Contact our experts to determine a configuration which meets your requirements.
| Data Rate | Theoretical Bandwidth (unidirectional) | End-to-End Latency | Technology |
|---|---|---|---|
| Gigabit Ethernet | 125 MB/s | 25 ~ 65 microseconds | |
| 10G Ethernet | 1.25 GB/s | 1.3 microseconds (RDMA application) 4 microseconds (sockets application) | Mellanox ConnectX-3 VPI |
| 40G Ethernet | 5 GB/s | 1.3 microseconds (RDMA application) 4 microseconds (sockets application) | Mellanox ConnectX-3 VPI |
InfiniBand and Omni-Path Fabrics
Although these fabrics typically offer the highest throughput and lowest latency, much depends on the configuration of the fabric and the method in which your software application accesses the fabric. The figures below describe the best possible performance – contact one of our experts to learn more.
The MPI Bandwidths are measured with large messages and MVAPICH2 MPI. The end-to-end latencies are measured with small messages and presume a single switch connecting two host adapters. Each additional switch hop will add latency (see table below).
| Data Rate | MPI Bandwidth (unidirectional) | End-to-End Latency | Generation |
|---|---|---|---|
| 10Gb/s SDR | 1 GB/s | 2.6 microseconds | Mellanox InfiniHost III |
| 20Gb/s DDR | 2 GB/s | 2.6 microseconds | Mellanox InfiniHost III |
| 40Gb/s QDR | 4 GB/s | 1.07 microseconds | Mellanox ConnectX-3 |
| 40Gb/s FDR-10 | 5.16 GB/s | 1.07 microseconds | Mellanox ConnectX-3 |
| 56Gb/s FDR | 6.82 GB/s | 1.07 microseconds | Mellanox ConnectX-3 |
| 100Gb/s EDR | 12.08 GB/s | 1.01 microseconds | Mellanox ConnectX-4 |
| 100Gb/s Omni-Path | 12.36 GB/s | 1.04 microseconds | Intel 100G Omni-Path |
Larger fabrics require that multiple switches be connected to provide service to all nodes. In such a fabric, each additional switch hop adds a small amount of latency.
| Data Rate | Hop Latency | Generation |
|---|---|---|
| 40Gb/s QDR | 0.10 microseconds | Mellanox InfiniScale IV |
| 56Gb/s FDR | 0.20 microseconds | Mellanox SwitchX-2 |
| 100Gb/s EDR | 0.09 microseconds | Mellanox Switch-IB |
| 100Gb/s Omni-Path | 0.10 microseconds | Intel 100G Omni-Path |
| 200Gb/s HDR | <0.09 microseconds | Mellanox Quantum |
See also: Performance Characteristics of Common Transports and Buses