A network and infrastructure design view

Every part on a datacenter order can be correct on its own and the rack can still fail. The faults sit between the parts, so a bundled quote is where they get caught.

1. Power: size the rack, not the server

• Circuit capacity. Continuous loads are normally limited to 80% of the breaker rating. A 208V single-phase 30A circuit gives about 5 kW usable (24A × 208V). A 208V three-phase 30A circuit gives about 8.6 kW (√3 × 208V × 24A).
• GPU servers. An 8-GPU server can draw 10 kW or more at full load, so a single unit can exceed one single-phase circuit. The PDU, feed and breaker have to be chosen with the server, not after it.
• A/B redundancy. If each server has two feeds, one feed must carry the full load when the other fails. In normal operation each feed should therefore sit at or below 40% of its capacity.
• Connectors. C13/C14 and C19/C20 are not interchangeable, and the supply voltage (208V, 230V) and wall plug must match the site. A 230V-only PSU on a 120V PDU won't run.
• Inrush. Powering many servers at once can trip a breaker. PDUs with staggered start-up avoid this.

2. Cooling: airflow must match end to end

• A rough guide is CFM = 3.16 × watts ÷ ΔT (°F). A 10 kW load at a 20°F temperature rise needs about 1,580 CFM of cold air.
• Switch fan and PSU modules are airflow-specific. A back-to-front switch in a front-to-back rack draws in hot exhaust air.
• Rack depth and width have to fit the server, rail kit, PDUs and cable bundles together. Many deep servers need racks of 1,000 mm or more, and 750 mm width if PDUs and cabling share the sides.

3. Cabling and optics: speed, reach and FEC

• Passive DAC is typically limited to about 3 m at 25G and 100G, and shorter at 400G. Longer runs need active cables or optics.
• FEC must match at both ends. A 25G link can need RS-FEC or BASE-R FEC depending on the cable or optic, and a mismatch gives no link or CRC errors.
• 100G SR4 needs an MPO-12 connection with correct polarity. Reach is about 100 m on OM4 and 70 m on OM3.
• Breakout cables depend on port-group rules on the switch.
• Some switches and NICs reject transceivers they don't recognise.

4. Compute, NIC and fabric

• A 400G NIC needs a PCIe Gen5 x16 slot. Gen4 x16 gives roughly 250 Gbps, so the NIC can't reach full speed.
• Oversubscription should be decided up front. Leaf switches with 48 × 25G down and 8 × 100G up are 1.5:1. An AI back-end fabric is usually built 1:1.
• RoCEv2 needs PFC and ECN configured on the switches, and the MTU must match on every hop. A single 1500 byte link in a 9000 byte path causes drops.
• Out-of-band management (BMC ports and an OOB switch) is often left off single-item orders.

5. Firmware and licensing

• NIC firmware, drivers and the switch OS need to appear on the same support matrix.
• Some platforms need a licence tier for features such as EVPN-VXLAN.
• Some access points use the same hardware but a different software image for standalone or managed use. The wrong one won't join the controller until it is reflashed.

6. Logistics and customs

• Lead times differ. Optics may arrive in two weeks and switches in twelve. Split orders leave expensive servers idle. A bundle lets you plan one delivery or a staged one.
• Customs. Each line has its own HTS code, duty rate and country of origin. Many IT products are duty-free under the Information Technology Agreement, but US Section 301 duties can still apply to Chinese-origin items. One consolidated shipment means one import record.
• Spares. A bundled quote is the right place to add spare optics, cables and PSUs.

What a good bundle includes

1. A bill of materials with exact SKUs
2. A rack power and heat budget
3. A cable and optics schedule with lengths and polarity
4. A firmware and licence list
5. HTS and country-of-origin data per line
6. Delivery planned in one or more agreed stages

Send us the parts list and the destination, and we'll check each of these before the quote goes out.