400Vdc Data Center Material for External Audiences

Transcription

1 400Vdc Data Center Material for External Audiences Joint work prepared by: Intel Corporate Technology Group Emerson Network ower EY Mission Critical Facilities

2 AC and DC power distribution Facility-level 480V/208Vac & 400Vdc MV 480V AC US AC/DC DC/AC Bypass 208V AC AC/DC V DC DC/DC 12V VR Chrg Battery cabinet(s) DU Rack SU Server MV 480V AC Rect AC/DC 400V DC 400V DC DC/DC 12V VR Battery cabinet(s) DC/DC DU Rack SU Server roposed distribution bus of 400Vdc ± 5% Batteries connected through bi-directional DC/DC Battery voltage can be optimized since not directly connected to bus 2

3 Comparisons Comparisons There s more to 400Vdc than efficiency! Improved efficiency achieved by using fewer conversion stages => Fewer components => Space savings Improvement in reliability Lower electrical component cost Comparison of 400Vdc to 480V/208Vac high efficiency designs For specific data centers 480V/208Vac design 400Vdc design Space Efficiency Reliability Cost Data Center Facility 5.5 MW IT load 3.3 MW IT load ** 3 ** Modeled on a 3.3 MW load rather than the 5.5 MW load due to available data

4 Intel facility used for comparisons Modules C - E: Future Full Build Out Module E Module D Module C 5.5MW IT Load Module C: Next phase (220 racks x 25 kw/rack) Loading Dock Facility Electrical Room Utility Spine Chilled Water Storage Tanks High-Temp Chiller lant Module B 3.3MW IT Load Module A 3.3MW IT Load Modules A - B: Existing (220 racks x 15 kw/rack) Low-Temp Chiller lant High-Temp Chiller lant Expansion 400Vdc proposal for Module C - Intel - Emerson Network ower - EY Mission Critical Facilities (H) 4

5 Intel facility for comparisons Two-story, vertical flow-through air design Designed for ~500 watts per square foot Centralized air cooling plant 6,000 square feet of raised metal floor (RMF) per module 240 cabinets per module (20 network) 5

6 Results & Comparisons Data Center Facility AC design Module E Module D Utility Spine extension Module C 5.5MW IT Load Loading Dock Utility Spine AC design requires 50% more space for 5.5MW than 3.3MW IT load Facility Electrical Room Chilled Water Storage Tanks High-Temp Chiller lant Module B 3.3MW IT Load Module A 3.3MW IT Load Low-Temp Chiller lant High-Temp Chiller lant Expansion 400 Vdc design can fit in the same space as 3.3MW facility 6 33% Space Savings: Fits within original Module C footprint

7 400Vdc concept : Architecture 400Vdc Module C : Server (Top) Floor 1.1 MW Module (44 cabinets) 7

8 Comparisons : hysical layout 400Vdc Module C : Utility (Ground) Floor BC BC BC D/D BC BC BC D/D BC BC BC D/D BC BC BC D/D BC BC BC D/D MV Transf MV Transf BC BC BC D/D BC BC BC D/D MV Transf IS R IS R IS R IS R IS R BC BC BC D/D BC BC BC D/D R R R R R BC BC BC D/D BC BC BC D/D MV Transf R R R R R BC BC BC D/D BC BC BC D/D BC BC BC D/D BC BC BC D/D MV Transf OS OS OS OS OS BC = Battery cabinet D/D = DC/DC Battery Converter R = Rectifier IS = Input Switchgear OS = Output Switchgear Busduct beneath Server Cabinets 8

9 400Vdc concept : Energy savings Energy savings at rack Rack power reduced with higher efficiency SU 400Vdc rack power ~ 0.98 x 208Vac rack power 400Vdc power supplies are prototypes from Delta Electronics Will be able to optimize further, but not significantly so Efficiency [%] Typical operating range Load [pu] 208V ac eff 400V dc eff 400V dc est Eff delta Est eff delta Increase in efficiency [%] Efficiency [%] Typical operating range Load [pu] 208V ac eff 400V dc eff 400V dc est in ratio Est in ratio in ratio dc/ac 9

10 Comparisons : Efficiency Estimated total energy savings Compare proposed 480V/208Vac & 400Vdc designs for Intel facility ** For 5.5MW facility Iso-redundant 480Vac Distributed-redundant 400Vdc Assumed 2.15% total cable loss for both Non-redundant and redundant SU cases modeled Cooling estimates included in total energy savings ower delivery efficiency [%] High eff 480VAC 250 Facility 400VDC Facility 400Vdc (Red SU) 125 High Eff 480Vac (Red SU) Total savings [kw] Load [%] Total input power savings [kw] *Facility savings includes cooling power 10 ~7.5% % Energy savings compared to 480/208Vac distribution ** Results will vary for different facilities and for different AC architectures

11 Results & Comparisons: Reliability Reliability Analysis Relex calculations by EY Mission Critical Facilities Option Availability Unavailability robability of failure in 5 years AC Tier IV configuration e % DC configuration ** e % DC Improvement 62.5% 200% 2X lower probability of failure compared to equivalent Tier IV AC facility 11 ** Reliability numbers for Rectifier from simulation of power train only, will be lower with all components included

12 Relative Electrical Cost Comparison Description AC Data Center [pu] DC Data Center [pu] Ratio Remarks Labor Due to smaller wiring sizes Materials Wire, pipes, Supports, anel boards, Busways etc Owner furnished items Equipment, US, static switches, transformers, etc. Electrical Cost of Work (COW) Total cost to implement all the electrical work including equipment AC DC Remarks US / Rectifier Inclusive of batteries, DC/DC converter, input & output switchgear 480/208V Transformers Distribution transformers are eliminated in DC Data Center Server ower Supply Assumed same cost although 6% reduction in BOM cost estimated Electrical cost for dc Data Center estimated 15% lower than for ac 12 Cost comparison for a 3.3 MW load rather than the 5.5 MW load due to available data. Cost estimate for maturity, i.e. high volume.

13 AC and DC power distribution : Benefits of 400Vdc Comparison summary Comparison for Intel facility shows ** : ~7% Facility energy savings, incl. cooling 7.7% at 50% load; 6.9% at 80% load 33% Space Savings No DUs, simplified switchgear 200% Reliability improvement 2x lower probability of failure in 5 years 15% Electrical facility capital cost savings Electrical is ~40% of total facility cost, i.e. saves 15% of 40% ~ 6% of total 480V/208Vac design 400Vdc design Space Efficiency Reliability Cost ** Results will vary for different facilities 13

14 AC and DC power distribution : Benefits of 400Vdc And that s not all Additional benefits to DC distribution No phase balancing => reduces power strip & wiring complexity No synchronization required to parallel multiple sources No harmonics => no FC circuits Fewer breakers required because of fewer stages Simplifies wiring, since only two wires required No need for complex interlocks - simpler procedures, less time Only resistive voltage drop in wires Additional benefits specific to higher voltage DC Simplifies wiring, especially at higher power densities Lower currents than at 48Vdc, so smaller physical wires Use fewer natural resources & less energy to process materials ~400Vdc bus in light ballasts and Adjustable Speed Drives (ASDs) Simpler/more efficient connection to renewable energy sources hotovoltaics, fuel cells, wind with variable frequency drives 14

15 Benefits of 400Vdc V DC V DC/AC Total facility impact X XAC/DC DC/AC Ballast Lighting loads FC 60 Hz AC 480V AC/DC Chrg X X DC/AC XAC/DC DC/DC Electronic loads VR SU Motor loads AC/DC DC/AC XAC/DC DC/AC ASD 15

16 Benefits of 400Vdc V DC V DC/AC 400Vdc facility vision DC/AC Ballast Lighting loads FC 60 Hz AC 480V AC/DC DC/DC DC/DC SU Electronic loads VR Motor loads AC/DC DC/AC DC/AC ASD 16

17 Benefits of 400Vdc V DC V X DC/AC 400Vdc facility vision DC/AC Ballast Lighting loads FC 60 Hz AC 480V AC/DC DC/DC DC/DC SU Electronic loads VR Motor loads X AC/DC DC/AC DC/AC ASD 17 V : hoto Voltaic FC : Fuel Cell

18 Benefits of 400Vdc DC V DC/DC 400Vdc facility vision DC/AC Ballast Lighting loads 60 Hz AC 480V AC/DC DC/DC DC/DC SU Electronic loads VR Motor loads AC/DC DC/AC ASD 18

19 How to make 400Vdc a reality? Need new/modified equipment May need new standards The US National Electrical Code (NEC) covers up to 600Vdc, but open to interpretation, so have to engage local authorities Market acceptance Need end user pull 19

20 Equipment : SU 400Vdc server For servers, only power supply enabling required Remove FC stage, re-use existing DC/DC stage Enabling circuit proposed only for input current > 5A Vac 50/60 Hz AC fuse L BOOST + EMI Filter (AC) s BOOST C O DC/DC 12V GND Chassis Modified supply to accept 400Vdc input - 400Vdc DC fuse EMI Filter (DC) + C O DC/DC 12V Chassis GND EN - 20

21 Equipment : SU 400Vdc Appliance Coupler Current SU prototypes from Delta use Anderson ower roducts connector from their owerak series ower contacts rated for up to 600V (Intel s version is rated 10 A) connectors have UL, CSA and TUV approvals. receptacle fits standard chassis opening for IEC320 C22 inlet For the next step Receptacle compatible with the IEC320 C14 chassis opening Molded rather than modular plug for reduced costs roposed 15A continuous rating Expect 5 8A disconnect rating rototype under development Will include 48Vdc keying option GND (make first) + 400Vdc 400Vdc return Enable (make last, break first) 21

22 Equipment : CDU 400Vdc rack power strip No phase balancing IEC 309 connector Standard for international AC power strips Rated for hot disconnect Recommend mechanical interlock IEC specified for >250VDC Not currently UL listed in the US 22

23 Equipment : CDU 400Vdc rack power strip CDU Either CB or fuse/relay, same as AC CDU ower supply without enable ckt (<5A?) Stby Hardwired to CB off busway Floor box Mechanical Interlock for IEC 309 Connector IEC 309 Cable ositive Negative Ground Connector Connector Connector Connector Cable ositive Enable Negative Ground Cable ositive Enable Connector Connector Directly connect enable pin to ground inside connector Connector Connector Lf Lf Cf Cf Soft start ckt not shown ower supply with enable ckt (>5A?) Optocoupler May use relay D1 Cb Cb DC/DC Stby DC/DC Negative 23 Ground

24 AC and DC power distribution 400Vdc rectifier concept roposed distribution bus of 400Vdc ± 5% Batteries connected through bi-directional DC/DC Battery voltage can be optimized since not directly connected to bus Rectifier concept by Emerson Network ower No full scale prototype Results from simulation and small scale prototype Not in product development yet, need market pull MV 480V AC Rect AC/DC 400V DC 400V DC DC/DC 12V VR Battery cabinet(s) DC/DC DU Rack SU Server 24

25 Equipment : Rectifier 400Vdc Rectifier roposal 2.0MW 1000A AC DC 3000A Two wire distribution AC input 12.54kV 1000A AC DC 3000A 400Vdc +/- 5% Y 12.54kV/480V 1000A 2000A Input Switchgear AC DC Rectifier Module 3000A Output Switchgear Battery DC DC Kept in standby mode Battery Battery Battery Cabinets DC DC DC DC DC/DC Converter Cabinet 3 x modules 2 modules on-line for 1.1MW avg load N+1 redundancy with 3 rd module 25

26 Equipment : Rectifier Rectifier Module % Load Efficiency 480Vac Current Source Rectifier AC 400Vdc +/- 15% DC +/- 5% 10% % 78 H x 60 W x 39 D % % % % 97.1 Simulated MTBF 6.94 million hrs Input THD MTTR 4 hrs Input F for most subassemblies CSR rectifier data compiled from computer simulations on 500kW design concept and 6kW lab testing Simulated MTBF includes only power components and basic controls, will be lower when all components included 6 ulse Rectifier with Trap Filter Actual % Load Efficiency 10% % % % % % 97.3 MTBF 2.58 million hrs Input THD Input F pulse rectifier data from actual AC US testing MTTR 4 hrs for most subassemblies Leading F red Lagging F - green 26

27 400Vdc rectifier module ~2% higher efficiency than high efficiency (double conversion) AC USs Average of a couple of manufacturers Non-isolated topology (Current Source Rectifier) MV/LV transformer at input provides required isolation 27 Efficiency [%] Output power [p.u.] 400Vdc rectifier proposed High efficiency AC US (avg)

28 Equipment : Rectifier Input & Output Switchgear Input Switchgear Output Switchgear Input Voltage Output Voltage ower Rating Output Breakers AIC Dimensions 480Vac 480Vac 2000A Main Bus 3 x 1000A 100k 91.5 H x 36 W x 24 D 400Vdc 400Vdc 3000A Main Bus 2 x 3000A >20k 91.5 H x 72 W x 48 D 28

29 Equipment : Rectifier MV Transformer MV Transformer Input Voltage Output Voltage ower Rating Efficiency Type Dimensions 12.47kVac wye 480Vac delta 2.0MW 99% (typical operation at 50% load at 35% of xfmr rating NEMA T-1 95 H x 99 W x 63 D 29

30 Equipment : Rectifier 1.1MW Rectifier (N+1 redundant) From MV Tranf. 480Vac 24 Top View To Busduct 400Vdc Input Switchgear 36 Wide Rectifier Cabinet 60 Wide Rectifier Cabinet Rectifier Cabinet 60 Wide 60 Wide Front elevation details for illustration purposes only. Dimensions may change Output Switchgear 72 Wide 30

31 Equipment : Batteries Battery Cabinet A battery string is a series of 2V cells that form a string at system voltage Multiple strings are placed in parallel to achieve desired capacity Cells are available in various ampere-hour (AH) capacity Use a cabinet that holds one string of 12V battery modules 40 modules of 2.25V/cell = 540V Battery capacity depends on the useable voltage drop Battery cell End Voltage in proposed configurations is 1.6V / cell 31

32 Equipment : Batteries Battery Cabinet Required Capacity DC/DC efficiency 97%, 540kw = 557kW 557kW / 3 battery strings = 186kW / string 186kW / 240 cells = 773W / cell Battery Disconnect Battery Cabinet Front View Battery Module (4 deep) Battery Module Cabinet contains 40 modules Enersys DataSafe HX500 (HX400) 12V Module Capacity at 5 Min with HX V end voltage = 771W / cell (384V) Battery Cabinet Front View (Door removed) 32

33 Equipment : Batteries Battery System To Rectifier Cabinet 78 Battery Cabinet #1 Battery Cabinet #2 Battery Cabinet #3 Battery DC/DC Converter Front elevation details for illustration purposes only. Dimensions may change 33

34 400Vdc concept : Architecture 400Vdc Distribution Architecture 400Vdc distribution can support any redundancy option In order to optimize DC system: Decided to use ground referenced DC rails (positive ground) Two wire distribution 2+1 redundancy within rectifier, modules Combined with distributed redundant wiring AC Tier IV equivalent With redundant power supplies in servers 34

35 400Vdc concept : Architecture Distributed Redundant Architecture DC US 2 DC/ DC A 100A Each rack fed from two buses Each rectifier feeds two buses NO Battery DC US 3 Battery A 100A 1.1 MW Module (44 cabinets) DC US A 35 Battery DC US A 100A

36 400Vdc concept : Architecture NO 36 Distributed Redundant Architecture DC US 2 Battery DC US 3 DC US 4 DC US 5 DC/ DC Battery Battery X X X A 100A 100A 100A 100A 100A 100A Rectifier 3 fails : Load transferred to Rectifiers 1 and 5 Rating required 3/2 x 1.1MW = 1.65MW 1.1 MW Module (44 cabinets)

37 400Vdc concept : Architecture DC US 2 DC/ DC Non-Redundant Architecture 100A 100A A module in Rectifier 3 fails : Load transferred to redundant module Battery 100A O DC US 3 X Battery 100A 1.1 MW Module (44 cabinets) DC US 4 100A Battery 100A 37 DC US 5 100A

38 Comparisons : Reliability roposed 400Vdc Distribution Fault Tolerance Example TYE OF FAILURE ANY 500KW RECTIFIER OR CONVERTER OR COMBINATION TWO 550KW RECTIFIERS OR CONVERTERS OR COMBINATION 1.65MW BLOCK 1.65MW BLOCK + ADDITIONAL 550KW RECTIFIER ANY FOUR 550KW RECTIFIERS ANY FOUR DC/DC CONVERTERS OR BATTERIES DURING DISCHARGE ANY BLOCK DISTRIBUTION BUS SHORT ANY INUT TRANSFORMER MANY COMBINATIONS OF DIFFERENT SYSTEM COMONENTS AFFECT ON LOAD NONE NONE NONE NONE NONE NONE NONE NONE NONE SYSTEM STATUS REDUNDANCY MAINTAINED REDUNDANCY MAINTAINED REDUNDANCY MAINTAINED LOSS OF REDUNDANCY LOSS OF REDUNDANCY LOSS OF REDUNDANCY ARTIAL LOSS OF REDUNDANCY LOSS OF REDUNDANCY LOSS OR ARTIAL LOSS OF REDUNDANCY CAN TAKE ENTIRE BLOCK FOR SERVICING AND HAVE FAILURES WITH NO AFFECT ON LOAD. 38

39 400Vdc concept : Distribution Distribution Cabling detail Rack C D U Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack FB Electrical Room Subfloor 12.47kV C B Bus A 400Vdc 400Vdc 480Vac BC BC BC DC/DC MV/ LV TXF Input Distr Rect Module Rect Module Rect Module Output Distr 39

40 400Vdc concept : Distribution Distribution Facility cables : 2 conductors in conduit Standard low voltage conductors (600V) Only resistive losses (G) N Busways standard products from Eaton, Siemens Voltage rating 600V Current rating up to 6,000A (GE) DC CBs off busway use 2 poles of std 3 pole device 40

41 Comparisons : Efficiency Elegance in distribution Expect lower conduction losses : 5 vs. 10 breakers 480Vac 3W + Grd 280Vac 4W + Grd 1600A AC US 600A 12.47kV 4000A 1600A 10 breaker in series Load Bank Floor Box 480Vac 3W + Grd 400Vdc 2W Busduct 12.47kV 2500A 1000A DC US 3000A 41 5 breaker in series Floor Box 100A ** It may be possible to eliminate the 2,500A breaker at rectifier input

42 400Vdc concept : Grounding Facility considerations : grounding ositive output rail of rectifier is grounded Circuit breakers required in negative rail only Offer two poles in series (250Vdc rating per pole) Use std 3 pole device possible Many manufacturers : ABB, Square D, GE Rectifier -400VDC + Gnd Busduct High R Battery DC/DC Ground Collector Floor Box MGB ower Strip + - Gnd ** MGB = Master Ground Bar 42

43 400Vdc concept : Grounding Frame Grounding Method 22 Equipment (2 rows) Frame Conductor Raised Floor Single oint Ground Method Floor Box Second Floor Ground Collector MGB Frame Conductor Input Swgr Rectifier Rectifier Rectifier Output Swgr First Floor ** MGB = Master Ground Bar 43

44 Maintenance Maintenance ersonnel access equivalent to AC system ower supply/server replacement Appliance couplers will have UL, CSA and TUV approvals Rack connection and disconnect DC rated IEC 309 or Anderson connectors Rated for hot disconnect, but propose mechanical interlock Rectifier servicing same as AC US Addressed by ability to isolate each rectifier 44

45 45 Back-up

46 ower efficiency modeling and results ower efficiency modeling Analytical model for power delivery efficiency in data centers/telco facilities Compares performance over a wide load range Each conversion stage modeled with three loss parameters 46

47 ower efficiency modeling and results Loss models Measure efficiency and loss Loss can be calculated from given efficiency curve Losses [p.u.] % % % % % 0.04 Calculated loss 85% 0.02 Measured efficiency 83% % Output power [p.u.] Efficiency [%] η = o, pu loss, pu o, pu = o, pu + loss, pu = 1 1 η loss, pu = o o, rated loss o, rated o, pu 47

48 ower efficiency modeling and results Curve fit losses Loss models Quadratic (a 2 ), Linear (a 1 ) and Constant (a 0 ) loss coefficients 2 = a + a loss, pu 2 o, pu 1 o, pu + a0 Losses [p.u.] y = x x % % % % % 0.04 Calculated loss 85% Measured efficiency % oly. (Calculated loss) % Output power [p.u.] Efficiency [%] η = o, pu loss, pu o, pu = o, pu + loss, pu = 1 1 η loss, pu = o o, rated loss o, rated o, pu 48

49 ower efficiency modeling and results Model Work from load upstream, adding losses Arrive at total input power Load [%] Full load eff [%] Rating [W] Voltage [V] AC SU output power [W] AC SU output power [p.u.] AC SU SU loss 1 [p.u.] a SU loss 2 [W] a AC SU input power [W] a Rack cable current [A] Max Voltage drop rack cable [%] 0.25 Max Voltage drop rack cable [V] 0.52 Rack cable R 3 [mohm] Cu ratio Cable loss [W] Cable input power [W] o, pu = 4 o o, rated 2 loss, pu = a2o, pu + a1o, pu + a0 3 4 R cable cable < = V I cable,max 2 o I R o cable D2D AC/DC w/ FC EMI 49

50 ower efficiency modeling and results Curve fit losses Loss models Quadratic (a 2 ), Linear (a 1 ) and Constant (a 0 ) loss coefficients 2 loss, pu = a2o, pu + a1o, pu + a0 Losses [p.u.] y = x x % % % % % 0.04 Calculated loss 85% Measured efficiency 0.02 Efficiency fit 83% oly. (Calculated loss) % Output power [p.u.] Efficiency [%] η = o, pu loss, pu o, pu = o, pu + loss, pu = 1 1 η loss, pu = o o, rated loss o, rated o, pu 50

51 ower efficiency modeling and results Results Observations : Confirms 400Vdc provides highest efficiency Spread 5-10% between all high efficiency options oor light load efficiency No modularity assumed ower delivery efficiency [%] Specialized AC US Facility 48Vdc Facility 400Vdc Load [%] Facility 550V/48Vdc, 400Vac High efficiency 480Vac, Rack 400Vdc, Rack 48Vdc Baseline 480Vac * Where architectures yield very similar results, only one graph is shown. 51

52 ower efficiency modeling and results Full load input power, including cooling Facility 400Vdc Facility 550V/48Vdc Facility 48Vdc Rack-level 400Vdc Rack-level 48Vdc Specialized AC US 400Vac High efficiency AC Baseline AC ~$140k ~$1M Energy cost savings Total facility input power [MW] For 8820 servers with 300 W load power each; ~20% of power is for cooling * 6.2c/kWh 52

53 ower efficiency modeling and results Full load efficiency of power delivery Facility 400Vdc Facility 550V/48Vdc Facility 48Vdc Rack-level 400Vdc Rack-level 48Vdc Specialized AC US 400Vac High efficiency AC ~4% ower delivery loss reduction: 17% Baseline AC ~20% 60% ower delivery efficiency [%] 53

54 Cost Comparison: Assumptions Cost comparison analysis performed for ModA/B with 3.3MW IT load Electrical cost of proposed AC design for ModC design not available Cost comparison for maturity, i.e. high volume Both AC and DC USs are iso-redundant Where commercial equipment is not available : Rectifier : two sources Actual cost estimates from Emerson Network ower for 1.1MW (N+1) Rectifier AC to DC US ratio from Emerson Network ower based on Rectifier module CDU : assume similar cost IEC 309 standard international connector Remove phase balancing complexity SU : fewer components due to removal of FC 1 st pass estimate of similar cost in comparable volumes Emerson Network ower: BOM cost reduction of ~6% Delta: ~1.5x at 20% of AC volume (3 yrs), 1.1x at same volume (5 yrs) ModA used as a baseline 2x 15kV -> 480V transformer upstream of US 3-phase 480Vac input to US 5x 800kVA/720kW US with iso-redundancy 3-phase, 5 wire distribution (3 phases, 1 neutral, 1 ground) 480V/208V transformers used in DU s DC Data Center assumptions 2x 15kV -> 480V transformer upstream of US No change in the distribution and cost upstream of US 3-phase 480Vac input to US 5x 720kW DC US (Rectifier + DC/DC converter in the battery cabinet) with isoredundancy 5 wire distribution (2 positive, 2 negative, 1 ground) Assume standard AC cable, and DC current carrying ability is 67% higher than AC rating Operational cost savings due to reduced energy use is not included 54