mopdf.com

Dragon Slayer ConsultingIntroduction to the Value Proposition ofInfiniBandMarc Staimer – marcstaim[email protected](503) 579-37635/27/2002Dragon Slayer Consulting1

Introduction to InfiniBand (IB) Agenda IB definedIB vs. FC & GbEIB architectureReal market problems IB solvesMarket projectionsConclusions5/27/2002Dragon Slayer Consulting2

Definition of Input/Output “The transfer of data into and out of a computer”Maintain data integrity Protect all other data in the computer from corruption Through the use of Operating System defined mechanisms Usually5/27/2002Dragon Slayer Consulting3

Three (3) Distinct Classes of I/O Block protocol Network protocol Typically disk orientedTypically IP orientedInter-Process Communication IPC5/27/2002Dragon Slayer Consulting4

Characteristics I/O ClassesBlock ProtocolLatencyToleranceAvg ork ProtocolDozens of milliseconds 100s of MillisecondsVery largeData center/campusFCFibre ChannelProtocol (FCP)IPCDozens ofMicrosecondsSmall to largeSmall to largeGlobalServer cluster/datacenterEthernet / TCP/IPEmerging - VIDragon Slayer Consulting5

IB Defined The 1st unified, simplified, & consolidated I/O FabricDesigned from the ground up for all aspects of I/O Shared memory vs. shared bus Leverages virtual lanes or pipes (multiple fabrics in one) Spec’d for today & tomorrow 1x 2.5Gbps 4x 10Gbps 12x 30Gbps InfiniBand (VI Protocol)Native VI protocol OS bypassCredit based flow control Key: extends server I/O FC GbE Outside the box5/27/2002UltraSCSI SATA SASVirtual LanesDragon Slayer Consulting6

Why Do We Need Yet Another Fabric? The issue is not the fabric, the issue is server I/OCurrent GbE & FC fabrics do not solve server I/O bottlenecks Bus contentionGbE & FC fabrics weren’t specifically designed for clustering They can do it AND Message queue depths and performance not optimal Performance is often inadequate5/27/2002Dragon Slayer Consulting7

IB vs. FC vs. GbE Conclusion Initially complimentary – IB will not replace FC or GbE Investment protectionEventually competitive and complimentary They will compete for some of the same budget dollars5/27/2002Dragon Slayer Consulting8

IB ArchitectureTarget Channel AdapterInterface to I/O ControllerFC, GbE, SCSI, eSwitchSysMemSwitchTCAHost Channel AdapterProtocol EngineMoves data via messagesqueued in memoryRouterRouterControllerInternal or ExternalSimple, low cost Multi-stage5/27/2002LinkI/OCntlrLinkCPUSystem s subnets togetherDragon Slayer Consulting9

IB Fabric BW Increases as Switches are eEndNodeEndNodeSubnet BDragon Slayer Consulting10

I/O Architecture Today Traditional Server & Infrastructure w/dedicated I/OComplex and expensiveEthernetLANTraditional Server ArchitectureRouterLocal Bus (PCI)NICMemorySystem geIPCCPUIPCiSCSI?5/27/2002IPCNetworkDAFS?Dragon Slayer Consulting11

InfiniBand Based I/OInfiniBand ServerHardware ArchitectureMultiple IBA links 2.5 Gbps 10 GbpsMemorySystem BusCPUMemoryControllerHostChannelAdapterSolve redundancy problem onceCPU5/27/2002Dragon Slayer Consulting12

InfiniBand Based I/OInfiniBand ServerHardware ArchitectureInfiniBand I/O UnitHardware ArchitectureEthernet I/OControllerMemorySystem tchTargetChannelAdapter(TCA)CPUFibre ChannelI/O ControlleriSCSII/O ControllerUltraSCSII/O ControllerRDMA based protocols5/27/2002Dragon Slayer Consulting13

Market Problems IB Solves Higher performance lower cost I/O (Shared I/O) Converges clustering, networking, & storage into one fabric The IAN (I/O Area Fabric) Reduces: IT management tasks Server workloads TCO PCI Bus I/O constraintsLow cost HP/HA server clustering Lowers the cost of server blade systems Enables higher density server blade clusters5/27/2002Dragon Slayer Consulting14

Higher Performance Lower Cost I/O (Shared I/O)New IBServerClustersFCSANIB FabricI/O UnitEthernetLAN/WANTCAIB FCIB E-netIB IBIBStorageIB iSCSIMgtLAN5/27/2002ModemRemoteMonitoringDragon Slayer ConsultingEthernetSAN15

Current High Availability I/O Configuration 16 Rack mount servers with dedicated I/O per Edge2450PowerEdge2450PowerEdge2450MaintenanceLAN 210 connections 5/27/2002(2) HBA FC paths/server to FC fabric(4) FC paths to storage to FC fabric(2) Ethernet paths/server to network(1) Ethernet maint path/server to networkDragon Slayer Consulting16

Non-Productive Costly ConnectivityProductive eLAN5/27/2002Non-productive connectivityDragon Slayer Consulting17

InfiniBand Shared I/O Chassis Example 19” rack mount environment3U highIBA single high single wide stdIntegrated IBA fabricUp to 45 watts / linecard slotHot swappable componentsChassis Management Entity(CME)Front-to-Back cooling3UFabric CardInfiniBand ports5/27/2002Line CardsUp to 8 - GE/FC/IB portsDragon Slayer ConsultingFabric CardInfiniBand ports18

IB Enabled High Availability Shared I/O Add dual redundant IB I/O e2450PowerEdge2450Eliminate FCEdge owerEdge2450PowerEdge2450MaintenanceLAN 10 slots each IB form factor I/O cardsMulti-protocol FC, GigE, FastE, iSCSI, etc. 5/27/2002Dragon Slayer ConsultingEliminate FC edge switches19

IB Enabled High Availability Shared I/O Reduces LAN switch eLAN5/27/2002 Total Connections 116 45% reduction (2) IB paths/server - IB fabric (6) FC paths to storage - FC fabric (2) Ethernet paths/I/O subsystem – network (2) E-net maint path/I/O subsystem - networkDragon Slayer Consulting20

Potential Savings Current dedicated I/O subsystem/server Costs 225,000IB shared I/O System with Improved BW, connectivity, manageability, availabilityCosts 112,500 Savings 50% Additional non-hardware TCO gains Operational Expense Estimated at 3x – 8x Capital Expense reduction Simpler system design to manage5/27/2002Dragon Slayer Consulting21

System Benefits Increased BW & connectivity per serverReduced infrastructure complexityReduced power & spaceBW migration to bursting serversNatural low latency IPC network5/27/2002Dragon Slayer Consulting22

Managing Scalability w/Traditional I/O What happens when just 2 more servers are erEdge2450MaintenanceLAN In the FC SAN Maintenance LAN will also need to change 5/27/2002(1) new switch has to be addedFabric will need to be reconfiguredFrom a 16-pt switch/router to 24-portDragon Slayer Consulting23

Managing Scalability w/Traditional I/O Adding servers takes a lot of hard work & Edge2450MaintenanceLAN 5/27/200220 Net new connectionsDisruptive FC fabric reconfigurationsDragon Slayer Consulting24

Managing Scalability w/IB based I/O Adding additional servers is significantly simpler & erEdge2450MaintenanceLAN5/27/2002 8 net new connections a 60% reduction (2) IB paths/new server - IB fabric No new switches or reconfigurations Faster & non-disruptive implementationDragon Slayer Consulting25

Scalability Net Results w/IB shared I/O Making adds, moves, or changes meansLess time Less cost Less effort Less complexity Less personnel Less disruptions More control More simplicity More stability Better RAS 5/27/2002Dragon Slayer Consulting26

PCI Bus Constraints PCI bus limitations have been strangling CPU I/O Like trying to drink from a fire hose5/27/2002Dragon Slayer Consulting27

PCI Bus ConstraintsPCIMax BWI/O ConstraintArchitectureIssues5/27/2002PCI (66Mhz)PCI-X(133 Mhz)DDRQDR3GIO4 Gbps8 Gbps16Gbps32Gbps64Gbps(1) 4x IB(1) 10gigE,FC, or 4x IB(2) 10gigE,FC, or 4x IB(4) GbE w/TOE(4) SCSI 320or (2) 2gig FCShared Parallel BusSwitchedserialBus contentionNot until 04Dragon Slayer Consulting28

PCI Bus vs. IBComparisonPCI PCI-X DDRQDR 3GIOAdvantagesLower costSimpler for chip-to-chipProtects software baseDisadvantagesUntil there is 3GIO, buscontentionClusteringScalability:Ports & BWOut-of-box connectivityInfiniBandQoSSecurityFault ToleranceSoftwareMulti-castFabric ConvergencePCB, Copper, & Fiber5/27/2002Dragon Slayer Consulting29

Solution: PCI Bus AND IB It’s not “either:or” The best solutions takes advantage of both They are complimentary not mutually exclusiveThis is why you rarely hear anymore that IB is the PCI replacementThere are new HCAs WITH PCI-X interfacesExpect DDR, QDR, & 3GIO as well The IB benefits are almost as great Eliminates bus contention Preserves PCI software base Provides IB benefits NOW Don’t have to wait for native server IBPCI-X HCA Example5/27/2002Dragon Slayer Consulting30

Low Cost HP/HA Server Clustering IB clustering costs less for scaling out than SMP or NUMA scaling upIB eliminates fabric messaging performance Issues with clustering Long queuesPCI bus contentionIB enables low cost server (shared I/O arguments even stronger here) Diskless blades Personality on the storage Higher Fault Tolerance and Availability One connection for clustering and shared I/O Less I/O interfaces than any other interconnect Higher performance Lower TCO5/27/2002Dragon Slayer Consulting31

Industry Analyst’s IB Enabled Server ForecastAnalysts are split in their forecast of IB’s TAM; but, not on its 00200220032004Gartner5/27/20022005IDCDragon Slayer Consulting32

IB Enabled Servers as a % of 0.0%200220032004Gartner5/27/20022005IDCDragon Slayer Consulting33

Conclusions Even if the analysts views are optimisticHuge % of servers will be I/B enabled The value proposition is far too strong to ignore Initial deployment will utilize PCI-X HCAs Native deployments will enable lower cost server blade clusters As more and more servers become IB enabled Clever IT people will realize that they can run IB native for: Clustering, Networking, and Storage When IB becomes native with the server motherboard The perception becomes that it’s free There is always high market demand for free.5/27/2002Dragon Slayer Consulting34

Dragon Slayer Consulting?Questions?5/27/2002Dragon Slayer Consulting35

Why Not Just Use GbE or FC? GbE and FC are the current fabric infrastructuresIT personnel already know & understand the technologiesFC & GbE are already battling it out for SAN infrastructures FCP vs. iSCSI5/27/2002Dragon Slayer Consulting36

IB vs. FC vs. GbETechnologyGigabitEthernetStandardsBodyIEEE & 2SignalingSpeedFirstStandard1.25 GbpsMaximumFrame SizePrimaryApplication1.5KLAN: LocalArea Network19882KSAN: StorageArea Network20014KIAN: I/O AreaNetwork19992.125 GbpsInfiniBand 2.5Gbps (1x)10Gbps (4x)TradeAssociation 30Gbps (12x)Dragon Slayer Consulting37

How IB compares w/GbE & FC in OSIUpper LevelProtocolsTransportLayerEthernet TCPUDP(FC-3)NetworkLayerIPLogical Link ControlLink LayerPhysicalLayerFC-2:FramingService ClassNetworkLineEncodingMedia Access ControlFC-1:EncodingMedia AccessControlPhysical Layer EntitiesFC-0:PhysicalMediaPhysical5/27/2002Dragon Slayer Consulting layers not included in theprotocol standards38

Data Center Fabric & I/O Consolidation IB enables convergence through shared server I/O One I/O interface for Clustering Network Storage Eliminates the need for multiple server I/O blades/ports IB virtual lanes provides Multiple independent logical fabrics multiplexed on one physical one QoS to prioritize traffic The benefits of independent fabrics with: The management and maintenance of one fabric Switches, directors, and routers provide Scalability, redundancy, availability, and flexibility5/27/2002Dragon Slayer Consulting39