mopdf.com

16 Dell Ready Bundle for Cloudera Hadoop OverviewIntroducing the Dell Ready Bundle for Cloudera HadoopAlthough Hadoop is popular and widely used, installing, configuring, and running a production Hadoopcluster involves multiple considerations, including: The appropriate Hadoop software distribution and extensionsMonitoring and management softwareAllocation of Hadoop services to physical nodesSelection of appropriate server hardwareDesign of the network fabricSizing and scalabilityPerformanceThese considerations are complicated by the need to understand the type of workloads that will be runningon the cluster, the fast-moving pace of the core Hadoop project and the challenges of managing a systemdesigned to scale to thousands of nodes in a single cluster.Dell’s customer-centered approach is to create rapidly deployable and highly optimized end-to-end Hadoopsolutions running on hyperscale hardware. Dell listened to its customers and designed a Hadoop solutionthat is unique in the marketplace, combining optimized hardware, software, and services to streamlinedeployment and improve the customer experience.The Dell Ready Bundle for Cloudera Hadoop was jointly designed by Dell and Cloudera, and embodies allthe hardware, software, resources and services needed to run Hadoop in a production environment. Thisend-to-end solution approach means that you can be in production with Hadoop in a shorter time than istypically possible with homegrown solutions.The solution is based on the Cloudera Enterprise and Dell PowerEdge and Dell Networking hardware. Thissolution includes components that span the entire solution stack: Dell Ready Bundle for Cloudera Hadoop Architecture Guide and best practicesOptimized server configurationsOptimized network infrastructureCloudera EnterpriseSolution Use Case SummaryThe Dell Ready Bundle for Cloudera Hadoop is designed to address the use cases described in Table 1:Big Data Solution Use Cases on page 16:Table 1: Big Data Solution Use CasesUse caseDescriptionBig data analyticsAbility to query in real time at the speed of thoughton petabyte scale unstructured and semi-structureddata using HBase and Hive.Data storageCollect and store unstructured and semi-structureddata in a secure, fault-resilient scalable data storethat can be organized and sorted for indexing andanalysis.Dell Ready Bundle for Cloudera Hadoop

Dell Ready Bundle for Cloudera Hadoop Overview 17Use caseDescriptionBatch processing of unstructured dataAbility to batch-process (index, analyze, etc.) tensto hundreds of petabytes of unstructured and semistructured data.Data archiveActive archival of medium-term (12–36 months)data from EDW/DBMS to expedite access, increasedata retention time, or meet data retention policiesor compliance requirements.Big data visualizationCapture, index and visualize unstructured andsemi-structured big data in real time.Search and predictive analyticsCrawl, extract, index and transform semi-structuredand unstructured data for search and predictiveanalytics.The Dell Ready Bundle for Cloudera Hadoop with SyncSort is designed to address the use casesdescribed in Table 2: ETL Solution Use Cases on page 17:Table 2: ETL Solution Use CasesUse caseDescriptionETL offloadOffload ETL processing from a RDBMS or enterprise data warehouseinto a Hadoop cluster.Data warehouse optimizationAugment the traditional relational management database or enterprisedata warehouse with Hadoop. Hadoop acts as single data hub for all datatypes.Integration with datawarehouseExtract, transfer and load data into and out of Hadoop into a separateDBMS for advanced analytics.High-Performance datatransformationsIncludes high-performance sort, joins, aggregations, multi-key lookup,advanced text processing, hashing functions, and source/record/fieldlevel operations.Mainframe data ingestion &translationReads files directly from the mainframe, parses and transforms the data– packed decimal, occurs depending on, EBCDIC/ASCII, multi-formatrecords, and more –- without installing any software on the mainframeand without writing any code.Solution ComponentsFigure 1: Dell Ready Bundle for Cloudera Hadoop Components on page 18 illustrates the primarycomponents in the Dell Ready Bundle for Cloudera Hadoop.Dell Ready Bundle for Cloudera Hadoop

18 Dell Ready Bundle for Cloudera Hadoop OverviewFigure 1: Dell Ready Bundle for Cloudera Hadoop ComponentsThe Dell PowerEdge servers, Dell Networking switches, and the operating system make up the foundationon which the solution software stack runs.The Store layer components provide multiple layers of functionality on top of this foundation. TheHadoop Distributed File System (HDFS) provides the core storage for data files in the system. HDFS is adistributed, scalable, reliable and portable file system. Apache Kudu provides a columnar relational storageoption, while Apache HBase provides NoSQL access to storage. Object storage is also available.The Integrate layer shows the components that can be used to move data in and out of the Hadoopsystem. Apache Sqoop provides data transfer to and from relational databases while Apache Flume andApache Kafka are optimized for real-time processing event and log data. The HDFS API and tools can alsobe used to move data files to and from the Hadoop system.YARN provides a resource management framework for running distributed applications under Hadoop. Themost popular distributed application is Hadoop’s MapReduce, but other applications also run under YARN,such as Apache Spark, Apache Hive, Apache Pig, etc. Enterprise grade Security services are provided byApache Sentry and RecordService.The right side of the diagram shows the data management capabilities that are integrated across the entiresystem, while the left side shows the operational components for Hadoop administration and managementprovided by Cloudera Manager and Cloudera Director.Sitting atop the Cloudera Enterprise core are multiple complementary processing and access alternatives. Batch data processingStream data processingSQL queryData searchAccess APIAll of these layers can be used simultaneously or independently, depending on the workload and problemsbeing solved.Dell Ready Bundle for Cloudera Hadoop

Dell Ready Bundle for Cloudera Hadoop Overview 19Table 3: Data processing and access componentsAccess LayerDescriptionBatch data processingSpark, Hive, Pig, MapReduce provide access to the massivelyparallel Hadoop data processing framework.Stream data processingSpark can also be used for stream processing.SQL queryImpala provides SQL query access to data.Data searchApache SOLR provides real-time search of indexed data.General AccessKite provides a high level data API for Hadoop.ETL Solution ComponentsThe ETL Solution is a variation of the architecture that adds Syncsort DMX-h to the installation. Figure 2:ETL Solution Components on page 19 illustrates this variation.Figure 2: ETL Solution ComponentsThe Syncsort DMX-h ETL engine runs under the YARN Resource management framework, and addsscalable ETL processing to the cluster. DMX-h can access mainframe and RDBMS data, while events canbe handled using either Flume or Kafka.DMX-h coexists with all other available Hadoop components, andcan be used in conjunction with them.Software SupportTable 4: Dell Ready Bundle for Cloudera Hadoop Support Matrix on page 19 describes where you canobtain technical support for the various components of the Dell Ready Bundle for Cloudera Hadoop.Table 4: Dell Ready Bundle for Cloudera Hadoop Support MatrixCategoryComponentVersionAvailable SupportOperating SystemRed Hat EnterpriseLinux Server7.3Red Hat Linux supportDell Ready Bundle for Cloudera Hadoop

20 Dell Ready Bundle for Cloudera Hadoop OverviewCategoryComponentVersionAvailable SupportOperating SystemCentOS7.3Dell Hardware supportJava Virtual MachineSun Oracle JVMJava 7 (1.7.0 67)N/AJava 8 (1.8.0 60)HadoopCloudera Enterprise5.10Cloudera supportHadoopCloudera Manager5.10Cloudera supportHadoopCloudera Navigator2.9Cloudera supportETL EngineSyncsort DMX-h9.2Syncsort supportCloudera Enterprise Software OverviewCloudera Enterprise helps you become information-driven by leveraging the best of the open sourcecommunity with the enterprise capabilities you need to succeed with Apache Hadoop in your organization.Hadoop for the EnterpriseDesigned specifically for mission-critical environments, Cloudera Enterprise includes CDH, the world’smost popular open source Hadoop-based platform, as well as advanced system management and datamanagement tools plus dedicated support and community advocacy from our world-class team of Hadoopdevelopers and experts. Cloudera is your partner on the path to big data.Cloudera Enterprise, with Apache Hadoop at the core, is: Unified – one integrated system, bringing diverse users and application workloads to one pool of dataon common infrastructure; no data movement requiredSecure – perimeter security, authentication, granular authorization, and data protectionGoverned – enterprise-grade data auditing, data lineage, and data discoveryManaged – native high-availability, fault-tolerance and self-healing storage, automated backup anddisaster recovery, and advanced system and data managementOpen – Apache-licensed open source to ensure your data and applications remain yours, and an openplatform to connect with all of your existing investments in technology and skillsRethink Data Management One massively scalable platform to store any amount or type of data, in its original form, for as long asdesired or requiredIntegrated with your existing infrastructure and toolsFlexible to run a variety of enterprise workloads - including batch processing, interactive SQL,enterprise search and advanced analyticsRobust security, governance, data protection, and management that enterprises requireWith Cloudera Enterprise, today’s leading organizations put their data at the center of their operations,to increase business visibility and reduce costs, while successfully managing risk and compliancerequirements.Dell Ready Bundle for Cloudera Hadoop

Dell Ready Bundle for Cloudera Hadoop Overview 21What's Inside?Table 5: Included ProductsProductDescriptionCDHAt the core of Cloudera Enterprise is CDH, which combinesApache Hadoop with a number of other open sourceprojects to create a single, massively scalable systemwhere you can unite storage with an array of powerfulprocessing and analytic frameworks.Automated Cluster Management - Cloudera Cloudera Enterprise includes Cloudera Manager to helpManageryou easily deploy, manage, monitor, and diagnose issueswith your cluster. Cloudera Manager is critical for operatingclusters at scale.Cloudera SupportGet the industry’s best technical support for Hadoop.With Cloudera Support, you’ll experience more uptime,faster issue resolution, better performance to support yourmission critical applications, and faster delivery of theplatform features you care about.Cloudera Enterprise Data HubCloudera Enterprise also offers support for several advanced components that extend and complement thevalue of Apache Hadoop:Table 6: Advanced ComponentsComponentDescriptionOnline NoSQL – HBaseHBase is a distributed key-value store that helpsyou build real-time applications on massive tables(billions of rows, millions of columns) with fast,random access.Analytic SQL – ImpalaImpala is the industry’s leading massively-parallelprocessing (MPP) SQL engine built for Hadoop.Search – Cloudera SearchCloudera Search, based on Apache Solr, letsyour users query and browse data in Hadoop justas they would search Google or their favorite ecommerce site.In-Memory Machine Learning and StreamProcessing – Apache SparkSpark delivers fast, in-memory analytics and realtime stream processing for Hadoop.Data Management – Cloudera NavigatorCloudera Navigator provides critical enterprise dataaudit, lineage, and data discovery capabilities thatenterprises require. Cloudera Navigator includesActive Data Optimization (Cloudera NavigatorOptimizer), Governance & Data Manage

Dell Ready Bundle for Cloudera Hadoop Architecture Guide and best practices Optimized server configurations Optimized network infrastructure Cloudera Enterprise Solution Use Case Summary The Dell Ready Bundle for Cloudera Hadoop is designed to address the use cases desc