mopdf.com

2.2 NETAPP’S STRATEGYFor NetApp Inc. the proper use of Big Data technology in our project will have a positiveimpact on its future business, because the successful deployment of Big Data technology on Eseries products will “necessitates secure, robust and low-cost solutions for data storage andmanagement”, as emphasized in NetApp Strategy Paper.1 When AutoSupport was firstintroduced in NetApp white paper in 2007, it was highlighted that NetApp would have a morethan “65% chance of resolving a customer case in less than one day” instead of only “35%[chances] without AutoSupport data” [10].On the other hand, as the database structure has become increasingly complex, ourstrategy for NetApp in the project is a radical evolution in the industry. MapReduce was themilestone in data mining, processing and management, like Dr. Jeff Ullman claimed in his bookMining of Massive Datasets, “Implementations of MapReduce enable many of the most commoncalculations on large-scale data to be performed on computing clusters efficiently” [11]. Later,the MapReduce methodology was integrated with Hadoop Hive, specifically, HiveQL “which arecompiled into map-reduce jobs executed on Hadoop” as demonstrated by Ashish Thusoo et al. inthe paper entitled Hive - A Warehousing Solution Over a Map-Reduce Framework in 2011 [12].Since then, the tool was tailored to handle large data set and was very powerful, and manycompanies still relied on it. However, we chose to use Berkeley Shark, which was Spark on topof Hadoop Hive with SQL compatibility. One of the special features of Shark was the fact thatShark could implement MapReduce functions approximately a hundred times faster [2], whichwas an ideal choice for fast-paced big data analysis. As illustrated in Table 1 above, with thehelp of Berkeley Shark technology, our data analysis function which required the predictivePage 6

nature and real- time feature over large-scale data set became feasible. This was innovative andwould dramatically improve the user experience of NetApp’s customers.Actually, for all the IT companies in this Big Data era, the key to the success is whetherthe company can master the advanced technology and seize the opportunity in a niche market.Our project on E-series ASUP framework will revolutionize NetApp’s data storage and customersupport business and help the company exploit its niche market in the Big Data industry.Page 7

Chapter 3: MethodologyOne of our tasks in this project was to gain extensive knowledge by researching,analyzing and testing various Big Data technologies for the E-series ASUP framework. Initially,we made our technology selection list with Spark/Shark from Berkeley AMPLab [2], Impala [13]and Parquet [14] from Cloudera, Phoenix from Saleforce.com [15] and Clydesdale from Googleand IBM [16]. We then set up various benchmarks to compare these technologies in order tonarrow down our list. After we finalized the decision to utilize the latest Berkeley Spark/Shark asour key technology, we developed the data storage schema, constructed the data repositorythereafter and parsed the ASUP raw log files into tabular format data for the repository. At thesame time, we made progress on Berkeley Shark configuration based on NetApp’s computingclusters, with which we could store the large-scale parsed data, perform analysis and offerpredictive solutions using machine learning techniques. Since my work is majorly focused ondata parsing, this paper will be centered on data processing accordingly.Page 8

3.1 ASUP ENVIRONMENTFigure 2 on the right is a demonstration of ASUPinfrastructure from NetApp’s AutoSupport documentsonline [4]. NetApp developed this technology manyyears ago, and integrated it with several branded productlines in order to continuously and efficiently monitor theFigure 2: NetApp AutoSupport Infrastructure [4]health of storage systems. It is achieved by constantlysending ASUP reports back to NetApp headquarter and “My AutoSupport” online platform. Asan effective troubleshooting tool, AutoSupport targets both of the NetApp support engineers andproduct customers.Although AutoSupport was already deployed in many other NetApp products, it had notbeen integrated with NetApp’s E-series product line. Since E-series products are becoming oneof NetApp’s featured products, the company is desired to make this integration accomplishedsoon. And that is the goal of our capstone project.Page 9

3.2 DATASETSince one storage system can generate multiple AutoSupport reports continuously in justa short period of time, it is a pressure for us to do data cleaning and analysis. Likely, it is due to ahardware failure or a system warning occurred before. But within each of the AutoSupportreport, most of the contents are duplicated. Therefore, how to efficiently extract the root cause ofthe problem becomes significant.The size of an ASUP dataset varies greatly from a few megabytes to several hundreds ofmegabytes in total, depending on how large the storage system is and whether the AutoSupportdata is a daily log or a system warning type.These are the raw datasets that we used for our capstone project. With access to theNetApp’s repository of AutoSupport raw data, we can continuously collect these data globally.However, to process and integrate the huge dataset demands novel Big Data technologies ratherthan traditional database and data management solutions.3.3 TECHNOLOGY COMPARISONWe made technical comparison of five eligible Big Data technologies, namely BerkeleySpark/Shark, Cloudera Impala and Parquet, Salesforce.com Phoenix and Google Clydesdale.They all have various advantages and disadvantages. And one of our tasks in this project was tonarrow down this list, and made a final decision on which technology we were going to use toPage 10

construct the framework. In order to achieve that goal, we did research on their hardwarelimitations and computing constraints one by one, and list our evaluation standards and results toexamine each single technology.3.4 DATA STORAGE MECHANISMIn order to efficiently organize and store all of the normalized data, we utilized the StarSchema data structure. The Star Schema consisted of fact tables and dimension tables, in whichfact tables stored the central metrics and information, whereas dimension tables were datawarehouse linked to the fact tables.3.5 DATA PARSING AND STORINGAfter choosing Berkeley Spark/Shark technology, it was important to install andconfigure it properly on the NetApp’s company computing cluster. Our computing clusterconsisted of one master node and three worker nodes. And we installed the BerkeleySpark/Shark with the latest release on February 2014 on all of the cluster nodes. With thataccomplished, I began to work on data parser, convert the ASUP raw data into tabular format tostore in data repository.Page 11

3.6 DATA QUERYING AND INSIGHTSLast but not least, we spent time and effort on identifying example use cases forNetApp’s E-series products, and generating insightful data queries. Because this was one of ourkey tasks for the project, we wanted to offer valuable and predictive solutions for our customer.A simple use case would be to collect any drive errors from one system, performinganalysis on its system configuration, record of repairing, capacity usage and device running timeetc., aggregating similar errors and identifying the root cause, and predicting what the next timethat the potential failure would occur. We applied K-means clustering and K- fold crossvalidation machine learning algorithms on our dataset and generated insightful conclusionsaccordingly.Page 12

Chapter 4: Discussion4.1 TECHNOLOGY COMPARISON MATRIXTable 2 below presents the technology comparison results we concluded for five majoradvanced Big Data technologies.Table 2: Technology Comparison Matrix [17]NameCompanyEase of SetupCompatibilitySQL LikeStar SchemaUnstructuredDataAcceleratedStorage FormatBulk Data LoadIn-memoryUDFPredictiveAnalyticsAvailable APIsMaturityNoteUsersOur ChoiceSparkUCB/ApacheEasy Hive ImpalaClouderaEasy Hive PhoenixSalesforceEasy HBase ParquetCloudera/TwitterMediumHadoop ClydesdaleGoogle/IBMHard Not for Nonscalar Data Columnar Columnar Columnar JavaJavaJavaJavaMediumMediumMediumLow In-memorycomputing, fasterdata queries,ideally suited formach ine learningBestintegrationwith ParquetTable Joinfunction notavailable inPhoenixVersion2.1.2Requires extensiveconfiguration, querydependent, notsuitable for mult iplequeriesStill a researchprototype.Performancevariesdepending onquery typeIBM , Yahoo!,Intel, GrouponClouderaSalesforceSalesforce, Couldera,TwitterGoogle, IBM Java, Python,ScalaHigh Page 13

The evaluation standards are:1.Co mpany: The entity who developed and supported such technology2.Ease of Setup: To measure how easy it is for users to setup and configure such technology3.Co mpatibility: To examine whether such technology is compatible with HIVE/ HBase/Hadoop4.SQL Like: To examine whether such technology has the SQL skin, wh ich is easy to develop5.Star Schema: To examine whether such technology supports “Star Schema”6.Unstructured Data: To survey how well such technology handles the unstructured data like t xt7.Accelerated Storage Format: To identify if such technology utilizes Co lu mnar data format8.Bulk Data Load: To examine whether such technology supports large size data bulk loa ding9.In-memory : To observe if such technology has the function of in -memory co mputation10. UDF: To examine if such technology has the User Defined Function features11. Predictive Analytics: To survey whether such technology has the predictive analytics function12. Available APIs: To examine what APIs it supports, like Java, Python or Scala13. Maturity: To measure how mature such technology is, in the level of High, Mediu m and Low14. Extra Note: Other significant features, functions, releasing or development notes15. Users: The examp le of co mpanies/entities who utilize or deploy such technology4.2 DATA PARSINGOne of my major tasks in this capstone project was to parse the raw log files and extractvaluable data from them. At the early stage of our project, we discovered that there was a binaryfile in the log file jar. Utilizing an internal java-based parser, we could convert these binary filesinto semi-structured xml files for preliminary analysis. However, xml file was not valuable to us,because this type of data format was not compatible with databases and none of the machinelearning algorithms could be applied upon. We needed to further normalize these data andconvert them into tabular format, then store them into our databases residing on powerfulcomputing clusters, aggregate them further to perform the predictive analysis using modernPage 14

machine learning algorithms and finally generate insightful solutions. To achieve these goals, wedevoted our effort on creating a new parser based on Python, to convert these xml files into csv(comma-separated values) format with organized data in it.However, the internal binary to xmlparser is just a preliminary version. As we parseddifferent ASUP log files later on, it failed severaltimes. On the other hand, the binary to xml parserhadits drawbackasinefficiencyindataprocessing. Because when we needed to use itevery time, we had to convert those raw log filesinto xml format first and then further transforminto tabular format using the parser developed byour team. This can be illustrated clearly in Figure3.Hence, in consideration of the efficiency ofour AutoSupport ecosystem, we needed to explorean alternative approach. We found that there weremany text files within the same ASUP log jar.Though they are all unstructured data, those text-Figure 3: ASUP Data Processing Using Binary - XML - Tabular Format ApproachPage 15

based data contain almost equally sufficient and valuable information as the binary files. So wedecided to set aside the previous approach, and begin to develop a new parser aiming to parsethese text-based data. This process is clearly illustrated in the appendix.I wrote a parser based on Python, which took in the text files in ASUP log file jar, andgenerated all the tables automatically. The parser would extract all the key words in the text file,like the ID of various physical components, the generation date of the ASUP report, date ofmanufacturing etc. as column names in the table, and the associated value or description to thosekey words would be stored in a tabular format in a csv file.As discussed in section 3.2, the same storage system could continuously generatemultiple ASUP reports in a short period of time. These datasets were mostly alike to each other,so to simply bulk load those in the data repository without proper process might causeoverwriting problems. To deal with such issue, I utilized the “Partitioning” function in Hive, aswell as Shark (because Shark was Spark on top of Hive), to solve this issue. Since the ASUPgeneration date had sufficient precision, we decided to use it as the partition field to differentiatedistinct ASUP data, or the data generated by the same ASUP but in different time period.Page 16

4.3 STAR SCHEMA DATA STRUCTUREI also participated in designing the data structure for the repository. Figure 4 below is thesample Star Schema we created for ASUP data warehousing:As discussedlarge-scaleASUPin previous sections,datawereallstoredfollowing this structure on computing cluster.For different dimension tables, we used IDs ofvarious components as the primary key to linkto the fact tables. And as claimed above, eachtable contained a partition field when storing inFigure 4: Star Schema Structure [17]data repository.Page 17

Chapter 5: ConclusionAt this time, we have successfully designed the data structure, configured Spark/Shark oncomputing clusters, had the majority of ASUP data parsed and cleaned for use, and generatedseveral use cases insights based on machine learning algorithm already.To review my part of work, the biggest difficulty I encountered was to parse theunstructured text-based AutoSupport data. Yet at the same time, I realized that it was importantto have a clean and normalized dataset generated for machine learning application, user interfacedevelopment and future predictive analysis. The difficulty lay in the fact that my parser mightwork well on one version of AutoSupport, but turn out to be a failure totally when testing onother AutoSupport versions, simply due to the new lines added in other AutoSupport versions.To overcome this difficulty, I had to run and test my parser on multiple AutoSupport versionsone by one. Fortunately, I found out that all of the AutoSupport reports were in a “normalized”format to some degree. The total amount of information, or the column names in tabular dataformat after conversion, was set and fixed. The only difference was that some AutoSupportversions tended to omit certain hardware information, which might not be configured in thestorage system. Therefore, I drawn the conclusion that to develop a parser for such unstructuredtext format data, it was important to aggregate all of the possible information first, no matterwhether it existed in the current dataset or not. Furthermore, some parsing techniques, like thelook- up table mechanism, should be used to parse the dataset completely, instead of parsing thedataset line by line or by searching key words in it.Page 18

In the future, continued work can be done in the areas of implementing more machinelearning algorithms on the whole set of E-series AutoSupport data, constructing a friendly userinterface for potential customers, and continuing working to make the E-series AutoSupportecosystem more efficient and robust.Page 19

Appendix A: Data Parser Presentation Slides233 Referencing “NetApp Capstone Team Final Presentation” in May, 2014Page 20

Page 21

Page 22

Bibliography[1] -systems/e5400/e5400-productcomparison.aspx[2] [Online] http://spark.apache.org/research.html[3] [Online] http://www.idc.com/getdoc.jsp?containerId prUS24302513[4] [Online] ort.aspx[5] plorer/index.htm#content/which/analytics/, last access Jan. 30, 2014.[6] ateral/ns340/ns1176/datacenter/BigData Case Study-1.html/, last access Jan. 31, 2014.[7] [Online]. http://www-01.ibm.com/software/data/bigdata/, last access Jan. 31, 2014.[8] [Online].http://storageeffect.media.seagate.com/ ion-a-case-study/, last access Feb. 1, 2014.[9] igital/, last access Feb. 2, 2014.[10] "Proactive Health Management with AutoSupport" NetApp White Paper. NetworkAppliance, Inc. Technical Report. WP-7027-0907. Sept. 2007.[11] J. Ullman, "Mining of Massive Datasets" pp.19, Cambridge University Press,December 30, 2011.[12] A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoffand R. Murthy, “Hive - A Warehousing Solution Over a Map-Reduce Framework”.Proceedings of the VLDB Endowment. Vol. 2 Issue 2. Pp. 1626-1629. August ] [Online] http://parquet.io/[15] [Online] https://github.com/forcedotcom/phoenix[16] T. Kaldewey, E. Shekita, S. Tata, “Clyd

2Using NetApp internal AutoSupport data search engine . Page 3 1) Helped to investigate and understand the hardware configuration of NetApp’s E-series product and how ASUP worked. 2) c Participated in designing t