Transcription

WhitepaperHow To Integrate iDRAC9 Telemetry Data IntoThe Splunk PlatformAbstractDell EMC PowerEdge Servers running iDRAC9 version 4.0 or higher with the Datacenter license canstream DMTF Redfish telemetry data. This information helps IT administrators better understand the innerworkings of their server environment. Telemetry data, simply put, is a series of timestamped numbers thatrepresent different data points about your server.This technical paper explains the steps required to setup Splunk to consume iDRAC9 telemetry. We willassume that you have Splunk set up but, if you do not, we will provide an abbreviated setup to assistgetting a Splunk environment set up.There are multiple methods for streaming the DMTF Redfish telemetry from each server as well as forimplementing the Splunk environment. This is not an exhaustive document for all the different options.Additionally, services required in this implementation guide could be deployed on standalone hardware,virtualized and/or containerized (i.e. Docker) depending upon specific customer environments and needs.September 2020

RevisionsDateDescription9/24/2020Version 1.0AcknowledgementsThis paper was produced by the following:NameKim KinahanDell TechnologiesMichael BrownDell TechnologiesKevin TollyThe Tolly GroupMatheus VieiraDell TechnologiesTim PaclDell TechnologiesKyle PrinsDell TechnologiesDean JacksonSplunk Inc.Addison LawrenceDell TechnologiesRafael MarreroTorresDell TechnologiesTanuj ArcotDell TechnologiesChris StahlDell Technologies2 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

Table of ContentsAbstract . 1Overview . 4iDRAC9 Setup. 6Infrastructure Setup – Splunk . 7Splunk Add-on For Redfish Telemetry Reports Configuration. 83 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

OverviewThe promise of “big data” can only be realized when that data is captured, stored and analyzed.Dell EMC PowerEdge Servers provide a stream of telemetry data via iDRAC9 4.x in conjunction with aDatacenter license.1 The Splunk platform can be used both to store that data from servers as well as toanalyze that data visually. Telemetry streaming is more scalable and efficient than prior methods. Fordetails concerning the efficiency, you can see the Tolly report on this topic.2While iDRAC9 and Splunk are the only two systems needed for the end-to-end solution, there aremultiple steps involved in bringing massive volumes of granular data to life. The functions span thesecomponents (and are discussed below): Data Generation (iDRAC9), Ingress Collectors, AnalysisDatabase, and Visualization. These are shown in the following figure:1iDRAC9 with v4.0 or later firmware and the Datacenter license are minimum server requirements for building thesystem described in this document.2See the “Telemetry” tab under the Resources/White Papers section at www.dell.com/support/idrac4 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

iDRAC9 Telemetry OverviewThe iDRAC9 supports the standard DMTF Redfish “Telemetry” interface. You can read more informationabout this in the iDRAC Telemetry whitepaper. 3 This document, however, will guide you through a basicsetup, specific for Splunk, that integrates the full power of iDRAC9 telemetry.The telemetry interface in iDRAC9 will build “Metric Reports.” These reports are JSON documents thatare easy to consume programmatically, but are also human-readable text. The main idea of the reports isthat they have data similar in nature to the table below:Metric eReadingTemperatureReadingDeviceSystem Inlet TempSystem Inlet TempSystem Inlet TempSystem Inlet TempTimestamp2020-08-27 08:50:012020-08-27 08:51:012020-08-27 08:52:012020-08-27 08:53:01Value22242627As one can see by looking at the sample data, it is easy to build things like graphs or other visualizationsand do things like trend analysis or even more advanced things like predictive analytics.Splunk PlatformSplunk is a commercial solution that can gather and store lots of data of different types: unstructured textlogs as well as “metrics.”4 Splunk then can perform search, analysis, even going up to advancedpredictive alerting on that stored data in real time. When you set up Splunk with “Metric” collection, moreadvanced analytics and visualization options are available. It is not the intent of this paper to provide fulland complete Splunk setup and configuration, however, we will go over a brief setup for a containerizedSplunk environment just for completeness. We will show how to “ingest” the iDRAC9 Telemetry data aswell as the iDRAC9 logs and alerts, so you have both the unstructured text and the metric data available.In this paper, we will set up a Splunk add-on from Splunkbase to handle Telemetry ingest as well as theHTTP Event Collector (HEC) to handle logs and alerts. We will also go over setup for Splunk visualizationand “dashboard” type additions that are also posted on Splunkbase. References below for all the utilitiesthat we will be using in this doc.Splunk: “Setting up Splunk in a priseinsideDockercontainersAdditional, supplemental information on setting up s://www.splunk.com/en ers-and-kubernetes.htmlSplunk: HTTP Event Collector /8.0.5/Data/UsetheHTTPEventCollectorSplunkbase –Redfish Telemetry App:https://splunkbase.splunk.com/app/5228/34See the “Telemetry” tab under the Resources/White Papers section at www.dell.com/support/idracThis add-on does not use Splunk “metrics.”5 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

Splunkbase – Redfish Telemetry iDRAC9 SetupThe following steps should be performed on each iDRAC9. Download the Dell telemetry utilities. Thesewill simplify configuration and let you script installation across many iDRACs without requiring manualGUI setup. The utilities are hosted at GitHub at the following URL: tep 1: Download the telemetry utilities: wget archive/master.zip -O iDRAC-TelemetryScripting-master.zip unzip iDRAC-Telemetry-Scripting-master.zip cd iDRAC-Telemetry-Scripting-masterStep 2: Enable Telemetry and Metric Reports. Note in the command below, replace target with the IPaddress or DNS name of the iDRAC9, replace user with an iDRAC9 username with administratorprivileges, and replace password with the specified user’s password. python3 Reports.py -ip target -u user -p passwordINFO:root:Successfully pulled configuration attributesINFO:root:iDRAC Telemetry is currently 'Enabled'.INFO:root:Successfully 'Enabled' iDRAC Telemetry and all reports.Step 3 (Optional, see note below): Enable Redfish Logs and Alerts: this step will enable Redfish alerting,turn on the ability for iDRAC9 to publish Lifecycle Logs and Alerts, and also set up IDRAC9 to forward allof these to your Splunk server. (If setting up Splunk from this doc, come back to this step after you haveset up your Splunk instance.) Note in the command below, replace target with the IP address or DNSname of the iDRAC9, replace user with an iDRAC9 username with administrator privileges, and replace password with the specified user’s password. Additionally, replace splunkserver with the IP addressor DNS name of your Splunk HTTP Event Collector instance. python3 SH.py -ip target -u user -p password-c y -D https:// splunkserver/services/collector/raw -E Alert -V Event- WARNING, checking current value for iDRAC attribute "IPMILan.1.AlertEnable"- WARNING, current value for iDRAC attribute "IPMILan.1.AlertEnable" is set to Disabled, settingvalue to Enabled- PASS, PATCH command passed to set iDRAC attribute "IPMILan.1.AlertEnable" to enabled- PASS, iDRAC attribute "IPMILan.1.AlertEnable" successfully set to Enabled- PASS, POST command passed, status code 201 returned, subscription successfully set forEventServiceThe indication to look for with the above command, is the bolded last line, “status code 201, subscriptionsuccessfully set for EventService.”6 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

NOTE: The step to enable Redfish Logs and Alerts is not strictly necessary for pure iDRAC9 Telemetryprocessing in Splunk. This step will forward all iDRAC9 Lifecycle Logs to your Splunk instance. We findthat most customers want to have all Lifecycle Logs forwarded to Splunk, so this step is recommendedbut optional if you only want Telemetry.Infrastructure Setup – SplunkSection 1:This section will walk you through a basic Splunk container setup. You can skip this if you already have aSplunk environment setup, or you can use this to set up a separate test environment to understand thisnew data before bringing it into your production environment.Splunk publishes a Docker container with a simple installation, the details are here:https://hub.docker.com/r/splunk/splunk/, and the below steps are taken from that document.Install Docker: See your Linux OS documentation or Docker documentation for how to install Docker onyour server.Start Docker and enable on boot systemctl start docker systemctl enable dockerStep 1: download the Splunk Docker image: docker pull splunk/splunk:latestStep 2: run the Docker image (replace password below with your own unique password). Note thatbelow you are accepting the Splunk licensing terms by specifying the –accept-license parameter. docker run -d -p 8000:8000 -e "SPLUNK START ARGS --accept-license" -e "SPLUNK PASSWORD password "--name splunk splunk/splunk:latestAfter the Docker image is running, you can access your new Splunk install at http://localhost:80007 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

Splunk Add-on For Redfish Telemetry ReportsConfigurationNote to reader: This section is intended for clustered environments where one needs to load the app filesfor all servers. For standalone implementations, one can use the GUI to install the app.StandaloneTo install the app in a standalone environment1. Click on Apps menu, click Find More Apps.2. Search keywords “Redfish Telemetry”3. Click Install.4. A message will state Splunk needs to restart. Click Restart.Clustered EnvironmentAfter your Splunk Docker container is up and running, we will install the Add-on For Redfish TelemetryReports. This add-on will be responsible for running in the Splunk environment, contacting the individualiDRACs and streaming telemetry from iDRAC9 to import into Splunk as Metrics.To install apps and add-ons from within Splunk Enterprise1. Log into Splunk Enterprise (Heavy Forwarder server’s web interface).2. On the Apps menu, click Manage Apps.3. Click Install app from file.4. In the Upload app window, click Choose File.5. Locate the tar.gz file you downloaded from Splunkbase5, and then click Open or Choose.6. Click Upload.7. Click Restart Splunk, and then confirm that you want to restart.To install apps and add-ons directly into Splunk Enterprise1. On the Heavy Forwarder server, put the downloaded file inthe SPLUNK HOME/etc/apps directory.2. Untar and ungzip y-on, using a tool like tar -xvf (on *nix) or WinZip (on Windows).3. Restart Splunk.Perform either one of the following set of instructions, depending upon if you are using a standalone orclustered environment.Search Head (Standalone environment)1. Deploy the add-on to your Search Head server (Please refer to Heavy Forwarder installationsteps.6)2. Once done, navigate to SPLUNK 3. Copy props.conf file over from default folder into ./local/1. Create local directory if it doesn’t exist.4. Edit the props.conf file, under local directory, and add the following parameters under each oneof the redfish AddMcafeeCloud/InstallHWF8 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

KV MODE noneAUTO KV JSON false5. Restart the Search Head.Search Head (Clustered environment)1. Download and deploy the add-on to your Search Head Deployer under the following location SPLUNK HOME/etc/shcluster/apps2. Once done, navigate to SPLUNK k/default3. Copy props.conf file over from default folder into ./local/1. Create local directory if it doesn’t exist.4. Edit the props.conf file, under local directory, and add the following parameters under each oneof the Redfish stanzas:KV MODE noneAUTO KV JSON false5. Push the changes from the Deployer server to the Search Head Cluster members by running thefollowing command:splunk apply shcluster-bundle -target https:// search head cluster captain :80896. This procedure will trigger a rolling restart of your cluster; once done, the changes are in effect.To configure new inputs from Redfish add-on for Splunk:1.2.3.4.5.Once installed, open Redfish add-on for Splunk from Splunk UINavigate through Configuration tabOn Account tab, click the Add buttonAdd the credentials with access to Redfish Telemetry reports on client serverOnce added the account, switch back to Inputs tab and click Create New Input; the followingoptions need to be filled in the form:Name: Specify input nameInterval: Time interval in secs for reports to be pulled from client serverIndex: Where the events are going to be stored in SplunkGlobal Account: The account added in the previous stepHostname: Client host name or IP addressChassis Collection Options: Default options are Overview, Power, Thermal. You can either add tothem or remove them.System Collection Options: Default options are Overview, Processors, Ethernet, Memory, Storage,Storage Subsystem.6. Save the form, and the logs should start flowing into the selected index.Note that the user may need to create an “Index” if one does not currently exist. Data Type is “Events.”9 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

Additional ReferencesTolly iDRAC9 ReportThis Tolly-Dell report from February 2020 validates both the number of data points generated bytelemetry streaming (compared with polling) along with quantifying the network efficiency of telemetrystreaming over traditional polling. 7iDRAC9 WebinarJoin the discussion with Rick Hall (Systems Management Product Planning and Strategy), Doug Iler(iDRAC Product Manager), Michael E. Brown (Distinguished Engineer), and Kevin Tolly (3rd PartyTesting & Validation) as they cover the new iDRAC9 v4.x Telemetry Streaming feature. In this 30-minutewebinar, an overview of Telemetry Streaming, use cases, description of the feature validation testing andengineering perspective are covered. February 2020. See the link 223/idrac9-v4-0-telemetry-streamingTransform Datacenter Analytics with iDRAC9 Telemetry StreamingThis 5-page tech note discusses the Telemetry Streaming feature part of the iDRAC9 Datacenter license.The Telemetry feature provides high-performance streaming of over 180 unique server and peripheralmetrics with our industry-leading agent-free m-developmentdatacenter-telemetry.pdfDell iDRAC9 Telemetry Performance ReportsThis technical paper details the Performance Reports and how to use them for monitoring and analyzingPowerEdge server utilization.8Tolly iDRAC9 Splunk Use Case ReportThis Tolly-Dell report from October 2020 demonstrates a use case with iDRAC9 telemetry integrated withSplunk and provides example visualizations in some areas key to delivering superior user experience. 9Unofficial Telemetry Streaming “How To” Blog by Dell’s Jonas WernerJonas is a Dell Cloud Architect. In this very informative blog post, it details the steps required toimplement both telemetry streaming and visualization using open-source tools (rather than using oninfluxdb-and-grafana/7See the “Telemetry” tab under the Resources/White Papers section at www.dell.com/support/idracSee the “Telemetry” tab under the Resources/White Papers section at www.dell.com/support/idrac9See the “Telemetry” tab under the Resources/White Papers section at www.dell.com/support/idrac810 How To Integrate iDRAC9 Telemetry Data Into The Splunk Platform

To install apps and add-ons from within Splunk Enterprise 1. Log into Splunk Enterprise (Heavy Forwarder server’s web interface). 2. On the Apps menu, click Manage Apps. 3. Click Install app from file. 4. In the Upload app window, click Choose File. 5. Locate the tar.gz file you download