Transcription

Nmon Performance monitor Splunk appfor Unix and Linux systemsDocumentationRelease 1.9.0Guilhem MarchandNov 06, 2019

Contents12Overview:1.1 About Nmon Performance monitor for Splunk1.2 Release notes . . . . . . . . . . . . . . . . . .1.3 Known Issues . . . . . . . . . . . . . . . . . .1.4 Support . . . . . . . . . . . . . . . . . . . . .1.5 Issues and enhancement requests . . . . . . .1.6 Scripts and Binaries . . . . . . . . . . . . . .1.7 licence . . . . . . . . . . . . . . . . . . . . .3363939404142Documentation:2.1 Introduction . . . . . . . . . . . . . . . . . . . . . .2.2 Deployment Matrix . . . . . . . . . . . . . . . . . .2.3 Deployment topologies . . . . . . . . . . . . . . . . .2.4 Download . . . . . . . . . . . . . . . . . . . . . . . .2.5 Running on Windows . . . . . . . . . . . . . . . . .2.6 Deploy to single server instance . . . . . . . . . . . .2.7 Deploy to distributed deployment . . . . . . . . . . .2.8 Deploying Nmon Performance Monitor in SH Clusters2.9 Deploy to Splunk Cloud . . . . . . . . . . . . . . . .2.10 Managing Nmon Central Repositories . . . . . . . . .2.11 Eventgen testing . . . . . . . . . . . . . . . . . . . .2.12 Upgrade . . . . . . . . . . . . . . . . . . . . . . . .2.13 Splunk HEC / nmon-logger deployment . . . . . . . .2.14 rsyslog / nmon-logger deployment . . . . . . . . . . .2.15 syslog-ng / nmon-logger deployment . . . . . . . . .2.16 frameID mapping management . . . . . . . . . . . .2.17 Userguide . . . . . . . . . . . . . . . . . . . . . . . .2.18 Total cost of Ownership . . . . . . . . . . . . . . . .2.19 Large scale deployment considerations . . . . . . . .2.20 Reference Materials . . . . . . . . . . . . . . . . . .4343454652535457656566687073798896100139142146.i

ii

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0Nmon Performance is now associated with Octamis to provide professional solutions for your business, andprofessional support for the Nmon Performance solution.For more information: Octamis professional support for businessContents1

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.02Contents

CHAPTER1Overview:1.1 About Nmon Performance monitor for Splunk Author: Guilhem Marchand First release was published on starting 2014 Purposes:The Nmon Performance application for Splunk implements the excellent and powerful nmon binary known as Nigel’sperformance monitor. Originally developed for IBM AIX performance monitoring and analysis, it is now an Opensource project that made it available to many other systems. It is fully available for any Linux flavor, and thanks to theexcellent work of Guy Deffaux, it also available for Solaris 10/11 systems using the sarmon project.The Nmon Performance monitor application for Splunk will generate performance and inventory data for your servers,and provides a rich number of monitors and tools to manage your AIX / Linux / Solaris systems.Nmon Performance is now associated with Octamis to provide professional solutions for your business, andprofessional support for the Nmon Performance solution.For more information: Octamis professional support for business1.1.1 Splunk versionsIt is recommended to use Splunk 6.5.x or superior to run the latest core application release. (in distributed deployments,only search heads may have this requirement)The last release can be downloaded from Splunk base: y matrix for core application:3

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0 Current major release Version 1.9.x: Splunk 6.5.x or superior are officially supportedSplunk 6.4 will globally perform as expected, but there might be some unwanted behaviors such as css issues as thisSplunk version is not supported anymore by the core application.Stopped versions for older Splunk releases: Last version compatible with Splunk 6.4.x with release 1.7.9 (Splunk shttps://github.com/ Last version compatible with Splunk 6.2.x with release 1.6.15 (Splunk shttps://github.com/ Last version compatible with Splunk 6.1.x, with release 1.4.902 (not Splunk certified): /blob/last release splunk 61xCompatibility matrix for TA-nmon addon:Consult the TA-nmon documentation: http://ta-nmon.readthedocs.io Both add-ons are compatible with any Splunk version 6.x (full instance of Universal Forwarder)The TA-nmon add-on is designed to be deployed on full Splunk instances or Universal Forwarders, it is only compatible with Splunk 6.x.The PA-nmon light add-on is a minimal add-on designed to be installed on indexers (clusters or standalone), thispackage contains the default “nmon” index definition and parsing configuration. It excludes any kind of binaries,inputs or scripts, and does not collect nmon data.1.1.2 Index time operationsThe application operates index time operations, the PA-nmon light add-on must be installed in indexers in order forthe application to operate normally.If there are any Heavy forwarders acting as intermediate forwarders between indexers and Universal Forwarders, theTA-nmon add-on must deployed on the intermediate forwarders to achieve successfully index time extractions.1.1.3 Index creationThe Nmon core application does not create any index at installation time.An index called “nmon” must be created manually by Splunk administrators to use the default TA-nmon indexingparameters. (this can be tuned)However, deploying the PA-nmon light will automatically defines the default “nmon” index. (pre-configured forclusters replication)Note: The application supports any index starting with the “nmon*” name, however the default index for the TA-nmoninputs is set to “nmon” index.In distributed deployments using clusters of indexers, the PA-nmon add-on will automatically creates the “nmon”replicated index.1.1.4 Summarization implementationAccelerated data models:Nmon for Splunk App intensively uses data model acceleration in almost every user interfaces, reports and dashboards.Splunk certification requirements prohibit the default activation of data models acceleration.4Chapter 1. Overview:

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0Since version 1.9.12, none of the data models are accelerated by default, this is your responsibility to decide ifyou wish to do so, bellow are the recommended acceleration parameters: metrics related data models accelerated over a period of 1 year non metrics data models accelerated over the last 30 days (Nmon config, Nmon processing)Splunk Accelerated data models provide a great and efficient user experience.Accelerated reports:The following report(s) use report acceleration feature: Volume of Data indexed Today, accelerated for last 7 days Number of notable events in Data Processing or Data Collect since last 24 Hours, accelerated for last 24 hoursPlease review the Large scale deployment considerations documentation.1.1.5 About Nmon Performance MonitorNmon Performance Monitor for Splunk is provided in Open Source, you are totally free to use it for personal orprofessional use without any limitation, and you are free to modify sources or participate in the development if youwish.Feedback and rating the application will be greatly appreciated. Join the Google group: https://groups.google.com/d/forum/nmon-splunk-app App’s Github page: https://github.com/guilhemmarchand/nmon-for-splunk Videos: kyHQcQ Gallery: https://flic.kr/s/aHskFZcQBn1.1.6 Open source and licensed materials reference css materials from http://www.w3schools.com d3 from Michael Bostock: https://bl.ocks.org various extensions and components from the Splunk 6.x Dashboard Examples application: https://splunkbase.splunk.com/app/1603 dark.css from: -looks-more-beautiful.html Take the tour component from lection hover.css from http://ianlunn.github.io/Hover free of use icons from /www.iconfinder.com Javascript tips (inputs highlighting) from https://splunkbase.splunk.com/app/3171 - hlighting-required-inputs1.1. About Nmon Performance monitor for Splunk5

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.01.2 Release notes1.2.1 Requirements Splunk 6.5.x and later Only, for 6.4.x and prior download release: V1.7.9, for prior to 6.2.x download release:V1.6.15, for 6.1.x and prior download release: V1.4.902 Universal Forwarder v6.x is required for clients Universal Forwarders clients system lacking a Python 2.7.x interpreter requires Perl WITH Time::HiRes moduleavailable1.2.2 What has been fixed by releaseV1.9.20:Version 1.9.20 fix: AIX - field alias for VP Idle PCT results in missing field after Splunk behaviour change regarding fieldaliasing to non existing fields #122V1.9.19:Version 1.9.19 fix: Solaris NMON ANALYSER view issue with drilldown field names causing URL malformed #121V1.9.18:Version 1.9.18 fix: AIX lpar measurement issue in some queries when comparing to cpu all #118 fix: Safecenter - user feedback on headings color #119V1.9.17:Version 1.9.17 fix: JQuery vulnerability issue for the integrated viz addons (bullet chart amd radial meter, CVE-201610707/CVE-2015-9251) #114 fix: KVstore collections management interfaces improvements #115 fix: Nav improvement: merge search and builtin menu into search menu #116 fix: Nmon summary improvement #117V1.9.16:Version 1.9.16 - multiple updates #112 review props.conf sourcetypes definition for Splunk best practices update of horseshoe-meter and bullet-graph to their latest version6Chapter 1. Overview:

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0 removal of calendar heatmap views managing processing and nmon data availability New dashboard: Heatmap daily CPU usage calendar with drilldown New alerting scheme with multi-layer KVstore based: Threshold management, frameID mapping and thresholdstemplating Splunk 7.1 minor compatibility issuesV1.9.15:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html# fix: NMON Data PAGE data model issue: Comparator ‘ ’ has an invalid term #106 fix: PAGE interface for AIX - duplicated ID #107V1.9.14: intermediate release unpublishedV1.9.13:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html# fix: CONFIG DF dead link in home pages (was replaced by STORAGE ui in 1.9.12) fix: props.conf and Nmon config datamodel issue with AIX combo cpu #100 fix: Nmon Summary dashboard - nmon span referenced instead of variable #102 feature request: allow deactivation for auto-refresh feature #103 fix: Summary dashboard stacking issues with Splunk 7 #104V1.9.12:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the release-prior-to-version-1-7-x1.2. Release st/upgrade.html#7

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release ocs.io/en/latest/upgrade.html# fix: Splunk certification requirements update, avoid global default parameter in ui-prefs.conf (config file hasbeen removed) #98 fix: Splunk certification requirements update, default activation of data model acceleration is now prohibited#98 fix: DF STORAGE vs JFSFILE compatibility for STORAGE UI and Dark monitoringV1.9.11:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html feature: DF STORAGE and DF INODES implementation in replacement of JFSFILE (extended file systemutilisation statistics) feature: New interface for STORAGE statistics management feature: metric catalog lookup implementation feature: review and refresh of various interfaces, including comparative and predictive interfaces fix: dynamic tokens in dashboard improvements fix: Nmon Config datamodel OStype extraction #95V1.9.10:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html feature: index and search time configuration for the TA-nmon-hec / nmon-logger-splunk-hec (agent less packageusing the Splunk http input) fix: UI Compare - fix frameID mapping for non CSV source data (nmon-logger) #92 fix: UI Predictive - issue when time range is changed, bad MEM metric label #93 fix: UI Summary / WOF - token auto-selection issue when time range is changed #948Chapter 1. Overview:

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0V1.9.9:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html fix: Large scale issue - Optimize Nmon inventory generation runtime #85 fix: Nmon inventory - Uptime data analysis issue #86 fix: Nmon Dark dashboard - missing reset auto-refresh #87 fix: TOP datamodel issue - error in distributed search for ALL OS node (nmon summary. . . ) #88 fix: Drilldown correction for the number of last 7 days hosts in home pages #89 evolution: Large scale consideration - restricted default limits for datamodel acceleration (1y for metrics) #90 fix: Use nmon inventory to retrieve configuration data instead of using datamodel #91V1.9.8: intermediate unpublished releaseV1.9.7:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html fix: Large scale issue - Optimize search refresh values for large deployments #84 fix: Nmon Config data model issues in some clustered environments #83 fix: baseline future charting not working due to mismatch between host and hostname #82 fix: Large scale issue - Optimize the run time of the Hosts with data within last 7 days #81 fix: Large scale issue - restrict the nmon processing data model to the last 30 days by default #80 fix: report issue - TA-nmon package deployment reporting can includes non deployment events #79 fix: Large scale issue - Optimize run time of the Volume of Data indexed today report #78 fix: Large scale issue - Nmon inventory generation report may fail due to report lengh #771.2. Release notes9

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0V1.9.6:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html fix: Alerting for CPU is broken since 1.9.5 due to unexpected missing sort time #73 fix: nmon data from syslog, missing indexed time creation and OStype and type fields #74 fix: nmon data from syslog - uptime extraction failure #75 fix: Alerting - Show the real number of alerts instead of triggered alerts #76V1.9.5:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html fix: missing oshost tag for ITSI fix: Nmon Summary dashboard not retrieving expected results in CPU usage summary with Splunk 6.6.1V1.9.4:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html fix alerting macros issues: transaction incorrect usage filter out events in excess of allowed limits #70 fix eventtype related messages for nmon:performance:cpu/mem due to WLM stats #71 fix Safe Center: reduce the number of searches and add refresh selector dropdown fix: CIM compliance improvements and corrections feature: introduce a smart auto refresh feature to prevent from having auto refresh enabled when out of currenttime range10Chapter 1. Overview:

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0 feature: red highlighting of forms waiting for inputs in views feature: Take the tour updateV1.9.3:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html fix certification issues: TA-nmon and PA-nmon light are not anymore embedded in the core application andmust be downloaded externally Lower data model acceleration load with per data model schedule configuration #68 Net stats not associated with time range selector in Nmon Summary IOPS and NET stats rendering improvements in Analyser viewsV1.9.2:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the st/upgrade.html#For the TA-nmon complete release notes: es.html Splunk 6.6 tstats issue over non existing field generates nan value instead of null values #67 Introducing the Dark monitoring dashboard, interfaces review Linux Nmon Analyser view issue in DG chart for IOPS Nmon external load average extraction failure for some OS Be time relative to show indexing evolution in home page UPTIME external collection integration TA-nmon local/nmon.conf from the SHC deployer is not compatible #23, AIX issues with old topas-nmon,external collection stops on AIX 6.1/7.1, . . .V1.9.1:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements below)This is a major release of the Nmon application and the TA-nmon:1.2. Release notes11

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release upgrade.html#For the TA-nmon complete release notes: es.html TA-nmon new branch: fantastic foot print reduction with the fifo implementation, extend data with nmon external, various bug fixes (read TA-nmon release notes) PA-nmon and TA-nmon selfmode are now deprecated (unified by the new TA-nmon features) Optimization and rationalisation (globally use the host Splunk Metadata instead of historical hostname field) Nmon cores issues (multisearch and tstats incompatible in distributed for the Disk KV generation)V1.8.6:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements version-1-7-x Invalid error number of events count in TCO dashboard when running multiple indexes Update of Nmon baseline generation for Disk I/O, and relevant macro update (use DG stats when available) app certification failure correction (custom viz issues in savedsearches.conf) Addons update to version 1.2.54 Removal of the static “nmon” index abstraction layer: the app supports natively any index(es) starting with the“nmon” pattern Native support for multiple indexes Introducing the new frameID management using KVstore, and the frameID mapping management interface Improved multi-line events management for rsyslog with nmon-logger agent TA-nmon issue: implementation of linux disks groups caused issues with old nmon releases Improvement of multi line event management for rsyslog deployments populating forms issues in DG interfaceV1.8.5: Intermediate release unpublishedV1.8.4: Intermediate release unpublished12Chapter 1. Overview:

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0V1.8.3:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements version-1-7-x Octamis release, Nmon Performance suite is now a company supported software ITSI better compatibility (most ITSI OS module builtin will work, entities dynamic inventory. . . ) Nmon WOF dashboard correction (single forms mot linked to shared time picker) Adding direct link to Data model manager, updating to datasets link, correction to removed interfaces (UI RT) Implementation of Linux disks extended statistics (DG* sections), new data model, interfaces, Howto Nmon Analyser update, Nmon Summary and WOF will now automatically choose disks extended statisticswhen available Implementation of monitors assets description (monitor description enrichment) Allow nmon.conf on a per server basis (/etc/nmon.conf can be set to customize parameters on a per server basis) Generic Nmon binaries not recognized for Linux 32 bits systems TA-nmon and PA-nmon update to v1.2.51V1.8.2:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements version-1-7-x Drilldown error with Splunk 6.5.1 #60 - Various drilldown errors since 6.5 when a pipeline is split in more thanone line (carriage return) Errors in Nmon analyser views (Since 6.5 renming an non existing field removes the existing field, this wascausing various Disks charts not to be displayed) TA-nmon update - Allow host name override #58 (feature request) TA-nmon and PA-nmon update to v1.2.50V1.8.1:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements version-1-7-x Technical addons issue with Oracle Solaris 10 using Python interpreter /11) TA-nmon and PA-nmon update to v1.2.481.2. Release notes13

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release 1.9.0V1.8.0:CAUTION: For Splunk 6.5 and later (for prior versions of Splunk, see requirements version-1-7-x Implementation of Splunk 6.5 auto refresh features Minor improvements and evolutions for best Splunk 6.5 ease-prior-to-version-1-7-x Adding the PA-nmon light add-on for indexers that need parsing configuration only (for people that do not wantor must not monitor performance of indexers such as Splunk cloud indexers instances) Documentation ior-to-version-1-7-x Add-ons update to 1.2.47 (Linux unlimited capture improvement #9, Nmon binary issue with SLES 11.3 #10) Adding CONFIG df (filesystems stats) reports & -prior-to-version-1-7-x Drilldown to inventory issues and improvements (Issue #55) Performance improvement of the TCO per server search (use datamodel for dcount) Add-ons Perl parser (nmon2csv.pl) is lacking OStype field in raw data for TOP/UARG, causing data to beunavailable Removal of nmon inventory OStype mapping had removed OStype mapping for historical data Add-ons update (PA-nmon/TA-nmon/TA-nmon selfmode) to ior-to-version-1-7-x Fix TCO scheduling searches analysis when running in Search Head Cluster Updating alerting menu Broken links to removed django views (Issue #54)14Chapter 1. Overview:

Nmon Performance monitor Splunk app for Unix and Linux systems Documentation, Release or-to-version-1-7-x Prevent unwanted server filtering in nmon inventory interfaces due to null fields in nmon inventory KV Correct labels for LPAR stats (for Powerlinux), correct series name to match Physical raw field names Integrating the TA-nmon selfmode as an alternative to the standard TA in case of unsolved unarchive processorfailure Rewritten Internal dashboard as the Total Cost of Ownership dashboard Rewritten Add-ons reporting to provide the global picture of add-ons deployment The Nmon app customization tool now offers the option to build a core app that supports Linux only Nmon core app Fix Git Issues: #48 to #53 TA-nmon and PA-nmon rior-to-version-1-7-x Preven

Current major release Version 1.9.x: Splunk 6.5.x or superior are officially supported Splunk 6.4 will globally perform as expected, but there might be some unwanted behaviors such as css issues as this Splunk version is not supported anymore by the core application. Stopped versions for older Splunk releases: