Transcription

Document:Authors:Company:Date:File:EditionFaronics Anti-Virus vs. Seven Competitors (August 2014)M. Baquiran, D. WrenPassMark Software12 August 2014Faronics Antivirus vs Competitors.docx1

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareTABLE OF CONTENTS. 2REVISION HISTORY. 3REFERENCES. 3EXECUTIVE SUMMARY . 4OVERALL SCORE . 5PRODUCTS AND VERSIONS . 6PERFORMANCE METRICS SUMMARY . 7TEST RESULTS . 9BENCHMARK 1 – INSTALLATION TIME . 9BENCHMARK 2 – INSTALLATION SIZE . 9BENCHMARK 3 – BOOT TIME . 10BENCHMARK 4 – CPU USAGE DURING SCAN . 10BENCHMARK 5 – MEMORY USAGE DURING INITIAL SCAN . 11BENCHMARK 6 – SCHEDULED SCAN TIME . 11BENCHMARK 7 – FILE COPY, MOVE, AND DELETE . 12BENCHMARK 8 – FILE COMPRESSION AND DECOMPRESSION . 12BENCHMARK 9 – FILE WRITE, OPEN, AND CLOSE . 13BENCHMARK 10 – NETWORK THROUGHPUT . 13DISCLAIMER AND DISCLOSURE . 14CONTACT DETAILS . 14APPENDIX 1 – TEST ENVIRONMENT . 15APPENDIX 2 – METHODOLOGY DESCRIPTION . 16Performance BenchmarkPage 2 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsRevEdition 1Revision History2DateInitial version of this report. Competitor results taken from a previously publishedreport (See References below).Ref #1PassMark SoftwareDocumentWhat Really Slows Windows Down (URL)Webroot SecureAnywhere Business Endpoint Protection vs.Seven Competitors (February 2014)Performance Benchmark12 August 2014AuthorDateO. Warner,2001-2014The PC SpyM. Baquiran, D. Wren19 February 2014Page 3 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwarePassMark Software conducted objective performance testing on eight (8) security software products, onWindows 7 Ultimate Edition (64-bit) between January and July 2014. This report presents our results and findingsas a result of performance benchmark testing conducted for these endpoint security products.The aim of this report is to compare the performance impact of Faronics Anti-Virus product with seven (7)competitor products, of which the results have been taken from a previously published performance benchmarkreport (see References).Testing was performed on all products using ten (10) performance metrics. These performance metrics are asfollows: Installation Time; Installation Size; Boot Time; CPU Usage during Scan; Memory Usage during Initial Scan; Scheduled Scan Time; File Copy, Move, and Delete; File Compression and Decompression; File Write, Open, and Close; and Network Throughput.Performance BenchmarkPage 4 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwarePassMark Software assigned every product a score depending on its ranking in each metric compared to otherproducts in the same category. In the following table, the highest possible score attainable is 80; in a hypotheticalsituation where a product has attained first place in all ten (10) metrics. Endpoint products have been ranked bytheir overall scores:Product NameFaronics Anti-VirusESET NOD32 Antivirus BusinessPerformance BenchmarkOverall Score6255Microsoft Security Center Endpoint Protection51Symantec Endpoint Protection Small Business Edition48Sophos EndUser Protection – Business46Trend Micro Worry Free Business Security Standard33McAfee Complete Endpoint Protection – Business29Kaspersky Endpoint Security29Page 5 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareFor each security product, we have tested the most current and available version.ManufacturerFaronics CorporationTrend Micro Inc.Kaspersky LabSophosMcAfee, Inc.Symantec CorpESET, spol. s r.o.Microsoft CorporationPerformance BenchmarkProduct VersionDateTested3.42.2102.251July 2014Trend Micro Worry Free Business SecurityStandard7.0.1638Jan 2014Kaspersky Endpoint Security10.2.1.23Jan 2014SophosEndpointSecurity andControl 10.3Jan 2014VirusScan,AntiSpywareEnterprise 8.8Jan 2014Product NameFaronics Anti-VirusSophos EndUser Protection – BusinessMcAfee Complete Endpoint Protection - BusinessSymantec Endpoint Protection Small BusinessEdition 2013 (Symantec .cloud)Cloud Agent x642.03.23.2539EndpointProtection NIS20.4.0.40Jan 2014ESET NOD32 Antivirus Business4.2.76.0Jan 2014Microsoft System Center Endpoint Protection4.3.220.0Jan 2014Page 6 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareWe have selected a set of objective metrics which provide a comprehensive and realistic indication of the areas inwhich endpoint protection products may impact system performance for end users. Our metrics test the impactof the software on common tasks that end-users would perform on a daily basis.All of PassMark Software’s test methods can be replicated by third parties using the same environment to obtainsimilar benchmark results. Detailed descriptions of the methodologies used in our tests are available as “Appendix2 – Methodology Description” of this report.The speed and ease of the installation process will strongly influence the user’s first impression of the securitysoftware. This test measures the installation time required by the security software to be fully functional and readyfor use by the end-user. Lower installation times represent security products which are quicker for a user to install.In offering new features and functionality to users, security software products tend to increase in size with eachnew release. Although new technologies push the size limits of hard drives each year, the growing disk spacerequirements of common applications and the increasing popularity of large media files (such as movies, photosand music) ensure that a product's installation size will remain of interest to home users.This metric aims to measure a product’s total installation size. This metric is defined as the total disk spaceconsumed by all new files added during a product's installation.This metric measures the amount of time taken for the machine to boot into the operating system. Securitysoftware is generally launched at Windows startup, adding an additional amount of time and delaying the startupof the operating system. Shorter boot times indicate that the application has had less impact on the normaloperation of the machine.The amount of load on the CPU while security software conducts a malware scan may prevent the reasonable useof the endpoint machine until the scan has completed. This metric measured the percentage of CPU used bysecurity software when performing a scan.This metric measures the amount of memory (RAM) used by the product during an initial security scan. The totalmemory usage was calculated by identifying all security software processes and the amount of memory used byeach process during the scan.Performance BenchmarkPage 7 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareMost anti-virus solutions are scheduled by default to scan the system regularly for viruses and malware. This metricmeasured the amount of time required to run a scheduled scan on the system. The scan is set to run at a specifiedtime via the client user interface.This metric measures the amount of time taken to copy, move and delete a sample set of files. The sample file setcontains several types of file formats that a Windows user would encounter in daily use. These formats includedocuments (for example, Microsoft Office documents, Adobe PDF, Zip files, etc), media formats (for example,images, movies and music) and system files (for example, executables, libraries, etc).This metric measures the amount of time taken to compress and decompress different types of files. Files formatsused in this test included documents, movies and images.This benchmark was derived from Oli Warner’s File I/O test at http://www.thepcspy.com (please see Reference#1: What Really Slows Windows Down). This metric measures the amount of time taken to write a file, then openand close that file.The metric measures the amount of time taken to download a variety of files from a local server using theHyperText Transfer Protocol (HTTP), which is the main protocol used on the web for browsing, linking and datatransfer. Files used in this test include file formats that users would typically download from the web, such asimages, archives, music files and movie files.Performance BenchmarkPage 8 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareIn the following charts, we have highlighted the results we obtained for Faronics Anti-Virus in red. The competitoraverage has also been highlighted in blue for ease of comparison.The following chart compares the minimum installation time it takes for endpoint security products to be fullyfunctional and ready for use by the end user. Products with lower installation times are considered betterperforming products in this category.Microsoft System Center EP39.2ESET NOD32 AV Business41.3Faronics Anti-Virus114.0Sophos EUP - Business133.8Kaspersky ES 10185.3Trend Micro WFBS Standard199.0Symantec EP SBE 2013200.3Average244.5McAfee CEP - Business1043.30s200 s400 s600 s800 s1,000 s1,200 sThe following chart compares the total size of files added during the installation of endpoint security products.Products with lower installation sizes are considered better performing products in this category.ESET NOD32 AV Business238Microsoft System Center EP244Faronics Anti-Virus297Sophos EUP - Business476Average618Trend Micro WFBS Standard620McAfee CEP - Business770Symantec EP SBE 2013843Kaspersky ES 100 MBPerformance Benchmark1453200 MB400 MB600 MB800 MB 1,000 MB 1,200 MB 1,400 MB 1,600 MBPage 9 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time taken for the system to boot (from a sample of five boots) for eachendpoint security product tested. Products with lower boot times are considered better performing products inthis category.Microsoft System Center EP14.3Faronics Anti-Virus14.5ESET NOD32 AV Business16.1Sophos EUP - Business17.8Trend Micro WFBS Standard17.9Average18.4McAfee CEP - Business18.9Symantec EP SBE 201320.5Kaspersky ES 1027.10s5s10 s15 s20 s25 s30 sThe following chart compares the average CPU usage during a scan of a set of media files, system files andMicrosoft Office documents that totaled 5.42 GB. Products with lower CPU usage are considered betterperforming products in this category.Trend Micro WFBS Standard9.8%ESET NOD32 AV Business20.1%Sophos EUP - Business20.3%Symantec EP SBE 201320.5%Faronics Anti-Virus24.2%Average25.4%McAfee CEP - Business26.9%Microsoft System Center EP39.8%Kaspersky ES 100.0%Performance .0%45.0%Page 10 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average amount of RAM in use by an endpoint security product during an initialscan on the main drive. This average is taken from a sample of ten memory snapshots taken at five second intervalsduring a scan of sample files which have not been previously scanned by the software. Products that use lessmemory during a scan are considered better performing products in this category.ESET NOD32 AV Business102.9Faronics Anti-Virus104.9Microsoft System Center EP130.2Sophos EUP - Business175.8Kaspersky ES 10185.2Average207.5Trend Micro WFBS Standard246.3Symantec EP SBE 2013261.4McAfee CEP - Business453.50 MB50 MB 100 MB 150 MB 200 MB 250 MB 300 MB 350 MB 400 MB 450 MB 500 MBThe following chart compares the average time taken to run a scheduled scan on the system for each securityproduct tested.*Symantec EP SBE 201334Faronics Anti-Virus59Microsoft System Center EP104Sophos EUP - Business357Average584Kaspersky ES 10986McAfee CEP - Business1218ESET NOD32 AV Business13290s200 s400 s600 s800 s1,000 s1,200 s1,400 s*Trend Micro’s product was omitted from the chart and given the lowest score. The scheduled scan time could not be run to completion due to what appears to be a bug.Performance BenchmarkPage 11 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time taken to copy, move and delete several sets of sample files foreach endpoint security product tested. Products with lower times are considered better performing products inthis category.Symantec EP SBE 201310.9ESET NOD32 AV Business13.7Trend Micro WFBS Standard13.8McAfee CEP - Business14.2Average16.3Sophos EUP - Business18.3Faronics Anti-Virus18.5Microsoft System Center EP19.0Kaspersky ES 1021.60s5s10 s15 s20 s25 sThe following chart compares the average time it takes for sample files to be compressed and decompressed foreach endpoint security product tested. Products with lower times are considered better performing products inthis category.Symantec EP SBE 201344.6Faronics Anti-Virus46.4McAfee CEP - Business48.0Kaspersky ES 1048.3Sophos EUP - Business49.0Average49.1ESET NOD32 AV Business49.1Microsoft System Center EP50.6Trend Micro WFBS Standard56.60sPerformance Benchmark10 s20 s30 s40 s50 s60 sPage 12 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time it takes for a file to be written to the hard drive then opened andclosed 180,000 times, for each endpoint security product tested. Products with lower times are considered betterperforming products in this category.Faronics Anti-Virus15.0Kaspersky ES 1018.9Symantec EP SBE 201319.3McAfee CEP - Business27.8Sophos EUP - Business72.6ESET NOD32 AV Business114.2Average165.8Microsoft System Center EP339.1Trend Micro WFBS Standard719.50s100 s200 s300 s400 s500 s600 s700 s800 sThe following chart compares the average time to download a sample set of common file types for each endpointsecurity product tested. Products with lower times are considered better performing products in this category.Microsoft System Center EP6.7Faronics Anti-Virus6.9McAfee CEP - Business7.3Symantec EP SBE 20137.6ESET NOD32 AV Business7.6Average7.7Sophos EUP - Business7.9Trend Micro WFBS Standard8.2Kaspersky ES 109.30sPerformance Benchmark1s2s3s4s5s6s7s8s9s10 sPage 13 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareThis report only covers versions of products that were available at the time of testing. The tested versions are asnoted in the “Products and Versions” section of this report. The products we have tested are not an exhaustivelist of all products available in these very competitive product categories.While every effort has been made to ensure that the information presented in this report is accurate, PassMarkSoftware Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not be liablein any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting from theavailability of, use of, access of, or inability to use this information.Faronics Corporation funded the writing of this report. The list of products tested and the metrics included in thereport were selected by Faronics.All trademarks are the property of their respective owners.PassMark Software Pty LtdSuite 202, Level 235 Buckingham St.Surry Hills, 2010Sydney, AustraliaPhone 61 (2) 9690 0444Fax 61 (2) 9690 0445Webwww.passmark.comPerformance BenchmarkPage 14 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareFor our testing, PassMark Software used a test environment running Windows 7 Ultimate (64-bit) with thefollowing hardware specifications:Model:CPU:Video Card:Motherboard:RAM:HDD:Network:HP Pavilion P6-2300AIntel Core i5 3330 @ 2.66GHz1GB nVIDIA GeForce GT 620MFoxconn 2ABF 3.106GB DDR3 RAMHitachi HDS721010CLA630 931.51GBGigabit (1GB/s)The Web and File server was not benchmarked directly, but served the web pages and files to the endpointmachine during performance testing.CPU:Video Card:Motherboard:RAM:SSD:Network:Performance BenchmarkIntel Xeon E3-1220v2 CPUKingston 8GB (2 x 4GB ECC RAM)Intel S1200BTL ServerKingston 8GB (2 x 4GB) ECC RAM, 1333MhzOCZ 128GB 2.5” Solid State DiskGigabit (1GB/s)Page 15 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareAs with testing on Windows Vista, Norton Ghost was used to create a “clean” baseline image prior to testing. Ouraim is to create a baseline image with the smallest possible footprint and reduce the possibility of variation causedby external operating system factors.The baseline image was restored prior to testing of each different product. This process ensures that we installand test all products on the same, “clean” machine.The steps taken to create the base Windows 7 image are as follows:1.Installation and activation of Windows 7 Ultimate Edition.2.Disabled Automatic Updates.3.Changed User Account Control settings to “Never Notify”.4.Disable Windows Defender automatic scans to avoid unexpected background activity.5.Disable the Windows firewall to avoid interference with security software.6.Installed Norton Ghost for imaging purposes.7.Disabled Superfetch to ensure consistent results.8.Installed HTTP Watch for Browse Time testing.9.Installed Windows Performance Toolkit x64 for Boot Time testing.10.Installed Active Perl for interpretation of some test scripts.11.Install OSForensics for testing (Installation Size test) purposes.12.Disabled updates, accelerators and compatibility view updates in Internet Explorer 8.13.Update to Windows Service Pack 114.Created a baseline image using Norton Ghost.This test measures the minimum Installation Time a product requires to be fully functional and ready for use bythe end user. Installation time can usually be divided in three major phases: The Extraction and Setup phase consists of file extraction, the EULA prompt, product activation and userconfigurable options for installation. The File Copy phase occurs when the product is being installed; usually this phase is indicated by a progressbar. The Post-Installation phase is any part of the installation that occurs after the File Copy phase. This phasevaries widely between products; the time recorded in this phase may include a required reboot to finalize theinstallation or include the time the program takes to become idle in the system tray.To reduce the impact of disk drive variables, each product was copied to the Desktop before initializing installation.Each step of the installation process was manually timed with a stopwatch and recorded in as much detail aspossible. Where input was required by the end user, the stopwatch was paused and the input noted in the rawresults in parenthesis after the phase description.Performance BenchmarkPage 16 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareWhere possible, all requests by products to pre-scan or post-install scan were declined or skipped. Where it wasnot possible to skip a scan, the time to scan was included as part of the installation time. Where an optionalcomponent of the installation formed a reasonable part of the functionality of the software, it was also installed(for example, website link checking software as part of a Security Product).Installation time includes the time taken by the product installer to download components required in theinstallation. This may include mandatory updates or the delivery of the application itself from a downloadmanager. We have noted in our results where a product has downloaded components for product installation.We have excluded product activation times due to network variability in contacting vendor servers or time takenin account creation. For all products tested, the installation was performed directly on the endpoint, either usinga standalone installation package or via the management server web console.A product's Installation Size was previously defined as the difference between the initial snapshot of the Disk Space(C: drive) before installation and the subsequent snapshot taken after the product is installed on the system.Although this is a widely used methodology, we noticed that the results it yielded were not always reproduciblein Windows Vista due to random OS operations that may take place between the two snapshots. We improvedthe Installation Size methodology by removing as many Operating System and disk space variables as possible.Using PassMark’s OSForensics 2.2 we created initial and post-installation disk signatures for each product. Thesedisk signatures recorded the amount of files and directories, and complete details of all files on that drive (includingfile name, file size, checksum, etc) at the time the signature was taken.The initial disk signature was taken immediately prior to installation of the product. A subsequent disk signaturewas taken immediately following a system reboot after product installation. Using OSForensics, we compared thetwo signatures and calculated the total disk space consumed by files that were new, modified, and deleted duringproduct installation. Our result for this metric reflects the total size of all newly added files during installation.The scope of this metric includes only an ‘out of the box’ installation size for each product. Our result does notcover the size of files downloaded by the product after its installation (such as engine or signature updates), orany files created by system restore points, pre-fetch files and other temporary files.PassMark Software uses tools available from the Windows Performance Toolkit version 4.6 (as part of theMicrosoft Windows 7 SDK obtainable from the Microsoft Website) with a view to obtaining more precise andconsistent boot time results on the Windows 7 platform.The boot process is first optimized with xbootmgr.exe using the command “xbootmgr.exe -trace boot –prepSystem” which prepares the system for the test over six optimization boots. The boot traces obtained fromthe optimization process are discarded.After boot optimization, the benchmark is conducted using the command "xbootmgr.exe -trace boot -numruns 5”.This command boots the system five times in succession, taking detailed boot traces for each boot cycle.Performance BenchmarkPage 17 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareFinally, a post-processing tool was used to parse the boot traces and obtain the BootTimeViaPostBoot value. Thisvalue reflects the amount of time it takes the system to complete all (and only) boot time processes. Our finalresult is an average of five boot traces.CPUAvg is a command-line tool which samples the amount of CPU load approximately two times per second. Fromthis, CPUAvg calculates and displays the average CPU load for the interval of time for which it has been active.For this metric, CPUAvg was used to measure the CPU load on average (as a percentage) by the system while theOn-Demand Scan Time test was being conducted. The final result was calculated as an average five sets of thirtyCPU load samples.The MemLog utility was used to record memory usage on the system while a malware scan is in progress. Pleaserefer to the metric “Memory usage – System Idle” above for a description of the MemLog Utility and anexplanation of the method by which memory usage is calculated.As some products cache scan locations, we take reasonable precautions to ensure that the security software doesnot scan the C:\ drive at any point before conducting this test. A manual scan on the C:\ drive is initiated at thesame time as the MemLog utility, enabling MemLog to record memory usage for 120 seconds at 12 secondintervals.This scan is configured as a full system scheduled scan from user interface. The default scheduled scan settings arekept (except for the start time) and the scan is scheduled to run at the next convenient time. To record the scantime, we have used product’s built-in scan timer or reporting system. Where this was not possible, scan timeswere taken manually with a stopwatch.The scan is run three times with a reboot between each run to remove potential caching effects. In the past, manyproducts have shown a substantial difference between the initial scan time (first scan) and subsequent scan times(scans 2 to 5). We believe this behavior is due to products themselves caching recently scanned files. As a resultof this mechanism, we have averaged the four subsequent scan times to obtain an average subsequent scan time.Our final result for this test is an average of the subsequent scan average and the initial scan time. Where thisoption is not available, the product is omitted from the metric, and given the lowest score for this metric.We used a single script in testing Benchmarks 8-10. The script consecutively executes tests for Benchmarks 10-13.The script times each phase in these benchmarks using CommandTimer.exe and appends results to a log file.This test measures the amount of time required for the system to copy, move and delete samples of files in variousfile formats. This sample was made up of 812 files over 760,867,636 bytes and can be categorized as documents[26% of total], media files [54% of total] and PE files (i.e. System Files) [20% of total].Performance BenchmarkPage 18 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark SoftwareThe breakdown of the main file types, file numbers and total sizes of the files in the sample set is shown in thefollowing table:Performance BenchmarkFile formatNumberSize ,952,914DLL10429,261,568AX118,432CPL22,109,440Page 19 of 2111 September 2014

Faronics Anti-Virus vs. Seven Endpoint Security ProductsPassMark 35,580TSK11,152UCE122,984Total812760,867,636This test was conducted five times to obtain the average time to copy, move and delete the sample files, with thetest machine rebooted between each sample to remove potential caching effects.This test measured the amount of time required to compress and decompress a sample set of files. For this test,we used a subset of the media and documents files used in the File Copy, Move, and Delete benchmark.CommandTimer.exe recorded the amount of time required for 7zip.exe to compress the files into a *.zip andsubsequently decompress the created *.zip file.This subset comprised 1,218 files over 783 MB. The breakdown of the file types, file numbers and total sizes of thefiles in the sample set is shown in the following table:Performance BenchmarkFile TypeFile NumberTotal Size.xls139.23 MB.xlsx93.51 MB.ppt97.37 MB.pptx1117.4 MB.doc1735.9 MB.docx1924.5 MB.gif1771.10 MB.jpg73766.2 MB.png15948.9 MB.mov754.7 MB.rm15.39 MB.avi46459 MB.wma1148.6 MB.avi46459 MBPage 2

Enterprise 8.8 Jan 2014 Symantec Corp Symantec Endpoint Protection Small Business Edition 2013 (Symantec .cloud) Cloud Agent x64 2.03.23.2539 Endpoint Protection NIS-20.4.0.40 Jan 2014 ESET, spol. s r.o. ESET NOD32 Antivirus Business 4.2.76.0 Jan 2014 Microsoft Corporation Micro