Best PracticesDell EMC SC Series: Microsoft ExchangeServer Best PracticesAbstractThis document provides best-practices for using Microsoft Exchange Server2016 or 2019 with Dell EMC SC Series storage.June 2019CML1037

RevisionsRevisionsDateDescriptionOctober 2012Initial releaseApril 2013UpdateSeptember 2015Updated with new SC Series products and best practices updatesAugust 2016Updated for Exchange Server 2016May 2019Updated for Exchange Server 2019; template updateAcknowledgementsEngineering: Damon ZaylskieThe information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in thispublication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.Use, copying, and distribution of any software described in this publication requires an applicable software license.Copyright 2012–2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or itssubsidiaries. Other trademarks may be trademarks of their respective owners. [5/30/2019] [Best Practices] [CML1037]2Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

AcknowledgementsTable of contentsRevisions.2Acknowledgements .2Executive summary.51Predeployment .61.1Understanding storage virtualization .61.2Understanding Exchange I/O .61.3Exchange architecture .71.3.1 Exchange 2016 .71.3.2 Exchange 2019 .82Configuration .92.1Disk layout .92.2SC Series page size considerations .102.3Storage profiles and Data Progression .112.3.1 Microsoft recommends ReFS for Exchange 2016 database volumes (JBOD) .112.3.2 Default allocation unit size .112.3.3 Partition type .112.4Mailbox server sizing considerations .112.4.1 Archive mailboxes.132.5Online database maintenance .132.5.1 Online defragmentation .132.5.2 Online database scanning (checksumming) .132.5.3 Database fragmentation .143Performance and monitoring .153.1Monitoring .153.1.1 Monitoring the database defragmentation process .153.1.2 Monitoring with System Center Operations Manager.1645Troubleshooting .174.1Dell Storage Manager .174.2Microsoft Exchange 2013/2016 Diagnostic Service logs .174.3Microsoft Exchange Troubleshooting Assistant (2007/2010) .184.4Microsoft Exchange Best Practices Analyzer (2007/2010) .18Backup and recovery .195.1Backups and snapshots .195.1.1 Backup schedules.193Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Acknowledgements5.1.2 Snapshots .195.1.3 Database verification .196Recovery .206.1Recovery databases .206.1.1 Single item recovery .207Disaster recovery.217.1ATechnical support and resources .22A.14Database copies and database availability groups .21Related documentation .22Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Executive summaryExecutive summaryThis document provides best-practices for using Microsoft Exchange Server 2016 or 2019 with Dell EMC SC Series storage. These guidelines apply to previous versions of Exchange still under support.These guidelines should be evaluated thoroughly since every environment configuration is different. Theyshould not be construed as final recommendations in the configuration of SC Series or Microsoft Exchangeenvironments.5Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Predeployment1Predeployment1.1Understanding storage virtualizationTraditional SAN solutions require a LUN to be created by carving out a specific amount of disk space from aspecified set of spindles. This process is time-consuming and hard to manage because it involves keepingtrack of which servers are mapped to which storage by which worldwide name (WWN). This often requiresadministrators to manage the storage configuration using a spreadsheet or other means outside of thestorage management interface.Dell EMC SC Series arrays virtualize storage at the disk level and use all available spindles as a single virtualdisk pool. They can create volumes in which all drives are used together, causing performance to improvebecause the blocks of data are written in parallel to all managed drives at once.Traditional disk mapping compared to the SC Series approach1.2Understanding Exchange I/OThe SAN configuration is an important part of any application configuration, and this is especially true withExchange Server. Understanding how Exchange Server works with storage helps administrators make surethat systems run in their most capable state. To ensure that Exchange Server will run in its optimalenvironment, performing some simple tests can determine whether a server and disk subsystem can providethe necessary performance.Several tools exist to put a load against and test the performance of Exchange Server and disk storage,including Exchange Load Generator (LoadGen) and Jetstress. Each of these tools has the capability tosimulate Exchange I/O patterns as well as the client experience, which can provide the estimatedperformance numbers to expect from the disk subsystem. LoadGen and Jetstress are available fromMicrosoft as free downloads and are discussed further in the section, Mailbox server sizing considerations.Another useful tool is the Microsoft Windows Performance Monitor, which can help define a baseline andshow how the application may perform in the current environment. This tool is discussed further in thesection, Performance and monitoring.6Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Predeployment1.3Exchange architecture1.3.1Exchange 2016With CPU hardware generally less expensive than in the past, the constraint of expensive server hardwarehas been alleviated. Exchange 2016 takes advantage of this with a primary design goal of simplicity in scaleand hardware utilization. The number of server roles has been reduced to two: Mailbox and Edge Transportserver roles.The Exchange 2016 Mailbox server role includes all server components from Exchange 2013 Mailbox andClient Access roles: Client Access services: provide authentication, limited re-direction, and proxy services offering theusual client access protocols: HTTP, POP, IMAP, and SMTP.Mailbox services: including the back-end client access protocols, Transport service, Mailboxdatabases, as well as Unified Messaging. The Mailbox server manages all active mailboxes on thatserver.Other enhanced features that are notable for storage considerations include the following:In-place archiving, retention, and eDiscovery: Public folder support for In-Place eDiscovery and In-Place HoldCompliance Search: available only in Exchange Management Shell (EMS)Improved performance and scalability: Search architecture redesigned as asynchronousImproved search scalability from 5K mailboxes to 10K mailboxes, or unlimited in EMSTo provide Exchange Native Data Protection, Exchange 2016 continues to use database availability groups(DAGs) and mailbox database copies, along with features such as single item recovery, retention policies,lagged database copies, and others. The high availability platform, the Exchange Information Store, and theExtensible Storage Engine (ESE), have all been enhanced to provide greater availability, easier management,and reduced costs.With respect to storage, these enhancements include the following: 7Reduced IOPS compared to Exchange 2013: A reduction in IOPS/mailbox size enables largerdisks to be better utilized, providing capacity and IOPS as efficiently as possible.Multiple databases per volume: This enables multiple databases (mixtures of active and passivecopies) to be hosted on the same volume and is another enhancement that allows larger disks to beused.Automatic Reseed for DAS disk failures: This provides a quick restore to database redundancyafter a DAS disk failure. If a physical disk fails, the database copy stored on that disk is copied fromthe active database copy to a spare physical DAS disk on the same server. If multiple databasecopies were stored on the failed disk, they can all be automatically reseeded on a spare disk. Thisenables faster reseeds because the active databases are likely to be on multiple servers and the datais copied in parallel.Automatic recovery from storage failures: This allows the system to recover from failures thataffect resiliency or redundancy. Exchange 2013 includes recovery behaviors for long I/O times,Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Predeployment 1.3.2excessive memory consumption by the Microsoft Exchange Replication service(MSExchangeRepl.exe), and also for severe cases in which the system is in such a bad state thatthreads cannot be scheduled.DAG lagged copy enhancements: Lagged copies can care for themselves to a certain extent usingautomatic log play down. In addition, lagged copies can leverage Safety.Net (previously TransportDumpster in Exchange 2010), making recovery or activation much easier.Exchange 2019Exchange 2019 builds upon the performance enhancements of Exchange 2016 with a number of additionalenhancements. The changes for Exchange 2019 are mainly focused around security, but there are somestorage changes designed to improve performance and scale. These changes can reduce I/O usage, lowerlatency, and improve scale — all at the same time. 8Tiered storage: The biggest change is around tiered storage. Exchange now can dual-write to SSDand spinning media. This allows Exchange to access more frequently used data on faster, lowerlatency media. This is due to the new Metacache Database implementation. The performanceimprovements increase user-per-server density and lower I/O latency.Search indexes: A new search index architecture also improves the quality and performance ofsearch indexes. To increase performance for users, the index is stored per mailbox. This reduces theindex sizes and reduces the number of rebuilds required. The overall result is fewer index rebuilds forusers and smaller indexes, requiring less storage performance.CPU utilization: The core count Exchange can leverage has increased to 48 cores in Exchange2019. This supports more parallel tasks and can increase the number of I/O streams a server cangenerate based on the workload it is processing. This needs to be considered when designingstorage pools.Dynamic database cache: The goal of the Dynamic Cache is to tune memory more effectively foractive and passive databases. This results in more memory for active databases, increasing thecache read hits. This will result in fewer reads.Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Configuration2Configuration2.1Disk layoutMicrosoft provides the following storage configuration best practices for Exchange 2016/2019:RAID is often used to both improve the performance characteristics of individual disks (by striping data acrossseveral disks) and provide protection from individual disk failures. With the advancements in Exchange2016/2019 high availability, RAID is not a required component for Exchange storage design. However, RAIDis still an essential component of Exchange storage design for standalone servers as well as solutions thatrequire storage fault tolerance.1Determining the storage layout before the installation of Microsoft Exchange Server is an important step sinceit can have direct impact on performance when using other disk solutions.With Exchange Server 2016, due to the reduced IOPS required, Microsoft allows placement of logs anddatabases on the same volume for DAG-protected databases. The Jet database (EDB) activity resemblessequential reading and writing to 32 KB blocks. The transaction logs see 100 percent sequential reads andwrites.The following table shows a sample disk layout based on best practices.DriveRecommended configurationContentsC:DAS/SANWindows, Exchange binariesD:DAS/SANPagefileE:SANDatabase 1 / Logs 1F:SANDatabase 2 / Logs 2When using Exchange Server 2016/2019 Database Availability Groups (DAGs) to create a highly resilientdatabase infrastructure, Microsoft Preferred Architecture guidance discusses distributing three copies plus alagged copy across DAG members, and utilizing the same spindle on each of four servers to host an activecopy, as well as two passive copies and a lagged copy of each of the four server's databases.1 Microsoftdesign guidance is specifically for JBOD (non-SAN) environments where larger, slower, single spindles areused to provide storage for Exchange databases. Therefore, this Microsoft guidance does not apply to SCSeries SAN volumes. This is due to the following reasons: Because RAID is inherent in SC Series arrays, 3-copy DAGs are not an absolute requirement forresilience. 2-copy DAGs are more common.When taking volume snapshots (Replays) with Replay Manager, an additional lagged database copyis not needed.Ross Smith IV, “The Exchange 2016 Preferred Architecture,” October 19Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

Configuration The risk of database reseeding is very low when compared to a single disk JBOD. If a disk drive in anSC Series array fails, a SAN spare is automatically used and the server and application are unawareof any hardware change.The main reason Microsoft best practices put these multiple databases on a disk is to allow for fasterreseeding of the storage space from multiple sources. Since this is not a factor for SC Seriesvolumes, Dell EMC does not recommend this as a best practice.Exchange 2016/2019 supports 100 databases per server with up to 16 copies of each to other mailboxservers in a DAG organization. A best practice is to minimize the number of databases, using as few asrecovery objectives will allow. As the number of databases increases, so does the I/O required to support theadditional data streams. This can have a negative impact on the I/O load of a system.As environments grow larger and larger, it becomes common to run out of drive letters for the volumes. Forthe purpose of scalability, it may be suitable to use volume mount points (VMP) for database and logvolumes.The following table shows a sample disk layout based on best practices using mount points.DriveRecommended configurationContentsC:DAS/SANWindows, Exchange binariesD:DAS (if available)PagefileC:\DBSANMount pointsVMPC:\DB\Database1Database 1 / Database 1 logsVMPC:\DB\Database2Database 2 / Database 2 logsVMPC:\DB\Database3Database 3 / Database 3 logsThe performance enhancements in Exchange 2019 allow a greater number of users per server with improvedlatency. The benefits of increased user count per server need to balance user impact during maintenance oroutages. If a server needs to be replaced or fails it will impact additional users. This needs to be factored inwhen sizing databases or server.The failover and re-seed performance of Exchange has been improved in Exchange 2019. This is due toindex location and architecture changes, as well as reseed and failover behavior.2.2SC Series page size considerationsBy default, the SC Series array writes data to individual 2 MB pages in a hybrid or spinning drive system.Systems with only SSD drives will use a 512 KB page size. Exchange 2016/2019 writes database pages of 32KB each. Write concatenation will attempt to put as many I/Os together to fill a page where possible. This willresult in I/O sizes of 256–384 KB in size.It is very likely that during one day, many pieces of the database will be accessed and changed. That beingconsidered, it is also important to understand that a single block being altered constitutes a change of that 2MB or 512 KB page where it is located on the SC Series array. Potentially, that could be the only block on thatpage that was changed. Updating a single email also requires updates to multiple indexes. This can result inlarger-than-expected snapshot sizes.10Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

ConfigurationSC Series customers who experience large snapshots (75 percent or greater of database size) can considerthe 512 KB page option (not available on SCv2xxx arrays) for their Exchange volumes on the SC Series arrayif they are running hybrid or spinning drive systems. This will reduce the number of blocks written to a page,thus reducing the snapshot sizes. The tradeoff with this smaller page size is reduced storage capacity.2.3Storage profiles and Data ProgressionStorage profiles were introduced in Storage Center OS (SCOS) 4.2. For SC Series models, theRecommended storage profile will provide the best Data Progression configuration for most implementations.Essentially, this storage profile will write new data to RAID10 with snapshot data being placed on RAID5–9.For SCv2xxx models, either the Balanced or Maximize Efficiency storage profiles are recommended.Maximize Efficiency provides the most available storage capacity using RAID5/6, while Balanced providesRAID10 performance on all writes and RAID5/6 for Replays. SCv2xxx models only provide Data Progressionbetween RAID levels when using Balanced.2.3.1Microsoft recommends ReFS for Exchange 2016 database volumes (JBOD)This recommendation is for JBOD and not SAN volumes and should not be used for Exchange 2016database and log volumes on SC Series arrays. According to Microsoft guidance in The Exchange 2016Preferred Architecture, each disk that houses an Exchange database is formatted with ReFS (with theintegrity feature disabled) and the DAG is configured such that AutoReseed formats the disks with ReFS.2.3.2Default allocation unit sizeExchange reads and writes to the database in 32 KB chunks. Formatting volumes with the correct defaultallocation unit size is also an important best practice. The default allocation unit size of 64 KB isrecommended.2.3.3Partition typeGUID Partition Table (GPT) is recommended for Exchange 2016/2019 volumes. Volumes larger than 2 TBrequire that GPT is used to access the entire size of the volume. Master Boot Record (MBR), the defaultpartition type, is also supported but the partition size is limited to less than 2 TB.2.4Mailbox server sizing considerationsUnderstanding user patterns is an important part of determining sizing parameters for any Exchange Server.Specifically, tools like Jetstress, Profile Analyzer, and LoadGen (described below) can help determine thebest settings for the type of mailbox user, mailbox sizes, and server configuration. By having information suchas mailbox count, required mailbox size, send/receive statistics, and average message size, Jetstress canuse these parameters to run a performance test to simulate how the specific configuration would perform.Servers should be sized before they are moved into production. This process should include a Jetstress testto determine if the hardware and disk configurations are suitable for the type and amount of traffic to beexpected. Once a successful Jetstress test is completed, Exchange Server can be installed and client testingcan be performed using LoadGen. LoadGen simulates actual client traffic operations such as sending andreceiving mail, creating calendar appointments, and accessing public folder data. Several other tests areavailable as part of the profile configuration and can be customized as necessary.11Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

ConfigurationMailbox I/O requirements are substantially lowered since Exchange 2013. The overall I/O requirement forExchange 2016 mailboxes has decreased by over 50 percent from Exchange 2010. Exchange 2019 data isnot yet available.Mailbox profile I/O requirements0.350.325IOPS per 0072010Exchange version(profile: Heavy)Change in IOPS per mailbox requirements by version of Exchange ServerMicrosoft provides the following information on simulation and analysis tools for Exchange (links provided):Jetstress 2013 should be used to simulate disk I/O load on a test server running Exchange 2013 or 2016 toverify the performance and stability of the disk subsystem before putting the server into a productionenvironment. Find more information on Load Generator is a simulation tool used to measure the impact of MAPI, OWA, ActiveSync,IMAP, POP, and SMTP clients on Exchange 2013–2016 servers. Find more information on the pageExchange Load Generator 2013 (64 bit).Exchange Profile Analyzer collects estimated statistical information from a single mailbox store or across anExchange Server 2007 or 2010 organization. The collected data can be used to analyze the performance andsizing of a server with mailboxes that will be migrated to Exchange 2016. Microsoft Exchange Server ProfileAnalyzer (64 bit).Exchange Server Role Requirements Calculator, a spreadsheet-based calculator, is very helpful indesigning the best environment (both storage and hardware) based on criteria including the following: 12User profile: message profile, mailbox size, and number of usersHigh availability architecture: number of database copies planned to deploy, whether the solution willbe site resilient or not, and desired number of mailbox serversServer CPU platformStorage architecture: disk capacity/type and storage solutionBackup architecture: whether to use hardware/software VSS and specify the frequency of thebackups, or leverage the Exchange native data protection featuresNetwork architecture: utilization, throughput, and latency aspectsMailbox I/O: at this time, the Exchange 2019 mailbox I/O load is still pending (document will beupdated when data is available)Dell EMC SC Series: Microsoft Exchange Server Best Practices CML1037

ConfigurationDownload the Exchange Server Role Requirements Calculator.2.4.1Archive mailboxesEach mailbox in the database can have an archive mailbox. This allows users to have an archive repositorythat does not impact existing mailbox quotas and allows them to retain data for longer periods of time. Archivemailboxes can help eliminate PST files and keep information contained within Exchange Server. It isimportant to note that an archive mailbox is a separate object from the user mailbox.Beginning with Exchange 2010 Service Pack 1 (SP1), it is possible to provision a user's personal archive onthe same mailbox database as the user's primary mailbox, on another mailbox database on the same mailboxserver, or on a mailbox database on another mailbox server in the same Active Directory site. This featureoffers the flexibility to utilize SC Series tiered storage architecture to store archive mailboxes on a separatevolume utilizing a tier-3 RAID5 storage profile, putting this archive data on less expensive disks. Snapshotscan be taken much less frequently because this data changes less frequently.Note: Archive mailboxes are only available while a user is online. They are not available if a user is running incached mode and is not connected to Exchange.2.5Online database maintenanceDatabase maintenance for Exchange 2013/2016/2019 has also been modified to work more efficiently. One ofthese changes regards background database maintenance (BDM), which is now throttled back from 5 MBper-sec/copy to 1 MB-per-sec/copy.In addition, the updates to ESE can reduce the cost of maintaining and managing a database. Databasemaintenance is comprised of storage mailbox maintenance and ESE database maintenance.2.5.1Online defragmentationOnline defragmentation now runs 24x7 in the background by default. There are no configurable settings withthe default feature. Exchange monitors the database as it is being used and small changes are made overtime to keep it defragmented for space and contiguity. Online defragmentation is also throttled so that it doesnot negatively impact client performance.2.5.2Online database scanning (checksumming)The checksumming process can run in two different modes on the active database copies:Default option: This has checksumming run in the background 24x7. This option is intended for mostdatabases which can require longer periods of time to complete a checksum. Exchange scans the fulldatabase no more than once a day and will generate an alert if it does not finish scanning within seven days.Second option: This runs as the last task in the custom-scheduled mailbox database maintenance process.The amount of time it runs can be configured by altering the schedule calendar, which is recommended fordatabases

The Exchange 2016 Mailbox server role includes all server components from Exchange 2013 Mailbox and Client Access roles: . copies) to be hosted on the same volume and is