GALAXY REDUNDANT SERVER SOLUTIONSQL SERVER BACKUP & RECOVERY PROCEDUREUsing Carbonite Availability 8.1 powered by Double-TakeWHAT’S IN THIS GUIDEThis manual is a procedural guide that provides » concept overview of the data replication and failover process» server, network and system operation requirements» procedural steps to manage data replication jobs, failovers, and related tasksMicrosoft , Windows Server and Hyper-V are registered trademarks of the Microsoft Corporation in the U.S. and other countries.Double-Take is the registered trademarks of the owners of Carbonite Availability Carbonite Availability powered by Double-Take Software version 2017 Galaxy Control Systems All Rights ReservedNovember 10, 2017


Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1INTRODUCTION OF REDUNDANT SERVER SOLUTIONGalaxy Control Systems offers an engineered redundant server solution using the Carbonite AvailabilitySolution for data replication with Microsoft Hyper-V virtual environment.The Primary and Standby Servers are prebuilt at Galaxy Control Systems. Both servers are preloaded withCarbonite Replication Solution, Hyper-V , Windows Server OS, and appropriate services. System GalaxyAccess Control software is also included.The server designated as the Primary will begin as the live host (source). The Standby server willbegin as the replication target server. When the primary system is “healthy” and the datareplication job is actively “protecting”, the backup copy is being synchronized in 10-sec intervals.IMPORTANT SECTIONS IN THIS GUIDEINTRODUCTION:PROCEDURES:» Overview of Healthy Redundant Server» Services» Stages of Disaster Recovery» Set Up Email Notification (failure alerts)» Terminology Used» Type of Failover» Recent Change History» New Replication Job» View the Health of Replication Job» Identify a Replication Failure (Jobs View)REQUIREMENTS & SPECIFICATIONS» Perform a Manual Failover» System & Operational Requirements» Resume Replication (after Failover Complete)» Network Requirements» Specifications for Redundant ServersCopyright 2017 Galaxy Control SystemsPage 3 of 36Revision: Jan 2017

GALAXY 'HOW TO' GUIDEServer Replication & FailoverCarbonite/Double-Take 8.1TERMS USED IN THIS GUIDESOFTWARECarbonite Availabilitythe data replication & site recovery solution; manages replication between servers.Replication Consolethe software UI used to configure, manage, monitor the replication jobs, perform thefailover and related tasks.Hyper-V(VM) the Microsoft software used to create & maintain a virtual environment; SystemGalaxy software & database will install and operate on the VM.System Galaxy(SG) the integrated access control software from Galaxy Control Systems.SERVERPrimary Serverlive hostthe physical server currently operating as the “live host” – i.e. the server that is activelyhosting the running copy of System Galaxy on a virtual environment (i.e. Hyper-V) .Target Serverstandby serverthe physical server operating as the “standby server” or backup server during replication,which is storing an up-to-date image of the live host while standing by for failover.PROCESS“failed”the activity state that indicates the Double-Take replication job has detected a failure and is nolonger updating or backing-up to the standby server (replication job is not protected).Failoverthe act of performing a failover or transferring live operations to the target/standby server.“failed-over”the activity state that indicates that the standby server has resumed live operation; however,data is not yet being replicated/protected.healthy job statuswhen the data/software is being replicated and job activity state is “Protecting”.notificationan email that is distributed to designated addresses to notify of replication errors or warnings(optional / recommended).“protecting”replication job activity state that indicates data is being backed-up or copied to a standby server.replicationthe operation of storing the current, up-to-date copy of the data and software from a live/sourceserver onto a standby/target server.replication failurea condition that interrupts the active data replication due to a fault or failure of the serverhardware, system or connectivity between the live/source server and the standby/target server.resume replicationmanually recreating the replication job to re-establish duplication after a failover has completed.[“Resume replication” replaces the operation of ‘reverse roles’. Once the failover is completedand the standby Target server assumes live-operation in the “production” environment, it mayremain in place as the Primary (host) server. A replaced or repaired server becomes the newStandby/Target. The replication job must be recreated to re-establish data duplication.]“stopped”the activity state showing that data replication has stopped.“synchronizing”the activity state showing that the Double-Take software is synchronizing the mirror image onthe standby server to match the source server image.Copyright 2017 Galaxy Control SystemsPage 4 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1IMPORTANT CHANGES SINCE v7 (IN THIS GUIDE)1. The “Replication Console” replaces “Double-Take Console” (UI & terminology). The new Carbonite ReplicationConsole is considered to be “powered by Double-Take”. The basic features are still available for a virtualenvironment with the exceptions below.a) Removed “Undo-Failover” button/option (the automated feature is currently unavailable when using avirtual machine in the replication process).b) Replaced “Reverse Roles” with “Resume Replication”. The Reverse Roles automated feature/button isunavailable to the virtual environment. The steps to reverse roles has been replaced with manual stepsto recreate a new replication job after a failover completes – see “RESUME DATA REPLICATION (afterFailover Completed)”.2. Updated instructions throughout this guide to reflect the currently supported process & terminology.a) Updated Requirements and Diagrams with applicable removals and substitutions.b) All screens have been updated. Updated Terms & Definitions table.Copyright 2017 Galaxy Control SystemsPage 5 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1OVERVIEW OF HALTHY REDUNDANT SERVERThe redundant server solution provides a means for quick disaster recovery in case of a catastrophic failureon the live/source server. The down-time is minimized, by the ability to switch to a stand-by server thathas been mirroring the live server. This cutover to the stand-by server is called a FAIL-OVER.IMPORTANT: There is a difference between a “failed” condition and a “failed-over” condition in the job status.During a failure you should be aware of these things! During a server failure condition, the System Galaxy database is offline (i.e. not running on eitherserver) and the data replication is stopped/failed (see Figure 2).However, all Galaxy control panels remain fully functional and do not degrade in performance because allnecessary programming is stored in the panel. This means all door & lock schedules, access, credentials/cards,input & output activation, arming/disarming, relays, etc. and all hardware remains fully operational. Eventswill be transmitted to the database when the server is online again. Once a Failover is complete, the System Galaxy comes back online on the standby server.Data synchronization (mirroring) is still stopped/failed until you recreate the job that resumesreplication. Since SG is online, the panels will reconnect and transmit their events to the SGdatabase. System Galaxy extended functions can resume (e.g. badging, enrolling, monitoring, etc.).See Figure 3. At this point replication has not resumed. After the failure is repaired and the downed server comes back online, the administrator can resumereplication by recreating the job. the server images will re-synchronize/replication job will resume aprotected state (fig. 4).FIGURE 1: Redundant Servers in their Initial Roles (live host & stand-by servers are both healthy)When the replication job’s Activity is in the “protecting” state and the condition is healthy, then a replica of theSG database is being updated and stored on the target/standby server. See the section on Stages of a DisasterRecovery Process for details about interim states such as failed, failed-over, .Copyright 2017 Galaxy Control SystemsPage 6 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1STAGES OF FAILOVERFIGURE 2: Redundant Servers are in Failure Condition (SG is not running on either server)When a failure condition occurs at the source server, the data replication is interrupted. The state of thereplication job will change from ‘protecting’ to ‘failure condition met’ and replication will “stop”.NOTE: Galaxy database will not be running on either server at this point, but the System Galaxy access control and all otherhardware functions (scheduled locks/unlocks, access approval/denial, detection of hardware, arming/disarming, etc.) will allremain fully operation in a non-degraded performance. Events are buffered at the panel until transmission is possible.IMPORTANT: SG Control Panels are designed to continue operating in full operational mode (non-degraded) when powered on.All cards, schedules, access rules and door/device programming is stored in the panels to prevent failure of the access controlsystem if the server goes offline. Events are buffered in panel memory until the panel reconnects to the server. Events aretransmitted to the server when the server communication is re-established.Copyright 2017 Galaxy Control SystemsPage 7 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1FIGURE 3: Target Server is Failed-Over (SG resumes running on stand-by server/Hyper-V)Once the FAILOVER process is performed at the target server, System Galaxy will start up (running onHyper-V ) on the standby/target server(02). Although System Galaxy is running again, redundancy or datareplication has not been restored.FIGURE 4: Replication has Resumed (live & stand-by servers are both healthy)Once the failure condition is corrected/repaired, the administrator can current Primary (virtual machine) mayremain in it's current location after the failover is completed, thereby preventing additional downtime. The newreplication job would then provision a brand new virtual machine as a backup, on the opposing physical host.The system is considered healthy again when the replication job is recreated and resumes replication (i.e.both servers are re-synchronized and data replication state indicates “protecting”).Copyright 2017 Galaxy Control SystemsPage 8 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1REQUIREMENTS & SPECIFICATIONSSYSTEM & OPERATIONAL REQUIREMENTS1. Galaxy Control Systems supports server replication using Hyper-V and Double-Take software when theredundant server solution is purchased through Galaxy.2. Microsoft Windows Server 2012 2016 will be running on the host operating system of the primary andstandby/target servers, as well as on the virtual machines.3. Hyper-V and Double-Take software must be installed and running on the operating system of both thesource/host server and the standby/target servers.4. The Double-Take services must be set to run automatically. These services must be running on thePrimary/Host OS & virtual machine, as well as on the Target server. The target virtual machine image is “off”during protected status.» Double Take (Automatic Start)» DT Management Service (Automatic Delayed Start)5. The Double-Take firewall ports/exceptions must be open on both host servers and on network devicesacross the LAN as required (6320 TCP & UDP, 6330 TCP & UDP, 6332 TCP & UDP, and 6325 TCP & UDP).6. The active replication job will copy a backup image of the Hosted System Galaxy/Hyper-V session from theLive Primary server onto the Standby Target server. The replication job updates the Target server every fewseconds to maintain synchronization with the Live Primary.7. Double-Take should be configured to send email notifications for warnings and errors when a replicationfailure occurs. This is especially recommended for servers using manual failover procedures.8. The failover process may be initiated when a replication failure occurs.CAUTION: The cause of a replication failure should be carefully evaluated in order to determine whetherperforming a cutover/failover is the best course of action. IF a failover is necessary, the failover processmust be “completed” before the new live-host server is operational(1). The data replication job must berecreated before protected replication can resume.IMPORTANT: to resume replication after a cutover, a new replication job must be manually recreated. Theservers must be resynchronized and in the “protecting” state before redundancy is truly re-established.(1) NOTICE:System Galaxy access control panels continue to be fully operational (i.e. no degraded operation)because all access control information is stored in the panel. Events are stored at the panel and transmittedto the SG database after the cutover is completed and System Galaxy is back online.9. The automated option to “reverse roles” is no longer supported within the virtual environment. The onlyrecovery method available to a virtual environment is to resume replication by recreating a new job. -see“Resuming Replication after Failover is Completed”, when the Target server becomes the Live-Primary and anew Target is setup as the standby server.10. Resuming replication is now a manual process after failover is completed. You must rename your instancewhen you recreate the new replication job.11. In Carbonite 8.1 you cannot undo a failover when hosting SG in a virtual environment.Copyright 2017 Galaxy Control SystemsPage 9 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1NETWORK SETUP REQUIREMENTS1. You need three (3) static IP Addresses. One for each server and one for the Virtual Machine. AssignIP addresses to 2nd NIC card on each host server.2. IP Addresses must be valid within the Network IP Range for each server.3. It is recommended the System Administrator tags or labels the source server at all times.SYSTEM SPECIFICATIONS FOR REDUNDANT SERVERSIn the Galaxy redundant server implementation, the following infrastructure will be observed:HOST OS: SERVER-A and SERVER-B» Both servers (A/B) will run Microsoft Server 2012 2016 as the host operating system.» Both servers (A/B) will run the Double-Take services set to start automatically: Double Take, DT Hyper V, DT Management.» Both servers (A/B) will open both TCP & UDP ports 6320, 6330, 6332, and 6325 (necessary for Double-Take).» Both servers (A/B) will have the Double-Take Console software installed and able to run when in the role of thestandby/target.GUEST OS:» The Hyper-V guest virtual environment will run Microsoft Server 2012 2016 (or appropriate version) as the operating system.SERVER ROLES and RESPONSIBILITIES» The server operating in the live role (source) hosts the virtual machine where System Galaxy, database & services are running.» The server operating in the standby role (target) is where the data replication is updating the synchronized backup copy.» The live/source server will support card editing and event monitoring of the System Galaxy software.» The standby/target server will perform failover in the event of a catastrophic failure of the live server.» Once a failover is completed, the standby server will go-live and begin hosting System Galaxy. Replication must be resumedby recreating the replication job (manually).FIGURE 5: HEALTHY REDUNDANT SERVER SOLUTION (PROTECTED)Copyright 2017 Galaxy Control SystemsPage 10 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1REPLICATION CONSOLE PROCEDURESThis section contains the step-by-step instructions for the main tasks involved with configuring andoperating the Carbonite /Double-Take server.SETUP & CONFIGURATION STEPS:» About Services (services should auto-start when platform is powered on)» Set Up Email Notification (for failure alerts)» Configure Type of FailoverMANAGE REPLICATION» Create a New Replication Job» View the Health of a Replication Job» Identify a Replication Failure (Jobs View)» Perform a Manual Failover from the Target Server» Resume Replication after Failover is CompletedSYSTEM ERRORS» See the appropriate documentation from the solution manufacturer’s for identifying errors or contacttechnical support as needed.Copyright 2017 Galaxy Control SystemsPage 11 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1ABOUT SERVICESIMPORTANT» Services must be running before any configuration or operation can occur.» Carbonite/Double-Take Services must be running all servers (Primary, Target, & in the liveHyper-V session).QUICK STEPS1. Open the Services window at each server:»START Control Panel Administrative Tools Services.2. Locate the two Double-Take services. Ensure Status is “started” and Startup Type is “automatic”.» Double-Take (or name of Replication Console)» Double-Take Management (or name of Replication Console)3. Services should be set services to start automatically.Copyright 2017 Galaxy Control SystemsPage 12 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1SET UP E-MAIL NOTIFICATION (FAILURE ALERTS)IMPORTANT Double-Take services must be running at both servers (Primary, Target, and Hyper-V session). You must configure E-mail Notification at both servers (Primary and Target).QUICK STEPS1. Go to the Target Server’s host OS and double-click the [Double-Take Console] desktop icon to openthe Double-Take Console software.2. From the console toolbar, click on the[ SERVERS] button.3. Right-click the desired server name and select ‘View Server Details’ in the shortcut menu.Copyright 2017 Galaxy Control SystemsPage 13 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.14. Click the ‘Edit Server Properties’ in the TASKS list on the right side the screen.5. Expand the E-mail Notification option in the Server Properties screen.Copyright 2017 Galaxy Control SystemsPage 14 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.16. Check the [Enable Notification] option and enter the Mail Server Address ( Check the [Log onto email server] option and enter the Login Credentials for the e-mail server.8. Enter the From Address, and Send To address ([email protected]).9. Enter the desired text in the Subject Prefix field.10. (optional) You do not need to enable the add event description option – this will make your subjectline longer and you will see this in the message content anyway.11. Check the [Warning] and [Error] options.12. Click [TEST] to verify your e-mail will work with these settings.13. Save your settings and repeat these steps for the other Server.Copyright 2017 Galaxy Control SystemsPage 15 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1CONFIGURE TYPE OF FAILOVERYou must determine whether you want to depend on a manual failover process or an automatic failoverprocess. A manual failover/cutover must be performed by the system administrator after Double-Takesoftware has reported that a ‘condition for failure’ has been met. An automatic failover is initiated by thesoftware whenever the system has met a ‘condition for failure’.IMPORTANT If using a manual failover, the user must perform the failover from the TARGET SERVER. An automatic failover will cut-over replication to the target sever without user intervention.QUICK STEPSThe type of failover is configured in the Job Properties screen. This can be done from either the target orthe source server once the job has been created.1. Click the[JOBS] button on the main toolbar, then click thethe lower toolbar.[VIEW JOB DETAIL] button on2. Click the EDIT JOB PROPERTIES link under the TASKS list, on the right side of the screen.Copyright 2017 Galaxy Control SystemsPage 16 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.13. In the Edit Job Properties page, expand the Failover Monitor section and set the following:a. Enable (“check”) the [Monitor for Failover] option.b. (Recommended) choose [Total Time to Failure] option and set Time to the desired value.c. Set the [monitor on this interval] value to an appropriate value.d. Enable (“check”) the [Wait for User to Initiate Failover] option if you want the system towait for the administrator to manually initiate a failover/cutover. Unchecking this optionwill allow the system to automatically failover when a condition for failure is met.The “Wait for User to Initiate” option: Enabling (checking) this option means the systemadministrator must manually initiate a failover when the software reports that a ‘conditionfor failure’ has been met (e.g. Target server looses communication to the Source server).Disabling (unchecking) this option means the Double-Take system automatically initiates afail-over when any ‘condition for failure’ has been detected by the software. An automaticfailover may be undesirable in controlled tests/planned outages (e.g. server/router reboot).Copyright 2017 Galaxy Control SystemsPage 17 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1CREATE A NEW REPLICATION JOBThe steps below show how to create a new replication job in the case that the target server goes downand the replication job is dropped.1. Go to the server host operating system and double-click the Replication Console startup icon onthe desktop to open Replication Console (Main Screen – All Servers).2. Right-click on the bottom (licensed) instance of the Double-Take Availability Virtual server (in thelist of servers) and choose “Protect” from the shortcut menu.Copyright 2017 Galaxy Control SystemsPage 18 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.13. Choose the ‘Full Server to Hyper-V ” option and accept all defaults.4. Choose the ‘Protect files and folders, application or entire server’ task.Copyright 2017 Galaxy Control SystemsPage 19 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.15. Under the CURRENT SERVERS list, select (highlight) the name of the physical server that you wantto operate as the Target/standby server.The target is the machine that is NOT currently hosting the live session running System Galaxy.IMPORTANT! if you are recreating a job after failover completed, you must verify which machine isnow the Primary and is currently hosting the “running” virtual session. Use the Hyper-V Console toview determine which server is currently “running” the live instance of Hyper-V/SG. Do not selectthe currently running server as the Target.WARNING! Failure to choose the correct server could result in total loss of data in the event of afailure. Never Back-up your synchronized copy onto the currently running (live) server. If you set therunning host as the target you are not truly protected, even if though the replication job indicatesthe “protecting” state. A catastrophic failure will prevent a successfully failover and your backupdata can be permanently lost. Proceed with caution!6. Click the NEXT button to continue.Copyright 2017 Galaxy Control SystemsPage 20 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.17. Under GENERAL, enter the Job Name (be unique/descriptive).8. Under REPLICA VM CONFIGURATION, enter the Display Name (be unique/descriptive).9. Accept all the defaults.10. Click NEXT to continue.Copyright 2017 Galaxy Control SystemsPage 21 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.111. Double-take will perform the obligatory checks.12. Click FINISH to save and continue. Your standby Target server will begin creating the backup image.Copyright 2017 Galaxy Control SystemsPage 22 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.113. The user is returned to theJOBS screen and the replication will connect and beginsynchronizing with the Primary (source) virtual machine.The process will cycle through several states of activity – i.e. Provisioning, Synchronizing Calc,Synchronizing %, – be before reaching the “Protecting” status. This process will take a while.14. Swap over to Hyper V Console and verify the Target server instance was created as intended.Target state should be “off”.Copyright 2017 Galaxy Control SystemsPage 23 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.115. Return to the Replication Console to monitor the activity status of the new replication job.The synchronization is complete when the job indicates “Protecting”, and Data State is “OK”.Copyright 2017 Galaxy Control SystemsPage 24 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.116. In Hyper-V Console, verify the states of both Hyper-V sessionsa. the Primary (live/host) HYPER-V session should be “running”.b. the Standby/Target HYPER-V session should be “off”Primary (live) Hyper-V SessionTarget (standby) Hyper-V SessionCopyright 2017 Galaxy Control SystemsPage 25 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1VIEW THE ‘HEALTH’ OF A REPLICATION JOB1. Go to the Target Server’s host OS: Double-click the Replication Console desktop icon to openDouble-Take Console.2. Click theJOBS button on the main toolbar.3. List of Jobs Panel: When data is actively replicating, the Job status indicates a green arrow and theActivity column displays “Protecting”.4. Job Highlights Panel: The Target Data State should indicate “OK”Copyright 2017 Galaxy Control SystemsPage 26 of 36Revision: Jan 2017

Server Replication & FailoverGALAXY 'HOW TO' GUIDECarbonite/Double-Take 8.1IDENTIFY A REPLICATION FAILURE (IN JOBS VIEW)A replication failure at the source server will interrupt the source server’s ability to continue protecting thesystem. The failure will be reported through emails and status changes. You will go to the Target Serverto confirm the failure.1. Go to the Target Server’s host operating system and double-click the Carbonite ReplicationConsole startup icon on the desktop to open Replication Console.2. Click theJOBS button.a. The replication job may display a Red-X (it depends on the reason for failure)b. You should get email(s) for a failure if you are properly set up to receive notifications.c. Job Highlights panel may also show a red-X or caution symbol indicating the failure.d. The FAILOVER button will be enabled/ON at the target server.e. When a failure condition occurs, yo

that indicates the Double-Take replication job has detected a failure and is no longer updating or backing-up to the standby server (replication job is not protected). . showing that the Double-Take software is synchronizing the mirror image on the standby server to match the source server image. Server Replication & Failover .