DR (Disaster Recover) process

Prerequisites

• Brain Versions: The version of the backup that is being restored, must match the major version running on the target Brain. The version of the backup is shown in the filename of the backup.

• Target Brain Capacity: Ensure target Brain is of equal or comparable capacity/performance to the source Brain.

• DNS Configuration: DNS TTL for the domain used for the Brain hostname should be set to 120 seconds to facilitate quick DNS updates during failover scenarios when pairing by hostname is used.

• Backup Configuration: It is a best practice to ensure backups are configured on the source Brain for both Brain-to-Brain and at least one other external target.

  • When using an external target, if the same external target is not configured identically on the target Brain, you will need to provide a URL to use with a restore operation for S3, SFTP, or SCP external targets.

  • Please see Restoring Backups in Backup and Restore for Vectra Brain Appliances (v8.5+) for details and Backup Configuration in the parent page of this page.

• Connectivity:

  • For Brain-to-Brain backup use - TCP/22 and TCP/443 are open between the Brains.

  • For S3 external targets - TCP/443 is open to AWS S3 from the Brain.

  • For SCP/SFTP - TCP/22 is open from the Brain to the external target.

Definitions

• Brain1 - Primary Brain

  • This Brain is the normal Brain in use for day to day operations.

  • A Brain-to-Brain external target should be configured that points to Brain2.

  • Another external target should be configured that points to AWS S3 or a server reachable via SFTP or SCP.

• Brain2 - Spare Brain

  • This Brain may be in another data center or other secure location but it is online and reachable from Brain 1.

  • This Brain may have its own backup configuration but it should be noted that once a backup is restored to this Brain, the backup configuration will be overwritten. Please see Backup Configuration in the parent page of this page for details.

Assumptions

• Brain1 is configured with a hostname in your DNS and pairing of network Sensors was done by hostname.

  • If you have only an IP address configured for your Brain and it is not in DNS or your network Sensors were paired by IP, then you will need to adjust the procedure to re-pair the network Sensors to the Brain2 IP after a backup is restored to Brain2.

  • See Vectra Network Sensor Pairing Considerations for more detail.

• If you are not using S3 for the additional external target, then you must ensure the public key for Brain2 is also in the external target's ~/.ssh/authorized_keys file before you try step 8 below.

Alternative

• While Brain-to-Brain backup is preferred because it means there is already another Brain available to be restored to in the event of a failure, any other external target can also be used.

  • This procedure will need to be adjusted for the specific external targets configured for your deployment.

  • Without another Brain available, you will not be able to simulate failure as part of a DR exercise.

Test DR Process

1. Log into Brain1 GUI or your Respond UX UI and confirm it's operational with successful connectivity to all configured network Sensors.

2. Confirm that Brain-to-Brain and at least one other external target backup is configured on Brain1 and then login to the Brain2 CLI and confirm that at least one recent backup has been stored on Brain2.

3. Simulate failure of Brain1 by disconnecting or disabling the network port for the management interface on Brain1.

4. Log into Brain2 and restore the most recent Brain-to-Brain backup file available that originated from Brain1's serial number.

   • Ensure that the --replace option was used when restoring the backup file (Brain must be connected to the Vectra Cloud).

   • Be careful that you restore the correct backup if the target Brain stores backups for itself and for another source Brain.

5. Reconfigure DNS to to point the Brain1 hostname to Brain2's IP address.

   • If pairing by hostname was not done on Brain1, recovery is still possible but network Sensors will need to be re-paired to Brain2. See Vectra Network Sensor Pairing Considerations for more detail.

6. Log into Brain2 GUI or your Respond UX UI and confirm it's operational with successful connectivity to all configured network Sensors (~ 10-15 minutes).

7. Reconnect or enable the management interface to simulate recovery of Brain1.

8. Manually execute a backup that includes the external target on Brain2 and restore it on Brain1.

   • Ensure that the --replace option was used when restoring the backup file (Brain must be connected to the Vectra cloud).

   • Using an S3, SCP, or SFTP external target and not Brain-to-Brain backup is recommended because the configuration for Brain-to-Brain backup will delete Brain2 as an external target when the Brain1 backup is restored on Brain2.

9. Reconfigure DNS to point the Brain1 hostname back to Brain1's IP address.

10. Perform a reboot on Brain2 to break the established Sensor tunnels to re-establish them to Brain1.

    • Since the Sensors were paired by hostname and the backup contains Sensor state information, they will automatically pair to Brain1.

11. Log into Brain1 GUI and confirm it's operational with successful connectivity to all configured network Sensors. (~10-15 minutes)

12. Confirm Brain1 backups are still configured.

13. Reconfigure Brain-to-Brain backup to again point to Brain2 as its external target.

Actual DR Event

• In the event of an actual disaster or failure of Brain1, only steps 4 to 6 above are required to switch to using Brain2.

  • Since Brain1 is down, unreachable, etc, network Sensor tunnels should terminate on their own and after DNS is changed to point to Brain2, they will pair to Brain2 once it has restored the backup from Brain1.

• If you replace or fix Brain1's issue, then only steps 8-13 are required to revert back to Brain1.

• For Quadrant UX deployments only

  • If you are a recall customer, please work with Vectra support to ensure your Recall deployment is properly configured to work with the replacement Brain.

• If remote support was enabled through a proxy by Vectra support in the past for your original Brain, please work with Vectra support again to re-enable remote support through a proxy to your replacement Brain.