# Troubleshooting and additional commands ## Additional Commands Example output (v8.5): ```language-markup Backup Schedule: Scheduled backups are enabled on Sundays at 5:00 UTC. External target upload is disabled. External Targets: scp-1: SCP Target ubuntu@192.168.12.34:/backups, max backups 5 sftp-2: SFTP Target ubuntu@192.168.56.78:/backups, max backups 5, using password authentication s3-3: S3 Target us-west-1 bucket backups-bucket, path ., max backups 7 to-brain-4: Brain-to-Brain Target example.com Allow-list: This brain will accept backups from: all hosts Events: Last successful backup: Thu 27 Jun 2024, 03:04PM UTC Last backup failure: (Never) Reason for last backup failure: (None) Last upload to external targets: (Never) Last failure to upload to external targets: Thu 27 Jun 2024, 03:06PM UTC Reason for last copy failure: ssh: connect to host 192.168.12.34 port 22: Connection timed out Public Key for SCP / SFTP Backups: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsoX+hHcZYEljEpMbaf135A14NMNER512UHPNUazhwc/J17Ob0KyQWclY463E3Z ``` **Clearing Backup Configuration** Executing the below command will reset the backup configuration to default values: ```language-markup backup clear ``` **Showing Backup Events** ```language-markup show backup events ``` Example output: ```language-markup Last successful backup: Tue 16 Apr 2024, 04:15PM Last backup failure: (Never) Reason for last backup failure: (None) Last upload to external targets: Tue 16 Apr 2024, 04:18PM Last failure to upload to external targets: (Never) Reason for last copy failure: (None) ``` In version 8.6 and higher the `show backup summary` command will show the last backup duration. The duration shown will NOT include the time it took to copy the backup to any external targets. **Viewing Backup Configuration** To see the currently backup configuration: ```language-markup show backup summary ``` **Generating backup link for download** It will prompt a link which can for download the backup file. This feature only works for Quadrant UX. Respond UX will prompt the link but the download page will not be available (it will be in the future). ```language-markup show backup download-link ``` ## Troubleshooting ### **Status Messages Related to Backup/Restore** You may receive status messages when running commands related to backup/restore. When running interactively, these can easily be viewed at the CLI. When you've setup a backup schedule it is necessary to monitor the audit log for messages related to backup/restore. Audit log messages can be sent as part of system logs which can be retrieved via API in Quadrant UX deployments, or sent via Syslog or Kafka for Quadrant UX deployments. If you are unsure of your deployment type. For information related to using QUX APIs, please see: * Quadrant UX * [How to query Vectra REST API using Postman platform](https://docs.vectra.ai/configuration/access/api-qux/v25-postman-quick-start-guide-using-token-auth) * [REST API Guide v2.5](https://docs.vectra.ai/configuration/access/api-qux/v25-api-guide-qux) * [Vectra v2.5 Public Postman Collection](https://www.postman.com/vectra-internal/workspace/vectra-ai/collection/7429442-509c593b-8de4-4aa3-ac05-e07934eb9f17) ### **Example Messages** Messages are sent for the following in the source Brain's audit log: * `Backup started` * `Error running backup: …` / `(Backup|Restore|Factory Restore) is already running. Try again later.` * `Backup complete` * `Restore started` * `Error running restore: …` / `(Backup|Restore|Factory Restore) is already running. Try again later.` * `Restore complete` * `Attempting backup copy to …` * `Attempting to rotate files on external target …` * `Backup copy to … completed. SHA-256 sum is …` * `Could not copy to …: …` * `Could not list backups on …` * `Failed to list directory on SFTP External target …` * `Could not delete existing backups on target system.` * `Attempting to stage backup to Brain-to-Brain Target …` * `Staging backup to Brain-to-Brain Target … failed. Error: …` * `Staging backup to Brain-to-Brain Target … succeeded` * `Failed to perform scheduled backup: …` * `Uncaught error running scheduled backup: …` * `Failed to upload migration-… to …` Messages are sent for the following in the target Brain's audit log (when using Brain-to-Brain backups): * `Failure retrieving remote backup size from …` * `Successfully transferred remote backup from …` * `Failure transferring remote backup from …` Local Backup Started/Completed : ```language-markup 2024-07-02T20:21:56+00:00 VHEef07obfuscated4e02f3d08b0c7e5a36 vectra_json_localaudit -: {"user": "root", "role": null, "source_ip": null, "headend_addr": "192.168.49.238", "dvchost": "192.168.49.238", "version": "8.5", "result": "pending", "message": "Backup started", "vectra_timestamp": "1719951716"} 2024-07-02T20:24:36+00:00 VHEef07obfuscated4e02f3d08b0c7e5a36 vectra_json_localaudit -: {"user": "root", "role": null, "source_ip": null, "headend_addr": "192.168.49.238", "dvchost": "192.168.49.238", "version": "8.5", "result": "success", "message": "Backup complete", "vectra_timestamp": "1719951876"} ``` External Target Copy Pending/Success: ```language-markup 2024-07-02T20:26:00+00:00 VHEef07obfuscated4e02f3d08b0c7e5a36 vectra_json_localaudit -: {"user": "root", "role": null, "source_ip": null, "headend_addr": "192.168.49.238", "dvchost": "192.168.49.238", "version": "8.5", "result": "pending", "message": "Attempting backup copy to S3 Target us-west-2 bucket vectra-test-bc4ef5a390obfuscated0ee4de0, path .", "vectra_timestamp": "1719951960"} 2024-07-02T20:26:02+00:00 VHEef07obfuscated4e02f3d08b0c7e5a36 vectra_json_localaudit -: {"user": "root", "role": null, "source_ip": null, "headend_addr": "192.168.49.238", "dvchost": "192.168.49.238", "version": "8.5", "result": "success", "message": "Backup copy to S3 Target us-west-2 bucket vectra-test-bc4ef5a3900146dobfuscated4de0, path . completed. SHA-256 sum is '6735b95cb8a3756074c9f3f0afece007d28d320d5ca918443257e4cd9fe7d738'", "vectra_timestamp": "1719951962"} ``` ### **Brain-to-Brain Common Issues** **Incorrect or missing token** * If the token is missing or incorrect, Brain-to-Brain backup test or execution will fail. * On the target Brain, execute `show backup from-brain token` to display the token and ensure this token is used to configure the external target on the source Brain. **Required firewall rules missing** * The source and target Brains need to have TCP/22 and TCP/443 open between them. This should typically be configured bidirectionally so that either end can initiate a connection to allow the Brains to backup to each other. ### **SCP/SFTP Common Issues** **Permission denied error** * Ensure the user, server, and path are configured correctly for the target server. * Ensure the public key from the source Brain is copied to the `~/.ssh/authorized_keys` file on the target server. * Check that the permissions are correct for both the .ssh directory and the `authorized_keys` file on the target server: * `~/.ssh` must have 700 permissions. If not, run: * `chmod 700 ~/.ssh` * `~/.ssh/authorized_keys` must have 600 permissions. * If not, run `chmod 600 ~/.ssh/authorized_keys` * Does the backup target path have read/write permissions for the configured user? * Is there sufficient space on the target path? * Is there SSH Inspection or firewall level block between the Brain and target server? * The source Brain must be able to reach the external target server over TCP/22. **Command** `backup external-targets test` **succeeds, but a full backup run fails** * This scenario usually occurs when connection to remote server has been closed abruptly or the network between Brain and target server is unstable. ### **Collecting Information for Vectra Support** If you need to open a Vectra support case to get additional assistance it will be valuable to provide information from your system(s) to help with the troubleshooting process. Please collect the following information and provide it to Vectra in your support ticket. **From Source Brain:** ```language-markup show version show interface show backup debug connectivity 443 --bypass-proxy debug connectivity 22 --bypass-proxy backup external-targets test ``` **From Target Brain (when using Brain-to-Brain backup):** ```language-markup show version show interface show backup debug connectivity 443 --bypass-proxy debug connectivity 22 --bypass-proxy ``` Generate system report on the source Brain with the below command. It should take few minuted to run all the checks and collect logs: ```language-markup status-report generate ``` Use the following command to list all reports: ```language-markup status-report list ``` Copy the URL associated with latest report (i.e. highest ID number). Put the URL ion your favorite browser to download the report.