Traffic visibility drop alerting

Configure health checks and alerts for drops in traffic visibility, including IP counts, bandwidth, and packet counts.

Overview

As of the v9.11 release, Brain appliances monitor the below metrics for abnormality:

• The number of observed IP addresses - Aggregate for entire deployment
• The observed traffic bandwidth (per Sensor)
• The observed packet counts (per Sensor)

For each of these metrics, the system will collect data, determine a baseline, and create a system health alert when it is determined that any of these metrics have crossed a critical threshold. The exact cause will not be known to the system but it is likely a situation where an external network event (e.g. switch reconfiguration) has caused a reduction in network observability.

Vectra expects a high correlation between losses in network observability and the system health alerts related to traffic visibility drops, and advises customers receiving these notifications that they investigate their network traffic monitoring to determine what event(s) may have caused a loss in visibility.

Alerts can be received by email, webhook, retrieved via API call, or received via Syslog (QUX deployments only).

Alerting Configuration

Traffic visibility drop alerts are a sub category of system health alerts and if you are receiving system health alerts by any supported method, you will receive these traffic visibility drop alerts. An alert is created when a metric becomes critical and when the metric returns to normal.

There are also alerts related to the quality of the traffic being ingested. For detail on those alerts, please see Traffic validation (ENTV) alerting.

System Health Alert Notification Methods

Email notification: Please see System alerts for details.

Webhook notification: Please see External app alerts (webhook) for details.

Please Note:

For Webhook notifications, an alert will be sent when a metric becomes critical, but there will be no alert sent when the metric returns to normal. This is expected behavior and is not cause for alarm.

API:

• Perform a GET against the /events/health endpoint

  • Example URL: https://VECTRA_PORTAL_URL/api/v3.4/events/health/
• RUX API details please see https://apidocs.vectra.ai

  • Example URL: https://VECTRA_PORTAL_URL/api/v3.4/events/health
• QUX API details please see v2.5 API guide (QUX)

  • Example URL: http://your_QUX_URL_or_IP//api/v2.5/events/health

Syslog (for QUX deployments only): Please see the Syslog Guide (QUX)

Metric Thresholds

There are no specific thresholds shared because the system calculates what is normal for each system automatically.

In general, the system monitors traffic levels over 3-6 weeks and builds a baseline from that. It then periodically compares the previous 8 hours to the baseline.

Please Note:

Your system will not create any alerts until the system has confidence in a generated baseline. Vectra may change the thresholds and parameters discussed above in future updates.

Customers with extremely noisy traffic graphs (wide fluctuations vs predictable patterns) or very low bandwidth may create alerts when not desired or fail to create alerts when desired. If this happens in your deployment, please open a support ticket. There are some underlying parameters that Vectra can alter to attempt to optimize for your deployment that the system may not be able to calculate automatically.

Example Alert Messages

Below are some example email alert messages for a system crossing the critical threshold for observed IPs and another message for when the observed IP count returns to normal (a "recovery" alert).

Please Note:

Recovery alerts are sent via most channels (see alerting configuration above for details) when the traffic level was abnormal but is now normal again. This can occurr when:

• The observed metric is fluctuating and transitioning in and out of the calculated critical threshold.
• The traffic level has stabilized, and the system has determined a new baseline.

Alerts for other visibility drops (bandwidth, packet counts) would look similar. Alerts through other channels such as Webhook, API, or syslog (QUX only) would also contain similar language.

Additional Details

When you have received an alert, you may wish to gather additional information or check at the CLI what current status the checks are in. Please see SSH login process for CLI for details on how to login to the Brain CLI.

show system-health --verbose

The above command can be used to see the Sensor Bandwidth. Alternatively you can perform a GET against the /events/health endpoint using the API.

Example:

Client <serial-number of the brain>: Sensor Traffic

Sensor Traffic exceeds critical threshold for <sensor>

Potential traffic anomaly detected. Please investigate recent traffic pattern changes in your environment.

This alert is an indicator of change and does not by itself confirm an issue.

The system can also sends a recovery notification. The sentence Potential traffic anomaly cleared is the indicator that the alert is the recovery or clearing of the previous alert.

Client <serial-number of the brain>: Sensor Traffic

Sensor Traffic is normal for xx sensor(s); unable to determine status for <sensor>, <sensor>, ..., and xx more

Potential traffic anomaly cleared.

The recovery notification can also contain Unable to determine status for xxxx when there are unknown values. This usually means that there isn't enough traffic data to generate a baseline. This can happen if the traffic levels are very low, or if the sensor is newly deployed. Unable to determine status should not be considered a problem unless those sensors have been receiving good quantities for traffic for a long time (over a month).