Troubleshooting

This guide is designed to provide Vectra Match customers with basic troubleshooting capabilities when operating the Vectra Match platform based upon use cases.

Some potential problems and solutions are below. If you are not able to resolve your issue or you want additional assistance, please feel free to contact Vectra's Customer Support team https://support.vectra.ai/s/login/.

Enabling Rules

Problem: I cannot enable Vectra Match in my deployment.

• Do you have a Vectra Match license? A Vectra Match license is required to enable Vectra Match. Tou can validate this by going to the Vectra CLI and issuing the show license command or looking in the GUI at Configuration → SETUP → Licensing.

• Is the Sensor you are attempting to enable paired and connected? Sensors must be paired and connected.

Uploading Rules

Problem: I cannot upload a rule file to the Vectra Brain

• Does your ruleset include any rules with File Store, Lua, or Datasets? These are not currently supported in Vectra Match

• Does the ruleset load into Suricata locally if you attempt to load it? You can always load Suricata on a workstation or endpoint in your environment with the suricata -vvT -S <ruleset-file> command. This will attempt to load your ruleset into a local instance of Suricata to test the configuration. If there are any issues with the ruleset, they will print out as ERRORs.

Not Seeing Expected Detections

Problem: I am running Vectra Match but I am not seeing detections I expect to see.

• Matches are output in a variety of ways:

  • RUX Deployments

    • Vectra Stream

    • Advanced Investigation

  • QUX Deployments

    • Vectra Stream

    • Vectra Recall

    • Syslog/Kafka

• If you aren't seeing the expected results downstream, you should validate:

  • Your downstream SIEM/SOAR/Log receiver configuration is properly setup, and Vectra Match has been enabled in the Notification settings when using syslog/kafka or Vectra Stream is properly configured.

  • That Vectra Match has been enabled on your sensor.

  • That your sensor has been assigned a ruleset.

  • Check the health of your sensors with the Stats API (include example.)

• Are you sure that your traffic is able to match the signatures that you are trying to match?

  • You can always replay packet captures through Suricata with the ruleset of your choice to validate that there are alerts. For instance, you can run the following on a local system to test your rule file + a PCAP: suricata -r <pcap-file> -S <signature-file>. The output will show how many alerts were detected, and you can review the fast.log or eve.json for additional information about what detections triggered. This will help to validate whether there are detections even in a local instance or if there is potentially something wrong with your deployment.

• Is your Vectra Match instance generating too many alerts triggering alert rate limiting?

  • Check the health stats of the sensor, if rate limiting is occurring you will see a message identifying it.

  • Rate limiting occurs when too many messages are sent within a single second, and the remain messages above the threshold are exempted to protect the platform. This is evaluated on a second by second basis.

  • When rate limiting occurs, you have a few options on how to address it.

    • First, you should determine which signatures are causing excessive rate limiting.

    • This can often be observed in your SIEM as these signatures will result in a large number of detections.

  • Once the signatures have been identified, you have a few options.

    1. If the signature is not relevant, you can disable it.

    2. If the signature is relevant, can it be tuned to avoid noise.

    3. You can leverage the Limit and Threshold features within signatures: https://suricata.readthedocs.io/en/suricata-6.0.10/rules/thresholding.html?highlight=threshold#threshold.

  • Make sure that there isn't an alert loop. Vectra has observed that certain detections like ET SID's 2031296 and 2031297 can alert on TCP based Syslog which can trigger an alert loop.

    • You can set a Suricata pass rule to exempt any traffic from the Vectra Brain IP: https://suricata.readthedocs.io/en/suricata-6.0.0/performance/ignoring-traffic.html?highlight=pass#pass-rules to avoid loops

    • You can exempt those specific signatures that are causing the alert storm, but this is less desirable.

  • For RUX customers, SQL Search can be used to see which signatures are firing most often:

SELECT alert.signature AS signature_name, alert.signature_id, COUNT(*) AS signature_count
FROM network.match
WHERE timestamp > date_add('hour', -24, now())
GROUP BY alert.signature, alert.signature_id
ORDER BY signature_count DESC

• Both QUX and RUX customers can use the alert-stats Match API endpoint to collect the same information.

  • RUX

    • Use a GET to /api/v3.4/vectra-match/alert-stats?device_serial={{device_serial}}

  • QUX

    • Use a GET to /api/v2.5/vectra-match/alert-stats?device_serial={{device_serial}}

  • Example output below (as you can see, the 1st result shows over 9.5 million hits ad is a good candidate for tuning):

{
    "alert_stats": [
        {
            "top_alert_counts": [
                {
                    "count": 9,592,816,
                    "signature": "ET EXPLOIT Apache log4j RCE Attempt (udp ldap) (CVE-2021-44228)",
                    "signature_id": 2034651
                },
                {
                    "count": 42,
                    "signature": "ET EXPLOIT Apache log4j RCE Attempt - lower/upper UDP Bypass M2 (CVE-2021-44228)",
                    "signature_id": 2034701
                },
                {
                    "count": 25,
                    "signature": "ET EXPLOIT Apache log4j RCE Attempt (udp ldap) (Outbound) (CVE-2021-44228)",
                    "signature_id": 2034761
                },
                {
                    "count": 17,
                    "signature": "ET EXPLOIT Apache log4j RCE Attempt - lower/upper UDP Bypass M2 (Outbound) (CVE-2021-44228)",
                    "signature_id": 2034799
                }
            ],
            "device_serial": "U22020000001284",
            "eve_log_rotated_time": ""
        }
    ]
}

• Are your rules making use of HOME_NET, EXTERNAL_NET, PROXY_IPs, EXCLUDE_NET variables?

  • Please see the Vectra Match FAQ for details on how these variables are set and the impact they will have on rules.

  • Check the following headings for details once on the above FAQ:

    • How does Vectra populate the HOME_NET variable?

    • Does Vectra support setting other variables besides HOME_NET?

Ruleset Compilation Error

Problem: I am seeing an error returned in Vectra Match as part of the ruleset compilation process, what should I do?

• Have you confirmed that you're not using an unsupported feature?

• Have you tried to test the ruleset file on a local Suricata instance to see if it works using the suricata -vvT -S <ruleset-file> on a local system?

• You may have invalid Variables in your Suricata rules. Today only HOME_NET, EXTERNAL_NET, PROXY_IPs, and EXCLUDE_NET are supported.

Recall Custom Model Unpected Result

Problem: I'm running Vectra Match in Recall (QUX deployments only) and I'm not seeing all of the expected Custom Models show up in Vectra NDR

• First validate that you see entries return with your saved search. If there are no entries, then you need to adjust the saved search.

• If you see the entries in the saved search but not in Vectra NDR, check that the Custom Model is enabled with the appropriate configuration in the Configuration → COVERAGE → Custom Models page. If it is, you may be hitting rate limits if you have a broad Custom Model and a non-tuned Vectra Match ruleset. Please contact support https://support.vectra.ai/.