Splunk TA - Changing from CEF to JSON for Vectra Detect (QUX)
Migrate Splunk ingestion for Vectra Detect from legacy CEF syslog to full JSON using the new TA, with install, configuration, and validation steps.
Overview
A Splunk Technology Add-on (TA) is available which uses JSON format instead of the CEF format (which was used by the original TA from Vectra). Please refer to the Splunk Integration Guide for Vectra AI for an overview of Vectra's Splunk Integrations, which TA's and Apps to use, and installation instructions. The remainder of this article is only for customers who are moving from the older CEF TA to the new JSON TA.
Why a new TA?
In releases 5.9 and 6.0, we introduced a new feature which allowed users to include additional information in a syslog event. It came first for Account and Host Scoring events (in 5.9), and then for Account and Host Detections (in 6.0). The goal of this feature is to provide additional detailed information in the syslog events to the SIEM and limit the need for a SOC Analyst to pivot to Detect. This is configurable in the Detect UI per destination. When using CEF format, all this additional information included in the syslog events are stored in a single attribute in JSON format. Parsing syslog events in different formats (CEF and JSON) presents some challenges and the solution can be cumbersome. For this reason, we decided to create a new TA which parses syslog events sent by Detect in full JSON format. This is something already supported by Detect (CEF, JSON and Standard are the 3 format supported today).
Installation
Remove the legacy TA that used CEF format - If applicable
To allow the Vectra Cognito Detect App to be compatible with both TAs (CEF and JSON), we used the same Sourcetypes inside the index (after transformation during input) in both Add-ons. For that reason, you need to remove the previous TA before installing the new one.
Follow those instructions to remove the add-on: https://docs.splunk.com/Documentation/Splunk/8.0.6/Admin/Managingappobjects .
You need to remove this Add-on everywhere it has been installed: Heavy Forwarders, Indexers and Search Heads (it should not have been installed on Universal Forwarders)
If you want to keep Detect data (in CEF), do not delete the index or data inside. You can decide to send the Detect syslog events in JSON format into the same index or create a new one. It does not really matter, you just need to be aware that the app (Vectra Detect or any other apps like Search & Reporting ) would only be able to parse the data properly in JSON format. The data in CEF format would be accessible in RAW format.
Where to install the new TA
Roles
TA Required
Search Head
Yes
Indexer
Yes
Heavy Forwarder
Yes
Universal Forwarder
No
Install the "Technology Add-on for Vectra Detect (JSON)"
Download the app from Splunkbase and then navigate in Splunk to Apps > Manage Apps:
Click on "Install app from file":
Upload the Add-on previously downloaded and click Upload. The new Add-on should be visible:
Once it is installed, edit the Permissions:
Select "All apps" then Save.
Select "All apps" then Save.
This allow any apps to read objects defined in this Add-on!
Configuration
Create a new index - If applicable
If you are moving from CEF to JSON you can use the same index if you desire. If you are performing a new install, you will need to create a new index. To create a new dedicated index, follow the steps as described below:
Go to Settings >Indexes:
Then click on New Index on the top right corner:
Give it a name which is easy to remember and recognize. Set the App to Vectra Cognito.
Click Save
Create a new Data Input
Most production deployments would have a separate syslog server like syslog-ng or rsyslog to send the syslog events to. This guide is not going to cover what the configuration should be in that case. Here, we are creating a new TCP inputs directly in Splunk.
Go to Settings > Data Inputs:
Click on Add new for TCP type input:
Configure a Port number (in this example, it is 5141/TCP) and click Next:
In** Source type**, search for vectra keyword in the dropdown list:
If the Add-on has been properly installed, you should see in the list the sourcetype: vectra:cognito:json. Select it! In the case you don't see it, it means that the Add-on has not been installed successfully. Select Vectra Cognito for the** App Context** and the newly created index (detect_json in this case).
Click Next the Submit if everything looks good:
Configure Detect Syslog
In the Detect UI, go to Settings > Notification. At the bottom of the page, you have the Syslog configuration, click the Edit button:
Configure the** IP address of your syslog server**, the Port and the Protocol. Select JSON for the format. Then you can choose which log types you want to receive. For most cases, just select all of them! Click Save when you are done.
There is additional configuration you can do in the second step to fine tune some other aspects. Edit again the syslog configuration. On the right side, you can see a couple of switch buttons (all off by default):
Include triaged Detections: When turned off, syslog messages will not be sent when triaged detections are created or updated.
Include detections in Info category: When turned off, syslog messages will not be sent when detections in the info category are created or updated.
Include host/account score decreases: When turned off, syslog messages will not be sent when threat and certainty scores are both decreasing and/or remain the same. This applies to both hosts and accounts.
Change the configuration of those 3 switch buttons based on your preferences.
The last piece of configuration is the checkbox for Enhanced Details. When it is on, event logs will include additional host, account, and detection attributes. This will benefit users looking for more detail in syslog, such as those that utilize a SIEM as their primary dashboard.
Click Save
Validation
The First step would be to validate that data is received by Splunk and is indexed. The easiest way to do that is to open the **Search & Reporting App **and filter on the index name:
Below you can see the data is received and is in JSON format:
The Second step would be to validate that the data is parsed successfully by the Vectra Detect Add-on. To validate that it is working as expected at the** indexer level**, look at the different Source Type:
If you see multiple Sourcetypes, it means that the Add-on in the indexer is doing the transformation as expected. Then, to validate it is working as expected at the Search Head level, expand one syslog event and look the list of fields:
Certain fields are added by the Add-on, like:
vectra_url
vectra_timestamp
src
tags
etc.
Fields are not the same for all source types but look at couple of different type of events (a scoring or detection event are good candidates) and validate that you can see those. If that is the case, the Add-on is working as expected!
After couple of hours (or the next day), open the Vectra Cognito App and check if the dashboard are populated.
**** Make sure the macro uses by the Vectra Detect app named vectra_cognito_index is pointing to the right index! ****
FAQ
Do I need a new version of Detect App to use the TA in JSON?
No. The Detect App is compatible with both Add-ons. It does not matter if you ingest syslog in CEF or JSON format as long as you have the right Add-on installed.
Why are syslog headers are not visible when I look at the RAW syslog event?
Splunk can parse all the attributes in a JSON document automatically but it needs to be exclusively in JSON. Syslog headers are not in JSON, only the message is. Actually, it does not matter which format we are using for the message (CEF or JSON or standard), the syslog header structure would be exactly the same and include:
Priority
Timestamp
Hostname
application
Example below where the syslog header is: <13>Oct 1 22:23:07 A21000000000354 vectra_json_v2 -:
In our case, syslog header information are redundant with the information stored in the syslog message. In order to allow Splunk to have only JSON data, the add-on is stripping the syslog header and what you would see in Splunk would be only:
Last updated
Was this helpful?