Estimating Log Volume

Guidance to estimate AWS log volume and expected ingestion impact for CDR for AWS.

You can view how much data Vectra will monitor by reviewing the daily volume of CloudTrail trail logs added to S3 via the AWS Cloudtrail management page. As long as you have permissions to view the S3 bucket which your CloudTrail logs are ingested to, its simple to determine the usage.

• In AWS, navigate to your CloudTrail management page.

• Select the trail which you want CDR for AWS to monitor.

  • Generally, we recommend creating a single organization trail which logs activity from all of your AWS accounts.

• You should see a view like this screenshot below. Take note of the “Trail Log Location” name.

Example organizational trail showing Trail Log Location

• Click on the Trail Log Location to be taken to S3.

• In the left hand Menu, click on “Dashboards” Under “Storage Lens”.

• You should see a “default-account-dashboard”.

Navigating to the S3 bucket data added per day data

• Open this Dashboard, and click on the “Bucket” tab.

• You should see a graph “Trend of Buckets”, showing activity for all of your buckets.

  • Drill down to your CloudTrail S3 bucket by deselecting any S3 buckets without the name you noted in Step 3.

• You may need to change “Top N Buckets” to show more buckets.

• Once you only have your CloudTrail S3 bucket selected, you should see a graph like this screenshot below:

Example graph showing S3 bucket data growth in last 30 days

• Note the difference in size between 2 points in time, and divide this difference by the number of days between these points to calculate your average daily ingestion.

Troubleshooting

The number is smaller than expected, I suspect we are using lifecycle management:

If your CloudTrail log S3 bucket deletes management events after a certain time period, then this graph may not reflect actual usage. In this case, you can measure total expected volume by checking the total storage volume in your S3 and measuring how long these logs are retained for, and calculating the average daily usage from that.

• Using the steps above, measure the total storage in your CloudTrail S3 bucket.

• Navigate to your CloudTrail S3 bucket.

• Select the “Management Tab”, and note the lifecycle rules for this S3 bucket.

• It may say something like “Expire objects after 90 days” or similar.

S3 bucket management tab showing lifecycle rules

• To get your daily log volume, divide the total storage in your S3 bucket by the number of days before expiry.

Example

• If the total storage in your CloudTrail S3 Bucket is 900GB.

• And you have a lifecycle management policy of 90 days.

• Then your CloudTrail logs would come to 10GB per day.

I cannot see my S3 bucket in the Top N buckets:

You may need to create a new Storage Lens Dashboard, to see CloudTrail management event activity.

• Navigate to “Storage Lens Dashboards”, and click “Create Dashboard”.

• Name your dashboard.

• In “Dashboard Scope”, untick “include all buckets”.

• Select the Cloudtrail S3 bucket you noted in step 3 above, click “Create Dashboard”.

• The dashboard will take some time to populate data, once this is done, follow the steps above.