Data Portability

Follow

Introduction

The data portability feature provides an efficient way to bulk export raw event data from the Threat Stack Cloud Security PlatformⓇ into your Amazon Web Service (AWS) Simple Storage Service (S3) bucket(s). From there, you can integrate Threat Stack data with your own tools, where you can streamline security operations without adding more screens to your workflows. You can also choose to store Threat Stack data in your own long-term storage locations to meet applicable compliance requirements.

Set Up Data Portability Integration

To set up the data portability integration between AWS and Threat Stack, perform the following steps:

  • Create an AWS IAM role specifically for the data portability integration
  • Use the API endpoint to set up the data portability integration
IAM Role

Prerequisites

  • Administrator access to the AWS Console
  • Access to the Threat Stack console

Set up IAM Role

  1. Log into the AWS Console as an administrator.
  2. Go to Services > Security, Identity, & Compliance.
  3. Select IAM. The Welcome to the Identity and Access Management page displays.
  4. Create a new policy.
    1. In the left navigation pane, click Policies. The Policy page displays.
    2. Click the Create policy button. The Create Policy page displays.
    3. Click the JSON tab. The JSON field displays.
    4. Copy and paste the following text into the JSON field:
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "s3:GetObject",
                      "s3:PutObject",
                      "s3:ListBucket",
                      "s3:DeleteObject"
                  ],
                  "Resource": [
                      "arn:aws:s3:::<Receive-Events-Bucket-Name>",
                      "arn:aws:s3:::<Receive-Events-Bucket-Name>/*"
             
                  ]
              }
          ]
      }           

      Notes:

      • Replace <Receive-Events-Buckets-Name> with the name of your S3 bucket that will receive the events from Threat Stack.
      • EMR is used to stream the data into the S3 bucket, which then renames the files. In order to rename, we need permissions to move those files (to rename), and then to delete old files.
    5. Click the Review Policy button. The Review policy page displays.
    6. In the Name field, type a name for the policy. Threat Stack suggests naming the policy something like “TSDataPortabilityPolicy.”
    7. Optionally, in the Description field, type a description of this IAM policy.
    8. Click the Create policy button. You return to the Policy page. A confirmation message on creation of the new IAM policy displays at the top of the page.
  5. Create a new IAM role.
    1. In the left navigation pane, click Roles. The Roles page displays.
    2. Click the Create role button. The Create role page displays.
    3. Click the Another AWS Account button. New fields display.
    4. In the Account ID field, type “896126563706”.
    5. Select the Require external ID (Best practice when a third party will assume this role) checkbox. An additional field displays.
    6. In the External ID field, type an identifier that is easy for you to track. You need this ID in later steps.
    7. Click the Next: Permissions button. The Attach permissions policies page displays.
    8. In the Filter policies field, type the name of the policy you created in step 4f and press ENTER.
    9. Select the checkbox next to the policy name.
    10. Click the Next: Tags button. The Add tags (optional) page displays.
    11. Optionally, add key / value tag pairs to the role.
    12. Click the Next: Review button. The Review page displays.
    13. In the Role name field, type a name for the IAM role. Threat Stack recommends the name be descriptive of the role, such as “TSDataPortabilityRole.”
    14. Optionally, in the Description field, type a description of this IAM role.
    15. Click the Create Role button. You return to the Roles page. A confirmation message on creation of the new IAM role displays at the top of the page.
  6. On the Roles page, in the Search field, type the name of the role you created in step 5o.
  7. Click the role name. The Summary page displays.
  8. In the Role ARN field, click the Copy icon.
  9. Continue to the next section to create the API endpoint and link it to this IAM role.
API Endpoint

Prerequisites

  • Access to your API integration
  • Access to the Threat Stack console
  • A data portability IAM role and the External ID associated with the role (created in the previous section)
  • Your AWS S3 bucket name and, if applicable, prefix (folder)
  • Your AWS S3 bucket region

Set up API Endpoint

Update S3 Export Enrollment API endpoint

Data Structure and Display

The Threat Stack Cloud Security PlatformⓇ (CSP) batches raw events before exporting them to your AWS S3 bucket.

Raw Event Batching

Your AWS S3 bucket receives a batched Threat Stack CSP event file every three to five minutes.

The Threat Stack CSP batches raw events when one of two situations occur:

  • Raw events batched more than 120 seconds ago.
  • 128 megabytes (MB) of raw event data entered the Threat Stack CSP since the last event batch occurred.

The Threat Stack CSP waits approximately 60 seconds to collect any batches ready for export. The Threat Stack CSP then processes the batches into individual newline-delimited gzipped JSON files. The Threat Stack CSP then exports the batch file to your AWS S3 bucket.

Folder Structure

The Threat Stack CSP uses a consistent folder structure to deliver the batch JSON file to your AWS S3 bucket. The folder structure is:

s3://<bucket-name>/<optional-prefix>/<orgId>/YYYY/MM/DD/<event-batch-type>/events-<N>-<M>.ndjson.gz

  • <bucket-name> – Replaced with the name of the S3 bucket in which to store Threat Stack CSP events.
  • <optional-prefix> – Optionally, replace with the top-level path within your AWS S3 bucket. If you do not have or do not want to use a top-level path, then Threat Stack CSP ignores this part of the folder structure.
  • <orgId> – Replaced with either your Threat Stack organization ID or the AWS organization ID of a consumer whose data you receive on their behalf
  • <YYYY/MM/DD> – Replaced with the year, month, and day at which the Threat Stack CSP ingested the batched JSON file. The datestamp is in UTC.
  • <event-batch-type> – Replaced with the event source type. Currently, the only value is agent-events.
  • events-<N>-<M> – Replaced with a Threat Stack CSP file name.

Raw Event Format

The raw event data passed from the Threat Stack CSP is structured in a specific way. See the Raw Event Format article for more information.

Frequently Asked Questions (FAQs)

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.