Troubleshooting: Agent 1.x Series Performance Issues

Follow

Performance Issue(s) Information Gathering

This section is designed to help you gather information if you experience performance issues with the Threat Stack Cloud Security Platform® (CSP). If you gather this information at the time of the incident, then future troubleshooting steps that may put your production environment in a risky state will be reduced.

Gather information to answer the following questions:

  • How many hosts/workloads are affected by the issue?
  • What is the AWS instance type for the affected workload?
  • Is the Threat Stack Agent installation on the affected systems/workloads new or existing?
  • Have there been any recent changes to the affected systems/workloads? (Examples: Kernel upgrade; a new version of Java for our workload; etc.)
  • Is your Threat Stack Agent deployment scripted? If so, what script tools do you use? Have there been any recent changes to the script?
  • Does the affected environment(s) use the Threat Stack Agent File Integrity Monitoring (FIM) or Container Monitoring features?
  • Are there any security tools besides the Threat Stack Agent installed on the affected environment(s)?
  • On an unaffected host with a similar workload, what is the typical resource utilization?
  • How can Threat Stack best replicate the workload on the affected environment(s) in our environments for further troubleshooting?

Host Reproduction Testing

This section is designed to help you reproduce and troubleshoot the reported performance issue without affecting your production environment. Perform these steps in the listed order in your development environment:

  1. Install the most recent Threat Stack Agent version.

    If your package manager or similar does not handle install:

    https://threatstack.zendesk.com/hc/en-us/articles/360015273172-Deploy-the-Threat-Stack-Agent

  2. Prior to enabling the Threat Stack Agent change Threat Stack Agent logging to “debug.”
    sudo cloudsight config log_level=debug

    https://threatstack.zendesk.com/hc/en-us/articles/360002685831-FAQ-Change-level-of-logging-on-Agents

  3. Install the Threat Stack support tools.
    apt-get install threatstack-agent-support

    https://threatstack.zendesk.com/hc/en-us/articles/208906046-Diagnostic-Tools-and-Support-Logs

  4. Start the Threat Stack Agent and monitor its usage.
    sudo cloudsight setup …

    https://threatstack.zendesk.com/hc/en-us/articles/360015273172-Deploy-the-Threat-Stack-Agent

  5. If the Threat Stack Agent is found in distress, then run the support tools.
    cd /opt/threatstack-agent-support
    sudo ./diagnostics.sh

    https://threatstack.zendesk.com/hc/en-us/articles/208906046-Diagnostic-Tools-and-Support-Logs

  6. From the Command Line gather the following information:
    • auditctl -s
    • auditctl -l
    • htop or top output of a host in trouble.
  7. Forward these captures and the gpg output of the diagnostics to Threat Stack Support for review.
  8. Disable FIM tracking and restart the Threat Stack Agent. (This disables our service which adds inotify and fanotify watches on files which are configured to be monitored. We expect this to reduce resource load, especially during times when files are accessed or changed)
    • sudo cloudsight config disable_fim=1
    • sudo cloudsight restart
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.