Troubleshooting: Linux Agent 1.x and 2.x Performance Issues

Agent 1.x Series

Performance Issue(s) Information Gathering

This section is designed to help you gather information if you experience performance issues with the Threat Stack Cloud Security Platform® (CSP). If you gather this information at the time of the incident, then future troubleshooting steps that may put your production environment in a risky state will be reduced.

Gather information to answer the following questions:

  • How many hosts/workloads are affected by the issue?
  • What is the AWS instance type for the affected workload?
  • Is the Threat Stack Agent installation on the affected systems/workloads new or existing?
  • Have there been any recent changes to the affected systems/workloads? (Examples: Kernel upgrade; a new version of Java for our workload; etc.)
  • Is your Threat Stack Agent deployment scripted? If so, what script tools do you use? Have there been any recent changes to the script?
  • Does the affected environment(s) use the Threat Stack Agent File Integrity Monitoring (FIM) or Container Monitoring features?
  • Are there any security tools besides the Threat Stack Agent installed on the affected environment(s)?
  • On an unaffected host with a similar workload, what is the typical resource utilization?
  • How can Threat Stack best replicate the workload on the affected environment(s) in our environments for further troubleshooting?

Host Reproduction Testing

This section is designed to help you reproduce and troubleshoot the reported performance issue without affecting your production environment. Perform these steps in the listed order in your development environment:

  1. Install the most recent Threat Stack Agent version.
  2. Prior to enabling the Threat Stack Agent, change Threat Stack Agent logging to “debug”:
    sudo cloudsight config log_level=debug

    For more information, please see FAQ: Change level of logging on Agents.

  3. Install the Threat Stack support tools, using the following command:
    apt-get install threatstack-agent-support

    For more information, please see Diagnostic Tools and Support Logs.

  4. Start the Threat Stack Agent and monitor its usage, using the following command:
    sudo cloudsight setup name=value

    For more information, please see Deploy Threat Stack Linux Agent 1.x Series.

  5. If the Threat Stack Agent is found in distress, then run the support tools:
    cd /opt/threatstack-agent-support
    sudo ./diagnostics.sh

    For more information, please see Diagnostic Tools and Support Logs.

  6. From the Command Line, gather the following information:
    • auditctl -s
    • auditctl -l
    • htop or top output of a host in trouble.
  7. Forward these captures and the gpg output of the diagnostics to Threat Stack Support for review.
  8. Disable FIM tracking and restart the Threat Stack Agent (This disables our service which adds inotify and fanotify watches on files which are configured to be monitored. We expect this to reduce resource load, especially during times when files are accessed or changed).
    • sudo cloudsight config disable_fim=1
    • sudo cloudsight restart
Agent 2.x Series

Performance Issue(s) Information Gathering

This section is designed to help you gather information if you experience performance issues with the Threat Stack Cloud Security Platform® (CSP). If you gather this information at the time of the incident, then future troubleshooting steps that may put your production environment in a risky state will be reduced.

Gather information to answer the following questions:

  • How many hosts/workloads are affected by the issue?
  • What is the AWS instance type for the affected workload?
  • Is the Threat Stack Agent installation on the affected systems/workloads new or existing?
  • Have there been any recent changes to the affected systems/workloads? (Examples: Kernel upgrade; a new version of Java for our workload; etc.)
  • Is your Threat Stack Agent deployment scripted? If so, what script tools do you use? Have there been any recent changes to the script?
  • Does the affected environment(s) use the Threat Stack Agent File Integrity Monitoring (FIM) or Container Monitoring features?
  • Are there any security tools besides the Threat Stack Agent installed on the affected environment(s)?
  • On an unaffected host with a similar workload, what is the typical resource utilization?
  • How can Threat Stack best replicate the workload on the affected environment(s) in our environments for further troubleshooting?

Host Reproduction Testing

This section is designed to help you reproduce and troubleshoot the reported performance issue without affecting your production environment. Perform these steps in the listed order in your development environment:

  1. Install the most recent Threat Stack Agent version.
  2. Prior to enabling the Threat Stack Agent, change Threat Stack Agent logging to “debug”:
    sudo tsagent config --set log.level debug

    For more information, please see FAQ: Change level of logging on Agents.

  3. Install the Threat Stack support tools, using the following commands:
    apt-get install threatstack-agent-support
    yum install threatstack-agent-support

    For more information, please see Diagnostic Tools and Support Logs.

  4. Start the Threat Stack Agent and monitor its usage, using the following commands:
    sudo tsagent setup --deploy-key=<your deploy key> --ruleset="Base Rule Set"
    sudo systemctl start threatstack

    For more information, please see Deploy Threat Stack Linux Agent 2.x Series.

  5. If the Threat Stack Agent is found in distress, then run the support tools:
    cd /opt/threatstack-agent-support
    sudo ./diagnostics.sh

    For more information, please see Diagnostic Tools and Support Logs.

  6. From the Command Line, gather the following information from a host experiencing trouble:
    • sudo lsof | grep tsauditd | grep netlink
    • ss -a | grep audit
    • sudo auditctl -s 

      Confirm the failure flag is set to 0

    • sudo auditctl -l 

      Confirm this matches the Threat Stack configuration. Also confirm that the uid and gid match Threat Stack's uid and gid.

      • getent group threatstack
      • id -u threatstack
    • sudo systemctl | grep running
    • sudo systemctl list-unit-files | grep enabled
    • sudo systemctl | grep audit
    • ps -ef 
    • sudo strace -o ts-processes-strace.out -f -p <tsagent pid>

      where you replace <tsagent pid> with the Threat Stack Agent pid.

    • Take a screenshot of the htop or top output of a host in trouble.
  7. Forward these captures and the gpg output of the diagnostics to Threat Stack Support for review.

Related Articles

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request