Troubleshoot Linux Agent Performance Issues

Performance Issue(s) Information Gathering

This section is designed to help you gather information if you experience performance issues with Distributed Cloud AIP. If you gather this information at the time of the incident, then future troubleshooting steps that may put your production environment in a risky state will be reduced.

Gather information to answer the following questions:

  • How many hosts/workloads are affected by the issue?
  • What is the AWS instance type for the affected workload?
  • Is the Distributed Cloud AIP Agent installation on the affected systems/workloads new or existing?
  • Have there been any recent changes to the affected systems/workloads? (Examples: Kernel upgrade; a new version of Java for our workload; etc.)
  • Is your Distributed Cloud AIP Agent deployment scripted? If so, what script tools do you use? Have there been any recent changes to the script?
  • Does the affected environment(s) use the Distributed Cloud AIP Agent File Integrity Monitoring (FIM) or Container Monitoring features?
  • Are there any security tools besides the Distributed Cloud AIP Agent installed on the affected environment(s)?
  • On an unaffected host with a similar workload, what is the typical resource utilization?
  • How can Distributed Cloud AIP best replicate the workload on the affected environment(s) in our environments for further troubleshooting?

Host Reproduction Testing

This section is designed to help you reproduce and troubleshoot the reported performance issue without affecting your production environment. Perform these steps in the listed order in your development environment:

  1. Install the most recent Distributed Cloud AIP Agent version.
  2. Prior to enabling the Distributed Cloud AIP Agent, change Distributed Cloud AIP Agent logging to “debug”:
    sudo tsagent config --set log.level debug

    For more information, please see FAQ: Change level of logging on Agents.

  3. Install the Distributed Cloud AIP support tools, using the following commands:
    apt-get install threatstack-agent-support
    yum install threatstack-agent-support

    For more information, please see Diagnostic Tools and Support Logs.

  4. Start the Distributed Cloud AIP Agent and monitor its usage, using the following commands:
    sudo tsagent setup --deploy-key=<your deploy key> --ruleset="Base Rule Set"
    sudo systemctl start threatstack

    For more information, please see Deploy Distributed Cloud AIP Linux Agent 3.x Series.

  5. If the Distributed Cloud AIP Agent is found in distress, then run the support tools:
    cd /opt/threatstack-agent-support
    sudo ./diagnostics.sh

    For more information, please see Diagnostic Tools and Support Logs.

  6. From the Command Line, gather the following information from a host experiencing trouble:
    • sudo lsof | grep tsauditd | grep netlink
    • ss -a | grep audit
    • sudo auditctl -s 

      Confirm the failure flag is set to 0

    • sudo auditctl -l 

      Confirm this matches the Distributed Cloud AIP configuration. Also confirm that the uid and gid match Distributed Cloud AIP's uid and gid.

      • getent group threatstack
      • id -u threatstack
    • sudo systemctl | grep running
    • sudo systemctl list-unit-files | grep enabled
    • sudo systemctl | grep audit
    • ps -ef 
    • sudo strace -o ts-processes-strace.out -f -p <tsagent pid>

      where you replace <tsagent pid> with the Distributed Cloud AIP Agent pid.

    • Take a screenshot of the htop or top output of a host in trouble.
  7. Forward these captures and the gpg output of the diagnostics to Distributed Cloud AIP Support for review.
Was this article helpful?
0 out of 0 found this helpful