Profile the Linux Agent

Overview

If you notice the Agent over-utilizing system resources, F5 Distributed Cloud App Infrastructure Protection (AIP) Support may ask for a memory or CPU profile of the Agent. The profile provides useful insight into how much memory or CPU the Agent is using and how it allocates those resources.

CPU

For Linux Host Agent 2.2.2 and above:

  1. In the Command Line, type the following command and press ENTER:
    sudo tsagent config --set pprof.enabled true
  2. Type the following command and press ENTER:
    sudo systemctl restart threatstack
  3. Type the following command and press ENTER:
    sudo tsagent pprof profile --seconds=300 > cpu.5m.pprof

Tip

Edit the --seconds= value and title (cpu.1h.pprof) for the pprof file to fit your needs. For example:

sudo tsagent pprof profile --seconds=120 cpu.2m.pprof.

You will now see a pprof file in the directory you ran the command in. Please gather the file and send it to Distributed Cloud AIP Support for review.

For Containerized Agent 2.2.2c and above:

This relies on your deployment method (Helm chart, DaemonSet, Puppet etc.). For this guide, we will use a Kubernetes cluster and our DaemonSet as an example.

  1. Locate line 138 in the DaemonSet yaml file:
    value: "enable_kubes 1 log.level info"
  2. Edit the line to include the Agent configuration to enable profiling:
    value: "enable_kubes 1 log.level info pprof.enabled true"
  3. Redeploy the Agent in your cluster (edit the .yaml file name accordingly):
    kubectl apply -f TSKubernetesDaemonSet.yaml
  4. In the Command Line, type the following command and press ENTER to hop into the Agent pod and generate the CPU profile:
    kubectl exec <agent pod name> -- tsagent pprof profile --seconds=300 > cpu.5m.pprof
  5. Gather the pprof file and send it to the Distributed Cloud AIP Support team for review.

Memory

The process for collecting a heap file is the same for CPU. Follow the steps above, but replace the pprof command:

sudo tsagent pprof heap > mem.pprof

The memory profile is a snapshot of the Agent's memory consumption at the moment of running the command. It does not support run for "x" amount of time before creating the profile.

Goroutine

The process for collecting a goroutine dump is the same for CPU. Follow the steps above, but replace the pprof command:

sudo tsagent pprof goroutine > goroutine.pprof

You can also gather a full stack trace by adding the --full flag:

sudo tsagent pprof goroutine --full > goroutine_full.txt

A goroutine trace is helpful in situations where we think the Agent is leaking threads.

Was this article helpful?
0 out of 0 found this helpful