Trex gui is consuming tons of cpu and memory after some time

[phschoen]

I usually leave the gui running and after some time (5h+) the gui gets unusable and slows down the whole system. here is a ps dump after 2 days letting it running without any traffic generation:

USER      PID    %CPU %MEM    VSZ      RSS     TTY        STAT START   TIME    COMMAND
phschoen  18763  93.3 3.6     12493080 1139896 pts/42 SI+ Nov 16       3758:10 java -jar trex-stateless-gui.jar

[phschoen]

was also reproduceable in 3.4

[EgorBlagov]

Hi,

is the GUI connected to trex during all this period? Or there is only GUI window open and nothing else?

[phschoen]

gui is opened and connected to a remote server which is not sending something. however some ports are reserved (2 out of 8). I will try if its also happening if no server is connected.

[phschoen]

actually its even growing just by opening the trex gui and leave it running.
Also each click in a drop down menu is increasing the memory e.g. opening and closing the help dropdown is increasing the memory by 40k each time

[EgorBlagov]

@phschoen, thanks for the update
I'll take a look into it.

[phschoen]

any updates?

[EgorBlagov]

Hi,

regarding menubar memory leak, I've checked on very lightweight jfx app on Java 8 (which we are using), and the same leak is reproduced, I've just selected dropdown and hold right arrow, and memory started to grow. So this issue is related to jfx itself, maybe in future if we will migrate to newer java and javafx it will be resolved (if newer javafx doesn't have this issue).

regrading main memory leak seems that I've found an error (looks like it related to logs, because once I've disabled logging for some classes who spam to logs and connected to trex memory is stalled around 257 MiB.

Please let me know your platforms (linux/windows/mac) and I'll prepare a build for your so you can check on your scenario.

Thanks,
Egor.

[EgorBlagov]

HI, @phschoen

Please check latest v4.5.5. Once I've validated it was around 305-307 maximal after 30-40 minutes.

[phschoen]

hi,
i tested the v4.5.5 on Ubuntu Linux and used java 8.211:

$java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

memory seems fine but cpu usage still goes over the roof and slows the pc down after some time (1 core with ~95% use no mater what).

[EgorBlagov]

Please, let me know latest scenario, how did you test it, and how long? ср, 15 мая 2019 г., 23:50 Philipp Schönberger <[email protected]>:

…

hi, i tested the v4.5.5 on Ubuntu Linux and used java 8.211: $java -version java version "1.8.0_211" Java(TM) SE Runtime Environment (build 1.8.0_211-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode) memory seems fine but cpu usage still goes over the roof and slows the pc down after some time (1 core with ~95% use no mater what). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#101?email_source=notifications&email_token=AINDXDI57TTXJWP7LKX3KITPVQ5MZA5CNFSM4GFDRAR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPIQCQ#issuecomment-492734474>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AINDXDMO2ESB44IC464NQQDPVQ5MZANCNFSM4GFDRARQ> .

[phschoen]

i created a small script to plot it over time and gather some infos
i started trex and created two simple ip/udp traffic with minimal pps and started it and let this running for some time.

trafic.yml

---
- name: "ip/udp-1"
  
  stream:
    action_count: 0
    enabled: true
    flags: 2
    flow_stats:
      enabled: true
      rule_type: "latency"
      stream_id: 2
    isg: 0.0
    mode:
      rate:
        type: "pps"
        value: 1.0
      type: "continuous"
      total_pkts: 2
      pkts_per_burst: 0
      ibg: 0.0
      count: 1
    
    packet:
      binary: "AAAAAAAAAAAAAAAAgQAAZAgARQAATgTSAAB/Ece3CgEtCgoBLgoAZABkADIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
      meta: "eyJwcm90b2NvbF9zZWxlY3Rpb24iOnsiaXNfdGFnZ2VkX3ZsYW5fc2VsZWN0ZWQiOnRydWUsImlzX2lwdjRfc2VsZWN0ZWQiOnRydWUsImlzX3RjcF9zZWxlY3RlZCI6ZmFsc2UsImlzX3VkcF9zZWxlY3RlZCI6dHJ1ZSwiaXNfcGF0dGVybl9zZWxlY3RlZCI6ZmFsc2UsImZyYW1lX2xlbmd0aF90eXBlIjoiRml4ZWQiLCJmcmFtZV9sZW5ndGgiOiIxMDAiLCJtYXhfbGVuZ3RoIjoiMTUxOCIsIm1pbl9sZW5ndGgiOiI2NCJ9LCJldGhlcm5ldCI6eyJ0eXBlIjoiMDgwMCIsImlzX292ZXJyaWRlIjpmYWxzZX0sIm1hYyI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjAwOjAwOjAwOjAwOjAwOjAwIiwiY291bnQiOiIxNiIsIm1vZGUiOiJUUmV4IENvbmZpZyIsInN0ZXAiOiIxIn0sImRlc3RpbmF0aW9uIjp7ImFkZHJlc3MiOiIwMDowMDowMDowMDowMDowMCIsImNvdW50IjoiMTYiLCJtb2RlIjoiRml4ZWQiLCJzdGVwIjoiMSJ9fSwiaXB2NCI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjEwLjEuNDUuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifSwiZGVzdGluYXRpb24iOnsiYWRkcmVzcyI6IjEwLjEuNDYuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifX0sInRjcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6ZmFsc2UsImNoZWNrc3VtIjoiQjNFMyIsImRzdF9wb3J0IjoiMTAyNSIsInNyY19wb3J0IjoiMTAyNSIsImlzX292ZXJyaWRlX3NyY19wb3J0IjpmYWxzZSwiYWNrX251bWJlciI6IjAiLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsInVyZ2VudF9wb2ludGVyIjoiMCIsImlzX3VyZyI6ZmFsc2UsImlzX2FjayI6ZmFsc2UsImlzX3BzaCI6ZmFsc2UsImlzX3JzdCI6ZmFsc2UsImlzX3N5bmMiOmZhbHNlLCJpc19maW4iOmZhbHNlLCJzZXF1ZW5jZV9udW1iZXIiOiIxMjkwMTgiLCJ3aW5kb3ciOiIxMDI0In0sInVkcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6dHJ1ZSwiY2hlY2tzdW0iOiJGRkJBIiwiZHN0X3BvcnQiOiIxMDAiLCJzcmNfcG9ydCI6IjEwMCIsImlzX292ZXJyaWRlX2xlbmd0aCI6ZmFsc2UsImlzX292ZXJyaWRlX3NyY19wb3J0Ijp0cnVlLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsImxlbmd0aCI6IjI2In0sInBheWxvYWQiOnsidHlwZSI6IkZpeGVkIFdvcmQiLCJwYXR0ZXJuIjoiMDAifSwidmxhbiI6eyJjZmkiOiIwIiwiaXNfb3ZlcnJpZGVfdHBfaWQiOmZhbHNlLCJ0cF9pZCI6IkZGRkYiLCJ2X2lkIjoiMTAwIiwicHJpb3JpdHkiOiIwIn0sImFkdmFuc2VkX3Byb3BlcnRpZXMiOnsiY2FjaGVfdmFsdWUiOiI1MDAwIiwiY2FjaGVfc2l6ZV90eXBlIjoiRW5hYmxlIn0sIm1vZGVsX3ZlcnNpb24iOiIxLjAifQ=="
      model: ""
    self_start: true
    advanced_mode: false
    vm:
      split_by_var: ""
      instructions: []
      cache_size: 5000
  stream_id: 0
- name: "ip/udp-2"
  
  stream:
    action_count: 0
    enabled: true
    flags: 2
    flow_stats:
      enabled: true
      rule_type: "latency"
      stream_id: 3
    isg: 0.0
    mode:
      rate:
        type: "pps"
        value: 1.0
      type: "continuous"
      total_pkts: 2
      pkts_per_burst: 0
      ibg: 0.0
      count: 1
    
    packet:
      binary: "AAAAAAAAAAAAAAAAgQAAZAgARQAATgTSAAB/Ece3CgEtCgoBLgoAyADIADIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
      meta: "eyJwcm90b2NvbF9zZWxlY3Rpb24iOnsiaXNfdGFnZ2VkX3ZsYW5fc2VsZWN0ZWQiOnRydWUsImlzX2lwdjRfc2VsZWN0ZWQiOnRydWUsImlzX3RjcF9zZWxlY3RlZCI6ZmFsc2UsImlzX3VkcF9zZWxlY3RlZCI6dHJ1ZSwiaXNfcGF0dGVybl9zZWxlY3RlZCI6ZmFsc2UsImZyYW1lX2xlbmd0aF90eXBlIjoiRml4ZWQiLCJmcmFtZV9sZW5ndGgiOiIxMDAiLCJtYXhfbGVuZ3RoIjoiMTUxOCIsIm1pbl9sZW5ndGgiOiI2NCJ9LCJldGhlcm5ldCI6eyJ0eXBlIjoiMDgwMCIsImlzX292ZXJyaWRlIjpmYWxzZX0sIm1hYyI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjAwOjAwOjAwOjAwOjAwOjAwIiwiY291bnQiOiIxNiIsIm1vZGUiOiJUUmV4IENvbmZpZyIsInN0ZXAiOiIxIn0sImRlc3RpbmF0aW9uIjp7ImFkZHJlc3MiOiIwMDowMDowMDowMDowMDowMCIsImNvdW50IjoiMTYiLCJtb2RlIjoiRml4ZWQiLCJzdGVwIjoiMSJ9fSwiaXB2NCI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjEwLjEuNDUuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifSwiZGVzdGluYXRpb24iOnsiYWRkcmVzcyI6IjEwLjEuNDYuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifX0sInRjcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6ZmFsc2UsImNoZWNrc3VtIjoiQjNFMyIsImRzdF9wb3J0IjoiMTAyNSIsInNyY19wb3J0IjoiMTAyNSIsImlzX292ZXJyaWRlX3NyY19wb3J0IjpmYWxzZSwiYWNrX251bWJlciI6IjAiLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsInVyZ2VudF9wb2ludGVyIjoiMCIsImlzX3VyZyI6ZmFsc2UsImlzX2FjayI6ZmFsc2UsImlzX3BzaCI6ZmFsc2UsImlzX3JzdCI6ZmFsc2UsImlzX3N5bmMiOmZhbHNlLCJpc19maW4iOmZhbHNlLCJzZXF1ZW5jZV9udW1iZXIiOiIxMjkwMTgiLCJ3aW5kb3ciOiIxMDI0In0sInVkcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6dHJ1ZSwiY2hlY2tzdW0iOiJGRkJBIiwiZHN0X3BvcnQiOiIyMDAiLCJzcmNfcG9ydCI6IjIwMCIsImlzX292ZXJyaWRlX2xlbmd0aCI6ZmFsc2UsImlzX292ZXJyaWRlX3NyY19wb3J0Ijp0cnVlLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsImxlbmd0aCI6IjI2In0sInBheWxvYWQiOnsidHlwZSI6IkZpeGVkIFdvcmQiLCJwYXR0ZXJuIjoiMDAifSwidmxhbiI6eyJjZmkiOiIwIiwiaXNfb3ZlcnJpZGVfdHBfaWQiOmZhbHNlLCJ0cF9pZCI6IkZGRkYiLCJ2X2lkIjoiMTAwIiwicHJpb3JpdHkiOiIwIn0sImFkdmFuc2VkX3Byb3BlcnRpZXMiOnsiY2FjaGVfdmFsdWUiOiI1MDAwIiwiY2FjaGVfc2l6ZV90eXBlIjoiRW5hYmxlIn0sIm1vZGVsX3ZlcnNpb24iOiIxLjAifQ=="
      model: ""
    self_start: true
    advanced_mode: false
    vm:
      split_by_var: ""
      instructions: []
      cache_size: 5000
  stream_id: 2

start.sh output

###############
CPU:
model name      : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz

###############
Mem:
             total       used       free     shared    buffers     cached
Mem:         30918      11050      19868        289        106       2243
-/+ buffers/cache:       8699      22218
Swap:        32518        658      31860

###############
java:
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

###############
trex version
version=v4.5.5-0-gb80ac59

###############
start trex:

###############
pid is 18080

start.sh to get cpu and mem

#!/bin/bash

echo "###############"
echo "CPU:"
cat /proc/cpuinfo  | grep "model name"| uniq
echo ""

echo "###############"
echo "Mem:"
free -m
echo ""

echo "###############"
echo "java:"
java -version
echo ""

echo "###############"
echo "trex version"
unzip -p trex-stateless-gui.jar version.properties
echo ""
echo ""

echo "###############"
echo "start trex:"
java -jar trex-stateless-gui.jar > trex.log &
pid=$!
trap "echo '' kill $pid ! ; kill $pid ; exit -1; ./plot.sh" SIGINT SIGTERM
START_TIME=$SECONDS

echo ""
echo "###############"
echo "pid is $pid"
rm trex_stats.dat
touch trex_stats.dat

while true
do

    ELAPSED_TIME=$(($SECONDS - $START_TIME))
    echo -n "$ELAPSED_TIME  ">>trex_stats.dat
    ps -o pid,pcpu,rss,pmem -p $pid | tail -n1 >> trex_stats.dat

    sleep 10
    if [[ $? -ne 0 ]]
    then
        break
    fi
done

plot.sh to get graph

#!/bin/bash 
gnuplot << eor
set title 'trex cpu/mem usage'
set ylabel 'cpu %'
set y2label 'mem kilobytes'
set xlabel 'time in sec'
set grid
set terminal pngcairo size 2000,1000
set output 'cpu.png'
set datafile separator whitespace
set style line 1 \
    linecolor rgb '#0060ad' \
    linetype 1 linewidth 2 \
    pointtype 7 pointsize 0.5

set style line 2 \
    linecolor rgb '#ff60ad' \
    linetype 2 linewidth 2 \
    pointtype 7 pointsize 0.5

set ytics 10 nomirror
set y2tics
set key outside below 
plot 'trex_stats.dat' using 1:3 title "CPU" with linespoints linestyle 1 , \
'trex_stats.dat' using 1:4 title "Mem" with linespoints linestyle 2 axes x1y2
eor