nfdump-1.7 Beta testers wanted.

[phaag]

The code for nfdump-1.7 settles slowly. Beta testers are wanted and welcome. If you want to test nfdump-1.7 please check out the unicorn branch:

git clone -b unicorn https://github.com/phaag/nfdump.git nfdump-1.7

Comments and bug reports are welcome.

[gnought]

Any good replacement to inject data in influxdb if nfprofile and nftrack are removed?

[piorek94]

Problem:

Issue is related to setting PID file for nfcapd/sfcapd/nfpcapd.

Description:

In version 1.7.0 there is requirement of creating pid file before launching mentioned programs (but is removed automatically on exit) which sometimes could lead to incompatibility problems.

Maybe requirement should only apply to dirname of pid path.

The problem is caused by using realpath() on argument parsing and removing code which creates pid file (by open() call).

For version 1.6.x pid file is created by mentioned programs, which is appropriate in my opinion.

Expected behavior

Pid file should be created, if possible, by nfcapd/sfcapd/nfpcapd.

[piorek94]

Problem:

Issue is related to w flag (sync file rotation) for nfcapd and sfcapd.

Description:

In version 1.7.0 flag w was removed from nfcapd, but it still available for sfcapd. In recent versions 1.6.x flag w is allowed for compatibility (nfcapd), but timeslot is always synced - this could be also applied to sfcapd.

Expected behavior

Consistence between nfcapd and sfcapd arguments (remove w from both or keep for compatibility reasons).

[piorek94]

Problem:

Issue is related to T flag (extension tags) for nfpcapd and sfcapd.

Description:

In version 1.7.0 flag T was removed from nfpcapd and sfcapd. For nfcapd it is still but it still allowed for compatibility.

Expected behavior

Consistence of nfcapd,sfcapd and nfpcad arguments (remove T from all or keep for compatibility reasons, man should also be updated).

[piorek94]

Problem:

Typos and missing man/usage info.

Description:

Typos:

• *cpad -> capd (man/, README, configure)
• tpye -> type (grammar)
• sflowd -> sfcapd (man.nfdump)

Missing info/usage examples:

• flag T for nfdump
• flag T for nfexpire
• flag y for sfcapd man

Expected behavior

Remove typos and update man/usage.

[piorek94]

Problem:

Verbose output of nfcapd.

Description:

Debug info (DumpHex) after launching nfcapd in non debug mode.

Expected behavior

Use dbg_printf instead printfin DumpHex() (in nfxV3.c)

[piorek94]

Are you planning full adapting sflow code from sflowtool to nfdump? I have noticed that you have made a step towards it (ex. add sflow_drop.h but not add readDiscardSample()).

To extend sflow code i can address my PR (master branch): #290

[piorek94]

Problem:

Wrong pipe output format (-o pipe) with aggregation (-A)

Description:

For version 1.7.0 pipe output format together with aggregation mode results with one long line instead newline separated records/flows (with s flag everything is ok).

Expected behavior

Records/flow in pipe output format with aggregation should be newline separated.

[phaag]

Problem:

Issue is related to w flag (sync file rotation) for nfcapd and sfcapd.

Description:

In version 1.7.0 flag w was removed from nfcapd, but it still available for sfcapd. In recent versions 1.6.x flag w is allowed for compatibility (nfcapd), but timeslot is always synced - this could be also applied to sfcapd.

Expected behavior

Consistence between nfcapd and sfcapd arguments (remove w from both or keep for compatibility reasons).

Thanks - Fixed in 47595c7

[phaag]

Problem:

Issue is related to T flag (extension tags) for nfpcapd and sfcapd.

Description:

In version 1.7.0 flag T was removed from nfpcapd and sfcapd. For nfcapd it is still but it still allowed for compatibility.

Expected behavior

Consistence of nfcapd,sfcapd and nfpcad arguments (remove T from all or keep for compatibility reasons, man should also be updated).

Thanks - Fixed in bbd3408

[phaag]

Problem:

Typos and missing man/usage info.

Description:

Typos:

• *cpad -> capd (man/, README, configure)
• tpye -> type (grammar)
• sflowd -> sfcapd (man.nfdump)

Missing info/usage examples:

• flag T for nfdump
• flag T for nfexpire
• flag y for sfcapd man

Expected behavior

Remove typos and update man/usage.

Thanks! Fixed in 3b9731a

[phaag]

Debug

Problem:

Verbose output of nfcapd.

Description:

Debug info (DumpHex) after launching nfcapd in non debug mode.

Expected behavior

Use dbg_printf instead printfin DumpHex() (in nfxV3.c)

Thanks! Fixed in ef739d8

[phaag]

Problem:

Wrong pipe output format (-o pipe) with aggregation (-A)

Description:

For version 1.7.0 pipe output format together with aggregation mode results with one long line instead newline separated records/flows (with s flag everything is ok).

Expected behavior

Records/flow in pipe output format with aggregation should be newline separated.

Thanks. Fixed in 67efa0e

[phaag]

Are you planning full adapting sflow code from sflowtool to nfdump? I have noticed that you have made a step towards it (ex. add sflow_drop.h but not add readDiscardSample()).

To extend sflow code i can address my PR (master branch): #290

What information are you exactly missing? So far, the goal was to have pretty much the same information from netflow/sflow collectors - with variations of course. I am not an sflow user and have no sflow equipment. I am open to integrate fields which are of interest to users, but do not want to blindly integrate each and every information.

[phaag]

Problem:

Issue is related to setting PID file for nfcapd/sfcapd/nfpcapd.

Description:

In version 1.7.0 there is requirement of creating pid file before launching mentioned programs (but is removed automatically on exit) which sometimes could lead to incompatibility problems.

Maybe requirement should only apply to dirname of pid path.

The problem is caused by using realpath() on argument parsing and removing code which creates pid file (by open() call).

For version 1.6.x pid file is created by mentioned programs, which is appropriate in my opinion.

Expected behavior

Pid file should be created, if possible, by nfcapd/sfcapd/nfpcapd.

I am not sure if I understand you correctly. The pid files are created by nfcapd/sfcapd/nfpcapd.. Could you please elaborate more detailed?
Thanks!

[tarungoswami]

@tarungoswami - What OS/Version are you using?

"Ubuntu 16.04.6 LTS"

On what CPU (endian type)? Intel AMD, PowerPC?

Now I moved to CentOS Linux 8 - Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz

[piorek94]

Problem:

Issue is related to time window reporting and processing by nfdump.

Enviroment

OS: Debian 9, Debian 8

Description:

Pcap and generated nfcapd file:nfdump_issue_data.tar.gz

Flag -t timewin is not properly processed by nfdump 1.7.0. Additionally reported time window in summary has strange values. Example is related to sflow (sfcapd), but netflow/ipfix (nfcapd) are also affected.

• file: nfcapd.202106141052
• collecting: sfcapd -p 10000 -l ./ -t 60
• Wrong summary:

$ nfdump -r nfcapd.202106141052
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2021-06-14 10:52:03.115     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:03.620     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
...
2021-06-14 10:52:31.252     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:31.756     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
Summary: total flows: 91, total bytes: 3.0 G, total packets: 2.0 M, avg bps: 837.1 M, avg pps: 69407, avg bpp: 1507
Time window: 53421-10-05 13:31:55 - 53421-10-05 21:29:16
Total flows processed: 91, passed: 91, Blocks skipped: 0, Bytes read: 15732
Sys: 0.0000s User: 0.0000s Wall: 0.0039s flows/second: 23531.5 Runtime: 0.0040s

• Wrong filtering (no effect):

$ nfdump -r nfcapd.202106141052 -t "2021/06/14.10:52:27-2021/06/14.10:52:32"
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2021-06-14 10:52:03.115     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:03.620     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
...
2021-06-14 10:52:31.252     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:31.756     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
Summary: total flows: 91, total bytes: 3.0 G, total packets: 2.0 M, avg bps: 837.1 M, avg pps: 69407, avg bpp: 1507
Time window: 53421-10-05 13:31:55 - 53421-10-05 21:29:16
Total flows processed: 91, passed: 91, Blocks skipped: 0, Bytes read: 15732
Sys: 0.0040s User: 0.0000s Wall: 0.0050s flows/second: 18210.9 Runtime: 0.0051s

NOTES:
Version 1.6.23 works properly.

Error source:

• timeWindow variable in main() function of src/nfdump.c is always set to NULL (line 708) - missing reference to proper value (should be set in if block in line 1033 or remove it and use flist.timeWindow instead)
• division of t_first_flow and t_last_flow should be done before if block (line 1258) - TimeString() func expects time in seconds - in case of not provided time window filtering TimeString() func receives time in miliseconds

Expected behavior

Time window value should be properly handled by nfdump (and valid reported time window).

[tarungoswami]

Problem : Getting "unknown extension '25' " Error
When we are running following command:

We are continuously getting error in between the output : "unknown extension '25' ". See below

We are running Centos server

@tarungoswami
It look like you mixed up versions please make sure to remove old version of all nfdump-1.7 tools and data, when retesting.
Extension 25 is an NSEL extension which is unknown, if not processed with an nfdump version compiled with nsel-enabled.
As a precaution I can catch that with anoher error message.

Thanks @phaag, please confirm when that is handled. We will take the checkout post your confirmation

@phaag is this handled now?

[piorek94]

Problem:

Issue is related to occassionaly error during collecting IPFIX records by nfcapd.

Enviroment

OS: Debian 9, Debian 8

Description:

• collecting: nfcapd -p 3311 -l data/ -t 10 -B 250000 -P nfcapd.pid -e

Some records during collecting IPFIX records are malformed (I can not detect why on my own).

I had noticed that more malformed records occurred for shorter time interval (t flag) and longer runtime.

• Example output:

$ nfdump -r nfcapd.20210625140510_anon 'dst ip 204.35.59.220'
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2021-06-25 14:05:04.640     0.000 TCP        178.76.8.72:80    ->    204.35.59.220:64027      500   752000     1
457400-10-01 13:12:34.816     0.000 TCP     127.21.137.106:80    ->    204.35.59.220:6911       500 9927059.5 T     1
2021-06-25 14:05:07.456     3.072 TCP     127.21.137.106:80    ->    204.35.59.220:6911      1000    1.2 M     1
2021-06-25 14:05:07.456    11.008 TCP     127.21.137.124:80    ->    204.35.59.220:6836      3000    4.5 M     1
Summary: total flows: 4, total bytes: 9927059.5 T, total packets: 5000, avg bps: 391743, avg pps: 0, avg bpp: 1985.4 T
Time window: 53450-05-05 03:12:32 - 847894544-05-27 19:08:00
Total flows processed: 4018, passed: 4, Blocks skipped: 0, Bytes read: 592692
Sys: 0.0000s User: 0.0000s Wall: 0.0007s flows/second: 5429353.6 Runtime: 0.0008s

Sometimes other fields (other than time, bytes, packets, etc) are also malformed, eg.

• fields related to source/destination ip are swapped in form: D.C.B.A instead A.B.C.D (which is valid)

NOTES:

• Such behavior was not noticed for version 1.6.23.
• Pcap file can not be attached due confidency reasons - but was gathered during error. It was also analyzed by external tool (eg. wireshark) - pcap file is ok so I suppose it is internal error in nfcapd. I have also tried reproduce this error by replaying pcap file but new generated file is ok.
• I have also noticed very high virtual memory (i suppose it has not impact) for nfcapd process - ca. 50GB for above command after day of running (for version 1.6.23 it is much smaller)

Expected behavior

Please detect and remove problem source - no malformed records.

[phaag]

Problem:

Issue is related to time window reporting and processing by nfdump.

Enviroment

OS: Debian 9, Debian 8

Description:

Pcap and generated nfcapd file:nfdump_issue_data.tar.gz

Flag -t timewin is not properly processed by nfdump 1.7.0. Additionally reported time window in summary has strange values. Example is related to sflow (sfcapd), but netflow/ipfix (nfcapd) are also affected.

• file: nfcapd.202106141052
• collecting: sfcapd -p 10000 -l ./ -t 60
• Wrong summary:

$ nfdump -r nfcapd.202106141052
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2021-06-14 10:52:03.115     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:03.620     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
...
2021-06-14 10:52:31.252     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:31.756     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
Summary: total flows: 91, total bytes: 3.0 G, total packets: 2.0 M, avg bps: 837.1 M, avg pps: 69407, avg bpp: 1507
Time window: 53421-10-05 13:31:55 - 53421-10-05 21:29:16
Total flows processed: 91, passed: 91, Blocks skipped: 0, Bytes read: 15732
Sys: 0.0000s User: 0.0000s Wall: 0.0039s flows/second: 23531.5 Runtime: 0.0040s

• Wrong filtering (no effect):

$ nfdump -r nfcapd.202106141052 -t "2021/06/14.10:52:27-2021/06/14.10:52:32"
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2021-06-14 10:52:03.115     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:03.620     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
...
2021-06-14 10:52:31.252     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
2021-06-14 10:52:31.756     0.000 UDP           10.0.0.0:2012  ->   192.168.100.10:3000     21845   32.9 M     1
Summary: total flows: 91, total bytes: 3.0 G, total packets: 2.0 M, avg bps: 837.1 M, avg pps: 69407, avg bpp: 1507
Time window: 53421-10-05 13:31:55 - 53421-10-05 21:29:16
Total flows processed: 91, passed: 91, Blocks skipped: 0, Bytes read: 15732
Sys: 0.0040s User: 0.0000s Wall: 0.0050s flows/second: 18210.9 Runtime: 0.0051s

NOTES:
Version 1.6.23 works properly.

Error source:

• timeWindow variable in main() function of src/nfdump.c is always set to NULL (line 708) - missing reference to proper value (should be set in if block in line 1033 or remove it and use flist.timeWindow instead)
• division of t_first_flow and t_last_flow should be done before if block (line 1258) - TimeString() func expects time in seconds - in case of not provided time window filtering TimeString() func receives time in miliseconds

Expected behavior

Time window value should be properly handled by nfdump (and valid reported time window).

Fixed in 8ee9de8

[phaag]

Problem:

Issue is related to occassionaly error during collecting IPFIX records by nfcapd.

Enviroment

OS: Debian 9, Debian 8

Description:

• collecting: nfcapd -p 3311 -l data/ -t 10 -B 250000 -P nfcapd.pid -e

Some records during collecting IPFIX records are malformed (I can not detect why on my own).

I had noticed that more malformed records occurred for shorter time interval (t flag) and longer runtime.

• Example output:

$ nfdump -r nfcapd.20210625140510_anon 'dst ip 204.35.59.220'
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2021-06-25 14:05:04.640     0.000 TCP        178.76.8.72:80    ->    204.35.59.220:64027      500   752000     1
457400-10-01 13:12:34.816     0.000 TCP     127.21.137.106:80    ->    204.35.59.220:6911       500 9927059.5 T     1
2021-06-25 14:05:07.456     3.072 TCP     127.21.137.106:80    ->    204.35.59.220:6911      1000    1.2 M     1
2021-06-25 14:05:07.456    11.008 TCP     127.21.137.124:80    ->    204.35.59.220:6836      3000    4.5 M     1
Summary: total flows: 4, total bytes: 9927059.5 T, total packets: 5000, avg bps: 391743, avg pps: 0, avg bpp: 1985.4 T
Time window: 53450-05-05 03:12:32 - 847894544-05-27 19:08:00
Total flows processed: 4018, passed: 4, Blocks skipped: 0, Bytes read: 592692
Sys: 0.0000s User: 0.0000s Wall: 0.0007s flows/second: 5429353.6 Runtime: 0.0008s

Sometimes other fields (other than time, bytes, packets, etc) are also malformed, eg.

• fields related to source/destination ip are swapped in form: D.C.B.A instead A.B.C.D (which is valid)

NOTES:

• Such behavior was not noticed for version 1.6.23.
• Pcap file can not be attached due confidency reasons - but was gathered during error. It was also analyzed by external tool (eg. wireshark) - pcap file is ok so I suppose it is internal error in nfcapd. I have also tried reproduce this error by replaying pcap file but new generated file is ok.
• I have also noticed very high virtual memory (i suppose it has not impact) for nfcapd process - ca. 50GB for above command after day of running (for version 1.6.23 it is much smaller)

Expected behavior

Please detect and remove problem source - no malformed records.

I had the virt. memory issue once on a Debian 5.9.15-1kali1, however, it resulted from a dirty git repo, after a git pull - not sure, what caused this effect. Deleting the entire repo and re-downloading/rebuilding fixed this. The virt memory is comparable with nfdump-1.6.x. So far I could not reproduce malformed records. I tested with the same command line options. I keep testing

[piorek94]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.

Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

---

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.

NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).

In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

[phaag]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.

Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.

NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).

In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

[piorek94]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.
Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.
NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).
In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

Hi,
It seems that virtual memory issue gone after that commit - 17h running nfcapd process with 10s rotation time reports virtual memory size of 737 MiB and not grows.

After that commit I get sometimes error during reading files:

nfdump -R ./ -s dstip/bytes
pthread_join() error in nffile.c line 1227: Success
pthread_join() error in nffile.c line 1227: Success
...

Unfortunately I'm still experiencing issue related to malformed records - I'm trying to generate appropriate pcap file with no confidential data. Malformed elements are usually 'bytes' (the most common), 'packets', and time related fields. Maybe have you any tool/method to anonymize ipfix/netflow records in pcap files? I have tried capturing traffic generated by nfreplay on anonymized nfcapd.* file using netflow v9 but segmentation fault occurs (maybe conversion from ipfix to netflow v9 is not possible?)

[phaag]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.
Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.
NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).
In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

Hi,
It seems that virtual memory issue gone after that commit - 17h running nfcapd process with 10s rotation time reports virtual memory size of 737 MiB and not grows.

After that commit I get sometimes error during reading files:

nfdump -R ./ -s dstip/bytes
pthread_join() error in nffile.c line 1227: Success
pthread_join() error in nffile.c line 1227: Success
...

Unfortunately I'm still experiencing issue related to malformed records - I'm trying to generate appropriate pcap file with no confidential data. Malformed elements are usually 'bytes' (the most common), 'packets', and time related fields. Maybe have you any tool/method to anonymize ipfix/netflow records in pcap files? I have tried capturing traffic generated by nfreplay on anonymized nfcapd.* file using netflow v9 but segmentation fault occurs (maybe conversion from ipfix to netflow v9 is not possible?)

The pthread_join() messages are removed in the latest commit. I also fixed some threading issues as well.
As of malformed data - I have a collector running over days with an IPFIX simulator (sender) without any malformed records. If you can not share the original pcap, I would like to ask you to send me at least the IPFIX template, as displayed in wireshark.
You may also send my the anonymised nfcapd file, so I can fix the coredump. You may send all this to my email address found in the Authors file.

[piorek94]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.
Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.
NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).
In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

Hi,
It seems that virtual memory issue gone after that commit - 17h running nfcapd process with 10s rotation time reports virtual memory size of 737 MiB and not grows.
After that commit I get sometimes error during reading files:

nfdump -R ./ -s dstip/bytes
pthread_join() error in nffile.c line 1227: Success
pthread_join() error in nffile.c line 1227: Success
...

Unfortunately I'm still experiencing issue related to malformed records - I'm trying to generate appropriate pcap file with no confidential data. Malformed elements are usually 'bytes' (the most common), 'packets', and time related fields. Maybe have you any tool/method to anonymize ipfix/netflow records in pcap files? I have tried capturing traffic generated by nfreplay on anonymized nfcapd.* file using netflow v9 but segmentation fault occurs (maybe conversion from ipfix to netflow v9 is not possible?)

The pthread_join() messages are removed in the latest commit. I also fixed some threading issues as well.
As of malformed data - I have a collector running over days with an IPFIX simulator (sender) without any malformed records. If you can not share the original pcap, I would like to ask you to send me at least the IPFIX template, as displayed in wireshark.
You may also send my the anonymised nfcapd file, so I can fix the coredump. You may send all this to my email address found in the Authors file.

Yes, after rebuilding image (commit 87fb3ba) error messages gone (and virtual memory usage decrease ci. 330M). I have noticed that problem occurs if there are more IPFIX exporters (in my case 4) - I also wasn't able to reproduce this error with single exporter. I have sent all mentioned data to your email address.

[phaag]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.
Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.
NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).
In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

Hi,
It seems that virtual memory issue gone after that commit - 17h running nfcapd process with 10s rotation time reports virtual memory size of 737 MiB and not grows.
After that commit I get sometimes error during reading files:

nfdump -R ./ -s dstip/bytes
pthread_join() error in nffile.c line 1227: Success
pthread_join() error in nffile.c line 1227: Success
...

Unfortunately I'm still experiencing issue related to malformed records - I'm trying to generate appropriate pcap file with no confidential data. Malformed elements are usually 'bytes' (the most common), 'packets', and time related fields. Maybe have you any tool/method to anonymize ipfix/netflow records in pcap files? I have tried capturing traffic generated by nfreplay on anonymized nfcapd.* file using netflow v9 but segmentation fault occurs (maybe conversion from ipfix to netflow v9 is not possible?)

The pthread_join() messages are removed in the latest commit. I also fixed some threading issues as well.
As of malformed data - I have a collector running over days with an IPFIX simulator (sender) without any malformed records. If you can not share the original pcap, I would like to ask you to send me at least the IPFIX template, as displayed in wireshark.
You may also send my the anonymised nfcapd file, so I can fix the coredump. You may send all this to my email address found in the Authors file.

Yes, after rebuilding image (commit 87fb3ba) error messages gone (and virtual memory usage decrease ci. 330M). I have noticed that problem occurs if there are more IPFIX exporters (in my case 4) - I also wasn't able to reproduce this error with single exporter. I have sent all mentioned data to your email address.

Thanks! The memory footprint is further reduced with commit 81b3f12 as well as the coredump with nfreplay.

[phaag]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.
Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.
NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).
In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

Hi,
It seems that virtual memory issue gone after that commit - 17h running nfcapd process with 10s rotation time reports virtual memory size of 737 MiB and not grows.
After that commit I get sometimes error during reading files:

nfdump -R ./ -s dstip/bytes
pthread_join() error in nffile.c line 1227: Success
pthread_join() error in nffile.c line 1227: Success
...

Unfortunately I'm still experiencing issue related to malformed records - I'm trying to generate appropriate pcap file with no confidential data. Malformed elements are usually 'bytes' (the most common), 'packets', and time related fields. Maybe have you any tool/method to anonymize ipfix/netflow records in pcap files? I have tried capturing traffic generated by nfreplay on anonymized nfcapd.* file using netflow v9 but segmentation fault occurs (maybe conversion from ipfix to netflow v9 is not possible?)

The pthread_join() messages are removed in the latest commit. I also fixed some threading issues as well.
As of malformed data - I have a collector running over days with an IPFIX simulator (sender) without any malformed records. If you can not share the original pcap, I would like to ask you to send me at least the IPFIX template, as displayed in wireshark.
You may also send my the anonymised nfcapd file, so I can fix the coredump. You may send all this to my email address found in the Authors file.

Yes, after rebuilding image (commit 87fb3ba) error messages gone (and virtual memory usage decrease ci. 330M). I have noticed that problem occurs if there are more IPFIX exporters (in my case 4) - I also wasn't able to reproduce this error with single exporter. I have sent all mentioned data to your email address.

Thanks! The memory footprint is further reduced with commit 81b3f12 as well as the coredump with nfreplay.

If you have 4 IPFIX exporters sending data to same collector, they must have different observation domain IDs. Otherwise you will run into corrupt data. There is no way to avoid this.

[piorek94]

I'm still experiencing two issues:

• growing virtual memory size of nfcapd/sfcapd process which lead to blocked process
• occasionally error during collecting IPFIX records by nfcapd

Both issues was reproduced in docker environment using prepared project
nfdump-tcpreplay-docker.
Issues should be treated separately (it seems there is no direct relation between).

Environment

• Nfdump version: unicorn branch
• Docker host: Debian 10/9/8, Ubuntu 20.04

Problem: Growing virtual memory size - blocked process

Description:

Running process of nfcapd/nfcapd increases its virtual memory size on every file
rotation. After reaching virtual memory size limit (on 64-bit Linux virtual
memory size is usually unlimited - but there is always limit, in my case it was
256GB) process is blocked and no error is returned.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with no changes.
NOTES:
tcpreplay container is not even required (running nfdump container without
traffic is enough)

Expected behavior

Reduce virtual memory usage.

Problem: Error during collecting IPFIX records by nfcapd

Description:

This is copy of previously mentioned problem - some records during collecting
IPFIX records are malformed.

How to reproduce

I was able to reproduce this issue in testing environment nfdump-tcpreplay-docker with following notes:

• used pcap was a record of production traffic (1min)
• first malformed record was observed after circa 36h
• issue was also reproduced on online traffic, docker compose file example:

  version: "3"
  
  services:
    mynfdump:
      build: ./nfdump
      restart: unless-stopped
      container_name: "mynfdump"
      ports:
        - "10000:10000/udp"
      environment:
        - NFEXPIRE=on
        - NFEXPIRE_TIME=2H
      command: ["nfcapd","-p","10000","-l",".","-t","10","-B","250000","-e"]
  
  networks:
    default:
      driver: bridge

I have noticed that reproduction of this issue is more likely using more
'sophisticated' traffic (example.pcap in repo won't be sufficient).
In my case there is 4 netflow exporters which generate 10s nfcapd files with
following summary:

Summary: total flows: 4104, total bytes: 3.6 G, total packets: 4.3 M, avg bps: 478177, avg pps: 70, avg bpp: 848
Time window: 2021-06-24 19:17:57 - 2021-06-25 12:05:26
Total flows processed: 4104, passed: 4104, Blocks skipped: 0, Bytes read: 608224

Expected behavior

No malformed records in nfcapd.* files.

The virtual memory issue should be gone with commit 4871d29. To speed up, you may set -t 2 down to 2 s rotation time. Please check, if it works now.
I assume, the malformed records resulted from same virt memory problem. If you still face malformed records, I would need to have a pcap to replay. Please specify which elements are malformed. Always the same or random.

Hi,
It seems that virtual memory issue gone after that commit - 17h running nfcapd process with 10s rotation time reports virtual memory size of 737 MiB and not grows.
After that commit I get sometimes error during reading files:

nfdump -R ./ -s dstip/bytes
pthread_join() error in nffile.c line 1227: Success
pthread_join() error in nffile.c line 1227: Success
...

Unfortunately I'm still experiencing issue related to malformed records - I'm trying to generate appropriate pcap file with no confidential data. Malformed elements are usually 'bytes' (the most common), 'packets', and time related fields. Maybe have you any tool/method to anonymize ipfix/netflow records in pcap files? I have tried capturing traffic generated by nfreplay on anonymized nfcapd.* file using netflow v9 but segmentation fault occurs (maybe conversion from ipfix to netflow v9 is not possible?)

The pthread_join() messages are removed in the latest commit. I also fixed some threading issues as well.
As of malformed data - I have a collector running over days with an IPFIX simulator (sender) without any malformed records. If you can not share the original pcap, I would like to ask you to send me at least the IPFIX template, as displayed in wireshark.
You may also send my the anonymised nfcapd file, so I can fix the coredump. You may send all this to my email address found in the Authors file.

Yes, after rebuilding image (commit 87fb3ba) error messages gone (and virtual memory usage decrease ci. 330M). I have noticed that problem occurs if there are more IPFIX exporters (in my case 4) - I also wasn't able to reproduce this error with single exporter. I have sent all mentioned data to your email address.

Thanks! The memory footprint is further reduced with commit 81b3f12 as well as the coredump with nfreplay.

If you have 4 IPFIX exporters sending data to same collector, they must have different observation domain IDs. Otherwise you will run into corrupt data. There is no way to avoid this.

Yes, they have different observation domain IDs - as I have said before: I replay recorded data and errors occur occasionally (most of the time everything is ok).

[phaag]

I convert this issue in a discussion, as it is a better place. nfdump-1.7-beta testers still wanted :)