Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spine 1.2.28 encountered segfault #356

Open
kiwildante opened this issue Oct 16, 2024 · 27 comments
Open

Spine 1.2.28 encountered segfault #356

kiwildante opened this issue Oct 16, 2024 · 27 comments

Comments

@kiwildante
Copy link

Describe the bug
i have a few cacti server that collect data for my device, i have server that only have about a hundred devices, and i have server that have a thousand devices, both produce segfault, but the hundred one only produce a few, but in the thousand device produce a lot with no backtrace, a few have a backtrace, i already set nofiles to a larger one, and have mysql max connection larger, but it still produce the same result, i run with 20/10 p/t for the large one, and 3/3 for the hundred.
it affect the polling time to be maximum, cause maybe the poller is waiting the spine to complete but the spine produce a segfault

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'logs'
  2. Change to cacti_stderr.log
  3. Some Spine Error segfault
  4. See error

Expected behavior
poller runs normally without segfault

Screenshots
here for the hundred device log
image
here for the thousand device
image
here's with backtrace
image
here's poller information for the thousand one
image

Server (please complete the following information):

  • OS: Ubuntu 16.04.7 LTS
  • Version 1.2.28

Compiling (please complete the following information):

  • compiler: gcc 5.4.0
  • autoconf: autoconf 2.69
  • glibc: ldd 2.23
  • source: files cacti spine latest

Note: you can find glibc version by running ldd --version

Note: if source is github, please include last commit reference

Additional context

full backtrace log

2024-10-16 12:23:44 - FATAL: Spine Encountered a Segmentation Fault

Generating backtrace...0 line(s)...
*** Error in `/usr/local/spine/bin/spine': double free or corruption (!prev): 0x000000000255aa40 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f041833d7f5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7f041834638a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f041834a58c]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x56846)[0x7f0418c0c846]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x281ba)[0x7f0418bde1ba]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x2ec11)[0x7f0418be4c11]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_send_query+0x57)[0x7f0418be1187]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_real_query+0x11)[0x7f0418be1201]
/usr/local/spine/bin/spine[0x4057ab]
/usr/local/spine/bin/spine[0x40e479]
/usr/local/spine/bin/spine[0x412ab2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f04186976ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f04183cd51d]
======= Memory map: ========
00400000-00424000 r-xp 00000000 08:01 922807 /usr/local/spine/bin/spine
00623000-00624000 r--p 00023000 08:01 922807 /usr/local/spine/bin/spine
00624000-00625000 rw-p 00024000 08:01 922807 /usr/local/spine/bin/spine
00625000-0069a000 rw-p 00000000 00:00 0
024d4000-0258a000 rw-p 00000000 00:00 0 [heap]
7f03e4000000-7f03e4039000 rw-p 00000000 00:00 0
7f03e4039000-7f03e8000000 ---p 00000000 00:00 0
7f03e8000000-7f03e8023000 rw-p 00000000 00:00 0
7f03e8023000-7f03ec000000 ---p 00000000 00:00 0
7f03ec000000-7f03ec022000 rw-p 00000000 00:00 0
7f03ec022000-7f03f0000000 ---p 00000000 00:00 0
7f03f0000000-7f03f003f000 rw-p 00000000 00:00 0
7f03f003f000-7f03f4000000 ---p 00000000 00:00 0
7f03f4000000-7f03f4022000 rw-p 00000000 00:00 0
7f03f4022000-7f03f8000000 ---p 00000000 00:00 0
7f03f8000000-7f03f8022000 rw-p 00000000 00:00 0
7f03f8022000-7f03fc000000 ---p 00000000 00:00 0
7f03fdffc000-7f03fdffd000 ---p 00000000 00:00 0
7f03fdffd000-7f03fe7fd000 rw-p 00000000 00:00 0
7f03fe7fd000-7f03fe7fe000 ---p 00000000 00:00 0
7f03fe7fe000-7f03feffe000 rw-p 00000000 00:00 0
7f03feffe000-7f03fefff000 ---p 00000000 00:00 0
7f03fefff000-7f03ff7ff000 rw-p 00000000 00:00 0
7f03ff7ff000-7f03ff800000 ---p 00000000 00:00 0
7f03ff800000-7f0400000000 rw-p 00000000 00:00 0
7f0400000000-7f0400022000 rw-p 00000000 00:00 0
7f0400022000-7f0404000000 ---p 00000000 00:00 0
7f0404000000-7f0404039000 rw-p 00000000 00:00 0
7f0404039000-7f0408000000 ---p 00000000 00:00 0
7f0408000000-7f0408022000 rw-p 00000000 00:00 0
7f0408022000-7f040c000000 ---p 00000000 00:00 0
7f040c000000-7f040c022000 rw-p 00000000 00:00 0
7f040c022000-7f0410000000 ---p 00000000 00:00 0
7f0410000000-7f0410022000 rw-p 00000000 00:00 0
7f0410022000-7f0414000000 ---p 00000000 00:00 0
7f041404b000-7f041404c000 ---p 00000000 00:00 0
7f041404c000-7f041484c000 rw-p 00000000 00:00 0
7f041484c000-7f041484d000 ---p 00000000 00:00 0
7f041484d000-7f041504d000 rw-p 00000000 00:00 0
7f041504d000-7f041504e000 ---p 00000000 00:00 0
7f041504e000-7f041584e000 rw-p 00000000 00:00 0
7f041584e000-7f041584f000 ---p 00000000 00:00 0
7f041584f000-7f041604f000 rw-p 00000000 00:00 0
7f041604f000-7f0416050000 ---p 00000000 00:00 0
7f0416050000-7f0416850000 rw-p 00000000 00:00 0
7f0416850000-7f0416851000 ---p 00000000 00:00 0
7f0416851000-7f0417051000 rw-p 00000000 00:00 0
7f0417051000-7f041705c000 r-xp 00000000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041705c000-7f041725b000 ---p 0000b000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041725b000-7f041725c000 r--p 0000a000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041725c000-7f041725d000 rw-p 0000b000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041725d000-7f0417263000 rw-p 00000000 00:00 0
7f0417263000-7f0417279000 r-xp 00000000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0417279000-7f0417478000 ---p 00016000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0417478000-7f0417479000 rw-p 00015000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0417479000-7f04175eb000 r-xp 00000000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04175eb000-7f04177eb000 ---p 00172000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04177eb000-7f04177f5000 r--p 00172000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04177f5000-7f04177f7000 rw-p 0017c000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04177f7000-7f04177fb000 rw-p 00000000 00:00 0
7f04177fb000-7f0417859000 r-xp 00000000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417859000-7f0417a59000 ---p 0005e000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417a59000-7f0417a5d000 r--p 0005e000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417a5d000-7f0417a63000 rw-p 00062000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417a63000-7f0417a7c000 r-xp 00000000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417a7c000-7f0417c7b000 ---p 00019000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417c7b000-7f0417c7c000 r--p 00018000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417c7c000-7f0417c7d000 rw-p 00019000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417c7d000-7f0417c80000 r-xp 00000000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417c80000-7f0417e7f000 ---p 00003000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417e7f000-7f0417e80000 r--p 00002000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417e80000-7f0417e81000 rw-p 00003000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417e81000-7f041809c000 r-xp 00000000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f041809c000-7f041829b000 ---p 0021b000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f041829b000-7f04182b7000 r--p 0021a000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f04182b7000-7f04182c3000 rw-p 00236000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f04182c3000-7f04182c6000 rw-p 00000000 00:00 0
7f04182c6000-7f0418486000 r-xp 00000000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f0418486000-7f0418686000 ---p 001c0000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f0418686000-7f041868a000 r--p 001c0000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f041868a000-7f041868c000 rw-p 001c4000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f041868c000-7f0418690000 rw-p 00000000 00:00 0
7f0418690000-7f04186a8000 r-xp 00000000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04186a8000-7f04188a7000 ---p 00018000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04188a7000-7f04188a8000 r--p 00017000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04188a8000-7f04188a9000 rw-p 00018000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04188a9000-7f04188ad000 rw-p 00000000 00:00 0
7f04188ad000-7f04189b5000 r-xp 00000000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f04189b5000-7f0418bb4000 ---p 00108000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f0418bb4000-7f0418bb5000 r--p 00107000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f0418bb5000-7f0418bb6000 rw-p 00108000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f0418bb6000-7f0418ef2000 r-xp 00000000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f0418ef2000-7f04190f1000 ---p 0033c000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f04190f1000-7f04190f5000 r--p 0033b000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f04190f5000-7f0419167000 rw-p 0033f000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f0419167000-7f041916d000 rw-p 00000000 00:00 0
7f041916d000-7f0419210000 r-xp 00000000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419210000-7f0419410000 ---p 000a3000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419410000-7f0419411000 r--p 000a3000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419411000-7f0419413000 rw-p 000a4000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419413000-7f0419449000 rw-p 00000000 00:00 0
7f0419449000-7f041946f000 r-xp 00000000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so
7f04194c2000-7f041965a000 r--p 00000000 08:01 788398 /usr/lib/locale/locale-archive
7f041965a000-7f0419663000 rw-p 00000000 00:00 0
7f041966d000-7f041966e000 rw-p 00000000 00:00 0
7f041966e000-7f041966f000 r--p 00025000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so
7f041966f000-7f0419670000 rw-p 00026000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so
7f0419670000-7f0419671000 rw-p 00000000 00:00 0
7ffc62213000-7ffc62314000 rw-p 00000000 00:00 0 [stack]
7ffc623cd000-7ffc623cf000 r--p 00000000 00:00 0 [vvar]
7ffc623cf000-7ffc623d1000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
2024-10-16 12:30:27 - FATAL: Spine Interrupted by Abort Signal

@TheWitness
Copy link
Member

You likely ran out of connections. Look at the connections logic on the technical support page.

@TheWitness
Copy link
Member

Also, for 1000 devices, I would normally use 2 processes and maybe 20 threads.

@kiwildante
Copy link
Author

i already sent the screenshot for the connection logic, i set the max connection to 2000 and the min req is 400, so i guess it's not the connection problem,
allright i'll try to change it to 2 process and 20 threads and see if it still produce the segfault

@kiwildante
Copy link
Author

i already set the process to 2 and threads to 20, but it makes the segfault occur more often from before
image
image

@TheWitness
Copy link
Member

What's you log level?

@TheWitness
Copy link
Member

What operating system and version?

@kiwildante
Copy link
Author

i'm setting it to low right now,
i'm using Ubuntu 16.04.7 LTS

@kiwildante
Copy link
Author

in the poller setting i set the invalid data logging to detailed

@TheWitness
Copy link
Member

Did you get a core file? Can you do a simple backtrace of it one that provides line numbers and files?

@kiwildante
Copy link
Author

i'm sorry, i already tried to get the core files, but it didn't record anything, i'll try it again later

@kiwildante
Copy link
Author

i can only provide you with this cacti log backtrace that happens in one of my 100 devices server, it uses 3 process and 3 thread

2024-10-28 11:40:20 - FATAL: Spine Interrupted by Abort Signal -- *** Error in `/usr/local/spine/bin/spine': double free or corruption (!prev): 0x0000000000917260 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f66c85007f5] /lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7f66c850938a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f66c850d58c] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x56846)[0x7f66c8dcf846] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x281ba)[0x7f66c8da11ba] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x2ec11)[0x7f66c8da7c11] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_send_query+0x57)[0x7f66c8da4187] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_real_query+0x11)[0x7f66c8da4201] /usr/local/spine/bin/spine[0x4055fb] /usr/local/spine/bin/spine[0x40e565] /usr/local/spine/bin/spine[0x412422] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f66c885a6ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f66c85904dd] ======= Memory map: ======== 00400000-00422000 r-xp 00000000 09:00 25692760 /usr/local/spine/bin/spine 00622000-00623000 r--p 00022000 09:00 25692760 /usr/local/spine/bin/spine 00623000-00624000 rw-p 00023000 09:00 25692760 /usr/local/spine/bin/spine 00624000-00699000 rw-p 00000000 00:00 0 008c5000-0093c000 rw-p 00000000 00:00 0 [heap] 7f66b0000000-7f66b0023000 rw-p 00000000 00:00 0 7f66b0023000-7f66b4000000 ---p 00000000 00:00 0 7f66b8000000-7f66b803a000 rw-p 00000000 00:00 0 7f66b803a000-7f66bc000000 ---p 00000000 00:00 0 7f66bc000000-7f66bc040000 rw-p 00000000 00:00 0 7f66bc040000-7f66c0000000 ---p 00000000 00:00 0 7f66c0000000-7f66c003a000 rw-p 00000000 00:00 0 7f66c003a000-7f66c4000000 ---p 00000000 00:00 0 7f66c5210000-7f66c5211000 ---p 00000000 00:00 0 7f66c5211000-7f66c5a11000 rw-p 00000000 00:00 0 7f66c5a11000-7f66c5a12000 ---p 00000000 00:00 0 7f66c5a12000-7f66c6212000 rw-p 00000000 00:00 0 7f66c6212000-7f66c6213000 ---p 00000000 00:00 0 7f66c6213000-7f66c6a13000 rw-p 00000000 00:00 0 7f66c6a13000-7f66c6a14000 ---p 00000000 00:00 0 7f66c6a14000-7f66c7214000 rw-p 00000000 00:00 0 7f66c7214000-7f66c721f000 r-xp 00000000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c721f000-7f66c741e000 ---p 0000b000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c741e000-7f66c741f000 r--p 0000a000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c741f000-7f66c7420000 rw-p 0000b000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c7420000-7f66c7426000 rw-p 00000000 00:00 0 7f66c7426000-7f66c743c000 r-xp 00000000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c743c000-7f66c763b000 ---p 00016000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c763b000-7f66c763c000 rw-p 00015000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c763c000-7f66c77ae000 r-xp 00000000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c77ae000-7f66c79ae000 ---p 00172000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79ae000-7f66c79b8000 r--p 00172000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79b8000-7f66c79ba000 rw-p 0017c000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79ba000-7f66c79be000 rw-p 00000000 00:00 0 7f66c79be000-7f66c7a1c000 r-xp 00000000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7a1c000-7f66c7c1c000 ---p 0005e000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c1c000-7f66c7c20000 r--p 0005e000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c20000-7f66c7c27000 rw-p 00062000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c27000-7f66c7c40000 r-xp 00000000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7c40000-7f66c7e3f000 ---p 00019000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e3f000-7f66c7e40000 r--p 00018000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e40000-7f66c7e41000 rw-p 00019000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e41000-7f66c7e44000 r-xp 00000000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c7e44000-7f66c8043000 ---p 00003000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8043000-7f66c8044000 r--p 00002000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8044000-7f66c8045000 rw-p 00003000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8045000-7f66c825f000 r-xp 00000000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c825f000-7f66c845e000 ---p 0021a000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c845e000-7f66c847a000 r--p 00219000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c847a000-7f66c8486000 rw-p 00235000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c8486000-7f66c8489000 rw-p 00000000 00:00 0 7f66c8489000-7f66c8649000 r-xp 00000000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c8649000-7f66c8849000 ---p 001c0000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c8849000-7f66c884d000 r--p 001c0000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c884d000-7f66c884f000 rw-p 001c4000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c884f000-7f66c8853000 rw-p 00000000 00:00 0 7f66c8853000-7f66c886b000 r-xp 00000000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c886b000-7f66c8a6a000 ---p 00018000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6a000-7f66c8a6b000 r--p 00017000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6b000-7f66c8a6c000 rw-p 00018000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6c000-7f66c8a70000 rw-p 00000000 00:00 0 7f66c8a70000-7f66c8b78000 r-xp 00000000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8b78000-7f66c8d77000 ---p 00108000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d77000-7f66c8d78000 r--p 00107000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d78000-7f66c8d79000 rw-p 00108000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d79000-7f66c90b5000 r-xp 00000000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c90b5000-7f66c92b4000 ---p 0033c000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c92b4000-7f66c92b8000 r--p 0033b000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c92b8000-7f66c932a000 rw-p 0033f000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c932a000-7f66c9330000 rw-p 00000000 00:00 0 7f66c9330000-7f66c93d3000 r-xp 00000000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c93d3000-7f66c95d3000 ---p 000a3000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d3000-7f66c95d4000 r--p 000a3000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d4000-7f66c95d6000 rw-p 000a4000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d6000-7f66c960c000 rw-p 00000000 00:00 0 7f66c960c000-7f66c9632000 r-xp 00000000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9686000-7f66c981e000 r--p 00000000 09:00 24908536 /usr/lib/locale/locale-archive 7f66c981e000-7f66c9827000 rw-p 00000000 00:00 0 7f66c9830000-7f66c9831000 rw-p 00000000 00:00 0 7f66c9831000-7f66c9832000 r--p 00025000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9832000-7f66c9833000 rw-p 00026000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9833000-7f66c9834000 rw-p 00000000 00:00 0 7ffe15453000-7ffe15552000 rw-p 00000000 00:00 0 [stack] 7ffe155f7000-7ffe155fa000 r--p 00000000 00:00 0 [vvar] 7ffe155fa000-7ffe155fc000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

@TheWitness
Copy link
Member

Where are you saving core files? Can you enable the capture of the core file and then backtrace the core file? If now. Go to your build directory, copy the spine.conf file into that build location. And then run spine this way:

cd /spine/build/directory
gdb ./spine
run -V 3 -S
# wait for crash
bt
# capture output
quit

@TheWitness
Copy link
Member

@kiwildante, any update?

@kiwildante
Copy link
Author

i'm sorry for the late update, i'm still not get the core files, i set it in the tmp folder, but it still not showing
i'll try with the gdb

@kiwildante
Copy link
Author

here's the bt
(gdb) bt
#0 0x00007fac1dbc13b0 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#1 0x00007fac1db89e31 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#2 0x00007fac1db8f162 in mysql_send_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#3 0x00007fac1db8f201 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#4 0x00000000004057ab in db_query (mysql=mysql@entry=0x7fabf8fd3910, type=type@entry=0,
query=query@entry=0x7fabf8fd8c90 "SELECT SQL_NO_CACHE id, hostname, snmp_community, snmp_version, snmp_username, snmp_password, snmp_auth_protocol, snmp_priv_passphrase, snmp_priv_protocol, snmp_context, snmp_engine_id, snmp_port, snm"...) at sql.c:159
#5 0x000000000040e479 in poll_host (device_counter=device_counter@entry=72, host_id=host_id@entry=1495, host_thread=host_thread@entry=1,
host_threads=host_threads@entry=1, host_data_ids=host_data_ids@entry=8, host_time=host_time@entry=0x7fabf8ff8de0 "1730715374",
host_errors=0x7fabf8ff8d0c, host_time_double=host_time_double@entry=1730715374.817122) at poller.c:565
#6 0x0000000000412ab2 in child (arg=0x17a3f50) at poller.c:109
#7 0x00007fac1d6456ba in start_thread (arg=0x7fabf8ff9700) at pthread_create.c:333
#8 0x00007fac1d37b51d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

@TheWitness
Copy link
Member

Yikes, that looks like a mysqlclient issue. Are you using MariaDB or MySQL? What version? You might want to see if there is a client update somewhere. It's an old server os though right Ubuntu 16.04?

@TheWitness
Copy link
Member

Also, if you were using a previous spine version, which one was it?

@kiwildante
Copy link
Author

i'm using mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrapper, it already the latest update, yes i'm using Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial

@kiwildante
Copy link
Author

i don't remember exactly, cause i have a few server, and each one i installed it with the new version, but lately i've been upgrading all of them to the latest, i think it's around 1.2.3, 1.2.14. and 1.2.18 i guess

@TheWitness
Copy link
Member

I might recommend you to stay on the older spine until you get your OS updated to something more recent. I don't think there's any harm to be honest.

@kiwildante
Copy link
Author

for now i set it to 50/20, and total connection is max 2048 min req 1250

@kiwildante
Copy link
Author

what version u can recommend using for ubuntu 16.04 and 18.04, and if the version of the spine and cacti are different is there any issue ? or do i have to downgrade the cacti version too ?

@TheWitness
Copy link
Member

First, make sure you compile the spine on the Linux you are running on, or at least on an older release. Second, as long as you are at Cacti 1.2.x or above, whatever version you are using will even work with 1.3.

I don't see spine architecture changing drastically in my lifetime any way. It's design is set in stone. There are a few enhancements around SNMP that will happen over time, other than that and maybe some optimization on startup, and ipv6 ping, but not much more going on. Regardless of those changes, it will remain compatible with future releases.

@TheWitness
Copy link
Member

TheWitness commented Nov 5, 2024

In more recent versions of spine, I went to connection pooling that creates all the MySQL/MariaDB connections up front. My guess is that is causing issues with your mysqlclient. Likely a memory leak of some kind in the client API with so many connections sharing the same memory space.

I debated giving people the ability to switch from pooling to on demand connections again, but thus far, too many other things going on in my life.

@kiwildante
Copy link
Author

can u tell me which version of spine that more stable for ubuntu 16.04 or 18.04
or u can tell me which version of the spine dan use on demand connection

@TheWitness
Copy link
Member

Review the change log for the term 'connection pool'. Use a version before that. Make sure you compile it using the machine it runs on.

@kiwildante
Copy link
Author

allright, thanks for the help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants