Skip to content
This repository has been archived by the owner on Apr 18, 2024. It is now read-only.

Soft lockup on arm64 v0.96 #513

Open
arter97 opened this issue May 7, 2023 · 2 comments
Open

Soft lockup on arm64 v0.96 #513

arter97 opened this issue May 7, 2023 · 2 comments

Comments

@arter97
Copy link
Contributor

arter97 commented May 7, 2023

Hi, everyone.

I'm hitting soft lockup issues on mptcp_v0.96 on an arm64 server, using shadowsocks-rust.

The issue happens randomly - sometimes once a few weeks, sometimes less than a single day of uptime.

The debugging is limited due to kernel completely locking itself up, but systemd managed to save the following bits:

May  6 16:04:52 arter97-ampere kernel: [1950069.732944] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:04:52 arter97-ampere kernel: [1950069.744985] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:04:52 arter97-ampere kernel: [1950069.745058] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O      5.4.240-arter97+ #1
May  6 16:04:52 arter97-ampere kernel: [1950069.745058] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:04:52 arter97-ampere kernel: [1950069.745059] pstate: 00400005 (nzcv daif +PAN -UAO)
May  6 16:04:52 arter97-ampere kernel: [1950069.745062] pc : mptcp_data_ready+0x5c/0x1ad8
May  6 16:04:52 arter97-ampere kernel: [1950069.745063] lr : mptcp_data_ready+0x1790/0x1ad8
May  6 16:04:52 arter97-ampere kernel: [1950069.745063] sp : ffffff85d0297970
May  6 16:04:52 arter97-ampere kernel: [1950069.745063] x29: ffffff85d0297ff0 x28: ffffff85ce6040c0 
May  6 16:04:52 arter97-ampere kernel: [1950069.745064] x27: ffffff85d5a49bc0 x26: ffffff84ab19ab28 
May  6 16:04:52 arter97-ampere kernel: [1950069.745065] x25: ffffff85d5a49c90 x24: ffffff84ab19aa00 
May  6 16:04:52 arter97-ampere kernel: [1950069.745065] x23: ffffff85ce6040c0 x22: ffffff84ab19aa00 
May  6 16:04:52 arter97-ampere kernel: [1950069.745066] x21: 00000000253f0aac x20: ffffff85fa11dc00 
May  6 16:04:52 arter97-ampere kernel: [1950069.745066] x19: ffffff85d5a49c90 x18: 0000000000000000 
May  6 16:04:52 arter97-ampere kernel: [1950069.745067] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:04:52 arter97-ampere kernel: [1950069.745067] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:04:52 arter97-ampere kernel: [1950069.745068] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:04:52 arter97-ampere kernel: [1950069.745068] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:04:52 arter97-ampere kernel: [1950069.745069] x9 : 00000000253f0aac x8 : 000000000000004e 
May  6 16:04:52 arter97-ampere kernel: [1950069.745069] x7 : 0000000000000080 x6 : 00000000b4792267 
May  6 16:04:52 arter97-ampere kernel: [1950069.745070] x5 : ffffff84ab19a100 x4 : ffffff85fffa05a0 
May  6 16:04:52 arter97-ampere kernel: [1950069.745070] x3 : 00000000a0a77e4d x2 : 0000000000000000 
May  6 16:04:52 arter97-ampere kernel: [1950069.745071] x1 : 0000000000000032 x0 : 0000000000000002 
May  6 16:04:52 arter97-ampere kernel: [1950069.745072] Call trace:
May  6 16:04:52 arter97-ampere kernel: [1950069.745072]  mptcp_data_ready+0x5c/0x1ad8
May  6 16:06:12 arter97-ampere kernel: [1950149.972522] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:06:12 arter97-ampere kernel: [1950149.974950] rcu: 	1-....: (60333 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=29993 
May  6 16:06:12 arter97-ampere kernel: [1950149.975757] 	(t=60003 jiffies g=39741721 q=1720)
May  6 16:06:12 arter97-ampere kernel: [1950149.975758] Task dump for CPU 1:
May  6 16:06:12 arter97-ampere kernel: [1950149.975759] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:06:12 arter97-ampere kernel: [1950149.975761] Call trace:
May  6 16:06:12 arter97-ampere kernel: [1950149.975765]  dump_backtrace+0x0/0x160
May  6 16:06:12 arter97-ampere kernel: [1950149.975767]  sched_show_task+0x110/0x13c
May  6 16:06:12 arter97-ampere kernel: [1950149.975770]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:06:12 arter97-ampere kernel: [1950149.975777]  kfree+0x7c/0x238
May  6 16:08:04 arter97-ampere kernel: [1950261.731939] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:08:04 arter97-ampere kernel: [1950261.733349] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:08:04 arter97-ampere kernel: [1950261.733377] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:08:04 arter97-ampere kernel: [1950261.733378] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:08:04 arter97-ampere kernel: [1950261.733378] pstate: a0400005 (NzCv daif +PAN -UAO)
May  6 16:08:04 arter97-ampere kernel: [1950261.733382] pc : __kmalloc_track_caller+0xc4/0x280
May  6 16:08:04 arter97-ampere kernel: [1950261.733383] lr : __kmalloc_track_caller+0x4c/0x280
May  6 16:08:04 arter97-ampere kernel: [1950261.733384] sp : ffffff85d02978d0
May  6 16:08:04 arter97-ampere kernel: [1950261.733384] x29: ffffff85d0297ff0 x28: 0000000000000034 
May  6 16:08:04 arter97-ampere kernel: [1950261.733385] x27: ffffff85d5a49bc0 x26: ffffffc0111aa300 
May  6 16:08:04 arter97-ampere kernel: [1950261.733385] x25: 0000000000000a20 x24: ffffff84ab19aa00 
May  6 16:08:04 arter97-ampere kernel: [1950261.733386] x23: ffffffc0111aa300 x22: 0000000000082a20 
May  6 16:08:04 arter97-ampere kernel: [1950261.733386] x21: 0000000000000180 x20: ffffff85f1b6b800 
May  6 16:08:04 arter97-ampere kernel: [1950261.733386] x19: ffffff85f8b42c00 x18: 0000000000000000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733387] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:08:04 arter97-ampere kernel: [1950261.733387] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:08:04 arter97-ampere kernel: [1950261.733388] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:08:04 arter97-ampere kernel: [1950261.733388] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:08:04 arter97-ampere kernel: [1950261.733389] x9 : ffffff84c02b3680 x8 : 0000000000010000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733389] x7 : 0000000000000080 x6 : 0000000136ab2517 
May  6 16:08:04 arter97-ampere kernel: [1950261.733390] x5 : 000000010b7ef967 x4 : ffffff85fff9efa0 
May  6 16:08:04 arter97-ampere kernel: [1950261.733391] x3 : 000000010b7ef968 x2 : 0000000000000000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733391] x1 : 0000000000000000 x0 : 0000000000000000 
May  6 16:08:04 arter97-ampere kernel: [1950261.733392] Call trace:
May  6 16:08:04 arter97-ampere kernel: [1950261.733393]  __kmalloc_track_caller+0xc4/0x280
May  6 16:09:13 arter97-ampere kernel: [1950329.983581] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:09:13 arter97-ampere kernel: [1950329.984740] rcu: 	1-....: (105336 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=52494 
May  6 16:09:13 arter97-ampere kernel: [1950329.985548] 	(t=105006 jiffies g=39741721 q=2651)
May  6 16:09:13 arter97-ampere kernel: [1950329.985549] Task dump for CPU 1:
May  6 16:09:13 arter97-ampere kernel: [1950329.985549] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:09:13 arter97-ampere kernel: [1950329.985551] Call trace:
May  6 16:09:13 arter97-ampere kernel: [1950329.985555]  dump_backtrace+0x0/0x160
May  6 16:09:13 arter97-ampere kernel: [1950329.985558]  sched_show_task+0x110/0x13c
May  6 16:09:13 arter97-ampere kernel: [1950329.985561]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:09:13 arter97-ampere kernel: [1950329.985562]  skb_split+0x1ac/0x2b4
May  6 16:11:00 arter97-ampere kernel: [1950437.731020] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:11:00 arter97-ampere kernel: [1950437.732698] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:11:00 arter97-ampere kernel: [1950437.732725] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:11:00 arter97-ampere kernel: [1950437.732725] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:11:00 arter97-ampere kernel: [1950437.732726] pstate: 40400005 (nZcv daif +PAN -UAO)
May  6 16:11:00 arter97-ampere kernel: [1950437.732730] pc : kfree+0x44/0x238
May  6 16:11:00 arter97-ampere kernel: [1950437.732732] lr : __kfree_skb+0x1c/0xc0
May  6 16:11:00 arter97-ampere kernel: [1950437.732732] sp : ffffff85d0297940
May  6 16:11:00 arter97-ampere kernel: [1950437.732733] x29: ffffff85d0297ff0 x28: ffffff85ce6040c0 
May  6 16:11:00 arter97-ampere kernel: [1950437.732733] x27: ffffff85d5a49bc0 x26: ffffff84ab19aa28 
May  6 16:11:00 arter97-ampere kernel: [1950437.732734] x25: ffffff85d5a49c90 x24: ffffff84ab19aa00 
May  6 16:11:00 arter97-ampere kernel: [1950437.732734] x23: ffffff85ce6040c0 x22: ffffff84ab19ab00 
May  6 16:11:00 arter97-ampere kernel: [1950437.732735] x21: ffffffff17a6da80 x20: ffffff85f1b6a600 
May  6 16:11:00 arter97-ampere kernel: [1950437.732735] x19: ffffff84ab19aa00 x18: 0000000000000000 
May  6 16:11:00 arter97-ampere kernel: [1950437.732736] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:11:00 arter97-ampere kernel: [1950437.732736] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:11:00 arter97-ampere kernel: [1950437.732737] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:11:00 arter97-ampere kernel: [1950437.732737] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:11:00 arter97-ampere kernel: [1950437.732738] x9 : 00000000253f0aac x8 : 000000000000004e 
May  6 16:11:00 arter97-ampere kernel: [1950437.732738] x7 : 0000000000000080 x6 : 0000000000000006 
May  6 16:11:00 arter97-ampere kernel: [1950437.732739] x5 : ffffff84ab19ab00 x4 : 0000000000008a34 
May  6 16:11:00 arter97-ampere kernel: [1950437.732739] x3 : 00000000001fffff x2 : 0000000000000000 
May  6 16:11:00 arter97-ampere kernel: [1950437.732740] x1 : 0000000000000000 x0 : ffffffff17a51e80 
May  6 16:11:00 arter97-ampere kernel: [1950437.732741] Call trace:
May  6 16:11:00 arter97-ampere kernel: [1950437.732742]  kfree+0x44/0x238
May  6 16:12:13 arter97-ampere kernel: [1950509.994642] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:12:13 arter97-ampere kernel: [1950509.998005] rcu: 	1-....: (150339 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=74995 
May  6 16:12:13 arter97-ampere kernel: [1950509.999143] 	(t=150009 jiffies g=39741721 q=3325)
May  6 16:12:13 arter97-ampere kernel: [1950509.999143] Task dump for CPU 1:
May  6 16:12:13 arter97-ampere kernel: [1950509.999144] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:12:13 arter97-ampere kernel: [1950509.999146] Call trace:
May  6 16:12:13 arter97-ampere kernel: [1950509.999150]  dump_backtrace+0x0/0x160
May  6 16:12:13 arter97-ampere kernel: [1950509.999152]  sched_show_task+0x110/0x13c
May  6 16:12:13 arter97-ampere kernel: [1950509.999155]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:12:13 arter97-ampere kernel: [1950509.999156]  mptcp_data_ready+0x550/0x1ad8
May  6 16:13:56 arter97-ampere kernel: [1950613.730100] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:13:56 arter97-ampere kernel: [1950613.734286] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:13:56 arter97-ampere kernel: [1950613.734312] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:13:56 arter97-ampere kernel: [1950613.734312] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:13:56 arter97-ampere kernel: [1950613.734313] pstate: 00400005 (nzcv daif +PAN -UAO)
May  6 16:13:56 arter97-ampere kernel: [1950613.734317] pc : skb_split+0x4c/0x2b4
May  6 16:13:56 arter97-ampere kernel: [1950613.734319] lr : mptcp_data_ready+0xed0/0x1ad8
May  6 16:13:56 arter97-ampere kernel: [1950613.734319] sp : ffffff85d0297930
May  6 16:13:56 arter97-ampere kernel: [1950613.734320] x29: ffffff85d0297ff0 x28: 0000000000000034 
May  6 16:13:56 arter97-ampere kernel: [1950613.734320] x27: ffffff85d5a49bc0 x26: ffffff84ab19ab28 
May  6 16:13:56 arter97-ampere kernel: [1950613.734321] x25: ffffff85d5a49c90 x24: 0000000000010000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734321] x23: ffffff85ce6040c0 x22: 0000000000010000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734322] x21: 0000000000000000 x20: ffffff84ab19aa00 
May  6 16:13:56 arter97-ampere kernel: [1950613.734322] x19: ffffff84ab19ab00 x18: 0000000000000000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734323] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:13:56 arter97-ampere kernel: [1950613.734323] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:13:56 arter97-ampere kernel: [1950613.734324] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:13:56 arter97-ampere kernel: [1950613.734324] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:13:56 arter97-ampere kernel: [1950613.734325] x9 : 0000000000000000 x8 : 0000000000000300 
May  6 16:13:56 arter97-ampere kernel: [1950613.734325] x7 : 0000000000000000 x6 : 00000000000000c0 
May  6 16:13:56 arter97-ampere kernel: [1950613.734325] x5 : 0000000000000000 x4 : ffffff85f1b6a6c0 
May  6 16:13:56 arter97-ampere kernel: [1950613.734326] x3 : 0000000000000000 x2 : 0000000000000000 
May  6 16:13:56 arter97-ampere kernel: [1950613.734327] x1 : ffffff84ab19ab00 x0 : ffffff84ab19aa00 
May  6 16:13:56 arter97-ampere kernel: [1950613.734327] Call trace:
May  6 16:13:56 arter97-ampere kernel: [1950613.734328]  skb_split+0x4c/0x2b4
May  6 16:15:13 arter97-ampere kernel: [1950690.005700] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:15:13 arter97-ampere kernel: [1950690.013846] rcu: 	1-....: (195341 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=97495 
May  6 16:15:13 arter97-ampere kernel: [1950690.014720] 	(t=195012 jiffies g=39741721 q=3771)
May  6 16:15:13 arter97-ampere kernel: [1950690.014721] Task dump for CPU 1:
May  6 16:15:13 arter97-ampere kernel: [1950690.014722] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:15:13 arter97-ampere kernel: [1950690.014724] Call trace:
May  6 16:15:13 arter97-ampere kernel: [1950690.014728]  dump_backtrace+0x0/0x160
May  6 16:15:13 arter97-ampere kernel: [1950690.014730]  sched_show_task+0x110/0x13c
May  6 16:15:13 arter97-ampere kernel: [1950690.014732]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:15:13 arter97-ampere kernel: [1950690.014734]  skb_split+0x19c/0x2b4
May  6 16:16:52 arter97-ampere kernel: [1950789.729181] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1740]
May  6 16:16:52 arter97-ampere kernel: [1950789.738604] Modules linked in: xt_geoip(O) xt_set xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_ce ghash_ce sha2_ce fb_sys_fops sha256_arm64 sha1_ce drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
May  6 16:16:52 arter97-ampere kernel: [1950789.738660] CPU: 1 PID: 1740 Comm: tokio-runtime-w Tainted: G        W  O L    5.4.240-arter97+ #1
May  6 16:16:52 arter97-ampere kernel: [1950789.738660] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
May  6 16:16:52 arter97-ampere kernel: [1950789.738661] pstate: a0400005 (NzCv daif +PAN -UAO)
May  6 16:16:52 arter97-ampere kernel: [1950789.738666] pc : __kmalloc_track_caller+0xc4/0x280
May  6 16:16:52 arter97-ampere kernel: [1950789.738667] lr : __kmalloc_track_caller+0x4c/0x280
May  6 16:16:52 arter97-ampere kernel: [1950789.738667] sp : ffffff85d02978d0
May  6 16:16:52 arter97-ampere kernel: [1950789.738667] x29: ffffff85d0297ff0 x28: 0000000000000034 
May  6 16:16:52 arter97-ampere kernel: [1950789.738668] x27: ffffff85d5a49bc0 x26: ffffffc0111aa300 
May  6 16:16:52 arter97-ampere kernel: [1950789.738668] x25: 0000000000000a20 x24: ffffff84ab19aa00 
May  6 16:16:52 arter97-ampere kernel: [1950789.738669] x23: ffffffc0111aa300 x22: 0000000000082a20 
May  6 16:16:52 arter97-ampere kernel: [1950789.738669] x21: 0000000000000180 x20: ffffff85f1b6b800 
May  6 16:16:52 arter97-ampere kernel: [1950789.738670] x19: ffffff85f8b42c00 x18: 0000000000000000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738670] x17: 0000000000000000 x16: ffffffc010ec40c0 
May  6 16:16:52 arter97-ampere kernel: [1950789.738671] x15: ffffff845377804e x14: 0fff882eb3961a8d 
May  6 16:16:52 arter97-ampere kernel: [1950789.738671] x13: 000000000000144e x12: ffffffff1296a200 
May  6 16:16:52 arter97-ampere kernel: [1950789.738672] x11: 00000000253ea0ac x10: 87c352c0253f0aac 
May  6 16:16:52 arter97-ampere kernel: [1950789.738672] x9 : ffffff84c02b3680 x8 : 0000000000010000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738673] x7 : 0000000000000080 x6 : 000000029cb76c4f 
May  6 16:16:52 arter97-ampere kernel: [1950789.738673] x5 : 00000002718b409f x4 : ffffff85fff9efa0 
May  6 16:16:52 arter97-ampere kernel: [1950789.738674] x3 : 00000002718b40a0 x2 : 0000000000000000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738674] x1 : 0000000000000000 x0 : 0000000000000000 
May  6 16:16:52 arter97-ampere kernel: [1950789.738675] Call trace:
May  6 16:16:52 arter97-ampere kernel: [1950789.738676]  __kmalloc_track_caller+0xc4/0x280
May  6 16:18:13 arter97-ampere kernel: [1950870.016760] rcu: INFO: rcu_sched self-detected stall on CPU
May  6 16:18:13 arter97-ampere kernel: [1950870.027787] rcu: 	1-....: (240341 ticks this GP) idle=b42/1/0x4000000000000002 softirq=12945615/12945616 fqs=119993 
May  6 16:18:13 arter97-ampere kernel: [1950870.028576] 	(t=240015 jiffies g=39741721 q=4222)
May  6 16:18:13 arter97-ampere kernel: [1950870.028577] Task dump for CPU 1:
May  6 16:18:13 arter97-ampere kernel: [1950870.028578] tokio-runtime-w R  running task        0  1740   1724 0x00000002
May  6 16:18:13 arter97-ampere kernel: [1950870.028582] Call trace:
May  6 16:18:13 arter97-ampere kernel: [1950870.028586]  dump_backtrace+0x0/0x160
May  6 16:18:13 arter97-ampere kernel: [1950870.028588]  sched_show_task+0x110/0x13c
May  6 16:18:13 arter97-ampere kernel: [1950870.028591]  rcu_dump_cpu_stacks+0x9c/0xdc
May  6 16:18:13 arter97-ampere kernel: [1950870.028595]  __kmalloc_track_caller+0xc4/0x280

This is the MPTCP's configuration:

root@arter97-ampere:/proc/sys/net/mptcp# grep . *
mptcp_checksum:1
mptcp_debug:0
mptcp_enabled:1
mptcp_path_manager:fullmesh
mptcp_scheduler:default
mptcp_syn_retries:3
mptcp_version:0

Thanks in advance.
@matttbe
Copy link
Member

matttbe commented Jun 2, 2023

Hi,

Thank you for the bug report.

Sorry, I was off and I missed the notification.

Do you mind executing ./scripts/decode_stacktrace.sh on the stacktrace you shared (and maybe share the source code if the lines doesn't correspond to what we have here)

I have to admit that there is not so much info in the stacktrace (maybe there are debug kernel options that could help?) + no easy way to reproduce it, it doesn't look good :-/

@arter97
Copy link
Contributor Author

arter97 commented Jun 4, 2023

Hi @matttbe, I've managed to capture a decoded stacktrace.

[20299.002613] rcu: INFO: rcu_sched self-detected stall on CPU
[20299.023487] rcu:      1-....: (15169 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=7191
[20299.027940]   (t=15007 jiffies g=266325 q=301)
[20299.027941] Task dump for CPU 1:
[20299.027942] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20299.027944] Call trace:
[20299.027968] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20299.027974] sched_show_task (kernel/sched/core.c:6101)
[20299.027976] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20299.027980] skb_split (net/core/skbuff.c:3242)
[20399.922033] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20399.924383] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20399.924495] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O      5.4.244-arter97+ #1
[20399.924495] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20399.924496] pstate: 80400005 (Nzcv daif +PAN -UAO)
[20399.924500] pc : skb_split (net/core/skbuff.c:3248 (discriminator 2))
[20399.924500] lr : skb_split (net/core/skbuff.c:3286)
[20399.924501] sp : ffffff85cf35f940
[20399.924501] x29: ffffff85cf35fff0 x28: 0000000000000034
[20399.924502] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[20399.924502] x25: ffffff85d6719c90 x24: 0000000000010000
[20399.924503] x23: 0000000000000000 x22: 0000000000000000
[20399.924503] x21: 0000000000000000 x20: ffffff85a1daf300
[20399.924504] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[20399.924504] x17: 0000000000000001 x16: ffffffc010edad60
[20399.924505] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20399.924505] x13: 000039ce00003cc0 x12: ffffffff16262400
[20399.924506] x11: 00000000038382bf x10: 00000000000000c0
[20399.924511] x9 : 0000000000000001 x8 : 0000000000000008
[20399.924512] x7 : 0000000000000040 x6 : 0000000000000001
[20399.924512] x5 : 0000000000000972 x4 : 0000000000000001
[20399.924513] x3 : 0000000000003096 x2 : ffffffff16177a00
[20399.924513] x1 : ffffff85ccc62ec0 x0 : 0000000000003a08
[20399.924514] Call trace:
[20399.924515] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[20479.013589] rcu: INFO: rcu_sched self-detected stall on CPU
[20479.025167] rcu:      1-....: (60167 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=28163
[20479.026135]   (t=60007 jiffies g=266325 q=1128)
[20479.026136] Task dump for CPU 1:
[20479.026137] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20479.026139] Call trace:
[20479.026143] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20479.026145] sched_show_task (kernel/sched/core.c:6101)
[20479.026147] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20479.026156] kfree (mm/slub.c:4018 (discriminator 2))
[20575.921070] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20575.935785] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20575.935941] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[20575.935941] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20575.935942] pstate: 80400005 (Nzcv daif +PAN -UAO)
[20575.935946] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:932 (discriminator 1))
[20575.935947] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[20575.935948] sp : ffffff85cf35f980
[20575.935948] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[20575.935949] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[20575.935949] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[20575.935950] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[20575.935950] x21: 000000007cd21279 x20: ffffff85a1daf300
[20575.935951] x19: ffffff85d6719c90 x18: ffffff85d31c7720
[20575.935951] x17: 0000000000000001 x16: ffffffc010edad60
[20575.935952] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20575.935952] x13: 000039ce00003cc0 x12: ffffffff16262400
[20575.935953] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[20575.935953] x9 : ffffff85d1b5f080 x8 : 000000007cd21279
[20575.935954] x7 : 00000000000000a0 x6 : 0000000103eaed34
[20575.935954] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[20575.935955] x3 : 000000007cd21279 x2 : ffffff85d1b5f080
[20575.935955] x1 : 0000000000000000 x0 : 000000007cd31279
[20575.935956] Call trace:
[20575.935957] mptcp_data_ready (net/mptcp/mptcp_input.c:932 (discriminator 1))
[20659.024622] rcu: INFO: rcu_sched self-detected stall on CPU
[20659.026880] rcu:      1-....: (105165 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=48876
[20659.027610]   (t=105007 jiffies g=266325 q=1683)
[20659.027610] Task dump for CPU 1:
[20659.027611] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20659.027613] Call trace:
[20659.027617] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20659.027620] sched_show_task (kernel/sched/core.c:6101)
[20659.027622] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20659.027624] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[20767.920032] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20767.929994] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20767.934512] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[20767.934516] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20767.934517] pstate: 40400005 (nZcv daif +PAN -UAO)
[20767.934531] pc : sock_rfree (include/net/sock.h:1469 (discriminator 1))
[20767.934532] lr : skb_release_head_state (include/linux/skbuff.h:4090)
[20767.934532] sp : ffffff85cf35f960
[20767.934533] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[20767.934533] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[20767.934534] x25: ffffff85d6719c90 x24: ffffff85a1dafd00
[20767.934534] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[20767.934535] x21: 000000000383f80f x20: ffffff85d31ed400
[20767.934535] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[20767.934536] x17: 0000000000000001 x16: ffffffc010edad60
[20767.934536] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20767.934537] x13: 000039ce00003cc0 x12: ffffffff16262400
[20767.934537] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[20767.934538] x9 : 000000000383f80f x8 : 000000000000004e
[20767.934538] x7 : 00000000000000a0 x6 : 0000000000000008
[20767.934539] x5 : ffffff85a1daf300 x4 : fe0fcb6f0383f80f
[20767.934540] x3 : ffffffc011aa6380 x2 : ffffffc0119d7238
[20767.934540] x1 : 0000000000000300 x0 : ffffff85d6719bc0
[20767.934541] Call trace:
[20767.934542] sock_rfree (include/net/sock.h:1469 (discriminator 1))
[20839.035645] rcu: INFO: rcu_sched self-detected stall on CPU
[20839.037368] rcu:      1-....: (150165 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=68832
[20839.038103]   (t=150010 jiffies g=266325 q=2155)
[20839.038103] Task dump for CPU 1:
[20839.038104] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[20839.038106] Call trace:
[20839.038110] dump_backtrace (arch/arm64/kernel/traps.c:86)
[20839.038116] sched_show_task (kernel/sched/core.c:6101)
[20839.038118] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[20839.038123] __kmalloc_track_caller (arch/arm64/include/asm/atomic_lse.h:317)
[20943.919071] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[20943.920924] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[20943.920955] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[20943.920956] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[20943.920957] pstate: 20400005 (nzCv daif +PAN -UAO)
[20943.920960] pc : skb_split (net/core/skbuff.c:3247)
[20943.920961] lr : skb_split (net/core/skbuff.c:3286)
[20943.920961] sp : ffffff85cf35f940
[20943.920962] x29: ffffff85cf35fff0 x28: 0000000000000034
[20943.920962] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[20943.920963] x25: ffffff85d6719c90 x24: 0000000000010000
[20943.920963] x23: 0000000000000000 x22: 0000000000000000
[20943.920964] x21: 0000000000000000 x20: ffffff85a1daf300
[20943.920964] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[20943.920965] x17: 0000000000000001 x16: ffffffc010edad60
[20943.920965] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[20943.920966] x13: 000039ce00003cc0 x12: ffffffff16262400
[20943.920966] x11: 00000000038382bf x10: 00000000000000c0
[20943.920967] x9 : 0000000000000001 x8 : 0000000000000008
[20943.920967] x7 : 0000000000000090 x6 : 0000000000000006
[20943.920968] x5 : 000000000000c1c8 x4 : 0000000000000006
[20943.920968] x3 : ffffff85ccc62f50 x2 : 0000000000002232
[20943.920969] x1 : ffffff85ccc62ec0 x0 : 000000000000c1c8
[20943.920970] Call trace:
[20943.920970] skb_split (net/core/skbuff.c:3247)
[21019.046661] rcu: INFO: rcu_sched self-detected stall on CPU
[21019.048544] rcu:      1-....: (195168 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=88349
[21019.049660]   (t=195013 jiffies g=266325 q=2419)
[21019.049661] Task dump for CPU 1:
[21019.049662] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21019.049664] Call trace:
[21019.049668] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21019.049670] sched_show_task (kernel/sched/core.c:6101)
[21019.049672] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21019.049674] mptcp_data_ready (net/mptcp/mptcp_input.c:689)
[21119.918105] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[21119.920736] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21119.920778] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21119.920779] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21119.920780] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21119.920783] pc : skb_split (net/core/skbuff.c:3248 (discriminator 2))
[21119.920784] lr : skb_split (net/core/skbuff.c:3286)
[21119.920784] sp : ffffff85cf35f940
[21119.920784] x29: ffffff85cf35fff0 x28: 0000000000000034
[21119.920785] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21119.920786] x25: ffffff85d6719c90 x24: 0000000000010000
[21119.920786] x23: 0000000000000000 x22: 0000000000000000
[21119.920787] x21: 0000000000000000 x20: ffffff85a1dafd00
[21119.920787] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[21119.920787] x17: 0000000000000001 x16: ffffffc010edad60
[21119.920788] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21119.920788] x13: 000039ce00003cc0 x12: ffffffff16262400
[21119.920789] x11: 00000000038382bf x10: 00000000000000c0
[21119.920789] x9 : 0000000000000001 x8 : 0000000000000008
[21119.920790] x7 : 0000000000000060 x6 : 0000000000000003
[21119.920790] x5 : 000000000000803a x4 : 0000000000000003
[21119.920791] x3 : ffffff85ccc63520 x2 : 0000000000000356
[21119.920791] x1 : ffffff85ccc634c0 x0 : 0000000000008390
[21119.920792] Call trace:
[21119.920793] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[21199.057667] rcu: INFO: rcu_sched self-detected stall on CPU
[21199.061108] rcu:      1-....: (240171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=108570
[21199.066967]   (t=240018 jiffies g=266325 q=2750)
[21199.066967] Task dump for CPU 1:
[21199.066969] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21199.066971] Call trace:
[21199.066975] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21199.066977] sched_show_task (kernel/sched/core.c:6101)
[21199.066979] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21199.066981] kmem_cache_free (arch/arm64/include/asm/atomic_lse.h:317)
[21295.917138] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21295.922540] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21295.922568] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21295.922569] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21295.922570] pstate: 00400005 (nzcv daif +PAN -UAO)
[21295.922573] pc : skb_split (net/core/skbuff.c:3280)
[21295.922576] lr : mptcp_data_ready (include/linux/skbuff.h:1988)
[21295.922577] sp : ffffff85cf35f940
[21295.922577] x29: ffffff85cf35fff0 x28: 0000000000000034
[21295.922577] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21295.922578] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[21295.922578] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[21295.922579] x21: 0000000000000000 x20: ffffff85a1dafd00
[21295.922579] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[21295.922580] x17: 0000000000000001 x16: ffffffc010edad60
[21295.922580] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21295.922581] x13: 000039ce00003cc0 x12: ffffffff16262400
[21295.922581] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21295.922582] x9 : 0000000000000000 x8 : 0000000000000300
[21295.922582] x7 : 0000000000000000 x6 : 000000000000003f
[21295.922583] x5 : 0000000000000040 x4 : 0000000000000000
[21295.922583] x3 : 000000007cd31279 x2 : 0000000000000000
[21295.922584] x1 : ffffff85a1daf300 x0 : ffffff85a1daf300
[21295.922585] Call trace:
[21295.922585] skb_split (net/core/skbuff.c:3280)
[21379.068677] rcu: INFO: rcu_sched self-detected stall on CPU
[21379.071036] rcu:      1-....: (285172 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=129306
[21379.072911]   (t=285020 jiffies g=266325 q=3248)
[21379.072912] Task dump for CPU 1:
[21379.072913] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21379.072915] Call trace:
[21379.072931] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21379.072934] sched_show_task (kernel/sched/core.c:6101)
[21379.072937] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21379.072939] __memset (arch/arm64/lib/memset.S:153)
[21487.916072] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21487.918185] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21487.918213] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21487.918214] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21487.918215] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21487.918219] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:907)
[21487.918220] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[21487.918220] sp : ffffff85cf35f980
[21487.918221] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[21487.918221] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21487.918222] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[21487.918222] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[21487.918223] x21: ffffff85d31ed400 x20: ffffff85cf36ef00
[21487.918223] x19: ffffff85d6719c90 x18: ffffff85d31c7720
[21487.918224] x17: 0000000000000001 x16: ffffffc010edad60
[21487.918224] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21487.918225] x13: 000039ce00003cc0 x12: ffffffff16262400
[21487.918225] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21487.918226] x9 : ffffff85d1b5f080 x8 : 000000007cd21279
[21487.918227] x7 : 00000000000000a0 x6 : 00000003bed4acf8
[21487.918227] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[21487.918228] x3 : 000000007cd21279 x2 : 0000000000000018
[21487.918228] x1 : 000000000000004e x0 : ffffff85d1b5f080
[21487.918229] Call trace:
[21487.918230] mptcp_data_ready (net/mptcp/mptcp_input.c:907)
[21559.079678] rcu: INFO: rcu_sched self-detected stall on CPU
[21559.081909] rcu:      1-....: (330175 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=150025
[21559.083604]   (t=330022 jiffies g=266325 q=3504)
[21559.083605] Task dump for CPU 1:
[21559.083606] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21559.083608] Call trace:
[21559.083612] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21559.083615] sched_show_task (kernel/sched/core.c:6101)
[21559.083617] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21559.083618] skb_split (net/core/skbuff.c:3239 (discriminator 1))
[21663.915098] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21663.929143] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21663.929171] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21663.929172] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21663.929172] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21663.929176] pc : skb_release_data (include/linux/skbuff.h:1388)
[21663.929177] lr : __kfree_skb (net/core/skbuff.c:630)
[21663.929177] sp : ffffff85cf35f940
[21663.929177] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[21663.929178] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[21663.929178] x25: ffffff85d6719c90 x24: ffffff85a1dafd00
[21663.929179] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[21663.929179] x21: 000000000383f80f x20: ffffff85ccc62e00
[21663.929180] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[21663.929180] x17: 0000000000000001 x16: ffffffc010edad60
[21663.929181] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21663.929181] x13: 000039ce00003cc0 x12: ffffffff16262400
[21663.929182] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21663.929182] x9 : 000000000383f80f x8 : 000000000000004e
[21663.929183] x7 : 00000000000000a0 x6 : 0000000000000008
[21663.929183] x5 : ffffff85a1daf300 x4 : 0000000000009a34
[21663.929184] x3 : 00000000001fffff x2 : ffffffc0119d7238
[21663.929184] x1 : 00000000000000c0 x0 : 0000000000000000
[21663.929185] Call trace:
[21663.929185] skb_release_data (include/linux/skbuff.h:1388)
[21739.090679] rcu: INFO: rcu_sched self-detected stall on CPU
[21739.106352] rcu:      1-....: (375175 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=171058
[21739.107298]   (t=375029 jiffies g=266325 q=3956)
[21739.107298] Task dump for CPU 1:
[21739.107299] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21739.107301] Call trace:
[21739.107304] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21739.107307] sched_show_task (kernel/sched/core.c:6101)
[21739.107309] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21739.107311] mptcp_data_ready (include/net/sock.h:1423)
[21839.914120] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[21839.930519] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[21839.930570] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[21839.930570] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[21839.930571] pstate: 80400005 (Nzcv daif +PAN -UAO)
[21839.930574] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:1014)
[21839.930575] lr : mptcp_data_ready (include/linux/skbuff.h:1988)
[21839.930576] sp : ffffff85cf35f980
[21839.930576] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[21839.930577] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[21839.930577] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[21839.930578] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[21839.930578] x21: 000000000383f80f x20: ffffff85d31ed400
[21839.930579] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[21839.930579] x17: 0000000000000001 x16: ffffffc010edad60
[21839.930579] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[21839.930580] x13: 000039ce00003cc0 x12: ffffffff16262400
[21839.930580] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[21839.930581] x9 : 000000000383f80f x8 : 000000000000004e
[21839.930582] x7 : 00000000000000a0 x6 : 0000000000000008
[21839.930582] x5 : ffffff85a1dafd00 x4 : fe0fcb6f0383f80f
[21839.930583] x3 : ffffff85d6719c90 x2 : ffffff85a1daf328
[21839.930583] x1 : 000000007cd21279 x0 : ffffff85a1daf300
[21839.930584] Call trace:
[21839.930585] mptcp_data_ready (net/mptcp/mptcp_input.c:1014)
[21919.101680] rcu: INFO: rcu_sched self-detected stall on CPU
[21919.110663] rcu:      1-....: (420171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=190960
[21919.111406]   (t=420030 jiffies g=266325 q=4335)
[21919.111407] Task dump for CPU 1:
[21919.111407] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[21919.111409] Call trace:
[21919.111413] dump_backtrace (arch/arm64/kernel/traps.c:86)
[21919.111415] sched_show_task (kernel/sched/core.c:6101)
[21919.111416] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[21919.111419] kmem_cache_alloc (mm/slub.c:2823)
[22015.913142] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22015.924876] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22015.925026] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22015.925027] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22015.925028] pstate: 80400005 (Nzcv daif +PAN -UAO)
[22015.925031] pc : skb_split (net/core/skbuff.c:3244 (discriminator 2))
[22015.925032] lr : skb_split (net/core/skbuff.c:3286)
[22015.925032] sp : ffffff85cf35f940
[22015.925032] x29: ffffff85cf35fff0 x28: 0000000000000034
[22015.925033] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22015.925033] x25: ffffff85d6719c90 x24: 0000000000010000
[22015.925034] x23: 0000000000000000 x22: 0000000000000000
[22015.925035] x21: 0000000000000000 x20: ffffff85a1dafd00
[22015.925035] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[22015.925036] x17: 0000000000000001 x16: ffffffc010edad60
[22015.925036] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22015.925037] x13: 000039ce00003cc0 x12: ffffffff16262400
[22015.925038] x11: 00000000038382bf x10: 00000000000000c0
[22015.925038] x9 : 0000000000000001 x8 : 0000000000000008
[22015.925039] x7 : 0000000000000070 x6 : 0000000000000005
[22015.925039] x5 : 0000000000008390 x4 : 0000000000000004
[22015.925040] x3 : 00000bce00002d40 x2 : ffffffff15ec9c00
[22015.925040] x1 : ffffff85ccc62f00 x0 : 000000000000b0d0
[22015.925041] Call trace:
[22015.925042] skb_split (net/core/skbuff.c:3244 (discriminator 2))
[22099.112680] rcu: INFO: rcu_sched self-detected stall on CPU
[22099.124232] rcu:      1-....: (465171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=211501
[22099.129635]   (t=465034 jiffies g=266325 q=5253)
[22099.129636] Task dump for CPU 1:
[22099.129637] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22099.129803] Call trace:
[22099.129963] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22099.129968] sched_show_task (kernel/sched/core.c:6101)
[22099.129970] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22099.129974] kmem_cache_free (include/linux/compiler.h:268 (discriminator 5))
[22207.912076] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22207.913725] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22207.913754] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22207.913755] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22207.913756] pstate: 20400005 (nzCv daif +PAN -UAO)
[22207.913771] pc : skb_split (include/linux/skbuff.h:1388)
[22207.913771] lr : skb_split (net/core/skbuff.c:3286)
[22207.913772] sp : ffffff85cf35f940
[22207.913772] x29: ffffff85cf35fff0 x28: 0000000000000034
[22207.913773] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22207.913773] x25: ffffff85d6719c90 x24: 0000000000010000
[22207.913774] x23: 0000000000000000 x22: 0000000000000000
[22207.913774] x21: 0000000000000000 x20: ffffff85a1dafd00
[22207.913775] x19: ffffff85a1daf300 x18: ffffff85d31c7720
[22207.913775] x17: 0000000000000001 x16: ffffffc010edad60
[22207.913775] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22207.913776] x13: 000039ce00003cc0 x12: ffffffff16262400
[22207.913776] x11: 00000000038382bf x10: 00000000000000c0
[22207.913777] x9 : 0000000000000001 x8 : 0000000000000008
[22207.913777] x7 : 0000000000000070 x6 : 0000000000000005
[22207.913778] x5 : 000000000000b0d0 x4 : 0000000000000005
[22207.913778] x3 : 0000000000000008 x2 : ffffffff15ec9c00
[22207.913779] x1 : ffffff85ccc63400 x0 : 000000000000b0d0
[22207.913780] Call trace:
[22207.913780] skb_split (include/linux/skbuff.h:1388)
[22279.123679] rcu: INFO: rcu_sched self-detected stall on CPU
[22279.124669] rcu:      1-....: (510171 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=231776
[22279.125441]   (t=510034 jiffies g=266325 q=5520)
[22279.125442] Task dump for CPU 1:
[22279.125442] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22279.125444] Call trace:
[22279.125448] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22279.125450] sched_show_task (kernel/sched/core.c:6101)
[22279.125452] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22279.125455] __kmalloc_track_caller (arch/arm64/include/asm/atomic_lse.h:317)
[22383.911097] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22383.912262] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22383.912291] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22383.912292] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22383.912293] pstate: 80400005 (Nzcv daif +PAN -UAO)
[22383.912298] pc : mptcp_data_ready (net/mptcp/mptcp_input.c:688)
[22383.912299] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[22383.912300] sp : ffffff85cf35f980
[22383.912300] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[22383.912300] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22383.912301] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[22383.912301] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[22383.912302] x21: ffffff85d31ed400 x20: ffffff85cf36ef00
[22383.912302] x19: ffffff85d6719c90 x18: ffffff85d31c7720
[22383.912303] x17: 0000000000000001 x16: ffffffc010edad60
[22383.912303] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22383.912304] x13: 000039ce00003cc0 x12: ffffffff16262400
[22383.912304] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[22383.912305] x9 : ffffff85d1b5f080 x8 : 000000007cd21279
[22383.912305] x7 : 00000000000000a0 x6 : 000000066e62dd58
[22383.912306] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[22383.912306] x3 : 000000007cd21279 x2 : 0000000000000000
[22383.912307] x1 : 0000000000010000 x0 : ffffff85d1b5f080
[22383.912308] Call trace:
[22383.912309] mptcp_data_ready (net/mptcp/mptcp_input.c:688)
[22459.134678] rcu: INFO: rcu_sched self-detected stall on CPU
[22459.135619] rcu:      1-....: (555174 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=251634
[22459.136475]   (t=555037 jiffies g=266325 q=6012)
[22459.136476] Task dump for CPU 1:
[22459.136477] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22459.136479] Call trace:
[22459.136483] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22459.136485] sched_show_task (kernel/sched/core.c:6101)
[22459.136487] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22459.136490] kmem_cache_alloc (mm/slub.c:2772)
[22559.910119] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22559.911203] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22559.911230] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22559.911230] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22559.911231] pstate: 20400005 (nzCv daif +PAN -UAO)
[22559.911235] pc : skb_split (net/core/skbuff.c:3247)
[22559.911235] lr : skb_split (net/core/skbuff.c:3286)
[22559.911236] sp : ffffff85cf35f940
[22559.911236] x29: ffffff85cf35fff0 x28: 0000000000000034
[22559.911237] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[22559.911237] x25: ffffff85d6719c90 x24: 0000000000010000
[22559.911238] x23: 0000000000000000 x22: 0000000000000000
[22559.911238] x21: 0000000000000000 x20: ffffff85a1daf300
[22559.911239] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[22559.911239] x17: 0000000000000001 x16: ffffffc010edad60
[22559.911240] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22559.911240] x13: 000039ce00003cc0 x12: ffffffff16262400
[22559.911241] x11: 00000000038382bf x10: 00000000000000c0
[22559.911241] x9 : 0000000000000001 x8 : 0000000000000008
[22559.911242] x7 : 0000000000000060 x6 : 0000000000000003
[22559.911242] x5 : 000000000000803a x4 : 0000000000000003
[22559.911243] x3 : ffffff85ccc62f20 x2 : 0000000000000356
[22559.911243] x1 : ffffff85ccc62ec0 x0 : 000000000000803a
[22559.911244] Call trace:
[22559.911245] skb_split (net/core/skbuff.c:3247)
[22639.145676] rcu: INFO: rcu_sched self-detected stall on CPU
[22639.146553] rcu:      1-....: (600177 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=272376
[22639.147309]   (t=600040 jiffies g=266325 q=6479)
[22639.147310] Task dump for CPU 1:
[22639.147311] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22639.147313] Call trace:
[22639.147316] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22639.147318] sched_show_task (kernel/sched/core.c:6101)
[22639.147321] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22639.147323] mptcp_data_ready (net/mptcp/mptcp_input.c:1019)
[22735.909139] watchdog: BUG: soft lockup - CPU#1 stuck for 89s! [tokio-runtime-w:1689]
[22735.910153] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22735.910189] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22735.910190] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22735.910191] pstate: 60400005 (nZCv daif +PAN -UAO)
[22735.910195] pc : kmem_cache_free (arch/arm64/include/asm/atomic_lse.h:317)
[22735.910197] lr : mptcp_data_ready (net/mptcp/mptcp_input.c:1017 (discriminator 1))
[22735.910198] sp : ffffff85cf35f950
[22735.910198] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[22735.910199] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22735.910199] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[22735.910200] x23: ffffff85cf36ef00 x22: ffffffc0111ce85c
[22735.910200] x21: ffffffff16676bc0 x20: ffffff85a1daf300
[22735.910201] x19: ffffff85ff75b200 x18: ffffff85d31c7720
[22735.910201] x17: 0000000000000001 x16: ffffffc010edad60
[22735.910202] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22735.910202] x13: 000039ce00003cc0 x12: ffffffff16262400
[22735.910202] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[22735.910203] x9 : 000000000383f80f x8 : 000000000000004e
[22735.910203] x7 : 00000000000000a0 x6 : 000000077c3109fe
[22735.910204] x5 : ffffff85a1daf100 x4 : ffffff85fffa05a0
[22735.910204] x3 : 000000077c3109ff x2 : ffffff85a1daf300
[22735.910205] x1 : 000000077c3109fe x0 : ffffff85a1daf100
[22735.910206] Call trace:
[22735.910207] kmem_cache_free (arch/arm64/include/asm/atomic_lse.h:317)
[22819.156674] rcu: INFO: rcu_sched self-detected stall on CPU
[22819.157603] rcu:      1-....: (645180 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=292064
[22819.158443]   (t=645043 jiffies g=266325 q=6921)
[22819.158444] Task dump for CPU 1:
[22819.158444] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22819.158446] Call trace:
[22819.158450] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22819.158452] sched_show_task (kernel/sched/core.c:6101)
[22819.158455] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22819.158456] skb_split (net/core/skbuff.c:3248 (discriminator 2))
[22927.908071] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[22927.911116] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[22927.911143] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[22927.911143] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[22927.911144] pstate: 40400005 (nZcv daif +PAN -UAO)
[22927.911148] pc : kfree (mm/slub.c:3026)
[22927.911150] lr : __kfree_skb (net/core/skbuff.c:630)
[22927.911150] sp : ffffff85cf35f950
[22927.911150] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[22927.911151] x27: ffffff85d6719bc0 x26: ffffff85a1daf328
[22927.911151] x25: ffffff85d6719c90 x24: ffffff85a1daf300
[22927.911152] x23: ffffff85cf36ef00 x22: ffffff85a1dafd00
[22927.911152] x21: ffffffff17131880 x20: ffffff85ccc63400
[22927.911153] x19: ffffff85f8b42c00 x18: ffffff85d31c7720
[22927.911153] x17: 0000000000000001 x16: ffffffc010edad60
[22927.911154] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[22927.911154] x13: 000039ce00003cc0 x12: ffffffff16262400
[22927.911155] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[22927.911155] x9 : 000000000383f80f x8 : 000000000000004e
[22927.911156] x7 : 00000000000000a0 x6 : 0000000000000008
[22927.911156] x5 : ffffff85a1dafd00 x4 : 0000000000009a34
[22927.911157] x3 : 00000000001fffff x2 : ffffff85cf35f950
[22927.911157] x1 : 000000080e5aeaae x0 : ffffffc01183bfa8
[22927.911158] Call trace:
[22927.911159] kfree (mm/slub.c:3026)
[22999.167676] rcu: INFO: rcu_sched self-detected stall on CPU
[22999.171318] rcu:      1-....: (690183 ticks this GP) idle=83e/1/0x4000000000000002 softirq=94682/94684 fqs=311736
[22999.173046]   (t=690047 jiffies g=266325 q=7179)
[22999.173047] Task dump for CPU 1:
[22999.173048] tokio-runtime-w R  running task        0  1689   1668 0x0000000e
[22999.173050] Call trace:
[22999.173053] dump_backtrace (arch/arm64/kernel/traps.c:86)
[22999.173056] sched_show_task (kernel/sched/core.c:6101)
[22999.173058] rcu_dump_cpu_stacks (kernel/rcu/tree_stall.h:254)
[22999.173061] kfree (arch/arm64/include/asm/atomic_lse.h:317)
[23103.907091] watchdog: BUG: soft lockup - CPU#1 stuck for 90s! [tokio-runtime-w:1689]
[23103.920965] Modules linked in: xt_set xt_geoip(O) xt_multiport xt_conntrack ip_set_hash_ip ip_set tls binfmt_misc nft_counter nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink joydev qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor xor_neon raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net net_failover failover aes_neon_bs aes_neon_blk aes_ce_blk aes_ce_cipher
[23103.920992] CPU: 1 PID: 1689 Comm: tokio-runtime-w Tainted: G           O L    5.4.244-arter97+ #1
[23103.920993] Hardware name: QEMU KVM Virtual Machine, BIOS 1.4.1 12/03/2020
[23103.920994] pstate: 40400005 (nZcv daif +PAN -UAO)
[23103.920998] pc : kfree (include/linux/page-flags.h:182)
[23103.921000] lr : __kfree_skb (net/core/skbuff.c:630)
[23103.921000] sp : ffffff85cf35f950
[23103.921001] x29: ffffff85cf35fff0 x28: ffffff85cf36ef00
[23103.921001] x27: ffffff85d6719bc0 x26: ffffff85a1dafd28
[23103.921002] x25: ffffff85d6719c90 x24: ffffff85a1dafd00
[23103.921002] x23: ffffff85cf36ef00 x22: ffffff85a1daf300
[23103.921003] x21: ffffffff17131880 x20: ffffff85ccc62e00
[23103.921003] x19: ffffff85a1dafd00 x18: ffffff85d31c7720
[23103.921004] x17: 0000000000000001 x16: ffffffc010edad60
[23103.921004] x15: ffffff85d12ecfce x14: 92d93866c79e5930
[23103.921005] x13: 000039ce00003cc0 x12: ffffffff16262400
[23103.921006] x11: 00000000038382bf x10: fe0fcb6f0383f80f
[23103.921006] x9 : 000000000383f80f x8 : 000000000000004e
[23103.921007] x7 : 00000000000000a0 x6 : 0000000000000008
[23103.921007] x5 : ffffff85a1daf300 x4 : 0000000000009a34
[23103.921008] x3 : 00000000001fffff x2 : 0000000000000000
[23103.921008] x1 : 0000000000000000 x0 : ffffffff17202000
[23103.921009] Call trace:
[23103.921011] kfree (include/linux/page-flags.h:182)

The code is at https://github.com/arter97/x86-kernel/tree/5.4 (commit 9bc70a46b024621d1dc2a931940b9ac01ce6cbcf) and the defconfig is at https://github.com/arter97/x86-kernel/blob/5.4/arch/arm64/configs/oracle_defconfig.

(Ignore the repo name, it's misleading lol)

Thanks.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@matttbe @arter97