Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gfx906 (AMD MI60) is failing on run_and_save_benchmarks.sh and llama.cpp #180

Open
Said-Akbar opened this issue Nov 22, 2024 · 10 comments
Open

gfx906 (AMD MI60) is failing on run_and_save_benchmarks.sh and llama.cpp #180

Said-Akbar opened this issue Nov 22, 2024 · 10 comments

Comments

@Said-Akbar
Copy link

Said-Akbar commented Nov 22, 2024 •
edited
Loading

Hi @lamikr,

I built rocm_sdk_builder on a freshly installed Ubuntu 24.04.1. It took 5 hours, 120GB of storage and many hours of fixing small issues during building the repo (reference: #175).
Also, I chose gfx906 from ./babs.sh -c.

When I ran ./run_and_save_benchmarks.sh, I got this message.

./run_and_save_benchmarks.sh
Timestamp for benchmark results: 20241121_190404
Saving to file: 20241121_190404_cpu_vs_gpu_simple.txt
Benchmarking CPU and GPUs
Pytorch version: 2.4.1
ROCM HIP version: 6.1.40093-61a06a2f8
       Device:  AMD Ryzen 9 5950X 16-Core Processor
    'CPU time: 26.503 sec
       Device: AMD Radeon Graphics
    'GPU time: 0.399 sec
       Device: AMD Radeon Graphics
    'GPU time: 0.353 sec
Benchmark ready

Saving to file: 20241121_190404_pytorch_dot_products.txt
Pytorch version: 2.4.1
dot product calculation test
tensor([[[ 0.2042, -0.5683,  0.5711,  1.5666, -0.8859, -0.4255, -0.6103,
          -0.5932],
         [-0.1816, -1.0552,  0.3676,  2.1399, -0.8622,  0.1185, -0.4614,
          -0.4577],
         [ 0.2491, -0.5238,  0.5873,  1.5027, -0.8808, -0.4906, -0.6309,
          -0.6083]],

        [[-0.0812,  0.5027, -0.0134, -0.1771, -1.6389,  0.0154, -1.1964,
          -0.3948],
         [-0.3459, -0.4265,  0.0969,  0.0608, -0.9923, -0.4199, -0.7190,
          -0.0208],
         [-0.2615, -0.6958,  0.1066, -0.1948, -1.2152, -0.1223, -0.6278,
           0.1627]]], device='cuda:0')

Benchmarking cuda and cpu with Default, Math, Flash Attention amd Memory pytorch backends
Device: AMD Radeon Graphics / cuda:0
    Default benchmark:
:0:/home/saidp/Downloads/rocm_sdk_builder/src_projects/clr/hipamd/src/hip_global.cpp:114 : 8471950880 us: [pid:454884 tid:0x7ad2a9db0b80] Cannot find Symbol with name: Cijk_Alik_Bljk_HHS_BH_MT128x64x16_SE_APM1_AF0EM2_AF1EM1_AMAS3_ASAE01_ASCE01_ASEM2_BL1_BS1_DTLA0_DTLB0_EPS1_FL1_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA1_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT8_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG16_16_1_WGM1

Note the error at the bottom 'Cannot find Symbol with name'. I thought this would not be an issue with llama.cpp.
However, I got a similar error in llama.cpp as well (I built it using ./babs.sh -b binfo/extra/ai_tools.blist).

source /opt/rocm_sdk_612/bin/env_rocm.sh
llama-server -m /media/saidp/datasets/text_generation/models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf -c 2048 -ngl 99 --metrics
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 9.0, VMM: no
  Device 1: AMD Radeon Graphics, compute capability 9.0, VMM: no
build: 3901 (49f4671b) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 31
main: loading model
llama_model_loader: loaded meta data with 38 key-value pairs and 339 tensors from /media/saidp/datasets/text_generation/models/Qwen2.5-7B-Instruct-Q8_0/Qwen2.5-7B-Instruct-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 7B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5
llama_model_loader: - kv   5:                         general.size_label str              = 7B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-7...
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 7B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-7B
llama_model_loader: - kv  12:                               general.tags arr[str,2]       = ["chat", "text-generation"]
llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  14:                          qwen2.block_count u32              = 28
llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 3584
llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 18944
llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 28
llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                          general.file_type u32              = 7
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - kv  34:                      quantize.imatrix.file str              = /models_out/Qwen2.5-7B-Instruct-GGUF/...
llama_model_loader: - kv  35:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  36:             quantize.imatrix.entries_count i32              = 196
llama_model_loader: - kv  37:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type q8_0:  198 tensors
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.9310 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 3584
llm_load_print_meta: n_layer          = 28
llm_load_print_meta: n_head           = 28
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 7
llm_load_print_meta: n_embd_k_gqa     = 512
llm_load_print_meta: n_embd_v_gqa     = 512
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 18944
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 7.62 B
llm_load_print_meta: model size       = 7.54 GiB (8.50 BPW) 
llm_load_print_meta: general.name     = Qwen2.5 7B Instruct
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 'ÄĬ'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: ggml ctx size =    0.45 MiB
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  3542.78 MiB
llm_load_tensors:      ROCm1 buffer size =  3622.66 MiB
llm_load_tensors:        CPU buffer size =   552.23 MiB
......................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      ROCm0 KV buffer size =    60.00 MiB
llama_kv_cache_init:      ROCm1 KV buffer size =    52.00 MiB
llama_new_context_with_model: KV self size  =  112.00 MiB, K (f16):   56.00 MiB, V (f16):   56.00 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     1.16 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model:      ROCm0 compute buffer size =   184.01 MiB
llama_new_context_with_model:      ROCm1 compute buffer size =   348.02 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =    23.02 MiB
llama_new_context_with_model: graph nodes  = 986
llama_new_context_with_model: graph splits = 3
llama_init_from_gpt_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
:0:/home/saidp/Downloads/rocm_sdk_builder/src_projects/clr/hipamd/src/hip_global.cpp:114 : 10662878012 us: [pid:465832 tid:0x7268ce2a2c40] Cannot find Symbol with name: Cijk_Alik_Bljk_HB_GB_MT32x32x16_SN_APM1_AF0EM2_AF1EM2_AMAS3_ASAE01_ASCE01_ASEM1_BL1_BS1_DTLA0_DTLB0_EPS1_FL0_GLVWA4_GLVWB4_GRVW4_GSU1_GSUASB_ISA906_IU1_K1_KLA_LPA0_LPB0_LDL1_LRVW4_MDA2_MMFGLC_NLCA1_NLCB1_ONLL1_PK0_PGR1_PLR1_SIA1_SU32_SUM0_SUS256_SVW4_SNLL0_TT4_4_USFGROn1_VAW2_VSn1_VW4_VWB4_WG8_8_1_WGM1
Aborted (core dumped)

llama.cpp is failing with a similar error. Note that this llama.cpp worked with the CPU when I do not set the ngl parameter (layer offloading). Please let me know if there is a fix.
@Said-Akbar
Copy link
Author

@lamikr ,
That error line comes from https://github.com/ROCm/clr/blob/rocm-6.1.x/hipamd/src/hip_global.cpp#L114 .

But I am not sure how to fix my issue above. Please, let me know if you have time to review this today.
Thanks!

@lamikr
Copy link
Owner

lamikr commented Nov 24, 2024

Hi, unfortunately I do not have myself the gfx906 for debug, so I only added added some patches that would be needed at least to get it build and start testing and added it's support as an experimental.

About your error, I have not never seen that kind of error, but it could be some kind of misconfiguration in rocBLAS related to src_projects/rocBLAS/library/src/blas3/Tensile/Logic/asm_full/vega10/vega10_Cijk_Alik_Bljk_HB_GB.yaml

But let's try to check first couple of basic issues step by step so I get basic info.

  1. Can you paste me first the output of rocminfo command? I am interested in whether it detects your gpu
    and what information it shows from it.

  2. Then are you able to build and run this test these test apps:

/opt/rocm_sdk_612/docs/examples/hipcc/hello_world
/opt/rocm_sdk_612/docs/examples/opencl/check_opencl_caps

@Said-Akbar
Copy link
Author

Hello @lamikr,
Sure, here is the output of rocminfo.

rocminfo
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
Runtime Ext Version:     1.4
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 9 5950X 16-Core Processor
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 9 5950X 16-Core Processor
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3400                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            32                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    98773496(0x5e329f8) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    98773496(0x5e329f8) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    98773496(0x5e329f8) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx906                             
  Uuid:                    GPU-161620e172e17d3d               
  Marketing Name:          AMD Radeon Graphics                
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      8192(0x2000) KB                    
  Chip ID:                 26273(0x66a1)                      
  ASIC Revision:           1(0x1)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   1800                               
  BDFID:                   3328                               
  Internal Node ID:        1                                  
  Compute Unit:            64                                 
  SIMDs per CU:            4                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   1                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          64(0x40)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        40(0x28)                           
  Max Work-item Per CU:    2560(0xa00)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 471                                
  SDMA engine uCode::      145                                
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    33538048(0x1ffc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    33538048(0x1ffc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Recommended Granule:0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*******                  
Agent 3                  
*******                  
  Name:                    gfx906                             
  Uuid:                    GPU-915e294172fd62d2               
  Marketing Name:          AMD Radeon Graphics                
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    2                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      8192(0x2000) KB                    
  Chip ID:                 26273(0x66a1)                      
  ASIC Revision:           1(0x1)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   1800                               
  BDFID:                   4096                               
  Internal Node ID:        2                                  
  Compute Unit:            64                                 
  SIMDs per CU:            4                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   1                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          64(0x40)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        40(0x28)                           
  Max Work-item Per CU:    2560(0xa00)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 471                                
  SDMA engine uCode::      145                                
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    33538048(0x1ffc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    33538048(0x1ffc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Recommended Granule:2048KB                             
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Recommended Granule:0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***

@Said-Akbar
Copy link
Author

tests:

cd /opt/rocm_sdk_612/docs/examples/hipcc/hello_world/
./build.sh 
rm -f ./hello_world
rm -f hello_world.o
rm -f /opt/rocm_sdk_612/src/*.o
/opt/rocm_sdk_612/bin/hipcc -g -fPIE   -c -o hello_world.o hello_world.cpp
/opt/rocm_sdk_612/bin/hipcc hello_world.o -fPIE -o hello_world
./hello_world
 System minor: 0
 System major: 9
 Agent name: AMD Radeon Graphics
Kernel input: GdkknVnqkc
Expecting that kernel increases each character from input string by one
Kernel output string: HelloWorld
Output string matched with HelloWorld
Test ok!

@Said-Akbar
Copy link
Author

Opencl test:

cd /opt/rocm_sdk_612/docs/examples/opencl/check_opencl_caps
make
/check_opencl_caps 
number of opencl platform devices: 1
==============================
Platform id: 0
AMD Accelerated Parallel Processing
Advanced Micro Devices, Inc.
OpenCL 2.1 AMD-APP (3614.0)
FULL_PROFILE
cl_khr_icd cl_amd_event_callback 
Number of devices found for platform: 2
    ---------------------------
    Device id: 0
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_TYPE:  GPU
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_MAX_COMPUTE_UNITS: 0x40
    CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: 0x3
    CL_DEVICE_MAX_WORK_GROUP_SIZE: 0x3
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_CHAR: 0x4
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_SHORT: 0x2
    todo more information...
   ---------------------------
    ---------------------------
    Device id: 1
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_TYPE:  GPU
    CL_DEVICE_VENDOR_ID: 0x1002
    CL_DEVICE_MAX_COMPUTE_UNITS: 0x40
    CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: 0x3
    CL_DEVICE_MAX_WORK_GROUP_SIZE: 0x3
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_CHAR: 0x4
    CL_DEVICE_PREFERRED_VECTOR_WIDTH_SHORT: 0x2
    todo more information...
   ---------------------------
==============================

@Said-Akbar
Copy link
Author

by the way, gfx906 has 'Vega 20' GPUs, but not 'Vega 10' GPUs. Not sure if some instruction that does not exist in gfx906 is being called from llama.cpp.

@Said-Akbar
Copy link
Author

Said-Akbar commented Nov 24, 2024 •
edited
Loading

Here is the app crash log :

cat /var/crash/_opt_rocm_sdk_612_bin_llama-server.1000.crash
ApportVersion: 2.28.1-0ubuntu3.1
CasperMD5CheckResult: pass
Disassembly:
 => 0x7b73e609eb1c <__GI___pthread_kill+284>:	mov    %eax,%r14d
    0x7b73e609eb1f <__GI___pthread_kill+287>:	neg    %r14d
    0x7b73e609eb22 <__GI___pthread_kill+290>:	cmp    $0xfffff000,%eax
    0x7b73e609eb27 <__GI___pthread_kill+295>:	mov    $0x0,%eax
    0x7b73e609eb2c <__GI___pthread_kill+300>:	cmovbe %eax,%r14d
    0x7b73e609eb30 <__GI___pthread_kill+304>:	jmp    0x7b73e609eab0 <__GI___pthread_kill+176>
    0x7b73e609eb35 <__GI___pthread_kill+309>:	nopl   (%rax)
    0x7b73e609eb38 <__GI___pthread_kill+312>:	mov    %r13,%rdi
    0x7b73e609eb3b <__GI___pthread_kill+315>:	call   0x7b73e6098ed0 <__GI___lll_lock_wait_private>
    0x7b73e609eb40 <__GI___pthread_kill+320>:	jmp    0x7b73e609ea7e <__GI___pthread_kill+126>
    0x7b73e609eb45 <__GI___pthread_kill+325>:	nopl   (%rax)
    0x7b73e609eb48 <__GI___pthread_kill+328>:	mov    %r13,%rdi
    0x7b73e609eb4b <__GI___pthread_kill+331>:	call   0x7b73e6098f90 <__GI___lll_lock_wake_private>
    0x7b73e609eb50 <__GI___pthread_kill+336>:	jmp    0x7b73e609ea99 <__GI___pthread_kill+153>
    0x7b73e609eb55 <__GI___pthread_kill+341>:	call   0x7b73e6137e90 <__stack_chk_fail>
    0x7b73e609eb5a:	nopw   0x0(%rax,%rax,1)
InstallationDate: Installed on 2024-11-20 (4 days ago)
InstallationMedia: Ubuntu 24.04.1 LTS "Noble Numbat" - Release amd64 (20240827.1)
JournalErrors: -- No entries --
ProcCpuinfoMinimal:
 processor	: 31
 vendor_id	: AuthenticAMD
 cpu family	: 25
 model		: 33
 model name	: AMD Ryzen 9 5950X 16-Core Processor
 stepping	: 0
 microcode	: 0xa201016
 cpu MHz		: 2200.000
 cache size	: 512 KB
 physical id	: 0
 siblings	: 32
 core id		: 15
 cpu cores	: 16
 apicid		: 31
 initial apicid	: 31
 fpu		: yes
 fpu_exception	: yes
 cpuid level	: 16
 wp		: yes
 flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
 bugs		: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
 bogomips	: 6799.86
 TLB size	: 2560 4K pages
 clflush size	: 64
 cache_alignment	: 64
 address sizes	: 48 bits physical, 48 bits virtual
 power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
ProcVersionSignature: Ubuntu 6.8.0-49.49-generic 6.8.12
Registers:
 rax            0x0                 0
 rbx            0x189e              6302
 rcx            0x7b73e609eb1c      135737710865180
 rdx            0x6                 6
 rsi            0x189e              6302
 rdi            0x189e              6302
 rbp            0x7ffe465f42b0      0x7ffe465f42b0
 rsp            0x7ffe465f4270      0x7ffe465f4270
 r8             0x57                87
 r9             0x0                 0
 r10            0x8                 8
 r11            0x246               582
 r12            0x6                 6
 r13            0x0                 0
 r14            0x16                22
 r15            0x627599932160      108257227252064
 rip            0x7b73e609eb1c      0x7b73e609eb1c <__GI___pthread_kill+284>
 eflags         0x246               [ PF ZF IF ]
 cs             0x33                51
 ss             0x2b                43
 ds             0x0                 0
 es             0x0                 0
 fs             0x0                 0
 gs             0x0                 0
 fs_base        0x7b73e628ec40      135737712897088
 gs_base        0x0                 0
Stacktrace:
 #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
         tid = <optimized out>
         ret = 0
         pd = <optimized out>
         old_mask = {__val = {0}}
         ret = <optimized out>
         pd = <optimized out>
         old_mask = <optimized out>
         ret = <optimized out>
         tid = <optimized out>
         ret = <optimized out>
         resultvar = <optimized out>
         resultvar = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
         __futex = <optimized out>
         resultvar = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
         __futex = <optimized out>
         __private = <optimized out>
         __oldval = <optimized out>
 #1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
 No locals.
 #2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
 No locals.
 #3  0x00007b73e604526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
         ret = <optimized out>
 #4  0x00007b73e60288ff in __GI_abort () at ./stdlib/abort.c:79
         save_stage = 1
         act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {108257077776080, 59, 14422071311227648676, 140730079068992, 135737623843607, 135737712327360, 18446744073709551512, 108257227250632, 303, 108257227129664, 303, 303, 2, 14, 6983489619661282816, 140730079069200}}, sa_flags = -1718412448, sa_restorer = 0x7ffe465f4410}
 #5  0x00007b73e0a2e0ff in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #6  0x00007b73e0b2b201 in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #7  0x00007b73e0ad5983 in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #8  0x00007b73e0c9f9ed in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #9  0x00007b73e0c799df in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #10 0x00007b73e204161e in Tensile::hip::SolutionAdapter::getKernel(ihipModuleSymbol_t*&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #11 0x00007b73e2042257 in Tensile::hip::SolutionAdapter::launchKernel(Tensile::KernelInvocation const&, ihipStream_t*, ihipEvent_t*, ihipEvent_t*) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #12 0x00007b73e2042a68 in Tensile::hip::SolutionAdapter::launchKernels(std::vector<Tensile::KernelInvocation, std::allocator<Tensile::KernelInvocation> > const&, ihipStream_t*, ihipEvent_t*, ihipEvent_t*) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #13 0x00007b73e184224f in rocblas_status_ runContractionProblem<_Float16, _Float16, _Float16, _Float16, _Float16, _Float16>(RocblasContractionProblem<_Float16, _Float16, _Float16, _Float16, _Float16, _Float16> const&, rocblas_gemm_algo_, int) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #14 0x00007b73e1968b6c in rocblas_status_ gemm_ex_batched_template<_Float16, _Float16, _Float16>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, _Float16 const*, _Float16 const* const*, long, int, long, _Float16 const* const*, long, int, long, _Float16 const*, _Float16 const* const*, long, int, long, _Float16* const*, long, int, long, int, rocblas_gemm_algo_, int, rocblas_gemm_flags_) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #15 0x00007b73e1966f0d in rocblas_status_ gemm_ex_typecasting<true, _Float16, _Float16, _Float16>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, void const*, void const*, long, int, long, void const*, long, int, long, void const*, void const*, long, int, long, void*, long, int, long, int, rocblas_gemm_algo_, int, rocblas_gemm_flags_) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #16 0x00007b73e1960db2 in rocblas_status_ rocblas_gemm_ex_template<true>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, void const*, void const*, rocblas_datatype_, long, int, long, void const*, rocblas_datatype_, long, int, long, void const*, void const*, rocblas_datatype_, long, int, long, void*, rocblas_datatype_, long, int, long, int, rocblas_datatype_, rocblas_gemm_algo_, int, unsigned int) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #17 0x00007b73e195ff43 in rocblas_gemm_batched_ex () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #18 0x00007b73e67b3e38 in hipblasGemmBatchedEx () from /opt/rocm_sdk_612/lib64/libhipblas.so.2
 No symbol table info available.
 #19 0x00007b73e6903068 in ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) () from /opt/rocm_sdk_612/lib64/libggml.so
 No symbol table info available.
 #20 0x00007b73e68f33b9 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /opt/rocm_sdk_612/lib64/libggml.so
 No symbol table info available.
 #21 0x00007b73e6873703 in ggml_backend_sched_graph_compute_async () from /opt/rocm_sdk_612/lib64/libggml.so
 No symbol table info available.
 #22 0x00007b73e8d6dfd2 in llama_decode () from /opt/rocm_sdk_612/lib64/libllama.so
 No symbol table info available.
 #23 0x0000627586b5d704 in llama_init_from_gpt_params(gpt_params&) ()
 No symbol table info available.
 #24 0x0000627586af0822 in server_context::load_model(gpt_params const&) ()
 No symbol table info available.
 #25 0x0000627586aa2820 in main ()
 No symbol table info available.
StacktraceAddressSignature: /opt/rocm_sdk_612/bin/llama-server:6:/usr/lib/x86_64-linux-gnu/libc.so.6+1d26e:/usr/lib/x86_64-linux-gnu/libc.so.6+8ff:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+b0ff:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+108201:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+b2983:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+27c9ed:/opt/rocm_sdk_612/lib64/libamdhip64.so.6.1.40093-61a06a2f8+2569df:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+fae61e:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+faf257:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+fafa68:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+7af24f:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8d5b6c:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8d3f0d:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8cddb2:/opt/rocm_sdk_612/lib64/librocblas.so.4.1.60102+8ccf43
StacktraceTop:
 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
Tags: noble wayland-session
ThreadStacktrace:
 .
 Thread 35 (Thread 0x7b71930006c0 (LWP 6336)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7192fff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 56, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 56
         seq = 28
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727727773376, 1370407099822688401, 135727727773376, -160, 0, 135728109450704, 1370407099742996625, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 34 (Thread 0x7b71926006c0 (LWP 6337)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71925ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 58, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 58
         seq = 29
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727717287616, 1370405725433153681, 135727717287616, -160, 0, 135728109450704, 1370405725353461905, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 33 (Thread 0x7b7194e006c0 (LWP 6333)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7194dff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 50, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 50
         seq = 25
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727759230656, 1370402426898270353, 135727759230656, -160, 0, 135728109450704, 1370402426818578577, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 32 (Thread 0x7b7196c006c0 (LWP 6330)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7196bff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 44, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 44
         seq = 22
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727790687936, 1370397753973852305, 135727790687936, -160, 0, 135728109450704, 1370397753894160529, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 31 (Thread 0x7b71958006c0 (LWP 6332)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71957ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 48, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 48
         seq = 24
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727769716416, 1370403801287805073, 135727769716416, -160, 0, 135728109450704, 1370403801208113297, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 30 (Thread 0x7b71962006c0 (LWP 6331)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71961ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 46, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 46
         seq = 23
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727780202176, 1370396379584317585, 135727780202176, -160, 0, 135728109450704, 1370396379504625809, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 29 (Thread 0x7b7193a006c0 (LWP 6335)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71939ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 52, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 52
         seq = 26
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727738259136, 1370408474212223121, 135727738259136, -160, 0, 135728109450704, 1370408474132531345, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 28 (Thread 0x7b7191c006c0 (LWP 6338)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7191bff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 60, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 60
         seq = 30
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727706801856, 1370413147136641169, 135727706801856, -160, 0, 135728109450704, 1370413147056949393, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 27 (Thread 0x7b71944006c0 (LWP 6334)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71943ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 54, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 54
         seq = 27
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727748744896, 1370401052508735633, 135727748744896, -160, 0, 135728109450704, 1370401052429043857, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 26 (Thread 0x7b71976006c0 (LWP 6329)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71975ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 42, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 42
         seq = 21
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727801173696, 1370399128363387025, 135727801173696, -160, 0, 135728109450704, 1370399128283695249, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 25 (Thread 0x7b7198a006c0 (LWP 6327)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71989ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 38, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 38
         seq = 19
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727822145216, 1370428265421523089, 135727822145216, -160, 0, 135728109450704, 1370428265341831313, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 24 (Thread 0x7b71980006c0 (LWP 6328)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7197fff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 40, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 40
         seq = 20
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727811659456, 1370400502752921745, 135727811659456, -160, 0, 135728109450704, 1370400502673229969, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 23 (Thread 0x7b719e4006c0 (LWP 6318)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719e3ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 20, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 20
         seq = 10
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727916517056, 1370414246648268945, 135727916517056, -160, 0, 135728109450704, 1370414246568577169, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 22 (Thread 0x7b71a0c006c0 (LWP 6314)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a0bff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 12, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 12
         seq = 6
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727958460096, 1370446132485474449, 135727958460096, -160, 0, 135728109450704, 1370446132405782673, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 21 (Thread 0x7b71994006c0 (LWP 6326)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71993ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 36, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 36
         seq = 18
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727832630976, 1370429639811057809, 135727832630976, -160, 0, 135728109450704, 1370429639731366033, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 20 (Thread 0x7b7199e006c0 (LWP 6325)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b7199dff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 34, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 34
         seq = 17
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727843116736, 1370431014200592529, 135727843116736, -160, 0, 135728109450704, 1370431014120900753, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 19 (Thread 0x7b71a20006c0 (LWP 6312)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a1fff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 8, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 8
         seq = 4
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727979431616, 1370448881264543889, 135727979431616, -160, 0, 135728109450608, 1370448881184852113, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 18 (Thread 0x7b719b2006c0 (LWP 6323)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719b1ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 30, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 30
         seq = 15
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727864088256, 1370424966886639761, 135727864088256, -160, 0, 135728109450704, 1370424966806947985, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 17 (Thread 0x7b719a8006c0 (LWP 6324)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719a7ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 32, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 32
         seq = 16
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727853602496, 1370423592497105041, 135727853602496, -160, 0, 135728109450608, 1370423592417413265, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 16 (Thread 0x7b71a3e006c0 (LWP 6309)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a3dff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 2, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 2
         seq = 1
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135728010888896, 1370444208340125841, 135728010888896, -160, 0, 135728109450608, 1370444208260434065, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 15 (Thread 0x7b719bc006c0 (LWP 6322)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719bbff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 28, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 28
         seq = 14
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727874574016, 1370426341276174481, 135727874574016, -160, 0, 135728109450704, 1370426341196482705, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 14 (Thread 0x7b719ee006c0 (LWP 6317)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719edff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 18, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 18
         seq = 9
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727927002816, 1370415621037803665, 135727927002816, -160, 0, 135728109450704, 1370415620958111889, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 13 (Thread 0x7b719c6006c0 (LWP 6321)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719c5ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 24, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 24
         seq = 12
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727885059776, 1370418919572686993, 135727885059776, -160, 0, 135728109450704, 1370418919492995217, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 12 (Thread 0x7b719da006c0 (LWP 6319)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719d9ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 22, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 22
         seq = 11
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727906031296, 1370421668351756433, 135727906031296, -160, 0, 135728109450704, 1370421668272064657, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 11 (Thread 0x7b719d0006c0 (LWP 6320)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719cfff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 26, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 26
         seq = 13
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727895545536, 1370420293962221713, 135727895545536, -160, 0, 135728109450704, 1370420293882529937, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 10 (Thread 0x7b719f8006c0 (LWP 6316)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b719f7ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 16, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 16
         seq = 8
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727937488576, 1370416995427338385, 135727937488576, -160, 0, 135728109450608, 1370416995347646609, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 9 (Thread 0x7b71a02006c0 (LWP 6315)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a01ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 14, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 14
         seq = 7
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727947974336, 1370444758095939729, 135727947974336, -160, 0, 135728109450704, 1370444758016247953, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 8 (Thread 0x7b71a9c006c0 (LWP 6307)):
 #0  0x00007b73e612b83d in __libc_accept (fd=11, addr=..., len=0x0) at ../sysdeps/unix/sysv/linux/accept.c:26
         sc_ret = -512
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
 #1  0x0000627586aa98db in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::{lambda()#3}> > >::_M_run() ()
 No symbol table info available.
 #2  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #3  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135728109455040, 1370465923694774417, 135728109455040, -160, 34, 140730079079232, 1370465923615082641, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #4  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 7 (Thread 0x7b71a16006c0 (LWP 6313)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a15ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 10, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 10
         seq = 5
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727968945856, 1370447506875009169, 135727968945856, -160, 0, 135728109450704, 1370447506795317393, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 6 (Thread 0x7b71a2a006c0 (LWP 6311)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a29ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 6, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 6
         seq = 3
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135727989917376, 1370441459561056401, 135727989917376, -160, 0, 135728109450704, 1370441459481364625, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 5 (Thread 0x7b71a34006c0 (LWP 6310)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a33ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 4, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 4
         seq = 2
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135728000403136, 1370442833950591121, 135728000403136, -160, 0, 135728109450608, 1370442833870899345, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 4 (Thread 0x7b73c44006c0 (LWP 6303)):
 #0  __GI___ioctl (fd=3, request=3222817548) at ../sysdeps/unix/sysv/linux/ioctl.c:36
         args = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7b73c43ff670, reg_save_area = 0x7b73c43ff630}}
         arg = <optimized out>
         r = -4
 #1  0x00007b73ce52dc30 in kmtIoctl () from /opt/rocm_sdk_612/lib64/libhsa-runtime64.so.1
 No symbol table info available.
 #2  0x00007b73ce526ab8 in hsaKmtWaitOnMultipleEvents_Ext () from /opt/rocm_sdk_612/lib64/libhsa-runtime64.so.1
 No symbol table info available.
 #3  0x00007b73ce497a89 in rocr::core::Signal::WaitAny(unsigned int, hsa_signal_s const*, hsa_signal_condition_t const*, long const*, unsigned long, hsa_wait_state_t, long*) () from /opt/rocm_sdk_612/lib64/libhsa-runtime64.so.1
 No symbol table info available.
 #4  0x00007b73ce46cae6 in rocr::AMD::hsa_amd_signal_wait_any(unsigned int, hsa_signal_s*, hsa_signal_condition_t*, long*, unsigned long, hsa_wait_state_t, long*) () from /opt/rocm_sdk_612/lib64/libhsa-runtime64.so.1
 No symbol table info available.
 #5  0x00007b73ce48efdf in rocr::core::Runtime::AsyncEventsLoop(void*) () from /opt/rocm_sdk_612/lib64/libhsa-runtime64.so.1
 No symbol table info available.
 #6  0x00007b73ce42cf9b in rocr::os::ThreadTrampoline(void*) () from /opt/rocm_sdk_612/lib64/libhsa-runtime64.so.1
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135737143985856, 1369099230741448849, 0, -160, 0, 140730079067040, 1369099230661757073, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 3 (Thread 0x7b71a92006c0 (LWP 6308)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=31601, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=31601, abstime=0x0, clockid=0, expected=0, futex_word=0x7b71a4000be0) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7b71a4000be0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7b71a4000be8, cond=0x7b71a4000bb8) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b71a91ff8c0, __canceltype = 0, __prev = 0x0}
         cbuffer = {wseq = 0, cond = 0x7b71a4000bb8, mutex = 0x7b71a4000be8, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 0
         seq = 0
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x7b71a4000bb8, mutex=0x7b71a4000be8) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586ad07ca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<httplib::ThreadPool::worker> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135728098969280, 1370464549305239697, 135728098969280, -160, 0, 135728109450608, 1370464549225547921, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 2 (Thread 0x7b72c26006c0 (LWP 6306)):
 #0  0x00007b73e6098d61 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x627586bfc938 <gpt_log_main()::log+88>) at ./nptl/futex-internal.c:57
         sc_cancel_oldtype = 0
         sc_ret = <optimized out>
         resultvar = <optimized out>
         __arg6 = <optimized out>
         __arg5 = <optimized out>
         __arg4 = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a6 = <optimized out>
         _a5 = <optimized out>
         _a4 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
 #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x627586bfc938 <gpt_log_main()::log+88>) at ./nptl/futex-internal.c:87
         err = <optimized out>
         clockbit = 256
         op = 393
         err = <optimized out>
         clockbit = <optimized out>
         op = <optimized out>
 #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x627586bfc938 <gpt_log_main()::log+88>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
 No locals.
 #3  0x00007b73e609b7dd in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x627586bfc8e0 <gpt_log_main()::log>, cond=0x627586bfc910 <gpt_log_main()::log+48>) at ./nptl/pthread_cond_wait.c:503
         spin = 0
         buffer = {__routine = 0x7b73e609b4a0 <__condvar_cleanup_waiting>, __arg = 0x7b72c25ff8d0, __canceltype = -2001090544, __prev = 0x0}
         cbuffer = {wseq = 204, cond = 0x627586bfc910 <gpt_log_main()::log+48>, mutex = 0x627586bfc8e0 <gpt_log_main()::log>, private = 0}
         err = <optimized out>
         g = 0
         flags = <optimized out>
         g1_start = <optimized out>
         maxspin = 0
         signals = <optimized out>
         result = 0
         wseq = 204
         seq = 102
         private = 0
         maxspin = <optimized out>
         err = <optimized out>
         result = <optimized out>
         wseq = <optimized out>
         g = <optimized out>
         seq = <optimized out>
         flags = <optimized out>
         private = <optimized out>
         signals = <optimized out>
         done = <optimized out>
         g1_start = <optimized out>
         spin = <optimized out>
         buffer = <optimized out>
         cbuffer = <optimized out>
         s = <optimized out>
 #4  ___pthread_cond_wait (cond=0x627586bfc910 <gpt_log_main()::log+48>, mutex=0x627586bfc8e0 <gpt_log_main()::log>) at ./nptl/pthread_cond_wait.c:627
 No locals.
 #5  0x0000627586b7e623 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<gpt_log::resume()::{lambda()#1}> > >::_M_run() ()
 No symbol table info available.
 #6  0x00007b73e64ecdb4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
 No symbol table info available.
 #7  0x00007b73e609ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
         ret = <optimized out>
         pd = <optimized out>
         out = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {135732817561280, 1369666853619288209, 135732817561280, -160, 2, 140730079079088, 1369666853539596433, 1369164738774222993}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
 #8  0x00007b73e6129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
 No locals.
 .
 Thread 1 (Thread 0x7b73e628ec40 (LWP 6302)):
 #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
         tid = <optimized out>
         ret = 0
         pd = <optimized out>
         old_mask = {__val = {0}}
         ret = <optimized out>
         pd = <optimized out>
         old_mask = <optimized out>
         ret = <optimized out>
         tid = <optimized out>
         ret = <optimized out>
         resultvar = <optimized out>
         resultvar = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
         __futex = <optimized out>
         resultvar = <optimized out>
         __arg3 = <optimized out>
         __arg2 = <optimized out>
         __arg1 = <optimized out>
         _a3 = <optimized out>
         _a2 = <optimized out>
         _a1 = <optimized out>
         __futex = <optimized out>
         __private = <optimized out>
         __oldval = <optimized out>
 #1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
 No locals.
 #2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
 No locals.
 #3  0x00007b73e604526e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
         ret = <optimized out>
 #4  0x00007b73e60288ff in __GI_abort () at ./stdlib/abort.c:79
         save_stage = 1
         act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {108257077776080, 59, 14422071311227648676, 140730079068992, 135737623843607, 135737712327360, 18446744073709551512, 108257227250632, 303, 108257227129664, 303, 303, 2, 14, 6983489619661282816, 140730079069200}}, sa_flags = -1718412448, sa_restorer = 0x7ffe465f4410}
 #5  0x00007b73e0a2e0ff in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #6  0x00007b73e0b2b201 in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #7  0x00007b73e0ad5983 in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #8  0x00007b73e0c9f9ed in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #9  0x00007b73e0c799df in ?? () from /opt/rocm_sdk_612/lib64/libamdhip64.so.6
 No symbol table info available.
 #10 0x00007b73e204161e in Tensile::hip::SolutionAdapter::getKernel(ihipModuleSymbol_t*&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #11 0x00007b73e2042257 in Tensile::hip::SolutionAdapter::launchKernel(Tensile::KernelInvocation const&, ihipStream_t*, ihipEvent_t*, ihipEvent_t*) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #12 0x00007b73e2042a68 in Tensile::hip::SolutionAdapter::launchKernels(std::vector<Tensile::KernelInvocation, std::allocator<Tensile::KernelInvocation> > const&, ihipStream_t*, ihipEvent_t*, ihipEvent_t*) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #13 0x00007b73e184224f in rocblas_status_ runContractionProblem<_Float16, _Float16, _Float16, _Float16, _Float16, _Float16>(RocblasContractionProblem<_Float16, _Float16, _Float16, _Float16, _Float16, _Float16> const&, rocblas_gemm_algo_, int) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #14 0x00007b73e1968b6c in rocblas_status_ gemm_ex_batched_template<_Float16, _Float16, _Float16>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, _Float16 const*, _Float16 const* const*, long, int, long, _Float16 const* const*, long, int, long, _Float16 const*, _Float16 const* const*, long, int, long, _Float16* const*, long, int, long, int, rocblas_gemm_algo_, int, rocblas_gemm_flags_) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #15 0x00007b73e1966f0d in rocblas_status_ gemm_ex_typecasting<true, _Float16, _Float16, _Float16>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, void const*, void const*, long, int, long, void const*, long, int, long, void const*, void const*, long, int, long, void*, long, int, long, int, rocblas_gemm_algo_, int, rocblas_gemm_flags_) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #16 0x00007b73e1960db2 in rocblas_status_ rocblas_gemm_ex_template<true>(_rocblas_handle*, rocblas_operation_, rocblas_operation_, int, int, int, void const*, void const*, rocblas_datatype_, long, int, long, void const*, rocblas_datatype_, long, int, long, void const*, void const*, rocblas_datatype_, long, int, long, void*, rocblas_datatype_, long, int, long, int, rocblas_datatype_, rocblas_gemm_algo_, int, unsigned int) () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #17 0x00007b73e195ff43 in rocblas_gemm_batched_ex () from /opt/rocm_sdk_612/lib64/librocblas.so.4
 No symbol table info available.
 #18 0x00007b73e67b3e38 in hipblasGemmBatchedEx () from /opt/rocm_sdk_612/lib64/libhipblas.so.2
 No symbol table info available.
 #19 0x00007b73e6903068 in ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) () from /opt/rocm_sdk_612/lib64/libggml.so
 No symbol table info available.
 #20 0x00007b73e68f33b9 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /opt/rocm_sdk_612/lib64/libggml.so
 No symbol table info available.
 #21 0x00007b73e6873703 in ggml_backend_sched_graph_compute_async () from /opt/rocm_sdk_612/lib64/libggml.so
 No symbol table info available.
 #22 0x00007b73e8d6dfd2 in llama_decode () from /opt/rocm_sdk_612/lib64/libllama.so
 No symbol table info available.
 #23 0x0000627586b5d704 in llama_init_from_gpt_params(gpt_params&) ()
 No symbol table info available.
 #24 0x0000627586af0822 in server_context::load_model(gpt_params const&) ()
 No symbol table info available.
 #25 0x0000627586aa2820 in main ()
 No symbol table info available.
Title: llama-server crashed with SIGABRT
UnreportableReason: This package does not seem to be installed correctly
UpgradeStatus: No upgrade log present (probably fresh install)
_MarkForUpload: True
separator:

@Said-Akbar
Copy link
Author

Based on app crash logs, I see that rocm is not able to find the symbol table 'No symbol table info available.' Not sure what that means. Let me know. Thanks!

@lamikr
Copy link
Owner

lamikr commented Nov 24, 2024

Thanks, good to see that the the basic applications works. I will start my gfx906 build and try to check if I can figure out fix for those build errors with llama.cpp.

@Said-Akbar
Copy link
Author

Thank you! Looking forward to your updates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lamikr @Said-Akbar