ERROR: unable to read data from DB node 0

[tferfce]

Hello,

Lately we are experiencing this error using pgpool. The pgpool is deployed on kubernetes (AKS version 1.28.9) using this image image: pgpool/pgpool:4.4.3. Our DB is a 2 node cluster deployed on 2xVMs on the same VNet on Azure.

2024-10-10 11:45:46.817: rest_api pid 164: WARNING:  ssl read: error: 5
2024-10-10 11:45:46.817: rest_api pid 164: ERROR:  unable to read data from DB node 0
2024-10-10 11:45:46.817: rest_api pid 164: DETAIL:  do not failover because failover_on_backend_error is off
2024-10-10 11:45:46.817: rest_api pid 164: WARNING:  ssl write: error: 5
2024-10-10 11:45:46.817: rest_api pid 164: WARNING:  write on backend 0 failed with error :"No error information"
2024-10-10 11:45:46.817: rest_api pid 164: DETAIL:  while trying to write data from offset: 0 wlen: 17
2024-10-10 11:45:46.817: rest_api pid 164: LOG:  pool_ssl: "SSL_write": "bad length"
2024-10-10 11:45:46.817: rest_api pid 164: WARNING:  write on backend 0 failed with error :"No error information"
2024-10-10 11:45:46.817: rest_api pid 164: DETAIL:  while trying to write data from offset: 0 wlen: 5
2024-10-10 11:49:56.473: rest_api pid 160: WARNING:  ssl read: error: 5
2024-10-10 11:49:56.473: rest_api pid 160: ERROR:  unable to read data from DB node 1
2024-10-10 11:49:56.473: rest_api pid 160: DETAIL:  do not failover because failover_on_backend_error is off
2024-10-10 11:49:56.474: rest_api pid 160: WARNING:  ssl write: error: 5
2024-10-10 11:49:56.474: rest_api pid 160: WARNING:  write on backend 1 failed with error :"No error information"
2024-10-10 11:49:56.474: rest_api pid 160: DETAIL:  while trying to write data from offset: 0 wlen: 17
2024-10-10 11:49:56.474: rest_api pid 160: LOG:  pool_ssl: "SSL_write": "bad length"
2024-10-10 11:49:56.474: rest_api pid 160: WARNING:  write on backend 1 failed with error :"No error information"
2024-10-10 11:49:56.474: rest_api pid 160: DETAIL:  while trying to write data from offset: 0 wlen: 5

That results some errors on our api service as well.

Severity: ERROR
          SqlState: XX000
          MessageText: unable to read data from DB node 0
          Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
          File: pool_stream.c
          Line: 407

our current configuration is

    pcp_listen_addresses = '*'
    pcp_port = 9898
    pcp_socket_dir = '/var/run/pgpool'
    listen_addresses = '*'
    backend_hostname0 = ###
    backend_hostname1 = ###
    backend_port0 = ###
    backend_port1 = ###
    backend_weight0 = 1
    backend_weight1 = 1
    backend_flag0 = 'ALWAYS_PRIMARY|DISALLOW_TO_FAILOVER'
    backend_flag1 = 'DISALLOW_TO_FAILOVER'
    failover_on_backend_error = off
    load_balance_mode = on
    connection_cache = on
    serialize_accept = on
    num_init_children = 90
    max_pool = 2
    child_life_time = 600
    child_max_connections = 0
    connection_life_time = 600
    statement_level_load_balance = on
    client_idle_limit = 1800
    log_min_messages = warning
    log_statement = off
    log_per_node_statement = off
    sr_check_period = 10
    sr_check_database = ###
    sr_check_user = ###
    sr_check_password = ###
    ssl = on
    ssl_key = '/pgpool-tls/tls.key'
    ssl_cert = '/pgpool-tls/tls.crt'
    process_management_mode = 'dynamic'
    process_management_strategy = 'gentle'
    min_spare_children = 5
    max_spare_children = 10

Also we investigate the issue on db and side and it seems the problem is between client and pgpool as the requests doesn't reach db at all. Regarding SSL they are self signed but valid (in terms of expiration date).

Any ideas on the issue?

Thanks,
Thomas