AttributeError: 'NoneType' object has no attribute 'timestamp' in last_xlog_upload_callback

[MPV]

Potential bug?

Our exporter has worked fine for months, and a few days ago it started emitting logs like this one:

Traceback (most recent call last):
AttributeError: 'NoneType' object has no attribute 'timestamp'
 File "exporter.py", line 239, in last_xlog_upload_callback
 File "prometheus_client/metrics.py", line 420, in samples
 File "prometheus_client/metrics.py", line 213, in _samples
 File "prometheus_client/metrics.py", line 220, in _multi_samples
 File "prometheus_client/metrics.py", line 94, in collect
 File "prometheus_client/registry.py", line 83, in collect
 File "prometheus_client/openmetrics/exposition.py", line 22, in generate_latest
 File "prometheus_client/exposition.py", line 104, in _bake_output
 File "prometheus_client/exposition.py", line 122, in prometheus_app
 File "wsgiref/handlers.py", line 137, in run

i.e. here:

wal-g-prometheus-exporter/exporter.py

Line 238 in b5a0a46

return archive_status['last_archived_time'].timestamp()

(I have different line numbers, due to us using this fork with the changes from #3: https://github.com/pagero/wal-g-prometheus-exporter)

[MPV]

For reference, this is what the basebackups_005 directory looks like:

                           PRE base_0000000D00000073000000CE/
                           PRE base_0000000D00000074000000C2/
                           PRE base_0000000D000000760000009C/
                           PRE base_0000000D000000780000002C/
                           PRE base_0000000D00000079000000C4/
                           PRE base_0000000D0000007B00000069/
                           PRE base_0000000D0000007D0000001C/
                           PRE base_0000000E0000007E0000000E/
                           PRE base_0000000E0000007F00000002/
2022-05-15 04:19:37  185.1 KiB base_0000000D00000073000000CE_backup_stop_sentinel.json
2022-05-16 03:25:17  185.1 KiB base_0000000D00000074000000C2_backup_stop_sentinel.json
2022-05-17 03:27:45  185.1 KiB base_0000000D000000760000009C_backup_stop_sentinel.json
2022-05-18 03:24:57  185.1 KiB base_0000000D000000780000002C_backup_stop_sentinel.json
2022-05-19 03:26:34  185.1 KiB base_0000000D00000079000000C4_backup_stop_sentinel.json
2022-05-20 03:22:54  185.2 KiB base_0000000D0000007B00000069_backup_stop_sentinel.json
2022-05-23 03:22:27  185.4 KiB base_0000000E0000007F00000002_backup_stop_sentinel.json

And here is the pg_stat_archiver table:

$ psql -c "SELECT archived_count, failed_count, last_archived_wal, last_archived_time, last_failed_wal, last_failed_time FROM pg_stat_archiver" postgres
 archived_count | failed_count | last_archived_wal | last_archived_time | last_failed_wal | last_failed_time
----------------+--------------+-------------------+--------------------+-----------------+------------------
              0 |            0 |                   |                    |                 |
(1 row)

[MPV]

Getting the full table (including the interesting stats_reset column), we can see that metrics were reset (around the time we started getting this issue):

$ psql -c "SELECT * FROM pg_stat_archiver" postgres
 archived_count | last_archived_wal | last_archived_time | failed_count | last_failed_wal | last_failed_time |          stats_reset
----------------+-------------------+--------------------+--------------+-----------------+------------------+-------------------------------
              0 |                   |                    |            0 |                 |                  | 2022-05-21 11:50:02.279099+00
(1 row)

i.e:

$ psql -c "SELECT stats_reset FROM pg_stat_archiver" postgres
          stats_reset
-------------------------------
 2022-05-21 11:50:02.279099+00
(1 row)

Reading the docs, it seems they can be reset at times:

[...]
When recovery is performed at server start (e.g., after immediate shutdown, server crash, and point-in-time recovery), all statistics counters are reset.
-- https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-STATS-FUNCTIONS