fixing fsnotify behaviour upon change in files which are volume mounted through secrets (New)

[abodhekar]

issue - https://infoblox.atlassian.net/browse/ATLAS-16938

Cause of the issue -

when a watch is added on a file, which is volume mounted through secrets, fsnotify emits 1 or more events including REMOVE.
This is due to the underlying behaviour of updation of files upon updation of corresponding secret keys of kubernetes.
the REMOVE event emitted by fsnotify , internally removes a watch on the path/file.
We were additionally removing the watcher, which resulted into "removal of non-existent watcher"

Solution -

Avoided FATAL, when fsnotify errors "removal of non-existent watcher"
Also sometimes there are more than one event emitted on secret/file change such as REMOVE and CHMOD etc. Upon which pgbouncer could have reloaded similar configuration a few times. Avoided this by caching latest db creds and verify the newer ones are not same as latest used ones.
Due to REMOVE event of file upon change in secret, precautionary checked whether the files are available on the paths before adding a new watcher