Add distributed lock to discover hosts

[s3rj1k]

refs:

• https://gerrit.mcp.mirantis.com/c/packaging/sources/nova/+/199367

[s3rj1k]

we have discover_hosts_in_cells_interval set to 2 min by default, and at same time we start discovery job each time number of computes is changed in daemonset. I wondered if we can fix code in discover_cells mechanism so concurrent runs of nova-manage cell_v2 discover_hosts and periodic tasks do not lead to failures in scheduling

periodics: https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L113
host_mapping_obj.discover_hosts uses https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L248

CLI: https://docs.openstack.org/nova/latest/cli/nova-manage.html#cell-v2-discover-hosts

Coordinator: https://docs.openstack.org/tooz/1.53.0/tutorial/coordinator.html

[s3rj1k]

RFE proposed: https://bugs.launchpad.net/nova/+bug/2089386

[s3rj1k]

Meeting: https://meetings.opendev.org/meetings/nova/2024/nova.2024-11-26-16.00.log.html
(scroll down to "#topic Open discussion")

[s3rj1k]

Spec: https://review.opendev.org/c/openstack/nova-specs/+/936389

[s3rj1k]

<sean-k-mooney> where as we ahve dicussed how to remove tooz form teh code base 
<sean-k-mooney> at least twice in the last year or two
...
<sean-k-mooney> the main concern with tooz by the way is its overall mantance
...
<s3rj1k> btw, is there a formal doc on tooz deprecation?
<sean-k-mooney> its not deprecated as a lib
<sean-k-mooney> so the only usage of it in nova is the ironic driver and we deprecated the fature that uses it in that
<s3rj1k> so for nova removal in this case
<sean-k-mooney> currently the ironic drivcer can use manual sharding or a dynmic hashring(provided by tooz)
<sean-k-mooney> the dynmaic hash ring is deprecated for removal which would have removed tooz as a side effect