Skip to content

Commit

Permalink
Merge pull request #415 from Cray-HPE/casmcms-9225-itertools-csm1.4
Browse files Browse the repository at this point in the history
CASMCMS-9225: Do not use itertools.batched
  • Loading branch information
mharding-hpe authored Dec 19, 2024
2 parents c9bbfcc + 2bc9563 commit bed4ec4
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 3 deletions.
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [2.0.49] - 2024-12-19
### Fixed
- Use correct path to requests_retry_session module
- Do not use `itertools.batched`, because it is not available in this Python version.

## [2.0.48] - 2024-12-18
### Changed
Expand Down
7 changes: 4 additions & 3 deletions src/bos/server/redis_db_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
from itertools import batched
import json
import logging
from typing import Callable
Expand Down Expand Up @@ -111,9 +110,11 @@ def iter_values(self):
"""
Iterate through every item in the database. Parse each item as JSON and yield it.
"""
for next_keys in batched(self.client.scan_iter(), 500):
for datastr in self.client.mget(next_keys):
all_keys = list(self.client.scan_iter())
while all_keys:
for datastr in self.client.mget(all_keys[:500]):
yield json.loads(datastr) if datastr else None
all_keys = all_keys[500:]

def put(self, key, new_data):
"""Put data in to the database, replacing any old data."""
Expand Down

0 comments on commit bed4ec4

Please sign in to comment.