Skip to content

Commit

Permalink
Merge pull request #414 from Cray-HPE/casmcms-9225-itertools
Browse files Browse the repository at this point in the history
CASMCMS-9225: Do not use itertools.batched
  • Loading branch information
mharding-hpe authored Dec 19, 2024
2 parents d2312f8 + 1039bbe commit 2dbbd84
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 3 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [2.10.30] - 2024-12-19
### Fixed
- Do not use `itertools.batched`, because it is not available in this Python version.

## [2.10.29] - 2024-12-18
### Changed
- Improve performance of large `GET` components requests.
Expand Down
7 changes: 4 additions & 3 deletions src/bos/server/redis_db_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
from itertools import batched
import functools
import json
import logging
Expand Down Expand Up @@ -110,9 +109,11 @@ def iter_values(self):
"""
Iterate through every item in the database. Parse each item as JSON and yield it.
"""
for next_keys in batched(self.client.scan_iter(), 500):
for datastr in self.client.mget(next_keys):
all_keys = list(self.client.scan_iter())
while all_keys:
for datastr in self.client.mget(all_keys[:500]):
yield json.loads(datastr) if datastr else None
all_keys = all_keys[500:]

def get_keys(self):
"""Get an array of all keys"""
Expand Down

0 comments on commit 2dbbd84

Please sign in to comment.