Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add RedisHashKeyValueDB #16

Merged
merged 14 commits into from
Oct 2, 2024
Merged

feat: add RedisHashKeyValueDB #16

merged 14 commits into from
Oct 2, 2024

Conversation

mrnagydavid
Copy link
Contributor

@mrnagydavid mrnagydavid commented Sep 26, 2024

In this PR, I implemented the RedisHashKeyValueDB which is a special class providing the usual CommonKeyValueDB functions on a hash field in Redis.

The underlying use-case is to

  1. keep certain data in a hash together for easier fetching (similar to having all fields top-level with a table: prefix)
  2. be able to increment values of certain fields

Compared to using the more idiomatic table: prefix approach, the hash field approach has the benefit of fetching the entire hash with an O(n) operation n being the size of the hash - meanwhile with a table: prefix, the scan operation would be O(m) where m is the number of keys in redis. (At least that is my understanding.)

const db = new RedisHashKeyValueDB({ 
  client: new RedisClient()
  hashKey: 'pageStats' // the name of the field in Redis which will store the hash
})

const dao = new CommonKeyValueDao({
  db,
  table: 'table' // this would become the prefix of the property names in the hash
})

const result = await dao.increment('viewIndexPage')
// --> HINCRBY pageStats table:viewIndexPage 1
expect(result).toBe(1)

This PR depends on NaturalCycles/db-lib#18

@mrnagydavid mrnagydavid marked this pull request as ready for review September 30, 2024 13:55
@mrnagydavid
Copy link
Contributor Author

mrnagydavid commented Sep 30, 2024

Looks like yarn test-manual is flaky.
At least 1 out of 4 runs it fails a few tests in runCommonKeyValueDBTest with the same error:
ReplyError: ERR wrong number of arguments for 'mget' command

I think it is may be related to the fact that the tests are not isolated.

Copy link
Member

@kirillgroshkov kirillgroshkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with minor comments.

Great job with tests! 🙏

return this.redis().hscanStream(key, opt)
}

async hScanCount(key: string, opt: ScanStreamOptions): Promise<number> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be in lowercase? (consistent)

Suggested change
async hScanCount(key: string, opt: ScanStreamOptions): Promise<number> {
async hscanCount(key: string, opt: ScanStreamOptions): Promise<number> {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My impression was that you tried to follow the same naming pattern whenever we are just providing a wrapper:
for example async mget() wraps this.redis().mget() (instead of async mGet()).

And you followed camelCase whenever this lib provides extra funtionality, e.g.: mgetBuffer or scanStreamFlat.

So my logic was:

  • hscanStream() wraps this.redis().hscanStream() call, hence identically named
  • hScanCount() provides custom functionality (i.e. there is no redis().hscanCount call), hence camelCase

TBH I don't know and I don't feel strongly about it, so I trust your judgement in using lowercase

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No strong opinion either, I just thought it's a typo. You can pick how you prefer.

My gut feeling is with lowercase hscan though

): Promise<void> {
if (!entries.length) return

const map: StringMap<any> = _mapObject(Object.fromEntries(entries), (k, v) => [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p3: I'd use native map here (instead of _mapObject), like:

const map = Object.fromEntries(entries.map(...))

why?
Since you're already using native Object.fromEntries.

match: `${table}:*`,
})
.flatMap(
async keyValueList => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: why is it async (and has concurrency), while I see no await inside?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy-pasted from redisKeyValueDB where it is correctly async.
As I re-wrote the function, I didn't notice that I removed the awaited calls in it.

match: `${table}:*`,
})
.flatMap(
async keyValueList => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q why async?

@mrnagydavid
Copy link
Contributor Author

Also confirmed that the flakiness is due to parallel running the test suits on the same REDIS with the same table name.
yarn test-manual --runInBand removes flakiness in this repo.
Other fix should probably go to db-lib where the TEST_TABLE is defined.

@mrnagydavid mrnagydavid merged commit 6522566 into master Oct 2, 2024
2 checks passed
@kirillgroshkov
Copy link
Member

Also confirmed that the flakiness is due to parallel running the test suits on the same REDIS with the same table name. yarn test-manual --runInBand removes flakiness in this repo. Other fix should probably go to db-lib where the TEST_TABLE is defined.

I never ran manual tests in parallel, so, never encountered this.

"Manual tests" are not very well defined, I must admit. Maybe we can define them as "only to be run manually one by one, and never in bulk".

@kirillgroshkov kirillgroshkov deleted the DEV-19274 branch October 2, 2024 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants