feat: implement cache on getPrompt and getPromptById methods #86

Matthieu-OD · 2024-11-19T09:12:55Z

Add Prompt Caching Functionality

Changes

Implemented a prompt caching system using a separate class SharedCachePrompt

Implementation Details

Cache keys are generated in three formats:
id
name
name:version
Caching logic implemented on both getPrompt and getPromptById
On each successful get prompt we will cache the prompt with keys id, name, name:version
Added fallback to cached prompts when API calls fail, with warning logs
To keep in mind this cache will be cleared each time the application Is restarted

clementsirieix

Thanks that's great, I'm being a bit cautious with adding core layers such as caching as we will need to reuse and maintain them.

clementsirieix · 2024-11-20T16:34:08Z

src/api.ts

    if (id) {
      return id;
-    } else if (name && version) {
+    } else if (name && (version || version === 0)) {


There is actually a way to do that explicitly:

Suggested change

} else if (name && (version || version === 0)) {

} else if (name && typeof version === "number") {

clementsirieix · 2024-11-20T16:35:24Z

src/api.ts

+    id?: string,
+    name?: string,
+    version?: number


When handling only optional params in js it's recommended to use an object because it's not great ux to do:

instance.getPromptCacheKey(undefined, undefined, version) // vs instance.getPromptCacheKey({ version })

now I understand why auto-complete suggested a object thx

clementsirieix · 2024-11-20T16:36:31Z

src/api.ts

+    return SharedCache.instance;
+  }
+
+  public getPromptCacheKey(


Now that the cache is not prompt only, let's not leak that business logic in the cache layer.

src/api.ts

clementsirieix · 2024-11-20T16:37:05Z

src/api.ts

+    return this.get(key);
+  }
+
+  public putPrompt(prompt: Prompt): void {


clementsirieix · 2024-11-20T16:38:25Z

src/api.ts

@@ -325,6 +325,63 @@ type CreateAttachmentParams = {
  metadata?: Maybe<Record<string, any>>;
 };

+export class SharedCache {
+  private static instance: SharedCache | null = null;


Let's simplify even more:

have a sharedCache.ts file

and apply the singleton pattern https://www.patterns.dev/vanilla/singleton-pattern/

clementsirieix · 2024-11-20T16:40:03Z

src/api.ts

@@ -340,6 +397,8 @@ type CreateAttachmentParams = {
 * Then you can use the `api` object to make calls to the Literal service.
 */
 export class API {
+  /** @ignore */
+  public cache: SharedCache;


This should be internal

used it to simplify my tests implementation and forgot to change it later

clementsirieix

Almost there, 1 blocking and 2 nits.

src/api.ts

src/cache/utils.ts

Matthieu-OD · 2024-11-22T09:11:08Z

I'm going to wait for the PR on the python sdk to be approved to merge both at the same time

Matthieu-OD added 3 commits November 19, 2024 10:13

feat: implement cache class in api.ts

b3828dd

feat: adds cache to getPrompt methods

e2706f0

fix: tests and implementation of nameVersionIndex

15c1428

Matthieu-OD force-pushed the matt/eng-2115-add-client-caching-for-prompts branch from 2a95ead to 15c1428 Compare November 19, 2024 09:15

Matthieu-OD self-assigned this Nov 19, 2024

refactor: move logic to getPromptWithQuery

9625e9f

Matthieu-OD force-pushed the matt/eng-2115-add-client-caching-for-prompts branch from f04386c to 9625e9f Compare November 19, 2024 09:59

Matthieu-OD added 4 commits November 19, 2024 11:04

refactor: if null in response get from cache

bbe697e

feat: adds timeout logic

fc9888a

feat: wip removing lock, simplifying and generalising

0460dc4

fix: finishes to fix all tests

3dab603

Matthieu-OD requested a review from clementsirieix November 20, 2024 15:06

clementsirieix requested changes Nov 20, 2024

View reviewed changes

Matthieu-OD requested a review from clementsirieix November 21, 2024 15:11

Matthieu-OD force-pushed the matt/eng-2115-add-client-caching-for-prompts branch 2 times, most recently from 2cae4a1 to 856baf6 Compare November 21, 2024 16:03

feat: refactor and fix tests

f1902e7

Matthieu-OD force-pushed the matt/eng-2115-add-client-caching-for-prompts branch from 856baf6 to f1902e7 Compare November 21, 2024 16:04

feat: replace manager by utils

b03d76d

clementsirieix requested changes Nov 22, 2024

View reviewed changes

src/api.ts Outdated Show resolved Hide resolved

src/api.ts Outdated Show resolved Hide resolved

src/cache/utils.ts Outdated Show resolved Hide resolved

refactor: apply clem comments

d2b3b53

Matthieu-OD requested a review from clementsirieix November 22, 2024 08:39

clementsirieix approved these changes Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement cache on getPrompt and getPromptById methods #86

feat: implement cache on getPrompt and getPromptById methods #86

Matthieu-OD commented Nov 19, 2024

clementsirieix left a comment

clementsirieix Nov 20, 2024

clementsirieix Nov 20, 2024

Matthieu-OD Nov 21, 2024

clementsirieix Nov 20, 2024

clementsirieix Nov 20, 2024

clementsirieix Nov 20, 2024

clementsirieix Nov 20, 2024

Matthieu-OD Nov 21, 2024

clementsirieix left a comment

Matthieu-OD commented Nov 22, 2024

	} else if (name && (version \|\| version === 0)) {
	} else if (name && typeof version === "number") {

feat: implement cache on getPrompt and getPromptById methods #86

Are you sure you want to change the base?

feat: implement cache on getPrompt and getPromptById methods #86

Conversation

Matthieu-OD commented Nov 19, 2024

Add Prompt Caching Functionality

Changes

Implementation Details

clementsirieix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clementsirieix left a comment

Choose a reason for hiding this comment

Matthieu-OD commented Nov 22, 2024