Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature add fault tolerance #74

Closed
wants to merge 8 commits into from
Closed

Feature add fault tolerance #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
13135c7
Added ModerationModel ChatMemoryProvider (but might need rework).
Nov 11, 2024
494e646
Enable the Fault Tolerance Interceptor.
Nov 23, 2024
0e73479
Enable the Fault Tolerance Interceptor.
Nov 23, 2024
6e7f3a8
Enable the Fault Tolerance Interceptor.
Nov 23, 2024
b2688a0
Merge branch 'feature-add-fault-tolerance' of https://github.com/smal…
Dec 16, 2024
dcf66ac
Enable the Fault Tolerance Interceptor.
Dec 17, 2024
8a7bd8f
Enable the Fault Tolerance Interceptor.
Nov 23, 2024
28f3e0d
Merge branch 'feature-add-fault-tolerance' of https://github.com/smal…
Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/helidon-car-booking/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

<properties>
<mainClass>io.helidon.Main</mainClass>
<dev.langchain4j.version>0.34.0</dev.langchain4j.version>
<dev.langchain4j.version>0.36.1</dev.langchain4j.version>
ehsavoie marked this conversation as resolved.
Show resolved Hide resolved
</properties>

<dependencyManagement>
Expand Down
2 changes: 1 addition & 1 deletion examples/liberty-car-booking/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ To package the application in JVM mode run: `mvn package`.

## Configuration

All configuration is centralized in `microprofile-config.properties`(found is `resources\META-INF` folder) and can be redefined using environment variables.
All configuration is centralized in `microprofile-config.properties` (found is `resources\META-INF` folder) and can be redefined using environment variables.

## Running the application

Expand Down
101 changes: 29 additions & 72 deletions examples/liberty-car-booking/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,24 @@
</developer>
</developers>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.report.sourceEncoding>UTF-8</project.report.sourceEncoding>
<jakartaee-api.version>10.0.0</jakartaee-api.version>
<microprofile-api.version>6.1</microprofile-api.version>
<compiler-plugin.version>3.13.0</compiler-plugin.version>
<war-plugin.version>3.4.0</war-plugin.version>
<!--Strictly for OpenLiberty-->
<liberty.env.ENGINE_CACHE_DIR>${project.build.directory}/liberty/wlp/usr/shared/resources/lib/</liberty.env.ENGINE_CACHE_DIR>
</properties>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.report.sourceEncoding>UTF-8</project.report.sourceEncoding>
<jakartaee-api.version>10.0.0</jakartaee-api.version>
<microprofile-api.version>6.1</microprofile-api.version>
<compiler-plugin.version>3.13.0</compiler-plugin.version>
<war-plugin.version>3.4.0</war-plugin.version>

<!-- We need only javac to use the release parameter. OpenLiberty doesn't depend on maven-compiler-plugin.' -->
<maven.compiler.source combine.self="override" />
ehsavoie marked this conversation as resolved.
Show resolved Hide resolved
<maven.compiler.target combine.self="override" />

<!--Strictly for OpenLiberty-->
<liberty.env.ENGINE_CACHE_DIR>${project.build.directory}/liberty/wlp/usr/shared/resources/lib/</liberty.env.ENGINE_CACHE_DIR>
</properties>

<dependencyManagement>
<dependencies>
<!-- https://mvnrepository.com/artifact/jakarta.platform/jakarta.jakartaee-api -->
<dependency>
<groupId>jakarta.platform</groupId>
<artifactId>jakarta.jakartaee-api</artifactId>
Expand All @@ -45,46 +49,21 @@
<type>pom</type>
<scope>provided</scope>
</dependency>

<!--
<dependency>
<groupId>io.smallrye.llm.liberty-bundle</groupId>
<artifactId>smallrye-llm-langchain4j-features-bom</artifactId>
<version>1.0.0-SNAPSHOT</version>
<type>pom</type>
</dependency>
-->
<!--
<dependency>
<groupId>io.smallrye.llm</groupId>
<artifactId>smallrye-llm-langchain4j-buildcompatible-extension</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
-->
<!-- https://mvnrepository.com/artifact/ai.djl.huggingface/tokenizers -->
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
<version>0.30.0</version>
</dependency>
</dependencies>
</dependencyManagement>

<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
<version>0.30.0</version>
</dependency>
</dependencies>
ehsavoie marked this conversation as resolved.
Show resolved Hide resolved
</dependencyManagement>

<dependencies>
<!-- https://mvnrepository.com/artifact/jakarta.platform/jakarta.jakartaee-api -->
<!--
<dependency>
<groupId>jakarta.platform</groupId>
<artifactId>jakarta.jakartaee-api</artifactId>
<scope>provided</scope>
</dependency>
-->

<dependency>
<groupId>org.eclipse.microprofile</groupId>
<artifactId>microprofile</artifactId>
<type>pom</type>
</dependency>
<dependencies>
<dependency>
<groupId>org.eclipse.microprofile</groupId>
<artifactId>microprofile</artifactId>
<type>pom</type>
</dependency>

<dependency>
<groupId>io.smallrye.llm</groupId>
Expand All @@ -96,27 +75,9 @@
<artifactId>smallrye-llm-langchain4j-portable-extension</artifactId>
</dependency>

<!--
<dependency>
<groupId>io.smallrye.llm</groupId>
<artifactId>smallrye-llm-langchain4j-buildcompatible-extension</artifactId>
</dependency>
-->

<!--
<dependency>
<groupId>io.smallrye.llm.liberty-bundle</groupId>
<artifactId>smallrye-llm-langchain4j-feature</artifactId>
<version>1.0.0-SNAPSHOT</version>
<type>pom</type>
<scope>provided</scope>
</dependency>
-->

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>

<!-- langchain4j dependencies -->
Expand All @@ -130,7 +91,6 @@
<artifactId>langchain4j-hugging-face</artifactId>
</dependency>

<!-- https://mvnrepository.com/artifact/dev.langchain4j/langchain4j-azure-open-ai -->
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-azure-open-ai</artifactId>
Expand All @@ -146,7 +106,6 @@
<artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId>
</dependency>

<!-- https://mvnrepository.com/artifact/ai.djl.huggingface/tokenizers -->
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
Expand All @@ -156,15 +115,13 @@
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<scope>runtime</scope>
</dependency>
</dependencies>

<build>
<finalName>${project.artifactId}</finalName>
<pluginManagement>
<plugins>
<!-- https://mvnrepository.com/artifact/io.openliberty.tools/liberty-maven-plugin -->
<plugin>
<groupId>io.openliberty.tools</groupId>
<artifactId>liberty-maven-plugin</artifactId>
Expand Down
2 changes: 0 additions & 2 deletions examples/liberty-car-booking/src/main/java/io/jefrajames/booking/CarBookingResource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,7 @@ public String chatWithAssistant(
@Path("/fraud")
public FraudResponse detectFraudForCustomer(
@QueryParam("name") String name,

ehsavoie marked this conversation as resolved.
Show resolved Hide resolved
@QueryParam("surname") String surname) {
return fraudService.detectFraudForCustomer(name, surname);
}

}
4 changes: 3 additions & 1 deletion examples/liberty-car-booking/src/main/java/io/jefrajames/booking/ChatAiService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import java.time.temporal.ChronoUnit;

import jakarta.enterprise.context.ApplicationScoped;

import org.eclipse.microprofile.faulttolerance.Fallback;
import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.faulttolerance.Timeout;
Expand All @@ -10,7 +12,7 @@
import io.smallrye.llm.spi.RegisterAIService;

//@SuppressWarnings("CdiManagedBeanInconsistencyInspection")
@RegisterAIService(tools = BookingService.class, chatMemoryMaxMessages = 10)
@RegisterAIService(scope = ApplicationScoped.class, tools = BookingService.class, chatMemoryMaxMessages = 10)
ehsavoie marked this conversation as resolved.
Show resolved Hide resolved
public interface ChatAiService {

@SystemMessage("""
Expand Down
11 changes: 3 additions & 8 deletions examples/liberty-car-booking/src/main/java/io/jefrajames/booking/DocRagIngestor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@
import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.parser.TextDocumentParser;
import dev.langchain4j.data.document.splitter.DocumentSplitters;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.embedding.EmbeddingModel;
import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
import dev.langchain4j.store.embedding.EmbeddingStore;
import dev.langchain4j.store.embedding.EmbeddingStoreIngestor;
import dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore;
import lombok.extern.java.Log;
Expand All @@ -32,9 +34,7 @@ public class DocRagIngestor {

// Used by ContentRetriever
@Produces
private InMemoryEmbeddingStore embeddingStore = new InMemoryEmbeddingStore<>();

// private File docs = new File(System.getProperty("docragdir"));
private EmbeddingStore<TextSegment> embeddingStore = new InMemoryEmbeddingStore<>();

@Inject
@ConfigProperty(name = "app.docs-for-rag.dir")
Expand All @@ -60,9 +60,4 @@ public void ingest(@Observes @Initialized(ApplicationScoped.class) Object pointl
log.info(String.format("DEMO %d documents ingested in %d msec", docs.size(),
System.currentTimeMillis() - start));
}

public static void main(String[] args) {

System.out.println(InMemoryEmbeddingStore.class.getInterfaces()[0]);
}
}
14 changes: 10 additions & 4 deletions examples/liberty-car-booking/src/main/java/io/jefrajames/booking/FraudAiService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
package io.jefrajames.booking;

import java.time.temporal.ChronoUnit;

import org.eclipse.microprofile.faulttolerance.Fallback;
import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.faulttolerance.Timeout;

import dev.langchain4j.service.SystemMessage;
import dev.langchain4j.service.UserMessage;
import dev.langchain4j.service.V;
import io.smallrye.llm.spi.RegisterAIService;

@SuppressWarnings("CdiManagedBeanInconsistencyInspection")
@RegisterAIService(chatMemoryMaxMessages = 5,

chatLanguageModelName = "chat-model")
@RegisterAIService(chatMemoryMaxMessages = 5, chatLanguageModelName = "chat-model")
public interface FraudAiService {

@SystemMessage("""
Expand Down Expand Up @@ -44,6 +47,9 @@ A booking overlap (and hence a fraud) occurs when there are several bookings for

You must not wrap JSON response in backticks, markdown, or in any other way, but return it as plain text.
""")
@Timeout(unit = ChronoUnit.MINUTES, value = 5)
@Retry(maxRetries = 2)
@Fallback(fallbackMethod = "fraudFallback")
FraudResponse detectFraudForCustomer(@V("name") String name, @V("surname") String surname);

default FraudResponse fraudFallback(String name, String surname) {
Expand Down
2 changes: 1 addition & 1 deletion examples/liberty-car-booking/src/main/resources/META-INF/microprofile-config.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ smallrye.llm.plugin.chat-model.config.temperature=0.1
smallrye.llm.plugin.chat-model.config.topP=0.1
smallrye.llm.plugin.chat-model.config.timeout=120s
smallrye.llm.plugin.chat-model.config.max-retries=2
#smallrye.llm.plugin.chat-model.config.logRequestsAndResponsess=false
#smallrye.llm.plugin.chat-model.config.logRequestsAndResponses=true


smallrye.llm.plugin.docRagRetriever.class=dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever
Expand Down
2 changes: 1 addition & 1 deletion examples/quarkus-car-booking/src/main/resources/application.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ smallrye.llm.plugin.chat-model.config.temperature=0.1
smallrye.llm.plugin.chat-model.config.topP=0.1
smallrye.llm.plugin.chat-model.config.timeout=120s
smallrye.llm.plugin.chat-model.config.max-retries=2
#smallrye.llm.plugin.chat-model.config.logRequestsAndResponsess=false
#smallrye.llm.plugin.chat-model.config.logRequestsAndResponses=false


smallrye.llm.plugin.docRagRetriever.class=dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever
Expand Down
Loading
Loading