Scrum spike: Research prompt engineering techniques

[tubamos]

Domain

data pipeline optimisation

Description

The focus of this scrum spike is to help the team familiarize with prompt engineering techniques that we will use to improve the interaction with the LLM(s). Popular techniques such as Chain-of-Thought (CoT) prompting and ReACT (integration of reasoning with actionable prompts) should be expored along with other techniques. The developers should engage in hands-on testing and and rank their proposals in relation to our use case.

Example documentaion:
https://www.promptingguide.ai/techniques

User Story

• As a developer in our team,
• I want to learn and experiment with prompt engineering techniques
• so that I can effectively optimize the responses generated by our custom agents.

Acceptance Criteria

• The developer understands the different techniques for prompt engineering in terms of strengths and weaknesses.
• The developer has documented at least 3 sample prompts an their results using CoT and ReACT techniques (and possibly others) and has evaluated their effectivenes for our purposes
• The developer can reason about which techniques are relevant to our use case and select three of them to focus on for the project.
• The developer has documented further resources for the team as well as proposals regarding which order the techniques should be pursued in our case and what the combination of the technicques should be (mixed technique prompt enginnering).
• The developer has shared their findings with the team in a knowledge-sharing session.

[preetvadaliya]

@tubamos ,

Just an idea for research tasks can we create a notebook and submit them in sprint-dilevriables so that can be stored there and in next sprint we dont need to remove or fix them.

[tubamos]

@preetvadaliya ,

yes, let's do this. We actually discussed this option in the past but we did not conclude on whether we do it that way or not because we wanted to check what options exist in github to make such tasks trackable for formal reasons.

The option to commit any documents in the project folder as you suggest has been one option. I am fine with that if the devs agree.

[manikg08]

Different Prompt engineering techniques have been added to the code and have been pushed into the main branch. The current problem with prompt engineering is that all the scraped data is in a different format like each scraper has a different JSON structure due to which chunking is quite problematic. We need to create different classes for chunking different scraped data into the Vector Database.

[tubamos]

@manikg08 Let us disucss this during the sprint meeting today.