Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added deployment option for hackathon participants #2

Closed
wants to merge 12 commits into from
Closed
Next Next commit
First commit
kongzii committed Jun 5, 2024
commit 7510133f6dc6cffb468dc7dee3da6c1902f92f2d
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
OPENAI_API_KEY=
SERPER_API_KEY=
BET_FROM_PRIVATE_KEY=
15 changes: 15 additions & 0 deletions .github/actions/python_prepare/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: "Prepare Python environment"
description: "Set up Python and install dependencies"
runs:
using: "composite"
steps:
- name: Set up Python 3.10
uses: actions/setup-python@v2
with:
python-version: 3.10.14
- name: Install Poetry
shell: bash
run: curl -sSL https://install.python-poetry.org | python3 -
- name: Install dependencies
shell: bash
run: poetry install
18 changes: 18 additions & 0 deletions .github/workflows/python_ci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Python CI

on:
pull_request:
push:
branches: [main]
workflow_dispatch:

jobs:
mypy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
submodules: true
- uses: ./.github/actions/python_prepare
- name: Run mypy
run: poetry run mypy
73 changes: 73 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,75 @@
# gnosis-labs-zuberlin2024

Repository for the hackathon run by Gnosis Labs at ZuBerlin 2024.

## Support

Contact us at https://t.me/+Fb0trLKZdMw2MTQ8.

## Setup

Install the project dependencies with `poetry`, using Python 3.10 (you can use [pyenv](https://github.com/pyenv/pyenv) to manage multiple Python versions):

```bash
python3.10 -m pip install poetry
python3.10 -m poetry install
python3.10 -m poetry shell
```

Copy `.env.example` to `.env` and fill in the values:

### OpenAI API key

We will provide you with OpenAI key that's allowed to use gpt-3.5-turbo and embedding models, contact us on the TG group above.

However, everyone is welcome to use arbitrary LLM if wanted.

### Tavily API key

Create a free acount on https://tavily.com and get the key there.

Again, everyone is welcome to use arbitrary search engines, combine them, or even do a totally different approaches!

### Private key on Gnosis Chain

Use your existing or create a new wallet on Gnosis Chain.

By default the script will do only very tiny bets (0.00001 xDai per market), but of course, you can contact us on the TG group above with your public key to get some free xDai.

## Task

Your task is to modify `predict` function in `trader/prediction.py` by any means necessary.

Goal of the `predict` function is, given an `question` about the future, answer it with either `True` (the answer is `yes`), `False` (if the answer is `no`) or `None` (if the prediction failed).

All the questions are guaranteed to be about the future and to be in a binary yes/no format.

You can play with the prompts, different approaches, different LLMs, search engines, or anything you can think of.

### Experimenting

Run

```bash
PYTHONPATH=. streamlit run trader/app.py
```

to start a Streamlit application where you can give your prediction method either question [from the Omen market](https://aiomen.eth.limo/), or write your own.

### Submission

1. Run `python trader/main.py`, it will place bets on all markets that will be used for the evaluation. You can run the script multiple times, but we will always look only at the latest bet on the market from your public key.
2. Open a PR against this repository with your implementation and public key used for placing bets.
3. Make sure the CI pipeline is all green.

### Evaluation

1. Quantitative
1. We will create N markets from the address `0xa7E93F5A0e718bDDC654e525ea668c64Fd572882` by the end of the June, and they will be resolved in roughly two weeks after the creation.
2. We will measure the accuracy of your agent's answers (by the last bet on each market).

2. Qualitative
1. We will look into implementation and judge the creativity of the improvements.

3. Cheating
1. For example, sometimes, the exactly same markets can be found on other prediction market platforms. If we see in the code that the prediction isn’t doing anything practical, we will disqualify it. That being said, it's okay to look at other markets if they are not about the same question, for example, given the evaluation question `Will GNO hit $1000 by the end of 2025?` it's okay to use markets such as `Will GNO hit $500 by the mid of 2025?` as a guidance, but it's not okay to look at the market `Will GNO hit $1000 by the end of 2025?` and copy-paste current probabilities.
36 changes: 36 additions & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
[mypy]
python_version = 3.10
files = trader/
plugins = pydantic.mypy
warn_redundant_casts = True
warn_unused_ignores = True
disallow_any_generics = True
warn_return_any = True
check_untyped_defs = True
show_error_codes = True
strict_equality = True
explicit_package_bases = True
show_traceback = True
disallow_incomplete_defs = True
disallow_untyped_defs = True
ignore_missing_imports = True
# Exclude submodules that are themselves not type-checked
exclude = prediction_market_agent/tools/mech/mech/

# See https://github.com/python/mypy/issues/3905#issuecomment-421065323
# We don't want to ignore all missing imports as we want to catch those in our own code
# But for certain libraries they don't have a stub file, so we only enforce import checking for our own libraries.
# Another alternative would be to list out every single dependency that does not have a stub.
[mypy-prediction_market_agent.*]
ignore_missing_imports = False
[mypy-scripts.*]
ignore_missing_imports = False
[mypy-tests.*]
ignore_missing_imports = False

[pydantic-mypy]
# See https://pydantic-docs.helpmanual.io/mypy_plugin/
init_forbid_extra = True
init_typed = True
warn_required_dynamic_aliases = True
warn_untyped_fields = True
7,374 changes: 7,374 additions & 0 deletions poetry.lock

Large diffs are not rendered by default.

23 changes: 23 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[tool.poetry]
name = "trader"
version = "0.1.0"
description = ""
authors = ["Gnosis Labs <ai@gnosis.io>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10.0"
prediction-market-agent-tooling = { version = "^0.34.0" }
prediction-prophet = { git = "https://github.com/polywrap/predictionprophet.git", branch = "main" }
pydantic-settings = "^2.1.0"

[tool.poetry.group.dev.dependencies]
pytest = "^8.2.1"
mypy = "^1.8.0"
black = "^23.12.1"
autoflake = "^2.2.1"
isort = "^5.13.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
38 changes: 38 additions & 0 deletions trader/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import streamlit as st
from dotenv import load_dotenv
from prediction_market_agent_tooling.markets.agent_market import SortBy
from prediction_market_agent_tooling.markets.markets import (
MarketType,
get_binary_markets,
)

from trader.prediction import predict

# Load the environment variables.
load_dotenv()

# Get some open markets from Omen.
markets = get_binary_markets(42, MarketType.OMEN, sort_by=SortBy.CLOSING_SOONEST)

# Either select a market or provide a custom question.
custom_question_input = st.checkbox("Provide a custom question", value=False)
question = (
st.text_input("Question")
if custom_question_input
else st.selectbox("Select a question", [m.question for m in markets])
)
if not question:
st.warning("Please enter a question.")
st.stop()

# Get the prediction.
prediction = predict(question)

# Display the prediction.
if prediction is None:
st.error("The agent failed to generate a prediction")
st.stop()

st.write(
f"Prediction for the question: {'Yes, it will happen.' if question else 'No, it will not.'}",
)
62 changes: 62 additions & 0 deletions trader/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import pandas as pd
import typer
from dotenv import load_dotenv
from prediction_market_agent_tooling.gtypes import HexAddress, HexStr
from prediction_market_agent_tooling.loggers import logger
from prediction_market_agent_tooling.markets.omen.omen import OmenAgentMarket
from prediction_market_agent_tooling.markets.omen.omen_subgraph_handler import (
OmenSubgraphHandler,
)
from prediction_market_agent_tooling.tools.utils import utcnow

from trader.prediction import predict


def main(creator: str = "0xa7E93F5A0e718bDDC654e525ea668c64Fd572882") -> None:
# Load the environment variables.
load_dotenv()

# Get all markets created by the specified creator.
markets = OmenSubgraphHandler().get_omen_binary_markets(
limit=None,
opened_after=utcnow(),
creator=HexAddress(HexStr(creator)),
)

if not markets:
logger.error("No markets found, please try again later.")
return

results: dict[str, list[str | bool | None]] = {
"market_id": [],
"question": [],
"prediction": [],
}

for market_idx, market in enumerate(markets):
# AgentMarket class contains the logic to interact with the market.
agent_market = OmenAgentMarket.from_data_model(market)

# Get the prediction.
prediction = predict(agent_market.question)

# Place a bet.
if prediction is not None:
agent_market.place_bet(
prediction,
amount=agent_market.get_tiny_bet_amount(), # Just 0.00001 xDai.
)

results["market_id"].append(market.id)
results["question"].append(agent_market.question)
results["prediction"].append(prediction)

logger.info(
f"[{market_idx + 1} / {len(markets)}] Placed {prediction} bet on market {market.url}."
)

pd.DataFrame.from_dict(results).to_csv("results.csv", index=False)


if __name__ == "__main__":
typer.run(main)
103 changes: 103 additions & 0 deletions trader/prediction.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
import streamlit as st
from langchain.text_splitter import RecursiveCharacterTextSplitter
from prediction_prophet.benchmark.agents import _make_prediction
from prediction_prophet.functions.create_embeddings_from_results import (
create_embeddings_from_results,
)
from prediction_prophet.functions.generate_subqueries import generate_subqueries
from prediction_prophet.functions.prepare_report import prepare_report
from prediction_prophet.functions.scrape_results import scrape_results
from prediction_prophet.functions.search import search

DEFAULT_MODEL = "gpt-3.5-turbo-0125"


def research(
goal: str,
scrape_content_split_chunk_size: int = 800,
scrape_content_split_chunk_overlap: int = 225,
top_k_per_query: int = 8,
) -> str:
with st.status("Generating subqueries"):
# Generate subqueries out of the original question, to have more chances of finding relevant information.
queries = generate_subqueries(query=goal, limit=5, model=DEFAULT_MODEL)
st.write(f"Generated subqueries:" + "\n- " + "\n- ".join(queries))

with st.status("Searching the web"):
# For each subquery, do a search using Tavily.
search_results = [r for _, r in search(queries)]

if not search_results:
raise ValueError(f"No search results found for the goal {goal}.")

st.write(
f"Found the following relevant results"
+ "\n- "
+ "\n- ".join(set(result.url for result in search_results))
)

with st.status(f"Scraping web results"):
# Scrap content of each page.
scraped = [
result
for result in scrape_results(search_results)
if result.content.strip()
]
st.write(f"Scraped content from {len(scraped)} websites")

text_splitter = RecursiveCharacterTextSplitter(
separators=["\n\n", "\n", ". ", " "],
chunk_size=scrape_content_split_chunk_size,
chunk_overlap=scrape_content_split_chunk_overlap,
)

with st.status(f"Performing similarity searches"):
# Chunk scraped contents into chunks and create embeddings for them using OpenAI.
collection = create_embeddings_from_results(scraped, text_splitter)
st.write("Created embeddings")

vector_result_texts: list[str] = []

# For each subquery, do a similarity search against the chunks and collect the most relevant ones.
for query in queries:
top_k_per_query_results = collection.similarity_search(
query, k=top_k_per_query
)
vector_result_texts += [
result.page_content
for result in top_k_per_query_results
if result.page_content not in vector_result_texts
]
st.write(f"Similarity searched for: {query}")

st.write(f"Found {len(vector_result_texts)} relevant information chunks")

with st.status(f"Preparing report"):
# Prepare a report based on all relevant collected information.
report = prepare_report(goal, vector_result_texts, model=DEFAULT_MODEL)
st.markdown(report)

return report


def predict(question: str) -> bool | None:
"""
Customize this function to make a prediction about the question.
You can keep using `research` and `_make_prediction` functions and just improve them, or create something new entirely.
"""
# Create a report about the question.
report = research(goal=question)
# Make an informed prediction based on the report.
prediction = _make_prediction(
market_question=question,
additional_information=report,
engine=DEFAULT_MODEL,
temperature=0.0,
)
# Answer yes if p_yes > 0.5, no otherwise.
return (
prediction.outcome_prediction.p_yes > 0.5
if prediction.outcome_prediction
else None
)