podcast script generation component #15

daavoo · 2024-11-21T12:18:17Z

What's changing

Added new text_to_podcast module.
Updated demo/app.py to be able to try different OLMoE quatitized versions and system prompts.

How to test it

Create a new Codespace using the New with options:

Select this branch (AH-104-Initial-Podcast-Script-Generation-Component).

Inside the codespace, run:

pip install -e . --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

python -m streamlit run demo/app.py

…t-Script-Generation-Component

Kostis-S-Z

I do like the API docs, but I guess we can take that at a later stage. Maybe we can create an item for documentation specifically
I would be happy with simple smoke tests like

assert isinstantce(load_model(...), Llama)
assert isinstantce(text_to_podcast(...), str)

but again, we can take it later if we feel like we are in a rush for an MVP.

Kostis-S-Z

I ll approve it cause there is nothing blocking here so feel free to leave the docs&tests for a later or the current PR.

stefanfrench · 2024-11-21T16:21:03Z

Thanks @daavoo. This is looking good and worked smoothy in Codespaces!

A couple notes from my side before approval:

It would be good if there was a clear/easy way for the developer to input an alternative hf model (both in the backend as a parameter and also in the app as free-text). We should still keep the OLMOE model q8 as the default option. do you see any issues with this
I think we should try to set the text limit from the input doc in a bit of a more rigorous way, e.g. doing it based on the actual number of real tokens as you suggest in your comment. If you prefer we can do this as a separate issue
I think we should do some more experimentation to find a good default prompt to make the script generated feel more natural. I think the current one will feel very robotic once it is audio generated. If you prefer, we can create a seperate issue for this as well. (also - happy to support on this prompt experimentation is its quite time consuming!)

demo/app.py

daavoo · 2024-11-22T09:43:18Z

demo/app.py

-        st.write(clean_text[:200])
+        st.text_area(f"Total Length: {len(clean_text)}", f"{clean_text[:500]} . . .")
+
+    # I set this value as a quick safeguard but we should actually tokenize the text and count the number of real tokens.


@stefanfrench :

I think we should try to set the text limit from the input doc in a bit of a more rigorous way, e.g. doing it based on the actual number of real tokens as you suggest in your comment. If you prefer we can do this as a separate issue

I am currently looking into this. I am trying to find a way to use the llama_cpp API to don't waste the tokenization call just for the sake of filtering.
If I don't find an easy solution today, maybe we can consider it a follow-up to not block the full PR

@daavoo - sounds good!

I gave it a try (to encode first) but the code became way more complicated.
However, it seems that people consider 1 token ~= 4 characters a common default and it is the value used when trying to estimate token consumption without expending calls.

So, I updated the code to use this 4 approximation and made a small improvement to use the context lenght associated to each model (before I was too lazy and just hardcoded the number to 4096 as it was the value for OLMoE)

… token as a resonable default.

tests/integration/test_model_load_and_inference.py

Kostis-S-Z

Thank you for adding the tests, all LGTM!

* Add devcontainer and requirements * Add pyproject.toml * Add data_loaders and tests * Add data_cleaners and tests * Update demo * Add `LOADERS` and `CLEANERS` * Add markdown and docx * Add API Reference * Update tests * Update install * Add initial scripts * More tests * fix merge * Add podcast writing to demo/app * Add missing deps * Add text_to_podcast module * Expose model options and prompt tuning in the app * pre-commit * Strip system_prompt * Rename to inference module. Add docstrings * pre-commit * Add CURATED_REPOS * JSON prompt * Update API docs * Fix format * Make text cutoff based on `model.n_ctx()`. Consider ~4 characters per token as a resonable default. * Add inference tests * Drop __init__ imports * Fix outdated arg * Drop redundant JSON output in prompt * Update default stop

* update read.me with guidance docs initial draft * minor update to read.me * podcast script generation component (#15) * Add devcontainer and requirements * Add pyproject.toml * Add data_loaders and tests * Add data_cleaners and tests * Update demo * Add `LOADERS` and `CLEANERS` * Add markdown and docx * Add API Reference * Update tests * Update install * Add initial scripts * More tests * fix merge * Add podcast writing to demo/app * Add missing deps * Add text_to_podcast module * Expose model options and prompt tuning in the app * pre-commit * Strip system_prompt * Rename to inference module. Add docstrings * pre-commit * Add CURATED_REPOS * JSON prompt * Update API docs * Fix format * Make text cutoff based on `model.n_ctx()`. Consider ~4 characters per token as a resonable default. * Add inference tests * Drop __init__ imports * Fix outdated arg * Drop redundant JSON output in prompt * Update default stop * updates to read.me to simplify down and add diagram * update read.me with guidance docs initial draft * minor update to read.me * updates to read.me to simplify down and add diagram * updated docs and added new pages and assets * Changes to the docs files * deleting contributing.md from docs * Add tests workflow (#18) * Add new `tests` workflow * Use pip cache * Unify env setup. Drop UV in favor of setup-python * Update tests * podcast script generation component (#15) * Add devcontainer and requirements * Add pyproject.toml * Add data_loaders and tests * Add data_cleaners and tests * Update demo * Add `LOADERS` and `CLEANERS` * Add markdown and docx * Add API Reference * Update tests * Update install * Add initial scripts * More tests * fix merge * Add podcast writing to demo/app * Add missing deps * Add text_to_podcast module * Expose model options and prompt tuning in the app * pre-commit * Strip system_prompt * Rename to inference module. Add docstrings * pre-commit * Add CURATED_REPOS * JSON prompt * Update API docs * Fix format * Make text cutoff based on `model.n_ctx()`. Consider ~4 characters per token as a resonable default. * Add inference tests * Drop __init__ imports * Fix outdated arg * Drop redundant JSON output in prompt * Update default stop * pre commit checks * Apply suggestions from code review * Update docs/step-by-step-guide.md * lint * changes based on peer reviews * changes based pre commit checks --------- Co-authored-by: David de la Iglesia Castro <[email protected]>

daavoo added 16 commits November 14, 2024 12:21

Add devcontainer and requirements

536c98d

Add pyproject.toml

0ae661e

Add data_loaders and tests

c4a1ee1

Add data_cleaners and tests

d2b276c

Update demo

8629481

Add LOADERS and CLEANERS

cef92b3

Add markdown and docx

acd50a9

Add API Reference

2a8f005

Update tests

95c342a

Update install

e8ac586

Add initial scripts

ee7d299

More tests

fb38207

Merge remote-tracking branch 'origin/main' into AH-104-Initial-Podcas…

29df436

…t-Script-Generation-Component

fix merge

d4b6066

Add podcast writing to demo/app

abeb3c0

Add missing deps

4bcc57b

daavoo linked an issue Nov 21, 2024 that may be closed by this pull request

Initial Podcast Script Generation Component #4

Closed

daavoo added 3 commits November 21, 2024 14:01

Add text_to_podcast module

06627fa

Expose model options and prompt tuning in the app

4457813

pre-commit

c73d4d3

daavoo self-assigned this Nov 21, 2024

daavoo marked this pull request as ready for review November 21, 2024 14:29

daavoo requested review from Kostis-S-Z and stefanfrench November 21, 2024 14:29

Strip system_prompt

a868093

Kostis-S-Z reviewed Nov 21, 2024

View reviewed changes

Kostis-S-Z approved these changes Nov 21, 2024

View reviewed changes

daavoo added 2 commits November 22, 2024 08:49

Rename to inference module. Add docstrings

8b2c57b

pre-commit

d2c75c9

Add CURATED_REPOS

7a7e39c

daavoo commented Nov 22, 2024

View reviewed changes

demo/app.py Show resolved Hide resolved

daavoo commented Nov 22, 2024

View reviewed changes

demo/app.py Show resolved Hide resolved

daavoo commented Nov 22, 2024

View reviewed changes

daavoo added 5 commits November 22, 2024 11:05

JSON prompt

e1c6ccb

Update API docs

72413a1

Fix format

8817ea0

Make text cutoff based on model.n_ctx(). Consider ~4 characters per…

06a2c3d

… token as a resonable default.

Add inference tests

39fb3b3

daavoo requested a review from Kostis-S-Z November 25, 2024 10:03

daavoo commented Nov 25, 2024

View reviewed changes

tests/integration/test_model_load_and_inference.py Outdated Show resolved Hide resolved

daavoo added 3 commits November 25, 2024 11:06

Drop __init__ imports

1968278

Fix outdated arg

f88b713

Drop redundant JSON output in prompt

73ac7bf

Kostis-S-Z approved these changes Nov 25, 2024

View reviewed changes

Update default stop

1c11c98

daavoo merged commit 3fc7ff1 into main Nov 25, 2024
1 check passed

daavoo deleted the AH-104-Initial-Podcast-Script-Generation-Component branch November 25, 2024 12:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podcast script generation component #15

podcast script generation component #15

daavoo commented Nov 21, 2024 •

edited

Loading

Kostis-S-Z left a comment

Kostis-S-Z left a comment

stefanfrench commented Nov 21, 2024 •

edited

Loading

daavoo Nov 22, 2024

stefanfrench Nov 22, 2024

daavoo Nov 25, 2024

Kostis-S-Z left a comment

podcast script generation component #15

podcast script generation component #15

Conversation

daavoo commented Nov 21, 2024 • edited Loading

What's changing

How to test it

Kostis-S-Z left a comment

Choose a reason for hiding this comment

Kostis-S-Z left a comment

Choose a reason for hiding this comment

stefanfrench commented Nov 21, 2024 • edited Loading

daavoo Nov 22, 2024

Choose a reason for hiding this comment

stefanfrench Nov 22, 2024

Choose a reason for hiding this comment

daavoo Nov 25, 2024

Choose a reason for hiding this comment

Kostis-S-Z left a comment

Choose a reason for hiding this comment

daavoo commented Nov 21, 2024 •

edited

Loading

stefanfrench commented Nov 21, 2024 •

edited

Loading