New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use continuous batching by default #882

Open

Wovchena wants to merge 80 commits into openvinotoolkit:master from Wovchena:use-continuos-batching-by-default

+28 −6

Collaborator

Wovchena commented Sep 19, 2024

No description provided.


          Use continuous batching by default

ec5f305

Wovchena requested a review from andrei-kochin

September 19, 2024 10:33

andrei-kochin added do_not_merge do_not_review labels


          Merge branch 'master' into use-continuos-batching-by-default

dd7a5cf

andrei-kochin marked this pull request as ready for review

September 19, 2024 14:52


          Merge branch 'master' into use-continuos-batching-by-default

229b7c5

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 20, 2024 16:41


          Update src/cpp/src/llm_pipeline.cpp

41d1fe7


          Update src/cpp/src/llm_pipeline.cpp

36150c4

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin and others added 3 commits

September 20, 2024 17:32


          Update src/cpp/src/llm_pipeline.cpp

4a4a09e


          Update src/cpp/src/llm_pipeline.cpp

1a58b5e


          Reorder cout

90d81e6

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 20, 2024 18:48


          Update src/cpp/src/llm_pipeline.cpp

6dc43a3


          Update src/cpp/src/llm_pipeline.cpp

03e2f32

ilya-lavrenov added this to the 2024.5 milestone

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 20, 2024 19:03


          Update src/cpp/src/llm_pipeline.cpp

e561e93


          Update src/cpp/src/llm_pipeline.cpp

37ea2ad

andrei-kochin approved these changes

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 23, 2024 11:45


          Update src/cpp/src/llm_pipeline.cpp

b62aee9


          Update src/cpp/src/llm_pipeline.cpp

07505b3

andrei-kochin reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 23, 2024 11:57


          Update src/cpp/src/llm_pipeline.cpp

e078818


          Update src/cpp/src/llm_pipeline.cpp

001d3a0

andrei-kochin requested changes

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added 2 commits

September 25, 2024 18:32


          Update src/cpp/src/llm_pipeline.cpp

a0a964f


          Update src/cpp/src/llm_pipeline.cpp

3cb2105

ilya-lavrenov added 4 commits

November 11, 2024 12:52


          Update windows.yml

35f4ff2


          Update mac.yml

ad78839


          Update linux.yml

c5201e4


          Merge branch 'master' into use-continuos-batching-by-default

d8d397a

ilya-lavrenov mentioned this pull request

Fixed endless generations in test_callback_one_string test #1189

Merged

ilya-lavrenov added a commit that referenced this pull request


          Fixed endless generations in test_callback_one_string test (#1189)

29f78de

See
https://productionresultssa7.blob.core.windows.net/actions-results/1d9c7ed7-f60f-463a-8dba-2898f6f39e0a/workflow-job-run-2ad6080a-b02b-5d2d-feb5-675d9367578e/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-11T12%3A35%3A03Z&sig=cE1uxQm2H0TswczNt4jESUwM1%2BzN2uQBHxsm7Tqw0AI%3D&ske=2024-11-11T22%3A38%3A18Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-11T10%3A38%3A18Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-11T12%3A24%3A58Z&sv=2024-08-04

A lot of words are printed as max_new_tokens is quite huge number, while
exception is expected as streamer cannot work with batch.

To fix hanging of tests in this PR
#882


          Merge branch 'master' into use-continuos-batching-by-default

dc2673c

github-actions bot removed the category: continuous batching label

ilya-lavrenov added 3 commits

November 11, 2024 23:23


          Update llm_pipeline.cpp

4dd053c


          Merge branch 'master' into use-continuos-batching-by-default

31ea070


          Update llm_pipeline.cpp

ilya-lavrenov reviewed

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

ilya-lavrenov added 3 commits

November 12, 2024 16:45


          Apply suggestions from code review

d11db7e


          Update llm_pipeline.cpp

9acf368


          Update causal_lm_cpp.yml

3c835af

ilya-lavrenov added this to the 2025.0 milestone

github-actions bot removed the category: GHA label


          Merge branch 'master' into use-continuos-batching-by-default

046c017

ilya-lavrenov added a commit to ilya-lavrenov/openvino.genai that referenced this pull request


          Fixed endless generations in test_callback_one_string test (openvinot…

9509c9f

…oolkit#1189)

See
https://productionresultssa7.blob.core.windows.net/actions-results/1d9c7ed7-f60f-463a-8dba-2898f6f39e0a/workflow-job-run-2ad6080a-b02b-5d2d-feb5-675d9367578e/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-11T12%3A35%3A03Z&sig=cE1uxQm2H0TswczNt4jESUwM1%2BzN2uQBHxsm7Tqw0AI%3D&ske=2024-11-11T22%3A38%3A18Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-11T10%3A38%3A18Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-11T12%3A24%3A58Z&sv=2024-08-04

A lot of words are printed as max_new_tokens is quite huge number, while
exception is expected as streamer cannot work with batch.

To fix hanging of tests in this PR
openvinotoolkit#882


          Merge branch 'master' into use-continuos-batching-by-default

badaa5b

github-actions bot removed the category: sampling label


          Fix validation

c807011

Contributor

ilya-lavrenov commented Nov 25, 2024

for some reason only a single test is run:

======================== 1 failed, 3 warnings in 45.75s ========================

@Wovchena any ideas?

Collaborator Author

Wovchena commented Nov 25, 2024

It was meant to run a single test only: the command is python -m pytest -v ./tests/python_tests/test_chat_generate_api.py::test_set_chat_template. The full testing is in the next step.


          Update linux.yml

github-actions bot added the category: GHA label

ilya-lavrenov and others added 4 commits

November 25, 2024 11:07


          Update windows.yml

3538bbe


          Merge branch 'master' into use-continuos-batching-by-default

9eb4176


          Update linux.yml

a510e77


          Update windows.yml

eb0b0f4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: GHA category: LLM do_not_merge do_not_review