Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hotfix #18

Closed
wants to merge 34 commits into from
Closed

hotfix #18

wants to merge 34 commits into from

Conversation

Saibo-creator
Copy link
Collaborator

No description provided.

@Saibo-creator
add unicode test
191aa3a
@Saibo-creator
support parsing unicode
1036680
@Saibo-creator
use dynamical char_acceptance_at_element implementation
ab2f2bd
@Saibo-creator
change rule_offsets from dict to array, aligning with llama-cpp
f1b6e7e
@Saibo-creator
more flexible GrammarRecognizer construct, aligning with llama-cpp
79511b5
@Saibo-creator
add utf8 decoding utils
d0c2ccf
@Saibo-creator
add a bit more on decode utf8, including two NotImplementedError:
1f3807f 
1. _consume_byte_partial_match
2. _consume_bytes_partial_match
@Saibo-creator
unicode acceptance is working
783b1bc
@Saibo-creator
fix repetition_penalty issue; add tests for running examples
704e5a5
@Saibo-creator
v0.1.2
bb7225f
@Saibo-creator
add github workflow, add badges on README and skip example testing lo…
ebc6c20 
…cally
@Saibo-creator
install pytest dependency in github workflow
369cf8c
@Saibo-creator
update requirements.txt
7d6767d
@Saibo-creator
set numpy version to 1.24.2
9900dda
@Saibo-creator
del workflow
15f154f
@Saibo-creator
support parsing unicode
b762873
@Saibo-creator
use dynamical char_acceptance_at_element implementation
af65c70
@Saibo-creator
add utf8 decoding utils
c8e9ee6
@Saibo-creator
add a bit more on decode utf8, including two NotImplementedError:
a524120 
1. _consume_byte_partial_match
2. _consume_bytes_partial_match
@Saibo-creator
unicode acceptance is working
15c446d
@Saibo-creator
fix repetition_penalty issue; add tests for running examples
e11c442
@Saibo-creator
add github workflow, add badges on README and skip example testing lo…
da1afb7 
…cally
@Saibo-creator
add test_accept_unicode_bytes.py
90b96f5 
refactor tests
@Saibo-creator
add ByteEncoding
1d146e5
@Saibo-creator
unicode is working, but very slow and the last stack element has problem
222ddf8
@Saibo-creator
unicode is working and fast: reset_utf8_decode is reduces the speed f…
1ded206 
…rom 40s to 15s
@Saibo-creator
freeze generate_json.py, repetition_penalty=1.1 is critical
be527cf
@Saibo-creator
add generate_c_code.py and fix test_grammar_constrained_decoding.py
d22c432
@Saibo-creator
Two modes: unicode(slower) or non-unicode(fast)
6373590 
All tests except t5 are passed
next step: add llama tokenizer support for unicode
@Saibo-creator
add more languages
d2f0ebf
@Saibo-creator Saibo-creator deleted the dev_unicode_1 branch February 29, 2024 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Saibo-creator