Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More amy/anysplit modifications #1337

Merged
merged 23 commits into from
Aug 13, 2022
Merged

More amy/anysplit modifications #1337

merged 23 commits into from
Aug 13, 2022

Commits on Aug 12, 2022

  1. amy/4.0.affix,amy/4.0.regex: Simplify the regexes 

    This doesn't work yet for splitting on grapheme boundaries, because ^X
matches at leas one codepoint so it matches a split initial morpheme in a part.

This change is needed for the upcoming new code to split at grapheme
boundaries.
    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    5825dd2 View commit details
    Browse the repository at this point in the history

  2. anysplit(): Move the sanity checks to the start

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    36901d0 View commit details
    Browse the repository at this point in the history

  3. free_anysplit(): Move it to be near its usage

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    f3c7d8a View commit details
    Browse the repository at this point in the history

  4. morpheme_match(): Rename prefix_string to word_part

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    8df81c5 View commit details
    Browse the repository at this point in the history

  5. anysplit(): Remove commented-out code line

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    b22706c View commit details
    Browse the repository at this point in the history

  6. anysplit,c: Fix a comment rot

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    97fd0c7 View commit details
    Browse the repository at this point in the history

  7. anysplit.c: Rename p_start to p_end

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    73eaff6 View commit details
    Browse the repository at this point in the history

  8. morpheme_match(): Update description

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    af8e481 View commit details
    Browse the repository at this point in the history

  9. anysplit.c: Define D_ANYS as the verbosity level for this file

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    1576152 View commit details
    Browse the repository at this point in the history

  10. anysplit(): Move 0 length check to the start

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    128fccd View commit details
    Browse the repository at this point in the history

  11. anysplit.c: Include pcre2.h

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    53dd29f View commit details
    Browse the repository at this point in the history

  12. anysplit.c: Add data structure for grapheme separation

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    aa3e7f1 View commit details
    Browse the repository at this point in the history

  13. anysplit.c: Add functions for grapheme separation

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    3193e89 View commit details
    Browse the repository at this point in the history

  14. anysplit.c: Add ability to split on grapheme boundaries

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    cf6a498 View commit details
    Browse the repository at this point in the history

  15. anysplit.c: Add #define atomic-unit instead of a hardcoded value

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    c295754 View commit details
    Browse the repository at this point in the history

  16. amy/4.0.affix: Remove regexes for REG* 

    No need for them after the grapheme-aware separation modification.
    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    74a1468 View commit details
    Browse the repository at this point in the history

  17. amy/4.0.regex: Include trailing mark codepoints in atomic-unit 

    This way morpheme candidates (split parts) are not starting with marks.
This looks nicer and gives less splits. I don't know it is more useful.
    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    54423ab View commit details
    Browse the repository at this point in the history

  18. anysplit.c: Change p_end to unsigned int

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    e236030 View commit details
    Browse the repository at this point in the history

  19. any/affix_punc: Replace one-char affixes by [[:punct:]]

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    82a97b0 View commit details
    Browse the repository at this point in the history

  20. any/affix_punc: Add comments

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    ff92f6a View commit details
    Browse the repository at this point in the history

  21. amy/4.0.regex: Accept subscripted punctuation

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    8b9f32e View commit details
    Browse the repository at this point in the history

  22. amy/4.0.regex: Update the comments

    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    9fb80f0 View commit details
    Browse the repository at this point in the history

  23. afdict_init(): Validate affixes w/dictionary_word_is_known() 

    ...instead of dict_has_word(), to allow punctuation that match a regex.
    @ampli
    ampli committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    56247cb View commit details
    Browse the repository at this point in the history