expose decoder

[pelikhan]

---

Pull Request Summary

Documentation 📚

• Added a new tokenizers.md file to the documentation, providing an overview of the tokenizers module and its functionalities, including count, truncate, and chunk.

Code Enhancements 🛠️

• Refactored resolveTokenEncoder to return an object with encode and decode methods, improving code clarity and consistency.
• Implemented a new chunk function in the encoders module to split text into chunks based on token size, enhancing text processing capabilities.

Tests ✅

• Updated tests to use the new encode method from the resolveTokenEncoder function.
• Added a test for the chunk function to ensure it splits text correctly and maintains line numbers.

Bug Fixes 🐛

• Fixed an issue with promptyParse to correctly handle undefined filenames, improving robustness.

New Features ✨

• Introduced a new sample script chunk.genai.mjs demonstrating the use of the chunk function to process and summarize text chunks.

Miscellaneous 🔄

• Updated various function calls to align with the new resolveTokenEncoder return structure.

generated by prd

[github-actions]

Investigator report

• run failure

, 5af3bff

generated by gai