Panic when there is CJK character in the document

[ghost]

When trying the demo code with CJK characters,

writer.add_document(tantivy.Document(
    title=["老人与海"],
    body=[""" ..."""],
))

the thread panic at

thread '' panicked at 'assertion failed: self.is_char_boundary(new_len)', /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9\src\libcore\macros\mod.rs:10:9

[fulmicoton]

The bug is confiremd.

The problem is in the implementation of Debug for Document. https://github.com/tantivy-search/tantivy-py/blob/master/src/document.rs#L102

@acc557 We will fix this swiftly. In the meanwhile, the workaround is fairly obvious...

You cannot call repr(doc). But you can do anything else.

print(doc["title"]) for instance, should work fine.

Also note that tantivy-py does not come with a japanese tokenizer.
Tantivy has a good and maintained tokenizer called Lindera. If you know rust, you may have to compile your own version of tantivy-py.

[ghost]

Thank you for the really fast response!
I also created another issue #25 about tokenizers.