span_info on combined unicode characters

[kkaiser]

This fixes issue: #81

Lowering a sentence with combined unicode chars changes the length of a sentence.

s = 'İ love Big Apple and Bay Area.'
len(s)  # 30
len(s.lower())  # 31

Lowering keywords and search sentence now works on a per char basis to return the correct span_info

from flashtext import KeywordProcessor
keyword_processor = KeywordProcessor()
keyword_processor.add_keyword('Big Apple', 'New York')
keyword_processor.add_keyword('Bay Area')
keyword_processor.add_keyword('İ love')
s = 'İ love Big Apple and Bay Area.'
keywords_found = keyword_processor.extract_keywords(s, span_info=True)
keywords_found
# old: [('İ love', 0, 7), ('New York', 8, 17), ('Bay Area', 22, 30)]
# new: [('İ love', 0, 6), ('New York', 7, 16), ('Bay Area', 21, 29)]
for k in keywords_found:
    print(s[k[1]:k[2]])
# new: İ love
# old: İ love
# new: Big Apple
# old: ig Apple
# new: Bay Area
# old: ay Area.

[coveralls]

Coverage increased (+0.02%) to 99.327% when pulling 40633c9 on kkaiser:master into 50c45f1 on vi3k6i5:master.

[coveralls]

Coverage increased (+0.02%) to 99.327% when pulling 40633c9 on kkaiser:master into 50c45f1 on vi3k6i5:master.

[vi3k6i5]

Moving the check inside the loop ads a little of execution overhead, please share a test case for the change. Thanks

[vi3k6i5]

Can you please resolve the conflict.

[kkaiser]

Ready for review

[laurenegerton]

This is still happening with flashtext-2.7
Looks like the fix was never merged with master...

[spencertollefson]

@vi3k6i5 - Hi there! Is this ready for and can you complete merging this PR?