Fix csv reader encoding issue #53
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… personal/ranxia/csv_reader
… personal/ranxia/csv_reader # Conflicts: # poetry.lock
moria97
reviewed
Jun 6, 2024
| ), | ||
| ".xlsx": PaiPandasExcelReader( | ||
| concat_rows=self.reader_config.get("concat_rows", False), | ||
| pandas_config={ | ||
| "encoding": self.reader_config.get("encoding", "GB18030") |
There was a problem hiding this comment.
使用GB18030加载utf8编码的中文文件会有问题吗？
There was a problem hiding this comment.
是有可能会出问题的，虽然GB18030可以兼容Unicode，但编码机制不一样。最好是在前端页面透出可以让用户填encoding的地方。
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
moria97
reviewed
Jun 6, 2024
moria97
reviewed
Jun 6, 2024
| @@ -82,7 +83,10 @@ def load_data( | |||
| if self._concat_rows: | |||
| return [Document(text="\n".join(text_list), metadata=metadata)] | |||
| else: | |||
| return [Document(text=text, metadata=metadata) for text in text_list] | |||
| return [ | |||
| Document(text=text, metadata={**metadata, **{"row_number": i}}) | |||
There was a problem hiding this comment.
直接metadata["row_number"] = i ? 另外，这里是不是应该用i+1
moria97
reviewed
Jun 6, 2024
| df = pd.read_csv(f, **self._pandas_config) | ||
| encoding = chardet.detect(f.read(100000))["encoding"] | ||
| f.seek(0) | ||
| if encoding.upper() in ["GB18030", "GBK"]: |
moria97
reviewed
Jun 6, 2024
| try: | ||
| df = pd.read_csv(f, **self._pandas_config) | ||
| except UnicodeDecodeError: | ||
| print(f"Error: The file {file} encoding could not be decoded.") |
moria97
reviewed
Jun 6, 2024
| try: | ||
| df = pd.read_csv(file, **self._pandas_config) | ||
| except UnicodeDecodeError: | ||
| print(f"Error: The file {file} encoding could not be decoded.") |
moria97
reviewed
Jun 6, 2024
… personal/ranxia/csv_reader
moria97
approved these changes
Jun 6, 2024
moria97
added a commit
that referenced
this pull request
Jun 14, 2024
* tabular reader * tabular reader * tabular reader * tabular reader * tabular reader * tabular reader * BugFix:add an encoding parameter * BugFix:add an encoding parameter * BugFix:add an encoding parameter * BugFix:add an encoding parameter * BugFix: add row number to metadata * BugFix: add row number to metadata --------- Co-authored-by: Yue Fei <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.