Preprocessing error reporting

[1607722534]

Preprocessing error reporting：
Parse ./data/train/news.tsv
malloc(): invalid next size (unsorted)
已放弃 (核心已转储)

Is it the code or the environment?

[yusanshi]

I'm not sure but looks like a out-of-memory error?

What's the capacity of the memory? Please check the available memory while running the script. And if it runs out of memory, one possible solution would be enlarging the swap size.

[1607722534]

I use two kinds of servers. The memory occupancy rate is less than 3%, and the cup occupancy rate is 100%.

[yusanshi]

Could you find out on which line of code the error occurs? (e.g., use debugging tools, or many print)

[1607722534]

Resource punkt not found.
Please use the NLTK Downloader to obtain the resource:

[yusanshi]

Resource punkt not found.
Please use the NLTK Downloader to obtain the resource:

Please refer to delip/PyTorchNLPBook#14 or https://stackoverflow.com/questions/4867197/failed-loading-english-pickle-with-nltk-data-load.

PS: you can try searching it with search engines first. See this, this and this. (No offense.)

[1607722534]

I got the preprocessed files. Thank you for your patient reply. My undergraduate is not a computer, so my code ability is too poor.
(╥﹏╥)

[yusanshi]

My pleasure. 😁