Code

联合国6国语料的整条管线接入，爬虫重做#62

Code

Open

voidf wants to merge 25 commits intomainmnbvc-parallel-corpus-team/parallel_corpus_mnbvc:mainfrom pipeline_reworkmnbvc-parallel-corpus-team/parallel_corpus_mnbvc:pipeline_reworkCopy head branch name to clipboard

+2,901-6

Commits

Commits on Mar 24, 2024

• adapt to new web api

  voidf

  committed
  2cf43f3

• upload doc2docx utility

  voidf

  committed
  d7eb056

• upload translate and align utility

  voidf

  committed
  b78d791

• efficient async based get doc utility

  voidf

  committed
  b85781a

Commits on Mar 25, 2024

• distributed computation utilities for translation

  voidf

  committed
  ed70577

• pipeline execution order sample

  voidf

  committed
  a57cb13

Commits on Apr 2, 2024

• add comment for why some files being discarded in final step

  voidf

  committed
  54b0240

• bleu score utility

  voidf

  committed
  c445f3d

Commits on Apr 3, 2024

• support resume from last exit

  voidf

  committed
  6b7b51d

Commits on Jul 9, 2024

• 还是传一下统计用脚本吧，不然过几年真的忘了

  voidf

  committed
  456907b

Commits on Sep 19, 2024

• 处理列数行数规整的grid table，附带dump样例查错脚本

  voidf

  committed
  2fe888c

Commits on Sep 23, 2024

• 表格处理脚本上传

  voidf

  committed
  fa614fc

Commits on Sep 24, 2024

• 为colab准备的脚本

  voidf

  committed
  a683403

Commits on Sep 26, 2024

• 暂存：一种基于字符例外的grid_tables处理方式

  voidf

  committed
  8593808

• 把目前遇到的情况都讨论完了，但多行表似乎会吃掉一些行，正在查

  voidf

  committed
  a82a1d6

• 修复漏掉的末尾处理

  voidf

  committed
  8361f4e

Commits on Sep 27, 2024

• 部署脚本

  voidf

  committed
  5001429

• google cloud shell用脚本

  voidf

  committed
  c2355cc

• client默认关掉cuda

  voidf

  committed
  2db1040

• 脚本换成两步

  voidf

  committed
  c247df8

Commits on Oct 7, 2024

• add check script

  voidf

  committed
  b31dbbc

• update jsonl_checker

  voidf

  committed
  56d61bb

Commits on Oct 19, 2024

• 单元格内双\n\n直接切分

  voidf

  committed
  a11f70a

Commits on Nov 2, 2024

• 两行忘记改了，翻译白跑了

  voidf

  committed
  e1a21c4

Commits on Nov 3, 2024

• 打开几个const变量，不再起一层无意义的别名

  voidf

  committed
  acecb98