Code

SDP data preparation pipeline#37

Code

Merged

Kipok merged 27 commits intoNVIDIA:mainNVIDIA/NeMo-Skills:mainfrom i-vainn:imoshkov/sdp_pipelinei-vainn/NeMo-Skills:imoshkov/sdp_pipelineCopy head branch name to clipboardJun 27, 2024

+1,341-440

Commits

Commits on Jun 14, 2024

• added sdp pipeline

  i-vainn

  committed
  5c01b51

• removed redundant config

  i-vainn

  committed
  0371406

• added arithmetic processing utils

  i-vainn

  committed
  f31e489

• removed old scripts

  i-vainn

  committed
  7e962d8

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 7a2ccf4
  pre-commit-ci[bot]

  committed
  7a2ccf4

• changed config structure

  i-vainn

  committed
  7cff70f

• formatting

  i-vainn

  committed
  9644c9a

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for b9724d7
  pre-commit-ci[bot]

  committed
  b9724d7

• added dependency

  i-vainn

  committed
  dbf0146

• Merge branch 'imoshkov/sdp_pipeline' of https://github.com/i-vainn/NeMo-Skills into imoshkov/sdp_pipeline

  i-vainn

  committed
  d8b86a4

Commits on Jun 15, 2024

• bug and formatting fixes

  i-vainn

  committed
  6d450fd

Commits on Jun 17, 2024

• fixed tests

  i-vainn

  committed
  7ab03a3

• minor changes

  i-vainn

  committed
  6484d3a

• simplified tests

  i-vainn

  committed
  53c95d8

• fixed tests

  i-vainn

  committed
  469331c

• added test data

  i-vainn

  committed
  055123c

Commits on Jun 19, 2024

• minor improvements

  i-vainn

  committed
  5ba6830

• minor fixes

  i-vainn

  committed
  060e0ef

Commits on Jun 24, 2024

• added metrics, added parallel read data, fixed deduplication bug

  i-vainn

  committed
  69eae1a

• Merge branch 'main' into imoshkov/sdp_pipeline

  i-vainn

  committed
  dbdc72d

Commits on Jun 26, 2024

• fixed deduplication issue and improved processing speed

  i-vainn

  committed
  515b32a

• added sdp to setup and requirements

  

  i-vainn

  committed
  75c745c

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 59b96db
  pre-commit-ci[bot]

  committed
  59b96db

• read requirements list from requirements.txt

  

  i-vainn

  committed
  756d750

• read requirements list from requirements.txt

  

  i-vainn

  committed
  ab0c0bb

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 5ed0948
  pre-commit-ci[bot]

  committed
  5ed0948

• Merge branch 'main' into imoshkov/sdp_pipeline

  Kipok

  authored
  1262484