Code

Lumi changes#26

Code

Draft

ZJaume wants to merge 35 commits intomasterparacrawl/cirrus-scripts:masterfrom lumiparacrawl/cirrus-scripts:lumiCopy head branch name to clipboard

+330-58

Commits

Commits on Apr 26, 2023

• Ignore vim swap files

  ZJaume

  committed
  78818c7

• Update giashard install instructions

  Show description for e56da94
  ZJaume

  committed
  e56da94

Commits on May 9, 2023

• LUMI config environment

  ZJaume

  committed
  10db860

Commits on May 11, 2023

• Add translate-bergamot.sh script

  ZJaume

  committed
  da369d3

• Override env variables for DOCALIGN and BLEUALIGN

  ZJaume

  committed
  a91e303

• Fix bicleaner env variables setup

  ZJaume

  committed
  02aeadf

Commits on May 12, 2023

• Fix typo

  ZJaume

  committed
  605b176

• Stop requesting more memory on align

  ZJaume

  committed
  c5d150f

• Increase mem-per-cpu on small partition

  ZJaume

  committed
  f3ca994

Commits on May 15, 2023

• Write directly to $SCRATCH instead of node temp in align and split

  ZJaume

  committed
  1477b17

• Cut out docalign scores before docjoin

  Show description for 92f0d38
  ZJaume

  committed
  92f0d38

• Docjoin does not expect docalign score in Bitextor 8.1

  ZJaume

  committed
  96f2ccf

• Ignore the model directory of individual language pairs

  ZJaume

  committed
  9b23df1

Commits on May 16, 2023

• Remove pairs.txt debugging output in 06.align

  ZJaume

  committed
  93f95f3

Commits on May 19, 2023

• Set path to stable bitextor env

  ZJaume

  committed
  2cc6047

• Set HF cache for bicleaner-ai

  ZJaume

  committed
  d1cbfc1

• Add install script for paracrawl cpp code

  ZJaume

  committed
  4cfef0b

• Update bifixer and kenlm installations

  ZJaume

  committed
  78bff5c

• Override binaries locations in init not setup

  ZJaume

  committed
  e6c3d57

• Use foldfilter in translate-bergamot

  ZJaume

  committed
  2c489e3

Commits on Jun 6, 2023

• Undo executables env variables overrided in init instead of config

  ZJaume

  committed
  c5046eb

Commits on Jul 6, 2023

• Fix scoring step

  ZJaume

  committed
  de9dbfa

• Fix 09.clean step

  Show description for 65a7094
  ZJaume

  committed
  65a7094

• Create cleaning directory if it doesn't exist

  ZJaume

  committed
  b665411

• Fix 11.reduce filtered step

  Show description for 04e918f
  ZJaume

  committed
  04e918f

• Fix 12.reduce.tmx

  Show description for 7bcdbd9
  ZJaume

  committed
  7bcdbd9

Commits on Jul 18, 2023

• 07.fix read model data from huggingface cache

  jelmervdl

  committed
  aa69628

• Remove cache from tokenize pipeline

  jelmervdl

  committed
  036e160

• Add --will-cite where necessary

  jelmervdl

  committed
  9fbe192

• Fix bergamot model configuration path

  jelmervdl

  committed
  7821e8d

• Add LUMI base module

  jelmervdl

  committed
  d822f60

Commits on Sep 26, 2024

• Use singularity container

  ZJaume

  committed
  5bc7971

• Use loomchild segmenter and text.gz

  ZJaume

  committed
  0511f8a

• Adapt translate-bergamot to marian container path and use config.yml

  Show description for d4e0acf
  ZJaume

  committed
  d4e0acf

• Use tokeniser.perl inside the container

  ZJaume

  committed
  615ef60