How can I use multiple spacy.train files in one training run?

Question

I've downloaded the UD Treebank dataset, set up a shell script to discover all folders for a given language and converted the .conllu files to .spacy.

Now I have a collection of files like this: de_gsd_ud-train.spacy, de_hdt-ud-train.spacy, ... I'd like to use all of them together to train a new nlp pipeline.

I have configured a training config, and it seems to work well on a single .spacy file (well, two, one for train and one for dev). But it's not clear to me how I can pass multiple files for the trainset.

Here's the failing command for training:

python -m spacy train "$FILLED_CONFIG" --output "$OUTPUT_DIR" --gpu-id 0 --verbose \
  --paths.train "${TRAIN_SPACY_FILES[@]}" \
  --paths.dev "${DEV_SPACY_FILES[@]}"

This will make spacy complain :

✘ Invalid config override './training_data/de_gsd-ud-train.spacy': name
should start with --

So, it parses the second training file (the de_gsd-ud-train.spacy) as a new config override and complains that this config override is not starting with --.

Ivan Kleshnin · Accepted Answer · 2025-02-18 08:41:56Z

You should rather concatenate treebanks, transforming them into a single .spacy data file. You can do the concatenation manually, but fortunately Spacy supports that.

Split your treebanks between train, dev, test folders.
Update convert/preprocess script to call spacy convert with --concatenate flag

project.yml

python -m spacy convert
  assets/EWT/train/ # <-- points to a folder, not a '*.conllu' file
  corpus/EWT/ 
  --converter conllu 
  --n-sents 10 
  --concatenate

The name of the output file will match the name of the last folder (train for the above). Update deps and output accordingly:

project.yml

deps:
  - "assets/EWT/train/ewt.conllu"
  - "assets/EWT/train/extra.conllu"
  - "assets/EWT/dev/ewt.conllu"
  - "assets/EWT/dev/extra.conllu"
  - "assets/EWT/test/ewt.conllu"
  - "assets/EWT/test/extra.conllu"
outputs:
  - "corpus/EWT/train.spacy"
  - "corpus/EWT/dev.spacy"
  - "corpus/EWT/test.spacy"

Recheck your script for unnecessary logic. Due to the above naming, the cleanup lines (present in default templates) are unnecessary:

project.yml

- "mv corpus/EWT/... corpus/EWT/train.spacy" # -- junk, remove this from the `preprocess`/`convert` weasel command

The output file is already named train.spacy and sits in the right folder.

Collectives™ on Stack Overflow

How can I use multiple spacy.train files in one training run?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related