Error executing job with overrides: ['model.sample_rate=16000', 'train_dataset=datasets/dev_clean.json', 'validation_datasets=datasets/test_clean.json', 'phoneme_dict_path=conf/ipa_cmudict-0.7b_nv23.01.txt', 'heteronyms_path=conf/heteronyms-052722', 'trainer.max_epochs=4', 'trainer.accelerator=gpu', 'trainer.check_val_every_n_epoch=1', 'trainer.devices=1']
Traceback (most recent call last):
  File "/home/sergey/vits.py", line 32, in <module>
    main()  # noqa pylint: disable=no-value-for-parameter
  File "/home/sergey/.local/lib/python3.10/site-packages/nemo/core/config/hydra_runner.py", line 126, in wrapper
    _run_hydra(
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/sergey/.local/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/sergey/vits.py", line 28, in main
    trainer.fit(model)
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
    call._call_and_handle_interrupt(
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
    return function(*args, **kwargs)
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
    results = self._run_stage()
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
    self.fit_loop.run()
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
    self.advance()
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 137, in run
    self.on_advance_end(data_fetcher)
  File "/home/sergey/.local/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 274, in on_advance_end
    should_check_val = self._should_check_val_fx(data_fetcher)
TypeError: SkipResumeTrainingValidationLoop._should_check_val_fx() takes 1 positional argument but 2 were given
Epoch 0:   1%|          | 1/84 [00:06<08:22,  0.17it/s, v_num=9-43, train_step_timing in s=3.740]         
Environment
  
Current environment
CUDA:

- GPU:

- Tesla T4

- available:         True

- version:           12.1
Lightning:

- lightning-utilities: 0.9.0

- pytorch-lightning: 2.1.0

- torch:             2.1.0

- torchaudio:        2.1.0

- torchmetrics:      1.2.0

- torchvision:       0.16.0
Packages:

- absl-py:           2.0.0

- aiohttp:           3.8.6

- aiosignal:         1.3.1

- alabaster:         0.7.13

- aniso8601:         9.0.1

- antlr4-python3-runtime: 4.9.3

- appdirs:           1.4.4

- asciitree:         0.3.3

- asttokens:         2.4.1

- async-timeout:     4.0.3

- attrdict:          2.0.1

- attrs:             23.1.0

- audioread:         3.0.1

- babel:             2.13.1

- beautifulsoup4:    4.12.2

- black:             19.10b0

- boto3:             1.28.78

- botocore:          1.31.78

- braceexpand:       0.1.7

- cachetools:        5.3.2

- cdifflib:          1.2.6

- certifi:           2023.7.22

- cffi:              1.16.0

- charset-normalizer: 3.3.2

- click:             8.0.2

- colorama:          0.4.6

- comm:              0.1.4

- contourpy:         1.2.0

- cycler:            0.12.1

- cython:            3.0.5

- datasets:          2.14.6

- decorator:         5.1.1

- dill:              0.3.7

- distance:          0.1.3

- docker-pycreds:    0.4.0

- docopt:            0.6.2

- docutils:          0.20.1

- editdistance:      0.6.2

- einops:            0.7.0

- exceptiongroup:    1.1.3

- executing:         2.0.1

- faiss-cpu:         1.7.4

- fasteners:         0.19

- fasttext:          0.9.2

- filelock:          3.13.1

- flask:             2.2.5

- flask-restful:     0.3.10

- fonttools:         4.44.0

- frozenlist:        1.4.0

- fsspec:            2023.10.0

- ftfy:              6.1.1

- g2p-en:            2.1.0

- gdown:             4.7.1

- gitdb:             4.0.11

- gitpython:         3.1.40

- google-auth:       2.23.4

- google-auth-oauthlib: 1.1.0

- grpcio:            1.59.2

- h5py:              3.10.0

- huggingface-hub:   0.18.0

- hydra-core:        1.3.2

- idna:              3.4

- ijson:             3.2.3

- imagesize:         1.4.1

- inflect:           7.0.0

- iniconfig:         2.0.0

- ipython:           8.17.2

- ipywidgets:        8.1.1

- isort:             5.12.0

- itsdangerous:      2.1.2

- jedi:              0.19.1

- jieba:             0.42.1

- jinja2:            3.1.2

- jiwer:             2.5.2

- jmespath:          1.0.1

- joblib:            1.3.2

- jupyterlab-widgets: 3.0.9

- kaldi-python-io:   1.2.2

- kaldiio:           2.18.0

- kiwisolver:        1.4.5

- kornia:            0.7.0

- latexcodec:        2.0.1

- lazy-loader:       0.3

- levenshtein:       0.22.0

- librosa:           0.10.1

- lightning-utilities: 0.9.0

- llvmlite:          0.41.1

- loguru:            0.7.2

- lxml:              4.9.3

- markdown:          3.5.1

- markdown-it-py:    3.0.0

- markdown2:         2.4.10

- markupsafe:        2.1.3

- marshmallow:       3.20.1

- matplotlib:        3.8.1

- matplotlib-inline: 0.1.6

- mdurl:             0.1.2

- megatron-core:     0.3.0

- mpmath:            1.3.0

- msgpack:           1.0.7

- multidict:         6.0.4

- multiprocess:      0.70.15

- nemo-text-processing: 0.2.2rc0

- nemo-toolkit:      1.21.0rc0

- networkx:          3.2.1

- nltk:              3.8.1

- numba:             0.58.1

- numcodecs:         0.12.1

- numpy:             1.23.5

- nvidia-cublas-cu12: 12.1.3.1

- nvidia-cuda-cupti-cu12: 12.1.105

- nvidia-cuda-nvrtc-cu12: 12.1.105

- nvidia-cuda-runtime-cu12: 12.1.105

- nvidia-cudnn-cu12: 8.9.2.26

- nvidia-cufft-cu12: 11.0.2.54

- nvidia-curand-cu12: 10.3.2.106

- nvidia-cusolver-cu12: 11.4.5.107

- nvidia-cusparse-cu12: 12.1.0.106

- nvidia-nccl-cu12:  2.18.1

- nvidia-nvjitlink-cu12: 12.3.52

- nvidia-nvtx-cu12:  12.1.105

- oauthlib:          3.2.2

- omegaconf:         2.3.0

- onnx:              1.15.0

- opencc:            1.1.6

- packaging:         23.2

- pandas:            2.1.2

- pangu:             4.0.6.1

- parameterized:     0.9.0

- parso:             0.8.3

- pathspec:          0.11.2

- pathtools:         0.1.2

- pexpect:           4.8.0

- pillow:            10.1.0

- pip:               23.3.1

- plac:              1.4.1

- platformdirs:      3.11.0

- pluggy:            1.3.0

- pooch:             1.8.0

- portalocker:       2.8.2

- progress:          1.6

- prompt-toolkit:    3.0.39

- protobuf:          4.23.4

- psutil:            5.9.6

- ptyprocess:        0.7.0

- pure-eval:         0.2.2

- pyannote.core:     5.0.0

- pyannote.database: 5.0.1

- pyannote.metrics:  3.2.1

- pyarrow:           14.0.0

- pyasn1:            0.5.0

- pyasn1-modules:    0.3.0

- pybind11:          2.11.1

- pybtex:            0.24.0

- pybtex-docutils:   1.0.3

- pycparser:         2.21

- pydantic:          1.10.13

- pydub:             0.25.1

- pygments:          2.16.1

- pynini:            2.1.5

- pyparsing:         3.1.1

- pypinyin:          0.49.0

- pypinyin-dict:     0.6.0

- pysocks:           1.7.1

- pytest:            7.4.3

- pytest-runner:     6.0.0

- python-dateutil:   2.8.2

- pytorch-lightning: 2.1.0

- pytz:              2023.3.post1

- pyyaml:            6.0.1

- rapidfuzz:         2.13.7

- regex:             2023.10.3

- requests:          2.31.0

- requests-oauthlib: 1.3.1

- rich:              13.6.0

- rouge-score:       0.1.2

- rsa:               4.9

- ruamel.yaml:       0.18.5

- ruamel.yaml.clib:  0.2.8

- s3transfer:        0.7.0

- sacrebleu:         2.3.1

- sacremoses:        0.1.1

- safetensors:       0.4.0

- scikit-learn:      1.3.2

- scipy:             1.11.3

- sentence-transformers: 2.2.2

- sentencepiece:     0.1.99

- sentry-sdk:        1.34.0

- setproctitle:      1.3.3

- setuptools:        68.2.2

- shellingham:       1.5.4

- six:               1.16.0

- smmap:             5.0.1

- snowballstemmer:   2.2.0

- sortedcontainers:  2.4.0

- soundfile:         0.12.1

- soupsieve:         2.5

- sox:               1.4.1

- soxr:              0.3.7

- sphinx:            7.2.6

- sphinxcontrib-applehelp: 1.0.7

- sphinxcontrib-bibtex: 2.6.1

- sphinxcontrib-devhelp: 1.0.5

- sphinxcontrib-htmlhelp: 2.0.4

- sphinxcontrib-jsmath: 1.0.1

- sphinxcontrib-qthelp: 1.0.6

- sphinxcontrib-serializinghtml: 1.1.9

- stack-data:        0.6.3

- sympy:             1.12

- tabulate:          0.9.0

- tensorboard:       2.15.1

- tensorboard-data-server: 0.7.2

- tensorstore:       0.1.45

- termcolor:         2.3.0

- text-unidecode:    1.3

- textdistance:      4.6.0

- texterrors:        0.4.4

- threadpoolctl:     3.2.0

- tokenizers:        0.13.3

- toml:              0.10.2

- tomli:             2.0.1

- torch:             2.1.0

- torchaudio:        2.1.0

- torchmetrics:      1.2.0

- torchvision:       0.16.0

- tqdm:              4.66.1

- traitlets:         5.13.0

- transformers:      4.33.3

- triton:            2.1.0

- typed-ast:         1.5.5

- typer:             0.9.0

- typing-extensions: 4.8.0

- tzdata:            2023.3

- urllib3:           2.0.7

- wandb:             0.15.12

- wcwidth:           0.2.9

- webdataset:        0.1.62

- werkzeug:          3.0.1

- wget:              3.2

- wheel:             0.41.3

- widgetsnbextension: 4.0.9

- wrapt:             1.15.0

- xxhash:            3.4.1

- yarl:              1.9.2

- youtokentome:      1.0.6

- zarr:              2.16.1
System:

- OS:                Linux

- architecture:

- 64bit

- ELF

- processor:

- python:            3.10.0

- release:           4.19.0-25-cloud-amd64

- version:           Proposal for help #1 SMP Debian 4.19.289-2 (2023-08-08)
More info
No response