Add pin to NeMo Toolkit (#407)

* Add pin to nemo toolkit

Signed-off-by: Sarah Yurick <[email protected]>

* update version

Signed-off-by: Sarah Yurick <[email protected]>

* Update requirements.txt

Signed-off-by: Sarah Yurick <[email protected]>

---------

Signed-off-by: Sarah Yurick <[email protected]>

Author: sarahyurick

requirements/requirements.txt

@@ -13,7 +13,7 @@ jieba==0.42.1
 justext==3.0.1
 lxml_html_clean
 mwparserfromhell==0.6.5
-nemo_toolkit[nlp]>=1.23.0
+nemo_toolkit[nlp]>=1.23.0, <2.0.0
 numpy<2
 openai
 peft

setup.py

@@ -55,7 +55,7 @@ def req_file(filename, folder="requirements"):
 
 setup(
     name="nemo_curator",
-    version="0.5.0",
+    version="0.5.1",
     description="Scalable Data Preprocessing Tool for "
     "Training Large Language Models",
     long_description=long_description,