Comparative Analysis of the Intrinsic Metrics for Tokenizers and their effect on Downstream Tasks for Hindi and Marathi auto_eval contains code for the automated evaluation framework for the question answering tasks eval contains code for model inference for QA, and inference + evaluation for word level tasks train_llm contains code for training T5 for QA tasks and the word-level tasks training_tokenizers contains code for training and implementation of all tokenizers used in the study.