NVFP4 e2e training with Wikitext

[matthiasdiener]

To run it:

$ pip install datasets regex transformers

$ python -c "
from datasets import load_dataset
import json
ds = load_dataset('wikitext', 'wikitext-103-raw-v1', split='train')
with open('wikitext103_train.jsonl', 'w') as f:
    for row in ds:
        text = row['text'].strip()
        if text:
            json.dump({'text': text}, f)
            f.write('\n')
"

$ python tools/preprocess_data.py \
    --input wikitext103_train.jsonl \
    --output-prefix wikitext103_train_text_document \
    --tokenizer-type HuggingFaceTokenizer \
    --tokenizer-model NousResearch/Meta-Llama-3-8B \
    --workers 8 \
    --append-eod

$ ./‎run_loss_wikitext.sh

$ ./plot_losses.py

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.