Conversation
…, and resilience experiment report
…ility evaluation suite - Expand Korean POS dataset in get_dataset.sh with OPUS-100 and KLUE task splits (DP, NER, MRC, NLI, RE, STS, YNAT) and update lane_metadata.json - Add --pos_loss_weight argument in train_args.py and handle POS loss weighting and milestone checkpoint saving in train.py - Add --mc_ckpt and --base_ckpt path override support to benchmarks/run_phonetic_slang_eval.py and benchmarks/run_vocab_tail_perplexity.py - Add 4-capability evaluation benchmark suite (benchmarks/run_four_capability_evals.py) covering KLUE-NER, KLUE-DP, noisy text resilience (NSMC/UnSmile), and rare vocabulary/OOV (KorMedMCQA) - Add evaluation runner demos/run_all_epoch_evals.sh, Option 1 sweep runner run_option1_sweep.py, and 10-epoch experiment runner run_opt1_10ep_experiment.py
… POS tagsets, and benchmark support - Implement HangulFullPosFactorizedTokenizer (46 Sejong tags) and HangulCoarsePosFactorizedTokenizer (17 mapped macro tags) in hangul_factorizer.py - Add make_byte_fallback_meta() to support 256-byte companion character stream without OOV drop - Update POS lane metadata and unit tests in test_hangul_factorizer.py - Add 59.5M token milestone checkpoint saves (3ep: 10899, 5ep: 18165, 10ep: 36330) in train.py - Add prepare_pos_and_byte_lanes.py to prepare Full POS and 256-Byte Fallback companion stream - Add run_pos_byte_experiments.py automation runner for training and evaluating Full vs Coarse POS under Weighted and Unweighted loss - Update evaluation benchmarks (run_four_capability_evals.py, run_ko_hellaswag.py, run_vocab_tail_perplexity.py, run_phonetic_slang_eval.py) to support byte fallback and full/coarse POS models
… and tokenizer byte-fallback improvements - Add curated 20-prompt OOV and Unicode benchmark suite in benchmarks/prompts/ covering archaic Hangul, rare Hanja, ancient scripts, complex emojis, and mathematical notation - Add benchmarks/run_oov_evaluations.py to evaluate baseline and multicontext models on OOV prompts - Optimize CharBPETokenizerWithByteFallback encoding loop with length bucketing and interval progress updates - Update benchmark encoders across capability tests to properly use get_tokenizer_functions - Update milestone checkpoint save iterations in train.py - Update .gitignore to track benchmark prompt files
…m overhead, training pipeline, and downstream evaluation suite - Add HangulHybridPosTokenizer in data/template/utils/korean/hangul_pos_hybrid_tokenizer.py - Surface-level script segmentation: pure Korean segments factorized into 22 phonetic/articulatory lanes + 1 Kiwi POS lane - Non-Korean segments tokenized into Lane 0 via SentencePiece byte-fallback BPE (vocab 4,096) with 256-byte fallback - Korean syllable steps flagged with <hangul> user symbol in Lane 0, zero-padded in factor lanes - 100% lossless roundtrip decoding across Korean, English, numbers, symbols, emojis, and Hanja - Add prepare_hybrid_pos_lanes.py with 8-worker parallel Kiwi preprocessing and lane bin generation (37,075,473 sequence tokens, ~37% sequence compression) - Add run_nochar_hybrid_pos_experiments.py multi-stage training pipeline (3, 5, 10 epochs) across coarse/full POS and weighted/unweighted loss - Update benchmarks (run_four_capability_evals.py, run_ko_hellaswag.py, run_phonetic_slang_eval.py, run_vocab_tail_perplexity.py, run_oov_evaluations.py) to support 24-lane hybrid models - Add run_comprehensive_evaluation_suite.py evaluating all 12 checkpoint variants and 10 baselines TAG=agy CONV=93668715-1772-4398-915c-c5b81edba888
…ree-Hot Tokenizer, NSMC adversarial, and KLUE zero-shot evaluations - Add English-to-Korean Seq2Seq training pipeline for 23-lane Hangul Factorizer vs Three-Hot Tokenizer (EACL 2023 conditional RNN and independent heads) - Add subcharacter canonicalization, BPJ (Bits-Per-Jamo), BLEU, and chrF order 18 evaluation metrics - Add adversarial noise and slang benchmark on NSMC (uncorrupted vs 80%-corrupted) - Add OOV contextual safety stress test battery (Middle Korean, complex Hanja, Emoji ligatures, Zalgo, logic notation) - Add zero-shot probability evaluation suite on KLUE-NER and KLUE-DP - Include comprehensive comparative benchmark report TAG=agy CONV=6107adad-8ad5-480d-9a0a-6e7f85e34df5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Overview
This PR introduces a comprehensive evaluation and benchmark suite comparing our 23-Lane Hangul Factorizer against the Three-Hot Tokenizer introduced in Cognetta et al. (EACL 2023), alongside the 24-lane hybrid Hangul factorizer architecture without character stream overhead.
Key Changes
benchmarks/seq2seq_hangul_comparison/):Seq2SeqThreeHotConditional(3-step unrolled RNN:Seq2SeqThreeHotIndependent(3 independent projection heads)Seq2SeqHangulFactorizer(23 multi-lane parallel heads)benchmarks/seq2seq_hangul_comparison/benchmark_report.mdandREADME.mdcontaining all tables, architectural diagrams, and analysis.Benchmark Highlights