Add files via upload#248
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces task-specific prompt routing (M03, CoT) and dynamic example selection strategies (M05, M09, M10, M11, M19, M20) to optimize in-context learning (ICL) annotations. It also switches the annotation backend to Huawei Ascend with differentiated temperature and token configurations. However, several critical issues were identified: caching examples_str defeats dynamic retrieval and breaks the mixed context length strategy; a typo in the regex pattern (<think&rt;) prevents thinking process removal; hardcoded absolute paths limit portability; unsafe indexing of example['output'][0] can truncate string outputs; repeatedly initializing the tokenizer creates a performance bottleneck; and the calculate_similarity function is duplicated across multiple methods.
| if examples_str is None: | ||
| examples_str = select_examples(icl_examples, task_description, text2annotate) | ||
| # M11优化:使用Task 7 Jeopardy线索拆解策略 | ||
| examples_str = select_examples_M11(icl_examples, task_description, text2annotate, task_id, sample_idx) |
There was a problem hiding this comment.
Caching examples_str with if examples_str is None: completely defeats the purpose of dynamic retrieval (e.g., select_examples_M11 which selects examples based on the current text2annotate). It also breaks the mixed context length strategy: since sample_idx = 0 uses a 30k context, the selected 30k token examples will be reused for all subsequent samples, bypassing the 8k context efficiency optimization for sample_idx >= 50 and potentially causing out-of-memory or context limit issues.
# M11优化:使用Task 7 Jeopardy线索拆解策略
examples_str = select_examples_M11(icl_examples, task_description, text2annotate, task_id, sample_idx)| # Step 1: 过滤<think&rt;标签内容(去思维链) | ||
| cleaned_result = re.sub(r'<think&rt;.*?</think&rt;', '', whole_result, flags=re.DOTALL) |
There was a problem hiding this comment.
The regex pattern contains a typo <think&rt; and </think&rt;. Standard reasoning models output <think> and </think>. This typo will prevent the thinking process from being matched and removed, leading to incorrect code extraction or parsing failures.
| # Step 1: 过滤<think&rt;标签内容(去思维链) | |
| cleaned_result = re.sub(r'<think&rt;.*?</think&rt;', '', whole_result, flags=re.DOTALL) | |
| # Step 1: 过滤<think>标签内容(去思维链) | |
| cleaned_result = re.sub(r'<think>.*?</think>', '', whole_result, flags=re.DOTALL) |
| TASK_FILES = { | ||
| 1: './data/openseek-1_closest_integers.json', | ||
| 2: './data/openseek-2_count_nouns_verbs.json', | ||
| 3: './data/openseek-3_collatz_conjecture.json', | ||
| 4: './data/openseek-4_conala_concat_strings.json', | ||
| 5: './data/openseek-5_semeval_2018_task1_tweet_sadness_detection.json', | ||
| 6: './data/openseek-6_mnli_same_genre_classification.json', | ||
| 7: './data/openseek-7_jeopardy_answer_generation_all.json', | ||
| 8: '../data/openseek-8_kernel_generation.json', | ||
| 1: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-1_closest_integers.json', | ||
| 2: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-2_count_nouns_verbs.json', | ||
| 3: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-3_collatz_conjecture.json', | ||
| 4: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-4_conala_concat_strings.json', | ||
| 5: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-5_semeval_2018_task1_tweet_sadness_detection.json', | ||
| 6: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-6_mnli_same_genre_classification.json', | ||
| 7: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-7_jeopardy_answer_generation_all.json', | ||
| 8: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-8_kernel_generation.json', | ||
| } |
There was a problem hiding this comment.
Hardcoding absolute paths like /root/OpenSeek/... makes the codebase non-portable and will cause FileNotFoundError on other environments. It is highly recommended to resolve these paths dynamically relative to the script's location.
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
_DATA_DIR = os.path.join(os.path.dirname(_SCRIPT_DIR), 'data')
TASK_FILES = {
1: os.path.join(_DATA_DIR, 'openseek-1_closest_integers.json'),
2: os.path.join(_DATA_DIR, 'openseek-2_count_nouns_verbs.json'),
3: os.path.join(_DATA_DIR, 'openseek-3_collatz_conjecture.json'),
4: os.path.join(_DATA_DIR, 'openseek-4_conala_concat_strings.json'),
5: os.path.join(_DATA_DIR, 'openseek-5_semeval_2018_task1_tweet_sadness_detection.json'),
6: os.path.join(_DATA_DIR, 'openseek-6_mnli_same_genre_classification.json'),
7: os.path.join(_DATA_DIR, 'openseek-7_jeopardy_answer_generation_all.json'),
8: os.path.join(_DATA_DIR, 'openseek-8_kernel_generation.json'),
}| input_text = example['input'] | ||
| output_text = example['output'][0] |
There was a problem hiding this comment.
Accessing example['output'][0] assumes that example['output'] is always a list. If example['output'] is a string (e.g., "Good Review"), this will silently return only the first character (e.g., "G"), resulting in truncated labels in the prompt. This issue also exists in select_examples_M19, select_examples_M09, select_examples_M10, and select_examples_M11.
| input_text = example['input'] | |
| output_text = example['output'][0] | |
| input_text = example['input'] | |
| output_text = example['output'][0] if isinstance(example['output'], list) else example['output'] |
| # 初始化tokenizer | ||
| tokenizer = AutoTokenizer.from_pretrained("/root/Qwen3-4B", trust_remote_code=True) |
There was a problem hiding this comment.
Initializing the tokenizer from disk inside the example selection function will cause a massive performance bottleneck if called per-sample during dynamic retrieval. It is highly recommended to use a global tokenizer singleton (similar to get_tokenizer_m20) or pass the pre-loaded tokenizer as an argument.
| # 初始化tokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("/root/Qwen3-4B", trust_remote_code=True) | |
| # 使用全局tokenizer单例避免重复加载 | |
| tokenizer = get_tokenizer_m20() |
| def calculate_similarity(text1: str, text2: str) -> float: | ||
| """计算文本相似度(基于词重叠)""" | ||
| words1 = set(re.findall(r'\w+', text1.lower())) | ||
| words2 = set(re.findall(r'\w+', text2.lower())) | ||
| if not words1 or not words2: | ||
| return 0.0 | ||
| intersection = words1.intersection(words2) | ||
| union = words1.union(words2) | ||
| return len(intersection) / len(union) if union else 0.0 |
There was a problem hiding this comment.
No description provided.