Skip to content

Add files via upload#248

Open
quoteven wants to merge 1 commit into
FlagAI-Open:mainfrom
quoteven:main
Open

Add files via upload#248
quoteven wants to merge 1 commit into
FlagAI-Open:mainfrom
quoteven:main

Conversation

@quoteven

Copy link
Copy Markdown

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces task-specific prompt routing (M03, CoT) and dynamic example selection strategies (M05, M09, M10, M11, M19, M20) to optimize in-context learning (ICL) annotations. It also switches the annotation backend to Huawei Ascend with differentiated temperature and token configurations. However, several critical issues were identified: caching examples_str defeats dynamic retrieval and breaks the mixed context length strategy; a typo in the regex pattern (<think&rt;) prevents thinking process removal; hardcoded absolute paths limit portability; unsafe indexing of example['output'][0] can truncate string outputs; repeatedly initializing the tokenizer creates a performance bottleneck; and the calculate_similarity function is duplicated across multiple methods.

Comment on lines 78 to +80
if examples_str is None:
examples_str = select_examples(icl_examples, task_description, text2annotate)
# M11优化:使用Task 7 Jeopardy线索拆解策略
examples_str = select_examples_M11(icl_examples, task_description, text2annotate, task_id, sample_idx)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Caching examples_str with if examples_str is None: completely defeats the purpose of dynamic retrieval (e.g., select_examples_M11 which selects examples based on the current text2annotate). It also breaks the mixed context length strategy: since sample_idx = 0 uses a 30k context, the selected 30k token examples will be reused for all subsequent samples, bypassing the 8k context efficiency optimization for sample_idx >= 50 and potentially causing out-of-memory or context limit issues.

        # M11优化:使用Task 7 Jeopardy线索拆解策略
        examples_str = select_examples_M11(icl_examples, task_description, text2annotate, task_id, sample_idx)

Comment on lines +584 to +585
# Step 1: 过滤<think&rt;标签内容(去思维链)
cleaned_result = re.sub(r'<think&rt;.*?</think&rt;', '', whole_result, flags=re.DOTALL)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The regex pattern contains a typo <think&rt; and </think&rt;. Standard reasoning models output <think> and </think>. This typo will prevent the thinking process from being matched and removed, leading to incorrect code extraction or parsing failures.

Suggested change
# Step 1: 过滤<think&rt;标签内容(去思维链)
cleaned_result = re.sub(r'<think&rt;.*?</think&rt;', '', whole_result, flags=re.DOTALL)
# Step 1: 过滤<think>标签内容(去思维链)
cleaned_result = re.sub(r'<think>.*?</think>', '', whole_result, flags=re.DOTALL)

Comment on lines 12 to 21
TASK_FILES = {
1: './data/openseek-1_closest_integers.json',
2: './data/openseek-2_count_nouns_verbs.json',
3: './data/openseek-3_collatz_conjecture.json',
4: './data/openseek-4_conala_concat_strings.json',
5: './data/openseek-5_semeval_2018_task1_tweet_sadness_detection.json',
6: './data/openseek-6_mnli_same_genre_classification.json',
7: './data/openseek-7_jeopardy_answer_generation_all.json',
8: '../data/openseek-8_kernel_generation.json',
1: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-1_closest_integers.json',
2: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-2_count_nouns_verbs.json',
3: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-3_collatz_conjecture.json',
4: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-4_conala_concat_strings.json',
5: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-5_semeval_2018_task1_tweet_sadness_detection.json',
6: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-6_mnli_same_genre_classification.json',
7: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-7_jeopardy_answer_generation_all.json',
8: '/root/OpenSeek/openseek/competition/LongContext-ICL-Annotation/data/openseek-8_kernel_generation.json',
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Hardcoding absolute paths like /root/OpenSeek/... makes the codebase non-portable and will cause FileNotFoundError on other environments. It is highly recommended to resolve these paths dynamically relative to the script's location.

_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
_DATA_DIR = os.path.join(os.path.dirname(_SCRIPT_DIR), 'data')

TASK_FILES = {
    1: os.path.join(_DATA_DIR, 'openseek-1_closest_integers.json'),
    2: os.path.join(_DATA_DIR, 'openseek-2_count_nouns_verbs.json'),
    3: os.path.join(_DATA_DIR, 'openseek-3_collatz_conjecture.json'),
    4: os.path.join(_DATA_DIR, 'openseek-4_conala_concat_strings.json'),
    5: os.path.join(_DATA_DIR, 'openseek-5_semeval_2018_task1_tweet_sadness_detection.json'),
    6: os.path.join(_DATA_DIR, 'openseek-6_mnli_same_genre_classification.json'),
    7: os.path.join(_DATA_DIR, 'openseek-7_jeopardy_answer_generation_all.json'),
    8: os.path.join(_DATA_DIR, 'openseek-8_kernel_generation.json'),
}

Comment on lines +653 to +654
input_text = example['input']
output_text = example['output'][0]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Accessing example['output'][0] assumes that example['output'] is always a list. If example['output'] is a string (e.g., "Good Review"), this will silently return only the first character (e.g., "G"), resulting in truncated labels in the prompt. This issue also exists in select_examples_M19, select_examples_M09, select_examples_M10, and select_examples_M11.

Suggested change
input_text = example['input']
output_text = example['output'][0]
input_text = example['input']
output_text = example['output'][0] if isinstance(example['output'], list) else example['output']

Comment on lines +638 to +639
# 初始化tokenizer
tokenizer = AutoTokenizer.from_pretrained("/root/Qwen3-4B", trust_remote_code=True)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Initializing the tokenizer from disk inside the example selection function will cause a massive performance bottleneck if called per-sample during dynamic retrieval. It is highly recommended to use a global tokenizer singleton (similar to get_tokenizer_m20) or pass the pre-loaded tokenizer as an argument.

Suggested change
# 初始化tokenizer
tokenizer = AutoTokenizer.from_pretrained("/root/Qwen3-4B", trust_remote_code=True)
# 使用全局tokenizer单例避免重复加载
tokenizer = get_tokenizer_m20()

Comment on lines +628 to +636
def calculate_similarity(text1: str, text2: str) -> float:
"""计算文本相似度(基于词重叠)"""
words1 = set(re.findall(r'\w+', text1.lower()))
words2 = set(re.findall(r'\w+', text2.lower()))
if not words1 or not words2:
return 0.0
intersection = words1.intersection(words2)
union = words1.union(words2)
return len(intersection) / len(union) if union else 0.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The calculate_similarity function is defined identically inside select_examples_M05, select_examples_M20 (as calculate_similarity_m20), and select_examples_M19 (as calculate_similarity_m19). Defining it once at the module level will improve maintainability and reduce code duplication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant