Skip to content

feat: add new long-context ICL annotation method#261

Open
xiamaozi11 wants to merge 1 commit into
FlagAI-Open:mainfrom
xiamaozi11:my-annotation-method
Open

feat: add new long-context ICL annotation method#261
xiamaozi11 wants to merge 1 commit into
FlagAI-Open:mainfrom
xiamaozi11:my-annotation-method

Conversation

@xiamaozi11

Copy link
Copy Markdown

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a comprehensive automatic data annotation solution for the OpenSeek competition (Tasks 1-8), featuring hybrid ICL retrieval, spaCy-based POS tagging, and ReAct-style verification loops. The code review feedback highlights critical security concerns regarding hardcoded DeepSeek API keys in build_cot_training_set.py and deepseek_client.py. Additionally, there are opportunities to improve portability by removing hardcoded CUDA_VISIBLE_DEVICES settings, clean up leftover debugging print statements in the Task 5 inference scripts, and enhance the robustness of the output comparison script by supporting alternative field names.

Comment on lines +368 to +369
if not args.dry_run and not api_key:
api_key = 'sk-340c73f8209e4ba3ae44c7e6cf570506'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Hardcoding API keys is a severe security risk. It is highly recommended to remove the hardcoded key and raise an error if the DEEPSEEK_API_KEY environment variable is not set.

Suggested change
if not args.dry_run and not api_key:
api_key = 'sk-340c73f8209e4ba3ae44c7e6cf570506'
if not args.dry_run and not api_key:
raise ValueError("DEEPSEEK_API_KEY environment variable is not set.")


from openai import OpenAI

DEEPSEEK_API_KEY = "sk-340c73f8209e4ba3ae44c7e6cf570506"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Hardcoding API keys is a severe security risk. Please retrieve the API key from the environment variable DEEPSEEK_API_KEY instead.

Suggested change
DEEPSEEK_API_KEY = "sk-340c73f8209e4ba3ae44c7e6cf570506"
DEEPSEEK_API_KEY = os.environ.get("DEEPSEEK_API_KEY", "")

continue
try:
row = json.loads(line)
except json.JSONDecodeError:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This print statement appears to be a leftover debugging aid. It should be removed to keep the output clean.

converted.append(EMOJI_TEXT_MAP_ZH.get(ch, _emoji_to_alias(ch)))
else:
converted.append(_emoji_to_alias(ch))
print('converted:', converted)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This print statement appears to be a leftover debugging aid. It should be removed to keep the output clean.

Comment on lines +21 to +26
sample_id = obj.get("test_sample_id")
prediction = obj.get("prediction")
if sample_id is None or prediction is None:
raise ValueError(
f"{path} 第 {line_no} 行缺少字段,必须包含 test_sample_id 和 prediction"
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation only supports test_sample_id and prediction fields. Supporting both test_sample_id/example_id and prediction/model_output makes this comparison script much more robust and compatible with both examples and test samples.

Suggested change
sample_id = obj.get("test_sample_id")
prediction = obj.get("prediction")
if sample_id is None or prediction is None:
raise ValueError(
f"{path} {line_no} 行缺少字段,必须包含 test_sample_id prediction"
)
sample_id = obj.get("test_sample_id") or obj.get("example_id")
prediction = obj.get("prediction") or obj.get("model_output")
if sample_id is None or prediction is None:
raise ValueError(
f"{path} line {line_no} is missing required fields (must contain test_sample_id/example_id and prediction/model_output)"
)

import concurrent.futures
import json
import os
os.environ.setdefault("CUDA_VISIBLE_DEVICES", "2")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Hardcoding CUDA_VISIBLE_DEVICES to a specific index like "2" limits portability and can cause execution failures on systems with fewer GPUs. It is better to let the user configure this via their environment.

Signed-off-by: xiamaozi11 <879989535@qq.com>
@xiamaozi11 xiamaozi11 force-pushed the my-annotation-method branch from 498c1b0 to 6802a68 Compare May 31, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant