[Track 3] Submission for LongContext-ICL-Annotation (kima) by HPUhushicheng · Pull Request #260 · FlagAI-Open/OpenSeek

HPUhushicheng · 2026-05-30T13:31:25Z

This PR adds the HPUhushicheng team submission for LongContext-ICL-Annotation.

Included materials:

Technical report: Technical-report-kima.pdf
Complete code: qwen3_experiment/
Usage and reproduction instructions: README.md

Commit record requested by the submitter: kima 's Technical report and code

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a complete Qwen3-4B “experiment” submission for the LongContext-ICL-Annotation competition, including scripts to run all tasks, package results, and an optimized prompting/example-selection/label-extraction pipeline.

Changes:

Introduces main.py + method.py implementing task-aware prompts, BM25/diversity ICL selection, voting, and optional structure alignment.
Adds convenience shell scripts to run all 8 tasks and to zip outputs for submission.
Adds a requirements.txt and a detailed Chinese README with setup and deployment instructions.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
openseek/competition/LongContext-ICL-Annotation/submissions/HPUhushicheng/qwen3_experiment/run_all.sh	Batch script to run all 8 tasks with a shared configuration.
openseek/competition/LongContext-ICL-Annotation/submissions/HPUhushicheng/qwen3_experiment/requirements.txt	Declares Python dependencies for running the submission.
openseek/competition/LongContext-ICL-Annotation/submissions/HPUhushicheng/qwen3_experiment/package_results.sh	Zips generated `openseek--v.jsonl` files for submission.
openseek/competition/LongContext-ICL-Annotation/submissions/HPUhushicheng/qwen3_experiment/method.py	Core logic: prompts, ICL example selection, inference calls, label extraction, and alignment.
openseek/competition/LongContext-ICL-Annotation/submissions/HPUhushicheng/qwen3_experiment/main.py	CLI entry + evaluation loop writing predictions to versioned JSONL outputs.
openseek/competition/LongContext-ICL-Annotation/submissions/HPUhushicheng/README.md	End-to-end instructions for env setup, deployment, running, and packaging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+
+    for ex in all_examples:
+        input_text = ex.get('input', '')
+        output_text = ex.get('output', [''])[0] if isinstance(ex.get('output'), list) else str(ex.get('output'), '')


+    """
+    import requests
+
+    URL = "http://0.0.0.0:2026/v1/completions"


+            _tokenizer_cache[key] = AutoTokenizer.from_pretrained(
+                candidates[0] or "/root/autodl-tmp/qwen3-4b",
+                trust_remote_code=True
+            )


+    parser.add_argument('--enable_alignment', action='store_true', default=True,
+                        help='Enable output structure alignment (default: True).')
+    parser.add_argument('--no_alignment', action='store_true',
+                        help='Disable structure alignment.')


+    examples_str = select_examples(
+        icl_examples, task_description, first_sample_input,
+        task_id=task_id
+    )


+            examples_str = select_examples_simple(
+                icl_examples, task_description, text2annotate,
+                tokenizer_local, 
+                max_context_length=max_input_length - 500,
+                prompt_template_length=600
+            )
+            input_prompt = prompt.replace("[[EXAMPLES]]\n\n", examples_str + '\n\n')


+OUTPUT_ZIP=${2:-result.zip}
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+INPUT_DIR="${SCRIPT_DIR}/${INPUT_DIR}"


+import re
+import math
+import json
+import random


+    tokenizer: AutoTokenizer,
+    max_context_length: int = 100_000,
+    prompt_template_length: int = 500,
+    diversity_factor: float = 0.3


+        "### Instructions\n"
+        "1. Think about the category and clue carefully.\n"
+        "2. Provide the most specific and accurate answer.\n"
+        "3. Answers should be in all lower cased letters.\n"


gemini-code-assist

Code Review

This pull request introduces a solution for the Long-Context ICL Data Annotation challenge, featuring evaluation scripts, core annotation methods, and helper bash scripts. The code review identifies several areas for improvement, including a missing parameter propagation for tokenizer_path, logical errors in argparse configuration, a type annotation mismatch, and performance bottlenecks from imports and file I/O inside loops. Additionally, the feedback highlights robustness issues such as incomplete exception handling, a potential division-by-zero error in BM25, hardcoded API parameters, and path resolution bugs in the bash scripts.

gemini-code-assist · 2026-05-30T13:33:08Z

+    examples_str = select_examples(
+        icl_examples, task_description, first_sample_input,
+        task_id=task_id
+    )


在调用 select_examples 时，没有传入 tokenizer_path 参数。这会导致 select_examples 内部忽略用户通过命令行指定的 --tokenizer_path，转而使用硬编码的默认路径。如果默认路径不存在，可能会导致分词器加载失败或尝试从 Hugging Face 在线下载（在无网环境下会报错）。

建议传入 tokenizer_path=qwen_tokenizer.name_or_path，以确保与主流程使用的分词器路径保持一致。

Suggested change

examples_str = select_examples(

icl_examples, task_description, first_sample_input,

task_id=task_id

)

examples_str = select_examples(

icl_examples, task_description, first_sample_input,

task_id=task_id, tokenizer_path=qwen_tokenizer.name_or_path

)

gemini-code-assist · 2026-05-30T13:33:08Z

+    parser.add_argument('--enable_alignment', action='store_true', default=True,
+                        help='Enable output structure alignment (default: True).')
+    parser.add_argument('--no_alignment', action='store_true',
+                        help='Disable structure alignment.')


--enable_alignment 参数使用了 action='store_true' 且设置了 default=True。在 argparse 中，这会导致该参数无论是否在命令行中指定，其值都始终为 True，从而使该参数失去实际控制作用。

既然已经提供了 --no_alignment 参数来禁用对齐，建议直接移除 --enable_alignment 参数，并在代码中通过 not args.no_alignment 来判断是否启用对齐。这样可以简化参数解析逻辑并避免混淆。

Suggested change

parser.add_argument('--enable_alignment', action='store_true', default=True,

help='Enable output structure alignment (default: True).')

parser.add_argument('--no_alignment', action='store_true',

help='Disable structure alignment.')

parser.add_argument('--no_alignment', action='store_true',

help='Disable structure alignment.')

gemini-code-assist · 2026-05-30T13:33:08Z

+    num_samples: int = 1,
+    max_examples: int = 100,
+    enable_alignment: bool = True,
+) -> float:


函数 evaluate 的返回类型标注为 float，但实际上它在第 154 行返回的是 output_file（一个字符串 str）。这会导致类型检查工具报错。建议将返回类型标注修改为 str。

Suggested change

) -> float:

) -> str:

gemini-code-assist · 2026-05-30T13:33:08Z

+            from method import select_examples_simple
+            tokenizer_local = qwen_tokenizer


from method import select_examples_simple 导入语句位于测试样本的循环内部。由于测试样本数量较多（如 500 个），在循环中重复导入会带来不必要的性能开销。建议将该导入语句移至文件顶部（例如第 13 行附近）。

Suggested change

from method import select_examples_simple

tokenizer_local = qwen_tokenizer

tokenizer_local = qwen_tokenizer

gemini-code-assist · 2026-05-30T13:33:08Z

+        with open(output_file, 'a') as f:
+            f.write(json.dumps(test_record) + '\n')


在循环内部频繁地以追加模式（'a'）打开和关闭输出文件，会产生大量的磁盘 I/O 开销，影响评估效率。

建议在循环外部使用 with open(output_file, 'w') as f: 上下文管理器一次性打开文件，并在循环内部直接进行写入。

gemini-code-assist · 2026-05-30T13:33:08Z

+                tf = doc_freq[qt]
+                idf = self.idf[qt]
+                numerator = tf * (self.k1 + 1)
+                denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avgdl)


如果语料库中的所有文档都为空，self.avgdl 将为 0，这会在计算 doc_len / self.avgdl 时引发 ZeroDivisionError。建议对 self.avgdl 进行非零检查以提高代码的健壮性。

Suggested change

denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avgdl)

denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avgdl) if self.avgdl > 0 else tf + self.k1 * (1 - self.b)

gemini-code-assist · 2026-05-30T13:33:08Z

+    tokenizer: AutoTokenizer,
+    max_context_length: int = 100_000,
+    prompt_template_length: int = 500,
+    diversity_factor: float = 0.3


参数 diversity_factor 在 select_examples_diverse 函数体内完全没有被使用。建议移除该未使用的参数，或者在多样性采样逻辑中实际应用它。

gemini-code-assist · 2026-05-30T13:33:08Z

+            "model": "/root/autodl-tmp/qwen3-4b",
+            "prompt": input_prompt,
+            "max_tokens": 10_000,
+            "temperature": 0.3 if num_samples == 1 else 0.7,
+            "top_p": 0.9,
+        }


在 annotate_nvidia 中，请求体中的 "model" 参数被硬编码为 "/root/autodl-tmp/qwen3-4b"。如果用户在启动 vLLM 服务时使用了不同的模型路径（例如 README 中推荐的 ${MODEL_PATH}）， vLLM 可能会因为模型名称不匹配而拒绝请求。

建议动态从 vLLM 的 /v1/models 接口获取当前加载的模型名称，或者允许通过参数传递，以提高部署的灵活性。

# 尝试动态获取 vLLM 加载的模型名称，获取失败时使用默认值 try: models_resp = requests.get("http://0.0.0.0:2026/v1/models", timeout=5) model_name = models_resp.json()["data"][0]["id"] except Exception: model_name = "/root/autodl-tmp/qwen3-4b" data = { "model": model_name, "prompt": input_prompt, "max_tokens": 10_000, "temperature": 0.3 if num_samples == 1 else 0.7, "top_p": 0.9, }

gemini-code-assist · 2026-05-30T13:33:08Z

+zip -j "${SCRIPT_DIR}/${OUTPUT_ZIP}" openseek-*-v1.jsonl
+
+echo ""
+echo "Package created: ${SCRIPT_DIR}/${OUTPUT_ZIP}"
+echo "Contents:"
+unzip -l "${SCRIPT_DIR}/${OUTPUT_ZIP}"


如果用户传入的 OUTPUT_ZIP 是绝对路径（例如 /tmp/result.zip），"${SCRIPT_DIR}/${OUTPUT_ZIP}" 的拼接方式会导致路径错误。

建议在脚本中判断并正确处理绝对路径。

Suggested change

zip -j "${SCRIPT_DIR}/${OUTPUT_ZIP}" openseek-*-v1.jsonl

echo ""

echo "Package created: ${SCRIPT_DIR}/${OUTPUT_ZIP}"

echo "Contents:"

unzip -l "${SCRIPT_DIR}/${OUTPUT_ZIP}"

if [[ "$OUTPUT_ZIP" = /* ]]; then

ZIP_PATH="$OUTPUT_ZIP"

else

ZIP_PATH="${SCRIPT_DIR}/${OUTPUT_ZIP}"

fi

zip -j "$ZIP_PATH" openseek-*-v1.jsonl

echo ""

echo "Package created: $ZIP_PATH"

echo "Contents:"

unzip -l "$ZIP_PATH"

gemini-code-assist · 2026-05-30T13:33:08Z

+OUTPUT_DIR=${2:-./outputs}
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"


如果用户从不同的工作目录运行 run_all.sh，默认的相对路径 OUTPUT_DIR=${2:-./outputs} 可能会被解析为不同的物理路径，从而导致生成的 JSONL 文件位置与 package_results.sh 预期的路径不一致。

建议在脚本中将相对的 OUTPUT_DIR 转换为相对于 SCRIPT_DIR 的绝对路径。

Suggested change

OUTPUT_DIR=${2:-./outputs}

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

OUTPUT_DIR=${2:-./outputs}

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

if [[ "$OUTPUT_DIR" != /* ]]; then

OUTPUT_DIR="${SCRIPT_DIR}/${OUTPUT_DIR}"

fi

Signed-off-by: HPUhushicheng <h18790492316@163.com>

Copilot AI review requested due to automatic review settings May 30, 2026 13:31

HPUhushicheng force-pushed the submission-kima branch from ea99147 to 4a7212c Compare May 30, 2026 13:32

Copilot AI reviewed May 30, 2026

View reviewed changes

gemini-code-assist Bot reviewed May 30, 2026

View reviewed changes

HPUhushicheng changed the title ~~Add HPUhushicheng LongContext-ICL-Annotation submission~~ [Track 3] Submission for LongContext-ICL-Annotation (kima) May 30, 2026

HPUhushicheng force-pushed the submission-kima branch from 4a7212c to 32cddb2 Compare May 30, 2026 13:41

kima 's Technical report and code

68108a9

Signed-off-by: HPUhushicheng <h18790492316@163.com>

HPUhushicheng force-pushed the submission-kima branch from 32cddb2 to 68108a9 Compare May 30, 2026 13:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Track 3] Submission for LongContext-ICL-Annotation (kima)#260

[Track 3] Submission for LongContext-ICL-Annotation (kima)#260
HPUhushicheng wants to merge 1 commit into
FlagAI-Open:mainfrom
HPUhushicheng:submission-kima

HPUhushicheng commented May 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

gemini-code-assist Bot May 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		from method import select_examples_simple
		tokenizer_local = qwen_tokenizer

		with open(output_file, 'a') as f:
		f.write(json.dumps(test_record) + '\n')

	denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avgdl)
	denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avgdl) if self.avgdl > 0 else tf + self.k1 * (1 - self.b)

-zip -j "${SCRIPT_DIR}/${OUTPUT_ZIP}" openseek-*-v1.jsonl
-echo ""
-echo "Package created: ${SCRIPT_DIR}/${OUTPUT_ZIP}"
-echo "Contents:"
-unzip -l "${SCRIPT_DIR}/${OUTPUT_ZIP}"
+if [[ "$OUTPUT_ZIP" = /* ]]; then
+    ZIP_PATH="$OUTPUT_ZIP"
+else
+    ZIP_PATH="${SCRIPT_DIR}/${OUTPUT_ZIP}"
+fi
+zip -j "$ZIP_PATH" openseek-*-v1.jsonl
+echo ""
+echo "Package created: $ZIP_PATH"
+echo "Contents:"
+unzip -l "$ZIP_PATH"

		OUTPUT_DIR=${2:-./outputs}

		SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

Uh oh!

Conversation

HPUhushicheng commented May 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants