Skip to content

Add OpenSeek submission for LongContext-ICL-Annotation#249

Open
hdxdfff wants to merge 3 commits into
FlagAI-Open:mainfrom
hdxdfff:submission-openseek-longcontext-icl
Open

Add OpenSeek submission for LongContext-ICL-Annotation#249
hdxdfff wants to merge 3 commits into
FlagAI-Open:mainfrom
hdxdfff:submission-openseek-longcontext-icl

Conversation

@hdxdfff

@hdxdfff hdxdfff commented May 27, 2026

Copy link
Copy Markdown

Summary

  • Add the OpenSeek submission for the LongContext-ICL-Annotation challenge.
  • Include the final prediction archive, the final technical report, and the final source-code archive used for the competition submission.
  • Also include an extracted code/ directory for direct review and reproduction.
  • Final confirmed platform score for this submission line: 87.35.

Compliance

  • Uses Qwen3-4B as the only large language model.
  • Uses FlagScale as the deployment and inference framework.
  • Uses only the official competition data.
  • Does not rely on fine-tuning, external data, or external models.
  • Public materials have been cleaned to remove unnecessary internal diagnostics and account-specific bindings, while keeping the final submitted artifacts unchanged.

Reproduction

  • See openseek/competition/LongContext-ICL-Annotation/submissions/OpenSeek/README.md for the submission overview.
  • See openseek/competition/LongContext-ICL-Annotation/submissions/OpenSeek/code/README.md for environment setup, model deployment, inference, packaging, and validation instructions.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the complete OpenSeek submission codebase, including configuration files, prompt templates, utility scripts, source code for retrieval, decision-making, and pipeline execution, as well as tests and a technical report. The code review identified several critical issues and improvement opportunities: incorrect dataset paths (RAW_TASK8) in scripts that would cause FileNotFoundErrors, potential runtime NameErrors in fallback code generation, a bug where max_samples being 0 would be treated as falsy, and a logger that creates a directory but fails to write logs to a file. Additionally, feedback was provided to enforce strict error handling when packaging missing prediction files and to clean up unused dead code.



ROOT = Path(__file__).resolve().parents[1]
RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The path to RAW_TASK8 is incorrect. According to the directory structure defined in data/README.md and the behavior of import_official_data.sh, the official dataset files are imported into data/raw/openseek/. Therefore, the path should include the openseek subdirectory to avoid a FileNotFoundError when running this script.

Suggested change
RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json"
RAW_TASK8 = ROOT / "data/raw/openseek/openseek-8_kernel_generation.json"

from build_task8_fallback_submission import _extract_name_and_args # noqa: E402


RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The path to RAW_TASK8 is incorrect. The official dataset files are imported into data/raw/openseek/ by import_official_data.sh. The path should include the openseek subdirectory to avoid a FileNotFoundError when running this script.

Suggested change
RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json"
RAW_TASK8 = ROOT / "data/raw/openseek/openseek-8_kernel_generation.json"

names = _param_names(args)
lname = name.lower()
desc = text.lower()
first = names[0] if names else "input"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If names is empty (e.g., when args is "*args, **kwargs"), setting first to "input" will cause a NameError during execution because input is not defined in the function signature. Since the parameter is *args, using "args[0]" instead of "input" is a robust fallback that correctly references the first positional argument.

Suggested change
first = names[0] if names else "input"
first = names[0] if names else "args[0]"

if _has(names, "x") and _has(names, "conv_weight"):
expr = "torch.nn.functional.conv2d(x, conv_weight, conv_bias, conv_stride, conv_padding, conv_dilation, conv_groups)"
else:
expr = "torch.nn.functional.conv2d(input, weight, bias, stride, padding, dilation, groups)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Hardcoding optional parameters like bias, stride, padding, dilation, and groups directly as positional arguments can cause a NameError if they are not defined in the wrapper's signature. Using locals().get(name, default) is a much more robust approach that safely retrieves these parameters if they exist, or falls back to their PyTorch defaults.

Suggested change
expr = "torch.nn.functional.conv2d(input, weight, bias, stride, padding, dilation, groups)"
expr = "torch.nn.functional.conv2d(input, weight, locals().get('bias', None), locals().get('stride', 1), locals().get('padding', 0), locals().get('dilation', 1), locals().get('groups', 1))"

Comment on lines +441 to +442
if missing:
logger.warning("Packaging with missing prediction files: %s", ", ".join(missing))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Packaging the submission when required prediction files are missing will result in an incomplete and invalid submission package. Instead of just logging a warning, the script should raise a FileNotFoundError to fail the packaging process and alert the user immediately.

Suggested change
if missing:
logger.warning("Packaging with missing prediction files: %s", ", ".join(missing))
if missing:
raise FileNotFoundError(f"Cannot package submission. Missing required prediction files: {', '.join(missing)}")

protocols = _load_protocol_set(config["prompt"], dataset.task_type, dataset.task_name)
outputs: List[Dict[str, Any]] = []
max_samples = config["runtime"].get("max_samples_per_task")
sample_records = dataset.test_samples[: int(max_samples)] if max_samples else dataset.test_samples

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using if max_samples to check if max_samples is set can lead to incorrect behavior if max_samples_per_task is set to 0 in the configuration (as 0 is falsy in Python, it would fall back to evaluating all samples instead of 0). It is safer and more idiomatic to explicitly check if max_samples is not None.

Suggested change
sample_records = dataset.test_samples[: int(max_samples)] if max_samples else dataset.test_samples
sample_records = dataset.test_samples[: int(max_samples)] if max_samples is not None else dataset.test_samples

from typing import Optional


class TokenCounter:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The TokenCounter class and the maybe_build_token_counter function are completely unused in the codebase. Additionally, the chunk_budget_tokens parameter in configs/base.yaml is also unused. Removing dead code and unused configuration parameters improves maintainability and clarity.

Comment on lines +5 to +16
def build_logger(log_dir: str, name: str = "ai-lab") -> logging.Logger:
Path(log_dir).mkdir(parents=True, exist_ok=True)
logger = logging.getLogger(name)
logger.setLevel(logging.INFO)

if not logger.handlers:
handler = logging.StreamHandler()
formatter = logging.Formatter(
"%(asctime)s | %(levelname)s | %(name)s | %(message)s"
)
handler.setFormatter(formatter)
logger.addHandler(handler)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The build_logger function creates the log_dir directory but only configures a StreamHandler to log to the console. It does not add a FileHandler to actually write the logs to a file in log_dir. Adding a FileHandler ensures that logs are persisted for future diagnostics.

    log_path = Path(log_dir)
    log_path.mkdir(parents=True, exist_ok=True)
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)

    if not logger.handlers:
        formatter = logging.Formatter(
            "%(asctime)s | %(levelname)s | %(name)s | %(message)s"
        )
        
        stream_handler = logging.StreamHandler()
        stream_handler.setFormatter(formatter)
        logger.addHandler(stream_handler)
        
        file_handler = logging.FileHandler(log_path / "run.log", encoding="utf-8")
        file_handler.setFormatter(formatter)
        logger.addHandler(file_handler)

from typing import Any, Dict, List


def format_example(example: Dict[str, Any]) -> str:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The functions format_example, build_output_instruction, and build_prompt are unused in the codebase. Removing unused utility functions keeps the codebase clean and maintainable.

@hdxdfff hdxdfff changed the title Add openseek submission for LongContext-ICL-Annotation Add OpenSeek submission for LongContext-ICL-Annotation May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant