Add OpenSeek submission for LongContext-ICL-Annotation#249
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the complete OpenSeek submission codebase, including configuration files, prompt templates, utility scripts, source code for retrieval, decision-making, and pipeline execution, as well as tests and a technical report. The code review identified several critical issues and improvement opportunities: incorrect dataset paths (RAW_TASK8) in scripts that would cause FileNotFoundErrors, potential runtime NameErrors in fallback code generation, a bug where max_samples being 0 would be treated as falsy, and a logger that creates a directory but fails to write logs to a file. Additionally, feedback was provided to enforce strict error handling when packaging missing prediction files and to clean up unused dead code.
|
|
||
|
|
||
| ROOT = Path(__file__).resolve().parents[1] | ||
| RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json" |
There was a problem hiding this comment.
The path to RAW_TASK8 is incorrect. According to the directory structure defined in data/README.md and the behavior of import_official_data.sh, the official dataset files are imported into data/raw/openseek/. Therefore, the path should include the openseek subdirectory to avoid a FileNotFoundError when running this script.
| RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json" | |
| RAW_TASK8 = ROOT / "data/raw/openseek/openseek-8_kernel_generation.json" |
| from build_task8_fallback_submission import _extract_name_and_args # noqa: E402 | ||
|
|
||
|
|
||
| RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json" |
There was a problem hiding this comment.
The path to RAW_TASK8 is incorrect. The official dataset files are imported into data/raw/openseek/ by import_official_data.sh. The path should include the openseek subdirectory to avoid a FileNotFoundError when running this script.
| RAW_TASK8 = ROOT / "data/raw/openseek-8_kernel_generation.json" | |
| RAW_TASK8 = ROOT / "data/raw/openseek/openseek-8_kernel_generation.json" |
| names = _param_names(args) | ||
| lname = name.lower() | ||
| desc = text.lower() | ||
| first = names[0] if names else "input" |
There was a problem hiding this comment.
If names is empty (e.g., when args is "*args, **kwargs"), setting first to "input" will cause a NameError during execution because input is not defined in the function signature. Since the parameter is *args, using "args[0]" instead of "input" is a robust fallback that correctly references the first positional argument.
| first = names[0] if names else "input" | |
| first = names[0] if names else "args[0]" |
| if _has(names, "x") and _has(names, "conv_weight"): | ||
| expr = "torch.nn.functional.conv2d(x, conv_weight, conv_bias, conv_stride, conv_padding, conv_dilation, conv_groups)" | ||
| else: | ||
| expr = "torch.nn.functional.conv2d(input, weight, bias, stride, padding, dilation, groups)" |
There was a problem hiding this comment.
Hardcoding optional parameters like bias, stride, padding, dilation, and groups directly as positional arguments can cause a NameError if they are not defined in the wrapper's signature. Using locals().get(name, default) is a much more robust approach that safely retrieves these parameters if they exist, or falls back to their PyTorch defaults.
| expr = "torch.nn.functional.conv2d(input, weight, bias, stride, padding, dilation, groups)" | |
| expr = "torch.nn.functional.conv2d(input, weight, locals().get('bias', None), locals().get('stride', 1), locals().get('padding', 0), locals().get('dilation', 1), locals().get('groups', 1))" |
| if missing: | ||
| logger.warning("Packaging with missing prediction files: %s", ", ".join(missing)) |
There was a problem hiding this comment.
Packaging the submission when required prediction files are missing will result in an incomplete and invalid submission package. Instead of just logging a warning, the script should raise a FileNotFoundError to fail the packaging process and alert the user immediately.
| if missing: | |
| logger.warning("Packaging with missing prediction files: %s", ", ".join(missing)) | |
| if missing: | |
| raise FileNotFoundError(f"Cannot package submission. Missing required prediction files: {', '.join(missing)}") |
| protocols = _load_protocol_set(config["prompt"], dataset.task_type, dataset.task_name) | ||
| outputs: List[Dict[str, Any]] = [] | ||
| max_samples = config["runtime"].get("max_samples_per_task") | ||
| sample_records = dataset.test_samples[: int(max_samples)] if max_samples else dataset.test_samples |
There was a problem hiding this comment.
Using if max_samples to check if max_samples is set can lead to incorrect behavior if max_samples_per_task is set to 0 in the configuration (as 0 is falsy in Python, it would fall back to evaluating all samples instead of 0). It is safer and more idiomatic to explicitly check if max_samples is not None.
| sample_records = dataset.test_samples[: int(max_samples)] if max_samples else dataset.test_samples | |
| sample_records = dataset.test_samples[: int(max_samples)] if max_samples is not None else dataset.test_samples |
| from typing import Optional | ||
|
|
||
|
|
||
| class TokenCounter: |
There was a problem hiding this comment.
| def build_logger(log_dir: str, name: str = "ai-lab") -> logging.Logger: | ||
| Path(log_dir).mkdir(parents=True, exist_ok=True) | ||
| logger = logging.getLogger(name) | ||
| logger.setLevel(logging.INFO) | ||
|
|
||
| if not logger.handlers: | ||
| handler = logging.StreamHandler() | ||
| formatter = logging.Formatter( | ||
| "%(asctime)s | %(levelname)s | %(name)s | %(message)s" | ||
| ) | ||
| handler.setFormatter(formatter) | ||
| logger.addHandler(handler) |
There was a problem hiding this comment.
The build_logger function creates the log_dir directory but only configures a StreamHandler to log to the console. It does not add a FileHandler to actually write the logs to a file in log_dir. Adding a FileHandler ensures that logs are persisted for future diagnostics.
log_path = Path(log_dir)
log_path.mkdir(parents=True, exist_ok=True)
logger = logging.getLogger(name)
logger.setLevel(logging.INFO)
if not logger.handlers:
formatter = logging.Formatter(
"%(asctime)s | %(levelname)s | %(name)s | %(message)s"
)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
file_handler = logging.FileHandler(log_path / "run.log", encoding="utf-8")
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)| from typing import Any, Dict, List | ||
|
|
||
|
|
||
| def format_example(example: Dict[str, Any]) -> str: |
Summary
OpenSeeksubmission for theLongContext-ICL-Annotationchallenge.code/directory for direct review and reproduction.87.35.Compliance
Qwen3-4Bas the only large language model.FlagScaleas the deployment and inference framework.Reproduction
openseek/competition/LongContext-ICL-Annotation/submissions/OpenSeek/README.mdfor the submission overview.openseek/competition/LongContext-ICL-Annotation/submissions/OpenSeek/code/README.mdfor environment setup, model deployment, inference, packaging, and validation instructions.