feat: Create rmagic cli (#7)

* feat(cli): Implement robust CLI argument parsing and file type detection

- Add comprehensive CLI argument structure using clap derive macros
- Implement platform-specific magic file path resolution
- Add support for JSON and text output formats
- Create `Args` struct with flexible configuration options
- Add default magic file path detection for Unix and Windows
- Implement output format selection with `OutputFormat` enum
- Add basic error handling and file existence validation
- Include initial test coverage for CLI argument parsing
Enhances the CLI tool's usability and provides a flexible interface for file type identification across different platforms.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(tests): Add compatibility testing framework and CI integration

- Introduced a new compatibility testing suite to ensure that libmagic-rs produces identical results to the original libmagic implementation.
- Added a `tests/compatibility_tests.rs` file to run tests against the original test files from the file/file repository.
- Created a `justfile` with commands for initializing and running compatibility tests, including CI integration for automated testing on push and pull request events.
- Implemented a GitHub Actions workflow in `compatibility.yml` to run compatibility tests across multiple platforms, ensuring consistent results.
- Added a `.gitmodules` file to manage the file-tests submodule for compatibility tests.

These enhancements improve the testing infrastructure, ensuring robust validation of the library's functionality against established benchmarks.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(magic): Add basic magic file for testing and enhance CLI functionality

- Introduced a basic magic file (`missing.magic` and `nonexistent.magic`) for testing and CI/CD environments, containing signatures for various file types including ELF, PE, ZIP, JPEG, PNG, GIF, PDF, and common text patterns.
- Updated the `justfile` to include specific actionlint commands for various GitHub workflows, ensuring comprehensive linting coverage.
- Enhanced the CLI in `main.rs` to check for the existence of the magic file and attempt to download it if missing, providing a fallback mechanism for CI/CD environments.
- Implemented a new function `download_magic_files` to create a basic magic file when none exists, improving usability in testing scenarios.

These changes improve the testing infrastructure and ensure robust functionality for file type detection in various environments.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(cli): Enhance error handling and validation in CLI

- Implemented comprehensive error handling in the CLI, providing user-friendly messages for various error scenarios, including file not found, permission denied, and invalid input.
- Added validation functions to ensure command-line arguments are correctly formatted and accessible before processing.
- Updated the `run_analysis` function to utilize the new error handling mechanism, improving the robustness of the CLI tool.
- Introduced new tests to validate error handling and argument validation, ensuring consistent behavior across different input scenarios.

These enhancements improve the user experience by providing clearer feedback and ensuring that the CLI operates reliably under various conditions.

Signed-off-by: UncleSp1d3r <[email protected]>

* chore(third_party): Update third-party test files and configurations

- Added NOTICE and COPYING from `github.com/file/file` to provide proper attribution for test files.
- Refactored various third-party test files and configurations to ensure consistency and maintainability.
- Updated `.gitattributes`, `.mdformat.toml`, and `.prettierignore` for prevent alterations to the test files.
- Cleaned up and organized test results and test files across multiple formats to enhance clarity and usability.

These changes contribute to a more organized testing framework and improve the overall structure of third-party resources.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(third_party): Add magic.mgc file for file type detection testing

- Introduced a new binary file `magic.mgc` in the `third_party` directory to support file type detection testing.
- This addition enhances the testing framework by providing a standard magic file for validating the functionality of the library.

These changes contribute to a more robust testing environment and ensure accurate file type detection capabilities.

Signed-off-by: UncleSp1d3r <[email protected]>

* chore(tests): Remove submodule for compatibility tests and update verification process

- Deleted the `.gitmodules` entry for the `file-tests` submodule, streamlining the project structure.
- Updated the `justfile` to replace the submodule initialization command with a verification command that checks for the existence of compatibility test files in the `third_party` directory.
- Adjusted the GitHub Actions workflow to reflect the new verification process for compatibility test files, ensuring that the necessary files are available before proceeding with builds.

These changes simplify the setup process for compatibility tests and enhance the reliability of the testing framework.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(output): Add JSON output support for file type detection

- Implement JSON output formatting with JsonMatchResult struct
- Add JSON output integration in main.rs with --json flag handling
- Create src/output/json.rs to define JSON serialization structure
- Update tasks.md to mark JSON output implementation tasks as completed
- Add tempfile dependency in Cargo.toml for testing
- Convert evaluation results to structured JSON match results
- Enhance CLI output routing to support JSON format
- Provide machine-readable file type detection results
Closes #11 (JSON match result implementation)

Signed-off-by: UncleSp1d3r <[email protected]>

* refactor(validation): Improve configuration validation with modular checks

- Split `validate()` method into separate validation functions
- Add constants for safe thresholds in recursion depth, string length, and timeout
- Enhance error messages with more descriptive and context-specific details
- Separate concerns by creating individual validation methods for different aspects
- Improve error handling and prevent potential resource exhaustion scenarios
- Maintain consistent error reporting and validation logic
This refactoring improves the robustness and readability of configuration validation, making it easier to understand and maintain security checks.

Signed-off-by: UncleSp1d3r <[email protected]>

* refactor(error): Improve error handling and message formatting in CLI and library

- Refactor error handling functions to use more concise, multi-line error messages
- Optimize error message generation with reduced string allocations
- Update error handling in main.rs to consolidate error printing logic
- Modify create_basic_magic_content() to use a const for magic file content
- Improve JSON output generation by using pre-allocated vector
- Standardize error message formatting across different error types
- Reduce redundant string formatting and improve readability of error messages
These changes enhance the error reporting mechanism, making error messages more consistent and slightly more memory-efficient while maintaining clear and informative output.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(evaluator): Add support for string type evaluation in magic rules

- Implement string type reading in evaluator with null-termination support
- Update test cases to validate string type evaluation
- Remove unsupported string type error handling
- Add test for matching and non-matching string rules
- Mark string type implementation tasks as completed in tasks.md

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(magic): Add basic string type support in AST and matching

- Implement basic string type in AST
- Add string matching support with UTF-8 validation
- Extend read_typed_value function to handle String type
- Mark tasks 12, 12.1, and 12.3 as completed in implementation plan
- Prepare for advanced string parsing and validation

Signed-off-by: UncleSp1d3r <[email protected]>

* refactor(validation): Improve error handling and message formatting in configuration validation

- Update error messages to use dynamic formatting with constants
- Add more descriptive error messages for configuration validation
- Enhance error context in validation methods for recursion depth, string length, and timeout
- Improve test assertions to check for more specific error message details
- Remove TODO comments and replace with actual implementation improvements
- Add runtime context to error messages using predefined constants

Signed-off-by: UncleSp1d3r <[email protected]>

* refactor(performance): Optimize JSON and parsing operations with iterator methods

- Replace manual vector building with iterator methods in JSON output functions
- Use `with_capacity()` to pre-allocate vectors for better performance
- Improve hex byte parsing in grammar parser with direct digit conversion
- Remove unnecessary `format!()` allocations in hex parsing
- Reduce memory allocations and improve parsing efficiency
These changes focus on reducing memory allocations and improving performance in key parsing and output generation functions.

Signed-off-by: UncleSp1d3r <[email protected]>

* chore(hooks): Simplify CI auto-fix hook configuration

- Remove unnecessary file patterns from hook trigger configuration
- Reduce scope of hook to only watch Rust source files
- Trim trailing newline for consistency
- Maintain core auto-fix agent prompt and functionality

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(error): Add comprehensive error handling for libmagic-rs

- Create src/error.rs with detailed error types using thiserror
- Implement LibmagicError enum with variants for parsing, evaluation, and I/O errors
- Add detailed ParseError enum with specific error types for magic file parsing
- Create EvaluationError enum to handle rule evaluation and type reading errors
- Implement helper methods for creating specific error instances
- Mark task 13.1 as completed in tasks.md
- Provide comprehensive error reporting with context and line numbers

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(error): Add evaluation error types for runtime detection

- Create EvaluationError enum in error.rs module
- Add variants for BufferOverrun, InvalidOffset, and UnsupportedType
- Mark task 13.2 as completed in implementation plan
- Improve error handling for runtime evaluation scenarios

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(evaluator): Implement graceful error handling in rule evaluation

- Add comprehensive error handling for individual rule evaluation
- Implement graceful degradation to skip problematic rules
- Log warning messages for skipped rules with specific error details
- Enhance error context in evaluate_rules and evaluate_single_rule functions
- Preserve evaluation flow by continuing processing after encountering rule-level errors
- Maintain critical error handling for timeout and recursion limit scenarios
- Update documentation to reflect new error handling approach

Signed-off-by: UncleSp1d3r <[email protected]>

* chore(ci): Update CI configuration for libmagic-rs

- Change Codecov slug to match the repository name
- Update Rust toolchain version to 1.90
- Remove caching of cargo dependencies and replace with installation of the Just task runner

This update streamlines the CI process and ensures compatibility with the latest Rust toolchain.

Signed-off-by: UncleSp1d3r <[email protected]>

* chore(dependencies): Update dependencies and improve test handling

- Add `regex` and `temp-env` dependencies for enhanced functionality.
- Update `read_long` documentation to clarify behavior regarding NUL characters.
- Introduce `BuildNoiseFilter` struct to filter out build noise in CLI integration tests.
- Refactor test file creation to use `NamedTempFile` for automatic cleanup.
- Normalize output in CLI integration tests to improve snapshot consistency.

These changes enhance the testing framework and clarify the behavior of string reading functions, contributing to overall code quality and maintainability.

Signed-off-by: UncleSp1d3r <[email protected]>

* test(cli): normalize rmagic(.exe) via insta filters for cross-platform snapshots

- Add normalize_cli_output() helper in tests/common/mod.rs to handle:
  - Convert 'rmagic.exe' to 'rmagic' for Windows compatibility
  - Remove Windows path prefixes (\\?\)
  - Filter cargo-specific error messages
- Update CLI integration tests to use normalization helper
- Add comprehensive regression tests in tests/cli_normalization.rs
- Update all affected insta snapshots to use normalized output

Fixes cross-platform test failures where Windows .exe suffix caused
snapshot mismatches between Unix and Windows environments.

* build(justfile): improve cross-platform coverage commands

- Split coverage and coverage-check targets into [unix] and [windows] variants
- Add proper environment setup with RUSTFLAGS for coverage builds
- Clean coverage artifacts before generating reports on both platforms
- Use bash script with set -euo pipefail for Unix reliability
- Use PowerShell syntax for Windows commands

Ensures consistent coverage generation across different operating systems
and improves reliability of coverage reports in CI environments.

* refactor(project-structure): Reorganize test files and update documentation paths

- Remove `test_files/` directory from project
- Update documentation references to use `third_party/` for test files and magic files
- Modify code examples and test paths to reflect new directory structure
- Update development, getting-started, and testing documentation
- Adjust `.gitignore` to remove `test_files/` entry
- Update task tracking to reflect ongoing implementation status
Rationale: Improve project organization and standardize test file management by using a more descriptive `third_party/` directory for external test resources and magic files.

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(tests): normalize file paths in CLI integration test snapshots

- Add path normalization utilities in tests/common/mod.rs
- Normalize absolute paths to relative filenames in CLI test outputs
- Update CLI integration test to use path normalization before snapshot assertion
- Add unit tests for path normalization functions
- Accept updated snapshot with normalized paths for cross-platform compatibility

This ensures snapshot tests work consistently across different development environments
by removing system-specific absolute paths from test outputs.

* chore(tasks): Update implementation progress tracking in project tasks

- Mark tasks 11-13 as completed in implementation plan
- Update task status for JSON match result structure
- Update task status for basic string type in AST
- Update task status for basic error types
- Add new task for text-based magic file parsing
- Add detailed notes and requirements for upcoming implementation tasks
- Prepare project roadmap for next development phases

Signed-off-by: UncleSp1d3r <[email protected]>

* chore(tasks): Update implementation progress tracking in project tasks

- Completed comprehensive project implementation milestones
- Updated task tracking with detailed progress descriptions
- Documented completed stages for Rust libmagic implementation
- Included summaries for key development areas:
* Project structure and dependencies
* AST type definitions
* Parser components
* Memory-mapped file I/O
* Offset resolution
* Type reading and interpretation
* Operator evaluation
* Rule evaluation engine
* Output formatting system
- Provides clear overview of project development status and achievements

Signed-off-by: UncleSp1d3r <[email protected]>

* feat(parser): Enhance magic rule parsing with advanced value and operator support

- Add BitwiseAndMask variant to Operator enum for masked bitwise comparisons
- Implement bitwise masking logic in apply_operator function
- Extend hex byte parsing to support mixed hex and ASCII sequences
- Update test coverage for new bitwise masking operator
- Mark text format magic rule parsing task as complete in implementation tasks
- Improve parsing flexibility for complex magic file rule formats

Signed-off-by: UncleSp1d3r <[email protected]>

* fix: resolve clippy warning in test_all_offset_spec_variants (#8)

* Initial plan

* fix: replace vec! with array in test_all_offset_spec_variants

Co-authored-by: unclesp1d3r <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: unclesp1d3r <[email protected]>

* fix(tests): Handle Windows drive letters in CLI output normalization (#10)

* Initial plan

* fix(tests): Handle Windows drive letters in path normalization

Fixed the CLI integration test's normalize_cli_output function to correctly
handle Windows paths that contain colons (e.g., D:\path\to\file). The previous
implementation would incorrectly find the first colon (from the drive letter)
instead of the colon separating the filename from the description.

The fix searches for the filename followed by ": " pattern, ensuring we find
the correct delimiter regardless of drive letters or other colons in the path.

This resolves the test-cross-platform Windows CI failures where the output
was showing "CVE-2014-1943.testfile: data" instead of just "data".

Co-authored-by: unclesp1d3r <[email protected]>

* chore: Remove accidentally committed test file

Co-authored-by: unclesp1d3r <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: unclesp1d3r <[email protected]>

---------

Signed-off-by: UncleSp1d3r <[email protected]>
Co-authored-by: Copilot <[email protected]>

Author: unclesp1d3r

.coderabbitai.yaml

@@ -73,6 +73,7 @@ reviews:
             "!*.svg",
             "!*.ico",
             "!*.wxs",
+            "!third_party/**",
         ]
     path_instructions:
         - path: "src/lib.rs"

.gitattributes

@@ -58,6 +58,10 @@ LICENSE text
 *.ico binary
 *.svg text
 
+# Test files
+**/*.result binary
+**/*.testfile binary
+
 # Font files
 *.ttf binary
 *.eot binary

.github/workflows/ci.yml

@@ -156,7 +156,7 @@ jobs:
                   files: lcov.info
                   fail_ci_if_error: false
                   token: ${{ secrets.CODECOV_TOKEN }}
-                  slug: EvilBit-Labs/StringyMcStringFace
+                  slug: EvilBit-Labs/libmagic-rs
             - uses: qltysh/qlty-action/coverage@v2
               with:
                   token: ${{ secrets.QLTY_COVERAGE_TOKEN }}

.github/workflows/compatibility.yml

@@ -0,0 +1,39 @@
+name: Compatibility Tests
+
+on:
+    push:
+        branches: [main, develop]
+    pull_request:
+        branches: [main, develop]
+    schedule:
+        # Run daily at 2 AM UTC to catch any regressions
+        - cron: "0 2 * * *"
+
+jobs:
+    compatibility-tests:
+        name: Compatibility Tests
+        runs-on: ${{ matrix.os }}
+
+        strategy:
+            matrix:
+                os: [ubuntu-latest, windows-latest, macos-latest]
+
+        steps:
+            - name: Checkout code
+              uses: actions/checkout@v5
+
+            - uses: dtolnay/[email protected]
+
+            - name: Install just task runner
+              uses: taiki-e/install-action@v2
+              with:
+                tool: just
+
+            - name: Verify compatibility test files are available
+              run: just verify-compatibility-tests
+
+            - name: Build rmagic
+              run: cargo build --release
+
+            - name: Run compatibility tests
+              run: cargo test test_compatibility_with_original_libmagic -- --ignored

.gitignore

@@ -129,4 +129,3 @@ megalinter-reports/
 .intentionally-empty-file.o
 
 # Files for validating the tooling
-test_files/

.gitmodules

@@ -6,16 +6,11 @@
   "when": {
     "type": "userTriggered",
     "patterns": [
-      "**/*.rs",
-      "**/Cargo.toml",
-      "**/Cargo.lock",
-      "justfile",
-      "deny.toml",
-      "rust-toolchain.toml"
+      "**/*.rs"
     ]
   },
   "then": {
     "type": "askAgent",
     "prompt": "1. First, run `just ci-check` to identify any failures\n2. Analyze the output to understand what specific checks are failing. If everything passes, you're done.\n3. Make minimal, targeted fixes to address ONLY the failing checks:\n- For formatting issues: run `just format`\n- For linting issues (clippy): fix the specific violations reported (rerun with `just lint-rust` / `just lint-rust-min`)\n- For compilation/type errors: fix the underlying Rust code until `just check` (or `cargo check`) succeeds\n- For test failures: fix the failing tests or underlying code (verify with `just test` or `just test-ci`)\n- For dependency security/advisory issues: run `just audit` (cargo-audit) and/or update `Cargo.toml` then `cargo update`\n- For license/compliance issues: run `just deny` and address cargo-deny findings\n4. After making fixes, run `just ci-check` again to verify all checks pass\n5. If any checks still fail, repeat steps 2-4 until all checks pass\n6. Provide a summary of what was fixed and confirm that `just ci-check` now passes completely\nKeep changes minimal and focused - only fix what's actually causing the CI failures. Do not make unnecessary refactoring or style changes beyond what's required to pass the checks."
   }
-}
+}

.kiro/hooks/ci-auto-fix.kiro.hook

@@ -8,6 +8,8 @@ exclude = [
     "**/CHANGELOG.md",
     "target/**",
     "megalinter-reports/**",
+    "**/*.result",
+    "**/*.testfile",
 ]
 validate = true
 number = true

.kiro/specs/rust-libmagic-implementation/tasks.md

@@ -54,25 +54,25 @@ repos:
       hooks:
           - id: cargo-machete
 
-    # Markdown formatting and linting (temporarily disabled in CI due to path issues)
-    - repo: https://github.com/hukkin/mdformat
-      rev: 0.7.22
-      hooks:
-          - id: mdformat
-            additional_dependencies:
-                - mdformat-gfm
-                - mdformat-admon
-                - mdformat-config
-                - mdformat-footnote
-                - mdformat-frontmatter
-                - mdformat-simple-breaks
-                - mdformat-tables
-                - mdformat-web
-                - mdformat-wikilink
-                - mdformat-gfm-alerts
-                - mdformat-rustfmt
-                - mdformat-toc
-            files: \.(md|mdx)$
+    # Markdown formatting and linting (temporarily disabled due to rust code block formatting issues)
+    # - repo: https://github.com/hukkin/mdformat
+    #   rev: 0.7.22
+    #   hooks:
+    #       - id: mdformat
+    #         additional_dependencies:
+    #             - mdformat-gfm
+    #             - mdformat-admon
+    #             - mdformat-config
+    #             - mdformat-footnote
+    #             - mdformat-frontmatter
+    #             - mdformat-simple-breaks
+    #             - mdformat-tables
+    #             - mdformat-web
+    #             - mdformat-wikilink
+    #             - mdformat-gfm-alerts
+    #             - mdformat-toc
+    #         files: \.(md|mdx)$
+    #         exclude: ^target/|
 
     # Security audit for Rust dependencies (moved to CI)
     - repo: local