Framework tests / test validators now return exit codes (-1) when failures are detected by marc-casavant · Pull Request #9 · InkbridgeNetworks/freeradius-multi-server

marc-casavant · 2026-03-27T18:00:25Z

Understand the codebase and identify all sys.exit uses
Fix sys.exit(1) in run_tests() async coroutine → use had_exception flag and return -1
Guard task.result() calls against cancelled/exceptional tasks in run_tests()
Fix isinstance(result, Exception) → isinstance(result, BaseException) in run_all_tests_and_shutdown()
Change main() to return int instead of calling sys.exit
Update interface() to call sys.exit based on main() return value
All remaining sys.exit calls are properly in interface() (topmost entry point)

⚡ Quickly spin up Copilot coding agent tasks from anywhere on your macOS or Windows machine with Raycast.

Copilot

Pull request overview

This PR updates the multi-server test framework so individual validators/tests return explicit result codes (0 on success, non-zero on failure) and attempts to propagate those codes up to the framework’s top-level execution.

Changes:

Validator.get_results_str() now returns (results_str, result_code) where result_code is 0 or -1.
Test.run() / Test.__run() now return an integer exit code derived from per-state validator results (and timeout handling).
multi_server_test.run_tests() / main() were updated to aggregate test return codes and exit non-zero on failures.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
`src/multi_server_test.py`	Propagates async test return codes to `main()` and attempts to convert failures into a non-zero process exit.
`src/custom_test.py`	Makes `Test.run()` and internal state runner return an int result code; logs validator result codes per state.
`src/Validator.py`	Extends validator result reporting to include an explicit pass/fail result code alongside the formatted output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ethan-thompson · 2026-04-01T20:18:24Z

                "%s %s", f"Test.{self.name}.{self.compose_file.stem}", result
            )
+
+        return ExitCodes.VALIDATION_FAILURE if any(has_failure for has_failure in test_has_failures) else ExitCodes.SUCCESS


return ExitCodes.VALIDATION_FAILURE if (True in test_has_failures) else ExitCodes.SUCCESS

ethan-thompson · 2026-04-01T20:19:16Z

        tests (list[Test]): List of Test objects to run.
+
+    Returns:
+        int: 0 if all tests passed, -1 if any test had failures.


Does not return -1. Returns a specific ExitCode

ethan-thompson · 2026-04-01T20:28:56Z

+    for task in tasks:
+        if task.cancelled():
+            logger.error("A test task was cancelled: %s", task.get_name())
+            return ExitCodes.GENERAL_EXCEPTION
+        exc = task.exception()
+        if exc is not None:
+            logger.error("A test task raised an exception: %s", exc)
+            return ExitCodes.GENERAL_EXCEPTION
+        if task.result() != 0:
+            return ExitCodes.VALIDATION_FAILURE


This is going to exit at the first test that had a problem. In that case, you would loose the results of the other tests.

ethan-thompson · 2026-04-01T20:30:50Z

+                codes.append(ExitCodes.GENERAL_EXCEPTION if isinstance(result, BaseException) else result)
+            await cleanup_and_shutdown()
+
+        loop.create_task(run_all_tests_and_shutdown())


This means that the shutdown of the tests is going to be blocked/waiting for all of the tests to finish up.

Instead, look at the callback that is being registered for when the test_task is "done". That method should look at the "result" and then track that using some shared thing. It looks like currently that could be the "codes" list.

ethan-thompson · 2026-04-01T20:31:37Z

    logger.info("Multi-server tests completed.")

+    # If any of the test exit codes are non-zero, return 1 to indicate a failure
+    if any(code != ExitCodes.SUCCESS for code in codes):


Might be able to just do if ExitCodes.SUCCESS in codes:

ethan-thompson · 2026-04-01T20:32:10Z

        )
        logger.info("Exiting.")
-        sys.exit(1)
+        #sys.exit(1)


Remove commented code

ethan-thompson · 2026-04-01T20:35:12Z

+    if ret:
+        sys.exit(ret)


Set a default return value for ret at the top. Let it be overwritten by the other returns. Then, just do sys.exit(ret) here with no conditional

ethan-thompson · 2026-04-06T17:57:06Z

 logger = logging_helper.get_logger()

-async def cleanup_and_shutdown() -> None:
+ret = ExitCodes.SUCCESS


This should be at the top of interface() since it does not need to be globally scoped.

marc-casavant · 2026-04-07T16:22:15Z

@ethan-thompson last minor update included, all commits squashed. I think we are good to merge.

marc-casavant requested review from arr2036, Copilot and ethan-thompson March 27, 2026 18:00

Copilot started reviewing on behalf of marc-casavant March 27, 2026 18:00 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Comment thread src/multi_server_test.py Outdated

Comment thread src/multi_server_test.py Outdated

Comment thread src/multi_server_test.py Outdated

Comment thread src/custom_test.py Outdated

Comment thread src/Validator.py Outdated

marc-casavant requested a review from Copilot March 27, 2026 19:03

Copilot started reviewing on behalf of marc-casavant March 27, 2026 19:03 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Comment thread src/multi_server_test.py Outdated

Comment thread src/multi_server_test.py Outdated

Comment thread src/Validator.py Outdated

Comment thread src/multi_server_test.py Outdated

Comment thread src/multi_server_test.py

Copilot started work on behalf of arr2036 March 27, 2026 20:57 View session

Copilot finished work on behalf of arr2036 March 27, 2026 21:04

marc-casavant changed the title ~~Framework tests now return exit codes, -1 on failure~~ Framework tests / test validators now return exit codes (-1) when failures are detected Mar 30, 2026

arr2036 reviewed Mar 31, 2026

View reviewed changes

Comment thread src/multi_server_test.py Outdated

marc-casavant requested a review from arr2036 March 31, 2026 19:05

ethan-thompson requested changes Apr 1, 2026

View reviewed changes

marc-casavant requested a review from ethan-thompson April 6, 2026 14:27

ethan-thompson requested changes Apr 6, 2026

View reviewed changes

Fix exit codes

971cd1f

marc-casavant force-pushed the fix-non-zero-code-on-failure branch from 0e76f88 to 971cd1f Compare April 7, 2026 16:20

ethan-thompson approved these changes Apr 7, 2026

View reviewed changes

ethan-thompson merged commit b4000ef into main Apr 7, 2026

Conversation

marc-casavant commented Mar 27, 2026 • edited by Copilot AI Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

marc-casavant commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

marc-casavant commented Mar 27, 2026 •

edited by Copilot AI

Loading