Add Python client and server for UiPath.Ipc

[eduard-dumitru]

Summary

Adds a Python client and server for UiPath.Ipc that speaks the same wire protocol as the .NET package, plus the CI/CD plumbing to build, test, and publish it. Published and resolvable on the project-scoped uipath-ipc-deps feed. The Python package version tracks UiPath.CoreIpc (<Version>2.5.1</Version>), stamped to a PEP 440 local-version string per CI run (e.g. 2.5.1+<build>) by src/CI/stamp-python-version.py — so pyproject.toml's version is a placeholder the pipeline overwrites.

Scope — landed incrementally, each in its own commit:

1. Client — dynamic-proxy RPC over Named Pipe / TCP.
2. Bidirectional callbacks — the client hosts callback services the server invokes.
3. Server — IpcServer hosts services that a .NET or Python client calls.
4. Handler reach-back — message.client.get_callback(...), the Python analog of .NET's m.Client.GetCallback<T>().

Streams (upload / download) remain explicitly out of scope.

Python client (src/Clients/python/uipath-ipc/)

• Pure-Python, asyncio-based; talks the same wire to any UiPath.Ipc peer.
• Transports: NamedPipe (Win + POSIX via /tmp/CoreFxPipe_*) and TCP — both client and server halves — with bounded retry on FileNotFoundError to ride out the peer's accept-loop warm-up.
• IpcClient(transport=..., callbacks={IClientCallback: impl}): lazy connect, dynamic __getattr__ proxy via get_proxy(IContract), auto-reconnect on transport drop, optional request_timeout.
• RemoteException preserves message / type_name / stack_trace / inner chain, sets __cause__ for traceback display.
• Cancellation is task-based; no CancellationToken parameter on signatures. The proxy forwards CancellationRequest on the wire when its task is cancelled; peer-initiated cancellation aborts the matching in-flight handler.
• Wheel ships as py3-none-any with PEP 561 py.typed.

Callback (server → client) support

• IpcClient(callbacks={Contract: instance}) registers handler objects keyed by interface name.
• IpcConnection dispatches incoming REQUEST frames: looks up the endpoint by interface name, resolves the method by name, json.loads each parameter individually (matching the .NET convention), invokes (awaitable or sync), and writes a Response back. A write lock keeps concurrent outbound frames aligned.
• Handler exceptions wire back as RemoteException. Peer-initiated CancellationRequest cancels the matching handler task; the emitted error mimics .NET's OperationCanceledException.
• Callback methods must NOT declare CancellationToken parameters — the caller doesn't include CT in the wire Parameters array (matches the existing .NET IComputingCallback convention).

Server — IpcServer (NEW)

• IpcServer(transport, services={IContract: impl}, request_timeout=None): listens via a ServerTransport and wraps each accepted client in the existing symmetric IpcConnection, whose callbacks dict is the hosted-service set. Duck-typed dispatch by contract __name__ (the instance need not inherit the contract).
• Server transports: TcpServerTransport (asyncio.start_server) and NamedPipeServerTransport (start_serving_pipe on Windows, start_unix_server at /tmp/CoreFxPipe_<name> on POSIX).
• start / serve_forever / aclose + async context manager; connection_count introspection. Live connections are pruned via a new IpcConnection.add_close_callback hook that fires once on explicit close or peer disconnect.
• aclose closes connections before awaiting the listener, so Python 3.12+'s Server.wait_closed() (which blocks until active connections finish) doesn't hang.

Handler reach-back — Message.Client / GetCallback (NEW)

• message.py: Message / Message[T] + an IClient protocol. A service or callback method opts into a handle on its caller by declaring a Message parameter; the wire never carries it (mirrors .NET's trailing-Message convention — the caller sends the same args either way).
• IpcConnection.get_callback(Contract) returns a proxy bound to that connection, so a handler can call the peer back mid-request — the inverse direction of the in-flight call. Python analog of m.Client.GetCallback<T>(). Reached as m.client.get_callback(IContract).
• Dispatch (_invoke_callback) does signature-aware binding: wire args fill the real parameters, a Message (.client = the connection, .request_timeout carried through) is injected at its slot; the no-Message fast path is unchanged and cached per function. Works symmetrically for client-hosted callbacks and server-hosted services, since both run through the one IpcConnection.

Dedicated .NET test server (src/IpcSample.PythonClientTestServer/)

Purpose-built for the Python integration suite — net8.0, console logging on, stable READY pipe=<name> startup marker. Hosts IComputingService / ISystemService (callback-free) plus ICallbackTester, whose handlers call m.Client.GetCallback<IClientCallback>() to exercise the server→client callback path against the real .NET Message.Client reach-back.

Wire-shape tests

tests/wire/test_dotnet_compatibility.py asserts our serialized JSON against the .NET schema literally (field sets, types, TimeoutInSeconds is a non-null double, etc.) — catches a class of regression at unit-test time that previously only surfaced when the live integration suite ran. This came out of finding (and fixing) an actual mismatch: TimeoutInSeconds: null made Newtonsoft drop the entire Request silently.

CI / pipeline redesign

• Parameterized publish: publishNuGet, publishNpm, publishPyPI (all default false). Default CI runs build + test only. Old behavior of always-pushing NuGet on every branch and always-popping an approval prompt is gone — and a rejected approval no longer leaves the run as Failed.
• Parameterized build (opt-out): buildNuGet, buildNpm, buildPython (all default true).
• Per-technology Build stages so NuGet / NPM / Python race to the finish line independently.
• reuseArtifactsFromBuildId parameter lets a Publish stage replay against a prior successful build, skipping Build entirely.
• Job names shifted from environment-centric (".NET on Windows", "node.js on Ubuntu") to deliverable-centric ("NuGet — .NET on Windows", "NPM — Node + Web on Linux (test-only)", "Python — Windows"). The (test-only) marker disambiguates the Linux matrix runs from the artifact-producing Windows ones.
• New environments NuGet-Packages and PyPI-Packages mirror the existing NPM-Packages approval shape (same approver, same 12h timeout).

Feed swap + Supply Chain Guard bypass

• npm install customFeed moved from the org-level npm-packages (managed outside CoreIpc) to a project-scoped uipath-ipc-deps (CoreIpc-owned, mirrors npmjs.org + PyPI + NuGet).
• Pipeline-level SCG_KILL_SWITCH=true to bypass the org-wide Aikido Safe Chain shim, which was interfering with npm tarball downloads. Comments in azp-start.yaml and azp-nodejs.yaml document the trail (with #devops Slack links) so the next person on this doesn't redo the dead-ends.

NPM publish

• Now writes to uipath-ipc-deps first (uses the pipeline's built-in identity — no PAT). GitHub Packages stays wired up as continueOnError: true until the platform team ships the post-Mini-Shai-Hulud pipeline-auth replacement (tracked in Liviu Bud's #dev thread).
• Runs of Publish_NPM finish as "Succeeded with issues" until the GitHub side resolves — packages still ship to uipath-ipc-deps.

Test plan

• CI green on Windows + Linux for Build (all jobs).
• 111 non-.NET + 11 .NET-interop Python tests pass (122 total) locally (pytest) and in CI.
• Client RPC over Named Pipe + TCP, incl. RemoteException propagation and task-based cancellation.
• Server→client callbacks: 3 round-trips against the .NET ICallbackTester (real m.Client.GetCallback<IClientCallback>()) — test_dotnet_interop.py#L136-L183.
• Server (IpcServer): TCP + named-pipe loopback (Python client ↔ Python server), lifecycle, connection-count pruning — test_ipc_server.py.
• Handler reach-back: full-duplex re-entrancy E2E — client → server handler → get_callback → client callback → back — test_ipc_server.py#L196-L242; plus Message-injection / get_callback units — test_message_injection.py.
• Close-callback hook (server connection pruning) — test_connection.py#L133-L173.
• Publish_PyPI publishes the wheel to uipath-ipc-deps, version stamped from UiPath.CoreIpc + build number (PEP 440 local segment, e.g. 2.5.1+<build>).
• Consumer-side install round-trips: uipath-robot-client (separate repo, own venv, index-url = uipath-ipc-deps) pins uipath-ipc>=2.5.1+20260611.6 and its 100-test suite passes against the published wheel.
• .NET-client ↔ Python-IpcServer interop (reverse of IpcSample.PythonClientTestServer): a real .NET client drives the Python server — direct calls, reach-back into a .NET-hosted callback, and a RemoteException round-trip — test_dotnet_client_interop.py + IpcSample.PythonServerTestClient. Surfaced two dispatch fixes: bind wire args to the handler signature (tolerate a .NET client's trailing CancellationToken), and close the writer on peer disconnect (transport-leak fix).
• High-effort code-review pass (7 finder angles) — fixed a real bug (serve_forever() returned immediately for Windows named pipes, tearing the server down), made Message injection keyword-based so keyword-only / Optional[Message] params inject correctly, unified connection teardown (now cancels in-flight handlers on peer disconnect), and dropped the _ConnectionInvoker shim. 4 regression tests added.
• (deferred) Pre-existing .NET flake SystemTestsOverTcp.NotPassingAnOptionalMessage_ShouldWork (tight 800ms timeout, pre-dates this PR). Tracked separately.
• (deferred) Restore GitHub Packages NPM publish + remove continueOnError once platform team ships sanctioned pipeline auth.
• (deferred) Re-enable Safe Chain Guard once DevOps fixes the underlying shim.
• Deleted src/Clients/python/_attempt0/ (the reference port; shipping impl is src/Clients/python/uipath-ipc/).

🤖 Generated with Claude Code

[eduard-dumitru]

/azp run CI

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Radu-Niculae]

payload_len is a signed int32 fed to readexactly with no bound — a header claiming ~2 GB OOMs the process, and a negative length silently desyncs the stream. .NET validates the length against _maxMessageSize before allocating (Connection.cs:260; 2 MB server default). Bound it here; ideally thread a configurable cap from IpcConnection to mirror MaxReceivedMessageSizeInMegabytes.

Suggested change

	payload = await reader.readexactly(payload_len) if payload_len > 0 else b""
	if not 0 <= payload_len <= 2 * 1024 * 1024: # match .NET's 2 MB server cap
	raise ValueError(f"frame payload length {payload_len} out of bounds")
	payload = await reader.readexactly(payload_len) if payload_len > 0 else b""

Claude said this was a problem on JS as well.

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

macOS interop: don't hardcode /tmp/ .NET builds this path as Path.Combine(Path.GetTempPath(), "CoreFxPipe_"), and GetTempPath() honors $TMPDIR (the TS port does too — Platform.ts:66-73). On macOS $TMPDIR is always set, so .NET binds under /var/folders/… while we look in /tmp/ and connect fails; vanilla Linux/Windows are unaffected. Same fix needed on the server property at line 136.

Suggested change

	return f"/tmp/CoreFxPipe_{self.pipe_name}"
	return os.path.join(os.environ.get("TMPDIR") or "/tmp", f"CoreFxPipe_{self.pipe_name}")

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

Same $TMPDIR fix here on the server side — see the client property above.

Suggested change

	return f"/tmp/CoreFxPipe_{self.pipe_name}"
	return os.path.join(os.environ.get("TMPDIR") or "/tmp", f"CoreFxPipe_{self.pipe_name}")

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

When an stream requests is initiated from the other side the stream frame silently desyncs the connection. Skipping UPLOAD_REQUEST/DOWNLOAD_RESPONSE leaves the trailing 8-byte length + raw stream bytes on the wire, so the next read_frame reads them as a header and every later frame is garbage. Streams are out of scope, but fail closed instead of mis-framing (the JS port errors the channel on unknown types — RpcChannel.ts:167).

Suggested change

	# UPLOAD_REQUEST / DOWNLOAD_RESPONSE are not yet handled.
	else:
	# streams are out of scope; we can't consume the trailing
	# length+bytes, so fail closed instead of desyncing
	raise ValueError(f"unsupported message type {msg_type!r}")

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

Receive loop swallows unexpected errors silently — add logging (and consider narrowing the blast radius)

_receive_loop's catch-all except Exception funnels every non-transport error into _fail_pending(ex) and then closes the connection, with no log anywhere on the failure path. The intent (per the # noqa: BLE001 comment) is to "surface anything unexpected via futures" — but that channel only works if a call happens to be in flight. The result is a few sharp edges:

• Silent loss. If the error originates while parsing an incoming frame (_handle_incoming_request / _handle_incoming_cancellation run synchronously on the loop) and there are no pending outgoing requests, _fail_pending iterates an empty dict and the exception evaporates — the connection just drops with zero diagnostics.
• Misattribution. If a call is in flight, an unrelated incoming-frame parse error (e.g. a malformed CancellationRequest) fails that call with a JSONDecodeError / KeyError that has nothing to do with it — looks like that call returned garbage.
• Bugs masquerade as disconnects. A genuine programming error in the dispatch path is indistinguishable from a peer hangup.

For reference, the .NET receive loop logs both exits — Connection_ReceiveLoopEndedSuccessfully on clean EOF and Connection_ReceiveLoopFailed(DebugName, ex) in its catch (Connection.cs:267/271). We have no equivalent here.

Minimal ask: log on the failure path, and split the clean-EOF case from the unexpected case:

# module level
import logging
_logger = logging.getLogger(__name__)

# in _receive_loop:
        except (asyncio.IncompleteReadError, ConnectionResetError, OSError) as ex:
            _logger.debug("receive loop ended (transport closed): %r", ex)
            self._fail_pending(ex)
        except Exception as ex:  # noqa: BLE001
            # Unexpected: a protocol/parse error or a genuine bug. The futures
            # channel only surfaces this when a call is in flight, so log it.
            _logger.exception("receive loop failed")
            self._fail_pending(ex)

Optional, if you want to go further (happy to file as a follow-up):

• Treat a payload parse error as recoverable rather than fatal — read_frame has already consumed the whole frame, so the stream is still aligned; only a true framing desync needs to tear the connection down. Logging + continue on a single bad payload would localize the blast radius.
• Don't fail unrelated outgoing _pending futures with an error that came from an incoming frame; if the request id parsed, an Error response back to the peer is more honest than nuking local calls.

Not blocking — the fail-fast instinct (vs. hanging callers forever) is right. This is about observability and not letting one bad frame disappear without a trace. Relatedly, there's no test that feeds a malformed frame; worth adding one that asserts the intended behavior here.

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

except BaseException here is broader than the C# behavior it mirrors — it swallows SystemExit / KeyboardInterrupt

The dedicated except asyncio.CancelledError just above is correct and necessary: CancelledError derives from BaseException (not Exception) since 3.8, so it must be caught explicitly to emit the System.OperationCanceledException wire type. (.NET gets this for free — its OperationCanceledException derives from Exception, so the server's single catch (Exception) handles cancellation as just another exception — Server.cs:103.)

But the follow-on except BaseException is wider than that catch (Exception). Once CancelledError is already handled above, the only extra types BaseException catches over Exception are SystemExit, KeyboardInterrupt, and GeneratorExit — and here they're converted into an Error response and never re-raised, so a process-termination signal raised inside a handler is silently turned into a wire error and swallowed. C#'s catch (Exception) never touches those (Ctrl+C is Console.CancelKeyPress, not an exception injected into running code), so this is strictly broader than the behavior we're matching.

The intent is presumably "always answer the peer so its pending future never hangs," which is reasonable. The best-of-both keeps that guarantee but still lets fatal signals propagate:

except BaseException as ex:
    resp = Response(request_id=req.id, error=Error(
        message=str(ex) or type(ex).__name__,
        type_name=type(ex).__name__, stack_trace=traceback.format_exc()))
    _logger.exception("callback %s.%s failed", req.endpoint, req.method_name)
    if isinstance(ex, (SystemExit, KeyboardInterrupt)):
        if not self._closed:
            try: await self._send_frame(MessageType.RESPONSE, resp.to_json().encode("utf-8"))
            except Exception: pass
        raise

Minor / non-blocking — under asyncio these signals are usually delivered to the loop driver rather than into a task, so it rarely triggers. Two related notes: (1) there's no log on this path either (same observability gap as the receive-loop comment), and (2) swallowing CancelledError without re-raising in the branch above is the usual asyncio smell — fine when the cancel is peer-initiated for this task, but on connection teardown you'd ideally re-raise (the self._closed check already guards the teardown send, so this is cosmetic).

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

CI only runs Python 3.12, but pyproject.toml declares requires-python = ">=3.10" — and the suite does not pass on 3.10/3.11

Running pytest on Python 3.10 (the declared minimum) fails 3 tests:

• tests/client/test_ipc_client.py::test_message_arg_sets_per_call_timeout
• tests/client/test_ipc_client.py::test_message_arg_with_payload_serializes_payload
• tests/client/test_timeout.py::test_request_includes_timeout_in_seconds_field

All three set a per-call/client timeout, so the proxy goes through asyncio.wait_for(conn.send_request(req), ...) (client/proxy.py:101). Each then does:

task = asyncio.create_task(svc.DoWork(Message(request_timeout=2.0)))
await asyncio.sleep(0)         # assumes ONE event-loop turn is enough
req = _sent_request(t.writer)  # buffer still empty -> JSONDecodeError / IndexError

On 3.10/3.11, asyncio.wait_for wraps the coroutine in a Task (ensure_future) that is scheduled after the test's own sleep(0) resumption, so the REQUEST frame is written one event-loop turn later than the test reads the buffer. On 3.12, wait_for was reimplemented to await the coroutine inline, so the write lands in the same turn. (The no-timeout tests take the plain await send_request path and pass on every version — which is why only these three, the timeout path, fail.)

This is a test-synchronization bug, not a product bug — the request is written correctly, just one turn later. Repro on 3.10.19:

no-timeout (direct await):    bytes after 1x sleep(0) = 105
per-call timeout (wait_for):  bytes after 1x sleep(0) = 0;  written after 1 extra turn = 109

Two fixes, ideally both:

1. Make the 3 tests poll for the bytes instead of assuming a single turn — the codebase already has the right pattern in test_timeout_sends_cancellation_to_server (client/test_timeout.py:86):

   for _ in range(20):
       await asyncio.sleep(0)
       if len(t.writer.buffer) >= 5:
           break

2. Add 3.10 (and 3.11) to this CI matrix, or bump requires-python to >=3.12 if 3.10 is not actually supported. As-is, CI runs only 3.12, so it can never catch a regression on the declared-minimum interpreter.

Not blocking the protocol work, but a consumer on 3.10 (the stated floor) hits this the moment they run the suite.

[eduard-dumitru]

System.Collections.Hashtable.body

[Radu-Niculae]

byte[] / BCL value types. byte[] round-trips silently wrong — .NET returns it base64, so the caller gets 'BAMCAQ==', not [4,3,2,1] (same shape issue for DateTime / Decimal / Guid). The robot client's _wire.py doesn't cover these either — only enums + nested dataclasses.

ISystemService.ReverseBytes is declared in the Python contract but no test actually calls it, and the obvious round-trip test wouldn't pass — for exactly the base64 reason above.

Question: the robot client had to hand-roll _wire.py (dataclass↔dict via fields + type hints, enum→value, PascalCase) to bridge all of this. Should we pull something like that into uipath-ipc itself as a base conversion layer — typed (de)serialization out of the box, instead of every consumer reinventing it?

[Radu-Niculae]

Key-mismatch silent loss is unguarded. A wrong, snake_case, typo'd, or nested-mismatched key is silently dropped and the field defaults, with no error (Person snake_case → {FirstName: None, LastName: None}).

small test of behavior:

class ISerializationProbe(ABC):
    @abstractmethod
    async def EchoPerson(self, p: dict) -> dict: ...

async def test_C_person_snake_case_silently_lost(dotnet_server) -> None:
    """Idiomatic snake_case keys can't be case-bridged (the underscore), so
    every field is silently dropped: the .NET Person comes back all-null, no
    error raised."""
    async with _client() as client:
        probe = client.get_proxy(ISerializationProbe)
        result = await probe.EchoPerson(
            {"first_name": "Ada", "last_name": "Lovelace"}
        )
        assert result == {"FirstName": None, "LastName": None}

.NET declares Person like this:

public sealed record Person
{
    public string? FirstName { get; init; }
    public string? LastName { get; init; }
    public override string ToString() =>
        $"FirstName={FirstName ?? "<null>"}, LastName={LastName ?? "<null>"}";
}

I think we should be more strict on deserialization, add required fields and type mismatch errors. Extra fields can be ignored for future version compatibility. On .NET strong typing should help us, but on python we might need to bring the _wire.py (or Claude suggested to use pydantic v2) into the lib to check type integrity (ex for dataclass) . What do you think?