Skip to content

websocket-protocol.md

Latest commit

 

History

History
130 lines (112 loc) · 7.6 KB

websocket-protocol.md

File metadata and controls

130 lines (112 loc) · 7.6 KB
Raw

WebSocket protocol — InteractiveInkCanvas ↔ MyScript backend

Full message lifecycle used by InteractiveInkCanvas (src/canvas/variants/InteractiveInkCanvas.ts) via WebSocketClient (src/client/WebSocketClient.ts) against the MyScript Cloud recognition server. All message shapes come from TWebSocketClientMessageType in src/client/WebSocketClientMessage.ts.

Note: the debug-websocket and recognizer-system project skills describe a different, outdated protocol (src/recognizer/..., message types like strokesAdded/sessionInitialized) — that path no longer exists in the codebase. This document reflects the current src/client/WebSocketClient.ts implementation; treat it as the source of truth.

sequenceDiagram
    participant App as Application
    participant Canvas as InteractiveInkCanvas
    participant Client as WebSocketClient
    participant Server as MyScript backend

    %% 1. Connection + HMAC auth
    App->>Canvas: Canvas.load(el, "INTERACTIVE_INK", options)
    Canvas->>Client: new WebSocketClient(configuration.server)
    Client->>Server: WS connect wss://.../offscreen?applicationKey=...
    Client->>Server: {type: "authenticate", myscript-client-name, myscript-client-version}
    Server-->>Client: {type: "hmacChallenge", hmacChallenge, iinkSessionId}
    Client->>Client: computeHmac(hmacChallenge, applicationKey, hmacKey)
    Client->>Server: {type: "hmac", hmac}
    Server-->>Client: {type: "authenticated"}

    %% 2. Session / content part init
    Client->>Server: {type: "initSession", scaleX, scaleY, configuration}
    Server-->>Client: {type: "sessionDescription", contentPartCount, iinkSessionId}
    Client->>Server: {type: "newContentPart", contentType, mimeTypes}
    Server-->>Client: {type: "newPart", id, idx}
    Client->>Canvas: resolve initialized promise
    Canvas-->>App: await editor.initialized.promise resolves

    %% keepalive (parallel, independent worker)
    loop every pingDelay (15s)
        Client->>Server: {type: "ping"}
        Server-->>Client: {type: "pong"}
    end

    %% 3. Stroke input + 4. real-time recognition
    App->>Canvas: user writes (pointer events)
    Canvas->>Canvas: writer / gesture / transform / erase managers build stroke ops
    Canvas->>Client: addStrokes(strokes) / eraseStrokes / transform / replaceStrokes
    Client->>Server: {type: "addStrokes", processGestures, strokes}
    Server-->>Client: {type: "gestureDetected", ...} (if a gesture matched)
    Server-->>Client: {type: "contentChanged", canUndo, canRedo, undoStackIndex}
    Client->>Canvas: emit CONTENT_CHANGED (debounced 500ms)
    Canvas->>Canvas: IISynchronizerManager.synchronize()

    %% 5. Export / JIIX
    Canvas->>Client: export(["application/vnd.myscript.jiix"])
    Client->>Server: {type: "export", partId, mimeTypes}
    Server-->>Client: {type: "exported", partId, exports}
    Client->>Canvas: resolve export promise (JIIX parsed)
    Canvas->>Canvas: IIJiixQueryManager rebuilds block index

    %% math solver round trip (optional, on-demand)
    Canvas->>Client: mathSolver request (get-variables / evaluate / ...)
    Client->>Server: {type: "mathSolver", action, blockId, ...}
    Server-->>Client: {type: "mathSolverResult", blockId, action, result}

    %% 6. Undo / redo (client-authoritative stack, round-tripped to server)
    App->>Canvas: undo()
    Canvas->>Canvas: IIHistoryManager pops local stack, extractIIBackendChanges()
    Canvas->>Client: undo(changes)
    Client->>Server: {type: "undo", changes}
    Server-->>Client: {type: "contentChanged", ...} (no dedicated ack)

    %% 7. Session close
    App->>Canvas: destroy()
    Canvas->>Client: destroy()
    Client->>Server: WS close frame (code 1000, "Client destroyed")

    %% 8. Error handling / reconnection
    Server--)Client: {type: "error", code, message} (e.g. no.activity, access.not.granted, session.too.old)
    alt reconnect (autoReconnect / offline queue)
        Client->>Client: enqueue offline messages (max 50)
        loop every reconnectDelay (3s), up to maxReconnectAttempts (10)
            Client->>Server: reconnect attempt (init())
        end
        Client->>Server: {type: "restoreSession", iinkSessionId} (on reconnect with known sessionId)
        Server-->>Client: {type: "sessionDescription", ...}
        Client->>Client: drain offline queue, replay queued messages
    else give up
        Client->>Canvas: emit CONNECTION_STATUS_CHANGED("error")
    end
Loading

Message reference

Direction type Purpose Handler
C→S authenticate Open handshake, client identity WebSocketClient.openCallback()
S→C hmacChallenge Challenge string to HMAC-sign manageHMACChallenge()
C→S hmac Computed HMAC response —
S→C authenticated Auth accepted manageAuthenticated()
C→S initSession / restoreSession Open new or resume existing session manageAuthenticated()
S→C sessionDescription Session id + content part count manageSessionDescriptionMessage()
C→S newContentPart / openContentPart Create or reopen a content part —
S→C newPart / partChanged Part ready, resolves initialized manageNewPartMessage() / managePartChangeMessage()
C→S ping Keepalive (every 15s, worker-driven) initPing()
S→C pong Keepalive ack messageCallback()
C→S addStrokes New stroke batch (chunks of 1000) addStrokes()
C→S replaceStrokes / transform / eraseStrokes / contextlessGesture Edit operations see WebSocketClient.ts
S→C contentChanged Model changed server-side, triggers re-sync manageContentChangedMessage()
S→C gestureDetected / contextlessGesture Gesture classification result manageGestureDetected()
C→S export Request export in given mime types export()
S→C exported Export payload (e.g. JIIX) manageExportMessage()
C→S mathSolver Math variable/evaluation request —
S→C mathSolverResult Math solver response manageMathSolverResult()
C→S undo / redo Backend-side change replay for undo/redo undo() / redo()
S→C error Server error (no.activity, access.not.granted, session.too.old, restore.session.not.found) manageErrorMessage()
— WS close frame Clean/unclean disconnect closeCallback(), ClientError.mapCloseCodeToMessage()

Reconnection & offline queue

Two independent mechanisms in WebSocketClient.ts:

  1. Legacy auto-reconnect in send() — if the socket is closing/closed and configuration.server.websocket.autoReconnect is true (default), retries init() + the send, up to maxRetryCount (default 2).
  2. Offline queue + background reconnect loop — if offlineQueueEnabled (default true), addStrokes() queues locally (max 50) while disconnected; a timer retries every reconnectDelay (default 3000ms) up to maxReconnectAttempts (default 10), then drains the queue in order. Gives up and emits CONNECTION_STATUS_CHANGED("error") if exhausted.

Source files

Concern File
Protocol state machine src/client/WebSocketClient.ts
Message type enum + payload shapes src/client/WebSocketClientMessage.ts
HMAC/applicationKey config src/client/ServerConfiguration.ts
HMAC computation src/utils/crypto.ts
Public event surface src/client/ClientEvent.ts
Close-code → message mapping src/client/ClientError.ts
Editor-side wiring (init/undo/redo/export/destroy) src/canvas/variants/InteractiveInkCanvas.ts
Debounced JIIX re-sync after contentChanged src/manager/interactive/IISynchronizerManager.ts