Tracker Protocol

The tracker protocol is really simple. It uses Websocket connections to reduce overhead. Connections have no state, except for HTTP Basic authentication to determine which pipeline the tracker is communicating with.

All commands must be atomic. Either they succeed, or they do not succeed, and there is no intermediate state. If something goes wrong, all persistent state must remain consistent. This allows for retries and makes the system more robust.

Schema

Requests

Requests currently have two fields—type and seq. type is the message type, and seq is a sequence number that the client generates to ensure messages don't get mixed up. Some requests will also require a third field, request, containing any additional parameters. The request field is a dictionary; keys must be strings.

Requests are sent from the client to the server.

Responses

Responses currently have three fields—status, payload, and seq. status is an HTTP status code corresponding to the result of the request. payload is the return value. seq is a copy of the seq value from the request.

In this article, when a response is described as having no payload, that should be interpreted as a payload of null. A server must provide a payload field even if it is null.

Responses are sent from the server to the client in response to requests.

Advisories

Advisories are a special case. They are not directly triggered by a request; instead, the server may subscribe a client to certain advisories as a result of a request. Advisories have three fields—type, payload, and seq. seq is always None, to denote the message as an advisory.

Commands

Requests

System:ping

No parameters.

Heartbeat; signals to the server that it is still connected and ready. Also ensures the connection is still functional.

Elicits a response with a status code of 204 and no payload.

Item:claim

Request parameters:

• pipeline_type: Pipeline type to dequeue from. (e.g. brozzler, zeno)

Claims an item corresponding to the given pipeline type.

Elicits a response with a status code of 200. If an item was found, it is contained in the item field of the payload. Otherwise, the item field of the payload is null, and the message field is set to a human-readable reason why an item could not be provided.

Item:fail

Request parameters:

• id: Identifier of the item.
• message: Failure reason.
• attempt: The client's understanding of what attempt the item is on, because it's possible it's out of sync with the database.
• fatal: Whether or not the error is fatal. (A fatal error won't requeue the item.)

Fails an item. It may or may not be cycled back into the queue.

Elicits a response with a status code of 204 and no payload.

Item:store

Request parameters:

• id: The identifier of the item.
• result: The result to store.
• attempt: The client's understanding of what attempt the item is on, because it's possible it's out of sync with the database.
• result_type: The type of result.
• decode_fields: Sometimes, like with screenshots, the field can't be stored in JSON. If this field is present, it should be an array of fields to decode (with Python's base64.b85decode base85 function).

Saves a result associated with the item.

Elicits a response with a status code of 201, and a new_id field containing the generated primary key of the result.

Item:finish

Request parameters:

• id: The identifier of the item.

Marks an item as finished.

Elicits a response with a status code of 204 and no payload.

Advisories