From b98b1d8d123c7b325d7c3c571517cad1b2a698f4 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:59:37 +0100 Subject: [PATCH 1/8] thinking about structure --- docs/deploy/portability-v1.md | 439 ++++++++++++++++++++++++++++++++++ docs/deploy/portability.md | 58 +++-- 2 files changed, 482 insertions(+), 15 deletions(-) create mode 100644 docs/deploy/portability-v1.md diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v1.md new file mode 100644 index 00000000000..679872d94e2 --- /dev/null +++ b/docs/deploy/portability-v1.md @@ -0,0 +1,439 @@ +--- +title: Portability v1 (legacy) +--- + +The portability specification allows for the representations of entire workflow +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. + +If you're interested in contributing to the specification, reach out to OpenFn +via the [community forum](https://community.openfn.org), write to us, or suggest +changes by submitting a pull request here. + +:::warning + +This is the legacy version of the OpenFn Portability spec. + +For the latest version, see [Portability](portability) + +::: + +## Projects "as code" + +Entire projects (groups of workflows with their associated triggers, edges, +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by (a) allowing workflows to be +built and tested locally; (b) enabling project version control and an audit +trail of project changes; and (c) allowing users to port existing projects +between different instances (i.e., deployments) of Lightning. + +### Directory structure + +Many users keep OpenFn projects in git repositories, and this is a common +structure: + +``` +myProject/ +├── workflow-a/ +│ ├── job-1.js +│ ├── job-2.js +│ └── job-3.js +├── workflow-b/ +│ └── job-4.js +├── project.yaml +├── projectState.json +└── config.json +``` + +:::info Directory Structure + +There are commonly used 3 directory structure for OpenFn projects namely: +standard, production & test, and monorepo. To learn more, please see the OpenFn +[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). + +::: + +### The project **_spec_** + +The project specification (or "spec") is often saved as a `project.yaml` file. +While most of the spec is written inline, many developers prefer to track their +job bodies in separate `.js` files and they then reference them with a relative +path. + +```yaml +name: openhie-project +description: Some sample +credentials: + jane-smith@test.com-HAPI-FHIR: + owner: jane-smith@test.com + name: HAPI FHIR +workflows: + OpenHIE-Workflow: + name: OpenHIE Workflow + jobs: + FHIR-standard-Data-with-change: + name: FHIR-standard-Data-with-change + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/my-fancy-script.js + + Send-to-OpenHIM-to-route-to-SHR: + name: Send-to-OpenHIM-to-route-to-SHR + adaptor: '@openfn/language-http@latest' + enabled: true + credential: jane-smith@test.com-HAPI-FHIR + body: | + fn(state => { + console.log("hello github integration") + return state + }); + + Notify-CHW-upload-successful: + name: Notify-CHW-upload-successful + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: fn(state => state); + + Notify-CHW-upload-failed: + name: Notify-CHW-upload-failed + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/notify-failure.js + + triggers: + webhook: + type: webhook + edges: + webhook->FHIR-standard-Data-with-change: + source_trigger: webhook + target_job: FHIR-standard-Data-with-change + condition: always + FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: + source_job: FHIR-standard-Data-with-change + target_job: Send-to-OpenHIM-to-route-to-SHR + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-successful + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-failed + condition: on_job_failure +``` + +In this spec, you can see the different ways of defining a job's body: + +1. Inline body: Used in the `FHIR-standard-Data-with-change` and + `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the + YAML file. + +2. External file reference: Used in both `Notify-CHW-upload-successful` and + `Notify-CHW-upload-failed` jobs. The body is stored in separate files, + referenced by the path key. This allows for better organization of complex + job logic. + +When using file paths: + +- Paths are relative to the location of the `project.yaml` file. +- Ensure that the referenced files exist and contain valid job body code. +- This method is particularly useful for complex jobs or when you want to reuse + job bodies across different projects. + +### The project **_state_** + +The project state is a representation of a particular project as _on a specific +Lightning instance_. It is often saved as `projectState.json` and contains UUIDs +for resources on a particular Lightning deployment. + +```json +{ + "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", + "name": "openhie-project", + "description": null, + "inserted_at": "2023-08-25T08:57:31", + "updated_at": "2023-08-25T08:57:31", + "scheduled_deletion": null, + "requires_mfa": false, + "project_credentials": { + "jane-smith@test.com-HAPI-FHIR": { + "id": "25f48989-d349-4eb8-99c3-923ebba5b116", + "name": "HAPI FHIR", + "owner": "jane-smith@test.com" + } + }, + "workflows": { + "OpenHIE-Workflow": { + "id": "27ae2937-0959-48b8-a597-b1646aae8c14", + "name": "OpenHIE Workflow", + "jobs": { + "Transform-data-to-FHIR-standard": { + "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR": { + "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "name": "Send-to-OpenHIM-to-route-to-SHR", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-successful": { + "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", + "name": "Notify-CHW-upload-successful", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-failed": { + "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", + "name": "Notify-CHW-upload-failed", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "FHIR-standard-Data": { + "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", + "delete": true + }, + "FHIR-standard-Data-with-change": { + "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "name": "FHIR-standard-Data-with-change", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + } + }, + "triggers": { + "webhook": { + "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "type": "webhook" + } + }, + "edges": { + "webhook->Transform-data-to-FHIR-standard": { + "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", + "delete": true + }, + "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { + "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { + "id": "26c12f7f-7806-4008-87cd-6747998f95f4", + "condition": "on_job_success", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { + "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", + "condition": "on_job_failure", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" + }, + "webhook->FHIR-standard-Data": { + "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", + "delete": true + }, + "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { + "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", + "delete": true + }, + "webhook->FHIR-standard-Data-with-change": { + "id": "75e7f7d8-274b-410d-9600-730bbd535229", + "condition": "always", + "source_job_id": null, + "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" + }, + "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { + "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", + "condition": "on_job_success", + "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "source_trigger_id": null, + "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" + } + } + } + } +} +``` + +## Using the CLI interact with projects + +The project spec and project state can be used for a variety of reasons, e.g. +one could generate the state and spec as backups of the project or one could +generate these files and use them for auditing and record keeping, etc. The +OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with +commands that can be used to pull project configurations down from a running +Lightning server, and to deploy or push updates to existing projects on a +Lightning server. To learn more about automated version control via pull and +deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) +docs. + +:::info Don't have the CLI yet? + +Install it by running `npm install -g @openfn/cli` + +::: + +Before using the CLI, configure it either by passing in environment variables: + +``` +OPENFN_ENDPOINT=https://app.openfn.org +OPENFN_API_KEY=yourSecretApiToken +``` + +Or through a `config.json` file: + +```json +{ + // Required, can be overridden or set with `OPENFN_API_KEY` env var + "apiKey": "***", + + // Optional: can be set using the -p, defaults to project.yaml + "specPath": "project.yaml", + + // Optional: can be set using -s, defaults to .state.json + "statePath": ".state.json", + + // Optional: defaults to OpenFn.org's API, can be overridden or set with + // `OPENFN_ENDPOINT` env var + "endpoint": "https://app.openfn.org" +} +``` + +More details on the CLI can be found +[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). + +### `openfn pull` to generate spec & state + +To generate the spec and state files for an existing project, use: + +```sh +openfn pull {YOUR-PROJECT-UUID} -c ./config.json +``` + +This command will save (or overwrite) a project spec and state file based on the +path you've set in your configuration. + +### `openfn deploy` to create new projects + +To deploy a new project to a Lightning instance from a project spec (without a +project state) file use: + +```sh +openfn deploy -c config.json +``` + +### `openfn deploy` to update existing projects + +With a valid project state defined in your `config.json`, the same +`openfn deploy` command will beam up your changes as described by a difference +between your project spec and what's found on the server. + +```sh +openfn deploy -c config.json +Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. +Project found. +[CLI] ♦ Changes: + { + workflows: [ + { + jobs: [ + { +- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" ++ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" + } + ... + ... + ... + ] + } + ] + } + +? Deploy? yes +[CLI] ♦ Deployed. +``` + +## Getting Help with the cli + +The cli package comes with an inbuilt `help`. Adding `--help` to a command such +as `openfn deploy --help` will result in a help message describing the command +and the options available when using this command. See an example below + +```sh +openfn deploy --help +openfn deploy + +Deploy a project's config to a remote Lightning instance + +Options: + --version Show version number [boolean] + --help Show help [boolean] + -c, --config, --config-path The location of your config file [default: "./.config.json"] + --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] + --describe Downloads the project yaml from the specified instance [boolean] + -l, --log Set the log level [string] + --log-json Output all logs as JSON objects [boolean] + -p, --project-path The location of your project.yaml file [string] + -s, --state-path Path to the state file +``` + +## Troubleshooting + +This section covers solutions to some errors you might come across when using +OpenFn pull or deploy in your projects. + +### Extraneous Workflow ID + +#### Description + +This error occurs when you run `openfn deploy` and there is a mismatch between +between IDs of workflows in your projectSpec and your OpenFn instance. When this +occurs, the error will be written out in an error object as shown below: + +``` +[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: +{ + "errors": { + "workflows": { + "1-ingest-messages": { + "base": [ + "extraneous parameters: workflow_id" + ] + }, + "2-calculate-indicators": { + "base": [ + "extraneous parameters: workflow_id" + ] + } + } + } +``` + +#### Solution + +Run `openfn pull` to update your local instance and keep IDs in sync, +incorporate your changes and run `openfn deploy` again. + +## Other Versions + +- [Portability Spec v2](portability-versions#v2) +- [Portability Spec v1](portability-versions#v1) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index cc6d1474526..c8742218ae2 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,30 +2,58 @@ title: Portability --- -## Intent - The portability specification allows for the representations of entire workflow -projects "as code", lets user move between various deployment pathways (cloud, -local, DIY, etc.) and proposes a globally-applicable way of **_specifying -workflow automation_** and **_systems integration_** that might be applied -across workflow-engines/integration platforms across the sector. Nothing about -the spec _must_ be specific to OpenFn or any one of our individual products. We -envision a future in which software built with Lightning, the OpenFn Integration -Toolkit, and entirely new and different integration/workflow tools can adopt -this specification. +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. +:::info + +This document describes v2 of the OpenFn deployment spec, in-line with +[OpenFn Sync](/documentation/sync). + +For the v1 spec (using state, project and config files), see +[Legacy Portability](./portability-v1). + +::: + ## Projects "as code" Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. This improves the OpenFn -developer experience by (a) allowing workflows to be built and tested locally; -(b) enabling project version control and an audit trail of project changes; and -(c) allowing users to port existing projects between different instances (i.e., -deployments) of Lightning. +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by: + +1. Allowing workflows to be built and tested locally +2. Enabling project version control and an audit trail of project changes +3. Allowing users to port existing projects between different instances (i.e., + deployments) of Lightning. + +# Joe's Thoughts + +What should go in this file? + +The Sync document is the best equivalent. Should I just reference that? + +This document talks about a portability "spec". But I don't really see that +explained anywhere. Should I rewrite that stuff into a higher level manifesto? + +There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? + +I should chat with Taylor about what to do here. + +I am super unconfortable having multiple versions of the sync workflow on the +site ### Directory structure From 993c1cbe7c82d1287ce591ad044be17046a3df67 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 12:13:33 +0100 Subject: [PATCH 2/8] remove notes --- docs/deploy/portability.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index c8742218ae2..c385a80364c 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -39,22 +39,6 @@ This improves the OpenFn developer experience by: 3. Allowing users to port existing projects between different instances (i.e., deployments) of Lightning. -# Joe's Thoughts - -What should go in this file? - -The Sync document is the best equivalent. Should I just reference that? - -This document talks about a portability "spec". But I don't really see that -explained anywhere. Should I rewrite that stuff into a higher level manifesto? - -There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? - -I should chat with Taylor about what to do here. - -I am super unconfortable having multiple versions of the sync workflow on the -site - ### Directory structure Many users keep OpenFn projects in git repositories, and this is a common From 28b32d1e1895e84610a134db39fd3a7fb6c65ddf Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:24:54 +0100 Subject: [PATCH 3/8] typo --- docs/deploy/portability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index c385a80364c..fc5a2e9b312 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -3,7 +3,7 @@ title: Portability --- The portability specification allows for the representations of entire workflow -projects "as code"; lets user moves between various deployment pathways (such as +projects "as code"; lets users move between various deployment pathways (such as cloud, local, hosted); and proposes a globally-applicable way of defining workflow automation and system integration rules that might be applied across workflow-engines/integration platforms across the sector. From 188f902a71b0a341811898d832d9566ac8de5466 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:26:19 +0100 Subject: [PATCH 4/8] rollback changes --- docs/build-for-developers/cli-intro.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index bf58e97c5c4..944c7712318 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Sync workflows between OpenFn and a local filesystem or GitHub -- Securely run OpenFn workflows +- Securely run OpenFn steps and workflows - Troubleshoot and debug OpenFn steps -- Read and write Collections data +- Access adaptor documentation +- Deploy workflows to OpenFn --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output +Expand to see the expected output. ``` [CLI] ♦ Versions: From e2e01e0364b23109414d545b84d1314d7a850476 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:32:47 +0100 Subject: [PATCH 5/8] rollback again --- sidebars-main.js | 1 - 1 file changed, 1 deletion(-) diff --git a/sidebars-main.js b/sidebars-main.js index 6cb20f464cb..404fc48fe25 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,7 +124,6 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', - 'build-for-developers/cli-sync', 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', From 78e8ee3809a993aa431a2caff0c4d40b825e6ef6 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 7 May 2026 18:27:34 +0100 Subject: [PATCH 6/8] updates --- .../{portability-v1.md => portability-v3.md} | 2 +- docs/deploy/portability.md | 539 +++++------------- 2 files changed, 142 insertions(+), 399 deletions(-) rename docs/deploy/{portability-v1.md => portability-v3.md} (99%) diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v3.md similarity index 99% rename from docs/deploy/portability-v1.md rename to docs/deploy/portability-v3.md index 679872d94e2..51e848e6fcb 100644 --- a/docs/deploy/portability-v1.md +++ b/docs/deploy/portability-v3.md @@ -1,5 +1,5 @@ --- -title: Portability v1 (legacy) +title: Portability v3 (legacy) --- The portability specification allows for the representations of entire workflow diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index fc5a2e9b312..6951301e1e4 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,11 +2,30 @@ title: Portability --- -The portability specification allows for the representations of entire workflow -projects "as code"; lets users move between various deployment pathways (such as -cloud, local, hosted); and proposes a globally-applicable way of defining -workflow automation and system integration rules that might be applied across -workflow-engines/integration platforms across the sector. +The Portability Specification is an idea right at the heart of OpenFn Projects. +It is both a technical standard and an ongoing commitment. + +The Portability Specification ensures that code written in an OpenFn application +can be: + +- Deployed to another OpenFn instance (critical for production services running + in-country) +- Executed on a local machine (great news for developers) +- Ejected from OpenFn entirely and executed through a generic JavaScript runtime + +The Portability Specification drives the core functionality of OpenFn Sync, CLI +Deploy, Sandbox merging, and Project export/import from the app. + +:::info Legacy Portability Specifications + +Our commitment to portability hasn't changed within OpenFn's lifetime - but our +approach and implementation of this commitment have taken many forms. + +This document describes the latest Portability Specification, published in +May 2026. For older specifications, see +[Portability Versions](portability-versions) + +::: Nothing about the spec _must_ be specific to OpenFn or any one of our individual products. We envision a future in which software built with Lightning, the @@ -17,20 +36,10 @@ If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. -:::info - -This document describes v2 of the OpenFn deployment spec, in-line with -[OpenFn Sync](/documentation/sync). +## Projects As Code -For the v1 spec (using state, project and config files), see -[Legacy Portability](./portability-v1). - -::: - -## Projects "as code" - -Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. +A core tenant of OpenFn Projects is that they can be represented as code, on a +file system or a git branch. This improves the OpenFn developer experience by: @@ -39,415 +48,149 @@ This improves the OpenFn developer experience by: 3. Allowing users to port existing projects between different instances (i.e., deployments) of Lightning. -### Directory structure +## Project Spec -Many users keep OpenFn projects in git repositories, and this is a common -structure: +:::warning TODO -``` -myProject/ -├── workflow-a/ -│ ├── job-1.js -│ ├── job-2.js -│ └── job-3.js -├── workflow-b/ -│ └── job-4.js -├── project.yaml -├── projectState.json -└── config.json -``` +project file? Project text? Project source? Oh I like source + +::: -:::info Directory Structure +The unit of portability - the thing that encodes a Project and allows it to be +shared, synced, deployed and edited - is called a Project Spec. -There are 3 commonly used directory structures for OpenFn projects, namely: -standard, production & test, and monorepo. To learn more, please see the OpenFn -[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). +It is a structured artifact which defines a set of workflows, and for each +workflow, its, its core configuration, and the sequence of steps which it +executes. We usually represent this structure as YAML, because it's convenient +for humans and machines, but it can be represented in any text format. -::: +With a copy of a project spec, users can: -### The project **_spec_** +- Import a project into an OpenFn app instance +- Execute workflows locally with the CLI +- Deploy a project to an OpenFn app instance +- Merge sandbox projects locally -The project specification (or "spec") is often saved as a `project.yaml` file. -While most of the spec is written inline, many developers prefer to track their -job bodies in separate `.js` files and they then reference them with a relative -path. +Keys are regularly added to this structure as new features are introduced. We +expect and ensure that these keys are supported in all applications of the spec. + +Project specs can be exported from the app via the Settings page. + +Workflows can also be interchanged independently, using the same spec. So you +can import a Workflow to an existing project, or execute it locally without +cloning a whole project. + +## Spec Example + +Here is an example Project spec in YAML format: ```yaml -name: openhie-project -description: Some sample +id: portability-example +name: Portability Example +collections: + - cache credentials: - jane-smith@test.com-HAPI-FHIR: - owner: jane-smith@test.com - name: HAPI FHIR + - name: local login + owner: editor@openfn.org workflows: - OpenHIE-Workflow: - name: OpenHIE Workflow - jobs: - FHIR-standard-Data-with-change: - name: FHIR-standard-Data-with-change - adaptor: '@openfn/language-http@latest' + - name: Event-based workflow + steps: + - id: transform-data + name: Transform data + expression: fn(s => s) + adaptor: '@openfn/language-common@latest' + - id: webhook + type: webhook + webhook_reply: before_start enabled: true - credential: null - body: - path: ./jobs/my-fancy-script.js - - Send-to-OpenHIM-to-route-to-SHR: - name: Send-to-OpenHIM-to-route-to-SHR - adaptor: '@openfn/language-http@latest' + next: + transform-data: + disabled: false + condition: always + id: event-based-workflow + start: webhook + - name: Scheduled workflow + steps: + - id: common + name: Common + expression: fn(s => s) + adaptor: '@openfn/language-common@3.3.1' + - id: cron + type: cron enabled: true - credential: jane-smith@test.com-HAPI-FHIR - body: | - fn(state => { - console.log("hello github integration") - return state - }); - - Notify-CHW-upload-successful: - name: Notify-CHW-upload-successful + cron_expression: 00 00 * * 1-5 + cron_cursor_job_id: fb95ea89-17d2-4773-823c-09770317aaed + next: + get-data: + disabled: false + condition: always + - id: get-data + name: Get data + expression: fn(s => s) adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: fn(state => state); - - Notify-CHW-upload-failed: - name: Notify-CHW-upload-failed - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/notify-failure.js - - triggers: - webhook: - type: webhook - edges: - webhook->FHIR-standard-Data-with-change: - source_trigger: webhook - target_job: FHIR-standard-Data-with-change - condition: always - FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: - source_job: FHIR-standard-Data-with-change - target_job: Send-to-OpenHIM-to-route-to-SHR - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-successful - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-failed - condition: on_job_failure -``` - -In this spec, you can see the different ways of defining a job's body: - -1. Inline body: Used in the `FHIR-standard-Data-with-change` and - `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the - YAML file. - -2. External file reference: Used in both `Notify-CHW-upload-successful` and - `Notify-CHW-upload-failed` jobs. The body is stored in separate files, - referenced by the path key. This allows for better organization of complex - job logic. - -When using file paths: - -- Paths are relative to the location of the `project.yaml` file. -- Ensure that the referenced files exist and contain valid job body code. -- This method is particularly useful for complex jobs or when you want to reuse - job bodies across different projects. - -### The project **_state_** - -The project state is a representation of a particular project as _on a specific -Lightning instance_. It is often saved as `projectState.json` and contains UUIDs -for resources on a particular Lightning deployment. - -```json -{ - "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", - "name": "openhie-project", - "description": null, - "inserted_at": "2023-08-25T08:57:31", - "updated_at": "2023-08-25T08:57:31", - "scheduled_deletion": null, - "requires_mfa": false, - "project_credentials": { - "jane-smith@test.com-HAPI-FHIR": { - "id": "25f48989-d349-4eb8-99c3-923ebba5b116", - "name": "HAPI FHIR", - "owner": "jane-smith@test.com" - } - }, - "workflows": { - "OpenHIE-Workflow": { - "id": "27ae2937-0959-48b8-a597-b1646aae8c14", - "name": "OpenHIE Workflow", - "jobs": { - "Transform-data-to-FHIR-standard": { - "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR": { - "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "name": "Send-to-OpenHIM-to-route-to-SHR", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-successful": { - "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", - "name": "Notify-CHW-upload-successful", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-failed": { - "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", - "name": "Notify-CHW-upload-failed", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "FHIR-standard-Data": { - "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", - "delete": true - }, - "FHIR-standard-Data-with-change": { - "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "name": "FHIR-standard-Data-with-change", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - } - }, - "triggers": { - "webhook": { - "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "type": "webhook" - } - }, - "edges": { - "webhook->Transform-data-to-FHIR-standard": { - "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", - "delete": true - }, - "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { - "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { - "id": "26c12f7f-7806-4008-87cd-6747998f95f4", - "condition": "on_job_success", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { - "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", - "condition": "on_job_failure", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" - }, - "webhook->FHIR-standard-Data": { - "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", - "delete": true - }, - "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { - "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", - "delete": true - }, - "webhook->FHIR-standard-Data-with-change": { - "id": "75e7f7d8-274b-410d-9600-730bbd535229", - "condition": "always", - "source_job_id": null, - "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" - }, - "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { - "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", - "condition": "on_job_success", - "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "source_trigger_id": null, - "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" - } - } - } - } -} + configuration: editor@openfn.org|local login + next: + throw-error: + disabled: false + condition: on_job_failure + common: + disabled: false + condition: '!state.error' + label: sometimes + never: + disabled: true + condition: on_job_success + - id: never + name: never + expression: fn(s => s) + adaptor: '@openfn/language-http@7.2.10' + - id: throw-error + name: throw error + expression: fn(s => s) + adaptor: '@openfn/language-common@3.3.1' + id: scheduled-workflow + start: cron ``` -## Using the CLI interact with projects +The latest schema for a project spec file is defined in TypeScript +[here](https://github.com/OpenFn/kit/blob/5e4d65af25a6854886c15294ed4cf17f93ecbc19/packages/lexicon/portability.d.ts) -:::warning +## Linked Resources -This document describes the legacy OpenFn Sync format used by older versions of -the CLI and GitHub Sync. +:::warning TODO -See the [CLI Sync](/documentation/sync) docs for the best experience syncing and -deploying projects. +I don't really want to talk about state here - but I do want to touch on the +idea that the spec is uncoupled from an instance, and that we have these "linked +resources" which on deploy get assigned UUIDs -::: +But the new statefile, main@app.openfn.org.yaml, is not part of the portability +spec! It's just an artefact used by the CLI to track and instance. -The project spec and project state can be used for a variety of reasons, e.g. -one could generate the state and spec as backups of the project or one could -generate these files and use them for auditing and record keeping, etc. The -OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with -commands that can be used to pull project configurations down from a running -Lightning server, and to deploy or push updates to existing projects on a -Lightning server. To learn more about automated version control via pull and -deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) -docs. - -:::info Don't have the CLI yet? - -Install it by running `npm install -g @openfn/cli` +If the SPEC is standard for interopability which might go beyond openfn, the +STATEFILE isa proprietary artefact of sync which is 100% coupled to the app ::: -Before using the CLI, configure it either by passing in environment variables: - -``` -OPENFN_ENDPOINT=https://app.openfn.org -OPENFN_API_KEY=yourSecretApiToken -``` - -Or through a `config.json` file: - -```json -{ - // Required, can be overridden or set with `OPENFN_API_KEY` env var - "apiKey": "***", - - // Optional: can be set using the -p, defaults to project.yaml - "specPath": "project.yaml", - - // Optional: can be set using -s, defaults to .state.json - "statePath": ".state.json", - - // Optional: defaults to OpenFn.org's API, can be overridden or set with - // `OPENFN_ENDPOINT` env var - "endpoint": "https://app.openfn.org" -} -``` - -More details on the CLI can be found -[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). - -### `openfn pull` to generate spec & state - -To generate the spec and state files for an existing project, use: - -```sh -openfn pull {YOUR-PROJECT-UUID} -c ./config.json -``` - -This command will save (or overwrite) a project spec and state file based on the -path you've set in your configuration. +Project State links a project to related artifacts which live on a specific +instance of the app - like a user or a credential. -### `openfn deploy` to create new projects +The Project Spec does not link directly to those resources. Rather, it +references those resources by an identifier string. -To deploy a new project to a Lightning instance from a project spec (without a -project state) file use: +When syncing or deploying a project, the hosting app will attempt to reference +an artifact with the same identifier within the project's scope, and connect it. +That connection is local to that instance and usually requires a UUID reference. +This connection is not portable. -```sh -openfn deploy -c config.json -``` - -### `openfn deploy` to update existing projects - -With a valid project state defined in your `config.json`, the same -`openfn deploy` command will beam up your changes as described by a difference -between your project spec and what's found on the server. - -```sh -openfn deploy -c config.json -Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. -Project found. -[CLI] ♦ Changes: - { - workflows: [ - { - jobs: [ - { -- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" -+ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" - } - ... - ... - ... - ] - } - ] - } - -? Deploy? yes -[CLI] ♦ Deployed. -``` - -## Getting Help with the cli - -The cli package comes with an inbuilt `help`. Adding `--help` to a command such -as `openfn deploy --help` will result in a help message describing the command -and the options available when using this command. See an example below - -```sh -openfn deploy --help -openfn deploy - -Deploy a project's config to a remote Lightning instance - -Options: - --version Show version number [boolean] - --help Show help [boolean] - -c, --config, --config-path The location of your config file [default: "./.config.json"] - --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] - --describe Downloads the project yaml from the specified instance [boolean] - -l, --log Set the log level [string] - --log-json Output all logs as JSON objects [boolean] - -p, --project-path The location of your project.yaml file [string] - -s, --state-path Path to the state file -``` +:::warning TODO -## Troubleshooting +I want to neatly reference users of the portability spec. -This section covers solutions to some errors you might come across when using -OpenFn pull or deploy in your projects. +Like "see CLI Sync" to read more about how portability works in practice. -### Extraneous Workflow ID - -#### Description - -This error occurs when you run `openfn deploy` and there is a mismatch between -IDs of workflows in your projectSpec and your OpenFn instance. When this -occurs, the error will be written out in an error object as shown below: - -``` -[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: -{ - "errors": { - "workflows": { - "1-ingest-messages": { - "base": [ - "extraneous parameters: workflow_id" - ] - }, - "2-calculate-indicators": { - "base": [ - "extraneous parameters: workflow_id" - ] - } - } - } -``` - -#### Solution - -Run `openfn pull` to update your local instance and keep IDs in sync, -incorporate your changes and run `openfn deploy` again. +::: -## Other Versions +## Execution -- [Portability Spec v2](portability-versions#v2) -- [Portability Spec v1](portability-versions#v1) +TODO: touch on executing workflows with CLI, and also compiling a workflow and +running that with native node. From 48f07f1e682c805adedcbf8ab3b350a0d900978f Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sat, 9 May 2026 12:32:59 +0100 Subject: [PATCH 7/8] remove UUID from example --- docs/deploy/portability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 6951301e1e4..eea225e4254 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -119,7 +119,7 @@ workflows: type: cron enabled: true cron_expression: 00 00 * * 1-5 - cron_cursor_job_id: fb95ea89-17d2-4773-823c-09770317aaed + cron_cursor_job_id: get-data next: get-data: disabled: false From 446e8e07b320827e5cdb5f7b6d7baab33aacf8b3 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sun, 10 May 2026 12:02:44 +0100 Subject: [PATCH 8/8] words --- docs/deploy/portability.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index eea225e4254..6885c76f4b9 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -3,18 +3,17 @@ title: Portability --- The Portability Specification is an idea right at the heart of OpenFn Projects. -It is both a technical standard and an ongoing commitment. - -The Portability Specification ensures that code written in an OpenFn application -can be: +It is both a technical standard and an ongoing commitment. It ensures that code +written in an OpenFn application can be: - Deployed to another OpenFn instance (critical for production services running in-country) -- Executed on a local machine (great news for developers) +- Executed on a local machine (great news for developers building workflows or + adaptors) - Ejected from OpenFn entirely and executed through a generic JavaScript runtime -The Portability Specification drives the core functionality of OpenFn Sync, CLI -Deploy, Sandbox merging, and Project export/import from the app. +This manifesto drives the core functionality of OpenFn Sync, CLI Deploy, Sandbox +merging, and Project export/import from the app. :::info Legacy Portability Specifications