docs: durable execution + update self-hosted defaults to use v1 (#1484)

* docs: update self-hosted defaults to use v1

* docs: durable execution

* rm dep

* lint: run black

* redundant readme

* more wording

* other small things

* isort

Author: abelanger5

README.md

@@ -36,7 +36,7 @@ Hatchet is a platform for running background tasks, built on top of Postgres. In
 
 Background tasks are critical for offloading work from your main web application. Usually background tasks are sent through a FIFO (first-in-first-out) queue, which helps guard against traffic spikes (queues can absorb a lot of load) and ensures that tasks are retried when your task handlers error out. Most stacks begin with a library-based queue backed by Redis or RabbitMQ (like Celery or BullMQ). But as your tasks become more complex, these queues become difficult to debug, monitor and start to fail in unexpected ways.
 
-This is where Hatchet comes in. Hatchet is a full-featured background task management platform, with built-in support for chaining complex tasks together into workflows, alerting on failures, and building complex workflows which would otherwise take months of engineering effort.
+This is where Hatchet comes in. Hatchet is a full-featured background task management platform, with built-in support for chaining complex tasks together into workflows, alerting on failures, making tasks more durable, and viewing tasks in a real-time web dashboard.
 
 ### Features
 
@@ -730,7 +730,7 @@ Most AI frameworks are built to run in-memory, with horizontal scaling and durab
 
 ### Issues
 
-Please submit any bugs that you encounter via Github issues. However, please reach out on [Discord](https://hatchet.run/discord) before submitting a feature request - as the project is very early, we'd like to build a solid foundation before adding more complex features.
+Please submit any bugs that you encounter via Github issues.
 
 ### I'd Like to Contribute
 

examples/v1/workflows/durable-event.go

@@ -21,9 +21,10 @@ type DurableEventOutput struct {
 }
 
 func DurableEvent(hatchet v1.HatchetClient) workflow.WorkflowDeclaration[DurableEventInput, DurableEventOutput] {
-	durableEventWorkflow := factory.NewDurableTask(
+	// ❓ Durable Event
+	durableEventTask := factory.NewDurableTask(
 		create.StandaloneTask{
-			Name: "durable-sleep",
+			Name: "durable-event",
 		},
 		func(ctx worker.DurableHatchetContext, input DurableEventInput) (*DurableEventOutput, error) {
 			eventData, err := ctx.WaitForEvent("user:update", "")
@@ -45,6 +46,34 @@ func DurableEvent(hatchet v1.HatchetClient) workflow.WorkflowDeclaration[Durable
 		},
 		hatchet,
 	)
+	// !!
 
-	return durableEventWorkflow
+	factory.NewDurableTask(
+		create.StandaloneTask{
+			Name: "durable-event",
+		},
+		func(ctx worker.DurableHatchetContext, input DurableEventInput) (*DurableEventOutput, error) {
+			// ❓ Durable Event With Filter
+			eventData, err := ctx.WaitForEvent("user:update", "input.user_id == '1234'")
+			// !!
+
+			if err != nil {
+				return nil, err
+			}
+
+			v := EventData{}
+			err = eventData.Unmarshal(&v)
+
+			if err != nil {
+				return nil, err
+			}
+
+			return &DurableEventOutput{
+				Data: v,
+			}, nil
+		},
+		hatchet,
+	)
+
+	return durableEventTask
 }

examples/v1/workflows/durable-sleep.go

@@ -20,7 +20,7 @@ type DurableSleepOutput struct {
 }
 
 func DurableSleep(hatchet v1.HatchetClient) workflow.WorkflowDeclaration[DurableSleepInput, DurableSleepOutput] {
-	// ctx as first param of NewTask
+	// ❓ Durable Sleep
 	simple := factory.NewDurableTask(
 		create.StandaloneTask{
 			Name: "durable-sleep",
@@ -38,6 +38,7 @@ func DurableSleep(hatchet v1.HatchetClient) workflow.WorkflowDeclaration[Durable
 		},
 		hatchet,
 	)
+	// !!
 
 	return simple
 }

frontend/docs/pages/home/_meta.js

@@ -49,15 +49,28 @@ export default {
   },
   "conditional-workflows": "Conditional Workflows",
   "on-failure-tasks": "On Failure Tasks",
-  "durable-execution": {
-    "title": "Durable Execution"
-  },
   "child-spawning": {
     "title": "Child Spawning"
   },
   "additional-metadata": {
     "title": "Additional Metadata"
   },
+  "--durable-execution": {
+    "title": "Durable Execution",
+    "type": "separator"
+  },
+  "durable-execution": {
+    "title": "Durable Execution"
+  },
+  "durable-events": {
+    "title": "Durable Events"
+  },
+  "durable-sleep": {
+    "title": "Durable Sleep"
+  },
+  "durable-best-practices": {
+    "title": "Best Practices"
+  },
   "--error-handling": {
     "title": "Error Handling",
     "type": "separator"

frontend/docs/pages/home/durable-best-practices.mdx

@@ -0,0 +1,19 @@
+## Durable Execution Best Practices
+
+Durable tasks require a bit of extra work to ensure that they are not misused. An important concept in running a durable task is that the task should be **deterministic**. This means that the task should always perform the same sequence of operations in between retries.
+
+The deterministic nature of durable tasks is what allows Hatchet to replay the task from the last checkpoint. If a task is not deterministic, it may produce different results on each retry, which can lead to unexpected behavior.
+
+## Maintaining Determinism
+
+By following a few simple rules, you can ensure that your durable tasks are deterministic:
+
+1. **Only call methods available on the `DurableContext`**: a very common way to introduce non-determinism is to call methods within your application code which produces side effects. If you need to call a method in your application code which fetches data from a database, calls any sort of i/o operation, or otherwise interacts with other systems, you should spawn those tasks as a **child task** or **child workflow** using `RunChild`.
+
+2. **When updating durable tasks, always guarantee backwards compatibility**: if you change the order of operations in a durable task, you may break determinism. For example, if you call `SleepFor` followed by `WaitFor`, and then change the order of those calls, Hatchet will not be able to replay the task correctly. This is because the task may have already been checkpointed at the first call to `SleepFor`, and if you change the order of operations, the checkpoint is meaningless.
+
+## Using DAGs instead of durable tasks
+
+[DAGs](./dags) are generally a much easier, more intuitive way to run a durable, deterministic workflow. DAGs are inherently deterministic, as their shape of work is predefined, and they cache intermediate results. If you are running simple workflows that can be represented as a DAG, you should use DAGs instead of durable tasks. DAGs also have conditional execution primitives which match the behavior of `SleepFor` and `WaitFor` in durable tasks.
+
+Durable tasks are useful if you need to run a workflow that is not easily represented as a DAG.

frontend/docs/pages/home/durable-events.mdx

@@ -0,0 +1,66 @@
+import { Callout, Card, Cards, Steps, Tabs } from "nextra/components";
+import UniversalTabs from "@/components/UniversalTabs";
+import { GithubSnippet, getSnippets } from "@/components/code";
+
+export const DurableGo = {
+  path: "examples/v1/workflows/durable-event.go",
+};
+
+export const DurableTs = {
+  path: "src/v1/examples/durable-event/workflow.ts",
+};
+
+export const DurablePy = {
+  path: "examples/durable_event/worker.py",
+};
+
+export const getStaticProps = ({}) =>
+  getSnippets([DurableGo, DurableTs, DurablePy]);
+
+## Durable Events
+
+Durable events are a feature of **durable tasks** which allow tasks to wait for an event to occur before continuing. This is useful in cases where a task needs to wait for a long time for an external action. Durable events are useful, because even if your task is interrupted and requeued while waiting for an event, the event will still be processed. When the task is resumed, it will read the event from the durable event log and continue processing.
+
+## Declaring durable events
+
+Durable events are declared using the context method `WaitFor` (or utility method `WaitForEvent`) on the `DurableContext` object.
+
+<UniversalTabs items={["Python", "Typescript", "Go"]}>
+  <Tabs.Tab title="Python">
+
+<GithubSnippet src={DurablePy} target="Durable Event" />
+
+</Tabs.Tab>
+<Tabs.Tab title="Typescript">
+
+<GithubSnippet src={DurableTs} target="Durable Event" />
+
+</Tabs.Tab>
+<Tabs.Tab title="Go">
+
+<GithubSnippet src={DurableGo} target="Durable Event" />
+
+</Tabs.Tab>
+</UniversalTabs>
+
+## Durable event filters
+
+Durable events can be filtered using [CEL](https://github.com/google/cel-spec) expressions. For example, to only receive `user:update` events for a specific user, you can use the following filter:
+
+<UniversalTabs items={["Python", "Typescript", "Go"]}>
+  <Tabs.Tab title="Python">
+
+<GithubSnippet src={DurablePy} target="Durable Event With Filter" />
+
+</Tabs.Tab>
+<Tabs.Tab title="Typescript">
+
+<GithubSnippet src={DurableTs} target="Durable Event With Filter" />
+
+</Tabs.Tab>
+<Tabs.Tab title="Go">
+
+<GithubSnippet src={DurableGo} target="Durable Event With Filter" />
+
+</Tabs.Tab>
+</UniversalTabs>

frontend/docs/pages/home/durable-execution.mdx

@@ -21,9 +21,7 @@ This is especially useful in cases such as:
 
 ## How Hatchet Runs Durable Tasks
 
-When you register a durable task, Hatchet marks the entire task as durable. Then, when you start your worker, Hatchet will start a second worker in the background for running durable tasks.
-
-If you don't register any durable tasks, the durable worker will not be started. Similarly, if you start a worker with _only_ durable tasks, the "main" worker will not start, and _only_ the durable worker will run. The durable worker will show up as a second worker in the Hatchet Dashboard.
+When you register a durable task, Hatchet will start a second worker in the background for running durable tasks. If you don't register any durable workflows, the durable worker will not be started. Similarly, if you start a worker with _only_ durable workflows, the "main" worker will not start, and _only_ the durable worker will run. The durable worker will show up as a second worker in the Hatchet Dashboard.
 
 Tasks that are declared as being durable (using `durable_task` instead of `task`), will receive a `DurableContext` object instead of a normal `Context,` which extends the `Context` by providing some additional tools for working with durable execution features.
 

frontend/docs/pages/home/durable-sleep.mdx

@@ -0,0 +1,46 @@
+import { Callout, Card, Cards, Steps, Tabs } from "nextra/components";
+import UniversalTabs from "@/components/UniversalTabs";
+import { GithubSnippet, getSnippets } from "@/components/code";
+
+export const DurableGo = {
+  path: "examples/v1/workflows/durable-sleep.go",
+};
+
+export const DurableTs = {
+  path: "src/v1/examples/durable-sleep/workflow.ts",
+};
+
+export const DurablePy = {
+  path: "examples/durable_sleep/worker.py",
+};
+
+export const getStaticProps = ({}) =>
+  getSnippets([DurableGo, DurableTs, DurablePy]);
+
+## Durable Sleep
+
+Durable sleep is a feature of **durable tasks** which allow tasks to pause execution for a specified amount of time. Instead of a regular `sleep` call in your task, durable sleep is guaranteed to only sleep for the specified amount of time after the first time it was called.
+
+For example, say you'd like to send a notification to a user after 24 hours. With a regular `sleep`, if the task is interrupted after 23 hours, it will restart and call `sleep` for 24 hours again. This means that the task will sleep for 47 hours in total, which is not what you want. With durable sleep, the task will respect the original sleep duration on restart -- that is, if the task calls `ctx.aio_sleep_for` for 24 hours and is interrupted after 23 hours, it will only sleep for 1 more hour on restart.
+
+## Using durable sleep
+
+Durable sleep can be used by calling the `SleepFor` method on the `DurableContext` object. This method takes a duration as an argument and will sleep for that duration.
+
+<UniversalTabs items={["Python", "Typescript", "Go"]}>
+  <Tabs.Tab title="Python">
+
+<GithubSnippet src={DurablePy} target="Durable Sleep" />
+
+</Tabs.Tab>
+<Tabs.Tab title="Typescript">
+
+<GithubSnippet src={DurableTs} target="Durable Sleep" />
+
+</Tabs.Tab>
+<Tabs.Tab title="Go">
+
+<GithubSnippet src={DurableGo} target="Durable Sleep" />
+
+</Tabs.Tab>
+</UniversalTabs>