Single workflow pause API #8605

gow · 2025-11-06T23:58:38Z

What changed?

Note: This API is part of an experimental feature - "pause single workflow"

Added a new API (in Frontend & History) to pause workflow execution.
Updated DescribeWorkflowExecution API to show the pause info.
Added a dynamic config to control the access to this feature. The PauseWorkflowExecution API fails if the config is not set.
Added simple acceptance functional tests. These tests will evolve and new ones added as the feature implementation progresses.

Followup PRs:

Idempotency check in pause api.
Validate input sizes.
Intercept workflow task creation.
Intercept and pause activities.
Unpause API

Why?

This API is needed to implement the "pause single workflow" feature.

How did you test it?

Note

Adds a new PauseWorkflowExecution API (frontend + history) gated by per-namespace config and exposes pause info via DescribeWorkflowExecution, with client support and tests.

Historyservice (API + server):
- New RPC: PauseWorkflowExecution with messages PauseWorkflowExecutionRequest/Response in request_response.proto and service definitions; generated Go, gRPC handlers, and mocks.
- Engine/Handler: Implements pause via pauseworkflow.Invoke and Handler.PauseWorkflowExecution (adds workflow paused event; no WFT creation).
- Telemetry/Logging: Adds routing/log tags and telemetry allowlist for PauseWorkflowExecution.
Frontend:
- Endpoint: WorkflowHandler.PauseWorkflowExecution forwarding to history; gated by WorkflowPauseEnabled.
- Dynamic Config: Adds frontend.WorkflowPauseEnabled (namespace-scoped).
DescribeWorkflowExecution:
- Populates WorkflowExecutionExtendedInfo.PauseInfo when present.
Clients:
- Adds client, metric, and retryable wrappers for PauseWorkflowExecution.
Tests:
- Functional tests for pausing a workflow and config-disabled behavior.
Deps:
- Bumps go.temporal.io/api version.

^{Written by Cursor Bugbot for commit ed7ce3a. This will update automatically on new commits. Configure here.}

cursor · 2025-11-14T19:28:42Z

cmd/tools/genrpcwrappers/main.go

 		f, ok := t.FieldByName(fieldName)
 		if !ok {
+			fmt.Printf("\nstack trace: path: [%s]\n", path)
+			fmt.Println(string(debug.Stack()))


Bug: Remove Accidental Debug Output

Debug print statements were accidentally left in the code generation tool. The fmt.Printf and fmt.Println statements with stack trace output appear to be temporary debugging code that should be removed before committing.

spkane31 · 2025-11-14T20:51:49Z

proto/internal/temporal/server/api/historyservice/v1/service.proto

    }

    // VerifyChildExecutionCompletionRecorded checks if child completion result is recorded in parent workflow.
-    // This is only used by standby transfer close execution logic to make sure parent workflow has the result 


Wonder if we can add a linter for these extra spaces at the end of lines, just muddles up your PR and they're hard to undo because it's likely a diff in editor settings

Yeah. I too agree this should be a lint rule. I'm not even sure which setting in my editor causes this on save.

spkane31 · 2025-11-14T22:11:15Z

tests/pause_workflow_execution_test.go

+	const (
+		testEndSignal = "test-end"
+		pauseIdentity = "functional-test"
+		pauseReason   = "pausing workflow for acceptance test"
+	)


these are duplicated in the tests, could they be package level consts?

This is a very large package. So adding a package level const is probably not a good idea. I'll initialize these in test setup so that they are done only once and scoped to this test.

spkane31 · 2025-11-14T22:11:58Z

tests/pause_workflow_execution_test.go

+	workflowFn := func(ctx workflow.Context) (string, error) {
+		signalCh := workflow.GetSignalChannel(ctx, testEndSignal)
+		var signalPayload string
+		signalCh.Receive(ctx, &signalPayload)
+		return signalPayload, nil
+	}


workflowFn is duplicated, could also break this out

spkane31 · 2025-11-14T22:14:47Z

tests/pause_workflow_execution_test.go

+	// TODO: currently pause workflow execution does not intercept workflow creation. Fix the reset of this test when that is implemented.
+	// For now sending this signal will complete the workflow and finish the test.
+	err = s.SdkClient().SignalWorkflow(ctx, workflowID, runID, testEndSignal, "test end signal")
+	s.NoError(err)
+
+	s.EventuallyWithT(func(t *assert.CollectT) {
+		desc, err := s.SdkClient().DescribeWorkflowExecution(ctx, workflowID, runID)
+		require.NoError(t, err)
+		info := desc.GetWorkflowExecutionInfo()
+		require.NotNil(t, info)
+		require.Equal(t, enumspb.WORKFLOW_EXECUTION_STATUS_COMPLETED, info.GetStatus())
+	}, 5*time.Second, 200*time.Millisecond)
+}


Making sure I understand this correctly, we don't intercept the signal yet, but we're planning on adding that in a future PR before the first release?

Yes. This workflow and the tests will change a bit in subsequent PRs.

service/history/api/pauseworkflow/api.go

yycptt · 2025-11-18T23:08:42Z

common/dynamicconfig/constants.go

 	)
+
+	WorkflowPauseEnabled = NewNamespaceBoolSetting(
+		"history.WorkflowPauseEnabled",


yycptt · 2025-11-19T01:16:46Z

service/frontend/workflow_handler.go

 }
+
+// PauseWorkflowExecution pauses a workflow execution.
+func (wh *WorkflowHandler) PauseWorkflowExecution(ctx context.Context, request *workflowservice.PauseWorkflowExecutionRequest) (_ *workflowservice.PauseWorkflowExecutionResponse, retError error) {


does pause workflow count as an action? if so we need to update the telemetry interceptor to emit an action metric with the api is called.

common/persistence/workflow_state_status_validator.go

yycptt · 2025-11-19T01:28:06Z

service/history/api/describeworkflow/api.go


+	// copy pause info to the response if it exists
+	if executionInfo.PauseInfo != nil {
+		result.WorkflowExecutionInfo.PauseInfo = &workflowpb.WorkflowExecutionPauseInfo{


WorkflowExecutionInfo is used by visibility as well. Shall we add PauseInfo to WorkflowExtendedInfo instead?

ok. I'll add it to WorkflowExtendedInfo and then update this PR.

yycptt · 2025-11-19T01:28:37Z

service/history/api/pauseworkflow/api.go

+		return nil, serviceerror.NewInvalidArgument("pause request is nil.")
+	}
+
+	if !shard.GetConfig().WorkflowPauseEnabled(namespaceID.String()) {


I think we should do this in frontend.

cursor · 2025-11-19T01:42:59Z

client/history/client_gen.go

+		return nil, err
+	}
+	return response, nil
+}


Bug: Potential null pointer dereference in client shard routing

The PauseWorkflowExecution client method calls request.GetPauseRequest().GetWorkflowId() without first checking if GetPauseRequest() returns nil. If the pause_request field is nil, this will cause a nil pointer dereference. While the history service API layer checks for nil and returns an error, the client-side wrapper should be defensive, similar to how CompleteNexusOperationChasm handles potential nil values by deserializing and checking before accessing nested fields.

This is in generated code.

service/frontend/workflow_handler.go

gow requested review from a team as code owners November 6, 2025 23:58

gow marked this pull request as draft November 6, 2025 23:58

gow mentioned this pull request Nov 7, 2025

Mutable state changes for workflow pause. #8560

Merged

5 tasks

gow force-pushed the cg/wf_pause_1 branch 2 times, most recently from 5a861df to 5c592f0 Compare November 14, 2025 06:48

gow force-pushed the cg/wf_pause_2_api branch from c76e337 to 728fbd9 Compare November 14, 2025 14:53

gow requested review from bergundy and spkane31 November 14, 2025 19:24

gow marked this pull request as ready for review November 14, 2025 19:24

cursor bot reviewed Nov 14, 2025

View reviewed changes

spkane31 reviewed Nov 14, 2025

View reviewed changes

cursor bot reviewed Nov 17, 2025

View reviewed changes

service/history/api/pauseworkflow/api.go Show resolved Hide resolved

gow mentioned this pull request Nov 18, 2025

[WorkflowPause] Input validation and idempotency check in pause API #8660

Open

5 tasks

gow requested a review from yycptt November 18, 2025 18:39

gow force-pushed the cg/wf_pause_1 branch from 5c592f0 to f7b64fe Compare November 18, 2025 19:09

yycptt reviewed Nov 19, 2025

View reviewed changes

gow force-pushed the cg/wf_pause_2_api branch from 2155224 to c002113 Compare November 19, 2025 01:36

gow force-pushed the cg/wf_pause_1 branch from 8a23d80 to 4754c85 Compare November 19, 2025 01:41

gow force-pushed the cg/wf_pause_2_api branch from c002113 to 408fa90 Compare November 19, 2025 01:41

cursor bot reviewed Nov 19, 2025

View reviewed changes

gow force-pushed the cg/wf_pause_1 branch from 4754c85 to 951c9c4 Compare November 19, 2025 06:42

gow force-pushed the cg/wf_pause_2_api branch from 408fa90 to 1d4e330 Compare November 19, 2025 06:43

Base automatically changed from cg/wf_pause_1 to main November 19, 2025 18:37

gow added 6 commits November 20, 2025 12:27

Added dynamicconfig.WorkflowPauseEnabled

f552430

API definitions for Pause operation in History service

a0a0cb9

Added pauseworkflow history api package

a76ac62

Add PauseWorkflowExecution gRPC API in history engine

63865d1

Use mutableState.AddWorkflowExecutionPausedEvent()

045f6c7

Implement PauseWorkflowExecution API in frontend

c1d1e1e

gow added 13 commits November 20, 2025 12:27

Get workflow ID from the request

b88c44d

fix getters

09f8031

fixes

4ced8f9

Remove usage of IsWorkflowExecutionStatusPaused()

6509fd8

Add functional acceptance test

6b6f83a

Added TestPauseWorkflowExecutionFailsWhenDisabled()

e3dffd9

nit fix

6e318a0

Remove debug statements

dbe04d2

Use s.ErrorAs()

0db7115

Address review comments

3ada932

Emit an action metric when pause api is called

e9771f4

Move WorkflowPauseEnabled() dynamic config check to frontend

98ef3ac

Move pause info into WorkflowExtendedInfo.PauseInfo in describe results

35acc0c

gow force-pushed the cg/wf_pause_2_api branch from 36c201e to 35acc0c Compare November 20, 2025 20:29

cursor bot reviewed Nov 20, 2025

View reviewed changes

service/frontend/workflow_handler.go Show resolved Hide resolved

Move nil check before accessing namespace

ed7ce3a

Single workflow pause API #8605

Are you sure you want to change the base?

Single workflow pause API #8605

Conversation

gow commented Nov 6, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Uh oh!

cursor bot Nov 14, 2025

Choose a reason for hiding this comment

Bug: Remove Accidental Debug Output

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Nov 19, 2025

Choose a reason for hiding this comment

Bug: Potential null pointer dereference in client shard routing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gow commented Nov 6, 2025 •

edited by cursor bot

Loading