Issue: Cookies Not Persisting Between Workflows in Persistent Browser Sessions

[fillipe-revido]

Problem Description

When using persistent browser sessions, cookies are not persisting between workflow runs. After executing a login workflow successfully, subsequent workflows using the same browser_session_id are not reusing the cookies, resulting in "NOT_LOGGED_IN" errors.

Expected Behavior

When a persistent browser session is created and a login workflow is executed:

1. Cookies should be saved to a persistent user_data_dir (e.g., /app/skyvern/persistent_browser_sessions/{org_id}/{session_id})
2. When a subsequent workflow uses the same browser_session_id, it should:
   • Connect to the existing browser via CDP using the stored browser_address
   • Use the same persistent user_data_dir to load cookies
   • Preserve authentication state between workflows

Actual Behavior

1. Session Creation: The persistent session is created correctly with the right user_data_dir:

   [persistent_sessions_manager.py:453] Using persistent user_data_dir for session
   browser_session_id=pbs_457827170959729696
   user_data_dir=/app/skyvern/persistent_browser_sessions/o_454165536444531154/pbs_457827170959729696
   browser_address=None
   

2. Browser Address Persisted: The browser address is correctly stored in the database:

   [persistent_sessions_manager.py:376] Persisted browser address for session
   browser_session_id=pbs_457827170959729696
   browser_address=http://127.0.0.1:9222
   

3. Workflow Execution Issue: When a workflow run is created for this session, it's creating a new browser state with a temporary user_data_dir instead of using the persistent one:

   [browser_manager.py:215] Creating browser state for workflow run
   workflow_run_id=wr_457827265449010210
   
   [browser_factory.py:557] Using temporary user_data_dir
   user_data_dir=./temp/skyvern_browser_3gaxhz4_
   

Root Cause Analysis

The issue appears to be in browser_manager.py -> get_or_create_for_workflow_run():

1. When get_browser_state() from PersistentSessionsManager returns None (browser state not in memory)
2. The code creates a new browser state via _create_browser_state()
3. At this point, workflow_run.browser_address is None, so it doesn't retrieve the browser_address from the database
4. Without browser_address, the code falls back to creating a new browser instead of connecting via CDP
5. Without explicit browser_session_id propagation in some code paths, user_data_dir is not calculated correctly, resulting in a temporary directory

Attempted Fixes

We've tried several approaches:

1. Modified _create_headless_chromium and _create_headful_chromium to calculate user_data_dir from browser_session_id when not explicitly provided
2. Added logic to retrieve browser_address from database when browser_session_id is present but workflow_run.browser_address is None
3. Added retry logic in _connect_to_cdp_browser to find existing contexts

However, we're still seeing the "Using temporary user_data_dir" log, indicating the issue persists.

Relevant Logs

Session Creation (Correct):

[persistent_sessions_manager.py:453] Using persistent user_data_dir for session
browser_session_id=pbs_457827170959729696
user_data_dir=/app/skyvern/persistent_browser_sessions/o_454165536444531154/pbs_457827170959729696

[browser_factory.py:536] Using provided user_data_dir for persistent session
user_data_dir=/app/skyvern/persistent_browser_sessions/o_454165536444531154/pbs_457827170959729696

[persistent_sessions_manager.py:376] Persisted browser address for session
browser_session_id=pbs_457827170959729696
browser_address=http://127.0.0.1:9222

Workflow Execution (Problem):

[browser_manager.py:215] Creating browser state for workflow run
workflow_run_id=wr_457827265449010210

[browser_factory.py:557] Using temporary user_data_dir
user_data_dir=./temp/skyvern_browser_3gaxhz4_

Questions

1. Has anyone else encountered this cookie persistence issue with persistent browser sessions?
2. Is there a recommended pattern for ensuring browser_session_id is correctly propagated throughout the browser creation chain?
3. Should we always retrieve browser_address from the database when browser_session_id is present, even if workflow_run.browser_address exists?

Environment

• Skyvern version: Latest (Docker deployment)
• Python version: As per Docker image
• Playwright version: As per Skyvern dependencies

We're willing to pay someone to help us hourly

[coderabbitai]

A summary of the changes CodeRabbit can apply:

• Add database lookup logic to skyvern/webeye/browser_manager.py (in get_or_create_for_task and get_or_create_for_workflow_run) to retrieve and use the persistent session's browser_address when browser_session_id is provided but browser state/address is missing, ensuring the process connects to the existing CDP browser and preserves cookies across workflow runs.

• Update skyvern/webeye/browser_manager.py to, when an in-memory browser_state is missing and a browser_session_id exists, attempt to retrieve browser_address from the persistent browser session in the database (logging success, warnings, or errors), use that address in logs and pass it into _create_browser_state instead of the original workflow/task browser_address.

• ✅ Create PR with these edits
• 📋 Get copyable edits

[fillipe-revido]

Fix: Cookie Persistence Between Workflows in Persistent Browser Sessions

Problem

Cookies were not persisting between workflows when using the same persistent browser session. When a second workflow ran after a login workflow, it would fail with "NOT_LOGGED_IN" errors because cookies were lost.

Root Cause

The issue had multiple contributing factors:

1. Missing browser_session_id propagation: The browser_session_id from workflow_run.browser_session_id was not being used as a fallback when not explicitly passed to get_or_create_for_workflow_run(), causing the system to create a new browser state instead of reusing the existing one.

2. Temporary user_data_dir usage: When creating a new browser state for a workflow run, the system would use a temporary user_data_dir instead of the persistent one associated with the session, causing cookies to be stored in a temporary location that was discarded after workflow completion.

3. Missing browser_address retrieval: When browser_address was not present in the workflow_run object, the system didn't retrieve it from the persistent session database, preventing reconnection to the existing browser instance.

Solution

1. Browser Session ID Fallback (browser_manager.py)

Added fallback logic in get_or_create_for_workflow_run() to use workflow_run.browser_session_id when browser_session_id parameter is None.

2. Browser Address Retrieval (browser_manager.py)

Added logic to retrieve browser_address from the persistent session database when it's missing.

3. Persistent user_data_dir Usage (Multiple Files)

Ensured that when a browser_session_id is present, the system uses a consistent persistent user_data_dir based on organization_id and session_id.

Files Modified

1. skyvern/webeye/browser_manager.py
2. skyvern/webeye/browser_factory.py
3. skyvern/webeye/persistent_sessions_manager.py (referenced for context)

Verification

• Login workflow completes and saves cookies to persistent user_data_dir
• Subsequent workflows in the same session reuse the existing browser state
• Cookies persist between workflows (no more "NOT_LOGGED_IN" errors)
• Browser state is properly shared across all contexts in the same session

Testing

Tested with a login workflow followed by a "spare part search" workflow using the same persistent session. Both workflows completed successfully, confirming cookie persistence.