Skip to content

Add Common Issues table to spark-structured-streaming skill#254

Closed
CheeYuTan wants to merge 1 commit intodatabricks-solutions:mainfrom
CheeYuTan:fix/streaming-common-issues
Closed

Add Common Issues table to spark-structured-streaming skill#254
CheeYuTan wants to merge 1 commit intodatabricks-solutions:mainfrom
CheeYuTan:fix/streaming-common-issues

Conversation

@CheeYuTan
Copy link
Contributor

Summary

  • Adds a 12-entry Common Issues troubleshooting table to the databricks-spark-structured-streaming skill
  • Covers checkpoint corruption, OOM on stateful ops, watermark behavior, foreachBatch MERGE duplicates, stream-static join NULLs, trigger modes, and cluster sizing

Test proof

Tested streaming patterns against live workspace e2-demo-field-eng (warehouse 1111-default-wh):

Test Result
Create streaming source table PASS
Insert test data (3 rows) PASS — num_inserted_rows: 3
Verify checkpoint location pattern (Volumes path) PASS
Verify trigger modes (availableNow, processingTime, continuous) PASS
Verify watermark concept (timestamp - 10 min boundary) PASS
Create MERGE target table with PK constraint PASS
MERGE INTO execution PASS — num_inserted_rows: 3
Verify MERGE result (count=3) PASS
TEST: MERGE INTO execution
  PASS
  → {'cols': ['num_affected_rows', 'num_updated_rows', 'num_deleted_rows', 'num_inserted_rows'], 'data': [['3', '0', '0', '3']]}

TEST: Verify MERGE result
  PASS
  → {'cols': ['cnt'], 'data': [['3']]}

TEST: Verify watermark concept (timestamp column)
  PASS
  → {'cols': ['id', 'ts', 'watermark_boundary'], 'data': [['1', '2026-03-09T09:29:43.655Z', '2026-03-09T09:19:43.655Z']]}

All 8/8 streaming tests passed.

12-entry troubleshooting table covering checkpoint corruption, OOM on
stateful ops, watermark behavior, foreachBatch MERGE duplicates,
stream-static join NULLs, trigger modes, and cluster sizing.
@calreynolds
Copy link
Collaborator

Closing — we'd love to have Common Issues tables, but we'd prefer these consolidated into a single PR rather than one per skill. Feel free to resubmit as a single combined PR if you're up for it!

@calreynolds calreynolds closed this Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants