Skip to content

New stream: Agentic Data Bootkon#2

Merged
fhirschmann merged 32 commits into
GoogleCloudPlatform:mainfrom
fhirschmann:agenticdata-stream
Jul 3, 2026
Merged

New stream: Agentic Data Bootkon#2
fhirschmann merged 32 commits into
GoogleCloudPlatform:mainfrom
fhirschmann:agenticdata-stream

Conversation

@fhirschmann

@fhirschmann fhirschmann commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a new stream under content/agenticdata: a 4-hour hands-on event (13:00–17:00) in which participants build an end-to-end agentic data platform for the fictional Cymbal commerce company, with the Antigravity CLI (agy) acting as an AI co-engineer throughout.

Pipeline built during the event:

Cloud SQL for PostgreSQL (PSC-only, no public IPs) → Datastream CDC → BigQuery cymbal_bronze → Dataform CLI (agy-authored SQLX) → cymbal_silver/cymbal_gold → Knowledge Catalog governance (tier aspects, auto data quality, PII policy tags, glossary, lineage) → BigQuery conversational data agent on gold → ADK multi-agent finale where a concierge agent delegates analytics over the A2A protocol and answers live operational lookups straight from Postgres.

What's included

  • Six labs + TUTORIAL.md + README.md, following the existing stream conventions (walkthrough directives, spotlights, per-lab duration/difficulty, challenge sections)
  • bk-bootstrap (APIs, IAM, DQ service account, generated secrets)
  • Fully synthetic, deterministic Cymbal orders dataset generated in the participant's own project (~2.3M rows, deliberately planted data-quality flaws) plus a live change simulator to keep CDC visibly flowing
  • Dataform starter project + complete reference models with assertions
  • ADK reference agents: an A2A-exposed analyst wrapping the BigQuery data agent (Conversational Analytics API) and a concierge with a live Cloud SQL tool
  • option1-vpc-peering-proxy.diff: a git-applyable fallback switching Labs 1–2 from PSC to VPC-peering + reverse-proxy connectivity

Design principles

  • Shell operates, agy authors, console verifies — participants type the infrastructure commands, agy writes SQLX/configs/agents (every artifact has a reference fallback in src/), and each lab verifies results in the console via spotlights
  • Long-running builds (Cloud SQL, Datastream private connection) are kicked off async in Lab 1 so no lab stalls on provisioning
  • Everything runs in a single fresh sandbox project from Cloud Shell; no external dataset downloads or pre-provisioned resources

Status

Draft until a full smoke test on a sandbox account:

  • End-to-end PSC path (instance → endpoint → Datastream private connection → stream) — verified live: stream RUNNING, CDC latency ~1–2 min
  • dataFreshness: "0s" accepted; change-to-queryable latency ~85 s
  • Dependency pins verified and fixed (psycopg 3.2.13, a2a-sdk http-server extras)
  • Dataform CLI 3.0.61 + ADC auth via .df-credentials.json — compile and runs green
  • Spotlight locators clicked through in the live console (esp. Lab 4)
  • Add the stream row to the repository root README.md

…sanitizer-safe wait command, drop redundant . bk
…outing rule

- dataform AGENTS.md: enumerate the exact valid status set (agy guessed
  'placed' and missed 'paid' without it, failing the assertion on real data)
- adk AGENTS.md: pin exact imports, the singular 'instruction' kwarg, and
  the full Conversational Analytics chat request shape (API is newer than
  model training data; agy hallucinated DataAgentContext/stream_chat forms)
- Lab 6 prompt (+ prompts.md): action-explicit wording -- 'create the files
  ... exactly as it describes' (print-mode agy plans without acting on the
  softer 'implement' phrasing)
- concierge: routing rule 3 for questions mixing both planes (tool first,
  then transfer) -- verified the combined demo question now answers both halves
- data-tier aspect type must be created in location 'global', not
  'us-central1': a regional aspect type does not appear in the attach
  dialog of the BigQuery dataset entries (they live in multi-region 'us'),
  so bronze/silver/gold tiers could never be stamped. Verified live.
- spell out the enum-value and optional-aspect UI steps that the console
  actually shows.
- bk-bootstrap: wait for the freshly created data-quality service account
  to become visible to IAM before granting roles (the first binding
  otherwise fails with 'service account does not exist').
@fhirschmann fhirschmann marked this pull request as ready for review July 3, 2026 14:21
@fhirschmann fhirschmann merged commit 55e341c into GoogleCloudPlatform:main Jul 3, 2026
7 checks passed
fhirschmann added a commit that referenced this pull request Jul 10, 2026
…gnal

Review item #25b (P2). Since the no-project fail-fast landed, BK_INITIALIZED
was set in the same validated branch as PROJECT_NUMBER, making the two
signals exactly equivalent -- a second flag carrying the same bit, and one
more line in the participant-visible vars.local.sh.

Consumers now guard on the precondition they actually consume:
  * stream bootstraps: [[ -z "$PROJECT_NUMBER" ]] -- strictly better, since
    the service-agent emails they create are built from $PROJECT_NUMBER;
    this also closes item #2's belt-and-braces gap (a bootstrap running
    with an empty project number) for free. data's pip install now runs
    after the guard instead of before it.
  * bk-start: same check, same message.
  * bk's first-run-vs-reload detection captures ${PROJECT_NUMBER:-} (present
    on reload via the ~/.bashrc source, absent on a first or failed run --
    so a retry after failed validation shows the full welcome banner again).
  * bk-info drops the redundant display line; bk-deactivate's hint says
    'unset PROJECT_NUMBER'.

Older vars.local.sh files are migrated opportunistically: the stale
BK_INITIALIZED line (and its comment) is deleted; nothing reads it anymore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant