Consolidate Cloud Shell env into a single vars.local.sh#3
Merged
Conversation
- abort with an example when BK_STREAM is unset instead of continuing - abort with the list of available streams when content/$BK_STREAM does not exist - persist BK_DIR, vars.sh/vars.local.sh sourcing, PROJECT_NUMBER and BK_INITIALIZED in ~/.bashrc so that new terminals no longer require re-running '. bk'; BK_INITIALIZED/PROJECT_NUMBER are only persisted once a valid PROJECT_ID is configured because the stream bootstrap scripts guard on them
Move all per-participant state into a single git-ignored vars.local.sh instead of scattering per-variable exports across ~/.bashrc, and reduce ~/.bashrc to one guarded block that sources it. Autodetect PROJECT_ID and GCP_USERNAME from the Cloud Shell environment (filling only empty values), so most participants no longer edit any config by hand. `. bk` is dual-mode: full bootstrap on first run, fast reload afterwards. - add bk-set-var: idempotent export upserter, atomic (same-dir temp + mv) so an interrupted write can't truncate the file's generated secrets - add bk-remove-block: robust ~/.bashrc block remover -- CRLF-safe, keeps an unterminated block instead of deleting to EOF, strips trailing blank lines - fixed clone path (~/bootkon); derive the repo identity from the checkout's git origin so rendered image links always match the files on disk - persist seed vars and install the ~/.bashrc block before validating the project, and mark the shell initialized unconditionally, so a failed or empty project still leaves new terminals working and still opens the tutorial - surface expired-auth as a likely cause of a failed project check - move agenticdata stream config (secrets + runtime env) into vars.local.sh - chmod 600 vars.local.sh (it holds generated database passwords) - update tutorials and contributor docs for the new flow
fhirschmann
changed the title
- agenticdata lab 1 / bk-bootstrap / simulate.py now reload with `. bk` instead of `source ~/.bashrc`, consistent with the tutorial intro and the dual-mode bk reload - contributing: add "Test a branch end-to-end" — the branch must go in the wget URL (the stream READMEs hard-code main), and ~/bootkon is reused if it already exists; BK_BRANCH and image links follow the checkout
…ine bootstrap
- bk-set-var gains --comment, written as a `# text` line above the export on
first append, so a fresh vars.local.sh is self-documenting; not duplicated on
reload, and config comments from vars.sh are preserved on in-place replace
- bk / bk-bootstrap pass a one-line description for every managed and stream var
- bk fills MY_NAME from the launch command (captured before sourcing; fill-if-
empty so a later manual edit wins), so no vars.local.sh edit is needed
- READMEs: two-line bootstrap (MY_NAME on line 1, fixed wget on line 2); this
also fixes the agents one-liner (was fhirschmann fork + empty ${BK_BRANCH})
- TUTORIAL intros: note MY_NAME comes from the launch command
…OOGLE_CLOUD_PROJECT Cloud Shell already exports GOOGLE_CLOUD_PROJECT (= the active project = PROJECT_ID, which bk autodetects from it), so persisting a duplicate in vars.local.sh was redundant. bootkon code now uses PROJECT_ID; the Vertex SDK and agy keep reading GOOGLE_CLOUD_PROJECT straight from Cloud Shell. - bk-bootstrap: drop the GOOGLE_CLOUD_PROJECT write (LOCATION + USE_VERTEXAI stay) - ca_tool.py: read PROJECT_ID instead of GOOGLE_CLOUD_PROJECT - docs (Lab 6, ADK AGENTS.md, agent docstring): PROJECT_ID is the canonical var
PROJECT_ID is no longer autodetected-and-persisted. bk derives it live from
${DEVSHELL_PROJECT_ID:-$GOOGLE_CLOUD_PROJECT} (Cloud Shell refreshes it every
prompt and keeps gcloud pointed at it), so it cannot go stale and there is
nothing to edit -- to use another project, switch it in the Cloud Shell picker.
The ~/.bashrc block exports it live in every new terminal.
PROJECT_NUMBER is the only project value not in the environment, so it is cached
in vars.local.sh keyed by project (BK_PROJECT_NUMBER_OF): computed once, reused
on same-project reloads with zero network calls, auto-refreshed on a project
switch. An empty project number is now a hard error, and the redundant
`gcloud config set project` is dropped.
- vars.sh: drop PROJECT_ID (now live)
- TUTORIAL intros (data, agenticdata, devex): a wrong PROJECT_ID is fixed via the
Cloud Shell project picker, not by editing vars.local.sh
Every lab needs a project and PROJECT_ID is derived live from the Cloud Shell picker, so bk now checks it up front: if no project is selected (DEVSHELL_PROJECT_ID/GOOGLE_CLOUD_PROJECT empty) it errors and returns BEFORE cloning or opening the editor, instead of setting up a project-less dead end and launching the tutorial anyway. Because the tutorial-without-a-project path is gone, BK_INITIALIZED is now exported only AFTER the project validates -- fixing the regression where the stream-bootstrap guard was defeated on a failed/aborted setup (bk-bootstrap would run with an empty PROJECT_NUMBER and silently skip the Datastream service-agent grant).
…first render
`. bk` now runs content/$BK_STREAM/bk-init (if present and executable) on every
initialization and reload -- after project validation and BEFORE the first
tutorial render -- then re-sources vars.local.sh. Hook contract: idempotent
(write via bk-set-var --if-empty), local-only and instant; heavy, visible GCP
work stays in the participant-run bk-bootstrap. Streams without a hook are
unaffected; a failing hook aborts with an actionable message.
agenticdata uses the hook to generate BK_DB_PASSWORD/BK_DS_PASSWORD and write
the deterministic ADK/agy runtime config. This closes the empty-password render
window: {{ BK_DB_PASSWORD }} in Lab 2 renders correctly from the very first
render, so Lab 1 no longer needs the "run '. bk', then 'bk-start' to load the
passwords" reload step -- bk-bootstrap keeps only APIs, IAM, service accounts,
pip and the agy config.
Review item #1 (P0). `wget -qO-` swallows every error (404, DNS, proxy, GitHub outage) and `. <(...)` happily sources the empty stream with exit 0 -- a participant pasting the launch command saw literally nothing happen. The fallback echo inside the process substitution prints a clear error with a recovery instruction instead. Identical in all five stream READMEs.
Review item #12 (P1). `cd $BK_DIR` was unquoted and unchecked: with a deleted checkout it printed the raw shell error, carried on in $HOME, tracebacked in the renderer, opened a stale tutorial and then opened $HOME as the workspace -- three misleading symptoms for one cause. The initialization guard now runs first (so an unloaded environment gets the '. bk' hint instead of a confusing cd error), and a missing folder stops immediately with a reinstall instruction.
…'s cwd Review item #11 (P1) plus the P3 double-launch note. Two confirmed failure modes: (a) a bare `bk-tutorial` with no memory file wrote an empty line into .tutorial.memory BEFORE rendering, dumped a raw Python traceback and still launched the stale /tmp/bootkon.md -- and every later bare invocation reproduced the traceback; (b) the memory stores a checkout-relative path and the renderer's include searchpath is a relative 'content', so invoking it from a lab directory (src/dataform, src/adk) tracebacked as well. Now: cd "$BK_DIR" first (with a clear message when the env is not loaded or the checkout is gone), refuse cleanly when there is nothing to open, write the memory only after a successful render, and quote all paths. The duplicated launch-tutorial call is now documented as a deliberate workaround.
…rets
Review item #8 renderer half (P1) and the residual belt-and-braces from
item #18. os.getenv() returns Python None for unset variables, and Jinja
rendered {{ PROJECT_NUMBER }} as the literal string 'None' inside copy-paste
gcloud commands (service-None@gcp-sa-dataplex...) -- unlike undefined
variables, which render empty. All scalar context values are now coerced to
''.
Additionally, any BK_*PASSWORD/SECRET/TOKEN/KEY variable referenced by the
template that is empty at render time now produces a loud stderr warning
with the recovery command, instead of silently baking blank credentials
into the labs.
Review items #26 and #27 (P3), plus the defensive chmod from item #4. Three edge cases: (a) a --comment containing a newline would write its second line outside the '# ' prefix -- executed by every terminal that sources vars.local.sh -- and (b) '--comment' as the last argument died on set -u with a cryptic message; both are now rejected with a clear error. (c) Appending to a hand-edited file without a trailing newline concatenated two exports onto one line; a newline is now ensured first. Additionally, when bk-set-var itself creates the target file it now chmods it to 600 immediately: the file may receive secrets later, and the default umask would leave it world-readable.
Review item #28 (P3). The atomic temp-file + mv rewrite replaced a symlinked ~/.bashrc with a regular file, silently detaching dotfile-managed setups from their repository. Resolve the target with readlink -f first so the rewrite lands in the real file and the symlink stays intact.
…gnal
Review item #25b (P2). Since the no-project fail-fast landed, BK_INITIALIZED
was set in the same validated branch as PROJECT_NUMBER, making the two
signals exactly equivalent -- a second flag carrying the same bit, and one
more line in the participant-visible vars.local.sh.
Consumers now guard on the precondition they actually consume:
* stream bootstraps: [[ -z "$PROJECT_NUMBER" ]] -- strictly better, since
the service-agent emails they create are built from $PROJECT_NUMBER;
this also closes item #2's belt-and-braces gap (a bootstrap running
with an empty project number) for free. data's pip install now runs
after the guard instead of before it.
* bk-start: same check, same message.
* bk's first-run-vs-reload detection captures ${PROJECT_NUMBER:-} (present
on reload via the ~/.bashrc source, absent on a first or failed run --
so a retry after failed validation shows the full welcome banner again).
* bk-info drops the redundant display line; bk-deactivate's hint says
'unset PROJECT_NUMBER'.
Older vars.local.sh files are migrated opportunistically: the stale
BK_INITIALIZED line (and its comment) is deleted; nothing reads it anymore.
Review item #24 (P2). The block was unreachable behind the CLOUD_SHELL guard (Cloud Shell ships git) -- and had it ever run, it would have greeted a first-day participant with a sudo password prompt and 30 seconds of apt noise. A missing git now produces one clear error pointing back to a fresh Cloud Shell.
…rminal Review items #3 (P0), #13 and #31 (P1/P3). The worst confirmed dead-end: rejecting the 'Authorize Cloud Shell' popup fails validation, the auto bk-start never runs, and no message anywhere mentioned bk-start -- the participant ended up authorized, environment loaded, staring at a bare prompt. Fixes: * Validation failure now disambiguates auth-vs-wrong-project itself (via gcloud auth print-access-token) and prints exactly one instruction, always ending with 'type bk-start to open the tutorial'. * The reload message gains '-- tutorial closed? Type bk-start.' * The stream-typo error lists only real streams (directories, minus common/contributing/example -- it previously offered 'contributing' and even CONCEPT.md) and says how to recover: fix BK_STREAM, paste again. * 'Script is not sourced. Please source it.' and 'Please run this script in Cloud Shell.' are reworded without jargon.
…ckout Review items #19b, #4 (P0) and #16 (P1), all confirmed with repros. * Pre-clone probe: one HEAD request against the raw content/$BK_STREAM/TUTORIAL.md validates BK_REPO, BK_BRANCH and BK_STREAM at once, in about a second, BEFORE any clone happens or any workspace opens. A typo now fails with one clear message and leaves nothing behind (works for forks too -- the probe uses the launch-provided BK_REPO). The post-clone stream check stays as belt and braces. * The clone itself is checked; a failure explains the recovery instead of letting the sourced script keep running without set -e. * The reuse guard now requires .git AND vars.sh: a clone that died between the two (disk full, closed tab) used to pass the old .git-only test, print a reassuring green message, and then cascade -- cp failed, sources failed, vars.local.sh got recreated world-readable via touch, and bk-start opened a blank tutorial. Such a directory is now rejected with an explicit rm -rf recovery line. * Re-pasting the launch command (the documented way to pick up mid-event content fixes) now fast-forwards a clean existing checkout quietly; plain '. bk' reloads stay offline. Local changes are left alone with a yellow note.
Review item #6 (P1, reproduced). The tutorial explicitly sends beginners
into vars.local.sh; a single lost quote meant every new terminal printed a
raw 'unexpected EOF' bash error and silently stopped loading everything
below the broken line (stream, passwords) -- with every follow-up error
('not initialized', 'BK_STREAM is not set') pointing away from the cause.
bk itself happily re-sourced the broken file.
Both consumers now run bash -n first: bk fails fast quoting the offending
line and saying exactly what to do; the ~/.bashrc block warns once per
terminal with the same instruction instead of spraying parser errors.
Review item #9 (P1). After a project switch in the picker, the current shell's ambient GOOGLE_CLOUD_PROJECT still names the old project until the next prompt refresh -- and agy plus the google-genai/Vertex SDKs read exactly that variable (the agenticdata stream deliberately no longer pins it). Re-export it to the freshly validated PROJECT_ID when the two differ, so SDK calls made in this very session land in the right project.
Review items #25 and #29 (P2). BK_GITHUB_USERNAME, BK_GITHUB_REPOSITORY and BK_REPO_URL have no consumer outside the current session (verified by full-repo grep): bk recomputes them from BK_REPO on every run and the persisted copies were never read back -- they only buried the two lines in vars.local.sh participants actually edit. The in-session exports stay (they cost nothing). Old files are migrated: the stale lines are dropped, which also prevents a stale BK_REPO_URL from clobbering the freshly derived value when vars.local.sh is re-sourced after the stream init hook. Fork safety re-checked in the review: forks rely on BK_REPO + BK_BRANCH (derived from the checkout's git origin), not on these three -- rendered image links keep following the fork automatically. bk-info: shows BK_REPO plus the derived clone URL instead of three redundant lines, prints an explicit 'run . bk' hint when the environment is not loaded (instead of a wall of blank values), and the bundled-dataset check is now gated to the data stream with neutral wording -- healthy agenticdata/devex/agents setups no longer see a red 'does not exist'.
Review item #14 (P1, confirmed). Because bk is sourced, its mid-script cd changed the participant's shell -- on every reload and every error path. Labs put people into src/dataform or src/adk, and the next relative lab command after a '. bk' reload failed with 'No such file or directory', with nothing connecting the failure to the reload. The script now works from absolute paths throughout and only a successful FIRST run ends with a cd to the repo root (where the labs' relative commands are meant to start). Reloads and all failure paths leave the current directory untouched.
…ne reload Review item #23 (P2). A first run used to open with the readlink'ed /proc/self/fd/... pipe path (literally the first terminal output a novice ever sees), two 'Sourcing /home/...' lines, a PATH notice and duplicated project/config lines; the closing banner said 'type bk-start' although on the wget path bk-start auto-runs one second later -- many participants dutifully typed it again. Internal narration now sits behind a BK_DEBUG gate (dbg helper). A first run prints one merged summary line (project · account · stream) plus the welcome banner, whose call-to-action is now path-aware: 'Opening the tutorial now -- if it ever closes, type bk-start' on the wget path. A reload prints exactly one line, ending with the bk-start hint. The existing-checkout notice only appears when re-pasting the launch command, not on every reload.
… name Review item *20* (P2 headline; supersedes #20, absorbs #33/#49). Editing MY_NAME inside a pasted quoted string was the single most error-prone moment of the launch: deleting a quote left a novice at a bare dquote> prompt with both lines swallowed, and retyping the value unquoted ran their name as a command. That edit no longer exists. On the very first launch (no vars.local.sh yet) the ASCII robot now greets UP FRONT, confirms the project it is about to use (with a pointer to the picker if it is the wrong one), and asks 'What should I call you?' via read. This works on the wget launch because '. <(wget ...)' sources through a process substitution and stdin stays on the tty -- unlike wget | bash. Guards: prompt only on a true first run AND an interactive stdin AND an empty MY_NAME; MY_NAME=... in the launch env still short-circuits it, so organizer/scripted/headless flows behave as before. The answer is persisted via the existing BK_REQ_MY_NAME capture (manual edits keep winning), and is sanitized either way -- newlines would corrupt vars.local.sh, quotes would look broken in the rendered greeting. The first run now closes with a compact 'All set, <name>!' line that is path-aware (the wget path opens the tutorial itself; only the manual path says to type bk-start). The MY_NAME line is gone from all five READMEs -- the launch is a single, edit-free paste again.
Review item #17 (P1, confirmed). The launch streams this file through '. <(wget -qO- ...)': with a plain script body, a connection drop mid- download executed everything up to the break. The worst window was the ~/.bashrc heredoc -- bash's at-EOF heredoc behavior would append a bootkon block WITHOUT its end marker, which bk-remove-block then (by design, data-loss safety) preserves forever, so blocks accumulate on every re-run and bk-deactivate refuses to clean up. The entire body now lives in _bk_main, invoked once at the very bottom: a truncated download is a parse error that executes NOTHING (verified in a sandbox: cut at 60% -> 'syntax error: unexpected end of file', zero side effects). Internal working variables became `local`, so early error returns no longer leak a dozen internals into the participant's shell; the helper functions are unset right after the call. The body is deliberately not re-indented to keep the diff and future history readable. Also verified in the sandbox: first run end-to-end (robot, name capture, config seeding, bashrc block, summary), 1-line reload that preserves the caller's cwd, broken-vars.local.sh error path, and status propagation for both the sourced and the executed case.
Review items #5 (P0, five confirmed issues) and #30 (P3).
* (a) No cd, no error tracking: invoked from a fresh tab (cwd=$HOME) the
relative pip path failed, the error scrolled away under 25 IAM lines,
and the script still ended with 'Bootstrap complete.' It now pins the
cwd to $BK_DIR, checks every step, keeps going past individual
failures, and only claims success when nothing failed -- otherwise:
'Bootstrap finished with N error(s)', pointing up and at a safe re-run.
* (b) The critical Datastream service-agent block -- the very reason this
script exists -- was fully silenced (>/dev/null 2>&1, unchecked); its
failure resurfaced an hour later as the exact Lab-1 PERMISSION_DENIED
it was added to prevent. stderr stays visible and each of the three
calls is checked.
* (c) Both IAM role loops now pass --condition=None (the service-agent
grant already did): any conditional binding in the sandbox project made
gcloud stop and prompt interactively mid-bootstrap, up to 18 times,
with the prompt half-hidden by the stdout redirect.
* (d) Service-account creation is idempotent (describe-before-create):
re-running the bootstrap is the natural novice recovery and no longer
prints a scary unexplained 'already exists' error.
* (e) The guard now requires the exact precondition the script consumes
(non-empty PROJECT_NUMBER; landed with the BK_INITIALIZED removal).
* #30: agy trustedWorkspaces interpolates $BK_DIR instead of hardcoding
$HOME/bootkon.
… fallback Review items #21, #10 and the tutorial half of #22 (P1/P2). On the wget launch path, the intro's '. bk' and 'bk-start' steps were pure no-ops (both had just run), costing two commands, ~10 lines of output, and a tutorial that closed and reopened itself while being read. The intro now leads with verification -- 'your PROJECT_ID is X, your GCP_USERNAME is Y; both correct? press START' -- and moves the commands into the only-if-something-is-wrong branch. That branch also fixes the confirmed never-refreshes loop (#10): it says '. bk && bk-start' explicitly, because a plain '. bk' reload never re-renders the pane, so the sidebar kept showing None and beginners re-edited a value that was already correct. A one-line tip covers reopening a closed tutorial. Same change in all four streams.
Review item #25's follow-up note. Forking is supported by design (BK_REPO and the rendered image links derive from the checkout's git origin), but three caveats were undocumented: the launch line must name the fork, the fork must be public for the tutorial images to load, and a few hardcoded upstream references (bk-legacy-download's dataset repo) do not follow the fork.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Reworks how the Cloud Shell environment is loaded so that new terminal tabs
just work and participants barely have to configure anything — and hardens the
whole participant-facing launch/setup flow so that every failure mode a
first-time terminal user can realistically hit ends in one clear instruction
instead of a dead end.
Previously
bkscattered per-variableexportlines across~/.bashrcandasked participants to hand-edit
vars.sh. This consolidates all per-participantstate into a single git-ignored
vars.local.sh, reduces~/.bashrcto oneguarded block that sources it, and makes the launch a single, edit-free paste.
Environment model
bkcreatesvars.local.shfromvars.shon firstrun and keeps config, seed vars, and generated secrets there (
chmod 600).~/.bashrcgets one guarded block (# >>> bootkon >>>) that setsBK_DIR,adds
.scriptstoPATH, and sourcesvars.local.sh— every new terminalhas the environment without re-running
. bk.PROJECT_IDis derived live from the CloudShell project picker (never persisted, can't go stale);
PROJECT_NUMBERiscached per project;
GCP_USERNAMEis autodetected (manual edits win). If noproject is selected,
bkfails fast before cloning or opening anything.. bkis dual-mode. Full bootstrap on first run; a fast, quiet,one-line reload afterwards that never changes the caller's working
directory. A non-empty
PROJECT_NUMBERis the "environment is ready"signal (there is no separate initialized flag); it is set only after the
project validated, so stream bootstraps can never run against a half
set-up project.
content/<stream>/bk-init(optional, idempotent,local-only) runs after validation and before the first render — agenticdata's
generated secrets exist from the very first tutorial render.
bk-set-var(idempotent, atomic export upserter) andbk-remove-block(robust~/.bashrcblock remover) replace the oldgrep/sed/echo churn.
Launch & onboarding UX
MY_NAME=""edit is gone from all READMEs; onfirst launch the ASCII robot greets up front, confirms the target project,
and asks for the name interactively (
MY_NAME=...in the env stillshort-circuits the prompt for scripted flows; non-tty runs skip it).
BK_DEBUG; a first runprints one summary line (project · account · stream), a reload exactly one
line. Messages are path-aware (no "type bk-start" when it auto-runs).
displayed values, press START.
. bk && bk-startmoved into theonly-if-something-is-wrong branch (a plain
. bknever re-renders the pane,which used to trap people in an edit/reload loop).
Failure hardening
wgetin the launch line now prints an error instead of sourcingan empty stream with exit 0 (literally nothing used to happen).
bkbody is wrapped in a function: a connection dropmid-download is a parse error that executes nothing (previously it could
half-run, worst case appending an unterminated
~/.bashrcblock).cloning; the clone itself is checked; an interrupted clone is detected and
rejected with an explicit recovery command instead of being silently reused.
content fixes actually reach participants).
vars.local.shis syntax-checked before every source (inbkand in the~/.bashrcblock): one lost quote no longer poisons every new terminalwith a raw bash error and half-loaded environment.
end with how to reopen the tutorial; the stream-typo error lists only real
streams and the exact fix.
bk-start/bk-tutorialguard missing checkouts and stale reload memory;the renderer renders unset values as empty (not the literal
Noneinsidecopy-paste commands) and warns loudly if a referenced secret is empty.
bk-bootstrapno longer fails silently: every step ischecked and counted, the critical Datastream service-agent block is loud,
IAM loops pass
--condition=None(no interactive prompts mid-bootstrap),service-account creation is idempotent, and success is only claimed when
nothing failed.
Notes
vars.local.shfiles are migrated automatically (stale lines dropped).the intro sections of the tutorials.
must name the fork, and it must be public for tutorial images).