feat: improve testing framework and add CI

.gitattributes

@@ -11,3 +11,7 @@ ell.sh      text eol=lf
 
 # Templates and configuration examples are line-oriented text; keep them LF.
 *.json      text eol=lf
+
+# CI workflow files are YAML; keep them LF for consistent parsing.
+*.yml       text eol=lf
+*.yaml      text eol=lf

.github/workflows/ci.yml

@@ -0,0 +1,59 @@
+name: CI
+
+on:
+  push:
+    branches: [ main, master ]
+  pull_request:
+
+# Cancel superseded runs on the same ref to save CI minutes.
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  # Static analysis: shellcheck every shell script. The `error` severity gate
+  # is a hard failure (the repo is clean at this level today). Warnings are
+  # reported separately and do not fail the build, so they can be cleaned up
+  # incrementally without blocking merges.
+  shellcheck:
+    name: ShellCheck
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install shellcheck
+        run: sudo apt-get update && sudo apt-get install -y shellcheck
+
+      - name: Collect shell scripts
+        id: collect
+        run: |
+          # ell and the launcher have no .sh extension; include them explicitly.
+          files="$(git ls-files '*.sh' 'ell' | tr '\n' ' ')"
+          echo "files=${files}" >> "${GITHUB_OUTPUT}"
+          echo "Scripts to check: ${files}"
+
+      - name: ShellCheck (errors, blocking)
+        run: shellcheck -S error ${{ steps.collect.outputs.files }}
+
+      - name: ShellCheck (warnings, non-blocking)
+        continue-on-error: true
+        run: shellcheck -S warning ${{ steps.collect.outputs.files }}
+
+  # Run the full test suite across the oldest and current supported bash
+  # versions using the project's own containerised entry point.
+  tests:
+    name: Tests (bash ${{ matrix.bash }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        bash: [ "4.1", "5.2" ]
+    container:
+      image: bash:${{ matrix.bash }}
+    steps:
+      - uses: actions/checkout@v4
+
+      # entry.sh derives all paths from its own location and best-effort installs
+      # curl and a global launcher, so no extra setup is needed here.
+      - name: Run test suite
+        run: bash tests/entry.sh

README.md

@@ -199,20 +199,82 @@ See [Risks Consideration](docs/Risk_Consideration.md).
 
 ## Testing
 
-The test suite is meant to be run inside Docker. The tests hard-code the
-container path `/ell` (the repository is mounted there read-only) and rely on
-a clean, predictable environment, so running them directly on your host is not
-supported and will fail (e.g. templates are looked up under `/ell/templates/`).
+The tests are self-checking Bash scripts: each asserts expected values and
+exits non-zero on any failure, so they can gate CI without human inspection.
+They use a tiny built-in assertion helper (`tests/assert.sh`) rather than an
+external framework, keeping the project dependency-free. The LLM backends are
+exercised offline via the `ell_echo` dummy backend and `file://` JSON fixtures,
+so no network or API key is required.
 
-Run the full suite against the supported Bash versions with:
+### Layout
+
+Tests come in two flavours:
+
+- **Unit tests live next to the source they cover**, named `<source>.test.sh`,
+  so a file's tests are easy to find right beside it:
+
+  ```
+  helpers/json.sh                        helpers/json.test.sh
+  helpers/logging.sh                     helpers/logging.test.sh
+  helpers/piping.sh                      helpers/piping.test.sh
+  helpers/render_to_text.awk             helpers/render_to_text.test.sh
+  plugins/redaction/50_post_input.sh     plugins/redaction/50_post_input.test.sh
+  ```
+
+- **End-to-end tests that drive the whole `ell` pipeline** (and their JSON
+  fixtures) live in `tests/`: `tests/templating.sh` and `tests/parse_output.sh`,
+  alongside the shared assertion helper `tests/assert.sh`.
+
+### Running
+
+Run the whole suite once, on your host:
+
+```bash
+bash tests/entry.sh
+```
+
+Or run it against the oldest and current supported Bash versions in Docker:
 
 ```bash
 bash tests/docker.sh
 ```
 
-This mounts the repository into `bash:4.1` and `bash:5.2` containers and runs
-`tests/entry.sh`, which installs the runtime dependencies and executes every
-test (`logging`, `piping`, `templating`, `parse_output` and `redaction`).
+`tests/entry.sh` auto-discovers every `*.test.sh` in the repository and then
+runs the end-to-end tests. `docker.sh` runs it inside `bash:4.1` and `bash:5.2`
+containers and fails if the suite fails under either version.
+
+### Continuous integration
+
+`.github/workflows/ci.yml` runs on every push and pull request:
+
+- **ShellCheck** over all shell scripts. Findings at `error` severity block the
+  build; warnings are reported but non-blocking so they can be cleaned up
+  incrementally.
+- **Tests** across a `bash:4.1` and `bash:5.2` matrix.
+
+### Adding a test
+
+For a unit test, create `<source>.test.sh` next to the file it covers, source
+the assertion helper (via a path relative to that location), write assertions,
+and end with `assert_summary`. It is picked up automatically by `entry.sh` —
+no registration needed:
+
+```bash
+#!/usr/bin/env bash
+set -o posix;
+DIR="$(dirname "${0}")";
+# From helpers/ this is ../tests/assert.sh; adjust the depth for other dirs.
+. "${DIR}/../tests/assert.sh";
+. "${DIR}/my_helper.sh";
+
+assert_equals "adds up" "3" "$((1 + 2))";
+
+assert_summary;
+```
+
+End-to-end tests that need JSON fixtures or the full pipeline go in `tests/`
+(sourcing `"${DIR}/assert.sh"`) and are registered with an explicit
+`run_test tests/<name>.sh` line in `tests/entry.sh`.
 
 ## Contributing
 

helpers/json.test.sh

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+
+# Unit tests for helpers/json.sh, the pure-bash JSON parser that replaced jq.
+#
+# The parser is security- and correctness-sensitive: it runs on arbitrary API
+# responses, decodes string escapes and \uXXXX (including surrogate pairs) into
+# UTF-8, and validates the full RFC 8259 number grammar. These tests exercise
+# path queries, escape/unicode decoding, presence checks and malformed-input
+# rejection.
+
+set -o posix;
+
+DIR="$(dirname "${0}")";
+. "${DIR}/../tests/assert.sh";
+. "${DIR}/json.sh";
+
+echo "json tests";
+echo "==========";
+
+# --- Path queries -----------------------------------------------------------
+
+json_parse '{"choices":[{"message":{"content":"hello"}}],"usage":{"prompt_tokens":7}}';
+assert_equals "nested object path"   "hello" "$(json_get choices.0.message.content)";
+assert_equals "nested number path"   "7"     "$(json_get usage.prompt_tokens)";
+
+json_parse '{"data":[10,20,30]}';
+assert_equals "array element by index" "20" "$(json_get data.1)";
+# NOTE: array lengths are stored in the JSON array under "<path>.length" but the
+# parser does not populate JSON_TYPE for them, so they are not reachable through
+# the public json_get API (which guards on JSON_TYPE). We therefore assert the
+# stored length via the internal array. If json_get is ever fixed to expose
+# lengths, switch this to: assert_equals ... "$(json_get data.length)".
+assert_equals "array length recorded"  "3"  "${JSON[data.length]}";
+
+json_parse '"just a scalar"';
+assert_equals "root scalar via empty path" "just a scalar" "$(json_get '')";
+
+# --- String escape decoding -------------------------------------------------
+
+json_parse '{"s":"a\nb\tc"}';
+assert_equals "decodes \\n and \\t" "$(printf 'a\nb\tc')" "$(json_get s)";
+
+json_parse '{"s":"quote:\" slash:\/ back:\\"}';
+assert_equals "decodes quote/slash/backslash" 'quote:" slash:/ back:\' "$(json_get s)";
+
+# --- Unicode decoding -------------------------------------------------------
+
+json_parse '{"s":"\u4f60\u597d"}';
+assert_equals "decodes BMP \\u escapes to UTF-8" "你好" "$(json_get s)";
+
+json_parse '{"s":"\ud83d\ude00"}';
+assert_equals "decodes surrogate pair to emoji" "😀" "$(json_get s)";
+
+# --- Presence / null semantics ----------------------------------------------
+
+json_parse '{"present":1,"empty":"","nothing":null}';
+assert_success  "json_has true for present key"  json_has present;
+assert_success  "json_has true for empty string" json_has empty;
+assert_failure  "json_has false for null value"  json_has nothing;
+assert_failure  "json_has false for missing key" json_has does_not_exist;
+assert_failure  "json_get fails on missing key"  json_get does_not_exist;
+
+# --- Validity: well-formed documents accepted -------------------------------
+
+assert_success "valid object"        json_is_valid '{"a":1,"b":[true,false,null]}';
+assert_success "valid nested array"  json_is_valid '[[1],[2,3],[]]';
+assert_success "valid number forms"  json_is_valid '{"a":-1.5e10,"b":0,"c":1.25}';
+
+# --- Validity: malformed documents rejected ---------------------------------
+
+assert_failure "rejects bare word"        json_is_valid '{bad}';
+assert_failure "rejects trailing comma"   json_is_valid '{"a":1,}';
+assert_failure "rejects trailing content" json_is_valid '{"a":1} extra';
+assert_failure "rejects leading zero"     json_is_valid '01';
+assert_failure "rejects bare dot number"  json_is_valid '1.';
+assert_failure "rejects incomplete exp"   json_is_valid '1e';
+assert_failure "rejects unterminated str" json_is_valid '{"a":"oops}';
+assert_failure "rejects empty input"      json_is_valid '';
+
+assert_summary;

helpers/logging.test.sh

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+
+# Self-checking tests for helpers/logging.sh.
+#
+# The logging helpers gate each level on ELL_LOG_LEVEL and write to stderr so
+# they don't pollute pipes. The message-visibility thresholds are:
+#   debug >= 5, info >= 4, warn >= 3, error >= 2, fatal >= 1.
+# These tests capture stderr and assert exactly which messages appear at each
+# level. TO_TTY=false disables colour codes so comparisons stay simple.
+
+set -o posix;
+
+DIR="$(dirname "${0}")";
+. "${DIR}/../tests/assert.sh";
+
+export TO_TTY=false;
+
+. "${DIR}/logging.sh";
+
+# emit_all <level>: set ELL_LOG_LEVEL and call every logging function, returning
+# everything written to stderr as a single string.
+emit_all() {
+  export ELL_LOG_LEVEL="${1}";
+  # Capture only stderr. The logging helpers write to stderr; we want that on
+  # the captured stdout while discarding any real stdout. Using a wrapper block
+  # `{ cmd >/dev/null; } 2>&1` makes the ordering unambiguous (see SC2069): the
+  # inner redirect sends stdout to /dev/null, then 2>&1 routes stderr to the
+  # command substitution's stdout.
+  {
+    {
+      logging_debug "MSG_DEBUG";
+      logging_info  "MSG_INFO";
+      logging_warn  "MSG_WARN";
+      logging_error "MSG_ERROR";
+      logging_fatal "MSG_FATAL";
+    } >/dev/null;
+  } 2>&1;
+}
+
+echo "logging tests";
+echo "=============";
+
+# Level 5 (debug): everything is visible.
+out="$(emit_all 5)";
+assert_contains "L5 shows debug" "${out}" "DEBUG MSG_DEBUG";
+assert_contains "L5 shows info"  "${out}" "INFO MSG_INFO";
+assert_contains "L5 shows warn"  "${out}" "WARN MSG_WARN";
+assert_contains "L5 shows error" "${out}" "ERROR MSG_ERROR";
+assert_contains "L5 shows fatal" "${out}" "FATAL MSG_FATAL";
+
+# Level 4 (info): debug hidden, info and below visible.
+out="$(emit_all 4)";
+assert_not_contains "L4 hides debug" "${out}" "DEBUG MSG_DEBUG";
+assert_contains     "L4 shows info"  "${out}" "INFO MSG_INFO";
+assert_contains     "L4 shows warn"  "${out}" "WARN MSG_WARN";
+assert_contains     "L4 shows error" "${out}" "ERROR MSG_ERROR";
+assert_contains     "L4 shows fatal" "${out}" "FATAL MSG_FATAL";
+
+# Level 3 (warn): info's message hidden, warn/error/fatal visible.
+out="$(emit_all 3)";
+assert_not_contains "L3 hides debug"    "${out}" "DEBUG MSG_DEBUG";
+assert_not_contains "L3 hides info msg" "${out}" "INFO MSG_INFO";
+assert_contains     "L3 shows warn"     "${out}" "WARN MSG_WARN";
+assert_contains     "L3 shows error"    "${out}" "ERROR MSG_ERROR";
+assert_contains     "L3 shows fatal"    "${out}" "FATAL MSG_FATAL";
+
+# Level 2 (error): warn's message hidden, error/fatal visible.
+out="$(emit_all 2)";
+assert_not_contains "L2 hides warn msg" "${out}" "WARN MSG_WARN";
+assert_contains     "L2 shows error"    "${out}" "ERROR MSG_ERROR";
+assert_contains     "L2 shows fatal"    "${out}" "FATAL MSG_FATAL";
+
+# Level 1 (fatal): only fatal's message visible.
+out="$(emit_all 1)";
+assert_not_contains "L1 hides error msg" "${out}" "ERROR MSG_ERROR";
+assert_contains     "L1 shows fatal"     "${out}" "FATAL MSG_FATAL";
+
+# Level 0: nothing is emitted at all.
+out="$(emit_all 0)";
+assert_equals "L0 silent" "" "${out}";
+
+# Logs must go to stderr, never stdout (so they don't corrupt piped output).
+export ELL_LOG_LEVEL=5;
+stdout="$(logging_info "ON_STDOUT" 2>/dev/null)";
+assert_equals "logs never reach stdout" "" "${stdout}";
+
+assert_summary;

helpers/piping.test.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+
+# Self-checking tests for helpers/piping.sh.
+#
+# piping() joins an array of command strings into a single shell pipeline and
+# runs it, feeding its own stdin to the first command. With no commands (or an
+# empty first command) it must behave as a transparent pass-through (cat -).
+
+set -o posix;
+
+DIR="$(dirname "${0}")";
+. "${DIR}/../tests/assert.sh";
+
+# inc: increment every input digit modulo 10. Used to prove that data really
+# flows through each stage of the pipeline in order.
+inc() {
+  while read -r -N 1 char; do
+    printf '%s' $(( (char + 1) % 10 ));
+  done
+}
+export -f inc;
+
+. "${DIR}/piping.sh";
+
+echo "piping tests";
+echo "============";
+
+# No commands: input passes through unchanged.
+out="$(printf 'hello' | piping)";
+assert_equals "no commands passes through" "hello" "${out}";
+
+# Empty first command: also treated as pass-through.
+out="$(printf 'hello' | piping '')";
+assert_equals "empty command passes through" "hello" "${out}";
+
+# Single command runs.
+out="$(printf '000' | piping inc)";
+assert_equals "single command applied once" "111" "${out}";
+
+# Multiple commands chain in order: three inc stages turn 0 into 3.
+out="$(printf '000' | piping inc inc inc)";
+assert_equals "three stages chained" "333" "${out}";
+
+# A real utility pipeline (not just inc) works too.
+out="$(printf 'c\na\nb\n' | piping 'sort' 'tr a-z A-Z')";
+assert_equals "sort then uppercase" "$(printf 'A\nB\nC')" "${out}";
+
+# The original scenario: 256 zeros through 12 inc stages -> all '2', length 256.
+out="$(head -c 256 < /dev/zero | tr '\0' '0' | piping inc inc inc inc inc inc inc inc inc inc inc inc)";
+assert_equals "12 stages over 256 zeros" "$(head -c 256 < /dev/zero | tr '\0' '2')" "${out}";
+
+assert_summary;

tests/render_to_text.sh → helpers/render_to_text.test.sh

@@ -11,7 +11,7 @@
 # Each case feeds a byte-exact input and asserts a byte-exact expected output.
 # The script exits non-zero if any case fails.
 
-RENDER="$(dirname "${0}")/../helpers/render_to_text.awk";
+RENDER="$(dirname "${0}")/render_to_text.awk";
 
 render() {
   LC_ALL=C awk -f "${RENDER}";