diff --git a/.github/workflows/rust-pipeline.yml b/.github/workflows/rust-pipeline.yml index add3e4e441..459b726c6b 100644 --- a/.github/workflows/rust-pipeline.yml +++ b/.github/workflows/rust-pipeline.yml @@ -8,47 +8,98 @@ on: env: CARGO_TERM_COLOR: always + # Incremental compilation hurts CI clean builds (extra disk I/O, larger + # target dirs, no benefit since each runner is ephemeral) — turn it off. + CARGO_INCREMENTAL: 0 + # Be more tolerant of registry hiccups on Windows. + CARGO_NET_RETRY: 10 + RUSTUP_MAX_RETRIES: 10 + # CI never runs a debugger, so full debug symbols (`debug=2`) are wasted + # compile + link time. `line-tables-only` keeps panic backtraces (file:line) + # so test failures stay readable, while cutting most debuginfo generation. + # Set via env (CI-only) rather than Cargo.toml so local dev keeps full symbols. + # Applies to every job, Windows included. `test` inherits from `dev`, but we + # set both explicitly so the override sticks regardless of inheritance. + CARGO_PROFILE_DEV_DEBUG: line-tables-only + CARGO_PROFILE_TEST_DEBUG: line-tables-only jobs: build: runs-on: ubuntu-22.04 + timeout-minutes: 45 steps: - uses: actions/checkout@v4 - name: Install X11 dev libs run: | - sudo apt-get update && sudo apt-get install -y libx11-dev protobuf-compiler libcurl4-openssl-dev + # `mold` is a fast linker — cuts a large slice off Rust link time + # (609 deps to link). Invoked below via `mold -run`, which redirects + # the linker through LD_PRELOAD, so it needs no .cargo/config.toml + # (which would force mold on every local dev too) and works with the + # runner's default gcc regardless of version. + sudo apt-get update && sudo apt-get install -y libx11-dev protobuf-compiler libcurl4-openssl-dev mold curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rust-install.sh bash ./rust-install.sh -y - - name: Build + - name: Cache cargo deps + uses: Swatinem/rust-cache@v2 + with: + workspaces: rust + cache-on-failure: true + # Build + test share a SINGLE (debug) profile so artifacts are reused — + # previously `cargo build --release` then `cargo test` compiled the whole + # workspace + all deps twice (release then debug), roughly doubling the + # job. CI only needs to prove it compiles and passes tests, which debug + # does. `--no-run` splits compile-vs-run timing; `--verbose` dropped to + # cut log I/O. + - name: Build tests run: | cd rust - cargo build --release --verbose + mold -run cargo test --no-run - name: Run tests run: | cd rust - cargo test --verbose + cargo test clippy: runs-on: ubuntu-22.04 + timeout-minutes: 30 steps: - uses: actions/checkout@v4 - name: Install X11 dev libs run: | - sudo apt-get update && sudo apt-get install -y libx11-dev protobuf-compiler libcurl4-openssl-dev + sudo apt-get update && sudo apt-get install -y libx11-dev protobuf-compiler libcurl4-openssl-dev mold curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rust-install.sh bash ./rust-install.sh -y + - name: Cache cargo deps + uses: Swatinem/rust-cache@v2 + with: + workspaces: rust + cache-on-failure: true - name: Run Clippy run: | cd rust - cargo clippy --verbose + mold -run cargo clippy --verbose windows-clippy: runs-on: windows-latest + timeout-minutes: 45 steps: - uses: actions/checkout@v4 + - name: Speed up Windows build (exclude build dirs from Defender) + shell: pwsh + run: | + # Real-time AV scanning of every compiler artifact (rustc writes + # thousands of .rlib/.o/.exe files) is the dominant hidden cost on + # Windows Rust CI. Exclude the workspace + cargo dirs and disable + # realtime monitoring outright — typically a 30-50% speedup. All + # calls are best-effort (Tamper Protection may block some). + Set-MpPreference -DisableRealtimeMonitoring $true -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionPath "$env:GITHUB_WORKSPACE" -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionPath "$env:USERPROFILE\.cargo" -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionProcess "cargo.exe" -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionProcess "rustc.exe" -ErrorAction SilentlyContinue - name: Install Rust uses: dtolnay/rust-toolchain@stable with: @@ -57,21 +108,49 @@ jobs: run: choco install protoc -y - name: Export PROTOC path run: echo "PROTOC=C:\ProgramData\chocolatey\bin\protoc.exe" >> $env:GITHUB_ENV + - name: Cache cargo deps + uses: Swatinem/rust-cache@v2 + with: + workspaces: rust + cache-on-failure: true - name: Run Clippy working-directory: rust - run: cargo clippy -p rqd --verbose + run: cargo clippy -p rqd windows-tests: runs-on: windows-latest + timeout-minutes: 60 steps: - uses: actions/checkout@v4 + - name: Speed up Windows build (exclude build dirs from Defender) + shell: pwsh + run: | + # See windows-clippy for rationale — AV scanning dominates Windows + # Rust CI time; excluding build dirs is a 30-50% win. Best-effort. + Set-MpPreference -DisableRealtimeMonitoring $true -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionPath "$env:GITHUB_WORKSPACE" -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionPath "$env:USERPROFILE\.cargo" -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionProcess "cargo.exe" -ErrorAction SilentlyContinue + Add-MpPreference -ExclusionProcess "rustc.exe" -ErrorAction SilentlyContinue - name: Install Rust uses: dtolnay/rust-toolchain@stable - name: Install Protobuf run: choco install protoc -y - name: Export PROTOC path run: echo "PROTOC=C:\ProgramData\chocolatey\bin\protoc.exe" >> $env:GITHUB_ENV + - name: Cache cargo deps + uses: Swatinem/rust-cache@v2 + with: + workspaces: rust + cache-on-failure: true + # Build tests first so we see compile-vs-run timing separately and the + # cache is populated even if a test later fails. `--verbose` is dropped + # — log volume noticeably slows Windows runners and the failure output + # is enough for triage. + - name: Build tests + working-directory: rust + run: cargo test -p rqd --no-run - name: Run tests working-directory: rust - run: cargo test -p rqd --verbose + run: cargo test -p rqd diff --git a/.github/workflows/scheduler-stress-pipeline.yml b/.github/workflows/scheduler-stress-pipeline.yml new file mode 100644 index 0000000000..a2021b3bf5 --- /dev/null +++ b/.github/workflows/scheduler-stress-pipeline.yml @@ -0,0 +1,154 @@ +name: OpenCue Scheduler Stress Pipeline + +# Runs the scheduler booking + accounting stress suite +# (rust/crates/scheduler/tests/stress_tests.rs): a full pipeline::run against a +# seeded farm, with an end-of-run audit that cross-checks the in-memory +# accounting store against SUM(proc) in Postgres and asserts cap enforcement. +# +# When it runs — and when it deliberately doesn't: +# - Pull requests: only when the scheduler crate, its proto dependency, the +# DB migrations, or this workflow change. The suite needs a migrated +# Postgres and takes several minutes — running it for Python/CueGUI/docs +# changes would burn runner time for zero signal. +# - Nightly on master: catches drift from changes that slipped past the +# paths filter (e.g. shared workspace dependencies) and gives a daily +# throughput data point under fixed scale. +# - Manually (workflow_dispatch): for benchmarking a branch at custom scale. +# +# What is a gate vs. what is informational: +# - The job FAILS on correctness regressions: accounting drift between the +# in-memory store and Postgres, cap breaches (subscription burst / job +# max-cores), booking +# liveness (<90% drain, no saturation rejections), or leftover test data. +# - The throughput numbers (frames/s) are reported in the step summary but +# are NOT asserted on: shared runners are too noisy for perf gating. For +# real benchmarking run the suite locally in release mode (see +# docs/_docs/developer-guide/scheduler-stress-testing.md). + +on: + pull_request: + branches: ["master"] + paths: + - "rust/crates/scheduler/**" + - "rust/crates/opencue-proto/**" + - "rust/Cargo.toml" + - "cuebot/src/main/resources/conf/ddl/postgres/migrations/**" + - ".github/workflows/scheduler-stress-pipeline.yml" + schedule: + # Nightly on master. Odd minute to avoid the top-of-hour scheduling rush. + - cron: "23 9 * * *" + workflow_dispatch: + inputs: + stress_jobs: + description: "Drain-phase job count (default 300)" + required: false + stress_hosts: + description: "Drain-phase host count (default 1200)" + required: false + stress_frames_per_layer: + description: "Drain-phase frames per layer (default 5)" + required: false + stress_timeout_secs: + description: "Per-phase hard timeout in seconds (default 600)" + required: false + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +env: + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: 0 + CARGO_NET_RETRY: 10 + RUSTUP_MAX_RETRIES: 10 + +jobs: + stress: + runs-on: ubuntu-22.04 + timeout-minutes: 45 + + # The suite's Postgres service. Accounting state lives in-process inside the + # test itself, so nothing else needs to be provisioned here. + services: + postgres: + image: postgres:15.1 + env: + POSTGRES_USER: cuebot + POSTGRES_PASSWORD: cuebot_password + POSTGRES_DB: cuebot + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U cuebot -d cuebot" + --health-interval 5s + --health-timeout 5s + --health-retries 10 + + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + # The suite only reads the repo (no git push), so don't leave the + # token on disk for later steps. + persist-credentials: false + + - name: Install build dependencies + run: | + sudo apt-get update && sudo apt-get install -y libx11-dev protobuf-compiler libcurl4-openssl-dev postgresql-client + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rust-install.sh + bash ./rust-install.sh -y + + - name: Cache cargo deps + uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1 + with: + workspaces: rust + cache-on-failure: true + + # Plain psql instead of the sandbox Flyway image: the migrations are + # versioned plain-SQL files, so applying them in numeric order is exactly + # what Flyway would do, without building a JDK image first. + - name: Apply database migrations + working-directory: cuebot/src/main/resources/conf/ddl/postgres/migrations + env: + PGPASSWORD: cuebot_password + run: | + for f in $(ls V*.sql | sort -t V -k2 -n); do + echo "== $f" + psql -q -v ON_ERROR_STOP=1 -h localhost -U cuebot -d cuebot -f "$f" + done + + - name: Run stress suite + working-directory: rust + shell: bash + env: + STRESS_JOBS: ${{ inputs.stress_jobs }} + STRESS_HOSTS: ${{ inputs.stress_hosts }} + STRESS_FRAMES_PER_LAYER: ${{ inputs.stress_frames_per_layer }} + STRESS_TIMEOUT_SECS: ${{ inputs.stress_timeout_secs }} + run: | + cargo test -p scheduler --features stress-tests --test stress_tests -- --nocapture 2>&1 | tee stress-output.log + + - name: Publish phase report + if: always() + working-directory: rust + shell: bash + run: | + if [ -f stress-output.log ] && grep -q "^================ phase" stress-output.log; then + { + echo "## Scheduler stress suite" + echo '```' + sed -n '/^================ phase/,$p' stress-output.log | sed -n '1,80p' + echo '```' + echo "_Throughput numbers are informational; only the accounting/enforcement assertions gate this job._" + } >> "$GITHUB_STEP_SUMMARY" + else + echo "## Scheduler stress suite: no phase report produced (failed before the run?)" >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Upload full output + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: scheduler-stress-output + path: rust/stress-output.log + if-no-files-found: ignore + retention-days: 30 diff --git a/.github/workflows/testing-pipeline.yml b/.github/workflows/testing-pipeline.yml index 3b6d155b92..5be4403aef 100644 --- a/.github/workflows/testing-pipeline.yml +++ b/.github/workflows/testing-pipeline.yml @@ -411,7 +411,9 @@ jobs: - name: Build CueWeb Docker image working-directory: cueweb run: | - docker buildx build . -f Dockerfile -t opencue/cueweb:test --load + docker buildx build . -f Dockerfile -t opencue/cueweb:test --load \ + --build-context project_root=.. \ + --build-arg NEXT_PUBLIC_GIT_SHA=$(git rev-parse --short HEAD) echo "CueWeb Docker image built successfully" build_rest_gateway: diff --git a/GOVERNANCE.md b/GOVERNANCE.md new file mode 100644 index 0000000000..83621c635d --- /dev/null +++ b/GOVERNANCE.md @@ -0,0 +1,207 @@ + + + +# OpenCue Project Governance + +OpenCue is a project of the Academy Software Foundation and relies on the +ASWF governance policies, supported by the Linux Foundation. + +There are three primary project roles: Contributors submit code to the +project; Committers approve code to be included into the project; and the +Technical Steering Committee (TSC) provides overall high-level project +guidance. + +* [Contributors](`#contributors`) +* [Committers](`#committers`) +* [Technical Steering Committee](`#technical-steering-committee`) + +## Contributors + +The OpenCue project grows and thrives from assistance from Contributors. +Contributors include anyone in the community who contributes code, +documentation, or other technical artifacts that have been incorporated into +the project repository. + +Anyone can be a Contributor. You need no formal approval from the project, +beyond the legal forms. + +### How to Become a Contributor + +* Review the coding standards and [contributing guidelines](CONTRIBUTING.md) + to ensure your contribution is in line with the project's coding and styling + guidelines. + +* Sign the Individual CLA, or if you are employed by an organization that + might have any claim to IP you create, have your organization sign the + Corporate CLA. + +* Submit your code as a PR with the appropriate DCO sign-off on each commit. + The easiest way to do this is to ensure that you commit your code with `git + commit -s`. + +## Committers + +Project Committers have merge access on the OpenCue GitHub repository +and are responsible for approving submissions by Contributors. + +### Committer Responsibilities + +Typical activities of a Committer include: + +* Helping users and novice contributors. + +* Ensuring a response to questions posted to the the project github + +* Contributing code and documentation changes that improve the project. + +* Reviewing and commenting on issues and pull requests. + +* Ensuring that changes and new code meet acceptable standards and are in + the long-term interest of the project. + +* Participation in working groups. + +* Merging pull requests. + +### How to Become a Committer + +Any member of the OpenCue community (though typically an existing +Committer or TSC member) may nominate an individual making significant and +valuable contributions to the OpenCue project to become a new Committer. +To nominate a new Committer, open an issue in the OpenCue repository, send +mail to the TSC mail list, or raise the issue at a TSC meeting. + +The TSC may periodically review the Committer list to identify inactive +Committers. Past Committers are typically given Emeritus status. Emeriti may +request that the TSC restore them to active Committer status. + +## Technical Steering Committee + +The Technical Steering Committee (TSC) has final authority over this project. +As defined in the project [technical charter](https://github.com/AcademySoftwareFoundation/foundation/blob/main/project_charters/opencue_charter.pdf), in +addition to committer activities, TSC responsibilities also include, but are +not limited to: + +* Coordinating technical direction of the Project. + +* Project governance and contribution policy. + +* GitHub repository administration. + +* Maintaining the list of additional Committers + +* Appointing representatives to work with other open source or open + standards communities. + +* Discussions, seeking consensus, and where necessary, voting on technical + matters relating to the code base that affect multiple projects. + +* Coordinating any marketing, events, or communications regarding the + project. + +Within the TSC are two elected leadership roles to be held by its members +and voted on annually. Any TSC member can express interest in serving in a +role, or nominate another member to serve. There are no term limits, and one +person may hold multiple roles simultaneously. Should a TSC member resign +from a leadership role before their term is complete, a successor shall be +elected through the standard nomination and voting process to complete the +remainder of the term. The leadership roles are: + +* **Chair**: This position acts as the project manager, organizing meetings + and providing oversight to project administration. + +* **Chief Architect**: This position makes all the final calls on design and + technical decisions, and is responsible for avoiding "design by committee" + pitfalls. In the absence of an architect, the chair position acts as an architect. + +The chair role is assumed to rotate periodically (though there are no term +limits, so the TSC may reelect an existing chair). The chief architect +position should be a source of stability and coherent design vision, so the +TSC is encouraged to choose an architect who can serve for many years and +only change architects when it is necessary for the health of the project +and its community. + +At the time of election, the TSC will also agree upon which of these two +leaders will serve as the OpenCue ASWF (Academy Software Foundation) TAC +(Technical Advisory Council) representative for the term. This member +represents the project at all ASWF TAC meetings. + +### TSC Members + +Current voting members of the TSC are: + +* **Chair and TAC representative**: Diego Tavares - Sony Pictures Imageworks +* Ramon Figueiredo - Sony Pictures Imageworks +* Jimmy Christensen - Ghost VFX + + +### TSC Nomination and Succession + +Any proposal for additional members of the TSC may be submitted by Committers, +TSC members, or other major stakeholders of the OpenCue community by +opening an issue outlining their case or raising the issue at a TSC meeting. +New TSC members are accepted or rejected by majority vote of the TSC. + +If a TSC member is for an extended period not regularly participating or +performing the responsibilities expected of TSC members, the TSC may by +majority vote request an alternate TSC member be submitted by that +organization, or remove the inactive member from the TSC. + +A voting member of the TSC may nominate a successor in the event that such +voting member decides to leave the TSC, and the TSC, including the departing +member, shall confirm or reject such nomination by a vote. In the event that +the departing member's nomination for successor is rejected by vote of the +TSC, the departing member shall be entitled to continue nominating successors +until one such successor is confirmed by vote of the TSC. If the departing +member fails or is unable to nominate a successor, the TSC may nominate one on +the departing member's behalf. + +TSC membership is presumed to be retained by the individual even if they +change employers, provided they remain active in the project. The TSC may take +action to ensure that organizational stakeholder representation not become +severely disproportionate, for example by urging an organization that loses +its sole TSC representative to nominate a new member, or by limiting the total +number of voting members from any one organization if too many members all +move to the same organization. + +### TSC Meetings + +Any meetings of the TSC are intended to be open to the public, except where +there is a reasonable need for privacy. The TSC meets regularly in a voice +conference call, at a cadence deemed appropriate by the TSC. The TSC Chair +moderates the meeting, or appoints another TSC member to moderate in his or +her absence. Meetings may also be streamed online where appropriate; +connection details will be posted to the opencue [slack channel](https://academysoftwarefdn.slack.com/archives/CMFPXV39Q) +in advance of the scheduled meeting. + +Items are added to the TSC agenda which are considered contentious or are +modifications of governance, contribution policy, TSC membership, or release +process, in addition to topics involving the high-level technical direction +of the project. + +The intention of the agenda is not to approve or review all patches. That +should happen continuously on GitHub and be handled by the larger group of +Committers. + +Any community member or Contributor can ask that something be reviewed by +the TSC at the meeting by logging a GitHub issue. Any Committer, TSC member, +or the meeting chair can bring the issue to the TSC's attention by applying +the `TSC` label. + +Prior to each TSC meeting, the meeting chair will share the agenda with +members of the TSC. TSC members can also add items to the agenda at the +beginning of each meeting. The meeting chair and the TSC cannot veto or +remove items. + +The TSC may invite additional persons to participate in a non-voting +capacity. + +The meeting chair is responsible for ensuring that minutes are taken and +archived in the project repository or other designated accessible location. + +Due to the challenges of scheduling a global meeting with participants in +several time zones, the TSC will seek to resolve as many agenda items as +possible outside of meetings on the public mailing list or through GitHub +issues, discussions, and pull requests. + +Meeting notes (usually very brief) can be found at: https://lf-aswf.atlassian.net/wiki/spaces/OPENCUE/pages/11283660/Meeting+Notes?atlOrigin=eyJpIjoiOTVlZjdjMjFjZDkwNDU0ZmE4ZGEyNWU0NmUxODE2YTgiLCJwIjoiYyJ9 diff --git a/VERSION.in b/VERSION.in index 5fb5a6b4f5..e0250ea3c7 100644 --- a/VERSION.in +++ b/VERSION.in @@ -1 +1 @@ -1.20 +1.28 diff --git a/api_docs/modules/opencue.wrappers.rst b/api_docs/modules/opencue.wrappers.rst index cc29047edf..4def99c957 100644 --- a/api_docs/modules/opencue.wrappers.rst +++ b/api_docs/modules/opencue.wrappers.rst @@ -28,6 +28,12 @@ opencue.wrappers.deed module .. automodule:: opencue.wrappers.deed :members: +opencue.wrappers.department module +---------------------------------- + +.. automodule:: opencue.wrappers.department + :members: + opencue.wrappers.depend module ------------------------------ diff --git a/ci/build_sphinx_docs.sh b/ci/build_sphinx_docs.sh index caa6f1ab24..6ebaa4c919 100755 --- a/ci/build_sphinx_docs.sh +++ b/ci/build_sphinx_docs.sh @@ -12,9 +12,9 @@ else fi # Sphinx has some additional requirements -pip install ${PIP_OPT} -r api_docs/requirements.txt +python -m pip install ${PIP_OPT} -r api_docs/requirements.txt -pip install ${PIP_OPT} proto/ pycue/ pyoutline/ cueadmin/ cuesubmit/ cuegui/ +python -m pip install ${PIP_OPT} proto/ pycue/ pyoutline/ cueadmin/ cuesubmit/ cuegui/ # ci/build_proto.sh # Build the docs and treat warnings as errors diff --git a/cueadmin/README.md b/cueadmin/README.md index 0ae007b68c..84f29e1563 100644 --- a/cueadmin/README.md +++ b/cueadmin/README.md @@ -39,6 +39,8 @@ cueadmin -lji # List hosts cueadmin -lh +cueadmin -lh -lock-state NIMBY_LOCKED # Only NIMBY-locked hosts +cueadmin -lh -lock-state OPEN -sort-idle # Unlocked hosts, most idle first # Job management cueadmin -pause JOB_NAME # Pause a job diff --git a/cueadmin/cueadmin/common.py b/cueadmin/cueadmin/common.py index 5a24f17d2d..afbfb51071 100644 --- a/cueadmin/cueadmin/common.py +++ b/cueadmin/cueadmin/common.py @@ -229,6 +229,17 @@ def getParser(): # choices=["UP", "DOWN", "REPAIR"], type=str.upper, help="Filter host search by hardware state, up or down.", ) + filter_grp.add_argument( + "-lock-state", + action="store", + choices=["OPEN", "LOCKED", "NIMBY_LOCKED"], + help="Filter -lh output to hosts in this lock state (e.g. NIMBY_LOCKED)", + ) + filter_grp.add_argument( + "-sort-idle", + action="store_true", + help="Sort -lh output by most idle resources first", + ) # # Show @@ -298,6 +309,17 @@ def getParser(): "Jobs submitted to the archived show will be executed by " "allocations subscribed to the target show.", ) + + show.add_argument( + "-scheduler-managed", + action="store", + nargs=2, + metavar="SHOW ON|OFF", + help="Set whether accounting for the given show is owned by the Rust " + "scheduler. When ON, Cuebot stops updating accounting tables " + "transactionally for this show and the Rust scheduler reconciles " + "them from the proc table.", + ) # # Allocation # @@ -1062,7 +1084,9 @@ def handleArgs(args): if args.lh: states = [Convert.strToHardwareState(s) for s in args.state] cueadmin.output.displayHosts( - opencue.api.getHosts(match=args.query, state=states, alloc=args.alloc) + opencue.api.getHosts(match=args.query, state=states, alloc=args.alloc), + lock_state=args.lock_state, + sort_idle=args.sort_idle, ) return @@ -1254,6 +1278,22 @@ def handleArgs(args): show.archive, target_show_name, ) + + elif args.scheduler_managed: + show_name, value = args.scheduler_managed + if value.lower() not in ("on", "off"): + raise ValueError( + "Invalid value for -scheduler-managed: %r (expected ON or OFF)" % value + ) + enabled = value.lower() == "on" + show = opencue.api.findShow(show_name) + verb = "Enable" if enabled else "Disable" + confirm( + "%s scheduler-managed accounting on %s" % (verb, opencue.rep(show)), + args.force, + show.setSchedulerManaged, + enabled, + ) # # Hosts are handled a bit differently than the rest # of the entities. To specify a host or hosts the user diff --git a/cueadmin/cueadmin/output.py b/cueadmin/cueadmin/output.py index dc38b5c11d..8ecef57646 100644 --- a/cueadmin/cueadmin/output.py +++ b/cueadmin/cueadmin/output.py @@ -56,11 +56,27 @@ def displayProcs(procs): ) -def displayHosts(hosts): +def displayHosts(hosts, lock_state=None, sort_idle=False): """Displays the host information on one line each. @type hosts: list @param hosts: Hosts to display information about + @type lock_state: str + @param lock_state: if set, only show hosts in this lock state + (OPEN, LOCKED or NIMBY_LOCKED) + @type sort_idle: bool + @param sort_idle: if True, sort by most idle resources first """ + hosts = list(hosts) + if lock_state: + hosts = [ + host + for host in hosts + if opencue.api.host_pb2.LockState.Name(host.data.lock_state) == lock_state + ] + if sort_idle: + hosts.sort(key=lambda v: (-v.data.idle_cores, -v.data.idle_memory)) + else: + hosts.sort(key=lambda v: v.data.name) host_format = ( "%-15s %-4s %-5s %-8s %-8s %-9s %-5s %-5s %-16s %-8s %-8s %-6s %-9s %-10s %-7s" ) @@ -84,7 +100,7 @@ def displayHosts(hosts): "Thread", ) ) - for host in sorted(hosts, key=lambda v: v.data.name): + for host in hosts: print( host_format % ( diff --git a/cueadmin/tests/test_common.py b/cueadmin/tests/test_common.py index eeb685d1ed..6e9830961a 100644 --- a/cueadmin/tests/test_common.py +++ b/cueadmin/tests/test_common.py @@ -155,6 +155,26 @@ def testDisableDispatch(self, getStubMock, findShowMock): showMock.enableDispatching.assert_called_with(False) + def testEnableSchedulerManaged(self, getStubMock, findShowMock): + args = self.parser.parse_args( + ["-scheduler-managed", TEST_SHOW, "on", "-force"]) + showMock = mock.Mock() + findShowMock.return_value = showMock + + cueadmin.common.handleArgs(args) + + showMock.setSchedulerManaged.assert_called_with(True) + + def testDisableSchedulerManaged(self, getStubMock, findShowMock): + args = self.parser.parse_args( + ["-scheduler-managed", TEST_SHOW, "off", "-force"]) + showMock = mock.Mock() + findShowMock.return_value = showMock + + cueadmin.common.handleArgs(args) + + showMock.setSchedulerManaged.assert_called_with(False) + def testDefaultMinCores(self, getStubMock, findShowMock): arbitraryCoreCount = 873 args = self.parser.parse_args( diff --git a/cuebot/scheduler-sim/.gitignore b/cuebot/scheduler-sim/.gitignore new file mode 100644 index 0000000000..32955b51aa --- /dev/null +++ b/cuebot/scheduler-sim/.gitignore @@ -0,0 +1,7 @@ +# Generated by setup.sh / simulate.py — never commit these. +venv/ +opencue_proto/ +sim_hosts +scheduler_sim.yaml +resolve_local.so +*.log diff --git a/cuebot/scheduler-sim/BUILD.md b/cuebot/scheduler-sim/BUILD.md new file mode 100644 index 0000000000..3c84a81fc7 --- /dev/null +++ b/cuebot/scheduler-sim/BUILD.md @@ -0,0 +1,77 @@ +# Building & running cuebot in this dev box (hard-won notes) + +This box has a toolchain trap. These are the exact steps that work. + +## The toolchain trap +- The repo pins **Gradle 7.6.2** (cuebot/gradle/wrapper). It does NOT run on the + box's default **JDK 21** ("Unsupported class file major version 65" — bundled + ASM too old). The standalone **Gradle 8.14.3** in /opt runs on 21 but is too + new for the Spring Boot 2.2.1 plugin ("ArchivePublishArtifact"). +- Correct combo: **wrapper Gradle 7.6.2 + JDK 17**. + +## JDK 17 (with the proxy CA) +A vanilla JDK 17 download can't fetch deps: the env's outbound proxy uses a TLS +CA that vanilla cacerts don't trust (Gradle reports "plugin not found"). Fix: +copy the managed JDK 21 truststore into the JDK 17. +```bash +# /tmp/jdk-17.0.2 was unpacked from openjdk-17.0.2_linux-x64; then: +cp /usr/lib/jvm/java-21-openjdk-amd64/lib/security/cacerts /tmp/jdk-17.0.2/lib/security/cacerts +``` + +## Repos: drop the dead ones (build-time only, do NOT commit) +cuebot/settings.gradle (pluginManagement) and build.gradle list `jcenter()` and +`repo.spring.io/plugins-snapshot`, which are dead and break resolution on 7.6.2. +Strip them before building: +```bash +# in cuebot/: remove the 'maven { url ".../plugins-snapshot" }' line and 'jcenter()' lines +``` + +## Postgres (must run as non-root; refuses root) +```bash +# run these as your own (non-root) user; postgres refuses root, no sudo needed +PGBIN=/usr/lib/postgresql/16/bin +rm -rf /tmp/pgdata && mkdir -p /tmp/pgdata /tmp/pgrun +$PGBIN/initdb -D /tmp/pgdata -U cue --auth=trust +$PGBIN/pg_ctl -D /tmp/pgdata -o "-p 5433 -k /tmp/pgrun -c listen_addresses=127.0.0.1" -l /tmp/pg.log start +$PGBIN/psql -h127.0.0.1 -p5433 -Ucue -dpostgres -c "CREATE DATABASE cuebot;" +# apply migrations in version order: +cd cuebot/src/main/resources/conf/ddl/postgres/migrations +for f in $(ls *.sql | sort -t_ -k1.2 -n); do $PGBIN/psql -h127.0.0.1 -p5433 -Ucue -dcuebot -v ON_ERROR_STOP=1 -q -f "$f"; done +# base data: dept/services/config from seed_data.sql + scheduler-sim/sim_seed.sql +``` + +## Build / run cuebot (as your own user, JDK 17, wrapper 7.6.2) +A dedicated gradle home /tmp/ghome-$USER holds the resolved deps. Build as the +user that owns the checkout (a fresh `git clone` already is); no specific account +is required. Remove cuebot/.gradle if you hit `checksums.lock (Permission denied)`. +```bash +cd cuebot && env \ + CUEBOT_DB_URL="jdbc:postgresql://127.0.0.1:5433/cuebot" CUEBOT_DB_USER=cue CUEBOT_DB_PASSWORD= \ + SCHEDULER_ENABLED=true SCHEDULER_INTERVAL_MS=250 SCHEDULER_RESERVATIONS_ENABLED=false \ + ./gradlew bootRun -g /tmp/ghome-$USER -Dorg.gradle.java.home=/tmp/jdk-17.0.2 --console=plain >/tmp/cuebot.log 2>&1 +``` +gRPC serves on **8443**. Compile-only check: swap `bootRun` for `compileJava` +(note `-Werror -Xlint:all` is on — warnings fail the build). Unit tests: +`./gradlew test --tests "...SchedulerTests"`. + +## CRITICAL launch pattern (process management) +Launch long-running procs (cuebot, pinger, fake_rqd) with the Bash tool's +`run_in_background: true` and **NO inner `&`**. An inner `&` double-backgrounds +and the JVM/Python gets SIGKILLed when the wrapper shell exits. +After a cuebot restart, restart status_pinger.py too (its gRPC channel goes +stale -> all ReportStatus fail -> hosts age to DOWN). + +## Reset the farm between runs +```bash +psql ... -c "DELETE FROM proc;" +psql ... -c "UPDATE host SET int_cores_idle=int_cores,int_mem_idle=int_mem,int_gpus_idle=int_gpus,int_gpu_mem_idle=int_gpu_mem;" +psql ... -c "UPDATE subscription SET int_cores=0,int_gpus=0;" +# clear the job backlog (frames then layers) for the sim show: +psql ... -c "DELETE FROM frame f USING job j WHERE f.pk_job=j.pk_job AND j.pk_show='10000000-0000-0000-0000-000000000003';" +psql ... -c "DELETE FROM layer l USING job j WHERE l.pk_job=j.pk_job AND j.pk_show='10000000-0000-0000-0000-000000000003';" +``` + +## Gotcha: "unable to allocate additional memory" +That is NOT a Postgres OOM. It's `trigger__verify_host_resources` raising when a +booking pushes a host's int_*_idle below 0 (overbooking protection). Treat it as +an overbooking/accounting signal, not a memory problem. diff --git a/cuebot/scheduler-sim/README.md b/cuebot/scheduler-sim/README.md new file mode 100644 index 0000000000..1310ddf13a --- /dev/null +++ b/cuebot/scheduler-sim/README.md @@ -0,0 +1,313 @@ +# Scheduler farm simulator + +A DB-backed integration harness for the cuebot `Scheduler`. It is not a model of +cuebot, it **is** cuebot: a **real cuebot + Postgres** driven over gRPC by a +**fake render farm**, so every booking goes through the exact production path +(the real `Scheduler`, the real SQL, the real frame-complete handler) — no +database writes from the driver. Hosts register like RQD, jobs are submitted like +a client, and a fake RQD runs/completes frames. It is the integration test unit +tests cannot be, and the place to observe behaviour that only shows up under +load: utilization, co-locality, throughput, reservations, dependency handling, +and big-job placement. + +This exercises the *real* booking path end to end and has already surfaced +several bugs. + +## Pieces +| file | role | +|------|------| +| **`simulate.py`** | **one command: tears down, resets DB, brings the whole stack up fresh, starts a workload** | +| `farm_spec.py` | the farm: 246 large/128c, 303 medium/32c, 1004 small/16c; mem 4 GB/core | +| `sim_model.py` | frame cores/mem/duration distribution (from real-farm CSVs); `SIM_COMPRESS` env scales durations | +| `sim_seed.sql` | one-time base data: facility/alloc/show/subscription | +| `register_hosts.py` | register all hosts via RQD ReportRqdStartup | +| `rqd_report.py [int]` | faithful host **+ running-frame** status heartbeat; refreshes `proc.ts_ping` so the 300s orphan sweep behaves like prod | +| `status_pinger.py` / `status_pinger_fast.py` | older empty-frame heartbeat (kept; superseded by `rqd_report.py`) | +| `fake_rqd.py [threads]` | fake RQD gRPC server on :8444; runs+completes frames. Used by new, old, and `rust --rust-real-launch`. `threads`=completion-report concurrency (1=serial, 64=concurrent RQDs) | +| `rqd_complete.py [int] [memfail]` | default `--mode rust`: polls the proc table for frames the Rust scheduler booked (dry-run) and reports them complete to cuebot after their `sim_model` run-time — the DB-poll analogue of `fake_rqd.py` | +| `gen_jobs.py` | submit a realistic job mix via LaunchSpec | +| `feed.py [dur] [target]` | paced feeder: hold a sustained backlog of ~`target` waiting frames | +| `drain_test.py