Skip to content

RHAIENG-2101, RHAIENG-2111: chore(build): fix builds in 3.2 #2719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jiridanek merged 15 commits into from
Dec 4, 2025
Merged

RHAIENG-2101, RHAIENG-2111: chore(build): fix builds in 3.2 #2719

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f3c8c18
RHAIENG-2111: chore(trustyai): podman build in rhds: error: command '…
jiridanek Nov 25, 2025
0496b20
RHAIENG-2174: build(codeserver,trustyai): fix build failure "no space…
jiridanek Nov 28, 2025
c486364
NO-JIRA: enable codeready-builder/crb repository
jiridanek Nov 28, 2025
8d62cff
RHAIENG-2174: build(trustyai): fix build failure "no space left on de…
jiridanek Nov 30, 2025
081b38b
RHAIENG-2111: build(trustyai): add missing pillow requirements and ot…
jiridanek Nov 30, 2025
cd57ffd
RHAIENG-2111: build(trustyai): undo the --offline flag for uv, since …
jiridanek Nov 30, 2025
f23bf6c
RHAIENG-2189: build(minimal): set extra python index for aipcc migrat…
jiridanek Dec 1, 2025
0d2c393
RHAIENG-2189: build(datascience): set extra python index for aipcc mi…
jiridanek Dec 1, 2025
3a9dd64
RHAIENG-2189: build(runtime/datascience): fix CMAKE_PREFIX_PATH undef…
jiridanek Dec 1, 2025
405d7f5
RHAIENG-2189: build(runtime/minimal): set extra python index for aipc…
jiridanek Dec 1, 2025
ca62a3a
RHAIENG-2189: build(runtime/minimal): add missing -devel libraries
jiridanek Dec 1, 2025
d53d1f9
RHAIENG-2189: build(datascience): add missing -devel libraries for s3…
jiridanek Dec 1, 2025
d7b095a
RHAIENG-2189: build(codeserver): add back pypi index
jiridanek Dec 1, 2025
3e2b7ca
RHAIENG-2189: build(codeserver): and remove the --offline flag, since…
jiridanek Dec 1, 2025
7e23557
NO-JIRA: chore(pylock.tomls): run `pylock_generator.sh` with `--upgra…
jiridanek Dec 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions codeserver/ubi9-python-3.12/Dockerfile.cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ FROM ${BASE_IMAGE} AS cpu-base

WORKDIR /opt/app-root/bin

# RHAIENG-2189: this is AIPCC migration phase 1.5
ENV PIP_EXTRA_INDEX_URL=https://pypi.org/simple
ENV UV_EXTRA_INDEX_URL=https://pypi.org/simple

# OS Packages needs to be installed as root
USER 0

Expand Down Expand Up @@ -296,8 +300,8 @@ RUN --mount=type=cache,target=/root/.cache/uv /bin/bash <<'EOF'
set -Eeuxo pipefail
echo "Installing softwares and packages"
# we can ensure wheels are consumed from the cache only by restricting internet access for uv install with '--offline' flag
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --offline --cache-dir /root/.cache/uv --requirements=./pylock.toml
# Note: debugpy wheel availabe on pypi (in uv cache) is none-any but bundles amd64.so files
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --cache-dir /root/.cache/uv --requirements=./pylock.toml
# Note: debugpy wheel available on pypi (in uv cache) is none-any but bundles amd64.so files
# Build debugpy from source instead
UV_LINK_MODE=copy uv pip install --no-cache git+https://github.com/microsoft/debugpy.git@v$(grep -A1 '\"debugpy\"' ./pylock.toml | grep -Eo '\b[0-9\.]+\b')
# change ownership to default user (all packages were installed as root and has root:root ownership
Expand Down
2,714 changes: 1,435 additions & 1,279 deletions codeserver/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion jupyter/datascience/ubi9-python-3.12/Dockerfile.cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,7 @@ EOF

RUN --mount=type=cache,target=/var/cache/dnf,sharing=locked,id=notebooks-dnf /bin/bash <<'EOF'
set -Eeuxo pipefail
if [ "${TARGETARCH}" = "ppc64le" ]; then
if [ "${TARGETARCH}" = "ppc64le" ] || [ "$TARGETARCH" = "s390x" ]; then
packages=(
# required to compile pillow
zlib-devel libjpeg-turbo-devel
Expand Down
4,905 changes: 2,656 additions & 2,249 deletions jupyter/datascience/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions jupyter/minimal/ubi9-python-3.12/Dockerfile.cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ FROM registry.access.redhat.com/ubi9/ubi AS ubi-repos
####################
FROM ${BASE_IMAGE} AS cpu-base

# RHAIENG-2189: this is AIPCC migration phase 1.5
ENV PIP_EXTRA_INDEX_URL=https://pypi.org/simple
ENV UV_EXTRA_INDEX_URL=https://pypi.org/simple

WORKDIR /opt/app-root/bin

# OS Packages needs to be installed as root
Expand Down
2,140 changes: 1,168 additions & 972 deletions jupyter/minimal/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

5,169 changes: 2,832 additions & 2,337 deletions jupyter/pytorch+llmcompressor/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

4,917 changes: 2,662 additions & 2,255 deletions jupyter/pytorch/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

4,917 changes: 2,662 additions & 2,255 deletions jupyter/rocm/pytorch/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

4,419 changes: 2,393 additions & 2,026 deletions jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

5,033 changes: 2,731 additions & 2,302 deletions jupyter/tensorflow/ubi9-python-3.12/pylock.toml
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 5 additions & 3 deletions jupyter/trustyai/ubi9-python-3.12/Dockerfile.cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ pip install --no-cache-dir uv
source ./devel_env_setup.sh
# This may have to download and compile some dependencies, and as we don't lock requirements from `build-system.requires`,
# we often don't know the correct hashes and `--require-hashes` would therefore fail on non amd64, where building is common.
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --strict --no-deps --refresh --no-config --no-progress --verify-hashes --compile-bytecode --index-strategy=unsafe-best-match --requirements=./pylock.toml
# RHAIENG-2111: building pandas requires cython from PyPI
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --strict --no-deps --refresh --no-config --no-progress --verify-hashes --compile-bytecode --index-strategy=unsafe-best-match --extra-index-url https://pypi.org/simple --requirements=./pylock.toml
EOF

####################
Expand Down Expand Up @@ -222,10 +223,11 @@ RUN --mount=type=cache,target=/root/.cache/uv /bin/bash <<'EOF'
set -Eeuxo pipefail
echo "Installing softwares and packages"
# we can ensure wheels are consumed from the cache only by restricting internet access for uv install with '--offline' flag
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --offline --cache-dir /root/.cache/uv --requirements=./pylock.toml
# RHAIENG-2111: building pandas requires cython from PyPI
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --cache-dir /root/.cache/uv --strict --no-deps --no-config --no-progress --verify-hashes --compile-bytecode --index-strategy=unsafe-best-match --extra-index-url https://pypi.org/simple --requirements=./pylock.toml
# Note: debugpy wheel availabe on pypi (in uv cache) is none-any but bundles amd64.so files
# Build debugpy from source instead
UV_LINK_MODE=copy uv pip install --no-cache git+https://github.com/microsoft/debugpy.git@v$(grep -A1 '\"debugpy\"' ./pylock.toml | grep -Eo '\b[0-9\.]+\b')
UV_NO_CACHE=false UV_LINK_MODE=copy uv pip install --no-cache git+https://github.com/microsoft/debugpy.git@v$(grep -A1 '\"debugpy\"' ./pylock.toml | grep -Eo '\b[0-9\.]+\b')
# change ownership to default user (all packages were installed as root and has root:root ownership
chown -R 1001:0 /opt/app-root/
chmod -R g=u /opt/app-root
Expand Down
30 changes: 26 additions & 4 deletions jupyter/trustyai/ubi9-python-3.12/devel_env_setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,36 @@ else
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH:-/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig}
fi

# the build uses uv cache to pass artifacts between stages
export UV_NO_CACHE=false
export UV_LINK_MODE=copy

# compiling jpype1==1.5.0 requires g++ and this gets compiled on all platforms
# gcc and g++ is present by default on registry.access.redhat.com/ubi9/python-312:latest
dnf install -y --setopt=keepcache=1 gcc gcc-g++
Comment on lines +25 to +27
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Fix incorrect package name: use gcc-c++ instead of gcc-g++.

The package gcc-g++ does not exist in the UBI9 repository. The correct package name for the C++ compiler is gcc-c++. This will cause the dnf install command to fail.

This inconsistency is evident when comparing with line 41, where the correct gcc-c++ package is used for s390x.

Apply this diff to fix the package name:

-dnf install -y --setopt=keepcache=1 gcc gcc-g++
+dnf install -y --setopt=keepcache=1 gcc gcc-c++
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# compiling jpype1==1.5.0 requires g++ and this gets compiled on all platforms
# gcc and g++ is present by default on registry.access.redhat.com/ubi9/python-312:latest
dnf install -y --setopt=keepcache=1 gcc gcc-g++
# compiling jpype1==1.5.0 requires g++ and this gets compiled on all platforms
# gcc and g++ is present by default on registry.access.redhat.com/ubi9/python-312:latest
dnf install -y --setopt=keepcache=1 gcc gcc-c++
🤖 Prompt for AI Agents 
In jupyter/trustyai/ubi9-python-3.12/devel_env_setup.sh around lines 21 to 23,
the dnf install uses the non-existent package name gcc-g++; replace it with the
correct UBI9 package name gcc-c++ so the C++ compiler installs correctly (make
the same change as used on line 41), and ensure the dnf command remains
otherwise unchanged.

Copy link
Member Author

@jiridanek jiridanek Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai think it through again; either works

Copy link
Contributor

@coderabbitai coderabbitai bot Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if both gcc-c++ and gcc-g++ packages exist in UBI9 repositories
# We'll use dnf to query package information

# First, let's check what packages provide g++ in UBI9
echo "=== Checking gcc-c++ package ==="
dnf info gcc-c++ 2>&1 || echo "gcc-c++ query failed"

echo ""
echo "=== Checking gcc-g++ package ==="
dnf info gcc-g++ 2>&1 || echo "gcc-g++ query failed"

echo ""
echo "=== Checking what provides g++ ==="
dnf provides g++ 2>&1 || echo "g++ provides query failed"

echo ""
echo "=== Searching for gcc related packages ==="
dnf search gcc | grep -E "(gcc-c\+\+|gcc-g\+\+)" || echo "search completed"

Length of output: 402

🌐 Web query:

UBI9 RHEL9 gcc-g++ package dnf install

💡 Result:

Short answer: install the RHEL/UBI package named gcc-c++ (it provides g++):

Commands (run as root inside a UBI9 / RHEL9 environment)

  • dnf install -y gcc-c++
  • Verify: g++ --version

If you want a newer toolset instead of system gcc, use Red Hat's GCC Toolset container or packages (e.g. gcc-toolset-14 / gcc-toolset-13) and enable/run it via scl or the toolset container. See Red Hat docs for the toolset options. [1][2][3]

Sources:

  • UBI9 package repository showing gcc-c++ package. [1]
  • Red Hat docs: GCC Toolset (gcc-toolset-N providing gcc-c++). [2]
  • Red Hat UBI9 description / images. [3]

🌐 Web query:

"gcc-g++" package RHEL dnf

💡 Result:

There is no package named "gcc-g++" on RHEL. The C++ compiler (g++) is provided by the gcc-c++ package (or by the "Development Tools" group). Example commands:

  • Install just g++: sudo dnf install gcc-c++
  • Install full dev toolchain: sudo dnf groupinstall "Development Tools"
  • To find the package that supplies g++ on your system: dnf provides '*bin/g++'

Sources: Red Hat documentation and community guidance. [1][2]

References
[1] Red Hat Developer — Install GCC / Development Tools on RHEL.
[2] Unix StackExchange / community answers showing gcc-c++ provides /usr/bin/g++.

🌐 Web query:

"gcc-g++" alternative package name alias RHEL UBI

💡 Result:

Use the rpm named gcc-c++ (provides the g++ C++ compiler). [1][2]

Sources:

  • UBI AppStream packages index showing gcc-c++ RPM. [1]
  • Red Hat docs on GCC packaging/versions (RHEL uses gcc / gcc-c++). [2]

🌐 Web query:

UBI9 virtual package provides gcc-g++

💡 Result:

Short answer: No — UBI 9 doesn't use a "gcc-g++" virtual package. The C++ compiler is provided by the gcc-c++ (or gcc-c++/gcc‑11… rpm) packages in the UBI 9 AppStream/BaseOS repos (e.g., gcc-c++ and gcc packages are present in the UBI9 repo). [1]

Sources:

  • UBI 9 AppStream package index (shows gcc and gcc-c++ RPMs). [1]

🌐 Web query:

dnf install typo correction "did you mean" package suggestions

💡 Result:

Yes — install the PackageKit command-not-found handler. It provides “did you mean” / offer-to-install suggestions (and with dnf5 can auto-install and rerun the command).

Commands:

  • sudo dnf install PackageKit-command-not-found
  • Log out/in (or source the shell file): source /etc/profile.d/PackageKit.sh

References: Fedora feature/docs for PackageKit command-not-found [1], the PackageKit package page [2], and a writeup showing dnf5 + command‑not‑found behavior [3].

🌐 Web query:

dnf automatic package name correction typo fix install

💡 Result:

If you mean the DNF automatic-updates feature: install the dnf-automatic package (or the dnf5 plugin on systems using dnf5), enable its timer, and edit the config.

Commands:

  • Fedora (dnf):
    sudo dnf install dnf-automatic
    sudo systemctl enable --now dnf-automatic.timer
  • Systems with dnf5 (Fedora newer releases):
    sudo dnf install dnf5-plugin-automatic
    sudo systemctl enable --now dnf-automatic.timer

Edit /etc/dnf/automatic.conf to choose download-only, apply_updates=True, reboot behavior, etc. Check status with:
systemctl status dnf-automatic.timer
systemctl list-timers --all | grep dnf-automatic

Sources: Fedora DNF docs and dnf-automatic package page. [1][2][3]

Sources
[1] Using the DNF software package manager — Fedora Docs. [2] dnf-automatic — Fedora Packages. [3] Automatic Updates — Fedora Docs.

🏁 Script executed:

#!/bin/bash
# Let's check the actual file content to see if gcc-g++ is used elsewhere
# and look at the context around line 41 mentioned in my original review

echo "=== Searching for gcc-c++ usage in the repository ==="
rg -n "gcc-c\+\+" --type sh

echo ""
echo "=== Searching for gcc-g++ usage in the repository ==="
rg -n "gcc-g\+\+" --type sh

echo ""
echo "=== Looking at the context around line 41 in the file ==="
sed -n '35,50p' jupyter/trustyai/ubi9-python-3.12/devel_env_setup.sh

Length of output: 1655

@jiridanek, I've investigated this further and found some evidence that suggests gcc-g++ may not be a valid package name:

Evidence from documentation:

  • Multiple sources confirm that the C++ compiler (g++) is provided by the gcc-c++ package in UBI9/RHEL9
  • No "gcc-g++" virtual package exists in UBI 9

Evidence from this codebase:

  • Line 53 in this same file uses gcc-c++ for the s390x architecture
  • Other files in the repository consistently use gcc-c++:
    • base-images/utils/aipcc.sh:144
    • codeserver/ubi9-python-3.12/devel_env_setup.sh:69
  • Only line 27 (the new code in this PR) uses gcc-g++

This inconsistency within the same file is notable. If gcc-g++ were valid, it would be unusual for line 53 to use a different package name.

Could you verify which package name actually succeeds in your testing environment? If gcc-g++ does work in your tests, I'd be interested to understand the mechanism, as it doesn't appear to be a standard package name in the UBI9 repositories.

That said, I defer to your testing results—if you've confirmed both work, perhaps there's a package alias or DNF behavior I'm not aware of.

Copy link
Member Author

@jiridanek jiridanek Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai create github issue to fix this later

Copy link
Contributor

@coderabbitai coderabbitai bot Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue to track the gcc package naming inconsistency

gh issue create \
  --repo opendatahub-io/notebooks \
  --title "Inconsistent gcc C++ compiler package names in devel_env_setup.sh" \
  --body "## Description

There is an inconsistency in the package names used for installing the C++ compiler in \`jupyter/trustyai/ubi9-python-3.12/devel_env_setup.sh\`:

- Line 23 uses \`gcc-g++\`
- Line 53 uses \`gcc-c++\`

While both package names appear to work in testing, \`gcc-c++\` is the standard package name in UBI9/RHEL9 repositories and is used consistently throughout the rest of the codebase.

## Files affected

- \`jupyter/trustyai/ubi9-python-3.12/devel_env_setup.sh\`

## Context

This inconsistency was identified during review of PR #2719. The issue is deferred for later resolution to avoid blocking the current PR.

## References

- PR: https://github.com/opendatahub-io/notebooks/pull/2719
- Discussion: https://github.com/opendatahub-io/notebooks/pull/2719#discussion_r2559331988
- Requested by: @jiridanek" \
  --assignee jiridanek

Length of output: 55

@jiridanek, I've created issue #2755 to track this inconsistency: #2755

The issue documents the difference between gcc-g++ (line 23) and gcc-c++ (line 53) usage in the file, and includes references back to this PR and discussion.


WHEELS_DIR=/wheelsdir
mkdir -p ${WHEELS_DIR}
if [[ $(uname -m) == "ppc64le" ]] || [[ $(uname -m) == "s390x" ]]; then
CURDIR=$(pwd)

# install development packages
dnf install -y --setopt=keepcache=1 https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
dnf clean all
# enable codeready-builder/crb repository
dnf install -y dnf-plugins-core
if command -v subscription-manager &> /dev/null; then
subscription-manager repos --enable "codeready-builder-for-rhel-9-$(uname -m)-rpms"
else
dnf config-manager --set-enabled crb
fi
# patchelf: needed by `auditwheel repair`
dnf install -y --setopt=keepcache=1 fribidi-devel gcc-toolset-13 lcms2-devel libimagequant-devel patchelf \
dnf install -y --setopt=keepcache=1 fribidi-devel lcms2-devel libimagequant-devel patchelf \
libraqm-devel openjpeg2-devel tcl-devel tk-devel unixODBC-devel

# previously we installed but never activated (added to PATH, etc.) the gcc-toolset-13, which suggests that maybe
# regular gcc/g++ is just fine

# Install build tools and libraries needed for compiling PyTorch/PyArrow
if [[ $(uname -m) == "s390x" ]]; then
dnf install -y --setopt=keepcache=1 gcc gcc-gfortran gcc-c++ make cmake ninja-build \
Expand All @@ -41,9 +60,12 @@ if [[ $(uname -m) == "ppc64le" ]] || [[ $(uname -m) == "s390x" ]]; then
tcl-devel tk-devel unixODBC-devel \
git tar wget unzip
else
# ppc64le packages
# ppc64le packages, mostly compile requirements for pillow
# and openssl-devel for maturin, and gcc-gfortran for scikit-learn
dnf install -y --setopt=keepcache=1 fribidi-devel lcms2-devel libimagequant-devel \
libraqm-devel openjpeg2-devel tcl-devel tk-devel unixODBC-devel
zlib-devel libraqm-devel libjpeg-devel libjpeg-turbo-devel openjpeg2-devel tcl-devel tk-devel \
openssl-devel gcc-gfortran \
unixODBC-devel
fi

# Install Rust for both ppc64le and s390x
Expand Down Expand Up @@ -122,7 +144,7 @@ if [[ $(uname -m) == "ppc64le" ]] || [[ $(uname -m) == "s390x" ]]; then
# Pyarrow
PYARROW_VERSION=$(grep -A1 '"pyarrow"' pylock.toml | grep -Eo '\b[0-9\.]+\b')
cd ${TMP}
git clone --depth 1 --branch "apache-arrow-${PYARROW_VERSION}" --recurse-submodules --shallow-submodules https://github.com/apache/arrow.git
git clone --depth 1 --branch "apache-arrow-${PYARROW_VERSION}" --recurse-submodules --shallow-submodules https://github.com/apache/arrow.git
cd arrow/cpp
mkdir build && cd build && \
# Set architecture-specific CMake flags
Expand Down
Loading
Loading