Skip to content

Commit 4dba200

Browse files
authored
Merge pull request #224 from inab/STABLE-1.0.x
Several fixes
2 parents 58ed08c + 5f5314f commit 4dba200

File tree

10 files changed

+112
-73
lines changed

10 files changed

+112
-73
lines changed

.pre-commit-config.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ repos:
1111
files: \.(py|pyi)$
1212
exclude: "/(site-packages|development-[^/]*|docs|node_modules|__pycache__|\\..*)/$"
1313
- id: check-json
14+
- id: check-toml
1415
- id: check-yaml
1516
types: [file]
1617
files: \.(yaml|yml|wfex\.(stage|export|ctxt))$
@@ -123,4 +124,4 @@ repos:
123124
hooks:
124125
- id: pip-audit
125126
stages: [manual]
126-
args: ["-r", "requirements.txt"]
127+
args: ["-r", "requirements.txt"]

CITATION.cff

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,5 +28,5 @@ repository-code: "https://github.com/inab/WfExS-backend"
2828
repository-artifact: "https://github.com/inab/WfExS-backend/pkgs/container/wfexs-backend"
2929
type: software
3030
title: "WfExS-backend"
31-
version: 1.0.2
31+
version: 1.0.3
3232
date-released: "2025-10-11"

INSTALL.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,12 @@ just using next bash pattern:
1414

1515
```bash
1616
# WFEXS_VER can be either a branch, a tag or a commit hash
17-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
17+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
1818

1919
# Alternatively, you can use local copy
2020
WFEXS_VER=$(git rev-parse HEAD)
2121

22-
docker build -t inab/wfexs-backend:${WFEXS_VER} \
22+
docker buildx build -t inab/wfexs-backend:${WFEXS_VER} \
2323
--build-arg wfexs_checkout="${WFEXS_VER}" \
2424
https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Dockerfile
2525
```
@@ -29,7 +29,7 @@ a local copy of the recipe, and next command line from the project root will hel
2929

3030
```bash
3131
# WFEXS_VER can be either a branch, a tag or a commit hash
32-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
32+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
3333

3434
# Alternatively, you can use local copy
3535
WFEXS_VER=$(git rev-parse HEAD)
@@ -38,7 +38,7 @@ mkdir WfExS_docker_build
3838
cd WfExS_docker_build
3939
curl -O https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Dockerfile
4040

41-
docker build -t inab/wfexs-backend:${WFEXS_VER} \
41+
docker buildx build -t inab/wfexs-backend:${WFEXS_VER} \
4242
--build-arg wfexs_checkout="${WFEXS_VER}" \
4343
Dockerfile
4444
```
@@ -54,7 +54,7 @@ just using next bash pattern:
5454

5555
```bash
5656
# WFEXS_VER can be either a branch, a tag or a commit hash
57-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
57+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
5858

5959
# Alternatively, you can use local copy
6060
WFEXS_VER=$(git rev-parse HEAD)
@@ -70,7 +70,7 @@ a local copy of the recipe, and next command line from the project root will hel
7070

7171
```bash
7272
# WFEXS_VER can be either a branch, a tag or a commit hash
73-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
73+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
7474

7575
# Alternatively, you can use local copy
7676
WFEXS_VER=$(git rev-parse HEAD)
@@ -95,7 +95,7 @@ The precondition is having either Apptainer or Singularity properly setup. There
9595

9696
```bash
9797
# WFEXS_VER can be either a branch, a tag or a commit hash
98-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
98+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
9999
mkdir WfExS_SIF_build
100100
cd WfExS_SIF_build
101101
curl -O https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Singularity.def
@@ -106,7 +106,7 @@ The precondition is having either Apptainer or Singularity properly setup. There
106106

107107
```bash
108108
# WFEXS_VER can be either a branch, a tag or a commit hash
109-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
109+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
110110

111111
# Alternatively, you can use local copy
112112
WFEXS_VER=$(git rev-parse HEAD)
@@ -122,7 +122,7 @@ The precondition is having either Apptainer or Singularity properly setup. There
122122

123123
```bash
124124
# Remember to use the correct tag!!!
125-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
125+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
126126
singularity build wfexs-${WFEXS_VER}.sif docker-daemon://inab/wfexs-backend:${WFEXS_VER}
127127
```
128128

@@ -139,7 +139,7 @@ The precondition is having either Apptainer or Singularity properly setup. There
139139
cd WfExS_SIF_build
140140

141141
# Remember to use the correct tag!!!
142-
WFEXS_VER=4f314f46ce2c18499837947081e74a1e9eba1f31
142+
WFEXS_VER=dae813eb000bdb0a921fb74c87b7ff356e3f7134
143143

144144
# Next command should be used if you used podman to build the local image
145145
podman save -o wfexs-backend-${WFEXS_VER}.tar inab/wfexs-backend:${WFEXS_VER}

MANIFEST.in

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
include LICENSE
2+
include *.md
3+
include requirements.txt

container_recipes/Dockerfile

Lines changed: 18 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,26 +10,25 @@ RUN git clone --filter=blob:none --no-checkout https://github.com/jmfernandez/su
1010
gcc -static -o suid_entrypoint suid_entrypoint.c
1111

1212
# The default images of python are based on debian
13-
FROM pypy:3.11-slim AS podman_build
13+
FROM pypy:3.11-slim-trixie AS podman_build
1414
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
15-
ARG apptainer_version=1.3.6
15+
ARG apptainer_version=1.4.3
16+
WORKDIR /
1617
# Install apptainer
1718
# hadolint ignore=DL3008
1819
RUN DPKG_ARCH=$(dpkg --print-architecture) && \
1920
apt-get update && \
20-
apt-get install -y wget build-essential git libxml2-dev libxslt-dev zlib1g-dev && \
21-
wget -O /tmp/rust-installer.sh https://sh.rustup.rs && \
21+
apt-get install -y --no-install-recommends wget build-essential git libxml2-dev libxslt-dev zlib1g-dev && \
22+
wget -nv -O /tmp/rust-installer.sh https://sh.rustup.rs && \
2223
chmod +x /tmp/rust-installer.sh && \
2324
/tmp/rust-installer.sh -y && \
2425
wget -nv \
25-
https://github.com/apptainer/apptainer/releases/download/v${apptainer_version}/apptainer_${apptainer_version}_${DPKG_ARCH}.deb \
26-
https://github.com/apptainer/apptainer/releases/download/v${apptainer_version}/apptainer-suid_${apptainer_version}_${DPKG_ARCH}.deb && \
27-
apt-get install -y ./*.deb && \
28-
rm -f *.deb
26+
https://github.com/apptainer/apptainer/releases/download/v"${apptainer_version}"/apptainer_"${apptainer_version}"-trixie+_"${DPKG_ARCH}".deb \
27+
https://github.com/apptainer/apptainer/releases/download/v"${apptainer_version}"/apptainer-suid_"${apptainer_version}"-trixie+_"${DPKG_ARCH}".deb && \
28+
apt-get install -y --no-install-recommends ./*.deb
2929
# Install WfExS, trusting the installers in the commit rather than the
3030
# one in the docker recipe
31-
ARG wfexs_checkout=70277f9708ecb338cfb192ddcbf24c7159d8f319
32-
WORKDIR /
31+
ARG wfexs_checkout=eb5d0f0fc95c60e545b0c96ec8da0efff0a7d74b
3332
# hadolint ignore=DL3003,DL3013
3433
RUN git clone --filter=blob:none --no-checkout https://github.com/inab/WfExS-backend.git && \
3534
cd WfExS-backend && \
@@ -41,9 +40,9 @@ RUN git clone --filter=blob:none --no-checkout https://github.com/inab/WfExS-bac
4140
container_recipes/full-installer.bash && \
4241
git checkout "${wfexs_checkout}" && \
4342
grep -vF git+ constraints-${PYVER}.txt > constraints-${PYVER}.txt-relaxed && \
44-
. ${HOME}/.cargo/env && \
43+
. "${HOME}"/.cargo/env && \
4544
pip install --no-cache-dir --upgrade pip wheel && \
46-
pip install --no-cache-dir "git+https://github.com/inab/WfExS-backend.git@${wfexs_checkout}" -c constraints-${PYVER}.txt-relaxed && \
45+
pip install --no-cache-dir "git+https://github.com/inab/WfExS-backend.git@${wfexs_checkout}" -c constraints-"${PYVER}".txt-relaxed && \
4746
python -m compileall "$(python -c 'import sys; print(sys.prefix)')" && \
4847
mv container_recipes/* /
4948

@@ -56,7 +55,7 @@ ARG OPENJ9_VER=0.26.0
5655
# Go version to compile
5756
ARG GO_VER=1.20.14
5857
# gocryptfs version
59-
ARG GOCRYPTFS_VER=v2.5.1
58+
ARG GOCRYPTFS_VER=v2.6.1
6059
# static bash version
6160
ARG STATIC_BASH_VER=5.1.004-1.2.2
6261
# static busybox version
@@ -73,13 +72,13 @@ RUN apt-get install -y --no-install-recommends curl libmagic1 graphviz && \
7372
BUSYBOX_VER="${BUSYBOX_VER}" \
7473
bash /full-installer.bash
7574

76-
FROM pypy:3.11-slim AS podman_image
75+
FROM pypy:3.11-slim-trixie AS podman_image
7776
COPY --from=podman_build /podman_containers_nokeyring.conf /etc/containers/containers.conf
7877
COPY --from=podman_build /opt/pypy /opt/pypy
78+
COPY --from=podman_build /*.deb /tmp/
7979
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
8080

8181
# Python 3 is installed so it can be used with cwltool
82-
ARG apptainer_version=1.3.6
8382
# hadolint ignore=DL3008
8483
RUN DPKG_ARCH=$(dpkg --print-architecture) && \
8584
python -c 'import urllib.request ; urllib.request.urlretrieve("https://download.docker.com/linux/debian/gpg", "/etc/apt/keyrings/docker.asc")' && \
@@ -90,11 +89,10 @@ RUN DPKG_ARCH=$(dpkg --print-architecture) && \
9089
tee /etc/apt/sources.list.d/docker.list > /dev/null && \
9190
apt-get update && \
9291
apt-get install -y --no-install-recommends git libmagic1 libxml2 libxslt1.1 zlib1g python3-venv podman encfs graphviz docker-ce-cli wget && \
93-
wget -nv \
94-
https://github.com/apptainer/apptainer/releases/download/v${apptainer_version}/apptainer_${apptainer_version}_${DPKG_ARCH}.deb \
95-
https://github.com/apptainer/apptainer/releases/download/v${apptainer_version}/apptainer-suid_${apptainer_version}_${DPKG_ARCH}.deb && \
96-
apt-get install -y ./*.deb && \
97-
rm -f *.deb /var/cache/apt/archives/*.deb
92+
apt-get install -y --no-install-recommends /tmp/*.deb && \
93+
apt-get clean && \
94+
rm -f /tmp/*.deb /var/cache/apt/archives/*.deb && \
95+
rm -rf /var/lib/apt/lists/*
9896

9997
CMD ["/bin/bash"]
10098

container_recipes/Singularity.def

Lines changed: 59 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
Bootstrap: docker
2-
From: python:3.12
3-
Stage: spython-base
2+
From: pypy:3.11-slim-trixie
3+
Stage: build
44

55
%arguments
66
# The default images of python are based on debian
77
# These arguments help customizing what it is included in the generated image
8-
wfexs_checkout=574fe343c0b59eecd95afbc67894456359ebe649
9-
apptainer_version=1.3.6
8+
wfexs_checkout=eb5d0f0fc95c60e545b0c96ec8da0efff0a7d74b
9+
apptainer_version=1.4.3
1010
# JDK version parameters
1111
JDK_MAJOR_VER=11
1212
# Nested arguments are not allowed
@@ -16,7 +16,7 @@ OPENJ9_VER=0.26.0
1616
# Go version to compile
1717
GO_VER=1.20.14
1818
# gocryptfs version
19-
GOCRYPTFS_VER=v2.4.0
19+
GOCRYPTFS_VER=v2.6.1
2020
# static bash version
2121
STATIC_BASH_VER=5.1.004-1.2.2
2222
# static busybox version
@@ -25,56 +25,79 @@ BUSYBOX_VER=1.35.0
2525

2626
%post
2727
set -ue
28-
# Install apptainer
28+
29+
cd /
30+
31+
# Install preconditions (including apptainer)
2932
DPKG_ARCH=$(dpkg --print-architecture) && \
30-
wget -nv \
31-
https://github.com/apptainer/apptainer/releases/download/v{{ apptainer_version }}/apptainer_{{ apptainer_version }}_${DPKG_ARCH}.deb \
32-
https://github.com/apptainer/apptainer/releases/download/v{{ apptainer_version }}/apptainer-suid_{{ apptainer_version }}_${DPKG_ARCH}.deb && \
3333
apt-get update && \
34-
apt-get install -y ./*.deb && \
35-
rm -f *.deb
36-
# Install docker-ce-cli
37-
# hadolint ignore=DL3008
38-
wget -nv -O /etc/apt/keyrings/docker.asc https://download.docker.com/linux/debian/gpg && \
39-
chmod a+r /etc/apt/keyrings/docker.asc && \
40-
echo \
41-
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
42-
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
43-
tee /etc/apt/sources.list.d/docker.list > /dev/null && \
44-
apt-get update && \
45-
apt-get install -y --no-install-recommends docker-ce-cli
46-
# Install both podman, encfs and graphviz
47-
# hadolint ignore=DL3008
48-
apt-get install -y podman encfs graphviz
34+
apt-get install -y --no-install-recommends wget build-essential git libxml2-dev libxslt-dev zlib1g-dev && \
35+
wget -nv -O /tmp/rust-installer.sh https://sh.rustup.rs && \
36+
chmod +x /tmp/rust-installer.sh && \
37+
/tmp/rust-installer.sh -y && \
38+
wget -nv \
39+
https://github.com/apptainer/apptainer/releases/download/v"{{ apptainer_version }}"/apptainer_"{{ apptainer_version }}"-trixie+_"${DPKG_ARCH}".deb \
40+
https://github.com/apptainer/apptainer/releases/download/v"{{ apptainer_version }}"/apptainer-suid_"{{ apptainer_version }}"-trixie+_"${DPKG_ARCH}".deb && \
41+
apt-get install -y --no-install-recommends ./*.deb
42+
4943
# Install WfExS, trusting the installers in the commit rather than the
5044
# one in the docker recipe
51-
mkdir -p /
52-
cd /
53-
# hadolint ignore=DL3003
5445
git clone --filter=blob:none --no-checkout https://github.com/inab/WfExS-backend.git && \
5546
cd WfExS-backend && \
5647
git sparse-checkout init --cone && \
57-
PYVER=$(python -c 'import sys; print("{}.{}".format(sys.version_info.major, sys.version_info.minor))') && \
58-
git sparse-checkout set constraints-${PYVER}.txt container_recipes/basic-installer.bash container_recipes/full-installer.bash && \
48+
PYVER=$(python -c 'import sys; import platform ; print("{}{}.{}".format("pypy" if platform.python_implementation() == "PyPy" else "", sys.version_info.major, sys.version_info.minor))') && \
49+
git sparse-checkout set constraints-${PYVER}.txt \
50+
podman_containers_nokeyring.conf \
51+
container_recipes/basic-installer.bash \
52+
container_recipes/full-installer.bash && \
5953
git checkout "{{ wfexs_checkout }}" && \
60-
pip install --no-cache-dir "git+https://github.com/inab/WfExS-backend.git@{{ wfexs_checkout }}" -c constraints-${PYVER}.txt && \
54+
grep -vF git+ constraints-${PYVER}.txt > constraints-${PYVER}.txt-relaxed && \
55+
. "${HOME}"/.cargo/env && \
56+
pip install --no-cache-dir --upgrade pip wheel && \
57+
pip install --no-cache-dir "git+https://github.com/inab/WfExS-backend.git@{{ wfexs_checkout }}" -c constraints-"${PYVER}".txt-relaxed && \
6158
python -m compileall "$(python -c 'import sys; print(sys.prefix)')" && \
62-
mv container_recipes/* / && \
63-
cd .. && \
64-
rm -rf WfExS-backend
65-
# Transfer other third party installation script
66-
# COPY basic-installer.bash full-installer.bash ./
59+
mv container_recipes/* /
60+
6761
# Install third party software
62+
apt-get install -y --no-install-recommends curl libmagic1 graphviz && \
6863
JDK_MAJOR_VER="{{ JDK_MAJOR_VER }}" \
6964
JDK_VER="{{ JDK_VER }}" \
70-
JDK_REV="{{JDK_REV }}" \
65+
JDK_REV="{{ JDK_REV }}" \
7166
OPENJ9_VER="{{ OPENJ9_VER }}" \
7267
GO_VER="{{ GO_VER }}" \
7368
GOCRYPTFS_VER="{{ GOCRYPTFS_VER }}" \
7469
STATIC_BASH_VER="{{ STATIC_BASH_VER }}" \
7570
BUSYBOX_VER="{{ BUSYBOX_VER }}" \
7671
bash /full-installer.bash
7772

73+
Bootstrap: docker
74+
From: pypy:3.11-slim-trixie
75+
Stage: image
76+
77+
%files from build
78+
/podman_containers_nokeyring.conf /etc/containers/containers.conf
79+
/opt/pypy /opt
80+
/*.deb /tmp/
81+
82+
%post
83+
set -ue
84+
85+
# Install docker-ce-cli
86+
# Cleanups
87+
DPKG_ARCH=$(dpkg --print-architecture) && \
88+
python -c 'import urllib.request ; urllib.request.urlretrieve("https://download.docker.com/linux/debian/gpg", "/etc/apt/keyrings/docker.asc")' && \
89+
chmod a+r /etc/apt/keyrings/docker.asc && \
90+
echo \
91+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
92+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
93+
tee /etc/apt/sources.list.d/docker.list > /dev/null && \
94+
apt-get update && \
95+
apt-get install -y --no-install-recommends git libmagic1 libxml2 libxslt1.1 zlib1g python3-venv podman encfs graphviz docker-ce-cli wget && \
96+
apt-get install -y --no-install-recommends /tmp/*.deb && \
97+
apt-get clean && \
98+
rm -f /tmp/*.deb /var/cache/apt/archives/*.deb && \
99+
rm -rf /var/lib/apt/lists/*
100+
78101
%runscript
79102
cd /
80103
exec /bin/bash bash "$@"

pyproject.toml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[build-system]
2+
requires = [
3+
"setuptools>=42",
4+
"wheel"
5+
]
6+
build-backend = "setuptools.build_meta"
7+

requirements.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ pyncclient-ext >= 0.1.2
3535
wiktionary-term-fetcher >= 0.1.1
3636
funny-passphrase >= 0.2.3
3737
pyxdg
38-
groovy-parser == 0.2.1
38+
groovy-parser == 0.2.2
3939
data-url >= 1.1.1
4040
pgzip
4141
defusedxml

wfexs_backend/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
__license__ = "Apache 2.0"
2222

2323
# https://www.python.org/dev/peps/pep-0396/
24-
__version__ = "1.0.2"
24+
__version__ = "1.0.3"
2525
__url__ = "https://github.com/inab/WfExS-backend"
2626
__official_name__ = "WfExS-backend"
2727

wfexs_backend/fetchers/git.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,8 @@
8686
import dulwich.porcelain
8787
import dulwich.repo
8888

89+
from dulwich import __version__ as dulwich_version
90+
8991
from . import (
9092
AbstractSchemeRepoFetcher,
9193
DocumentedStatefulProtocolFetcher,
@@ -296,9 +298,14 @@ def _find_git_repo_in_uri(
296298
# Possible sha in repoTag
297299
memory_repo = dulwich.repo.MemoryRepo()
298300
transport, path = get_transport_and_path(remote_uri_anc)
299-
fetch_pack_result = transport.fetch(
300-
path, cast("dulwich.repo.Repo", memory_repo)
301-
)
301+
if dulwich_version < (0, 24, 6):
302+
fetch_pack_result = transport.fetch(
303+
path, cast("dulwich.repo.Repo", memory_repo) # type: ignore[arg-type]
304+
)
305+
else:
306+
fetch_pack_result = transport.fetch(
307+
path.encode("utf-8"), cast("dulwich.repo.Repo", memory_repo) # type: ignore[arg-type]
308+
)
302309
try:
303310
memory_repo.get_object(repoTag.encode("utf-8"))
304311
b_default_repo_tag = repoTag

0 commit comments

Comments
 (0)