Skip to content

Commit 19a57e6

Browse files
EnmkEnmk
authored
Merge pull request #370 from Altinity/customizations/23.3.19
23.3.19 Pre-release PR
2 parents c4d4ca8 + ceabecc commit 19a57e6

File tree

773 files changed

+4937
-6738
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

773 files changed

+4937
-6738
lines changed

‎.github/actionlint.yml‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
self-hosted-runner:
22
labels:
3-
- builder
4-
- func-tester
5-
- func-tester-aarch64
6-
- fuzzer-unit-tester
7-
- stress-tester
8-
- style-checker
9-
- style-checker-aarch64
3+
- altinity-on-demand
4+
- altinity-in-fsn1
5+
- altinity-in-ash
6+
- altinity-image-arm-app-docker-ce
7+
- altinity-image-x86-app-docker-ce
8+
- altinity-type-cax41
9+
- altinity-type-cpx41
10+
- altinity-type-cpx51
11+
- altinity-type-ccx53

‎.github/actions/docker_setup/action.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Docker setup
2+
description: Setup docker
3+
inputs:
4+
nested_job:
5+
description: the fuse for unintended use inside of the reusable callable jobs
6+
default: true
7+
type: boolean
8+
DOCKER_USERNAME:
9+
description: username for the dockerhub login
10+
required: true
11+
type: string
12+
DOCKER_PASSWORD:
13+
description: password for the dockerhub login
14+
required: true
15+
type: string
16+
runs:
17+
using: "composite"
18+
steps:
19+
- name: Docker IPv6 configuration
20+
shell: bash
21+
run: |
22+
# make sure docker uses proper IPv6 config
23+
sudo touch /etc/docker/daemon.json
24+
sudo chown ubuntu:ubuntu /etc/docker/daemon.json
25+
sudo cat <<EOT > /etc/docker/daemon.json
26+
{
27+
"ipv6": true,
28+
"fixed-cidr-v6": "2001:3984:3989::/64"
29+
}
30+
EOT
31+
sudo chown root:root /etc/docker/daemon.json
32+
sudo systemctl restart docker
33+
sudo systemctl status docker
34+
- name: Docker login
35+
shell: bash
36+
run: |
37+
docker login -u ${{ inputs.DOCKER_USERNAME }} -p ${{ inputs.DOCKER_PASSWORD }}
38+
docker info

‎.github/docker-ipv6-setup.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
sudo touch /etc/docker/daemon.json
3+
sudo chown ubuntu:ubuntu /etc/docker/daemon.json
4+
sudo cat <<EOT > /etc/docker/daemon.json
5+
{
6+
"ipv6": true,
7+
"fixed-cidr-v6": "2001:3984:3989::/64"
8+
}
9+
EOT
10+
sudo chown root:root /etc/docker/daemon.json
11+
sudo systemctl restart docker

‎.github/retry.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
# Execute command until exitcode is 0 or
3+
# maximum number of retries is reached
4+
# Example:
5+
# ./retry <retries> <delay> <command>
6+
retries=$1
7+
delay=$2
8+
command="${@:3}"
9+
exitcode=0
10+
try=0
11+
until [ "$try" -ge $retries ]
12+
do
13+
echo "$command"
14+
eval "$command"
15+
exitcode=$?
16+
if [ $exitcode -eq 0 ]; then
17+
break
18+
fi
19+
try=$((try+1))
20+
sleep $2
21+
done
22+
exit $exitcode

‎.github/workflows/README.md‎

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# 23.3 GitHub CI/CD Pipeline
2+
3+
## Release Test Fails
4+
| Job | Test Name | Reason | Resolution | Report | Detail |
5+
| --- | --------- | ------ | ---------- | ------ | ------ |
6+
| FunctionalStatelessMsan3 | 00534_functions_bad_arguments5 | Timeout | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__msan__[4_6].html | |
7+
| FunctionalStatelessMsan3 | 00534_functions_bad_arguments6 | Timeout | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__msan__[4_6].html | |
8+
| FunctionalStatelessMsan3 | 00534_functions_bad_arguments9 | Timeout | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__msan__[4_6].html | |
9+
| FunctionalStatelessMsan4 | 00534_functions_bad_arguments2 | Timeout | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__msan__[5_6].html | |
10+
| FunctionalStatelessMsan4 | 00534_functions_bad_arguments7 | Timeout | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__msan__[5_6].html | |
11+
| FunctionalStatelessS3Debug0 | 02479_mysql_connect_to_self | Fail to connect to MySQL server | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__debug__s3_storage__[1_6].html |
12+
| FunctionalStatelessDebug1 | 02561_null_as_default_more_formats | Fail to write to fail | Flaky | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__debug__[2_5].html | Passed on the previous run |
13+
| FunctionalStatelessDebug1 | 02703_jit_external_aggregation | Timeout | Consistent fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stateless_tests__debug__[2_5].html | |
14+
| StressTestAsan | | | | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stress_test__tsan_.html | |
15+
| StressTestTsan | | | | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/stress_test__tsan_.html | |
16+
| SQLancer (release) | ALL | settings used not supported on 23.3 | OK to fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/sqlancer__release_.html | |
17+
| SQLancer (debug) | ALL | settings used not supported on 23.3 | OK to fail | https://s3.amazonaws.com/altinity-build-artifacts/315/40a1d584f651d64851fd801cb147abb3de03378d/sqlancer__debug_.html | |
18+
19+
20+
## Comparison to ClickHouse
21+
Comparison between ClickHouse and Altinity CI/CD pipeline. Primary focus on the comparison is on testing and builds.
22+
ClickHouse workflows from https://github.com/ClickHouse/ClickHouse/tree/ff5101070e4732977e72d810f0e0ca1042dd4d3a.
23+
24+
### Master only
25+
26+
The following workflow files are included but they're only run on the master branch:
27+
- jepsen.yml
28+
- nightly.yml
29+
- master.yml
30+
- woboq.yml
31+
32+
### Not included workflows
33+
34+
The following workflow files are not included in the comparison as they are minor, not used, or do not provide any new tests relative to other workflows:
35+
- backport_branches.yml
36+
- cancel.yml
37+
- cherry_pick.yml
38+
- debug.yml
39+
- pull_request_approved.yml
40+
- release.yml
41+
- tags_stable.yml
42+
- docs_check.yml
43+
44+
### pull_request.yml
45+
46+
https://github.com/ClickHouse/ClickHouse/blob/23.3/.github/workflows/pull_request.yml
47+
48+
| Job | In altinity pipeline? |
49+
| ------ | ------------ |
50+
| CheckLabels | no, ClickHouse PR label management tool. |
51+
| PythonUnitTests | no, ClickHouse infrastructure tests. |
52+
| DockerHubPushAarch64 | yes |
53+
| DockerHubPushAmd64 | yes |
54+
| DockerHubPush | yes |
55+
| StyleCheck | yes |
56+
| FastTest | no, ClickHouse infrastructure tests. |
57+
| CompatibilityCheckX86 | yes |
58+
| CompatibilityCheckAarch64 | yes |
59+
| BuilderDebRelease | yes |
60+
| BuilderBinRelease | yes |
61+
| BuilderDebAarch64 | yes |
62+
| BuilderDebAsan | yes |
63+
| BuilderDebUBsan | yes |
64+
| BuilderDebTsan | yes |
65+
| BuilderDebMsan | yes |
66+
| BuilderDebDebug | yes |
67+
| BuilderBinClangTidy | no, we are focused on deb. |
68+
| BuilderBinDarwin | no, we are focused on deb. |
69+
| BuilderBinAarch64 | no, we are focused on deb. |
70+
| BuilderBinFreeBSD | no, we are focused on deb. |
71+
| BuilderBinDarwinAarch64 | no, we are focused on deb. |
72+
| BuilderBinPPC64| no, we are focused on deb. |
73+
| BuilderBinAmd64Compat | no, we are focused on deb. |
74+
| BuilderBinAarch64V80Compat | no, we are focused on deb. |
75+
| DockerServerImages | yes |
76+
| BuilderReport | yes |
77+
| BuilderSpecialReport | no, depends on bin builder stages. |
78+
| InstallPackagesTestRelease | yes |
79+
| InstallPackagesTestAarch64 | yes |
80+
| FunctionalStatelessTestRelease | yes |
81+
| FunctionalStatelessTestReleaseDatabaseReplicated0 | yes |
82+
| FunctionalStatelessTestReleaseDatabaseReplicated1 | yes |
83+
| FunctionalStatelessTestReleaseDatabaseReplicated2 | yes |
84+
| FunctionalStatelessTestReleaseDatabaseReplicated3 | yes |
85+
| FunctionalStatelessTestReleaseWideParts | yes |
86+
| FunctionalStatelessTestReleaseS3_0 | yes |
87+
| FunctionalStatelessTestReleaseS3_1 | yes |
88+
| FunctionalStatelessTestS3Debug0 | yes |
89+
| FunctionalStatelessTestS3Debug1 | yes |
90+
| FunctionalStatelessTestS3Debug2 | yes |
91+
| FunctionalStatelessTestS3Debug3 | yes |
92+
| FunctionalStatelessTestS3Debug4 | yes |
93+
| FunctionalStatelessTestS3Debug5 | yes |
94+
| FunctionalStatelessTestS3Tsan0 | yes |
95+
| FunctionalStatelessTestS3Tsan1 | yes |
96+
| FunctionalStatelessTestS3Tsan2 | yes |
97+
| FunctionalStatelessTestS3Tsan3 | yes |
98+
| FunctionalStatelessTestS3Tsan4 | yes |
99+
| FunctionalStatelessTestAarch64 | yes |
100+
| FunctionalStatelessTestAsan0 | yes |
101+
| FunctionalStatelessTestAsan1 | yes |
102+
| FunctionalStatelessTestAsan2 | yes |
103+
| FunctionalStatelessTestAsan3 | yes |
104+
| FunctionalStatelessTestTsan0 | yes |
105+
| FunctionalStatelessTestTsan1 | yes |
106+
| FunctionalStatelessTestTsan2 | yes |
107+
| FunctionalStatelessTestTsan3 | yes |
108+
| FunctionalStatelessTestTsan4 | yes |
109+
| FunctionalStatelessTestUBsan0 | yes |
110+
| FunctionalStatelessTestUBsan1 | yes |
111+
| FunctionalStatelessTestMsan0 | yes |
112+
| FunctionalStatelessTestMsan1 | yes |
113+
| FunctionalStatelessTestMsan2 | yes |
114+
| FunctionalStatelessTestMsan3 | yes |
115+
| FunctionalStatelessTestMsan4 | yes |
116+
| FunctionalStatelessTestMsan5 | yes |
117+
| FunctionalStatelessTestDebug0 | yes |
118+
| FunctionalStatelessTestDebug1 | yes |
119+
| FunctionalStatelessTestDebug2 | yes |
120+
| FunctionalStatelessTestDebug3 | yes |
121+
| FunctionalStatelessTestDebug4 | yes |
122+
| FunctionalStatelessTestFlakyCheck | yes |
123+
| TestsBugfixCheck | no, depends on CheckLabels |
124+
| FunctionalStatefulTestRelease | yes |
125+
| FunctionalStatefulTestAarch64 | yes |
126+
| FunctionalStatefulTestAsan | yes |
127+
| FunctionalStatefulTestTsan | yes |
128+
| FunctionalStatefulTestMsan | yes |
129+
| FunctionalStatefulTestUBsan | yes |
130+
| FunctionalStatefulTestDebug | yes |
131+
| FunctionalStatefulTestDebugParallelReplicas | yes |
132+
| FunctionalStatefulTestUBsanParallelReplicas | yes |
133+
| FunctionalStatefulTestMsanParallelReplicas | yes |
134+
| FunctionalStatefulTestTsanParallelReplicas | yes |
135+
| FunctionalStatefulTestAsanParallelReplicas | yes |
136+
| FunctionalStatefulTestReleaseParallelReplicas | yes |
137+
| StressTestAsan | yes |
138+
| StressTestTsan | yes |
139+
| StressTestMsan | yes |
140+
| StressTestUBsan | yes |
141+
| StressTestDebug | yes |
142+
| UpgradeCheckAsan | no, in development. |
143+
| UpgradeCheckTsan | no, in development. |
144+
| UpgradeCheckMsan | no, in development. |
145+
| UpgradeCheckDebug | no, in development. |
146+
| ASTFuzzerTestAsan | yes |
147+
| ASTFuzzerTestTsan | yes |
148+
| ASTFuzzerTestUBSan | yes |
149+
| ASTFuzzerTestMSan | yes |
150+
| ASTFuzzerTestDebug | yes |
151+
| IntegrationTestsAsan0 | yes |
152+
| IntegrationTestsAsan1 | yes |
153+
| IntegrationTestsAsan2 | yes |
154+
| IntegrationTestsAsan3 | yes |
155+
| IntegrationTestsAsan4 | yes |
156+
| IntegrationTestsAsan5 | yes |
157+
| IntegrationTestsTsan0 | yes |
158+
| IntegrationTestsTsan1 | yes |
159+
| IntegrationTestsTsan2 | yes |
160+
| IntegrationTestsTsan3 | yes |
161+
| IntegrationTestsTsan4 | yes |
162+
| IntegrationTestsTsan5 | yes |
163+
| IntegrationTestsRelease0 | yes |
164+
| IntegrationTestsRelease1 | yes |
165+
| IntegrationTestsRelease2 | yes |
166+
| IntegrationTestsRelease3 | yes |
167+
| IntegrationTestsFlakyCheck | yes |
168+
| UnitTestsAsan | yes |
169+
| UnitTestsReleaseClang | yes |
170+
| UnitTestsTsan | yes |
171+
| UnitTestsMsan | yes |
172+
| UnitTestsUBsan | yes |
173+
| PerformanceComparisonX86-0 | no, to be added. |
174+
| PerformanceComparisonX86-1 | no, to be added. |
175+
| PerformanceComparisonX86-2 | no, to be added. |
176+
| PerformanceComparisonX86-3 | no, to be added. |
177+
| PerformanceComparisonAarch-0 | no, to be added. |
178+
| PerformanceComparisonAarch-1 | no, to be added. |
179+
| PerformanceComparisonAarch-2 | no, to be added. |
180+
| PerformanceComparisonAarch-3 | no, to be added. |
181+
| SQLancerTestRelease | yes |
182+
| SQLancerTestDebug | yes |
183+
| Jepsen | no, in development. |
184+
185+
Jepsen job only runs if the pull request name containes `jepsen-test`
186+
187+
### release_branches.yml
188+
189+
https://github.com/ClickHouse/ClickHouse/blob/23.3/.github/workflows/release_branches.yml
190+
191+
| Job | In altinity pipeline? |
192+
| ------ | ------------ |
193+
| DockerHubPushAarch64 | yes |
194+
| DockerHubPushAmd64 | yes |
195+
| DockerHubPush | yes |
196+
| CompatibilityCheckX86 | yes |
197+
| CompatibilityCheckAarch64 | yes |
198+
| BuilderDebRelease | yes |
199+
| BuilderDebAarch64 | yes |
200+
| BuilderDebAsan | yes |
201+
| BuilderDebUBsan | yes |
202+
| BuilderDebTsan | yes |
203+
| BuilderDebMsan | yes |
204+
| BuilderDebDebug | yes |
205+
| BuilderBinDarwin | no, we are only focused on deb. |
206+
| BuilderBinDarwinAarch64 | no, we are only focused on deb. |
207+
| DockerServerImages | yes |
208+
| BuilderReport | yes |
209+
| BuilderSpecialReport | no, depends on bin builder stages. |
210+
| MarkReleaseReady | yes |
211+
| InstallPackagesTestRelease | yes |
212+
| InstallPackagesTestAarch64 | yes |
213+
| FunctionalStatelessTestRelease | yes |
214+
| FunctionalStatelessTestAarch64 | yes |
215+
| FunctionalStatelessTestAsan0 | yes |
216+
| FunctionalStatelessTestAsan1 | yes |
217+
| FunctionalStatelessTestTsan0 | yes |
218+
| FunctionalStatelessTestTsan1 | yes |
219+
| FunctionalStatelessTestTsan2 | yes |
220+
| FunctionalStatelessTestUBsan | yes |
221+
| FunctionalStatelessTestMsan0 | yes |
222+
| FunctionalStatelessTestMsan1 | yes |
223+
| FunctionalStatelessTestMsan2 | yes |
224+
| FunctionalStatelessTestDebug0 | yes |
225+
| FunctionalStatelessTestDebug1 | yes |
226+
| FunctionalStatelessTestDebug2 | yes |
227+
| FunctionalStatefulTestRelease | yes |
228+
| FunctionalStatefulTestAarch64 | yes |
229+
| FunctionalStatefulTestAsan | yes |
230+
| FunctionalStatefulTestTsan | yes |
231+
| FunctionalStatefulTestMsan | yes |
232+
| FunctionalStatefulTestUBsan | yes |
233+
| FunctionalStatefulTestDebug | yes |
234+
| StressTestAsan | yes |
235+
| StressTestTsan | yes |
236+
| StressTestMsan | yes |
237+
| StressTestUBsan | yes |
238+
| StressTestDebug | yes |
239+
| IntegrationTestsAsan0 | yes |
240+
| IntegrationTestsAsan1 | yes |
241+
| IntegrationTestsAsan2 | yes |
242+
| IntegrationTestsTsan0 | yes |
243+
| IntegrationTestsTsan1 | yes |
244+
| IntegrationTestsTsan2 | yes |
245+
| IntegrationTestsTsan3 | yes |
246+
| IntegrationTestsRelease0 | yes |
247+
| IntegrationTestsRelease1 | yes |
248+
| FinishCheck | yes |

‎.github/workflows/backport_branches.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -176,13 +176,13 @@ jobs:
176176
clear-repository: true
177177
fetch-depth: 0 # It MUST BE THE SAME for all dependencies and the job itself
178178
filter: tree:0
179-
- name: Check docker clickhouse/clickhouse-server building
179+
- name: Check docker altinityinfra/clickhouse-server building
180180
run: |
181181
cd "$GITHUB_WORKSPACE/tests/ci"
182182
python3 docker_server.py --release-type head --no-push \
183-
--image-repo clickhouse/clickhouse-server --image-path docker/server
183+
--image-repo altinityinfra/clickhouse-server --image-path docker/server
184184
python3 docker_server.py --release-type head --no-push \
185-
--image-repo clickhouse/clickhouse-keeper --image-path docker/keeper
185+
--image-repo altinityinfra/clickhouse-keeper --image-path docker/keeper
186186
- name: Cleanup
187187
if: always()
188188
run: |

0 commit comments

Comments
 (0)