[ISel] Improve clmul fallback implementation - #204802
Open
folkertdev wants to merge 3 commits into
Open
Conversation
to narrower and wider integers
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
🪟 Windows x64 Test Results
✅ The build succeeded and all tests passed. |
folkertdev
marked this pull request as ready for review
June 19, 2026 13:36
llvmorg-github-actions
Bot
added
backend:AArch64
backend:RISC-V
backend:PowerPC
backend:X86
llvm:SelectionDAG
|
@llvm/pr-subscribers-llvm-analysis @llvm/pr-subscribers-backend-x86 Author: Folkert de Vries (folkertdev) ChangesGeneralize the approach from #203727 to narrower and wider integers. We still need the fallback for when multiplication isn't available, and it turns out that for some widths the fallback emits fewer instructions, the naive fallback is still used for Based on my local experiments, the Karatsuba approach (e.g. as in rust-lang/rust#152132 (comment)) is not actually better than zero extending the input and using multiplication with holes on the wider type. CC #203694 Patch is 1.41 MiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/204802.diff 18 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 0090bdcd4b55b..277dcfa6e30eb 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -3100,12 +3100,22 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
InstructionCost MulCost =
thisT()->getArithmeticInstrCost(Instruction::Mul, RetTy, CostKind);
- // When the multiplication with holes approach is used, that emits 16
- // MULs, 8 + 4 ANDs, 12 XORs and 3 ORs.
- if (BW >= 32 && BW <= 64 &&
- TLI->isOperationLegalOrCustom(ISD::MUL,
- TLI->getValueType(DL, RetTy))) {
- return 16 * MulCost + 12 * AndCost + 12 * XorCost + 3 * OrCost;
+ // When the multiplication with holes approach is used, it splits the
+ // operands into S phases (the smallest stride with ceil(BW/S) <= 2^S) and
+ // emits S*S MULs, 3*S ANDs, S*(S-1) XORs and S-1 ORs.
+ //
+ // * BW <= 8 uses S = 2
+ // * BW <= 24 uses S = 3
+ // * BW <= 64 uses S = 4
+ // * BW <= 160 uses S = 5
+ // * BW <= 384 uses S = 6
+ unsigned S = 1;
+ while (S < 32 && divideCeil(BW, S) > (1u << S))
+ ++S;
+ if (S * S < BW && TLI->isOperationLegalOrCustom(
+ ISD::MUL, TLI->getValueType(DL, RetTy))) {
+ return S * S * MulCost + 3 * S * AndCost + S * (S - 1) * XorCost +
+ (S - 1) * OrCost;
}
InstructionCost PerBitCostMul = AndCost + MulCost + XorCost;
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index 5ba36495ba4f6..a7b926ae43905 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -8903,37 +8903,94 @@ SDValue TargetLowering::expandCLMUL(SDNode *Node, SelectionDAG &DAG) const {
// do occur, they wind up in a "hole" and are subsequently masked out of the
// result.
//
- // A hole of 3 bits is optimal for 32-bit and 64-bit inputs. 128-bit
- // integers need a larger hole, and for smaller integers the fallback below
- // is more efficient.
+ // https://www.bearssl.org/constanttime.html#ghash-for-gcm describes this
+ // approach.
+
+ // Stride S handles operands up to S·2^S bits using S² multiplies.
+ //
+ // * BW <= 8 uses S = 2 (holes of 1 bit)
+ // * BW <= 24 uses S = 3 (holes of 2 bits)
+ // * BW <= 64 uses S = 4 (holes of 3 bits)
+ // * BW <= 160 uses S = 5 (holes of 4 bits)
+ // * BW <= 384 uses S = 6 (holes of 5 bits)
+ //
+ // We distribute the BW bits over S phases:
+ //
+ // phase 0 keeps bits: 0, S, 2S, ...
+ // phase 1 keeps bits: 1, S + 1, 2S + 1, ...
+ // ...
+ //
+ // Each phase has up to n = ceil(BW / S) bits set, and the holes are S-1
+ // bits wide.
+ //
+ // Take BW = 4, S = 2, n = 2. The worst case is a fully populated phase (all
+ // non-hole bits are set to 1) multiplied by itself, 0b0101 * 0b0101. Each
+ // set bit of one operand shifts a copy of the other, and we add the copies:
+ //
+ // col: 4 3 2 1 0
+ // 0b0101 << 0: 0 0 1 0 1
+ // 0b0101 << 2: 1 0 1 0 0
+ // ----------------- +
+ // count: 1 0 2 0 1
+ //
+ // Counting the number of one-bits in each column gives a triangle: the
+ // counts climb 1, 2, ..., n and back down (here 1, 2, 1 across the data
+ // columns). So a column holds at most n one-bits, and that maximum n is
+ // reached in only one column: the peak. Every other column holds at most n
+ // - 1 one-bits.
+ //
+ // A stack of one-bits in a column turns into carries: column 2 above really
+ // stores the value 1 + 1 = 2 = n. A column spans S bits, its kept bit
+ // plus S-1 hole bits, and the count is written from the kept bit upward,
+ // so any count <= 2^S - 1 stays within the column and never interferes with
+ // the next data bit S positions up. Every non-peak column holds at most n -
+ // 1, so they all fit as soon as n - 1 <= 2^S - 1.
+ //
+ // That leaves only the peak column. Because both operands set all data
+ // bits, the triangle peaks at the top of the word at the highest data bit
+ // still inside BW. Here the count reaches exactly n = 2^S and overflows.
+ // But its carry lands at bit n*S >= BW, off the top, where it (and the
+ // whole descending half of the triangle) is truncated.
+ //
+ // Hence the holes suffice exactly when n = ceil(BW / S) <= 2^S, i.e. BW <=
+ // S*2^S.
//
- // Based on bmul64 in bearssl and bmul in the rust polyval crate.
- if (BW >= 32 && BW <= 64 &&
+ // Here we find the smallest S that satisfies this inequality.
+ unsigned S = 1;
+ while (S < 32 && divideCeil(BW, S) > (1u << S))
+ ++S;
+
+ // Continue to use the naive fallback below if it seems cheaper. We compare
+ // the number of multiplications here (S * S) versus the number of
+ // iterations there (BW). The naive fallback is still used for i1, i3, i4
+ // and i9, and when multiplication isn't available.
+ if (S * S < BW &&
isOperationLegalOrCustom(ISD::MUL, getTypeToTransformTo(Ctx, VT))) {
- // Set every fourth bit of each nibble, equivalent to 0b00010001...0001.
- APInt MaskVal = APInt::getSplat(BW, APInt(4, 0b0001));
+ // Set a bit every S positions, e.g. for S = 4 this is equivalent to
+ // 0b...00010001...0001.
+ APInt MaskVal = APInt::getSplat(BW, APInt(S, 1));
- // Create versions of X and Y that keep only the I-th bit of
- // each nibble.
- SDValue M[4], Xp[4], Yp[4];
- for (unsigned I = 0; I < 4; ++I) {
+ // Create versions of X and Y that keep only the I-th bit of each S-bit
+ // slice.
+ SmallVector<SDValue, 4> M(S), Xp(S), Yp(S);
+ for (unsigned I = 0; I < S; ++I) {
M[I] = DAG.getConstant(MaskVal.shl(I), DL, VT);
Xp[I] = DAG.getNode(ISD::AND, DL, VT, X, M[I]);
Yp[I] = DAG.getNode(ISD::AND, DL, VT, Y, M[I]);
}
- // Codegens these expressions (16 multiplications):
+ // Codegens these expressions (S*S multiplications), e.g. for S=4:
//
// z0 = (x0 * y0) ^ (x1 * y3) ^ (x2 * y2) ^ (x3 * y1);
// z1 = (x0 * y1) ^ (x1 * y0) ^ (x2 * y3) ^ (x3 * y2);
// z2 = (x0 * y2) ^ (x1 * y1) ^ (x2 * y0) ^ (x3 * y3);
// z3 = (x0 * y3) ^ (x1 * y2) ^ (x2 * y1) ^ (x3 * y0);
SDValue Res = DAG.getConstant(0, DL, VT);
- for (unsigned I = 0; I < 4; ++I) {
+ for (unsigned I = 0; I < S; ++I) {
SDValue Zi = DAG.getConstant(0, DL, VT);
- for (unsigned J = 0; J < 4; ++J) {
- unsigned K = (I + 4 - J) % 4;
+ for (unsigned J = 0; J < S; ++J) {
+ unsigned K = (I + S - J) % S;
SDValue P = DAG.getNode(ISD::MUL, DL, VT, Xp[J], Yp[K]);
Zi = DAG.getNode(ISD::XOR, DL, VT, Zi, P);
}
diff --git a/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll b/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
index 9a40c6a915435..8594e24ac7354 100644
--- a/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
@@ -10,8 +10,8 @@ define void @clmul_fixed() {
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c1 = call <1 x i64> @llvm.clmul.v1i64(<1 x i64> poison, <1 x i64> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> poison, <4 x i32> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c2 = call <2 x i32> @llvm.clmul.v2i32(<2 x i32> poison, <2 x i32> poison)
-; NOAES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
-; NOAES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
+; NOAES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
+; NOAES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; AES-LABEL: 'clmul_fixed'
@@ -21,8 +21,8 @@ define void @clmul_fixed() {
; AES-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %c1 = call <1 x i64> @llvm.clmul.v1i64(<1 x i64> poison, <1 x i64> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> poison, <4 x i32> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %c2 = call <2 x i32> @llvm.clmul.v2i32(<2 x i32> poison, <2 x i32> poison)
-; AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
-; AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
+; AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
+; AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%c8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> poison, <16 x i8> poison)
diff --git a/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll b/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
index b8df50e351fa4..4705669c2207c 100644
--- a/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
@@ -8,15 +8,15 @@
define void @clmul_scalable() {
; SVE-LABEL: 'clmul_scalable'
-; SVE-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
-; SVE-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
+; SVE-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
+; SVE-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <vscale x 4 x i32> @llvm.clmul.nxv4i32(<vscale x 4 x i32> poison, <vscale x 4 x i32> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 192 for instruction: %c64 = call <vscale x 2 x i64> @llvm.clmul.nxv2i64(<vscale x 2 x i64> poison, <vscale x 2 x i64> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; SVE-AES-LABEL: 'clmul_scalable'
-; SVE-AES-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
-; SVE-AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
+; SVE-AES-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
+; SVE-AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <vscale x 4 x i32> @llvm.clmul.nxv4i32(<vscale x 4 x i32> poison, <vscale x 4 x i32> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %c64 = call <vscale x 2 x i64> @llvm.clmul.nxv2i64(<vscale x 2 x i64> poison, <vscale x 2 x i64> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
diff --git a/llvm/test/Analysis/CostModel/X86/clmul.ll b/llvm/test/Analysis/CostModel/X86/clmul.ll
index 8e5c71b311ea9..15ce73f5d0b8d 100644
--- a/llvm/test/Analysis/CostModel/X86/clmul.ll
+++ b/llvm/test/Analysis/CostModel/X86/clmul.ll
@@ -16,8 +16,8 @@ define void @clmul(i128 %a128, i128 %b128, i64 %a64, i64 %b64, i32 %a32, i32 %b3
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 768 for instruction: %call_i128 = call i128 @llvm.clmul.i128(i128 %a128, i128 %b128)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 59 for instruction: %call_i64 = call i64 @llvm.clmul.i64(i64 %a64, i64 %b64)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %call_i32 = call i32 @llvm.clmul.i32(i32 %a32, i32 %b32)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call i16 @llvm.clmul.i16(i16 %a16, i16 %b16)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %call_i8 = call i8 @llvm.clmul.i8(i8 %a8, i8 %b8)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 35 for instruction: %call_i16 = call i16 @llvm.clmul.i16(i16 %a16, i16 %b16)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 21 for instruction: %call_i8 = call i8 @llvm.clmul.i8(i8 %a8, i8 %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%call_i128 = call i128 @llvm.clmul.i128(i128 %a128, i128 %b128)
@@ -33,16 +33,16 @@ define void @clmul_128(<1 x i128> %a128, <1 x i128> %b128, <2 x i64> %a64, <2 x
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %call_i64 = call <2 x i64> @llvm.clmul.v2i64(<2 x i64> %a64, <2 x i64> %b64)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %call_i32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> %a32, <4 x i32> %b32)
-; PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
-; PCLMUL-NEXT: Cost Model: Found an estimated cost of 40 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
+; PCLMUL-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
+; PCLMUL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; NO-PCLMUL-LABEL: 'clmul_128'
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 768 for instruction: %call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 139 for instruction: %call_i64 = call <2 x i64> @llvm.clmul.v2i64(<2 x i64> %a64, <2 x i64> %b64)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 123 for instruction: %call_i32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> %a32, <4 x i32> %b32)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 40 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
diff --git a/llvm/test/CodeGen/AArch64/clmul-scalable.ll b/llvm/test/CodeGen/AArch64/clmul-scalable.ll
index 385f58639beba..2a2ad5c4223c8 100644
--- a/llvm/test/CodeGen/AArch64/clmul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/clmul-scalable.ll
@@ -10,71 +10,45 @@ define <vscale x 16 x i8> @clmul_nxv16i8(<vscale x 16 x i8> %x, <vscale x 16 x i
; CHECK-SVE-LABEL: clmul_nxv16i8:
; CHECK-SVE: // %bb.0:
; CHECK-SVE-NEXT: movprfx z2, z1
-; CHECK-SVE-NEXT: and z2.b, z2.b, #0x2
-; CHECK-SVE-NEXT: movprfx z3, z1
-; CHECK-SVE-NEXT: and z3.b, z3.b, #0x1
-; CHECK-SVE-NEXT: movprfx z4, z1
-; CHECK-SVE-NEXT: and z4.b, z4.b, #0x4
-; CHECK-SVE-NEXT: movprfx z5, z1
-; CHECK-SVE-NEXT: and z5.b, z5.b, #0x8
-; CHECK-SVE-NEXT: movprfx z6, z1
-; CHECK-SVE-NEXT: and z6.b, z6.b, #0x10
-; CHECK-SVE-NEXT: movprfx z7, z1
-; CHECK-SVE-NEXT: and z7.b, z7.b, #0x20
+; CHECK-SVE-NEXT: and z2.b, z2.b, #0x55
+; CHECK-SVE-NEXT: movprfx z3, z0
+; CHECK-SVE-NEXT: and z3.b, z3.b, #0xaa
+; CHECK-SVE-NEXT: and z1.b, z1.b, #0xaa
+; CHECK-SVE-NEXT: and z0.b, z0.b, #0x55
; CHECK-SVE-NEXT: ptrue p0.b
-; CHECK-SVE-NEXT: movprfx z24, z1
-; CHECK-SVE-NEXT: and z24.b, z24.b, #0x40
-; CHECK-SVE-NEXT: and z1.b, z1.b, #0x80
-; CHECK-SVE-NEXT: mul z2.b, p0/m, z2.b, z0.b
-; CHECK-SVE-NEXT: mul z3.b, p0/m, z3.b, z0.b
-; CHECK-SVE-NEXT: mul z4.b, p0/m, z4.b, z0.b
-; CHECK-SVE-NEXT: mul z5.b, p0/m, z5.b, z0.b
-; CHECK-SVE-NEXT: mul z6.b, p0/m, z6.b, z0.b
-; CHECK-SVE-NEXT: mul z7.b, p0/m, z7.b, z0.b
-; CHECK-SVE-NEXT: mul z24.b, p0/m, z24.b, z0.b
-; CHECK-SVE-NEXT: mul z0.b, p0/m, z0.b, z1.b
-; CHECK-SVE-NEXT: eor z2.d, z3.d, z2.d
-; CHECK-SVE-NEXT: eor z3.d, z4.d, z5.d
-; CHECK-SVE-NEXT: eor z4.d, z6.d, z7.d
-; CHECK-SVE-NEXT: eor z2.d, z2.d, z3.d
-; CHECK-SVE-NEXT: eor z3.d, z4.d, z24.d
-; CHECK-SVE-NEXT: eor z1.d, z2.d, z3.d
-; CHECK-SVE-NEXT: eor z0.d, z1.d, z0.d
+; CHECK-SVE-NEXT: movprfx z4, z3
+; CHECK-SVE-NEXT: mul z4.b, p0/m, z4.b, z2.b
+; CHECK-SVE-NEXT: movprfx z5, z0
+; CHECK-SVE-NEXT: mul z5.b, p0/m, z5.b, z1.b
+; CHECK-SVE-NEXT: mul z1.b, p0/m, z1.b, z3.b
+; CHECK-SVE-NEXT: mul z0.b, p0/m, z0.b, z2.b
+; CHECK-SVE-NEXT: eor z2.d, z5.d, z4.d
+; CHECK-SVE-NEXT: eor z0.d, z0.d, z1.d
+; CHECK-SVE-NEXT: and z2.b, z2.b, #0xaa
+; CHECK-SVE-NEXT: and z0.b, z0.b, #0x55
+; CHECK-SVE-NEXT: orr z0.d, z0.d, z2.d
; CHECK-SVE-NEXT: ret
;
; CHECK-SVE-AES-LABEL: clmul_nxv16i8:
; CHECK-SVE-AES: // %bb.0:
; CHECK-SVE-AES-NEXT: movprfx z2, z1
-; CHECK-SVE-AES-NEXT: and z2.b, z2.b, #0x2
-; CHECK-SVE-AES-NEXT: movprfx z3, z1
-; CHECK-SVE-AES-NEXT: and z3.b, z3.b, #0x1
-; CHECK-SVE-AES-NEXT: movprfx z4, z1
-; CHECK-SVE-AES-NEXT: and z4.b, z4.b, #0x4
-; CHECK-SVE-AES-NEXT: movprfx z5, z1
-; CHECK-SVE-AES-NEXT: and z5.b, z5.b, #0x8
-; CHECK-SVE-AES-NEXT: movprfx z6, z1
-; CHECK-SVE-AES-NEXT: and z6.b, z6.b, #0x10
-; CHECK-SVE-AES-NEXT: movprfx z7, z1
-; CHECK-SVE-AES-NEXT: and z7.b, z7.b, #0x20
+; CHECK-SVE-AES-NEXT: and z2.b, z2.b, #0x55
+; CHECK-SVE-AES-NEXT: movprfx z3, z0
+; CHECK-SVE-AES-NEXT: and z3.b, z3.b, #0xaa
+; CHECK-SVE-AES-NEXT: and z1.b, z1.b, #0xaa
+; CHECK-SVE-AES-NEXT: and z0.b, z0.b, #0x55
; CHECK-SVE-AES-NEXT: ptrue p0.b
-; CHECK-SVE-AES-NEXT: movprfx z24, z1
-; CHECK-SVE-AES-NEXT: and z24.b, z24.b, #0x40
-; CHECK-SVE-AES-NEXT: and z1.b, z1.b, #0x80
-; CHECK-SVE-AES-NEXT: mul z2.b, p0/m, z2.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z3.b, p0/m, z3.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z4.b, p0/m, z4.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z5.b, p0/m, z5.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z6.b, p0/m, z6.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z7.b, p0/m, z7.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z24.b, p0/m, z24.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z0.b, p0/m, z0.b, z1.b
-; CH...
[truncated]
|
|
@llvm/pr-subscribers-backend-powerpc Author: Folkert de Vries (folkertdev) ChangesGeneralize the approach from #203727 to narrower and wider integers. We still need the fallback for when multiplication isn't available, and it turns out that for some widths the fallback emits fewer instructions, the naive fallback is still used for Based on my local experiments, the Karatsuba approach (e.g. as in rust-lang/rust#152132 (comment)) is not actually better than zero extending the input and using multiplication with holes on the wider type. CC #203694 Patch is 1.41 MiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/204802.diff 18 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 0090bdcd4b55b..277dcfa6e30eb 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -3100,12 +3100,22 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
InstructionCost MulCost =
thisT()->getArithmeticInstrCost(Instruction::Mul, RetTy, CostKind);
- // When the multiplication with holes approach is used, that emits 16
- // MULs, 8 + 4 ANDs, 12 XORs and 3 ORs.
- if (BW >= 32 && BW <= 64 &&
- TLI->isOperationLegalOrCustom(ISD::MUL,
- TLI->getValueType(DL, RetTy))) {
- return 16 * MulCost + 12 * AndCost + 12 * XorCost + 3 * OrCost;
+ // When the multiplication with holes approach is used, it splits the
+ // operands into S phases (the smallest stride with ceil(BW/S) <= 2^S) and
+ // emits S*S MULs, 3*S ANDs, S*(S-1) XORs and S-1 ORs.
+ //
+ // * BW <= 8 uses S = 2
+ // * BW <= 24 uses S = 3
+ // * BW <= 64 uses S = 4
+ // * BW <= 160 uses S = 5
+ // * BW <= 384 uses S = 6
+ unsigned S = 1;
+ while (S < 32 && divideCeil(BW, S) > (1u << S))
+ ++S;
+ if (S * S < BW && TLI->isOperationLegalOrCustom(
+ ISD::MUL, TLI->getValueType(DL, RetTy))) {
+ return S * S * MulCost + 3 * S * AndCost + S * (S - 1) * XorCost +
+ (S - 1) * OrCost;
}
InstructionCost PerBitCostMul = AndCost + MulCost + XorCost;
diff --git a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
index 5ba36495ba4f6..a7b926ae43905 100644
--- a/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
@@ -8903,37 +8903,94 @@ SDValue TargetLowering::expandCLMUL(SDNode *Node, SelectionDAG &DAG) const {
// do occur, they wind up in a "hole" and are subsequently masked out of the
// result.
//
- // A hole of 3 bits is optimal for 32-bit and 64-bit inputs. 128-bit
- // integers need a larger hole, and for smaller integers the fallback below
- // is more efficient.
+ // https://www.bearssl.org/constanttime.html#ghash-for-gcm describes this
+ // approach.
+
+ // Stride S handles operands up to S·2^S bits using S² multiplies.
+ //
+ // * BW <= 8 uses S = 2 (holes of 1 bit)
+ // * BW <= 24 uses S = 3 (holes of 2 bits)
+ // * BW <= 64 uses S = 4 (holes of 3 bits)
+ // * BW <= 160 uses S = 5 (holes of 4 bits)
+ // * BW <= 384 uses S = 6 (holes of 5 bits)
+ //
+ // We distribute the BW bits over S phases:
+ //
+ // phase 0 keeps bits: 0, S, 2S, ...
+ // phase 1 keeps bits: 1, S + 1, 2S + 1, ...
+ // ...
+ //
+ // Each phase has up to n = ceil(BW / S) bits set, and the holes are S-1
+ // bits wide.
+ //
+ // Take BW = 4, S = 2, n = 2. The worst case is a fully populated phase (all
+ // non-hole bits are set to 1) multiplied by itself, 0b0101 * 0b0101. Each
+ // set bit of one operand shifts a copy of the other, and we add the copies:
+ //
+ // col: 4 3 2 1 0
+ // 0b0101 << 0: 0 0 1 0 1
+ // 0b0101 << 2: 1 0 1 0 0
+ // ----------------- +
+ // count: 1 0 2 0 1
+ //
+ // Counting the number of one-bits in each column gives a triangle: the
+ // counts climb 1, 2, ..., n and back down (here 1, 2, 1 across the data
+ // columns). So a column holds at most n one-bits, and that maximum n is
+ // reached in only one column: the peak. Every other column holds at most n
+ // - 1 one-bits.
+ //
+ // A stack of one-bits in a column turns into carries: column 2 above really
+ // stores the value 1 + 1 = 2 = n. A column spans S bits, its kept bit
+ // plus S-1 hole bits, and the count is written from the kept bit upward,
+ // so any count <= 2^S - 1 stays within the column and never interferes with
+ // the next data bit S positions up. Every non-peak column holds at most n -
+ // 1, so they all fit as soon as n - 1 <= 2^S - 1.
+ //
+ // That leaves only the peak column. Because both operands set all data
+ // bits, the triangle peaks at the top of the word at the highest data bit
+ // still inside BW. Here the count reaches exactly n = 2^S and overflows.
+ // But its carry lands at bit n*S >= BW, off the top, where it (and the
+ // whole descending half of the triangle) is truncated.
+ //
+ // Hence the holes suffice exactly when n = ceil(BW / S) <= 2^S, i.e. BW <=
+ // S*2^S.
//
- // Based on bmul64 in bearssl and bmul in the rust polyval crate.
- if (BW >= 32 && BW <= 64 &&
+ // Here we find the smallest S that satisfies this inequality.
+ unsigned S = 1;
+ while (S < 32 && divideCeil(BW, S) > (1u << S))
+ ++S;
+
+ // Continue to use the naive fallback below if it seems cheaper. We compare
+ // the number of multiplications here (S * S) versus the number of
+ // iterations there (BW). The naive fallback is still used for i1, i3, i4
+ // and i9, and when multiplication isn't available.
+ if (S * S < BW &&
isOperationLegalOrCustom(ISD::MUL, getTypeToTransformTo(Ctx, VT))) {
- // Set every fourth bit of each nibble, equivalent to 0b00010001...0001.
- APInt MaskVal = APInt::getSplat(BW, APInt(4, 0b0001));
+ // Set a bit every S positions, e.g. for S = 4 this is equivalent to
+ // 0b...00010001...0001.
+ APInt MaskVal = APInt::getSplat(BW, APInt(S, 1));
- // Create versions of X and Y that keep only the I-th bit of
- // each nibble.
- SDValue M[4], Xp[4], Yp[4];
- for (unsigned I = 0; I < 4; ++I) {
+ // Create versions of X and Y that keep only the I-th bit of each S-bit
+ // slice.
+ SmallVector<SDValue, 4> M(S), Xp(S), Yp(S);
+ for (unsigned I = 0; I < S; ++I) {
M[I] = DAG.getConstant(MaskVal.shl(I), DL, VT);
Xp[I] = DAG.getNode(ISD::AND, DL, VT, X, M[I]);
Yp[I] = DAG.getNode(ISD::AND, DL, VT, Y, M[I]);
}
- // Codegens these expressions (16 multiplications):
+ // Codegens these expressions (S*S multiplications), e.g. for S=4:
//
// z0 = (x0 * y0) ^ (x1 * y3) ^ (x2 * y2) ^ (x3 * y1);
// z1 = (x0 * y1) ^ (x1 * y0) ^ (x2 * y3) ^ (x3 * y2);
// z2 = (x0 * y2) ^ (x1 * y1) ^ (x2 * y0) ^ (x3 * y3);
// z3 = (x0 * y3) ^ (x1 * y2) ^ (x2 * y1) ^ (x3 * y0);
SDValue Res = DAG.getConstant(0, DL, VT);
- for (unsigned I = 0; I < 4; ++I) {
+ for (unsigned I = 0; I < S; ++I) {
SDValue Zi = DAG.getConstant(0, DL, VT);
- for (unsigned J = 0; J < 4; ++J) {
- unsigned K = (I + 4 - J) % 4;
+ for (unsigned J = 0; J < S; ++J) {
+ unsigned K = (I + S - J) % S;
SDValue P = DAG.getNode(ISD::MUL, DL, VT, Xp[J], Yp[K]);
Zi = DAG.getNode(ISD::XOR, DL, VT, Zi, P);
}
diff --git a/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll b/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
index 9a40c6a915435..8594e24ac7354 100644
--- a/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
@@ -10,8 +10,8 @@ define void @clmul_fixed() {
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c1 = call <1 x i64> @llvm.clmul.v1i64(<1 x i64> poison, <1 x i64> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> poison, <4 x i32> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c2 = call <2 x i32> @llvm.clmul.v2i32(<2 x i32> poison, <2 x i32> poison)
-; NOAES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
-; NOAES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
+; NOAES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
+; NOAES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; AES-LABEL: 'clmul_fixed'
@@ -21,8 +21,8 @@ define void @clmul_fixed() {
; AES-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %c1 = call <1 x i64> @llvm.clmul.v1i64(<1 x i64> poison, <1 x i64> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> poison, <4 x i32> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %c2 = call <2 x i32> @llvm.clmul.v2i32(<2 x i32> poison, <2 x i32> poison)
-; AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
-; AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
+; AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
+; AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%c8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> poison, <16 x i8> poison)
diff --git a/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll b/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
index b8df50e351fa4..4705669c2207c 100644
--- a/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
+++ b/llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
@@ -8,15 +8,15 @@
define void @clmul_scalable() {
; SVE-LABEL: 'clmul_scalable'
-; SVE-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
-; SVE-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
+; SVE-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
+; SVE-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <vscale x 4 x i32> @llvm.clmul.nxv4i32(<vscale x 4 x i32> poison, <vscale x 4 x i32> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 192 for instruction: %c64 = call <vscale x 2 x i64> @llvm.clmul.nxv2i64(<vscale x 2 x i64> poison, <vscale x 2 x i64> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; SVE-AES-LABEL: 'clmul_scalable'
-; SVE-AES-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
-; SVE-AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
+; SVE-AES-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
+; SVE-AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <vscale x 4 x i32> @llvm.clmul.nxv4i32(<vscale x 4 x i32> poison, <vscale x 4 x i32> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %c64 = call <vscale x 2 x i64> @llvm.clmul.nxv2i64(<vscale x 2 x i64> poison, <vscale x 2 x i64> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
diff --git a/llvm/test/Analysis/CostModel/X86/clmul.ll b/llvm/test/Analysis/CostModel/X86/clmul.ll
index 8e5c71b311ea9..15ce73f5d0b8d 100644
--- a/llvm/test/Analysis/CostModel/X86/clmul.ll
+++ b/llvm/test/Analysis/CostModel/X86/clmul.ll
@@ -16,8 +16,8 @@ define void @clmul(i128 %a128, i128 %b128, i64 %a64, i64 %b64, i32 %a32, i32 %b3
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 768 for instruction: %call_i128 = call i128 @llvm.clmul.i128(i128 %a128, i128 %b128)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 59 for instruction: %call_i64 = call i64 @llvm.clmul.i64(i64 %a64, i64 %b64)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %call_i32 = call i32 @llvm.clmul.i32(i32 %a32, i32 %b32)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call i16 @llvm.clmul.i16(i16 %a16, i16 %b16)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %call_i8 = call i8 @llvm.clmul.i8(i8 %a8, i8 %b8)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 35 for instruction: %call_i16 = call i16 @llvm.clmul.i16(i16 %a16, i16 %b16)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 21 for instruction: %call_i8 = call i8 @llvm.clmul.i8(i8 %a8, i8 %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%call_i128 = call i128 @llvm.clmul.i128(i128 %a128, i128 %b128)
@@ -33,16 +33,16 @@ define void @clmul_128(<1 x i128> %a128, <1 x i128> %b128, <2 x i64> %a64, <2 x
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %call_i64 = call <2 x i64> @llvm.clmul.v2i64(<2 x i64> %a64, <2 x i64> %b64)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %call_i32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> %a32, <4 x i32> %b32)
-; PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
-; PCLMUL-NEXT: Cost Model: Found an estimated cost of 40 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
+; PCLMUL-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
+; PCLMUL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; NO-PCLMUL-LABEL: 'clmul_128'
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 768 for instruction: %call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 139 for instruction: %call_i64 = call <2 x i64> @llvm.clmul.v2i64(<2 x i64> %a64, <2 x i64> %b64)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 123 for instruction: %call_i32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> %a32, <4 x i32> %b32)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
-; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 40 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
+; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
diff --git a/llvm/test/CodeGen/AArch64/clmul-scalable.ll b/llvm/test/CodeGen/AArch64/clmul-scalable.ll
index 385f58639beba..2a2ad5c4223c8 100644
--- a/llvm/test/CodeGen/AArch64/clmul-scalable.ll
+++ b/llvm/test/CodeGen/AArch64/clmul-scalable.ll
@@ -10,71 +10,45 @@ define <vscale x 16 x i8> @clmul_nxv16i8(<vscale x 16 x i8> %x, <vscale x 16 x i
; CHECK-SVE-LABEL: clmul_nxv16i8:
; CHECK-SVE: // %bb.0:
; CHECK-SVE-NEXT: movprfx z2, z1
-; CHECK-SVE-NEXT: and z2.b, z2.b, #0x2
-; CHECK-SVE-NEXT: movprfx z3, z1
-; CHECK-SVE-NEXT: and z3.b, z3.b, #0x1
-; CHECK-SVE-NEXT: movprfx z4, z1
-; CHECK-SVE-NEXT: and z4.b, z4.b, #0x4
-; CHECK-SVE-NEXT: movprfx z5, z1
-; CHECK-SVE-NEXT: and z5.b, z5.b, #0x8
-; CHECK-SVE-NEXT: movprfx z6, z1
-; CHECK-SVE-NEXT: and z6.b, z6.b, #0x10
-; CHECK-SVE-NEXT: movprfx z7, z1
-; CHECK-SVE-NEXT: and z7.b, z7.b, #0x20
+; CHECK-SVE-NEXT: and z2.b, z2.b, #0x55
+; CHECK-SVE-NEXT: movprfx z3, z0
+; CHECK-SVE-NEXT: and z3.b, z3.b, #0xaa
+; CHECK-SVE-NEXT: and z1.b, z1.b, #0xaa
+; CHECK-SVE-NEXT: and z0.b, z0.b, #0x55
; CHECK-SVE-NEXT: ptrue p0.b
-; CHECK-SVE-NEXT: movprfx z24, z1
-; CHECK-SVE-NEXT: and z24.b, z24.b, #0x40
-; CHECK-SVE-NEXT: and z1.b, z1.b, #0x80
-; CHECK-SVE-NEXT: mul z2.b, p0/m, z2.b, z0.b
-; CHECK-SVE-NEXT: mul z3.b, p0/m, z3.b, z0.b
-; CHECK-SVE-NEXT: mul z4.b, p0/m, z4.b, z0.b
-; CHECK-SVE-NEXT: mul z5.b, p0/m, z5.b, z0.b
-; CHECK-SVE-NEXT: mul z6.b, p0/m, z6.b, z0.b
-; CHECK-SVE-NEXT: mul z7.b, p0/m, z7.b, z0.b
-; CHECK-SVE-NEXT: mul z24.b, p0/m, z24.b, z0.b
-; CHECK-SVE-NEXT: mul z0.b, p0/m, z0.b, z1.b
-; CHECK-SVE-NEXT: eor z2.d, z3.d, z2.d
-; CHECK-SVE-NEXT: eor z3.d, z4.d, z5.d
-; CHECK-SVE-NEXT: eor z4.d, z6.d, z7.d
-; CHECK-SVE-NEXT: eor z2.d, z2.d, z3.d
-; CHECK-SVE-NEXT: eor z3.d, z4.d, z24.d
-; CHECK-SVE-NEXT: eor z1.d, z2.d, z3.d
-; CHECK-SVE-NEXT: eor z0.d, z1.d, z0.d
+; CHECK-SVE-NEXT: movprfx z4, z3
+; CHECK-SVE-NEXT: mul z4.b, p0/m, z4.b, z2.b
+; CHECK-SVE-NEXT: movprfx z5, z0
+; CHECK-SVE-NEXT: mul z5.b, p0/m, z5.b, z1.b
+; CHECK-SVE-NEXT: mul z1.b, p0/m, z1.b, z3.b
+; CHECK-SVE-NEXT: mul z0.b, p0/m, z0.b, z2.b
+; CHECK-SVE-NEXT: eor z2.d, z5.d, z4.d
+; CHECK-SVE-NEXT: eor z0.d, z0.d, z1.d
+; CHECK-SVE-NEXT: and z2.b, z2.b, #0xaa
+; CHECK-SVE-NEXT: and z0.b, z0.b, #0x55
+; CHECK-SVE-NEXT: orr z0.d, z0.d, z2.d
; CHECK-SVE-NEXT: ret
;
; CHECK-SVE-AES-LABEL: clmul_nxv16i8:
; CHECK-SVE-AES: // %bb.0:
; CHECK-SVE-AES-NEXT: movprfx z2, z1
-; CHECK-SVE-AES-NEXT: and z2.b, z2.b, #0x2
-; CHECK-SVE-AES-NEXT: movprfx z3, z1
-; CHECK-SVE-AES-NEXT: and z3.b, z3.b, #0x1
-; CHECK-SVE-AES-NEXT: movprfx z4, z1
-; CHECK-SVE-AES-NEXT: and z4.b, z4.b, #0x4
-; CHECK-SVE-AES-NEXT: movprfx z5, z1
-; CHECK-SVE-AES-NEXT: and z5.b, z5.b, #0x8
-; CHECK-SVE-AES-NEXT: movprfx z6, z1
-; CHECK-SVE-AES-NEXT: and z6.b, z6.b, #0x10
-; CHECK-SVE-AES-NEXT: movprfx z7, z1
-; CHECK-SVE-AES-NEXT: and z7.b, z7.b, #0x20
+; CHECK-SVE-AES-NEXT: and z2.b, z2.b, #0x55
+; CHECK-SVE-AES-NEXT: movprfx z3, z0
+; CHECK-SVE-AES-NEXT: and z3.b, z3.b, #0xaa
+; CHECK-SVE-AES-NEXT: and z1.b, z1.b, #0xaa
+; CHECK-SVE-AES-NEXT: and z0.b, z0.b, #0x55
; CHECK-SVE-AES-NEXT: ptrue p0.b
-; CHECK-SVE-AES-NEXT: movprfx z24, z1
-; CHECK-SVE-AES-NEXT: and z24.b, z24.b, #0x40
-; CHECK-SVE-AES-NEXT: and z1.b, z1.b, #0x80
-; CHECK-SVE-AES-NEXT: mul z2.b, p0/m, z2.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z3.b, p0/m, z3.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z4.b, p0/m, z4.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z5.b, p0/m, z5.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z6.b, p0/m, z6.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z7.b, p0/m, z7.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z24.b, p0/m, z24.b, z0.b
-; CHECK-SVE-AES-NEXT: mul z0.b, p0/m, z0.b, z1.b
-; CH...
[truncated]
|
folkertdev
commented
Jun 19, 2026
Comment on lines
+8917
to
+8924
| // We distribute the BW bits over S phases: | ||
| // | ||
| // phase 0 keeps bits: 0, S, 2S, ... | ||
| // phase 1 keeps bits: 1, S + 1, 2S + 1, ... | ||
| // ... | ||
| // | ||
| // Each phase has up to n = ceil(BW / S) bits set, and the holes are S-1 | ||
| // bits wide. |
Contributor
Author
There was a problem hiding this comment.
Lmk if this makes sense, it's kind of tricky to explain.
Contributor
Author
There was a problem hiding this comment.
Here the new approach is apparently worse in terms of instruction count. Based on some grepping a big difference is the number of spills. Anything we can/should do about that?
Contributor
Author
|
I need some guidance here: are the instruction count regressions acceptable, or should I somehow tweak the heuristic for when to use the multiplication with holes approach on vectors? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generalize the approach from #203727 to narrower and wider integers.
We still need the fallback for when multiplication isn't available, and it turns out that for some widths the fallback emits fewer instructions, the naive fallback is still used for
i1,i3,i4andi9. I've also now enabled wider integers (i128andi256have uses in cryptography).Based on my local experiments, the Karatsuba approach (e.g. as in rust-lang/rust#152132 (comment)) is not actually better than zero extending the input and using multiplication with holes on the wider type.
CC #203694
CC: @eisenwave