[GlobalIsel][X86] Legalize G_CTPOP and G_CTLZ

[tschuett]

Summary

G_BSWAP was reverted -> added to this diff.

check plan: ninja check-llvm-codegen-x86

Future work: G_SUB and G_ZEXT need some modernization.

[arsenm]

llvm/lib/Target/X86/X86LegalizerInfo.cpp

402

These both have buggy type lists, and I'm surprised nothing is blowing up. These have 2 type parameters, the result may be different from the source

[arsenm]

llvm/lib/Target/X86/X86LegalizerInfo.cpp

402

[tschuett]

Now, he fails to legalise:
unable to legalize instruction: %3:_(s16) = G_CTLZ %0:_(s8) (in function: test_ctlz8)
Note s8 and s16.

[tschuett]

llvm/test/CodeGen/X86/GlobalISel/legalize-leading-zeros.mir

44

The input is s8 and the output is s16.

[arsenm]

llvm/test/CodeGen/X86/GlobalISel/legalize-leading-zeros.mir

44

I think you're running into the over-constrained API problem. I think it only works as you would expect if you legalize the first operand and the second operand second. The current API cannot express your desired changes for both types at the same time, and the LegalizerHelper currently doesn't handle arbitrary requests.

[tschuett]

Any hints how to implement your approach?

[arsenm]

In D150677#4346713, @tschuett wrote:

Any hints how to implement your approach?

Which approach? Not sure precisely what you are trying to achieve in the example. I assume you want s8 to promote both operands? The -debug log should show the legalization steps it's trying to take

[tschuett]

G_CTLZ is only legal for s16, s32, and s64. I was hoping that it will somehow legally widened to s16 ops.

[arsenm]

In D150677#4346982, @tschuett wrote:

G_CTLZ is only legal for s16, s32, and s64. I was hoping that it will somehow legally widened to s16 ops.

Your legalize rules aren't constraining the result and source type to match though. They're expressing the constraints as though they are independently chosen

[tschuett]

changed the type of clampScalar

[arsenm]

llvm/lib/Target/X86/X86LegalizerInfo.cpp

403–404

Should probably swap these lines but it doesn't make much difference. The API could also use work for the common clamp-to-power-of-2-in-range case

[tschuett]

Thanks. I am only looking for low hanging fruit and not to maximize coverage. X86 Gisel fails fast and loud.

llvm/lib/Target/X86/X86LegalizerInfo.h

bool legalizeIntrinsic(LegalizerHelper &Helper,
                          MachineInstr &MI) const override;
 
 private:
   void setLegalizerInfo32bit();
   void setLegalizerInfo64bit();
   void setLegalizerInfoSSE1();
   void setLegalizerInfoSSE2();
   void setLegalizerInfoSSE41();
+  void setLegalizerInfoSSE42();
   void setLegalizerInfoAVX();
   void setLegalizerInfoAVX2();
   void setLegalizerInfoAVX512();
   void setLegalizerInfoAVX512DQ();
   void setLegalizerInfoAVX512BW();
 };
 } // namespace llvm
 #endif

llvm/lib/Target/X86/X86LegalizerInfo.cpp

X86LegalizerInfo::X86LegalizerInfo(const X86Subtarget &STI,
                                    const X86TargetMachine &TM)
     : Subtarget(STI), TM(TM) {
 
   setLegalizerInfo32bit();
   setLegalizerInfo64bit();
   setLegalizerInfoSSE1();
   setLegalizerInfoSSE2();
   setLegalizerInfoSSE41();
+  setLegalizerInfoSSE42();
   setLegalizerInfoAVX();
   setLegalizerInfoAVX2();
   setLegalizerInfoAVX512();
   setLegalizerInfoAVX512DQ();
   setLegalizerInfoAVX512BW();
 
   getActionDefinitionsBuilder(G_INTRINSIC_ROUNDEVEN)
     .scalarize(0)
getActionDefinitionsBuilder(
     .clampScalar(1, s8, s8);
 
   // Merge/Unmerge
   LegacyInfo.setAction({G_MERGE_VALUES, s128}, LegacyLegalizeActions::Legal);
   LegacyInfo.setAction({G_UNMERGE_VALUES, 1, s128},
                        LegacyLegalizeActions::Legal);
   LegacyInfo.setAction({G_MERGE_VALUES, 1, s128}, LegacyLegalizeActions::Legal);
   LegacyInfo.setAction({G_UNMERGE_VALUES, s128}, LegacyLegalizeActions::Legal);
+  getActionDefinitionsBuilder(G_BSWAP)
+    .legalFor({s32, s64})
+    .widenScalarToNextPow2(1, /*Min=*/32)
+    .clampScalar(0, s32, s64);
 }
 
 void X86LegalizerInfo::setLegalizerInfoSSE1() {
   if (!Subtarget.hasSSE1())
     return;
 
   const LLT s32 = LLT::scalar(32);
   const LLT s64 = LLT::scalar(64);
void X86LegalizerInfo::setLegalizerInfoSSE41() {
 
   const LLT v4s32 = LLT::fixed_vector(4, 32);
 
   auto &LegacyInfo = getLegacyLegalizerInfo();
 
   LegacyInfo.setAction({G_MUL, v4s32}, LegacyLegalizeActions::Legal);
 }
 
+void X86LegalizerInfo::setLegalizerInfoSSE42() {
+  if (!Subtarget.hasSSE42())
+    return;
+
+  const LLT s16 = LLT::scalar(16);
+  const LLT s32 = LLT::scalar(32);
+  const LLT s64 = LLT::scalar(64);
+
+  // popcount
+  getActionDefinitionsBuilder(G_CTPOP)
+    .legalFor({{s16, s16}, {s32, s32}, {s64, s64}})
+    .widenScalarToNextPow2(1, /*Min=*/16)
+    .clampScalar(1, s16, s64);
+
+  // count leading zeros (LZCNT)
+  getActionDefinitionsBuilder(G_CTLZ)
+    .legalFor({{s16, s16}, {s32, s32}, {s64, s64}})
+    .widenScalarToNextPow2(1, /*Min=*/16)
+    .clampScalar(1, s16, s64);
+}
+
+
 void X86LegalizerInfo::setLegalizerInfoAVX() {
   if (!Subtarget.hasAVX())
     return;
 
   const LLT v16s8 = LLT::fixed_vector(16, 8);
   const LLT v8s16 = LLT::fixed_vector(8, 16);
   const LLT v4s32 = LLT::fixed_vector(4, 32);
   const LLT v2s64 = LLT::fixed_vector(2, 64);

llvm/test/CodeGen/X86/GlobalISel/legalize-bswap.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -mtriple=x86_64-linux-gnu -run-pass=legalizer %s -o - | FileCheck %s
+
+# test BSWAP s32 and s64
+
+---
+name:            test_bswap_s96
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_bswap_s96
+    ; CHECK: [[DEF:%[0-9]+]]:_(s96) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[DEF]](s96)
+    ; CHECK-NEXT: [[BSWAP:%[0-9]+]]:_(s32) = G_BSWAP [[ANYEXT]]
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s8) = G_CONSTANT i8 -64
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[BSWAP]], [[C]](s8)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:_(s96) = G_TRUNC [[LSHR]](s32)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s96) = COPY [[TRUNC]](s96)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s96)
+    %0:_(s96) = IMPLICIT_DEF
+    %1:_(s96) = G_BSWAP %0
+    %2:_(s96) = COPY %1(s96)
+    RET 0, implicit %2
+
+...
+---
+name:            test_bswaps64
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_bswaps64
+    ; CHECK: [[DEF:%[0-9]+]]:_(s64) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[BSWAP:%[0-9]+]]:_(s64) = G_BSWAP [[DEF]]
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY [[BSWAP]](s64)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s64)
+    %0:_(s64) = IMPLICIT_DEF
+    %1:_(s64) = G_BSWAP %0
+    %2:_(s64) = COPY %1(s64)
+    RET 0, implicit %2
+
+...
+---
+name:            test_bswap_s32
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_bswap_s32
+    ; CHECK: [[DEF:%[0-9]+]]:_(s32) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[BSWAP:%[0-9]+]]:_(s32) = G_BSWAP [[DEF]]
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s32) = COPY [[BSWAP]](s32)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s32)
+    %0:_(s32) = IMPLICIT_DEF
+    %1:_(s32) = G_BSWAP %0
+    %2:_(s32) = COPY %1(s32)
+    RET 0, implicit %2
+
+...
+---
+name:            test_bswap_s16
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_bswap_s16
+    ; CHECK: [[DEF:%[0-9]+]]:_(s16) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[DEF]](s16)
+    ; CHECK-NEXT: [[BSWAP:%[0-9]+]]:_(s32) = G_BSWAP [[ANYEXT]]
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s8) = G_CONSTANT i8 16
+    ; CHECK-NEXT: [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[BSWAP]], [[C]](s8)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[LSHR]](s32)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s16) = COPY [[TRUNC]](s16)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s16)
+    %0:_(s16) = IMPLICIT_DEF
+    %1:_(s16) = G_BSWAP %0
+    %2:_(s16) = COPY %1(s16)
+    RET 0, implicit %2
+
+...

llvm/test/CodeGen/X86/GlobalISel/legalize-constant.mir

registers:
   - { id: 1, class: _ }
   - { id: 2, class: _ }
   - { id: 3, class: _ }
   - { id: 4, class: _ }
 body:             |
   bb.1 (%ir-block.0):
     ; X32-LABEL: name: test_constant
     ; X32: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
-    ; X32: $eax = COPY [[C]](s32)
-    ; X32: [[C1:%[0-9]+]]:_(s8) = G_CONSTANT i8 8
-    ; X32: $al = COPY [[C1]](s8)
-    ; X32: [[C2:%[0-9]+]]:_(s16) = G_CONSTANT i16 16
-    ; X32: $ax = COPY [[C2]](s16)
-    ; X32: [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 32
-    ; X32: $eax = COPY [[C3]](s32)
-    ; X32: [[C4:%[0-9]+]]:_(s32) = G_CONSTANT i32 64
-    ; X32: [[C5:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; X32: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[C4]](s32), [[C5]](s32)
-    ; X32: $rax = COPY [[MV]](s64)
-    ; X32: RET 0
+    ; X32-NEXT: $eax = COPY [[C]](s32)
+    ; X32-NEXT: [[C1:%[0-9]+]]:_(s8) = G_CONSTANT i8 8
+    ; X32-NEXT: $al = COPY [[C1]](s8)
+    ; X32-NEXT: [[C2:%[0-9]+]]:_(s16) = G_CONSTANT i16 16
+    ; X32-NEXT: $ax = COPY [[C2]](s16)
+    ; X32-NEXT: [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 32
+    ; X32-NEXT: $eax = COPY [[C3]](s32)
+    ; X32-NEXT: [[C4:%[0-9]+]]:_(s32) = G_CONSTANT i32 64
+    ; X32-NEXT: [[C5:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
+    ; X32-NEXT: [[MV:%[0-9]+]]:_(s64) = G_MERGE_VALUES [[C4]](s32), [[C5]](s32)
+    ; X32-NEXT: $rax = COPY [[MV]](s64)
+    ; X32-NEXT: RET 0
     ; X64-LABEL: name: test_constant
     ; X64: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
-    ; X64: $eax = COPY [[C]](s32)
-    ; X64: [[C1:%[0-9]+]]:_(s8) = G_CONSTANT i8 8
-    ; X64: $al = COPY [[C1]](s8)
-    ; X64: [[C2:%[0-9]+]]:_(s16) = G_CONSTANT i16 16
-    ; X64: $ax = COPY [[C2]](s16)
-    ; X64: [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 32
-    ; X64: $eax = COPY [[C3]](s32)
-    ; X64: [[C4:%[0-9]+]]:_(s64) = G_CONSTANT i64 64
-    ; X64: $rax = COPY [[C4]](s64)
-    ; X64: RET 0
+    ; X64-NEXT: $eax = COPY [[C]](s32)
+    ; X64-NEXT: [[C1:%[0-9]+]]:_(s8) = G_CONSTANT i8 8
+    ; X64-NEXT: $al = COPY [[C1]](s8)
+    ; X64-NEXT: [[C2:%[0-9]+]]:_(s16) = G_CONSTANT i16 16
+    ; X64-NEXT: $ax = COPY [[C2]](s16)
+    ; X64-NEXT: [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 32
+    ; X64-NEXT: $eax = COPY [[C3]](s32)
+    ; X64-NEXT: [[C4:%[0-9]+]]:_(s64) = G_CONSTANT i64 64
+    ; X64-NEXT: $rax = COPY [[C4]](s64)
+    ; X64-NEXT: RET 0
     %0(s1) = G_CONSTANT i1 1
     %5:_(s32) = G_ANYEXT %0
     $eax = COPY %5
 
     %1(s8) = G_CONSTANT i8 8
     $al = COPY %1
     %2(s16) = G_CONSTANT i16 16
     $ax = COPY %2

 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
 body: |
   bb.0:
 
     ; X32-LABEL: name: test_fconstant
     ; X32: [[C:%[0-9]+]]:_(s32) = G_FCONSTANT float 1.000000e+00
-    ; X32: $eax = COPY [[C]](s32)
-    ; X32: [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 2.000000e+00
-    ; X32: $rax = COPY [[C1]](s64)
+    ; X32-NEXT: $eax = COPY [[C]](s32)
+    ; X32-NEXT: [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 2.000000e+00
+    ; X32-NEXT: $rax = COPY [[C1]](s64)
     ; X64-LABEL: name: test_fconstant
     ; X64: [[C:%[0-9]+]]:_(s32) = G_FCONSTANT float 1.000000e+00
-    ; X64: $eax = COPY [[C]](s32)
-    ; X64: [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 2.000000e+00
-    ; X64: $rax = COPY [[C1]](s64)
+    ; X64-NEXT: $eax = COPY [[C]](s32)
+    ; X64-NEXT: [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 2.000000e+00
+    ; X64-NEXT: $rax = COPY [[C1]](s64)
     %0(s32) = G_FCONSTANT float 1.0
     $eax = COPY %0
     %1(s64) = G_FCONSTANT double 2.0
     $rax = COPY %1
 ...
 

llvm/test/CodeGen/X86/GlobalISel/legalize-ctpop.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -mtriple=x86_64-linux-gnu -mattr=+sse4.2 -run-pass=legalizer %s -o - | FileCheck %s
+
+# test popcount for s16, s32, and s64
+
+---
+name:            test_ctpop35
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctpop35
+    ; CHECK: [[DEF:%[0-9]+]]:_(s35) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[DEF]](s35)
+    ; CHECK-NEXT: [[CTPOP:%[0-9]+]]:_(s64) = G_CTPOP [[ZEXT]](s64)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:_(s35) = G_TRUNC [[CTPOP]](s64)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s35) = COPY [[TRUNC]](s35)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s35)
+    %0:_(s35) = IMPLICIT_DEF
+    %1:_(s35) = G_CTPOP %0
+    %2:_(s35) = COPY %1(s35)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctpop8
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctpop8
+    ; CHECK: [[DEF:%[0-9]+]]:_(s8) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[ZEXT:%[0-9]+]]:_(s16) = G_ZEXT [[DEF]](s8)
+    ; CHECK-NEXT: [[CTPOP:%[0-9]+]]:_(s16) = G_CTPOP [[ZEXT]](s16)
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[CTPOP]](s16)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s8) = COPY [[TRUNC]](s8)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s8)
+    %0:_(s8) = IMPLICIT_DEF
+    %1:_(s8) = G_CTPOP %0
+    %2:_(s8) = COPY %1(s8)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctpop64
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctpop64
+    ; CHECK: [[DEF:%[0-9]+]]:_(s64) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[CTPOP:%[0-9]+]]:_(s64) = G_CTPOP [[DEF]](s64)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY [[CTPOP]](s64)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s64)
+    %0:_(s64) = IMPLICIT_DEF
+    %1:_(s64) = G_CTPOP %0
+    %2:_(s64) = COPY %1(s64)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctpop32
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctpop32
+    ; CHECK: [[DEF:%[0-9]+]]:_(s32) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[CTPOP:%[0-9]+]]:_(s32) = G_CTPOP [[DEF]](s32)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s32) = COPY [[CTPOP]](s32)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s32)
+    %0:_(s32) = IMPLICIT_DEF
+    %1:_(s32) = G_CTPOP %0
+    %2:_(s32) = COPY %1(s32)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctpop16
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctpop16
+    ; CHECK: [[DEF:%[0-9]+]]:_(s16) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[CTPOP:%[0-9]+]]:_(s16) = G_CTPOP [[DEF]](s16)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s16) = COPY [[CTPOP]](s16)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s16)
+    %0:_(s16) = IMPLICIT_DEF
+    %1:_(s16) = G_CTPOP %0
+    %2:_(s16) = COPY %1(s16)
+    RET 0, implicit %2
+
+...

llvm/test/CodeGen/X86/GlobalISel/legalize-leading-zeros.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
+# RUN: llc -mtriple=x86_64-linux-gnu -mattr=+sse4.2 -run-pass=legalizer %s -o - | FileCheck %s
+
+# test count leading zeros for s16, s32, and s64
+
+---
+name:            test_ctlz35
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctlz35
+    ; CHECK: [[DEF:%[0-9]+]]:_(s35) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[ZEXT:%[0-9]+]]:_(s64) = G_ZEXT [[DEF]](s35)
+    ; CHECK-NEXT: [[CTLZ:%[0-9]+]]:_(s64) = G_CTLZ [[ZEXT]](s64)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 29
+    ; CHECK-NEXT: [[SUB:%[0-9]+]]:_(s64) = G_SUB [[CTLZ]], [[C]]
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:_(s35) = G_TRUNC [[SUB]](s64)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s35) = COPY [[TRUNC]](s35)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s35)
+    %0:_(s35) = IMPLICIT_DEF
+    %1:_(s35) = G_CTLZ %0
+    %2:_(s35) = COPY %1(s35)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctlz8
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctlz8
+    ; CHECK: [[DEF:%[0-9]+]]:_(s8) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[ZEXT:%[0-9]+]]:_(s16) = G_ZEXT [[DEF]](s8)
+    ; CHECK-NEXT: [[CTLZ:%[0-9]+]]:_(s16) = G_CTLZ [[ZEXT]](s16)
+    ; CHECK-NEXT: [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 8
+    ; CHECK-NEXT: [[SUB:%[0-9]+]]:_(s16) = G_SUB [[CTLZ]], [[C]]
+    ; CHECK-NEXT: [[TRUNC:%[0-9]+]]:_(s8) = G_TRUNC [[SUB]](s16)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s8) = COPY [[TRUNC]](s8)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s8)
+    %0:_(s8) = IMPLICIT_DEF
+    %1:_(s8) = G_CTLZ %0
+    %2:_(s8) = COPY %1(s8)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctlz64
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctlz64
+    ; CHECK: [[DEF:%[0-9]+]]:_(s64) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[CTLZ:%[0-9]+]]:_(s64) = G_CTLZ [[DEF]](s64)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s64) = COPY [[CTLZ]](s64)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s64)
+    %0:_(s64) = IMPLICIT_DEF
+    %1:_(s64) = G_CTLZ %0
+    %2:_(s64) = COPY %1(s64)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctlz32
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctlz32
+    ; CHECK: [[DEF:%[0-9]+]]:_(s32) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[CTLZ:%[0-9]+]]:_(s32) = G_CTLZ [[DEF]](s32)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s32) = COPY [[CTLZ]](s32)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s32)
+    %0:_(s32) = IMPLICIT_DEF
+    %1:_(s32) = G_CTLZ %0
+    %2:_(s32) = COPY %1(s32)
+    RET 0, implicit %2
+
+...
+---
+name:            test_ctlz16
+alignment:       16
+legalized:       false
+regBankSelected: false
+registers:
+  - { id: 0, class: _, preferred-register: '' }
+  - { id: 1, class: _, preferred-register: '' }
+body:             |
+  bb.1:
+    ; CHECK-LABEL: name: test_ctlz16
+    ; CHECK: [[DEF:%[0-9]+]]:_(s16) = IMPLICIT_DEF
+    ; CHECK-NEXT: [[CTLZ:%[0-9]+]]:_(s16) = G_CTLZ [[DEF]](s16)
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:_(s16) = COPY [[CTLZ]](s16)
+    ; CHECK-NEXT: RET 0, implicit [[COPY]](s16)
+    %0:_(s16) = IMPLICIT_DEF
+    %1:_(s16) = G_CTLZ %0
+    %2:_(s16) = COPY %1(s16)
+    RET 0, implicit %2
+
+...