[GlobalISel] Add G_ASSERT_ZEXT

[paquette]

Summary

This adds a generic opcode which communicates that a type has already been zero-extended from a narrower type.

This is intended to be similar to AssertZext in SelectionDAG.

For example,

%x_was_extended:_(s64) = G_ASSERT_ZEXT %x, 16

Signifies that the top 48 bits of %x are known to be 0.

This is useful in cases like this:

define i1 @zeroext_param(i8 zeroext %x) {
    %cmp = icmp ult i8 %x, -20
    ret i1 %cmp
}

In AArch64, %x must use a 32-bit register, which is then truncated to a 8-bit value.

If we know that %x is already zero-ed out in the relevant high bits, we can avoid the truncate.

Currently, in GISel, this looks like this:

_zeroext_param:
    and w8, w0, #0xff ; We don't actually need this!
    cmp w8, #236
    cset w0, lo
    ret

While SDAG does not produce the truncation, since it knows that it's unnecessary:

_zeroext_param:
    cmp w0, #236
    cset w0, lo
    ret

This patch

• Adds G_ASSERT_ZEXT
• Adds MIRBuilder support for it
• Adds MachineVerifier support for it
• Documents it

It also puts G_ASSERT_ZEXT into its own class of "hint instruction." (There should be a G_ASSERT_SEXT in the future, maybe a G_ASSERT_ALIGN as well.)

The reason for doing this rather than making it a pseudo like, say, COPY, is that these *are* pre-ISel generic opcodes. The only difference is

1. They should be ignored by the legalizer
2. They shouldn't impact register bank selection
3. They should select to nothing

It also puts G_ASSERT_ZEXT into its own class of "hint instruction." (There should be a G_ASSERT_SEXT in the future, maybe a G_ASSERT_ALIGN as well.)

[paquette]

• Remove requirement that these be removed before regbankselect. @arsenm mentioned that it's useful to have these in selection.
• Pull these out of PRE_ISEL_GENERIC_OPCODE_*, and place them above in their own markers. That way, isPreISelGenericOpcode doesn't return true for them, but we can still communicate that they are GlobalISel-specific hints.

[arsenm]

llvm/include/llvm/Target/GenericOpcodes.td

1345–1346

Should mention this behaves identically to a COPY

llvm/lib/CodeGen/MachineVerifier.cpp

946

I think these assert messages aren't consistent with using 0 indexing or not, but I guess this is the better direction

llvm/test/CodeGen/AArch64/GlobalISel/legalize-ignore-hint.mir

19

Should probably include a case with an illegal register type

[arsenm]

The copy-look-through util functions should probably look through this, although that's a separate patch

[paquette]

• Update documentation to include that this behaves like a COPY
• Add two testcases for invalid registers to the MachineVerifier test (type mismatch, physical register)

[aemerson]

We should clarify what kind of copies this is allowed to do. As a hint, I think it shouldn't be legal to use this copy to change regbanks, or regclasses, or to use/define physical registers. As a hint, it should be guaranteed to be safe to look through to the vreg source.

[paquette]

• Update docs to clarify what kinds of COPY instructions are allowed
• Update MachineVerifier to check regbank/regclass restrictions + add a test

[arsenm]

llvm/test/MachineVerifier/test_g_assert_zext_register_bank_class.mir

4–8

Don't need the IR section

llvm/docs/GlobalISel/GenericOpcode.rst

 ^^^^^^^^^^^^^^^^
 
 Dynamically realigns the stack pointer to the specified size and alignment.
 An alignment value of `0` or `1` mean no specific alignment.
 
 .. code-block:: none
 
   %8:_(p0) = G_DYN_STACKALLOC %7(s64), 32
+
+Optimization Hints
+------------------
+
+These instructions do not correspond to any target instructions. They act as
+hints for various combines.
+
+G_ASSERT_ZEXT
+^^^^^^^^^^^^^
+
+Signifies that the contents of a register were previously zero-extended from a
+smaller type.
+
+The smaller type is denoted using an immediate operand. For scalars, this is the
+width of the entire smaller type. For vectors, this is the width of the smaller
+element type.
+
+.. code-block:: none
+
+  %x_assert:_(s32) = G_ASSERT_ZEXT %x(s32), 16
+  %y_assert:_(<2 x s32>) = G_ASSERT_ZEXT %y(<2 x s32>), 16
+
+G_ASSERT_ZEXT acts like a restricted form of copy.
+
+The source and destination registers must
+
+- Be virtual
+- Belong to the same register class
+- Belong to the same register bank
+
+It should always be safe to
+
+- Look through the source register
+- Replace the destination register with the source register

llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h

public:
   ///
   /// Register-to-register COPY sets \p Res to \p Op.
   ///
   /// \pre setBasicBlock or setMI must have been called.
   ///
   /// \return a MachineInstrBuilder for the newly created instruction.
   MachineInstrBuilder buildCopy(const DstOp &Res, const SrcOp &Op);
 
+  /// Build and insert \p Res = G_ASSERT_ZEXT Op, Size
+  ///
+  /// \return a MachineInstrBuilder for the newly created instruction.
+  MachineInstrBuilder buildAssertZExt(const DstOp &Res, const SrcOp &Op,
+                                      unsigned Size);
+
   /// Build and insert `Res = G_LOAD Addr, MMO`.
   ///
   /// Loads the value stored at \p Addr. Puts the result in \p Res.
   ///
   /// \pre setBasicBlock or setMI must have been called.
   /// \pre \p Res must be a generic virtual register.
   /// \pre \p Addr must be a generic virtual register with pointer type.
   ///

llvm/include/llvm/CodeGen/TargetOpcodes.h

inline bool isPreISelGenericOpcode(unsigned Opcode) {
   return Opcode >= TargetOpcode::PRE_ISEL_GENERIC_OPCODE_START &&
          Opcode <= TargetOpcode::PRE_ISEL_GENERIC_OPCODE_END;
 }
 
 /// Check whether the given Opcode is a target-specific opcode.
 inline bool isTargetSpecificOpcode(unsigned Opcode) {
   return Opcode > TargetOpcode::PRE_ISEL_GENERIC_OPCODE_END;
 }
+
+/// \returns true if \p Opcode is an optimization hint opcode which is not
+/// supposed to appear after ISel.
+inline bool isPreISelGenericOptimizationHint(unsigned Opcode) {
+  return Opcode >= TargetOpcode::PRE_ISEL_GENERIC_OPTIMIZATION_HINT_START &&
+         Opcode <= TargetOpcode::PRE_ISEL_GENERIC_OPTIMIZATION_HINT_END;
+}
+
 } // end namespace llvm
 
 #endif

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(PATCHABLE_TYPED_EVENT_CALL)
 
 HANDLE_TARGET_OPCODE(ICALL_BRANCH_FUNNEL)
 
 /// The following generic opcodes are not supposed to appear after ISel.
 /// This is something we might want to relax, but for now, this is convenient
 /// to produce diagnostics.
 
+/// Instructions which should not exist past instruction selection, but do not
+/// generate code. These instructions only act as optimization hints.
+HANDLE_TARGET_OPCODE(G_ASSERT_ZEXT)
+HANDLE_TARGET_OPCODE_MARKER(PRE_ISEL_GENERIC_OPTIMIZATION_HINT_START,
+                            G_ASSERT_ZEXT)
+HANDLE_TARGET_OPCODE_MARKER(PRE_ISEL_GENERIC_OPTIMIZATION_HINT_END,
+                            G_ASSERT_ZEXT)
+
 /// Generic ADD instruction. This is an integer add.
 HANDLE_TARGET_OPCODE(G_ADD)
 HANDLE_TARGET_OPCODE_MARKER(PRE_ISEL_GENERIC_OPCODE_START, G_ADD)
 
 /// Generic SUB instruction. This is an integer sub.
 HANDLE_TARGET_OPCODE(G_SUB)
 
 // Generic multiply instruction.

llvm/include/llvm/Target/GenericOpcodes.td

 }
 
 def G_MEMSET : GenericInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins ptype0:$dst_addr, type1:$value, type2:$size, untyped_imm_0:$tailcall);
   let hasSideEffects = false;
   let mayStore = true;
 }
+
+//------------------------------------------------------------------------------
+// Optimization hints
+//------------------------------------------------------------------------------
+
+// Asserts that an operation has already been zero-extended from a specific
+// type.
+def G_ASSERT_ZEXT : GenericInstruction {
+  let OutOperandList = (outs type0:$dst);
+  let InOperandList = (ins type0:$src, untyped_imm_0:$sz);
+  let hasSideEffects = false;
+}

llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp

return buildInstr(TargetOpcode::G_BRJT)
       .addUse(IndexReg);
 }
 
 MachineInstrBuilder MachineIRBuilder::buildCopy(const DstOp &Res,
                                                 const SrcOp &Op) {
   return buildInstr(TargetOpcode::COPY, Res, Op);
 }
 
+MachineInstrBuilder MachineIRBuilder::buildAssertZExt(const DstOp &Res,
+                                                      const SrcOp &Op,
+                                                      unsigned Size) {
+  return buildInstr(TargetOpcode::G_ASSERT_ZEXT, Res, Op).addImm(Size);
+}
+
 MachineInstrBuilder MachineIRBuilder::buildConstant(const DstOp &Res,
                                                     const ConstantInt &Val) {
   LLT Ty = Res.getLLTTy(*getMRI());
   LLT EltTy = Ty.getScalarType();
   assert(EltTy.getScalarSizeInBits() == Val.getBitWidth() &&
          "creating constant with the wrong size");
 
   if (Ty.isVector()) {

llvm/lib/CodeGen/MachineVerifier.cpp

if (MI->getNumOperands() < MCID.getNumOperands())
     return;
 
   StringRef ErrorInfo;
   if (!TII->verifyInstruction(*MI, ErrorInfo))
     report(ErrorInfo.data(), MI);
 
   // Verify properties of various specific instruction types
   switch (MI->getOpcode()) {
+  case TargetOpcode::G_ASSERT_ZEXT: {
+    if (!MI->getOperand(2).isImm()) {
+      report("G_ASSERT_ZEXT expects an immediate operand #2", MI);
+      break;
+    }
+
+    Register Dst = MI->getOperand(0).getReg();
+    Register Src = MI->getOperand(1).getReg();
+    LLT DstTy = MRI->getType(Dst);
+    LLT SrcTy = MRI->getType(Src);
+    verifyVectorElementMatch(DstTy, SrcTy, MI);
+    int64_t Imm = MI->getOperand(2).getImm();
+    if (Imm <= 0) {
+      report("G_ASSERT_ZEXT size must be >= 1", MI);
+      break;
+    }
+
+    if (Imm >= SrcTy.getScalarSizeInBits()) {
+      report("G_ASSERT_ZEXT size must be less than source bit width", MI);
+      break;
+    }
+
+    if (MRI->getRegBankOrNull(Src) != MRI->getRegBankOrNull(Dst)) {
+      report("G_ASSERT_ZEXT source and destination register banks must match",
+             MI);
+      break;
+    }
+
+    if (MRI->getRegClassOrNull(Src) != MRI->getRegClassOrNull(Dst))
+      report("G_ASSERT_ZEXT source and destination register classes must match",
+             MI);
+
+    break;
+  }
+
   case TargetOpcode::G_CONSTANT:
   case TargetOpcode::G_FCONSTANT: {
     LLT DstTy = MRI->getType(MI->getOperand(0).getReg());
     if (DstTy.isVector())
       report("Instruction cannot use a vector result type", MI);
 
     if (MI->getOpcode() == TargetOpcode::G_CONSTANT) {
       if (!MI->getOperand(1).isCImm()) {
if (MI->isDebugInstr()) {
       if (mapped)
         report("Instruction inside bundle has a slot index", MI);
     } else {
       if (!mapped)
         report("Missing slot index", MI);
     }
   }
 
-  if (isPreISelGenericOpcode(MCID.getOpcode())) {
+  unsigned Opc = MCID.getOpcode();
+  if (isPreISelGenericOpcode(Opc) || isPreISelGenericOptimizationHint(Opc)) {
     verifyPreISelGenericInstruction(MI);
     return;
   }
 
   StringRef ErrorInfo;
   if (!TII->verifyInstruction(*MI, ErrorInfo))
     report(ErrorInfo.data(), MI);
 

llvm/test/CodeGen/AArch64/GlobalISel/legalize-ignore-hint.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=aarch64 -run-pass=legalizer -verify-machineinstrs %s -o - | FileCheck %s
+#
+# Verify that hint instructions are ignored by the legalizer.
+
+---
+name:            assert_zext
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: assert_zext
+    ; CHECK: %copy:_(s32) = COPY $w1
+    ; CHECK: %hint:_(s32) = G_ASSERT_ZEXT %copy, 16
+    ; CHECK: $w0 = COPY %hint(s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %copy:_(s32) = COPY $w1
+    %hint:_(s32) = G_ASSERT_ZEXT %copy, 16
+    $w0 = COPY %hint
+    RET_ReallyLR implicit $w0
+...

llvm/test/MachineVerifier/test_g_assert_ext.mir

+# RUN: not --crash llc -verify-machineinstrs -run-pass none -o /dev/null %s 2>&1 | FileCheck %s
+# REQUIRES: aarch64-registered-target
+
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+  target triple = "aarch64-unknown-unknown"
+  define void @test() { ret void }
+...
+---
+name:            test
+body: |
+  bb.0:
+   liveins: $x0, $w0
+   %0:_(s64) = COPY $x0
+   %1:_(<4 x s16>) = COPY $x0
+   %2:_(s32) = COPY $w0
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT expects an immediate operand #2 ***
+   ; CHECK: instruction: %assert_zext_1:_(s64) = G_ASSERT_ZEXT
+   %assert_zext_1:_(s64) = G_ASSERT_ZEXT %0, %0
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT expects an immediate operand #2 ***
+   ; CHECK: instruction: %assert_zext_2:_(s64) = G_ASSERT_ZEXT
+   %assert_zext_2:_(s64) = G_ASSERT_ZEXT %0, i8 8
+
+   ; CHECK: *** Bad machine code: Type mismatch in generic instruction ***
+   ; CHECK: instruction: %assert_zext_3:_(<2 x s32>) = G_ASSERT_ZEXT
+   ; CHECK: *** Bad machine code: operand types must be all-vector or all-scalar ***
+   ; CHECK: instruction: %assert_zext_3:_(<2 x s32>) = G_ASSERT_ZEXT
+   %assert_zext_3:_(<2 x s32>) = G_ASSERT_ZEXT %0, 8
+
+   ; CHECK: *** Bad machine code: operand types must preserve number of vector elements ***
+   ; CHECK: instruction: %assert_zext_4:_(<2 x s32>) = G_ASSERT_ZEXT
+   %assert_zext_4:_(<2 x s32>) = G_ASSERT_ZEXT %1, 8
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT size must be >= 1 ***
+   ; CHECK: instruction: %assert_zext_5:_(s64) = G_ASSERT_ZEXT
+   %assert_zext_5:_(s64) = G_ASSERT_ZEXT %0, 0
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT size must be less than source bit width ***
+   ; CHECK: instruction: %assert_zext_6:_(s64) = G_ASSERT_ZEXT
+   %assert_zext_6:_(s64) = G_ASSERT_ZEXT %0, 128
+
+   ; CHECK: *** Bad machine code: Type mismatch in generic instruction ***
+   ; CHECK: instruction: %assert_zext_7:_(s64) = G_ASSERT_ZEXT %2:_, 8
+   %assert_zext_7:_(s64) = G_ASSERT_ZEXT %2, 8
+
+   ; CHECK: *** Bad machine code: Generic instruction cannot have physical register ***
+   ; CHECK: instruction: %assert_zext_8:_(s64) = G_ASSERT_ZEXT $x0, 8
+   %assert_zext_8:_(s64) = G_ASSERT_ZEXT $x0, 8

llvm/test/MachineVerifier/test_g_assert_zext_register_bank_class.mir

+# RUN: not --crash llc -verify-machineinstrs -run-pass none -o /dev/null %s 2>&1 | FileCheck %s
+# REQUIRES: aarch64-registered-target
+
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+  target triple = "aarch64-unknown-unknown"
+  define void @test() { ret void }
+...
+---
+name:            test
+legalized:       true
+regBankSelected: true
+body: |
+  bb.0:
+   liveins: $w0, $x0, $w1
+   %bank:gpr(s32) = COPY $w0
+   %class:gpr32(s32) = COPY $w1
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT source and destination register banks must match ***
+   ; CHECK: instruction: %bank_mismatch:fpr(s32) = G_ASSERT_ZEXT %bank:gpr, 16
+   %bank_mismatch:fpr(s32) = G_ASSERT_ZEXT %bank, 16
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT source and destination register classes must match ***
+   ; CHECK: instruction: %class_mismatch_gpr:gpr32all(s32) = G_ASSERT_ZEXT %class:gpr32, 16
+   %class_mismatch_gpr:gpr32all(s32) = G_ASSERT_ZEXT %class, 16
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT source and destination register classes must match ***
+   ; CHECK: instruction: %class_mismatch_fpr:fpr32(s32) = G_ASSERT_ZEXT %class:gpr32, 16
+   %class_mismatch_fpr:fpr32(s32) = G_ASSERT_ZEXT %class, 16
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT source and destination register banks must match ***
+   ; CHECK: instruction: %dst_has_class_src_has_bank:gpr32all(s32) = G_ASSERT_ZEXT %bank:gpr, 16
+   %dst_has_class_src_has_bank:gpr32all(s32) = G_ASSERT_ZEXT %bank, 16
+
+   ; CHECK: *** Bad machine code: G_ASSERT_ZEXT source and destination register banks must match ***
+   ; CHECK: instruction: %dst_has_bank_src_has_class:gpr(s32) = G_ASSERT_ZEXT %class:gpr32, 16
+   %dst_has_bank_src_has_class:gpr(s32) = G_ASSERT_ZEXT %class, 16
+
+   ; CHECK: *** Bad machine code: Generic instruction cannot have physical register ***
+   ; CHECK: instruction: %implicit_physreg:gpr(s32) = G_ASSERT_ZEXT %class:gpr32, 16, implicit-def $w0
+   %implicit_physreg:gpr(s32) = G_ASSERT_ZEXT %class, 16, implicit-def $w0