[AArch64][GlobalISel] Change G_ANYEXT fed by scalar G_ICMP to G_ZEXT

[paquette]

Summary

This is a common pattern:

%icmp:_(s32) = G_ICMP intpred(eq), ...
%ext:_(s64) = G_ANYEXT %icmp(s32)
%and:_(s64) = G_AND %ext, 1

Here's an example: https://godbolt.org/z/T13f6o8zE

This pattern appears because of the following combine in the
LegalizationArtifactCombiner:

// zext(trunc x) - > and (aext/copy/trunc x), mask

Which kicks in when we widen the result of G_ICMP from 1 bit to 32 bits.

We know that, on AArch64, a scalar G_ICMP will produce 0 or 1. So the result of %ext will always be 0 or 1 as well.

We have some KnownBits combines which eliminate redundant G_ANDs with masks. These combines don't kick in with G_ANYEXT.

So, if we replace the G_ANYEXT with G_ZEXT in this situation, the KnownBits-based combines can remove the redundant G_AND.

I wasn't sure if it woud be more appropriate to

• Take this route
• Put this in the LegalizationArtifactCombiner.
• Allow 64 bit G_ICMP destinations

I decided on this route because

1. It's simple

2. I'm not sure if philosophically-speaking, we should be handling non-artifact instructions + target-specific details like TargetBooleanContents in the LegalizationArtifactCombiner

3. There is a lot of existing code which assumes we only have 32 bit G_ICMP destinations. So, adding support for 64-bit destinations seems rather invasive right now. I think that adding support for 64-bit destinations, or modelling G_ICMP as ADDS/SUBS/etc is probably cleaner long term though.

This gives minor code size savings on all CTMark benchmarks.

[jroelofs]

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerCombiner.cpp

271

G_FCMP too?

[paquette]

Add G_FCMP. This gives minor added improvements on CTMark/sqlite3 and CTMark/bullet.

Here's a similar IR example to the G_ICMP one for G_FCMP: https://godbolt.org/z/c4hW53Wrs

[jroelofs]

LGTM

llvm/lib/Target/AArch64/AArch64Combine.td

 
 def fold_merge_to_zext : GICombineRule<
   (defs root:$d),
   (match (wip_match_opcode G_MERGE_VALUES):$d,
           [{ return matchFoldMergeToZext(*${d}, MRI); }]),
   (apply [{ applyFoldMergeToZext(*${d}, MRI, B, Observer); }])
 >;
 
+def mutate_anyext_to_zext : GICombineRule<
+  (defs root:$d),
+  (match (wip_match_opcode G_ANYEXT):$d,
+          [{ return matchMutateAnyExtToZExt(*${d}, MRI); }]),
+  (apply [{ applyMutateAnyExtToZExt(*${d}, MRI, B, Observer); }])
+>;
+
 // Post-legalization combines which should happen at all optimization levels.
 // (E.g. ones that facilitate matching for the selector) For example, matching
 // pseudos.
 def AArch64PostLegalizerLoweringHelper
     : GICombinerHelper<"AArch64GenPostLegalizerLoweringHelper",
                        [shuffle_vector_lowering, vashr_vlshr_imm,
                         icmp_lowering, build_vector_lowering,
                         lower_vector_fcmp, form_truncstore]> {
   let DisableRuleOption = "aarch64postlegalizerlowering-disable-rule";
 }
 
 // Post-legalization combines which are primarily optimizations.
 def AArch64PostLegalizerCombinerHelper
     : GICombinerHelper<"AArch64GenPostLegalizerCombinerHelper",
                        [copy_prop, erase_undef_store, combines_for_extload,
-                        sext_trunc_sextload,
+                        sext_trunc_sextload, mutate_anyext_to_zext,
                         hoist_logic_op_with_same_opcode_hands,
                         redundant_and, xor_of_and_with_same_reg,
                         extractvecelt_pairwise_add, redundant_or,
                         mul_const, redundant_sext_inreg,
                         form_bitfield_extract, rotate_out_of_range,
                         icmp_to_true_false_known_bits, merge_unmerge,
                         select_combines, fold_merge_to_zext,
                         constant_fold, identity_combines,
                         ptr_add_immed_chain, overlapping_and]> {
   let DisableRuleOption = "aarch64postlegalizercombiner-disable-rule";
 }

llvm/lib/Target/AArch64/GISel/AArch64PostLegalizerCombiner.cpp

void applyFoldMergeToZext(MachineInstr &MI, MachineRegisterInfo &MRI,
   //  ->
   // %d(s64) = G_ZEXT %a(s32)
   Observer.changingInstr(MI);
   MI.setDesc(B.getTII().get(TargetOpcode::G_ZEXT));
   MI.RemoveOperand(2);
   Observer.changedInstr(MI);
 }
 
+/// \returns True if a G_ANYEXT instruction \p MI should be mutated to a G_ZEXT
+/// instruction.
+static bool matchMutateAnyExtToZExt(MachineInstr &MI, MachineRegisterInfo &MRI) {
+  // If this is coming from a scalar compare then we can use a G_ZEXT instead of
+  // a G_ANYEXT:
+  //
+  // %cmp:_(s32) = G_[I|F]CMP ... <-- produces 0/1.
+  // %ext:_(s64) = G_ANYEXT %cmp(s32)
+  //
+  // By doing this, we can leverage more KnownBits combines.
+  assert(MI.getOpcode() == TargetOpcode::G_ANYEXT);
+  Register Dst = MI.getOperand(0).getReg();
+  Register Src = MI.getOperand(1).getReg();
+  return MRI.getType(Dst).isScalar() &&
+         mi_match(Src, MRI,
+                  m_any_of(m_GICmp(m_Pred(), m_Reg(), m_Reg()),
+                           m_GFCmp(m_Pred(), m_Reg(), m_Reg())));
+}
+
+static void applyMutateAnyExtToZExt(MachineInstr &MI, MachineRegisterInfo &MRI,
+                              MachineIRBuilder &B,
+                              GISelChangeObserver &Observer) {
+  Observer.changingInstr(MI);
+  MI.setDesc(B.getTII().get(TargetOpcode::G_ZEXT));
+  Observer.changedInstr(MI);
+}
+
 #define AARCH64POSTLEGALIZERCOMBINERHELPER_GENCOMBINERHELPER_DEPS
 #include "AArch64GenPostLegalizeGICombiner.inc"
 #undef AARCH64POSTLEGALIZERCOMBINERHELPER_GENCOMBINERHELPER_DEPS
 
 namespace {
 #define AARCH64POSTLEGALIZERCOMBINERHELPER_GENCOMBINERHELPER_H
 #include "AArch64GenPostLegalizeGICombiner.inc"
 #undef AARCH64POSTLEGALIZERCOMBINERHELPER_GENCOMBINERHELPER_H

llvm/test/CodeGen/AArch64/GlobalISel/postlegalizer-combiner-anyext-to-zext.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64 -run-pass=aarch64-postlegalizer-combiner --aarch64postlegalizercombinerhelper-only-enable-rule="mutate_anyext_to_zext" -verify-machineinstrs %s -o - | FileCheck %s
+# REQUIRES: asserts
+
+...
+---
+name:            scalar_icmp
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x0
+    ; Scalars have 0 or 1, so we want a ZExt.
+
+    ; CHECK-LABEL: name: scalar_icmp
+    ; CHECK: liveins: $x0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: %copy:_(s64) = COPY $x0
+    ; CHECK-NEXT: %cst_1:_(s64) = G_CONSTANT i64 1
+    ; CHECK-NEXT: %icmp:_(s32) = G_ICMP intpred(eq), %copy(s64), %cst_1
+    ; CHECK-NEXT: %ext:_(s64) = G_ZEXT %icmp(s32)
+    ; CHECK-NEXT: $x0 = COPY %ext(s64)
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
+    %copy:_(s64) = COPY $x0
+    %cst_1:_(s64) = G_CONSTANT i64 1
+    %icmp:_(s32) = G_ICMP intpred(eq), %copy(s64), %cst_1
+    %ext:_(s64) = G_ANYEXT %icmp(s32)
+    $x0 = COPY %ext(s64)
+    RET_ReallyLR implicit $x0
+
+
+...
+---
+name:            vector_icmp
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x0, $d0
+    ; Vectors have 0 or negative 1, so we don't produce a zext.
+
+    ; CHECK-LABEL: name: vector_icmp
+    ; CHECK: liveins: $x0, $d0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: %copy:_(<2 x s32>) = COPY $d0
+    ; CHECK-NEXT: %cst_1:_(s32) = G_CONSTANT i32 1
+    ; CHECK-NEXT: %vec_cst_1:_(<2 x s32>) = G_BUILD_VECTOR %cst_1(s32), %cst_1(s32)
+    ; CHECK-NEXT: %icmp:_(<2 x s32>) = G_ICMP intpred(eq), %copy(<2 x s32>), %vec_cst_1
+    ; CHECK-NEXT: %ext:_(<2 x s64>) = G_ANYEXT %icmp(<2 x s32>)
+    ; CHECK-NEXT: $q0 = COPY %ext(<2 x s64>)
+    ; CHECK-NEXT: RET_ReallyLR implicit $q0
+    %copy:_(<2 x s32>) = COPY $d0
+    %cst_1:_(s32) = G_CONSTANT i32 1
+    %vec_cst_1:_(<2 x s32>) = G_BUILD_VECTOR %cst_1, %cst_1
+    %icmp:_(<2 x s32>) = G_ICMP intpred(eq), %copy(<2 x s32>), %vec_cst_1
+    %ext:_(<2 x s64>) = G_ANYEXT %icmp(<2 x s32>)
+    $q0 = COPY %ext(<2 x s64>)
+    RET_ReallyLR implicit $q0
+
+...
+---
+name:            scalar_fcmp
+legalized:       true
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    liveins: $x0, $d0
+    ; Scalars have 0 or 1, so we want a ZExt.
+
+    ; CHECK-LABEL: name: scalar_fcmp
+    ; CHECK: liveins: $x0, $d0
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: %copy:_(s64) = COPY $d0
+    ; CHECK-NEXT: %cst_1:_(s64) = G_FCONSTANT double 1.000000e+00
+    ; CHECK-NEXT: %fcmp:_(s32) = G_FCMP intpred(eq), %copy(s64), %cst_1
+    ; CHECK-NEXT: %ext:_(s64) = G_ZEXT %fcmp(s32)
+    ; CHECK-NEXT: $x0 = COPY %ext(s64)
+    ; CHECK-NEXT: RET_ReallyLR implicit $x0
+    %copy:_(s64) = COPY $d0
+    %cst_1:_(s64) = G_FCONSTANT double 1.0
+    %fcmp:_(s32) = G_FCMP intpred(eq), %copy(s64), %cst_1
+    %ext:_(s64) = G_ANYEXT %fcmp(s32)
+    $x0 = COPY %ext(s64)
+    RET_ReallyLR implicit $x0