[AArch64] Split LSLFast into Addr and ALU parts

[dmgreen]

Summary

As far as I can tell FeatureLSLFast was originally added to specify that a lsl of <= 3 was cheap when folded into an addressing operand, so should override the one-use checks usually intended to make sure we don't perform redundant work. At a later point it also came to also mean that add x0, x1, x2, lsl N with N <= 4 was cheap, in that it took a single cycle not multiple cycles that more complex adds usually take.

This patch splits those two concepts out into separate subtarget features. The biggest change is the change to AArch64DAGToDAGISel::isWorthFoldingALU, making ALU operations now produce a ADDWrs if the shift is <= 4.

Otherwise the patch is mostly an NFC as it tries to keep the subtarget features the same for each cpu. I believe that the Arm OoO CPUs should eventually be changed to a new subtarget feature that specifies that a shift of 2 or 3 with any extend should be treated as cheap (just not shifts of 1 or 4).

[chill]

LGTM, cheers!

llvm/lib/Target/AArch64/AArch64.td

 
 def FeatureNoNegativeImmediates : SubtargetFeature<"no-neg-immediates",
                                         "NegativeImmediates", "false",
                                         "Convert immediates and instructions "
                                         "to their negated or complemented "
                                         "equivalent when the immediate does "
                                         "not fit in the encoding.">;
 
-def FeatureLSLFast : SubtargetFeature<
-    "lsl-fast", "HasLSLFast", "true",
-    "CPU has a fastpath logical shift of up to 3 places">;
+def FeatureAddrLSLFast : SubtargetFeature<
+    "addr-lsl-fast", "HasAddrLSLFast", "true",
+    "Address operands with logical shift of up to 3 places are cheap">;
+
+def FeatureALULSLFast : SubtargetFeature<
+    "alu-lsl-fast", "HasALULSLFast", "true",
+    "Add/Sub operations with lsl shift <= 4 are cheap">;
 
 def FeatureAggressiveFMA :
   SubtargetFeature<"aggressive-fma",
                    "HasAggressiveFMA",
                    "true",
                    "Enable Aggressive FMA for floating-point.">;
 
 def FeatureAltFPCmp : SubtargetFeature<"altnzcv", "HasAlternativeNZCV", "true",
def TuneA75     : SubtargetFeature<"a75", "ARMProcFamily", "CortexA75",
                                    FeatureFuseAdrpAdd,
                                    FeatureEnableSelectOptimize,
                                    FeaturePredictableSelectIsExpensive]>;
 
 def TuneA76     : SubtargetFeature<"a76", "ARMProcFamily", "CortexA76",
                                    "Cortex-A76 ARM processors", [
                                    FeatureFuseAES,
                                    FeatureFuseAdrpAdd,
-                                   FeatureLSLFast,
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast,
                                    FeatureEnableSelectOptimize,
                                    FeaturePredictableSelectIsExpensive]>;
 
 def TuneA77     : SubtargetFeature<"a77", "ARMProcFamily", "CortexA77",
                                    "Cortex-A77 ARM processors", [
                                    FeatureCmpBccFusion,
                                    FeatureFuseAES,
                                    FeatureFuseAdrpAdd,
-                                   FeatureLSLFast,
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast,
                                    FeatureEnableSelectOptimize,
                                    FeaturePredictableSelectIsExpensive]>;
 
 def TuneA78 : SubtargetFeature<"a78", "ARMProcFamily", "CortexA78",
                                "Cortex-A78 ARM processors", [
                                FeatureCmpBccFusion,
                                FeatureFuseAES,
                                FeatureFuseAdrpAdd,
-                               FeatureLSLFast,
+                               FeatureAddrLSLFast,
+                               FeatureALULSLFast,
                                FeaturePostRAScheduler,
                                FeatureEnableSelectOptimize,
                                FeaturePredictableSelectIsExpensive]>;
 
 def TuneA78C : SubtargetFeature<"a78c", "ARMProcFamily",
                                 "CortexA78C",
                                 "Cortex-A78C ARM processors", [
                                 FeatureCmpBccFusion,
                                 FeatureFuseAES,
                                 FeatureFuseAdrpAdd,
-                                FeatureLSLFast,
+                                FeatureAddrLSLFast,
+                                FeatureALULSLFast,
                                 FeaturePostRAScheduler,
                                 FeatureEnableSelectOptimize,
                                 FeaturePredictableSelectIsExpensive]>;
 
 def TuneA710    : SubtargetFeature<"a710", "ARMProcFamily", "CortexA710",
                                    "Cortex-A710 ARM processors", [
                                    FeatureCmpBccFusion,
                                    FeatureFuseAES,
                                    FeatureFuseAdrpAdd,
-                                   FeatureLSLFast,
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast,
                                    FeaturePostRAScheduler,
                                    FeatureEnableSelectOptimize,
                                    FeaturePredictableSelectIsExpensive]>;
 
 def TuneA715 : SubtargetFeature<"a715", "ARMProcFamily", "CortexA715",
                                  "Cortex-A715 ARM processors", [
                                  FeatureFuseAES,
                                  FeaturePostRAScheduler,
                                  FeatureCmpBccFusion,
-                                 FeatureLSLFast,
+                                 FeatureAddrLSLFast,
+                                 FeatureALULSLFast,
                                  FeatureFuseAdrpAdd,
                                  FeatureEnableSelectOptimize,
                                  FeaturePredictableSelectIsExpensive]>;
 
 def TuneR82 : SubtargetFeature<"cortex-r82", "ARMProcFamily",
                                "CortexR82",
                                "Cortex-R82 ARM processors", [
                                FeaturePostRAScheduler]>;
 
 def TuneX1 : SubtargetFeature<"cortex-x1", "ARMProcFamily", "CortexX1",
                                   "Cortex-X1 ARM processors", [
                                   FeatureCmpBccFusion,
                                   FeatureFuseAES,
                                   FeatureFuseAdrpAdd,
-                                  FeatureLSLFast,
+                                  FeatureAddrLSLFast,
+                                  FeatureALULSLFast,
                                   FeaturePostRAScheduler,
                                   FeatureEnableSelectOptimize,
                                   FeaturePredictableSelectIsExpensive]>;
 
 def TuneX2 : SubtargetFeature<"cortex-x2", "ARMProcFamily", "CortexX2",
                                   "Cortex-X2 ARM processors", [
                                   FeatureCmpBccFusion,
                                   FeatureFuseAES,
                                   FeatureFuseAdrpAdd,
-                                  FeatureLSLFast,
+                                  FeatureAddrLSLFast,
+                                  FeatureALULSLFast,
                                   FeaturePostRAScheduler,
                                   FeatureEnableSelectOptimize,
                                   FeaturePredictableSelectIsExpensive]>;
 
 def TuneX3 : SubtargetFeature<"cortex-x3", "ARMProcFamily", "CortexX3",
                               "Cortex-X3 ARM processors", [
-                               FeatureLSLFast,
+                               FeatureAddrLSLFast,
+                               FeatureALULSLFast,
                                FeatureFuseAdrpAdd,
                                FeatureFuseAES,
                                FeaturePostRAScheduler,
                                FeatureEnableSelectOptimize,
                                FeaturePredictableSelectIsExpensive]>;
 
 def TuneA64FX : SubtargetFeature<"a64fx", "ARMProcFamily", "A64FX",
                                  "Fujitsu A64FX processors", [
def TuneExynosM3 : SubtargetFeature<"exynosm3", "ARMProcFamily", "ExynosM3",
                                     "Samsung Exynos-M3 processors",
                                     [FeatureExynosCheapAsMoveHandling,
                                      FeatureForce32BitJumpTables,
                                      FeatureFuseAddress,
                                      FeatureFuseAES,
                                      FeatureFuseCCSelect,
                                      FeatureFuseAdrpAdd,
                                      FeatureFuseLiterals,
-                                     FeatureLSLFast,
+                                     FeatureAddrLSLFast,
+                                     FeatureALULSLFast,
                                      FeaturePostRAScheduler,
                                      FeaturePredictableSelectIsExpensive]>;
 
 // Re-uses some scheduling and tunings from the ExynosM3 proc family.
 def TuneExynosM4 : SubtargetFeature<"exynosm4", "ARMProcFamily", "ExynosM3",
                                     "Samsung Exynos-M4 processors",
                                     [FeatureArithmeticBccFusion,
                                      FeatureArithmeticCbzFusion,
                                      FeatureExynosCheapAsMoveHandling,
                                      FeatureForce32BitJumpTables,
                                      FeatureFuseAddress,
                                      FeatureFuseAES,
                                      FeatureFuseArithmeticLogic,
                                      FeatureFuseCCSelect,
                                      FeatureFuseAdrpAdd,
                                      FeatureFuseLiterals,
-                                     FeatureLSLFast,
+                                     FeatureAddrLSLFast,
+                                     FeatureALULSLFast,
                                      FeaturePostRAScheduler,
                                      FeatureZCZeroing]>;
 
 def TuneKryo    : SubtargetFeature<"kryo", "ARMProcFamily", "Kryo",
                                    "Qualcomm Kryo processors", [
                                    FeatureCustomCheapAsMoveHandling,
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive,
                                    FeatureZCZeroing,
-                                   FeatureLSLFast]
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast]
                                    >;
 
 def TuneFalkor  : SubtargetFeature<"falkor", "ARMProcFamily", "Falkor",
                                    "Qualcomm Falkor processors", [
                                    FeatureCustomCheapAsMoveHandling,
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive,
                                    FeatureZCZeroing,
-                                   FeatureLSLFast,
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast,
                                    FeatureSlowSTRQro
                                    ]>;
 
 def TuneNeoverseE1 : SubtargetFeature<"neoversee1", "ARMProcFamily", "NeoverseE1",
                                       "Neoverse E1 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
                                       FeaturePostRAScheduler]>;
 
 def TuneNeoverseN1 : SubtargetFeature<"neoversen1", "ARMProcFamily", "NeoverseN1",
                                       "Neoverse N1 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
-                                      FeatureLSLFast,
+                                      FeatureAddrLSLFast,
+                                      FeatureALULSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive]>;
 
 def TuneNeoverseN2 : SubtargetFeature<"neoversen2", "ARMProcFamily", "NeoverseN2",
                                       "Neoverse N2 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
-                                      FeatureLSLFast,
+                                      FeatureAddrLSLFast,
+                                      FeatureALULSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive]>;
 
 def TuneNeoverse512TVB : SubtargetFeature<"neoverse512tvb", "ARMProcFamily", "Neoverse512TVB",
                                       "Neoverse 512-TVB ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
-                                      FeatureLSLFast,
+                                      FeatureAddrLSLFast,
+                                      FeatureALULSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive]>;
 
 def TuneNeoverseV1 : SubtargetFeature<"neoversev1", "ARMProcFamily", "NeoverseV1",
                                       "Neoverse V1 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
-                                      FeatureLSLFast,
+                                      FeatureAddrLSLFast,
+                                      FeatureALULSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive,
                                       FeatureNoSVEFPLD1R]>;
 
 def TuneNeoverseV2 : SubtargetFeature<"neoversev2", "ARMProcFamily", "NeoverseV2",
                                       "Neoverse V2 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
-                                      FeatureLSLFast,
+                                      FeatureAddrLSLFast,
+                                      FeatureALULSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive]>;
 
 def TuneSaphira  : SubtargetFeature<"saphira", "ARMProcFamily", "Saphira",
                                    "Qualcomm Saphira processors", [
                                    FeatureCustomCheapAsMoveHandling,
                                    FeaturePostRAScheduler,
                                    FeaturePredictableSelectIsExpensive,
                                    FeatureZCZeroing,
-                                   FeatureLSLFast]>;
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast]>;
 
 def TuneThunderX2T99  : SubtargetFeature<"thunderx2t99", "ARMProcFamily", "ThunderX2T99",
                                          "Cavium ThunderX2 processors", [
                                           FeatureAggressiveFMA,
                                           FeatureArithmeticBccFusion,
                                           FeaturePostRAScheduler,
                                           FeaturePredictableSelectIsExpensive]>;
 
def TuneTSV110 : SubtargetFeature<"tsv110", "ARMProcFamily", "TSV110",
                                   FeatureCustomCheapAsMoveHandling,
                                   FeatureFuseAES,
                                   FeaturePostRAScheduler]>;
 
 def TuneAmpere1 : SubtargetFeature<"ampere1", "ARMProcFamily", "Ampere1",
                                    "Ampere Computing Ampere-1 processors", [
                                    FeaturePostRAScheduler,
                                    FeatureFuseAES,
-                                   FeatureLSLFast,
+                                   FeatureAddrLSLFast,
+                                   FeatureALULSLFast,
                                    FeatureAggressiveFMA,
                                    FeatureArithmeticBccFusion,
                                    FeatureCmpBccFusion,
                                    FeatureFuseAddress,
                                    FeatureFuseLiterals]>;
 
 def TuneAmpere1A : SubtargetFeature<"ampere1a", "ARMProcFamily", "Ampere1A",
                                     "Ampere Computing Ampere-1A processors", [
                                     FeaturePostRAScheduler,
                                     FeatureFuseAES,
-                                    FeatureLSLFast,
+                                    FeatureAddrLSLFast,
+                                    FeatureALULSLFast,
                                     FeatureAggressiveFMA,
                                     FeatureArithmeticBccFusion,
                                     FeatureCmpBccFusion,
                                     FeatureFuseAddress,
                                     FeatureFuseLiterals,
                                     FeatureFuseLiterals]>;
 
 def ProcessorFeatures {

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

private:
   bool SelectAddrModeUnscaled(SDValue N, unsigned Size, SDValue &Base,
                               SDValue &OffImm);
   bool SelectAddrModeWRO(SDValue N, unsigned Size, SDValue &Base,
                          SDValue &Offset, SDValue &SignExtend,
                          SDValue &DoShift);
   bool SelectAddrModeXRO(SDValue N, unsigned Size, SDValue &Base,
                          SDValue &Offset, SDValue &SignExtend,
                          SDValue &DoShift);
-  bool isWorthFolding(SDValue V) const;
+  bool isWorthFoldingALU(SDValue V, bool LSL = false) const;
+  bool isWorthFoldingAddr(SDValue V) const;
   bool SelectExtendedSHL(SDValue N, unsigned Size, bool WantExtend,
                          SDValue &Offset, SDValue &SignExtend);
 
   template<unsigned RegWidth>
   bool SelectCVTFixedPosOperand(SDValue N, SDValue &FixedPos) {
     return SelectCVTFixedPosOperand(N, FixedPos, RegWidth);
   }
 
static bool isWorthFoldingSHL(SDValue V) {
   for (SDNode *UI : Node->uses())
     if (!isa<MemSDNode>(*UI))
       for (SDNode *UII : UI->uses())
         if (!isa<MemSDNode>(*UII))
           return false;
   return true;
 }
 
-/// Determine whether it is worth to fold V into an extended register.
-bool AArch64DAGToDAGISel::isWorthFolding(SDValue V) const {
+/// Determine whether it is worth to fold V into an extended register addressing
+/// mode.
+bool AArch64DAGToDAGISel::isWorthFoldingAddr(SDValue V) const {
   // Trivial if we are optimizing for code size or if there is only
   // one use of the value.
   if (CurDAG->shouldOptForSize() || V.hasOneUse())
     return true;
   // If a subtarget has a fastpath LSL we can fold a logical shift into
   // the addressing mode and save a cycle.
-  if (Subtarget->hasLSLFast() && V.getOpcode() == ISD::SHL &&
+  if (Subtarget->hasAddrLSLFast() && V.getOpcode() == ISD::SHL &&
       isWorthFoldingSHL(V))
     return true;
-  if (Subtarget->hasLSLFast() && V.getOpcode() == ISD::ADD) {
+  if (Subtarget->hasAddrLSLFast() && V.getOpcode() == ISD::ADD) {
     const SDValue LHS = V.getOperand(0);
     const SDValue RHS = V.getOperand(1);
     if (LHS.getOpcode() == ISD::SHL && isWorthFoldingSHL(LHS))
       return true;
     if (RHS.getOpcode() == ISD::SHL && isWorthFoldingSHL(RHS))
       return true;
   }
 
bool AArch64DAGToDAGISel::SelectShiftedRegisterFromAnd(SDValue N, SDValue &Reg,
   Reg = SDValue(CurDAG->getMachineNode(NewShiftOp, DL, VT, LHS->getOperand(0),
                                        NewShiftAmt, BitWidthMinus1),
                 0);
   unsigned ShVal = AArch64_AM::getShifterImm(AArch64_AM::LSL, LowZBits);
   Shift = CurDAG->getTargetConstant(ShVal, DL, MVT::i32);
   return true;
 }
 
-/// SelectShiftedRegister - Select a "shifted register" operand.  If the value
-/// is not shifted, set the Shift operand to default of "LSL 0".  The logical
-/// instructions allow the shifted register to be rotated, but the arithmetic
-/// instructions do not.  The AllowROR parameter specifies whether ROR is
-/// supported.
-bool AArch64DAGToDAGISel::SelectShiftedRegister(SDValue N, bool AllowROR,
-                                                SDValue &Reg, SDValue &Shift) {
-  if (SelectShiftedRegisterFromAnd(N, Reg, Shift))
-    return true;
-
-  AArch64_AM::ShiftExtendType ShType = getShiftTypeForNode(N);
-  if (ShType == AArch64_AM::InvalidShiftExtend)
-    return false;
-  if (!AllowROR && ShType == AArch64_AM::ROR)
-    return false;
-
-  if (ConstantSDNode *RHS = dyn_cast<ConstantSDNode>(N.getOperand(1))) {
-    unsigned BitSize = N.getValueSizeInBits();
-    unsigned Val = RHS->getZExtValue() & (BitSize - 1);
-    unsigned ShVal = AArch64_AM::getShifterImm(ShType, Val);
-
-    Reg = N.getOperand(0);
-    Shift = CurDAG->getTargetConstant(ShVal, SDLoc(N), MVT::i32);
-    return isWorthFolding(N);
-  }
-
-  return false;
-}
-
 /// getExtendTypeForNode - Translate an extend node to the corresponding
 /// ExtendType value.
 static AArch64_AM::ShiftExtendType
 getExtendTypeForNode(SDValue N, bool IsLoadStore = false) {
   if (N.getOpcode() == ISD::SIGN_EXTEND ||
       N.getOpcode() == ISD::SIGN_EXTEND_INREG) {
     EVT SrcVT;
     if (N.getOpcode() == ISD::SIGN_EXTEND_INREG)
if (N.getOpcode() == ISD::SIGN_EXTEND ||
     case 0xFFFFFFFF:
       return AArch64_AM::UXTW;
     }
   }
 
   return AArch64_AM::InvalidShiftExtend;
 }
 
+/// Determine whether it is worth to fold V into an extended register of an
+/// Add/Sub. LSL means we are folding into an `add w0, w1, w2, lsl #N`
+/// instruction, and the shift should be treated as worth folding even if has
+/// multiple uses.
+bool AArch64DAGToDAGISel::isWorthFoldingALU(SDValue V, bool LSL) const {
+  // Trivial if we are optimizing for code size or if there is only
+  // one use of the value.
+  if (CurDAG->shouldOptForSize() || V.hasOneUse())
+    return true;
+
+  // If a subtarget has a fastpath LSL we can fold a logical shift into
+  // the add/sub and save a cycle.
+  if (LSL && Subtarget->hasALULSLFast() && V.getOpcode() == ISD::SHL &&
+      V.getConstantOperandVal(1) <= 4 &&
+      getExtendTypeForNode(V.getOperand(0)) == AArch64_AM::InvalidShiftExtend)
+    return true;
+
+  // It hurts otherwise, since the value will be reused.
+  return false;
+}
+
+/// SelectShiftedRegister - Select a "shifted register" operand.  If the value
+/// is not shifted, set the Shift operand to default of "LSL 0".  The logical
+/// instructions allow the shifted register to be rotated, but the arithmetic
+/// instructions do not.  The AllowROR parameter specifies whether ROR is
+/// supported.
+bool AArch64DAGToDAGISel::SelectShiftedRegister(SDValue N, bool AllowROR,
+                                                SDValue &Reg, SDValue &Shift) {
+  if (SelectShiftedRegisterFromAnd(N, Reg, Shift))
+    return true;
+
+  AArch64_AM::ShiftExtendType ShType = getShiftTypeForNode(N);
+  if (ShType == AArch64_AM::InvalidShiftExtend)
+    return false;
+  if (!AllowROR && ShType == AArch64_AM::ROR)
+    return false;
+
+  if (ConstantSDNode *RHS = dyn_cast<ConstantSDNode>(N.getOperand(1))) {
+    unsigned BitSize = N.getValueSizeInBits();
+    unsigned Val = RHS->getZExtValue() & (BitSize - 1);
+    unsigned ShVal = AArch64_AM::getShifterImm(ShType, Val);
+
+    Reg = N.getOperand(0);
+    Shift = CurDAG->getTargetConstant(ShVal, SDLoc(N), MVT::i32);
+    return isWorthFoldingALU(N, true);
+  }
+
+  return false;
+}
+
 /// Instructions that accept extend modifiers like UXTW expect the register
 /// being extended to be a GPR32, but the incoming DAG might be acting on a
 /// GPR64 (either via SEXT_INREG or AND). Extract the appropriate low bits if
 /// this is the case.
 static SDValue narrowIfNeeded(SelectionDAG *CurDAG, SDValue N) {
   if (N.getValueType() == MVT::i32)
     return N;
 
bool AArch64DAGToDAGISel::SelectArithExtendedRegister(SDValue N, SDValue &Reg,
   // register class that could contain the size being extended from.  Thus,
   // if we're folding a (sext i8), we need the RHS to be a GPR32, even though
   // there might not be an actual 32-bit value in the program.  We can
   // (harmlessly) synthesize one by injected an EXTRACT_SUBREG here.
   assert(Ext != AArch64_AM::UXTX && Ext != AArch64_AM::SXTX);
   Reg = narrowIfNeeded(CurDAG, Reg);
   Shift = CurDAG->getTargetConstant(getArithExtendImm(Ext, ShiftVal), SDLoc(N),
                                     MVT::i32);
-  return isWorthFolding(N);
+  return isWorthFoldingALU(N);
 }
 
 /// SelectArithUXTXRegister - Select a "UXTX register" operand. This
 /// operand is refered by the instructions have SP operand
 bool AArch64DAGToDAGISel::SelectArithUXTXRegister(SDValue N, SDValue &Reg,
                                                   SDValue &Shift) {
   unsigned ShiftVal = 0;
   AArch64_AM::ShiftExtendType Ext;
 
   if (N.getOpcode() != ISD::SHL)
     return false;
 
   ConstantSDNode *CSD = dyn_cast<ConstantSDNode>(N.getOperand(1));
   if (!CSD)
     return false;
   ShiftVal = CSD->getZExtValue();
   if (ShiftVal > 4)
     return false;
 
   Ext = AArch64_AM::UXTX;
   Reg = N.getOperand(0);
   Shift = CurDAG->getTargetConstant(getArithExtendImm(Ext, ShiftVal), SDLoc(N),
                                     MVT::i32);
-  return isWorthFolding(N);
+  return isWorthFoldingALU(N);
 }
 
 /// If there's a use of this ADDlow that's not itself a load/store then we'll
 /// need to create a real ADD instruction from it anyway and there's no point in
 /// folding it into the mem op. Theoretically, it shouldn't matter, but there's
 /// a single pseudo-instruction for an ADRP/ADD pair so over-aggressive folding
 /// leads to duplicated ADRP instructions.
 static bool isWorthFoldingADDlow(SDValue N) {
bool AArch64DAGToDAGISel::SelectExtendedSHL(SDValue N, unsigned Size,
   }
 
   unsigned LegalShiftVal = Log2_32(Size);
   unsigned ShiftVal = CSD->getZExtValue();
 
   if (ShiftVal != 0 && ShiftVal != LegalShiftVal)
     return false;
 
-  return isWorthFolding(N);
+  return isWorthFoldingAddr(N);
 }
 
 bool AArch64DAGToDAGISel::SelectAddrModeWRO(SDValue N, unsigned Size,
                                             SDValue &Base, SDValue &Offset,
                                             SDValue &SignExtend,
                                             SDValue &DoShift) {
   if (N.getOpcode() != ISD::ADD)
     return false;
bool AArch64DAGToDAGISel::SelectAddrModeWRO(SDValue N, unsigned Size,
   // computation, since the computation will be kept.
   const SDNode *Node = N.getNode();
   for (SDNode *UI : Node->uses()) {
     if (!isa<MemSDNode>(*UI))
       return false;
   }
 
   // Remember if it is worth folding N when it produces extended register.
-  bool IsExtendedRegisterWorthFolding = isWorthFolding(N);
+  bool IsExtendedRegisterWorthFolding = isWorthFoldingAddr(N);
 
   // Try to match a shifted extend on the RHS.
   if (IsExtendedRegisterWorthFolding && RHS.getOpcode() == ISD::SHL &&
       SelectExtendedSHL(RHS, Size, true, Offset, SignExtend)) {
     Base = LHS;
     DoShift = CurDAG->getTargetConstant(true, dl, MVT::i32);
     return true;
   }
bool AArch64DAGToDAGISel::SelectAddrModeWRO(SDValue N, unsigned Size,
   // Try to match an unshifted extend on the LHS.
   if (IsExtendedRegisterWorthFolding &&
       (Ext = getExtendTypeForNode(LHS, true)) !=
           AArch64_AM::InvalidShiftExtend) {
     Base = RHS;
     Offset = narrowIfNeeded(CurDAG, LHS.getOperand(0));
     SignExtend = CurDAG->getTargetConstant(Ext == AArch64_AM::SXTW, dl,
                                            MVT::i32);
-    if (isWorthFolding(LHS))
+    if (isWorthFoldingAddr(LHS))
       return true;
   }
 
   // Try to match an unshifted extend on the RHS.
   if (IsExtendedRegisterWorthFolding &&
       (Ext = getExtendTypeForNode(RHS, true)) !=
           AArch64_AM::InvalidShiftExtend) {
     Base = LHS;
     Offset = narrowIfNeeded(CurDAG, RHS.getOperand(0));
     SignExtend = CurDAG->getTargetConstant(Ext == AArch64_AM::SXTW, dl,
                                            MVT::i32);
-    if (isWorthFolding(RHS))
+    if (isWorthFoldingAddr(RHS))
       return true;
   }
 
   return false;
 }
 
 // Check if the given immediate is preferred by ADD. If an immediate can be
 // encoded in an ADD, or it can be encoded in an "ADD LSL #12" and can not be
if (isa<ConstantSDNode>(RHS)) {
     SDNode *MOVI =
         CurDAG->getMachineNode(AArch64::MOVi64imm, DL, MVT::i64, Ops);
     SDValue MOVIV = SDValue(MOVI, 0);
     // This ADD of two X register will be selected into [Reg+Reg] mode.
     N = CurDAG->getNode(ISD::ADD, DL, MVT::i64, LHS, MOVIV);
   }
 
   // Remember if it is worth folding N when it produces extended register.
-  bool IsExtendedRegisterWorthFolding = isWorthFolding(N);
+  bool IsExtendedRegisterWorthFolding = isWorthFoldingAddr(N);
 
   // Try to match a shifted extend on the RHS.
   if (IsExtendedRegisterWorthFolding && RHS.getOpcode() == ISD::SHL &&
       SelectExtendedSHL(RHS, Size, false, Offset, SignExtend)) {
     Base = LHS;
     DoShift = CurDAG->getTargetConstant(true, DL, MVT::i32);
     return true;
   }

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (SCVMinus1.isPowerOf2()) {
       ShiftAmt = SCVMinus1.logBase2();
       return Shl(Add(Shl(N0, ShiftAmt), N0), TrailingZeroes);
     } else if (CVPlus1.isPowerOf2()) {
       ShiftAmt = CVPlus1.logBase2();
       return Sub(Shl(N0, ShiftAmt), N0);
     } else if (SCVPlus1.isPowerOf2()) {
       ShiftAmt = SCVPlus1.logBase2() + TrailingZeroes;
       return Sub(Shl(N0, ShiftAmt), Shl(N0, TrailingZeroes));
-    } else if (Subtarget->hasLSLFast() &&
+    } else if (Subtarget->hasALULSLFast() &&
                isPowPlusPlusConst(ConstValue, CVM, CVN)) {
       APInt CVMMinus1 = CVM - 1;
       APInt CVNMinus1 = CVN - 1;
       unsigned ShiftM1 = CVMMinus1.logBase2();
       unsigned ShiftN1 = CVNMinus1.logBase2();
       // LSLFast implicate that Shifts <= 3 places are fast
       if (ShiftM1 <= 3 && ShiftN1 <= 3) {
         SDValue MVal = Add(Shl(N0, ShiftM1), N0);

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

bool AArch64InstructionSelector::isWorthFoldingIntoExtendedReg(
   // Always fold if there is one use, or if we're optimizing for size.
   Register DefReg = MI.getOperand(0).getReg();
   if (MRI.hasOneNonDBGUse(DefReg) ||
       MI.getParent()->getParent()->getFunction().hasOptSize())
     return true;
 
   // It's better to avoid folding and recomputing shifts when we don't have a
   // fastpath.
-  if (!STI.hasLSLFast())
+  if (!STI.hasAddrLSLFast())
     return false;
 
   // We have a fastpath, so folding a shift in and potentially computing it
   // many times may be beneficial. Check if this is only used in memory ops.
   // If it is, then we should fold.
   return all_of(MRI.use_nodbg_instructions(DefReg),
                 [](MachineInstr &Use) { return Use.mayLoadOrStore(); });
 }

llvm/test/CodeGen/AArch64/GlobalISel/load-addressing-modes.mir

--- |
   define void @more_than_one_use_shl_lsl_slow(i64* %addr) { ret void }
   define void @more_than_one_use_shl_minsize(i64* %addr) #0 { ret void }
   define void @ldrwrox(i64* %addr) { ret void }
   define void @ldrsrox(i64* %addr) { ret void }
   define void @ldrhrox(i64* %addr) { ret void }
   define void @ldbbrox(i64* %addr) { ret void }
   define void @ldrqrox(i64* %addr) { ret void }
   attributes #0 = { optsize }
-  attributes #1 = { "target-features"="+lsl-fast" }
+  attributes #1 = { "target-features"="+addr-lsl-fast" }
 ...
 
 ---
 name:            ldrxrox_breg_oreg
 alignment:       4
 legalized:       true
 regBankSelected: true
 tracksRegLiveness: true

llvm/test/CodeGen/AArch64/aarch64-fold-lslfast.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s --check-prefixes=CHECK,CHECK0
-; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+lsl-fast | FileCheck %s --check-prefixes=CHECK,CHECK3
+; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+addr-lsl-fast | FileCheck %s --check-prefixes=CHECK,CHECK3
 
 %struct.a = type [256 x i16]
 %struct.b = type [256 x i32]
 %struct.c = type [256 x i64]
 
 declare void @foo()
 define i16 @halfword(ptr %ctx, i32 %xor72) nounwind {
 ; CHECK0-LABEL: halfword:

llvm/test/CodeGen/AArch64/lslfast.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s --check-prefixes=CHECK,CHECK-SLOW
+; RUN: llc < %s -mtriple=aarch64-linux-gnu -mattr=+alu-lsl-fast | FileCheck %s --check-prefixes=CHECK,CHECK-FAST
+
+define i32 @testmul3(i32 noundef %x, i32 noundef %y, i32 noundef %z) {
+; CHECK-SLOW-LABEL: testmul3:
+; CHECK-SLOW:       // %bb.0: // %entry
+; CHECK-SLOW-NEXT:    lsl w8, w0, #3
+; CHECK-SLOW-NEXT:    add w9, w8, w1
+; CHECK-SLOW-NEXT:    add w8, w8, w2
+; CHECK-SLOW-NEXT:    mul w0, w8, w9
+; CHECK-SLOW-NEXT:    ret
+;
+; CHECK-FAST-LABEL: testmul3:
+; CHECK-FAST:       // %bb.0: // %entry
+; CHECK-FAST-NEXT:    add w8, w1, w0, lsl #3
+; CHECK-FAST-NEXT:    add w9, w2, w0, lsl #3
+; CHECK-FAST-NEXT:    mul w0, w9, w8
+; CHECK-FAST-NEXT:    ret
+entry:
+  %shl = shl i32 %x, 3
+  %add = add nsw i32 %shl, %y
+  %add2 = add nsw i32 %shl, %z
+  %mul = mul nsw i32 %add2, %add
+  ret i32 %mul
+}
+
+define i32 @testvar(i32 noundef %x, i32 noundef %y, i32 noundef %z, i32 %zz) {
+; CHECK-LABEL: testvar:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    lsl w8, w0, w3
+; CHECK-NEXT:    add w9, w8, w1
+; CHECK-NEXT:    add w8, w8, w2
+; CHECK-NEXT:    mul w0, w8, w9
+; CHECK-NEXT:    ret
+entry:
+  %shl = shl i32 %x, %zz
+  %add = add nsw i32 %shl, %y
+  %add2 = add nsw i32 %shl, %z
+  %mul = mul nsw i32 %add2, %add
+  ret i32 %mul
+}
+
+define i32 @testmul5(i32 noundef %x, i32 noundef %y, i32 noundef %z) {
+; CHECK-LABEL: testmul5:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    lsl w8, w0, #5
+; CHECK-NEXT:    add w9, w8, w1
+; CHECK-NEXT:    add w8, w8, w2
+; CHECK-NEXT:    mul w0, w8, w9
+; CHECK-NEXT:    ret
+entry:
+  %shl = shl i32 %x, 5
+  %add = add nsw i32 %shl, %y
+  %add2 = add nsw i32 %shl, %z
+  %mul = mul nsw i32 %add2, %add
+  ret i32 %mul
+}
+
+define i64 @testsext3(i32 noundef %x, i64 noundef %y, i64 noundef %z) {
+; CHECK-LABEL: testsext3:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
+; CHECK-NEXT:    sbfiz x8, x0, #3, #32
+; CHECK-NEXT:    add x9, x8, x1
+; CHECK-NEXT:    add x8, x8, x2
+; CHECK-NEXT:    mul x0, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %conv = sext i32 %x to i64
+  %shl = shl nsw i64 %conv, 3
+  %add = add nsw i64 %shl, %y
+  %add3 = add nsw i64 %shl, %z
+  %mul = mul nsw i64 %add, %add3
+  ret i64 %mul
+}
+
+define i64 @testzext3(i32 noundef %x, i64 noundef %y, i64 noundef %z) {
+; CHECK-LABEL: testzext3:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    // kill: def $w0 killed $w0 def $x0
+; CHECK-NEXT:    ubfiz x8, x0, #3, #32
+; CHECK-NEXT:    add x9, x8, x1
+; CHECK-NEXT:    add x8, x8, x2
+; CHECK-NEXT:    mul x0, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %conv = zext i32 %x to i64
+  %shl = shl nsw i64 %conv, 3
+  %add = add nsw i64 %shl, %y
+  %add3 = add nsw i64 %shl, %z
+  %mul = mul nsw i64 %add, %add3
+  ret i64 %mul
+}
+
+define i64 @test3sext(i32 noundef %x, i64 noundef %y, i64 noundef %z) {
+; CHECK-LABEL: test3sext:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    lsl w8, w0, #3
+; CHECK-NEXT:    sxtw x8, w8
+; CHECK-NEXT:    add x9, x8, x1
+; CHECK-NEXT:    add x8, x8, x2
+; CHECK-NEXT:    mul x0, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %shl = shl i32 %x, 3
+  %conv = sext i32 %shl to i64
+  %add = add nsw i64 %conv, %y
+  %add3 = add nsw i64 %conv, %z
+  %mul = mul nsw i64 %add, %add3
+  ret i64 %mul
+}
+
+define i64 @test3zext(i32 noundef %x, i64 noundef %y, i64 noundef %z) {
+; CHECK-LABEL: test3zext:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    lsl w8, w0, #3
+; CHECK-NEXT:    add x9, x8, x1
+; CHECK-NEXT:    add x8, x8, x2
+; CHECK-NEXT:    mul x0, x9, x8
+; CHECK-NEXT:    ret
+entry:
+  %shl = shl i32 %x, 3
+  %conv = zext i32 %shl to i64
+  %add = add nsw i64 %conv, %y
+  %add3 = add nsw i64 %conv, %z
+  %mul = mul nsw i64 %add, %add3
+  ret i64 %mul
+}

llvm/test/CodeGen/AArch64/mul_pow2.ll

 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    lsl w0, w0, #4
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 16
   ret i32 %mul
 }
 
-define i32 @test25_fast_shift(i32 %x) "target-features"="+lsl-fast" {
+define i32 @test25_fast_shift(i32 %x) "target-features"="+alu-lsl-fast" {
 ; CHECK-LABEL: test25_fast_shift:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add w8, w0, w0, lsl #2
 ; CHECK-NEXT:    add w0, w8, w8, lsl #2
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test25_fast_shift:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #25
 ; GISEL-NEXT:    mul w0, w0, w8
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 25 ; 25 = (1+4)*(1+4)
   ret i32 %mul
 }
 
-define i32 @test45_fast_shift(i32 %x) "target-features"="+lsl-fast" {
+define i32 @test45_fast_shift(i32 %x) "target-features"="+alu-lsl-fast" {
 ; CHECK-LABEL: test45_fast_shift:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add w8, w0, w0, lsl #2
 ; CHECK-NEXT:    add w0, w8, w8, lsl #3
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test45_fast_shift:
 ; GISEL:       // %bb.0:

 ; GISEL-NEXT:    mul w0, w0, w8
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 45 ; 45 = (1+4)*(1+8)
   ret i32 %mul
 }
 
 ; Negative test: The shift amount 4 larger than 3
-define i32 @test85_fast_shift(i32 %x) "target-features"="+lsl-fast" {
+define i32 @test85_fast_shift(i32 %x) "target-features"="+alu-lsl-fast" {
 ; CHECK-LABEL: test85_fast_shift:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #85
 ; CHECK-NEXT:    mul w0, w0, w8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test85_fast_shift:
 ; GISEL:       // %bb.0:
 ; GISEL-NEXT:    mov w8, #85
 ; GISEL-NEXT:    mul w0, w0, w8
 ; GISEL-NEXT:    ret
 
   %mul = mul nsw i32 %x, 85 ; 85 = (1+4)*(1+16)
   ret i32 %mul
 }
 
 ; Negative test: The shift amount 5 larger than 3
-define i32 @test297_fast_shift(i32 %x) "target-features"="+lsl-fast" {
+define i32 @test297_fast_shift(i32 %x) "target-features"="+alu-lsl-fast" {
 ; CHECK-LABEL: test297_fast_shift:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #297
 ; CHECK-NEXT:    mul w0, w0, w8
 ; CHECK-NEXT:    ret
 ;
 ; GISEL-LABEL: test297_fast_shift:
 ; GISEL:       // %bb.0: