[PowerPC] exclude ICmpZero Use in LSR if icmp can be replaced inside hardware loop.

[shchenz]

Summary

Currently pass LSR runs before PPCCTRLoops, we meet some problem with this order because of the icmp inside loop which compares iteration indexed and loop count. In pass LSR, we always treat that icmp as a valid ICmpZero type LSRUse. But this is not true because in later pass HardwareLoops pass, we may replace this icmp with ctrloop instruction mtctr/bdnz. So we may get suboptimal code based on this order.

In Patch https://reviews.llvm.org/D62847, I made a patch which reorders LSR Pass and HardwareLoops pass. By reordering, LSR knows precisely whether the icmp is replaced or kept in HardwareLoops pass. But reviewers is uneasy about that patch and suggests we should add a target hook in LSR.

This patch add a target hook canSaveCmp() in TTI, it returns true only when we are sure that current handling loop in LSR is a hardware loop. If the loop is a hardware loop, we should not generate formulae for this icmp.

[shchenz]

llvm/test/CodeGen/PowerPC/negctr.ll

38 ↗

(On Diff #205268)

This is a known deg. With this patch,
Before Codegen Prepare Pass:

for.body:                                         ; preds = %for.body.preheader, %for.body
  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 1, %for.body.preheader ]
  %indvars.iv.next = add i64 %indvars.iv, 1
  %exitcond = icmp eq i64 %indvars.iv.next, 0
  br i1 %exitcond, label %for.end.loopexit, label %for.body

After Codegen Prepare Pass:

for.body:                                         ; preds = %for.body.preheader, %for.body
  %indvars.iv = phi i64 [ %math, %for.body ], [ 1, %for.body.preheader ]
  %0 = call { i64, i1 } @llvm.uadd.with.overflow.i64(i64 %indvars.iv, i64 1)
  %math = extractvalue { i64, i1 } %0, 0
  %ov = extractvalue { i64, i1 } %0, 1
  br i1 %ov, label %for.end, label %for.body

HardwareLoops Pass currently can not recognize uadd Intrinsic + extractvalue as a hardware loop(can not get loop exit count). Maybe we need to modify the logic in Codegen Prepare Pass: if the loop is a hardware loop, we should not optimize the cmp to uadd/usub intrinsic?

[shchenz]

rebase

[joanlluch]

I recently sent a new LSR patch proposal, D63692, that to some extent runs in the opposite direction than this one. I mean, I am adding an additional initial formula for ICmpZero as opposed to excluding it. I think that both patches are however fully compatible and probably worth merging together to get the best of all platforms. I added @shchenz, the author of this, as a D63692 reviewer so he can post any considerations.

[hfinkel]

llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

893 ↗

(On Diff #205493)

This is duplicating applicibility logic from the HardwareLoops pass? Can we move this into a utility function, maybe something like HardwareLoopInfo::canAnalyze(Loop *L)?

llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp

3271 ↗

(On Diff #205493)

saveCmp -> SaveCmp

llvm/test/CodeGen/PowerPC/negctr.ll

38 ↗

(On Diff #205268)

Yes, or teach SCEV about the uadd/usub pattern? I lean toward making SCEV smarter.

[hfinkel]

llvm/test/CodeGen/PowerPC/negctr.ll

38 ↗

(On Diff #205268)

(and, just to clarify, that should be a separate patch, so the FIXME here is fine. although please add a note about the uadd/usub and GCP in the comment).

[shchenz]

Thanks for your comments @hfinkel .
Updated accordingly.

For the FIXME case, yes, I plan to fix it in a seperated patch later. Get your idea about fixing it by making SCEV recognize loops with uadd/usub intrinsic.

[shchenz]

call target hook function outside the loop, it should be called once per loop.

[hfinkel]

LGTM

llvm/trunk/include/llvm/Analysis/TargetTransformInfo.h

 #include "llvm/IR/Operator.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/AtomicOrdering.h"
 #include "llvm/Support/DataTypes.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/ScalarEvolution.h"
 #include "llvm/IR/Dominators.h"
+#include "llvm/Analysis/AssumptionCache.h"
 #include <functional>
 
 namespace llvm {
 
 namespace Intrinsic {
 enum ID : unsigned;
 }
 
public:
   bool isLSRCostLess(TargetTransformInfo::LSRCost &C1,
                      TargetTransformInfo::LSRCost &C2) const;
 
   /// Return true if the target can fuse a compare and branch.
   /// Loop-strength-reduction (LSR) uses that knowledge to adjust its cost
   /// calculation for the instructions in a loop.
   bool canMacroFuseCmp() const;
 
+  /// Return true if the target can save a compare for loop count, for example
+  /// hardware loop saves a compare.
+  bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE, LoopInfo *LI,
+                  DominatorTree *DT, AssumptionCache *AC,
+                  TargetLibraryInfo *LibInfo) const;
+
   /// \return True is LSR should make efforts to create/preserve post-inc
   /// addressing mode expressions.
   bool shouldFavorPostInc() const;
 
   /// Return true if LSR should make efforts to generate indexed addressing
   /// modes that operate across loop iterations.
   bool shouldFavorBackedgeIndex(const Loop *L) const;
 
public:
   virtual bool isLegalAddressingMode(Type *Ty, GlobalValue *BaseGV,
                                      int64_t BaseOffset, bool HasBaseReg,
                                      int64_t Scale,
                                      unsigned AddrSpace,
                                      Instruction *I) = 0;
   virtual bool isLSRCostLess(TargetTransformInfo::LSRCost &C1,
                              TargetTransformInfo::LSRCost &C2) = 0;
   virtual bool canMacroFuseCmp() = 0;
+  virtual bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE,
+                          LoopInfo *LI, DominatorTree *DT, AssumptionCache *AC,
+                          TargetLibraryInfo *LibInfo) = 0;
   virtual bool shouldFavorPostInc() const = 0;
   virtual bool shouldFavorBackedgeIndex(const Loop *L) const = 0;
   virtual bool isLegalMaskedStore(Type *DataType) = 0;
   virtual bool isLegalMaskedLoad(Type *DataType) = 0;
   virtual bool isLegalNTStore(Type *DataType, unsigned Alignment) = 0;
   virtual bool isLegalNTLoad(Type *DataType, unsigned Alignment) = 0;
   virtual bool isLegalMaskedScatter(Type *DataType) = 0;
   virtual bool isLegalMaskedGather(Type *DataType) = 0;
public:
   }
   bool isLSRCostLess(TargetTransformInfo::LSRCost &C1,
                      TargetTransformInfo::LSRCost &C2) override {
     return Impl.isLSRCostLess(C1, C2);
   }
   bool canMacroFuseCmp() override {
     return Impl.canMacroFuseCmp();
   }
+  bool canSaveCmp(Loop *L, BranchInst **BI,
+                        ScalarEvolution *SE,
+                        LoopInfo *LI, DominatorTree *DT, AssumptionCache *AC,
+                        TargetLibraryInfo *LibInfo) override {
+    return Impl.canSaveCmp(L, BI, SE, LI, DT, AC, LibInfo);
+  }
   bool shouldFavorPostInc() const override {
     return Impl.shouldFavorPostInc();
   }
   bool shouldFavorBackedgeIndex(const Loop *L) const override {
     return Impl.shouldFavorBackedgeIndex(L);
   }
   bool isLegalMaskedStore(Type *DataType) override {
     return Impl.isLegalMaskedStore(DataType);

llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h

bool isLSRCostLess(TTI::LSRCost &C1, TTI::LSRCost &C2) {
     return std::tie(C1.NumRegs, C1.AddRecCost, C1.NumIVMuls, C1.NumBaseAdds,
                     C1.ScaleCost, C1.ImmCost, C1.SetupCost) <
            std::tie(C2.NumRegs, C2.AddRecCost, C2.NumIVMuls, C2.NumBaseAdds,
                     C2.ScaleCost, C2.ImmCost, C2.SetupCost);
   }
 
   bool canMacroFuseCmp() { return false; }
 
+  bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE, LoopInfo *LI,
+                  DominatorTree *DT, AssumptionCache *AC,
+                  TargetLibraryInfo *LibInfo) {
+    return false;
+  }
+
   bool shouldFavorPostInc() const { return false; }
 
   bool shouldFavorBackedgeIndex(const Loop *L) const { return false; }
 
   bool isLegalMaskedStore(Type *DataType) { return false; }
 
   bool isLegalMaskedLoad(Type *DataType) { return false; }
 

llvm/trunk/lib/Analysis/TargetTransformInfo.cpp

 bool TargetTransformInfo::isLSRCostLess(LSRCost &C1, LSRCost &C2) const {
   return TTIImpl->isLSRCostLess(C1, C2);
 }
 
 bool TargetTransformInfo::canMacroFuseCmp() const {
   return TTIImpl->canMacroFuseCmp();
 }
 
+bool TargetTransformInfo::canSaveCmp(Loop *L, BranchInst **BI,
+                                     ScalarEvolution *SE, LoopInfo *LI,
+                                     DominatorTree *DT, AssumptionCache *AC,
+                                     TargetLibraryInfo *LibInfo) const {
+  return TTIImpl->canSaveCmp(L, BI, SE, LI, DT, AC, LibInfo);
+}
+
 bool TargetTransformInfo::shouldFavorPostInc() const {
   return TTIImpl->shouldFavorPostInc();
 }
 
 bool TargetTransformInfo::shouldFavorBackedgeIndex(const Loop *L) const {
   return TTIImpl->shouldFavorBackedgeIndex(L);
 }
 

llvm/trunk/lib/Target/PowerPC/PPCTargetTransformInfo.h

public:
 
   unsigned getUserCost(const User *U, ArrayRef<const Value *> Operands);
 
   TTI::PopcntSupportKind getPopcntSupport(unsigned TyWidth);
   bool isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,
                                 AssumptionCache &AC,
                                 TargetLibraryInfo *LibInfo,
                                 HardwareLoopInfo &HWLoopInfo);
+  bool canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE, LoopInfo *LI,
+                  DominatorTree *DT, AssumptionCache *AC,
+                  TargetLibraryInfo *LibInfo);
   void getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                TTI::UnrollingPreferences &UP);
 
   /// @}
 
   /// \name Vector TTI Implementations
   /// @{
   bool useColdCCForColdCall(Function &F);

llvm/trunk/lib/Target/PowerPC/PPCTargetTransformInfo.cpp

int PPCTTIImpl::getInterleavedMemoryOpCost(unsigned Opcode, Type *VecTy,
   // instruction). For each result vector, we need one shuffle per incoming
   // vector (except that the first shuffle can take two incoming vectors
   // because it does not need to take itself).
   Cost += Factor*(LT.first-1);
 
   return Cost;
 }
 
+bool PPCTTIImpl::canSaveCmp(Loop *L, BranchInst **BI, ScalarEvolution *SE,
+                            LoopInfo *LI, DominatorTree *DT,
+                            AssumptionCache *AC, TargetLibraryInfo *LibInfo) {
+  // Process nested loops first.
+  for (Loop::iterator I = L->begin(), E = L->end(); I != E; ++I)
+    if (canSaveCmp(*I, BI, SE, LI, DT, AC, LibInfo))
+      return false; // Stop search.
+
+  HardwareLoopInfo HWLoopInfo(L);
+
+  if (!HWLoopInfo.canAnalyze(*LI))
+    return false;
+
+  if (!isHardwareLoopProfitable(L, *SE, *AC, LibInfo, HWLoopInfo))
+    return false;
+
+  if (!HWLoopInfo.isHardwareLoopCandidate(*SE, *LI, *DT))
+    return false;
+
+  *BI = HWLoopInfo.ExitBranch;
+  return true;
+}

llvm/trunk/lib/Transforms/Scalar/LoopStrengthReduce.cpp

 };
 
 /// This class holds state for the main loop strength reduction logic.
 class LSRInstance {
   IVUsers &IU;
   ScalarEvolution &SE;
   DominatorTree &DT;
   LoopInfo &LI;
+  AssumptionCache ∾
+  TargetLibraryInfo &LibInfo;
   const TargetTransformInfo &TTI;
   Loop *const L;
   bool FavorBackedgeIndex = false;
   bool Changed = false;
 
   /// This is the insert position that the current loop's induction variable
   /// increment should be placed. In simple loops, this is the latch block's
   /// terminator. But in more complicated cases, this is a position which will
void RewriteForPHI(PHINode *PN, const LSRUse &LU, const LSRFixup &LF,
                      SmallVectorImpl<WeakTrackingVH> &DeadInsts) const;
   void Rewrite(const LSRUse &LU, const LSRFixup &LF, const Formula &F,
                SCEVExpander &Rewriter,
                SmallVectorImpl<WeakTrackingVH> &DeadInsts) const;
   void ImplementSolution(const SmallVectorImpl<const Formula *> &Solution);
 
 public:
   LSRInstance(Loop *L, IVUsers &IU, ScalarEvolution &SE, DominatorTree &DT,
-              LoopInfo &LI, const TargetTransformInfo &TTI);
+              LoopInfo &LI, const TargetTransformInfo &TTI, AssumptionCache &AC,
+              TargetLibraryInfo &LibInfo);
 
   bool getChanged() const { return Changed; }
 
   void print_factors_and_types(raw_ostream &OS) const;
   void print_fixups(raw_ostream &OS) const;
   void print_uses(raw_ostream &OS) const;
   void print(raw_ostream &OS) const;
   void dump() const;
for (PHINode &Phi : L->getHeader()->phis()) {
       }
       Phi.replaceUsesOfWith(PostIncV, IVOper);
       DeadInsts.emplace_back(PostIncV);
     }
   }
 }
 
 void LSRInstance::CollectFixupsAndInitialFormulae() {
+  BranchInst *ExitBranch = nullptr;
+  bool SaveCmp = TTI.canSaveCmp(L, &ExitBranch, &SE, &LI, &DT, &AC, &LibInfo);
+
   for (const IVStrideUse &U : IU) {
     Instruction *UserInst = U.getUser();
     // Skip IV users that are part of profitable IV Chains.
     User::op_iterator UseI =
         find(UserInst->operands(), U.getOperandValToReplace());
     assert(UseI != UserInst->op_end() && "cannot find IV operand");
     if (IVIncSet.count(UseI)) {
       LLVM_DEBUG(dbgs() << "Use is in profitable chain: " << **UseI << '\n');
for (const IVStrideUse &U : IU) {
     // Equality (== and !=) ICmps are special. We can rewrite (i == N) as
     // (N - i == 0), and this allows (N - i) to be the expression that we work
     // with rather than just N or i, so we can consider the register
     // requirements for both N and i at the same time. Limiting this code to
     // equality icmps is not a problem because all interesting loops use
     // equality icmps, thanks to IndVarSimplify.
     if (ICmpInst *CI = dyn_cast<ICmpInst>(UserInst))
       if (CI->isEquality()) {
+        // If CI can be saved in some target, like replaced inside hardware loop
+        // in PowerPC, no need to generate initial formulae for it.
+        if (SaveCmp && CI == cast<ICmpInst>(ExitBranch->getCondition()))
+          continue;
         // Swap the operands if needed to put the OperandValToReplace on the
         // left, for consistency.
         Value *NV = CI->getOperand(1);
         if (NV == U.getOperandValToReplace()) {
           CI->setOperand(1, CI->getOperand(0));
           CI->setOperand(0, NV);
           NV = CI->getOperand(1);
           Changed = true;
#endif
   // instructions.
   Rewriter.clear();
 
   Changed |= DeleteTriviallyDeadInstructions(DeadInsts);
 }
 
 LSRInstance::LSRInstance(Loop *L, IVUsers &IU, ScalarEvolution &SE,
                          DominatorTree &DT, LoopInfo &LI,
-                         const TargetTransformInfo &TTI)
-    : IU(IU), SE(SE), DT(DT), LI(LI), TTI(TTI), L(L),
+                         const TargetTransformInfo &TTI, AssumptionCache &AC,
+                         TargetLibraryInfo &LibInfo)
+    : IU(IU), SE(SE), DT(DT), LI(LI), AC(AC), LibInfo(LibInfo), TTI(TTI), L(L),
       FavorBackedgeIndex(EnableBackedgeIndexing &&
                          TTI.shouldFavorBackedgeIndex(L)) {
   // If LoopSimplify form is not available, stay out of trouble.
   if (!L->isLoopSimplifyForm())
     return;
 
   // If there's no interesting work to be done, bail early.
   if (IU.empty()) return;
void LoopStrengthReduce::getAnalysisUsage(AnalysisUsage &AU) const {
 
   AU.addRequired<LoopInfoWrapperPass>();
   AU.addPreserved<LoopInfoWrapperPass>();
   AU.addRequiredID(LoopSimplifyID);
   AU.addRequired<DominatorTreeWrapperPass>();
   AU.addPreserved<DominatorTreeWrapperPass>();
   AU.addRequired<ScalarEvolutionWrapperPass>();
   AU.addPreserved<ScalarEvolutionWrapperPass>();
+  AU.addRequired<AssumptionCacheTracker>();
+  AU.addRequired<TargetLibraryInfoWrapperPass>();
   // Requiring LoopSimplify a second time here prevents IVUsers from running
   // twice, since LoopSimplify was invalidated by running ScalarEvolution.
   AU.addRequiredID(LoopSimplifyID);
   AU.addRequired<IVUsersWrapperPass>();
   AU.addPreserved<IVUsersWrapperPass>();
   AU.addRequired<TargetTransformInfoWrapperPass>();
 }
 
 static bool ReduceLoopStrength(Loop *L, IVUsers &IU, ScalarEvolution &SE,
                                DominatorTree &DT, LoopInfo &LI,
-                               const TargetTransformInfo &TTI) {
+                               const TargetTransformInfo &TTI,
+                               AssumptionCache &AC,
+                               TargetLibraryInfo &LibInfo) {
+
   bool Changed = false;
 
   // Run the main LSR transformation.
-  Changed |= LSRInstance(L, IU, SE, DT, LI, TTI).getChanged();
+  Changed |= LSRInstance(L, IU, SE, DT, LI, TTI, AC, LibInfo).getChanged();
 
   // Remove any extra phis created by processing inner loops.
   Changed |= DeleteDeadPHIs(L->getHeader());
   if (EnablePhiElim && L->isLoopSimplifyForm()) {
     SmallVector<WeakTrackingVH, 16> DeadInsts;
     const DataLayout &DL = L->getHeader()->getModule()->getDataLayout();
     SCEVExpander Rewriter(SE, DL, "lsr");
 #ifndef NDEBUG
if (skipLoop(L))
     return false;
 
   auto &IU = getAnalysis<IVUsersWrapperPass>().getIU();
   auto &SE = getAnalysis<ScalarEvolutionWrapperPass>().getSE();
   auto &DT = getAnalysis<DominatorTreeWrapperPass>().getDomTree();
   auto &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
   const auto &TTI = getAnalysis<TargetTransformInfoWrapperPass>().getTTI(
       *L->getHeader()->getParent());
-  return ReduceLoopStrength(L, IU, SE, DT, LI, TTI);
+  auto &AC = getAnalysis<AssumptionCacheTracker>().getAssumptionCache(
+      *L->getHeader()->getParent());
+  auto &LibInfo = getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
+  return ReduceLoopStrength(L, IU, SE, DT, LI, TTI, AC, LibInfo);
 }
 
 PreservedAnalyses LoopStrengthReducePass::run(Loop &L, LoopAnalysisManager &AM,
                                               LoopStandardAnalysisResults &AR,
                                               LPMUpdater &) {
   if (!ReduceLoopStrength(&L, AM.getResult<IVUsersAnalysis>(L, AR), AR.SE,
-                          AR.DT, AR.LI, AR.TTI))
+                          AR.DT, AR.LI, AR.TTI, AR.AC, AR.TLI))
     return PreservedAnalyses::all();
 
   return getLoopPassPreservedAnalyses();
 }
 
 char LoopStrengthReduce::ID = 0;
 
 INITIALIZE_PASS_BEGIN(LoopStrengthReduce, "loop-reduce",

llvm/trunk/test/CodeGen/PowerPC/addi-licm.ll

entry:
   %1 = bitcast [2048 x float]* %y to i8*
   call void @llvm.lifetime.start.p0i8(i64 8192, i8* %1) #2
   br label %for.body.i
 
 ; CHECK-LABEL: @foo
 ; CHECK: addi [[REG1:[0-9]+]], 1,
 ; CHECK: addi [[REG2:[0-9]+]], 1,
 ; CHECK: %for.body.i
-; CHECK-DAG: lfsx {{[0-9]+}}, [[REG1]],
-; CHECK-DAG: lfsx {{[0-9]+}}, [[REG2]],
+; CHECK-DAG: lfs {{[0-9]+}}, 0([[REG1]])
+; CHECK-DAG: lfs {{[0-9]+}}, 0([[REG2]])
 ; CHECK: blr
 
 ; PIP-LABEL: @foo
 ; PIP: addi [[REG1:[0-9]+]], 1,
 ; PIP: addi [[REG2:[0-9]+]], 1,
 ; PIP: %for.body.i
 ; PIP-DAG: lfsu {{[0-9]+}}, 4([[REG1]])
 ; PIP-DAG: lfsu {{[0-9]+}}, 4([[REG2]])

llvm/trunk/test/CodeGen/PowerPC/ctrloop-ne.ll

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos2_ir_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos2_ir_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 29554, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos4_ir_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos4_ir_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 15692, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos8_ir_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos8_ir_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 10449, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos16_ir_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos16_ir_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 32087, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos2_ri_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos2_ri_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, 8730
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos4_ri_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos4_ri_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, 1493
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos8_ri_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos8_ri_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, 1706
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos16_ri_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos16_ri_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, 1886
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos2_rr_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos2_rr_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos4_rr_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos4_rr_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos8_rr_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos8_rr_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 
 
 ; CHECK: test_pos16_rr_ne
-; FIXME: Support this loop!
-; CHECK-NOT: bdnz
+; CHECK: bdnz
 ; a < b
 define void @test_pos16_rr_ne(i8* nocapture %p, i32 %a, i32 %b) nounwind {
 entry:
   %cmp3 = icmp slt i32 %a, %b
   br i1 %cmp3, label %for.body.lr.ph, label %for.end
 
 for.body.lr.ph:                                   ; preds = %entry
   br label %for.body

llvm/trunk/test/CodeGen/PowerPC/ctrloop-shortLoops.ll

for.body:                                         ; preds = %entry, %for.body
   %0 = load i32, i32* %arrayidx, align 4
   %add = add nsw i32 %0, %Sum.05
   %indvars.iv.next = add nsw i64 %indvars.iv, -1
   %tobool = icmp eq i64 %indvars.iv, 0
   br i1 %tobool, label %for.cond.cleanup, label %for.body
 }
 
 ; Function Attrs: norecurse nounwind
+; On core a2q, IssueWidth is 1. On core pwr8, IssueWidth is 8.
+; a2q should use mtctr, but pwr8 should not use mtctr.
 define signext i32 @testTripCount2NonSmallLoop() {
 ; CHECK-LABEL: testTripCount2NonSmallLoop:
-; CHECK: blt
-; CHECK: beq
+; CHECK-A2Q: mtctr
+; CHECK-PWR8-NOT: mtctr
 ; CHECK: blr
 
 entry:
   %.pre = load i32, i32* @a, align 4
   br label %for.body
 
 for.body:                                         ; preds = %entry, %if.end
   %0 = phi i32 [ %.pre, %entry ], [ %1, %if.end ]

llvm/trunk/test/CodeGen/PowerPC/lsr-ctrloop.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mcpu=pwr9 -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu < %s | FileCheck %s
 
 ; void foo(float *data, float d) {
 ;   long i;
 ;   for (i = 0; i < 8000; i++)
 ;     data[i] = d;
 ; }
 ; 
 ; This loop will be unrolled by 96 and vectorized on power9.
 ; icmp for loop iteration index and loop trip count(384) has LSRUse for 'reg({0,+,384})'.
 ; Make sure above icmp does not impact LSR choose best formulae sets based on 'reg({(192 + %0),+,384})' 
 
 define void @foo(float* nocapture %data, float %d) {
 ; CHECK-LABEL: foo:
 ; CHECK:  .LBB0_1: # %vector.body
-; CHECK:         add 5, 3, 4
-; CHECK-NEXT:    stxvx 0, 3, 4
+; CHECK:         stxv 0, -192(4)
+; CHECK-NEXT:    stxv 0, -176(4)
+; CHECK-NEXT:    stxv 0, -160(4)
+; CHECK-NEXT:    stxv 0, -144(4)
+; CHECK-NEXT:    stxv 0, -128(4)
+; CHECK-NEXT:    stxv 0, -112(4)
+; CHECK-NEXT:    stxv 0, -96(4)
+; CHECK-NEXT:    stxv 0, -80(4)
+; CHECK-NEXT:    stxv 0, -64(4)
+; CHECK-NEXT:    stxv 0, -48(4)
+; CHECK-NEXT:    stxv 0, -32(4)
+; CHECK-NEXT:    stxv 0, -16(4)
+; CHECK-NEXT:    stxv 0, 0(4)
+; CHECK-NEXT:    stxv 0, 16(4)
+; CHECK-NEXT:    stxv 0, 32(4)
+; CHECK-NEXT:    stxv 0, 48(4)
+; CHECK-NEXT:    stxv 0, 64(4)
+; CHECK-NEXT:    stxv 0, 80(4)
+; CHECK-NEXT:    stxv 0, 96(4)
+; CHECK-NEXT:    stxv 0, 112(4)
+; CHECK-NEXT:    stxv 0, 128(4)
+; CHECK-NEXT:    stxv 0, 144(4)
+; CHECK-NEXT:    stxv 0, 160(4)
+; CHECK-NEXT:    stxv 0, 176(4)
 ; CHECK-NEXT:    addi 4, 4, 384
-; CHECK-NEXT:    stxv 0, 16(5)
-; CHECK-NEXT:    stxv 0, 32(5)
-; CHECK-NEXT:    stxv 0, 48(5)
-; CHECK-NEXT:    stxv 0, 64(5)
-; CHECK-NEXT:    stxv 0, 80(5)
-; CHECK-NEXT:    stxv 0, 96(5)
-; CHECK-NEXT:    stxv 0, 112(5)
-; CHECK-NEXT:    stxv 0, 128(5)
-; CHECK-NEXT:    stxv 0, 144(5)
-; CHECK-NEXT:    stxv 0, 160(5)
-; CHECK-NEXT:    stxv 0, 176(5)
-; CHECK-NEXT:    stxv 0, 192(5)
-; CHECK-NEXT:    stxv 0, 208(5)
-; CHECK-NEXT:    stxv 0, 224(5)
-; CHECK-NEXT:    stxv 0, 240(5)
-; CHECK-NEXT:    stxv 0, 256(5)
-; CHECK-NEXT:    stxv 0, 272(5)
-; CHECK-NEXT:    stxv 0, 288(5)
-; CHECK-NEXT:    stxv 0, 304(5)
-; CHECK-NEXT:    stxv 0, 320(5)
-; CHECK-NEXT:    stxv 0, 336(5)
-; CHECK-NEXT:    stxv 0, 352(5)
-; CHECK-NEXT:    stxv 0, 368(5)
 ; CHECK-NEXT:    bdnz .LBB0_1
 
 entry:
   %broadcast.splatinsert16 = insertelement <4 x float> undef, float %d, i32 0
   %broadcast.splat17 = shufflevector <4 x float> %broadcast.splatinsert16, <4 x float> undef, <4 x i32> zeroinitializer
   %broadcast.splatinsert18 = insertelement <4 x float> undef, float %d, i32 0
   %broadcast.splat19 = shufflevector <4 x float> %broadcast.splatinsert18, <4 x float> undef, <4 x i32> zeroinitializer
   %broadcast.splatinsert20 = insertelement <4 x float> undef, float %d, i32 0

llvm/trunk/test/CodeGen/PowerPC/negctr.ll

entry:
   br i1 undef, label %for.end, label %for.body
 
 for.body:                                         ; preds = %for.body, %entry
   %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 1, %entry ]
   %indvars.iv.next = add i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, 0
   br i1 %exitcond, label %for.end, label %for.body
 
+; FIXME: This should be a hardware loop.
+; cmp is optimized to uadd intrinsic in CGP pass which can not be recognized in
+; later HardwareLoops Pass.
 ; CHECK: @main1
-; CHECK: li [[REG:[0-9]+]], -1
-; CHECK: mtctr [[REG]]
-; CHECK: bdnz
+; CHECK: li [[REG:[0-9]+]], 1
+; CHECK: addi [[REG2:[0-9]+]], [[REG]], 1
+; CHECK: cmpld
+; CHECK: bge
 
 for.end:                                          ; preds = %for.body, %entry
   ret void
 }
 
 define void @main2() #0 {
 entry:
   br i1 undef, label %for.end, label %for.body

llvm/trunk/test/CodeGen/PowerPC/stwu-sched.ll

 ; RUN: llc  -mcpu=pwr9 -mtriple=powerpc64-unknown-linux-gnu < %s -verify-machineinstrs | FileCheck %s
 ; RUN: llc  -mcpu=pwr9 -mtriple=powerpc64le-unknown-linux-gnu < %s -verify-machineinstrs | FileCheck %s
-; RUN: llc  -mcpu=pwr8 -mtriple=powerpc64-unknown-linux-gnu < %s -verify-machineinstrs | FileCheck %s \
-; RUN: --check-prefix=CHECK-ITIN
-; RUN: llc  -mcpu=pwr8 -mtriple=powerpc64le-unknown-linux-gnu < %s -verify-machineinstrs | FileCheck %s \
-; RUN: --check-prefix=CHECK-ITIN
+; RUN: llc  -mcpu=pwr8 -mtriple=powerpc64-unknown-linux-gnu -disable-ppc-ctrloops < %s -verify-machineinstrs \
+; RUN: | FileCheck %s --check-prefix=CHECK-ITIN
+; RUN: llc  -mcpu=pwr8 -mtriple=powerpc64le-unknown-linux-gnu -disable-ppc-ctrloops < %s -verify-machineinstrs \
+; RUN: | FileCheck %s --check-prefix=CHECK-ITIN
 
 
 %0 = type { i32, i32 }
 
 ; Function Attrs: norecurse nounwind writeonly
 define void @initCombList(%0* nocapture, i32 signext) local_unnamed_addr #0 {
 ; CHECK-LABEL: initCombList:
 ; CHECK: addi 4, 4, -8
-; CHECK: stwu 5, 64(3)
+; CHECK: stwu [[REG:[0-9]+]], 64(3)
 
 ; CHECK-ITIN-LABEL: initCombList:
-; CHECK-ITIN: stwu 5, 64(4)
-; CHECK-ITIN-NEXT:   addi 3, 3, -8
+; CHECK-ITIN: stwu [[REG:[0-9]+]], 64(3)
+; CHECK-ITIN-NEXT:   addi [[REG2:[0-9]+]], [[REG2]], 8
 
 
   %3 = zext i32 %1 to i64
   br i1 undef, label %6, label %4
 
 ; <label>:4:                                      ; preds = %2
   store i32 0, i32* undef, align 4, !tbaa !1
   %5 = add nuw nsw i64 0, 1

llvm/trunk/test/CodeGen/PowerPC/unal-altivec.ll

vector.body:                                      ; preds = %vector.body, %vector.ph
   %8 = getelementptr float, float* %a, i64 %.sum12
   %9 = bitcast float* %8 to <4 x float>*
   store <4 x float> %5, <4 x float>* %9, align 4
   %index.next = add i64 %index, 8
   %10 = icmp eq i64 %index.next, 16000
   br i1 %10, label %for.end, label %vector.body
 
 ; CHECK: @foo
-; CHECK-DAG: li [[C0:[0-9]+]], 0
+; CHECK-DAG: li [[C16:[0-9]+]], 16
 ; CHECK-DAG: lvx [[CNST:[0-9]+]],
 ; CHECK: .LBB0_1:
-; CHECK-DAG: lvsl [[MASK1:[0-9]+]], [[B1:[0-9]+]], [[C0]]
-; CHECK-DAG: add [[B3:[0-9]+]], [[B1]], [[C0]]
-; CHECK-DAG: lvx [[LD1:[0-9]+]], [[B1]], [[C0]]
-; CHECK-DAG: lvx [[LD2:[0-9]+]], [[B3]],
-; CHECK-DAG: vperm [[R1:[0-9]+]], [[LD1]], [[LD2]], [[MASK1]]
-; CHECK-DAG: vaddfp {{[0-9]+}}, [[R1]], [[CNST]]
+; CHECK-DAG: lvx [[LD1:[0-9]+]], 0, [[C0:[0-9]+]]
+; CHECK-DAG: lvx [[LD2:[0-9]+]], [[C0]], [[C16]]
+; CHECK-DAG: lvsl [[MASK1:[0-9]+]], 0, [[C0]]
+; CHECK-DAG: vperm [[VR1:[0-9]+]], [[LD1]], [[LD2]], [[MASK1]]
+; CHECK-DAG: vaddfp {{[0-9]+}}, [[VR1]], [[CNST]]
 ; CHECK: blr
 
 for.end:                                          ; preds = %vector.body
   ret void
 }
 
 attributes #0 = { nounwind }