[HardwareLoops] Optimisation remarks

[SjoerdMeijer]

Summary

I reverted this because of a PowerPC buildbot failure. I don't have access to the build/test logs anymore, and I can't reproduce this locally.
Could one of the PowerPC guys perhaps do a check-all with this patch?

Original review/patch here: https://reviews.llvm.org/D68579

[shchenz]

I will have a look

[shchenz]

Many cases fail due to internal crash.

******************** TEST 'LLVM :: ExecutionEngine/OrcMCJIT/non-extern-addend.ll' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli -jit-kind=orc-mcjit /home/czhengsz/llvm_new/llvm-project/llvm/test/ExecutionEngine/OrcMCJIT/non-extern-addend.ll > /dev/null
--
Exit Code: 134

Command Output (stderr):
--
Pass 'Hardware Loop Insertion' is not initialized.
Verify if there is a pass dependency cycle.
Required Passes:
        Natural Loop Information
        Dominator Tree Construction
        Scalar Evolution Analysis
        Assumption Cache Tracker
        Target Transform Information
lli: /home/czhengsz/llvm_new/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:695: void llvm::PMTopLevelManager::schedulePass(llvm::Pass*): Assertion `PI && "Expected required passes to be initialized"' failed.
Stack dump:
0.      Program arguments: /home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli -jit-kind=orc-mcjit /home/czhengsz/llvm_new/llvm-project/llvm/test/ExecutionEngine/OrcMCJIT/non-extern-addend.ll 
 #0 0x0000000010eb6848 llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10eb6848)
 #1 0x0000000010eb6970 PrintStackTraceSignalHandler(void*) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10eb6970)
 #2 0x0000000010eb4404 llvm::sys::RunSignalHandlers() (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10eb4404)
 #3 0x0000000010eb45dc SignalHandler(int) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10eb45dc)
 #4 0x00003fffa7f40478 (linux-vdso64.so.1+0x478)
 #5 0x00003fffa77de100 raise (/opt/at12.0/lib64/power8/libc.so.6+0x4e100)
 #6 0x00003fffa77b4598 abort (/opt/at12.0/lib64/power8/libc.so.6+0x24598)
 #7 0x00003fffa77cb624 (/opt/at12.0/lib64/power8/libc.so.6+0x3b624)
 #8 0x00003fffa77cb6c4 __assert_fail (/opt/at12.0/lib64/power8/libc.so.6+0x3b6c4)
 #9 0x0000000010841da0 llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (.localalias.532) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10841da0)
#10 0x00000000108421b0 llvm::legacy::PassManager::add(llvm::Pass*) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x108421b0)
#11 0x0000000010456500 llvm::TargetPassConfig::addPass(llvm::Pass*, bool, bool) (.localalias.227) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10456500)
#12 0x0000000010fcad6c (anonymous namespace)::PPCPassConfig::addPreISel() (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10fcad6c)
#13 0x000000001045714c llvm::TargetPassConfig::addISelPrepare() (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x1045714c)
#14 0x0000000010458f84 llvm::TargetPassConfig::addISelPasses() (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10458f84)
#15 0x0000000011226f94 addPassesToGenerateCode(llvm::LLVMTargetMachine&, llvm::legacy::PassManagerBase&, bool, llvm::MachineModuleInfoWrapperPass&) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x11226f94)
#16 0x000000001122a460 llvm::LLVMTargetMachine::addPassesToEmitMC(llvm::legacy::PassManagerBase&, llvm::MCContext*&, llvm::raw_pwrite_stream&, bool) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x1122a460)
#17 0x0000000010aeeeb4 llvm::orc::SimpleCompiler::operator()(llvm::Module&) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10aeeeb4)
#18 0x0000000010ad93e4 llvm::Expected<unsigned long> llvm::unique_function<llvm::Expected<unsigned long> ()>::CallImpl<llvm::orc::LazyEmittingLayer<llvm::orc::LegacyIRCompileLayer<llvm::orc::LegacyRTDyldObjectLinkingLayer, llvm::orc::SimpleCompiler> >::EmissionDeferredModule::find(llvm::StringRef, bool, llvm::orc::LegacyIRCompileLayer<llvm::orc::LegacyRTDyldObjectLinkingLayer, llvm::orc::SimpleCompiler>&)::'lambda'()>(void*) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10ad93e4)
#19 0x0000000010ad45dc llvm::JITSymbol::getAddress() (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10ad45dc)
#20 0x0000000010adbfa4 llvm::orc::OrcMCJITReplacement::getPointerToFunction(llvm::Function*) (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x10adbfa4)
#21 0x00000000101ff694 main (/home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli+0x101ff694)
#22 0x00003fffa77b4bf8 (/opt/at12.0/lib64/power8/libc.so.6+0x24bf8)
#23 0x00003fffa77b4e04 __libc_start_main (/opt/at12.0/lib64/power8/libc.so.6+0x24e04)
/home/czhengsz/llvm_new/llvm-project/llvm/build/test/ExecutionEngine/OrcMCJIT/Output/non-extern-addend.ll.script: line 1: 132292 Aborted                 /home/czhengsz/llvm_new/llvm-project/llvm/build/bin/lli -jit-kind=orc-mcjit /home/czhengsz/llvm_new/llvm-project/llvm/test/ExecutionEngine/OrcMCJIT/non-extern-addend.ll > /dev/null

--

[SjoerdMeijer]

Many thanks!!!
Now I've got something to stare at. :-)

[SjoerdMeijer]

This is a pass manager's problem. By comparing the hardwareloop pass with other passes also using the optimisation remarks, I noticed a missing:

INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)

which I have now added. I am fairly confident that this will solve the problem, but as I haven't been able to reproduce locally, I am not entirely certain.
Could I ask for one more favour, would you be able to verify this? If this doesn't solve it, I need to see why I can't reproduce this, but that is a bit of bigger task which I'm obviously trying to avoid... :-/

[shchenz]

Confirmed internal crash fixed with the dependancy decl change.

Now only one test case fail:

: 'RUN: at line 2';   /home/czhengsz/llvm_new/llvm-project/llvm/build/bin/llc -mtriple=thumbv8.1m.main-arm-none-eabi /home/czhengsz/llvm_new/llvm-project/llvm/test/Transforms/HardwareLoops/ARM/structure.ll -o - -pass-remarks-analysis=hardware-loops  2>&1 | /home/czhengsz/llvm_new/llvm-project/llvm/build/bin/FileCheck /home/czhengsz/llvm_new/llvm-project/llvm/test/Transforms/HardwareLoops/ARM/structure.ll --check-prefix=CHECK-LLC

--
Exit Code: 1

Command Output (stderr):
--
/home/czhengsz/llvm_new/llvm-project/llvm/test/Transforms/HardwareLoops/ARM/structure.ll:181:14: error: CHECK-LLC: expected string not found in input
; CHECK-LLC: remark: <unknown>:0:0: hardware-loop not created: loop is not a candidate
             ^
<stdin>:90:10: note: scanning from here
.LBB1_5: @ %while.end7
         ^

Strange thing is that it only fails on PowerPC but ok on ARM, this is a ARM case?

I think you can not mixup checking for remark and assembly both based on CHECK-LLC. All remarks will be put in the front of the output and after checking assembly with CHECK-LLC in func nested, there is no any remark logs after.

[shchenz]

In D69660#1730379, @SjoerdMeijer wrote:

This is a pass manager's problem. By comparing the hardwareloop pass with other passes also using the optimisation remarks, I noticed a missing:

INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)

which I have now added. I am fairly confident that this will solve the problem, but as I haven't been able to reproduce locally, I am not entirely certain.
Could I ask for one more favour, would you be able to verify this? If this doesn't solve it, I need to see why I can't reproduce this, but that is a bit of bigger task which I'm obviously trying to avoid... :-/

I think the reason it crashs inside llc of PowerPC is HardwareLoop Pass becomes the first pass in PowerPC llc pipeline which uses analysis pass OptimizationRemarkEmitterWrapperPass after your change. But you don't register it by adding the dependancy marco.

I guess on ARM HardwareLoop Pass is not the first pass in llc pipeline so OptimizationRemarkEmitterWrapperPass should be registered before. So it is ok on ARM.

[SjoerdMeijer]

Many thanks again!

Yes, I guess it is platform dependent how output to stderr and stdout is 'interleaved', and I now remember having run into this before. So now I have just created a separate CHECK-REMARKS run line for this.

[shchenz]

Verified we get a clean make check-llvm on PowerPC. I guess reason you delete the case test/Transforms/HardwareLoops/unconditional-latch.ll is this case becomes redudant after adding more checking in test/Transforms/HardwareLoops/ARM/structure.ll?
LGTM.

[SjoerdMeijer]

I guess reason you delete the case test/Transforms/HardwareLoops/unconditional-latch.ll is this case becomes redudant after adding more checking in test/Transforms/HardwareLoops/ARM/structure.ll?

Yep, exactly that.

Thanks again for all your help.

llvm/lib/CodeGen/HardwareLoops.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/Pass.h"
 #include "llvm/PassRegistry.h"
 #include "llvm/PassSupport.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/LoopInfo.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ScalarEvolution.h"
 #include "llvm/Analysis/ScalarEvolutionExpander.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/Dominators.h"

 
 static cl::opt<bool>
 ForceGuardLoopEntry(
   "force-hardware-loop-guard", cl::Hidden, cl::init(false),
   cl::desc("Force generation of loop guard intrinsic"));
 
 STATISTIC(NumHWLoops, "Number of loops converted to hardware loops");
 
+#ifndef NDEBUG
+static void debugHWLoopFailure(const StringRef DebugMsg,
+    Instruction *I) {
+  dbgs() << "HWLoops: " << DebugMsg;
+  if (I)
+    dbgs() << ' ' << *I;
+  else
+    dbgs() << '.';
+  dbgs() << '\n';
+}
+#endif
+
+static OptimizationRemarkAnalysis
+createHWLoopAnalysis(StringRef RemarkName, Loop *L, Instruction *I) {
+  Value *CodeRegion = L->getHeader();
+  DebugLoc DL = L->getStartLoc();
+
+  if (I) {
+    CodeRegion = I->getParent();
+    // If there is no debug location attached to the instruction, revert back to
+    // using the loop's.
+    if (I->getDebugLoc())
+      DL = I->getDebugLoc();
+  }
+
+  OptimizationRemarkAnalysis R(DEBUG_TYPE, RemarkName, DL, CodeRegion);
+  R << "hardware-loop not created: ";
+  return R;
+}
+
 namespace {
 
+  void reportHWLoopFailure(const StringRef Msg, const StringRef ORETag,
+      OptimizationRemarkEmitter *ORE, Loop *TheLoop, Instruction *I = nullptr) {
+    LLVM_DEBUG(debugHWLoopFailure(Msg, I));
+    ORE->emit(createHWLoopAnalysis(ORETag, TheLoop, I) << Msg);
+  }
+
   using TTI = TargetTransformInfo;
 
   class HardwareLoops : public FunctionPass {
   public:
     static char ID;
 
     HardwareLoops() : FunctionPass(ID) {
       initializeHardwareLoopsPass(*PassRegistry::getPassRegistry());
     }
 
     bool runOnFunction(Function &F) override;
 
     void getAnalysisUsage(AnalysisUsage &AU) const override {
       AU.addRequired<LoopInfoWrapperPass>();
       AU.addPreserved<LoopInfoWrapperPass>();
       AU.addRequired<DominatorTreeWrapperPass>();
       AU.addPreserved<DominatorTreeWrapperPass>();
       AU.addRequired<ScalarEvolutionWrapperPass>();
       AU.addRequired<AssumptionCacheTracker>();
       AU.addRequired<TargetTransformInfoWrapperPass>();
+      AU.addRequired<OptimizationRemarkEmitterWrapperPass>();
     }
 
     // Try to convert the given Loop into a hardware loop.
     bool TryConvertLoop(Loop *L);
 
     // Given that the target believes the loop to be profitable, try to
     // convert it.
     bool TryConvertLoop(HardwareLoopInfo &HWLoopInfo);
 
   private:
     ScalarEvolution *SE = nullptr;
     LoopInfo *LI = nullptr;
     const DataLayout *DL = nullptr;
+    OptimizationRemarkEmitter *ORE = nullptr;
     const TargetTransformInfo *TTI = nullptr;
     DominatorTree *DT = nullptr;
     bool PreserveLCSSA = false;
     AssumptionCache *AC = nullptr;
     TargetLibraryInfo *LibInfo = nullptr;
     Module *M = nullptr;
     bool MadeChange = false;
   };
class HardwareLoop {
     PHINode *InsertPHICounter(Value *NumElts, Value *EltsRem);
 
     // Create a new cmp, that checks the returned value of loop_decrement*,
     // and update the exit branch to use it.
     void UpdateBranch(Value *EltsRem);
 
   public:
     HardwareLoop(HardwareLoopInfo &Info, ScalarEvolution &SE,
-                 const DataLayout &DL) :
-      SE(SE), DL(DL), L(Info.L), M(L->getHeader()->getModule()),
+                 const DataLayout &DL,
+                 OptimizationRemarkEmitter *ORE) :
+      SE(SE), DL(DL), ORE(ORE), L(Info.L), M(L->getHeader()->getModule()),
       ExitCount(Info.ExitCount),
       CountType(Info.CountType),
       ExitBranch(Info.ExitBranch),
       LoopDecrement(Info.LoopDecrement),
       UsePHICounter(Info.CounterInReg),
       UseLoopGuard(Info.PerformEntryTest) { }
 
     void Create();
 
   private:
     ScalarEvolution &SE;
     const DataLayout &DL;
+    OptimizationRemarkEmitter *ORE = nullptr;
     Loop *L                 = nullptr;
     Module *M               = nullptr;
     const SCEV *ExitCount   = nullptr;
     Type *CountType         = nullptr;
     BranchInst *ExitBranch  = nullptr;
     Value *LoopDecrement    = nullptr;
     bool UsePHICounter      = false;
     bool UseLoopGuard       = false;
bool HardwareLoops::runOnFunction(Function &F) {
 
   LLVM_DEBUG(dbgs() << "HWLoops: Running on " << F.getName() << "\n");
 
   LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
   SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
   DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
   TTI = &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
   DL = &F.getParent()->getDataLayout();
+  ORE = &getAnalysis<OptimizationRemarkEmitterWrapperPass>().getORE();
   auto *TLIP = getAnalysisIfAvailable<TargetLibraryInfoWrapperPass>();
   LibInfo = TLIP ? &TLIP->getTLI(F) : nullptr;
   PreserveLCSSA = mustPreserveAnalysisID(LCSSAID);
   AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
   M = F.getParent();
 
   for (LoopInfo::iterator I = LI->begin(), E = LI->end(); I != E; ++I) {
     Loop *L = *I;
     if (!L->getParentLoop())
       TryConvertLoop(L);
   }
 
   return MadeChange;
 }
 
 // Return true if the search should stop, which will be when an inner loop is
 // converted and the parent loop doesn't support containing a hardware loop.
 bool HardwareLoops::TryConvertLoop(Loop *L) {
   // Process nested loops first.
-  for (Loop::iterator I = L->begin(), E = L->end(); I != E; ++I)
-    if (TryConvertLoop(*I))
+  for (Loop::iterator I = L->begin(), E = L->end(); I != E; ++I) {
+    if (TryConvertLoop(*I)) {
+      reportHWLoopFailure("nested hardware-loops not supported", "HWLoopNested",
+                          ORE, L);
       return true; // Stop search.
+    }
+  }
 
   HardwareLoopInfo HWLoopInfo(L);
-  if (!HWLoopInfo.canAnalyze(*LI))
+  if (!HWLoopInfo.canAnalyze(*LI)) {
+    reportHWLoopFailure("cannot analyze loop, irreducible control flow",
+                        "HWLoopCannotAnalyze", ORE, L);
     return false;
+  }
 
-  if (TTI->isHardwareLoopProfitable(L, *SE, *AC, LibInfo, HWLoopInfo) ||
-      ForceHardwareLoops) {
+  if (!ForceHardwareLoops &&
+      !TTI->isHardwareLoopProfitable(L, *SE, *AC, LibInfo, HWLoopInfo)) {
+    reportHWLoopFailure("it's not profitable to create a hardware-loop",
+                        "HWLoopNotProfitable", ORE, L);
+    return false;
+  }
 
-    // Allow overriding of the counter width and loop decrement value.
-    if (CounterBitWidth.getNumOccurrences())
-      HWLoopInfo.CountType =
-        IntegerType::get(M->getContext(), CounterBitWidth);
 
-    if (LoopDecrement.getNumOccurrences())
-      HWLoopInfo.LoopDecrement =
-        ConstantInt::get(HWLoopInfo.CountType, LoopDecrement);
 
-    MadeChange |= TryConvertLoop(HWLoopInfo);
-    return MadeChange && (!HWLoopInfo.IsNestingLegal && !ForceNestedLoop);
-  }
 
-  return false;
-}
-
 bool HardwareLoops::TryConvertLoop(HardwareLoopInfo &HWLoopInfo) {
 
   Loop *L = HWLoopInfo.L;
   LLVM_DEBUG(dbgs() << "HWLoops: Try to convert profitable loop: " << *L);
 
   if (!HWLoopInfo.isHardwareLoopCandidate(*SE, *LI, *DT, ForceNestedLoop,
-                                          ForceHardwareLoopPHI))
+                                          ForceHardwareLoopPHI)) {
+    // TODO: there can be many reasons a loop is not considered a
+    // candidate, so we should let isHardwareLoopCandidate fill in the
+    // reason and then report a better message here.
+    reportHWLoopFailure("loop is not a candidate", "HWLoopNoCandidate", ORE, L);
     return false;
+  }
 
   assert(
       (HWLoopInfo.ExitBlock && HWLoopInfo.ExitBranch && HWLoopInfo.ExitCount) &&
       "Hardware Loop must have set exit info.");
 
   BasicBlock *Preheader = L->getLoopPreheader();
 
   // If we don't have a preheader, then insert one.
   if (!Preheader)
     Preheader = InsertPreheaderForLoop(L, DT, LI, nullptr, PreserveLCSSA);
   if (!Preheader)
     return false;
 
-  HardwareLoop HWLoop(HWLoopInfo, *SE, *DL);
+  HardwareLoop HWLoop(HWLoopInfo, *SE, *DL, ORE);
   HWLoop.Create();
   ++NumHWLoops;
   return true;
 }
 
 void HardwareLoop::Create() {
   LLVM_DEBUG(dbgs() << "HWLoops: Converting loop..\n");
- 
   Value *LoopCountInit = InitLoopCount();
-  if (!LoopCountInit)
+  if (!LoopCountInit) {
+    reportHWLoopFailure("could not safely create a loop count expression",
+                        "HWLoopNotSafe", ORE, L);
     return;
+  }
 
   InsertIterationSetup(LoopCountInit);
 
   if (UsePHICounter || ForceHardwareLoopPHI) {
     Instruction *LoopDec = InsertLoopRegDec(LoopCountInit);
     Value *EltsRem = InsertPHICounter(LoopCountInit, LoopDec);
     LoopDec->setOperand(0, EltsRem);
     UpdateBranch(LoopDec);
void HardwareLoop::UpdateBranch(Value *EltsRem) {
   // (the original induction variable).
   RecursivelyDeleteTriviallyDeadInstructions(OldCond);
 }
 
 INITIALIZE_PASS_BEGIN(HardwareLoops, DEBUG_TYPE, HW_LOOPS_NAME, false, false)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(OptimizationRemarkEmitterWrapperPass)
 INITIALIZE_PASS_END(HardwareLoops, DEBUG_TYPE, HW_LOOPS_NAME, false, false)
 
 FunctionPass *llvm::createHardwareLoopsPass() { return new HardwareLoops(); }

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:    Rewrite Symbols
 ; CHECK-NEXT:    FunctionPass Manager
 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Exception handling preparation
 ; CHECK-NEXT:      Merge internal globals
 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Scalar Evolution Analysis
+; CHECK-NEXT:      Lazy Branch Probability Analysis
+; CHECK-NEXT:      Lazy Block Frequency Analysis
+; CHECK-NEXT:      Optimization Remark Emitter
 ; CHECK-NEXT:      Hardware Loop Insertion
 ; CHECK-NEXT:      Scalar Evolution Analysis
 ; CHECK-NEXT:      Loop Pass Manager
 ; CHECK-NEXT:        Transform predicated vector loops to use MVE tail predication
 ; CHECK-NEXT:      Safe Stack instrumentation pass
 ; CHECK-NEXT:      Insert stack protectors
 ; CHECK-NEXT:      Module Verifier
 ; CHECK-NEXT:      Dominator Tree Construction

llvm/test/Transforms/HardwareLoops/ARM/structure.ll

 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops %s -S -o - | FileCheck %s
-; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi %s -o - | FileCheck %s --check-prefix=CHECK-LLC
+; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi %s -o - -pass-remarks-analysis=hardware-loops  2>&1 | FileCheck %s --check-prefix=CHECK-LLC
 ; RUN: opt -mtriple=thumbv8.1m.main -loop-unroll -unroll-remainder=false -S < %s | llc -mtriple=thumbv8.1m.main | FileCheck %s --check-prefix=CHECK-UNROLL
 
+; CHECK-LLC: remark: <unknown>:0:0: hardware-loop not created: it's not profitable to create a hardware-loop
 ; CHECK-LABEL: early_exit
 ; CHECK-NOT: llvm.set.loop.iterations
 ; CHECK-NOT: llvm.loop.decrement
 define i32 @early_exit(i32* nocapture readonly %a, i32 %max, i32 %n) {
 entry:
   br label %do.body
 
 do.body:

 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ %N, %while.cond1.preheader.us ], [ [[LOOP_DEC:%[^ ]+]], %while.body3.us ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body3.us, label %while.cond1.while.end_crit_edge.us
 
 ; CHECK-NOT: [[LOOP_DEC1:%[^ ]+]] = call i1 @llvm.loop.decrement.i32(i32 1)
 ; CHECK-NOT: br i1 [[LOOP_DEC1]], label %while.cond1.preheader.us, label %while.end7
 
+; CHECK-LLC: remark: <unknown>:0:0: hardware-loop not created: nested hardware-loops not supported
 ; CHECK-LLC:      nested:
 ; CHECK-LLC-NOT:    mov lr, r1
 ; CHECK-LLC:        dls lr, r1
 ; CHECK-LLC-NOT:    mov lr, r1
 ; CHECK-LLC:      [[LOOP_HEADER:\.LBB[0-9._]+]]:
 ; CHECK-LLC:        le lr, [[LOOP_HEADER]]
 ; CHECK-LLC-NOT:    b [[LOOP_EXIT:\.LBB[0-9._]+]]
 ; CHECK-LLC:      [[LOOP_EXIT:\.LBB[0-9._]+]]:
while.cond1.while.end_crit_edge.us:
   %inc6.us = add nuw i32 %i.021.us, 1
   %exitcond23 = icmp eq i32 %inc6.us, %N
   br i1 %exitcond23, label %while.end7, label %while.cond1.preheader.us
 
 while.end7:
   ret void
 }
 
+
+; CHECK-LLC: remark: <unknown>:0:0: hardware-loop not created: loop is not a candidate
+; CHECK-LLC: remark: <unknown>:0:0: hardware-loop not created: nested hardware-loops not supported
 ; CHECK-LABEL: not_rotated
 ; CHECK-NOT: call void @llvm.set.loop.iterations
 ; CHECK-NOT: call i32 @llvm.loop.decrement.i32
 define void @not_rotated(i32, i16* nocapture, i16 signext) {
   br label %4
 
 4:
   %5 = phi i32 [ 0, %3 ], [ %19, %18 ]

llvm/test/Transforms/HardwareLoops/unconditional-latch.ll

 ; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -hardware-loops -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-ALLOW
 ; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -hardware-loops -force-hardware-loop-guard=true -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-ALLOW
-; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-phi=true -hardware-loops -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-LATCH
+;
+; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 \
+; RUN:     -hardware-loop-counter-bitwidth=32 -force-hardware-loop-phi=true \
+; RUN:     -hardware-loops -S -pass-remarks-analysis=hardware-loops %s -o - \
+; RUN:     2>&1 | FileCheck %s --check-prefixes=CHECK,CHECK-LATCH
+
+; CHECK-LATCH: remark: <unknown>:0:0: hardware-loop not created: loop is not a candidate
 
 ; CHECK-LABEL: not_rotated
 ; CHECK-LATCH-NOT: call void @llvm.set.loop.iterations
 ; CHECK-LATCH-NOT: call i1 @llvm.loop.decrement
 
 ; CHECK-ALLOW: bb:
 ; CHECK-ALLOW:   [[COUNT:%[^ ]+]] = add i32 %arg, 1
 ; CHECK-ALLOW:   br label %bb3