[AlwaysInliner] Enable call site inlining to make flatten attribute working again (#53360)

[xbolva00]

Summary

Problem: Migration to new PM broke flatten attribute.

This is one use case why LLVM should support inlining call-site with alwaysinline. The flatten attribute is nowdays broken, so we should either land patch like this one or remove everything related to flatten attribute from Clang.

Second use case is something like "per call site inlining intrinsics" to control inlining even more; mentioned in
https://lists.llvm.org/pipermail/cfe-dev/2018-September/059232.html

Fixes https://github.com/llvm/llvm-project/issues/53360

[xbolva00]

llvm/lib/Transforms/IPO/AlwaysInliner.cpp

98

avoid DCE of internal functions without always_inline; check test pr2945 in always-inline.ll

[xbolva00]

Aiming this fix for LLVM 14.0.

[aeubanks]

The existing implementation only inlines top level call sites, which doesn't match gcc where all calls are recursively inlined [1]. gcc's implementation makes more sense IMO, only inlining top level call sites doesn't seem super useful.

I'd vote for properly implementing [1] or removing our support for flatten.

[1] https://lists.llvm.org/pipermail/llvm-dev/2019-November/136514.html

[xbolva00]

There is also second use case which I mentioned and I am interested in.

I'd vote for properly implementing [1] or removing our support for flatten.

I would rather prefer to restore atleast something which was working before new PM and then improve it, instead of full user unfriendly removal thanks to oversight during new PM transition.

Imagine also possible use case with those intrinsics:

...
__builtin_flatten(foo());
...

[aeubanks]

sounds good for the purposes of going back to what we had before, but we should reconsider the attribute support

llvm/lib/Transforms/IPO/AlwaysInliner.cpp

98

no need to change in this patch, but it looks like that was a past limitation, removing any dead internal function should be fine (subject to comdats). either we should not delete any function or we should attempt to remove any function we can

llvm/test/Transforms/Inline/always-inline.ll

147

s/CHECK-CALL/CHECK/g should be good enough? any reason for outer7b? it's exactly the same as outer7a

[xbolva00]

Thanks, addressed review comments.

we should reconsider the attribute support

I agree.

[xbolva00]

llvm/lib/Transforms/IPO/AlwaysInliner.cpp

98

Thanks for info, changed to delete them.

llvm/test/Transforms/Inline/always-inline.ll

147

Yes, simplified.

[aeubanks]

llvm/test/Transforms/Coroutines/coro-retcon-once-private.ll

6

CHECK-NOT: define {{.*}} @f is probably a bit more robust

[thakis]

Could this have caused http://45.33.8.238/linux/66347/step_7.txt ?

[xbolva00]

In D117965#3270744, @thakis wrote:

Could this have caused http://45.33.8.238/linux/66347/step_7.txt ?

Yeah, possible as inliner removes internal unused functions (small change before commit, missed check-clang).

I will revert and fix.

clang/test/CodeGen/flatten.c

-// UNSUPPORTED: experimental-new-pass-manager
-// Currently, different code seems to be intentionally generated under the new
-// PM since we alwaysinline functions and not callsites under new PM.
-// Under new PM, f() will not be inlined from g() since f is not marked as
-// alwaysinline.
-
 // RUN: %clang_cc1 -triple=x86_64-linux-gnu %s -emit-llvm -o - | FileCheck %s
 
 void f(void) {}
 
 __attribute__((noinline)) void ni(void) {}
 
 __attribute__((flatten))
 // CHECK: define{{.*}} void @g()

clang/test/CodeGenCXX/flatten.cpp

-// UNSUPPORTED: experimental-new-pass-manager
-// See the comment for CodeGen/flatten.c on why this is unsupported with the new
-// PM.
-
 // RUN: %clang_cc1 -triple=x86_64-linux-gnu -std=c++11 %s -emit-llvm -o - | FileCheck %s
 
 void f(void) {}
 
 [[gnu::flatten]]
 // CHECK: define{{.*}} void @_Z1gv()
 void g(void) {
   // CHECK-NOT: call {{.*}} @_Z1fv
   f();
 }

llvm/lib/Transforms/IPO/AlwaysInliner.cpp

PreservedAnalyses AlwaysInlinerPass::run(Module &M,
   for (Function &F : M) {
     // When callee coroutine function is inlined into caller coroutine function
     // before coro-split pass,
     // coro-early pass can not handle this quiet well.
     // So we won't inline the coroutine function if it have not been unsplited
     if (F.isPresplitCoroutine())
       continue;
 
-    if (!F.isDeclaration() && F.hasFnAttribute(Attribute::AlwaysInline) &&
-        isInlineViable(F).isSuccess()) {
+    if (!F.isDeclaration() && isInlineViable(F).isSuccess()) {
       Calls.clear();
 
       for (User *U : F.users())
         if (auto *CB = dyn_cast<CallBase>(U))
-          if (CB->getCalledFunction() == &F)
+          if (CB->getCalledFunction() == &F &&
+              CB->hasFnAttr(Attribute::AlwaysInline))
             Calls.insert(CB);
 
       for (CallBase *CB : Calls) {
         Function *Caller = CB->getCaller();
         OptimizationRemarkEmitter ORE(Caller);
         auto OIC = shouldInline(
             *CB,
             [&](CallBase &CB) {
if (!F.isDeclaration() && isInlineViable(F).isSuccess()) {
         AttributeFuncs::mergeAttributesForInlining(*Caller, F);
 
         Changed = true;
       }
 
       // Remember to try and delete this function afterward. This both avoids
       // re-walking the rest of the module and avoids dealing with any iterator
       // invalidation issues while deleting functions.
       InlinedFunctions.push_back(&F);
     }
   }
 
   // Remove any live functions.
   erase_if(InlinedFunctions, [&](Function *F) {
     F->removeDeadConstantUsers();
     return !F->isDefTriviallyDead();
   });

llvm/test/CodeGen/X86/x86-cmov-converter.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=x86_64-pc-linux -x86-cmov-converter=true -verify-machineinstrs -disable-block-placement < %s | FileCheck -allow-deprecated-dag-overlap %s
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; This test checks that x86-cmov-converter optimization transform CMOV
 ;; instruction into branches when it is profitable.
 ;; There are 5 cases below:
 ;;   1. CmovInCriticalPath:
 ;;        CMOV depends on the condition and it is in the hot path.

 ;;      t = 10;
 ;;    c[i] = t;
 ;;  }
 ;;}
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
 %struct.Node = type { i32, %struct.Node*, %struct.Node* }
 
-; CHECK-LABEL: CmovInHotPath
-; CHECK-NOT: cmov
-; CHECK: jg
-
 define void @CmovInHotPath(i32 %n, i32 %a, i32 %b, i32* nocapture %c, i32* nocapture readnone %d) #0 {
+; CHECK-LABEL: CmovInHotPath:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    testl %edi, %edi
+; CHECK-NEXT:    jle .LBB0_5
+; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    movl %edi, %r8d
+; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:  .LBB0_2: # %for.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movl (%rcx,%rdi,4), %eax
+; CHECK-NEXT:    leal 1(%rax), %r9d
+; CHECK-NEXT:    imull %esi, %eax
+; CHECK-NEXT:    movl $10, %r10d
+; CHECK-NEXT:    cmpl %edx, %eax
+; CHECK-NEXT:    jg .LBB0_4
+; CHECK-NEXT:  # %bb.3: # %for.body
+; CHECK-NEXT:    # in Loop: Header=BB0_2 Depth=1
+; CHECK-NEXT:    movl %r9d, %r10d
+; CHECK-NEXT:  .LBB0_4: # %for.body
+; CHECK-NEXT:    # in Loop: Header=BB0_2 Depth=1
+; CHECK-NEXT:    imull %r9d, %r10d
+; CHECK-NEXT:    movl %r10d, (%rcx,%rdi,4)
+; CHECK-NEXT:    addq $1, %rdi
+; CHECK-NEXT:    cmpq %rdi, %r8
+; CHECK-NEXT:    jne .LBB0_2
+; CHECK-NEXT:  .LBB0_5: # %for.cond.cleanup
+; CHECK-NEXT:    retq
 entry:
   %cmp14 = icmp sgt i32 %n, 0
   br i1 %cmp14, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %n to i64
   br label %for.body
 
for.body:                                         ; preds = %for.body.preheader, %for.body
   %. = select i1 %cmp3, i32 10, i32 %add
   %mul7 = mul nsw i32 %., %add
   store i32 %mul7, i32* %arrayidx, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-; CHECK-LABEL: CmovNotInHotPath
-; CHECK: cmovg
-
 define void @CmovNotInHotPath(i32 %n, i32 %a, i32 %b, i32* nocapture %c, i32* nocapture %d) #0 {
+; CHECK-LABEL: CmovNotInHotPath:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    testl %edi, %edi
+; CHECK-NEXT:    jle .LBB1_3
+; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    movl %edx, %r9d
+; CHECK-NEXT:    movl %edi, %r10d
+; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:    movl $10, %r11d
+; CHECK-NEXT:  .LBB1_2: # %for.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movl (%rcx,%rdi,4), %eax
+; CHECK-NEXT:    movl %eax, %edx
+; CHECK-NEXT:    imull %esi, %edx
+; CHECK-NEXT:    cmpl %r9d, %edx
+; CHECK-NEXT:    cmovgl %r11d, %eax
+; CHECK-NEXT:    movl %eax, (%rcx,%rdi,4)
+; CHECK-NEXT:    movl (%r8,%rdi,4), %eax
+; CHECK-NEXT:    cltd
+; CHECK-NEXT:    idivl %r9d
+; CHECK-NEXT:    movl %eax, (%r8,%rdi,4)
+; CHECK-NEXT:    addq $1, %rdi
+; CHECK-NEXT:    cmpq %rdi, %r10
+; CHECK-NEXT:    jne .LBB1_2
+; CHECK-NEXT:  .LBB1_3: # %for.cond.cleanup
+; CHECK-NEXT:    retq
 entry:
   %cmp18 = icmp sgt i32 %n, 0
   br i1 %cmp18, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %n to i64
   br label %for.body
 
for.body:                                         ; preds = %for.body.preheader, %for.body
   %1 = load i32, i32* %arrayidx7, align 4
   %div = sdiv i32 %1, %b
   store i32 %div, i32* %arrayidx7, align 4
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-; CHECK-LABEL: MaxIndex
-; CHECK-NOT: cmov
-; CHECK: jg
-
 define i32 @MaxIndex(i32 %n, i32* nocapture readonly %a) #0 {
+; CHECK-LABEL: MaxIndex:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    cmpl $2, %edi
+; CHECK-NEXT:    jl .LBB2_5
+; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    movl %edi, %r8d
+; CHECK-NEXT:    xorl %edi, %edi
+; CHECK-NEXT:    movl $1, %edx
+; CHECK-NEXT:  .LBB2_2: # %for.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movl (%rsi,%rdx,4), %r9d
+; CHECK-NEXT:    movslq %edi, %rcx
+; CHECK-NEXT:    movl %edx, %eax
+; CHECK-NEXT:    cmpl (%rsi,%rcx,4), %r9d
+; CHECK-NEXT:    jg .LBB2_4
+; CHECK-NEXT:  # %bb.3: # %for.body
+; CHECK-NEXT:    # in Loop: Header=BB2_2 Depth=1
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:  .LBB2_4: # %for.body
+; CHECK-NEXT:    # in Loop: Header=BB2_2 Depth=1
+; CHECK-NEXT:    addq $1, %rdx
+; CHECK-NEXT:    movl %eax, %edi
+; CHECK-NEXT:    cmpq %rdx, %r8
+; CHECK-NEXT:    jne .LBB2_2
+; CHECK-NEXT:  .LBB2_5: # %for.cond.cleanup
+; CHECK-NEXT:    retq
 entry:
   %cmp14 = icmp sgt i32 %n, 1
   br i1 %cmp14, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %n to i64
   br label %for.body
 
for.body:                                         ; preds = %for.body.preheader, %for.body
   %cmp3 = icmp sgt i32 %0, %1
   %2 = trunc i64 %indvars.iv to i32
   %i.0.t.0 = select i1 %cmp3, i32 %2, i32 %t.015
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-; CHECK-LABEL: MaxValue
-; CHECK-NOT: jg
-; CHECK: cmovg
+; TODO: If cmov instruction is marked as unpredicatable, do not convert it to branch.
+define i32 @MaxIndex_unpredictable(i32 %n, i32* nocapture readonly %a) #0 {
+; CHECK-LABEL: MaxIndex_unpredictable:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    cmpl $2, %edi
+; CHECK-NEXT:    jl .LBB3_3
+; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    movl %edi, %ecx
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    movl $1, %edx
+; CHECK-NEXT:  .LBB3_2: # %for.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movl (%rsi,%rdx,4), %edi
+; CHECK-NEXT:    cltq
+; CHECK-NEXT:    cmpl (%rsi,%rax,4), %edi
+; CHECK-NEXT:    cmovgl %edx, %eax
+; CHECK-NEXT:    addq $1, %rdx
+; CHECK-NEXT:    cmpq %rdx, %rcx
+; CHECK-NEXT:    jne .LBB3_2
+; CHECK-NEXT:  .LBB3_3: # %for.cond.cleanup
+; CHECK-NEXT:    # kill: def $eax killed $eax killed $rax
+; CHECK-NEXT:    retq
+entry:
+  %cmp14 = icmp sgt i32 %n, 1
+  br i1 %cmp14, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:                               ; preds = %entry
+  %wide.trip.count = zext i32 %n to i64
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  %t.0.lcssa = phi i32 [ 0, %entry ], [ %i.0.t.0, %for.body ]
+  ret i32 %t.0.lcssa
+
+for.body:                                         ; preds = %for.body.preheader, %for.body
+  %indvars.iv = phi i64 [ %indvars.iv.next, %for.body ], [ 1, %for.body.preheader ]
+  %t.015 = phi i32 [ %i.0.t.0, %for.body ], [ 0, %for.body.preheader ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i64 %indvars.iv
+  %0 = load i32, i32* %arrayidx, align 4
+  %idxprom1 = sext i32 %t.015 to i64
+  %arrayidx2 = getelementptr inbounds i32, i32* %a, i64 %idxprom1
+  %1 = load i32, i32* %arrayidx2, align 4
+  %cmp3 = icmp sgt i32 %0, %1
+  %2 = trunc i64 %indvars.iv to i32
+  %i.0.t.0 = select i1 %cmp3, i32 %2, i32 %t.015, !unpredictable !0
+  %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
+  %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
 
 define i32 @MaxValue(i32 %n, i32* nocapture readonly %a) #0 {
+; CHECK-LABEL: MaxValue:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl (%rsi), %eax
+; CHECK-NEXT:    cmpl $2, %edi
+; CHECK-NEXT:    jl .LBB4_3
+; CHECK-NEXT:  # %bb.1: # %for.body.preheader
+; CHECK-NEXT:    movl %edi, %ecx
+; CHECK-NEXT:    movl $1, %edx
+; CHECK-NEXT:  .LBB4_2: # %for.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movl (%rsi,%rdx,4), %edi
+; CHECK-NEXT:    cmpl %eax, %edi
+; CHECK-NEXT:    cmovgl %edi, %eax
+; CHECK-NEXT:    addq $1, %rdx
+; CHECK-NEXT:    cmpq %rdx, %rcx
+; CHECK-NEXT:    jne .LBB4_2
+; CHECK-NEXT:  .LBB4_3: # %for.cond.cleanup
+; CHECK-NEXT:    retq
 entry:
   %0 = load i32, i32* %a, align 4
   %cmp13 = icmp sgt i32 %n, 1
   br i1 %cmp13, label %for.body.preheader, label %for.cond.cleanup
 
 for.body.preheader:                               ; preds = %entry
   %wide.trip.count = zext i32 %n to i64
   br label %for.body
for.body:                                         ; preds = %for.body.preheader, %for.body
   %1 = load i32, i32* %arrayidx1, align 4
   %cmp2 = icmp sgt i32 %1, %t.014
   %.t.0 = select i1 %cmp2, i32 %1, i32 %t.014
   %indvars.iv.next = add nuw nsw i64 %indvars.iv, 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-; CHECK-LABEL: BinarySearch
-; CHECK: set
-
 define i32 @BinarySearch(i32 %Mask, %struct.Node* nocapture readonly %Curr, %struct.Node* nocapture readonly %Next) #0 {
+; CHECK-LABEL: BinarySearch:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl (%rsi), %eax
+; CHECK-NEXT:    jmp .LBB5_2
+; CHECK-NEXT:  .LBB5_1: # %while.body
+; CHECK-NEXT:    # in Loop: Header=BB5_2 Depth=1
+; CHECK-NEXT:    movl %ecx, %eax
+; CHECK-NEXT:    xorl %ecx, %ecx
+; CHECK-NEXT:    btl %eax, %edi
+; CHECK-NEXT:    setae %cl
+; CHECK-NEXT:    movq 8(%rdx,%rcx,8), %rdx
+; CHECK-NEXT:  .LBB5_2: # %while.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movl (%rdx), %ecx
+; CHECK-NEXT:    cmpl %ecx, %eax
+; CHECK-NEXT:    ja .LBB5_1
+; CHECK-NEXT:  # %bb.3: # %while.end
+; CHECK-NEXT:    retq
 entry:
   %Val8 = getelementptr inbounds %struct.Node, %struct.Node* %Curr, i64 0, i32 0
   %0 = load i32, i32* %Val8, align 8
   %Val19 = getelementptr inbounds %struct.Node, %struct.Node* %Next, i64 0, i32 0
   %1 = load i32, i32* %Val19, align 8
   %cmp10 = icmp ugt i32 %0, %1
   br i1 %cmp10, label %while.body, label %while.end
 

 ;; SinkMBB:
 ;;   %v1 = phi[%f1, %FalseMBB], [%t1, %MBB]
 ;;   %v2 = phi[%f1, %FalseMBB], [%f2, %MBB] ; For CMOV with OppCC switch
 ;;                                          ; true-value with false-value
 ;;                                          ; Phi instruction cannot use
 ;;                                          ; previous Phi instruction result
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
-; CHECK-LABEL: Transform
-; CHECK-NOT: cmov
-; CHECK:         divl    [[a:%[0-9a-z]*]]
-; CHECK:         movl    $11, [[s1:%[0-9a-z]*]]
-; CHECK:         movl    [[a]], [[s2:%[0-9a-z]*]]
-; CHECK:         cmpl    [[a]], %edx
-; CHECK:         ja      [[SinkBB:.*]]
-; CHECK: [[FalseBB:.*]]:
-; CHECK:         movl    $22, [[s1]]
-; CHECK:         movl    $22, [[s2]]
-; CHECK: [[SinkBB]]:
-; CHECK:         ja
-
 define void @Transform(i32 *%arr, i32 *%arr2, i32 %a, i32 %b, i32 %c, i32 %n) #0 {
+; CHECK-LABEL: Transform:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movb $1, %al
+; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    jne .LBB6_5
+; CHECK-NEXT:  # %bb.1: # %while.body.preheader
+; CHECK-NEXT:    movl %edx, %r8d
+; CHECK-NEXT:    xorl %esi, %esi
+; CHECK-NEXT:  .LBB6_2: # %while.body
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movslq %esi, %rsi
+; CHECK-NEXT:    movl (%rdi,%rsi,4), %eax
+; CHECK-NEXT:    xorl %edx, %edx
+; CHECK-NEXT:    divl %r8d
+; CHECK-NEXT:    movl %eax, %edx
+; CHECK-NEXT:    movl $11, %eax
+; CHECK-NEXT:    movl %r8d, %ecx
+; CHECK-NEXT:    cmpl %r8d, %edx
+; CHECK-NEXT:    ja .LBB6_4
+; CHECK-NEXT:  # %bb.3: # %while.body
+; CHECK-NEXT:    # in Loop: Header=BB6_2 Depth=1
+; CHECK-NEXT:    movl $22, %eax
+; CHECK-NEXT:    movl $22, %ecx
+; CHECK-NEXT:  .LBB6_4: # %while.body
+; CHECK-NEXT:    # in Loop: Header=BB6_2 Depth=1
+; CHECK-NEXT:    xorl %edx, %edx
+; CHECK-NEXT:    divl %ecx
+; CHECK-NEXT:    movl %edx, (%rdi,%rsi,4)
+; CHECK-NEXT:    addl $1, %esi
+; CHECK-NEXT:    cmpl %r9d, %esi
+; CHECK-NEXT:    ja .LBB6_2
+; CHECK-NEXT:  .LBB6_5: # %while.end
+; CHECK-NEXT:    retq
 entry:
   %cmp10 = icmp ugt i32 0, %n
   br i1 %cmp10, label %while.body, label %while.end
 
 while.body:                                       ; preds = %entry, %while.body
   %i = phi i32 [ %i_inc, %while.body ], [ 0, %entry ]
   %arr_i = getelementptr inbounds i32, i32* %arr, i32 %i
   %x = load i32, i32* %arr_i, align 4

 while.end:                                        ; preds = %while.body, %entry
   ret void
 }
 
 ; Test that we always will convert a cmov with a memory operand into a branch,
 ; even outside of a loop.
 define i32 @test_cmov_memoperand(i32 %a, i32 %b, i32 %x, i32* %y) #0 {
 ; CHECK-LABEL: test_cmov_memoperand:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edx, %eax
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    ja .LBB7_2
+; CHECK-NEXT:  # %bb.1: # %entry
+; CHECK-NEXT:    movl (%rcx), %eax
+; CHECK-NEXT:  .LBB7_2: # %entry
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         movl %edx, %eax
-; CHECK:         cmpl
   %load = load i32, i32* %y
   %z = select i1 %cond, i32 %x, i32 %load
-; CHECK-NOT:     cmov
-; CHECK:         ja [[FALSE_BB:.*]]
-; CHECK:         movl (%rcx), %eax
-; CHECK:       [[FALSE_BB]]:
+  ret i32 %z
+}
+
+; TODO: If cmov instruction is marked as unpredicatable, do not convert it to branch.
+define i32 @test_cmov_memoperand_unpredictable(i32 %a, i32 %b, i32 %x, i32* %y) #0 {
+; CHECK-LABEL: test_cmov_memoperand_unpredictable:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edx, %eax
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    cmovbel (%rcx), %eax
+; CHECK-NEXT:    retq
+entry:
+  %cond = icmp ugt i32 %a, %b
+  %load = load i32, i32* %y
+  %z = select i1 %cond, i32 %x, i32 %load, !unpredictable !0
   ret i32 %z
 }
 
 ; Test that we can convert a group of cmovs where only one has a memory
 ; operand.
 define i32 @test_cmov_memoperand_in_group(i32 %a, i32 %b, i32 %x, i32* %y.ptr) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_in_group:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edx, %eax
+; CHECK-NEXT:    movl %edx, %r8d
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    ja .LBB9_2
+; CHECK-NEXT:  # %bb.1: # %entry
+; CHECK-NEXT:    movl (%rcx), %r8d
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    movl %esi, %edx
+; CHECK-NEXT:  .LBB9_2: # %entry
+; CHECK-NEXT:    addl %r8d, %eax
+; CHECK-NEXT:    addl %edx, %eax
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         movl %edx, %eax
-; CHECK:         cmpl
   %y = load i32, i32* %y.ptr
   %z1 = select i1 %cond, i32 %x, i32 %a
   %z2 = select i1 %cond, i32 %x, i32 %y
   %z3 = select i1 %cond, i32 %x, i32 %b
-; CHECK-NOT:     cmov
-; CHECK:         ja [[FALSE_BB:.*]]
-; CHECK-DAG:     movl %{{.*}}, %[[R1:.*]]
-; CHECK-DAG:     movl (%r{{..}}), %[[R2:.*]]
-; CHECK-DAG:     movl %{{.*}} %eax
-; CHECK:       [[FALSE_BB]]:
-; CHECK:         addl
-; CHECK-DAG:       %[[R1]]
-; CHECK-DAG:       ,
-; CHECK-DAG:       %eax
-; CHECK-DAG:     addl
-; CHECK-DAG:       %[[R2]]
-; CHECK-DAG:       ,
-; CHECK-DAG:       %eax
-; CHECK:         retq
   %s1 = add i32 %z1, %z2
   %s2 = add i32 %s1, %z3
   ret i32 %s2
 }
 
 ; Same as before but with operands reversed in the select with a load.
 define i32 @test_cmov_memoperand_in_group2(i32 %a, i32 %b, i32 %x, i32* %y.ptr) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_in_group2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edx, %eax
+; CHECK-NEXT:    movl %edx, %r8d
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    jbe .LBB10_2
+; CHECK-NEXT:  # %bb.1: # %entry
+; CHECK-NEXT:    movl (%rcx), %r8d
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    movl %esi, %edx
+; CHECK-NEXT:  .LBB10_2: # %entry
+; CHECK-NEXT:    addl %r8d, %eax
+; CHECK-NEXT:    addl %edx, %eax
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         movl %edx, %eax
-; CHECK:         cmpl
   %y = load i32, i32* %y.ptr
   %z2 = select i1 %cond, i32 %a, i32 %x
   %z1 = select i1 %cond, i32 %y, i32 %x
   %z3 = select i1 %cond, i32 %b, i32 %x
-; CHECK-NOT:     cmov
-; CHECK:         jbe [[FALSE_BB:.*]]
-; CHECK-DAG:     movl %{{.*}}, %[[R1:.*]]
-; CHECK-DAG:     movl (%r{{..}}), %[[R2:.*]]
-; CHECK-DAG:     movl %{{.*}} %eax
-; CHECK:       [[FALSE_BB]]:
-; CHECK:         addl
-; CHECK-DAG:       %[[R1]]
-; CHECK-DAG:       ,
-; CHECK-DAG:       %eax
-; CHECK-DAG:     addl
-; CHECK-DAG:       %[[R2]]
-; CHECK-DAG:       ,
-; CHECK-DAG:       %eax
-; CHECK:         retq
   %s1 = add i32 %z1, %z2
   %s2 = add i32 %s1, %z3
   ret i32 %s2
 }
 
 ; Test that we don't convert a group of cmovs with conflicting directions of
 ; loads.
 define i32 @test_cmov_memoperand_conflicting_dir(i32 %a, i32 %b, i32 %x, i32* %y1.ptr, i32* %y2.ptr) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_conflicting_dir:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    movl (%rcx), %eax
+; CHECK-NEXT:    cmoval %edx, %eax
+; CHECK-NEXT:    cmoval (%r8), %edx
+; CHECK-NEXT:    addl %edx, %eax
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         cmpl
   %y1 = load i32, i32* %y1.ptr
   %y2 = load i32, i32* %y2.ptr
   %z1 = select i1 %cond, i32 %x, i32 %y1
   %z2 = select i1 %cond, i32 %y2, i32 %x
-; CHECK:         cmoval
-; CHECK:         cmoval
   %s1 = add i32 %z1, %z2
   ret i32 %s1
 }
 
 ; Test that we can convert a group of cmovs where only one has a memory
 ; operand and where that memory operand's registers come from a prior cmov in
 ; the group.
 define i32 @test_cmov_memoperand_in_group_reuse_for_addr(i32 %a, i32 %b, i32* %x, i32* %y) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_in_group_reuse_for_addr:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    ja .LBB12_2
+; CHECK-NEXT:  # %bb.1: # %entry
+; CHECK-NEXT:    movl (%rcx), %eax
+; CHECK-NEXT:  .LBB12_2: # %entry
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         movl %edi, %eax
-; CHECK:         cmpl
   %p = select i1 %cond, i32* %x, i32* %y
   %load = load i32, i32* %p
   %z = select i1 %cond, i32 %a, i32 %load
-; CHECK-NOT:     cmov
-; CHECK:         ja [[FALSE_BB:.*]]
-; CHECK:         movl (%r{{..}}), %eax
-; CHECK:       [[FALSE_BB]]:
-; CHECK:         retq
   ret i32 %z
 }
 
 ; Test that we can convert a group of two cmovs with memory operands where one
 ; uses the result of the other as part of the address.
 define i32 @test_cmov_memoperand_in_group_reuse_for_addr2(i32 %a, i32 %b, i32* %x, i32** %y) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_in_group_reuse_for_addr2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    ja .LBB13_2
+; CHECK-NEXT:  # %bb.1: # %entry
+; CHECK-NEXT:    movq (%rcx), %rax
+; CHECK-NEXT:    movl (%rax), %eax
+; CHECK-NEXT:  .LBB13_2: # %entry
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         movl %edi, %eax
-; CHECK:         cmpl
   %load1 = load i32*, i32** %y
   %p = select i1 %cond, i32* %x, i32* %load1
   %load2 = load i32, i32* %p
   %z = select i1 %cond, i32 %a, i32 %load2
-; CHECK-NOT:     cmov
-; CHECK:         ja [[FALSE_BB:.*]]
-; CHECK:         movq (%r{{..}}), %[[R1:.*]]
-; CHECK:         movl (%[[R1]]), %eax
-; CHECK:       [[FALSE_BB]]:
-; CHECK:         retq
   ret i32 %z
 }
 
 ; Test that we can convert a group of cmovs where only one has a memory
 ; operand and where that memory operand's registers come from a prior cmov and
 ; where that cmov gets *its* input from a prior cmov in the group.
 define i32 @test_cmov_memoperand_in_group_reuse_for_addr3(i32 %a, i32 %b, i32* %x, i32* %y, i32* %z) #0 {
 ; CHECK-LABEL: test_cmov_memoperand_in_group_reuse_for_addr3:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    cmpl %esi, %edi
+; CHECK-NEXT:    ja .LBB14_2
+; CHECK-NEXT:  # %bb.1: # %entry
+; CHECK-NEXT:    movl (%rcx), %eax
+; CHECK-NEXT:  .LBB14_2: # %entry
+; CHECK-NEXT:    retq
 entry:
   %cond = icmp ugt i32 %a, %b
-; CHECK:         movl %edi, %eax
-; CHECK:         cmpl
   %p = select i1 %cond, i32* %x, i32* %y
   %p2 = select i1 %cond, i32* %z, i32* %p
   %load = load i32, i32* %p2
   %r = select i1 %cond, i32 %a, i32 %load
-; CHECK-NOT:     cmov
-; CHECK:         ja [[FALSE_BB:.*]]
-; CHECK:         movl (%r{{..}}), %eax
-; CHECK:       [[FALSE_BB]]:
-; CHECK:         retq
   ret i32 %r
 }
 
 @begin = external global i32*
 @end = external global i32*
 
 define void @test_memoperand_loop(i32 %data) #0 {
 ; CHECK-LABEL: test_memoperand_loop:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq begin@GOTPCREL(%rip), %r8
 ; CHECK-NEXT:    movq (%r8), %rax
 ; CHECK-NEXT:    movq end@GOTPCREL(%rip), %rcx
 ; CHECK-NEXT:    movq (%rcx), %rdx
 ; CHECK-NEXT:    xorl %esi, %esi
 ; CHECK-NEXT:    movq %rax, %rcx
-entry:
-  %begin = load i32*, i32** @begin, align 8
-  %end = load i32*, i32** @end, align 8
-  br label %loop.body
-
-; CHECK-NEXT:  .LBB13_1: # %loop.body
+; CHECK-NEXT:  .LBB15_1: # %loop.body
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    addq $8, %rcx
 ; CHECK-NEXT:    cmpq %rdx, %rcx
-; CHECK-NEXT:    ja .LBB13_3
+; CHECK-NEXT:    ja .LBB15_3
 ; CHECK-NEXT:  # %bb.2: # %loop.body
-; CHECK-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; CHECK-NEXT:    # in Loop: Header=BB15_1 Depth=1
 ; CHECK-NEXT:    movq (%r8), %rcx
-; CHECK-NEXT:  .LBB13_3: # %loop.body
-; CHECK-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; CHECK-NEXT:  .LBB15_3: # %loop.body
+; CHECK-NEXT:    # in Loop: Header=BB15_1 Depth=1
 ; CHECK-NEXT:    movl %edi, (%rcx)
 ; CHECK-NEXT:    addq $8, %rcx
 ; CHECK-NEXT:    cmpq %rdx, %rcx
-; CHECK-NEXT:    ja .LBB13_5
+; CHECK-NEXT:    ja .LBB15_5
 ; CHECK-NEXT:  # %bb.4: # %loop.body
-; CHECK-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; CHECK-NEXT:    # in Loop: Header=BB15_1 Depth=1
 ; CHECK-NEXT:    movq %rax, %rcx
-; CHECK-NEXT:  .LBB13_5: # %loop.body
-; CHECK-NEXT:    # in Loop: Header=BB13_1 Depth=1
+; CHECK-NEXT:  .LBB15_5: # %loop.body
+; CHECK-NEXT:    # in Loop: Header=BB15_1 Depth=1
 ; CHECK-NEXT:    movl %edi, (%rcx)
 ; CHECK-NEXT:    addl $1, %esi
 ; CHECK-NEXT:    cmpl $1024, %esi # imm = 0x400
-; CHECK-NEXT:    jl .LBB13_1
+; CHECK-NEXT:    jl .LBB15_1
+; CHECK-NEXT:  # %bb.6: # %exit
+; CHECK-NEXT:    retq
+entry:
+  %begin = load i32*, i32** @begin, align 8
+  %end = load i32*, i32** @end, align 8
+  br label %loop.body
 loop.body:
   %phi.iv = phi i32 [ 0, %entry ], [ %iv.next, %loop.body ]
   %phi.ptr = phi i32* [ %begin, %entry ], [ %dst2, %loop.body ]
   %gep1 = getelementptr inbounds i32, i32 *%phi.ptr, i64 2
   %cmp1 = icmp ugt i32* %gep1, %end
   %begin_dup = load i32*, i32** @begin, align 8
   %dst1 = select i1 %cmp1, i32* %gep1, i32* %begin_dup
   store i32 %data, i32 *%dst1, align 4
   %gep2 = getelementptr inbounds i32, i32 *%dst1, i64 2
   %cmp2 = icmp ugt i32* %gep2, %end
   %dst2 = select i1 %cmp2, i32* %gep2, i32* %begin
   store i32 %data, i32 *%dst2, align 4
   %iv.next = add i32 %phi.iv, 1
   %cond = icmp slt i32 %iv.next, 1024
   br i1 %cond, label %loop.body, label %exit
-
-; CHECK-NEXT:  # %bb.6: # %exit
-; CHECK-NEXT:    retq
 exit:
   ret void
 }
 
 attributes #0 = {"target-cpu"="x86-64"}
+!0 = !{}

llvm/test/Transforms/Coroutines/coro-retcon-once-private.ll

 ; RUN: opt < %s -enable-coroutines -passes='default<O0>' -S | FileCheck %s
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.12.0"
 
-; CHECK: define internal { i8*, i32 } @f(i8* %buffer, i32* %array)
-; CHECK-NEXT: entry:
-; CHECK-NEXT:  unreachable
+; CHECK-NOT: define internal { i8*, i32 } @f(i8* %buffer, i32* %array)
 
 define internal {i8*, i32} @f(i8* %buffer, i32* %array) {
 entry:
   %id = call token @llvm.coro.id.retcon.once(i32 8, i32 8, i8* %buffer, i8* bitcast (void (i8*, i1)* @prototype to i8*), i8* bitcast (i8* (i32)* @allocate to i8*), i8* bitcast (void (i8*)* @deallocate to i8*))
   %hdl = call i8* @llvm.coro.begin(token %id, i8* null)
   %load = load i32, i32* %array
   %load.pos = icmp sgt i32 %load, 0
   br i1 %load.pos, label %pos, label %neg

llvm/test/Transforms/Inline/always-inline.ll

-; RUN: opt < %s -inline-threshold=0 -always-inline -enable-new-pm=0 -S | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-CALL
+; RUN: opt < %s -inline-threshold=0 -always-inline -enable-new-pm=0 -S | FileCheck %s --check-prefix=CHECK
 ;
 ; Ensure the threshold has no impact on these decisions.
-; RUN: opt < %s -inline-threshold=20000000 -always-inline -enable-new-pm=0 -S | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-CALL
-; RUN: opt < %s -inline-threshold=-20000000 -always-inline -enable-new-pm=0 -S | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-CALL
+; RUN: opt < %s -inline-threshold=20000000 -always-inline -enable-new-pm=0 -S | FileCheck %s --check-prefix=CHECK
+; RUN: opt < %s -inline-threshold=-20000000 -always-inline -enable-new-pm=0 -S | FileCheck %s --check-prefix=CHECK
 ;
 ; The new pass manager doesn't re-use any threshold based infrastructure for
-; the always inliner, but test that we get the correct result. The new PM
-; always inliner also doesn't support inlining call-site alwaysinline
-; annotations. It isn't clear that this is a reasonable use case for
-; 'alwaysinline'.
+; the always inliner, but test that we get the correct result.
 ; RUN: opt < %s -inline-threshold=0 -passes=always-inline -S | FileCheck %s --check-prefix=CHECK
 ; RUN: opt < %s -inline-threshold=20000000 -passes=always-inline -S | FileCheck %s --check-prefix=CHECK
 ; RUN: opt < %s -inline-threshold=-20000000 -passes=always-inline -S | FileCheck %s --check-prefix=CHECK
 
 define internal i32 @inner1() alwaysinline {
 ; CHECK-NOT: @inner1(
   ret i32 1
 }
 define i32 @outer1() {
 ; CHECK-LABEL: @outer1(
 ; CHECK-NOT: call
 ; CHECK: ret
 
    %r = call i32 @inner1()
    ret i32 %r
 }
 
-; The always inliner can't DCE arbitrary internal functions. PR2945
-define internal i32 @pr2945() nounwind {
-; CHECK-LABEL: @pr2945(
-  ret i32 0
-}
-
 define internal void @inner2(i32 %N) alwaysinline {
 ; CHECK-NOT: @inner2(
   %P = alloca i32, i32 %N
   ret void
 }
 define void @outer2(i32 %N) {
 ; The always inliner (unlike the normal one) should be willing to inline
 ; a function with a dynamic alloca into one without a dynamic alloca.

 }
 
 ; This is not an alwaysinline function and is actually external.
 define i32 @inner7() {
 ; CHECK-LABEL: @inner7(
   ret i32 1
 }
 define i32 @outer7() {
-; CHECK-CALL-LABEL: @outer7(
-; CHECK-CALL-NOT: call
-; CHECK-CALL: ret
-
+; CHECK-LABEL: @outer7(
+; CHECK-NOT: call
+; CHECK: ret
    %r = call i32 @inner7() alwaysinline
    ret i32 %r
 }
 
 define internal float* @inner8(float* nocapture align 128 %a) alwaysinline {
 ; CHECK-NOT: @inner8(
   ret float* %a
 }
 define float @outer8(float* nocapture %a) {
 ; CHECK-LABEL: @outer8(
 ; CHECK-NOT: call float* @inner8
 ; CHECK: ret