[SelectionDAG][RISCV] Add preferred extend of value used for PHI node

[Luhaocong]

Summary

If a value is used for PHI node，and the PHI node is only used for return instruction which requires
signext or zeroext, we set default preferred extend kind to SIGN_EXTEND or ZERO_EXTEND.

With this optimization, we could perform bit extend in necessary BB，instead of return BB .
At the same time, increase more machine CSE opportunities.

[craig.topper]

llvm/test/CodeGen/RISCV/prefer-extend.ll

3

What does the S mean after RV32/RV64?

6

Remove the dso_local

[jrtc27]

The test case is quite complicated for what should be a simple thing to test. and is rather redundant in its IR; ignoring the nsw aspect, sext+sub+trunc is just sub of the smaller type, and the sext+icmp can just be icmp of the smaller type. Changing the latter seems to have no effect on the generated code, but for whatever reason changing the former to just sub on i16 already sign-extends at the computation site. Also, your ; preds comments don't belong in IR tests, and the naming looks like you just took Clang output and hacked it into a test.

define signext i16 @foo(i16 signext %a, i1 zeroext %b) {
entry:
  br i1 %b, label %add, label %ret

add:
  %0 = add i16 %a, 1
  br label %ret

ret:
  %1 = phi i16 [ %a, %entry ], [ %0, %add ]
  ret i16 %1
}

is a much simpler from-scratch test that should show the behaviour you want.

llvm/test/CodeGen/RISCV/prefer-extend.ll

3

I assume it's for signext, but that has no place here

[craig.topper]

With this optimization, we could perform bit extend in necessary BB，instead of all predecessors of
return BB. At the same time, increase more machine CSE opportunities.

Why would the extend be in all predecessors? Wouldn't it just be in the block with the return?

Wouldn't this patch increase code size if none of the phi inputs were known to be sign extended? Your test just moves an extend to the block that produced %conv2 because %left and %right are already extended. But if they weren't wouldn't it increase the number of extends?

[Luhaocong]

1. fix failed test case.
2. To avoid increasing the loop body, we exclude values within the loop，

[Luhaocong]

In D116398#3214097, @jrtc27 wrote:

The test case is quite complicated for what should be a simple thing to test. and is rather redundant in its IR; ignoring the nsw aspect, sext+sub+trunc is just sub of the smaller type, and the sext+icmp can just be icmp of the smaller type. Changing the latter seems to have no effect on the generated code, but for whatever reason changing the former to just sub on i16 already sign-extends at the computation site. Also, your ; preds comments don't belong in IR tests, and the naming looks like you just took Clang output and hacked it into a test.

define signext i16 @foo(i16 signext %a, i1 zeroext %b) {
entry:
  br i1 %b, label %add, label %ret

add:
  %0 = add i16 %a, 1
  br label %ret

ret:
  %1 = phi i16 [ %a, %entry ], [ %0, %add ]
  ret i16 %1
}

is a much simpler from-scratch test that should show the behaviour you want.

Thanks for your professional guidance，I will focus more on simplifying test cases

[Luhaocong]

In D116398#3214114, @craig.topper wrote:

With this optimization, we could perform bit extend in necessary BB，instead of all predecessors of
return BB. At the same time, increase more machine CSE opportunities.

Why would the extend be in all predecessors? Wouldn't it just be in the block with the return?

Wouldn't this patch increase code size if none of the phi inputs were known to be sign extended? Your test just moves an extend to the block that produced %conv2 because %left and %right are already extended. But if they weren't wouldn't it increase the number of extends?

It‘s my mistake, what I want to express is that not all paths to return BB should do the extend.
Sometimes this does increase code size, but I think it is valuable if you can get optimization opportunities.

llvm/test/CodeGen/RISCV/prefer-extend.ll

3

I've deleted it

6

Done

llvm/include/llvm/CodeGen/FunctionLoweringInfo.h

 #ifndef LLVM_CODEGEN_FUNCTIONLOWERINGINFO_H
 #define LLVM_CODEGEN_FUNCTIONLOWERINGINFO_H
 
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/IndexedMap.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/Analysis/LoopInfo.h"
 #include "llvm/CodeGen/ISDOpcodes.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Support/KnownBits.h"
 #include <cassert>
public:
   /// If the current MBB is a landing pad, the exception pointer and exception
   /// selector registers are copied into these virtual registers by
   /// SelectionDAGISel::PrepareEHLandingPad().
   unsigned ExceptionPointerVirtReg, ExceptionSelectorVirtReg;
 
   /// set - Initialize this FunctionLoweringInfo with the given Function
   /// and its associated MachineFunction.
   ///
-  void set(const Function &Fn, MachineFunction &MF, SelectionDAG *DAG);
+  void set(const Function &Fn, MachineFunction &MF, SelectionDAG *DAG,
+           const LoopInfo *LI);
 
   /// clear - Clear out all the function-specific state. This returns this
   /// FunctionLoweringInfo to an empty state, ready to be used for a
   /// different function.
   void clear();
 
   /// isExportedInst - Return true if the specified value is an instruction
   /// exported from its block.

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

static bool isUsedOutsideOfDefiningBlock(const Instruction *I) {
   const BasicBlock *BB = I->getParent();
   for (const User *U : I->users())
     if (cast<Instruction>(U)->getParent() != BB || isa<PHINode>(U))
       return true;
 
   return false;
 }
 
-static ISD::NodeType getPreferredExtendForValue(const Value *V) {
-  // For the users of the source value being used for compare instruction, if
-  // the number of signed predicate is greater than unsigned predicate, we
-  // prefer to use SIGN_EXTEND.
+static bool isIncomingBBInLoop(const PHINode *PHI, const LoopInfo *LI) {
+  if (!LI)
+    return true;
+
+  for (BasicBlock *IncomingBB : PHI->blocks()) {
+    if (LI->getLoopDepth(IncomingBB) > 0)
+      return true;
+  }
+
+  return false;
+}
+
+static ISD::NodeType getPreferredExtendForValue(const Value *V,
+                                                const LoopInfo *LI) {
+  // For the users of the source value being used for compare instruction or
+  // PHI node:
+  //
+  // 1. if PHI node is only used for return instruction which is required
+  //    to signext or zeroext, we set default ExtendKind to SIGN_EXTEND or
+  //    ZERO_EXTEND.
+  // 2. if the number of signed predicate is greater than unsigned
+  //    predicate, we prefer to use SIGN_EXTEND.
   //
   // With this optimization, we would be able to reduce some redundant sign or
   // zero extension instruction, and eventually more machine CSE opportunities
   // can be exposed.
   ISD::NodeType ExtendKind = ISD::ANY_EXTEND;
   unsigned NumOfSigned = 0, NumOfUnsigned = 0;
   for (const User *U : V->users()) {
     if (const auto *CI = dyn_cast<CmpInst>(U)) {
       NumOfSigned += CI->isSigned();
       NumOfUnsigned += CI->isUnsigned();
+    } else if (const auto *PHI = dyn_cast<PHINode>(U)) {
+      if (PHI->hasOneUser() && !isIncomingBBInLoop(PHI, LI)) {
+        if (const auto RI = dyn_cast<ReturnInst>(U->user_back())) {
+          const Function *F = RI->getParent()->getParent();
+          if (F->getAttributes().hasRetAttr(Attribute::SExt)) {
+            ExtendKind = ISD::SIGN_EXTEND;
+            NumOfSigned += 1;
+          } else if (F->getAttributes().hasRetAttr(Attribute::ZExt)) {
+            ExtendKind = ISD::ZERO_EXTEND;
+            NumOfUnsigned += 1;
+          }
+        }
+      }
     }
   }
   if (NumOfSigned > NumOfUnsigned)
     ExtendKind = ISD::SIGN_EXTEND;
 
   return ExtendKind;
 }
 
 void FunctionLoweringInfo::set(const Function &fn, MachineFunction &mf,
-                               SelectionDAG *DAG) {
+                               SelectionDAG *DAG, const LoopInfo *LI) {
   Fn = &fn;
   MF = &mf;
   TLI = MF->getSubtarget().getTargetLowering();
   RegInfo = &MF->getRegInfo();
   const TargetFrameLowering *TFI = MF->getSubtarget().getFrameLowering();
   DA = DAG->getDivergenceAnalysis();
 
   // Check whether the function can return without sret-demotion.
for (const Instruction &I : BB) {
 
       // Mark values used outside their block as exported, by allocating
       // a virtual register for them.
       if (isUsedOutsideOfDefiningBlock(&I))
         if (!isa<AllocaInst>(I) || !StaticAllocaMap.count(cast<AllocaInst>(&I)))
           InitializeRegForValue(&I);
 
       // Decide the preferred extend type for a value.
-      PreferredExtendType[&I] = getPreferredExtendForValue(&I);
+      PreferredExtendType[&I] = getPreferredExtendForValue(&I, LI);
     }
   }
 
   // Create an initial MachineBasicBlock for each LLVM BasicBlock in F.  This
   // also creates the initial PHI MachineInstrs, though none of the input
   // operands are populated.
   for (const BasicBlock &BB : *Fn) {
     // Don't create MachineBasicBlocks for imaginary EH pad blocks. These blocks

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

if (PSI && PSI->hasProfileSummary() && OptLevel != CodeGenOpt::None)
     BFI = &getAnalysis<LazyBlockFrequencyInfoPass>().getBFI();
 
   LLVM_DEBUG(dbgs() << "\n\n\n=== " << Fn.getName() << "\n");
 
   SplitCriticalSideEffectEdges(const_cast<Function &>(Fn), DT, LI);
 
   CurDAG->init(*MF, *ORE, this, LibInfo,
                getAnalysisIfAvailable<LegacyDivergenceAnalysis>(), PSI, BFI);
-  FuncInfo->set(Fn, *MF, CurDAG);
+  FuncInfo->set(Fn, *MF, CurDAG, LI);
   SwiftError->setFunction(*MF);
 
   // Now get the optional analyzes if we want to.
   // This is based on the possibly changed OptLevel (after optnone is taken
   // into account).  That's unfortunate but OK because it just means we won't
   // ask for passes that have been required anyway.
 
   if (UseMBPI && OptLevel != CodeGenOpt::None)

llvm/test/CodeGen/ARM/load-global2.ll

 ; LINUX-PIC-NEXT:    .save {r4, lr}
 ; LINUX-PIC-NEXT:    push {r4, lr}
 ; LINUX-PIC-NEXT:    ldr r4, .LCPI0_0
 ; LINUX-PIC-NEXT:    mov r0, #0
 ; LINUX-PIC-NEXT:  .LPC0_0:
 ; LINUX-PIC-NEXT:    ldr r4, [pc, r4]
 ; LINUX-PIC-NEXT:    ldrb r1, [r4]
 ; LINUX-PIC-NEXT:    cmp r1, #0
-; LINUX-PIC-NEXT:    beq .LBB0_2
-; LINUX-PIC-NEXT:  @ %bb.1: @ %common.ret
-; LINUX-PIC-NEXT:    sxtb r0, r0
-; LINUX-PIC-NEXT:    pop {r4, pc}
-; LINUX-PIC-NEXT:  .LBB0_2: @ %bb1
+; LINUX-PIC-NEXT:    popne {r4, pc}
+; LINUX-PIC-NEXT:  .LBB0_1: @ %bb1
 ; LINUX-PIC-NEXT:    bl bar
-; LINUX-PIC-NEXT:    ldrb r0, [r4]
-; LINUX-PIC-NEXT:    sxtb r0, r0
+; LINUX-PIC-NEXT:    ldrsb r0, [r4]
 ; LINUX-PIC-NEXT:    pop {r4, pc}
 ; LINUX-PIC-NEXT:    .p2align 2
-; LINUX-PIC-NEXT:  @ %bb.3:
+; LINUX-PIC-NEXT:  @ %bb.2:
 ; LINUX-PIC-NEXT:  .LCPI0_0:
 ; LINUX-PIC-NEXT:  .Ltmp0:
 ; LINUX-PIC-NEXT:    .long x(GOT_PREL)-((.LPC0_0+8)-.Ltmp0)
 entry:
   %0 = load i8, i8* @x
   %tobool = icmp eq i8 %0, 0
   br i1 %tobool, label %bb1, label %bb2
 

llvm/test/CodeGen/RISCV/prefer-extend.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 < %s | FileCheck -check-prefixes=RV32 %s
+; RUN: llc -mtriple=riscv64 < %s | FileCheck -check-prefixes=RV64 %s
+
+define signext i16 @foo(i16 signext %a, i1 zeroext %b) {
+; RV32-LABEL: foo:
+; RV32:       # %bb.0: # %entry
+; RV32-NEXT:    beqz a1, .LBB0_2
+; RV32-NEXT:  # %bb.1: # %add
+; RV32-NEXT:    addi a0, a0, 1
+; RV32-NEXT:    slli a0, a0, 16
+; RV32-NEXT:    srai a0, a0, 16
+; RV32-NEXT:  .LBB0_2: # %ret
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: foo:
+; RV64:       # %bb.0: # %entry
+; RV64-NEXT:    beqz a1, .LBB0_2
+; RV64-NEXT:  # %bb.1: # %add
+; RV64-NEXT:    addiw a0, a0, 1
+; RV64-NEXT:    slli a0, a0, 48
+; RV64-NEXT:    srai a0, a0, 48
+; RV64-NEXT:  .LBB0_2: # %ret
+; RV64-NEXT:    ret
+entry:
+  br i1 %b, label %add, label %ret
+
+add:
+  %0 = add i16 %a, 1
+  br label %ret
+
+ret:
+  %1 = phi i16 [ %a, %entry ], [ %0, %add ]
+  ret i16 %1
+}
+
+define zeroext i16 @foo1(i16 zeroext %a, i1 zeroext %b) {
+; RV32-LABEL: foo1:
+; RV32:       # %bb.0: # %entry
+; RV32-NEXT:    beqz a1, .LBB1_2
+; RV32-NEXT:  # %bb.1: # %add
+; RV32-NEXT:    addi a0, a0, 1
+; RV32-NEXT:    lui a1, 16
+; RV32-NEXT:    addi a1, a1, -1
+; RV32-NEXT:    and a0, a0, a1
+; RV32-NEXT:  .LBB1_2: # %ret
+; RV32-NEXT:    ret
+;
+; RV64-LABEL: foo1:
+; RV64:       # %bb.0: # %entry
+; RV64-NEXT:    beqz a1, .LBB1_2
+; RV64-NEXT:  # %bb.1: # %add
+; RV64-NEXT:    addi a0, a0, 1
+; RV64-NEXT:    lui a1, 16
+; RV64-NEXT:    addiw a1, a1, -1
+; RV64-NEXT:    and a0, a0, a1
+; RV64-NEXT:  .LBB1_2: # %ret
+; RV64-NEXT:    ret
+entry:
+  br i1 %b, label %add, label %ret
+
+add:
+  %0 = add i16 %a, 1
+  br label %ret
+
+ret:
+  %1 = phi i16 [ %a, %entry ], [ %0, %add ]
+  ret i16 %1
+}

llvm/test/CodeGen/Thumb2/LowOverheadLoops/reductions.ll

for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i8 [ 0, %entry ], [ %i6, %middle.block ]
   ret i8 %res.0.lcssa
 }
 
 define dso_local arm_aapcs_vfpcc signext i16 @one_loop_add_add_v8i16(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: one_loop_add_add_v8i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    ittt eq
+; CHECK-NEXT:    itt eq
 ; CHECK-NEXT:    moveq r0, #0
-; CHECK-NEXT:    sxtheq r0, r0
 ; CHECK-NEXT:    bxeq lr
 ; CHECK-NEXT:  .LBB1_1: @ %vector.ph
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #7
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w r3, r3, r12, lsr #3
 ; CHECK-NEXT:    dls lr, r3
 ; CHECK-NEXT:  .LBB1_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u16 q1, [r0], #8
 ; CHECK-NEXT:    vldrbt.u16 q2, [r1], #8
 ; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vadd.i16 q1, q0, q1
 ; CHECK-NEXT:    vadd.i16 q1, q1, q2
 ; CHECK-NEXT:    le lr, .LBB1_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q1, q0
 ; CHECK-NEXT:    vaddv.u16 r0, q0
-; CHECK-NEXT:    pop.w {r7, lr}
 ; CHECK-NEXT:    sxth r0, r0
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %cmp12 = icmp eq i32 %N, 0
   br i1 %cmp12, label %for.cond.cleanup, label %vector.ph
 
 vector.ph:                                        ; preds = %entry
   %n.rnd.up = add i32 %N, 7
   %n.vec = and i32 %n.rnd.up, -8
   br label %vector.body
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i16 [ 0, %entry ], [ %i10, %middle.block ]
   ret i16 %res.0.lcssa
 }
 
 define dso_local arm_aapcs_vfpcc zeroext i8 @one_loop_sub_add_v16i8(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: one_loop_sub_add_v16i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    ittt eq
+; CHECK-NEXT:    itt eq
 ; CHECK-NEXT:    moveq r0, #0
-; CHECK-NEXT:    uxtbeq r0, r0
 ; CHECK-NEXT:    bxeq lr
 ; CHECK-NEXT:  .LBB2_1: @ %vector.ph
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    add.w r3, r2, #15
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #15
 ; CHECK-NEXT:    sub.w r12, r3, #16
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w r3, r3, r12, lsr #4
 ; CHECK-NEXT:    dls lr, r3
 ; CHECK-NEXT:  .LBB2_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.8 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u8 q1, [r1], #16
 ; CHECK-NEXT:    vldrbt.u8 q2, [r0], #16
 ; CHECK-NEXT:    subs r2, #16
 ; CHECK-NEXT:    vsub.i8 q1, q2, q1
 ; CHECK-NEXT:    vadd.i8 q1, q1, q0
 ; CHECK-NEXT:    le lr, .LBB2_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q1, q0
 ; CHECK-NEXT:    vaddv.u8 r0, q0
-; CHECK-NEXT:    pop.w {r7, lr}
 ; CHECK-NEXT:    uxtb r0, r0
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %cmp11 = icmp eq i32 %N, 0
   br i1 %cmp11, label %for.cond.cleanup, label %vector.ph
 
 vector.ph:                                        ; preds = %entry
   %n.rnd.up = add i32 %N, 15
   %n.vec = and i32 %n.rnd.up, -16
   br label %vector.body
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i8 [ 0, %entry ], [ %i8, %middle.block ]
   ret i8 %res.0.lcssa
 }
 
 define dso_local arm_aapcs_vfpcc signext i16 @one_loop_sub_add_v8i16(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: one_loop_sub_add_v8i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    ittt eq
+; CHECK-NEXT:    itt eq
 ; CHECK-NEXT:    moveq r0, #0
-; CHECK-NEXT:    sxtheq r0, r0
 ; CHECK-NEXT:    bxeq lr
 ; CHECK-NEXT:  .LBB3_1: @ %vector.ph
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #7
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w r3, r3, r12, lsr #3
 ; CHECK-NEXT:    dls lr, r3
 ; CHECK-NEXT:  .LBB3_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u16 q1, [r0], #8
 ; CHECK-NEXT:    vldrbt.u16 q2, [r1], #8
 ; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vsub.i16 q1, q2, q1
 ; CHECK-NEXT:    vadd.i16 q1, q1, q0
 ; CHECK-NEXT:    le lr, .LBB3_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q1, q0
 ; CHECK-NEXT:    vaddv.u16 r0, q0
-; CHECK-NEXT:    pop.w {r7, lr}
 ; CHECK-NEXT:    sxth r0, r0
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %cmp12 = icmp eq i32 %N, 0
   br i1 %cmp12, label %for.cond.cleanup, label %vector.ph
 
 vector.ph:                                        ; preds = %entry
   %n.rnd.up = add i32 %N, 7
   %n.vec = and i32 %n.rnd.up, -8
   br label %vector.body
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i16 [ 0, %entry ], [ %i10, %middle.block ]
   ret i16 %res.0.lcssa
 }
 
 define dso_local arm_aapcs_vfpcc zeroext i8 @one_loop_mul_add_v16i8(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: one_loop_mul_add_v16i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    ittt eq
+; CHECK-NEXT:    itt eq
 ; CHECK-NEXT:    moveq r0, #0
-; CHECK-NEXT:    uxtbeq r0, r0
 ; CHECK-NEXT:    bxeq lr
 ; CHECK-NEXT:  .LBB4_1: @ %vector.ph
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    add.w r3, r2, #15
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #15
 ; CHECK-NEXT:    sub.w r12, r3, #16
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w r3, r3, r12, lsr #4
 ; CHECK-NEXT:    dls lr, r3
 ; CHECK-NEXT:  .LBB4_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.8 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u8 q1, [r0], #16
 ; CHECK-NEXT:    vldrbt.u8 q2, [r1], #16
 ; CHECK-NEXT:    subs r2, #16
 ; CHECK-NEXT:    vmul.i8 q1, q2, q1
 ; CHECK-NEXT:    vadd.i8 q1, q1, q0
 ; CHECK-NEXT:    le lr, .LBB4_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q1, q0
 ; CHECK-NEXT:    vaddv.u8 r0, q0
-; CHECK-NEXT:    pop.w {r7, lr}
 ; CHECK-NEXT:    uxtb r0, r0
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %cmp10 = icmp eq i32 %N, 0
   br i1 %cmp10, label %for.cond.cleanup, label %vector.ph
 
 vector.ph:                                        ; preds = %entry
   %n.rnd.up = add i32 %N, 15
   %n.vec = and i32 %n.rnd.up, -16
   br label %vector.body
for.cond.cleanup:                                 ; preds = %middle.block, %entry
   %res.0.lcssa = phi i8 [ 0, %entry ], [ %i8, %middle.block ]
   ret i8 %res.0.lcssa
 }
 
 define dso_local arm_aapcs_vfpcc signext i16 @one_loop_mul_add_v8i16(i8* nocapture readonly %a, i8* nocapture readonly %b, i32 %N) local_unnamed_addr {
 ; CHECK-LABEL: one_loop_mul_add_v8i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #0
-; CHECK-NEXT:    ittt eq
+; CHECK-NEXT:    itt eq
 ; CHECK-NEXT:    moveq r0, #0
-; CHECK-NEXT:    sxtheq r0, r0
 ; CHECK-NEXT:    bxeq lr
 ; CHECK-NEXT:  .LBB5_1: @ %vector.ph
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    adds r3, r2, #7
 ; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    bic r3, r3, #7
 ; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
 ; CHECK-NEXT:    add.w r3, r3, r12, lsr #3
 ; CHECK-NEXT:    dls lr, r3
 ; CHECK-NEXT:  .LBB5_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vctp.16 r2
 ; CHECK-NEXT:    vmov q0, q1
 ; CHECK-NEXT:    vpstt
 ; CHECK-NEXT:    vldrbt.u16 q1, [r0], #8
 ; CHECK-NEXT:    vldrbt.u16 q2, [r1], #8
 ; CHECK-NEXT:    subs r2, #8
 ; CHECK-NEXT:    vmul.i16 q1, q2, q1
 ; CHECK-NEXT:    vadd.i16 q1, q1, q0
 ; CHECK-NEXT:    le lr, .LBB5_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    vpsel q0, q1, q0
 ; CHECK-NEXT:    vaddv.u16 r0, q0
-; CHECK-NEXT:    pop.w {r7, lr}
 ; CHECK-NEXT:    sxth r0, r0
-; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    pop {r7, pc}
 entry:
   %cmp12 = icmp eq i32 %N, 0
   br i1 %cmp12, label %for.cond.cleanup, label %vector.ph
 
 vector.ph:                                        ; preds = %entry
   %n.rnd.up = add i32 %N, 7
   %n.vec = and i32 %n.rnd.up, -8
   br label %vector.body