[ARM] Code-generation infrastructure for MVE.

[simon_tatham]

Summary

This provides the low-level support to start using MVE vector types in
LLVM IR, loading and storing them, passing them to __asm__ statements
containing hand-written MVE vector instructions, and *if* you have the
hard-float ABI turned on, using them as function parameters.

(In the soft-float ABI, vector types are passed in integer registers,
and combining all those 32-bit integers into a q-reg requires support
for selection DAG nodes like insert_vector_elt and build_vector which
aren't implemented yet for MVE.)

Specifically, this commit adds support for:

• spills, reloads and register moves for MVE vector registers
• ditto for the VPT predication mask that lives in VPR.P0
• make all the MVE vector types legal in ISel, and provide selection DAG patterns for BITCAST, LOAD and STORE
• make loads and stores of scalar FP types conditional on hasFPRegs() rather than hasVFP2Base(). As a result a few existing tests needed their llc command lines updating to use -mattr=-fpregs as their method of turning off all hardware FP support.

[ostannard]

Can any of this be tested yet, or are we still missing some patches?

llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp

809 ↗

(On Diff #195169)

Style nit: opening brace should be on the line above.

933 ↗

(On Diff #195169)

Should we assert that the other register is a GPR here?

llvm/lib/Target/ARM/Thumb2ITBlockPass.cpp

206 ↗

(On Diff #195169)

Does anything generate VPT-predicated instructions yet? If not, this should probably be split into a separate patch.

[t.p.northover]

llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp

808–809 ↗

(On Diff #195169)

Braces go on the same line as the ) in LLVM style.

llvm/lib/Target/ARM/ARMISelDAGToDAG.cpp

1284–1286 ↗

(On Diff #195169)

I think this would be clearer if you used the isInt<7> functions from MathExtras.h.

llvm/lib/Target/ARM/Thumb2ITBlockPass.cpp

208–209 ↗

(On Diff #195169)

I think this function would be a good place to assert that all instructions have at most one kind of predication.

300 ↗

(On Diff #195169)

This continue isn't needed with the new control flow (and is inconsistent with the other branches).

llvm/lib/Target/ARM/Thumb2InstrInfo.cpp

694 ↗

(On Diff #195169)

Possibly better as a static helper; then you don't have to explain it at all in the documentation comments in the header.

[simon_tatham]

This infrastructure should be enough to let you use the vector types in C as a means of getting them to and from __asm__ blocks that do the real work. I suppose that means I should include some tests that exercise exactly that, using asm operations in IR.

llvm/lib/Target/ARM/Thumb2ITBlockPass.cpp

206 ↗

(On Diff #195169)

You have a point – predicated instructions won't be generated by anything I can think of in this patch series, and if you put one in an __asm__ block then it won't be represented in a way that this phase will notice.

So yes, perhaps I should back out this part of the patch until the ACLE intrinsics implementation is ready for review.

[simon_tatham]

Updated this patch so that it doesn't depend on the intrusive calling convention changes in D60707 (which no longer looks necessary, so I'm probably going to abandon it).

[ostannard]

There is still a lot of code here not tested, but I think it will all be easier to test once we have some more actual instruction selection, so LGTM.

[simon_tatham]

Remastered patch to apply cleanly against current trunk.

[dmgreen]

llvm/test/CodeGen/Thumb2/mve-basic.ll

2 ↗

(On Diff #204038)

Can you also add a -mattr=+mve run line?

[simon_tatham]

Revised this patch to work properly with all the other MVE-related changes we've been making.

In particular, I've incorporated the whole of D63252 into this patch, because after I added a RUN line to the new mve-basic.ll test to try it in integer-only MVE, I found the bitconvert selection patterns from that patch were needed to get it to pass.

A few other changes: I made all the vector types legal even in integer-only MVE (in that you can load, store and bitconvert them, but nothing else), which is necessary so that the calling convention can work (since it converts everything to v2f64, so that has to be legal!). I added a missing case in copyPhysReg, which I had previously switched from the NEON VORRq instruction to MVE_VORR for the purposes of copying a single q-reg, but forgot to do the same for copying multiple q-regs (the QQPR or QQQQPR reg classes). Integer MVE support also needed me to legalise the scalar FP types, so I changed the existing condition for that to hasFPRegs in place of hasVFP2Base, which had knock-on effects on a couple of existing tests.

[ostannard]

LGTM

[dmgreen]

Nice one

llvm/test/CodeGen/Thumb2/mve-bitcasts.ll

2 ↗

(On Diff #206454)

This test might as well get a -mattr=+mve line too, if we expect the output to be the same.

llvm/trunk/lib/Target/ARM/ARMBaseInstrInfo.h

bool registerDefinedBetween(unsigned Reg, MachineBasicBlock::iterator From,
                             MachineBasicBlock::iterator To,
                             const TargetRegisterInfo *TRI);
 
 /// Search backwards from a tBcc to find a tCMPi8 against 0, meaning
 /// we can convert them to a tCBZ or tCBNZ. Return nullptr if not found.
 MachineInstr *findCMPToFoldIntoCBZ(MachineInstr *Br,
                                    const TargetRegisterInfo *TRI);
 
+void addUnpredicatedMveVpredNOp(MachineInstrBuilder &MIB);
+void addUnpredicatedMveVpredROp(MachineInstrBuilder &MIB, unsigned DestReg);
+
+void addPredicatedMveVpredNOp(MachineInstrBuilder &MIB, unsigned Cond);
+void addPredicatedMveVpredROp(MachineInstrBuilder &MIB, unsigned Cond,
+                              unsigned Inactive);
+
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_ARM_ARMBASEINSTRINFO_H

llvm/trunk/lib/Target/ARM/ARMBaseInstrInfo.cpp

void ARMBaseInstrInfo::copyToCPSR(MachineBasicBlock &MBB,
   else
     MIB.addImm(8);
 
   MIB.addReg(SrcReg, getKillRegState(KillSrc))
      .add(predOps(ARMCC::AL))
      .addReg(ARM::CPSR, RegState::Implicit | RegState::Define);
 }
 
+void llvm::addUnpredicatedMveVpredNOp(MachineInstrBuilder &MIB) {
+  MIB.addImm(ARMVCC::None);
+  MIB.addReg(0);
+}
+
+void llvm::addUnpredicatedMveVpredROp(MachineInstrBuilder &MIB,
+                                      unsigned DestReg) {
+  addUnpredicatedMveVpredNOp(MIB);
+  MIB.addReg(DestReg, RegState::Undef);
+}
+
+void llvm::addPredicatedMveVpredNOp(MachineInstrBuilder &MIB, unsigned Cond) {
+  MIB.addImm(Cond);
+  MIB.addReg(ARM::VPR, RegState::Implicit);
+}
+
+void llvm::addPredicatedMveVpredROp(MachineInstrBuilder &MIB,
+                                    unsigned Cond, unsigned Inactive) {
+  addPredicatedMveVpredNOp(MIB, Cond);
+  MIB.addReg(Inactive);
+}
+
 void ARMBaseInstrInfo::copyPhysReg(MachineBasicBlock &MBB,
                                    MachineBasicBlock::iterator I,
                                    const DebugLoc &DL, unsigned DestReg,
                                    unsigned SrcReg, bool KillSrc) const {
   bool GPRDest = ARM::GPRRegClass.contains(DestReg);
   bool GPRSrc = ARM::GPRRegClass.contains(SrcReg);
 
   if (GPRDest && GPRSrc) {
if (SPRDest && SPRSrc)
     Opc = ARM::VMOVS;
   else if (GPRDest && SPRSrc)
     Opc = ARM::VMOVRS;
   else if (SPRDest && GPRSrc)
     Opc = ARM::VMOVSR;
   else if (ARM::DPRRegClass.contains(DestReg, SrcReg) && Subtarget.hasFP64())
     Opc = ARM::VMOVD;
   else if (ARM::QPRRegClass.contains(DestReg, SrcReg))
-    Opc = ARM::VORRq;
+    Opc = Subtarget.hasNEON() ? ARM::VORRq : ARM::MVE_VORR;
 
   if (Opc) {
     MachineInstrBuilder MIB = BuildMI(MBB, I, DL, get(Opc), DestReg);
     MIB.addReg(SrcReg, getKillRegState(KillSrc));
-    if (Opc == ARM::VORRq)
+    if (Opc == ARM::VORRq || Opc == ARM::MVE_VORR)
       MIB.addReg(SrcReg, getKillRegState(KillSrc));
+    if (Opc == ARM::MVE_VORR)
+      addUnpredicatedMveVpredROp(MIB, DestReg);
+    else
       MIB.add(predOps(ARMCC::AL));
     return;
   }
 
   // Handle register classes that require multiple instructions.
   unsigned BeginIdx = 0;
   unsigned SubRegs = 0;
   int Spacing = 1;
 
   // Use VORRq when possible.
   if (ARM::QQPRRegClass.contains(DestReg, SrcReg)) {
-    Opc = ARM::VORRq;
+    Opc = Subtarget.hasNEON() ? ARM::VORRq : ARM::MVE_VORR;
     BeginIdx = ARM::qsub_0;
     SubRegs = 2;
   } else if (ARM::QQQQPRRegClass.contains(DestReg, SrcReg)) {
-    Opc = ARM::VORRq;
+    Opc = Subtarget.hasNEON() ? ARM::VORRq : ARM::MVE_VORR;
     BeginIdx = ARM::qsub_0;
     SubRegs = 4;
   // Fall back to VMOVD.
   } else if (ARM::DPairRegClass.contains(DestReg, SrcReg)) {
     Opc = ARM::VMOVD;
     BeginIdx = ARM::dsub_0;
     SubRegs = 2;
   } else if (ARM::DTripleRegClass.contains(DestReg, SrcReg)) {
// Fall back to VMOVD.
     BeginIdx = ARM::ssub_0;
     SubRegs = 2;
   } else if (SrcReg == ARM::CPSR) {
     copyFromCPSR(MBB, I, DestReg, KillSrc, Subtarget);
     return;
   } else if (DestReg == ARM::CPSR) {
     copyToCPSR(MBB, I, SrcReg, KillSrc, Subtarget);
     return;
+  } else if (DestReg == ARM::VPR) {
+    assert(ARM::GPRPairRegClass.contains(SrcReg));
+    BuildMI(MBB, I, I->getDebugLoc(), get(ARM::VMSR_P0), DestReg)
+        .addReg(SrcReg, getKillRegState(KillSrc))
+        .add(predOps(ARMCC::AL));
+    return;
+  } else if (SrcReg == ARM::VPR) {
+    assert(ARM::GPRPairRegClass.contains(DestReg));
+    BuildMI(MBB, I, I->getDebugLoc(), get(ARM::VMRS_P0), DestReg)
+        .addReg(SrcReg, getKillRegState(KillSrc))
+        .add(predOps(ARMCC::AL));
+    return;
+  } else if (DestReg == ARM::FPSCR_NZCV) {
+    assert(ARM::GPRPairRegClass.contains(SrcReg));
+    BuildMI(MBB, I, I->getDebugLoc(), get(ARM::VMSR_FPSCR_NZCVQC), DestReg)
+        .addReg(SrcReg, getKillRegState(KillSrc))
+        .add(predOps(ARMCC::AL));
+    return;
+  } else if (SrcReg == ARM::FPSCR_NZCV) {
+    assert(ARM::GPRPairRegClass.contains(DestReg));
+    BuildMI(MBB, I, I->getDebugLoc(), get(ARM::VMRS_FPSCR_NZCVQC), DestReg)
+        .addReg(SrcReg, getKillRegState(KillSrc))
+        .add(predOps(ARMCC::AL));
+    return;
   }
 
   assert(Opc && "Impossible reg-to-reg copy");
 
   const TargetRegisterInfo *TRI = &getRegisterInfo();
   MachineInstrBuilder Mov;
 
   // Copy register tuples backward when the first Dest reg overlaps with SrcReg.
   if (TRI->regsOverlap(SrcReg, TRI->getSubReg(DestReg, BeginIdx))) {
     BeginIdx = BeginIdx + ((SubRegs - 1) * Spacing);
     Spacing = -Spacing;
   }
 #ifndef NDEBUG
   SmallSet<unsigned, 4> DstRegs;
 #endif
   for (unsigned i = 0; i != SubRegs; ++i) {
     unsigned Dst = TRI->getSubReg(DestReg, BeginIdx + i * Spacing);
     unsigned Src = TRI->getSubReg(SrcReg, BeginIdx + i * Spacing);
     assert(Dst && Src && "Bad sub-register");
 #ifndef NDEBUG
     assert(!DstRegs.count(Src) && "destructive vector copy");
     DstRegs.insert(Dst);
 #endif
     Mov = BuildMI(MBB, I, I->getDebugLoc(), get(Opc), Dst).addReg(Src);
-    // VORR takes two source operands.
-    if (Opc == ARM::VORRq)
+    // VORR (NEON or MVE) takes two source operands.
+    if (Opc == ARM::VORRq || Opc == ARM::MVE_VORR) {
       Mov.addReg(Src);
+    }
+    // MVE VORR takes predicate operands in place of an ordinary condition.
+    if (Opc == ARM::MVE_VORR)
+      addUnpredicatedMveVpredROp(Mov, Dst);
+    else
       Mov = Mov.add(predOps(ARMCC::AL));
     // MOVr can set CC.
     if (Opc == ARM::MOVr)
       Mov = Mov.add(condCodeOp());
   }
   // Add implicit super-register defs and kills to the last instruction.
   Mov->addRegisterDefined(DestReg, TRI);
   if (KillSrc)
     Mov->addRegisterKilled(SrcReg, TRI);
case 4:
             .add(predOps(ARMCC::AL));
       } else if (ARM::SPRRegClass.hasSubClassEq(RC)) {
         BuildMI(MBB, I, DebugLoc(), get(ARM::VSTRS))
             .addReg(SrcReg, getKillRegState(isKill))
             .addFrameIndex(FI)
             .addImm(0)
             .addMemOperand(MMO)
             .add(predOps(ARMCC::AL));
+      } else if (ARM::VCCRRegClass.hasSubClassEq(RC)) {
+        BuildMI(MBB, I, DebugLoc(), get(ARM::VSTR_P0_off))
+            .addReg(SrcReg, getKillRegState(isKill))
+            .addFrameIndex(FI)
+            .addImm(0)
+            .addMemOperand(MMO)
+            .add(predOps(ARMCC::AL));
       } else
         llvm_unreachable("Unknown reg class!");
       break;
     case 8:
       if (ARM::DPRRegClass.hasSubClassEq(RC)) {
         BuildMI(MBB, I, DebugLoc(), get(ARM::VSTRD))
             .addReg(SrcReg, getKillRegState(isKill))
             .addFrameIndex(FI)
case 8:
                                         .add(predOps(ARMCC::AL));
           AddDReg(MIB, SrcReg, ARM::gsub_0, getKillRegState(isKill), TRI);
           AddDReg(MIB, SrcReg, ARM::gsub_1, 0, TRI);
         }
       } else
         llvm_unreachable("Unknown reg class!");
       break;
     case 16:
-      if (ARM::DPairRegClass.hasSubClassEq(RC)) {
+      if (ARM::DPairRegClass.hasSubClassEq(RC) && Subtarget.hasNEON()) {
         // Use aligned spills if the stack can be realigned.
         if (Align >= 16 && getRegisterInfo().canRealignStack(MF)) {
           BuildMI(MBB, I, DebugLoc(), get(ARM::VST1q64))
               .addFrameIndex(FI)
               .addImm(16)
               .addReg(SrcReg, getKillRegState(isKill))
               .addMemOperand(MMO)
               .add(predOps(ARMCC::AL));
         } else {
           BuildMI(MBB, I, DebugLoc(), get(ARM::VSTMQIA))
               .addReg(SrcReg, getKillRegState(isKill))
               .addFrameIndex(FI)
               .addMemOperand(MMO)
               .add(predOps(ARMCC::AL));
         }
+      } else if (ARM::QPRRegClass.hasSubClassEq(RC) &&
+                 Subtarget.hasMVEIntegerOps()) {
+        auto MIB = BuildMI(MBB, I, DebugLoc(), get(ARM::MVE_VSTRWU32));
+        MIB.addReg(SrcReg, getKillRegState(isKill))
+          .addFrameIndex(FI)
+          .addImm(0)
+          .addMemOperand(MMO);
+        addUnpredicatedMveVpredNOp(MIB);
       } else
         llvm_unreachable("Unknown reg class!");
       break;
     case 24:
       if (ARM::DTripleRegClass.hasSubClassEq(RC)) {
         // Use aligned spills if the stack can be realigned.
         if (Align >= 16 && getRegisterInfo().canRealignStack(MF)) {
           BuildMI(MBB, I, DebugLoc(), get(ARM::VST1d64TPseudo))
unsigned ARMBaseInstrInfo::isStoreToStackSlot(const MachineInstr &MI,
   case ARM::VSTRD:
   case ARM::VSTRS:
     if (MI.getOperand(1).isFI() && MI.getOperand(2).isImm() &&
         MI.getOperand(2).getImm() == 0) {
       FrameIndex = MI.getOperand(1).getIndex();
       return MI.getOperand(0).getReg();
     }
     break;
+  case ARM::VSTR_P0_off:
+    if (MI.getOperand(0).isFI() && MI.getOperand(1).isImm() &&
+        MI.getOperand(1).getImm() == 0) {
+      FrameIndex = MI.getOperand(0).getIndex();
+      return ARM::P0;
+    }
+    break;
   case ARM::VST1q64:
   case ARM::VST1d64TPseudo:
   case ARM::VST1d64QPseudo:
     if (MI.getOperand(0).isFI() && MI.getOperand(2).getSubReg() == 0) {
       FrameIndex = MI.getOperand(0).getIndex();
       return MI.getOperand(2).getReg();
     }
     break;
if (ARM::GPRRegClass.hasSubClassEq(RC)) {
           .addMemOperand(MMO)
           .add(predOps(ARMCC::AL));
     } else if (ARM::SPRRegClass.hasSubClassEq(RC)) {
       BuildMI(MBB, I, DL, get(ARM::VLDRS), DestReg)
           .addFrameIndex(FI)
           .addImm(0)
           .addMemOperand(MMO)
           .add(predOps(ARMCC::AL));
+    } else if (ARM::VCCRRegClass.hasSubClassEq(RC)) {
+      BuildMI(MBB, I, DL, get(ARM::VLDR_P0_off), DestReg)
+          .addFrameIndex(FI)
+          .addImm(0)
+          .addMemOperand(MMO)
+          .add(predOps(ARMCC::AL));
     } else
       llvm_unreachable("Unknown reg class!");
     break;
   case 8:
     if (ARM::DPRRegClass.hasSubClassEq(RC)) {
       BuildMI(MBB, I, DL, get(ARM::VLDRD), DestReg)
           .addFrameIndex(FI)
           .addImm(0)
if (ARM::DPRRegClass.hasSubClassEq(RC)) {
       }
 
       if (TargetRegisterInfo::isPhysicalRegister(DestReg))
         MIB.addReg(DestReg, RegState::ImplicitDefine);
     } else
       llvm_unreachable("Unknown reg class!");
     break;
   case 16:
-    if (ARM::DPairRegClass.hasSubClassEq(RC)) {
+    if (ARM::DPairRegClass.hasSubClassEq(RC) && Subtarget.hasNEON()) {
       if (Align >= 16 && getRegisterInfo().canRealignStack(MF)) {
         BuildMI(MBB, I, DL, get(ARM::VLD1q64), DestReg)
             .addFrameIndex(FI)
             .addImm(16)
             .addMemOperand(MMO)
             .add(predOps(ARMCC::AL));
       } else {
         BuildMI(MBB, I, DL, get(ARM::VLDMQIA), DestReg)
             .addFrameIndex(FI)
             .addMemOperand(MMO)
             .add(predOps(ARMCC::AL));
       }
+    } else if (ARM::QPRRegClass.hasSubClassEq(RC) &&
+               Subtarget.hasMVEIntegerOps()) {
+      auto MIB = BuildMI(MBB, I, DL, get(ARM::MVE_VLDRWU32), DestReg);
+      MIB.addFrameIndex(FI)
+        .addImm(0)
+        .addMemOperand(MMO);
+      addUnpredicatedMveVpredNOp(MIB);
     } else
       llvm_unreachable("Unknown reg class!");
     break;
   case 24:
     if (ARM::DTripleRegClass.hasSubClassEq(RC)) {
       if (Align >= 16 && getRegisterInfo().canRealignStack(MF)) {
         BuildMI(MBB, I, DL, get(ARM::VLD1d64TPseudo), DestReg)
             .addFrameIndex(FI)
unsigned ARMBaseInstrInfo::isLoadFromStackSlot(const MachineInstr &MI,
   case ARM::VLDRD:
   case ARM::VLDRS:
     if (MI.getOperand(1).isFI() && MI.getOperand(2).isImm() &&
         MI.getOperand(2).getImm() == 0) {
       FrameIndex = MI.getOperand(1).getIndex();
       return MI.getOperand(0).getReg();
     }
     break;
+  case ARM::VLDR_P0_off:
+    if (MI.getOperand(0).isFI() && MI.getOperand(1).isImm() &&
+        MI.getOperand(1).getImm() == 0) {
+      FrameIndex = MI.getOperand(0).getIndex();
+      return ARM::P0;
+    }
+    break;
   case ARM::VLD1q64:
   case ARM::VLD1d8TPseudo:
   case ARM::VLD1d16TPseudo:
   case ARM::VLD1d32TPseudo:
   case ARM::VLD1d64TPseudo:
   case ARM::VLD1d8QPseudo:
   case ARM::VLD1d16QPseudo:
   case ARM::VLD1d32QPseudo:

llvm/trunk/lib/Target/ARM/ARMISelDAGToDAG.cpp

public:
   bool SelectThumbAddrModeSP(SDValue N, SDValue &Base, SDValue &OffImm);
 
   // Thumb 2 Addressing Modes:
   bool SelectT2AddrModeImm12(SDValue N, SDValue &Base, SDValue &OffImm);
   bool SelectT2AddrModeImm8(SDValue N, SDValue &Base,
                             SDValue &OffImm);
   bool SelectT2AddrModeImm8Offset(SDNode *Op, SDValue N,
                                  SDValue &OffImm);
+  template<unsigned Shift>
+  bool SelectT2AddrModeImm7(SDValue N, SDValue &Base,
+                            SDValue &OffImm);
   bool SelectT2AddrModeSoReg(SDValue N, SDValue &Base,
                              SDValue &OffReg, SDValue &ShImm);
   bool SelectT2AddrModeExclusive(SDValue N, SDValue &Base, SDValue &OffImm);
 
   inline bool is_so_imm(unsigned Imm) const {
     return ARM_AM::getSOImmVal(Imm) != -1;
   }
 
OffImm = ((AM == ISD::PRE_INC) || (AM == ISD::POST_INC))
       ? CurDAG->getTargetConstant(RHSC, SDLoc(N), MVT::i32)
       : CurDAG->getTargetConstant(-RHSC, SDLoc(N), MVT::i32);
     return true;
   }
 
   return false;
 }
 
+template<unsigned Shift>
+bool ARMDAGToDAGISel::SelectT2AddrModeImm7(SDValue N,
+                                           SDValue &Base, SDValue &OffImm) {
+  if (N.getOpcode() == ISD::SUB ||
+      CurDAG->isBaseWithConstantOffset(N)) {
+    if (auto RHS = dyn_cast<ConstantSDNode>(N.getOperand(1))) {
+      int RHSC = (int)RHS->getZExtValue();
+      if (N.getOpcode() == ISD::SUB)
+        RHSC = -RHSC;
+
+      if (isShiftedInt<7, Shift>(RHSC)) {
+        Base = N.getOperand(0);
+        if (Base.getOpcode() == ISD::FrameIndex) {
+          int FI = cast<FrameIndexSDNode>(Base)->getIndex();
+          Base = CurDAG->getTargetFrameIndex(
+            FI, TLI->getPointerTy(CurDAG->getDataLayout()));
+        }
+        OffImm = CurDAG->getTargetConstant(RHSC, SDLoc(N), MVT::i32);
+        return true;
+      }
+    }
+  }
+
+  // Base only.
+  Base = N;
+  OffImm  = CurDAG->getTargetConstant(0, SDLoc(N), MVT::i32);
+  return true;
+}
+
 bool ARMDAGToDAGISel::SelectT2AddrModeSoReg(SDValue N,
                                             SDValue &Base,
                                             SDValue &OffReg, SDValue &ShImm) {
   // (R - imm8) should be handled by t2LDRi8. The rest are handled by t2LDRi12.
   if (N.getOpcode() != ISD::ADD && !CurDAG->isBaseWithConstantOffset(N))
     return false;
 
   // Leave (R + imm12) for t2LDRi12, (R - imm8) for t2LDRi8.

llvm/trunk/lib/Target/ARM/ARMISelLowering.h

private:
 
     MachineBasicBlock *EmitStructByval(MachineInstr &MI,
                                        MachineBasicBlock *MBB) const;
 
     MachineBasicBlock *EmitLowered__chkstk(MachineInstr &MI,
                                            MachineBasicBlock *MBB) const;
     MachineBasicBlock *EmitLowered__dbzchk(MachineInstr &MI,
                                            MachineBasicBlock *MBB) const;
+    void addMVEVectorTypes();
   };
 
   enum NEONModImmType {
     VMOVModImm,
     VMVNModImm,
     OtherModImm
   };
 

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

void ARMTargetLowering::addDRTypeForNEON(MVT VT) {
   addTypeForNEON(VT, MVT::f64, MVT::v2i32);
 }
 
 void ARMTargetLowering::addQRTypeForNEON(MVT VT) {
   addRegisterClass(VT, &ARM::DPairRegClass);
   addTypeForNEON(VT, MVT::v2f64, MVT::v4i32);
 }
 
+void ARMTargetLowering::addMVEVectorTypes() {
+  // We 'support' these types up to bitcast/load/store level, regardless of
+  // MVE integer-only / float support. Only doing FP data processing on the FP
+  // vector types is inhibited at integer-only level.
+
+  const MVT VecTypes[] = {
+      MVT::v2i64, MVT::v4i32, MVT::v8i16, MVT::v16i8,
+      MVT::v2f64, MVT::v4f32, MVT::v8f16,
+  };
+
+  for (auto VT : VecTypes) {
+    addRegisterClass(VT, &ARM::QPRRegClass);
+    for (unsigned Opc = 0; Opc < ISD::BUILTIN_OP_END; ++Opc)
+      setOperationAction(Opc, VT, Expand);
+    setOperationAction(ISD::BITCAST, VT, Legal);
+    setOperationAction(ISD::LOAD, VT, Legal);
+    setOperationAction(ISD::STORE, VT, Legal);
+  }
+}
+
 ARMTargetLowering::ARMTargetLowering(const TargetMachine &TM,
                                      const ARMSubtarget &STI)
     : TargetLowering(TM), Subtarget(&STI) {
   RegInfo = Subtarget->getRegisterInfo();
   Itins = Subtarget->getInstrItineraryData();
 
   setBooleanContents(ZeroOrOneBooleanContent);
   setBooleanVectorContents(ZeroOrNegativeOneBooleanContent);
if (Subtarget->isTargetAEABI()) {
     }
   }
 
   if (Subtarget->isThumb1Only())
     addRegisterClass(MVT::i32, &ARM::tGPRRegClass);
   else
     addRegisterClass(MVT::i32, &ARM::GPRRegClass);
 
-  if (!Subtarget->useSoftFloat() && Subtarget->hasVFP2Base() &&
+  if (!Subtarget->useSoftFloat() && Subtarget->hasFPRegs() &&
       !Subtarget->isThumb1Only()) {
     addRegisterClass(MVT::f32, &ARM::SPRRegClass);
     addRegisterClass(MVT::f64, &ARM::DPRRegClass);
   }
 
   if (Subtarget->hasFullFP16()) {
     addRegisterClass(MVT::f16, &ARM::HPRRegClass);
     setOperationAction(ISD::BITCAST, MVT::i16, Custom);
ARMTargetLowering::ARMTargetLowering(const TargetMachine &TM,
   }
 
   setOperationAction(ISD::ConstantFP, MVT::f32, Custom);
   setOperationAction(ISD::ConstantFP, MVT::f64, Custom);
 
   setOperationAction(ISD::READ_REGISTER, MVT::i64, Custom);
   setOperationAction(ISD::WRITE_REGISTER, MVT::i64, Custom);
 
+  if (Subtarget->hasMVEIntegerOps())
+    addMVEVectorTypes();
+
   if (Subtarget->hasNEON()) {
     addDRTypeForNEON(MVT::v2f32);
     addDRTypeForNEON(MVT::v8i8);
     addDRTypeForNEON(MVT::v4i16);
     addDRTypeForNEON(MVT::v2i32);
     addDRTypeForNEON(MVT::v1i64);
 
     addQRTypeForNEON(MVT::v4f32);
     addQRTypeForNEON(MVT::v2f64);
     addQRTypeForNEON(MVT::v16i8);
     addQRTypeForNEON(MVT::v8i16);
     addQRTypeForNEON(MVT::v4i32);
     addQRTypeForNEON(MVT::v2i64);
 
     if (Subtarget->hasFullFP16()) {
       addQRTypeForNEON(MVT::v8f16);
       addDRTypeForNEON(MVT::v4f16);
     }
+  }
 
+  if (Subtarget->hasMVEIntegerOps() || Subtarget->hasNEON()) {
     // v2f64 is legal so that QR subregs can be extracted as f64 elements, but
-    // neither Neon nor VFP support any arithmetic operations on it.
-    // The same with v4f32. But keep in mind that vadd, vsub, vmul are natively
-    // supported for v4f32.
+    // none of Neon, MVE or VFP supports any arithmetic operations on it.
     setOperationAction(ISD::FADD, MVT::v2f64, Expand);
     setOperationAction(ISD::FSUB, MVT::v2f64, Expand);
     setOperationAction(ISD::FMUL, MVT::v2f64, Expand);
     // FIXME: Code duplication: FDIV and FREM are expanded always, see
     // ARMTargetLowering::addTypeForNEON method for details.
     setOperationAction(ISD::FDIV, MVT::v2f64, Expand);
     setOperationAction(ISD::FREM, MVT::v2f64, Expand);
     // FIXME: Create unittest.
if (Subtarget->hasMVEIntegerOps() || Subtarget->hasNEON()) {
     setOperationAction(ISD::FEXP2, MVT::v2f64, Expand);
     // FIXME: Create unittest for FCEIL, FTRUNC, FRINT, FNEARBYINT, FFLOOR.
     setOperationAction(ISD::FCEIL, MVT::v2f64, Expand);
     setOperationAction(ISD::FTRUNC, MVT::v2f64, Expand);
     setOperationAction(ISD::FRINT, MVT::v2f64, Expand);
     setOperationAction(ISD::FNEARBYINT, MVT::v2f64, Expand);
     setOperationAction(ISD::FFLOOR, MVT::v2f64, Expand);
     setOperationAction(ISD::FMA, MVT::v2f64, Expand);
+  }
 
+  if (Subtarget->hasNEON()) {
+    // The same with v4f32. But keep in mind that vadd, vsub, vmul are natively
+    // supported for v4f32.
     setOperationAction(ISD::FSQRT, MVT::v4f32, Expand);
     setOperationAction(ISD::FSIN, MVT::v4f32, Expand);
     setOperationAction(ISD::FCOS, MVT::v4f32, Expand);
     setOperationAction(ISD::FPOW, MVT::v4f32, Expand);
     setOperationAction(ISD::FLOG, MVT::v4f32, Expand);
     setOperationAction(ISD::FLOG2, MVT::v4f32, Expand);
     setOperationAction(ISD::FLOG10, MVT::v4f32, Expand);
     setOperationAction(ISD::FEXP, MVT::v4f32, Expand);
ARMTargetLowering::ARMTargetLowering(const TargetMachine &TM,
 
   // Requires SXTB/SXTH, available on v6 and up in both ARM and Thumb modes.
   if (!Subtarget->hasV6Ops()) {
     setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i16, Expand);
     setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i8,  Expand);
   }
   setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i1, Expand);
 
-  if (!Subtarget->useSoftFloat() && Subtarget->hasVFP2Base() &&
+  if (!Subtarget->useSoftFloat() && Subtarget->hasFPRegs() &&
       !Subtarget->isThumb1Only()) {
     // Turn f64->i64 into VMOVRRD, i64 -> f64 to VMOVDRR
     // iff target supports vfp2.
     setOperationAction(ISD::BITCAST, MVT::i64, Custom);
     setOperationAction(ISD::FLT_ROUNDS_, MVT::i32, Custom);
   }
 
   // We want to custom lower some of our intrinsics.

llvm/trunk/lib/Target/ARM/ARMInstrMVE.td

 
 def MVE_LCTP : MVE_loltp_end<(outs), (ins pred:$p), "lctp${p}", ""> {
   let Inst{20} = 0b0;
   let Inst{13} = 0b1;
   let Inst{11-1} = 0b00000000000;
   let Unpredictable{21-20} = 0b11;
   let Unpredictable{11-1} = 0b11111111111;
 }
+
+
+//===----------------------------------------------------------------------===//
+// Patterns
+//===----------------------------------------------------------------------===//
+
+class MVE_unpred_vector_store_typed<ValueType Ty, Instruction RegImmInst,
+                                    PatFrag StoreKind, int shift>
+      : Pat<(StoreKind (Ty MQPR:$val), t2addrmode_imm7<shift>:$addr),
+           (RegImmInst (Ty MQPR:$val), t2addrmode_imm7<shift>:$addr)>;
+
+multiclass MVE_unpred_vector_store<Instruction RegImmInst, PatFrag StoreKind,
+                                   int shift> {
+  def : MVE_unpred_vector_store_typed<v16i8, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_store_typed<v8i16, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_store_typed<v8f16, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_store_typed<v4i32, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_store_typed<v4f32, RegImmInst, StoreKind, shift>;
+  def : MVE_unpred_vector_store_typed<v2i64, RegImmInst, StoreKind, shift>;
+}
+
+class MVE_unpred_vector_load_typed<ValueType Ty, Instruction RegImmInst,
+                                   PatFrag LoadKind, int shift>
+      : Pat<(Ty (LoadKind t2addrmode_imm7<shift>:$addr)),
+          (Ty (RegImmInst t2addrmode_imm7<shift>:$addr))>;
+multiclass MVE_unpred_vector_load<Instruction RegImmInst, PatFrag LoadKind,
+                                  int shift> {
+  def : MVE_unpred_vector_load_typed<v16i8, RegImmInst, LoadKind, shift>;
+  def : MVE_unpred_vector_load_typed<v8i16, RegImmInst, LoadKind, shift>;
+  def : MVE_unpred_vector_load_typed<v8f16, RegImmInst, LoadKind, shift>;
+  def : MVE_unpred_vector_load_typed<v4i32, RegImmInst, LoadKind, shift>;
+  def : MVE_unpred_vector_load_typed<v4f32, RegImmInst, LoadKind, shift>;
+  def : MVE_unpred_vector_load_typed<v2i64, RegImmInst, LoadKind, shift>;
+}
+
+let Predicates = [HasMVEInt, IsLE] in {
+  defm : MVE_unpred_vector_store<MVE_VSTRBU8, byte_alignedstore, 0>;
+  defm : MVE_unpred_vector_store<MVE_VSTRHU16, hword_alignedstore, 1>;
+  defm : MVE_unpred_vector_store<MVE_VSTRWU32, alignedstore32, 2>;
+
+  defm : MVE_unpred_vector_load<MVE_VLDRBU8, byte_alignedload, 0>;
+  defm : MVE_unpred_vector_load<MVE_VLDRHU16, hword_alignedload, 1>;
+  defm : MVE_unpred_vector_load<MVE_VLDRWU32, alignedload32, 2>;
+
+  def  : Pat<(v16i1 (load t2addrmode_imm7<2>:$addr)),
+             (v16i1 (VLDR_P0_off t2addrmode_imm7<2>:$addr))>;
+  def  : Pat<(v8i1 (load t2addrmode_imm7<2>:$addr)),
+             (v8i1 (VLDR_P0_off t2addrmode_imm7<2>:$addr))>;
+  def  : Pat<(v4i1 (load t2addrmode_imm7<2>:$addr)),
+             (v4i1 (VLDR_P0_off t2addrmode_imm7<2>:$addr))>;
+}
+
+let Predicates = [HasMVEInt, IsBE] in {
+  def : MVE_unpred_vector_store_typed<v16i8, MVE_VSTRBU8, store, 0>;
+  def : MVE_unpred_vector_store_typed<v8i16, MVE_VSTRHU16, alignedstore16, 1>;
+  def : MVE_unpred_vector_store_typed<v8f16, MVE_VSTRHU16, alignedstore16, 1>;
+  def : MVE_unpred_vector_store_typed<v4i32, MVE_VSTRWU32, alignedstore32, 2>;
+  def : MVE_unpred_vector_store_typed<v4f32, MVE_VSTRWU32, alignedstore32, 2>;
+
+  def : MVE_unpred_vector_load_typed<v16i8, MVE_VLDRBU8, load, 0>;
+  def : MVE_unpred_vector_load_typed<v8i16, MVE_VLDRHU16, alignedload16, 1>;
+  def : MVE_unpred_vector_load_typed<v8f16, MVE_VLDRHU16, alignedload16, 1>;
+  def : MVE_unpred_vector_load_typed<v4i32, MVE_VLDRWU32, alignedload32, 2>;
+  def : MVE_unpred_vector_load_typed<v4f32, MVE_VLDRWU32, alignedload32, 2>;
+}
+
+// Bit convert patterns
+
+let Predicates = [HasMVEInt] in {
+  def : Pat<(v2f64 (bitconvert (v2i64 QPR:$src))), (v2f64 QPR:$src)>;
+  def : Pat<(v2i64 (bitconvert (v2f64 QPR:$src))), (v2i64 QPR:$src)>;
+
+  def : Pat<(v4i32 (bitconvert (v4f32 QPR:$src))), (v4i32 QPR:$src)>;
+  def : Pat<(v4f32 (bitconvert (v4i32 QPR:$src))), (v4f32 QPR:$src)>;
+
+  def : Pat<(v8i16 (bitconvert (v8f16 QPR:$src))), (v8i16  QPR:$src)>;
+  def : Pat<(v8f16 (bitconvert (v8i16 QPR:$src))), (v8f16  QPR:$src)>;
+}
+
+let Predicates = [IsLE,HasMVEInt] in {
+  def : Pat<(v2f64 (bitconvert (v4f32 QPR:$src))), (v2f64 QPR:$src)>;
+  def : Pat<(v2f64 (bitconvert (v4i32 QPR:$src))), (v2f64 QPR:$src)>;
+  def : Pat<(v2f64 (bitconvert (v8f16 QPR:$src))), (v2f64 QPR:$src)>;
+  def : Pat<(v2f64 (bitconvert (v8i16 QPR:$src))), (v2f64 QPR:$src)>;
+  def : Pat<(v2f64 (bitconvert (v16i8 QPR:$src))), (v2f64 QPR:$src)>;
+
+  def : Pat<(v2i64 (bitconvert (v4f32 QPR:$src))), (v2i64 QPR:$src)>;
+  def : Pat<(v2i64 (bitconvert (v4i32 QPR:$src))), (v2i64 QPR:$src)>;
+  def : Pat<(v2i64 (bitconvert (v8f16 QPR:$src))), (v2i64 QPR:$src)>;
+  def : Pat<(v2i64 (bitconvert (v8i16 QPR:$src))), (v2i64 QPR:$src)>;
+  def : Pat<(v2i64 (bitconvert (v16i8 QPR:$src))), (v2i64 QPR:$src)>;
+
+  def : Pat<(v4f32 (bitconvert (v2f64 QPR:$src))), (v4f32 QPR:$src)>;
+  def : Pat<(v4f32 (bitconvert (v2i64 QPR:$src))), (v4f32 QPR:$src)>;
+  def : Pat<(v4f32 (bitconvert (v8f16 QPR:$src))), (v4f32 QPR:$src)>;
+  def : Pat<(v4f32 (bitconvert (v8i16 QPR:$src))), (v4f32 QPR:$src)>;
+  def : Pat<(v4f32 (bitconvert (v16i8 QPR:$src))), (v4f32 QPR:$src)>;
+
+  def : Pat<(v4i32 (bitconvert (v2f64 QPR:$src))), (v4i32 QPR:$src)>;
+  def : Pat<(v4i32 (bitconvert (v2i64 QPR:$src))), (v4i32 QPR:$src)>;
+  def : Pat<(v4i32 (bitconvert (v8f16 QPR:$src))), (v4i32 QPR:$src)>;
+  def : Pat<(v4i32 (bitconvert (v8i16 QPR:$src))), (v4i32 QPR:$src)>;
+  def : Pat<(v4i32 (bitconvert (v16i8 QPR:$src))), (v4i32 QPR:$src)>;
+
+  def : Pat<(v8f16 (bitconvert (v2f64 QPR:$src))), (v8f16 QPR:$src)>;
+  def : Pat<(v8f16 (bitconvert (v2i64 QPR:$src))), (v8f16 QPR:$src)>;
+  def : Pat<(v8f16 (bitconvert (v4f32 QPR:$src))), (v8f16 QPR:$src)>;
+  def : Pat<(v8f16 (bitconvert (v4i32 QPR:$src))), (v8f16 QPR:$src)>;
+  def : Pat<(v8f16 (bitconvert (v16i8 QPR:$src))), (v8f16 QPR:$src)>;
+
+  def : Pat<(v8i16 (bitconvert (v2f64 QPR:$src))), (v8i16 QPR:$src)>;
+  def : Pat<(v8i16 (bitconvert (v2i64 QPR:$src))), (v8i16 QPR:$src)>;
+  def : Pat<(v8i16 (bitconvert (v4f32 QPR:$src))), (v8i16 QPR:$src)>;
+  def : Pat<(v8i16 (bitconvert (v4i32 QPR:$src))), (v8i16 QPR:$src)>;
+  def : Pat<(v8i16 (bitconvert (v16i8 QPR:$src))), (v8i16 QPR:$src)>;
+
+  def : Pat<(v16i8 (bitconvert (v2f64 QPR:$src))), (v16i8 QPR:$src)>;
+  def : Pat<(v16i8 (bitconvert (v2i64 QPR:$src))), (v16i8 QPR:$src)>;
+  def : Pat<(v16i8 (bitconvert (v4f32 QPR:$src))), (v16i8 QPR:$src)>;
+  def : Pat<(v16i8 (bitconvert (v4i32 QPR:$src))), (v16i8 QPR:$src)>;
+  def : Pat<(v16i8 (bitconvert (v8f16 QPR:$src))), (v16i8 QPR:$src)>;
+  def : Pat<(v16i8 (bitconvert (v8i16 QPR:$src))), (v16i8 QPR:$src)>;
+}

llvm/trunk/lib/Target/ARM/ARMRegisterInfo.td

def TuplesOE2D : RegisterTuples<[dsub_0, dsub_1],
                                  (decimate (shl DPR, 2), 2)]>;
 
 // Register class representing a pair of consecutive D registers.
 // Use the Q registers for the even-odd pairs.
 def DPair : RegisterClass<"ARM", [v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
                           128, (interleave QPR, TuplesOE2D)> {
   // Allocate starting at non-VFP2 registers D16-D31 first.
   // Prefer even-odd pairs as they are easier to copy.
-  let AltOrders = [(add (rotl QPR, 8), (rotl DPair, 16))];
-  let AltOrderSelect = [{ return 1; }];
+  let AltOrders = [(add (rotl QPR, 8),  (rotl DPair, 16)),
+                   (add (trunc QPR, 8), (trunc DPair, 16))];
+  let AltOrderSelect = [{
+    return 1 + MF.getSubtarget<ARMSubtarget>().hasMVEIntegerOps();
+  }];
 }
 
 // Pseudo-registers representing even-odd pairs of GPRs from R1 to R13/SP.
 // These are needed by instructions (e.g. ldrexd/strexd) requiring even-odd GPRs.
 def Tuples2R : RegisterTuples<[gsub_0, gsub_1],
                               [(add R0, R2, R4, R6, R8, R10, R12),
                                (add R1, R3, R5, R7, R9, R11, SP)]>;
 

llvm/trunk/test/CodeGen/ARM/fast-isel-call.ll

 ; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-apple-ios | FileCheck %s --check-prefix=ARM
 ; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-linux-gnueabi | FileCheck %s --check-prefix=ARM
 ; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=thumbv7-apple-ios | FileCheck %s --check-prefix=THUMB
 ; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-apple-ios -mattr=+long-calls | FileCheck %s --check-prefix=ARM-LONG --check-prefix=ARM-LONG-MACHO
 ; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-linux-gnueabi -mattr=+long-calls | FileCheck %s --check-prefix=ARM-LONG --check-prefix=ARM-LONG-ELF
 ; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=thumbv7-apple-ios -mattr=+long-calls | FileCheck %s --check-prefix=THUMB-LONG
-; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-apple-ios -mattr=-vfp2d16sp | FileCheck %s --check-prefix=ARM-NOVFP
-; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-linux-gnueabi -mattr=-vfp2d16sp | FileCheck %s --check-prefix=ARM-NOVFP
-; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=thumbv7-apple-ios -mattr=-vfp2d16sp | FileCheck %s --check-prefix=THUMB-NOVFP
+; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-apple-ios -mattr=-fpregs | FileCheck %s --check-prefix=ARM-NOVFP
+; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=armv7-linux-gnueabi -mattr=-fpregs | FileCheck %s --check-prefix=ARM-NOVFP
+; RUN: llc -fast-isel-sink-local-values < %s -O0 -verify-machineinstrs -fast-isel-abort=1 -relocation-model=dynamic-no-pic -mtriple=thumbv7-apple-ios -mattr=-fpregs | FileCheck %s --check-prefix=THUMB-NOVFP
 
 ; Note that some of these tests assume that relocations are either
 ; movw/movt or constant pool loads. Different platforms will select
 ; different approaches.
 
 define i32 @t0(i1 zeroext %a) nounwind {
   %1 = zext i1 %a to i32
   ret i32 %1

llvm/trunk/test/CodeGen/ARM/fp16-promote.ll

 ; RUN: llc -asm-verbose=false < %s -mattr=+vfp3,+fp16 | FileCheck -allow-deprecated-dag-overlap %s -check-prefix=CHECK-FP16  --check-prefix=CHECK-VFP -check-prefix=CHECK-ALL
 ; RUN: llc -asm-verbose=false < %s | FileCheck -allow-deprecated-dag-overlap %s -check-prefix=CHECK-LIBCALL --check-prefix=CHECK-VFP -check-prefix=CHECK-ALL --check-prefix=CHECK-LIBCALL-VFP
-; RUN: llc -asm-verbose=false < %s -mattr=-vfp2d16sp | FileCheck -allow-deprecated-dag-overlap %s --check-prefix=CHECK-LIBCALL -check-prefix=CHECK-NOVFP -check-prefix=CHECK-ALL
+; RUN: llc -asm-verbose=false < %s -mattr=-fpregs | FileCheck -allow-deprecated-dag-overlap %s --check-prefix=CHECK-LIBCALL -check-prefix=CHECK-NOVFP -check-prefix=CHECK-ALL
 
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-n32"
 target triple = "armv7---eabihf"
 
 ; CHECK-ALL-LABEL: test_fadd:
 ; CHECK-FP16: vcvtb.f32.f16
 ; CHECK-FP16: vcvtb.f32.f16
 ; CHECK-LIBCALL: bl __aeabi_h2f

llvm/trunk/test/CodeGen/ARM/no-fpu.ll

-; RUN: llc < %s -mtriple=armv7-none-gnueabi -mattr=-neon,-vfp2d16sp | FileCheck --check-prefix=NONEON-NOVFP %s
+; RUN: llc < %s -mtriple=armv7-none-gnueabi -mattr=-neon,-fpregs | FileCheck --check-prefix=NONEON-NOVFP %s
 ; RUN: llc < %s -mtriple=armv7-none-gnueabi -mattr=-neon | FileCheck --check-prefix=NONEON %s
-; RUN: llc < %s -mtriple=armv7-none-gnueabi -mattr=-vfp2d16sp | FileCheck --check-prefix=NOVFP %s
+; RUN: llc < %s -mtriple=armv7-none-gnueabi -mattr=-fpregs | FileCheck --check-prefix=NOVFP %s
 ; RUN: llc < %s -mtriple=armv7-none-gnueabi -mattr=-neon,+vfp2 | FileCheck --check-prefix=NONEON-VFP %s
 
 ; Check no NEON instructions are selected when feature is disabled.
 define void @neonop(i64* nocapture readonly %a, i64* nocapture %b) #0 {
   %1 = bitcast i64* %a to <2 x i64>*
   %wide.load = load <2 x i64>, <2 x i64>* %1, align 8
   ; NONEON-NOVFP-NOT: vld1.64
   ; NONEON-NOT: vld1.64

llvm/trunk/test/CodeGen/Thumb2/mve-basic.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve.fp -o - %s | FileCheck %s
+; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve -o - %s | FileCheck %s
+
+define arm_aapcs_vfpcc <4 x i32> @vector_add_by_value(<4 x i32> %lhs, <4 x i32>%rhs) {
+; CHECK-LABEL: vector_add_by_value:
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    @APP
+; CHECK-NEXT:    vadd.i32 q0, q0, q1
+; CHECK-NEXT:    @NO_APP
+; CHECK-NEXT:    bx lr
+  %result = tail call <4 x i32> asm "vadd.i32 $0,$1,$2", "=t,t,t"(<4 x i32> %lhs, <4 x i32> %rhs)
+  ret <4 x i32> %result
+}
+
+define void @vector_add_by_reference(<4 x i32>* %resultp, <4 x i32>* %lhsp, <4 x i32>* %rhsp) {
+; CHECK-LABEL: vector_add_by_reference:
+; CHECK:       @ %bb.0:
+; CHECK-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-NEXT:    vldrw.u32 q1, [r2]
+; CHECK-NEXT:    @APP
+; CHECK-NEXT:    vadd.i32 q0, q0, q1
+; CHECK-NEXT:    @NO_APP
+; CHECK-NEXT:    vstrw.32 q0, [r0]
+; CHECK-NEXT:    bx lr
+  %lhs = load <4 x i32>, <4 x i32>* %lhsp, align 16
+  %rhs = load <4 x i32>, <4 x i32>* %rhsp, align 16
+  %result = tail call <4 x i32> asm "vadd.i32 $0,$1,$2", "=t,t,t"(<4 x i32> %lhs, <4 x i32> %rhs)
+  store <4 x i32> %result, <4 x i32>* %resultp, align 16
+  ret void
+}

llvm/trunk/test/CodeGen/Thumb2/mve-bitcasts.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp %s -o - | FileCheck %s
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_i64_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_i64_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_i64_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_i64_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_i64_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_i64_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+define arm_aapcs_vfpcc <2 x i64> @bitcast_i64_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_i64_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <2 x i64>
+  ret <2 x i64> %r
+}
+
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_i32_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_i32_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_i32_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_i32_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_i32_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_i32_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+define arm_aapcs_vfpcc <4 x i32> @bitcast_i32_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_i32_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <4 x i32>
+  ret <4 x i32> %r
+}
+
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_i16_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_i16_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_i16_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_i16_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_i16_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_i16_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+define arm_aapcs_vfpcc <8 x i16> @bitcast_i16_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_i16_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <8 x i16>
+  ret <8 x i16> %r
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_i8_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_i8_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_i8_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_i8_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_i8_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_i8_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+define arm_aapcs_vfpcc <16 x i8> @bitcast_i8_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_i8_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <16 x i8>
+  ret <16 x i8> %r
+}
+
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_f64_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_f64_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_f64_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_f64_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_f64_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_f64_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+define arm_aapcs_vfpcc <2 x double> @bitcast_f64_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_f64_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <2 x double>
+  ret <2 x double> %r
+}
+
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_f32_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_f32_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_f32_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_f32_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_f32_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_f32_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+define arm_aapcs_vfpcc <4 x float> @bitcast_f32_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_f32_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <4 x float>
+  ret <4 x float> %r
+}
+
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_i64(<2 x i64> %src) {
+; CHECK-LABEL: bitcast_f16_i64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x i64> %src to <8 x half>
+  ret <8 x half> %r
+}
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_i32(<4 x i32> %src) {
+; CHECK-LABEL: bitcast_f16_i32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x i32> %src to <8 x half>
+  ret <8 x half> %r
+}
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_i16(<8 x i16> %src) {
+; CHECK-LABEL: bitcast_f16_i16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x i16> %src to <8 x half>
+  ret <8 x half> %r
+}
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_i8(<16 x i8> %src) {
+; CHECK-LABEL: bitcast_f16_i8:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <16 x i8> %src to <8 x half>
+  ret <8 x half> %r
+}
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_f64(<2 x double> %src) {
+; CHECK-LABEL: bitcast_f16_f64:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <2 x double> %src to <8 x half>
+  ret <8 x half> %r
+}
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_f32(<4 x float> %src) {
+; CHECK-LABEL: bitcast_f16_f32:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <4 x float> %src to <8 x half>
+  ret <8 x half> %r
+}
+
+define arm_aapcs_vfpcc <8 x half> @bitcast_f16_f16(<8 x half> %src) {
+; CHECK-LABEL: bitcast_f16_f16:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    bx lr
+entry:
+  %r = bitcast <8 x half> %src to <8 x half>
+  ret <8 x half> %r
+}

llvm/trunk/test/Transforms/HardwareLoops/ARM/calls.ll

 ; CHECK-LABEL: test_masked_i32
 ; CHECK-NOT: call void @llvm.set.loop.iterations
 ; CHECK-MVEFP: call void @llvm.set.loop.iterations
 ; CHECK-MVE: call void @llvm.set.loop.iterations.i32(i32 100)
 ; CHECK-MVE: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
 ; CHECK-MVE: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
 ; CHECK-MVE: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-MVE: br i1 [[CMP]], label %loop, label %exit
-define void @test_masked_i32(<4 x i1> %mask, <4 x i32>* %a, <4 x i32>* %b, <4 x i32>* %c, <4 x i32> %passthru) {
+define arm_aapcs_vfpcc void @test_masked_i32(<4 x i1> %mask, <4 x i32>* %a, <4 x i32>* %b, <4 x i32>* %c, <4 x i32> %passthru) {
 entry:
   br label %loop
 loop:
   %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
   %addr.a = getelementptr <4 x i32>, <4 x i32>* %a, i32 %count
   %addr.b = getelementptr <4 x i32>, <4 x i32>* %b, i32 %count
   %addr.c = getelementptr <4 x i32>, <4 x i32>* %c, i32 %count
   %load.a = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %addr.a, i32 4, <4 x i1> %mask, <4 x i32> %passthru)

 ; CHECK-LABEL: test_masked_f32
 ; CHECK-NOT: call void @llvm.set.loop.iterations
 ; CHECK-MVEFP: call void @llvm.set.loop.iterations
 ; CHECK-MVE: call void @llvm.set.loop.iterations.i32(i32 100)
 ; CHECK-MVE: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
 ; CHECK-MVE: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
 ; CHECK-MVE: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-MVE: br i1 [[CMP]], label %loop, label %exit
-define void @test_masked_f32(<4 x i1> %mask, <4 x float>* %a, <4 x float>* %b, <4 x float>* %c, <4 x float> %passthru) {
+define arm_aapcs_vfpcc void @test_masked_f32(<4 x i1> %mask, <4 x float>* %a, <4 x float>* %b, <4 x float>* %c, <4 x float> %passthru) {
 entry:
   br label %loop
 loop:
   %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
   %addr.a = getelementptr <4 x float>, <4 x float>* %a, i32 %count
   %addr.b = getelementptr <4 x float>, <4 x float>* %b, i32 %count
   %addr.c = getelementptr <4 x float>, <4 x float>* %c, i32 %count
   %load.a = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %addr.a, i32 4, <4 x i1> %mask, <4 x float> %passthru)

 ; CHECK-LABEL: test_gather_scatter
 ; CHECK-NOT: call void @llvm.set.loop.iterations
 ; CHECK-MVEFP: call void @llvm.set.loop.iterations
 ; CHECK-MVE: call void @llvm.set.loop.iterations.i32(i32 100)
 ; CHECK-MVE: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
 ; CHECK-MVE: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
 ; CHECK-MVE: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-MVE: br i1 [[CMP]], label %loop, label %exit
-define void @test_gather_scatter(<4 x i1> %mask, <4 x float*> %a, <4 x float*> %b, <4 x float*> %c, <4 x float> %passthru) {
+define arm_aapcs_vfpcc void @test_gather_scatter(<4 x i1> %mask, <4 x float*> %a, <4 x float*> %b, <4 x float*> %c, <4 x float> %passthru) {
 entry:
   br label %loop
 loop:
   %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
   %load.a = call <4 x float> @llvm.masked.gather.v4f32.p0v4f32(<4 x float*> %a, i32 4, <4 x i1> %mask, <4 x float> %passthru)
   %load.b = call <4 x float> @llvm.masked.gather.v4f32.p0v4f32(<4 x float*> %b, i32 4, <4 x i1> %mask, <4 x float> %passthru)
   %res = fadd <4 x float> %load.a, %load.b
   call void @llvm.masked.scatter.v4f32.p0v4f32(<4 x float> %res, <4 x float*> %c, i32 4, <4 x i1> %mask)