[ARM] Supporting lowering of half-precision FP arguments and returns in AArch32's backend

[pratlucas]

Summary

Half-precision floating point arguments and returns are currently
promoted to either float or int32 in clang's CodeGen and there's
no existing support for the lowering of half arguments and returns
from IR in AArch32's backend.

Such frontend coercions, implemented as coercion through memory
in clang, can cause a series of issues in argument lowering, as causing
arguments to be stored on the wrong bits on big-endian architectures
and incurring in missing overflow detections in the return of certain
functions.

This patch introduces the handling of half-precision arguments and returns in
the backend using the actual "half" type on the IR. Using the "half"
type the backend is able to properly enforce the AAPCS' directions for
those arguments, making sure they are stored on the proper bits of the
registers and performing the necessary floating point convertions.

[pratlucas]

Ping.

[lattner]

I'm not a qualified reviewer for this at this point.

[rjmccall]

Ugh. I'd hate to introduce yet another weird little tweak to ABIArgInfo that's used on exactly one target. For 16-bit composite types, we seem to coerce to a type like [1 x i32]; would that be okay here?

You don't have a test that checks that you get the IR you want.

[pratlucas]

Hi @rjmccall,
I agree those kind of tweaks do not look good. The issue here, though, is that argument coercion currently ignores the target's endian information when performing coercion through memory.
This happens for any type that requires memory coercion, so unfortunately using [1 x i32] does not do the trick.
Let me know if you have any other sugestions for handling this, I'd be glad to avoid the ABIArgInfo approach.

[rjmccall]

In D75169#1926545, @pratlucas wrote:

Hi @rjmccall,
I agree those kind of tweaks do not look good. The issue here, though, is that argument coercion currently ignores the target's endian information when performing coercion through memory.
This happens for any type that requires memory coercion, so unfortunately using [1 x i32] does not do the trick.

Oh, wait, AAPCS wants half values to be passed in the *least* significant bits of a GPR, even on big-endian machines? That's certainly more convenient, but it's a weird inconsistency with the otherwise iron rule of the calling convention, which that it's exactly as if you laid all of the arguments out in memory and then popped the first four 32-bit values off. We're talking about a calling convention here that literally skips registers in order to "align" arguments.

Can we not just coerce to i16? Will LLVM not pass an i16 in the least-significant bits of a register?

[pratlucas]

Oh, wait, AAPCS wants half values to be passed in the *least* significant bits of a GPR, even on big-endian machines? That's certainly more convenient, but it's a weird inconsistency with the otherwise iron rule of the calling convention, which that it's exactly as if you laid all of the arguments out in memory and then popped the first four 32-bit values off. We're talking about a calling convention here that literally skips registers in order to "align" arguments.

Can we not just coerce to i16? Will LLVM not pass an i16 in the least-significant bits of a register?

Yes, AAPCS specifies that they should go into the LSBs:

B.2 [...] If the argument is a Half-precision Floating Point Type its size is set to 4 bytes as if it had been copied to the least significant bits of a 32-bit register and the remaining bits filled with unspecified values.

Coercing to i16 solves it for the general case, when the argumetns are going into GPRs, but is not suficient when those arguments are required to go into FP registers - e.g. -mfloat-abi=hard.

[rjmccall]

In D75169#1928696, @pratlucas wrote:

Oh, wait, AAPCS wants half values to be passed in the *least* significant bits of a GPR, even on big-endian machines? That's certainly more convenient, but it's a weird inconsistency with the otherwise iron rule of the calling convention, which that it's exactly as if you laid all of the arguments out in memory and then popped the first four 32-bit values off. We're talking about a calling convention here that literally skips registers in order to "align" arguments.

Can we not just coerce to i16? Will LLVM not pass an i16 in the least-significant bits of a register?

Yes, AAPCS specifies that they should go into the LSBs:

B.2 [...] If the argument is a Half-precision Floating Point Type its size is set to 4 bytes as if it had been copied to the least significant bits of a 32-bit register and the remaining bits filled with unspecified values.

Coercing to i16 solves it for the general case, when the argumetns are going into GPRs, but is not suficient when those arguments are required to go into FP registers - e.g. -mfloat-abi=hard.

Why not just make half as an argument do the right thing for that case?

[rudkx]

I did some refactoring here years ago but I'm not that familiar with the ABIs or the handling in clang.

[pratlucas]

Why not just make half as an argument do the right thing for that case?

That would be the ideal approach, but currently there's a limitation on the backend's calling convention lowering that gets in the way.
The lowering of calls in SelectionDAGBuilder includes a target-independent step that is responsible for spliting or promoting each argument into "legal registers" and takes place before the targets' calling convention lowering.
As f16 is not a legal type on many of the AAPCS_VFP targets, it gets promoted to f32 before the target's lowering code has a chance to define how to handle it.
Ideally, this stpe should only take place if lowering calling conventions after type legalization - there's a FIXME there already capturing that -, but that would involve a major rewriting that would impact multiple targets.
Inserting a hacky target-dependent fix in this step also didn't look very good.
Do you see other alternatives for handling it? If not, which approach would you suggest?

[pratlucas]

Ping.

[pratlucas]

Ping.

[pratlucas]

Ping.

[dnsampaio]

In D75169#1952159, @pratlucas wrote:

Why not just make half as an argument do the right thing for that case?

That would be the ideal approach, but currently there's a limitation on the backend's calling convention lowering that gets in the way.
The lowering of calls in SelectionDAGBuilder includes a target-independent step that is responsible for spliting or promoting each argument into "legal registers" and takes place before the targets' calling convention lowering.
As f16 is not a legal type on many of the AAPCS_VFP targets, it gets promoted to f32 before the target's lowering code has a chance to define how to handle it.
Ideally, this stpe should only take place if lowering calling conventions after type legalization - there's a FIXME there already capturing that -, but that would involve a major rewriting that would impact multiple targets.
Inserting a hacky target-dependent fix in this step also didn't look very good.
Do you see other alternatives for handling it? If not, which approach would you suggest?

Would it be possible to pass a half argument and fix-it-up at CodeGenPrepare?

[pratlucas]

Re-writing the handling of fp16 arguments, moving their lowering to be performed
in the backend.

This removes the necesity of coercing the arguments in clang and allows the
backend to have full control over how those arguments should be lowered according
to each calling convention.

To achieve this, a couple of new hooks had to be introduced to allow the target
to have a say on how an argument value should be split or combined together into
a legal type.

There are still two tests failing - one due to the lack of support for the fp16
handling in the GlobalISel flow and one due to an issue lowering the result of
an fp_round library call - but I would like to start getting feedback while
I work on the fix for those.

[pratlucas]

Splitting the patch into two parts: one for introducing the half-precision
handling into AArch32's backend and one for removing the existing coercion
of those arguments from Clang.

[pratlucas]

Formatting patch.

[dmgreen]

Love it. This will help optimise these more too. We can currently get into places where we can't prove only the bottom bits of a returned value are demanded so can't remove unnecessary vmovs. Test look good too from my understanding of how fp16 calling should work.

I unfortunately can't really claim to be an expert on calling conventions code or if these new functions are needed. Hopefully someone I added can take a look, and if not I can always go find out how it works.

[efriedma]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

524

I'm not sure I understand why the standard getCopyFromParts/getCopyToParts codepath doesn't work. Is the issue that it uses FP_ROUND/FP_EXTEND to promote from f16 to f32?

llvm/lib/Target/ARM/ARMCallingConv.cpp

294

It isn't obvious to me why you need this; can you not use CCBitConvertToType/CCAssignToReg?

[pratlucas]

Clean-ups + fixing failure in CodeGen/ARM/half.ll test.

[pratlucas]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

524

Yes, the issue is the usage of FP_ROUND/FP_EXTEND indeed. Those cause the argument to be converted from f16 into f32 - with a vcvtb.f16.f32 for instance - instead of simply being placed the value in the LSBs as required by the AAPCS.

llvm/lib/Target/ARM/ARMCallingConv.cpp

294

For hard floats, using CCAssingToReg would indeed work well in the majority of the scenarios, but would get in the way of the CMSE handling from D81428. Using the f16 loc type causes the clearing of the top 16 bits to be optimized out in the DAG.
Also, the AAPCS expects the argument sized to be extended to 4 bytes, so using the f32 loc type attends to that rule.

For soft floats, on the other hand, simply convering it to i32 causes the code on ARMISel lowering to be quite cumbersome. The loc info becomes either CCValAssign::BCvt (f16 -> f32 - >i32) or CCValAssign::AExt ( f16 -> i16 -> i32), so checking for when we need to handle things differently for f16 becomes less clear.
Using this flow we have the isCustom flag assigned and can have a more explicit handling of this.

[efriedma]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

524

That makes sense.

It feels a little weird to have a TLI method to do the splitting, as opposed to adding an extra check to the shared codepath, but I guess this way is more flexible if someone else needs a similar change in the future.

One other thing to consider is that we could make f16 a "legal" type for all ARM subtargets with floating-point registers, regardless of whether the target actually has native f16 arithmetic instructions. We do this on AArch64. That would reduce the number of different ways to handle f16 values, and I think this change would be unnecessary.

916

(Accidental change?)

[pratlucas]

Fixing failure on CodeGen/ARM/GlobalISel/arm-unsupported.ll and making clang-format happy.

[SjoerdMeijer]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

524

One other thing to consider is that we could make f16 a "legal" type for all ARM subtargets with floating-point registers, regardless of whether the target actually has native f16 arithmetic instructions. We do this on AArch64.

I am partly guilty here and there when I added _Float16 and v8.2 FP16 support.
When I did this, life was easy for AArch64, because it doesn't have a soft float ABI support and it has or hasn't got FP16 support, and so everything is straightforward. Life became an awful lot less pleasant for AArch32, because of the hard/soft float and different FP16 support, i.e. there are early FPU versions with limited fp16 support for the storage only type (some conversion), and from v8.2 and up the native fp16 instruction. It's been a few years now so can't remember exactly, which is a bit unhelpful, but somewhere here for these corner cases I got into trouble by treating f16 as a legal type.

But in case you're interested / this might be useful, I think this is the mail that I wrote to the llvm dev list when I got into trouble here (which I actually need to reread too for details):

http://lists.llvm.org/pipermail/llvm-dev/2018-January/120537.html

and as referred in that mail, earlier revisions of this:

https://reviews.llvm.org/D38315

might have taken exactly that approach. Long story short, I am not saying we shouldn't do it, just pointing out some background info. And since we're all a few years wiser now since that happened, perhaps we should try again ;-)

[dnsampaio]

Perhaps we could move to making half a valid type for the arm back-end as follow up patches. Allowing half as argument through the IR is already a step to that direction.
IMO this patch is already quite big and it excels in fixing the bugs it proposed.

[pratlucas]

Hi @efriedma,

From @SjoerdMeijer's comment and the links he pointed to, it seems to me that making f16 types legal for all ARM subtargets would be a major undertaking and far from trivial to implement. It's also not clear to me how significant would be the returns of this effort.
My feeling is that we could proceed with the current approach and discuss the possbility of making f16 legal in a separate follow up effort, as mentioned by @dnsampaio.

What's your view on this?

[ostannard]

I don't think it makes sense to make f16 legal for targets which don't have any arithmetic operations on it, since that would be contrary to the definition of "legal". I'd also expect doing so to introduce a lot more complexity than this patch.

I think all of the previous comments have been addressed now, and this LGTM.

[plotfi]

@pratlucas @ostannard @rjmccall

I have noticed this change break on the building of the Swift standard library in downstream apple/swift/master-next. I reduced the test case and found that the chain of bitcasts/extends in ARMTargetLowering::splitValueIntoRegisterParts ends up not being legal and causing an assert in SelectionDag.cpp when compiling the following with llc:

target datalayout = "e-m:o-p:32:32-Fi8-f64:32:64-v64:32:64-v128:32:128-a:0:32-n32-S32"
target triple = "thumbv7s-apple-ios7.0.0"

define fastcc { <8 x half>, <8 x half> } @f() {
  ret { <8 x half>, <8 x half> } zeroinitializer
}

This was originally swiftcc, but I changed it to fastcc so it would compile with upstream llvm.org/main while also having a calling convention that would satisfy the conditional here: https://github.com/llvm/llvm-project/blob/bfec030e69afc73b29aa1b66902ae802a448fc19/llvm/lib/Target/ARM/ARMISelLowering.cpp#L4155

The assert is as follows:

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp:1307: llvm::SDValue llvm::SelectionDAG::getConstant(const llvm::ConstantInt &, const llvm::SDLoc &, llvm::EVT, bool, bool): Assertion `Elt->getBitWidth() == EltVT.getSizeInBits() && "APInt size does not match type size!"' failed.

The DAG nodes generated before:

t5: f16 = extract_vector_elt t3, Constant:i32<0>
  t3: v8f16,v8f16 = merge_values t2, t2
    t2: v8f16 = BUILD_VECTOR ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>

And After:

t22: f32 = bitcast t21
  t21: i32 = any_extend t20
    t20: i16 = bitcast t5
      t5: f16 = extract_vector_elt t3, Constant:i32<0>
        t3: v8f16,v8f16 = merge_values t2, t2
          t2: v8f16 = BUILD_VECTOR ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>, ConstantFP:f16<APFloat(0)>

I suspect here there could be a the EltVT in the assert was expecting a 16-bit size, but is now getting 32.

I hope this is helpful.

[plotfi]

llvm/lib/Target/ARM/ARMISelLowering.cpp

4106

I'm not well versed in arm CCs, but I suspect you might want this to check if the CC is arm_apcscc, arm_aapcscc, or arm_aapcs_vfpcc instead of checking merely if it has a CC.

[plotfi]

@pratlucas @ostannard @rjmccall

I've posted a diff D82443 to address what I think could be a potential fix for the assert I was seeing on the provided reduced IR.

[hans]

Lucas, this seems to have casued https://bugs.llvm.org/show_bug.cgi?id=47001. Can you take a look? (I would cc you on the bug, but I couldn't find your email in bugzilla.)

[pratlucas]

Hi @hans , I'll have a look at it!

llvm/include/llvm/CodeGen/TargetLowering.h

SDValue getNegatedExpression(SDValue Op, SelectionDAG &DAG, bool LegalOps,
     return getNegatedExpression(Op, DAG, LegalOps, OptForSize, Cost, Depth);
   }
 
   //===--------------------------------------------------------------------===//
   // Lowering methods - These methods must be implemented by targets so that
   // the SelectionDAGBuilder code knows how to lower these.
   //
 
+  /// Target-specific splitting of values into parts that fit a register
+  /// storing a legal type
+  virtual bool splitValueIntoRegisterParts(SelectionDAG &DAG, const SDLoc &DL,
+                                           SDValue Val, SDValue *Parts,
+                                           unsigned NumParts, MVT PartVT,
+                                           Optional<CallingConv::ID> CC) const {
+    return false;
+  }
+
+  /// Target-specific combining of register parts into its original value
+  virtual SDValue
+  joinRegisterPartsIntoValue(SelectionDAG &DAG, const SDLoc &DL,
+                             const SDValue *Parts, unsigned NumParts,
+                             MVT PartVT, EVT ValueVT,
+                             Optional<CallingConv::ID> CC) const {
+    return SDValue();
+  }
+
   /// This hook must be implemented to lower the incoming (formal) arguments,
   /// described by the Ins array, into the specified DAG. The implementation
   /// should fill in the InVals array with legal-type argument values, and
   /// return the resulting token chain value.
   virtual SDValue LowerFormalArguments(
       SDValue /*Chain*/, CallingConv::ID /*CallConv*/, bool /*isVarArg*/,
       const SmallVectorImpl<ISD::InputArg> & /*Ins*/, const SDLoc & /*dl*/,
       SelectionDAG & /*DAG*/, SmallVectorImpl<SDValue> & /*InVals*/) const {

llvm/lib/CodeGen/GlobalISel/CallLowering.cpp

bool CallLowering::handleAssignments(CCState &CCInfo,
 
   for (unsigned i = 0, e = Args.size(), j = 0; i != e; ++i, ++j) {
     assert(j < ArgLocs.size() && "Skipped too many arg locs");
 
     CCValAssign &VA = ArgLocs[j];
     assert(VA.getValNo() == i && "Location doesn't correspond to current arg");
 
     if (VA.needsCustom()) {
-      j += Handler.assignCustomValue(Args[i], makeArrayRef(ArgLocs).slice(j));
+      unsigned NumArgRegs =
+          Handler.assignCustomValue(Args[i], makeArrayRef(ArgLocs).slice(j));
+      if (!NumArgRegs)
+        return false;
+      j += NumArgRegs;
       continue;
     }
 
     // FIXME: Pack registers if we have more than one.
     Register ArgReg = Args[i].Regs[0];
 
     MVT OrigVT = MVT::getVT(Args[i].Ty);
     MVT VAVT = VA.getValVT();

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

 /// larger than ValueVT then AssertOp can be used to specify whether the extra
 /// bits are known to be zero (ISD::AssertZext) or sign extended from ValueVT
 /// (ISD::AssertSext).
 static SDValue getCopyFromParts(SelectionDAG &DAG, const SDLoc &DL,
                                 const SDValue *Parts, unsigned NumParts,
                                 MVT PartVT, EVT ValueVT, const Value *V,
                                 Optional<CallingConv::ID> CC = None,
                                 Optional<ISD::NodeType> AssertOp = None) {
+  // Let the target assemble the parts if it wants to
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  if (SDValue Val = TLI.joinRegisterPartsIntoValue(DAG, DL, Parts, NumParts,
+                                                   PartVT, ValueVT, CC))
+    return Val;
+
   if (ValueVT.isVector())
     return getCopyFromPartsVector(DAG, DL, Parts, NumParts, PartVT, ValueVT, V,
                                   CC);
 
   assert(NumParts > 0 && "No parts to assemble!");
-  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   SDValue Val = Parts[0];
 
   if (NumParts > 1) {
     // Assemble the value from multiple parts.
     if (ValueVT.isInteger()) {
       unsigned PartBits = PartVT.getSizeInBits();
       unsigned ValueBits = ValueVT.getSizeInBits();
 

 /// getCopyToParts - Create a series of nodes that contain the specified value
 /// split into legal parts.  If the parts contain more bits than Val, then, for
 /// integers, ExtendKind can be used to specify how to generate the extra bits.
 static void getCopyToParts(SelectionDAG &DAG, const SDLoc &DL, SDValue Val,
                            SDValue *Parts, unsigned NumParts, MVT PartVT,
                            const Value *V,
                            Optional<CallingConv::ID> CallConv = None,
                            ISD::NodeType ExtendKind = ISD::ANY_EXTEND) {
+  // Let the target split the parts if it wants to
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  if (TLI.splitValueIntoRegisterParts(DAG, DL, Val, Parts, NumParts, PartVT,
+                                      CallConv))
+    return;
   EVT ValueVT = Val.getValueType();
 
   // Handle the vector case separately.
   if (ValueVT.isVector())
     return getCopyToPartsVector(DAG, DL, Val, Parts, NumParts, PartVT, V,
                                 CallConv);
 
   unsigned PartBits = PartVT.getSizeInBits();
for (unsigned Value = 0, Part = 0, e = ValueVTs.size(); Value != e; ++Value) {
 
     MVT RegisterVT = isABIMangled() ? TLI.getRegisterTypeForCallingConv(
                                           *DAG.getContext(),
                                           CallConv.getValue(), RegVTs[Value])
                                     : RegVTs[Value];
 
     if (ExtendKind == ISD::ANY_EXTEND && TLI.isZExtFree(Val, RegisterVT))
       ExtendKind = ISD::ZERO_EXTEND;
 
     getCopyToParts(DAG, dl, Val.getValue(Val.getResNo() + Value), &Parts[Part],
                    NumParts, RegisterVT, V, CallConv, ExtendKind);
     Part += NumParts;
   }
 
   // Copy the parts into the registers.
   SmallVector<SDValue, 8> Chains(NumRegs);
   for (unsigned i = 0; i != NumRegs; ++i) {

llvm/lib/Target/ARM/ARMCallLowering.cpp

struct OutgoingValueHandler : public CallLowering::ValueHandler {
   }
 
   unsigned assignCustomValue(const CallLowering::ArgInfo &Arg,
                              ArrayRef<CCValAssign> VAs) override {
     assert(Arg.Regs.size() == 1 && "Can't handle multple regs yet");
 
     CCValAssign VA = VAs[0];
     assert(VA.needsCustom() && "Value doesn't need custom handling");
-    assert(VA.getValVT() == MVT::f64 && "Unsupported type");
+
+    // Custom lowering for other types, such as f16, is currently not supported
+    if (VA.getValVT() != MVT::f64)
+      return 0;
 
     CCValAssign NextVA = VAs[1];
     assert(NextVA.needsCustom() && "Value doesn't need custom handling");
     assert(NextVA.getValVT() == MVT::f64 && "Unsupported type");
 
     assert(VA.getValNo() == NextVA.getValNo() &&
            "Values belong to different arguments");
 
struct IncomingValueHandler : public CallLowering::ValueHandler {
   }
 
   unsigned assignCustomValue(const ARMCallLowering::ArgInfo &Arg,
                              ArrayRef<CCValAssign> VAs) override {
     assert(Arg.Regs.size() == 1 && "Can't handle multple regs yet");
 
     CCValAssign VA = VAs[0];
     assert(VA.needsCustom() && "Value doesn't need custom handling");
-    assert(VA.getValVT() == MVT::f64 && "Unsupported type");
+
+    // Custom lowering for other types, such as f16, is currently not supported
+    if (VA.getValVT() != MVT::f64)
+      return 0;
 
     CCValAssign NextVA = VAs[1];
     assert(NextVA.needsCustom() && "Value doesn't need custom handling");
     assert(NextVA.getValVT() == MVT::f64 && "Unsupported type");
 
     assert(VA.getValNo() == NextVA.getValNo() &&
            "Values belong to different arguments");
 

llvm/lib/Target/ARM/ARMCallingConv.cpp

static bool CC_ARM_AAPCS_Custom_Aggregate(unsigned ValNo, MVT ValVT,
 
   // All pending members have now been allocated
   PendingMembers.clear();
 
   // This will be allocated by the last member of the aggregate
   return true;
 }
 
+static bool CustomAssignInRegList(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                  CCValAssign::LocInfo LocInfo, CCState &State,
+                                  ArrayRef<MCPhysReg> RegList) {
+  unsigned Reg = State.AllocateReg(RegList);
+  if (Reg) {
+    State.addLoc(CCValAssign::getCustomReg(ValNo, ValVT, Reg, LocVT, LocInfo));
+    return true;
+  }
+  return false;
+}
+
+static bool CC_ARM_AAPCS_Custom_f16(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                    CCValAssign::LocInfo LocInfo,
+                                    ISD::ArgFlagsTy ArgFlags, CCState &State) {
+  // f16 arguments are extended to i32 and assigned to a register in [r0, r3]
+  return CustomAssignInRegList(ValNo, ValVT, MVT::i32, LocInfo, State,
+                               RRegList);
+}
+
+static bool CC_ARM_AAPCS_VFP_Custom_f16(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                        CCValAssign::LocInfo LocInfo,
+                                        ISD::ArgFlagsTy ArgFlags,
+                                        CCState &State) {
+  // f16 arguments are extended to f32 and assigned to a register in [s0, s15]
+  return CustomAssignInRegList(ValNo, ValVT, MVT::f32, LocInfo, State,
+                               SRegList);
+}
+
 // Include the table generated calling convention implementations.
 #include "ARMGenCallingConv.inc"

llvm/lib/Target/ARM/ARMCallingConv.td

def CC_ARM_AAPCS_Common : CallingConv<[
   // i64 is 8-aligned i32 here, so we may need to eat R1 as a pad register
   // (and the same is true for f64 if VFP is not enabled)
   CCIfType<[i32], CCIfAlign<"8", CCAssignToRegWithShadow<[R0, R2], [R0, R1]>>>,
   CCIfType<[i32], CCIf<"ArgFlags.getOrigAlign() != 8",
                        CCAssignToReg<[R0, R1, R2, R3]>>>,
 
   CCIfType<[i32], CCIfAlign<"8", CCAssignToStackWithShadow<4, 8, [R0, R1, R2, R3]>>>,
   CCIfType<[i32], CCAssignToStackWithShadow<4, 4, [R0, R1, R2, R3]>>,
-  CCIfType<[f32], CCAssignToStackWithShadow<4, 4, [Q0, Q1, Q2, Q3]>>,
+  CCIfType<[f16, f32], CCAssignToStackWithShadow<4, 4, [Q0, Q1, Q2, Q3]>>,
   CCIfType<[f64], CCAssignToStackWithShadow<8, 8, [Q0, Q1, Q2, Q3]>>,
   CCIfType<[v2f64], CCIfAlign<"16",
            CCAssignToStackWithShadow<16, 16, [Q0, Q1, Q2, Q3]>>>,
   CCIfType<[v2f64], CCAssignToStackWithShadow<16, 8, [Q0, Q1, Q2, Q3]>>
 ]>;
 
 def RetCC_ARM_AAPCS_Common : CallingConv<[
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
def CC_ARM_AAPCS : CallingConv<[
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is passed in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   CCIfType<[f64, v2f64], CCCustom<"CC_ARM_AAPCS_Custom_f64">>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
+  CCIfType<[f16], CCCustom<"CC_ARM_AAPCS_Custom_f16">>,
   CCDelegateTo<CC_ARM_AAPCS_Common>
 ]>;
 
 let Entry = 1 in
 def RetCC_ARM_AAPCS : CallingConv<[
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v4f16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v8f16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is returned in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   CCIfType<[f64, v2f64], CCCustom<"RetCC_ARM_AAPCS_Custom_f64">>,
   CCIfType<[f32], CCBitConvertToType<i32>>,
+  CCIfType<[f16], CCCustom<"CC_ARM_AAPCS_Custom_f16">>,
 
   CCDelegateTo<RetCC_ARM_AAPCS_Common>
 ]>;
 
 //===----------------------------------------------------------------------===//
 // ARM AAPCS-VFP (EABI) Calling Convention
 // Also used for FastCC (when VFP2 or later is available)
 //===----------------------------------------------------------------------===//
def CC_ARM_AAPCS_VFP : CallingConv<[
 
   // HFAs are passed in a contiguous block of registers, or on the stack
   CCIfConsecutiveRegs<CCCustom<"CC_ARM_AAPCS_Custom_Aggregate">>,
 
   CCIfType<[v2f64], CCAssignToReg<[Q0, Q1, Q2, Q3]>>,
   CCIfType<[f64], CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f32], CCAssignToReg<[S0, S1, S2, S3, S4, S5, S6, S7, S8,
                                  S9, S10, S11, S12, S13, S14, S15]>>,
+  CCIfType<[f16], CCCustom<"CC_ARM_AAPCS_VFP_Custom_f16">>,
   CCDelegateTo<CC_ARM_AAPCS_Common>
 ]>;
 
 let Entry = 1 in
 def RetCC_ARM_AAPCS_VFP : CallingConv<[
   // Handle all vector types as either f64 or v2f64.
   CCIfType<[v1i64, v2i32, v4i16, v4f16, v8i8, v2f32], CCBitConvertToType<f64>>,
   CCIfType<[v2i64, v4i32, v8i16, v8f16, v16i8, v4f32], CCBitConvertToType<v2f64>>,
 
   // Pass SwiftSelf in a callee saved register.
   CCIfSwiftSelf<CCIfType<[i32], CCAssignToReg<[R10]>>>,
 
   // A SwiftError is returned in R8.
   CCIfSwiftError<CCIfType<[i32], CCAssignToReg<[R8]>>>,
 
   CCIfType<[v2f64], CCAssignToReg<[Q0, Q1, Q2, Q3]>>,
   CCIfType<[f64], CCAssignToReg<[D0, D1, D2, D3, D4, D5, D6, D7]>>,
   CCIfType<[f32], CCAssignToReg<[S0, S1, S2, S3, S4, S5, S6, S7, S8,
-                                      S9, S10, S11, S12, S13, S14, S15]>>,
+  CCIfType<[f16], CCCustom<"CC_ARM_AAPCS_VFP_Custom_f16">>,
   CCDelegateTo<RetCC_ARM_AAPCS_Common>
 ]>;
 
 
 // Windows Control Flow Guard checks take a single argument (the target function
 // address) and have no return value.
 let Entry = 1 in
 def CC_ARM_Win32_CFGuard_Check : CallingConv<[

llvm/lib/Target/ARM/ARMISelLowering.h

bool supportSplitCSR(MachineFunction *MF) const override {
           MF->getFunction().hasFnAttribute(Attribute::NoUnwind);
     }
 
     void initializeSplitCSR(MachineBasicBlock *Entry) const override;
     void insertCopiesSplitCSR(
       MachineBasicBlock *Entry,
       const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
 
+    bool
+    splitValueIntoRegisterParts(SelectionDAG &DAG, const SDLoc &DL, SDValue Val,
+                                SDValue *Parts, unsigned NumParts, MVT PartVT,
+                                Optional<CallingConv::ID> CC) const override;
+
+    SDValue
+    joinRegisterPartsIntoValue(SelectionDAG &DAG, const SDLoc &DL,
+                               const SDValue *Parts, unsigned NumParts,
+                               MVT PartVT, EVT ValueVT,
+                               Optional<CallingConv::ID> CC) const override;
+
     SDValue
     LowerFormalArguments(SDValue Chain, CallingConv::ID CallConv, bool isVarArg,
                          const SmallVectorImpl<ISD::InputArg> &Ins,
                          const SDLoc &dl, SelectionDAG &DAG,
                          SmallVectorImpl<SDValue> &InVals) const override;
 
     int StoreByValRegs(CCState &CCInfo, SelectionDAG &DAG, const SDLoc &dl,
                        SDValue &Chain, const Value *OrigArg,

llvm/lib/Target/ARM/ARMISelLowering.cpp

for (unsigned i = 0; i != RVLocs.size(); ++i) {
     if (i == 0 && isThisReturn) {
       assert(!VA.needsCustom() && VA.getLocVT() == MVT::i32 &&
              "unexpected return calling convention register assignment");
       InVals.push_back(ThisVal);
       continue;
     }
 
     SDValue Val;
-    if (VA.needsCustom()) {
+    if (VA.needsCustom() &&
+        (VA.getLocVT() == MVT::f64 || VA.getLocVT() == MVT::v2f64)) {
       // Handle f64 or half of a v2f64.
       SDValue Lo = DAG.getCopyFromReg(Chain, dl, VA.getLocReg(), MVT::i32,
                                       InFlag);
       Chain = Lo.getValue(1);
       InFlag = Lo.getValue(2);
       VA = RVLocs[++i]; // skip ahead to next loc
       SDValue Hi = DAG.getCopyFromReg(Chain, dl, VA.getLocReg(), MVT::i32,
                                       InFlag);
for (unsigned i = 0; i != RVLocs.size(); ++i) {
     switch (VA.getLocInfo()) {
     default: llvm_unreachable("Unknown loc info!");
     case CCValAssign::Full: break;
     case CCValAssign::BCvt:
       Val = DAG.getNode(ISD::BITCAST, dl, VA.getValVT(), Val);
       break;
     }
 
+    // f16 arguments have their size extended to 4 bytes and passed as if they
+    // had been copied to the LSBs of a 32-bit register.
+    // For that, it's passed extended to i32 (soft ABI) or to f32 (hard ABI)
+    if (VA.needsCustom() && VA.getValVT() == MVT::f16) {
+      assert(Subtarget->hasFullFP16() &&
+             "Lowering f16 type return without full fp16 support");
+      Val = DAG.getNode(ISD::BITCAST, dl,
+                        MVT::getIntegerVT(VA.getLocVT().getSizeInBits()), Val);
+      Val = DAG.getNode(ARMISD::VMOVhr, dl, VA.getValVT(), Val);
+    }
+
     InVals.push_back(Val);
   }
 
   return Chain;
 }
 
 /// LowerMemOpCallTo - Store the argument to the stack.
 SDValue ARMTargetLowering::LowerMemOpCallTo(SDValue Chain, SDValue StackPtr,
for (unsigned i = 0, realArgIdx = 0, e = ArgLocs.size();
     case CCValAssign::AExt:
       Arg = DAG.getNode(ISD::ANY_EXTEND, dl, VA.getLocVT(), Arg);
       break;
     case CCValAssign::BCvt:
       Arg = DAG.getNode(ISD::BITCAST, dl, VA.getLocVT(), Arg);
       break;
     }
 
+    // f16 arguments have their size extended to 4 bytes and passed as if they
+    // had been copied to the LSBs of a 32-bit register.
+    // For that, it's passed extended to i32 (soft ABI) or to f32 (hard ABI)
+    if (VA.needsCustom() && VA.getValVT() == MVT::f16) {
+      assert(Subtarget->hasFullFP16() &&
+             "Lowering f16 type argument without full fp16 support");
+      Arg = DAG.getNode(ARMISD::VMOVrh, dl,
+                        MVT::getIntegerVT(VA.getLocVT().getSizeInBits()), Arg);
+      Arg = DAG.getNode(ISD::BITCAST, dl, VA.getLocVT(), Arg);
+    }
+
     // f64 and v2f64 might be passed in i32 pairs and must be split into pieces
-    if (VA.needsCustom()) {
-      if (VA.getLocVT() == MVT::v2f64) {
-        SDValue Op0 = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::f64, Arg,
-                                  DAG.getConstant(0, dl, MVT::i32));
-        SDValue Op1 = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::f64, Arg,
-                                  DAG.getConstant(1, dl, MVT::i32));
+    if (VA.needsCustom() && VA.getLocVT() == MVT::v2f64) {
 
-        PassF64ArgInRegs(dl, DAG, Chain, Op0, RegsToPass,
-                         VA, ArgLocs[++i], StackPtr, MemOpChains, Flags);
+      PassF64ArgInRegs(dl, DAG, Chain, Op0, RegsToPass, VA, ArgLocs[++i],
+                       StackPtr, MemOpChains, Flags);
 
-        VA = ArgLocs[++i]; // skip ahead to next loc
-        if (VA.isRegLoc()) {
-          PassF64ArgInRegs(dl, DAG, Chain, Op1, RegsToPass,
-                           VA, ArgLocs[++i], StackPtr, MemOpChains, Flags);
-        } else {
-          assert(VA.isMemLoc());
+        PassF64ArgInRegs(dl, DAG, Chain, Op1, RegsToPass, VA, ArgLocs[++i],
+                         StackPtr, MemOpChains, Flags);
 
-          MemOpChains.push_back(LowerMemOpCallTo(Chain, StackPtr, Op1,
-                                                 dl, DAG, VA, Flags));
-        }
-      } else {
-        PassF64ArgInRegs(dl, DAG, Chain, Arg, RegsToPass, VA, ArgLocs[++i],
-                         StackPtr, MemOpChains, Flags);
-      }
+        MemOpChains.push_back(
+            LowerMemOpCallTo(Chain, StackPtr, Op1, dl, DAG, VA, Flags));
+    } else if (VA.needsCustom() && VA.getLocVT() == MVT::f64) {
     } else if (VA.isRegLoc()) {
       if (realArgIdx == 0 && Flags.isReturned() && !Flags.isSwiftSelf() &&
           Outs[0].VT == MVT::i32) {
         assert(VA.getLocVT() == MVT::i32 &&
                "unexpected calling convention register assignment");
         assert(!Ins.empty() && Ins[0].VT == MVT::i32 &&
                "unexpected use of 'returned'");
         isThisReturn = true;
if (CCInfo.getNextStackOffset()) {
            i != e;
            ++i, ++realArgIdx) {
         CCValAssign &VA = ArgLocs[i];
         EVT RegVT = VA.getLocVT();
         SDValue Arg = OutVals[realArgIdx];
         ISD::ArgFlagsTy Flags = Outs[realArgIdx].Flags;
         if (VA.getLocInfo() == CCValAssign::Indirect)
           return false;
-        if (VA.needsCustom()) {
+        if (VA.needsCustom() && (RegVT == MVT::f64 || RegVT == MVT::v2f64)) {
           // f64 and vector types are split into multiple registers or
           // register/stack-slot combinations.  The types will not match
           // the registers; give up on memory f64 refs until we figure
           // out what to do about this.
           if (!VA.isRegLoc())
             return false;
           if (!ArgLocs[++i].isRegLoc())
             return false;
for (unsigned i = 0, realRVLocIdx = 0;
     default: llvm_unreachable("Unknown loc info!");
     case CCValAssign::Full: break;
     case CCValAssign::BCvt:
       if (!ReturnF16)
         Arg = DAG.getNode(ISD::BITCAST, dl, VA.getLocVT(), Arg);
       break;
     }
 
-    if (VA.needsCustom()) {
+    if (VA.needsCustom() &&
+        (VA.getLocVT() == MVT::v2f64 || VA.getLocVT() == MVT::f64)) {
       if (VA.getLocVT() == MVT::v2f64) {
         // Extract the first half and return it in two registers.
         SDValue Half = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::f64, Arg,
                                    DAG.getConstant(0, dl, MVT::i32));
         SDValue HalfGPRs = DAG.getNode(ARMISD::VMOVRRD, dl,
                                        DAG.getVTList(MVT::i32, MVT::i32), Half);
 
-        Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(),
-                                 HalfGPRs.getValue(isLittleEndian ? 0 : 1),
-                                 Flag);
+        Chain =
+            DAG.getCopyToReg(Chain, dl, VA.getLocReg(),
+                             HalfGPRs.getValue(isLittleEndian ? 0 : 1), Flag);
         Flag = Chain.getValue(1);
         RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
         VA = RVLocs[++i]; // skip ahead to next loc
-        Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(),
-                                 HalfGPRs.getValue(isLittleEndian ? 1 : 0),
-                                 Flag);
+        Chain =
+            DAG.getCopyToReg(Chain, dl, VA.getLocReg(),
+                             HalfGPRs.getValue(isLittleEndian ? 1 : 0), Flag);
         Flag = Chain.getValue(1);
         RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
         VA = RVLocs[++i]; // skip ahead to next loc
 
         // Extract the 2nd half and fall through to handle it as an f64 value.
         Arg = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::f64, Arg,
                           DAG.getConstant(1, dl, MVT::i32));
       }
       // Legalize ret f64 -> ret 2 x i32.  We always have fmrrd if f64 is
       // available.
       SDValue fmrrd = DAG.getNode(ARMISD::VMOVRRD, dl,
                                   DAG.getVTList(MVT::i32, MVT::i32), Arg);
       Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(),
-                               fmrrd.getValue(isLittleEndian ? 0 : 1),
-                               Flag);
+                               fmrrd.getValue(isLittleEndian ? 0 : 1), Flag);
       Flag = Chain.getValue(1);
       RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
       VA = RVLocs[++i]; // skip ahead to next loc
       Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(),
-                               fmrrd.getValue(isLittleEndian ? 1 : 0),
-                               Flag);
+                               fmrrd.getValue(isLittleEndian ? 1 : 0), Flag);
     } else
       Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(), Arg, Flag);
 
     // Guarantee that all emitted copies are
     // stuck together, avoiding something bad.
     Flag = Chain.getValue(1);
     RetOps.push_back(DAG.getRegister(VA.getLocReg(),
                                      ReturnF16 ? MVT::f16 : VA.getLocVT()));
void ARMTargetLowering::VarArgStyleRegisters(CCState &CCInfo, SelectionDAG &DAG,
   // argument passed via stack.
   int FrameIndex = StoreByValRegs(CCInfo, DAG, dl, Chain, nullptr,
                                   CCInfo.getInRegsParamsCount(),
                                   CCInfo.getNextStackOffset(),
                                   std::max(4U, TotalArgRegsSaveSize));
   AFI->setVarArgsFrameIndex(FrameIndex);
 }
 
+bool ARMTargetLowering::splitValueIntoRegisterParts(
+    SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
+    unsigned NumParts, MVT PartVT, Optional<CallingConv::ID> CC) const {
+  bool IsABIRegCopy = CC.hasValue();
+  EVT ValueVT = Val.getValueType();
+  if (IsABIRegCopy && ValueVT == MVT::f16 && PartVT == MVT::f32) {
+    unsigned ValueBits = ValueVT.getSizeInBits();
+    unsigned PartBits = PartVT.getSizeInBits();
+    Val = DAG.getNode(ISD::BITCAST, DL, MVT::getIntegerVT(ValueBits), Val);
+    Val = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::getIntegerVT(PartBits), Val);
+    Val = DAG.getNode(ISD::BITCAST, DL, PartVT, Val);
+    Parts[0] = Val;
+    return true;
+  }
+  return false;
+}
+
+SDValue ARMTargetLowering::joinRegisterPartsIntoValue(
+    SelectionDAG &DAG, const SDLoc &DL, const SDValue *Parts, unsigned NumParts,
+    MVT PartVT, EVT ValueVT, Optional<CallingConv::ID> CC) const {
+  bool IsABIRegCopy = CC.hasValue();
+  if (IsABIRegCopy && ValueVT == MVT::f16 && PartVT == MVT::f32) {
+    unsigned ValueBits = ValueVT.getSizeInBits();
+    unsigned PartBits = PartVT.getSizeInBits();
+    SDValue Val = Parts[0];
+
+    Val = DAG.getNode(ISD::BITCAST, DL, MVT::getIntegerVT(PartBits), Val);
+    Val = DAG.getNode(ISD::TRUNCATE, DL, MVT::getIntegerVT(ValueBits), Val);
+    Val = DAG.getNode(ISD::BITCAST, DL, ValueVT, Val);
+    return Val;
+  }
+  return SDValue();
+}
+
 SDValue ARMTargetLowering::LowerFormalArguments(
     SDValue Chain, CallingConv::ID CallConv, bool isVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &dl,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();
 
   ARMFunctionInfo *AFI = MF.getInfo<ARMFunctionInfo>();
if (Ins[VA.getValNo()].isOrigArg()) {
       std::advance(CurOrigArg,
                    Ins[VA.getValNo()].getOrigArgIndex() - CurArgIdx);
       CurArgIdx = Ins[VA.getValNo()].getOrigArgIndex();
     }
     // Arguments stored in registers.
     if (VA.isRegLoc()) {
       EVT RegVT = VA.getLocVT();
 
-      if (VA.needsCustom()) {
+      if (VA.needsCustom() && VA.getLocVT() == MVT::v2f64) {
         // f64 and vector types are split up into multiple registers or
         // combinations of registers and stack slots.
-        if (VA.getLocVT() == MVT::v2f64) {
-          SDValue ArgValue1 = GetF64FormalArgument(VA, ArgLocs[++i],
-                                                   Chain, DAG, dl);
-          VA = ArgLocs[++i]; // skip ahead to next loc
-          SDValue ArgValue2;
-          if (VA.isMemLoc()) {
-            int FI = MFI.CreateFixedObject(8, VA.getLocMemOffset(), true);
-            SDValue FIN = DAG.getFrameIndex(FI, PtrVT);
-            ArgValue2 = DAG.getLoad(MVT::f64, dl, Chain, FIN,
-                                    MachinePointerInfo::getFixedStack(
-                                        DAG.getMachineFunction(), FI));
-          } else {
-            ArgValue2 = GetF64FormalArgument(VA, ArgLocs[++i],
-                                             Chain, DAG, dl);
-          }
-          ArgValue = DAG.getNode(ISD::UNDEF, dl, MVT::v2f64);
-          ArgValue = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, MVT::v2f64,
-                                 ArgValue, ArgValue1,
-                                 DAG.getIntPtrConstant(0, dl));
-          ArgValue = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, MVT::v2f64,
-                                 ArgValue, ArgValue2,
-                                 DAG.getIntPtrConstant(1, dl));
-        } else
-          ArgValue = GetF64FormalArgument(VA, ArgLocs[++i], Chain, DAG, dl);
+        SDValue ArgValue1 =
+            GetF64FormalArgument(VA, ArgLocs[++i], Chain, DAG, dl);
+          ArgValue2 = DAG.getLoad(
+              MVT::f64, dl, Chain, FIN,
+              MachinePointerInfo::getFixedStack(DAG.getMachineFunction(), FI));
+          ArgValue2 = GetF64FormalArgument(VA, ArgLocs[++i], Chain, DAG, dl);
+        ArgValue = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, MVT::v2f64, ArgValue,
+                               ArgValue1, DAG.getIntPtrConstant(0, dl));
+        ArgValue = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, MVT::v2f64, ArgValue,
+                               ArgValue2, DAG.getIntPtrConstant(1, dl));
+      } else if (VA.needsCustom() && VA.getLocVT() == MVT::f64) {
       } else {
         const TargetRegisterClass *RC;
 
 
         if (RegVT == MVT::f16)
           RC = &ARM::HPRRegClass;
         else if (RegVT == MVT::f32)
           RC = &ARM::SPRRegClass;
if (VA.isRegLoc()) {
         break;
       case CCValAssign::ZExt:
         ArgValue = DAG.getNode(ISD::AssertZext, dl, RegVT, ArgValue,
                                DAG.getValueType(VA.getValVT()));
         ArgValue = DAG.getNode(ISD::TRUNCATE, dl, VA.getValVT(), ArgValue);
         break;
       }
 
+      // f16 arguments have their size extended to 4 bytes and passed as if they
+      // had been copied to the LSBs of a 32-bit register.
+      // For that, it's passed extended to i32 (soft ABI) or to f32 (hard ABI)
+      if (VA.needsCustom() && VA.getValVT() == MVT::f16) {
+        assert(Subtarget->hasFullFP16() &&
+               "Lowering f16 type argument without full fp16 support");
+        ArgValue = DAG.getNode(ISD::BITCAST, dl,
+                               MVT::getIntegerVT(VA.getLocVT().getSizeInBits()),
+                               ArgValue);
+        ArgValue = DAG.getNode(ARMISD::VMOVhr, dl, VA.getValVT(), ArgValue);
+      }
+
       InVals.push_back(ArgValue);
     } else { // VA.isRegLoc()
       // sanity check
       assert(VA.isMemLoc());
       assert(VA.getValVT() != MVT::i64 && "i64 should already be lowered");
 
       int index = VA.getValNo();
 

llvm/test/CodeGen/ARM/GlobalISel/arm-unsupported.ll

 define i17 @test_funny_ints(i17 %a, i17 %b) {
 ; CHECK: remark: {{.*}} unable to lower arguments: i17 (i17, i17)*
 ; CHECK-LABEL: warning: Instruction selection used fallback path for test_funny_ints
   %res = add i17 %a, %b
   ret i17 %res
 }
 
 define half @test_half(half %a, half %b) {
-; CHECK: remark: {{.*}} unable to translate instruction: ret: '  ret half %res' (in function: test_half)
+; CHECK: remark: {{.*}} unable to lower arguments: half (half, half)* (in function: test_half)
 ; CHECK-LABEL: warning: Instruction selection used fallback path for test_half
   %res = fadd half %a, %b
   ret half %res
 }
 
 declare [16 x i32] @ret_demotion_target()
 
 define [16 x i32] @test_ret_demotion() {

llvm/test/CodeGen/ARM/fp16-args.ll

-; RUN: llc -float-abi soft -mattr=+fp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=SOFT
-; RUN: llc -float-abi hard -mattr=+fp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=HARD
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=armv7a--none-eabi -float-abi soft -mattr=+fp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=SOFT
+; RUN: llc -mtriple=armv7a--none-eabi -float-abi hard -mattr=+fp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=HARD
+; RUN: llc -mtriple=armv7a--none-eabi -float-abi soft -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=FULL-SOFT
+; RUN: llc -mtriple=armv7a--none-eabi -float-abi hard -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=FULL-HARD
+; RUN: llc -mtriple=armv7aeb--none-eabi -float-abi soft -mattr=+fp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=SOFT
+; RUN: llc -mtriple=armv7aeb--none-eabi -float-abi hard -mattr=+fp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=HARD
+; RUN: llc -mtriple=armv7aeb--none-eabi -float-abi soft -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=FULL-SOFT
+; RUN: llc -mtriple=armv7aeb--none-eabi -float-abi hard -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=FULL-HARD
 
-target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
-target triple = "armv7a--none-eabi"
-
-define float @foo(float %a.coerce, float %b.coerce) {
+define half @foo(half %a, half %b) {
+; SOFT-LABEL: foo:
+; SOFT:       @ %bb.0: @ %entry
+; SOFT-NEXT:    vmov s2, r1
+; SOFT-NEXT:    vmov s0, r0
+; SOFT-NEXT:    vcvtb.f32.f16 s2, s2
+; SOFT-NEXT:    vcvtb.f32.f16 s0, s0
+; SOFT-NEXT:    vadd.f32 s0, s0, s2
+; SOFT-NEXT:    vcvtb.f16.f32 s0, s0
+; SOFT-NEXT:    vmov r0, s0
+; SOFT-NEXT:    bx lr
+;
+; HARD-LABEL: foo:
+; HARD:       @ %bb.0: @ %entry
+; HARD-NEXT:    vcvtb.f32.f16 s2, s1
+; HARD-NEXT:    vcvtb.f32.f16 s0, s0
+; HARD-NEXT:    vadd.f32 s0, s0, s2
+; HARD-NEXT:    vcvtb.f16.f32 s0, s0
+; HARD-NEXT:    bx lr
+;
+; FULL-SOFT-LABEL: foo:
+; FULL-SOFT:       @ %bb.0: @ %entry
+; FULL-SOFT-NEXT:    vmov.f16 s0, r1
+; FULL-SOFT-NEXT:    vmov.f16 s2, r0
+; FULL-SOFT-NEXT:    vadd.f16 s0, s2, s0
+; FULL-SOFT-NEXT:    vmov r0, s0
+; FULL-SOFT-NEXT:    bx lr
+;
+; FULL-HARD-LABEL: foo:
+; FULL-HARD:       @ %bb.0: @ %entry
+; FULL-HARD-NEXT:    vadd.f16 s0, s0, s1
+; FULL-HARD-NEXT:    bx lr
 entry:
-  %0 = bitcast float %a.coerce to i32
-  %tmp.0.extract.trunc = trunc i32 %0 to i16
-  %1 = bitcast i16 %tmp.0.extract.trunc to half
-  %2 = bitcast float %b.coerce to i32
-  %tmp1.0.extract.trunc = trunc i32 %2 to i16
-  %3 = bitcast i16 %tmp1.0.extract.trunc to half
-  %4 = fadd half %1, %3
-  %5 = bitcast half %4 to i16
-  %tmp5.0.insert.ext = zext i16 %5 to i32
-  %6 = bitcast i32 %tmp5.0.insert.ext to float
-  ret float %6
-; CHECK: foo:
-
-; SOFT: vmov    {{s[0-9]+}}, r1
-; SOFT: vmov    {{s[0-9]+}}, r0
-; SOFT: vcvtb.f32.f16   {{s[0-9]+}}, {{s[0-9]+}}
-; SOFT: vcvtb.f32.f16   {{s[0-9]+}}, {{s[0-9]+}}
-; SOFT: vadd.f32        {{s[0-9]+}}, {{s[0-9]+}}, {{s[0-9]+}}
-; SOFT: vcvtb.f16.f32   {{s[0-9]+}}, {{s[0-9]+}}
-; SOFT: vmov    r0, {{s[0-9]+}}
-
-; HARD-NOT: vmov
-; HARD-NOT: uxth
-; HARD: vcvtb.f32.f16   {{s[0-9]+}}, s1
-; HARD: vcvtb.f32.f16   {{s[0-9]+}}, s0
-; HARD: vadd.f32        {{s[0-9]+}}, {{s[0-9]+}}, {{s[0-9]+}}
-; HARD: vcvtb.f16.f32   [[SREG:s[0-9]+]], {{s[0-9]+}}
-; HARD-NEXT: vmov            [[REG0:r[0-9]+]], [[SREG]]
-; HARD-NEXT: uxth            [[REG1:r[0-9]+]], [[REG0]]
-; HARD-NEXT: vmov            s0, [[REG1]]
-
-; CHECK: bx lr
+  %0 = fadd half %a, %b
+  ret half %0
 }

llvm/test/CodeGen/ARM/fp16-bitcast.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple thumbv8m.main-arm-unknown-eabi -mattr=+vfp4d16sp < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-VFPV4
-; RUN: llc -mtriple thumbv8.1m.main-arm-unknown-eabi -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-FP16
+; RUN: llc -mtriple thumbv8m.main-arm-unknown-eabi --float-abi=soft -mattr=+vfp4d16sp < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-VFPV4-SOFT
+; RUN: llc -mtriple thumbv8.1m.main-arm-unknown-eabi --float-abi=soft -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-FP16-SOFT
+; RUN: llc -mtriple thumbv8m.main-arm-unknown-eabi --float-abi=hard -mattr=+vfp4d16sp < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-VFPV4-HARD
+; RUN: llc -mtriple thumbv8.1m.main-arm-unknown-eabi --float-abi=hard -mattr=+fullfp16 < %s | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-FP16-HARD
 
 target triple = "thumbv8.1m.main-arm-unknown-eabi"
 
 define float @add(float %a, float %b) {
-; CHECK-LABEL: add:
-; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov s0, r1
-; CHECK-NEXT:    vmov s2, r0
-; CHECK-NEXT:    vadd.f32 s0, s2, s0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    bx lr
+; CHECK-VFPV4-SOFT-LABEL: add:
+; CHECK-VFPV4-SOFT:       @ %bb.0: @ %entry
+; CHECK-VFPV4-SOFT-NEXT:    vmov s0, r1
+; CHECK-VFPV4-SOFT-NEXT:    vmov s2, r0
+; CHECK-VFPV4-SOFT-NEXT:    vadd.f32 s0, s2, s0
+; CHECK-VFPV4-SOFT-NEXT:    vmov r0, s0
+; CHECK-VFPV4-SOFT-NEXT:    bx lr
+;
+; CHECK-FP16-SOFT-LABEL: add:
+; CHECK-FP16-SOFT:       @ %bb.0: @ %entry
+; CHECK-FP16-SOFT-NEXT:    vmov s0, r1
+; CHECK-FP16-SOFT-NEXT:    vmov s2, r0
+; CHECK-FP16-SOFT-NEXT:    vadd.f32 s0, s2, s0
+; CHECK-FP16-SOFT-NEXT:    vmov r0, s0
+; CHECK-FP16-SOFT-NEXT:    bx lr
+;
+; CHECK-VFPV4-HARD-LABEL: add:
+; CHECK-VFPV4-HARD:       @ %bb.0: @ %entry
+; CHECK-VFPV4-HARD-NEXT:    vadd.f32 s0, s0, s1
+; CHECK-VFPV4-HARD-NEXT:    bx lr
+;
+; CHECK-FP16-HARD-LABEL: add:
+; CHECK-FP16-HARD:       @ %bb.0: @ %entry
+; CHECK-FP16-HARD-NEXT:    vadd.f32 s0, s0, s1
+; CHECK-FP16-HARD-NEXT:    bx lr
 entry:
   %add = fadd float %a, %b
   ret float %add
 }
 
-define i32 @addf16(i32 %a.coerce, i32 %b.coerce) {
-; CHECK-VFPV4-LABEL: addf16:
-; CHECK-VFPV4:       @ %bb.0: @ %entry
-; CHECK-VFPV4-NEXT:    vmov s2, r1
-; CHECK-VFPV4-NEXT:    vmov s0, r0
-; CHECK-VFPV4-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-VFPV4-NEXT:    vcvtb.f32.f16 s0, s0
-; CHECK-VFPV4-NEXT:    vadd.f32 s0, s0, s2
-; CHECK-VFPV4-NEXT:    vcvtb.f16.f32 s0, s0
-; CHECK-VFPV4-NEXT:    vmov r0, s0
-; CHECK-VFPV4-NEXT:    uxth r0, r0
-; CHECK-VFPV4-NEXT:    bx lr
-;
-; CHECK-FP16-LABEL: addf16:
-; CHECK-FP16:       @ %bb.0: @ %entry
-; CHECK-FP16-NEXT:    vmov.f16 s0, r1
-; CHECK-FP16-NEXT:    vmov.f16 s2, r0
-; CHECK-FP16-NEXT:    vadd.f16 s0, s2, s0
-; CHECK-FP16-NEXT:    vmov.f16 r0, s0
-; CHECK-FP16-NEXT:    bx lr
+define half @addf16(half %a, half %b) {
+; CHECK-VFPV4-SOFT-LABEL: addf16:
+; CHECK-VFPV4-SOFT:       @ %bb.0: @ %entry
+; CHECK-VFPV4-SOFT-NEXT:    vmov s2, r1
+; CHECK-VFPV4-SOFT-NEXT:    vmov s0, r0
+; CHECK-VFPV4-SOFT-NEXT:    vcvtb.f32.f16 s2, s2
+; CHECK-VFPV4-SOFT-NEXT:    vcvtb.f32.f16 s0, s0
+; CHECK-VFPV4-SOFT-NEXT:    vadd.f32 s0, s0, s2
+; CHECK-VFPV4-SOFT-NEXT:    vcvtb.f16.f32 s0, s0
+; CHECK-VFPV4-SOFT-NEXT:    vmov r0, s0
+; CHECK-VFPV4-SOFT-NEXT:    bx lr
+;
+; CHECK-FP16-SOFT-LABEL: addf16:
+; CHECK-FP16-SOFT:       @ %bb.0: @ %entry
+; CHECK-FP16-SOFT-NEXT:    vmov.f16 s0, r1
+; CHECK-FP16-SOFT-NEXT:    vmov.f16 s2, r0
+; CHECK-FP16-SOFT-NEXT:    vadd.f16 s0, s2, s0
+; CHECK-FP16-SOFT-NEXT:    vmov r0, s0
+; CHECK-FP16-SOFT-NEXT:    bx lr
+;
+; CHECK-VFPV4-HARD-LABEL: addf16:
+; CHECK-VFPV4-HARD:       @ %bb.0: @ %entry
+; CHECK-VFPV4-HARD-NEXT:    vcvtb.f32.f16 s2, s1
+; CHECK-VFPV4-HARD-NEXT:    vcvtb.f32.f16 s0, s0
+; CHECK-VFPV4-HARD-NEXT:    vadd.f32 s0, s0, s2
+; CHECK-VFPV4-HARD-NEXT:    vcvtb.f16.f32 s0, s0
+; CHECK-VFPV4-HARD-NEXT:    bx lr
+;
+; CHECK-FP16-HARD-LABEL: addf16:
+; CHECK-FP16-HARD:       @ %bb.0: @ %entry
+; CHECK-FP16-HARD-NEXT:    vadd.f16 s0, s0, s1
+; CHECK-FP16-HARD-NEXT:    bx lr
 entry:
-  %tmp.0.extract.trunc = trunc i32 %a.coerce to i16
-  %0 = bitcast i16 %tmp.0.extract.trunc to half
-  %tmp1.0.extract.trunc = trunc i32 %b.coerce to i16
-  %1 = bitcast i16 %tmp1.0.extract.trunc to half
-  %add = fadd half %0, %1
-  %2 = bitcast half %add to i16
-  %tmp4.0.insert.ext = zext i16 %2 to i32
-  ret i32 %tmp4.0.insert.ext
+  %add = fadd half %a, %b
+  ret half %add
 }
 
 define half @load_i16(i16 *%hp) {
-; CHECK-VFPV4-LABEL: load_i16:
-; CHECK-VFPV4:       @ %bb.0: @ %entry
-; CHECK-VFPV4-NEXT:    vmov.f32 s0, #1.000000e+00
-; CHECK-VFPV4-NEXT:    ldrh r0, [r0]
-; CHECK-VFPV4-NEXT:    vmov s2, r0
-; CHECK-VFPV4-NEXT:    vcvtb.f32.f16 s2, s2
-; CHECK-VFPV4-NEXT:    vadd.f32 s0, s2, s0
-; CHECK-VFPV4-NEXT:    vmov r0, s0
-; CHECK-VFPV4-NEXT:    bx lr
-;
-; CHECK-FP16-LABEL: load_i16:
-; CHECK-FP16:       @ %bb.0: @ %entry
-; CHECK-FP16-NEXT:    vldr.16 s2, [r1]
-; CHECK-FP16-NEXT:    vmov.f16 s0, #1.000000e+00
-; CHECK-FP16-NEXT:    vadd.f16 s0, s2, s0
-; CHECK-FP16-NEXT:    vstr.16 s0, [r0]
-; CHECK-FP16-NEXT:    bx lr
+; CHECK-VFPV4-SOFT-LABEL: load_i16:
+; CHECK-VFPV4-SOFT:       @ %bb.0: @ %entry
+; CHECK-VFPV4-SOFT-NEXT:    vmov.f32 s0, #1.000000e+00
+; CHECK-VFPV4-SOFT-NEXT:    ldrh r0, [r0]
+; CHECK-VFPV4-SOFT-NEXT:    vmov s2, r0
+; CHECK-VFPV4-SOFT-NEXT:    vcvtb.f32.f16 s2, s2
+; CHECK-VFPV4-SOFT-NEXT:    vadd.f32 s0, s2, s0
+; CHECK-VFPV4-SOFT-NEXT:    vcvtb.f16.f32 s0, s0
+; CHECK-VFPV4-SOFT-NEXT:    vmov r0, s0
+; CHECK-VFPV4-SOFT-NEXT:    bx lr
+;
+; CHECK-FP16-SOFT-LABEL: load_i16:
+; CHECK-FP16-SOFT:       @ %bb.0: @ %entry
+; CHECK-FP16-SOFT-NEXT:    vldr.16 s2, [r0]
+; CHECK-FP16-SOFT-NEXT:    vmov.f16 s0, #1.000000e+00
+; CHECK-FP16-SOFT-NEXT:    vadd.f16 s0, s2, s0
+; CHECK-FP16-SOFT-NEXT:    vmov r0, s0
+; CHECK-FP16-SOFT-NEXT:    bx lr
+;
+; CHECK-VFPV4-HARD-LABEL: load_i16:
+; CHECK-VFPV4-HARD:       @ %bb.0: @ %entry
+; CHECK-VFPV4-HARD-NEXT:    vmov.f32 s0, #1.000000e+00
+; CHECK-VFPV4-HARD-NEXT:    ldrh r0, [r0]
+; CHECK-VFPV4-HARD-NEXT:    vmov s2, r0
+; CHECK-VFPV4-HARD-NEXT:    vcvtb.f32.f16 s2, s2
+; CHECK-VFPV4-HARD-NEXT:    vadd.f32 s0, s2, s0
+; CHECK-VFPV4-HARD-NEXT:    vcvtb.f16.f32 s0, s0
+; CHECK-VFPV4-HARD-NEXT:    bx lr
+;
+; CHECK-FP16-HARD-LABEL: load_i16:
+; CHECK-FP16-HARD:       @ %bb.0: @ %entry
+; CHECK-FP16-HARD-NEXT:    vldr.16 s2, [r0]
+; CHECK-FP16-HARD-NEXT:    vmov.f16 s0, #1.000000e+00
+; CHECK-FP16-HARD-NEXT:    vadd.f16 s0, s2, s0
+; CHECK-FP16-HARD-NEXT:    bx lr
 entry:
   %h = load i16, i16 *%hp, align 2
   %hc = bitcast i16 %h to half
   %add = fadd half %hc, 1.0
   ret half %add
 }
 
 define i16 @load_f16(half *%hp) {

llvm/test/CodeGen/ARM/fp16-promote.ll

 define void @test_extractvalue(%struct.dummy* %p, half* %q) {
   %a = load %struct.dummy, %struct.dummy* %p
   %b = extractvalue %struct.dummy %a, 1
   store half %b, half* %q
   ret void
 }
 
 ; CHECK-ALL-LABEL: test_struct_return:
-; CHECK-FP16: vcvtb.f32.f16
 ; CHECK-VFP-LIBCALL: bl __aeabi_h2f
 ; CHECK-NOVFP-DAG: ldr
 ; CHECK-NOVFP-DAG: ldrh
 define %struct.dummy @test_struct_return(%struct.dummy* %p) {
   %a = load %struct.dummy, %struct.dummy* %p
   ret %struct.dummy %a
 }
 

llvm/test/CodeGen/ARM/fp16-vminmaxnm-safe.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=armv8-eabi -mattr=+fullfp16 | FileCheck %s
 ; RUN: llc < %s -mtriple thumbv7a -mattr=+fullfp16 | FileCheck %s
 
-; TODO: we can't pass half-precision arguments as "half" types yet. We do
-; that for the time being by passing "float %f.coerce" and the necessary
-; bitconverts/truncates. In these tests we pass i16 and use 1 bitconvert, which
-; is the shortest way to get a half type. But when we can pass half types, we
-; want to use that here.
-
-define half @fp16_vminnm_o(i16 signext %a, i16 signext %b) {
+define half @fp16_vminnm_o(half %a, half %b) {
 ; CHECK-LABEL: fp16_vminnm_o:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r2
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp olt half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp olt half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vminnm_o_rev(i16 signext %a, i16 signext %b) {
+define half @fp16_vminnm_o_rev(half %a, half %b) {
 ; CHECK-LABEL: fp16_vminnm_o_rev:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r2
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ogt half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp ogt half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vminnm_u(i16 signext %a, i16 signext %b) {
+define half @fp16_vminnm_u(half %a, half %b) {
 ; CHECK-LABEL: fp16_vminnm_u:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
-; CHECK-NEXT:    vmov.f16 s2, r2
+; CHECK-NEXT:    vmov.f16 s0, r0
+; CHECK-NEXT:    vmov.f16 s2, r1
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ult half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp ult half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vminnm_ule(i16 signext %a, i16 signext %b) {
+define half @fp16_vminnm_ule(half %a, half %b) {
 ; CHECK-LABEL: fp16_vminnm_ule:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
-; CHECK-NEXT:    vmov.f16 s2, r2
+; CHECK-NEXT:    vmov.f16 s0, r0
+; CHECK-NEXT:    vmov.f16 s2, r1
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ule half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp ule half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vminnm_u_rev(i16 signext %a, i16 signext %b) {
+define half @fp16_vminnm_u_rev(half %a, half %b) {
 ; CHECK-LABEL: fp16_vminnm_u_rev:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r2
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ugt half %0, %1
-  %cond = select i1 %cmp, half %1, half %0
+  %cmp = fcmp ugt half %a, %b
+  %cond = select i1 %cmp, half %b, half %a
   ret half %cond
 }
 
-define half @fp16_vmaxnm_o(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_o(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_o:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r2
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ogt half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp ogt half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vmaxnm_oge(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_oge(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_oge:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r2
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp oge half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp oge half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vmaxnm_o_rev(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_o_rev(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_o_rev:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
-; CHECK-NEXT:    vmov.f16 s2, r2
+; CHECK-NEXT:    vmov.f16 s0, r0
+; CHECK-NEXT:    vmov.f16 s2, r1
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp olt half %0, %1
-  %cond = select i1 %cmp, half %1, half %0
+  %cmp = fcmp olt half %a, %b
+  %cond = select i1 %cmp, half %b, half %a
   ret half %cond
 }
 
-define half @fp16_vmaxnm_ole_rev(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_ole_rev(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_ole_rev:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
-; CHECK-NEXT:    vmov.f16 s2, r2
+; CHECK-NEXT:    vmov.f16 s0, r0
+; CHECK-NEXT:    vmov.f16 s2, r1
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ole half %0, %1
-  %cond = select i1 %cmp, half %1, half %0
+  %cmp = fcmp ole half %a, %b
+  %cond = select i1 %cmp, half %b, half %a
   ret half %cond
 }
 
-define half @fp16_vmaxnm_u(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_u(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_u:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
-; CHECK-NEXT:    vmov.f16 s2, r2
+; CHECK-NEXT:    vmov.f16 s0, r0
+; CHECK-NEXT:    vmov.f16 s2, r1
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ugt half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp ugt half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vmaxnm_uge(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_uge(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_uge:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
-; CHECK-NEXT:    vmov.f16 s2, r2
+; CHECK-NEXT:    vmov.f16 s0, r0
+; CHECK-NEXT:    vmov.f16 s2, r1
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp uge half %0, %1
-  %cond = select i1 %cmp, half %0, half %1
+  %cmp = fcmp uge half %a, %b
+  %cond = select i1 %cmp, half %a, half %b
   ret half %cond
 }
 
-define half @fp16_vmaxnm_u_rev(i16 signext %a, i16 signext %b) {
+define half @fp16_vmaxnm_u_rev(half %a, half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_u_rev:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r2
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 entry:
-  %0 = bitcast i16 %a to half
-  %1 = bitcast i16 %b to half
-  %cmp = fcmp ult half %0, %1
-  %cond = select i1 %cmp, half %1, half %0
+  %cmp = fcmp ult half %a, %b
+  %cond = select i1 %cmp, half %b, half %a
   ret half %cond
 }
 
 ; known non-NaNs
 
-define half @fp16_vminnm_NNNo(i16 signext %a) {
+define half @fp16_vminnm_NNNo(half %a) {
 ; CHECK-LABEL: fp16_vminnm_NNNo:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vmov.f16 s2, #1.200000e+01
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI12_0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI12_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp olt half %0, 12.
-  %cond1 = select i1 %cmp1, half %0, half 12.
+  %cmp1 = fcmp olt half %a, 12.
+  %cond1 = select i1 %cmp1, half %a, half 12.
   %cmp2 = fcmp olt half 34., %cond1
   %cond2 = select i1 %cmp2, half 34., half %cond1
   ret half %cond2
 }
 
-define half @fp16_vminnm_NNNo_rev(i16 signext %a) {
+define half @fp16_vminnm_NNNo_rev(half %a) {
 ; CHECK-LABEL: fp16_vminnm_NNNo_rev:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI13_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
 ; CHECK-NEXT:    vldr.16 s2, .LCPI13_1
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI13_0:
 ; CHECK-NEXT:    .short 0x5300 @ half 56
 ; CHECK-NEXT:  .LCPI13_1:
 ; CHECK-NEXT:    .short 0x54e0 @ half 78
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp ogt half %0, 56.
-  %cond1 = select i1 %cmp1, half 56., half %0
+  %cmp1 = fcmp ogt half %a, 56.
+  %cond1 = select i1 %cmp1, half 56., half %a
   %cmp2 = fcmp ogt half 78., %cond1
   %cond2 = select i1 %cmp2, half %cond1, half 78.
   ret half %cond2
 }
 
-define half @fp16_vminnm_NNNu(i16 signext %b) {
+define half @fp16_vminnm_NNNu(half %b) {
 ; CHECK-LABEL: fp16_vminnm_NNNu:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vmov.f16 s2, #1.200000e+01
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI14_0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI14_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 entry:
-  %0 = bitcast i16 %b to half
-  %cmp1 = fcmp ult half 12., %0
-  %cond1 = select i1 %cmp1, half 12., half %0
+  %cmp1 = fcmp ult half 12., %b
+  %cond1 = select i1 %cmp1, half 12., half %b
   %cmp2 = fcmp ult half %cond1, 34.
   %cond2 = select i1 %cmp2, half %cond1, half 34.
   ret half %cond2
 }
 
-define half @fp16_vminnm_NNNule(i16 signext %b) {
+define half @fp16_vminnm_NNNule(half %b) {
 ; CHECK-LABEL: fp16_vminnm_NNNule:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI15_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI15_1
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI15_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 ; CHECK-NEXT:  .LCPI15_1:
 ; CHECK-NEXT:    .short 0x5300 @ half 56
 
 entry:
-  %0 = bitcast i16 %b to half
-  %cmp1 = fcmp ule half 34., %0
-  %cond1 = select i1 %cmp1, half 34., half %0
+  %cmp1 = fcmp ule half 34., %b
+  %cond1 = select i1 %cmp1, half 34., half %b
   %cmp2 = fcmp ule half %cond1, 56.
   %cond2 = select i1 %cmp2, half %cond1, half 56.
   ret half %cond2
 }
 
-define half @fp16_vminnm_NNNu_rev(i16 signext %b) {
+define half @fp16_vminnm_NNNu_rev(half %b) {
 ; CHECK-LABEL: fp16_vminnm_NNNu_rev:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI16_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
 ; CHECK-NEXT:    vldr.16 s2, .LCPI16_1
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI16_0:
 ; CHECK-NEXT:    .short 0x5300 @ half 56
 ; CHECK-NEXT:  .LCPI16_1:
 ; CHECK-NEXT:    .short 0x54e0 @ half 78
 
 
 entry:
-  %0 = bitcast i16 %b to half
-  %cmp1 = fcmp ugt half 56., %0
-  %cond1 = select i1 %cmp1, half %0, half 56.
+  %cmp1 = fcmp ugt half 56., %b
+  %cond1 = select i1 %cmp1, half %b, half 56.
   %cmp2 = fcmp ugt half %cond1, 78.
   %cond2 = select i1 %cmp2, half 78., half %cond1
   ret half %cond2
 }
 
-define half @fp16_vmaxnm_NNNo(i16 signext %a) {
+define half @fp16_vmaxnm_NNNo(half %a) {
 ; CHECK-LABEL: fp16_vmaxnm_NNNo:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vmov.f16 s2, #1.200000e+01
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI17_0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI17_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp ogt half %0, 12.
-  %cond1 = select i1 %cmp1, half %0, half 12.
+  %cmp1 = fcmp ogt half %a, 12.
+  %cond1 = select i1 %cmp1, half %a, half 12.
   %cmp2 = fcmp ogt half 34., %cond1
   %cond2 = select i1 %cmp2, half 34., half %cond1
   ret half %cond2
 }
 
-define half @fp16_vmaxnm_NNNoge(i16 signext %a) {
+define half @fp16_vmaxnm_NNNoge(half %a) {
 ; CHECK-LABEL: fp16_vmaxnm_NNNoge:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI18_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI18_1
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI18_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 ; CHECK-NEXT:  .LCPI18_1:
 ; CHECK-NEXT:    .short 0x5300 @ half 56
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp oge half %0, 34.
-  %cond1 = select i1 %cmp1, half %0, half 34.
+  %cmp1 = fcmp oge half %a, 34.
+  %cond1 = select i1 %cmp1, half %a, half 34.
   %cmp2 = fcmp oge half 56., %cond1
   %cond2 = select i1 %cmp2, half 56., half %cond1
   ret half %cond2
 }
 
-define half @fp16_vmaxnm_NNNo_rev(i16 signext %a) {
+define half @fp16_vmaxnm_NNNo_rev(half %a) {
 ; CHECK-LABEL: fp16_vmaxnm_NNNo_rev:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI19_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
 ; CHECK-NEXT:    vldr.16 s2, .LCPI19_1
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI19_0:
 ; CHECK-NEXT:    .short 0x5300 @ half 56
 ; CHECK-NEXT:  .LCPI19_1:
 ; CHECK-NEXT:    .short 0x54e0 @ half 78
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp olt half %0, 56.
-  %cond1 = select i1 %cmp1, half 56., half %0
+  %cmp1 = fcmp olt half %a, 56.
+  %cond1 = select i1 %cmp1, half 56., half %a
   %cmp2 = fcmp olt half 78., %cond1
   %cond2 = select i1 %cmp2, half %cond1, half 78.
   ret half %cond2
 }
 
-define half @fp16_vmaxnm_NNNole_rev(i16 signext %a) {
+define half @fp16_vmaxnm_NNNole_rev(half %a) {
 ; CHECK-LABEL: fp16_vmaxnm_NNNole_rev:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI20_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
 ; CHECK-NEXT:    vldr.16 s2, .LCPI20_1
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI20_0:
 ; CHECK-NEXT:    .short 0x54e0 @ half 78
 ; CHECK-NEXT:  .LCPI20_1:
 ; CHECK-NEXT:    .short 0x55a0 @ half 90
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp ole half %0, 78.
-  %cond1 = select i1 %cmp1, half 78., half %0
+  %cmp1 = fcmp ole half %a, 78.
+  %cond1 = select i1 %cmp1, half 78., half %a
   %cmp2 = fcmp ole half 90., %cond1
   %cond2 = select i1 %cmp2, half %cond1, half 90.
   ret half %cond2
 }
 
-define half @fp16_vmaxnm_NNNu(i16 signext %b) {
+define half @fp16_vmaxnm_NNNu(half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_NNNu:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vmov.f16 s2, #1.200000e+01
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI21_0
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI21_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 entry:
-  %0 = bitcast i16 %b to half
-  %cmp1 = fcmp ugt half 12., %0
-  %cond1 = select i1 %cmp1, half 12., half %0
+  %cmp1 = fcmp ugt half 12., %b
+  %cond1 = select i1 %cmp1, half 12., half %b
   %cmp2 = fcmp ugt half %cond1, 34.
   %cond2 = select i1 %cmp2, half %cond1, half 34.
   ret half %cond2
 }
 
-define half @fp16_vmaxnm_NNNuge(i16 signext %b) {
+define half @fp16_vmaxnm_NNNuge(half %b) {
 ; CHECK-LABEL: fp16_vmaxnm_NNNuge:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s2, .LCPI22_0
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vldr.16 s2, .LCPI22_1
 ; CHECK-NEXT:    vcmp.f16 s2, s0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselgt.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI22_0:
 ; CHECK-NEXT:    .short 0x5040 @ half 34
 ; CHECK-NEXT:  .LCPI22_1:
 ; CHECK-NEXT:    .short 0x5300 @ half 56
 entry:
-  %0 = bitcast i16 %b to half
-  %cmp1 = fcmp uge half 34., %0
-  %cond1 = select i1 %cmp1, half 34., half %0
+  %cmp1 = fcmp uge half 34., %b
+  %cond1 = select i1 %cmp1, half 34., half %b
   %cmp2 = fcmp uge half %cond1, 56.
   %cond2 = select i1 %cmp2, half %cond1, half 56.
   ret half %cond2
 }
 
-define half @fp16_vminmaxnm_neg0(i16 signext %a) {
+define half @fp16_vminmaxnm_neg0(half %a) {
 ; CHECK-LABEL: fp16_vminmaxnm_neg0:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s0, .LCPI23_0
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vminnm.f16 s2, s2, s0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI23_0:
 ; CHECK-NEXT:    .short 0x8000 @ half -0
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp olt half %0, -0.
-  %cond1 = select i1 %cmp1, half %0, half -0.
+  %cmp1 = fcmp olt half %a, -0.
+  %cond1 = select i1 %cmp1, half %a, half -0.
   %cmp2 = fcmp ugt half %cond1, -0.
   %cond2 = select i1 %cmp2, half %cond1, half -0.
   ret half %cond2
 }
 
-define half @fp16_vminmaxnm_e_0(i16 signext %a) {
+define half @fp16_vminmaxnm_e_0(half %a) {
 ; CHECK-LABEL: fp16_vminmaxnm_e_0:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 s0, r1
+; CHECK-NEXT:    vmov.f16 s0, r0
 ; CHECK-NEXT:    vldr.16 s2, .LCPI24_0
 ; CHECK-NEXT:    vcmp.f16 s0, #0
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s2, s0
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI24_0:
 ; CHECK-NEXT:    .short 0x0000 @ half 0
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp nsz ole half 0., %0
-  %cond1 = select i1 %cmp1, half 0., half %0
+  %cmp1 = fcmp nsz ole half 0., %a
+  %cond1 = select i1 %cmp1, half 0., half %a
   %cmp2 = fcmp nsz uge half 0., %cond1
   %cond2 = select i1 %cmp2, half 0., half %cond1
   ret half %cond2
 }
 
-define half @fp16_vminmaxnm_e_neg0(i16 signext %a) {
+define half @fp16_vminmaxnm_e_neg0(half %a) {
 ; CHECK-LABEL: fp16_vminmaxnm_e_neg0:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldr.16 s0, .LCPI25_0
-; CHECK-NEXT:    vmov.f16 s2, r1
+; CHECK-NEXT:    vmov.f16 s2, r0
 ; CHECK-NEXT:    vminnm.f16 s2, s2, s0
 ; CHECK-NEXT:    vcmp.f16 s0, s2
 ; CHECK-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NEXT:    vselge.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI25_0:
 ; CHECK-NEXT:    .short 0x8000 @ half -0
 entry:
-  %0 = bitcast i16 %a to half
-  %cmp1 = fcmp nsz ule half -0., %0
-  %cond1 = select i1 %cmp1, half -0., half %0
+  %cmp1 = fcmp nsz ule half -0., %a
+  %cond1 = select i1 %cmp1, half -0., half %a
   %cmp2 = fcmp nsz oge half -0., %cond1
   %cond2 = select i1 %cmp2, half -0., half %cond1
   ret half %cond2
 }

llvm/test/CodeGen/ARM/vecreduce-fadd-legalization-strict.ll

 ; CHECK-NEXT:    .save {r11, lr}
 ; CHECK-NEXT:    push {r11, lr}
 ; CHECK-NEXT:    bl __aeabi_f2h
 ; CHECK-NEXT:    bl __aeabi_h2f
 ; CHECK-NEXT:    vldr s0, .LCPI0_0
 ; CHECK-NEXT:    vmov s2, r0
 ; CHECK-NEXT:    vadd.f32 s0, s2, s0
 ; CHECK-NEXT:    vmov r0, s0
+; CHECK-NEXT:    bl __aeabi_f2h
 ; CHECK-NEXT:    pop {r11, lr}
 ; CHECK-NEXT:    mov pc, lr
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 0x00000000 @ float 0
   %b = call half @llvm.experimental.vector.reduce.v2.fadd.f16.v1f16(half 0.0, <1 x half> %a)
   ret half %b

llvm/test/CodeGen/ARM/vecreduce-fmul-legalization-strict.ll

 ; CHECK-NEXT:    .save {r11, lr}
 ; CHECK-NEXT:    push {r11, lr}
 ; CHECK-NEXT:    bl __aeabi_f2h
 ; CHECK-NEXT:    bl __aeabi_h2f
 ; CHECK-NEXT:    vldr s0, .LCPI0_0
 ; CHECK-NEXT:    vmov s2, r0
 ; CHECK-NEXT:    vmul.f32 s0, s2, s0
 ; CHECK-NEXT:    vmov r0, s0
+; CHECK-NEXT:    bl __aeabi_f2h
 ; CHECK-NEXT:    pop {r11, lr}
 ; CHECK-NEXT:    mov pc, lr
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 0x00000000 @ float 0
   %b = call half @llvm.experimental.vector.reduce.v2.fmul.f16.v1f16(half 0.0, <1 x half> %a)
   ret half %b

llvm/test/CodeGen/Thumb2/mve-shuffle.ll

 entry:
   %res = extractelement <4 x float> %a, i32 3
   ret float %res
 }
 
 define arm_aapcs_vfpcc half @extract_f16_0(<8 x half> %a) {
 ; CHECK-LABEL: extract_f16_0:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vstr.16 s0, [r0]
+; CHECK-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-NEXT:    bx lr
 entry:
   %res = extractelement <8 x half> %a, i32 0
   ret half %res
 }
 
 define arm_aapcs_vfpcc half @extract_f16_3(<8 x half> %a) {
 ; CHECK-LABEL: extract_f16_3:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmovx.f16 s0, s1
-; CHECK-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %res = extractelement <8 x half> %a, i32 3
   ret half %res
 }
 
 define arm_aapcs_vfpcc double @extract_f64_0(<2 x double> %a) {
 ; CHECK-LABEL: extract_f64_0:

llvm/test/CodeGen/Thumb2/mve-vdup.ll

entry:
   %outbc = bitcast <4 x i32> %out to <4 x float>
   %ext = extractelement <4 x float> %outbc, i32 2
   ret float %ext
 }
 
 define arm_aapcs_vfpcc half @vdup_f16_extract(half* %src1, half* %src2) {
 ; CHECK-LABEL: vdup_f16_extract:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vldr.16 s2, [r1]
+; CHECK-NEXT:    vldr.16 s0, [r1]
+; CHECK-NEXT:    vldr.16 s2, [r0]
 ; CHECK-NEXT:    vadd.f16 s0, s2, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = load half, half *%src1, align 2
   %1 = load half, half *%src2, align 2
   %2 = fadd half %0, %1
   %bc = bitcast half %2 to i16
   %3 = insertelement <8 x i16> undef, i16 %bc, i32 0
   %out = shufflevector <8 x i16> %3, <8 x i16> undef, <8 x i32> zeroinitializer
   %outbc = bitcast <8 x i16> %out to <8 x half>
   %ext = extractelement <8 x half> %outbc, i32 2
   ret half %ext
 }

llvm/test/CodeGen/Thumb2/mve-vecreduce-fminmax.ll

 ; CHECK-NEXT:    vmovx.f16 s0, s1
 ; CHECK-NEXT:    vminnm.f16 s4, s4, s1
 ; CHECK-NEXT:    vldr.16 s2, .LCPI3_0
 ; CHECK-NEXT:    vminnm.f16 s0, s4, s0
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vminnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI3_0:
 ; CHECK-NEXT:    .short 0x7c00 @ half +Inf
 entry:
   %z = call fast half @llvm.experimental.vector.reduce.fmin.v4f16(<4 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmin_v8f16(<8 x half> %x) {
 ; CHECK-LABEL: fmin_v8f16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmovx.f16 s4, s0
 ; CHECK-NEXT:    vmovx.f16 s6, s1
 ; CHECK-NEXT:    vminnm.f16 s4, s0, s4
 ; CHECK-NEXT:    vmovx.f16 s0, s3
 ; CHECK-NEXT:    vminnm.f16 s4, s4, s1
 ; CHECK-NEXT:    vminnm.f16 s4, s4, s6
 ; CHECK-NEXT:    vmovx.f16 s6, s2
 ; CHECK-NEXT:    vminnm.f16 s4, s4, s2
 ; CHECK-NEXT:    vminnm.f16 s4, s4, s6
 ; CHECK-NEXT:    vminnm.f16 s4, s4, s3
 ; CHECK-NEXT:    vminnm.f16 s0, s4, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = call fast half @llvm.experimental.vector.reduce.fmin.v8f16(<8 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmin_v16f16(<16 x half> %x) {
 ; CHECK-FP-LABEL: fmin_v16f16:
 ; CHECK-FP:       @ %bb.0: @ %entry
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
 ; CHECK-FP-NEXT:    vmovx.f16 s4, s0
 ; CHECK-FP-NEXT:    vmovx.f16 s6, s1
 ; CHECK-FP-NEXT:    vminnm.f16 s4, s0, s4
 ; CHECK-FP-NEXT:    vmovx.f16 s0, s3
 ; CHECK-FP-NEXT:    vminnm.f16 s4, s4, s1
 ; CHECK-FP-NEXT:    vminnm.f16 s4, s4, s6
 ; CHECK-FP-NEXT:    vmovx.f16 s6, s2
 ; CHECK-FP-NEXT:    vminnm.f16 s4, s4, s2
 ; CHECK-FP-NEXT:    vminnm.f16 s4, s4, s6
 ; CHECK-FP-NEXT:    vminnm.f16 s4, s4, s3
 ; CHECK-FP-NEXT:    vminnm.f16 s0, s4, s0
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmin_v16f16:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s4
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s0
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s10
 ; CHECK-NOFP-NEXT:    vmovx.f16 s12, s1

 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vcmp.f16 s4, s0
 ; CHECK-NOFP-NEXT:    vminnm.f16 s8, s8, s10
 ; CHECK-NOFP-NEXT:    vselgt.f16 s10, s3, s7
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vminnm.f16 s8, s8, s10
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s4
 ; CHECK-NOFP-NEXT:    vminnm.f16 s0, s8, s0
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call fast half @llvm.experimental.vector.reduce.fmin.v16f16(<16 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc double @fmin_v1f64(<1 x double> %x) {
 ; CHECK-LABEL: fmin_v1f64:

 entry:
   %z = call float @llvm.experimental.vector.reduce.fmin.v8f32(<8 x float> %x)
   ret float %z
 }
 
 define arm_aapcs_vfpcc half @fmin_v4f16_nofast(<4 x half> %x) {
 ; CHECK-FP-LABEL: fmin_v4f16_nofast:
 ; CHECK-FP:       @ %bb.0: @ %entry
-; CHECK-FP-NEXT:    vmov r1, s1
-; CHECK-FP-NEXT:    vdup.32 q1, r1
+; CHECK-FP-NEXT:    vmov r0, s1
+; CHECK-FP-NEXT:    vdup.32 q1, r0
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov.u16 r1, q0[1]
-; CHECK-FP-NEXT:    vdup.16 q1, r1
+; CHECK-FP-NEXT:    vmov.u16 r0, q0[1]
+; CHECK-FP-NEXT:    vdup.16 q1, r0
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
+; CHECK-FP-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmin_v4f16_nofast:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
-; CHECK-NOFP-NEXT:    vmov r1, s1
+; CHECK-NOFP-NEXT:    vmov r0, s1
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s0
-; CHECK-NOFP-NEXT:    vdup.32 q1, r1
+; CHECK-NOFP-NEXT:    vdup.32 q1, r0
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s4
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s10
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vcmp.f16 s4, s0
 ; CHECK-NOFP-NEXT:    vselgt.f16 s8, s10, s8
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s4
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s0
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s8
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call half @llvm.experimental.vector.reduce.fmin.v4f16(<4 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmin_v8f16_nofast(<8 x half> %x) {
 ; CHECK-FP-LABEL: fmin_v8f16_nofast:
 ; CHECK-FP:       @ %bb.0: @ %entry
 ; CHECK-FP-NEXT:    vmov.f64 d2, d1
 ; CHECK-FP-NEXT:    vmov.f32 s5, s3
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov r1, s1
-; CHECK-FP-NEXT:    vdup.32 q1, r1
+; CHECK-FP-NEXT:    vmov r0, s1
+; CHECK-FP-NEXT:    vdup.32 q1, r0
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov.u16 r1, q0[1]
-; CHECK-FP-NEXT:    vdup.16 q1, r1
+; CHECK-FP-NEXT:    vmov.u16 r0, q0[1]
+; CHECK-FP-NEXT:    vdup.16 q1, r0
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
+; CHECK-FP-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmin_v8f16_nofast:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s3
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s1
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s10
 ; CHECK-NOFP-NEXT:    vmov.f64 d2, d1

 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s4
 ; CHECK-NOFP-NEXT:    vcmp.f16 s10, s0
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s10
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s0
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s8
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call half @llvm.experimental.vector.reduce.fmin.v8f16(<8 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmin_v16f16_nofast(<16 x half> %x) {
 ; CHECK-FP-LABEL: fmin_v16f16_nofast:
 ; CHECK-FP:       @ %bb.0: @ %entry
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
 ; CHECK-FP-NEXT:    vmov.f64 d2, d1
 ; CHECK-FP-NEXT:    vmov.f32 s5, s3
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov r1, s1
-; CHECK-FP-NEXT:    vdup.32 q1, r1
+; CHECK-FP-NEXT:    vmov r0, s1
+; CHECK-FP-NEXT:    vdup.32 q1, r0
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov.u16 r1, q0[1]
-; CHECK-FP-NEXT:    vdup.16 q1, r1
+; CHECK-FP-NEXT:    vmov.u16 r0, q0[1]
+; CHECK-FP-NEXT:    vdup.16 q1, r0
 ; CHECK-FP-NEXT:    vminnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
+; CHECK-FP-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmin_v16f16_nofast:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s7
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s3
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s10
 ; CHECK-NOFP-NEXT:    vmovx.f16 s12, s1

 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s12
 ; CHECK-NOFP-NEXT:    vcmp.f16 s10, s0
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s10
 ; CHECK-NOFP-NEXT:    vcmp.f16 s8, s0
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s8
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call half @llvm.experimental.vector.reduce.fmin.v16f16(<16 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc double @fmin_v1f64_nofast(<1 x double> %x) {
 ; CHECK-LABEL: fmin_v1f64_nofast:

 ; CHECK-NEXT:    vmovx.f16 s0, s1
 ; CHECK-NEXT:    vmaxnm.f16 s4, s4, s1
 ; CHECK-NEXT:    vldr.16 s2, .LCPI39_0
 ; CHECK-NEXT:    vmaxnm.f16 s0, s4, s0
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
 ; CHECK-NEXT:    vmaxnm.f16 s0, s0, s2
-; CHECK-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 1
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:  .LCPI39_0:
 ; CHECK-NEXT:    .short 0xfc00 @ half -Inf
 entry:
   %z = call fast half @llvm.experimental.vector.reduce.fmax.v4f16(<4 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmax_v8f16(<8 x half> %x) {
 ; CHECK-LABEL: fmax_v8f16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vmovx.f16 s4, s0
 ; CHECK-NEXT:    vmovx.f16 s6, s1
 ; CHECK-NEXT:    vmaxnm.f16 s4, s0, s4
 ; CHECK-NEXT:    vmovx.f16 s0, s3
 ; CHECK-NEXT:    vmaxnm.f16 s4, s4, s1
 ; CHECK-NEXT:    vmaxnm.f16 s4, s4, s6
 ; CHECK-NEXT:    vmovx.f16 s6, s2
 ; CHECK-NEXT:    vmaxnm.f16 s4, s4, s2
 ; CHECK-NEXT:    vmaxnm.f16 s4, s4, s6
 ; CHECK-NEXT:    vmaxnm.f16 s4, s4, s3
 ; CHECK-NEXT:    vmaxnm.f16 s0, s4, s0
-; CHECK-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NEXT:    bx lr
 entry:
   %z = call fast half @llvm.experimental.vector.reduce.fmax.v8f16(<8 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmax_v16f16(<16 x half> %x) {
 ; CHECK-FP-LABEL: fmax_v16f16:
 ; CHECK-FP:       @ %bb.0: @ %entry
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
 ; CHECK-FP-NEXT:    vmovx.f16 s4, s0
 ; CHECK-FP-NEXT:    vmovx.f16 s6, s1
 ; CHECK-FP-NEXT:    vmaxnm.f16 s4, s0, s4
 ; CHECK-FP-NEXT:    vmovx.f16 s0, s3
 ; CHECK-FP-NEXT:    vmaxnm.f16 s4, s4, s1
 ; CHECK-FP-NEXT:    vmaxnm.f16 s4, s4, s6
 ; CHECK-FP-NEXT:    vmovx.f16 s6, s2
 ; CHECK-FP-NEXT:    vmaxnm.f16 s4, s4, s2
 ; CHECK-FP-NEXT:    vmaxnm.f16 s4, s4, s6
 ; CHECK-FP-NEXT:    vmaxnm.f16 s4, s4, s3
 ; CHECK-FP-NEXT:    vmaxnm.f16 s0, s4, s0
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmax_v16f16:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s4
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s0
 ; CHECK-NOFP-NEXT:    vcmp.f16 s10, s8
 ; CHECK-NOFP-NEXT:    vmovx.f16 s12, s1

 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s4
 ; CHECK-NOFP-NEXT:    vmaxnm.f16 s8, s8, s10
 ; CHECK-NOFP-NEXT:    vselgt.f16 s10, s3, s7
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vmaxnm.f16 s8, s8, s10
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s4
 ; CHECK-NOFP-NEXT:    vmaxnm.f16 s0, s8, s0
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call fast half @llvm.experimental.vector.reduce.fmax.v16f16(<16 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc double @fmax_v1f64(<1 x double> %x) {
 ; CHECK-LABEL: fmax_v1f64:

 entry:
   %z = call float @llvm.experimental.vector.reduce.fmax.v8f32(<8 x float> %x)
   ret float %z
 }
 
 define arm_aapcs_vfpcc half @fmax_v4f16_nofast(<4 x half> %x) {
 ; CHECK-FP-LABEL: fmax_v4f16_nofast:
 ; CHECK-FP:       @ %bb.0: @ %entry
-; CHECK-FP-NEXT:    vmov r1, s1
-; CHECK-FP-NEXT:    vdup.32 q1, r1
+; CHECK-FP-NEXT:    vmov r0, s1
+; CHECK-FP-NEXT:    vdup.32 q1, r0
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov.u16 r1, q0[1]
-; CHECK-FP-NEXT:    vdup.16 q1, r1
+; CHECK-FP-NEXT:    vmov.u16 r0, q0[1]
+; CHECK-FP-NEXT:    vdup.16 q1, r0
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
+; CHECK-FP-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmax_v4f16_nofast:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
-; CHECK-NOFP-NEXT:    vmov r1, s1
+; CHECK-NOFP-NEXT:    vmov r0, s1
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s0
-; CHECK-NOFP-NEXT:    vdup.32 q1, r1
+; CHECK-NOFP-NEXT:    vdup.32 q1, r0
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s4
 ; CHECK-NOFP-NEXT:    vcmp.f16 s10, s8
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s4
 ; CHECK-NOFP-NEXT:    vselgt.f16 s8, s10, s8
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s4
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s8
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s8
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call half @llvm.experimental.vector.reduce.fmax.v4f16(<4 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmax_v8f16_nofast(<8 x half> %x) {
 ; CHECK-FP-LABEL: fmax_v8f16_nofast:
 ; CHECK-FP:       @ %bb.0: @ %entry
 ; CHECK-FP-NEXT:    vmov.f64 d2, d1
 ; CHECK-FP-NEXT:    vmov.f32 s5, s3
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov r1, s1
-; CHECK-FP-NEXT:    vdup.32 q1, r1
+; CHECK-FP-NEXT:    vmov r0, s1
+; CHECK-FP-NEXT:    vdup.32 q1, r0
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov.u16 r1, q0[1]
-; CHECK-FP-NEXT:    vdup.16 q1, r1
+; CHECK-FP-NEXT:    vmov.u16 r0, q0[1]
+; CHECK-FP-NEXT:    vdup.16 q1, r0
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
+; CHECK-FP-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmax_v8f16_nofast:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s3
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s1
 ; CHECK-NOFP-NEXT:    vcmp.f16 s10, s8
 ; CHECK-NOFP-NEXT:    vmov.f64 d2, d1

 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s4
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s10
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s10
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s8
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s8
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call half @llvm.experimental.vector.reduce.fmax.v8f16(<8 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc half @fmax_v16f16_nofast(<16 x half> %x) {
 ; CHECK-FP-LABEL: fmax_v16f16_nofast:
 ; CHECK-FP:       @ %bb.0: @ %entry
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
 ; CHECK-FP-NEXT:    vmov.f64 d2, d1
 ; CHECK-FP-NEXT:    vmov.f32 s5, s3
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov r1, s1
-; CHECK-FP-NEXT:    vdup.32 q1, r1
+; CHECK-FP-NEXT:    vmov r0, s1
+; CHECK-FP-NEXT:    vdup.32 q1, r0
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vmov.u16 r1, q0[1]
-; CHECK-FP-NEXT:    vdup.16 q1, r1
+; CHECK-FP-NEXT:    vmov.u16 r0, q0[1]
+; CHECK-FP-NEXT:    vdup.16 q1, r0
 ; CHECK-FP-NEXT:    vmaxnm.f16 q0, q0, q1
-; CHECK-FP-NEXT:    vstr.16 s0, [r0]
+; CHECK-FP-NEXT:    @ kill: def $s0 killed $s0 killed $q0
 ; CHECK-FP-NEXT:    bx lr
 ;
 ; CHECK-NOFP-LABEL: fmax_v16f16_nofast:
 ; CHECK-NOFP:       @ %bb.0: @ %entry
 ; CHECK-NOFP-NEXT:    vmovx.f16 s8, s7
 ; CHECK-NOFP-NEXT:    vmovx.f16 s10, s3
 ; CHECK-NOFP-NEXT:    vcmp.f16 s10, s8
 ; CHECK-NOFP-NEXT:    vmovx.f16 s12, s1

 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s12
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s10
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s10
 ; CHECK-NOFP-NEXT:    vcmp.f16 s0, s8
 ; CHECK-NOFP-NEXT:    vmrs APSR_nzcv, fpscr
 ; CHECK-NOFP-NEXT:    vselgt.f16 s0, s0, s8
-; CHECK-NOFP-NEXT:    vstr.16 s0, [r0]
 ; CHECK-NOFP-NEXT:    bx lr
 entry:
   %z = call half @llvm.experimental.vector.reduce.fmax.v16f16(<16 x half> %x)
   ret half %z
 }
 
 define arm_aapcs_vfpcc double @fmax_v1f64_nofast(<1 x double> %x) {
 ; CHECK-LABEL: fmax_v1f64_nofast: