[ARM] Use __rt_div functions for DIVREM on Windows

[mstorsjo]

Summary

This avoids falling back to calling out to the GCC rem functions (moddi3, umoddi3) when targeting Windows.

The rt_div functions have flipped the two arguments compared to the aeabi_divmod functions.

Contrary to the existing calls to division functions (and to what MSVC does), this doesn't add any check for division by zero (why does it have to, other than the fact that MSVC does it?)

In practice, the __rt_div family of functions check that themselves though, so I'm unsure why MSVC and clang needs to include it in the calling code.

Calls to rt_div functions for division aren't merged with calls to the same function with the same parameters for the remainder, which is more wasteful than a div + mls as before, but avoids calls to moddi3.

[rengolin]

Hi Martin,

The fact that divmod-eabi.ll changes is an indication that the interaction between the Dag nodes and the values they produce is weird. I suggest you walk through the back-end and print the Dag before and after the transformations.

One thing that, unfortunately, happens is that Div/Mod patters are passed multiple times over the same nodes, as they get converted from Div/Mod to Divmod, etc. This is particular to ARM and EABI, and may be getting in the way of your lowering. I can't imagine why, since it also returns two values, so it may be some corner case no one knew about.

divmod-eabi.ll is a good first approach to the problem, but you will probably have more corner cases, so please add them to the same file as you add a windows test to it.

cheers,
--renato

lib/Target/ARM/ARMISelLowering.cpp

800 ↗

(On Diff #69841)

hasDivide makes sense, but is unrelated and may break some expectations I don't know about. I'd leave it for later, since this shouldn't hit here if the sub-arch has divide anyway.

805 ↗

(On Diff #69841)

I'd have put this in a separate if block, since the function names are *all* different.

12090 ↗

(On Diff #69841)

This is not about windows or not, it's about rt_div instead of eabi_divmod. Please, make it dependent on the RT type (not OS).

Also, this induction fiddling is confusing. The easiest thing to do is to swap the two first arguments before the loop. It's clear and precise.

[mstorsjo]

I tested to extend this by removing the current custom code for SDIV/UDIV for windows, and divmod-eabi.ll now looks better (it produces two calls to __rt_sdiv instead of three).

Previously, all divisions included a check for whether the divisor is zero (with an udf.w #349 instruction in case of a division by zero), while when I made it work in the same way as __aeabi_idivmod I don't get those.

I'll upload the updated patch for further comments and clarifications.

lib/Target/ARM/ARMISelLowering.cpp

805 ↗

(On Diff #69841)

I chose putting it here since it shares the code for all the setLibcallCallingConv below anyway, instead of duplicating all the 15 lines of shared code above and below. But you'd still prefer me to duplicate all of it including setOperationAction(), HasStandaloneRem and setLibcallCallingConv?

12090 ↗

(On Diff #69841)

How do I find the RT type here - I didn't find it within ARMSubtarget on a quick look at least.

And how would I go about to swap the two arguments? N is a const SDNode here - is it ok to make it mutable and swap the arguments there?

[mstorsjo]

Now the output from divmod-eabi.ll makes more sense (I still need to update the test references though).

[rengolin]

Given that all of these changes are related to isTargetWindows everywhere, the assumption is that this can only work for windows. Once someone wants to use rt_div on non-Windows, we'll need to come up with a isWeirdABI which Windows and and that taget are members of, just like isEABI and others. So, for now, isWindows is ok.

Just a few more inline comments, and the tests and it should be good.

lib/Target/ARM/ARMISelLowering.cpp

964–982 ↗

(On Diff #71121)

Ah, makes sense. Looks good this way.

12206–12207 ↗

(On Diff #71121)

No, don't change the SDNode, just std::swap the two arguments in Args after the loop.

[mstorsjo]

In D24076#541117, @rengolin wrote:

Just a few more inline comments, and the tests and it should be good.

Actually, it turned out to be harder than this after all. With the current diff, div+rem gets nicely merged into one rt_sdiv call, but a lone div becomes a divsi3 instead.

Given all the changes back and forth in this area in the last year (see SVN revisions 248561, 253865, 254158, 263714 or git mirror commits 64ed61ca6b, ee5418798, 937e2d588c, a944db98cd) I'm a bit hesitant to change this further without input from @compnerd. I'll update the patch to a state which works (and doesn't produce calls to divsi3 or similar), but which doesn't merge the div+rem calls in all cases, and write a bug report for the actual issue that I'm trying to work around (LLVM producing calls to moddi3 which is unavailable on windows).

[mstorsjo]

Updated to a working patch

[mstorsjo]

Filed a plain bug report on the original issue at https://llvm.org/bugs/show_bug.cgi?id=30378 since it seems like my attempt at this won't be completable easily.

test/CodeGen/ARM/divmod-eabi.ll

159 ↗

(On Diff #71309)

Here, the old test reference even shows a call to __moddi3, which is not available on ARM/Windows.

[rengolin]

Right, the change now looks good, but I'll let @compnerd approve, since I'm not that well versed in the Windows ABI.

[mstorsjo]

In D24076#560523, @rengolin wrote:

Right, the change now looks good, but I'll let @compnerd approve, since I'm not that well versed in the Windows ABI.

FWIW, one of the remaining issues is that I don't (currently) add the DBZCHK (division by zero check) before the __rt_div calls for divrem or rem. I can add that as well (I got a patch for it locally). But IMO the question remains whether we should add that at all. MSVC does add it (both for div and rem, which are calls to the same function), but the actual implementations of the functions also do similar checks internally, so it feels redundant. But I guess the ABI of those functions mandate that the caller checks it.

The current implementation of DBZCHK also has got an issue, that it issues a CBZ, but later passes may move the branch target might out of range, see https://llvm.org/bugs/show_bug.cgi?id=30356 and the function EmitLowered__dbzchk.

If we can remove DBZCHK altogether, it'd be easy, but I'm not sure how to fix the current one.

[mstorsjo]

Added DBZCHK for the divrem/rem calls as well.

[compnerd]

lib/Target/ARM/ARMISelLowering.cpp

984 ↗

(On Diff #73448)

Wow, this is getting unwieldy. I wonder if its easier to negate the condition.

989–1007 ↗

(On Diff #73448)

Not your fault since you are replicating what is there, but I think that refactoring this a slight bit will be nicer:

for (const auto &LC : {RTLIB::SDIVREM_I8, RTLIB::SDIVREM_I16, RTLIB::SDIVREM_I32})
  setLibcallName(LC, Subtarget->isTargetWindows() ? "__rt_sdiv" : "__aeabi_idivmod");
setLibcallName(RTLIB::SDIVREM_I64, Subtarget->isTargetWindows() ? "__rt_sdiv64" : "__aeabi_ldivmod");
for (const auto &LC : {RTLIB::UDIVREM_I8, RTLIB::UDIVREM_I16, RTLIB::UDIVREM_I32})
  setLibcallName(LC, Subtarget->isTargetWindows() ? "__rt_udiv" : "__aeabi_uidivmod");
setLibcallName(RTLIB::UDIVREM_I64, Subtarget->isTargetWindows() ? "__rt_udiv64" : " __aeabi_uldivmod");

12520 ↗

(On Diff #73448)

Sink this inside the if condition, as it is unused otherwise.

12532 ↗

(On Diff #73448)

How about doing a trivial static function ExtractDenominator or even better, create a WinDBZCheckDenominator to handle the duplication of the DBZ check.

static SDValue WinDBZCheckDenominator(SDNode *N, SDValue InChain) {
  SDLoc DL(N);
  Value *Op = N->getOperand(1);
  if (N->getValueType(0) == MVT::32)
    return DAG.getNode(ARMISD::WIN__DBZCHK, DL, MVT::Other, InChain, Op);
  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i32, Op),
                           DAG.getConstant(0, DL, MVT::i32));
  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i32, Op),
                           DAG.getConstant(1, DL, MVT::i32));
  return DAG.getNode(ARMISD::WIN__DBZCHK, DL, MVT::Other, InChain,
                     DAG.getNode(ISD::OR, DL, MVT::i32, Lo, Hi));
}

test/CodeGen/ARM/Windows/dbzchk.ll

185 ↗

(On Diff #73448)

Why are you removing the check for the actual dbzchk codegen?

[mstorsjo]

Thanks for the comments, I'll update the patch accordingly.

test/CodeGen/ARM/Windows/dbzchk.ll

185 ↗

(On Diff #73448)

Sorry about that, that change is from an earlier version of the patch where some of the DBZCHKs vanished (when I hadn't added the new ones into REM and DIVREM). I restored this hunk (and some of the ones above as well).

[mstorsjo]

Updated based on the comments.

[compnerd]

test/CodeGen/ARM/Windows/dbzchk.ll

144 ↗

(On Diff #73686)

This is amazing. Glad to see this improved.

test/CodeGen/ARM/divmod-eabi.ll

45 ↗

(On Diff #73686)

Hmm, it would be nice to see that we can still generate the hardware division. But, I don't think that we need to hold up on that.

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

if (Subtarget->isTargetWindows() && !Subtarget->hasDivide()) {
     setOperationAction(ISD::SDIV, MVT::i64, Custom);
     setOperationAction(ISD::UDIV, MVT::i64, Custom);
   }
 
   setOperationAction(ISD::SREM,  MVT::i32, Expand);
   setOperationAction(ISD::UREM,  MVT::i32, Expand);
   // Register based DivRem for AEABI (RTABI 4.2)
   if (Subtarget->isTargetAEABI() || Subtarget->isTargetAndroid() ||
-      Subtarget->isTargetGNUAEABI() || Subtarget->isTargetMuslAEABI()) {
+      Subtarget->isTargetGNUAEABI() || Subtarget->isTargetMuslAEABI() ||
+      Subtarget->isTargetWindows()) {
     setOperationAction(ISD::SREM, MVT::i64, Custom);
     setOperationAction(ISD::UREM, MVT::i64, Custom);
     HasStandaloneRem = false;
 
-    setLibcallName(RTLIB::SDIVREM_I8,  "__aeabi_idivmod");
-    setLibcallName(RTLIB::SDIVREM_I16, "__aeabi_idivmod");
-    setLibcallName(RTLIB::SDIVREM_I32, "__aeabi_idivmod");
-    setLibcallName(RTLIB::SDIVREM_I64, "__aeabi_ldivmod");
-    setLibcallName(RTLIB::UDIVREM_I8,  "__aeabi_uidivmod");
-    setLibcallName(RTLIB::UDIVREM_I16, "__aeabi_uidivmod");
-    setLibcallName(RTLIB::UDIVREM_I32, "__aeabi_uidivmod");
-    setLibcallName(RTLIB::UDIVREM_I64, "__aeabi_uldivmod");
+    for (const auto &LC :
+         {RTLIB::SDIVREM_I8, RTLIB::SDIVREM_I16, RTLIB::SDIVREM_I32})
+      setLibcallName(LC, Subtarget->isTargetWindows() ? "__rt_sdiv"
+                                                      : "__aeabi_idivmod");
+    setLibcallName(RTLIB::SDIVREM_I64, Subtarget->isTargetWindows()
+                                           ? "__rt_sdiv64"
+                                           : "__aeabi_ldivmod");
+    for (const auto &LC :
+         {RTLIB::UDIVREM_I8, RTLIB::UDIVREM_I16, RTLIB::UDIVREM_I32})
+      setLibcallName(LC, Subtarget->isTargetWindows() ? "__rt_udiv"
+                                                      : "__aeabi_uidivmod");
+    setLibcallName(RTLIB::UDIVREM_I64, Subtarget->isTargetWindows()
+                                           ? "__rt_udiv64"
+                                           : " __aeabi_uldivmod");
 
     setLibcallCallingConv(RTLIB::SDIVREM_I8, CallingConv::ARM_AAPCS);
     setLibcallCallingConv(RTLIB::SDIVREM_I16, CallingConv::ARM_AAPCS);
     setLibcallCallingConv(RTLIB::SDIVREM_I32, CallingConv::ARM_AAPCS);
     setLibcallCallingConv(RTLIB::SDIVREM_I64, CallingConv::ARM_AAPCS);
     setLibcallCallingConv(RTLIB::UDIVREM_I8, CallingConv::ARM_AAPCS);
     setLibcallCallingConv(RTLIB::UDIVREM_I16, CallingConv::ARM_AAPCS);
     setLibcallCallingConv(RTLIB::UDIVREM_I32, CallingConv::ARM_AAPCS);
SDValue ARMTargetLowering::LowerDIV_Windows(SDValue Op, SelectionDAG &DAG,
   SDLoc dl(Op);
 
   SDValue DBZCHK = DAG.getNode(ARMISD::WIN__DBZCHK, dl, MVT::Other,
                                DAG.getEntryNode(), Op.getOperand(1));
 
   return LowerWindowsDIVLibCall(Op, DAG, Signed, DBZCHK);
 }
 
+static SDValue WinDBZCheckDenominator(SelectionDAG &DAG, SDNode *N, SDValue InChain) {
+  SDLoc DL(N);
+  SDValue Op = N->getOperand(1);
+  if (N->getValueType(0) == MVT::i32)
+    return DAG.getNode(ARMISD::WIN__DBZCHK, DL, MVT::Other, InChain, Op);
+  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i32, Op,
+                           DAG.getConstant(0, DL, MVT::i32));
+  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, DL, MVT::i32, Op,
+                           DAG.getConstant(1, DL, MVT::i32));
+  return DAG.getNode(ARMISD::WIN__DBZCHK, DL, MVT::Other, InChain,
+                     DAG.getNode(ISD::OR, DL, MVT::i32, Lo, Hi));
+}
+
 void ARMTargetLowering::ExpandDIV_Windows(
     SDValue Op, SelectionDAG &DAG, bool Signed,
     SmallVectorImpl<SDValue> &Results) const {
   const auto &DL = DAG.getDataLayout();
   const auto &TLI = DAG.getTargetLoweringInfo();
 
   assert(Op.getValueType() == MVT::i64 &&
          "unexpected type for custom lowering DIV");
   SDLoc dl(Op);
 
-  SDValue Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, dl, MVT::i32, Op.getOperand(1),
-                           DAG.getConstant(0, dl, MVT::i32));
-  SDValue Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, dl, MVT::i32, Op.getOperand(1),
-                           DAG.getConstant(1, dl, MVT::i32));
-  SDValue Or = DAG.getNode(ISD::OR, dl, MVT::i32, Lo, Hi);
-
-  SDValue DBZCHK =
-      DAG.getNode(ARMISD::WIN__DBZCHK, dl, MVT::Other, DAG.getEntryNode(), Or);
+  SDValue DBZCHK = WinDBZCheckDenominator(DAG, Op.getNode(), DAG.getEntryNode());
 
   SDValue Result = LowerWindowsDIVLibCall(Op, DAG, Signed, DBZCHK);
 
   SDValue Lower = DAG.getNode(ISD::TRUNCATE, dl, MVT::i32, Result);
   SDValue Upper = DAG.getNode(ISD::SRL, dl, MVT::i64, Result,
                               DAG.getConstant(32, dl, TLI.getPointerTy(DL)));
   Upper = DAG.getNode(ISD::TRUNCATE, dl, MVT::i32, Upper);
 
static RTLIB::Libcall getDivRemLibcall(
   case MVT::i16: LC = isSigned ? RTLIB::SDIVREM_I16 : RTLIB::UDIVREM_I16; break;
   case MVT::i32: LC = isSigned ? RTLIB::SDIVREM_I32 : RTLIB::UDIVREM_I32; break;
   case MVT::i64: LC = isSigned ? RTLIB::SDIVREM_I64 : RTLIB::UDIVREM_I64; break;
   }
   return LC;
 }
 
 static TargetLowering::ArgListTy getDivRemArgList(
-    const SDNode *N, LLVMContext *Context) {
+    const SDNode *N, LLVMContext *Context, const ARMSubtarget *Subtarget) {
   assert((N->getOpcode() == ISD::SDIVREM || N->getOpcode() == ISD::UDIVREM ||
           N->getOpcode() == ISD::SREM    || N->getOpcode() == ISD::UREM) &&
          "Unhandled Opcode in getDivRemArgList");
   bool isSigned = N->getOpcode() == ISD::SDIVREM ||
                   N->getOpcode() == ISD::SREM;
   TargetLowering::ArgListTy Args;
   TargetLowering::ArgListEntry Entry;
   for (unsigned i = 0, e = N->getNumOperands(); i != e; ++i) {
     EVT ArgVT = N->getOperand(i).getValueType();
     Type *ArgTy = ArgVT.getTypeForEVT(*Context);
     Entry.Node = N->getOperand(i);
     Entry.Ty = ArgTy;
     Entry.isSExt = isSigned;
     Entry.isZExt = !isSigned;
     Args.push_back(Entry);
   }
+  if (Subtarget->isTargetWindows() && Args.size() >= 2)
+    std::swap(Args[0], Args[1]);
   return Args;
 }
 
 SDValue ARMTargetLowering::LowerDivRem(SDValue Op, SelectionDAG &DAG) const {
   assert((Subtarget->isTargetAEABI() || Subtarget->isTargetAndroid() ||
-          Subtarget->isTargetGNUAEABI() || Subtarget->isTargetMuslAEABI()) &&
+          Subtarget->isTargetGNUAEABI() || Subtarget->isTargetMuslAEABI() ||
+          Subtarget->isTargetWindows()) &&
          "Register-based DivRem lowering only");
   unsigned Opcode = Op->getOpcode();
   assert((Opcode == ISD::SDIVREM || Opcode == ISD::UDIVREM) &&
          "Invalid opcode for Div/Rem lowering");
   bool isSigned = (Opcode == ISD::SDIVREM);
   EVT VT = Op->getValueType(0);
   Type *Ty = VT.getTypeForEVT(*DAG.getContext());
   SDLoc dl(Op);
if (Subtarget->hasDivide() && Op->getValueType(0).isSimple() &&
     return DAG.getNode(ISD::MERGE_VALUES, dl, DAG.getVTList(VT, VT), Values);
   }
 
   RTLIB::Libcall LC = getDivRemLibcall(Op.getNode(),
                                        VT.getSimpleVT().SimpleTy);
   SDValue InChain = DAG.getEntryNode();
 
   TargetLowering::ArgListTy Args = getDivRemArgList(Op.getNode(),
-                                                    DAG.getContext());
+                                                    DAG.getContext(),
+                                                    Subtarget);
 
   SDValue Callee = DAG.getExternalSymbol(getLibcallName(LC),
                                          getPointerTy(DAG.getDataLayout()));
 
   Type *RetTy = (Type*)StructType::get(Ty, Ty, nullptr);
 
+  if (Subtarget->isTargetWindows())
+    InChain = WinDBZCheckDenominator(DAG, Op.getNode(), InChain);
+
   TargetLowering::CallLoweringInfo CLI(DAG);
   CLI.setDebugLoc(dl).setChain(InChain)
     .setCallee(getLibcallCallingConv(LC), RetTy, Callee, std::move(Args))
     .setInRegister().setSExtResult(isSigned).setZExtResult(!isSigned);
 
   std::pair<SDValue, SDValue> CallInfo = LowerCallTo(CLI);
   return CallInfo.first;
 }
SDValue ARMTargetLowering::LowerREM(SDNode *N, SelectionDAG &DAG) const {
   RetTyParams.push_back(RetTyElement);
   RetTyParams.push_back(RetTyElement);
   ArrayRef<Type*> ret = ArrayRef<Type*>(RetTyParams);
   Type *RetTy = StructType::get(*DAG.getContext(), ret);
 
   RTLIB::Libcall LC = getDivRemLibcall(N, N->getValueType(0).getSimpleVT().
                                                              SimpleTy);
   SDValue InChain = DAG.getEntryNode();
-  TargetLowering::ArgListTy Args = getDivRemArgList(N, DAG.getContext());
+  TargetLowering::ArgListTy Args = getDivRemArgList(N, DAG.getContext(),
+                                                    Subtarget);
   bool isSigned = N->getOpcode() == ISD::SREM;
   SDValue Callee = DAG.getExternalSymbol(getLibcallName(LC),
                                          getPointerTy(DAG.getDataLayout()));
 
+  if (Subtarget->isTargetWindows())
+    InChain = WinDBZCheckDenominator(DAG, N, InChain);
+
   // Lower call
   CallLoweringInfo CLI(DAG);
   CLI.setChain(InChain)
      .setCallee(CallingConv::ARM_AAPCS, RetTy, Callee, std::move(Args))
      .setSExtResult(isSigned).setZExtResult(!isSigned).setDebugLoc(SDLoc(N));
   std::pair<SDValue, SDValue> CallResult = LowerCallTo(CLI);
 
   // Return second (rem) result operand (first contains div)

llvm/trunk/test/CodeGen/ARM/Windows/dbzchk.ll

 
 ; CHECK-CFG-ASM-LABEL: h:
 ; CHECK-CFG-ASM: cbz r{{[0-9]}}, .LBB2_2
 ; CHECK-CFG-ASM: b .LBB2_4
 ; CHECK-CFG-ASM-LABEL: .LBB2_2:
 ; CHECK-CFG-ASM-NEXT: udf.w #249
 ; CHECK-CFG-ASM-LABEL: .LBB2_4:
 ; CHECK-CFG-ASM: bl __rt_udiv
-; CHECK-CFG-ASM: pop.w {{{.*}}, r11, pc}
+; CHECK-CFG-ASM: pop.w {r11, pc}
 
 ; RUN: llc -O0 -mtriple thumbv7--windows-itanium -verify-machineinstrs -filetype asm -o - %s | FileCheck %s -check-prefix CHECK-WIN__DBZCHK
 
 ; long k(void);
 ; int l(void);
 ; int j(int i) {
 ;   if (l() == -1)
 ;     return 0;

llvm/trunk/test/CodeGen/ARM/divmod-eabi.ll

 ; EABI: mov [[div:r[0-9]+]], r0
 ; EABI: mov [[rem:r[0-9]+]], r1
 ; DARWIN: ___divsi3
 ; DARWIN: mov [[div:r[0-9]+]], r0
 ; DARWIN: __modsi3
 ; DARWIN-DEFAULT: add [[sum:r[0-9]+]], r0, [[div]]
 ; DARWIN-O0: mov [[rem:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
-; WINDOWS-DEFAULT: mls [[rem:r[0-9]+]], r0,
-; WINDOWS-DEFAULT: adds [[sum:r[0-9]+]], [[rem]], r0
-; WINDOWS-O0: mov [[div:r[0-9]+]], r0
-; WINDOWS-O0: mls [[rem:r[0-9]+]], [[div]],
+; WINDOWS: __rt_sdiv
+; WINDOWS-DEFAULT: add [[sum:r[0-9]+]], r1
+; WINDOWS-O0: mov [[rem:r[0-9]+]], r1
   %rem8 = srem i32 %conv1, %conv
 ; EABI: __aeabi_idivmod
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem1:r[0-9]+]], r0,
   %add = add nsw i32 %rem, %div
   %add13 = add nsw i32 %add, %rem8
   %conv14 = trunc i32 %add13 to i16
 ; EABI: add r0{{.*}}r1
 ; EABI: sxth r0, r0
 ; DARWIN-DEFAULT: add [[res:r[0-9]+]], [[sum]], r0
 ; DARWIN-O0: add [[sum:r[0-9]+]], [[rem]], [[div]]
 ; DARWIN-O0: add [[res:r[0-9]+]], [[sum]], r0
 ; DARWIN: sxth r0, [[res]]
-; WINDOWS-O0: adds [[sum:r[0-9]+]], [[rem]], [[div]]
-; WINDOWS: add [[rem1]], [[sum]]
-; WINDOWS: sxth [[res:r[0-9]+]], [[rem1]]
+; WINDOWS-DEFAULT: adds [[sum1:r[0-9]+]], [[sum]], r1
+; WINDOWS-O0: adds [[sum:r[0-9]+]], [[rem]],
+; WINDOWS-O0: add [[sum1:r[0-9]+]], r1
+; WINDOWS: sxth [[res:r[0-9]+]], [[sum1]]
   ret i16 %conv14
 }
 
 define i32 @f32(i32 %a, i32 %b) {
 ; EABI-LABEL: f32:
 ; DARWIN-LABEL: f32:
 ; WINDOWS-LABEL: f32:
 entry:
   %div = sdiv i32 %a, %b
   %rem = srem i32 %a, %b
 ; EABI: __aeabi_idivmod
 ; EABI: mov [[div:r[0-9]+]], r0
 ; EABI: mov [[rem:r[0-9]+]], r1
 ; DARWIN: ___divsi3
 ; DARWIN: mov [[div:r[0-9]+]], r0
 ; DARWIN: __modsi3
 ; DARWIN-DEFAULT: add [[sum:r[0-9]+]], r0, [[div]]
 ; DARWIN-O0: mov [[rem:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
 ; WINDOWS: mov [[div:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem:r[0-9]+]], r0,
-; WINDOWS-DEFAULT: add [[div]], [[rem]]
+; WINDOWS-DEFAULT: add [[div]], r1
   %rem1 = srem i32 %b, %a
 ; EABI: __aeabi_idivmod
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem1:r[0-9]+]], r0,
   %add = add nsw i32 %rem, %div
   %add2 = add nsw i32 %add, %rem1
 ; EABI: add r0{{.*}}r1
 ; DARWIN-DEFAULT: add r0, [[sum]], r0
 ; DARWIN-O0: add [[sum:r[0-9]+]], [[rem]], [[div]]
 ; DARWIN-O0: add [[res:r[0-9]+]], [[sum]], r0
-; WINDOWS-DEFAULT: add [[rem1]], [[div]]
+; WINDOWS-DEFAULT: adds r0, [[div]], r1
 ; WINDOWS-O0: adds [[sum:r[0-9]+]], [[rem]], [[div]]
-; WINDOWS-O0: add [[rem1]], [[sum]]
+; WINDOWS-O0: add [[sum]], r1
   ret i32 %add2
 }
 
 define i32 @uf(i32 %a, i32 %b) {
 ; EABI-LABEL: uf:
 ; DARWIN-LABEL: uf:
 ; WINDOWS-LABEL: uf:
 entry:
   %div = udiv i32 %a, %b
   %rem = urem i32 %a, %b
 ; EABI: __aeabi_uidivmod
 ; DARWIN: ___udivsi3
 ; DARWIN: mov [[div:r[0-9]+]], r0
 ; DARWIN: __umodsi3
 ; DARWIN-DEFAULT: add [[sum:r[0-9]+]], r0, [[div]]
 ; DARWIN-O0: mov [[rem:r[0-9]+]], r0
 ; WINDOWS: __rt_udiv
 ; WINDOWS: mov [[div:r[0-9]+]], r0
 ; WINDOWS: __rt_udiv
-; WINDOWS: mls [[rem:r[0-9]+]], r0,
-; WINDOWS-DEFAULT: add [[div]], [[rem]]
+; WINDOWS-DEFAULT: add [[div]], r1
   %rem1 = urem i32 %b, %a
 ; EABI: __aeabi_uidivmod
 ; DARWIN: __umodsi3
 ; WINDOWS: __rt_udiv
-; WINDOWS: mls [[rem1:r[0-9]+]], r0,
   %add = add nuw i32 %rem, %div
   %add2 = add nuw i32 %add, %rem1
 ; EABI: add r0{{.*}}r1
 ; DARWIN-DEFAULT: add r0, [[sum]], r0
 ; DARWIN-O0: add [[sum:r[0-9]+]], [[rem]], [[div]]
 ; DARWIN-O0: add [[res:r[0-9]+]], [[sum]], r0
-; WINDOWS-DEFAULT: add [[rem1]], [[div]]
-; WINDOWS-O0: adds [[sum:r[0-9]+]], [[rem]], [[div]]
-; WINDOWS-O0: add [[rem1]], [[sum]]
+; WINDOWS-DEFAULT: adds [[sum:r[0-9]+]], [[div]], r1
+; WINDOWS-O0: adds [[sum:r[0-9]+]],
+; WINDOWS-O0: add [[sum]], r1
   ret i32 %add2
 }
 
 define i64 @longf(i64 %a, i64 %b) {
 ; EABI-LABEL: longf:
 ; DARWIN-LABEL: longf:
 ; WINDOWS-LABEL: longf:
 entry:
   %div = sdiv i64 %a, %b
   %rem = srem i64 %a, %b
 ; EABI: __aeabi_ldivmod
 ; EABI-NEXT: adds r0
 ; EABI-NEXT: adc r1
 ; EABI-NOT: __aeabi_ldivmod
 ; DARWIN: ___divdi3
 ; DARWIN: mov [[div1:r[0-9]+]], r0
 ; DARWIN: mov [[div2:r[0-9]+]], r1
 ; DARWIN: __moddi3
 ; WINDOWS: __rt_sdiv64
-; WINDOWS: mov [[div1:r[0-9]+]], r0
-; WINDOWS: mov [[div2:r[0-9]+]], r1
-; WINDOWS: __moddi3
   %add = add nsw i64 %rem, %div
 ; DARWIN: adds r0{{.*}}[[div1]]
 ; DARWIN: adc r1{{.*}}[[div2]]
-; WINDOWS: adds.w r0, r0, [[div1]]
-; WINDOWS: adc.w r1, r1, [[div2]]
+; WINDOWS: adds r0, r0, r2
+; WINDOWS: adcs r1, r3
   ret i64 %add
 }
 
 define i16 @shortf(i16 %a, i16 %b) {
 ; EABI-LABEL: shortf:
 ; DARWIN-LABEL: shortf:
 ; WINDOWS-LABEL: shortf:
 entry:
   %div = sdiv i16 %a, %b
   %rem = srem i16 %a, %b
 ; EABI: __aeabi_idivmod
 ; DARWIN: ___divsi3
 ; DARWIN: mov [[div1:r[0-9]+]], r0
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
 ; WINDOWS: mov [[div:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem:r[0-9]+]], r0,
   %add = add nsw i16 %rem, %div
 ; EABI: add r0, r1
 ; DARWIN: add r0{{.*}}[[div1]]
-; WINDOWS: add [[rem]], [[div]]
+; WINDOWS: adds r0, r1, [[div]]
   ret i16 %add
 }
 
 define i32 @g1(i32 %a, i32 %b) {
 ; EABI-LABEL: g1:
 ; DARWIN-LABEL: g1:
 ; WINDOWS-LABEL: g1:
 entry:
   %div = sdiv i32 %a, %b
   %rem = srem i32 %a, %b
 ; EABI: __aeabi_idivmod
 ; DARWIN: ___divsi3
 ; DARWIN: mov [[sum:r[0-9]+]], r0
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
 ; WINDOWS: mov [[div:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem:r[0-9]+]], r0,
   %add = add nsw i32 %rem, %div
 ; EABI: add r0{{.*}}r1
 ; DARWIN: add r0{{.*}}[[sum]]
-; WINDOWS: add [[rem]], [[div]]
+; WINDOWS: adds r0, r1, [[div]]
   ret i32 %add
 }
 
 ; On both Darwin and Gnu, this is just a call to __modsi3
 define i32 @g2(i32 %a, i32 %b) {
 ; EABI-LABEL: g2:
 ; DARWIN-LABEL: g2:
 ; WINDOWS-LABEL: g2:
 entry:
   %rem = srem i32 %a, %b
 ; EABI: __aeabi_idivmod
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
   ret i32 %rem
 ; EABI: mov r0, r1
-; WINDOWS: mls r0, r0,
+; WINDOWS: mov  r0, r1
 }
 
 define i32 @g3(i32 %a, i32 %b) {
 ; EABI-LABEL: g3:
 ; DARWIN-LABEL: g3:
 ; WINDOWS-LABEL: g3:
 entry:
   %rem = srem i32 %a, %b
 ; EABI: __aeabi_idivmod
 ; EABI: mov [[mod:r[0-9]+]], r1
 ; DARWIN: __modsi3
 ; DARWIN: mov [[sum:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem:r[0-9]+]], r0,
+; WINDOWS: mov [[rem:r[0-9]+]], r1
   %rem1 = srem i32 %b, %rem
 ; EABI: __aeabi_idivmod
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem1:r[0-9]+]], r0,
   %add = add nsw i32 %rem1, %rem
 ; EABI: add r0, r1, [[mod]]
 ; DARWIN: add r0{{.*}}[[sum]]
-; WINDOWS: add [[rem1]], [[rem]]
+; WINDOWS: adds r0, r1, [[rem]]
   ret i32 %add
 }
 
 define i32 @g4(i32 %a, i32 %b) {
 ; EABI-LABEL: g4:
 ; DARWIN-LABEL: g4:
 ; WINDOWS-LABEL: g4:
 entry:
   %div = sdiv i32 %a, %b
 ; EABI: __aeabi_idiv{{$}}
 ; EABI: mov [[div:r[0-9]+]], r0
 ; DARWIN: ___divsi3
 ; DARWIN: mov [[sum:r[0-9]+]], r0
 ; WINDOWS: __rt_sdiv
 ; WINDOWS: mov [[div:r[0-9]+]], r0
   %rem = srem i32 %b, %div
 ; EABI: __aeabi_idivmod
 ; DARWIN: __modsi3
 ; WINDOWS: __rt_sdiv
-; WINDOWS: mls [[rem:r[0-9]+]], r0,
   %add = add nsw i32 %rem, %div
 ; EABI: add r0, r1, [[div]]
 ; DARWIN: add r0{{.*}}[[sum]]
-; WINDOWS: add [[rem]], [[div]]
+; WINDOWS: adds r0, r1, [[div]]
   ret i32 %add
 }