Disable DAGCombine for -O0 and optnone

[iid_iunknown]

Summary

Yet another attempt to disable DAGCombine for -O0 and optnone for better debugging experience.

Some additional info on the subject can be found in D7181, D8614 and in PR22346.

The previous attempt has been reverted due to the failures in instruction selection, which depends on the transforms done from DAGCombine. DAGCombine calls LegalizeOp that performs transformations such as ConstantPool -> <TargetConstantPool + Wrapper> and others. Their absence breaks instruction selection.

The patch disables combine leaving LegalizeOp enabled to satisfy instruction selection. The dependent tests were adjusted to reflect the changes in the code generated w/o combine.
It also has a fix in ARMISelDAGToDAG.cpp that prevents llc from crashing when compiling the CodeGen/ARM/2010-05-18-LocalAllocCrash.ll test with DAGCombine disabled, as described in PR22346.

I might be missing some important points so would like to have a sharp community's eye on this and hear any concerns / criticism about the patch.

*UPDATE*
The new version of the patch fixes broken FMA lowering on X86 (please find the comment from Michael Kuperstein).

Previously X86 FMA lowering was done at the combine step thereby requiring DAGCombine to get proper target specific FMA nodes. The patch moves this logic to the Legalize step to make FMA lowering independent on DAGCombine. It also adds a new CodeGen/X86/fma-no-dag-combine.ll test.

This broke fma_patterns.ll and other tests that expected operations like a * b - c to be combined into (fma a, b, (fneg c)). The patch was producing vfmadd instead of the expected vfmsub. The reason for this is that FNEG lowering is called before the added FMA lowering. FNEG gets transformed into other nodes making FMA lowering unable to match the pattern for vfmsub. To overcome this, the patch skips FNEG lowering if it is an FMA operand (LowerFABSorFNEG in X86ISelLowering.cpp).

[probinson]

I had one inline comment about how to test for the correct conditions.

I think the updated 'dag-optnone.ll' already covers everything that's in 'fastmath-optnone.ll' so you don't need to add the latter as a new test.

I'm not qualified to review all the other test changes, someone else will need to do that.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

1204

It should be sufficient to test OptLevel here. OptimizeNone should have reset OptLevel for the current function, so you should not need to check the attribute directly.

[iid_iunknown]

In D9992#178684, @probinson wrote:

Thank you for your feedback Paul!
I will update the patch according to your remarks. New version will also contain a fix for the CodeGen/ARM/2010-05-18-LocalAllocCrash.ll test, which crashes without DAGCombine as you noticed in PR22346. Just missed it somehow when creating the patch file.

[iid_iunknown]

Corrections according to the remarks from Paul.
Fix for CodeGen/ARM/2010-05-18-LocalAllocCrash.ll crash (ARMISelDAGToDAG.cpp).

[mkuper]

So, there's something that I'm fairly sure will break on x86, and it doesn't seem to be covered by a test. :-(
The x86 PerformFMACombine actually performs an essential part of isel - it lowers a target-independent ISD node into a target-dependent one.

Now, for cases where the FMA comes from an x86 intrinsic, it's not an issue, since we never get the target-independent ISD. For cases where the FMA itself is constructed by a DAGCombine (which is what fma_patterns tests), it's not an issue either, because the FMA never gets formed. But it will be hit when the FMA comes from a target-independent intrinsic.

TL;DR:
Try compiling this with -mattr=+fma -O0 with your patch, I expect it to fail:

declare <4 x float> @llvm.fma.v4f32(<4 x float> %a, <4 x float>  %b, <4 x float>  %c)

define <4 x float> @test_fma1(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2) {
  %res = call <4 x float> @llvm.fma.v4f32(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2)
  ret <4 x float> %res
}

[asl]

In D9992#179746, @mkuper wrote:

So, there's something that I'm fairly sure will break on x86, and it doesn't seem to be covered by a test. :-(
The x86 PerformFMACombine actually performs an essential part of isel - it lowers a target-independent ISD node into a target-dependent one.

Now, for cases where the FMA comes from an x86 intrinsic, it's not an issue, since we never get the target-independent ISD. For cases where the FMA itself is constructed by a DAGCombine (which is what fma_patterns tests), it's not an issue either, because the FMA never gets formed. But it will be hit when the FMA comes from a target-independent intrinsic.

TL;DR:
Try compiling this with -mattr=+fma -O0 with your patch, I expect it to fail:

declare <4 x float> @llvm.fma.v4f32(<4 x float> %a, <4 x float>  %b, <4 x float>  %c)

define <4 x float> @test_fma1(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2) {
  %res = call <4 x float> @llvm.fma.v4f32(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2)
  ret <4 x float> %res
}

Wow... then it should be a part of Legalize step.

[iid_iunknown]

In D9992#179746, @mkuper wrote:

So, there's something that I'm fairly sure will break on x86, and it doesn't seem to be covered by a test. :-(
The x86 PerformFMACombine actually performs an essential part of isel - it lowers a target-independent ISD node into a target-dependent one.

Now, for cases where the FMA comes from an x86 intrinsic, it's not an issue, since we never get the target-independent ISD. For cases where the FMA itself is constructed by a DAGCombine (which is what fma_patterns tests), it's not an issue either, because the FMA never gets formed. But it will be hit when the FMA comes from a target-independent intrinsic.

TL;DR:
Try compiling this with -mattr=+fma -O0 with your patch, I expect it to fail:

declare <4 x float> @llvm.fma.v4f32(<4 x float> %a, <4 x float>  %b, <4 x float>  %c)

define <4 x float> @test_fma1(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2) {
  %res = call <4 x float> @llvm.fma.v4f32(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2)
  ret <4 x float> %res
}

Thank you Michael! You are right - your test fails with the patch.
The next version of the patch moves FMA lowering on X86 from combine to the Legalize step, thus it no longer depends on DAGCombine availability. It also adds a test you provided. I will update the review summary with the description of the changes made. Would you be able to take a look, please?

[iid_iunknown]

Fix for broken FMA lowering on X86 according to the comment from Michael Kuperstein.

[mkuper]

I'm not sure it's a good idea to move the entire combine into lowering.
The FNEG treatment is an actual combine. And I'm not 100% it will work correctly now (e.g. in cases where there are multiple users for the FNEG).

It would probably be better to leave the combine as is, but add a trivial "safety-net" lowering for ISD::FMA -> X86ISD::FMADD.

[mkuper]

Adding Elena, who knows this code better than I do.

[delena]

All lit tests should pass in the -O0 mode. The code may be not optimal, but the compilation should not fail and the generated code should correct.

lib/Target/X86/X86ISelLowering.cpp

17616

You should not do the work of DAG-combiner in lowering. Lowering should deal with one SDNode.
Please put the FMA-combine back and just translate ISD::FMA to X86ISD::FMADD here.

[iid_iunknown]

In D9992#182775, @mkuper wrote:

I'm not sure it's a good idea to move the entire combine into lowering.
The FNEG treatment is an actual combine. And I'm not 100% it will work correctly now (e.g. in cases where there are multiple users for the FNEG).

It would probably be better to leave the combine as is, but add a trivial "safety-net" lowering for ISD::FMA -> X86ISD::FMADD.

Thanks Michael. I will look into this.

[iid_iunknown]

In D9992#182779, @delena wrote:

All lit tests should pass in the -O0 mode. The code may be not optimal, but the compilation should not fail and the generated code should correct.

Not sure I get this point correctly.
Many tests currently run without -O0 and their CHECK's expect some optimizations to happen. This doesn't mean the tests won't compile with -O0. This means the CHECK's will have to be changed if -O0 is used.

The patch has disabled DAGCombine for -O0. This broke some tests running with -O0 as their CHECK's were written with DAGCombine in mind. I have changed the CHECK's for those tests whose differences in the code generated with and without DAGCombine in -O0 were relatively small. Several tests however had quite significant differences in the generated code, and required full redesign of their CHECK's. Instead, DAGCombine was enabled for them by changing -O0 -> -O1 to get nearly the same code as the CHECK's expected. Please let me know if this is not the best way to handle such tests and we should rather stick to the 1st option (leave -O0 and redesign the CHECK's).

lib/Target/X86/X86ISelLowering.cpp

17616

Thank you for the clarification Elena. I will submit a new patch shortly.

[delena]

I just afraid that some tests will fail to select instruction if you run them without DAGCombine.
So you should run ALL tests with O0 and be sure that they don't fail.

• Elena

[mkuper]

I think what Elena means is that it may be a good idea to check that the tests that currently run with -O3 don't *crash* (because of selection failures) when run with -O0.
They can, of course, fail due to unmatched CHECK lines.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

 //  Main DAG Combiner implementation
 //===----------------------------------------------------------------------===//
 
 void DAGCombiner::Run(CombineLevel AtLevel) {
   // set the instance variables, so that the various visit routines may use it.
   Level = AtLevel;
   LegalOperations = Level >= AfterLegalizeVectorOps;
   LegalTypes = Level >= AfterLegalizeTypes;
+  bool LegalizeOps = Level == AfterLegalizeDAG;
+  bool DoCombine = OptLevel != CodeGenOpt::None;
+
+  if (!DoCombine && !LegalizeOps)
+    return;
 
   // Add all the dag nodes to the worklist.
   for (SelectionDAG::allnodes_iterator I = DAG.allnodes_begin(),
        E = DAG.allnodes_end(); I != E; ++I)
     AddToWorklist(I);
 
   // Create a dummy node (which is not added to allnodes), that adds a reference
   // to the root node, preventing it from being deleted, and tracking any
while (!WorklistMap.empty()) {
     // reduced number of uses, allowing other xforms.
     if (recursivelyDeleteUnusedNodes(N))
       continue;
 
     WorklistRemover DeadNodes(*this);
 
     // If this combine is running after legalizing the DAG, re-legalize any
     // nodes pulled off the worklist.
-    if (Level == AfterLegalizeDAG) {
+    if (LegalizeOps) {
       SmallSetVector<SDNode *, 16> UpdatedNodes;
       bool NIsValid = DAG.LegalizeOp(N, UpdatedNodes);
 
       for (SDNode *LN : UpdatedNodes) {
         AddToWorklist(LN);
         AddUsersToWorklist(LN);
       }
       if (!NIsValid)
         continue;
     }
 
+    if (DoCombine) {
       DEBUG(dbgs() << "\nCombining: "; N->dump(&DAG));
 
       // Add any operands of the new node which have not yet been combined to the
       // worklist as well. Because the worklist uniques things already, this
       // won't repeatedly process the same operand.
       CombinedNodes.insert(N);
       for (unsigned i = 0, e = N->getNumOperands(); i != e; ++i)
         if (!CombinedNodes.count(N->getOperand(i).getNode()))
           AddToWorklist(N->getOperand(i).getNode());
 
       SDValue RV = combine(N);
 
       if (!RV.getNode())
         continue;
 
       ++NodesCombined;
 
       // If we get back the same node we passed in, rather than a new node or
       // zero, we know that the node must have defined multiple values and
       // CombineTo was used.  Since CombineTo takes care of the worklist
       // mechanics for us, we have no work to do in this case.
       if (RV.getNode() == N)
         continue;
 
       assert(N->getOpcode() != ISD::DELETED_NODE &&
-           RV.getNode()->getOpcode() != ISD::DELETED_NODE &&
-           "Node was deleted but visit returned new node!");
 
       DEBUG(dbgs() << " ... into: ";
-          RV.getNode()->dump(&DAG));
 
       // Transfer debug value.
       DAG.TransferDbgValues(SDValue(N, 0), RV);
       if (N->getNumValues() == RV.getNode()->getNumValues())
         DAG.ReplaceAllUsesWith(N, RV.getNode());
       else {
         assert(N->getValueType(0) == RV.getValueType() &&
-             N->getNumValues() == 1 && "Type mismatch");
         SDValue OpV = RV;
         DAG.ReplaceAllUsesWith(N, &OpV);
       }
 
       // Push the new node and any users onto the worklist
       AddToWorklist(RV.getNode());
       AddUsersToWorklist(RV.getNode());
+    }
 
     // Finally, if the node is now dead, remove it from the graph.  The node
     // may not be dead if the replacement process recursively simplified to
     // something else needing this node. This will also take care of adding any
     // operands which have lost a user to the worklist.
     recursivelyDeleteUnusedNodes(N);
   }
 

lib/Target/ARM/ARMISelDAGToDAG.cpp

if (ConstantSDNode *C = dyn_cast<ConstantSDNode>(N->getOperand(1))) {
       if (isPowerOf2_32(RHSV-1)) {  // 2^n+1?
         unsigned ShImm = Log2_32(RHSV-1);
         if (ShImm >= 32)
           break;
         SDValue V = N->getOperand(0);
         ShImm = ARM_AM::getSORegOpc(ARM_AM::lsl, ShImm);
         SDValue ShImmOp = CurDAG->getTargetConstant(ShImm, dl, MVT::i32);
         SDValue Reg0 = CurDAG->getRegister(0, MVT::i32);
-        if (Subtarget->isThumb()) {
-          SDValue Ops[] = { V, V, ShImmOp, getAL(CurDAG, dl), Reg0, Reg0 };
-          return CurDAG->SelectNodeTo(N, ARM::t2ADDrs, MVT::i32, Ops);
-        } else {
-          SDValue Ops[] = { V, V, Reg0, ShImmOp, getAL(CurDAG, dl), Reg0,
-                            Reg0 };
-          return CurDAG->SelectNodeTo(N, ARM::ADDrsi, MVT::i32, Ops);
-        }
+        unsigned Opc = (Subtarget->isThumb() ? ARM::t2ADDrs : ARM::ADDrsi);
+        return CurDAG->SelectNodeTo(N, Opc, MVT::i32, Ops);
       }
       if (isPowerOf2_32(RHSV+1)) {  // 2^n-1?
         unsigned ShImm = Log2_32(RHSV+1);
         if (ShImm >= 32)
           break;
         SDValue V = N->getOperand(0);
         ShImm = ARM_AM::getSORegOpc(ARM_AM::lsl, ShImm);
         SDValue ShImmOp = CurDAG->getTargetConstant(ShImm, dl, MVT::i32);
         SDValue Reg0 = CurDAG->getRegister(0, MVT::i32);
-        if (Subtarget->isThumb()) {
-          SDValue Ops[] = { V, V, ShImmOp, getAL(CurDAG, dl), Reg0, Reg0 };
-          return CurDAG->SelectNodeTo(N, ARM::t2RSBrs, MVT::i32, Ops);
-        } else {
-          SDValue Ops[] = { V, V, Reg0, ShImmOp, getAL(CurDAG, dl), Reg0,
-                            Reg0 };
-          return CurDAG->SelectNodeTo(N, ARM::RSBrsi, MVT::i32, Ops);
-        }
+        unsigned Opc = (Subtarget->isThumb() ? ARM::t2RSBrs : ARM::RSBrsi);
+        return CurDAG->SelectNodeTo(N, Opc, MVT::i32, Ops);
       }
     }
     break;
   case ISD::AND: {
     // Check for unsigned bitfield extract
     if (SDNode *I = SelectV6T2BitfieldExtractOp(N, false))
       return I;
 

lib/Target/X86/X86ISelLowering.h

private:
     SDValue LowerEH_RETURN(SDValue Op, SelectionDAG &DAG) const;
     SDValue lowerEH_SJLJ_SETJMP(SDValue Op, SelectionDAG &DAG) const;
     SDValue lowerEH_SJLJ_LONGJMP(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerINIT_TRAMPOLINE(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerFLT_ROUNDS_(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerWin64_i128OP(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerGC_TRANSITION_START(SDValue Op, SelectionDAG &DAG) const;
     SDValue LowerGC_TRANSITION_END(SDValue Op, SelectionDAG &DAG) const;
+    SDValue LowerFMA(SDValue Op, SelectionDAG &DAG) const;
 
     SDValue
       LowerFormalArguments(SDValue Chain,
                            CallingConv::ID CallConv, bool isVarArg,
                            const SmallVectorImpl<ISD::InputArg> &Ins,
                            SDLoc dl, SelectionDAG &DAG,
                            SmallVectorImpl<SDValue> &InVals) const override;
     SDValue LowerCall(CallLoweringInfo &CLI,

lib/Target/X86/X86ISelLowering.cpp

for (MVT VT : MVT::vector_valuetypes()) {
     setOperationAction(ISD::EXTRACT_SUBVECTOR, VT,Expand);
     setOperationAction(ISD::INSERT_SUBVECTOR, VT,Expand);
     setOperationAction(ISD::FABS, VT, Expand);
     setOperationAction(ISD::FSIN, VT, Expand);
     setOperationAction(ISD::FSINCOS, VT, Expand);
     setOperationAction(ISD::FCOS, VT, Expand);
     setOperationAction(ISD::FSINCOS, VT, Expand);
     setOperationAction(ISD::FREM, VT, Expand);
-    setOperationAction(ISD::FMA,  VT, Expand);
+    setOperationAction(ISD::FMA, VT, Expand);
     setOperationAction(ISD::FPOWI, VT, Expand);
     setOperationAction(ISD::FSQRT, VT, Expand);
     setOperationAction(ISD::FCOPYSIGN, VT, Expand);
     setOperationAction(ISD::FFLOOR, VT, Expand);
     setOperationAction(ISD::FCEIL, VT, Expand);
     setOperationAction(ISD::FTRUNC, VT, Expand);
     setOperationAction(ISD::FRINT, VT, Expand);
     setOperationAction(ISD::FNEARBYINT, VT, Expand);
if (!Subtarget->useSoftFloat() && Subtarget->hasFp256()) {
     setOperationAction(ISD::ANY_EXTEND,        MVT::v4i64, Custom);
     setOperationAction(ISD::ANY_EXTEND,        MVT::v8i32, Custom);
     setOperationAction(ISD::ANY_EXTEND,        MVT::v16i16, Custom);
     setOperationAction(ISD::TRUNCATE,          MVT::v16i8, Custom);
     setOperationAction(ISD::TRUNCATE,          MVT::v8i16, Custom);
     setOperationAction(ISD::TRUNCATE,          MVT::v4i32, Custom);
 
     if (Subtarget->hasFMA() || Subtarget->hasFMA4()) {
-      setOperationAction(ISD::FMA,             MVT::v8f32, Legal);
-      setOperationAction(ISD::FMA,             MVT::v4f64, Legal);
-      setOperationAction(ISD::FMA,             MVT::v4f32, Legal);
-      setOperationAction(ISD::FMA,             MVT::v2f64, Legal);
-      setOperationAction(ISD::FMA,             MVT::f32, Legal);
-      setOperationAction(ISD::FMA,             MVT::f64, Legal);
+      setOperationAction(ISD::FMA,             MVT::v8f32, Custom);
+      setOperationAction(ISD::FMA,             MVT::v4f64, Custom);
+      setOperationAction(ISD::FMA,             MVT::v4f32, Custom);
+      setOperationAction(ISD::FMA,             MVT::v2f64, Custom);
+      setOperationAction(ISD::FMA,             MVT::f32, Custom);
+      setOperationAction(ISD::FMA,             MVT::f64, Custom);
     }
 
     if (Subtarget->hasInt256()) {
       setOperationAction(ISD::ADD,             MVT::v4i64, Legal);
       setOperationAction(ISD::ADD,             MVT::v8i32, Legal);
       setOperationAction(ISD::ADD,             MVT::v16i16, Legal);
       setOperationAction(ISD::ADD,             MVT::v32i8, Legal);
 
if (!Subtarget->useSoftFloat() && Subtarget->hasAVX512()) {
     setOperationAction(ISD::FNEG,               MVT::v16f32, Custom);
 
     setOperationAction(ISD::FADD,               MVT::v8f64, Legal);
     setOperationAction(ISD::FSUB,               MVT::v8f64, Legal);
     setOperationAction(ISD::FMUL,               MVT::v8f64, Legal);
     setOperationAction(ISD::FDIV,               MVT::v8f64, Legal);
     setOperationAction(ISD::FSQRT,              MVT::v8f64, Legal);
     setOperationAction(ISD::FNEG,               MVT::v8f64, Custom);
-    setOperationAction(ISD::FMA,                MVT::v8f64, Legal);
-    setOperationAction(ISD::FMA,                MVT::v16f32, Legal);
+    setOperationAction(ISD::FMA,                MVT::v8f64, Custom);
+    setOperationAction(ISD::FMA,                MVT::v16f32, Custom);
 
     setOperationAction(ISD::FP_TO_SINT,         MVT::i32, Legal);
     setOperationAction(ISD::FP_TO_UINT,         MVT::i32, Legal);
     setOperationAction(ISD::SINT_TO_FP,         MVT::i32, Legal);
     setOperationAction(ISD::UINT_TO_FP,         MVT::i32, Legal);
     if (Subtarget->is64Bit()) {
       setOperationAction(ISD::FP_TO_UINT,       MVT::i64, Legal);
       setOperationAction(ISD::FP_TO_SINT,       MVT::i64, Legal);

   setTargetDAGCombine(ISD::SHL);
   setTargetDAGCombine(ISD::SRA);
   setTargetDAGCombine(ISD::SRL);
   setTargetDAGCombine(ISD::OR);
   setTargetDAGCombine(ISD::AND);
   setTargetDAGCombine(ISD::ADD);
   setTargetDAGCombine(ISD::FADD);
   setTargetDAGCombine(ISD::FSUB);
-  setTargetDAGCombine(ISD::FMA);
   setTargetDAGCombine(ISD::SUB);
   setTargetDAGCombine(ISD::LOAD);
   setTargetDAGCombine(ISD::MLOAD);
   setTargetDAGCombine(ISD::STORE);
   setTargetDAGCombine(ISD::MSTORE);
   setTargetDAGCombine(ISD::ZERO_EXTEND);
   setTargetDAGCombine(ISD::ANY_EXTEND);
   setTargetDAGCombine(ISD::SIGN_EXTEND);
static SDValue LowerFABSorFNEG(SDValue Op, SelectionDAG &DAG) {
 
   // If this is a FABS and it has an FNEG user, bail out to fold the combination
   // into an FNABS. We'll lower the FABS after that if it is still in use.
   if (IsFABS)
     for (SDNode *User : Op->uses())
       if (User->getOpcode() == ISD::FNEG)
         return Op;
 
+  // If this is a FNEG and it has an FMA user, bail out to lower the combination
+  // into an X86ISD::FMSUB or X86ISD::FNMSUB.
+  bool IsFNEG = (Op.getOpcode() == ISD::FNEG);
+  if (IsFNEG)
+    for (SDNode *User : Op->uses())
+      if (User->getOpcode() == ISD::FMA || User->getOpcode() == ISD::FMAD)
+        return Op;
+
   SDValue Op0 = Op.getOperand(0);
   bool IsFNABS = !IsFABS && (Op0.getOpcode() == ISD::FABS);
 
   SDLoc dl(Op);
   MVT VT = Op.getSimpleValueType();
   // Assume scalar op for initialization; update for vector if needed.
   // Note that there are no scalar bitwise logical SSE/AVX instructions, so we
   // generate a 16-byte vector constant and logic op even for the scalar case.
SDValue X86TargetLowering::LowerGC_TRANSITION_END(SDValue Op,
 
   SDLoc OpDL(Op);
   SDVTList VTs = DAG.getVTList(MVT::Other, MVT::Glue);
   SDValue NOOP(DAG.getMachineNode(X86::NOOP, SDLoc(Op), VTs, Ops), 0);
 
   return NOOP;
 }
 
+SDValue X86TargetLowering::LowerFMA(SDValue Op, SelectionDAG &DAG) const {
+  SDLoc dl(Op);
+  EVT VT = Op.getNode()->getValueType(0);
+
+  // Let legalize expand this if it isn't a legal type yet.
+  if (!DAG.getTargetLoweringInfo().isTypeLegal(VT))
+    return SDValue();
+
+  EVT ScalarVT = VT.getScalarType();
+  if ((ScalarVT != MVT::f32 && ScalarVT != MVT::f64) ||
+    (!Subtarget->hasFMA() && !Subtarget->hasFMA4()))
+    return SDValue();
+
+  SDValue A = Op.getOperand(0);
+  SDValue B = Op.getOperand(1);
+  SDValue C = Op.getOperand(2);
+
+  bool NegA = (A.getOpcode() == ISD::FNEG);
+  bool NegB = (B.getOpcode() == ISD::FNEG);
+  bool NegC = (C.getOpcode() == ISD::FNEG);
+
+  // Negative multiplication when NegA xor NegB
+  bool NegMul = (NegA != NegB);
+  if (NegA)
+    A = A.getOperand(0);
+  if (NegB)
+    B = B.getOperand(0);
+  if (NegC)
+    C = C.getOperand(0);
+
+  unsigned Opcode;
+  if (!NegMul)
+    Opcode = (!NegC) ? X86ISD::FMADD : X86ISD::FMSUB;
+  else
+    Opcode = (!NegC) ? X86ISD::FNMADD : X86ISD::FNMSUB;
+
+  return DAG.getNode(Opcode, dl, VT, A, B, C);
+}
+
 /// LowerOperation - Provide custom lowering hooks for some operations.
 ///
 SDValue X86TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   switch (Op.getOpcode()) {
   default: llvm_unreachable("Should not custom lower this!");
   case ISD::ATOMIC_FENCE:       return LowerATOMIC_FENCE(Op, Subtarget, DAG);
   case ISD::ATOMIC_CMP_SWAP_WITH_SUCCESS:
     return LowerCMP_SWAP(Op, Subtarget, DAG);
SDValue X86TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::ADD:                return LowerADD(Op, DAG);
   case ISD::SUB:                return LowerSUB(Op, DAG);
   case ISD::FSINCOS:            return LowerFSINCOS(Op, Subtarget, DAG);
   case ISD::MGATHER:            return LowerMGATHER(Op, Subtarget, DAG);
   case ISD::MSCATTER:           return LowerMSCATTER(Op, Subtarget, DAG);
   case ISD::GC_TRANSITION_START:
                                 return LowerGC_TRANSITION_START(Op, DAG);
   case ISD::GC_TRANSITION_END:  return LowerGC_TRANSITION_END(Op, DAG);
+  case ISD::FMA:                return LowerFMA(Op, DAG);
   }
 }
 
 /// ReplaceNodeResults - Replace a node with an illegal result type
 /// with a new node built out of custom code.
 void X86TargetLowering::ReplaceNodeResults(SDNode *N,
                                            SmallVectorImpl<SDValue>&Results,
                                            SelectionDAG &DAG) const {
if (VT.isVector() && VT.getSizeInBits() == 256) {
     SDValue R = WidenMaskArithmetic(N, DAG, DCI, Subtarget);
     if (R.getNode())
       return R;
   }
 
   return SDValue();
 }
 
-static SDValue PerformFMACombine(SDNode *N, SelectionDAG &DAG,
-                                 const X86Subtarget* Subtarget) {
-  SDLoc dl(N);
-  EVT VT = N->getValueType(0);
-
-  // Let legalize expand this if it isn't a legal type yet.
-  if (!DAG.getTargetLoweringInfo().isTypeLegal(VT))
-    return SDValue();
-
-  EVT ScalarVT = VT.getScalarType();
-  if ((ScalarVT != MVT::f32 && ScalarVT != MVT::f64) ||
-      (!Subtarget->hasFMA() && !Subtarget->hasFMA4()))
-    return SDValue();
-
-  SDValue A = N->getOperand(0);
-  SDValue B = N->getOperand(1);
-  SDValue C = N->getOperand(2);
-
-  bool NegA = (A.getOpcode() == ISD::FNEG);
-  bool NegB = (B.getOpcode() == ISD::FNEG);
-  bool NegC = (C.getOpcode() == ISD::FNEG);
-
-  // Negative multiplication when NegA xor NegB
-  bool NegMul = (NegA != NegB);
-  if (NegA)
-    A = A.getOperand(0);
-  if (NegB)
-    B = B.getOperand(0);
-  if (NegC)
-    C = C.getOperand(0);
-
-  unsigned Opcode;
-  if (!NegMul)
-    Opcode = (!NegC) ? X86ISD::FMADD : X86ISD::FMSUB;
-  else
-    Opcode = (!NegC) ? X86ISD::FNMADD : X86ISD::FNMSUB;
-
-  return DAG.getNode(Opcode, dl, VT, A, B, C);
-}
-
 static SDValue PerformZExtCombine(SDNode *N, SelectionDAG &DAG,
                                   TargetLowering::DAGCombinerInfo &DCI,
                                   const X86Subtarget *Subtarget) {
   // (i32 zext (and (i8  x86isd::setcc_carry), 1)) ->
   //           (and (i32 x86isd::setcc_carry), 1)
   // This eliminates the zext. This transformation is necessary because
   // ISD::SETCC is always legalized to i8.
   SDLoc dl(N);
SDValue X86TargetLowering::PerformDAGCombine(SDNode *N,
   case X86ISD::PSHUFD:
   case X86ISD::PSHUFHW:
   case X86ISD::PSHUFLW:
   case X86ISD::MOVSS:
   case X86ISD::MOVSD:
   case X86ISD::VPERMILPI:
   case X86ISD::VPERM2X128:
   case ISD::VECTOR_SHUFFLE: return PerformShuffleCombine(N, DAG, DCI,Subtarget);
-  case ISD::FMA:            return PerformFMACombine(N, DAG, Subtarget);
   case ISD::INTRINSIC_WO_CHAIN:
     return PerformINTRINSIC_WO_CHAINCombine(N, DAG, Subtarget);
   case X86ISD::INSERTPS: {
     if (getTargetMachine().getOptLevel() > CodeGenOpt::None)
       return PerformINSERTPSCombine(N, DAG, Subtarget);
     break;
   }
   case X86ISD::BLENDI:    return PerformBLENDICombine(N, DAG);

test/CodeGen/AArch64/aarch64_f16_be.ll

; CHECK-BE: st1
   ret void
 }
 
 define void @test_bitcast_v8f16_to_fp128(<8 x half> %a) {
 ; CHECK-LABEL: test_bitcast_v8f16_to_fp128:
 ; CHECK-NOT: st1
 
 ; CHECK-BE-LABEL: test_bitcast_v8f16_to_fp128:
-; CHECK-BE: st1
+; CHECK-BE: rev64
+; CHECK-BE: ext
+; CHECK-BE: rev64
+; CHECK-BE: ext
+; CHECK-BE: str
 
   %x = alloca fp128, align 16
   %y = bitcast <8 x half> %a to fp128
   store fp128 %y, fp128* %x, align 16
   ret void
 }
 
 define void @test_bitcast_v4f16_to_v2f32(<4 x half> %a) {
; CHECK-BE: st1
   ret void
 }
 
 define void @test_bitcast_v4f16_to_v1f64(<4 x half> %a) {
 ; CHECK-LABEL: test_bitcast_v4f16_to_v1f64:
 ; CHECK-NOT: st1
 
 ; CHECK-BE-LABEL: test_bitcast_v4f16_to_v1f64:
-; CHECK-BE: st1
+; CHECK-BE: rev64
+; CHECK-BE: rev64
+; CHECK-BE: str
 
   %x = alloca <1 x double>, align 8
   %y = bitcast <4 x half> %a to <1 x double>
   store <1 x double> %y, <1 x double>* %x, align 8
   ret void
 }

test/CodeGen/AArch64/and-mask-removal.ll

-; RUN: llc -O0 -fast-isel=false -mtriple=arm64-apple-darwin  < %s  | FileCheck %s
+; RUN: llc -O1 -fast-isel=false -mtriple=arm64-apple-darwin  < %s  | FileCheck %s
 
 @board = common global [400 x i8] zeroinitializer, align 1
 @next_string = common global i32 0, align 4
 @string_number = common global [400 x i32] zeroinitializer, align 4
 
 ; Function Attrs: nounwind ssp
 define void @new_position(i32 %pos) {
 entry:

test/CodeGen/ARM/Windows/alloca.ll

-; RUN: llc -O0 -mtriple thumbv7-windows-itanium -filetype asm -o - %s | FileCheck %s
+; RUN: llc -O1 -mtriple thumbv7-windows-itanium -filetype asm -o - %s | FileCheck %s
 
 declare arm_aapcs_vfpcc i32 @num_entries()
 
 define arm_aapcs_vfpcc void @test___builtin_alloca() {
 entry:
   %array = alloca i8*, align 4
   %call = call arm_aapcs_vfpcc i32 @num_entries()
   %mul = mul i32 4, %call

test/CodeGen/ARM/alloc-no-stack-realign.ll

-; RUN: llc < %s -mtriple=armv7-apple-ios -O0 | FileCheck %s -check-prefix=NO-REALIGN
-; RUN: llc < %s -mtriple=armv7-apple-ios -O0 | FileCheck %s -check-prefix=REALIGN
+; RUN: llc < %s -mtriple=armv7-apple-ios -O1 | FileCheck %s -check-prefix=NO-REALIGN
+; RUN: llc < %s -mtriple=armv7-apple-ios -O1 | FileCheck %s -check-prefix=REALIGN
 
 ; rdar://12713765
 ; When realign-stack is set to false, make sure we are not creating stack
 ; objects that are assumed to be 64-byte aligned.
 @T3_retval = common global <16 x float> zeroinitializer, align 16
 
 define void @test1(<16 x float>* noalias sret %agg.result) nounwind ssp "no-realign-stack" {
 entry:
 ; NO-REALIGN-LABEL: test1
 ; NO-REALIGN: mov r[[R2:[0-9]+]], r[[R1:[0-9]+]]
 ; NO-REALIGN: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]!
-; NO-REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; NO-REALIGN: add r[[R2:[0-9]+]], r[[R1]], #32
-; NO-REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; NO-REALIGN: add r[[R2:[0-9]+]], r[[R1]], #48
+
+; NO-REALIGN: add r[[R9:[0-9]+]], r[[R1]], #32
+; NO-REALIGN: add r[[R1]], r[[R1]], #48
+; NO-REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; NO-REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R9]]:128]
 ; NO-REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
 
-; NO-REALIGN: add r[[R2:[0-9]+]], r[[R1:[0-9]+]], #48
-; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; NO-REALIGN: add r[[R2:[0-9]+]], r[[R1]], #32
-; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; NO-REALIGN: vst1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]!
+; NO-REALIGN: add r[[R1:[0-9]+]], r[[R3:[0-9]+]], #48
 ; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; NO-REALIGN: add r[[R1:[0-9]+]], r[[R3:[0-9]+]], #32
+; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; NO-REALIGN: vst1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[R3]]:128]!
+; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R3]]:128]
 
-; NO-REALIGN: add r[[R2:[0-9]+]], r[[R0:0]], #48
-; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; NO-REALIGN: add r[[R2:[0-9]+]], r[[R0]], #32
-; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
+; NO-REALIGN: add r[[R1:[0-9]+]], r[[R0:0]], #48
+; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; NO-REALIGN: add r[[R1:[0-9]+]], r[[R0]], #32
+; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
 ; NO-REALIGN: vst1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[R0]]:128]!
 ; NO-REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R0]]:128]
+
  %retval = alloca <16 x float>, align 16
  %0 = load <16 x float>, <16 x float>* @T3_retval, align 16
  store <16 x float> %0, <16 x float>* %retval
  %1 = load <16 x float>, <16 x float>* %retval
  store <16 x float> %1, <16 x float>* %agg.result, align 16
  ret void
 }
 
 define void @test2(<16 x float>* noalias sret %agg.result) nounwind ssp {
 entry:
 ; REALIGN-LABEL: test2
 ; REALIGN: bfc sp, #0, #6
 ; REALIGN: mov r[[R2:[0-9]+]], r[[R1:[0-9]+]]
 ; REALIGN: vld1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]!
-; REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; REALIGN: add r[[R2:[0-9]+]], r[[R1]], #32
-; REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; REALIGN: add r[[R2:[0-9]+]], r[[R1]], #48
+; REALIGN: add r[[R9:[0-9]+]], r[[R1]], #32
+; REALIGN: add r[[R1:[0-9]+]], r[[R1]], #48
+; REALIGN: mov r[[R3:[0-9]+]], sp
+; REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R9]]:128]
 ; REALIGN: vld1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
 
-
-; REALIGN: orr r[[R2:[0-9]+]], r[[R1:[0-9]+]], #48
-; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; REALIGN: orr r[[R2:[0-9]+]], r[[R1]], #32
-; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
-; REALIGN: orr r[[R2:[0-9]+]], r[[R1]], #16
-; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R2]]:128]
+; REALIGN: orr r[[R1:[0-9]+]], r[[R3:[0-9]+]], #48
+; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; REALIGN: orr r[[R1:[0-9]+]], r[[R3]], #32
+; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
+; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R3]]:128]
+; REALIGN: orr r[[R1:[0-9]+]], r[[R3]], #16
 ; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
 
 ; REALIGN: add r[[R1:[0-9]+]], r[[R0:0]], #48
 ; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
 ; REALIGN: add r[[R1:[0-9]+]], r[[R0]], #32
 ; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R1]]:128]
 ; REALIGN: vst1.32 {{{d[0-9]+, d[0-9]+}}}, [r[[R0]]:128]!
 ; REALIGN: vst1.64 {{{d[0-9]+, d[0-9]+}}}, [r[[R0]]:128]
  %retval = alloca <16 x float>, align 16
  %0 = load <16 x float>, <16 x float>* @T3_retval, align 16
  store <16 x float> %0, <16 x float>* %retval
  %1 = load <16 x float>, <16 x float>* %retval
  store <16 x float> %1, <16 x float>* %agg.result, align 16
  ret void
 }

test/CodeGen/ARM/big-endian-ret-f64.ll

 ; RUN: llc -mtriple=armebv7a-eabi %s -O0 -o - | FileCheck %s
 ; RUN: llc -mtriple=armebv8a-eabi %s -O0 -o - | FileCheck %s
 
 define double @fn() {
 ; CHECK-LABEL: fn
-; CHECK: ldr r0, [sp]
-; CHECK: ldr r1, [sp, #4]
+; CHECK: vldr [[REG:d[0-9]+]], [sp]
+; CHECK: vmov r1, r0, [[REG]]
   %r = alloca double, align 8
   %1 = load double, double* %r, align 8
   ret double %1
 }
 

test/CodeGen/ARM/vst3.ll

-; RUN: llc -mtriple=arm-eabi -mattr=+neon -fast-isel=0 -O0 %s -o - | FileCheck %s
+; RUN: llc -mtriple=arm-eabi -mattr=+neon -fast-isel=0 -O1 %s -o - | FileCheck %s
 
 define void @vst3i8(i8* %A, <8 x i8>* %B) nounwind {
 ;CHECK-LABEL: vst3i8:
 ;Check the alignment value.  Max for this instruction is 64 bits:
 ;This test runs at -O0 so do not check for specific register numbers.
 ;CHECK: vst3.8 {d{{.*}}, d{{.*}}, d{{.*}}}, [r{{.*}}:64]
   %tmp1 = load <8 x i8>, <8 x i8>* %B
   call void @llvm.arm.neon.vst3.v8i8(i8* %A, <8 x i8> %tmp1, <8 x i8> %tmp1, <8 x i8> %tmp1, i32 32)

test/CodeGen/X86/atomic16.ll

 ; X64-LABEL:   atomic_fetch_and16
 ; X32-LABEL:   atomic_fetch_and16
   %t1 = atomicrmw and  i16* @sc16, i16 3 acquire
 ; X64:       lock
 ; X64:       andw $3, {{.*}} # encoding: [0x66,0xf0
 ; X32:       lock
 ; X32:       andw $3
   %t2 = atomicrmw and  i16* @sc16, i16 5 acquire
-; X64:       andl
+; X64:       andw
 ; X64:       lock
 ; X64:       cmpxchgw
-; X32:       andl
+; X32:       andw
 ; X32:       lock
 ; X32:       cmpxchgw
   %t3 = atomicrmw and  i16* @sc16, i16 %t2 acquire
 ; X64:       lock
 ; X64:       andw {{.*}} # encoding: [0x66,0xf0
 ; X32:       lock
 ; X32:       andw
   ret void
 ; X64:       ret
 ; X32:       ret
 }
 
 define void @atomic_fetch_or16() nounwind {
 ; X64-LABEL:   atomic_fetch_or16
 ; X32-LABEL:   atomic_fetch_or16
   %t1 = atomicrmw or   i16* @sc16, i16 3 acquire
 ; X64:       lock
 ; X64:       orw $3, {{.*}} # encoding: [0x66,0xf0
 ; X32:       lock
 ; X32:       orw $3
   %t2 = atomicrmw or   i16* @sc16, i16 5 acquire
-; X64:       orl
+; X64:       orw
 ; X64:       lock
 ; X64:       cmpxchgw
-; X32:       orl
+; X32:       orw
 ; X32:       lock
 ; X32:       cmpxchgw
   %t3 = atomicrmw or   i16* @sc16, i16 %t2 acquire
 ; X64:       lock
 ; X64:       orw {{.*}} # encoding: [0x66,0xf0
 ; X32:       lock
 ; X32:       orw
   ret void
 ; X64:       ret
 ; X32:       ret
 }
 
 define void @atomic_fetch_xor16() nounwind {
 ; X64-LABEL:   atomic_fetch_xor16
 ; X32-LABEL:   atomic_fetch_xor16
   %t1 = atomicrmw xor  i16* @sc16, i16 3 acquire
 ; X64:       lock
 ; X64:       xorw $3, {{.*}} # encoding: [0x66,0xf0
 ; X32:       lock
 ; X32:       xorw $3
   %t2 = atomicrmw xor  i16* @sc16, i16 5 acquire
-; X64:       xorl
+; X64:       xorw
 ; X64:       lock
 ; X64:       cmpxchgw
-; X32:       xorl
+; X32:       xorw
 ; X32:       lock
 ; X32:       cmpxchgw
   %t3 = atomicrmw xor  i16* @sc16, i16 %t2 acquire
 ; X64:       lock
 ; X64:       xorw {{.*}} # encoding: [0x66,0xf0
 ; X32:       lock
 ; X32:       xorw
   ret void
 ; X64:       ret
 ; X32:       ret
 }
 
 define void @atomic_fetch_nand16(i16 %x) nounwind {
 ; X64-LABEL:   atomic_fetch_nand16
 ; X32-LABEL:   atomic_fetch_nand16
   %t1 = atomicrmw nand i16* @sc16, i16 %x acquire
-; X64:       andl
-; X64:       notl
+; X64:       andw
+; X64:       notw
 ; X64:       lock
 ; X64:       cmpxchgw
-; X32:       andl
-; X32:       notl
+; X32:       andw
+; X32:       notw
 ; X32:       lock
 ; X32:       cmpxchgw
   ret void
 ; X64:       ret
 ; X32:       ret
 }
 
 define void @atomic_fetch_max16(i16 %x) nounwind {

test/CodeGen/X86/atomic32.ll

; WITH-CMOV-LABEL: atomic_fetch_max32:
 
   %t1 = atomicrmw max  i32* @sc32, i32 %x acquire
 ; WITH-CMOV:       subl
 ; WITH-CMOV:       cmov
 ; WITH-CMOV:       lock
 ; WITH-CMOV:       cmpxchgl
 
 ; NOCMOV:    subl
-; NOCMOV:    jge
+; NOCMOV:    setg [[REG:%[a-z]+]]
+; NOCMOV:    testb $1, [[REG]]
+; NOCMOV:    jne
 ; NOCMOV:    lock
 ; NOCMOV:    cmpxchgl
   ret void
 ; WITH-CMOV:       ret
 ; NOCMOV:    ret
 }
 
 define void @atomic_fetch_min32(i32 %x) nounwind {
 ; WITH-CMOV-LABEL: atomic_fetch_min32:
 ; NOCMOV-LABEL: atomic_fetch_min32:
 
   %t1 = atomicrmw min  i32* @sc32, i32 %x acquire
 ; WITH-CMOV:       subl
 ; WITH-CMOV:       cmov
 ; WITH-CMOV:       lock
 ; WITH-CMOV:       cmpxchgl
 
 ; NOCMOV:    subl
-; NOCMOV:    jle
+; NOCMOV:    setle [[REG:%[a-z]+]]
+; NOCMOV:    testb $1, [[REG]]
+; NOCMOV:    jne
 ; NOCMOV:    lock
 ; NOCMOV:    cmpxchgl
   ret void
 ; WITH-CMOV:       ret
 ; NOCMOV:    ret
 }
 
 define void @atomic_fetch_umax32(i32 %x) nounwind {
 ; WITH-CMOV-LABEL: atomic_fetch_umax32:
 ; NOCMOV-LABEL: atomic_fetch_umax32:
 
   %t1 = atomicrmw umax i32* @sc32, i32 %x acquire
 ; WITH-CMOV:       subl
 ; WITH-CMOV:       cmov
 ; WITH-CMOV:       lock
 ; WITH-CMOV:       cmpxchgl
 
 ; NOCMOV:    subl
-; NOCMOV:    ja
+; NOCMOV:    seta [[REG:%[a-z]+]]
+; NOCMOV:    testb $1, [[REG]]
+; NOCMOV:    jne
 ; NOCMOV:    lock
 ; NOCMOV:    cmpxchgl
   ret void
 ; WITH-CMOV:       ret
 ; NOCMOV:    ret
 }
 
 define void @atomic_fetch_umin32(i32 %x) nounwind {
 ; WITH-CMOV-LABEL: atomic_fetch_umin32:
 ; NOCMOV-LABEL: atomic_fetch_umin32:
 
   %t1 = atomicrmw umin i32* @sc32, i32 %x acquire
 ; WITH-CMOV:       subl
 ; WITH-CMOV:       cmov
 ; WITH-CMOV:       lock
 ; WITH-CMOV:       cmpxchgl
 
 ; NOCMOV:    subl
-; NOCMOV:    jb
+; NOCMOV:    setbe [[REG:%[a-z]+]]
+; NOCMOV:    testb $1, [[REG]]
+; NOCMOV:    jne
 ; NOCMOV:    lock
 ; NOCMOV:    cmpxchgl
   ret void
 ; WITH-CMOV:       ret
 ; NOCMOV:    ret
 }
 
 define void @atomic_fetch_cmpxchg32() nounwind {

test/CodeGen/X86/atomic6432.ll

 ; X32:       cmpxchg8b
   ret void
 ; X32:       ret
 }
 
 define void @atomic_fetch_sub64() nounwind {
 ; X32-LABEL:   atomic_fetch_sub64:
   %t1 = atomicrmw sub  i64* @sc64, i64 1 acquire
-; X32:       addl $-1
-; X32:       adcl $-1
+; X32:       subl $1
+; X32:       sbbl $0
 ; X32:       lock
 ; X32:       cmpxchg8b
   %t2 = atomicrmw sub  i64* @sc64, i64 3 acquire
-; X32:       addl $-3
-; X32:       adcl $-1
+; X32:       subl $3
+; X32:       sbbl $0
 ; X32:       lock
 ; X32:       cmpxchg8b
   %t3 = atomicrmw sub  i64* @sc64, i64 5 acquire
-; X32:       addl $-5
-; X32:       adcl $-1
+; X32:       subl $5
+; X32:       sbbl $0
 ; X32:       lock
 ; X32:       cmpxchg8b
   %t4 = atomicrmw sub  i64* @sc64, i64 %t3 acquire
 ; X32:       subl
 ; X32:       sbbl
 ; X32:       lock
 ; X32:       cmpxchg8b
   ret void

test/CodeGen/X86/dag-optnone.ll

-; RUN: llc < %s -mtriple=x86_64-pc-win32 -O0 -mattr=+avx | FileCheck %s
+; RUN: llc < %s -mtriple=x86_64-pc-win32 -O0 -mattr=+avx | FileCheck %s --check-prefix=CHECK --check-prefix=OPT
+; RUN: llc < %s -mtriple=x86_64-pc-win32 -O1 -mattr=+avx | FileCheck %s --check-prefix=OPT
 
 ; Background:
-; If fast-isel bails out to normal selection, then the DAG combiner will run,
-; even at -O0. In principle this should not happen (those are optimizations,
-; and we said -O0) but as a practical matter there are some instruction
-; selection patterns that depend on the legalizations and transforms that the
-; DAG combiner does.
+; If fast-isel bails out to normal selection, then the DAG combiner should not
+; run at -O0.
 ;
 ; The 'optnone' attribute implicitly sets -O0 and fast-isel for the function.
-; The DAG combiner was disabled for 'optnone' (but not -O0) by r221168, then
-; re-enabled in r233153 because of problems with instruction selection patterns
-; mentioned above. (Note: because 'optnone' is supposed to match -O0, r221168
-; really should have disabled the combiner for both.)
 ;
-; If instruction selection eventually becomes smart enough to run without DAG
-; combiner, then the combiner can be turned off for -O0 (not just 'optnone')
-; and this test can go away. (To be replaced by a different test that verifies
-; the DAG combiner does *not* run at -O0 or for 'optnone' functions.)
-;
-; In the meantime, this test wants to make sure the combiner stays enabled for
-; 'optnone' functions, just as it is for -O0.
-
-
-; The test cases @foo[WithOptnone] prove that the same DAG combine happens
-; with -O0 and with 'optnone' set.  To prove this, we use a Windows triple to
+; The test cases @foo[WithOptnone] prove that no DAG combine happens with
+; -O0 and with 'optnone' set.  To prove this, we use a Windows triple to
 ; cause fast-isel to bail out (because something about the calling convention
 ; is not handled in fast-isel).  Then we have a repeated fadd that can be
-; combined into an fmul.  We show that this happens in both the non-optnone
-; function and the optnone function.
+; combined into an fmul.  We show that this does not happen in both the
+; non-optnone function and the optnone function.
 
 define float @foo(float %x) #0 {
 entry:
   %add = fadd fast float %x, %x
   %add1 = fadd fast float %add, %x
   ret float %add1
 }
 
 ; CHECK-LABEL: @foo
-; CHECK-NOT:   add
-; CHECK:       mul
+; CHECK-NOT:   mul
+; CHECK:       add
+; CHECK:       add
 ; CHECK-NEXT:  ret
 
 define float @fooWithOptnone(float %x) #1 {
 entry:
   %add = fadd fast float %x, %x
   %add1 = fadd fast float %add, %x
   ret float %add1
 }
 
-; CHECK-LABEL: @fooWithOptnone
-; CHECK-NOT:   add
-; CHECK:       mul
-; CHECK-NEXT:  ret
+; OPT-LABEL: @fooWithOptnone
+; OPT-NOT:   mul
+; OPT:       add
+; OPT:       add
+; OPT-NEXT:  ret
 
 
 ; The test case @bar is derived from an instruction selection failure case
 ; that was solved by r233153. It depends on -mattr=+avx.
 ; Really all we're trying to prove is that it doesn't crash any more.
 
 @id84 = common global <16 x i32> zeroinitializer, align 64
 

test/CodeGen/X86/fast-isel-gep.ll

entry:
   %tmp = load i64, i64* %x.addr                        ; <i64> [#uses=1]
   %add = add nsw i64 %tmp, 16                     ; <i64> [#uses=1]
   %tmp1 = load double*, double** %p.addr                   ; <double*> [#uses=1]
   %arrayidx = getelementptr inbounds double, double* %tmp1, i64 %add ; <double*> [#uses=1]
   %tmp2 = load double, double* %arrayidx                  ; <double> [#uses=1]
   ret double %tmp2
 
 ; X32-LABEL: test4:
-; X32: 128(%e{{.*}},%e{{.*}},8)
+; X32: addl $16, [[REG:%e[a-z]+]]
+; X32: (%e{{.*}},[[REG]],8)
 ; X64-LABEL: test4:
 ; X64: 128(%r{{.*}},%r{{.*}},8)
 }
 
 ; PR8961 - Make sure the sext for the GEP addressing comes before the load that
 ; is folded.
 define i64 @test5(i8* %A, i32 %I, i64 %B) nounwind {
   %v8 = getelementptr i8, i8* %A, i32 %I

test/CodeGen/X86/fma-no-dag-combine.ll

+; Check if an fma intrinsic is properly lowered to a target specific fma instruction when
+; DAGCombine is disabled by -O0.
+
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -march=x86-64 -O0 -mattr=+fma | FileCheck %s
+
+declare <4 x float> @llvm.fma.v4f32(<4 x float> %a, <4 x float>  %b, <4 x float>  %c)
+
+; CHECK: test_fma_no_combine
+; CHECK: vfmadd213ps
+; CHECK: ret  
+define <4 x float> @test_fma_no_combine(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2) {
+  %res = call <4 x float> @llvm.fma.v4f32(<4 x float> %a0, <4 x float> %a1, <4 x float> %a2)
+  ret <4 x float> %res
+}

test/CodeGen/X86/inline-asm-tied.ll

 ; RUN: llc < %s -mtriple=i386-apple-darwin9 -O0 -optimize-regalloc -regalloc=basic -no-integrated-as | FileCheck %s
 ; rdar://6992609
 
-; CHECK: movl [[EDX:%e..]], 4(%esp)
-; CHECK: movl [[EDX]], 4(%esp)
+; CHECK: movl [[REG1:%e..]], 4(%esp)
+; CHECK: movl 4(%esp), [[REG2:%e..]]
+; CHECK: movl [[REG2]], 4(%esp)
 target triple = "i386-apple-darwin9.0"
 @llvm.used = appending global [1 x i8*] [i8* bitcast (i64 (i64)* @_OSSwapInt64 to i8*)], section "llvm.metadata"    ; <[1 x i8*]*> [#uses=0]
 
 define i64 @_OSSwapInt64(i64 %_data) nounwind {
 entry:
   %retval = alloca i64    ; <i64*> [#uses=2]
   %_data.addr = alloca i64    ; <i64*> [#uses=4]
   store i64 %_data, i64* %_data.addr

test/CodeGen/X86/musttail.ll

 declare void @capture(i8*)
 declare void @t3_callee(i32)
 
 ; Test that we actually copy in and out stack arguments that aren't forwarded
 ; without modification.
 define i32 @t4({}* %fn, i32 %n, i32 %r) {
 ; CHECK-LABEL: t4:
 ; CHECK: incl %[[r:.*]]
-; CHECK: decl %[[n:.*]]
+; CHECK: {{decl|subl}}
+; CHECK-SAME: %[[n:.*]]
 ; CHECK: movl %[[r]], {{[0-9]+}}(%esp)
 ; CHECK: movl %[[n]], {{[0-9]+}}(%esp)
 ; CHECK: jmpl *%{{.*}}
 
 entry:
   %r1 = add i32 %r, 1
   %n1 = sub i32 %n, 1
   %fn_cast = bitcast {}* %fn to i32 ({}*, i32, i32)*
   %r2 = musttail call i32 %fn_cast({}* %fn, i32 %n1, i32 %r1)
   ret i32 %r2
 }
 
 ; Combine the complex stack frame with the parameter modification.
 define i32 @t5({}* %fn, i32 %n, i32 %r) alignstack(32) {
 ; CHECK-LABEL: t5:
 ; CHECK: pushl %ebp
 ; CHECK: movl %esp, %ebp
 ; CHECK: pushl %esi
 ;   Align the stack.
 ; CHECK: andl $-32, %esp
 ; CHECK: movl %esp, %esi
 ;   Modify the args.
 ; CHECK: incl %[[r:.*]]
-; CHECK: decl %[[n:.*]]
+; CHECK: {{decl|subl}}
+; CHECK-SAME: %[[n:.*]]
 ;   Store them through ebp, since that's the only stable arg pointer.
 ; CHECK: movl %[[r]], {{[0-9]+}}(%ebp)
 ; CHECK: movl %[[n]], {{[0-9]+}}(%ebp)
 ;   Epilogue.
 ; CHECK: leal {{[-0-9]+}}(%ebp), %esp
 ; CHECK: popl %esi
 ; CHECK: popl %ebp
 ; CHECK: jmpl *%{{.*}}

test/CodeGen/X86/switch.ll

 bb0: tail call void @g(i32 0) br label %return
 bb1: tail call void @g(i32 1) br label %return
 bb2: tail call void @g(i32 1) br label %return
 return: ret void
 
 ; Should be lowered as straight compares in -O0 mode.
 ; NOOPT-LABEL: basic
 ; NOOPT: subl $1, %eax
-; NOOPT: je
+; NOOPT: sete [[R1:%.+]]
+; NOOPT: testb $1, [[R1]]
+; NOOPT: jne
 ; NOOPT: subl $3, %eax
-; NOOPT: je
+; NOOPT: sete [[R1]]
+; NOOPT: testb $1, [[R1]]
+; NOOPT: jne
 ; NOOPT: subl $4, %eax
-; NOOPT: je
+; NOOPT: sete [[R1]]
+; NOOPT: testb $1, [[R1]]
+; NOOPT: jne
 ; NOOPT: subl $5, %eax
-; NOOPT: je
+; NOOPT: sete [[R1]]
+; NOOPT: testb $1, [[R1]]
+; NOOPT: jne
 
 ; Jump table otherwise.
 ; CHECK-LABEL: basic
 ; CHECK: decl
 ; CHECK: cmpl $4
 ; CHECK: ja
 ; CHECK: jmpq *.LJTI
 }

 ; CHECK: leal -100
 ; CHECK: cmpl $4
 ; CHECK: jae
 ; CHECK: cmpl $3
 ; CHECK: ja
 
 ; We do this even at -O0, because it's cheap and makes codegen faster.
 ; NOOPT-LABEL: simple_ranges
-; NOOPT: subl $4
-; NOOPT: jb
-; NOOPT: addl $-100
-; NOOPT: subl $4
-; NOOPT: jb
+; NOOPT: subl $3
+; NOOPT: setbe [[R1:%.+]]
+; NOOPT: testb $1, [[R1]]
+; NOOPT: jne
+; NOOPT: subl $100
+; NOOPT: subl $3
+; NOOPT: setbe [[R1]]
+; NOOPT: testb $1, [[R1]]
+; NOOPT: jne
 }
 
 
 define void @jt_is_better(i32 %x) {
 entry:
   switch i32 %x, label %return [
     i32 0, label %bb0
     i32 2, label %bb0

test/CodeGen/X86/win32_sret.ll

 ; We specify -mcpu explicitly to avoid instruction reordering that happens on
 ; some setups (e.g., Atom) from affecting the output.
-; RUN: llc < %s -mcpu=core2 -mtriple=i686-pc-win32 | FileCheck %s -check-prefix=WIN32
+; RUN: llc < %s -mcpu=core2 -mtriple=i686-pc-win32 | FileCheck %s -check-prefix=WIN32 -check-prefix=WIN32-NO-O0
 ; RUN: llc < %s -mcpu=core2 -mtriple=i686-pc-mingw32 | FileCheck %s -check-prefix=MINGW_X86
 ; RUN: llc < %s -mcpu=core2 -mtriple=i686-pc-cygwin | FileCheck %s -check-prefix=CYGWIN
 ; RUN: llc < %s -mcpu=core2 -mtriple=i386-pc-linux | FileCheck %s -check-prefix=LINUX
-; RUN: llc < %s -mcpu=core2 -O0 -mtriple=i686-pc-win32 | FileCheck %s -check-prefix=WIN32
+; RUN: llc < %s -mcpu=core2 -O0 -mtriple=i686-pc-win32 | FileCheck %s -check-prefix=WIN32 -check-prefix=WIN32-O0
 ; RUN: llc < %s -mcpu=core2 -O0 -mtriple=i686-pc-mingw32 | FileCheck %s -check-prefix=MINGW_X86
 ; RUN: llc < %s -mcpu=core2 -O0 -mtriple=i686-pc-cygwin | FileCheck %s -check-prefix=CYGWIN
 ; RUN: llc < %s -mcpu=core2 -O0 -mtriple=i386-pc-linux | FileCheck %s -check-prefix=LINUX
 
 ; The SysV ABI used by most Unixes and Mingw on x86 specifies that an sret pointer
 ; is callee-cleanup. However, in MSVC's cdecl calling convention, sret pointer
 ; arguments are caller-cleanup like normal arguments.
 
entry:
   store i32 42, i32* %x, align 4
   ret void
 ; WIN32-LABEL:     {{^}}"?foo@C5@@QAE?AUS5@@XZ":
 ; MINGW_X86-LABEL: {{^}}"?foo@C5@@QAE?AUS5@@XZ":
 ; CYGWIN-LABEL:    {{^}}"?foo@C5@@QAE?AUS5@@XZ":
 ; LINUX-LABEL:     {{^}}"?foo@C5@@QAE?AUS5@@XZ":
 
 ; The address of the return structure is passed as an implicit parameter.
-; In the -O0 build, %eax is spilled at the beginning of the function, hence we
-; should match both 4(%esp) and 8(%esp).
-; WIN32:     {{[48]}}(%esp), %eax
-; WIN32:     movl $42, (%eax)
-; WIN32:     retl $4
+; WIN32-NO-O0: {{[48]}}(%esp), %eax
+; WIN32-O0:    subl ${{[0-9]+}}, %esp
+; WIN32-O0:    {{[0-9]+}}(%esp), %eax
+; WIN32:       movl $42, (%eax)
+; WIN32:       retl $4
 }
 
 define void @call_foo5() {
 entry:
   %c = alloca %class.C5, align 1
   %s = alloca %struct.S5, align 4
   call x86_thiscallcc void @"\01?foo@C5@@QAE?AUS5@@XZ"(%struct.S5* sret %s, %class.C5* %c)
 ; WIN32-LABEL:      {{^}}_call_foo5:

test/CodeGen/X86/win64_eh.ll

 ; WIN64: .seh_stackalloc 8000
 ; WIN64: .seh_endprologue
 ; WIN64: addq $8000, %rsp
 ; WIN64: ret
 ; WIN64: .seh_endproc
 
 
 ; Checks stack push
-define i32 @foo3(i32 %f_arg, i32 %e_arg, i32 %d_arg, i32 %c_arg, i32 %b_arg, i32 %a_arg) uwtable {
+define i32 @foo3(i32 %g_arg, i32 %f_arg, i32 %e_arg, i32 %d_arg, i32 %c_arg, i32 %b_arg, i32 %a_arg) uwtable {
 entry:
   %a = alloca i32
   %b = alloca i32
   %c = alloca i32
   %d = alloca i32
   %e = alloca i32
   %f = alloca i32
+  %g = alloca i32
   store i32 %a_arg, i32* %a
   store i32 %b_arg, i32* %b
   store i32 %c_arg, i32* %c
   store i32 %d_arg, i32* %d
   store i32 %e_arg, i32* %e
   store i32 %f_arg, i32* %f
+  store i32 %g_arg, i32* %g
   %tmp = load i32, i32* %a
   %tmp1 = mul i32 %tmp, 2
   %tmp2 = load i32, i32* %b
   %tmp3 = mul i32 %tmp2, 3
   %tmp4 = add i32 %tmp1, %tmp3
   %tmp5 = load i32, i32* %c
   %tmp6 = mul i32 %tmp5, 5
   %tmp7 = add i32 %tmp4, %tmp6
   %tmp8 = load i32, i32* %d
-  %tmp9 = mul i32 %tmp8, 7
+  %tmp9 = mul i32 %tmp8, 9
   %tmp10 = add i32 %tmp7, %tmp9
   %tmp11 = load i32, i32* %e
   %tmp12 = mul i32 %tmp11, 11
   %tmp13 = add i32 %tmp10, %tmp12
   %tmp14 = load i32, i32* %f
   %tmp15 = mul i32 %tmp14, 13
   %tmp16 = add i32 %tmp13, %tmp15
   ret i32 %tmp16
 }
 ; WIN64-LABEL: foo3:
 ; WIN64: .seh_proc foo3
 ; WIN64: pushq %rsi
 ; WIN64: .seh_pushreg 6
-; NORM:  subq $24, %rsp
-; ATOM:  leaq -24(%rsp), %rsp
-; WIN64: .seh_stackalloc 24
+; NORM:  subq $32, %rsp
+; ATOM:  leaq -32(%rsp), %rsp
+; WIN64: .seh_stackalloc 32
 ; WIN64: .seh_endprologue
-; WIN64: addq $24, %rsp
+; WIN64: addq $32, %rsp
 ; WIN64: popq %rsi
 ; WIN64: ret
 ; WIN64: .seh_endproc
 
 
 ; Check emission of eh handler and handler data
 declare i32 @_d_eh_personality(i32, i32, i64, i8*, i8*)
 declare void @_d_eh_resume_unwind(i8*)

test/CodeGen/XCore/threads.ll

ConstantExpPhiNode:
   br label %ConstantExpPhiNode
 exit:
   ret void
 }
 
 define void @phiNode2( i1 %bool) {
 ; N.B. check an extra 'Node_crit_edge' (LBB12_1) is inserted
 ; PHINODE-LABEL: phiNode2:
-; PHINODE: bf {{r[0-9]}}, .LBB12_3
+; PHINODE: mkmsk [[MASK_REG:r[0-9]+]], 1
+; PHINODE: xor [[REG:r[0-9]+]], [[REG]], [[MASK_REG]]
+; PHINODE: bt [[REG]], .LBB12_3
 ; PHINODE: bu .LBB12_1
 ; PHINODE-LABEL: .LBB12_1:
 ; PHINODE: get r11, id
 ; PHINODE-LABEL: .LBB12_2:
 ; PHINODE: get r11, id
 ; PHINODE: bu .LBB12_2
 ; PHINODE-LABEL: .LBB12_3:
 entry:

test/DebugInfo/ARM/line.test

-; RUN: llc -mtriple=arm-none-linux -O0 -filetype=asm < %S/../Inputs/line.ll | FileCheck %S/../Inputs/line.ll
+; RUN: llc -mtriple=arm-none-linux -O1 -filetype=asm < %S/../Inputs/line.ll | FileCheck %S/../Inputs/line.ll
 
 ; This is more complex than it looked. It's mixed up somewhere in SelectionDAG
 ; (legalized as br_cc, losing the separation between the comparison and the
 ; branch, then further lowered to CMPri + brcc but without the fidelity that
 ; those two instructions are on separate lines)
 ; XFAIL: *

test/DebugInfo/X86/op_deref.ll

 
 ; CHECK-NOT: DW_TAG
 ; CHECK: DW_AT_name [DW_FORM_strp]  ( .debug_str[0x00000067] = "vla")
 
 ; Unfortunately llvm-dwarfdump can't unparse a list of DW_AT_locations
 ; right now, so we check the asm output:
 ; RUN: llc -O0 -mtriple=x86_64-apple-darwin %s -o - -filetype=asm | FileCheck %s -check-prefix=ASM-CHECK
 ; vla should have a register-indirect address at one point.
-; ASM-CHECK: DEBUG_VALUE: vla <- RCX
-; ASM-CHECK: DW_OP_breg2
+; ASM-CHECK: DEBUG_VALUE: vla <- RDX
+; ASM-CHECK: DW_OP_breg1
 
 ; RUN: llvm-as %s -o - | llvm-dis - | FileCheck %s --check-prefix=PRETTY-PRINT
 ; PRETTY-PRINT: DIExpression(DW_OP_deref, DW_OP_deref)
 
 define void @testVLAwithSize(i32 %s) nounwind uwtable ssp {
 entry:
   %s.addr = alloca i32, align 4
   %saved_stack = alloca i8*

test/DebugInfo/X86/vla.ll

 ; RUN: llc -O0 -mtriple=x86_64-apple-darwin -filetype=asm %s -o - | FileCheck %s
 ; Ensure that we generate an indirect location for the variable length array a.
-; CHECK: ##DEBUG_VALUE: vla:a <- RDX
-; CHECK: DW_OP_breg1
+; CHECK: ##DEBUG_VALUE: vla:a <- RSI
+; CHECK: DW_OP_breg4
 ; rdar://problem/13658587
 ;
 ; generated from:
 ;
 ; int vla(int n) {
 ;   int a[n];
 ;   a[0] = 42;
 ;   return a[n-1];

test/tools/llvm-symbolizer/ppc64.test

 // ppc64 was compiled from this source on a big-endian 64-bit PowerPC box
 // with just "clang -nostdlib":
 int foo() { return 0; }
 int bar() { return foo(); }
 int _start() { return bar(); }
 
-RUN: %python -c "print('0x1000014c\n0x1000018c\n0x100001cc')" | llvm-symbolizer -obj=%p/Inputs/ppc64 | FileCheck %s
+RUN: "%python" -c "print('0x1000014c\n0x1000018c\n0x100001cc')" | llvm-symbolizer -obj=%p/Inputs/ppc64 | FileCheck %s
 
 CHECK: foo
 CHECK: bar
 CHECK: _start