[SDAG] Introduce a combined set to the DAG combiner which tracks nodes which have successfully round-tripped through the combine phase, and use this to ensure all operands to DAG nodes are visited by the combiner, even if they are only added...

[chandlerc]

Summary

...during the combine phase.

This is critical to have the combiner reach nodes that are *introduced*
during combining. Previously these would sometimes be visited and
sometimes not be visited based on whether they happened to end up on the
worklist or not. Now we always run them through the combiner.

This fixes quite a few bad codegen test cases lurking in the suite while
also being more principled. Among these, the TLS codegeneration is
particularly exciting for programs that have this in the critical path
like TSan-instrumented binaries (although I think they engineer to use
a different TLS that is faster anyways).

However, there are also a worrying number of cases in x86 where this
causes us to combine things into a DAG that misses the expected patterns
and instruction selects to something pretty terrible. =/ The vector sign
extension patterns in particular seem quite broken by this.

Again, assistance would be appreciated from those familiar with other
targets to evaluate the nature of these changes. I haven't even really
vetted Mips or R600 for correctness. That's the primary reason for
needing pre-commit review here.

If anyone wants to take a look at the x86 regressions around sign
extension and add some patterns or fix the combines to survive this
change that would be amazingly helpful...

[chandlerc]

A couple of random other notes about this patch....

• I did benchmark the compile time impact. For a "worst case" scenario in

my mind of taking a single bitcode for all of clang and running it through
just llc I observed a 3% slowdown of llc with this patch. I expect that to
translate into more like 1% or 2% of slowdown for LTO, and well under that
for normal compiles (which have the frontend in the critical path). So the
runtime cost isn't great, but isn't terribly scary to me.

• An obvious question is, why still populate the worklist with all the

nodes? The answer is because just starting from the root and letting the
de-dup'ed operand walk handle the rest causes more tests to fail, and I'm
already struggling to drag the test suite kicking and screaming through
these changes. It still might be nice eventually so that we move to a more
principled graph-traversal strategy. It may recover a (very minor) fraction
of the runtime cost, but nothing huge.

• With this functionality added, I can add legalization support to the last

DAG combine phase and successfully legalize newly created illegal nodes as
we effectively traverse the new nodes. I've successfully implemented the
PSHUFB matching logic on top of this. So this isn't just idle hardening of
the DAG combiner.

[echristo]

I think mips test changes look fine and the patch looks fine as well.

[chandlerc]

FYI, discussed this with some folks on IRC too and they seem happy with it.

However, I want to land http://reviews.llvm.org/D4654 (or something similar) first as that will remove the regressions w.r.t. sext from this change.

[chandlerc]

Closed by commit rL213898 (authored by @chandlerc).

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

class DAGCombiner {
 
     /// \brief Mapping from an SDNode to its position on the worklist.
     ///
     /// This is used to find and remove nodes from the worklist (by nulling
     /// them) when they are deleted from the underlying DAG. It relies on
     /// stable indices of nodes within the worklist.
     DenseMap<SDNode *, unsigned> WorklistMap;
 
+    /// \brief Set of nodes which have been combined (at least once).
+    ///
+    /// This is used to allow us to reliably add any operands of a DAG node
+    /// which have not yet been combined to the worklist.
+    SmallPtrSet<SDNode *, 64> CombinedNodes;
+
     // AA - Used for DAG load/store alias analysis.
     AliasAnalysis &AA;
 
     /// AddUsersToWorklist - When an instruction is simplified, add all users of
     /// the instruction to the work lists because they might get more simplified
     /// now.
     ///
     void AddUsersToWorklist(SDNode *N) {
void AddToWorklist(SDNode *N) {
 
       if (WorklistMap.insert(std::make_pair(N, Worklist.size())).second)
         Worklist.push_back(N);
     }
 
     /// removeFromWorklist - remove all instances of N from the worklist.
     ///
     void removeFromWorklist(SDNode *N) {
+      CombinedNodes.erase(N);
+
       auto It = WorklistMap.find(N);
       if (It == WorklistMap.end())
         return; // Not in the worklist.
 
       // Null out the entry rather than erasing it to avoid a linear operation.
       Worklist[It->second] = nullptr;
       WorklistMap.erase(It);
     }
assert(GoodWorklistEntry &&
            "Found a worklist entry without a corresponding map entry!");
 
     // If N has no uses, it is dead.  Make sure to revisit all N's operands once
     // N is deleted from the DAG, since they too may now be dead or may have a
     // reduced number of uses, allowing other xforms.
     if (recursivelyDeleteUnusedNodes(N))
       continue;
 
+    DEBUG(dbgs() << "\nCombining: ";
+          N->dump(&DAG));
+
+    // Add any operands of the new node which have not yet been combined to the
+    // worklist as well. Because the worklist uniques things already, this
+    // won't repeatedly process the same operand.
+    CombinedNodes.insert(N);
+    for (unsigned i = 0, e = N->getNumOperands(); i != e; ++i)
+      if (!CombinedNodes.count(N->getOperand(i).getNode()))
+        AddToWorklist(N->getOperand(i).getNode());
+
     WorklistRemover DeadNodes(*this);
 
     SDValue RV = combine(N);
 
     if (!RV.getNode())
       continue;
 
     ++NodesCombined;
 
     // If we get back the same node we passed in, rather than a new node or
     // zero, we know that the node must have defined multiple values and
     // CombineTo was used.  Since CombineTo takes care of the worklist
     // mechanics for us, we have no work to do in this case.
     if (RV.getNode() == N)
       continue;
 
     assert(N->getOpcode() != ISD::DELETED_NODE &&
            RV.getNode()->getOpcode() != ISD::DELETED_NODE &&
            "Node was deleted but visit returned new node!");
 
-    DEBUG(dbgs() << "\nReplacing.3 ";
-          N->dump(&DAG);
-          dbgs() << "\nWith: ";
-          RV.getNode()->dump(&DAG);
-          dbgs() << '\n');
+    DEBUG(dbgs() << " ... into: ";
+          RV.getNode()->dump(&DAG));
 
     // Transfer debug value.
     DAG.TransferDbgValues(SDValue(N, 0), RV);
     if (N->getNumValues() == RV.getNode()->getNumValues())
       DAG.ReplaceAllUsesWith(N, RV.getNode());
     else {
       assert(N->getValueType(0) == RV.getValueType() &&
              N->getNumValues() == 1 && "Type mismatch");

llvm/trunk/test/CodeGen/AArch64/arm64-dagcombiner-indexed-load.ll

-; RUN: llc -O3 < %s | FileCheck %s
-; RUN: llc -O3 -addr-sink-using-gep=1 < %s | FileCheck %s
-; Test case for a DAG combiner bug where we combined an indexed load
-; with an extension (sext, zext, or any) into a regular extended load,
-; i.e., dropping the indexed value.
-; <rdar://problem/16389332>
-
-target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
-target triple = "arm64-apple-ios"
-
-%class.A = type { i64, i64 }
-%class.C = type { i64 }
-
-; CHECK-LABEL: XX:
-; CHECK: ldr
-define i32 @XX(%class.A* %K, i1 %tst, i32* %addr, %class.C** %ppC, %class.C* %pC) {
-entry:
-  br i1 %tst, label %if.then, label %lor.rhs.i
-
-lor.rhs.i:                                        ; preds = %entry
-  %tmp = load i32* %addr, align 4
-  %y.i.i.i = getelementptr inbounds %class.A* %K, i64 0, i32 1
-  %tmp1 = load i64* %y.i.i.i, align 8
-  %U.sroa.3.8.extract.trunc.i = trunc i64 %tmp1 to i32
-  %div11.i = sdiv i32 %U.sroa.3.8.extract.trunc.i, 17
-  %add12.i = add nsw i32 0, %div11.i
-  %U.sroa.3.12.extract.shift.i = lshr i64 %tmp1, 32
-  %U.sroa.3.12.extract.trunc.i = trunc i64 %U.sroa.3.12.extract.shift.i to i32
-  %div15.i = sdiv i32 %U.sroa.3.12.extract.trunc.i, 13
-  %add16.i = add nsw i32 %add12.i, %div15.i
-  %rem.i.i = srem i32 %add16.i, %tmp
-  %idxprom = sext i32 %rem.i.i to i64
-  %arrayidx = getelementptr inbounds %class.C** %ppC, i64 %idxprom
-  %tobool533 = icmp eq %class.C* %pC, null
-  br i1 %tobool533, label %while.end, label %while.body
-
-if.then:                                          ; preds = %entry
-  ret i32 42
-
-while.body:                                       ; preds = %lor.rhs.i
-  ret i32 5
-
-while.end:                                        ; preds = %lor.rhs.i
-  %tmp3 = load %class.C** %arrayidx, align 8
-  ret i32 50
-}

llvm/trunk/test/CodeGen/ARM/aapcs-hfa-code.ll

 }
 
 ; Final double argument might go at [sp, #4] if we're careless. Should go at
 ; [sp, #8] to preserve alignment.
 define arm_aapcs_vfpcc void @test_1double_misaligned([4 x double], [4 x double], float, double) {
   call arm_aapcs_vfpcc void @test_1double_misaligned([4 x double] undef, [4 x double] undef, float undef, double 1.0)
 
 ; CHECK-LABEL: test_1double_misaligned:
-; CHECK-DAG: mov [[ONELO:r[0-9]+]], #0
-; CHECK-DAG: mov r[[BASE:[0-9]+]], sp
 ; CHECK-DAG: movw [[ONEHI:r[0-9]+]], #0
+; CHECK-DAG: mov [[ONELO:r[0-9]+]], #0
 ; CHECK-DAG: movt [[ONEHI]], #16368
-; CHECK-DAG: str [[ONELO]], [r[[BASE]], #8]!
-; CHECK-DAG: str [[ONEHI]], [r[[BASE]], #4]
+; CHECK-DAG: strd [[ONELO]], [[ONEHI]], [sp, #8]
 
 ; CHECK-M4F-LABEL: test_1double_misaligned:
 ; CHECK-M4F: movs [[ONELO:r[0-9]+]], #0
 ; CHECK-M4F: movs [[ONEHI:r[0-9]+]], #0
 ; CHECK-M4F: movt [[ONEHI]], #16368
 ; CHECK-M4F-DAG: str [[ONELO]], [sp, #8]
 ; CHECK-M4F-DAG: str [[ONEHI]], [sp, #12]
 ; CHECK-M4F: bl test_1double_misaligned
 
   ret void
 }

llvm/trunk/test/CodeGen/Mips/cmov.ll

 define i32 @slti6(i32 %a) nounwind readnone {
   %1 = icmp slt i32 %a, 7
   %2 = select i1 %1, i32 3, i32 4
   ret i32 %2
 }
 
 ; ALL-LABEL: slti6:
 
-; 32-CMOV-DAG: slti [[R1:\$[0-9]+]], $4, 7
-; 32-CMOV-DAG: xori [[R1]], [[R1]], 1
-; 32-CMOV-DAG: addiu [[R2:\$[0-9]+]], [[R1]], 3
-; 32-CMOV-NOT: movn
-
-; 32-CMP-DAG:  slti [[R1:\$[0-9]+]], $4, 7
-; 32-CMP-DAG:  xori [[R1]], [[R1]], 1
-; 32-CMP-DAG:  addiu [[R2:\$[0-9]+]], [[R1]], 3
-; 32-CMP-NOT:  seleqz
-; 32-CMP-NOT:  selnez
-
-; 64-CMOV-DAG: slti [[R1:\$[0-9]+]], $4, 7
-; 64-CMOV-DAG: xori [[R1]], [[R1]], 1
-; 64-CMOV-DAG: addiu [[R2:\$[0-9]+]], [[R1]], 3
-; 64-CMOV-NOT: movn
-
-; 64-CMP-DAG:  slti [[R1:\$[0-9]+]], $4, 7
-; 64-CMP-DAG:  xori [[R1]], [[R1]], 1
-; 64-CMP-DAG:  addiu [[R2:\$[0-9]+]], [[R1]], 3
-; 64-CMP-NOT:  seleqz
-; 64-CMP-NOT:  selnez
+; ALL-DAG: addiu [[R1:\$[0-9]+]], $zero, 6
+; ALL-DAG: slt [[R1]], [[R1]], $4
+; ALL-DAG: addiu [[R2:\$[0-9]+]], [[R1]], 3
+; ALL-NOT: movn
+; ALL-NOT:  seleqz
+; ALL-NOT:  selnez

llvm/trunk/test/CodeGen/R600/add_i64.ll

define void @test_v2i64_vreg(<2 x i64> addrspace(1)* noalias %out, <2 x i64> addrspace(1)* noalias %inA, <2 x i64> addrspace(1)* noalias %inB) {
   %a = load <2 x i64> addrspace(1)* %a_ptr
   %b = load <2 x i64> addrspace(1)* %b_ptr
   %result = add <2 x i64> %a, %b
   store <2 x i64> %result, <2 x i64> addrspace(1)* %out
   ret void
 }
 
 ; SI-LABEL: @trunc_i64_add_to_i32
-; SI: S_LOAD_DWORDX2 s{{\[}}[[SREG0:[0-9]+]]
-; SI: S_LOAD_DWORDX2 s{{\[}}[[SREG1:[0-9]+]]
+; SI: S_LOAD_DWORD s[[SREG0:[0-9]+]]
+; SI: S_LOAD_DWORD s[[SREG1:[0-9]+]]
 ; SI: S_ADD_I32 [[SRESULT:s[0-9]+]], s[[SREG1]], s[[SREG0]]
 ; SI-NOT: ADDC
 ; SI: V_MOV_B32_e32 [[VRESULT:v[0-9]+]], [[SRESULT]]
 ; SI: BUFFER_STORE_DWORD [[VRESULT]],
 define void @trunc_i64_add_to_i32(i32 addrspace(1)* %out, i64 %a, i64 %b) {
   %add = add i64 %b, %a
   %trunc = trunc i64 %add to i32
   store i32 %trunc, i32 addrspace(1)* %out, align 8
   ret void
 }

llvm/trunk/test/CodeGen/R600/or.ll

 define void @vector_or_i64_imm(i64 addrspace(1)* %out, i64 addrspace(1)* %a, i64 addrspace(1)* %b) {
   %loada = load i64 addrspace(1)* %a, align 8
   %or = or i64 %loada, 8
   store i64 %or, i64 addrspace(1)* %out
   ret void
 }
 
 ; SI-LABEL: @trunc_i64_or_to_i32
-; SI: S_LOAD_DWORDX2 s{{\[}}[[SREG0:[0-9]+]]
-; SI: S_LOAD_DWORDX2 s{{\[}}[[SREG1:[0-9]+]]
-; SI: S_OR_B32 [[SRESULT:s[0-9]+]], s[[SREG1]], s[[SREG0]]
-; SI: V_MOV_B32_e32 [[VRESULT:v[0-9]+]], [[SRESULT]]
+; SI: S_LOAD_DWORD s[[SREG0:[0-9]+]]
+; SI: S_LOAD_DWORD s[[SREG1:[0-9]+]]
+; SI: S_OR_B32 s[[SRESULT:[0-9]+]], s[[SREG1]], s[[SREG0]]
+; SI: V_MOV_B32_e32 [[VRESULT:v[0-9]+]], s[[SRESULT]]
 ; SI: BUFFER_STORE_DWORD [[VRESULT]],
 define void @trunc_i64_or_to_i32(i32 addrspace(1)* %out, i64 %a, i64 %b) {
   %add = or i64 %b, %a
   %trunc = trunc i64 %add to i32
   store i32 %trunc, i32 addrspace(1)* %out, align 8
   ret void
 }
 

llvm/trunk/test/CodeGen/X86/2010-04-23-mmx-movdq2q.ll

 ; RUN: llc < %s -mtriple=x86_64-apple-darwin -mattr=+mmx,+sse2 | FileCheck %s
 ; There are no MMX operations here, so we use XMM or i64.
 
 ; CHECK: ti8
 define void @ti8(double %a, double %b) nounwind {
 entry:
         %tmp1 = bitcast double %a to <8 x i8>
         %tmp2 = bitcast double %b to <8 x i8>
         %tmp3 = add <8 x i8> %tmp1, %tmp2
-; CHECK:  paddw
+; CHECK:  paddb
         store <8 x i8> %tmp3, <8 x i8>* null
         ret void
 }
 
 ; CHECK: ti16
 define void @ti16(double %a, double %b) nounwind {
 entry:
         %tmp1 = bitcast double %a to <4 x i16>
         %tmp2 = bitcast double %b to <4 x i16>
         %tmp3 = add <4 x i16> %tmp1, %tmp2
-; CHECK:  paddd
+; CHECK:  paddw
         store <4 x i16> %tmp3, <4 x i16>* null
         ret void
 }
 
 ; CHECK: ti32
 define void @ti32(double %a, double %b) nounwind {
 entry:
         %tmp1 = bitcast double %a to <2 x i32>
         %tmp2 = bitcast double %b to <2 x i32>
         %tmp3 = add <2 x i32> %tmp1, %tmp2
-; CHECK:  paddq
+; CHECK:  paddd
         store <2 x i32> %tmp3, <2 x i32>* null
         ret void
 }
 
 ; CHECK: ti64
 define void @ti64(double %a, double %b) nounwind {
 entry:
         %tmp1 = bitcast double %a to <1 x i64>

llvm/trunk/test/CodeGen/X86/i8-umulo.ll

 ; RUN: llc -mcpu=generic -march=x86 < %s | FileCheck %s
 ; PR19858
 
 declare {i8, i1} @llvm.umul.with.overflow.i8(i8 %a, i8 %b)
 define i8 @testumulo(i32 %argc) {
-; CHECK: imulw
+; CHECK: imull
 ; CHECK: testb %{{.+}}, %{{.+}}
 ; CHECK: je [[NOOVERFLOWLABEL:.+]]
 ; CHECK: {{.*}}[[NOOVERFLOWLABEL]]:
 ; CHECK-NEXT: movb
 ; CHECK-NEXT: retl
 top:
   %RHS = trunc i32 %argc to i8
   %umul = call { i8, i1 } @llvm.umul.with.overflow.i8(i8 25, i8 %RHS)

llvm/trunk/test/CodeGen/X86/jump_sign.ll

 }
 
 ; PR13966
 @b = common global i32 0, align 4
 @a = common global i32 0, align 4
 define i32 @func_test1(i32 %p1) nounwind uwtable {
 entry:
 ; CHECK-LABEL: func_test1:
-; CHECK: testb
+; CHECK: andb
 ; CHECK: j
 ; CHECK: ret
   %0 = load i32* @b, align 4
   %cmp = icmp ult i32 %0, %p1
   %conv = zext i1 %cmp to i32
   %1 = load i32* @a, align 4
   %and = and i32 %conv, %1
   %conv1 = trunc i32 %and to i8

llvm/trunk/test/CodeGen/X86/lower-bitcast.ll

 
 
 define i64 @test4(i64 %A) {
   %1 = bitcast i64 %A to <2 x i32>
   %add = add <2 x i32> %1, <i32 3, i32 5>
   %2 = bitcast <2 x i32> %add to i64
   ret i64 %2
 }
-; FIXME: At the moment we still produce the sequence pshufd+paddq+pshufd.
+; FIXME: At the moment we still produce the sequence pshufd+paddd+pshufd.
 ; Ideally, we should fold that sequence into a single paddd. This is fixed with
 ; the widening legalization.
 ;
 ; CHECK-LABEL: test4
 ; CHECK: pshufd
-; CHECK-NEXT: paddq
+; CHECK-NEXT: paddd
 ; CHECK-NEXT: pshufd
 ; CHECK: ret
 ;
 ; CHECK-WIDE-LABEL: test4
 ; CHECK-WIDE: movd %{{rdi|rcx}},
 ; CHECK-WIDE-NEXT: paddd
 ; CHECK-WIDE-NEXT: movd {{.*}}, %rax
 ; CHECK-WIDE: ret

llvm/trunk/test/CodeGen/X86/pr15267.ll

 define <4 x i64> @test3(<4 x i1>* %in) nounwind {
   %wide.load35 = load <4 x i1>* %in, align 1
   %sext = sext <4 x i1> %wide.load35 to <4 x i64>
   ret <4 x i64> %sext
 }
 
 ; CHECK: test3
 ; CHECK: movzbl
-; CHECK: shrl
-; CHECK: andl $1
-; CHECK: andl $1
-; CHECK: vmovd
-; CHECK: pinsrd $1
-; CHECK: shrl $2
-; CHECK: andl $1
-; CHECK: pinsrd $2
-; CHECK: shrl $3
-; CHECK: andl $1
-; CHECK: pinsrd $3
-; CHECK: pslld
-; CHECK: psrad
-; CHECK: pmovsxdq
-; CHECK: pmovsxdq
+; CHECK: movq
+; CHECK: shlq
+; CHECK: sarq
+; CHECK: vmovq
+; CHECK: movq
+; CHECK: shlq
+; CHECK: sarq
+; CHECK: vmovq
+; CHECK: vpunpcklqdq
+; CHECK: movq
+; CHECK: shlq
+; CHECK: sarq
+; CHECK: vmovq
+; CHECK: shlq
+; CHECK: sarq
+; CHECK: vmovq
+; CHECK: vpunpcklqdq
+; CHECK: vinsertf128
 ; CHECK: ret

llvm/trunk/test/CodeGen/X86/store-narrow.ll

entry:
   %CS = shl i32 %C, 8
   %D = or i32 %B, %CS
   store i32 %D, i32* %a0, align 4
   ret void
 ; X64-LABEL: test2:
 ; X64: movb %sil, 1(%rdi)
 
 ; X32-LABEL: test2:
-; X32: movzbl 8(%esp), %e[[REG:[abcd]]]x
+; X32: movb 8(%esp), %[[REG:[abcd]]]l
 ; X32: movb %[[REG]]l, 1(%{{.*}})
 }
 
 define void @test3(i32* nocapture %a0, i16 zeroext %a1) nounwind ssp {
 entry:
   %A = load i32* %a0, align 4
   %B = and i32 %A, -65536    ; 0xFFFF0000
   %C = zext i16 %a1 to i32
entry:
   %CS = shl i32 %C, 16
   %D = or i32 %B, %CS
   store i32 %D, i32* %a0, align 4
   ret void
 ; X64-LABEL: test4:
 ; X64: movw %si, 2(%rdi)
 
 ; X32-LABEL: test4:
-; X32: movl 8(%esp), %e[[REG:[abcd]x]]
-; X32: movw %[[REG]], 2(%{{.*}})
+; X32: movw 8(%esp), %[[REG:[abcd]]]x
+; X32: movw %[[REG]]x, 2(%{{.*}})
 }
 
 define void @test5(i64* nocapture %a0, i16 zeroext %a1) nounwind ssp {
 entry:
   %A = load i64* %a0, align 4
   %B = and i64 %A, -4294901761    ; 0xFFFFFFFF0000FFFF
   %C = zext i16 %a1 to i64
   %CS = shl i64 %C, 16
   %D = or i64 %B, %CS
   store i64 %D, i64* %a0, align 4
   ret void
 ; X64-LABEL: test5:
 ; X64: movw %si, 2(%rdi)
 
 ; X32-LABEL: test5:
-; X32: movzwl 8(%esp), %e[[REG:[abcd]x]]
-; X32: movw %[[REG]], 2(%{{.*}})
+; X32: movw 8(%esp), %[[REG:[abcd]]]x
+; X32: movw %[[REG]]x, 2(%{{.*}})
 }
 
 define void @test6(i64* nocapture %a0, i8 zeroext %a1) nounwind ssp {
 entry:
   %A = load i64* %a0, align 4
   %B = and i64 %A, -280375465082881    ; 0xFFFF00FFFFFFFFFF
   %C = zext i8 %a1 to i64
   %CS = shl i64 %C, 40

llvm/trunk/test/CodeGen/X86/trunc-ext-ld-st.ll

define void @load_2_i16(<2 x i16>* %A)  {
    %T = load <2 x i16>* %A
    %G = add <2 x i16> %T, <i16 9, i16 7>
    store <2 x i16> %G, <2 x i16>* %A
    ret void
 } 
 
 ;CHECK-LABEL: load_2_i32:
 ;CHECK: pmovzxdq
-;CHECK: paddq
+;CHECK: paddd
 ;CHECK: pshufd
 ;CHECK: ret
 define void @load_2_i32(<2 x i32>* %A)  {
    %T = load <2 x i32>* %A
    %G = add <2 x i32> %T, <i32 9, i32 7>
    store <2 x i32> %G, <2 x i32>* %A
    ret void
 } 
 
 ;CHECK-LABEL: load_4_i8:
 ;CHECK: pmovzxbd
 ;CHECK: paddd
 ;CHECK: pshufb
 ;CHECK: ret
 define void @load_4_i8(<4 x i8>* %A)  {
    %T = load <4 x i8>* %A
    %G = add <4 x i8> %T, <i8 1, i8 4, i8 9, i8 7>
    store <4 x i8> %G, <4 x i8>* %A
    ret void
 } 
 
 ;CHECK-LABEL: load_4_i16:
 ;CHECK: pmovzxwd
-;CHECK: paddd
+;CHECK: paddw
 ;CHECK: pshufb
 ;CHECK: ret
 define void @load_4_i16(<4 x i16>* %A)  {
    %T = load <4 x i16>* %A
    %G = add <4 x i16> %T, <i16 1, i16 4, i16 9, i16 7>
    store <4 x i16> %G, <4 x i16>* %A
    ret void
 } 
 
 ;CHECK-LABEL: load_8_i8:
 ;CHECK: pmovzxbw
-;CHECK: paddw
+;CHECK: paddb
 ;CHECK: pshufb
 ;CHECK: ret
 define void @load_8_i8(<8 x i8>* %A)  {
    %T = load <8 x i8>* %A
    %G = add <8 x i8> %T, %T
    store <8 x i8> %G, <8 x i8>* %A
    ret void
 } 

llvm/trunk/test/CodeGen/X86/vector-idiv.ll

 
 define <4 x i32> @test8(<4 x i32> %a) {
   %div = sdiv <4 x i32> %a, <i32 7, i32 7, i32 7, i32 7>
   ret <4 x i32> %div
 
 ; SSE41-LABEL: test8:
 ; SSE41: pmuldq
 ; SSE41: pshufd $49
-; SSE41-NOT: pshufd $49
+; SSE41: pshufd $49
 ; SSE41: pmuldq
 ; SSE41: shufps $-35
 ; SSE41: pshufd $-40
 ; SSE41: padd
 ; SSE41: psrld $31
 ; SSE41: psrad $2
 ; SSE41: padd
 
 ; SSE-LABEL: test8:
 ; SSE: pmuludq
 ; SSE: pshufd $49
-; SSE-NOT: pshufd $49
+; SSE: pshufd $49
 ; SSE: pmuludq
 ; SSE: shufps $-35
 ; SSE: pshufd $-40
 ; SSE: psubd
 ; SSE: padd
 ; SSE: psrld $31
 ; SSE: psrad $2
 ; SSE: padd
 
 ; AVX-LABEL: test8:
 ; AVX: vpmuldq
 ; AVX: vpshufd  $49
-; AVX-NOT: vpshufd  $49
+; AVX: vpshufd  $49
 ; AVX: vpmuldq
 ; AVX: vshufps  $-35
 ; AVX: vpshufd  $-40
 ; AVX: vpadd
 ; AVX: vpsrld $31
 ; AVX: vpsrad $2
 ; AVX: vpadd
 }
 
 define <8 x i32> @test9(<8 x i32> %a) {
   %div = sdiv <8 x i32> %a, <i32 7, i32 7, i32 7, i32 7,i32 7, i32 7, i32 7, i32 7>
   ret <8 x i32> %div
 
 ; AVX-LABEL: test9:
-; AVX: vpalignr $4
 ; AVX: vpbroadcastd
+; AVX: vpalignr $4
+; AVX: vpalignr $4
 ; AVX: vpmuldq
 ; AVX: vpmuldq
+; AVX: vpalignr $4
 ; AVX: vpblendd $170
 ; AVX: vpadd
 ; AVX: vpsrld $31
 ; AVX: vpsrad $2
 ; AVX: vpadd
 }
 
 define <8 x i32> @test10(<8 x i32> %a) {

 ; AVX: vpmulld
 }
 
 define <8 x i32> @test11(<8 x i32> %a) {
   %rem = srem <8 x i32> %a, <i32 7, i32 7, i32 7, i32 7,i32 7, i32 7, i32 7, i32 7>
   ret <8 x i32> %rem
 
 ; AVX-LABEL: test11:
-; AVX: vpalignr $4
 ; AVX: vpbroadcastd
+; AVX: vpalignr $4
+; AVX: vpalignr $4
 ; AVX: vpmuldq
 ; AVX: vpmuldq
+; AVX: vpalignr $4
 ; AVX: vpblendd $170
 ; AVX: vpadd
 ; AVX: vpsrld $31
 ; AVX: vpsrad $2
 ; AVX: vpadd
 ; AVX: vpmulld
 }
 

llvm/trunk/test/CodeGen/X86/widen_cast-1.ll

 ; RUN: llc -march=x86 -mcpu=generic -mattr=+sse4.2 < %s | FileCheck %s
 ; RUN: llc -march=x86 -mcpu=atom < %s | FileCheck -check-prefix=ATOM %s
 
 ; CHECK: movl
-; CHECK: paddd
+; CHECK: paddw
 ; CHECK: movlpd
 
 ; Scheduler causes produce a different instruction order
 ; ATOM: movl
-; ATOM: paddd
+; ATOM: paddw
 ; ATOM: movlpd
 
 ; bitcast a v4i16 to v2i32
 
 define void @convert(<2 x i32>* %dst, <4 x i16>* %src) nounwind {
 entry:
   %dst.addr = alloca <2 x i32>*    ; <<2 x i32>**> [#uses=2]
   %src.addr = alloca <4 x i16>*    ; <<4 x i16>**> [#uses=2]

llvm/trunk/test/CodeGen/X86/widen_conv-1.ll

 ; RUN: llc < %s -march=x86 -mattr=+sse4.2 | FileCheck %s
-; CHECK: paddq
+; CHECK: paddd
 
 ; truncate v2i64 to v2i32
 
 define void @convert(<2 x i32>* %dst.addr, <2 x i64> %src) nounwind {
 entry:
   %val = trunc <2 x i64> %src to <2 x i32>
   %add = add <2 x i32> %val, < i32 1, i32 1 >
   store <2 x i32> %add, <2 x i32>* %dst.addr
   ret void
 }

llvm/trunk/test/CodeGen/X86/widen_load-2.ll

; CHECK-NEXT:    movd     %[[R0]], (%{{.*}})
   %x = add %i16vec3 %a, %b
   store %i16vec3 %x, %i16vec3* %ret, align 16
   ret void
 }
 
 %i16vec4 = type <4 x i16>
 define void @add4i16(%i16vec4* nocapture sret %ret, %i16vec4* %ap, %i16vec4* %bp) nounwind {
 ; CHECK-LABEL: add4i16:
-; CHECK:         pmovzxwd (%{{.*}}), %[[R0:xmm[0-9]+]]
-; CHECK-NEXT:    pmovzxwd (%{{.*}}), %[[R1:xmm[0-9]+]]
-; CHECK-NEXT:    paddd   %[[R0]], %[[R1]]
-; CHECK-NEXT:    pshufb  {{.*}}, %[[R1]]
+; CHECK:         movq    (%{{.*}}), %[[R0:xmm[0-9]+]]
+; CHECK-NEXT:    movq    (%{{.*}}), %[[R1:xmm[0-9]+]]
+; CHECK-NEXT:    paddw   %[[R0]], %[[R1]]
 ; CHECK-NEXT:    movq    %[[R1]], (%{{.*}})
   %a = load %i16vec4* %ap, align 16
   %b = load %i16vec4* %bp, align 16
   %x = add %i16vec4 %a, %b
   store %i16vec4 %x, %i16vec4* %ret, align 16
   ret void
 }
 

llvm/trunk/test/CodeGen/X86/x86-64-tls-1.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 @tm_nest_level = internal thread_local global i32 0
 define i64 @z() nounwind {
-; FIXME: The codegen here is primitive at best and could be much better.
-; The add and the moves can be folded together.
-; CHECK-DAG: movq    $tm_nest_level@TPOFF, %rcx
-; CHECK-DAG: movq    %fs:0, %rax
-; CHECK: addl    %ecx, %eax
+; CHECK:      movq    $tm_nest_level@TPOFF, %r[[R0:[abcd]]]x
+; CHECK-NEXT: addl    %fs:0, %e[[R0]]x
+; CHECK-NEXT: andq    $100, %r[[R0]]x
+
   ret i64 and (i64 ptrtoint (i32* @tm_nest_level to i64), i64 100)
 }

llvm/trunk/test/CodeGen/X86/x86-setcc-int-to-fp-combine.ll

 ; RUN: llc < %s -mtriple=x86_64-apple-darwin | FileCheck %s
 
 define <4 x float> @foo(<4 x float> %val, <4 x float> %test) nounwind {
 ; CHECK-LABEL: LCPI0_0:
-; CHECK-NEXT: .long 1065353216              ## float 1.000000e+00
-; CHECK-NEXT: .long 1065353216              ## float 1.000000e+00
-; CHECK-NEXT: .long 1065353216              ## float 1.000000e+00
-; CHECK-NEXT: .long 1065353216              ## float 1.000000e+00
+; CHECK-NEXT: .long 1065353216              ## 0x3f800000
+; CHECK-NEXT: .long 1065353216              ## 0x3f800000
+; CHECK-NEXT: .long 1065353216              ## 0x3f800000
+; CHECK-NEXT: .long 1065353216              ## 0x3f800000
 ; CHECK-LABEL: foo:
 ; CHECK: cmpeqps %xmm1, %xmm0
 ; CHECK-NEXT: andps LCPI0_0(%rip), %xmm0
 ; CHECK-NEXT: retq
 
   %cmp = fcmp oeq <4 x float> %val, %test
   %ext = zext <4 x i1> %cmp to <4 x i32>
   %result = sitofp <4 x i32> %ext to <4 x float>
; CHECK:  movaps LCPI2_0(%rip), %xmm0
   store <4 x float> %val, <4 x float>* %result
   ret void
 }
 
 ; Fold explicit AND operations when the constant isn't a splat of a single
 ; scalar value like what the zext creates.
 define <4 x float> @foo3(<4 x float> %val, <4 x float> %test) nounwind {
 ; CHECK-LABEL: LCPI3_0:
-; CHECK-NEXT: .long 1065353216              ## float 1.000000e+00
-; CHECK-NEXT: .long 0                       ## float 0.000000e+00
-; CHECK-NEXT: .long 1065353216              ## float 1.000000e+00
-; CHECK-NEXT: .long 0                       ## float 0.000000e+00
+; CHECK-NEXT: .long 1065353216              ## 0x3f800000
+; CHECK-NEXT: .long 0                       ## 0x0
+; CHECK-NEXT: .long 1065353216              ## 0x3f800000
+; CHECK-NEXT: .long 0                       ## 0x0
 ; CHECK-LABEL: foo3:
 ; CHECK: cmpeqps %xmm1, %xmm0
 ; CHECK-NEXT: andps LCPI3_0(%rip), %xmm0
   %cmp = fcmp oeq <4 x float> %val, %test
   %ext = zext <4 x i1> %cmp to <4 x i32>
   %and = and <4 x i32> %ext, <i32 255, i32 256, i32 257, i32 258>
   %result = sitofp <4 x i32> %and to <4 x float>
   ret <4 x float> %result
 }