Sparc: Implement i64 load/store support for 32-bit sparc.

[jyknight]

Summary

The LDD/STD instructions can load/store a 64bit quantity from/to
memory to/from a consecutive even/odd pair of (32-bit) registers. They
are part of SparcV8, and also present in SparcV9. (Although deprecated
there, as you can store 64bits in one register).

As recommended on llvmdev in the thread "How to enable use of 64bit
load/store for 32bit architecture" from Apr 2015, I've modeled the
64-bit load/store operations as working on a v2i32 type, rather than
making i64 a legal type, but with few legal operations. The latter
does not (currently) work, as there is much code in llvm which assumes
that if i64 is legal, operations like "add" will actually work on it.

The same assumption does not hold for v2i32 -- for vector types, it is
workable to support only load/store, and expand everything else.

This patch:

• Adds a new register class, IntPair, for even/odd pairs of registers.

• Modifies the list of reserved registers, the stack spilling code, and register copying code to support the IntPair register class.

• Adds support in AsmParser. (note that in asm text, you write the name of the first register of the pair only. So the parser has to morph the single register into the equivalent paired register).

• Adds the new instructions themselves (LDD/STD/LDDA/STDA).

• Hooks up the instructions and registers as a vector type v2i32. Adds custom legalizer to transform i64 load/stores into v2i32 load/stores and bitcasts, so that the new instructions can actually be generated, and marks all operations other than load/store on v2i32 as needing to be expanded.

• Copies the unfortunate SelectInlineAsm hack from ARMISelDAGToDAG. This hack undoes the transformation of i64 operands into two arbitrarily-allocated separate i32 registers in SelectionDAGBuilder. and instead passes them in a single IntPair. (Arbitrarily allocated registers are not useful, asm code expects to be receiving a pair, which can be passed to ldd/std.)

Also adds a bunch of test cases covering all the bugs I've added along
the way.

[jyknight]

Implemented support for int64 stores/loads using the new instructions.

[jyknight]

Adds more test cases.
Fixes inline asm register constraint register class selection.
Fixes remapRegsForLeafProc.

[jyknight]

Ping

[jyknight]

Discovered and fixed a bunch of bugs and added a bunch more test cases.

[t.p.northover]

lib/Target/Sparc/SparcFrameLowering.cpp

232–235 ↗

(On Diff #25709)

Don't you need to mark the pair unused as well? I wouldn't swear to this, but a quick look suggests that marking I0_I1 unused would implicitly mark I0 and I1 unused, but not the converse.

245 ↗

(On Diff #25709)

Is this delta correct? I'd have expected "reg - SP::I0_I1 + SP::O0_O1" unless I'm misunderstanding the logic.

lib/Target/Sparc/SparcISelLowering.cpp

2637–2638

I don't understand this: it's the only place a 128-bit type is mentioned at all in the patch. Why does the default action change?

3305

Commented code.

[jyknight]

Thanks a bunch for your notes!

lib/Target/Sparc/SparcFrameLowering.cpp

232–235 ↗

(On Diff #25709)

As far as I can tell, I0_I1 (etc) don't ever get used bits set on them, so they don't need to be unmarked either.

245 ↗

(On Diff #25709)

This code is indeed *not* correct. Thanks for catching that!! Any idea how to cover that in a test case?

lib/Target/Sparc/SparcISelLowering.cpp

2637–2638

Not quite the only place, see also LowerSTORE, which has a similar piece of code.

Note the call to LowerLOAD from LowerOperation: the code there used to say "return LowerF128Load(Op, DAG);", which simply assumed that the type passed in was *always* MVT::f128 when called with ISD::LOAD. That's no longer true, because MVT::i64 is also marked for Custom handling via setOperationAction, so LowerOperation could be called with i64, now too.

Unlike store, I don't actually need any custom handling for it here (because the custom handling for it occurs in ReplaceNodeResults, instead), but this code does need to not do the f128 stuff for it.

[t.p.northover]

lib/Target/Sparc/SparcFrameLowering.cpp

232–235 ↗

(On Diff #25709)

Ah, that makes sense. Their fundamental Units are already covered in the outer loop.

245 ↗

(On Diff #25709)

Not sure. It might be visible in "llc -debug", but I'm not sure we encourage tests like that (it would certainly need a "REQUIRES: asserts" line).

lib/Target/Sparc/SparcISelLowering.cpp

2637–2638

Ah yes, I completely missed the 2-line LowerF128 directly in the switch. Sorry about that.

[jyknight]

Minor tweaks as requested. Also included a new test file I'd neglected
to include last time (reserved-regs.ll)

[jyknight]

Note that there's still an issue with this: it breaks calling a function with an v2i32 type, because I didn't add calling convention support for it -- even though I want it to have the default behavior, were that not a legal type, it still needs explicit handling. (That was covered by a test case only run against the default target)

I also plan to fix the core inlineasm code, so that the SelectInlineAsm hack can be deleted.

[echristo]

Think you need to clang format bits, but otherwise looks ok. Request for a comment inline.

Thanks!

-eric

lib/Target/Sparc/SparcInstrInfo.cpp

299–304 ↗

(On Diff #25997)

This could use a comment.

[jyknight]

Update to trunk, fix CodeGen/Generic/vector.ll test failure by
implementing v2i32 calling convention.

lib/Target/Sparc/AsmParser/SparcAsmParser.cpp

static unsigned ASRRegs[32] = {
     SP::ASR4,  SP::ASR5,  SP::ASR6, SP::ASR7,
     SP::ASR8,  SP::ASR9,  SP::ASR10, SP::ASR11,
     SP::ASR12, SP::ASR13, SP::ASR14, SP::ASR15,
     SP::ASR16, SP::ASR17, SP::ASR18, SP::ASR19,
     SP::ASR20, SP::ASR21, SP::ASR22, SP::ASR23,
     SP::ASR24, SP::ASR25, SP::ASR26, SP::ASR27,
     SP::ASR28, SP::ASR29, SP::ASR30, SP::ASR31};
 
+  static unsigned IntPairRegs[] = {
+    Sparc::G0_G1, Sparc::G2_G3, Sparc::G4_G5, Sparc::G6_G7,
+    Sparc::O0_O1, Sparc::O2_O3, Sparc::O4_O5, Sparc::O6_O7,
+    Sparc::L0_L1, Sparc::L2_L3, Sparc::L4_L5, Sparc::L6_L7,
+    Sparc::I0_I1, Sparc::I2_I3, Sparc::I4_I5, Sparc::I6_I7};
+
 /// SparcOperand - Instances of this class represent a parsed Sparc machine
 /// instruction.
 class SparcOperand : public MCParsedAsmOperand {
 public:
   enum RegisterKind {
     rk_None,
     rk_IntReg,
+    rk_IntPairReg,
     rk_FloatReg,
     rk_DoubleReg,
     rk_QuadReg,
     rk_CCReg,
     rk_ASRReg
   };
 private:
   enum KindTy {
public:
 
   bool isToken() const override { return Kind == k_Token; }
   bool isReg() const override { return Kind == k_Register; }
   bool isImm() const override { return Kind == k_Immediate; }
   bool isMem() const override { return isMEMrr() || isMEMri(); }
   bool isMEMrr() const { return Kind == k_MemoryReg; }
   bool isMEMri() const { return Kind == k_MemoryImm; }
 
+  bool isIntReg() const {
+    return (Kind == k_Register && Reg.Kind == rk_IntReg);
+  }
+
   bool isFloatReg() const {
     return (Kind == k_Register && Reg.Kind == rk_FloatReg);
   }
 
   bool isFloatOrDoubleReg() const {
     return (Kind == k_Register && (Reg.Kind == rk_FloatReg
                                    || Reg.Kind == rk_DoubleReg));
   }
static std::unique_ptr<SparcOperand> CreateImm(const MCExpr *Val, SMLoc S,
                                                  SMLoc E) {
     auto Op = make_unique<SparcOperand>(k_Immediate);
     Op->Imm.Val = Val;
     Op->StartLoc = S;
     Op->EndLoc = E;
     return Op;
   }
 
+  static bool MorphToIntPairReg(SparcOperand &Op) {
+    unsigned Reg = Op.getReg();
+    assert(Op.Reg.Kind == rk_IntReg);
+    unsigned regIdx = 32;
+    if (Reg >= Sparc::G0 && Reg <= Sparc::G7)
+      regIdx = Reg - Sparc::G0;
+    else if (Reg >= Sparc::O0 && Reg <= Sparc::O7)
+      regIdx = Reg - Sparc::O0 + 8;
+    else if (Reg >= Sparc::L0 && Reg <= Sparc::L7)
+      regIdx = Reg - Sparc::L0 + 16;
+    else if (Reg >= Sparc::I0 && Reg <= Sparc::I7)
+      regIdx = Reg - Sparc::I0 + 24;
+    if (regIdx % 2 || regIdx > 31)
+      return false;
+    Op.Reg.RegNum = IntPairRegs[regIdx / 2];
+    Op.Reg.Kind = rk_IntPairReg;
+    return true;
+  }
+
   static bool MorphToDoubleReg(SparcOperand &Op) {
     unsigned Reg = Op.getReg();
     assert(Op.Reg.Kind == rk_FloatReg);
     unsigned regIdx = Reg - Sparc::F0;
     if (regIdx % 2 || regIdx > 31)
       return false;
     Op.Reg.RegNum = DoubleRegs[regIdx / 2];
     Op.Reg.Kind = rk_DoubleReg;
case MCK_DFPRegs:
         return MCTargetAsmParser::Match_Success;
       break;
     case MCK_QFPRegs:
       if (SparcOperand::MorphToQuadReg(Op))
         return MCTargetAsmParser::Match_Success;
       break;
     }
   }
+  if (Op.isIntReg() && Kind == MCK_IntPair) {
+    if (SparcOperand::MorphToIntPairReg(Op))
+      return MCTargetAsmParser::Match_Success;
+  }
   return Match_InvalidOperand;
 }

lib/Target/Sparc/Disassembler/SparcDisassembler.cpp

static const unsigned ASRRegDecoderTable[] = {
   SP::ASR4,  SP::ASR5,  SP::ASR6,  SP::ASR7,
   SP::ASR8,  SP::ASR9,  SP::ASR10, SP::ASR11,
   SP::ASR12, SP::ASR13, SP::ASR14, SP::ASR15,
   SP::ASR16, SP::ASR17, SP::ASR18, SP::ASR19,
   SP::ASR20, SP::ASR21, SP::ASR22, SP::ASR23,
   SP::ASR24, SP::ASR25, SP::ASR26, SP::ASR27,
   SP::ASR28, SP::ASR29, SP::ASR30, SP::ASR31};
 
+static const uint16_t IntPairDecoderTable[] = {
+  SP::G0_G1, SP::G2_G3, SP::G4_G5, SP::G6_G7,
+  SP::O0_O1, SP::O2_O3, SP::O4_O5, SP::O6_O7,
+  SP::L0_L1, SP::L2_L3, SP::L4_L5, SP::L6_L7,
+  SP::I0_I1, SP::I2_I3, SP::I4_I5, SP::I6_I7,
+};
+
 static DecodeStatus DecodeIntRegsRegisterClass(MCInst &Inst,
                                                unsigned RegNo,
                                                uint64_t Address,
                                                const void *Decoder) {
   if (RegNo > 31)
     return MCDisassembler::Fail;
   unsigned Reg = IntRegDecoderTable[RegNo];
   Inst.addOperand(MCOperand::CreateReg(Reg));
static DecodeStatus DecodeASRRegsRegisterClass(MCInst &Inst, unsigned RegNo,
                                                uint64_t Address,
                                                const void *Decoder) {
   if (RegNo > 31)
     return MCDisassembler::Fail;
   Inst.addOperand(MCOperand::CreateReg(ASRRegDecoderTable[RegNo]));
   return MCDisassembler::Success;
 }
 
+static DecodeStatus DecodeIntPairRegisterClass(MCInst &Inst, unsigned RegNo,
+                                   uint64_t Address, const void *Decoder) {
+  DecodeStatus S = MCDisassembler::Success;
+
+  if (RegNo > 31)
+    return MCDisassembler::Fail;
+
+  if ((RegNo & 1))
+    S = MCDisassembler::SoftFail;
+
+  unsigned RegisterPair = IntPairDecoderTable[RegNo/2];
+  Inst.addOperand(MCOperand::CreateReg(RegisterPair));
+  return S;
+}
 
 static DecodeStatus DecodeLoadInt(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder);
+static DecodeStatus DecodeLoadIntPair(MCInst &Inst, unsigned insn, uint64_t Address,
+                                  const void *Decoder);
 static DecodeStatus DecodeLoadFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                  const void *Decoder);
 static DecodeStatus DecodeLoadDFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder);
 static DecodeStatus DecodeLoadQFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder);
 static DecodeStatus DecodeStoreInt(MCInst &Inst, unsigned insn,
                                    uint64_t Address, const void *Decoder);
+static DecodeStatus DecodeStoreIntPair(MCInst &Inst, unsigned insn,
+                                   uint64_t Address, const void *Decoder);
 static DecodeStatus DecodeStoreFP(MCInst &Inst, unsigned insn,
                                   uint64_t Address, const void *Decoder);
 static DecodeStatus DecodeStoreDFP(MCInst &Inst, unsigned insn,
                                    uint64_t Address, const void *Decoder);
 static DecodeStatus DecodeStoreQFP(MCInst &Inst, unsigned insn,
                                    uint64_t Address, const void *Decoder);
 static DecodeStatus DecodeCall(MCInst &Inst, unsigned insn,
                                uint64_t Address, const void *Decoder);

 }
 
 static DecodeStatus DecodeLoadInt(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder) {
   return DecodeMem(Inst, insn, Address, Decoder, true,
                    DecodeIntRegsRegisterClass);
 }
 
+static DecodeStatus DecodeLoadIntPair(MCInst &Inst, unsigned insn, uint64_t Address,
+                                  const void *Decoder) {
+  return DecodeMem(Inst, insn, Address, Decoder, true,
+                   DecodeIntPairRegisterClass);
+}
+
 static DecodeStatus DecodeLoadFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                  const void *Decoder) {
   return DecodeMem(Inst, insn, Address, Decoder, true,
                    DecodeFPRegsRegisterClass);
 }
 
 static DecodeStatus DecodeLoadDFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder) {
   return DecodeMem(Inst, insn, Address, Decoder, true,
                    DecodeDFPRegsRegisterClass);
 }
 
 static DecodeStatus DecodeLoadQFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder) {
   return DecodeMem(Inst, insn, Address, Decoder, true,
                    DecodeQFPRegsRegisterClass);
 }
 
 static DecodeStatus DecodeStoreInt(MCInst &Inst, unsigned insn,
                                    uint64_t Address, const void *Decoder) {
   return DecodeMem(Inst, insn, Address, Decoder, false,
                    DecodeIntRegsRegisterClass);
 }
 
+static DecodeStatus DecodeStoreIntPair(MCInst &Inst, unsigned insn,
+                                   uint64_t Address, const void *Decoder) {
+  return DecodeMem(Inst, insn, Address, Decoder, false,
+                   DecodeIntPairRegisterClass);
+}
+
 static DecodeStatus DecodeStoreFP(MCInst &Inst, unsigned insn, uint64_t Address,
                                   const void *Decoder) {
   return DecodeMem(Inst, insn, Address, Decoder, false,
                    DecodeFPRegsRegisterClass);
 }
 
 static DecodeStatus DecodeStoreDFP(MCInst &Inst, unsigned insn,
                                    uint64_t Address, const void *Decoder) {

lib/Target/Sparc/SparcISelLowering.h

public:
 
     bool ShouldShrinkFPConstant(EVT VT) const override {
       // Do not shrink FP constpool if VT == MVT::f128.
       // (ldd, call _Q_fdtoq) is more expensive than two ldds.
       return VT != MVT::f128;
     }
 
     void ReplaceNodeResults(SDNode *N,
-                                    SmallVectorImpl<SDValue>& Results,
-                                    SelectionDAG &DAG) const override;
 
     MachineBasicBlock *expandSelectCC(MachineInstr *MI, MachineBasicBlock *BB,
                                       unsigned BROpcode) const;
     MachineBasicBlock *expandAtomicRMW(MachineInstr *MI,
                                        MachineBasicBlock *BB,
                                        unsigned Opcode,
                                        unsigned CondCode = 0) const;
+
+    //    SDValue PerformDAGCombine(SDNode *N, DAGCombinerInfo &DCI) const override;
   };
 } // end namespace llvm
 
 #endif    // SPARC_ISELLOWERING_H

lib/Target/Sparc/SparcISelLowering.cpp

 SparcTargetLowering::SparcTargetLowering(TargetMachine &TM,
                                          const SparcSubtarget &STI)
     : TargetLowering(TM), Subtarget(&STI) {
   // Set up the register classes.
   addRegisterClass(MVT::i32, &SP::IntRegsRegClass);
   addRegisterClass(MVT::f32, &SP::FPRegsRegClass);
   addRegisterClass(MVT::f64, &SP::DFPRegsRegClass);
   addRegisterClass(MVT::f128, &SP::QFPRegsRegClass);
-  if (Subtarget->is64Bit())
+  if (Subtarget->is64Bit()) {
     addRegisterClass(MVT::i64, &SP::I64RegsRegClass);
+  } else {
+    // On 32bit sparc, we define a double-register 32bit register
+    // class, as well. This is modeled in LLVM as a 2-vector of i32.
+    addRegisterClass(MVT::v2i32, &SP::IntPairRegClass);
+
+    // ...but almost all operations must be expanded, so set that as
+    // the default.
+    for (unsigned Op = 0; Op < ISD::BUILTIN_OP_END; ++Op) {
+      setOperationAction(Op, MVT::v2i32, Expand);
+    }
+    // FIXME: I'm not 100% sure if these are needed
+    for (MVT VT : MVT::integer_valuetypes()) {
+      setLoadExtAction(ISD::SEXTLOAD, VT, MVT::v2i32, Expand);
+      setLoadExtAction(ISD::ZEXTLOAD, VT, MVT::v2i32, Expand);
+      setLoadExtAction(ISD::EXTLOAD, VT, MVT::v2i32, Expand);
+
+      setLoadExtAction(ISD::SEXTLOAD, MVT::v2i32, VT, Expand);
+      setLoadExtAction(ISD::ZEXTLOAD, MVT::v2i32, VT, Expand);
+      setLoadExtAction(ISD::EXTLOAD, MVT::v2i32, VT, Expand);
+
+      setTruncStoreAction(VT, MVT::v2i32, Expand);
+      setTruncStoreAction(MVT::v2i32, VT, Expand);
+    }
+    // However, load and store *are* legal.
+    setOperationAction(ISD::LOAD, MVT::v2i32, Legal);
+    setOperationAction(ISD::STORE, MVT::v2i32, Legal);
+    setOperationAction(ISD::EXTRACT_VECTOR_ELT, MVT::v2i32, Legal);
+    setOperationAction(ISD::BUILD_VECTOR, MVT::v2i32, Legal);
+
+    // And we need to promote i64 loads/stores into vector load/store
+    setOperationAction(ISD::LOAD, MVT::i64, Custom);
+    setOperationAction(ISD::STORE, MVT::i64, Custom);
+
+    // Sadly, this doesn't work:
+    //    AddPromotedToType(ISD::LOAD, MVT::i64, MVT::v2i32);
+    //    AddPromotedToType(ISD::STORE, MVT::i64, MVT::v2i32);
+  }
 
   // Turn FP extload into load/fextend
   for (MVT VT : MVT::fp_valuetypes()) {
     setLoadExtAction(ISD::EXTLOAD, VT, MVT::f32, Expand);
     setLoadExtAction(ISD::EXTLOAD, VT, MVT::f64, Expand);
   }
 
   // Sparc doesn't have i1 sign extending load
InFP128 = DAG.getMachineNode(TargetOpcode::INSERT_SUBREG, dl,
                                SubRegOdd);
   SDValue OutChains[2] = { SDValue(Hi64.getNode(), 1),
                            SDValue(Lo64.getNode(), 1) };
   SDValue OutChain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, OutChains);
   SDValue Ops[2] = {SDValue(InFP128,0), OutChain};
   return DAG.getMergeValues(Ops, dl);
 }
 
+static SDValue LowerLOAD(SDValue Op, SelectionDAG &DAG)
+{
+  LoadSDNode *LdNode = cast<LoadSDNode>(Op.getNode());
+
+  EVT MemVT = LdNode->getMemoryVT();
+  if (MemVT == MVT::f128)
+    return LowerF128Load(Op, DAG);
+
+  return Op;
+}
+
 // Lower a f128 store into two f64 stores.
 static SDValue LowerF128Store(SDValue Op, SelectionDAG &DAG) {
   SDLoc dl(Op);
   StoreSDNode *StNode = dyn_cast<StoreSDNode>(Op.getNode());
   assert(StNode && StNode->getOffset().getOpcode() == ISD::UNDEF
          && "Unexpected node type");
   SDValue SubRegEven = DAG.getTargetConstant(SP::sub_even64, MVT::i32);
   SDValue SubRegOdd  = DAG.getTargetConstant(SP::sub_odd64, MVT::i32);
OutChains[1] = DAG.getStore(StNode->getChain(),
                              dl,
                              SDValue(Lo64, 0),
                              LoPtr,
                              MachinePointerInfo(),
                              false, false, alignment);
   return DAG.getNode(ISD::TokenFactor, dl, MVT::Other, OutChains);
 }
 
+static SDValue LowerSTORE(SDValue Op, SelectionDAG &DAG)
+{
+  SDLoc dl(Op);
+  StoreSDNode *St = cast<StoreSDNode>(Op.getNode());
+
+  EVT MemVT = St->getMemoryVT();
+  if (MemVT == MVT::f128)
+    return LowerF128Store(Op, DAG);
+
+  if (MemVT == MVT::i64) {
+    // Custom handling for i64 stores: turn it into a bitcast and a
+    // v2i32 store.
+    SDValue Val = DAG.getNode(ISD::BITCAST, dl, MVT::v2i32, St->getValue());
+    SDValue Chain = DAG.getStore(
+        St->getChain(), dl, Val, St->getBasePtr(), St->getPointerInfo(),
+        St->isVolatile(), St->isNonTemporal(), St->getAlignment(),
+        St->getAAInfo());
+    return Chain;
+  }
+
+  return Op;
+}
+
 static SDValue LowerFNEGorFABS(SDValue Op, SelectionDAG &DAG, bool isV9) {
   assert((Op.getOpcode() == ISD::FNEG || Op.getOpcode() == ISD::FABS)
          && "invalid opcode");
 
   if (Op.getValueType() == MVT::f64)
     return LowerF64Op(Op, DAG, Op.getOpcode());
   if (Op.getValueType() != MVT::f128)
     return Op;
static SDValue LowerATOMIC_LOAD_STORE(SDValue Op, SelectionDAG &DAG) {
   // Monotonic load/stores are legal.
   if (cast<AtomicSDNode>(Op)->getOrdering() <= Monotonic)
     return Op;
 
   // Otherwise, expand with a fence.
   return SDValue();
 }
 
-
 SDValue SparcTargetLowering::
 LowerOperation(SDValue Op, SelectionDAG &DAG) const {
 
   bool hasHardQuad = Subtarget->hasHardQuad();
   bool isV9        = Subtarget->isV9();
 
   switch (Op.getOpcode()) {
   default: llvm_unreachable("Should not custom lower this!");
case ISD::BR_CC:              return LowerBR_CC(Op, DAG, *this,
                                                   hasHardQuad);
   case ISD::SELECT_CC:          return LowerSELECT_CC(Op, DAG, *this,
                                                       hasHardQuad);
   case ISD::VASTART:            return LowerVASTART(Op, DAG, *this);
   case ISD::VAARG:              return LowerVAARG(Op, DAG);
   case ISD::DYNAMIC_STACKALLOC: return LowerDYNAMIC_STACKALLOC(Op, DAG,
                                                                Subtarget);
 
-  case ISD::LOAD:               return LowerF128Load(Op, DAG);
-  case ISD::STORE:              return LowerF128Store(Op, DAG);
+  case ISD::LOAD:               return LowerLOAD(Op, DAG);
+  case ISD::STORE:              return LowerSTORE(Op, DAG);
   case ISD::FADD:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::ADD_F128), 2);
   case ISD::FSUB:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::SUB_F128), 2);
   case ISD::FMUL:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::MUL_F128), 2);
   case ISD::FDIV:               return LowerF128Op(Op, DAG,
                                        getLibcallName(RTLIB::DIV_F128), 2);
libCall = ((N->getOpcode() == ISD::SINT_TO_FP)
                ? RTLIB::SINTTOFP_I64_F128
                : RTLIB::UINTTOFP_I64_F128);
 
     Results.push_back(LowerF128Op(SDValue(N, 0),
                                   DAG,
                                   getLibcallName(libCall),
                                   1));
     return;
+  case ISD::LOAD: {
+    LoadSDNode *Ld = cast<LoadSDNode>(N);
+    // Custom handling only for i64: turn i64 load into a v2i32 load,
+    // and a bitcast.
+    if (N->getValueType(0) != MVT::i64)
+      return;
+
+    SDLoc dl(N);
+    SDValue LoadRes = DAG.getExtLoad(
+        Ld->getExtensionType(), dl, MVT::v2i32,
+        Ld->getChain(), Ld->getBasePtr(), Ld->getPointerInfo(),
+        MVT::v2i32, Ld->isVolatile(), Ld->isNonTemporal(),
+        Ld->isInvariant(), Ld->getAlignment(), Ld->getAAInfo());
+
+    //    SDValue Lo =
+    //    Val = DAG.getNode(ISD::BUILD_PAIR, DL, RoundVT, Lo, Hi);
+    SDValue Res = DAG.getNode(ISD::BITCAST, dl, MVT::i64, LoadRes);
+    Results.push_back(Res);
+    Results.push_back(LoadRes.getValue(1));
+    return;
+  }
   }
 }

lib/Target/Sparc/SparcInstrInfo.td

 let DecoderMethod = "DecodeLoadInt" in {
   defm LDSB : LoadA<"ldsb", 0b001001, 0b011001, sextloadi8,  IntRegs, i32>;
   defm LDSH : LoadA<"ldsh", 0b001010, 0b011010, sextloadi16, IntRegs, i32>;
   defm LDUB : LoadA<"ldub", 0b000001, 0b010001, zextloadi8,  IntRegs, i32>;
   defm LDUH : LoadA<"lduh", 0b000010, 0b010010, zextloadi16, IntRegs, i32>;
   defm LD   : LoadA<"ld",   0b000000, 0b010000, load,        IntRegs, i32>;
 }
 
+let DecoderMethod = "DecodeLoadIntPair" in
+  defm LDD : LoadA<"ldd", 0b000011, 0b010011, load, IntPair, v2i32>;
+
 // Section B.2 - Load Floating-point Instructions, p. 92
 let DecoderMethod = "DecodeLoadFP" in
   defm LDF   : Load<"ld",  0b100000, load, FPRegs,  f32>;
 let DecoderMethod = "DecodeLoadDFP" in
   defm LDDF  : Load<"ldd", 0b100011, load, DFPRegs, f64>;
 let DecoderMethod = "DecodeLoadQFP" in
   defm LDQF  : Load<"ldq", 0b100010, load, QFPRegs, f128>,
                Requires<[HasV9, HasHardQuad]>;
 
 // Section B.4 - Store Integer Instructions, p. 95
 let DecoderMethod = "DecodeStoreInt" in {
   defm STB   : StoreA<"stb", 0b000101, 0b010101, truncstorei8,  IntRegs, i32>;
   defm STH   : StoreA<"sth", 0b000110, 0b010110, truncstorei16, IntRegs, i32>;
   defm ST    : StoreA<"st",  0b000100, 0b010100, store,         IntRegs, i32>;
 }
 
+let DecoderMethod = "DecodeStoreIntPair" in
+  defm STD   : StoreA<"std", 0b000111, 0b010111, store, IntPair, v2i32>;
+
 // Section B.5 - Store Floating-point Instructions, p. 97
 let DecoderMethod = "DecodeStoreFP" in
   defm STF   : Store<"st",  0b100100, store,         FPRegs,  f32>;
 let DecoderMethod = "DecodeStoreDFP" in
   defm STDF  : Store<"std", 0b100111, store,         DFPRegs, f64>;
 let DecoderMethod = "DecodeStoreQFP" in
   defm STQF  : Store<"stq", 0b100110, store,         QFPRegs, f128>,
                Requires<[HasV9, HasHardQuad]>;

 // atomic_load_32 addr -> load addr
 def : Pat<(i32 (atomic_load ADDRrr:$src)), (LDrr ADDRrr:$src)>;
 def : Pat<(i32 (atomic_load ADDRri:$src)), (LDri ADDRri:$src)>;
 
 // atomic_store_32 val, addr -> store val, addr
 def : Pat<(atomic_store ADDRrr:$dst, i32:$val), (STrr ADDRrr:$dst, $val)>;
 def : Pat<(atomic_store ADDRri:$dst, i32:$val), (STri ADDRri:$dst, $val)>;
 
+// extract_vector
+def : Pat<(vector_extract (v2i32 IntPair:$Rn), 0),
+          (i32 (EXTRACT_SUBREG IntPair:$Rn, sub_even))>;
+def : Pat<(vector_extract (v2i32 IntPair:$Rn), 1),
+          (i32 (EXTRACT_SUBREG IntPair:$Rn, sub_odd))>;
+
+// build_vector
+def : Pat<(build_vector (i32 IntRegs:$a1), (i32 IntRegs:$a2)),
+          (INSERT_SUBREG
+      (INSERT_SUBREG (v2i32 (IMPLICIT_DEF)), (i32 IntRegs:$a1), sub_even),
+            (i32 IntRegs:$a2), sub_odd)>;
+
 
 include "SparcInstr64Bit.td"
 include "SparcInstrVIS.td"
 include "SparcInstrAliases.td"

lib/Target/Sparc/SparcRegisterInfo.td

 def sub_even64 : SubRegIndex<64>;
 def sub_odd64  : SubRegIndex<64, 64>;
 }
 
 // Registers are identified with 5-bit ID numbers.
 // Ri - 32-bit integer registers
 class Ri<bits<16> Enc, string n> : SparcReg<Enc, n>;
 
+// Rdi - pairs of 32-bit integer registers
+class Rdi<bits<16> Enc, string n, list<Register> subregs> : SparcReg<Enc, n> {
+  let SubRegs = subregs;
+  let SubRegIndices = [sub_even, sub_odd];
+  let CoveredBySubRegs = 1;
+}
 // Rf - 32-bit floating-point registers
 class Rf<bits<16> Enc, string n> : SparcReg<Enc, n>;
 
 // Rd - Slots in the FP register file for 64-bit floating-point values.
 class Rd<bits<16> Enc, string n, list<Register> subregs> : SparcReg<Enc, n> {
   let SubRegs = subregs;
   let SubRegIndices = [sub_even, sub_odd];
   let CoveredBySubRegs = 1;

 def Q9  : Rq< 5, "F36", [D18, D19]>;
 def Q10 : Rq< 9, "F40", [D20, D21]>;
 def Q11 : Rq<13, "F44", [D22, D23]>;
 def Q12 : Rq<17, "F48", [D24, D25]>;
 def Q13 : Rq<21, "F52", [D26, D27]>;
 def Q14 : Rq<25, "F56", [D28, D29]>;
 def Q15 : Rq<29, "F60", [D30, D31]>;
 
+// Aliases of the integer registers used for LDD/STD double-word operations
+def G0_G1 : Rdi<0, "G0", [G0, G1]>;
+def G2_G3 : Rdi<2, "G2", [G2, G3]>;
+def G4_G5 : Rdi<4, "G4", [G4, G5]>;
+def G6_G7 : Rdi<6, "G6", [G6, G7]>;
+def O0_O1 : Rdi<8, "O0", [O0, O1]>;
+def O2_O3 : Rdi<10, "O2", [O2, O3]>;
+def O4_O5 : Rdi<12, "O4", [O4, O5]>;
+def O6_O7 : Rdi<14, "O6", [O6, O7]>;
+def L0_L1 : Rdi<16, "L0", [L0, L1]>;
+def L2_L3 : Rdi<18, "L2", [L2, L3]>;
+def L4_L5 : Rdi<20, "L4", [L4, L5]>;
+def L6_L7 : Rdi<22, "L6", [L6, L7]>;
+def I0_I1 : Rdi<24, "I0", [I0, I1]>;
+def I2_I3 : Rdi<26, "I2", [I2, I3]>;
+def I4_I5 : Rdi<28, "I4", [I4, I5]>;
+def I6_I7 : Rdi<30, "I6", [I6, I7]>;
+
 // Register classes.
 //
 // FIXME: the register order should be defined in terms of the preferred
 // allocation order...
 //
 // This register class should not be used to hold i64 values, use the I64Regs
 // register class for that. The i64 type is included here to allow i64 patterns
 // using the integer instructions.
 def IntRegs : RegisterClass<"SP", [i32, i64], 32,
                             (add (sequence "I%u", 0, 7),
                                  (sequence "G%u", 0, 7),
                                  (sequence "L%u", 0, 7),
                                  (sequence "O%u", 0, 7))>;
 
+// Should be in the same order as IntRegs.
+def IntPair : RegisterClass<"SP", [v2i32], 64,
+    (add I0_I1, I2_I3, I4_I5, I6_I7,
+         G0_G1, G2_G3, G4_G5, G6_G7,
+         L0_L1, L2_L3, L4_L5, L6_L7,
+         O0_O1, O2_O3, O4_O5, O6_O7)>;
+
 // Register class for 64-bit mode, with a 64-bit spill slot size.
 // These are the same as the 32-bit registers, so TableGen will consider this
 // to be a sub-class of IntRegs. That works out because requiring a 64-bit
 // spill slot is a stricter constraint than only requiring a 32-bit spill slot.
 def I64Regs : RegisterClass<"SP", [i64], 64, (add IntRegs)>;
 
 // Floating point register classes.
 def FPRegs : RegisterClass<"SP", [f32], 32, (sequence "F%u", 0, 31)>;

test/MC/Disassembler/Sparc/sparc-mem.txt

 # CHECK:     swap [%g1], %o2
 0xd4 0x78 0x40 0x00
 
 # CHECK:     swapa [%i0+%l6] 131, %o2
 0xd4 0xfe 0x10 0x76
 
 # CHECK:     swapa [%g1] 131, %o2
 0xd4 0xf8 0x50 0x60
+
+# CHECK:     ldd [%i0+%l6], %o2
+0xd4 0x1e 0x00 0x16
+
+# CHECK:     ldd [%i0+32], %o2
+0xd4 0x1e 0x20 0x20
+
+# CHECK:     ldd [%g1], %o2
+0xd4 0x18 0x60 0x00
+
+# CHECK:     ldd [%g1], %o2
+0xd4 0x18 0x40 0x00
+
+# CHECK:     std %o2, [%i0+%l6]
+0xd4 0x3e 0x00 0x16
+
+# CHECK:     std %o2, [%i0+32]
+0xd4 0x3e 0x20 0x20
+
+# CHECK:     std %o2, [%g1]
+0xd4 0x38 0x60 0x00
+
+# CHECK:     std %o2, [%g1]
+0xd4 0x38 0x40 0x00

test/MC/Sparc/sparc-mem-instructions.s

! RUN: llvm-mc %s -arch=sparcv9 -show-encoding | FileCheck %s
         ld [%i0 + %l6], %o2
         ! CHECK: ld [%i0+32], %o2     ! encoding: [0xd4,0x06,0x20,0x20]
         ld [%i0 + 32], %o2
         ! CHECK: ld [%g1], %o2        ! encoding: [0xd4,0x00,0x40,0x00]
         ld [%g1], %o2
         ! CHECK: lda [%i0+%l6] 131, %o2 ! encoding: [0xd4,0x86,0x10,0x76]
         lda [%i0 + %l6] 131, %o2
 
+        ! CHECK: ldd [%i0+%l6], %o2    ! encoding: [0xd4,0x1e,0x00,0x16]
+        ldd [%i0 + %l6], %o2
+        ! CHECK: ldd [%i0+32], %o2     ! encoding: [0xd4,0x1e,0x20,0x20]
+        ldd [%i0 + 32], %o2
+        ! CHECK: ldd [%g1], %o2        ! encoding: [0xd4,0x18,0x40,0x00]
+        ldd [%g1], %o2
+        ! CHECK: ldda [%i0+%l6] 131, %o2 ! encoding: [0xd4,0x9e,0x10,0x76]
+        ldda [%i0 + %l6] 131, %o2
+
         ! CHECK: stb %o2, [%i0+%l6]   ! encoding: [0xd4,0x2e,0x00,0x16]
         stb %o2, [%i0 + %l6]
         ! CHECK: stb %o2, [%i0+32]    ! encoding: [0xd4,0x2e,0x20,0x20]
         stb %o2, [%i0 + 32]
         ! CHECK: stb %o2, [%g1]       ! encoding: [0xd4,0x28,0x40,0x00]
         stb %o2, [%g1]
         ! CHECK: stba %o2, [%i0+%l6] 131 ! encoding: [0xd4,0xae,0x10,0x76]
         stba %o2, [%i0 + %l6] 131
! RUN: llvm-mc %s -arch=sparcv9 -show-encoding | FileCheck %s
         ! CHECK: st %o2, [%i0+%l6]    ! encoding: [0xd4,0x26,0x00,0x16]
         st %o2, [%i0 + %l6]
         ! CHECK: st %o2, [%i0+32]     ! encoding: [0xd4,0x26,0x20,0x20]
         st %o2, [%i0 + 32]
         ! CHECK: st %o2, [%g1]        ! encoding: [0xd4,0x20,0x40,0x00]
         st %o2, [%g1]
         ! CHECK: sta %o2, [%i0+%l6] 131 ! encoding: [0xd4,0xa6,0x10,0x76]
         sta %o2, [%i0 + %l6] 131
+
+        ! CHECK: std %o2, [%i0+%l6]    ! encoding: [0xd4,0x3e,0x00,0x16]
+        std %o2, [%i0 + %l6]
+        ! CHECK: std %o2, [%i0+32]     ! encoding: [0xd4,0x3e,0x20,0x20]
+        std %o2, [%i0 + 32]
+        ! CHECK: std %o2, [%g1]        ! encoding: [0xd4,0x38,0x40,0x00]
+        std %o2, [%g1]
+        ! CHECK: stda %o2, [%i0+%l6] 131 ! encoding: [0xd4,0xbe,0x10,0x76]
+        stda %o2, [%i0 + %l6] 131