[RISCV] Add vendor-defined XTheadMemPair (two-GPR Memory Operations) extension

[mtsamis]

Summary

The vendor-defined XTHeadMemPair (no comparable standard extension exists
at the time of writing) extension adds two-GPR load/store pair instructions.

It is supported by the C9xx cores (e.g., found in the wild in the
Allwinner D1) by Alibaba T-Head.

The current (as of this commit) public documentation for this
extension is available at:

https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf

Support for these instructions has already landed in GNU Binutils:

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=6e17ae625570ff8f3c12c8765b8d45d4db8694bd

Depends on D143847

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2665

Does this restrict apply for TH_SDD/TH_SWD? I don't see it in the spec.

llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp

454

Drop curly braces

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9744

Since the offset allows for a gap in the memory locations the size isn't 8/16. It needs to be UnknownSize or large enough to cover the gap.

9803

Drop curly braces

9842

adjacent*

9846

adjacent*

9847

How do we know MemVT is i64 here?

[philipp.tomsich]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2665

Given that QEMU had this constraint, but it was missing in the spec, I checked on an Allwinner D1.
Turns out that LDD works with with $rs1 == $rs2 and with $rs1 == $rs2 == $rd.

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2665

Did you mean SDD instead of LDD?

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Why does this 3rd operand exist at all?

2672

Oops, I meant fifth operand.

[philipp.tomsich]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2665

Yes, the store instructions (SDD and SWD) work.

[mtsamis]

• Do not restrict source and destination registers for TH_SDD/TH_SWD
• Add missing check for MemVT == i64
• Fix typos and remove unwanted braces

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Although I also don't see why this required, I implemented it as per the specification

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9744

The offset does not allow for a gap, all loads stores are from adjacent memory locations.
The offset only affects the base pointer from which the load/store happens.

E.g. per the specification:

addr := rs1 + (zero_extend(imm2) << 4)
tmp1 := mem[addr+7:addr]
tmp2 := mem[addr+15:addr+8]

Would that make the 8/16 correct then?

9847

You're correct that this is not known; I've added a check. Thanks

[philipp.tomsich]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

The problem really is the Specification on this one:

th.ldd rd1,rd2,(rs1),#imm2,4

appears to have 4 as an operand (even though it is the only permissible value and does not get encoded explicitly).

However, looking at it from another angle the ",4" is part of the operation mnemonic (or an %_ term when expressing the pattern in GNU notation):

th.ldd %1,%2,(%0),%3,4

Given that the 4 is not encoded, I am leaning with Craig at the moment and consider it as part of the mnemonic (i.e., a split off fragment of 'th.ldd'+',4').

I would much rather have seen this specified as

th.ldd rd1,rd2,#offset(rs1)

where offset would have a constraint to have its lower 4 bits cleared.

This would be a much more meaningful assembler syntax.
Can we have this as a pseudo (that also becomes the preferred disassembly)?
I'll volunteer to send out a PR against the T-Head specification to add that additional assembler-pseudo under the th.ldd definition…

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

From what I understand, even if we have an additional assembler-pseudo for a nicer syntax we would still have to support the current one from the specification. In that case, adding the 'nice' syntax won't make the implementation better because we'll have to support the other one as well. The "4" is indeed not encoded and could be part of the assembler string as + ", 4" but from what I understood when implementing it, this doesn't work great with the current infrastructure. You can also check this discussion about implicit operands from which I got ideas how to properly implent them: mc-layer-parsing-disassembling-implicit-operands. The constant assembler string makes the assembler not able to properly parse the instruction and also makes it hard to get proper error messages as in this implementation.

Tl;dr My personal conclusion is that if we have to support the current syntax then we don't benefit from creating an alternative nicer syntax.

[jrtc27]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

lw rd, rs1, imm is the syntax in the unprivileged spec, but lw rd, imm(rs1) is the canonical assembly format. Ditto for jalr. You could just view it the same way.

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

But the mempair instruction are of the form th.ldd rd1, rd2, (rs1), imm2, 4 where there is both imm but also a trailing 3 or 4 which has no function (as discussed above) and is not included in the instruction encoding. But it still has to be there as per the specification and the assembler/disassembler must work with it.

So I don't think this is the same as lw?

[philipp.tomsich]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

I agree with Jessica in principle, but we have a released GNU Binutils in the wild that implements the "weird" assembler syntax. So we should not have a hard break for backwards compatibility (i.e., will need coexistence).

[jrtc27]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

I wasn't suggesting ditching the weird syntax; lw rd, rs1, imm is legal assembly and works in both assemblers today. Just saying we can (a) have multiple syntaxes supported (b) change the canonical one.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9744

Yes. You're right. Sorry about that.

[mtsamis]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9744

No problem, marking as done then

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Jessica, I think I now better understand what you've meant and it sounds like a great way to do this.

Then, would it be possible for the alias to be "th.ldd $rd1, $rd2, ($rs1), $offset, 4" where the "4" part is discarded?
I'm not familiar enough with the infrastructure to know if this will work fine with the assembler/disassembler, but I can try it out.

Please correct me if I'm wrong or understood your proposal incorrectly.

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

I don't think InstAlias or the Instruction AsmString support a specific constant hardcoded into the string. So it would still need to be parsed as an immediate.

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Ok, I see. Then I can add the alias for the alternative syntax if we'd like to, without that affecting the current implementation.

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Is the 3 or 4 indicating the amount the immediate is shifted by? Instead of only accepting immediates that are divisible 8 or 16. The latter is what C.LW does for example.

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Is the 3 or 4 indicating the amount the immediate is shifted by?

Yes, that looks to be the case.

Instead of only accepting immediates that are divisible 8 or 16. The latter is what C.LW does for example.

I looked up the format of C.LW and more conveniently the (shifted) constant is a single argument of the instruction with the restriction you mentioned. In MemPair the two significant bits of the constant and the shift amount are separate arguments but the shift is reduntant anyway.

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

I don't know if this instruction was defined before C.LW or just ignored the precedent.

But I guess it doesn't matter now. We have to implement the spec.

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Yes, I agree with that; the format for these instructions is a bit unfortunate.

So, of all things that were discussed here, is there some desired change I should do on the current implementation?

[craig.topper]

llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp

455

Inst.addOperand(MCOperand::createImm(IsWordOp ? 3 : 4));

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9838

contraints -> constraints

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

14

What is "C" here?

llvm/test/CodeGen/RISCV/xtheadmempair.ll

2

Can we test i64 load/store on rv32 and i128 load on rv64? I assume we use these instructions to handle the split.

[craig.topper]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

I don't think we can change the canonical syntax without binutils also supporting or we will break -fno-integrated-as.

I'm ok with just accepting the syntax binutils supports.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9731

Just to make sure I understand. The first call processes the worklist. The second call will have an empty worklist so the only thing the second call does is check whether LSNode2 appears in the visited set from the first call?

[mtsamis]

Fix typos and add additional tests.

[mtsamis]

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

2672

Ok, marking this as done then.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9731

This code is taken from elsewhere in LLVM and as per the comments (e.g. in DAGCombiner.cpp::getPostIndexedLoadStoreOp) it checks if the two nodes are independent of each other.
Otherwise the merged load/store may be incorrect. Also I believe that the Worklist is not always empty in the second call; hasPredecessorHelper addes back some nodes to the worklist after it has finished.

Since this pattern is followed in other places, is this code fine to use in the same way here?

llvm/test/CodeGen/RISCV/xtheadmempair.ll

2

Ok, I have added four new test functions: ld64/ld128/sd64/sd128.
All tested for both rv32/rv64, I believe that should be fine.
The expected instructions are emmited for the loads/stores.

[philipp.tomsich]

Accidentially pushed and reverted.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9731

I think the DeferredNodes that are pushed back to the worklist only exist if TopologicalPrune is true which it isn't here.

The code is fine to use here. I just wanted to make sure I understood how it worked.

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

22

Can we name these with lower case letters. I think that's consistent with the usual style. That should also remove the need for the extra T prefix?

123

Can you indent these to line up with !shl on the previous line. Probably need a line break for 80 columns. But it will provide a better visual separation from the let statements in the body.

135

Same here

[mtsamis]

Correctly push the latest changes.

[mtsamis]

Change names of MemPair nodes and code alignment

[mtsamis]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9731

Indeed, that's true. After looking again at the code though there is a

if (Op == N)
  Found = true;
...
if (Found)
  break;

which means that the Worklist may still be empty at the end.
Given that the worklist is initially {LSNode1, LSNode2} and N is LSNode1 at the first call and LSNode2 at the second, I think the worklist potentially contains LSNode1 and/or LSNode2 at the second call too

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

22

Done, but I've kept the th prefix because otherwise there could be a potential name collision with future changes?
I also renamed SDT_TDBLD and SDT_TDBST to SDT_LoadPair and SDT_StorePair, which I think are more readable.

123

Done

[craig.topper]

LGTM with the noted tab characters fixed. Please double check there aren't any others.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9731

But if Found is true in the first call, I think the second call to hasPredecessorHelper won't happen because the first call will return true so we won't continue past the ||. So the worklist is empty in the case that the second call to hasPredecessorHelper happens.

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

22

I was referring to the second 'T' in TH_TLWD. Not the 'T' that was part of 'TH'

124

I think there's a tab character here instead of spaces

137

I think there's a tab character here instead of spaces

llvm/test/CodeGen/RISCV/xtheadmempair.ll

2

Thanks!

[mtsamis]

Replace tabs with spaces and fix alignment

[mtsamis]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

9731

That's true.
Then given that worklist is empty, the second call always returns false except for when LSnode2 has been visited in the first call?

if (Visited.count(N))
      return true;

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

22

Ok makes sense, I misread that.

llvm/docs/RISCVUsage.rst

``XTHeadBb``
   LLVM implements `the THeadBb (basic bit-manipulation) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
 
 ``XTHeadBs``
   LLVM implements `the THeadBs (single-bit operations) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
 
 ``XTheadMac``
   LLVM implements `the XTheadMac (multiply-accumulate instructions) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
 
+``XTHeadMemPair``
+  LLVM implements `the THeadMemPair (two-GPR memory operations) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
+
 ``XTHeadVdot``
   LLVM implements `version 1.0.0 of the THeadV-family custom instructions specification <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.0/xthead-2022-12-04-2.2.0.pdf>`_ by T-HEAD of Alibaba.  All instructions are prefixed with `th.` as described in the specification, and the riscv-toolchain-convention document linked above.
 
 ``XVentanaCondOps``
   LLVM implements `version 1.0.0 of the VTx-family custom instructions specification <https://github.com/ventanamicro/ventana-custom-extensions/releases/download/v1.0.0/ventana-custom-extensions-v1.0.0.pdf>`_ by Ventana Micro Systems.  All instructions are prefixed with `vt.` as described in the specification, and the riscv-toolchain-convention document linked above.  These instructions are only available for riscv64 at this time.
 
 Specification Documents
 =======================
 For ratified specifications, please refer to the `official RISC-V International
 page <https://riscv.org/technical/specifications/>`_.  Make sure to check the
 `wiki for not yet integrated extensions
 <https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions>`_.
 

llvm/docs/ReleaseNotes.rst

 * Assembler support for version 1.0.1 of the Zcb extension was added.
 * Zca, Zcf, and Zcd extensions were upgraded to version 1.0.1.
 * vsetvli intrinsics no longer have side effects. They may now be combined,
   moved, deleted, etc. by optimizations.
 * Adds support for the vendor-defined XTHeadBa (address-generation) extension.
 * Adds support for the vendor-defined XTHeadBb (basic bit-manipulation) extension.
 * Adds support for the vendor-defined XTHeadBs (single-bit) extension.
 * Adds support for the vendor-defined XTHeadMac (multiply-accumulate instructions) extension.
+* Added support for the vendor-defined XTHeadMemPair (two-GPR memory operations)
+  extension disassembler/assembler.
 
 Changes to the WebAssembly Backend
 ----------------------------------
 
 * ...
 
 Changes to the Windows Target
 -----------------------------

llvm/lib/Support/RISCVISAInfo.cpp

static const RISCVSupportedExtension SupportedExtensions[] = {
     {"svpbmt", RISCVExtensionVersion{1, 0}},
     {"svinval", RISCVExtensionVersion{1, 0}},
 
     // vendor-defined ('X') extensions
     {"xtheadba", RISCVExtensionVersion{1, 0}},
     {"xtheadbb", RISCVExtensionVersion{1, 0}},
     {"xtheadbs", RISCVExtensionVersion{1, 0}},
     {"xtheadmac", RISCVExtensionVersion{1, 0}},
+    {"xtheadmempair", RISCVExtensionVersion{1, 0}},
     {"xtheadvdot", RISCVExtensionVersion{1, 0}},
     {"xventanacondops", RISCVExtensionVersion{1, 0}},
 };
 
 static const RISCVSupportedExtension SupportedExperimentalExtensions[] = {
     {"zihintntl", RISCVExtensionVersion{0, 2}},
 
     {"zca", RISCVExtensionVersion{1, 0}},

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

if (Inst.getOpcode() == RISCV::PseudoVMSGEU_VX_M_T ||
     unsigned TempReg = Inst.getOperand(1).getReg();
     if (DestReg == TempReg) {
       SMLoc Loc = Operands.back()->getStartLoc();
       return Error(Loc, "The temporary vector register cannot be the same as "
                         "the destination register.");
     }
   }
 
+  unsigned Opcode = Inst.getOpcode();
+
+  if (Opcode == RISCV::TH_LDD || Opcode == RISCV::TH_LWUD ||
+      Opcode == RISCV::TH_LWD) {
+    unsigned Rd1 = Inst.getOperand(0).getReg();
+    unsigned Rd2 = Inst.getOperand(1).getReg();
+    unsigned Rs1 = Inst.getOperand(2).getReg();
+    // The encoding with rd1 == rd2 == rs1 is reserved for XTHead load pair.
+    if (Rs1 == Rd1 && Rs1 == Rd2) {
+      SMLoc Loc = Operands[1]->getStartLoc();
+      return Error(Loc, "The source register and destination registers "
+                        "cannot be equal.");
+    }
+  }
+
+  bool IsTHeadMemPair32 = (Opcode == RISCV::TH_LWD ||
+                           Opcode == RISCV::TH_LWUD || Opcode == RISCV::TH_SWD);
+  bool IsTHeadMemPair64 = (Opcode == RISCV::TH_LDD || Opcode == RISCV::TH_SDD);
+  // The last operand of XTHeadMemPair instructions must be constant 3 or 4
+  // depending on the data width.
+  if (IsTHeadMemPair32 && Inst.getOperand(4).getImm() != 3) {
+    SMLoc Loc = Operands.back()->getStartLoc();
+    return Error(Loc, "Operand must be constant 3.");
+  } else if (IsTHeadMemPair64 && Inst.getOperand(4).getImm() != 4) {
+    SMLoc Loc = Operands.back()->getStartLoc();
+    return Error(Loc, "Operand must be constant 4.");
+  }
+
   const MCInstrDesc &MCID = MII.get(Inst.getOpcode());
   RISCVII::VConstraintType Constraints = RISCVII::getConstraint(MCID.TSFlags);
   if (Constraints == RISCVII::NoConstraint)
     return false;
 
   unsigned DestReg = Inst.getOperand(0).getReg();
   // Operands[1] will be the first operand, DestReg.
   SMLoc Loc = Operands[1]->getStartLoc();
   if (Constraints & RISCVII::VS2Constraint) {
     unsigned CheckReg = Inst.getOperand(1).getReg();
     if (DestReg == CheckReg)
       return Error(Loc, "The destination vector register group cannot overlap"
                         " the source vector register group.");
   }
   if ((Constraints & RISCVII::VS1Constraint) && (Inst.getOperand(2).isReg())) {
     unsigned CheckReg = Inst.getOperand(2).getReg();
     if (DestReg == CheckReg)
       return Error(Loc, "The destination vector register group cannot overlap"
                         " the source vector register group.");
   }
   if ((Constraints & RISCVII::VMConstraint) && (DestReg == RISCV::V0)) {
     // vadc, vsbc are special cases. These instructions have no mask register.
     // The destination register could not be V0.
-    unsigned Opcode = Inst.getOpcode();
     if (Opcode == RISCV::VADC_VVM || Opcode == RISCV::VADC_VXM ||
         Opcode == RISCV::VADC_VIM || Opcode == RISCV::VSBC_VVM ||
         Opcode == RISCV::VSBC_VXM || Opcode == RISCV::VFMERGE_VFM ||
         Opcode == RISCV::VMERGE_VIM || Opcode == RISCV::VMERGE_VVM ||
         Opcode == RISCV::VMERGE_VXM)
       return Error(Loc, "The destination vector register group cannot be V0.");
 
     // Regardless masked or unmasked version, the number of operands is the

llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp

 static DecodeStatus decodeRVCInstrRdRs2(MCInst &Inst, uint32_t Insn,
                                         uint64_t Address,
                                         const MCDisassembler *Decoder);
 
 static DecodeStatus decodeRVCInstrRdRs1Rs2(MCInst &Inst, uint32_t Insn,
                                            uint64_t Address,
                                            const MCDisassembler *Decoder);
 
+static DecodeStatus decodeXTHeadMemPair(MCInst &Inst, uint32_t Insn,
+                                        uint64_t Address,
+                                        const MCDisassembler *Decoder);
+
 #include "RISCVGenDisassemblerTables.inc"
 
 static DecodeStatus decodeRVCInstrRdRs1ImmZero(MCInst &Inst, uint32_t Insn,
                                                uint64_t Address,
                                                const MCDisassembler *Decoder) {
   uint32_t Rd = fieldFromInstruction(Insn, 7, 5);
   DecodeStatus Result = DecodeGPRNoX0RegisterClass(Inst, Rd, Address, Decoder);
   (void)Result;
static DecodeStatus decodeRVCInstrRdRs1Rs2(MCInst &Inst, uint32_t Insn,
   uint32_t Rd = fieldFromInstruction(Insn, 7, 5);
   uint32_t Rs2 = fieldFromInstruction(Insn, 2, 5);
   DecodeGPRRegisterClass(Inst, Rd, Address, Decoder);
   Inst.addOperand(Inst.getOperand(0));
   DecodeGPRRegisterClass(Inst, Rs2, Address, Decoder);
   return MCDisassembler::Success;
 }
 
+static DecodeStatus decodeXTHeadMemPair(MCInst &Inst, uint32_t Insn,
+                                        uint64_t Address,
+                                        const MCDisassembler *Decoder) {
+  uint32_t Rd1 = fieldFromInstruction(Insn, 7, 5);
+  uint32_t Rs1 = fieldFromInstruction(Insn, 15, 5);
+  uint32_t Rd2 = fieldFromInstruction(Insn, 20, 5);
+  uint32_t UImm2 = fieldFromInstruction(Insn, 25, 2);
+  DecodeGPRRegisterClass(Inst, Rd1, Address, Decoder);
+  DecodeGPRRegisterClass(Inst, Rd2, Address, Decoder);
+  DecodeGPRRegisterClass(Inst, Rs1, Address, Decoder);
+  DecodeStatus Result = decodeUImmOperand<2>(Inst, UImm2, Address, Decoder);
+  (void)Result;
+  assert(Result == MCDisassembler::Success && "Invalid immediate");
+
+  // Disassemble the final operand which is implicit.
+  unsigned Opcode = Inst.getOpcode();
+  bool IsWordOp = (Opcode == RISCV::TH_LWD || Opcode == RISCV::TH_LWUD ||
+                   Opcode == RISCV::TH_SWD);
+  if (IsWordOp)
+    Inst.addOperand(MCOperand::createImm(3));
+  else
+    Inst.addOperand(MCOperand::createImm(4));
+
+  return MCDisassembler::Success;
+}
+
 DecodeStatus RISCVDisassembler::getInstruction(MCInst &MI, uint64_t &Size,
                                                ArrayRef<uint8_t> Bytes,
                                                uint64_t Address,
                                                raw_ostream &CS) const {
   // TODO: This will need modification when supporting instruction set
   // extensions with instructions > 32-bits (up to 176 bits wide).
   uint32_t Insn;
   DecodeStatus Result;
if ((Bytes[0] & 0x3) == 0x3) {
     }
     if (STI.getFeatureBits()[RISCV::FeatureVendorXTHeadMac]) {
       LLVM_DEBUG(dbgs() << "Trying XTHeadMac custom opcode table:\n");
       Result = decodeInstruction(DecoderTableTHeadMac32, MI, Insn, Address,
                                  this, STI);
       if (Result != MCDisassembler::Fail)
         return Result;
     }
+    if (STI.getFeatureBits()[RISCV::FeatureVendorXTHeadMemPair]) {
+      LLVM_DEBUG(dbgs() << "Trying XTHeadMemPair custom opcode table:\n");
+      Result = decodeInstruction(DecoderTableTHeadMemPair32, MI, Insn, Address,
+                                 this, STI);
+      if (Result != MCDisassembler::Fail)
+        return Result;
+    }
     if (STI.getFeatureBits()[RISCV::FeatureVendorXTHeadVdot]) {
       LLVM_DEBUG(dbgs() << "Trying XTHeadVdot custom opcode table:\n");
       Result =
           decodeInstruction(DecoderTableTHeadV32, MI, Insn, Address, this, STI);
       if (Result != MCDisassembler::Fail)
         return Result;
     }
 

llvm/lib/Target/RISCV/RISCVFeatures.td

 
 def FeatureVendorXTHeadMac
     : SubtargetFeature<"xtheadmac", "HasVendorXTHeadMac", "true",
                        "'xtheadmac' (T-Head Multiply-Accumulate Instructions)">;
 def HasVendorXTHeadMac : Predicate<"Subtarget->hasVendorXTHeadMac()">,
                                   AssemblerPredicate<(all_of FeatureVendorXTHeadMac),
                                   "'xtheadmac' (T-Head Multiply-Accumulate Instructions)">;
 
+def FeatureVendorXTHeadMemPair
+    : SubtargetFeature<"xtheadmempair", "HasVendorXTHeadMemPair", "true",
+                       "'xtheadmempair' (T-Head two-GPR Memory Operations)">;
+def HasVendorXTHeadMemPair : Predicate<"Subtarget->hasVendorXTHeadMemPair()">,
+                                    AssemblerPredicate<(all_of FeatureVendorXTHeadMemPair),
+                                    "'xtheadmempair' (T-Head two-GPR Memory Operations)">;
+
 def FeatureVendorXTHeadVdot
     : SubtargetFeature<"xtheadvdot", "HasVendorXTHeadVdot", "true",
                        "'xtheadvdot' (T-Head Vector Extensions for Dot)",
                        [FeatureStdExtV]>;
 def HasVendorXTHeadVdot : Predicate<"Subtarget->hasVendorXTHeadVdot()">,
                                     AssemblerPredicate<(all_of FeatureVendorXTHeadVdot),
                                     "'xtheadvdot' (T-Head Vector Extensions for Dot)">;
 

llvm/lib/Target/RISCV/RISCVISelLowering.h

enum NodeType : unsigned {
 
   // WARNING: Do not add anything in the end unless you want the node to
   // have memop! In fact, starting from FIRST_TARGET_MEMORY_OPCODE all
   // opcodes will be thought as target memory ops!
 
   // Load address.
   LA = ISD::FIRST_TARGET_MEMORY_OPCODE,
   LA_TLS_IE,
+
+  TH_LWD,
+  TH_LWUD,
+  TH_LDD,
+  TH_SWD,
+  TH_SDD,
 };
 } // namespace RISCVISD
 
 class RISCVTargetLowering : public TargetLowering {
   const RISCVSubtarget &Subtarget;
 
 public:
   explicit RISCVTargetLowering(const TargetMachine &TM,

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

if (Subtarget.hasStdExtZfhOrZfhmin())
     setTargetDAGCombine(ISD::SIGN_EXTEND_INREG);
   if (Subtarget.hasStdExtF())
     setTargetDAGCombine({ISD::ZERO_EXTEND, ISD::FP_TO_SINT, ISD::FP_TO_UINT,
                          ISD::FP_TO_SINT_SAT, ISD::FP_TO_UINT_SAT});
   if (Subtarget.hasVInstructions())
     setTargetDAGCombine({ISD::FCOPYSIGN, ISD::MGATHER, ISD::MSCATTER,
                          ISD::VP_GATHER, ISD::VP_SCATTER, ISD::SRA, ISD::SRL,
                          ISD::SHL, ISD::STORE, ISD::SPLAT_VECTOR});
+  if (Subtarget.hasVendorXTHeadMemPair())
+    setTargetDAGCombine({ISD::LOAD, ISD::STORE});
   if (Subtarget.useRVVForFixedLengthVectors())
     setTargetDAGCombine(ISD::BITCAST);
 
   setLibcallName(RTLIB::FPEXT_F16_F32, "__extendhfsf2");
   setLibcallName(RTLIB::FPROUND_F32_F16, "__truncsfhf2");
 }
 
 EVT RISCVTargetLowering::getSetCCResultType(const DataLayout &DL,
combineBinOp_VLToVWBinOp_VL(SDNode *N, TargetLowering::DAGCombinerInfo &DCI) {
   }
   for (std::pair<SDValue, SDValue> OldNewValues : ValuesToReplace) {
     DAG.ReplaceAllUsesOfValueWith(OldNewValues.first, OldNewValues.second);
     DCI.AddToWorklist(OldNewValues.second.getNode());
   }
   return InputRootReplacement;
 }
 
+// Helper function for performMemPairCombine.
+// Try to combine the memory loads/stores LSNode1 and LSNode2
+// into a single memory pair operation.
+static SDValue tryMemPairCombine(SelectionDAG &DAG, LSBaseSDNode *LSNode1,
+                                 LSBaseSDNode *LSNode2, SDValue BasePtr,
+                                 uint64_t Imm) {
+  SmallPtrSet<const SDNode *, 32> Visited;
+  SmallVector<const SDNode *, 8> Worklist = {LSNode1, LSNode2};
+
+  if (SDNode::hasPredecessorHelper(LSNode1, Visited, Worklist) ||
+      SDNode::hasPredecessorHelper(LSNode2, Visited, Worklist))
+    return SDValue();
+
+  MachineFunction &MF = DAG.getMachineFunction();
+  const RISCVSubtarget &Subtarget = MF.getSubtarget<RISCVSubtarget>();
+
+  // The new operation has twice the width.
+  MVT XLenVT = Subtarget.getXLenVT();
+  EVT MemVT = LSNode1->getMemoryVT();
+  EVT NewMemVT = (MemVT == MVT::i32) ? MVT::i64 : MVT::i128;
+  MachineMemOperand *MMO = LSNode1->getMemOperand();
+  MachineMemOperand *NewMMO = MF.getMachineMemOperand(
+      MMO, MMO->getPointerInfo(), MemVT == MVT::i32 ? 8 : 16);
+
+  if (LSNode1->getOpcode() == ISD::LOAD) {
+    auto Ext = cast<LoadSDNode>(LSNode1)->getExtensionType();
+    unsigned Opcode;
+    if (MemVT == MVT::i32)
+      Opcode = (Ext == ISD::ZEXTLOAD) ? RISCVISD::TH_LWUD : RISCVISD::TH_LWD;
+    else
+      Opcode = RISCVISD::TH_LDD;
+
+    SDValue Res = DAG.getMemIntrinsicNode(
+        Opcode, SDLoc(LSNode1), DAG.getVTList({XLenVT, XLenVT, MVT::Other}),
+        {LSNode1->getChain(), BasePtr,
+         DAG.getConstant(Imm, SDLoc(LSNode1), XLenVT)},
+        NewMemVT, NewMMO);
+
+    SDValue Node1 =
+        DAG.getMergeValues({Res.getValue(0), Res.getValue(2)}, SDLoc(LSNode1));
+    SDValue Node2 =
+        DAG.getMergeValues({Res.getValue(1), Res.getValue(2)}, SDLoc(LSNode2));
+
+    DAG.ReplaceAllUsesWith(LSNode2, Node2.getNode());
+    return Node1;
+  } else {
+    unsigned Opcode = (MemVT == MVT::i32) ? RISCVISD::TH_SWD : RISCVISD::TH_SDD;
+
+    SDValue Res = DAG.getMemIntrinsicNode(
+        Opcode, SDLoc(LSNode1), DAG.getVTList(MVT::Other),
+        {LSNode1->getChain(), LSNode1->getOperand(1), LSNode2->getOperand(1),
+         BasePtr, DAG.getConstant(Imm, SDLoc(LSNode1), XLenVT)},
+        NewMemVT, NewMMO);
+
+    DAG.ReplaceAllUsesWith(LSNode2, Res.getNode());
+    return Res;
+  }
+}
+
+// Try to combine two adjacent loads/stores to a single pair instruction from
+// the XTHeadMemPair vendor extension.
+static SDValue performMemPairCombine(SDNode *N,
+                                     TargetLowering::DAGCombinerInfo &DCI) {
+  SelectionDAG &DAG = DCI.DAG;
+  MachineFunction &MF = DAG.getMachineFunction();
+  const RISCVSubtarget &Subtarget = MF.getSubtarget<RISCVSubtarget>();
+
+  // Target does not support load/store pair.
+  if (!Subtarget.hasVendorXTHeadMemPair())
+    return SDValue();
+
+  LSBaseSDNode *LSNode1 = cast<LSBaseSDNode>(N);
+  EVT MemVT = LSNode1->getMemoryVT();
+  unsigned OpNum = LSNode1->getOpcode() == ISD::LOAD ? 1 : 2;
+
+  // No volatile, indexed or atomic loads/stores.
+  if (!LSNode1->isSimple() || LSNode1->isIndexed())
+    return SDValue();
+
+  // Function to get a base + constant representation from a memory value.
+  auto ExtractBaseAndOffset = [](SDValue Ptr) -> std::pair<SDValue, uint64_t> {
+    if (Ptr->getOpcode() == ISD::ADD)
+      if (auto *C1 = dyn_cast<ConstantSDNode>(Ptr->getOperand(1)))
+        return {Ptr->getOperand(0), C1->getZExtValue()};
+    return {Ptr, 0};
+  };
+
+  auto [Base1, Offset1] = ExtractBaseAndOffset(LSNode1->getOperand(OpNum));
+
+  SDValue Chain = N->getOperand(0);
+  for (SDNode::use_iterator UI = Chain->use_begin(), UE = Chain->use_end();
+       UI != UE; ++UI) {
+    SDUse &Use = UI.getUse();
+    if (Use.getUser() != N && Use.getResNo() == 0 &&
+        Use.getUser()->getOpcode() == N->getOpcode()) {
+      LSBaseSDNode *LSNode2 = cast<LSBaseSDNode>(Use.getUser());
+
+      // No volatile, indexed or atomic loads/stores.
+      if (!LSNode2->isSimple() || LSNode2->isIndexed())
+        continue;
+
+      // Check if LSNode1 and LSNode2 have the same type and extension.
+      if (LSNode1->getOpcode() == ISD::LOAD)
+        if (cast<LoadSDNode>(LSNode2)->getExtensionType() !=
+            cast<LoadSDNode>(LSNode1)->getExtensionType())
+          continue;
+
+      if (LSNode1->getMemoryVT() != LSNode2->getMemoryVT())
+        continue;
+
+      auto [Base2, Offset2] = ExtractBaseAndOffset(LSNode2->getOperand(OpNum));
+
+      // Check if the base pointer is the same for both instruction.
+      if (Base1 != Base2)
+        continue;
+
+      // Check if the offsets match the XTHeadMemPair encoding contraints.
+      if (MemVT == MVT::i32) {
+        // Check for adjacent i32 values and a 2-bit index.
+        if ((Offset1 + 4 != Offset2) || !isShiftedUInt<2, 3>(Offset1))
+          continue;
+      } else if (MemVT == MVT::i64) {
+        // Check for adjacent i64 values and a 2-bit index.
+        if ((Offset1 + 8 != Offset2) || !isShiftedUInt<2, 4>(Offset1))
+          continue;
+      }
+
+      // Try to combine.
+      if (SDValue Res =
+              tryMemPairCombine(DAG, LSNode1, LSNode2, Base1, Offset1))
+        return Res;
+    }
+  }
+
+  return SDValue();
+}
+
 // Fold
 //   (fp_to_int (froundeven X)) -> fcvt X, rne
 //   (fp_to_int (ftrunc X))     -> fcvt X, rtz
 //   (fp_to_int (ffloor X))     -> fcvt X, rdn
 //   (fp_to_int (fceil X))      -> fcvt X, rup
 //   (fp_to_int (fround X))     -> fcvt X, rmm
 static SDValue performFP_TO_INTCombine(SDNode *N,
                                        TargetLowering::DAGCombinerInfo &DCI,
case RISCVISD::VFNMSUB_VL: {
     // If no operands are negated, we're done.
     if (!NegA && !NegB && !NegC)
       return SDValue();
 
     unsigned NewOpcode = negateFMAOpcode(N->getOpcode(), NegA != NegB, NegC);
     return DAG.getNode(NewOpcode, SDLoc(N), N->getValueType(0), A, B, C, Mask,
                        VL);
   }
+  case ISD::LOAD:
   case ISD::STORE: {
+    if (DCI.isAfterLegalizeDAG())
+      if (SDValue V = performMemPairCombine(N, DCI))
+        return V;
+
+    if (N->getOpcode() != ISD::STORE)
+      break;
+
     auto *Store = cast<StoreSDNode>(N);
     SDValue Val = Store->getValue();
     // Combine store of vmv.x.s/vfmv.f.s to vse with VL of 1.
     // vfmv.f.s is represented as extract element from 0. Match it late to avoid
     // any illegal types.
     if (Val.getOpcode() == RISCVISD::VMV_X_S ||
         (DCI.isAfterLegalizeDAG() &&
          Val.getOpcode() == ISD::EXTRACT_VECTOR_ELT &&
#define NODE_NAME_CASE(NODE)                                                   \
   NODE_NAME_CASE(STRICT_FCVT_W_RV64)
   NODE_NAME_CASE(STRICT_FCVT_WU_RV64)
   NODE_NAME_CASE(FROUND)
   NODE_NAME_CASE(READ_CYCLE_WIDE)
   NODE_NAME_CASE(BREV8)
   NODE_NAME_CASE(ORC_B)
   NODE_NAME_CASE(ZIP)
   NODE_NAME_CASE(UNZIP)
+  NODE_NAME_CASE(TH_LWD)
+  NODE_NAME_CASE(TH_LWUD)
+  NODE_NAME_CASE(TH_LDD)
+  NODE_NAME_CASE(TH_SWD)
+  NODE_NAME_CASE(TH_SDD)
   NODE_NAME_CASE(VMV_V_X_VL)
   NODE_NAME_CASE(VFMV_V_F_VL)
   NODE_NAME_CASE(VMV_X_S)
   NODE_NAME_CASE(VMV_S_X_VL)
   NODE_NAME_CASE(VFMV_S_F_VL)
   NODE_NAME_CASE(SPLAT_VECTOR_SPLIT_I64_VL)
   NODE_NAME_CASE(READ_VLENB)
   NODE_NAME_CASE(TRUNCATE_VECTOR_VL)

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

 //===-- RISCVInstrInfoXTHead.td ----------------------------*- tablegen -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file describes the vendor extensions defined by T-Head of Alibaba.
 //
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
+// T-HEAD specific DAG Nodes.
+//===----------------------------------------------------------------------===//
+
+def SDT_TDBLD : SDTypeProfile<2, 2,
+  [SDTCisSameAs<0, 1>, SDTCisSameAs<1, 3>, SDTCisPtrTy<2>, SDTCisVT<3, XLenVT>]>;
+def SDT_TDBST : SDTypeProfile<0, 4,
+  [SDTCisSameAs<0, 1>, SDTCisSameAs<1, 3>, SDTCisPtrTy<2>, SDTCisVT<3, XLenVT>]>;
+
+def TH_TLWUD : SDNode<"RISCVISD::TH_LWUD", SDT_TDBLD,
+  [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
+def TH_TLWD : SDNode<"RISCVISD::TH_LWD", SDT_TDBLD,
+  [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
+def TH_TLDD : SDNode<"RISCVISD::TH_LDD", SDT_TDBLD,
+  [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
+def TH_TSWD : SDNode<"RISCVISD::TH_SWD", SDT_TDBST,
+  [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
+def TH_TSDD : SDNode<"RISCVISD::TH_SDD", SDT_TDBST,
+  [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
+
+//===----------------------------------------------------------------------===//
 // Instruction class templates
 //===----------------------------------------------------------------------===//
 class THInstVdotVV<bits<6> funct6, RISCVVFormat opv, dag outs, dag ins,
                    string opcodestr, string argstr>
     : RVInstVV<funct6, opv, outs, ins, opcodestr, argstr> {
   let Inst{26} = 0;
   let Opcode = OPC_CUSTOM_0.Value;
   let DecoderNamespace = "THeadV";
let Predicates = [HasVendorXTHeadMac], DecoderNamespace = "THeadMac",
   hasSideEffects = 0, mayLoad = 0, mayStore = 0, isCommutable = 1 in
 class THMulAccumulate_rr<bits<7> funct7, string opcodestr>
     : RVInstR<funct7, 0b001, OPC_CUSTOM_0, (outs GPR:$rd_up),
                     (ins GPR:$rd, GPR:$rs1, GPR:$rs2),
         opcodestr, "$rd, $rs1, $rs2"> {
   let Constraints = "$rd_up = $rd";
 }
 
+let Predicates = [HasVendorXTHeadMemPair], DecoderNamespace = "THeadMemPair",
+  hasSideEffects = 0, mayLoad = 1, mayStore = 0 in
+class THLoadPair<bits<5> funct5, string opcodestr>
+  : RVInstR<!shl(funct5, 2), 0b100, OPC_CUSTOM_0,
+  (outs GPR:$rd, GPR:$rs2), (ins GPR:$rs1, uimm2:$uimm2, uimm7:$const3or4),
+  opcodestr, "$rd, $rs2, (${rs1}), $uimm2, $const3or4"> {
+  bits<2> uimm2;
+  let Inst{26-25} = uimm2;
+  let DecoderMethod = "decodeXTHeadMemPair";
+  let Constraints = "@earlyclobber $rd,@earlyclobber $rs2";
+}
+
+let Predicates = [HasVendorXTHeadMemPair], DecoderNamespace = "THeadMemPair",
+  hasSideEffects = 0, mayLoad = 0, mayStore = 1 in
+class THStorePair<bits<5> funct5, string opcodestr>
+  : RVInstR<!shl(funct5, 2), 0b101, OPC_CUSTOM_0,
+  (outs), (ins GPR:$rd, GPR:$rs2, GPR:$rs1, uimm2:$uimm2, uimm7:$const3or4),
+  opcodestr, "$rd, $rs2, (${rs1}), $uimm2, $const3or4"> {
+  bits<2> uimm2;
+  let Inst{26-25} = uimm2;
+  let DecoderMethod = "decodeXTHeadMemPair";
+}
+
 
 //===----------------------------------------------------------------------===//
 // Combination of instruction classes.
 // Use these multiclasses to define instructions more easily.
 //===----------------------------------------------------------------------===//
 multiclass THVdotVMAQA_VX<string opcodestr, bits<6> funct6> {
   def _VX : THVdotALUrVX<funct6, OPMVX, opcodestr # ".vx">;
 }

 def TH_MULSH : THMulAccumulate_rr<0b0010101, "th.mulsh">;
 } // Predicates = [HasVendorXTHeadMac]
 
 let Predicates = [HasVendorXTHeadMac, IsRV64], IsSignExtendingOpW = 1 in {
 def TH_MULAW : THMulAccumulate_rr<0b0010010, "th.mulaw">;
 def TH_MULSW : THMulAccumulate_rr<0b0010011, "th.mulsw">;
 } // Predicates = [HasVendorXTHeadMac, IsRV64]
 
+let Predicates = [HasVendorXTHeadMemPair] in {
+def TH_LWUD : THLoadPair<0b11110, "th.lwud">,
+        Sched<[WriteLDW, WriteLDW, ReadMemBase]>;
+def TH_SWD  : THStorePair<0b11100, "th.swd">,
+        Sched<[WriteSTW, WriteSTW, ReadStoreData, ReadMemBase]>;
+let IsSignExtendingOpW = 1 in
+def TH_LWD  : THLoadPair<0b11100, "th.lwd">,
+        Sched<[WriteLDW, WriteLDW, ReadMemBase]>;
+}
+
+let Predicates = [HasVendorXTHeadMemPair, IsRV64] in {
+def TH_LDD  : THLoadPair<0b11111, "th.ldd">,
+        Sched<[WriteLDD, WriteLDD, ReadMemBase]>;
+def TH_SDD  : THStorePair<0b11111, "th.sdd">,
+        Sched<[WriteSTD, WriteSTD, ReadStoreData, ReadMemBase]>;
+}
+
 let Predicates = [HasVendorXTHeadVdot],
     Constraints = "@earlyclobber $vd",
     RVVConstraint = WidenV in {
 defm THVdotVMAQA      : THVdotVMAQA<"th.vmaqa",     0b100000>;
 defm THVdotVMAQAU     : THVdotVMAQA<"th.vmaqau",    0b100010>;
 defm THVdotVMAQASU    : THVdotVMAQA<"th.vmaqasu",   0b100100>;
 defm THVdotVMAQAUS    : THVdotVMAQA_VX<"th.vmaqaus",0b100110>;
 }

 defm PseudoTHVdotVMAQAUS    : VPseudoVMAQA_VX;
 
 let Predicates = [HasVendorXTHeadVdot] in {
 defm : VPatTernaryVMAQA_VV_VX<"int_riscv_th_vmaqa",  "PseudoTHVdotVMAQA",  AllQuadWidenableInt8NoVLMulVectors>;
 defm : VPatTernaryVMAQA_VV_VX<"int_riscv_th_vmaqau", "PseudoTHVdotVMAQAU", AllQuadWidenableInt8NoVLMulVectors>;
 defm : VPatTernaryVMAQA_VV_VX<"int_riscv_th_vmaqasu","PseudoTHVdotVMAQASU",AllQuadWidenableInt8NoVLMulVectors>;
 defm : VPatTernaryVMAQA_VX<"int_riscv_th_vmaqaus",   "PseudoTHVdotVMAQAUS",AllQuadWidenableInt8NoVLMulVectors>;
 }
+
+def uimm2_3_XFORM : SDNodeXForm<imm, [{
+  return CurDAG->getTargetConstant((N->getZExtValue() >> 3) & 0x3,
+           SDLoc(N), Subtarget->getXLenVT());
+}]>;
+
+def uimm2_3 : Operand<XLenVT>, ImmLeaf<XLenVT, [{
+  return isShiftedUInt<2, 3>(Imm);
+}], uimm2_3_XFORM>;
+
+def uimm2_4_XFORM : SDNodeXForm<imm, [{
+  return CurDAG->getTargetConstant((N->getZExtValue() >> 4) & 0x3,
+           SDLoc(N), Subtarget->getXLenVT());
+}]>;
+
+def uimm2_4 : Operand<XLenVT>, ImmLeaf<XLenVT, [{
+  return isShiftedUInt<2, 4>(Imm);
+}], uimm2_4_XFORM>;
+
+let Predicates = [HasVendorXTHeadMemPair, IsRV64] in {
+def : Pat<(TH_TLWUD i64:$rs1, uimm2_3:$uimm2_3), (TH_LWUD i64:$rs1, uimm2_3:$uimm2_3, 3)>;
+def : Pat<(TH_TLDD i64:$rs1, uimm2_4:$uimm2_4), (TH_LDD i64:$rs1, uimm2_4:$uimm2_4, 4)>;
+
+def : Pat<(TH_TSDD i64:$rd1, i64:$rd2, i64:$rs1, uimm2_4:$uimm2_4),
+  (TH_SDD i64:$rd1, i64:$rd2, i64:$rs1, uimm2_4:$uimm2_4, 4)>;
+}
+
+let Predicates = [HasVendorXTHeadMemPair] in {
+  def : Pat<(TH_TLWD GPR:$rs1, uimm2_3:$uimm2_3), (TH_LWD GPR:$rs1, uimm2_3:$uimm2_3, 3)>;
+  def : Pat<(TH_TSWD GPR:$rd1, GPR:$rd2, GPR:$rs1, uimm2_3:$uimm2_3),
+            (TH_SWD GPR:$rd1, GPR:$rd2, GPR:$rs1, uimm2_3:$uimm2_3, 3)>;
+}

llvm/test/CodeGen/RISCV/attributes.ll

 ; RUN: llc -mtriple=riscv32 -mattr=+zbkb,+zbkc,+zbkx,+zksed,+zksh %s -o - | FileCheck --check-prefixes=CHECK,RV32COMBINEINTOZKS %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicbom %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICBOM %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicboz %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICBOZ %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicbop %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICBOP %s
 ; RUN: llc -mtriple=riscv32 -mattr=+svnapot %s -o - | FileCheck --check-prefixes=CHECK,RV32SVNAPOT %s
 ; RUN: llc -mtriple=riscv32 -mattr=+svpbmt %s -o - | FileCheck --check-prefixes=CHECK,RV32SVPBMT %s
 ; RUN: llc -mtriple=riscv32 -mattr=+svinval %s -o - | FileCheck --check-prefixes=CHECK,RV32SVINVAL %s
 ; RUN: llc -mtriple=riscv32 -mattr=+xtheadmac %s -o - | FileCheck --check-prefixes=CHECK,RV32XTHEADMAC %s
+; RUN: llc -mtriple=riscv32 -mattr=+xtheadmempair %s -o - | FileCheck --check-prefix=RV32XTHEADMEMPAIR %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zca %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCA %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcb %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCB %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcd %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCD %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcf %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCF %s
 
 ; RUN: llc -mtriple=riscv64 %s -o - | FileCheck %s
 ; RUN: llc -mtriple=riscv64 -mattr=+m %s -o - | FileCheck --check-prefixes=CHECK,RV64M %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zmmul %s -o - | FileCheck --check-prefixes=CHECK,RV64ZMMUL %s

 ; RUN: llc -mtriple=riscv64 -mattr=+svnapot %s -o - | FileCheck --check-prefixes=CHECK,RV64SVNAPOT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+svpbmt %s -o - | FileCheck --check-prefixes=CHECK,RV64SVPBMT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+svinval %s -o - | FileCheck --check-prefixes=CHECK,RV64SVINVAL %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xventanacondops %s -o - | FileCheck --check-prefixes=CHECK,RV64XVENTANACONDOPS %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadba %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADBA %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadbb %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADBB %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadbs %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADBS %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadMAC %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADMAC %s
+; RUN: llc -mtriple=riscv64 -mattr=+xtheadmempair %s -o - | FileCheck --check-prefix=RV64XTHEADMEMPAIR %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadvdot %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADVDOT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zawrs %s -o - | FileCheck --check-prefixes=CHECK,RV64ZAWRS %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-ztso %s -o - | FileCheck --check-prefixes=CHECK,RV64ZTSO %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zca %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCA %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcb %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCB %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcd %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCD %s
 
 ; CHECK: .attribute 4, 16

 ; RV32COMBINEINTOZKS: .attribute 5, "rv32i2p0_zbkb1p0_zbkc1p0_zbkx1p0_zks1p0_zksed1p0_zksh1p0"
 ; RV32ZICBOM: .attribute 5, "rv32i2p0_zicbom1p0"
 ; RV32ZICBOZ: .attribute 5, "rv32i2p0_zicboz1p0"
 ; RV32ZICBOP: .attribute 5, "rv32i2p0_zicbop1p0"
 ; RV32SVNAPOT: .attribute 5, "rv32i2p0_svnapot1p0"
 ; RV32SVPBMT: .attribute 5, "rv32i2p0_svpbmt1p0"
 ; RV32SVINVAL: .attribute 5, "rv32i2p0_svinval1p0"
 ; RV32XTHEADMAC: .attribute 5, "rv32i2p0_xtheadmac1p0"
+; RV32XTHEADMEMPAIR: .attribute 5, "rv32i2p0_xtheadmempair1p0"
 ; RV32ZCA: .attribute 5, "rv32i2p0_zca1p0"
 ; RV32ZCB: .attribute 5, "rv32i2p0_zca1p0_zcb1p0"
 ; RV32ZCD: .attribute 5, "rv32i2p0_zcd1p0"
 ; RV32ZCF: .attribute 5, "rv32i2p0_zcf1p0"
 
 ; RV64M: .attribute 5, "rv64i2p0_m2p0"
 ; RV64ZMMUL: .attribute 5, "rv64i2p0_zmmul1p0"
 ; RV64MZMMUL: .attribute 5, "rv64i2p0_m2p0_zmmul1p0"

 ; RV64SVNAPOT: .attribute 5, "rv64i2p0_svnapot1p0"
 ; RV64SVPBMT: .attribute 5, "rv64i2p0_svpbmt1p0"
 ; RV64SVINVAL: .attribute 5, "rv64i2p0_svinval1p0"
 ; RV64XVENTANACONDOPS: .attribute 5, "rv64i2p0_xventanacondops1p0"
 ; RV64XTHEADBA: .attribute 5, "rv64i2p0_xtheadba1p0"
 ; RV64XTHEADBB: .attribute 5, "rv64i2p0_xtheadbb1p0"
 ; RV64XTHEADBS: .attribute 5, "rv64i2p0_xtheadbs1p0"
 ; RV64XTHEADMAC: .attribute 5, "rv64i2p0_xtheadmac1p0"
+; RV64XTHEADMEMPAIR: .attribute 5, "rv64i2p0_xtheadmempair1p0"
 ; RV64XTHEADVDOT: .attribute 5, "rv64i2p0_f2p0_d2p0_v1p0_zve32f1p0_zve32x1p0_zve64d1p0_zve64f1p0_zve64x1p0_zvl128b1p0_zvl32b1p0_zvl64b1p0_xtheadvdot1p0"
 ; RV64ZTSO: .attribute 5, "rv64i2p0_ztso0p1"
 ; RV64ZCA: .attribute 5, "rv64i2p0_zca1p0"
 ; RV64ZCB: .attribute 5, "rv64i2p0_zca1p0_zcb1p0"
 ; RV64ZCD: .attribute 5, "rv64i2p0_zcd1p0"
 
 define i32 @addi(i32 %a) {
   %1 = add i32 %a, 1
   ret i32 %1
 }

llvm/test/CodeGen/RISCV/xtheadmempair.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+xtheadmempair -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s -check-prefix=RV32XTHEADMEMPAIR
+; RUN: llc -mtriple=riscv64 -mattr=+xtheadmempair -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s -check-prefix=RV64XTHEADMEMPAIR
+
+define i64 @lwd(i32* %a) {
+; RV32XTHEADMEMPAIR-LABEL: lwd:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 2, 3
+; RV32XTHEADMEMPAIR-NEXT:    srai a3, a1, 31
+; RV32XTHEADMEMPAIR-NEXT:    srai a4, a2, 31
+; RV32XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV32XTHEADMEMPAIR-NEXT:    sltu a1, a0, a1
+; RV32XTHEADMEMPAIR-NEXT:    add a3, a3, a4
+; RV32XTHEADMEMPAIR-NEXT:    add a1, a3, a1
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: lwd:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 2, 3
+; RV64XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i32, i32* %a, i64 4
+  %2 = load i32, i32* %1, align 4
+  %3 = getelementptr i32, i32* %a, i64 5
+  %4 = load i32, i32* %3, align 4
+  %5 = sext i32 %2 to i64
+  %6 = sext i32 %4 to i64
+  %7 = add i64 %5, %6
+  ret i64 %7
+}
+
+define i64 @lwud(i32* %a) {
+; RV32XTHEADMEMPAIR-LABEL: lwud:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 2, 3
+; RV32XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV32XTHEADMEMPAIR-NEXT:    sltu a1, a0, a1
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: lwud:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.lwud a1, a2, (a0), 2, 3
+; RV64XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i32, i32* %a, i64 4
+  %2 = load i32, i32* %1, align 4
+  %3 = getelementptr i32, i32* %a, i64 5
+  %4 = load i32, i32* %3, align 4
+  %5 = zext i32 %2 to i64
+  %6 = zext i32 %4 to i64
+  %7 = add i64 %5, %6
+  ret i64 %7
+}
+
+define i64 @ldd(i64* %a) {
+; RV32XTHEADMEMPAIR-LABEL: ldd:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    lw a1, 32(a0)
+; RV32XTHEADMEMPAIR-NEXT:    lw a2, 36(a0)
+; RV32XTHEADMEMPAIR-NEXT:    lw a3, 44(a0)
+; RV32XTHEADMEMPAIR-NEXT:    lw a0, 40(a0)
+; RV32XTHEADMEMPAIR-NEXT:    add a2, a2, a3
+; RV32XTHEADMEMPAIR-NEXT:    add a0, a1, a0
+; RV32XTHEADMEMPAIR-NEXT:    sltu a1, a0, a1
+; RV32XTHEADMEMPAIR-NEXT:    add a1, a2, a1
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: ldd:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.ldd a1, a2, (a0), 2, 4
+; RV64XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i64, i64* %a, i64 4
+  %2 = load i64, i64* %1, align 8
+  %3 = getelementptr i64, i64* %a, i64 5
+  %4 = load i64, i64* %3, align 8
+  %5 = add i64 %2, %4
+  ret i64 %5
+}
+
+define i64 @lwd_0(i32* %a) {
+; RV32XTHEADMEMPAIR-LABEL: lwd_0:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    srai a3, a1, 31
+; RV32XTHEADMEMPAIR-NEXT:    srai a4, a2, 31
+; RV32XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV32XTHEADMEMPAIR-NEXT:    sltu a1, a0, a1
+; RV32XTHEADMEMPAIR-NEXT:    add a3, a3, a4
+; RV32XTHEADMEMPAIR-NEXT:    add a1, a3, a1
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: lwd_0:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 0, 3
+; RV64XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i32, i32* %a, i64 0
+  %2 = load i32, i32* %1, align 4
+  %3 = getelementptr i32, i32* %a, i64 1
+  %4 = load i32, i32* %3, align 4
+  %5 = sext i32 %2 to i64
+  %6 = sext i32 %4 to i64
+  %7 = add i64 %5, %6
+  ret i64 %7
+}
+
+define i64 @lwud_0(i32* %a) {
+; RV32XTHEADMEMPAIR-LABEL: lwud_0:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV32XTHEADMEMPAIR-NEXT:    sltu a1, a0, a1
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: lwud_0:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.lwud a1, a2, (a0), 0, 3
+; RV64XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i32, i32* %a, i64 0
+  %2 = load i32, i32* %1, align 4
+  %3 = getelementptr i32, i32* %a, i64 1
+  %4 = load i32, i32* %3, align 4
+  %5 = zext i32 %2 to i64
+  %6 = zext i32 %4 to i64
+  %7 = add i64 %5, %6
+  ret i64 %7
+}
+
+define i64 @ldd_0(i64* %a) {
+; RV32XTHEADMEMPAIR-LABEL: ldd_0:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a1, a2, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a3, a4, (a0), 1, 3
+; RV32XTHEADMEMPAIR-NEXT:    add a2, a2, a4
+; RV32XTHEADMEMPAIR-NEXT:    add a0, a1, a3
+; RV32XTHEADMEMPAIR-NEXT:    sltu a1, a0, a1
+; RV32XTHEADMEMPAIR-NEXT:    add a1, a2, a1
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: ldd_0:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.ldd a1, a2, (a0), 0, 4
+; RV64XTHEADMEMPAIR-NEXT:    add a0, a1, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i64, i64* %a, i64 0
+  %2 = load i64, i64* %1, align 8
+  %3 = getelementptr i64, i64* %a, i64 1
+  %4 = load i64, i64* %3, align 8
+  %5 = add i64 %2, %4
+  ret i64 %5
+}
+
+define void @swd(i32* %a, i32 %b, i32%c) {
+; RV32XTHEADMEMPAIR-LABEL: swd:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a1, a2, (a0), 2, 3
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: swd:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.swd a1, a2, (a0), 2, 3
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i32, i32* %a, i64 4
+  store i32 %b, i32* %1, align 4
+  %2 = getelementptr i32, i32* %a, i64 5
+  store i32 %c, i32* %2, align 4
+  ret void
+}
+
+define void @sdd(i64* %a, i64 %b, i64%c) {
+; RV32XTHEADMEMPAIR-LABEL: sdd:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    sw a2, 36(a0)
+; RV32XTHEADMEMPAIR-NEXT:    sw a1, 32(a0)
+; RV32XTHEADMEMPAIR-NEXT:    sw a4, 44(a0)
+; RV32XTHEADMEMPAIR-NEXT:    sw a3, 40(a0)
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: sdd:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.sdd a1, a2, (a0), 2, 4
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i64, i64* %a, i64 4
+  store i64 %b, i64* %1, align 8
+  %2 = getelementptr i64, i64* %a, i64 5
+  store i64 %c, i64* %2, align 8
+  ret void
+}
+
+define void @swd_0(i32* %a, i32 %b, i32%c) {
+; RV32XTHEADMEMPAIR-LABEL: swd_0:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a1, a2, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: swd_0:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.swd a1, a2, (a0), 0, 3
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i32, i32* %a, i64 0
+  store i32 %b, i32* %1, align 4
+  %2 = getelementptr i32, i32* %a, i64 1
+  store i32 %c, i32* %2, align 4
+  ret void
+}
+
+define void @sdd_0(i64* %a, i64 %b, i64%c) {
+; RV32XTHEADMEMPAIR-LABEL: sdd_0:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a1, a2, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a3, a4, (a0), 1, 3
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: sdd_0:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.sdd a1, a2, (a0), 0, 4
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i64, i64* %a, i64 0
+  store i64 %b, i64* %1, align 8
+  %2 = getelementptr i64, i64* %a, i64 1
+  store i64 %c, i64* %2, align 8
+  ret void
+}
+
+define i64 @ld64(i64* %a) {
+; RV32XTHEADMEMPAIR-LABEL: ld64:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a2, a1, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    mv a0, a2
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: ld64:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    ld a0, 0(a0)
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i64, i64* %a, i64 0
+  %2 = load i64, i64* %1, align 8
+  ret i64 %2
+}
+
+define i128 @ld128(i128* %a) {
+; RV32XTHEADMEMPAIR-LABEL: ld128:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a2, a3, (a1), 1, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a4, a5, (a1), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a2, a3, (a0), 1, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a4, a5, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: ld128:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.ldd a2, a1, (a0), 0, 4
+; RV64XTHEADMEMPAIR-NEXT:    mv a0, a2
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i128, i128* %a, i64 0
+  %2 = load i128, i128* %1, align 8
+  ret i128 %2
+}
+
+define void @sd64(i64* %a, i64 %b) {
+; RV32XTHEADMEMPAIR-LABEL: sd64:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a1, a2, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: sd64:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    sd a1, 0(a0)
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i64, i64* %a, i64 0
+  store i64 %b, i64* %1, align 8
+  ret void
+}
+
+define void @sd128(i128* %a, i128 %b) {
+; RV32XTHEADMEMPAIR-LABEL: sd128:
+; RV32XTHEADMEMPAIR:       # %bb.0:
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a2, a3, (a1), 1, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.lwd a4, a5, (a1), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a2, a3, (a0), 1, 3
+; RV32XTHEADMEMPAIR-NEXT:    th.swd a4, a5, (a0), 0, 3
+; RV32XTHEADMEMPAIR-NEXT:    ret
+;
+; RV64XTHEADMEMPAIR-LABEL: sd128:
+; RV64XTHEADMEMPAIR:       # %bb.0:
+; RV64XTHEADMEMPAIR-NEXT:    th.sdd a1, a2, (a0), 0, 4
+; RV64XTHEADMEMPAIR-NEXT:    ret
+  %1 = getelementptr i128, i128* %a, i64 0
+  store i128 %b, i128* %1, align 8
+  ret void
+}

llvm/test/MC/RISCV/rv32xtheadmempair-invalid.s

+# RUN: not llvm-mc -triple riscv32 -mattr=+xtheadmempair < %s 2>&1 | FileCheck %s
+
+th.ldd t0, t1, (t2), 5, 4   # CHECK: [[@LINE]]:22: error: invalid operand for instruction
+th.ldd t0, t1, (t2)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.ldd t0, t1, (t2), 3, 5   # CHECK: [[@LINE]]:1: error: instruction requires the following: RV64I Base Instruction Set{{$}}
+th.sdd a0, a1, (a2), 5, 4   # CHECK: [[@LINE]]:22: error: invalid operand for instruction
+th.sdd a0, a1, (a2)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.sdd a0, a1, (a2), 3, 5   # CHECK: [[@LINE]]:1: error: instruction requires the following: RV64I Base Instruction Set{{$}}
+th.lwud t0, t1, (t2), 5, 4  # CHECK: [[@LINE]]:23: error: immediate must be an integer in the range [0, 3]
+th.lwud t0, t1, (t2)        # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.lwud t0, t1, (t2), 3, 5  # CHECK: [[@LINE]]:26: error: Operand must be constant 3.
+th.lwd a3, a4, (a5), 5, 4   # CHECK: [[@LINE]]:22: error: immediate must be an integer in the range [0, 3]
+th.lwd a3, a4, (a5)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.lwd a3, a4, (a5), 3, 5   # CHECK: [[@LINE]]:25: error: Operand must be constant 3.
+th.swd t3, t4, (t5), 5, 4   # CHECK: [[@LINE]]:22: error: immediate must be an integer in the range [0, 3]
+th.swd t3, t4, (t5)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.swd t3, t4, (t5), 3, 5   # CHECK: [[@LINE]]:25: error: Operand must be constant 3.
+th.lwud x6, x6, (x6), 2, 3  # CHECK: [[@LINE]]:9: error: The source register and destination registers cannot be equal.
+th.ldd t0, t1, (t2), 2, 4   # CHECK: [[@LINE]]:1: error: instruction requires the following: RV64I Base Instruction Set{{$}}
+th.sdd t0, t1, (t2), 2, 4   # CHECK: [[@LINE]]:1: error: instruction requires the following: RV64I Base Instruction Set{{$}}

llvm/test/MC/RISCV/rv32xtheadmempair-valid.s

+# With Bitmanip base extension:
+# RUN: llvm-mc %s -triple=riscv32 -mattr=+xtheadmempair -show-encoding \
+# RUN:     | FileCheck -check-prefixes=CHECK-ASM,CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc -filetype=obj -triple=riscv32 -mattr=+xtheadmempair < %s \
+# RUN:     | llvm-objdump --mattr=+xtheadmempair -d -r - \
+# RUN:     | FileCheck --check-prefix=CHECK-ASM-AND-OBJ %s
+
+# CHECK-ASM-AND-OBJ: th.lwd
+# CHECK-ASM: encoding: [0x0b,0x45,0xb6,0xe2]
+th.lwd a0, a1, (a2), 1, 3
+
+# CHECK-ASM-AND-OBJ: th.lwud
+# CHECK-ASM: encoding: [0x0b,0x45,0xb6,0xf4]
+th.lwud a0, a1, (a2), 2, 3
+
+# CHECK-ASM-AND-OBJ: th.swd
+# CHECK-ASM: encoding: [0x0b,0x55,0xb6,0xe0]
+th.swd a0, a1, (a2), 0, 3

llvm/test/MC/RISCV/rv64xtheadmempair-invalid.s

+# RUN: not llvm-mc -triple riscv64 -mattr=+xtheadmempair < %s 2>&1 | FileCheck %s
+
+th.ldd t0, t1, (t2), 5, 4   # CHECK: [[@LINE]]:22: error: immediate must be an integer in the range [0, 3]
+th.ldd t0, t1, (t2)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.ldd t0, t1, (t2), 3, 5   # CHECK: [[@LINE]]:25: error: Operand must be constant 4.
+th.sdd a0, a1, (a2), 5, 4   # CHECK: [[@LINE]]:22: error: immediate must be an integer in the range [0, 3]
+th.sdd a0, a1, (a2)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.sdd a0, a1, (a2), 3, 5   # CHECK: [[@LINE]]:25: error: Operand must be constant 4.
+th.lwud t0, t1, (t2), 5, 4  # CHECK: [[@LINE]]:23: error: immediate must be an integer in the range [0, 3]
+th.lwud t0, t1, (t2)        # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.lwud t0, t1, (t2), 3, 5  # CHECK: [[@LINE]]:26: error: Operand must be constant 3.
+th.lwd a3, a4, (a5), 5, 4   # CHECK: [[@LINE]]:22: error: immediate must be an integer in the range [0, 3]
+th.lwd a3, a4, (a5)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.lwd a3, a4, (a5), 3, 5   # CHECK: [[@LINE]]:25: error: Operand must be constant 3.
+th.swd t3, t4, (t5), 5, 4   # CHECK: [[@LINE]]:22: error: immediate must be an integer in the range [0, 3]
+th.swd t3, t4, (t5)         # CHECK: [[@LINE]]:1: error: too few operands for instruction
+th.swd t3, t4, (t5), 3, 5   # CHECK: [[@LINE]]:25: error: Operand must be constant 3.
+th.lwud x6, x6, (x6), 2, 3  # CHECK: [[@LINE]]:9: error: The source register and destination registers cannot be equal.

llvm/test/MC/RISCV/rv64xtheadmempair-valid.s

+# With Bitmanip base extension:
+# RUN: llvm-mc %s -triple=riscv64 -mattr=+xtheadmempair -show-encoding \
+# RUN:     | FileCheck -check-prefixes=CHECK-ASM,CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc -filetype=obj -triple=riscv64 -mattr=+xtheadmempair < %s \
+# RUN:     | llvm-objdump --mattr=+xtheadmempair -d -r - \
+# RUN:     | FileCheck --check-prefix=CHECK-ASM-AND-OBJ %s
+
+# CHECK-ASM-AND-OBJ: th.lwd
+# CHECK-ASM: encoding: [0x0b,0x45,0xb6,0xe2]
+th.lwd a0, a1, (a2), 1, 3
+
+# CHECK-ASM-AND-OBJ: th.lwud
+# CHECK-ASM: encoding: [0x0b,0x45,0xb6,0xf4]
+th.lwud a0, a1, (a2), 2, 3
+
+# CHECK-ASM-AND-OBJ: th.swd
+# CHECK-ASM: encoding: [0x0b,0x55,0xb6,0xe0]
+th.swd a0, a1, (a2), 0, 3
+
+# CHECK-ASM-AND-OBJ: th.ldd
+# CHECK-ASM: encoding: [0x0b,0x45,0xb6,0xf8]
+th.ldd a0, a1, (a2), 0, 4
+
+# CHECK-ASM-AND-OBJ: th.sdd
+# CHECK-ASM: encoding: [0x0b,0x55,0xb6,0xfe]
+th.sdd a0, a1, (a2), 3, 4