[llvm][aarch64] SVE addressing modes.

[fpetrogalli]

Summary

Added register + immediate and register + register addressing modes for the following intrinsics:

1. Masked load and stores:
   • Sign and zero extended load and truncated stores.
   • No extension or truncation.
2. Masked non-temporal load and store.

[fpetrogalli]

The patch now covers all the LLVM intrinsic listed in the summary:

1. Masked load and stores:
   • Sign and zero extended load and truncated stores.
   • No extension or truncation.
2. Non-temporal load and store.

[fpetrogalli]

Notice that this patch is based (and depends) on https://reviews.llvm.org/D73602.

[andwar]

Hi @fpetrogalli, thank you for working on this.

• IMHO every test functions (e.g.:test_masked_ldst_sv2i8 ) should either test contiguous load or store (i.e. only one thing at a time). That will help triaging potential bugs in the future and also would be consistent with other test files in this folder.
• [nit] AFAIK, + is never used in file names (e.g. sve-pred-non-temporal-ldst-addressing-mode-reg+reg.ll). I couldn't find any specific rule for that though.
• [nit] The idea of %sv2i1 = type <vscale x 2 x i1> is very neat, but I'm slightly worried that that will complicate searching through test files (we are quite familiar with <vscale x 2 x i1>, not so much with %sv2i1, so you might miss cases when using grep).
• Are the DAG Combine rules required for this patch? If so, could you add tests for them?

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4420

[nit] Wouldn't assert be more suitable?

4439

Base and OffImm are only only needed when building in Debug mode and running e.g. llc with -debug. Maybe move that inside LLVM_DEBUG?

4441

[nit] Match found is very generic (and you use it twice). Maybe sth more specific?

4454

Used only once (so no need for a dedicated variable)

4455

Not used.

4464

Hm, why? Maybe document this method like you did for SelectAddrModeIndexedSVE?

llvm/lib/Target/AArch64/SVEInstrFormats.td

6979

Inconsistent naming with the records that follow this one.

llvm/test/CodeGen/AArch64/sve-gep.ll

7

Unrelated changes.

[fpetrogalli]

@andwar , thank you for the review.

I'll update the patch asap.

IMHO every test functions (e.g.:test_masked_ldst_sv2i8 ) should either test contiguous load or store (i.e. only one thing at a time). That will help triaging potential bugs in the future and also would be consistent with other test files in this folder.

I see your point, but the tests that use merge and store at the same time are using exactly the same addressing modes, it is not that they are using something different. So if something fails in the addressing mode of the load, it fails in the addressing mode of the store. Having them merged together saves quite some typing, and has no disadvantages in term of unit testing.

[nit] AFAIK, + is never used in file names (e.g. sve-pred-non-temporal-ldst-addressing-mode-reg+reg.ll). I couldn't find any specific rule for that though.

Fair point, I'll remove the plus.

[nit] The idea of %sv2i1 = type <vscale x 2 x i1> is very neat, but I'm slightly worried that that will complicate searching through test files (we are quite familiar with <vscale x 2 x i1>, not so much with %sv2i1, so you might miss cases when using grep).

Very good point. The type redefinition saved me a lot of typing, but now that the tests are there it is better o remove it.

Are the DAG Combine rules required for this patch? If so, could you add tests for them?

They are necessary for the reg+imm tests. They are needed to make sure that the ADD node of the base address is always in the format (ADD %BASE (VSCALE CONST)). I'll see if I can add some unit tests specifically for the combiner changes.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

1304

This is to be removed.

1321

This will be removed too.

llvm/test/CodeGen/AArch64/sve-gep.ll

7

No, I had to fix this because of the DAG combine changes.

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4439

No, they are output parameters, passed by reference. They are not just for debug purpose.

4441

I will remove these debug messages, they are not very helpful.

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4464

Good catch, it is not needed.

llvm/lib/Target/AArch64/SVEInstrFormats.td

6979

Used the file convention (lower case, separated with _, starts with am).

[fpetrogalli]

Address code review from @andwar.

I have added the tests for the DAG Combine changes.

[fpetrogalli]

I have uploaded PDF renderings of the DAGS before and after the changes. I am not sure the tests for the combiner are explicit about the fact that the combiner is run on this pattern, but the graphs attached here show the effect of my changes on the code. I think that some of the patterns that are needed to show the changes in the output code are missing.

before.dag.combine_sub_vscale_i64-bdd206.dot.pdf22 KBDownload

after.dag.combine_sub_vscale_i64-7f807e.dot.pdf22 KBDownload

before.dag.combine_mul_vscale_i64-f4259d.dot.pdf21 KBDownload

before.dag.combine_add_vscale_i64-62fd59.dot.pdf20 KBDownload

after.dag.combine_mul_vscale_i64-975c2a.dot.pdf20 KBDownload

after.dag.combine_add_vscale_i64-35445c.dot.pdf20 KBDownload

[fpetrogalli]

This comment was removed by fpetrogalli.

[fpetrogalli]

Cosmetic changes in the comments of the non-temporal reg+reg test.

[andwar]

In D74254#1872363, @fpetrogalli wrote:

IMHO every test functions (e.g.:test_masked_ldst_sv2i8 ) should either test contiguous load or store (i.e. only one thing at a time). That will help triaging potential bugs in the future and also would be consistent with other test files in this folder.

I see your point, but the tests that use merge and store at the same time are using exactly the same addressing modes, it is not that they are using something different. So if something fails in the addressing mode of the load, it fails in the addressing mode of the store. Having them merged together saves quite some typing, and has no disadvantages in term of unit testing.

Ah, I've realised that you split the files per addressing modes. As the name of the patch would suggest :-) OK, keep it as it is.

Thank you for adding DAG diagrams - they are very helpful!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

2330

vscale * C1 and vscale * (C0 + C1)?

3260

I don't quite understand the benefit of this transformation. The selection dag before and after are almost identical.

3601

vscale * (C0 + C1)?

7730

vscale * (C0 * C1)?

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4447

Missing doxstring.

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

11 ↗

(On Diff #244273)

; CHECK-NEXT? Here and in the following examples.

16 ↗

(On Diff #244273)

Could you format this line (and similar lines elsewhere)? E.g.: https://github.com/llvm/llvm-project/blob/318d0ede572080f18d0106dbc354e11c88329a84/llvm/test/CodeGen/AArch64/sve-intrinsics-stores.ll#L11

Makes it easier to parse for humans :) And will be consistent with other files too!

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

6 ↗

(On Diff #244273)

vscale * C1? and vscale * (C0 + C1)?

9 ↗

(On Diff #244273)

Perhaps ; CHECK-NOT: add?

25 ↗

(On Diff #244273)

• vscale * (C0 * C1)?
• What's C0 and what is C1 in this example?

33 ↗

(On Diff #244273)

[nit] Align with the following line (missing space)

38 ↗

(On Diff #244273)

Is nounwind needed in these examples?

55 ↗

(On Diff #244273)

Perhaps ; CHECK-NOT: sub?

77 ↗

(On Diff #244273)

; CHECK-NOT: shl?

80 ↗

(On Diff #244273)

Hm, since it's 6 here, shouldn't line 77 be ; CHECK-NEXT: rdvl x0, #6?

[fpetrogalli]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3260

It is needed to avoid the having to deal with SUB when computing the addressing mode.

3601

Almost: vscale * (C0 * C1).

7730

Almost: (vscale * (C0 << C1)).

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

16 ↗

(On Diff #244273)

I prefer keeping all parameters in one line.

The "consistency with other file" argument doesn't work, most of the tests for SVE uses the convention in this patch:

frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*)$" sve*.ll | grep -v "()" | wc -l
2076
frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*,$" sve*.ll | grep -v "()" | wc -l
1433

The result becomes even more unbalanced if you look at the totality of tests in the folder:

frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*)$" *.ll | grep -v "()" | wc -l
7260
frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*,$" *.ll | grep -v "()" | wc -l
1435

I run grep on master, @ a062a3ed7fd82c277812d80fb83dc6f05b939a84, _not_ on my dev branch :).

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

38 ↗

(On Diff #244273)

Yes, to be able to use CHECK-NEXT after CHECK-LABEL

80 ↗

(On Diff #244273)

At IR level, it is %shl = 2^6 * VSCALE

At Assembly level, the output of RDVL is 2^4 * VSCALE

Hence, to compute %shl we need to multiply RDVL output by 2^2 -> #4 is correct.

Does that make sense? I have actually added it as a comment, but I have modified the code so that it produces rdvl #1.

[fpetrogalli]

@andwar, thank you for you review.

I have updated the code accordingly.

Thank you!

[fpetrogalli]

I have added a CHECK-NOT: mul that was missing from the DAG combiner tests.

[andwar]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

700

DELETEME

4435

[nit] As per https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code, this could be simplified:

if ((MulImm % MemWidthBytes) != 0)
  return false;
 
signed Offset = MulImm / MemWidthBytes;
if ((Offset < Min) || (Offset > Max)) 
  return false;

Base = N.getOperand(0);
OffImm = CurDAG->getTargetConstant(Offset, SDLoc(N), MVT::i64);
return true;

This way we have fewer levels of indentation.

4436

This operation mixes sizes: unsigned vs int64_t. It would be safer to be either:

• consistently explicit about the size of integers (int64_t and int32_t), or
• consistently implicit about the size of integers (unsigned, unsigned long)

4468

[nit] As per https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code, this could be simplified as:

if (RHS.getOpcode() != ISD::SHL) 
  return false;

const SDValue SRHS = RHS.getOperand(1);
auto *C = dyn_cast<ConstantSDNode>(SRHS);
if (nullptr == C) 
  return false;

const uint64_t Shift = C->getZExtValue();
if (Shift == Scale) {
   Base = LHS;
   Offset = RHS.getOperand(0);
    return true;
}

This way you we have fewer levels of indentation.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

1210

[nit] Indent consistently

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

16 ↗

(On Diff #244273)

I think that your comparison misses the context, and that's entirely my fault because I didn't make it clear. If you compare long lines only, the split is roughly 50/50.

I kindly asked for this update because these lines are wrapped by Phabricator (they don't fit on my screen and I don't see how to make Phabricator stop doing that). This makes reviewing on Phab a bit frustrating.

Btw, I think that awk would be more fitting here ;-)

#! /bin/evn bash

TEST_FILES=(
"llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll"
"llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-reg.ll"
"llvm/test/CodeGen/AArch64/sve-pred-non-temporal-ldst-addressing-mode-reg-imm.ll"
"llvm/test/CodeGen/AArch64/sve-pred-non-temporal-ldst-addressing-mode-reg-reg.ll")

for test_file in "${TEST_FILES[@]}"; do
awk -F',' '{
  if ($1 ~ /call/) {
    # Counter number of character for indentation
    for (i = 1; i <= length($0); i++) {
      if (substr($0, i, 1) == "(") {
        lenght_col = i
        break
      }
    }

    # Split function call - 2 args
    if (NF == 2) {
      printf("%s,\n %*s %s\n", $1, lenght_col - 3, "", $2)
    }

    # Split function call - 3 args
    if (NF == 3) {
      printf("%s,\n %*s %s,\n %*s %s\n", $1, lenght_col - 3, "", $2, lenght_col - 3, "", $3)
    }

    # Split function call - 4 args
    if (NF == 4) {
      printf("%s,\n %*s %s,\n %*s %s,\n %*s %s\n", $1, lenght_col - 3, "", $2, lenght_col - 3, "", $3, lenght_col - 3, "", $4)
    }

  } else {
    # Not a call, not reformatting
    print $0
  }
}' ${test_file} > temp.ll

mv temp.ll ${test_file}

done

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-reg.ll

391 ↗

(On Diff #244499)

FIXME

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

6 ↗

(On Diff #244273)

What about vscale * C1?

13 ↗

(On Diff #244499)

Could C0 and C1 be different than 1?

28 ↗

(On Diff #244499)

Did you mean C0 = 2? And multiplication by C0 seems to be missing - there's only multiplication by 32. Shouldn't the IR look like this:

%vscale = call i64 @llvm.vscale.i64()
%mul_by_16 = mul i64 %vscale, 16
%mul_by_2 = mul i64 %mul_by_16, 32
ret i64 %mul_by_2

[andwar]

Cheers for updating this @fpetrogalli ! My most recent comments are mostly nits.

[fpetrogalli]

Thank you @adwar. I have addressed your comments.

Your awk script was very useful.

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

16 ↗

(On Diff #244273)

You win!

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

13 ↗

(On Diff #244499)

Yes, but to get C0 and C1 different from 1 I'd have to use a mul instruction, which is already tested in the combine_mul_vscale_* tests.

I think it is enough to test with C0=C1=1.

28 ↗

(On Diff #244499)

When targeting SVE, @llvm.vscale.*() returns the number of 16-byte chunks of and SVE register. If I multiply it by 32, it means it is returning the number of 4-bit (half-byte) elements in the SVE register. RDVL returns the number of 8-bit lanes in the register, hence the number of 4-bit lanes is given by RDVL Xn, %2.

I have updated the comment setting C1 = 32.

[sdesmalen]

Thanks for creating this patch @fpetrogalli!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

2331

Does it make sense to submit these DAGCombine changes + all related tests in a separate patch?
This patch is a bit of three patches in one at the moment: DAGCombines, reg+reg addressing modes, reg+imm addressing modes.

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

224

nit: This one is missing a comment.

4438

Are you purposely using signed instead of int64_t here?

4439

nit: unnecessary brackets, can be if (Offset < Min || Offset > Max)

4471

nit: Why const?

[fpetrogalli]

I have extracted the DAGCombiner changes in a separate patch as
requested by @sdesmalen: https://reviews.llvm.org/D74782

[fpetrogalli]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

2331

I have extracted the DAGCombiner changes here: https://reviews.llvm.org/D74782

I think that the code for the addressing mode is simple enough that it can stay in a single patch.

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4438

Should be int64_t for consistency.

4471

I try to use const indiscriminately when I know that the variable is not supposed to change inside the scope. The same it is done in other places in this file (not everywhere). I didn't marry this, so if you want I can remove it. :)

[fpetrogalli]

Thank you @sdesmalen for your review!

Francesco

[andwar]

Thank you for all the updates @fpetrogalli ! LGTM

[sdesmalen]

LGTM!

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4472

if (auto *C = dyn_cast<ConstantSDNode>(ShiftRHS)) {
  if (ShiftAmount == C->getZExtValue()) {
    ...
    return true;
  }
}

return false;

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

250 ↗

(On Diff #245224)

nit: did you chose a different alignment here on purpose?

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4472

@sdesmalen, your version saves some lines, I'll apply it and then submit the patch. Thank you!

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

250 ↗

(On Diff #245224)

No - it is not on purpose. Do you want me to set everything to 1? I wouldn't bother, non of the code added in this patch care of the value of the alignment...

[fpetrogalli]

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

250 ↗

(On Diff #245224)

Well, it was easier to change everything to i32 1 instead of doing another round of questions! :)

[fpetrogalli]

Update code as requested by @sdesmalen.

Thank you!

Francesco

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitADD(SDNode *N) {
   if (SDValue V = foldAddSubOfSignBit(N, DAG))
     return V;
 
   // fold (a+b) -> (a|b) iff a and b share no bits.
   if ((!LegalOperations || TLI.isOperationLegal(ISD::OR, VT)) &&
       DAG.haveNoCommonBitsSet(N0, N1))
     return DAG.getNode(ISD::OR, DL, VT, N0, N1);
 
+  // Fold (add (vscale * C0), (vscale C1)) to (vscale  C0 + C1))
+  if (N0.getOpcode() == ISD::VSCALE && N1.getOpcode() == ISD::VSCALE) {
+    APInt C0 = N0->getConstantOperandAPInt(0);
+    APInt C1 = N1->getConstantOperandAPInt(0);
+    return DAG.getVScale(DL, VT, C0 + C1);
+  }
+
   return SDValue();
 }
 
 SDValue DAGCombiner::visitADDSAT(SDNode *N) {
   unsigned Opcode = N->getOpcode();
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N0.getValueType();
if (N1.getOpcode() == ISD::SIGN_EXTEND_INREG) {
     VTSDNode *TN = cast<VTSDNode>(N1.getOperand(1));
     if (TN->getVT() == MVT::i1) {
       SDValue ZExt = DAG.getNode(ISD::AND, DL, VT, N1.getOperand(0),
                                  DAG.getConstant(1, DL, VT));
       return DAG.getNode(ISD::ADD, DL, VT, N0, ZExt);
     }
   }
 
+  // canonicalize (sub X, (vscale * C)) to (add X,  (vscale * -C))
+  if (N1.getOpcode() == ISD::VSCALE) {
+    APInt IntVal = N1.getConstantOperandAPInt(0);
+    return DAG.getNode(ISD::ADD, DL, VT, N0, DAG.getVScale(DL, VT, -IntVal));
+  }
+
   // Prefer an add for more folding potential and possibly better codegen:
   // sub N0, (lshr N10, width-1) --> add N0, (ashr N10, width-1)
   if (!LegalOperations && N1.getOpcode() == ISD::SRL && N1.hasOneUse()) {
     SDValue ShAmt = N1.getOperand(1);
     ConstantSDNode *ShAmtC = isConstOrConstSplat(ShAmt);
     if (ShAmtC &&
         ShAmtC->getAPIntValue() == (N1.getScalarValueSizeInBits() - 1)) {
       SDValue SRA = DAG.getNode(ISD::SRA, DL, VT, N1.getOperand(0), ShAmt);
if (DAG.isConstantIntBuildVectorOrConstantInt(N1) &&
       DAG.isConstantIntBuildVectorOrConstantInt(N0.getOperand(1)) &&
       isMulAddWithConstProfitable(N, N0, N1))
       return DAG.getNode(ISD::ADD, SDLoc(N), VT,
                          DAG.getNode(ISD::MUL, SDLoc(N0), VT,
                                      N0.getOperand(0), N1),
                          DAG.getNode(ISD::MUL, SDLoc(N1), VT,
                                      N0.getOperand(1), N1));
 
+  // Fold (mul (vscale * C0), C1) to (vscale  C0 * C1))
+  if (N0.getOpcode() == ISD::VSCALE)
+    if (ConstantSDNode *NC1 = isConstOrConstSplat(N1)) {
+      APInt C0 = N0.getConstantOperandAPInt(0);
+      APInt C1 = NC1->getAPIntValue();
+      return DAG.getVScale(SDLoc(N), VT, C0 * C1);
+    }
+
   // reassociate mul
   if (SDValue RMUL = reassociateOps(ISD::MUL, SDLoc(N), N0, N1, N->getFlags()))
     return RMUL;
 
   return SDValue();
 }
 
 /// Return true if divmod libcall is available.
if (N0.getOpcode() == ISD::MUL && N0.getNode()->hasOneUse() &&
     if (isConstantOrConstantVector(Shl))
       return DAG.getNode(ISD::MUL, SDLoc(N), VT, N0.getOperand(0), Shl);
   }
 
   if (N1C && !N1C->isOpaque())
     if (SDValue NewSHL = visitShiftByConstant(N))
       return NewSHL;
 
+  // Fold (shl (vscale * C0), C1) to (vscale  C0 << C1))
+  if (N0.getOpcode() == ISD::VSCALE)
+    if (ConstantSDNode *NC1 = isConstOrConstSplat(N->getOperand(1))) {
+      auto DL = SDLoc(N);
+      APInt C0 = N0.getConstantOperandAPInt(0);
+      APInt C1 = NC1->getAPIntValue();
+      return DAG.getVScale(DL, VT, C0 << C1);
+    }
+
   return SDValue();
 }
 
 SDValue DAGCombiner::visitSRA(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   if (SDValue V = DAG.simplifyShift(N0, N1))
     return V;

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

public:
   void SelectLoad(SDNode *N, unsigned NumVecs, unsigned Opc,
                      unsigned SubRegIdx);
   void SelectPostLoad(SDNode *N, unsigned NumVecs, unsigned Opc,
                          unsigned SubRegIdx);
   void SelectLoadLane(SDNode *N, unsigned NumVecs, unsigned Opc);
   void SelectPostLoadLane(SDNode *N, unsigned NumVecs, unsigned Opc);
 
   bool SelectAddrModeFrameIndexSVE(SDValue N, SDValue &Base, SDValue &OffImm);
+  template <int64_t Min, int64_t Max>
+  bool SelectAddrModeIndexedSVE(SDNode *Root, SDValue N, SDValue &Base,
+                                SDValue &OffImm);
+  /// SVE Reg+Reg address mode
+  template <unsigned Scale>
+  bool SelectSVERegRegAddrMode(SDValue N, SDValue &Base, SDValue &Offset) {
+    return SelectSVERegRegAddrMode(N, Scale, Base, Offset);
+  }
 
   void SelectStore(SDNode *N, unsigned NumVecs, unsigned Opc);
   void SelectPostStore(SDNode *N, unsigned NumVecs, unsigned Opc);
   void SelectStoreLane(SDNode *N, unsigned NumVecs, unsigned Opc);
   void SelectPostStoreLane(SDNode *N, unsigned NumVecs, unsigned Opc);
 
   bool tryBitfieldExtractOp(SDNode *N);
   bool tryBitfieldExtractOpFromSExt(SDNode *N);
private:
 
   bool SelectSVEAddSubImm(SDValue N, MVT VT, SDValue &Imm, SDValue &Shift);
 
   bool SelectSVELogicalImm(SDValue N, MVT VT, SDValue &Imm);
 
   bool SelectSVESignedArithImm(SDValue N, SDValue &Imm);
 
   bool SelectSVEArithImm(SDValue N, SDValue &Imm);
+  bool SelectSVERegRegAddrMode(SDValue N, unsigned Scale, SDValue &Base,
+                               SDValue &Offset);
 };
 } // end anonymous namespace
 
 /// isIntImmediate - This method tests to see if the node is a constant
 /// operand. If so Imm will receive the 32-bit value.
 static bool isIntImmediate(const SDNode *N, uint64_t &Imm) {
   if (const ConstantSDNode *C = dyn_cast<const ConstantSDNode>(N)) {
     Imm = C->getZExtValue();
static SDValue narrowIfNeeded(SelectionDAG *CurDAG, SDValue N) {
   MachineSDNode *Node = CurDAG->getMachineNode(TargetOpcode::EXTRACT_SUBREG,
                                                dl, MVT::i32, N, SubReg);
   return SDValue(Node, 0);
 }
 
 // Returns a suitable CNT/INC/DEC/RDVL multiplier to calculate VSCALE*N.
 template<signed Low, signed High, signed Scale>
 bool AArch64DAGToDAGISel::SelectRDVLImm(SDValue N, SDValue &Imm) {
   if (!isa<ConstantSDNode>(N))
     return false;
 
   int64_t MulImm = cast<ConstantSDNode>(N)->getSExtValue();
   if ((MulImm % std::abs(Scale)) == 0) {
     int64_t RDVLImm = MulImm / Scale;
     if ((RDVLImm >= Low) && (RDVLImm <= High)) {
       Imm = CurDAG->getTargetConstant(RDVLImm, SDLoc(N), MVT::i32);
       return true;

 }
 
 /// createAArch64ISelDag - This pass converts a legalized DAG into a
 /// AArch64-specific DAG, ready for instruction scheduling.
 FunctionPass *llvm::createAArch64ISelDag(AArch64TargetMachine &TM,
                                          CodeGenOpt::Level OptLevel) {
   return new AArch64DAGToDAGISel(TM, OptLevel);
 }
+
+/// SelectAddrModeIndexedSVE - Attempt selection of the addressing mode:
+/// Base + OffImm * sizeof(MemVT) for Min >= OffImm <= Max
+/// where Root is the memory access using N for its address.
+template <int64_t Min, int64_t Max>
+bool AArch64DAGToDAGISel::SelectAddrModeIndexedSVE(SDNode *Root, SDValue N,
+                                                   SDValue &Base,
+                                                   SDValue &OffImm) {
+  if (!isa<MemSDNode>(Root))
+    return false;
+
+  EVT MemVT = cast<MemSDNode>(Root)->getMemoryVT();
+
+  if (N.getOpcode() != ISD::ADD)
+    return false;
+
+  SDValue VScale = N.getOperand(1);
+  if (VScale.getOpcode() != ISD::VSCALE)
+    return false;
+
+  TypeSize TS = MemVT.getSizeInBits();
+  unsigned MemWidthBytes = TS.getKnownMinSize() / 8;
+  int64_t MulImm = cast<ConstantSDNode>(VScale.getOperand(0))->getSExtValue();
+
+  if ((MulImm % MemWidthBytes) == 0) {
+    signed Offset = MulImm / MemWidthBytes;
+    if ((Offset >= Min) && (Offset <= Max)) {
+      Base = N.getOperand(0);
+      OffImm = CurDAG->getTargetConstant(Offset, SDLoc(N), MVT::i64);
+      LLVM_DEBUG(dbgs() << "Match found:\n"; dbgs() << "ROOT:\n"; Root->dumpr();
+                 dbgs() << "BASE:\n"; Base.dumpr(); dbgs() << "OFFSET:\n";
+                 OffImm.dumpr());
+      return true;
+    }
+  }
+
+  return false;
+}
+
+bool AArch64DAGToDAGISel::SelectSVERegRegAddrMode(SDValue N, unsigned Scale,
+                                                  SDValue &Base,
+                                                  SDValue &Offset) {
+  const unsigned Opcode = N.getOpcode();
+  const SDLoc dl(N);
+
+  if (Opcode != ISD::ADD)
+    return false;
+
+  // Process an ADD node
+  const SDValue LHS = N.getOperand(0);
+  const SDValue RHS = N.getOperand(1);
+
+  // We don't match addition to constants
+  if (isa<ConstantSDNode>(RHS) || isa<ConstantSDNode>(LHS))
+    return false;
+
+  // 8 bit data don't have the SHL node, so we treat it separately.
+  if (Scale == 0) {
+    Base = LHS;
+    Offset = RHS;
+    return true;
+  }
+
+  // Check if the RHS is a shift node with a constant.
+  if (RHS.getOpcode() == ISD::SHL) {
+    const SDValue SRHS = RHS.getOperand(1);
+    if (auto C = dyn_cast<ConstantSDNode>(SRHS)) {
+      const uint64_t Shift = C->getZExtValue();
+      if (Shift == Scale) {
+        Base = LHS;
+        Offset = RHS.getOperand(0);
+        LLVM_DEBUG(dbgs() << "Match found, rewriting as:\n"
+                          << "BASE:\n";
+                   Base.dumpr(); dbgs() << "OFFSET:\n"; Offset.dumpr());
+        return true;
+      }
+    }
+  }
+
+  return false;
+}

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

   def : Pat<(nxv2f64 (bitconvert (nxv8i16 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv4i32 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv2i64 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv8f16 ZPR:$src))), (nxv2f64 ZPR:$src)>;
   def : Pat<(nxv2f64 (bitconvert (nxv4f32 ZPR:$src))), (nxv2f64 ZPR:$src)>;
 
   // Add more complex addressing modes here as required
   multiclass pred_load<ValueType Ty, ValueType PredTy, SDPatternOperator Load,
-                        Instruction RegImmInst> {
-
+                       Instruction RegRegInst, Instruction RegImmInst, ComplexPattern AddrCP> {
+    // reg + reg
+    let AddedComplexity = 1 in {
+      def _reg_reg_z : Pat<(Ty (Load (AddrCP GPR64:$base, GPR64:$offset), (PredTy PPR:$gp), (SVEDup0Undef))),
+                         (RegRegInst PPR:$gp, GPR64:$base, GPR64:$offset)>;
+    }
+    // reg + imm
+    let AddedComplexity = 2 in {
+      def _reg_imm_z : Pat<(Ty (Load (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), (PredTy PPR:$gp), (SVEDup0Undef))),
+                           (RegImmInst PPR:$gp, GPR64:$base, simm4s1:$offset)>;
+    }
     def _default_z : Pat<(Ty (Load  GPR64:$base, (PredTy PPR:$gp), (SVEDup0Undef))),
                          (RegImmInst PPR:$gp, GPR64:$base, (i64 0))>;
   }
 
   // 2-element contiguous loads
-  defm : pred_load<nxv2i64, nxv2i1, zext_masked_load_i8,   LD1B_D_IMM>;
-  defm : pred_load<nxv2i64, nxv2i1, asext_masked_load_i8,  LD1SB_D_IMM>;
-  defm : pred_load<nxv2i64, nxv2i1, zext_masked_load_i16,  LD1H_D_IMM>;
-  defm : pred_load<nxv2i64, nxv2i1, asext_masked_load_i16, LD1SH_D_IMM>;
-  defm : pred_load<nxv2i64, nxv2i1, zext_masked_load_i32,  LD1W_D_IMM>;
-  defm : pred_load<nxv2i64, nxv2i1, asext_masked_load_i32, LD1SW_D_IMM>;
-  defm : pred_load<nxv2i64, nxv2i1, nonext_masked_load,    LD1D_IMM>;
-  defm : pred_load<nxv2f16, nxv2i1, nonext_masked_load,    LD1H_D_IMM>;
-  defm : pred_load<nxv2f32, nxv2i1, nonext_masked_load,    LD1W_D_IMM>;
-  defm : pred_load<nxv2f64, nxv2i1, nonext_masked_load,    LD1D_IMM>;
+  defm : pred_load<nxv2i64, nxv2i1, zext_masked_load_i8,   LD1B_D,  LD1B_D_IMM,  SVEAddrModeRegReg8>;
+  defm : pred_load<nxv2i64, nxv2i1, asext_masked_load_i8,  LD1SB_D, LD1SB_D_IMM, SVEAddrModeRegReg8>;
+  defm : pred_load<nxv2i64, nxv2i1, zext_masked_load_i16,  LD1H_D,  LD1H_D_IMM,  SVEAddrModeRegReg16>;
+  defm : pred_load<nxv2i64, nxv2i1, asext_masked_load_i16, LD1SH_D, LD1SH_D_IMM, SVEAddrModeRegReg16>;
+  defm : pred_load<nxv2i64, nxv2i1, zext_masked_load_i32,  LD1W_D,  LD1W_D_IMM,  SVEAddrModeRegReg32>;
+  defm : pred_load<nxv2i64, nxv2i1, asext_masked_load_i32, LD1SW_D, LD1SW_D_IMM, SVEAddrModeRegReg32>;
+  defm : pred_load<nxv2i64, nxv2i1, nonext_masked_load,    LD1D,    LD1D_IMM,    SVEAddrModeRegReg64>;
+  defm : pred_load<nxv2f16, nxv2i1, nonext_masked_load,    LD1H_D,  LD1H_D_IMM,  SVEAddrModeRegReg16>;
+  defm : pred_load<nxv2f32, nxv2i1, nonext_masked_load,    LD1W_D,  LD1W_D_IMM,  SVEAddrModeRegReg32>;
+  defm : pred_load<nxv2f64, nxv2i1, nonext_masked_load,    LD1D,    LD1D_IMM,    SVEAddrModeRegReg64>;
 
   // 4-element contiguous loads
-  defm : pred_load<nxv4i32, nxv4i1, zext_masked_load_i8,   LD1B_S_IMM>;
-  defm : pred_load<nxv4i32, nxv4i1, asext_masked_load_i8,  LD1SB_S_IMM>;
-  defm : pred_load<nxv4i32, nxv4i1, zext_masked_load_i16,  LD1H_S_IMM>;
-  defm : pred_load<nxv4i32, nxv4i1, asext_masked_load_i16, LD1SH_S_IMM>;
-  defm : pred_load<nxv4i32, nxv4i1, nonext_masked_load,    LD1W_IMM>;
-  defm : pred_load<nxv4f16, nxv4i1, nonext_masked_load,    LD1H_S_IMM>;
-  defm : pred_load<nxv4f32, nxv4i1, nonext_masked_load,    LD1W_IMM>;
+  defm : pred_load<nxv4i32, nxv4i1, zext_masked_load_i8,   LD1B_S,  LD1B_S_IMM,  SVEAddrModeRegReg8>;
+  defm : pred_load<nxv4i32, nxv4i1, asext_masked_load_i8,  LD1SB_S, LD1SB_S_IMM, SVEAddrModeRegReg8>;
+  defm : pred_load<nxv4i32, nxv4i1, zext_masked_load_i16,  LD1H_S,  LD1H_S_IMM,  SVEAddrModeRegReg16>;
+  defm : pred_load<nxv4i32, nxv4i1, asext_masked_load_i16, LD1SH_S, LD1SH_S_IMM, SVEAddrModeRegReg16>;
+  defm : pred_load<nxv4i32, nxv4i1, nonext_masked_load,    LD1W,    LD1W_IMM,    SVEAddrModeRegReg32>;
+  defm : pred_load<nxv4f16, nxv4i1, nonext_masked_load,    LD1H_S,  LD1H_S_IMM,  SVEAddrModeRegReg16>;
+  defm : pred_load<nxv4f32, nxv4i1, nonext_masked_load,    LD1W,    LD1W_IMM,    SVEAddrModeRegReg32>;
 
   // 8-element contiguous loads
-  defm : pred_load<nxv8i16, nxv8i1, zext_masked_load_i8,  LD1B_H_IMM>;
-  defm : pred_load<nxv8i16, nxv8i1, asext_masked_load_i8, LD1SB_H_IMM>;
-  defm : pred_load<nxv8i16, nxv8i1, nonext_masked_load,   LD1H_IMM>;
-  defm : pred_load<nxv8f16, nxv8i1, nonext_masked_load,   LD1H_IMM>;
+  defm : pred_load<nxv8i16, nxv8i1, zext_masked_load_i8,  LD1B_H,  LD1B_H_IMM,  SVEAddrModeRegReg8>;
+  defm : pred_load<nxv8i16, nxv8i1, asext_masked_load_i8, LD1SB_H, LD1SB_H_IMM, SVEAddrModeRegReg8>;
+  defm : pred_load<nxv8i16, nxv8i1, nonext_masked_load,   LD1H,    LD1H_IMM,    SVEAddrModeRegReg16>;
+  defm : pred_load<nxv8f16, nxv8i1, nonext_masked_load,   LD1H,    LD1H_IMM,    SVEAddrModeRegReg16>;
 
   // 16-element contiguous loads
-  defm : pred_load<nxv16i8, nxv16i1, nonext_masked_load, LD1B_IMM>;
+  defm : pred_load<nxv16i8, nxv16i1, nonext_masked_load, LD1B, LD1B_IMM, SVEAddrModeRegReg8>;
 
   multiclass pred_store<ValueType Ty, ValueType PredTy, SDPatternOperator Store,
-                         Instruction RegImmInst> {
+                        Instruction RegRegInst, Instruction RegImmInst, ComplexPattern AddrCP> {
+    // reg + reg
+    let AddedComplexity = 1 in {
+      def _reg_reg : Pat<(Store (Ty ZPR:$vec), (AddrCP GPR64:$base, GPR64:$offset), (PredTy PPR:$gp)),
+                         (RegRegInst ZPR:$vec, PPR:$gp, GPR64:$base, GPR64:$offset)>;
+    }
+    // reg + imm
+    let AddedComplexity = 2 in {
+      def _reg_imm : Pat<(Store (Ty ZPR:$vec), (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), (PredTy PPR:$gp)),
+                         (RegImmInst ZPR:$vec, PPR:$gp, GPR64:$base, simm4s1:$offset)>;
+    }
     def _default : Pat<(Store (Ty ZPR:$vec), GPR64:$base, (PredTy PPR:$gp)),
                        (RegImmInst ZPR:$vec, PPR:$gp, GPR64:$base, (i64 0))>;
   }
 
   // 2-element contiguous stores
-  defm : pred_store<nxv2i64, nxv2i1, trunc_masked_store_i8,  ST1B_D_IMM>;
-  defm : pred_store<nxv2i64, nxv2i1, trunc_masked_store_i16, ST1H_D_IMM>;
-  defm : pred_store<nxv2i64, nxv2i1, trunc_masked_store_i32, ST1W_D_IMM>;
-  defm : pred_store<nxv2i64, nxv2i1, nontrunc_masked_store,  ST1D_IMM>;
-  defm : pred_store<nxv2f16, nxv2i1, nontrunc_masked_store,  ST1H_D_IMM>;
-  defm : pred_store<nxv2f32, nxv2i1, nontrunc_masked_store,  ST1W_D_IMM>;
-  defm : pred_store<nxv2f64, nxv2i1, nontrunc_masked_store,  ST1D_IMM>;
+  defm : pred_store<nxv2i64, nxv2i1, trunc_masked_store_i8,  ST1B_D, ST1B_D_IMM, SVEAddrModeRegReg8>;
+  defm : pred_store<nxv2i64, nxv2i1, trunc_masked_store_i16, ST1H_D, ST1H_D_IMM, SVEAddrModeRegReg16>;
+  defm : pred_store<nxv2i64, nxv2i1, trunc_masked_store_i32, ST1W_D, ST1W_D_IMM, SVEAddrModeRegReg32>;
+  defm : pred_store<nxv2i64, nxv2i1, nontrunc_masked_store,  ST1D,   ST1D_IMM,   SVEAddrModeRegReg64>;
+  defm : pred_store<nxv2f16, nxv2i1, nontrunc_masked_store,  ST1H_D, ST1H_D_IMM, SVEAddrModeRegReg16>;
+  defm : pred_store<nxv2f32, nxv2i1, nontrunc_masked_store,  ST1W_D, ST1W_D_IMM, SVEAddrModeRegReg32>;
+  defm : pred_store<nxv2f64, nxv2i1, nontrunc_masked_store,  ST1D,   ST1D_IMM,   SVEAddrModeRegReg64>;
 
   // 4-element contiguous stores
-  defm : pred_store<nxv4i32, nxv4i1, trunc_masked_store_i8,  ST1B_S_IMM>;
-  defm : pred_store<nxv4i32, nxv4i1, trunc_masked_store_i16, ST1H_S_IMM>;
-  defm : pred_store<nxv4i32, nxv4i1, nontrunc_masked_store,  ST1W_IMM>;
-  defm : pred_store<nxv4f16, nxv4i1, nontrunc_masked_store,  ST1H_S_IMM>;
-  defm : pred_store<nxv4f32, nxv4i1, nontrunc_masked_store,  ST1W_IMM>;
+  defm : pred_store<nxv4i32, nxv4i1, trunc_masked_store_i8,  ST1B_S, ST1B_S_IMM, SVEAddrModeRegReg8>;
+  defm : pred_store<nxv4i32, nxv4i1, trunc_masked_store_i16, ST1H_S, ST1H_S_IMM, SVEAddrModeRegReg16>;
+  defm : pred_store<nxv4i32, nxv4i1, nontrunc_masked_store,  ST1W,   ST1W_IMM,   SVEAddrModeRegReg32>;
+  defm : pred_store<nxv4f16, nxv4i1, nontrunc_masked_store,  ST1H_S, ST1H_S_IMM, SVEAddrModeRegReg16>;
+  defm : pred_store<nxv4f32, nxv4i1, nontrunc_masked_store,  ST1W,   ST1W_IMM,   SVEAddrModeRegReg32>;
 
   // 8-element contiguous stores
-  defm : pred_store<nxv8i16, nxv8i1, trunc_masked_store_i8, ST1B_H_IMM>;
-  defm : pred_store<nxv8i16, nxv8i1, nontrunc_masked_store, ST1H_IMM>;
-  defm : pred_store<nxv8f16, nxv8i1, nontrunc_masked_store, ST1H_IMM>;
+  defm : pred_store<nxv8i16, nxv8i1, trunc_masked_store_i8, ST1B_H, ST1B_H_IMM, SVEAddrModeRegReg8>;
+  defm : pred_store<nxv8i16, nxv8i1, nontrunc_masked_store, ST1H,   ST1H_IMM,   SVEAddrModeRegReg16>;
+  defm : pred_store<nxv8f16, nxv8i1, nontrunc_masked_store, ST1H,   ST1H_IMM,   SVEAddrModeRegReg16>;
 
   // 16-element contiguous stores
-  defm : pred_store<nxv16i8, nxv16i1, nontrunc_masked_store, ST1B_IMM>;
+  defm : pred_store<nxv16i8, nxv16i1, nontrunc_masked_store, ST1B, ST1B_IMM, SVEAddrModeRegReg8>;
 
-  defm : pred_load<nxv16i8, nxv16i1, non_temporal_load, LDNT1B_ZRI>;
-  defm : pred_load<nxv8i16, nxv8i1,  non_temporal_load, LDNT1H_ZRI>;
-  defm : pred_load<nxv4i32, nxv4i1,  non_temporal_load, LDNT1W_ZRI>;
-  defm : pred_load<nxv2i64, nxv2i1,  non_temporal_load, LDNT1D_ZRI>;
-
-  defm : pred_store<nxv16i8, nxv16i1, non_temporal_store, STNT1B_ZRI>;
-  defm : pred_store<nxv8i16, nxv8i1,  non_temporal_store, STNT1H_ZRI>;
-  defm : pred_store<nxv4i32, nxv4i1,  non_temporal_store, STNT1W_ZRI>;
-  defm : pred_store<nxv2i64, nxv2i1,  non_temporal_store, STNT1D_ZRI>;
+  defm : pred_load<nxv16i8, nxv16i1, non_temporal_load, LDNT1B_ZRR, LDNT1B_ZRI, SVEAddrModeRegReg8>;
+  defm : pred_load<nxv8i16, nxv8i1,  non_temporal_load, LDNT1H_ZRR, LDNT1H_ZRI, SVEAddrModeRegReg16>;
+  defm : pred_load<nxv4i32, nxv4i1,  non_temporal_load, LDNT1W_ZRR, LDNT1W_ZRI, SVEAddrModeRegReg32>;
+  defm : pred_load<nxv2i64, nxv2i1,  non_temporal_load, LDNT1D_ZRR, LDNT1D_ZRI, SVEAddrModeRegReg64>;
+
+  defm : pred_store<nxv16i8, nxv16i1, non_temporal_store, STNT1B_ZRR, STNT1B_ZRI, SVEAddrModeRegReg8>;
+  defm : pred_store<nxv8i16, nxv8i1,  non_temporal_store, STNT1H_ZRR, STNT1H_ZRI, SVEAddrModeRegReg16>;
+  defm : pred_store<nxv4i32, nxv4i1,  non_temporal_store, STNT1W_ZRR, STNT1W_ZRI, SVEAddrModeRegReg32>;
+  defm : pred_store<nxv2i64, nxv2i1,  non_temporal_store, STNT1D_ZRR, STNT1D_ZRI, SVEAddrModeRegReg64>;
 
   multiclass unpred_store<ValueType Ty, Instruction RegImmInst, Instruction PTrue> {
     def _fi : Pat<(store (Ty ZPR:$val), (am_sve_fi GPR64sp:$base, simm4s1:$offset)),
                        (RegImmInst ZPR:$val, (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    let AddedComplexity = 2 in {
+      def _reg_imm : Pat<(store(Ty ZPR:$val),  (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset)),
+                         (RegImmInst ZPR:$val, (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    }
   }
 
   defm Pat_ST1B        : unpred_store<nxv16i8, ST1B_IMM, PTRUE_B>;
   defm Pat_ST1H        : unpred_store<nxv8i16, ST1H_IMM, PTRUE_H>;
   defm Pat_ST1W        : unpred_store<nxv4i32, ST1W_IMM, PTRUE_S>;
   defm Pat_ST1D        : unpred_store<nxv2i64, ST1D_IMM, PTRUE_D>;
   defm Pat_ST1H_float16: unpred_store<nxv8f16, ST1H_IMM, PTRUE_H>;
   defm Pat_ST1W_float  : unpred_store<nxv4f32, ST1W_IMM, PTRUE_S>;
   defm Pat_ST1D_double : unpred_store<nxv2f64, ST1D_IMM, PTRUE_D>;
 
   multiclass unpred_load<ValueType Ty, Instruction RegImmInst, Instruction PTrue> {
     def _fi : Pat<(Ty (load  (am_sve_fi GPR64sp:$base, simm4s1:$offset))),
                   (RegImmInst (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    let AddedComplexity = 2 in {
+      def _reg_imm : Pat<(Ty (load  (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset))),
+                         (RegImmInst (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    }
   }
 
   defm Pat_LD1B        : unpred_load<nxv16i8, LD1B_IMM, PTRUE_B>;
   defm Pat_LD1H        : unpred_load<nxv8i16, LD1H_IMM, PTRUE_H>;
   defm Pat_LD1W        : unpred_load<nxv4i32, LD1W_IMM, PTRUE_S>;
   defm Pat_LD1D        : unpred_load<nxv2i64, LD1D_IMM, PTRUE_D>;
   defm Pat_LD1H_float16: unpred_load<nxv8f16, LD1H_IMM, PTRUE_H>;
   defm Pat_LD1W_float  : unpred_load<nxv4f32, LD1W_IMM, PTRUE_S>;

llvm/lib/Target/AArch64/SVEInstrFormats.td

: I<(outs ZPR8:$Zdn), (ins ZPR8:$_Zdn),
   bits<5> Zdn;
   let Inst{31-11} = 0b010001010010000011100;
   let Inst{10}    = opc;
   let Inst{9-5}   = 0b00000;
   let Inst{4-0}   = Zdn;
 
   let Constraints = "$Zdn = $_Zdn";
 }
+
+/// Addressing modes
+def am_sve_indexed_s4 :ComplexPattern<i64, 2, "SelectAddrModeIndexedSVE<-8,7>", [], [SDNPWantRoot]>;
+
+def SVEAddrModeRegReg8  : ComplexPattern<i64, 2, "SelectSVERegRegAddrMode<0>", []>;
+def SVEAddrModeRegReg16 : ComplexPattern<i64, 2, "SelectSVERegRegAddrMode<1>", []>;
+def SVEAddrModeRegReg32 : ComplexPattern<i64, 2, "SelectSVERegRegAddrMode<2>", []>;
+def SVEAddrModeRegReg64 : ComplexPattern<i64, 2, "SelectSVERegRegAddrMode<3>", []>;

llvm/test/CodeGen/AArch64/sve-gep.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
 
 define <vscale x 2 x i64>* @scalar_of_scalable_1(<vscale x 2 x i64>* %base) {
 ; CHECK-LABEL: scalar_of_scalable_1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    rdvl x8, #1
-; CHECK-NEXT:    add x0, x0, x8, lsl #2
+; CHECK-NEXT:    rdvl x8, #4
+; CHECK-NEXT:    add x0, x0, x8
 ; CHECK-NEXT:    ret
   %d = getelementptr <vscale x 2 x i64>, <vscale x 2 x i64>* %base, i64 4
   ret <vscale x 2 x i64>* %d
 }
 
 define <vscale x 2 x i64>* @scalar_of_scalable_2(<vscale x 2 x i64>* %base, i64 %offset) {
 ; CHECK-LABEL: scalar_of_scalable_2:
 ; CHECK:       // %bb.0:

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg+imm.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+%sv2i1  = type <vscale x 2 x i1>
+%sv2i8  = type <vscale x 2 x i8>
+%sv2i16 = type <vscale x 2 x i16>
+%sv2i32 = type <vscale x 2 x i32>
+%sv2i64 = type <vscale x 2 x i64>
+%sv2f16 = type <vscale x 2 x half>
+%sv2f32 = type <vscale x 2 x float>
+%sv2f64 = type <vscale x 2 x double>
+
+%sv4i1  = type <vscale x 4 x i1>
+%sv4i8  = type <vscale x 4 x i8>
+%sv4i16 = type <vscale x 4 x i16>
+%sv4i32 = type <vscale x 4 x i32>
+%sv4f16 = type <vscale x 4 x half>
+%sv4f32 = type <vscale x 4 x float>
+
+%sv8i1  = type <vscale x 8 x i1>
+%sv8i8  = type <vscale x 8 x i8>
+%sv8i16 = type <vscale x 8 x i16>
+%sv8f16 = type <vscale x 8 x half>
+
+%sv16i1  = type <vscale x 16 x i1>
+%sv16i8  = type <vscale x 16 x i8>
+
+; 2-element contiguous loads.
+declare %sv2i8  @llvm.masked.load.nxv2i8 (%sv2i8* , i32, %sv2i1, %sv2i8 )
+declare %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16*, i32, %sv2i1, %sv2i16)
+declare %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32*, i32, %sv2i1, %sv2i32)
+declare %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64*, i32, %sv2i1, %sv2i64)
+declare %sv2f16 @llvm.masked.load.nxv2f16(%sv2f16*, i32, %sv2i1, %sv2f16)
+declare %sv2f32 @llvm.masked.load.nxv2f32(%sv2f32*, i32, %sv2i1, %sv2f32)
+declare %sv2f64 @llvm.masked.load.nxv2f64(%sv2f64*, i32, %sv2i1, %sv2f64)
+
+; 4-element contiguous loads.
+declare %sv4i8  @llvm.masked.load.nxv4i8 (%sv4i8* , i32, %sv4i1, %sv4i8 )
+declare %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16*, i32, %sv4i1, %sv4i16)
+declare %sv4i32 @llvm.masked.load.nxv4i32(%sv4i32*, i32, %sv4i1, %sv4i32)
+declare %sv4f16 @llvm.masked.load.nxv4f16(%sv4f16*, i32, %sv4i1, %sv4f16)
+declare %sv4f32 @llvm.masked.load.nxv4f32(%sv4f32*, i32, %sv4i1, %sv4f32)
+
+; 8-element contiguous loads.
+declare %sv8i8  @llvm.masked.load.nxv8i8 (%sv8i8* , i32, %sv8i1, %sv8i8 )
+declare %sv8i16 @llvm.masked.load.nxv8i16(%sv8i16*, i32, %sv8i1, %sv8i16)
+declare %sv8f16 @llvm.masked.load.nxv8f16(%sv8f16*, i32, %sv8i1, %sv8f16)
+
+; 16-element contiguous loads.
+declare %sv16i8 @llvm.masked.load.nxv16i8(%sv16i8*, i32, %sv16i1, %sv16i8)
+
+; 2-element contiguous stores.
+declare void @llvm.masked.store.nxv2i8 (%sv2i8 , %sv2i8* , i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i16(%sv2i16, %sv2i16*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i32(%sv2i32, %sv2i32*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i64(%sv2i64, %sv2i64*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f16(%sv2f16, %sv2f16*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f32(%sv2f32, %sv2f32*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f64(%sv2f64, %sv2f64*, i32, %sv2i1)
+
+; 4-element contiguous stores.
+declare void @llvm.masked.store.nxv4i8 (%sv4i8 , %sv4i8* , i32, %sv4i1)
+declare void @llvm.masked.store.nxv4i16(%sv4i16, %sv4i16*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4i32(%sv4i32, %sv4i32*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4f16(%sv4f16, %sv4f16*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4f32(%sv4f32, %sv4f32*, i32, %sv4i1)
+
+; 8-element contiguous stores.
+declare void @llvm.masked.store.nxv8i8 (%sv8i8 , %sv8i8* , i32, %sv8i1)
+declare void @llvm.masked.store.nxv8i16(%sv8i16, %sv8i16*, i32, %sv8i1)
+declare void @llvm.masked.store.nxv8f16(%sv8f16, %sv8f16*, i32, %sv8i1)
+
+; 16-element contiguous stores.
+declare void @llvm.masked.store.nxv16i8(%sv16i8, %sv16i8*, i32, %sv16i1)
+
+; Range checks: for all the instruction tested in this file, the
+; immediate must be within the range [-8, 7] (4-bit immediate). Out of
+; range values are tested only in one case (following). Valid values
+; are tested all through the rest of the file.
+
+define void @imm_out_of_range(%sv2i64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: imm_out_of_range:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x{{[0-9]+}}]
+; CHECK: st1d { z[[DATA]].d }, p0, [x{{[0-9]+}}]
+; CHECK: ret
+  %base_load = getelementptr %sv2i64, %sv2i64* %base, i64 8
+  %data = call %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64* %base_load, i32 1, %sv2i1 %mask, %sv2i64 undef)
+  %base_store = getelementptr %sv2i64, %sv2i64 * %base, i64 -9
+  call void @llvm.masked.store.nxv2i64(%sv2i64 %data, %sv2i64* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+; 2-lane contiguous load/stores
+
+define void @test_masked_ldst_sv2i8(%sv2i8 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1b { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 -8
+  %data = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_load, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %base_store = getelementptr %sv2i8, %sv2i8 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i8(%sv2i8 %data, %sv2i8* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i16(%sv2i16 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i16:
+; CHECK: ld1sh { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1h { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 -8
+  %data = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_load, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %base_store = getelementptr %sv2i16, %sv2i16 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i16(%sv2i16 %data, %sv2i16* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+
+define void @test_masked_ldst_sv2i32(%sv2i32 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i32:
+; CHECK: ld1sw { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1w { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 -8
+  %data = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_load, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %base_store = getelementptr %sv2i32, %sv2i32 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i32(%sv2i32 %data, %sv2i32* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i64(%sv2i64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i64:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1d { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i64, %sv2i64* %base, i64 -8
+  %data = call %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64* %base_load, i32 1, %sv2i1 %mask, %sv2i64 undef)
+  %base_store = getelementptr %sv2i64, %sv2i64 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i64(%sv2i64 %data, %sv2i64* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f16(%sv2f16 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1h { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f16, %sv2f16* %base, i64 -8
+  %data = call %sv2f16 @llvm.masked.load.nxv2f16(%sv2f16* %base_load, i32 1, %sv2i1 %mask, %sv2f16 undef)
+  %base_store = getelementptr %sv2f16, %sv2f16 * %base, i64 -7
+  call void @llvm.masked.store.nxv2f16(%sv2f16 %data, %sv2f16* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+
+define void @test_masked_ldst_sv2f32(%sv2f32 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1w { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f32, %sv2f32* %base, i64 -8
+  %data = call %sv2f32 @llvm.masked.load.nxv2f32(%sv2f32* %base_load, i32 1, %sv2i1 %mask, %sv2f32 undef)
+  %base_store = getelementptr %sv2f32, %sv2f32 * %base, i64 -7
+  call void @llvm.masked.store.nxv2f32(%sv2f32 %data, %sv2f32* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f64(%sv2f64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f64:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-6, mul vl]
+; CHECK: st1d { z[[DATA]].d }, p0, [x0, #-5, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f64, %sv2f64* %base, i64 -6
+  %data = call %sv2f64 @llvm.masked.load.nxv2f64(%sv2f64* %base_load, i32 1, %sv2i1 %mask, %sv2f64 undef)
+  %base_store = getelementptr %sv2f64, %sv2f64 * %base, i64 -5
+  call void @llvm.masked.store.nxv2f64(%sv2f64 %data, %sv2f64* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+; 2-lane zero/sign extended contiguous loads.
+
+define %sv2i64 @masked_zload_sv2i8_to_sv2i64(%sv2i8* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_zload_sv2i8_to_sv2i64:
+; CHECK: ld1b { z0.d }, p0/z, [x0, #-4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 -4
+  %load = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_load, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %ext = zext %sv2i8 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i8_to_sv2i64(%sv2i8* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_sload_sv2i8_to_sv2i64:
+; CHECK: ld1sb { z0.d }, p0/z, [x0, #-3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 -3
+  %load = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_load, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %ext = sext %sv2i8 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_zload_sv2i16_to_sv2i64(%sv2i16* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_zload_sv2i16_to_sv2i64:
+; CHECK: ld1h { z0.d }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 1
+  %load = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_load, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %ext = zext %sv2i16 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i16_to_sv2i64(%sv2i16* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_sload_sv2i16_to_sv2i64:
+; CHECK: ld1sh { z0.d }, p0/z, [x0, #2, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 2
+  %load = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_load, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %ext = sext %sv2i16 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_zload_sv2i32_to_sv2i64(%sv2i32* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_zload_sv2i32_to_sv2i64:
+; CHECK: ld1w { z0.d }, p0/z, [x0, #-2, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 -2
+  %load = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_load, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %ext = zext %sv2i32 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i32_to_sv2i64(%sv2i32* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_sload_sv2i32_to_sv2i64:
+; CHECK: ld1sw { z0.d }, p0/z, [x0, #-1, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 -1
+  %load = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_load, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %ext = sext %sv2i32 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+; 2-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv2i64_to_sv2i8(%sv2i64 %val, %sv2i8 *%base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i8:
+; CHECK: st1b { z0.d }, p0, [x0, #3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 3
+  %trunc = trunc %sv2i64 %val to %sv2i8
+  call void @llvm.masked.store.nxv2i8(%sv2i8 %trunc, %sv2i8 *%base_load, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+
+define void @masked_trunc_store_sv2i64_to_sv2i16(%sv2i64 %val, %sv2i16 *%base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i16:
+; CHECK: st1h { z0.d }, p0, [x0, #4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 4
+  %trunc = trunc %sv2i64 %val to %sv2i16
+  call void @llvm.masked.store.nxv2i16(%sv2i16 %trunc, %sv2i16 *%base_load, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+define void @masked_trunc_store_sv2i64_to_sv2i32(%sv2i64 %val, %sv2i32 *%base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i32:
+; CHECK: st1w { z0.d }, p0, [x0, #5, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 5
+  %trunc = trunc %sv2i64 %val to %sv2i32
+  call void @llvm.masked.store.nxv2i32(%sv2i32 %trunc, %sv2i32 *%base_load, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+; 4-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv4i8(%sv4i8 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1b { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 -1
+  %data = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_load, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %base_store = getelementptr %sv4i8, %sv4i8 * %base, i64 2
+  call void @llvm.masked.store.nxv4i8(%sv4i8 %data, %sv4i8* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4i16(%sv4i16 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i16:
+; CHECK: ld1sh { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1h { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 -1
+  %data = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_load, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %base_store = getelementptr %sv4i16, %sv4i16 * %base, i64 2
+  call void @llvm.masked.store.nxv4i16(%sv4i16 %data, %sv4i16* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4i32(%sv4i32 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, #6, mul vl]
+; CHECK: st1w { z[[DATA]].s }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i32, %sv4i32* %base, i64 6
+  %data = call %sv4i32 @llvm.masked.load.nxv4i32(%sv4i32* %base_load, i32 1, %sv4i1 %mask, %sv4i32 undef)
+  %base_store = getelementptr %sv4i32, %sv4i32 * %base, i64 7
+  call void @llvm.masked.store.nxv4i32(%sv4i32 %data, %sv4i32* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f16(%sv4f16 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1h { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4f16, %sv4f16* %base, i64 -1
+  %data = call %sv4f16 @llvm.masked.load.nxv4f16(%sv4f16* %base_load, i32 1, %sv4i1 %mask, %sv4f16 undef)
+  %base_store = getelementptr %sv4f16, %sv4f16 * %base, i64 2
+  call void @llvm.masked.store.nxv4f16(%sv4f16 %data, %sv4f16* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f32(%sv4f32 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4f32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1w { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4f32, %sv4f32* %base, i64 -1
+  %data = call %sv4f32 @llvm.masked.load.nxv4f32(%sv4f32* %base_load, i32 1, %sv4i1 %mask, %sv4f32 undef)
+  %base_store = getelementptr %sv4f32, %sv4f32 * %base, i64 2
+  call void @llvm.masked.store.nxv4f32(%sv4f32 %data, %sv4f32* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+; 4-lane zero/sign extended contiguous loads.
+
+define %sv4i32 @masked_zload_sv4i8_to_sv4i32(%sv4i8* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_zload_sv4i8_to_sv4i32:
+; CHECK: ld1b { z0.s }, p0/z, [x0, #-4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 -4
+  %load = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_load, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %ext = zext %sv4i8 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_sload_sv4i8_to_sv4i32(%sv4i8* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_sload_sv4i8_to_sv4i32:
+; CHECK: ld1sb { z0.s }, p0/z, [x0, #-3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 -3
+  %load = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_load, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %ext = sext %sv4i8 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_zload_sv4i16_to_sv4i32(%sv4i16* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_zload_sv4i16_to_sv4i32:
+; CHECK: ld1h { z0.s }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 1
+  %load = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_load, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %ext = zext %sv4i16 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_sload_sv4i16_to_sv4i32(%sv4i16* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_sload_sv4i16_to_sv4i32:
+; CHECK: ld1sh { z0.s }, p0/z, [x0, #2, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 2
+  %load = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_load, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %ext = sext %sv4i16 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+; 4-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv4i32_to_sv4i8(%sv4i32 %val, %sv4i8 *%base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv4i32_to_sv4i8:
+; CHECK: st1b { z0.s }, p0, [x0, #3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 3
+  %trunc = trunc %sv4i32 %val to %sv4i8
+  call void @llvm.masked.store.nxv4i8(%sv4i8 %trunc, %sv4i8 *%base_load, i32 8, %sv4i1 %mask)
+  ret void
+}
+
+
+define void @masked_trunc_store_sv4i32_to_sv4i16(%sv4i32 %val, %sv4i16 *%base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv4i32_to_sv4i16:
+; CHECK: st1h { z0.s }, p0, [x0, #4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 4
+  %trunc = trunc %sv4i32 %val to %sv4i16
+  call void @llvm.masked.store.nxv4i16(%sv4i16 %trunc, %sv4i16 *%base_load, i32 8, %sv4i1 %mask)
+  ret void
+}
+
+; 8-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv8i8(%sv8i8 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].h }, p0/z, [x0, #6, mul vl]
+; CHECK: st1b { z[[DATA]].h }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 6
+  %data = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_load, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %base_store = getelementptr %sv8i8, %sv8i8 * %base, i64 7
+  call void @llvm.masked.store.nxv8i8(%sv8i8 %data, %sv8i8* %base_store,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv8i16(%sv8i16 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8i16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, #6, mul vl]
+; CHECK: st1h { z[[DATA]].h }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8i16, %sv8i16* %base, i64 6
+  %data = call %sv8i16 @llvm.masked.load.nxv8i16(%sv8i16* %base_load, i32 1, %sv8i1 %mask, %sv8i16 undef)
+  %base_store = getelementptr %sv8i16, %sv8i16 * %base, i64 7
+  call void @llvm.masked.store.nxv8i16(%sv8i16 %data, %sv8i16* %base_store,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv8f16(%sv8f16 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1h { z[[DATA]].h }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8f16, %sv8f16* %base, i64 -1
+  %data = call %sv8f16 @llvm.masked.load.nxv8f16(%sv8f16* %base_load, i32 1, %sv8i1 %mask, %sv8f16 undef)
+  %base_store = getelementptr %sv8f16, %sv8f16 * %base, i64 2
+  call void @llvm.masked.store.nxv8f16(%sv8f16 %data, %sv8f16* %base_store,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+; 8-lane zero/sign extended contiguous loads.
+
+define %sv8i16 @masked_zload_sv8i8_to_sv8i16(%sv8i8* %base, %sv8i1 %mask) {
+; CHECK-LABEL: masked_zload_sv8i8_to_sv8i16:
+; CHECK: ld1b { z0.h }, p0/z, [x0, #-4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 -4
+  %load = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_load, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %ext = zext %sv8i8 %load to %sv8i16
+  ret %sv8i16 %ext
+}
+
+define %sv8i16 @masked_sload_sv8i8_to_sv8i16(%sv8i8* %base, %sv8i1 %mask) {
+; CHECK-LABEL: masked_sload_sv8i8_to_sv8i16:
+; CHECK: ld1sb { z0.h }, p0/z, [x0, #-3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 -3
+  %load = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_load, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %ext = sext %sv8i8 %load to %sv8i16
+  ret %sv8i16 %ext
+}
+
+; 8-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv8i16_to_sv8i8(%sv8i16 %val, %sv8i8 *%base, %sv8i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv8i16_to_sv8i8:
+; CHECK: st1b { z0.h }, p0, [x0, #3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 3
+  %trunc = trunc %sv8i16 %val to %sv8i8
+  call void @llvm.masked.store.nxv8i8(%sv8i8 %trunc, %sv8i8 *%base_load, i32 8, %sv8i1 %mask)
+  ret void
+}
+
+; 16-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv16i8(%sv16i8 * %base, %sv16i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv16i8:
+; CHECK: ld1b { z[[DATA:[0-9]+]].b }, p0/z, [x0, #6, mul vl]
+; CHECK: st1b { z[[DATA]].b }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv16i8, %sv16i8* %base, i64 6
+  %data = call %sv16i8 @llvm.masked.load.nxv16i8(%sv16i8* %base_load, i32 1, %sv16i1 %mask, %sv16i8 undef)
+  %base_store = getelementptr %sv16i8, %sv16i8 * %base, i64 7
+  call void @llvm.masked.store.nxv16i8(%sv16i8 %data, %sv16i8* %base_store,  i32 1, %sv16i1 %mask)
+  ret void
+}

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg+reg.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+%sv2i1  = type <vscale x 2 x i1>
+%sv2i8  = type <vscale x 2 x i8>
+%sv2i16 = type <vscale x 2 x i16>
+%sv2i32 = type <vscale x 2 x i32>
+%sv2i64 = type <vscale x 2 x i64>
+%sv2f16 = type <vscale x 2 x half>
+%sv2f32 = type <vscale x 2 x float>
+%sv2f64 = type <vscale x 2 x double>
+
+%sv4i1  = type <vscale x 4 x i1>
+%sv4i8  = type <vscale x 4 x i8>
+%sv4i16 = type <vscale x 4 x i16>
+%sv4i32 = type <vscale x 4 x i32>
+%sv4f16 = type <vscale x 4 x half>
+%sv4f32 = type <vscale x 4 x float>
+
+%sv8i1  = type <vscale x 8 x i1>
+%sv8i8  = type <vscale x 8 x i8>
+%sv8i16 = type <vscale x 8 x i16>
+%sv8f16 = type <vscale x 8 x half>
+
+%sv16i1  = type <vscale x 16 x i1>
+%sv16i8  = type <vscale x 16 x i8>
+
+; 2-element contiguous loads.
+declare %sv2i8  @llvm.masked.load.nxv2i8 (%sv2i8* , i32, %sv2i1, %sv2i8 )
+declare %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16*, i32, %sv2i1, %sv2i16)
+declare %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32*, i32, %sv2i1, %sv2i32)
+declare %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64*, i32, %sv2i1, %sv2i64)
+declare %sv2f16 @llvm.masked.load.nxv2f16(%sv2f16*, i32, %sv2i1, %sv2f16)
+declare %sv2f32 @llvm.masked.load.nxv2f32(%sv2f32*, i32, %sv2i1, %sv2f32)
+declare %sv2f64 @llvm.masked.load.nxv2f64(%sv2f64*, i32, %sv2i1, %sv2f64)
+
+; 4-element contiguous loads.
+declare %sv4i8  @llvm.masked.load.nxv4i8 (%sv4i8* , i32, %sv4i1, %sv4i8 )
+declare %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16*, i32, %sv4i1, %sv4i16)
+declare %sv4i32 @llvm.masked.load.nxv4i32(%sv4i32*, i32, %sv4i1, %sv4i32)
+declare %sv4f16 @llvm.masked.load.nxv4f16(%sv4f16*, i32, %sv4i1, %sv4f16)
+declare %sv4f32 @llvm.masked.load.nxv4f32(%sv4f32*, i32, %sv4i1, %sv4f32)
+
+; 8-element contiguous loads.
+declare %sv8i8  @llvm.masked.load.nxv8i8 (%sv8i8* , i32, %sv8i1, %sv8i8 )
+declare %sv8i16 @llvm.masked.load.nxv8i16(%sv8i16*, i32, %sv8i1, %sv8i16)
+declare %sv8f16 @llvm.masked.load.nxv8f16(%sv8f16*, i32, %sv8i1, %sv8f16)
+
+; 16-element contiguous loads.
+declare %sv16i8 @llvm.masked.load.nxv16i8(%sv16i8*, i32, %sv16i1, %sv16i8)
+
+; 2-element contiguous stores.
+declare void @llvm.masked.store.nxv2i8 (%sv2i8 , %sv2i8* , i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i16(%sv2i16, %sv2i16*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i32(%sv2i32, %sv2i32*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i64(%sv2i64, %sv2i64*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f16(%sv2f16, %sv2f16*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f32(%sv2f32, %sv2f32*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f64(%sv2f64, %sv2f64*, i32, %sv2i1)
+
+; 4-element contiguous stores.
+declare void @llvm.masked.store.nxv4i8 (%sv4i8 , %sv4i8* , i32, %sv4i1)
+declare void @llvm.masked.store.nxv4i16(%sv4i16, %sv4i16*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4i32(%sv4i32, %sv4i32*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4f16(%sv4f16, %sv4f16*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4f32(%sv4f32, %sv4f32*, i32, %sv4i1)
+
+; 8-element contiguous stores.
+declare void @llvm.masked.store.nxv8i8 (%sv8i8 , %sv8i8* , i32, %sv8i1)
+declare void @llvm.masked.store.nxv8i16(%sv8i16, %sv8i16*, i32, %sv8i1)
+declare void @llvm.masked.store.nxv8f16(%sv8f16, %sv8f16*, i32, %sv8i1)
+
+; 16-element contiguous stores.
+declare void @llvm.masked.store.nxv16i8(%sv16i8, %sv16i8*, i32, %sv16i1)
+
+; 2-lane contiguous load/stores
+
+define void @test_masked_ldst_sv2i8(i8 * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1]
+; CHECK: st1b { z[[DATA]].d }, p0, [x0, x1]
+; CHECK: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv2i8*
+  %data = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_addr, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  call void @llvm.masked.store.nxv2i8(%sv2i8 %data, %sv2i8* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i16(i16 * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2i16:
+; CHECK: ld1sh { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1, lsl #1]
+; CHECK: st1h { z[[DATA]].d }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv2i16*
+  %data = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_addr, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  call void @llvm.masked.store.nxv2i16(%sv2i16 %data, %sv2i16* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i32(i32 * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2i32:
+; CHECK: ld1sw  { z0.d }, p0/z, [x0, x1, lsl #2]
+; CHECK: st1w  { z0.d }, p0, [x0, x1, lsl #2]
+; CHECK: ret
+  %base_i32 = getelementptr i32, i32* %base, i64 %offset
+  %base_addr = bitcast i32* %base_i32 to %sv2i32*
+  %data = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_addr, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  call void @llvm.masked.store.nxv2i32(%sv2i32 %data, %sv2i32* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i64(i64 * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2i64:
+; CHECK: ld1d  { z0.d }, p0/z, [x0, x1, lsl #3]
+; CHECK: st1d  { z0.d }, p0, [x0, x1, lsl #3]
+; CHECK: ret
+  %base_i64 = getelementptr i64, i64* %base, i64 %offset
+  %base_addr = bitcast i64* %base_i64 to %sv2i64*
+  %data = call %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64* %base_addr, i32 1, %sv2i1 %mask, %sv2i64 undef)
+  call void @llvm.masked.store.nxv2i64(%sv2i64 %data, %sv2i64* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f16(half * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1, lsl #1]
+; CHECK: st1h { z[[DATA]].d }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_half = getelementptr half, half* %base, i64 %offset
+  %base_addr = bitcast half* %base_half to %sv2f16* 
+  %data = call %sv2f16 @llvm.masked.load.nxv2f16(%sv2f16* %base_addr, i32 1, %sv2i1 %mask, %sv2f16 undef)
+  call void @llvm.masked.store.nxv2f16(%sv2f16 %data, %sv2f16* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+} 
+
+define void @test_masked_ldst_sv2f32(float * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2f32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1, lsl #2]
+; CHECK: st1w { z[[DATA]].d }, p0, [x0, x1, lsl #2]
+; CHECK: ret
+  %base_float = getelementptr float, float* %base, i64 %offset
+  %base_addr = bitcast float* %base_float to %sv2f32* 
+  %data = call %sv2f32 @llvm.masked.load.nxv2f32(%sv2f32* %base_addr, i32 1, %sv2i1 %mask, %sv2f32 undef)
+  call void @llvm.masked.store.nxv2f32(%sv2f32 %data, %sv2f32* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f64(double * %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2f64:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1, lsl #3]
+; CHECK: st1d { z[[DATA]].d }, p0, [x0, x1, lsl #3]
+; CHECK: ret
+  %base_double = getelementptr double, double* %base, i64 %offset
+  %base_addr = bitcast double* %base_double to %sv2f64* 
+  %data = call %sv2f64 @llvm.masked.load.nxv2f64(%sv2f64* %base_addr, i32 1, %sv2i1 %mask, %sv2f64 undef)
+  call void @llvm.masked.store.nxv2f64(%sv2f64 %data, %sv2f64* %base_addr,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+; 2-lane zero/sign extended contiguous loads.
+
+define %sv2i64 @masked_zload_sv2i8_to_sv2i64(i8* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_zload_sv2i8_to_sv2i64:
+; CHECK: ld1b { z0.d }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv2i8*
+  %load = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_addr, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %ext = zext %sv2i8 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i8_to_sv2i64(i8* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_sload_sv2i8_to_sv2i64:
+; CHECK: ld1sb { z0.d }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv2i8*
+  %load = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_addr, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %ext = sext %sv2i8 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_zload_sv2i16_to_sv2i64(i16* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_zload_sv2i16_to_sv2i64:
+; CHECK: ld1h { z0.d }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv2i16*
+  %load = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_addr, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %ext = zext %sv2i16 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i16_to_sv2i64(i16* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_sload_sv2i16_to_sv2i64:
+; CHECK: ld1sh { z0.d }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv2i16*
+  %load = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_addr, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %ext = sext %sv2i16 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+
+define %sv2i64 @masked_zload_sv2i32_to_sv2i64(i32* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_zload_sv2i32_to_sv2i64:
+; CHECK: ld1w { z0.d }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base_i32 = getelementptr i32, i32* %base, i64 %offset
+  %base_addr = bitcast i32* %base_i32 to %sv2i32*
+  %load = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_addr, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %ext = zext %sv2i32 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i32_to_sv2i64(i32* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_sload_sv2i32_to_sv2i64:
+; CHECK: ld1sw { z0.d }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base_i32 = getelementptr i32, i32* %base, i64 %offset
+  %base_addr = bitcast i32* %base_i32 to %sv2i32*
+  %load = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_addr, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %ext = sext %sv2i32 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+; 2-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv2i64_to_sv2i8(%sv2i64 %val, i8 *%base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i8:
+; CHECK: st1b { z0.d }, p0, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv2i8*
+  %trunc = trunc %sv2i64 %val to %sv2i8
+  call void @llvm.masked.store.nxv2i8(%sv2i8 %trunc, %sv2i8 *%base_addr, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+define void @masked_trunc_store_sv2i64_to_sv2i16(%sv2i64 %val, i16 *%base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i16:
+; CHECK: st1h { z0.d }, p0, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv2i16*
+  %trunc = trunc %sv2i64 %val to %sv2i16
+  call void @llvm.masked.store.nxv2i16(%sv2i16 %trunc, %sv2i16 *%base_addr, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+define void @masked_trunc_store_sv2i64_to_sv2i32(%sv2i64 %val, i32 *%base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i32:
+; CHECK: st1w { z0.d }, p0, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base_i32 = getelementptr i32, i32* %base, i64 %offset
+  %base_addr = bitcast i32* %base_i32 to %sv2i32*
+  %trunc = trunc %sv2i64 %val to %sv2i32
+  call void @llvm.masked.store.nxv2i32(%sv2i32 %trunc, %sv2i32 *%base_addr, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+; 4-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv4i8(i8 * %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1]
+; CHECK: st1b { z[[DATA]].s }, p0, [x0, x1]
+; CHECK: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv4i8*
+  %data = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_addr, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  call void @llvm.masked.store.nxv4i8(%sv4i8 %data, %sv4i8* %base_addr,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4i16(i16 * %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4i16:
+; CHECK: ld1sh { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1, lsl #1]
+; CHECK: st1h { z[[DATA]].s }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv4i16*
+  %data = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_addr, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  call void @llvm.masked.store.nxv4i16(%sv4i16 %data, %sv4i16* %base_addr,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4i32(i32 * %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4i32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1, lsl #2]
+; CHECK: st1w { z[[DATA]].s }, p0, [x0, x1, lsl #2]
+; CHECK: ret
+  %base_i32 = getelementptr i32, i32* %base, i64 %offset
+  %base_addr = bitcast i32* %base_i32 to %sv4i32*
+  %data = call %sv4i32 @llvm.masked.load.nxv4i32(%sv4i32* %base_addr, i32 1, %sv4i1 %mask, %sv4i32 undef)
+  call void @llvm.masked.store.nxv4i32(%sv4i32 %data, %sv4i32* %base_addr,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f16(half * %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1, lsl #1]
+; CHECK: st1h { z[[DATA]].s }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_f16 = getelementptr half, half* %base, i64 %offset
+  %base_addr = bitcast half* %base_f16 to %sv4f16*
+  %data = call %sv4f16 @llvm.masked.load.nxv4f16(%sv4f16* %base_addr, i32 1, %sv4i1 %mask, %sv4f16 undef)
+  call void @llvm.masked.store.nxv4f16(%sv4f16 %data, %sv4f16* %base_addr,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f32(float * %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4f32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1, lsl #2]
+; CHECK: st1w { z[[DATA]].s }, p0, [x0, x1, lsl #2]
+; CHECK: ret
+  %base_f32 = getelementptr float, float* %base, i64 %offset
+  %base_addr = bitcast float* %base_f32 to %sv4f32*
+  %data = call %sv4f32 @llvm.masked.load.nxv4f32(%sv4f32* %base_addr, i32 1, %sv4i1 %mask, %sv4f32 undef)
+  call void @llvm.masked.store.nxv4f32(%sv4f32 %data, %sv4f32* %base_addr,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+; 4-lane zero/sign extended contiguous loads.
+
+define %sv4i32 @masked_zload_sv4i8_to_sv4i32(i8* %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_zload_sv4i8_to_sv4i32:
+; CHECK: ld1b { z0.s }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv4i8*
+  %load = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_addr, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %ext = zext %sv4i8 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_sload_sv4i8_to_sv4i32(i8* %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_sload_sv4i8_to_sv4i32:
+; CHECK: ld1sb { z0.s }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv4i8*
+  %load = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_addr, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %ext = sext %sv4i8 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_zload_sv4i16_to_sv4i32(i16* %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_zload_sv4i16_to_sv4i32:
+; CHECK: ld1h { z0.s }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv4i16*
+  %load = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_addr, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %ext = zext %sv4i16 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_sload_sv4i16_to_sv4i32(i16* %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_sload_sv4i16_to_sv4i32:
+; CHECK: ld1sh { z0.s }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv4i16*
+  %load = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_addr, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %ext = sext %sv4i16 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+; 4-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv4i32_to_sv4i8(%sv4i32 %val, i8 *%base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_trunc_store_sv4i32_to_sv4i8:
+; CHECK: st1b { z0.s }, p0, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv4i8*
+  %trunc = trunc %sv4i32 %val to %sv4i8
+  call void @llvm.masked.store.nxv4i8(%sv4i8 %trunc, %sv4i8 *%base_addr, i32 8, %sv4i1 %mask)
+  ret void
+}
+
+define void @masked_trunc_store_sv4i32_to_sv4i16(%sv4i32 %val, i16 *%base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_trunc_store_sv4i32_to_sv4i16:
+; CHECK: st1h { z0.s }, p0, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv4i16*
+  %trunc = trunc %sv4i32 %val to %sv4i16
+  call void @llvm.masked.store.nxv4i16(%sv4i16 %trunc, %sv4i16 *%base_addr, i32 8, %sv4i1 %mask)
+  ret void
+}
+
+; 8-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv8i8(i8 * %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv8i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].h }, p0/z, [x0, x1]
+; CHECK: st1b { z[[DATA]].h }, p0, [x0, x1]
+; CHECK: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv8i8*
+  %data = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_addr, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  call void @llvm.masked.store.nxv8i8(%sv8i8 %data, %sv8i8* %base_addr,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv8i16(i16 * %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv8i16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, x1, lsl #1]
+; CHECK: st1h { z[[DATA]].h }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv8i16*
+  %data = call %sv8i16 @llvm.masked.load.nxv8i16(%sv8i16* %base_addr, i32 1, %sv8i1 %mask, %sv8i16 undef)
+  call void @llvm.masked.store.nxv8i16(%sv8i16 %data, %sv8i16* %base_addr,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv8f16(half * %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv8f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, x1, lsl #1]
+; CHECK: st1h { z[[DATA]].h }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_f16 = getelementptr half, half* %base, i64 %offset
+  %base_addr = bitcast half* %base_f16 to %sv8f16*
+  %data = call %sv8f16 @llvm.masked.load.nxv8f16(%sv8f16* %base_addr, i32 1, %sv8i1 %mask, %sv8f16 undef)
+  call void @llvm.masked.store.nxv8f16(%sv8f16 %data, %sv8f16* %base_addr,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+; 8-lane zero/sign extended contiguous loads.
+
+define %sv8i16 @masked_zload_sv8i8_to_sv8i16(i8* %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_zload_sv8i8_to_sv8i16:
+; CHECK: ld1b { z0.h }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv8i8*
+  %load = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_addr, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %ext = zext %sv8i8 %load to %sv8i16
+  ret %sv8i16 %ext
+}
+
+define %sv8i16 @masked_sload_sv8i8_to_sv8i16(i8* %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_sload_sv8i8_to_sv8i16:
+; CHECK: ld1sb { z0.h }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv8i8*
+  %load = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_addr, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %ext = sext %sv8i8 %load to %sv8i16
+  ret %sv8i16 %ext
+}
+
+; 8-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv8i16_to_sv8i8(%sv8i16 %val, i8 *%base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: masked_trunc_store_sv8i16_to_sv8i8:
+; C HECK: st1b { z0.h }, p0, [x0, x1]
+; C HECK-NEXT: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv8i8*
+  %trunc = trunc %sv8i16 %val to %sv8i8
+  call void @llvm.masked.store.nxv8i8(%sv8i8 %trunc, %sv8i8 *%base_addr, i32 8, %sv8i1 %mask)
+  ret void
+}
+
+; 16-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv16i8(i8 * %base, %sv16i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv16i8:
+; CHECK: ld1b { z[[DATA:[0-9]+]].b }, p0/z, [x0, x1]
+; CHECK: st1b { z[[DATA]].b }, p0, [x0, x1]
+; CHECK: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv16i8*
+  %data = call %sv16i8 @llvm.masked.load.nxv16i8(%sv16i8* %base_addr, i32 1, %sv16i1 %mask, %sv16i8 undef)
+  call void @llvm.masked.store.nxv16i8(%sv16i8 %data, %sv16i8* %base_addr,  i32 1, %sv16i1 %mask)
+  ret void
+}

llvm/test/CodeGen/AArch64/sve-pred-non-temporal-ldst-addressing-mode-reg+imm.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+%sv2i1  = type <vscale x 2 x i1>
+%sv2i64 = type <vscale x 2 x i64>
+%sv2f64 = type <vscale x 2 x double>
+
+%sv4i1  = type <vscale x 4 x i1>
+%sv4i32 = type <vscale x 4 x i32>
+%sv4f32 = type <vscale x 4 x float>
+
+%sv8i1  = type <vscale x 8 x i1>
+%sv8i16 = type <vscale x 8 x i16>
+%sv8f16 = type <vscale x 8 x half>
+
+%sv16i1  = type <vscale x 16 x i1>
+%sv16i8  = type <vscale x 16 x i8>
+
+; 2-element non-temporal loads.
+declare %sv2i64 @llvm.aarch64.sve.ldnt1.nxv2i64(%sv2i1, %sv2i64*)
+declare %sv2f64 @llvm.aarch64.sve.ldnt1.nxv2f64(%sv2i1, %sv2f64*)
+
+; 4-element non-temporal loads.
+declare %sv4i32 @llvm.aarch64.sve.ldnt1.nxv4i32(%sv4i1, %sv4i32*)
+declare %sv4f32 @llvm.aarch64.sve.ldnt1.nxv4f32(%sv4i1, %sv4f32*)
+
+; 8-element non-temporal loads.
+declare %sv8i16 @llvm.aarch64.sve.ldnt1.nxv8i16(%sv8i1, %sv8i16*)
+declare %sv8f16 @llvm.aarch64.sve.ldnt1.nxv8f16(%sv8i1, %sv8f16*)
+
+; 16-element non-temporal loads.
+declare %sv16i8 @llvm.aarch64.sve.ldnt1.nxv16i8(%sv16i1, %sv16i8*)
+
+; 2-element non-temporal stores.
+declare void @llvm.aarch64.sve.stnt1.nxv2i64(%sv2i64, %sv2i1, %sv2i64*)
+declare void @llvm.aarch64.sve.stnt1.nxv2f64(%sv2f64, %sv2i1, %sv2f64*)
+                                                                      
+; 4-element non-temporal stores.                                        
+declare void @llvm.aarch64.sve.stnt1.nxv4i32(%sv4i32, %sv4i1, %sv4i32*)
+declare void @llvm.aarch64.sve.stnt1.nxv4f32(%sv4f32, %sv4i1, %sv4f32*)
+                                                                      
+; 8-element non-temporal stores.                                        
+declare void @llvm.aarch64.sve.stnt1.nxv8i16(%sv8i16, %sv8i1, %sv8i16*)
+declare void @llvm.aarch64.sve.stnt1.nxv8f16(%sv8f16, %sv8i1, %sv8f16*)
+
+; 16-element non-temporal stores.
+declare void @llvm.aarch64.sve.stnt1.nxv16i8(%sv16i8, %sv16i1, %sv16i8*)
+
+; Range checks: for all the instruction tested in this file, the
+; immediate must be within the range [-8, 7] (4-bit immediate). Out of
+; range values are tested only in one case (following). Valid values
+; are tested all through the rest of the file.
+
+define void @imm_out_of_range(%sv2i64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: imm_out_of_range:
+; CHECK: ldnt1d { z[[DATA:[0-9]+]].d }, p0/z, [x{{[0-9]+}}]
+; CHECK: stnt1d { z[[DATA]].d }, p0, [x{{[0-9]+}}]
+; CHECK: ret
+  %base_load = getelementptr %sv2i64, %sv2i64* %base, i64 8
+  %data = call %sv2i64 @llvm.aarch64.sve.ldnt1.nxv2i64(%sv2i1 %mask, %sv2i64* %base_load)
+  %base_store = getelementptr %sv2i64, %sv2i64 * %base, i64 -9
+  call void @llvm.aarch64.sve.stnt1.nxv2i64(%sv2i64 %data, %sv2i1 %mask, %sv2i64* %base_store)
+  ret void
+}
+
+; 2-lane non-temporal load/stores
+
+
+define void @test_masked_ldst_sv2i64(%sv2i64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i64:
+; CHECK: ldnt1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: stnt1d { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i64, %sv2i64* %base, i64 -8
+  %data = call %sv2i64 @llvm.aarch64.sve.ldnt1.nxv2i64(%sv2i1 %mask, %sv2i64* %base_load)
+  %base_store = getelementptr %sv2i64, %sv2i64 * %base, i64 -7
+  call void @llvm.aarch64.sve.stnt1.nxv2i64(%sv2i64 %data, %sv2i1 %mask, %sv2i64* %base_store)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f64(%sv2f64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f64:
+; CHECK: ldnt1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-6, mul vl]
+; CHECK: stnt1d { z[[DATA]].d }, p0, [x0, #-5, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f64, %sv2f64* %base, i64 -6
+  %data = call %sv2f64 @llvm.aarch64.sve.ldnt1.nxv2f64(%sv2i1 %mask,%sv2f64* %base_load)
+  %base_store = getelementptr %sv2f64, %sv2f64 * %base, i64 -5
+  call void @llvm.aarch64.sve.stnt1.nxv2f64(%sv2f64 %data, %sv2i1 %mask, %sv2f64* %base_store)
+  ret void
+}
+
+; 4-lane non-temporal load/stores.
+
+define void @test_masked_ldst_sv4i32(%sv4i32 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i32:
+; CHECK: ldnt1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, #6, mul vl]
+; CHECK: stnt1w { z[[DATA]].s }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i32, %sv4i32* %base, i64 6
+  %data = call %sv4i32 @llvm.aarch64.sve.ldnt1.nxv4i32(%sv4i1 %mask, %sv4i32* %base_load)
+  %base_store = getelementptr %sv4i32, %sv4i32 * %base, i64 7
+  call void @llvm.aarch64.sve.stnt1.nxv4i32(%sv4i32 %data, %sv4i1 %mask, %sv4i32* %base_store)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f32(%sv4f32 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4f32:
+; CHECK: ldnt1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: stnt1w { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4f32, %sv4f32* %base, i64 -1
+  %data = call %sv4f32 @llvm.aarch64.sve.ldnt1.nxv4f32(%sv4i1 %mask, %sv4f32* %base_load)
+  %base_store = getelementptr %sv4f32, %sv4f32 * %base, i64 2
+  call void @llvm.aarch64.sve.stnt1.nxv4f32(%sv4f32 %data, %sv4i1 %mask, %sv4f32* %base_store)
+  ret void
+}
+
+
+; 8-lane non-temporal load/stores.
+
+define void @test_masked_ldst_sv8i16(%sv8i16 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8i16:
+; CHECK: ldnt1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, #6, mul vl]
+; CHECK: stnt1h { z[[DATA]].h }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8i16, %sv8i16* %base, i64 6
+  %data = call %sv8i16 @llvm.aarch64.sve.ldnt1.nxv8i16(%sv8i1 %mask, %sv8i16* %base_load)
+  %base_store = getelementptr %sv8i16, %sv8i16 * %base, i64 7
+  call void @llvm.aarch64.sve.stnt1.nxv8i16(%sv8i16 %data, %sv8i1 %mask, %sv8i16* %base_store)
+  ret void
+}
+
+define void @test_masked_ldst_sv8f16(%sv8f16 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8f16:
+; CHECK: ldnt1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, #-1, mul vl]
+; CHECK: stnt1h { z[[DATA]].h }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8f16, %sv8f16* %base, i64 -1
+  %data = call %sv8f16 @llvm.aarch64.sve.ldnt1.nxv8f16(%sv8i1 %mask, %sv8f16* %base_load)
+  %base_store = getelementptr %sv8f16, %sv8f16 * %base, i64 2
+  call void @llvm.aarch64.sve.stnt1.nxv8f16(%sv8f16 %data, %sv8i1 %mask, %sv8f16* %base_store)
+  ret void
+}
+
+; 16-lane non-temporal load/stores.
+
+define void @test_masked_ldst_sv16i8(%sv16i8 * %base, %sv16i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv16i8:
+; CHECK: ldnt1b { z[[DATA:[0-9]+]].b }, p0/z, [x0, #6, mul vl]
+; CHECK: stnt1b { z[[DATA]].b }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv16i8, %sv16i8* %base, i64 6
+  %data = call %sv16i8 @llvm.aarch64.sve.ldnt1.nxv16i8(%sv16i1 %mask, %sv16i8* %base_load)
+  %base_store = getelementptr %sv16i8, %sv16i8 * %base, i64 7
+  call void @llvm.aarch64.sve.stnt1.nxv16i8(%sv16i8 %data, %sv16i1 %mask, %sv16i8* %base_store)
+  ret void
+}

llvm/test/CodeGen/AArch64/sve-pred-non-temporal-ldst-addressing-mode-reg+reg.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+%sv2i1  = type <vscale x 2 x i1>
+%sv2i64 = type <vscale x 2 x i64>
+%sv2f64 = type <vscale x 2 x double>
+
+%sv4i1  = type <vscale x 4 x i1>
+%sv4i32 = type <vscale x 4 x i32>
+%sv4f32 = type <vscale x 4 x float>
+
+%sv8i1  = type <vscale x 8 x i1>
+%sv8i16 = type <vscale x 8 x i16>
+%sv8f16 = type <vscale x 8 x half>
+
+%sv16i1  = type <vscale x 16 x i1>
+%sv16i8  = type <vscale x 16 x i8>
+
+; 2-element contiguous loads.
+declare %sv2i64 @llvm.aarch64.sve.ldnt1.nxv2i64(%sv2i1, %sv2i64*)
+declare %sv2f64 @llvm.aarch64.sve.ldnt1.nxv2f64(%sv2i1, %sv2f64*)
+
+; 4-element contiguous loads.
+declare %sv4i32 @llvm.aarch64.sve.ldnt1.nxv4i32(%sv4i1, %sv4i32*)
+declare %sv4f32 @llvm.aarch64.sve.ldnt1.nxv4f32(%sv4i1, %sv4f32*)
+
+; 8-element contiguous loads.
+declare %sv8i16 @llvm.aarch64.sve.ldnt1.nxv8i16(%sv8i1, %sv8i16*)
+declare %sv8f16 @llvm.aarch64.sve.ldnt1.nxv8f16(%sv8i1, %sv8f16*)
+
+; 16-element contiguous loads.
+declare %sv16i8 @llvm.aarch64.sve.ldnt1.nxv16i8(%sv16i1, %sv16i8*)
+
+; 2-element contiguous stores.
+declare void @llvm.aarch64.sve.stnt1.nxv2i64(%sv2i64, %sv2i1, %sv2i64*)
+declare void @llvm.aarch64.sve.stnt1.nxv2f64(%sv2f64, %sv2i1, %sv2f64*)
+                                                                      
+; 4-element contiguous stores.                                        
+declare void @llvm.aarch64.sve.stnt1.nxv4i32(%sv4i32, %sv4i1, %sv4i32*)
+declare void @llvm.aarch64.sve.stnt1.nxv4f32(%sv4f32, %sv4i1, %sv4f32*)
+                                                                      
+; 8-element contiguous stores.                                        
+declare void @llvm.aarch64.sve.stnt1.nxv8i16(%sv8i16, %sv8i1, %sv8i16*)
+declare void @llvm.aarch64.sve.stnt1.nxv8f16(%sv8f16, %sv8i1, %sv8f16*)
+
+; 16-element contiguous stores.
+declare void @llvm.aarch64.sve.stnt1.nxv16i8(%sv16i8, %sv16i1, %sv16i8*)
+
+; 2-lane contiguous load/stores
+
+define void @test_masked_ldst_sv2i64(i64* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2i64:
+; CHECK: ldnt1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1, lsl #3]
+; CHECK: stnt1d { z[[DATA]].d }, p0, [x0, x1, lsl #3]
+; CHECK: ret
+  %base_i64 = getelementptr i64, i64* %base, i64 %offset
+  %base_addr = bitcast i64* %base_i64 to %sv2i64*
+  %data = call %sv2i64 @llvm.aarch64.sve.ldnt1.nxv2i64(%sv2i1 %mask, %sv2i64* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv2i64(%sv2i64 %data, %sv2i1 %mask, %sv2i64* %base_addr)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f64(double* %base, %sv2i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv2f64:
+; CHECK: ldnt1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, x1, lsl #3]
+; CHECK: stnt1d { z[[DATA]].d }, p0, [x0, x1, lsl #3]
+; CHECK: ret
+  %base_double = getelementptr double, double* %base, i64 %offset
+  %base_addr = bitcast double* %base_double to %sv2f64*
+  %data = call %sv2f64 @llvm.aarch64.sve.ldnt1.nxv2f64(%sv2i1 %mask,%sv2f64* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv2f64(%sv2f64 %data, %sv2i1 %mask, %sv2f64* %base_addr)
+  ret void
+}
+
+; 4-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv4i32(i32* %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4i32:
+; CHECK: ldnt1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1, lsl #2]
+; CHECK: stnt1w { z[[DATA]].s }, p0, [x0, x1, lsl #2]
+; CHECK: ret
+  %base_i32 = getelementptr i32, i32* %base, i64 %offset
+  %base_addr = bitcast i32* %base_i32 to %sv4i32*
+  %data = call %sv4i32 @llvm.aarch64.sve.ldnt1.nxv4i32(%sv4i1 %mask, %sv4i32* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv4i32(%sv4i32 %data, %sv4i1 %mask, %sv4i32* %base_addr)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f32(float* %base, %sv4i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv4f32:
+; CHECK: ldnt1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, x1, lsl #2]
+; CHECK: stnt1w { z[[DATA]].s }, p0, [x0, x1, lsl #2]
+; CHECK: ret
+  %base_float = getelementptr float, float* %base, i64 %offset
+  %base_addr = bitcast float* %base_float to %sv4f32*
+  %data = call %sv4f32 @llvm.aarch64.sve.ldnt1.nxv4f32(%sv4i1 %mask, %sv4f32* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv4f32(%sv4f32 %data, %sv4i1 %mask, %sv4f32* %base_addr)
+  ret void
+}
+
+
+; 8-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv8i16(i16* %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv8i16:
+; CHECK: ldnt1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, x1, lsl #1]
+; CHECK: stnt1h { z[[DATA]].h }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_i16 = getelementptr i16, i16* %base, i64 %offset
+  %base_addr = bitcast i16* %base_i16 to %sv8i16*
+  %data = call %sv8i16 @llvm.aarch64.sve.ldnt1.nxv8i16(%sv8i1 %mask, %sv8i16* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv8i16(%sv8i16 %data, %sv8i1 %mask, %sv8i16* %base_addr)
+  ret void
+}
+
+define void @test_masked_ldst_sv8f16(half* %base, %sv8i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv8f16:
+; CHECK: ldnt1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, x1, lsl #1]
+; CHECK: stnt1h { z[[DATA]].h }, p0, [x0, x1, lsl #1]
+; CHECK: ret
+  %base_half = getelementptr half, half* %base, i64 %offset
+  %base_addr = bitcast half* %base_half to %sv8f16*
+  %data = call %sv8f16 @llvm.aarch64.sve.ldnt1.nxv8f16(%sv8i1 %mask, %sv8f16* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv8f16(%sv8f16 %data, %sv8i1 %mask, %sv8f16* %base_addr)
+  ret void
+}
+
+; 16-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv16i8(i8* %base, %sv16i1 %mask, i64 %offset) {
+; CHECK-LABEL: test_masked_ldst_sv16i8:
+; CHECK: ldnt1b { z[[DATA:[0-9]+]].b }, p0/z, [x0, x1]
+; CHECK: stnt1b { z[[DATA]].b }, p0, [x0, x1]
+; CHECK: ret
+  %base_i8 = getelementptr i8, i8* %base, i64 %offset
+  %base_addr = bitcast i8* %base_i8 to %sv16i8*
+  %data = call %sv16i8 @llvm.aarch64.sve.ldnt1.nxv16i8(%sv16i1 %mask, %sv16i8* %base_addr)
+  call void @llvm.aarch64.sve.stnt1.nxv16i8(%sv16i8 %data, %sv16i1 %mask, %sv16i8* %base_addr)
+  ret void
+}