[llvm][aarch64] SVE addressing modes.

[fpetrogalli]

Summary

Added register + immediate and register + register addressing modes for the following intrinsics:

1. Masked load and stores:
   • Sign and zero extended load and truncated stores.
   • No extension or truncation.
2. Masked non-temporal load and store.

[fpetrogalli]

The patch now covers all the LLVM intrinsic listed in the summary:

1. Masked load and stores:
   • Sign and zero extended load and truncated stores.
   • No extension or truncation.
2. Non-temporal load and store.

[fpetrogalli]

Notice that this patch is based (and depends) on https://reviews.llvm.org/D73602.

[andwar]

Hi @fpetrogalli, thank you for working on this.

• IMHO every test functions (e.g.:test_masked_ldst_sv2i8 ) should either test contiguous load or store (i.e. only one thing at a time). That will help triaging potential bugs in the future and also would be consistent with other test files in this folder.
• [nit] AFAIK, + is never used in file names (e.g. sve-pred-non-temporal-ldst-addressing-mode-reg+reg.ll). I couldn't find any specific rule for that though.
• [nit] The idea of %sv2i1 = type <vscale x 2 x i1> is very neat, but I'm slightly worried that that will complicate searching through test files (we are quite familiar with <vscale x 2 x i1>, not so much with %sv2i1, so you might miss cases when using grep).
• Are the DAG Combine rules required for this patch? If so, could you add tests for them?

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4420

[nit] Wouldn't assert be more suitable?

4439

Base and OffImm are only only needed when building in Debug mode and running e.g. llc with -debug. Maybe move that inside LLVM_DEBUG?

4441

[nit] Match found is very generic (and you use it twice). Maybe sth more specific?

4454

Used only once (so no need for a dedicated variable)

4455

Not used.

4464

Hm, why? Maybe document this method like you did for SelectAddrModeIndexedSVE?

llvm/lib/Target/AArch64/SVEInstrFormats.td

6979

Inconsistent naming with the records that follow this one.

llvm/test/CodeGen/AArch64/sve-gep.ll

7 ↗

(On Diff #244061)

Unrelated changes.

[fpetrogalli]

@andwar , thank you for the review.

I'll update the patch asap.

IMHO every test functions (e.g.:test_masked_ldst_sv2i8 ) should either test contiguous load or store (i.e. only one thing at a time). That will help triaging potential bugs in the future and also would be consistent with other test files in this folder.

I see your point, but the tests that use merge and store at the same time are using exactly the same addressing modes, it is not that they are using something different. So if something fails in the addressing mode of the load, it fails in the addressing mode of the store. Having them merged together saves quite some typing, and has no disadvantages in term of unit testing.

[nit] AFAIK, + is never used in file names (e.g. sve-pred-non-temporal-ldst-addressing-mode-reg+reg.ll). I couldn't find any specific rule for that though.

Fair point, I'll remove the plus.

[nit] The idea of %sv2i1 = type <vscale x 2 x i1> is very neat, but I'm slightly worried that that will complicate searching through test files (we are quite familiar with <vscale x 2 x i1>, not so much with %sv2i1, so you might miss cases when using grep).

Very good point. The type redefinition saved me a lot of typing, but now that the tests are there it is better o remove it.

Are the DAG Combine rules required for this patch? If so, could you add tests for them?

They are necessary for the reg+imm tests. They are needed to make sure that the ADD node of the base address is always in the format (ADD %BASE (VSCALE CONST)). I'll see if I can add some unit tests specifically for the combiner changes.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

1304

This is to be removed.

1321

This will be removed too.

llvm/test/CodeGen/AArch64/sve-gep.ll

7 ↗

(On Diff #244061)

No, I had to fix this because of the DAG combine changes.

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4439

No, they are output parameters, passed by reference. They are not just for debug purpose.

4441

I will remove these debug messages, they are not very helpful.

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4464

Good catch, it is not needed.

llvm/lib/Target/AArch64/SVEInstrFormats.td

6979

Used the file convention (lower case, separated with _, starts with am).

[fpetrogalli]

Address code review from @andwar.

I have added the tests for the DAG Combine changes.

[fpetrogalli]

I have uploaded PDF renderings of the DAGS before and after the changes. I am not sure the tests for the combiner are explicit about the fact that the combiner is run on this pattern, but the graphs attached here show the effect of my changes on the code. I think that some of the patterns that are needed to show the changes in the output code are missing.

before.dag.combine_sub_vscale_i64-bdd206.dot.pdf22 KBDownload

after.dag.combine_sub_vscale_i64-7f807e.dot.pdf22 KBDownload

before.dag.combine_mul_vscale_i64-f4259d.dot.pdf21 KBDownload

before.dag.combine_add_vscale_i64-62fd59.dot.pdf20 KBDownload

after.dag.combine_mul_vscale_i64-975c2a.dot.pdf20 KBDownload

after.dag.combine_add_vscale_i64-35445c.dot.pdf20 KBDownload

[fpetrogalli]

This comment was removed by fpetrogalli.

[fpetrogalli]

Cosmetic changes in the comments of the non-temporal reg+reg test.

[andwar]

In D74254#1872363, @fpetrogalli wrote:

IMHO every test functions (e.g.:test_masked_ldst_sv2i8 ) should either test contiguous load or store (i.e. only one thing at a time). That will help triaging potential bugs in the future and also would be consistent with other test files in this folder.

I see your point, but the tests that use merge and store at the same time are using exactly the same addressing modes, it is not that they are using something different. So if something fails in the addressing mode of the load, it fails in the addressing mode of the store. Having them merged together saves quite some typing, and has no disadvantages in term of unit testing.

Ah, I've realised that you split the files per addressing modes. As the name of the patch would suggest :-) OK, keep it as it is.

Thank you for adding DAG diagrams - they are very helpful!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

2330 ↗

(On Diff #244273)

vscale * C1 and vscale * (C0 + C1)?

3260 ↗

(On Diff #244273)

I don't quite understand the benefit of this transformation. The selection dag before and after are almost identical.

3601 ↗

(On Diff #244273)

vscale * (C0 + C1)?

7730 ↗

(On Diff #244273)

vscale * (C0 * C1)?

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4447

Missing doxstring.

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

11 ↗

(On Diff #244273)

; CHECK-NEXT? Here and in the following examples.

16 ↗

(On Diff #244273)

Could you format this line (and similar lines elsewhere)? E.g.: https://github.com/llvm/llvm-project/blob/318d0ede572080f18d0106dbc354e11c88329a84/llvm/test/CodeGen/AArch64/sve-intrinsics-stores.ll#L11

Makes it easier to parse for humans :) And will be consistent with other files too!

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

6 ↗

(On Diff #244273)

vscale * C1? and vscale * (C0 + C1)?

9 ↗

(On Diff #244273)

Perhaps ; CHECK-NOT: add?

25 ↗

(On Diff #244273)

• vscale * (C0 * C1)?
• What's C0 and what is C1 in this example?

33 ↗

(On Diff #244273)

[nit] Align with the following line (missing space)

38 ↗

(On Diff #244273)

Is nounwind needed in these examples?

55 ↗

(On Diff #244273)

Perhaps ; CHECK-NOT: sub?

77 ↗

(On Diff #244273)

; CHECK-NOT: shl?

80 ↗

(On Diff #244273)

Hm, since it's 6 here, shouldn't line 77 be ; CHECK-NEXT: rdvl x0, #6?

[fpetrogalli]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

3260 ↗

(On Diff #244273)

It is needed to avoid the having to deal with SUB when computing the addressing mode.

3601 ↗

(On Diff #244273)

Almost: vscale * (C0 * C1).

7730 ↗

(On Diff #244273)

Almost: (vscale * (C0 << C1)).

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

16 ↗

(On Diff #244273)

I prefer keeping all parameters in one line.

The "consistency with other file" argument doesn't work, most of the tests for SVE uses the convention in this patch:

frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*)$" sve*.ll | grep -v "()" | wc -l
2076
frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*,$" sve*.ll | grep -v "()" | wc -l
1433

The result becomes even more unbalanced if you look at the totality of tests in the folder:

frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*)$" *.ll | grep -v "()" | wc -l
7260
frapet01@man-08:~/projects/upstream-clang/llvm-project/llvm/test/CodeGen/AArch64 (master)$ grep "@llvm.*,$" *.ll | grep -v "()" | wc -l
1435

I run grep on master, @ a062a3ed7fd82c277812d80fb83dc6f05b939a84, _not_ on my dev branch :).

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

38 ↗

(On Diff #244273)

Yes, to be able to use CHECK-NEXT after CHECK-LABEL

80 ↗

(On Diff #244273)

At IR level, it is %shl = 2^6 * VSCALE

At Assembly level, the output of RDVL is 2^4 * VSCALE

Hence, to compute %shl we need to multiply RDVL output by 2^2 -> #4 is correct.

Does that make sense? I have actually added it as a comment, but I have modified the code so that it produces rdvl #1.

[fpetrogalli]

@andwar, thank you for you review.

I have updated the code accordingly.

Thank you!

[fpetrogalli]

I have added a CHECK-NOT: mul that was missing from the DAG combiner tests.

[andwar]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

700

DELETEME

4436

[nit] As per https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code, this could be simplified:

if ((MulImm % MemWidthBytes) != 0)
  return false;
 
signed Offset = MulImm / MemWidthBytes;
if ((Offset < Min) || (Offset > Max)) 
  return false;

Base = N.getOperand(0);
OffImm = CurDAG->getTargetConstant(Offset, SDLoc(N), MVT::i64);
return true;

This way we have fewer levels of indentation.

4437

This operation mixes sizes: unsigned vs int64_t. It would be safer to be either:

• consistently explicit about the size of integers (int64_t and int32_t), or
• consistently implicit about the size of integers (unsigned, unsigned long)

4469

[nit] As per https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code, this could be simplified as:

if (RHS.getOpcode() != ISD::SHL) 
  return false;

const SDValue SRHS = RHS.getOperand(1);
auto *C = dyn_cast<ConstantSDNode>(SRHS);
if (nullptr == C) 
  return false;

const uint64_t Shift = C->getZExtValue();
if (Shift == Scale) {
   Base = LHS;
   Offset = RHS.getOperand(0);
    return true;
}

This way you we have fewer levels of indentation.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

1213

[nit] Indent consistently

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

16 ↗

(On Diff #244273)

I think that your comparison misses the context, and that's entirely my fault because I didn't make it clear. If you compare long lines only, the split is roughly 50/50.

I kindly asked for this update because these lines are wrapped by Phabricator (they don't fit on my screen and I don't see how to make Phabricator stop doing that). This makes reviewing on Phab a bit frustrating.

Btw, I think that awk would be more fitting here ;-)

#! /bin/evn bash

TEST_FILES=(
"llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll"
"llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-reg.ll"
"llvm/test/CodeGen/AArch64/sve-pred-non-temporal-ldst-addressing-mode-reg-imm.ll"
"llvm/test/CodeGen/AArch64/sve-pred-non-temporal-ldst-addressing-mode-reg-reg.ll")

for test_file in "${TEST_FILES[@]}"; do
awk -F',' '{
  if ($1 ~ /call/) {
    # Counter number of character for indentation
    for (i = 1; i <= length($0); i++) {
      if (substr($0, i, 1) == "(") {
        lenght_col = i
        break
      }
    }

    # Split function call - 2 args
    if (NF == 2) {
      printf("%s,\n %*s %s\n", $1, lenght_col - 3, "", $2)
    }

    # Split function call - 3 args
    if (NF == 3) {
      printf("%s,\n %*s %s,\n %*s %s\n", $1, lenght_col - 3, "", $2, lenght_col - 3, "", $3)
    }

    # Split function call - 4 args
    if (NF == 4) {
      printf("%s,\n %*s %s,\n %*s %s,\n %*s %s\n", $1, lenght_col - 3, "", $2, lenght_col - 3, "", $3, lenght_col - 3, "", $4)
    }

  } else {
    # Not a call, not reformatting
    print $0
  }
}' ${test_file} > temp.ll

mv temp.ll ${test_file}

done

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-reg.ll

391 ↗

(On Diff #244499)

FIXME

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

6 ↗

(On Diff #244273)

What about vscale * C1?

13 ↗

(On Diff #244499)

Could C0 and C1 be different than 1?

28 ↗

(On Diff #244499)

Did you mean C0 = 2? And multiplication by C0 seems to be missing - there's only multiplication by 32. Shouldn't the IR look like this:

%vscale = call i64 @llvm.vscale.i64()
%mul_by_16 = mul i64 %vscale, 16
%mul_by_2 = mul i64 %mul_by_16, 32
ret i64 %mul_by_2

[andwar]

Cheers for updating this @fpetrogalli ! My most recent comments are mostly nits.

[fpetrogalli]

Thank you @adwar. I have addressed your comments.

Your awk script was very useful.

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

16 ↗

(On Diff #244273)

You win!

llvm/test/CodeGen/AArch64/sve-vscale-combine.ll

13 ↗

(On Diff #244499)

Yes, but to get C0 and C1 different from 1 I'd have to use a mul instruction, which is already tested in the combine_mul_vscale_* tests.

I think it is enough to test with C0=C1=1.

28 ↗

(On Diff #244499)

When targeting SVE, @llvm.vscale.*() returns the number of 16-byte chunks of and SVE register. If I multiply it by 32, it means it is returning the number of 4-bit (half-byte) elements in the SVE register. RDVL returns the number of 8-bit lanes in the register, hence the number of 4-bit lanes is given by RDVL Xn, %2.

I have updated the comment setting C1 = 32.

[sdesmalen]

Thanks for creating this patch @fpetrogalli!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

2331 ↗

(On Diff #245180)

Does it make sense to submit these DAGCombine changes + all related tests in a separate patch?
This patch is a bit of three patches in one at the moment: DAGCombines, reg+reg addressing modes, reg+imm addressing modes.

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

224

nit: This one is missing a comment.

4438

Are you purposely using signed instead of int64_t here?

4439

nit: unnecessary brackets, can be if (Offset < Min || Offset > Max)

4471

nit: Why const?

[fpetrogalli]

I have extracted the DAGCombiner changes in a separate patch as
requested by @sdesmalen: https://reviews.llvm.org/D74782

[fpetrogalli]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

2331 ↗

(On Diff #245180)

I have extracted the DAGCombiner changes here: https://reviews.llvm.org/D74782

I think that the code for the addressing mode is simple enough that it can stay in a single patch.

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4438

Should be int64_t for consistency.

4471

I try to use const indiscriminately when I know that the variable is not supposed to change inside the scope. The same it is done in other places in this file (not everywhere). I didn't marry this, so if you want I can remove it. :)

[fpetrogalli]

Thank you @sdesmalen for your review!

Francesco

[andwar]

Thank you for all the updates @fpetrogalli ! LGTM

[sdesmalen]

LGTM!

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4473

if (auto *C = dyn_cast<ConstantSDNode>(ShiftRHS)) {
  if (ShiftAmount == C->getZExtValue()) {
    ...
    return true;
  }
}

return false;

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

250 ↗

(On Diff #245224)

nit: did you chose a different alignment here on purpose?

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4473

@sdesmalen, your version saves some lines, I'll apply it and then submit the patch. Thank you!

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

250 ↗

(On Diff #245224)

No - it is not on purpose. Do you want me to set everything to 1? I wouldn't bother, non of the code added in this patch care of the value of the alignment...

[fpetrogalli]

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg-imm.ll

250 ↗

(On Diff #245224)

Well, it was easier to change everything to i32 1 instead of doing another round of questions! :)

[fpetrogalli]

Update code as requested by @sdesmalen.

Thank you!

Francesco

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

       const auto *CI = dyn_cast_or_null<ConstantInt>(C);
       if (CI && CI->isZero())
         continue;
+
+      // Constant integer offsets on scalable vectors pointers.
+      VectorType *VecTy = dyn_cast<VectorType>(GTI.getIndexedType());
+      // VecTy->dump();
+      if (VecTy && VecTy->isScalable() && !IsVectorGEP) {
+        EVT PTy = TLI.getPointerTy(DAG.getDataLayout(), AS);
+
+        // If index is smaller or larger than intptr_t, truncate or extend it.
+        SDValue IdxN = getValue(Idx);
+        IdxN = DAG.getSExtOrTrunc(IdxN, dl, N.getValueType());
+
+        // Determine offset (in bits).
+        SDValue OffsVal;
+        if (CI) {
+          APInt MinSize = CI->getValue() * VecTy->getBitWidth();
+          OffsVal = DAG.getVScale(dl, PTy, MinSize);
+        } else {
+          SDValue VS =
+              DAG.getVScale(dl, PTy, /*MulImm=*/APInt(PTy.getSizeInBits(), 1));
+          OffsVal = DAG.getNode(ISD::MUL, dl, PTy, VS, IdxN);
+        }
+        N = DAG.getNode(ISD::ADD, dl, N.getValueType(), N, OffsVal);
+        continue;
+      }
+
       if (CI && !ElementScalable) {
         APInt Offs = ElementMul * CI->getValue().sextOrTrunc(IdxSize);
         LLVMContext &Context = *DAG.getContext();

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

   void SelectPostLoadLane(SDNode *N, unsigned NumVecs, unsigned Opc);
 
   bool SelectAddrModeFrameIndexSVE(SDValue N, SDValue &Base, SDValue &OffImm);
+  template <int64_t Min, int64_t Max>
+  bool SelectAddrModeIndexedSVE(SDNode *Root, SDValue N, SDValue &Base,
+                                SDValue &OffImm);
 
   void SelectStore(SDNode *N, unsigned NumVecs, unsigned Opc);
   void SelectPostStore(SDNode *N, unsigned NumVecs, unsigned Opc);

                                          CodeGenOpt::Level OptLevel) {
   return new AArch64DAGToDAGISel(TM, OptLevel);
 }
+
+/// SelectAddrModeIndexedSVE - Attempt selection of the addressing mode:
+/// Base + OffImm * sizeof(MemVT) for Min >= OffImm <= Max
+/// where Root is the memory access using N for its address.
+template <int64_t Min, int64_t Max>
+bool AArch64DAGToDAGISel::SelectAddrModeIndexedSVE(SDNode *Root, SDValue N,
+                                                   SDValue &Base,
+                                                   SDValue &OffImm) {
+  if (!isa<MemSDNode>(Root))
+    return false;
+
+  EVT MemVT = cast<MemSDNode>(Root)->getMemoryVT();
+
+  if (N.getOpcode() != ISD::ADD)
+    return false;
+
+  SDValue VScale = N.getOperand(1);
+  if (VScale.getOpcode() != ISD::VSCALE)
+    return false;
+
+  TypeSize TS = MemVT.getSizeInBits();
+  unsigned MemWidthBits = TS.getKnownMinSize();
+  int64_t MulImm = cast<ConstantSDNode>(VScale.getOperand(0))->getSExtValue();
+
+  if ((MulImm % MemWidthBits) == 0) {
+    signed Offset = MulImm / MemWidthBits;
+    if ((Offset >= Min) && (Offset <= Max)) {
+      Base = N.getOperand(0);
+      OffImm = CurDAG->getTargetConstant(Offset, SDLoc(N), MVT::i64);
+      LLVM_DEBUG(dbgs() << "Match found:\n"; dbgs() << "ROOT:\n"; Root->dumpr();
+                 dbgs() << "BASE:\n"; Base.dumpr(); dbgs() << "OFFSET:\n";
+                 OffImm.dumpr());
+      return true;
+    }
+  }
+
+  return false;
+}

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

   multiclass pred_load<ValueType Ty, ValueType PredTy, SDPatternOperator Load,
                         Instruction RegImmInst> {
 
+    // reg + imm
+    let AddedComplexity = 2 in {
+      def _reg_imm_z : Pat<(Ty (Load (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), (PredTy PPR:$gp), (SVEDup0Undef))),
+                           (RegImmInst PPR:$gp, GPR64:$base, simm4s1:$offset)>;
+    }
     def _default_z : Pat<(Ty (Load  GPR64:$base, (PredTy PPR:$gp), (SVEDup0Undef))),
                          (RegImmInst PPR:$gp, GPR64:$base, (i64 0))>;
   }

 
   multiclass pred_store<ValueType Ty, ValueType PredTy, SDPatternOperator Store,
                          Instruction RegImmInst> {
+    // reg + imm
+    let AddedComplexity = 2 in {
+      def _reg_imm : Pat<(Store (Ty ZPR:$vec), (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), (PredTy PPR:$gp)),
+                         (RegImmInst ZPR:$vec, PPR:$gp, GPR64:$base, simm4s1:$offset)>;
+    }
     def _default : Pat<(Store (Ty ZPR:$vec), GPR64:$base, (PredTy PPR:$gp)),
                        (RegImmInst ZPR:$vec, PPR:$gp, GPR64:$base, (i64 0))>;
   }

   multiclass unpred_store<ValueType Ty, Instruction RegImmInst, Instruction PTrue> {
     def _fi : Pat<(store (Ty ZPR:$val), (am_sve_fi GPR64sp:$base, simm4s1:$offset)),
                        (RegImmInst ZPR:$val, (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    let AddedComplexity = 2 in {
+      def _reg_imm : Pat<(store(Ty ZPR:$val),  (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset)),
+                         (RegImmInst ZPR:$val, (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    }
   }
 
   defm Pat_ST1B        : unpred_store<nxv16i8, ST1B_IMM, PTRUE_B>;

   multiclass unpred_load<ValueType Ty, Instruction RegImmInst, Instruction PTrue> {
     def _fi : Pat<(Ty (load  (am_sve_fi GPR64sp:$base, simm4s1:$offset))),
                   (RegImmInst (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    let AddedComplexity = 2 in {
+      def _reg_imm : Pat<(Ty (load  (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset))),
+                         (RegImmInst (PTrue 31), GPR64sp:$base, simm4s1:$offset)>;
+    }
   }
 
   defm Pat_LD1B        : unpred_load<nxv16i8, LD1B_IMM, PTRUE_B>;

llvm/lib/Target/AArch64/SVEInstrFormats.td

 
   let Constraints = "$Zdn = $_Zdn";
 }
+
+/// Addressing modes
+def am_sve_indexed_s4 :ComplexPattern<i64, 2, "SelectAddrModeIndexedSVE<-8,7>", [], [SDNPWantRoot]>;
+

llvm/test/CodeGen/AArch64/sve-pred-contiguous-ldst-addressing-mode-reg+imm.ll

+; RUN: llc -mtriple=aarch64--linux-gnu -mattr=+sve < %s | FileCheck %s
+
+%sv2i1  = type <vscale x 2 x i1>
+%sv2i8  = type <vscale x 2 x i8>
+%sv2i16 = type <vscale x 2 x i16>
+%sv2i32 = type <vscale x 2 x i32>
+%sv2i64 = type <vscale x 2 x i64>
+%sv2f16 = type <vscale x 2 x half>
+%sv2f32 = type <vscale x 2 x float>
+%sv2f64 = type <vscale x 2 x double>
+
+%sv4i1  = type <vscale x 4 x i1>
+%sv4i8  = type <vscale x 4 x i8>
+%sv4i16 = type <vscale x 4 x i16>
+%sv4i32 = type <vscale x 4 x i32>
+%sv4f16 = type <vscale x 4 x half>
+%sv4f32 = type <vscale x 4 x float>
+
+%sv8i1  = type <vscale x 8 x i1>
+%sv8i8  = type <vscale x 8 x i8>
+%sv8i16 = type <vscale x 8 x i16>
+%sv8f16 = type <vscale x 8 x half>
+
+%sv16i1  = type <vscale x 16 x i1>
+%sv16i8  = type <vscale x 16 x i8>
+
+; 2-element contiguous loads.
+declare %sv2i8  @llvm.masked.load.nxv2i8 (%sv2i8* , i32, %sv2i1, %sv2i8 )
+declare %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16*, i32, %sv2i1, %sv2i16)
+declare %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32*, i32, %sv2i1, %sv2i32)
+declare %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64*, i32, %sv2i1, %sv2i64)
+declare %sv2f16 @llvm.masked.load.nxv2f16(%sv2f16*, i32, %sv2i1, %sv2f16)
+declare %sv2f32 @llvm.masked.load.nxv2f32(%sv2f32*, i32, %sv2i1, %sv2f32)
+declare %sv2f64 @llvm.masked.load.nxv2f64(%sv2f64*, i32, %sv2i1, %sv2f64)
+
+; 4-element contiguous loads.
+declare %sv4i8  @llvm.masked.load.nxv4i8 (%sv4i8* , i32, %sv4i1, %sv4i8 )
+declare %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16*, i32, %sv4i1, %sv4i16)
+declare %sv4i32 @llvm.masked.load.nxv4i32(%sv4i32*, i32, %sv4i1, %sv4i32)
+declare %sv4f16 @llvm.masked.load.nxv4f16(%sv4f16*, i32, %sv4i1, %sv4f16)
+declare %sv4f32 @llvm.masked.load.nxv4f32(%sv4f32*, i32, %sv4i1, %sv4f32)
+
+; 8-element contiguous loads.
+declare %sv8i8  @llvm.masked.load.nxv8i8 (%sv8i8* , i32, %sv8i1, %sv8i8 )
+declare %sv8i16 @llvm.masked.load.nxv8i16(%sv8i16*, i32, %sv8i1, %sv8i16)
+declare %sv8f16 @llvm.masked.load.nxv8f16(%sv8f16*, i32, %sv8i1, %sv8f16)
+
+; 16-element contiguous loads.
+declare %sv16i8 @llvm.masked.load.nxv16i8(%sv16i8*, i32, %sv16i1, %sv16i8)
+
+; 2-element contiguous stores.
+declare void @llvm.masked.store.nxv2i8 (%sv2i8 , %sv2i8* , i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i16(%sv2i16, %sv2i16*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i32(%sv2i32, %sv2i32*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2i64(%sv2i64, %sv2i64*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f16(%sv2f16, %sv2f16*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f32(%sv2f32, %sv2f32*, i32, %sv2i1)
+declare void @llvm.masked.store.nxv2f64(%sv2f64, %sv2f64*, i32, %sv2i1)
+
+; 4-element contiguous stores.
+declare void @llvm.masked.store.nxv4i8 (%sv4i8 , %sv4i8* , i32, %sv4i1)
+declare void @llvm.masked.store.nxv4i16(%sv4i16, %sv4i16*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4i32(%sv4i32, %sv4i32*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4f16(%sv4f16, %sv4f16*, i32, %sv4i1)
+declare void @llvm.masked.store.nxv4f32(%sv4f32, %sv4f32*, i32, %sv4i1)
+
+; 8-element contiguous stores.
+declare void @llvm.masked.store.nxv8i8 (%sv8i8 , %sv8i8* , i32, %sv8i1)
+declare void @llvm.masked.store.nxv8i16(%sv8i16, %sv8i16*, i32, %sv8i1)
+declare void @llvm.masked.store.nxv8f16(%sv8f16, %sv8f16*, i32, %sv8i1)
+
+; 16-element contiguous stores.
+declare void @llvm.masked.store.nxv16i8(%sv16i8, %sv16i8*, i32, %sv16i1)
+
+; Range checks: for all the instruction tested in this file, the
+; immediate must be within the range [-8, 7] (4-bit immediate). Out of
+; range values are tested only in one case (following). Valid values
+; are tested all through the rest of the file.
+
+define void @imm_out_of_range(%sv2i64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: imm_out_of_range:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x{{[0-9]+}}]
+; CHECK: st1d { z[[DATA]].d }, p0, [x{{[0-9]+}}]
+; CHECK: ret
+  %base_load = getelementptr %sv2i64, %sv2i64* %base, i64 8
+  %data = call %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64* %base_load, i32 1, %sv2i1 %mask, %sv2i64 undef)
+  %base_store = getelementptr %sv2i64, %sv2i64 * %base, i64 -9
+  call void @llvm.masked.store.nxv2i64(%sv2i64 %data, %sv2i64* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+; 2-lane contiguous load/stores
+
+define void @test_masked_ldst_sv2i8(%sv2i8 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1b { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 -8
+  %data = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_load, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %base_store = getelementptr %sv2i8, %sv2i8 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i8(%sv2i8 %data, %sv2i8* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i16(%sv2i16 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i16:
+; CHECK: ld1sh { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1h { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 -8
+  %data = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_load, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %base_store = getelementptr %sv2i16, %sv2i16 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i16(%sv2i16 %data, %sv2i16* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+
+define void @test_masked_ldst_sv2i32(%sv2i32 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i32:
+; CHECK: ld1sw { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1w { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 -8
+  %data = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_load, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %base_store = getelementptr %sv2i32, %sv2i32 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i32(%sv2i32 %data, %sv2i32* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2i64(%sv2i64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2i64:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1d { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2i64, %sv2i64* %base, i64 -8
+  %data = call %sv2i64 @llvm.masked.load.nxv2i64(%sv2i64* %base_load, i32 1, %sv2i1 %mask, %sv2i64 undef)
+  %base_store = getelementptr %sv2i64, %sv2i64 * %base, i64 -7
+  call void @llvm.masked.store.nxv2i64(%sv2i64 %data, %sv2i64* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f16(%sv2f16 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1h { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f16, %sv2f16* %base, i64 -8
+  %data = call %sv2f16 @llvm.masked.load.nxv2f16(%sv2f16* %base_load, i32 1, %sv2i1 %mask, %sv2f16 undef)
+  %base_store = getelementptr %sv2f16, %sv2f16 * %base, i64 -7
+  call void @llvm.masked.store.nxv2f16(%sv2f16 %data, %sv2f16* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+
+define void @test_masked_ldst_sv2f32(%sv2f32 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-8, mul vl]
+; CHECK: st1w { z[[DATA]].d }, p0, [x0, #-7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f32, %sv2f32* %base, i64 -8
+  %data = call %sv2f32 @llvm.masked.load.nxv2f32(%sv2f32* %base_load, i32 1, %sv2i1 %mask, %sv2f32 undef)
+  %base_store = getelementptr %sv2f32, %sv2f32 * %base, i64 -7
+  call void @llvm.masked.store.nxv2f32(%sv2f32 %data, %sv2f32* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv2f64(%sv2f64 * %base, %sv2i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv2f64:
+; CHECK: ld1d { z[[DATA:[0-9]+]].d }, p0/z, [x0, #-6, mul vl]
+; CHECK: st1d { z[[DATA]].d }, p0, [x0, #-5, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv2f64, %sv2f64* %base, i64 -6
+  %data = call %sv2f64 @llvm.masked.load.nxv2f64(%sv2f64* %base_load, i32 1, %sv2i1 %mask, %sv2f64 undef)
+  %base_store = getelementptr %sv2f64, %sv2f64 * %base, i64 -5
+  call void @llvm.masked.store.nxv2f64(%sv2f64 %data, %sv2f64* %base_store,  i32 1, %sv2i1 %mask)
+  ret void
+}
+
+; 2-lane zero/sign extended contiguous loads.
+
+define %sv2i64 @masked_zload_sv2i8_to_sv2i64(%sv2i8* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_zload_sv2i8_to_sv2i64:
+; CHECK: ld1b { z0.d }, p0/z, [x0, #-4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 -4
+  %load = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_load, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %ext = zext %sv2i8 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i8_to_sv2i64(%sv2i8* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_sload_sv2i8_to_sv2i64:
+; CHECK: ld1sb { z0.d }, p0/z, [x0, #-3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 -3
+  %load = call %sv2i8 @llvm.masked.load.nxv2i8(%sv2i8* %base_load, i32 1, %sv2i1 %mask, %sv2i8 undef)
+  %ext = sext %sv2i8 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_zload_sv2i16_to_sv2i64(%sv2i16* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_zload_sv2i16_to_sv2i64:
+; CHECK: ld1h { z0.d }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 1
+  %load = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_load, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %ext = zext %sv2i16 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i16_to_sv2i64(%sv2i16* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_sload_sv2i16_to_sv2i64:
+; CHECK: ld1sh { z0.d }, p0/z, [x0, #2, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 2
+  %load = call %sv2i16 @llvm.masked.load.nxv2i16(%sv2i16* %base_load, i32 1, %sv2i1 %mask, %sv2i16 undef)
+  %ext = sext %sv2i16 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_zload_sv2i32_to_sv2i64(%sv2i32* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_zload_sv2i32_to_sv2i64:
+; CHECK: ld1w { z0.d }, p0/z, [x0, #-2, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 -2
+  %load = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_load, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %ext = zext %sv2i32 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+define %sv2i64 @masked_sload_sv2i32_to_sv2i64(%sv2i32* %base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_sload_sv2i32_to_sv2i64:
+; CHECK: ld1sw { z0.d }, p0/z, [x0, #-1, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 -1
+  %load = call %sv2i32 @llvm.masked.load.nxv2i32(%sv2i32* %base_load, i32 1, %sv2i1 %mask, %sv2i32 undef)
+  %ext = sext %sv2i32 %load to %sv2i64
+  ret %sv2i64 %ext
+}
+
+; 2-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv2i64_to_sv2i8(%sv2i64 %val, %sv2i8 *%base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i8:
+; CHECK: st1b { z0.d }, p0, [x0, #3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i8, %sv2i8* %base, i64 3
+  %trunc = trunc %sv2i64 %val to %sv2i8
+  call void @llvm.masked.store.nxv2i8(%sv2i8 %trunc, %sv2i8 *%base_load, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+
+define void @masked_trunc_store_sv2i64_to_sv2i16(%sv2i64 %val, %sv2i16 *%base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i16:
+; CHECK: st1h { z0.d }, p0, [x0, #4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i16, %sv2i16* %base, i64 4
+  %trunc = trunc %sv2i64 %val to %sv2i16
+  call void @llvm.masked.store.nxv2i16(%sv2i16 %trunc, %sv2i16 *%base_load, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+define void @masked_trunc_store_sv2i64_to_sv2i32(%sv2i64 %val, %sv2i32 *%base, %sv2i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv2i64_to_sv2i32:
+; CHECK: st1w { z0.d }, p0, [x0, #5, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv2i32, %sv2i32* %base, i64 5
+  %trunc = trunc %sv2i64 %val to %sv2i32
+  call void @llvm.masked.store.nxv2i32(%sv2i32 %trunc, %sv2i32 *%base_load, i32 8, %sv2i1 %mask)
+  ret void
+}
+
+; 4-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv4i8(%sv4i8 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1b { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 -1
+  %data = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_load, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %base_store = getelementptr %sv4i8, %sv4i8 * %base, i64 2
+  call void @llvm.masked.store.nxv4i8(%sv4i8 %data, %sv4i8* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4i16(%sv4i16 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i16:
+; CHECK: ld1sh { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1h { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 -1
+  %data = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_load, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %base_store = getelementptr %sv4i16, %sv4i16 * %base, i64 2
+  call void @llvm.masked.store.nxv4i16(%sv4i16 %data, %sv4i16* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4i32(%sv4i32 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4i32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, #6, mul vl]
+; CHECK: st1w { z[[DATA]].s }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4i32, %sv4i32* %base, i64 6
+  %data = call %sv4i32 @llvm.masked.load.nxv4i32(%sv4i32* %base_load, i32 1, %sv4i1 %mask, %sv4i32 undef)
+  %base_store = getelementptr %sv4i32, %sv4i32 * %base, i64 7
+  call void @llvm.masked.store.nxv4i32(%sv4i32 %data, %sv4i32* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f16(%sv4f16 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1h { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4f16, %sv4f16* %base, i64 -1
+  %data = call %sv4f16 @llvm.masked.load.nxv4f16(%sv4f16* %base_load, i32 1, %sv4i1 %mask, %sv4f16 undef)
+  %base_store = getelementptr %sv4f16, %sv4f16 * %base, i64 2
+  call void @llvm.masked.store.nxv4f16(%sv4f16 %data, %sv4f16* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv4f32(%sv4f32 * %base, %sv4i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv4f32:
+; CHECK: ld1w { z[[DATA:[0-9]+]].s }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1w { z[[DATA]].s }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv4f32, %sv4f32* %base, i64 -1
+  %data = call %sv4f32 @llvm.masked.load.nxv4f32(%sv4f32* %base_load, i32 1, %sv4i1 %mask, %sv4f32 undef)
+  %base_store = getelementptr %sv4f32, %sv4f32 * %base, i64 2
+  call void @llvm.masked.store.nxv4f32(%sv4f32 %data, %sv4f32* %base_store,  i32 1, %sv4i1 %mask)
+  ret void
+}
+
+; 4-lane zero/sign extended contiguous loads.
+
+define %sv4i32 @masked_zload_sv4i8_to_sv4i32(%sv4i8* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_zload_sv4i8_to_sv4i32:
+; CHECK: ld1b { z0.s }, p0/z, [x0, #-4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 -4
+  %load = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_load, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %ext = zext %sv4i8 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_sload_sv4i8_to_sv4i32(%sv4i8* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_sload_sv4i8_to_sv4i32:
+; CHECK: ld1sb { z0.s }, p0/z, [x0, #-3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 -3
+  %load = call %sv4i8 @llvm.masked.load.nxv4i8(%sv4i8* %base_load, i32 1, %sv4i1 %mask, %sv4i8 undef)
+  %ext = sext %sv4i8 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_zload_sv4i16_to_sv4i32(%sv4i16* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_zload_sv4i16_to_sv4i32:
+; CHECK: ld1h { z0.s }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 1
+  %load = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_load, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %ext = zext %sv4i16 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+define %sv4i32 @masked_sload_sv4i16_to_sv4i32(%sv4i16* %base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_sload_sv4i16_to_sv4i32:
+; CHECK: ld1sh { z0.s }, p0/z, [x0, #2, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 2
+  %load = call %sv4i16 @llvm.masked.load.nxv4i16(%sv4i16* %base_load, i32 1, %sv4i1 %mask, %sv4i16 undef)
+  %ext = sext %sv4i16 %load to %sv4i32
+  ret %sv4i32 %ext
+}
+
+; 4-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv4i32_to_sv4i8(%sv4i32 %val, %sv4i8 *%base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv4i32_to_sv4i8:
+; CHECK: st1b { z0.s }, p0, [x0, #3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i8, %sv4i8* %base, i64 3
+  %trunc = trunc %sv4i32 %val to %sv4i8
+  call void @llvm.masked.store.nxv4i8(%sv4i8 %trunc, %sv4i8 *%base_load, i32 8, %sv4i1 %mask)
+  ret void
+}
+
+
+define void @masked_trunc_store_sv4i32_to_sv4i16(%sv4i32 %val, %sv4i16 *%base, %sv4i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv4i32_to_sv4i16:
+; CHECK: st1h { z0.s }, p0, [x0, #4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv4i16, %sv4i16* %base, i64 4
+  %trunc = trunc %sv4i32 %val to %sv4i16
+  call void @llvm.masked.store.nxv4i16(%sv4i16 %trunc, %sv4i16 *%base_load, i32 8, %sv4i1 %mask)
+  ret void
+}
+
+; 8-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv8i8(%sv8i8 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8i8:
+; CHECK: ld1sb { z[[DATA:[0-9]+]].h }, p0/z, [x0, #6, mul vl]
+; CHECK: st1b { z[[DATA]].h }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 6
+  %data = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_load, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %base_store = getelementptr %sv8i8, %sv8i8 * %base, i64 7
+  call void @llvm.masked.store.nxv8i8(%sv8i8 %data, %sv8i8* %base_store,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv8i16(%sv8i16 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8i16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, #6, mul vl]
+; CHECK: st1h { z[[DATA]].h }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8i16, %sv8i16* %base, i64 6
+  %data = call %sv8i16 @llvm.masked.load.nxv8i16(%sv8i16* %base_load, i32 1, %sv8i1 %mask, %sv8i16 undef)
+  %base_store = getelementptr %sv8i16, %sv8i16 * %base, i64 7
+  call void @llvm.masked.store.nxv8i16(%sv8i16 %data, %sv8i16* %base_store,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+define void @test_masked_ldst_sv8f16(%sv8f16 * %base, %sv8i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv8f16:
+; CHECK: ld1h { z[[DATA:[0-9]+]].h }, p0/z, [x0, #-1, mul vl]
+; CHECK: st1h { z[[DATA]].h }, p0, [x0, #2, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv8f16, %sv8f16* %base, i64 -1
+  %data = call %sv8f16 @llvm.masked.load.nxv8f16(%sv8f16* %base_load, i32 1, %sv8i1 %mask, %sv8f16 undef)
+  %base_store = getelementptr %sv8f16, %sv8f16 * %base, i64 2
+  call void @llvm.masked.store.nxv8f16(%sv8f16 %data, %sv8f16* %base_store,  i32 1, %sv8i1 %mask)
+  ret void
+}
+
+; 8-lane zero/sign extended contiguous loads.
+
+define %sv8i16 @masked_zload_sv8i8_to_sv8i16(%sv8i8* %base, %sv8i1 %mask) {
+; CHECK-LABEL: masked_zload_sv8i8_to_sv8i16:
+; CHECK: ld1b { z0.h }, p0/z, [x0, #-4, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 -4
+  %load = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_load, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %ext = zext %sv8i8 %load to %sv8i16
+  ret %sv8i16 %ext
+}
+
+define %sv8i16 @masked_sload_sv8i8_to_sv8i16(%sv8i8* %base, %sv8i1 %mask) {
+; CHECK-LABEL: masked_sload_sv8i8_to_sv8i16:
+; CHECK: ld1sb { z0.h }, p0/z, [x0, #-3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 -3
+  %load = call %sv8i8 @llvm.masked.load.nxv8i8(%sv8i8* %base_load, i32 1, %sv8i1 %mask, %sv8i8 undef)
+  %ext = sext %sv8i8 %load to %sv8i16
+  ret %sv8i16 %ext
+}
+
+; 8-lane truncating contiguous stores.
+
+define void @masked_trunc_store_sv8i16_to_sv8i8(%sv8i16 %val, %sv8i8 *%base, %sv8i1 %mask) {
+; CHECK-LABEL: masked_trunc_store_sv8i16_to_sv8i8:
+; CHECK: st1b { z0.h }, p0, [x0, #3, mul vl]
+; CHECK-NEXT: ret
+  %base_load = getelementptr %sv8i8, %sv8i8* %base, i64 3
+  %trunc = trunc %sv8i16 %val to %sv8i8
+  call void @llvm.masked.store.nxv8i8(%sv8i8 %trunc, %sv8i8 *%base_load, i32 8, %sv8i1 %mask)
+  ret void
+}
+
+; 16-lane contiguous load/stores.
+
+define void @test_masked_ldst_sv16i8(%sv16i8 * %base, %sv16i1 %mask) {
+; CHECK-LABEL: test_masked_ldst_sv16i8:
+; CHECK: ld1b { z[[DATA:[0-9]+]].b }, p0/z, [x0, #6, mul vl]
+; CHECK: st1b { z[[DATA]].b }, p0, [x0, #7, mul vl]
+; CHECK: ret
+  %base_load = getelementptr %sv16i8, %sv16i8* %base, i64 6
+  %data = call %sv16i8 @llvm.masked.load.nxv16i8(%sv16i8* %base_load, i32 1, %sv16i1 %mask, %sv16i8 undef)
+  %base_store = getelementptr %sv16i8, %sv16i8 * %base, i64 7
+  call void @llvm.masked.store.nxv16i8(%sv16i8 %data, %sv16i8* %base_store,  i32 1, %sv16i1 %mask)
+  ret void
+}