[Clang][AArch64] Fine-grained ldp and stp policies.

[manosanag]

Summary

This patch enables fine-grained tuning control for ldp and stp.

It provides two new and concrete command-line options -aarch64-ldp-policy
and -aarch64-stp-policy to give the ability to control load and store
policies seperately with both clang and flang-new frontends including
when using -flto.

The accepted values for both options are:

• default: Use the ldp/stp policy currently used by the compiler (always).
• always: Emit ldp/stp regardless of alignment.
• never: Do not emit ldp/stp.
• aligned: In order to emit ldp/stp, first check if the load/store will be aligned to 2 * element_size.

[manosanag]

Updated Options.td to provide visibility for the options, because
it caused a regression for my fortran tests after rebasing to
current llvm main branch.

[dmgreen]

Can you give more details about why this is wanted and in which cases it helps with? Is it an optimization, as opposed to working around some correctness issue?

[manosanag]

Hello Dave,

thanks for replying.

Yes, this is an optimization.

On some AArch64 cores, including Ampere's ampere1 architecture that this is targeted for, load/store pair instructions are faster compared to simple loads/stores only when the alignment of the pair is at least twice that of the individual element being loaded. Based on the performance of various benchmarks, emitting ldp/stp instructions was disabled on GCC at some point (discussion is https://gcc.gnu.org/pipermail/gcc-patches/2023-April/615672.html). This patch improves on that and offers control over when the instructions are used.

Similar patch with the same flags has been recently submitted for review in the GCC mailing lists (https://gcc.gnu.org/pipermail/gcc-patches/2023-August/628590.html).

I have a fix ready for the fortran regressions shown by autotesting. I could include some of this information to the commit message of the diff.

[philipp.tomsich]

Should we move this to a GitHub PR instead?

[dmgreen]

We do not usually add front-end clang options for optimizations like this. Users are more likely to use them incorrectly, or just not know that they exist. The usual method would be to make a subtarget tuning feature that controls whether ldp are created, and enable it for -mcpu=ampere1.

Having an internal llvm option for it (-mllvm -aarch64-stp-policy=never) sounds fine, but should be considered an internal option. And adding a subtarget feature would make sense to have this be used from ampere1. If you get the option committed to GCC then it might be OK for clang too, but I would suggest splitting this into a patch for the backend part and another for the frontend option in either case.

[manosanag]

Moved to https://github.com/llvm/llvm-project/pull/66098.

clang/include/clang/Driver/Options.td

def mno_fix_cortex_a53_835769 : Flag<["-"], "mno-fix-cortex-a53-835769">,
   HelpText<"Don't workaround Cortex-A53 erratum 835769 (AArch64 only)">;
 def mmark_bti_property : Flag<["-"], "mmark-bti-property">,
   Group<m_aarch64_Features_Group>,
   HelpText<"Add .note.gnu.property with BTI to assembly files (AArch64 only)">;
 def mno_bti_at_return_twice : Flag<["-"], "mno-bti-at-return-twice">,
   Group<m_arm_Features_Group>,
   HelpText<"Do not add a BTI instruction after a setjmp or other"
            " return-twice construct (Arm/AArch64 only)">;
+def aarch64_ldp_policy_EQ : Joined<["-"], "aarch64-ldp-policy=">,
+  Group<m_aarch64_Features_Group>,
+  Visibility<[ClangOption, FlangOption, CC1Option, FC1Option]>,
+  HelpText<"Fine-grained load pair policy (AArch64 only)">;
+def aarch64_stp_policy_EQ : Joined<["-"], "aarch64-stp-policy=">,
+  Group<m_aarch64_Features_Group>,
+  Visibility<[ClangOption, FlangOption, CC1Option, FC1Option]>,
+  HelpText<"Fine-grained store pair policy (AArch64 only)">;
 
 foreach i = {1-31} in
   def ffixed_x#i : Flag<["-"], "ffixed-x"#i>, Group<m_Group>,
     HelpText<"Reserve the x"#i#" register (AArch64/RISC-V only)">;
 
 foreach i = {8-15,18} in
   def fcall_saved_x#i : Flag<["-"], "fcall-saved-x"#i>, Group<m_aarch64_Features_Group>,
     HelpText<"Make the x"#i#" register call-saved (AArch64 only)">;

clang/lib/Driver/ToolChains/Clang.cpp

if (const Arg *A = Args.getLastArg(clang::driver::options::OPT_mtune_EQ)) {
     CmdArgs.push_back("-tune-cpu");
     if (strcmp(A->getValue(), "native") == 0)
       CmdArgs.push_back(Args.MakeArgString(llvm::sys::getHostCPUName()));
     else
       CmdArgs.push_back(A->getValue());
   }
 
   AddUnalignedAccessWarning(CmdArgs);
+
+  // Handle -aarch64-ldp-policy=
+  if (Arg *A = Args.getLastArg(options::OPT_aarch64_ldp_policy_EQ)) {
+    StringRef Val = A->getValue();
+    const Driver &D = getToolChain().getDriver();
+    if (!Val.equals("aligned") && !Val.equals("never") &&
+        !Val.equals("always") && !Val.equals("default"))
+      // Handle the unsupported values passed to aarch64-ldp-policy.
+      D.Diag(diag::err_drv_unsupported_option_argument)
+          << A->getSpelling() << Val;
+    CmdArgs.push_back("-mllvm");
+    CmdArgs.push_back(Args.MakeArgString("-aarch64-ldp-policy=" + Val));
+  }
+
+  // Handle -aarch64-stp-policy=
+  if (Arg *A = Args.getLastArg(options::OPT_aarch64_stp_policy_EQ)) {
+    StringRef Val = A->getValue();
+    const Driver &D = getToolChain().getDriver();
+    if (!Val.equals("aligned") && !Val.equals("never") &&
+        !Val.equals("always") && !Val.equals("default"))
+      // Handle the unsupported values passed to aarch64-stp-policy.
+      D.Diag(diag::err_drv_unsupported_option_argument)
+          << A->getSpelling() << Val;
+    CmdArgs.push_back("-mllvm");
+    CmdArgs.push_back(Args.MakeArgString("-aarch64-stp-policy=" + Val));
+  }
 }
 
 void Clang::AddLoongArchTargetArgs(const ArgList &Args,
                                    ArgStringList &CmdArgs) const {
   const llvm::Triple &Triple = getToolChain().getTriple();
 
   CmdArgs.push_back("-target-abi");
   CmdArgs.push_back(

clang/lib/Driver/ToolChains/CommonArgs.cpp

if (willEmitRemarks(Args))
     renderRemarksOptions(Args, CmdArgs, ToolChain.getEffectiveTriple(), Input,
                          Output, PluginOptPrefix);
 
   // Handle remarks hotness/threshold related options.
   renderRemarksHotnessOptions(Args, CmdArgs, PluginOptPrefix);
 
   addMachineOutlinerArgs(D, Args, CmdArgs, ToolChain.getEffectiveTriple(),
                          /*IsLTO=*/true, PluginOptPrefix);
+
+  // Handle -aarch64-ldp-policy=
+  if (Arg *A = Args.getLastArg(options::OPT_aarch64_ldp_policy_EQ)) {
+    StringRef Val = A->getValue();
+    if (!Val.equals("aligned") && !Val.equals("never") &&
+        !Val.equals("always") && !Val.equals("default"))
+      // Handle the unsupported values passed to aarch64-ldp-policy.
+      D.Diag(diag::err_drv_unsupported_option_argument)
+          << A->getSpelling() << Val;
+    CmdArgs.push_back(Args.MakeArgString(Twine(PluginOptPrefix) +
+                                         "-aarch64-ldp-policy=" + Val));
+  }
+
+  // Handle -aarch64-stp-policy=
+  if (Arg *A = Args.getLastArg(options::OPT_aarch64_stp_policy_EQ)) {
+    StringRef Val = A->getValue();
+    if (!Val.equals("aligned") && !Val.equals("never") &&
+        !Val.equals("always") && !Val.equals("default"))
+      // Handle the unsupported values passed to aarch64-stp-policy.
+      D.Diag(diag::err_drv_unsupported_option_argument)
+          << A->getSpelling() << Val;
+    CmdArgs.push_back(Args.MakeArgString(Twine(PluginOptPrefix) +
+                                         "-aarch64-stp-policy=" + Val));
+  }
 }
 
 void tools::addOpenMPRuntimeLibraryPath(const ToolChain &TC,
                                         const ArgList &Args,
                                         ArgStringList &CmdArgs) {
   // Default to clang lib / lib64 folder, i.e. the same location as device
   // runtime.
   SmallString<256> DefaultLibPath =

clang/lib/Driver/ToolChains/Flang.cpp

void Flang::addTargetOptions(const ArgList &Args,
   case llvm::Triple::amdgcn:
   case llvm::Triple::aarch64:
   case llvm::Triple::riscv64:
   case llvm::Triple::x86_64:
     getTargetFeatures(D, Triple, Args, CmdArgs, /*ForAs*/ false);
     break;
   }
 
+  // Handle -aarch64-ldp-policy=
+  if (Arg *A = Args.getLastArg(options::OPT_aarch64_ldp_policy_EQ)) {
+    StringRef Val = A->getValue();
+    const Driver &D = getToolChain().getDriver();
+    if (!Val.equals("aligned") && !Val.equals("never") &&
+        !Val.equals("always") && !Val.equals("default"))
+      // Handle the unsupported values passed to aarch64-ldp-policy.
+      D.Diag(diag::err_drv_unsupported_option_argument)
+          << A->getSpelling() << Val;
+    CmdArgs.push_back("-mllvm");
+    CmdArgs.push_back(Args.MakeArgString("-aarch64-ldp-policy=" + Val));
+  }
+
+  // Handle -aarch64-stp-policy=
+  if (Arg *A = Args.getLastArg(options::OPT_aarch64_stp_policy_EQ)) {
+    StringRef Val = A->getValue();
+    const Driver &D = getToolChain().getDriver();
+    if (!Val.equals("aligned") && !Val.equals("never") &&
+        !Val.equals("always") && !Val.equals("default"))
+      // Handle the unsupported values passed to aarch64-stp-policy.
+      D.Diag(diag::err_drv_unsupported_option_argument)
+          << A->getSpelling() << Val;
+    CmdArgs.push_back("-mllvm");
+    CmdArgs.push_back(Args.MakeArgString("-aarch64-stp-policy=" + Val));
+  }
+
   // TODO: Add target specific flags, ABI, mtune option etc.
 }
 
 void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
                               const JobAction &JA, const ArgList &Args,
                               ArgStringList &CmdArgs) const {
   bool IsOpenMPDevice = JA.isDeviceOffloading(Action::OFK_OpenMP);
   bool IsHostOffloadingAction = JA.isHostOffloading(Action::OFK_OpenMP) ||

clang/test/Driver/aarch64-ldp-policy.c

+// RUN: %clang -### -target aarch64 -aarch64-ldp-policy=always %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALWAYS %s
+// RUN: %clang -### -target aarch64 -aarch64-ldp-policy=aligned %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALIGNED %s
+// RUN: %clang -### -target aarch64 -aarch64-ldp-policy=never %s -c 2>&1 | FileCheck -check-prefix=CHECK-NEVER %s
+// RUN: %clang -### -target aarch64 -aarch64-ldp-policy=default %s -c 2>&1 | FileCheck -check-prefix=CHECK-DEFAULT %s
+// RUN: not %clang -### -target aarch64 -aarch64-ldp-policy=def %s -c 2>&1 | FileCheck -check-prefix=CHECK-ARGUMENT %s
+// RUN: not %clang -c -target x86-64 -aarch64-ldp-policy=aligned %s 2>&1 | FileCheck -check-prefix=CHECK-TRIPLE %s
+
+// CHECK-ALWAYS: "-aarch64-ldp-policy=always"
+// CHECK-ALIGNED: "-aarch64-ldp-policy=aligned"
+// CHECK-NEVER: "-aarch64-ldp-policy=never"
+// CHECK-DEFAULT: "-aarch64-ldp-policy=default"
+// CHECK-ARGUMENT: clang: error: unsupported argument 'def' to option '-aarch64-ldp-policy='
+// CHECK-TRIPLE: clang: error: unsupported option '-aarch64-ldp-policy=' for target

clang/test/Driver/aarch64-stp-policy.c

+// RUN: %clang -### -target aarch64 -aarch64-stp-policy=always %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALWAYS %s
+// RUN: %clang -### -target aarch64 -aarch64-stp-policy=aligned %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALIGNED %s
+// RUN: %clang -### -target aarch64 -aarch64-stp-policy=never %s -c 2>&1 | FileCheck -check-prefix=CHECK-NEVER %s
+// RUN: %clang -### -target aarch64 -aarch64-stp-policy=default %s -c 2>&1 | FileCheck -check-prefix=CHECK-DEFAULT %s
+// RUN: not %clang -### -target aarch64 -aarch64-stp-policy=def %s -c 2>&1 | FileCheck -check-prefix=CHECK-ARGUMENT %s
+// RUN: not %clang -c -target x86-64 -aarch64-stp-policy=aligned %s 2>&1 | FileCheck -check-prefix=CHECK-TRIPLE %s
+
+// CHECK-ALWAYS: "-aarch64-stp-policy=always"
+// CHECK-ALIGNED: "-aarch64-stp-policy=aligned"
+// CHECK-NEVER: "-aarch64-stp-policy=never"
+// CHECK-DEFAULT: "-aarch64-stp-policy=default"
+// CHECK-ARGUMENT: clang: error: unsupported argument 'def' to option '-aarch64-stp-policy='
+// CHECK-TRIPLE: clang: error: unsupported option '-aarch64-stp-policy=' for target

clang/test/Driver/flang/aarch64-ldp-policy.f90

+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-ldp-policy=always %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALWAYS %s
+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-ldp-policy=aligned %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALIGNED %s
+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-ldp-policy=never %s -c 2>&1 | FileCheck -check-prefix=CHECK-NEVER %s
+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-ldp-policy=default %s -c 2>&1 | FileCheck -check-prefix=CHECK-DEFAULT %s
+! RUN: not %clang -### --driver-mode=flang -target aarch64 -aarch64-ldp-policy=def %s -c 2>&1 | FileCheck -check-prefix=CHECK-ARGUMENT %s
+
+! CHECK-ALWAYS: "-aarch64-ldp-policy=always"
+! CHECK-ALIGNED: "-aarch64-ldp-policy=aligned"
+! CHECK-NEVER: "-aarch64-ldp-policy=never"
+! CHECK-DEFAULT: "-aarch64-ldp-policy=default"
+! CHECK-ARGUMENT: clang: error: unsupported argument 'def' to option '-aarch64-ldp-policy='

clang/test/Driver/flang/aarch64-stp-policy.f90

+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-stp-policy=always %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALWAYS %s
+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-stp-policy=aligned %s -c 2>&1 | FileCheck -check-prefix=CHECK-ALIGNED %s
+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-stp-policy=never %s -c 2>&1 | FileCheck -check-prefix=CHECK-NEVER %s
+! RUN: %clang -### --driver-mode=flang -target aarch64 -aarch64-stp-policy=default %s -c 2>&1 | FileCheck -check-prefix=CHECK-DEFAULT %s
+! RUN: not %clang -### --driver-mode=flang -target aarch64 -aarch64-stp-policy=def %s -c 2>&1 | FileCheck -check-prefix=CHECK-ARGUMENT %s
+
+! CHECK-ALWAYS: "-aarch64-stp-policy=always"
+! CHECK-ALIGNED: "-aarch64-stp-policy=aligned"
+! CHECK-NEVER: "-aarch64-stp-policy=never"
+! CHECK-DEFAULT: "-aarch64-stp-policy=default"
+! CHECK-ARGUMENT: clang: error: unsupported argument 'def' to option '-aarch64-stp-policy='

llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

 STATISTIC(NumUnscaledPairCreated,
           "Number of load/store from unscaled generated");
 STATISTIC(NumZeroStoresPromoted, "Number of narrow zero stores promoted");
 STATISTIC(NumLoadsFromStoresPromoted, "Number of loads from stores promoted");
 
 DEBUG_COUNTER(RegRenamingCounter, DEBUG_TYPE "-reg-renaming",
               "Controls which pairs are considered for renaming");
 
+enum LdpPolicy {
+  LDP_POLICY_ALWAYS, ///< Emit ldp regardless of alignment.
+  LDP_POLICY_NEVER,  ///< Do not emit ldp.
+  LDP_POLICY_ALIGNED ///< In order to emit ldp, first check if the load will
+                     ///< be aligned to 2 * element_size.
+};
+
+enum StpPolicy {
+  STP_POLICY_ALWAYS, ///< Emit stp regardless of alignment.
+  STP_POLICY_NEVER,  ///< Do not emit stp.
+  STP_POLICY_ALIGNED ///< In order to emit stp, first check if the store will
+                     ///< be aligned to 2 * element_size.
+};
+
+static cl::opt<LdpPolicy> AArch64LdpPolicy(
+    "aarch64-ldp-policy", cl::Optional, cl::init(LDP_POLICY_ALWAYS),
+    cl::desc("AArch64 Specific: Load pair policy."),
+    cl::values(clEnumValN(LDP_POLICY_NEVER, "never", "Do not emit ldp."),
+               clEnumValN(LDP_POLICY_ALIGNED, "aligned",
+                          "Emit ldp only if the source pointer is aligned to "
+                          "at least double the alignment of the type."),
+               clEnumValN(LDP_POLICY_ALWAYS, "always",
+                          "Emit ldp regardless of alignment. (default)"),
+               clEnumValN(LDP_POLICY_ALWAYS, "default", "Use the default.")));
+
+static cl::opt<StpPolicy> AArch64StpPolicy(
+    "aarch64-stp-policy", cl::Optional, cl::init(STP_POLICY_ALWAYS),
+    cl::desc("AArch64 Specific: Store pair policy."),
+    cl::values(clEnumValN(STP_POLICY_NEVER, "never", "Do not emit stp."),
+               clEnumValN(STP_POLICY_ALIGNED, "aligned",
+                          "Emit stp only if the source pointer is aligned to "
+                          "at least double the alignment of the type."),
+               clEnumValN(STP_POLICY_ALWAYS, "always",
+                          "Emit stp regardless of alignment. (default)"),
+               clEnumValN(STP_POLICY_ALWAYS, "default", "Use the default.")));
+
 // The LdStLimit limits how far we search for load/store pairs.
 static cl::opt<unsigned> LdStLimit("aarch64-load-store-scan-limit",
                                    cl::init(20), cl::Hidden);
 
 // The UpdateLimit limits how far we search for update instructions when we form
 // pre-/post-index instructions.
 static cl::opt<unsigned> UpdateLimit("aarch64-update-scan-limit", cl::init(100),
                                      cl::Hidden);

 // instruction.
 bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
   MachineInstr &MI = *MBBI;
   MachineBasicBlock::iterator E = MI.getParent()->end();
 
   if (!TII->isCandidateToMergeOrPair(MI))
     return false;
 
+  // Fetch the memoperand of the load/store that is a candidate for combination.
+  MachineMemOperand *memOp = MBBI->memoperands().data()[0];
+
+  // If a load arrives and LDP_POLICY_NEVER is opted, do not emit ldp.
+  if (memOp->isLoad() && AArch64LdpPolicy == LDP_POLICY_NEVER)
+    return false;
+  // If a store arrives and STP_POLICY_NEVER is opted, do not emit stp.
+  if (memOp->isStore() && AArch64StpPolicy == STP_POLICY_NEVER)
+    return false;
+
   // Early exit if the offset is not possible to match. (6 bits of positive
   // range, plus allow an extra one in case we find a later insn that matches
   // with Offset-1)
   bool IsUnscaled = TII->hasUnscaledLdStOffset(MI);
   int Offset = AArch64InstrInfo::getLdStOffsetOp(MI).getImm();
   int OffsetStride = IsUnscaled ? TII->getMemScale(MI) : 1;
   // Allow one more for offset.
   if (Offset > 0)
     Offset -= OffsetStride;
   if (!inBoundsForPair(IsUnscaled, Offset, OffsetStride))
     return false;
 
   // Look ahead up to LdStLimit instructions for a pairable instruction.
   LdStPairFlags Flags;
   MachineBasicBlock::iterator Paired =
       findMatchingInsn(MBBI, Flags, LdStLimit, /* FindNarrowMerge = */ false);
   if (Paired != E) {
     ++NumPairCreated;
     if (TII->hasUnscaledLdStOffset(MI))
       ++NumUnscaledPairCreated;
     // Keeping the iterator straight is a pain, so we let the merge routine tell
     // us what the next instruction is after it's done mucking about.
     auto Prev = std::prev(MBBI);
+
+    // Get the needed alignments to check them if
+    // LDP_POLICY_ALIGNED/STP_POLICY_ALIGNED is opted.
+    uint64_t memAlignment = memOp->getAlign().value();
+    uint64_t typeAlignment = Align(memOp->getSize()).value();
+
+    // If a load arrives and LDP_POLICY_ALIGNED is opted, check that the
+    // alignment of the source pointer is at least double the alignment of the
+    // type.
+    if (memOp->isLoad() && AArch64LdpPolicy == LDP_POLICY_ALIGNED &&
+        memAlignment < 2 * typeAlignment)
+      return false;
+    // If a store arrives and STP_POLICY_ALIGNED is opted, check that the
+    // alignment of the source pointer is at least double the alignment of the
+    // type.
+    if (memOp->isStore() && AArch64StpPolicy == STP_POLICY_ALIGNED &&
+        memAlignment < 2 * typeAlignment)
+      return false;
     MBBI = mergePairedInsns(MBBI, Paired, Flags);
     // Collect liveness info for instructions between Prev and the new position
     // MBBI.
     for (auto I = std::next(Prev); I != MBBI; I++)
       updateDefinedRegisters(*I, DefinedInBB, TRI);
 
     return true;
   }

llvm/test/CodeGen/AArch64/ldp-aligned.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -O2 -mtriple=aarch64 -mcpu=generic -aarch64-ldp-policy=aligned | FileCheck %s
+
+define i32 @ldp_aligned_int32_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    ldp w9, w8, [x8]
+; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -64
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load i32, ptr %4, align 64
+  %6 = getelementptr inbounds i32, ptr %4, i64 1
+  %7 = load i32, ptr %6, align 4
+  %8 = add nsw i32 %7, %5
+  ret i32 %8
+}
+
+define i64 @ldp_aligned_int64_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    ldp x9, x8, [x8]
+; CHECK-NEXT:    add x0, x8, x9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -128
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load i64, ptr %4, align 128
+  %6 = getelementptr inbounds i64, ptr %4, i64 1
+  %7 = load i64, ptr %6, align 8
+  %8 = add nsw i64 %7, %5
+  ret i64 %8
+}
+
+define <4 x i32> @ldp_aligned_v4si(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    ldp q0, q1, [x8]
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -256
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load <4 x i32>, ptr %4, align 256
+  %6 = getelementptr inbounds <4 x i32>, ptr %4, i64 1
+  %7 = load <4 x i32>, ptr %6, align 16
+  %8 = add <4 x i32> %7, %5
+  ret <4 x i32> %8
+}
+
+define i32 @ldp_unaligned_int32_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    ldr w9, [x8, #4]
+; CHECK-NEXT:    ldr w8, [x8, #8]
+; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -64
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds i32, ptr %4, i64 1
+  %6 = load i32, ptr %5, align 4
+  %7 = getelementptr inbounds i32, ptr %4, i64 2
+  %8 = load i32, ptr %7, align 8
+  %9 = add nsw i32 %8, %6
+  ret i32 %9
+}
+
+define i64 @ldp_unaligned_int64_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    ldr x9, [x8, #8]
+; CHECK-NEXT:    ldr x8, [x8, #16]
+; CHECK-NEXT:    add x0, x8, x9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -128
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds i64, ptr %4, i64 1
+  %6 = load i64, ptr %5, align 8
+  %7 = getelementptr inbounds i64, ptr %4, i64 2
+  %8 = load i64, ptr %7, align 16
+  %9 = add nsw i64 %8, %6
+  ret i64 %9
+}
+
+define <4 x i32> @ldp_unaligned_v4si(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    ldr q0, [x8, #16]
+; CHECK-NEXT:    ldr q1, [x8, #32]
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -256
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds <4 x i32>, ptr %4, i64 1
+  %6 = load <4 x i32>, ptr %5, align 16
+  %7 = getelementptr inbounds <4 x i32>, ptr %4, i64 2
+  %8 = load <4 x i32>, ptr %7, align 32
+  %9 = add <4 x i32> %8, %6
+  ret <4 x i32> %9
+}

llvm/test/CodeGen/AArch64/ldp-always.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -O2 -mtriple=aarch64 -mcpu=generic -aarch64-ldp-policy=always | FileCheck %s
+
+define i32 @ldp_aligned_int32_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    ldp w9, w8, [x8]
+; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -64
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load i32, ptr %4, align 64
+  %6 = getelementptr inbounds i32, ptr %4, i64 1
+  %7 = load i32, ptr %6, align 4
+  %8 = add nsw i32 %7, %5
+  ret i32 %8
+}
+
+define i64 @ldp_aligned_int64_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    ldp x9, x8, [x8]
+; CHECK-NEXT:    add x0, x8, x9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -128
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load i64, ptr %4, align 128
+  %6 = getelementptr inbounds i64, ptr %4, i64 1
+  %7 = load i64, ptr %6, align 8
+  %8 = add nsw i64 %7, %5
+  ret i64 %8
+}
+
+define <4 x i32> @ldp_aligned_v4si(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    ldp q0, q1, [x8]
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -256
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load <4 x i32>, ptr %4, align 256
+  %6 = getelementptr inbounds <4 x i32>, ptr %4, i64 1
+  %7 = load <4 x i32>, ptr %6, align 16
+  %8 = add <4 x i32> %7, %5
+  ret <4 x i32> %8
+}
+
+define i32 @ldp_unaligned_int32_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    ldp w9, w8, [x8, #4]
+; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -64
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds i32, ptr %4, i64 1
+  %6 = load i32, ptr %5, align 4
+  %7 = getelementptr inbounds i32, ptr %4, i64 2
+  %8 = load i32, ptr %7, align 8
+  %9 = add nsw i32 %8, %6
+  ret i32 %9
+}
+
+define i64 @ldp_unaligned_int64_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    ldp x9, x8, [x8, #8]
+; CHECK-NEXT:    add x0, x8, x9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -128
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds i64, ptr %4, i64 1
+  %6 = load i64, ptr %5, align 8
+  %7 = getelementptr inbounds i64, ptr %4, i64 2
+  %8 = load i64, ptr %7, align 16
+  %9 = add nsw i64 %8, %6
+  ret i64 %9
+}
+
+define <4 x i32> @ldp_unaligned_v4si(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    ldp q0, q1, [x8, #16]
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -256
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds <4 x i32>, ptr %4, i64 1
+  %6 = load <4 x i32>, ptr %5, align 16
+  %7 = getelementptr inbounds <4 x i32>, ptr %4, i64 2
+  %8 = load <4 x i32>, ptr %7, align 32
+  %9 = add <4 x i32> %8, %6
+  ret <4 x i32> %9
+}
+

llvm/test/CodeGen/AArch64/ldp-never.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -O2 -mtriple=aarch64 -mcpu=generic -aarch64-ldp-policy=never | FileCheck %s
+
+define i32 @ldp_aligned_int32_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    ldr w9, [x8]
+; CHECK-NEXT:    ldr w8, [x8, #4]
+; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -64
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load i32, ptr %4, align 64
+  %6 = getelementptr inbounds i32, ptr %4, i64 1
+  %7 = load i32, ptr %6, align 4
+  %8 = add nsw i32 %7, %5
+  ret i32 %8
+}
+
+define i64 @ldp_aligned_int64_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    ldr x9, [x8]
+; CHECK-NEXT:    ldr x8, [x8, #8]
+; CHECK-NEXT:    add x0, x8, x9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -128
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load i64, ptr %4, align 128
+  %6 = getelementptr inbounds i64, ptr %4, i64 1
+  %7 = load i64, ptr %6, align 8
+  %8 = add nsw i64 %7, %5
+  ret i64 %8
+}
+
+define <4 x i32> @ldp_aligned_v4si(ptr %0) #0 {
+; CHECK-LABEL: ldp_aligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    ldr q0, [x8]
+; CHECK-NEXT:    ldr q1, [x8, #16]
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -256
+  %4 = inttoptr i64 %3 to ptr
+  %5 = load <4 x i32>, ptr %4, align 256
+  %6 = getelementptr inbounds <4 x i32>, ptr %4, i64 1
+  %7 = load <4 x i32>, ptr %6, align 16
+  %8 = add <4 x i32> %7, %5
+  ret <4 x i32> %8
+}
+
+define i32 @ldp_unaligned_int32_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    ldr w9, [x8, #4]
+; CHECK-NEXT:    ldr w8, [x8, #8]
+; CHECK-NEXT:    add w0, w8, w9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -64
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds i32, ptr %4, i64 1
+  %6 = load i32, ptr %5, align 4
+  %7 = getelementptr inbounds i32, ptr %4, i64 2
+  %8 = load i32, ptr %7, align 8
+  %9 = add nsw i32 %8, %6
+  ret i32 %9
+}
+
+define i64 @ldp_unaligned_int64_t(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    ldr x9, [x8, #8]
+; CHECK-NEXT:    ldr x8, [x8, #16]
+; CHECK-NEXT:    add x0, x8, x9
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -128
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds i64, ptr %4, i64 1
+  %6 = load i64, ptr %5, align 8
+  %7 = getelementptr inbounds i64, ptr %4, i64 2
+  %8 = load i64, ptr %7, align 16
+  %9 = add nsw i64 %8, %6
+  ret i64 %9
+}
+
+define <4 x i32> @ldp_unaligned_v4si(ptr %0) #0 {
+; CHECK-LABEL: ldp_unaligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    ldr q0, [x8, #16]
+; CHECK-NEXT:    ldr q1, [x8, #32]
+; CHECK-NEXT:    add v0.4s, v1.4s, v0.4s
+; CHECK-NEXT:    ret
+  %2 = ptrtoint ptr %0 to i64
+  %3 = and i64 %2, -256
+  %4 = inttoptr i64 %3 to ptr
+  %5 = getelementptr inbounds <4 x i32>, ptr %4, i64 1
+  %6 = load <4 x i32>, ptr %5, align 16
+  %7 = getelementptr inbounds <4 x i32>, ptr %4, i64 2
+  %8 = load <4 x i32>, ptr %7, align 32
+  %9 = add <4 x i32> %8, %6
+  ret <4 x i32> %9
+}
+

llvm/test/CodeGen/AArch64/stp-aligned.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -O2 -mtriple=aarch64 -mcpu=generic -aarch64-stp-policy=aligned | FileCheck %s
+
+define ptr @stp_aligned_int32_t(ptr %0, i32 %1) #0 {
+; CHECK-LABEL: stp_aligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    stp w1, w1, [x0]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -64
+  %5 = inttoptr i64 %4 to ptr
+  store i32 %1, ptr %5, align 64
+  %6 = getelementptr inbounds i32, ptr %5, i64 1
+  store i32 %1, ptr %6, align 4
+  ret ptr %5
+}
+
+define dso_local ptr @stp_aligned_int64_t(ptr %0, i64 %1) #0 {
+; CHECK-LABEL: stp_aligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffff80
+; CHECK-NEXT:    stp x1, x1, [x0]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -128
+  %5 = inttoptr i64 %4 to ptr
+  store i64 %1, ptr %5, align 128
+  %6 = getelementptr inbounds i64, ptr %5, i64 1
+  store i64 %1, ptr %6, align 8
+  ret ptr %5
+}
+
+define ptr @stp_aligned_v4si(ptr %0, <4 x i32> %1) #0 {
+; CHECK-LABEL: stp_aligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffff00
+; CHECK-NEXT:    stp q0, q0, [x0]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -256
+  %5 = inttoptr i64 %4 to ptr
+  store <4 x i32> %1, ptr %5, align 256
+  %6 = getelementptr inbounds <4 x i32>, ptr %5, i64 1
+  store <4 x i32> %1, ptr %6, align 16
+  ret ptr %5
+}
+
+define ptr @stp_unaligned_int32_t(ptr %0, i32 %1) #0 {
+; CHECK-LABEL: stp_unaligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    orr x0, x8, #0x4
+; CHECK-NEXT:    str w1, [x8, #4]
+; CHECK-NEXT:    str w1, [x8, #8]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -64
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds i32, ptr %5, i64 1
+  store i32 %1, ptr %6, align 4
+  %7 = getelementptr inbounds i32, ptr %5, i64 2
+  store i32 %1, ptr %7, align 8
+  ret ptr %6
+}
+
+define ptr @stp_unaligned_int64_t(ptr %0, i64 %1) #0 {
+; CHECK-LABEL: stp_unaligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    orr x0, x8, #0x8
+; CHECK-NEXT:    str x1, [x8, #8]
+; CHECK-NEXT:    str x1, [x8, #16]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -128
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds i64, ptr %5, i64 1
+  store i64 %1, ptr %6, align 8
+  %7 = getelementptr inbounds i64, ptr %5, i64 2
+  store i64 %1, ptr %7, align 16
+  ret ptr %6
+}
+
+define ptr @stp_unaligned_v4si(ptr %0, <4 x i32> %1) #0 {
+; CHECK-LABEL: stp_unaligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    orr x0, x8, #0x10
+; CHECK-NEXT:    str q0, [x8, #16]
+; CHECK-NEXT:    str q0, [x8, #32]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -256
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds <4 x i32>, ptr %5, i64 1
+  store <4 x i32> %1, ptr %6, align 16
+  %7 = getelementptr inbounds <4 x i32>, ptr %5, i64 2
+  store <4 x i32> %1, ptr %7, align 32
+  ret ptr %6
+}
+

llvm/test/CodeGen/AArch64/stp-always.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -O2 -mtriple=aarch64 -mcpu=generic -aarch64-stp-policy=always | FileCheck %s
+
+define ptr @stp_aligned_int32_t(ptr %0, i32 %1) #0 {
+; CHECK-LABEL: stp_aligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    stp w1, w1, [x0]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -64
+  %5 = inttoptr i64 %4 to ptr
+  store i32 %1, ptr %5, align 64
+  %6 = getelementptr inbounds i32, ptr %5, i64 1
+  store i32 %1, ptr %6, align 4
+  ret ptr %5
+}
+
+define ptr @stp_aligned_int64_t(ptr %0, i64 %1) #0 {
+; CHECK-LABEL: stp_aligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffff80
+; CHECK-NEXT:    stp x1, x1, [x0]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -128
+  %5 = inttoptr i64 %4 to ptr
+  store i64 %1, ptr %5, align 128
+  %6 = getelementptr inbounds i64, ptr %5, i64 1
+  store i64 %1, ptr %6, align 8
+  ret ptr %5
+}
+
+define ptr @stp_aligned_v4si(ptr %0, <4 x i32> %1) #0 {
+; CHECK-LABEL: stp_aligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffff00
+; CHECK-NEXT:    stp q0, q0, [x0]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -256
+  %5 = inttoptr i64 %4 to ptr
+  store <4 x i32> %1, ptr %5, align 256
+  %6 = getelementptr inbounds <4 x i32>, ptr %5, i64 1
+  store <4 x i32> %1, ptr %6, align 16
+  ret ptr %5
+}
+
+define ptr @stp_unaligned_int32_t(ptr %0, i32 %1) #0 {
+; CHECK-LABEL: stp_unaligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    orr x0, x8, #0x4
+; CHECK-NEXT:    stp w1, w1, [x8, #4]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -64
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds i32, ptr %5, i64 1
+  store i32 %1, ptr %6, align 4
+  %7 = getelementptr inbounds i32, ptr %5, i64 2
+  store i32 %1, ptr %7, align 8
+  ret ptr %6
+}
+
+define ptr @stp_unaligned_int64_t(ptr %0, i64 %1) #0 {
+; CHECK-LABEL: stp_unaligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    orr x0, x8, #0x8
+; CHECK-NEXT:    stp x1, x1, [x8, #8]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -128
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds i64, ptr %5, i64 1
+  store i64 %1, ptr %6, align 8
+  %7 = getelementptr inbounds i64, ptr %5, i64 2
+  store i64 %1, ptr %7, align 16
+  ret ptr %6
+}
+
+define ptr @stp_unaligned_v4si(ptr %0, <4 x i32> %1) #0 {
+; CHECK-LABEL: stp_unaligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    orr x0, x8, #0x10
+; CHECK-NEXT:    stp q0, q0, [x8, #16]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -256
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds <4 x i32>, ptr %5, i64 1
+  store <4 x i32> %1, ptr %6, align 16
+  %7 = getelementptr inbounds <4 x i32>, ptr %5, i64 2
+  store <4 x i32> %1, ptr %7, align 32
+  ret ptr %6
+}
+

llvm/test/CodeGen/AArch64/stp-never.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc < %s -O2 -mtriple=aarch64 -mcpu=generic -aarch64-stp-policy=never | FileCheck %s
+
+define ptr @stp_aligned_int32_t(ptr %0, i32 %1) #0 {
+; CHECK-LABEL: stp_aligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    str w1, [x0]
+; CHECK-NEXT:    str w1, [x0, #4]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -64
+  %5 = inttoptr i64 %4 to ptr
+  store i32 %1, ptr %5, align 64
+  %6 = getelementptr inbounds i32, ptr %5, i64 1
+  store i32 %1, ptr %6, align 4
+  ret ptr %5
+}
+
+define ptr @stp_aligned_int64_t(ptr %0, i64 %1) #0 {
+; CHECK-LABEL: stp_aligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffff80
+; CHECK-NEXT:    str x1, [x0]
+; CHECK-NEXT:    str x1, [x0, #8]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -128
+  %5 = inttoptr i64 %4 to ptr
+  store i64 %1, ptr %5, align 128
+  %6 = getelementptr inbounds i64, ptr %5, i64 1
+  store i64 %1, ptr %6, align 8
+  ret ptr %5
+}
+
+define ptr @stp_aligned_v4si(ptr %0, <4 x i32> %1) #0 {
+; CHECK-LABEL: stp_aligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x0, x0, #0xffffffffffffff00
+; CHECK-NEXT:    str q0, [x0]
+; CHECK-NEXT:    str q0, [x0, #16]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -256
+  %5 = inttoptr i64 %4 to ptr
+  store <4 x i32> %1, ptr %5, align 256
+  %6 = getelementptr inbounds <4 x i32>, ptr %5, i64 1
+  store <4 x i32> %1, ptr %6, align 16
+  ret ptr %5
+}
+
+define ptr @stp_unaligned_int32_t(ptr %0, i32 %1) #0 {
+; CHECK-LABEL: stp_unaligned_int32_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffffc0
+; CHECK-NEXT:    orr x0, x8, #0x4
+; CHECK-NEXT:    str w1, [x8, #4]
+; CHECK-NEXT:    str w1, [x8, #8]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -64
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds i32, ptr %5, i64 1
+  store i32 %1, ptr %6, align 4
+  %7 = getelementptr inbounds i32, ptr %5, i64 2
+  store i32 %1, ptr %7, align 8
+  ret ptr %6
+}
+
+define ptr @stp_unaligned_int64_t(ptr %0, i64 %1) #0 {
+; CHECK-LABEL: stp_unaligned_int64_t:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff80
+; CHECK-NEXT:    orr x0, x8, #0x8
+; CHECK-NEXT:    str x1, [x8, #8]
+; CHECK-NEXT:    str x1, [x8, #16]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -128
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds i64, ptr %5, i64 1
+  store i64 %1, ptr %6, align 8
+  %7 = getelementptr inbounds i64, ptr %5, i64 2
+  store i64 %1, ptr %7, align 16
+  ret ptr %6
+}
+
+define ptr @stp_unaligned_v4si(ptr %0, <4 x i32> %1) #0 {
+; CHECK-LABEL: stp_unaligned_v4si:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    and x8, x0, #0xffffffffffffff00
+; CHECK-NEXT:    orr x0, x8, #0x10
+; CHECK-NEXT:    str q0, [x8, #16]
+; CHECK-NEXT:    str q0, [x8, #32]
+; CHECK-NEXT:    ret
+  %3 = ptrtoint ptr %0 to i64
+  %4 = and i64 %3, -256
+  %5 = inttoptr i64 %4 to ptr
+  %6 = getelementptr inbounds <4 x i32>, ptr %5, i64 1
+  store <4 x i32> %1, ptr %6, align 16
+  %7 = getelementptr inbounds <4 x i32>, ptr %5, i64 2
+  store <4 x i32> %1, ptr %7, align 32
+  ret ptr %6
+}
+