[RISCV] Add a subtarget feature to enable unaligned scalar loads and stores

[reames]

Summary

A RISCV implementation can choose to implement unaligned load/store support. We currently don't have a way for such a processor to indicate a preference for unaligned load/stores, so add a subtarget feature.

There doesn't appear to be a formal extension for unaligned support. The RISCV Profiles (https://github.com/riscv/riscv-profiles/blob/main/profiles.adoc#rva20u64-profile) docs use the name Zicclsm, but a) that doesn't appear to actually been standardized, and b) isn't quite what we want here anyway due to the perf comment.

Instead, we can follow precedent from other backends and have a feature flag for the existence of misaligned load/stores with sufficient performance that user code should actually use them.

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11824

This seems like it should be a different feature for vectors.

[reames]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11824

There might be such a target where scalar vs vector matters, but on the motivating case, this is not expected to matter. We can split later if needed.

[jrtc27]

Is this meant to be "it works" or "it works without trapping for emulation"? Pretty much every EEI out there has misaligned accesses guaranteed to work, just not quickly, and in those cases you'd still want to avoid them as the inlined byte-wise code is far faster.

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11810–11814

If returning true and Fast isn't null we should be setting it; some users pre-initialise it, but not all (I found one in GlobalISel and stopped looking)

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11824

There are no vector tests and the description in the .td file says “scalar”. The patch should at least be self consistent.

[reames]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11810–11814

Good catch. I will unconditionally initialize to false to make sure we don't miss any paths. Thanks!

11824

You're absolutely correct on that. I'd originally had it split, then changed my mind. Let me rebase and correct both of those!

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11820

Not directly related to this patch, but it could be argued that *Fast = true is incorrect for vectors here. Being element aligned doesn't guarantee optimal access if you're splitting a cache line or other alignment requirements of your memory unit.

[reames]

In D126085#3528394, @jrtc27 wrote:

Is this meant to be "it works" or "it works without trapping for emulation"? Pretty much every EEI out there has misaligned accesses guaranteed to work, just not quickly, and in those cases you'd still want to avoid them as the inlined byte-wise code is far faster.

From what I can tell looking at existing targets, the three states for unaligned are basically.

• "not present" - either the hardware doesn't support, or performance is unreasonable
• "present" - hardware is present, and reasonable performant. unaligned accesses in source are lowered as such.
• "fast" - hardware is present, optimizer should use when doesn't exist in code already

I tried to follow that convention with this change.

[reames]

address review comments

[reames]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11824

It turns out writing vector tests was a useful exercise. :)

From what I can tell, we aggressively canonicalize towards byte element loads and stores for vectors. As such, no plain vector load or store is every treated as unaligned.

However, some indexed and strided load tests did fail if I asserted the fallthrough return was never taken.

For the moment, I decided to restrict the scope to only the scalar case. Mostly because there look to be enough other opportunities around indexed loads that this didn't seem terribly useful.

With that, I could go in two directions here.

1. Rename this to scalar and add a separate flag later for vector cases as warranted.
2. Document the intent that this covers all access, but leave the indexed/strided case currently unimplemented.

I have no strong preference and will do as reviewers request.

[asb]

llvm/lib/Target/RISCV/RISCV.td

435

Nit: reasonable => reasonably

[reames]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11824

@craig.topper ping?

[craig.topper]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

11824

Let's rename it to scalar and add a separate flag for vector later as warranted.

[reames]

Revise per reviewer request

[craig.topper]

llvm/lib/Target/RISCV/RISCV.td

435

asb's comment still needs to be addressed.

[reames]

llvm/lib/Target/RISCV/RISCV.td

435

Oops, yep, will rev.

[reames]

address missed review comment

[asb]

I don't think it affects this patch (as splitting scalar vs vector features for alignment makes sense regardless of the frontend option), but I had a quick look at the frontend option for this. Looks like on the GCC side it's -mno-strict-align. On Arm/AArch64 it's spelled -munaligned-access and __ARM_FEATURE_UNALIGNED is set. Unless there's some target generic #define that's set that I'm missing, it feels like it would be useful to set a define for the RISC-V case as well (and agree this with the GNU folks).

One thing I noticed that does affect this patch (and sorry I didn't spot this earlier!), is we should be setting Tag_RISCV_unaligned_access when this feature is set (see here).

[reames]

In D126085#3538293, @asb wrote:

I don't think it affects this patch (as splitting scalar vs vector features for alignment makes sense regardless of the frontend option), but I had a quick look at the frontend option for this. Looks like on the GCC side it's -mno-strict-align. On Arm/AArch64 it's spelled -munaligned-access and __ARM_FEATURE_UNALIGNED is set. Unless there's some target generic #define that's set that I'm missing, it feels like it would be useful to set a define for the RISC-V case as well (and agree this with the GNU folks).

I agree this is separate. What a target defaults to should not be blocked on what a frontend might chose to override. I'll note that these flags don't appear to influence x86 handling of the corresponding cases, so unless this is handled somewhere generically I missed, this isn't RISCV specific in any way.

One thing I noticed that does affect this patch (and sorry I didn't spot this earlier!), is we should be setting Tag_RISCV_unaligned_access when this feature is set (see here).

Er, I'm sorry, what? Why?

I agree this spec exists, but what purpose does the attribute bit actually serve? There's no interaction with the dynamic loader here. It's a compiler codegen choice.

If you really want, I'm fine adding the attribute wiring (in a separate patch), but this really looks like a spec that is a) out of sync with reality, and b) serves no obvious purpose.

[asb]

In D126085#3538325, @reames wrote:

In D126085#3538293, @asb wrote:

I don't think it affects this patch (as splitting scalar vs vector features for alignment makes sense regardless of the frontend option), but I had a quick look at the frontend option for this. Looks like on the GCC side it's -mno-strict-align. On Arm/AArch64 it's spelled -munaligned-access and __ARM_FEATURE_UNALIGNED is set. Unless there's some target generic #define that's set that I'm missing, it feels like it would be useful to set a define for the RISC-V case as well (and agree this with the GNU folks).

I agree this is separate. What a target defaults to should not be blocked on what a frontend might chose to override. I'll note that these flags don't appear to influence x86 handling of the corresponding cases, so unless this is handled somewhere generically I missed, this isn't RISCV specific in any way.

Yep, just mentioning for future reference.

One thing I noticed that does affect this patch (and sorry I didn't spot this earlier!), is we should be setting Tag_RISCV_unaligned_access when this feature is set (see here).

Er, I'm sorry, what? Why?

I agree this spec exists, but what purpose does the attribute bit actually serve? There's no interaction with the dynamic loader here. It's a compiler codegen choice.

If you really want, I'm fine adding the attribute wiring (in a separate patch), but this really looks like a spec that is a) out of sync with reality, and b) serves no obvious purpose.

I'm not sure the attribute is used in the wild in any meaningful way right now, but I think it does have purpose. There's no guarantee that a RISC-V EEI actually provides a trap handler for misaligned accesses, so on some targets (largely embedded), compiling with +unaligned-scalar-mem may result in code that is unsuitable. Given there's no way to describe support for misaligned accesses in the ISA naming string, it's probably similarly useful to the other attributes from the perspective of any tooling working with ELF files. Such tooling might be used to warn / error when a likely build misconfiguration is found within a project.

If I were authoring the patch, I'd likely put the attribute wiring in the same patch (or perhaps if I really wanted to split things, have patch 1 adding the feature but only impacting the attribute, and patch 2 adding codegen tests and changes). But tastes vary and I don't think it makes a big difference for this patch, so I'm happy with whatever sequencing you prefer here.

[jrtc27]

Technically it's may not will so you can just hard-wire it to 1...

[asb]

This is all that's needed to hook up the appropriate attribute:

diff --git a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVTargetStreamer.cpp b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVTargetStreamer.cpp
index 5f9ed77d07cf..ac0c8113135a 100644
--- a/llvm/lib/Target/RISCV/MCTargetDesc/RISCVTargetStreamer.cpp
+++ b/llvm/lib/Target/RISCV/MCTargetDesc/RISCVTargetStreamer.cpp
@@ -50,6 +50,9 @@ void RISCVTargetStreamer::emitTargetAttributes(const MCSubtargetInfo &STI) {
   else
     emitAttribute(RISCVAttrs::STACK_ALIGN, RISCVAttrs::ALIGN_16);
 
+  if (STI.hasFeature(RISCV::FeatureUnalignedScalarMem))
+    emitAttribute(RISCVAttrs::UNALIGNED_ACCESS, RISCVAttrs::ALLOWED);
+
   auto ParseResult = RISCVFeatures::parseFeatureBits(
       STI.hasFeature(RISCV::Feature64Bit), STI.getFeatureBits());
   if (!ParseResult) {
diff --git a/llvm/test/CodeGen/RISCV/attributes.ll b/llvm/test/CodeGen/RISCV/attributes.ll
index 9d9f02ce52cb..0734b5a01b45 100644
--- a/llvm/test/CodeGen/RISCV/attributes.ll
+++ b/llvm/test/CodeGen/RISCV/attributes.ll
@@ -147,6 +147,14 @@
 ; RV64COMBINEINTOZKN: .attribute 5, "rv64i2p0_zbkb1p0_zbkc1p0_zbkx1p0_zkn1p0_zknd1p0_zkne1p0_zknh1p0"
 ; RV64COMBINEINTOZKS: .attribute 5, "rv64i2p0_zbkb1p0_zbkc1p0_zbkx1p0_zks1p0_zksed1p0_zksh1p0"
 
+; RUN: llc -mtriple=riscv32 %s -o - | FileCheck --check-prefix=ALIGNED %s
+; RUN: llc -mtriple=riscv64 %s -o - | FileCheck --check-prefix=ALIGNED %s
+; RUN: llc -mtriple=riscv32 -mattr=+unaligned-scalar-mem %s -o - | FileCheck --check-prefix=UNALIGNED %s
+; RUN: llc -mtriple=riscv64 -mattr=+unaligned-scalar-mem %s -o - | FileCheck --check-prefix=UNALIGNED %s
+
+; ALIGNED-NOT: .attribute 6
+; UNALIGNED: .attribute 6, 1
+
 define i32 @addi(i32 %a) {
   %1 = add i32 %a, 1
   ret i32 %1

I think not emitting Tag_RISCV_unaligned_access in the absence of +unaligned-scalar-mem (as the above patch does) is probably the best we can do, as inline assembly or handwritten .s files may well include misaligned accesses, even without +unaligned-scalar-mem.

For the patch under review, ALIGN and NOALIGN are (to my eyes) reversed in meaning. i.e. I'd expect NOALIGN to cover RUN invocations with -mattr=+unaligned-scalar-mem. But maybe there's an alternate interpretation?

[reames]

The topic of Tag_RISCV_unaligned_access handling was discussed in today's risc-v sync up call. The conclusion of that discussion is that we're going to leave it unset in this patch, and that I'm going to file an issue against the psABI to clarify a couple points in the spec. Once that clarification has been done, we may revisit setting the tag.

In the short run, we believe that leaving the tag in the unset state is the lowest risk strategy, and the least likely to break assumptions which may have been made about the meaning of the tag.

[asb]

As just discussed in the sync-up call meeting, let's defer the question of setting / not setting the attribute to an upstream discussion in riscv-elf-psabi-doc on the precise semantics of the tag. Philip kindly volunteered to kick off an issue in that repo.

I don't think setting the tag or not needs to block this landing (though the degree to which ELF tags are opt-in is perhaps something that could be further clarified in the psABI doc too).

(EDIT: Ah, Philip beat me to it!)

[reames]

Address test check comment.

Anything remaining blocking this? I know I need to file the psABI issue and will do so before pushing. Asking about further review items.

[asb]

In D126085#3540577, @reames wrote:

Address test check comment.

Anything remaining blocking this? I know I need to file the psABI issue and will do so before pushing. Asking about further review items.

Nothing remaining from me - LGTM.

[reames]

Here is the relevant psABI issue: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/283

[BaoshanPang]

How to enable this feature through clang? I tried this:

$ ./bin/clang -S t.c -O2 -mllvm -mattr=+unaligned-scalar-mem

But get:
clang (LLVM option parsing): Unknown command line argument '-mattr=+unaligned-scalar-mem'. Try: 'clang (LLVM option parsing) --help'
clang (LLVM option parsing): Did you mean '--mv67t=+unaligned-scalar-mem'?

[craig.topper]

In D126085#3556502, @BaoshanPang wrote:

How to enable this feature through clang? I tried this:

$ ./bin/clang -S t.c -O2 -mllvm -mattr=+unaligned-scalar-mem

But get:
clang (LLVM option parsing): Unknown command line argument '-mattr=+unaligned-scalar-mem'. Try: 'clang (LLVM option parsing) --help'
clang (LLVM option parsing): Did you mean '--mv67t=+unaligned-scalar-mem'?

Try -Xclang -target-feature -Xclang +unaligned-scalar-mem

[BaoshanPang]

In D126085#3556528, @craig.topper wrote:

In D126085#3556502, @BaoshanPang wrote:

How to enable this feature through clang? I tried this:

$ ./bin/clang -S t.c -O2 -mllvm -mattr=+unaligned-scalar-mem

But get:
clang (LLVM option parsing): Unknown command line argument '-mattr=+unaligned-scalar-mem'. Try: 'clang (LLVM option parsing) --help'
clang (LLVM option parsing): Did you mean '--mv67t=+unaligned-scalar-mem'?

Try -Xclang -target-feature -Xclang +unaligned-scalar-mem

Thanks for the qucik response, it works for me.

llvm/lib/Target/RISCV/RISCV.td

 foreach i = {1-31} in
     def FeatureReserveX#i :
         SubtargetFeature<"reserve-x"#i, "UserReservedRegister[RISCV::X"#i#"]",
                          "true", "Reserve X"#i>;
 
 def FeatureSaveRestore : SubtargetFeature<"save-restore", "EnableSaveRestore",
                                           "true", "Enable save/restore.">;
 
+def FeatureUnalignedScalarMem
+   : SubtargetFeature<"unaligned-scalar-mem", "EnableUnalignedScalarMem",
+                      "true", "Has reasonable performant unaligned scalar "
+                      "loads and stores">;
+
 def TuneNoDefaultUnroll
     : SubtargetFeature<"no-default-unroll", "EnableDefaultUnroll", "false",
                        "Disable default unroll preference.">;
 
 def TuneSiFive7 : SubtargetFeature<"sifive7", "RISCVProcFamily", "SiFive7",
                                    "SiFive 7-Series processors">;
 
 //===----------------------------------------------------------------------===//

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

bool RISCVTargetLowering::isMulAddWithConstProfitable(SDValue AddNode,
 
   // Default to true and let the DAGCombiner decide.
   return true;
 }
 
 bool RISCVTargetLowering::allowsMisalignedMemoryAccesses(
     EVT VT, unsigned AddrSpace, Align Alignment, MachineMemOperand::Flags Flags,
     bool *Fast) const {
-  if (!VT.isVector())
-    return false;
+  if (!VT.isVector()) {
+    if (Fast)
+      *Fast = false;
+    return Subtarget.enableUnalignedScalarMem();
+  }
 
+  // All vector implementations must support element alignment
   EVT ElemVT = VT.getVectorElementType();
   if (Alignment >= ElemVT.getStoreSize()) {
     if (Fast)
       *Fast = true;
     return true;
   }
 
   return false;
 }
 
 bool RISCVTargetLowering::splitValueIntoRegisterParts(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Val, SDValue *Parts,
     unsigned NumParts, MVT PartVT, Optional<CallingConv::ID> CC) const {
   bool IsABIRegCopy = CC.hasValue();
   EVT ValueVT = Val.getValueType();
   if (IsABIRegCopy && ValueVT == MVT::f16 && PartVT == MVT::f32) {

llvm/lib/Target/RISCV/RISCVSubtarget.h

private:
   bool HasStdExtZkt = false;
   bool HasStdExtZk = false;
   bool HasRV64 = false;
   bool IsRV32E = false;
   bool EnableLinkerRelax = false;
   bool EnableRVCHintInstrs = true;
   bool EnableDefaultUnroll = true;
   bool EnableSaveRestore = false;
+  bool EnableUnalignedScalarMem = false;
   unsigned XLen = 32;
   unsigned ZvlLen = 0;
   MVT XLenVT = MVT::i32;
   uint8_t MaxInterleaveFactor = 2;
   RISCVABI::ABI TargetABI = RISCVABI::ABI_Unknown;
   BitVector UserReservedRegister;
   RISCVFrameLowering FrameLowering;
   RISCVInstrInfo InstrInfo;
public:
   bool hasStdExtZksh() const { return HasStdExtZksh; }
   bool hasStdExtZkr() const { return HasStdExtZkr; }
   bool is64Bit() const { return HasRV64; }
   bool isRV32E() const { return IsRV32E; }
   bool enableLinkerRelax() const { return EnableLinkerRelax; }
   bool enableRVCHintInstrs() const { return EnableRVCHintInstrs; }
   bool enableDefaultUnroll() const { return EnableDefaultUnroll; }
   bool enableSaveRestore() const { return EnableSaveRestore; }
+  bool enableUnalignedScalarMem() const { return EnableUnalignedScalarMem; }
   MVT getXLenVT() const { return XLenVT; }
   unsigned getXLen() const { return XLen; }
   unsigned getFLen() const {
     if (HasStdExtD)
       return 64;
 
     if (HasStdExtF)
       return 32;

llvm/test/CodeGen/RISCV/unaligned-load-store.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -verify-machineinstrs < %s \
-; RUN:   | FileCheck -check-prefixes=BOTH,RV32I %s
+; RUN:   | FileCheck -check-prefixes=ALL,NOALIGN,RV32I %s
 ; RUN: llc -mtriple=riscv64 -verify-machineinstrs < %s \
-; RUN:   | FileCheck -check-prefixes=BOTH,RV64I %s
+; RUN:   | FileCheck -check-prefixes=ALL,NOALIGN,RV64I %s
+; RUN: llc -mtriple=riscv32 -mattr=+unaligned-scalar-mem -verify-machineinstrs < %s \
+; RUN:   | FileCheck -check-prefixes=ALL,ALIGN,ALIGN-RV32I %s
+; RUN: llc -mtriple=riscv64 -mattr=+unaligned-scalar-mem -verify-machineinstrs < %s \
+; RUN:   | FileCheck -check-prefixes=ALL,ALIGN,ALIGN-RV64I %s
 
 ; A collection of cases showing codegen for unaligned loads and stores
 
 define i8 @load_i8(i8* %p) {
-; BOTH-LABEL: load_i8:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    lb a0, 0(a0)
-; BOTH-NEXT:    ret
+; ALL-LABEL: load_i8:
+; ALL:       # %bb.0:
+; ALL-NEXT:    lb a0, 0(a0)
+; ALL-NEXT:    ret
   %res = load i8, i8* %p, align 1
   ret i8 %res
 }
 
 define i16 @load_i16(i16* %p) {
-; BOTH-LABEL: load_i16:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    lb a1, 1(a0)
-; BOTH-NEXT:    lbu a0, 0(a0)
-; BOTH-NEXT:    slli a1, a1, 8
-; BOTH-NEXT:    or a0, a1, a0
-; BOTH-NEXT:    ret
+; NOALIGN-LABEL: load_i16:
+; NOALIGN:       # %bb.0:
+; NOALIGN-NEXT:    lb a1, 1(a0)
+; NOALIGN-NEXT:    lbu a0, 0(a0)
+; NOALIGN-NEXT:    slli a1, a1, 8
+; NOALIGN-NEXT:    or a0, a1, a0
+; NOALIGN-NEXT:    ret
+;
+; ALIGN-LABEL: load_i16:
+; ALIGN:       # %bb.0:
+; ALIGN-NEXT:    lh a0, 0(a0)
+; ALIGN-NEXT:    ret
   %res = load i16, i16* %p, align 1
   ret i16 %res
 }
 
 define i24 @load_i24(i24* %p) {
-; BOTH-LABEL: load_i24:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    lbu a1, 1(a0)
-; BOTH-NEXT:    lbu a2, 0(a0)
-; BOTH-NEXT:    lb a0, 2(a0)
-; BOTH-NEXT:    slli a1, a1, 8
-; BOTH-NEXT:    or a1, a1, a2
-; BOTH-NEXT:    slli a0, a0, 16
-; BOTH-NEXT:    or a0, a1, a0
-; BOTH-NEXT:    ret
+; NOALIGN-LABEL: load_i24:
+; NOALIGN:       # %bb.0:
+; NOALIGN-NEXT:    lbu a1, 1(a0)
+; NOALIGN-NEXT:    lbu a2, 0(a0)
+; NOALIGN-NEXT:    lb a0, 2(a0)
+; NOALIGN-NEXT:    slli a1, a1, 8
+; NOALIGN-NEXT:    or a1, a1, a2
+; NOALIGN-NEXT:    slli a0, a0, 16
+; NOALIGN-NEXT:    or a0, a1, a0
+; NOALIGN-NEXT:    ret
+;
+; ALIGN-LABEL: load_i24:
+; ALIGN:       # %bb.0:
+; ALIGN-NEXT:    lb a1, 2(a0)
+; ALIGN-NEXT:    lhu a0, 0(a0)
+; ALIGN-NEXT:    slli a1, a1, 16
+; ALIGN-NEXT:    or a0, a0, a1
+; ALIGN-NEXT:    ret
   %res = load i24, i24* %p, align 1
   ret i24 %res
 }
 
 define i32 @load_i32(i32* %p) {
 ; RV32I-LABEL: load_i32:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    lbu a1, 1(a0)

 ; RV64I-NEXT:    lbu a0, 2(a0)
 ; RV64I-NEXT:    slli a1, a1, 8
 ; RV64I-NEXT:    or a1, a1, a2
 ; RV64I-NEXT:    slli a2, a3, 8
 ; RV64I-NEXT:    or a0, a2, a0
 ; RV64I-NEXT:    slli a0, a0, 16
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    ret
+;
+; ALIGN-LABEL: load_i32:
+; ALIGN:       # %bb.0:
+; ALIGN-NEXT:    lw a0, 0(a0)
+; ALIGN-NEXT:    ret
   %res = load i32, i32* %p, align 1
   ret i32 %res
 }
 
 define i64 @load_i64(i64* %p) {
 ; RV32I-LABEL: load_i64:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    lbu a1, 1(a0)

 ; RV64I-NEXT:    or a2, a2, a3
 ; RV64I-NEXT:    slli a3, a4, 8
 ; RV64I-NEXT:    or a0, a3, a0
 ; RV64I-NEXT:    slli a0, a0, 16
 ; RV64I-NEXT:    or a0, a0, a2
 ; RV64I-NEXT:    slli a0, a0, 32
 ; RV64I-NEXT:    or a0, a0, a1
 ; RV64I-NEXT:    ret
+;
+; ALIGN-RV32I-LABEL: load_i64:
+; ALIGN-RV32I:       # %bb.0:
+; ALIGN-RV32I-NEXT:    lw a2, 0(a0)
+; ALIGN-RV32I-NEXT:    lw a1, 4(a0)
+; ALIGN-RV32I-NEXT:    mv a0, a2
+; ALIGN-RV32I-NEXT:    ret
+;
+; ALIGN-RV64I-LABEL: load_i64:
+; ALIGN-RV64I:       # %bb.0:
+; ALIGN-RV64I-NEXT:    ld a0, 0(a0)
+; ALIGN-RV64I-NEXT:    ret
   %res = load i64, i64* %p, align 1
   ret i64 %res
 }
 
 define void @store_i8(i8* %p, i8 %v) {
-; BOTH-LABEL: store_i8:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    sb a1, 0(a0)
-; BOTH-NEXT:    ret
+; ALL-LABEL: store_i8:
+; ALL:       # %bb.0:
+; ALL-NEXT:    sb a1, 0(a0)
+; ALL-NEXT:    ret
   store i8 %v, i8* %p, align 1
   ret void
 }
 
 define void @store_i16(i16* %p, i16 %v) {
-; BOTH-LABEL: store_i16:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    sb a1, 0(a0)
-; BOTH-NEXT:    srli a1, a1, 8
-; BOTH-NEXT:    sb a1, 1(a0)
-; BOTH-NEXT:    ret
+; NOALIGN-LABEL: store_i16:
+; NOALIGN:       # %bb.0:
+; NOALIGN-NEXT:    sb a1, 0(a0)
+; NOALIGN-NEXT:    srli a1, a1, 8
+; NOALIGN-NEXT:    sb a1, 1(a0)
+; NOALIGN-NEXT:    ret
+;
+; ALIGN-LABEL: store_i16:
+; ALIGN:       # %bb.0:
+; ALIGN-NEXT:    sh a1, 0(a0)
+; ALIGN-NEXT:    ret
   store i16 %v, i16* %p, align 1
   ret void
 }
 
 define void @store_i24(i24* %p, i24 %v) {
-; BOTH-LABEL: store_i24:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    sb a1, 0(a0)
-; BOTH-NEXT:    srli a2, a1, 8
-; BOTH-NEXT:    sb a2, 1(a0)
-; BOTH-NEXT:    srli a1, a1, 16
-; BOTH-NEXT:    sb a1, 2(a0)
-; BOTH-NEXT:    ret
+; NOALIGN-LABEL: store_i24:
+; NOALIGN:       # %bb.0:
+; NOALIGN-NEXT:    sb a1, 0(a0)
+; NOALIGN-NEXT:    srli a2, a1, 8
+; NOALIGN-NEXT:    sb a2, 1(a0)
+; NOALIGN-NEXT:    srli a1, a1, 16
+; NOALIGN-NEXT:    sb a1, 2(a0)
+; NOALIGN-NEXT:    ret
+;
+; ALIGN-LABEL: store_i24:
+; ALIGN:       # %bb.0:
+; ALIGN-NEXT:    sh a1, 0(a0)
+; ALIGN-NEXT:    srli a1, a1, 16
+; ALIGN-NEXT:    sb a1, 2(a0)
+; ALIGN-NEXT:    ret
   store i24 %v, i24* %p, align 1
   ret void
 }
 
 define void @store_i32(i32* %p, i32 %v) {
-; BOTH-LABEL: store_i32:
-; BOTH:       # %bb.0:
-; BOTH-NEXT:    sb a1, 0(a0)
-; BOTH-NEXT:    srli a2, a1, 24
-; BOTH-NEXT:    sb a2, 3(a0)
-; BOTH-NEXT:    srli a2, a1, 16
-; BOTH-NEXT:    sb a2, 2(a0)
-; BOTH-NEXT:    srli a1, a1, 8
-; BOTH-NEXT:    sb a1, 1(a0)
-; BOTH-NEXT:    ret
+; NOALIGN-LABEL: store_i32:
+; NOALIGN:       # %bb.0:
+; NOALIGN-NEXT:    sb a1, 0(a0)
+; NOALIGN-NEXT:    srli a2, a1, 24
+; NOALIGN-NEXT:    sb a2, 3(a0)
+; NOALIGN-NEXT:    srli a2, a1, 16
+; NOALIGN-NEXT:    sb a2, 2(a0)
+; NOALIGN-NEXT:    srli a1, a1, 8
+; NOALIGN-NEXT:    sb a1, 1(a0)
+; NOALIGN-NEXT:    ret
+;
+; ALIGN-LABEL: store_i32:
+; ALIGN:       # %bb.0:
+; ALIGN-NEXT:    sw a1, 0(a0)
+; ALIGN-NEXT:    ret
   store i32 %v, i32* %p, align 1
   ret void
 }
 
 define void @store_i64(i64* %p, i64 %v) {
 ; RV32I-LABEL: store_i64:
 ; RV32I:       # %bb.0:
 ; RV32I-NEXT:    sb a2, 4(a0)

 ; RV64I-NEXT:    sb a2, 4(a0)
 ; RV64I-NEXT:    srli a2, a1, 24
 ; RV64I-NEXT:    sb a2, 3(a0)
 ; RV64I-NEXT:    srli a2, a1, 16
 ; RV64I-NEXT:    sb a2, 2(a0)
 ; RV64I-NEXT:    srli a1, a1, 8
 ; RV64I-NEXT:    sb a1, 1(a0)
 ; RV64I-NEXT:    ret
+;
+; ALIGN-RV32I-LABEL: store_i64:
+; ALIGN-RV32I:       # %bb.0:
+; ALIGN-RV32I-NEXT:    sw a2, 4(a0)
+; ALIGN-RV32I-NEXT:    sw a1, 0(a0)
+; ALIGN-RV32I-NEXT:    ret
+;
+; ALIGN-RV64I-LABEL: store_i64:
+; ALIGN-RV64I:       # %bb.0:
+; ALIGN-RV64I-NEXT:    sd a1, 0(a0)
+; ALIGN-RV64I-NEXT:    ret
   store i64 %v, i64* %p, align 1
   ret void
 }