[LoongArch] Align functions and loops better according to uarch

[xen0n]

Summary

The LA464 micro-architecture is very sensitive to alignment of hot code,
with performance variation of up to ~12% in the go1 benchmark suite of
the Go language (as observed by me during my work on the Go loong64
port).
Manual alignment of certain loops and automatic alignment of loop heads
helps a lot there, by reducing much of the random variation and
generally increasing performance, so we naturally want to do the same
here.

Practically, LA464 is the only LoongArch micro-architecture in wide use,
and we are currently supporting just that. The first "4" in "LA464"
stands for "4-issue", in particular its instruction fetch and decode
stages are 4-wide; so functions and branch targets should be preferably
aligned to at least 16 bytes for best throughput.

The Loongson team has benchmarked various combinations of function,
loop, and branch target alignments with GCC.
The results
show that "16-byte label alignment together with 32-byte function
alignment gives best results in terms of SPEC score". A "label" in GCC
means a branch target; while we don't currently align branch targets,
we do align loops, so in this patch we default to 32-byte function
alignment and 16-byte loop alignment.

[xry111]

Hmm, it seems to me setMaxBytesForAlignment sets "the maximum amount of padding allowed for aligning the BB", not the max loop size. Or am I missing something?

I'll do the similar thing for GCC but we've already missed GCC 13 deadline :(.

[xen0n]

use STI consistently and fix one grammatical nit in comment

[xen0n]

In D148622#4277173, @xry111 wrote:

Hmm, it seems to me setMaxBytesForAlignment sets "the maximum amount of padding allowed for aligning the BB", not the max loop size. Or am I missing something?

I simply copied from AArch64 for that part. Maybe I should mention this in the commit message...

I'll do the similar thing for GCC but we've already missed GCC 13 deadline :(.

No worries, things happen, and it's not like everyone's gonna burn 2x more CPU cycles without such changes. I only discovered this because of surprising go1 benchmark results over and over again, getting 13% performance hits for a change that theoretically cannot increase dynamic instruction counts; it actually proves such fluctuations are largely unnoticed in everyday use.

[xry111]

In D148622#4277187, @xen0n wrote:

In D148622#4277173, @xry111 wrote:

Hmm, it seems to me setMaxBytesForAlignment sets "the maximum amount of padding allowed for aligning the BB", not the max loop size. Or am I missing something?

I simply copied from AArch64 for that part. Maybe I should mention this in the commit message...

I mean, I didn't see how the change only sets loop alignments for tight loops. Maybe I'm missing something, but with GCC there seems no way to limit -falign-loops only for tight loops.

[SixWeining]

In D148622#4277173, @xry111 wrote:

Hmm, it seems to me setMaxBytesForAlignment sets "the maximum amount of padding allowed for aligning the BB", not the max loop size. Or am I missing something?

I agree. It should be the third expression described in https://ftp.gnu.org/old-gnu/Manuals/gas-2.9.1/html_chapter/as_7.html#SEC113.

I mean, I didn't see how the change only sets loop alignments for tight loops. Maybe I'm missing something, but with GCC there seems no way to limit -falign-loops only for tight loops.

I have the same concern.

BTW, @xen0n Could you please add some dedicate tests for function alignment settings. For example:

; RUN: llc --mtriple=loongarch64 %s -o - | FileCheck %s
; RUN: llc --mtriple=loongarch64 -mcpu=la464 %s -o - | FileCheck %s

define void @foo() nounwind {
;CHECK: .p2align 4
;CHECK: foo:
entry:
  ret void
}

[xen0n]

In D148622#4279228, @SixWeining wrote:

In D148622#4277173, @xry111 wrote:

Hmm, it seems to me setMaxBytesForAlignment sets "the maximum amount of padding allowed for aligning the BB", not the max loop size. Or am I missing something?

I agree. It should be the third expression described in https://ftp.gnu.org/old-gnu/Manuals/gas-2.9.1/html_chapter/as_7.html#SEC113.

I mean, I didn't see how the change only sets loop alignments for tight loops. Maybe I'm missing something, but with GCC there seems no way to limit -falign-loops only for tight loops.

I have the same concern.

BTW, @xen0n Could you please add some dedicate tests for function alignment settings. For example:

; RUN: llc --mtriple=loongarch64 %s -o - | FileCheck %s
; RUN: llc --mtriple=loongarch64 -mcpu=la464 %s -o - | FileCheck %s

define void @foo() nounwind {
;CHECK: .p2align 4
;CHECK: foo:
entry:
  ret void
}

I've checked the logic behind MaxBytesForAlignment and it seems to be the case, sorry for the confusion. I'll fix in the next revision. Thank you both.

[xry111]

Lulu said a benchmark should be performed for such a change (via gcc-patches) and I agree, so please hold the change until a benchmark is done.

@SixWeining I guess you can collaborate with Lulu internally for the benchmark. I don't have an access to SPEC CPU either.

[SixWeining]

In D148622#4279817, @xry111 wrote:

Lulu said a benchmark should be performed for such a change (via gcc-patches) and I agree, so please hold the change until a benchmark is done.

@SixWeining I guess you can collaborate with Lulu internally for the benchmark. I don't have an access to SPEC CPU either.

Thanks. Let me talk with her offline.

[SixWeining]

llvm/lib/Target/LoongArch/LoongArchSubtarget.cpp

38

Should we pass CPU or TuneCPU to this function and use it to set different numbers?

[xen0n]

Rebased and added a dedicated test case for this change.

Also changed the preferred function alignment to 32 bytes per Loongson benchmark results.

[SixWeining]

llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp

199

Almost all parts of this constructor use the member variable Subtarget but not the parameter STI except line 187 which I will change it later.

llvm/lib/Target/LoongArch/LoongArchSubtarget.h

55

Is this initial value useful? PrefFunctionAlignment and PrefLoopAlignment do not have initial values?

[xen0n]

llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp

199

Okay I'll change to Subtarget for consistency with the local style shortly. (Although IMO local variables/arguments should be preferred over instance states which is more global, it's best done separately.)

llvm/lib/Target/LoongArch/LoongArchSubtarget.cpp

38

For now I've added the TuneCPU argument for smaller future diffs when we get to add more uarch data later.

llvm/lib/Target/LoongArch/LoongArchSubtarget.h

55

This is similar to the way AArch64 does it (that I obviously referred to). Apparently the default ctor of Align means "byte alignment" and zero MaxBytesForAlignment means disabling that feature.

[xen0n]

Move default alignment setting to member declaration and use Subtarget consistently during LoongArchISelLowering init.

[xen0n]

rebase

[SixWeining]

LGTM. Thanks.

llvm/lib/Target/LoongArch/LoongArchISelLowering.cpp

LoongArchTargetLowering::LoongArchTargetLowering(const TargetMachine &TM,
   setBooleanContents(ZeroOrOneBooleanContent);
 
   setMaxAtomicSizeInBitsSupported(Subtarget.getGRLen());
 
   setMinCmpXchgSizeInBits(32);
 
   // Function alignments.
   setMinFunctionAlignment(Align(4));
+  // Set preferred alignments.
+  setPrefFunctionAlignment(STI.getPrefFunctionAlignment());
+  setPrefLoopAlignment(STI.getPrefLoopAlignment());
+  setMaxBytesForAlignment(STI.getMaxBytesForAlignment());
 
   setTargetDAGCombine(ISD::AND);
   setTargetDAGCombine(ISD::OR);
   setTargetDAGCombine(ISD::SRL);
 }
 
 bool LoongArchTargetLowering::isOffsetFoldingLegal(
     const GlobalAddressSDNode *GA) const {

llvm/lib/Target/LoongArch/LoongArchSubtarget.h

class LoongArchSubtarget : public LoongArchGenSubtargetInfo {
   unsigned GRLen = 32;
   MVT GRLenVT = MVT::i32;
   LoongArchABI::ABI TargetABI = LoongArchABI::ABI_Unknown;
   LoongArchFrameLowering FrameLowering;
   LoongArchInstrInfo InstrInfo;
   LoongArchRegisterInfo RegInfo;
   LoongArchTargetLowering TLInfo;
   SelectionDAGTargetInfo TSInfo;
+  Align PrefFunctionAlignment;
+  Align PrefLoopAlignment;
+  unsigned MaxBytesForAlignment = 0;
 
   /// Initializes using the passed in CPU and feature strings so that we can
   /// use initializer lists for subtarget initialization.
   LoongArchSubtarget &initializeSubtargetDependencies(const Triple &TT,
                                                       StringRef CPU,
                                                       StringRef TuneCPU,
                                                       StringRef FS,
                                                       StringRef ABIName);
 
+  /// Initialize properties based on the selected processor family.
+  void initializeProperties(StringRef TuneCPU);
+
 public:
   // Initializes the data members to match that of the specified triple.
   LoongArchSubtarget(const Triple &TT, StringRef CPU, StringRef TuneCPU,
                      StringRef FS, StringRef ABIName, const TargetMachine &TM);
 
   // Parses features string setting specified subtarget options. The
   // definition of this function is auto-generated by tblgen.
   void ParseSubtargetFeatures(StringRef CPU, StringRef TuneCPU, StringRef FS);
public:
   bool hasExtLVZ() const { return HasExtLVZ; }
   bool hasExtLBT() const { return HasExtLBT; }
   bool hasLaGlobalWithPcrel() const { return HasLaGlobalWithPcrel; }
   bool hasLaGlobalWithAbs() const { return HasLaGlobalWithAbs; }
   bool hasLaLocalWithAbs() const { return HasLaLocalWithAbs; }
   MVT getGRLenVT() const { return GRLenVT; }
   unsigned getGRLen() const { return GRLen; }
   LoongArchABI::ABI getTargetABI() const { return TargetABI; }
+  Align getPrefFunctionAlignment() const { return PrefFunctionAlignment; }
+  Align getPrefLoopAlignment() const { return PrefLoopAlignment; }
+  unsigned getMaxBytesForAlignment() const { return MaxBytesForAlignment; }
 };
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_LOONGARCH_LOONGARCHSUBTARGET_H

llvm/lib/Target/LoongArch/LoongArchSubtarget.cpp

LoongArchSubtarget &LoongArchSubtarget::initializeSubtargetDependencies(
   bool Is64Bit = TT.isArch64Bit();
   if (CPU.empty() || CPU == "generic")
     CPU = Is64Bit ? "generic-la64" : "generic-la32";
 
   if (TuneCPU.empty())
     TuneCPU = CPU;
 
   ParseSubtargetFeatures(CPU, TuneCPU, FS);
+  initializeProperties(TuneCPU);
   if (Is64Bit) {
     GRLenVT = MVT::i64;
     GRLen = 64;
   }
 
   if (HasLA32 == HasLA64)
     report_fatal_error("Please use one feature of 32bit and 64bit.");
 
   if (Is64Bit && HasLA32)
     report_fatal_error("Feature 32bit should be used for loongarch32 target.");
 
   if (!Is64Bit && HasLA64)
     report_fatal_error("Feature 64bit should be used for loongarch64 target.");
 
   TargetABI = LoongArchABI::computeTargetABI(TT, ABIName);
 
   return *this;
 }
 
+void LoongArchSubtarget::initializeProperties(StringRef TuneCPU) {
+  // Initialize CPU specific properties. We should add a tablegen feature for
+  // this in the future so we can specify it together with the subtarget
+  // features.
+
+  // TODO: Make use of TuneCPU once we support optimizing for more uarchs.
+
+  // Default to the alignment settings empirically confirmed to perform best
+  // on LA464, with 4-wide instruction fetch and decode stages.
+  //
+  // We default to such higher-than-minimum alignments because we assume that:
+  //
+  // * future general-purpose LoongArch cores are likely to have issue widths
+  //   equal to or wider than 4,
+  // * instruction sequences best for LA464 should not pessimize other future
+  //   uarchs, and
+  // * narrower cores would not suffer much (aside from slightly increased
+  //   ICache footprint maybe), compared to the gains everywhere else.
+  PrefFunctionAlignment = Align(32);
+  PrefLoopAlignment = Align(16);
+  MaxBytesForAlignment = 16;
+}
+
 LoongArchSubtarget::LoongArchSubtarget(const Triple &TT, StringRef CPU,
                                        StringRef TuneCPU, StringRef FS,
                                        StringRef ABIName,
                                        const TargetMachine &TM)
     : LoongArchGenSubtargetInfo(TT, CPU, TuneCPU, FS),
       FrameLowering(
           initializeSubtargetDependencies(TT, CPU, TuneCPU, FS, ABIName)),
       InstrInfo(*this), RegInfo(getHwMode()), TLInfo(TM, *this) {}

llvm/test/CodeGen/LoongArch/atomicrmw-uinc-udec-wrap.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=loongarch64 < %s | FileCheck --check-prefix=LA64 %s
 
 define i8 @atomicrmw_uinc_wrap_i8(ptr %ptr, i8 %val) {
 ; LA64-LABEL: atomicrmw_uinc_wrap_i8:
 ; LA64:       # %bb.0:
 ; LA64-NEXT:    addi.w $a2, $zero, -4
 ; LA64-NEXT:    and $a2, $a0, $a2
 ; LA64-NEXT:    ld.w $a3, $a2, 0
 ; LA64-NEXT:    slli.d $a0, $a0, 3
 ; LA64-NEXT:    ori $a4, $zero, 255
 ; LA64-NEXT:    sll.w $a4, $a4, $a0
 ; LA64-NEXT:    andi $a0, $a0, 24
 ; LA64-NEXT:    nor $a4, $a4, $zero
 ; LA64-NEXT:    andi $a1, $a1, 255
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB0_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB0_3 Depth 2
 ; LA64-NEXT:    srl.w $a5, $a3, $a0
 ; LA64-NEXT:    andi $a6, $a5, 255
 ; LA64-NEXT:    sltu $a6, $a6, $a1
 ; LA64-NEXT:    addi.d $a5, $a5, 1
 ; LA64-NEXT:    xori $a6, $a6, 1

 ; LA64-NEXT:    ld.w $a3, $a2, 0
 ; LA64-NEXT:    slli.d $a0, $a0, 3
 ; LA64-NEXT:    lu12i.w $a4, 15
 ; LA64-NEXT:    ori $a4, $a4, 4095
 ; LA64-NEXT:    sll.w $a4, $a4, $a0
 ; LA64-NEXT:    andi $a0, $a0, 24
 ; LA64-NEXT:    nor $a4, $a4, $zero
 ; LA64-NEXT:    bstrpick.d $a1, $a1, 15, 0
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB1_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB1_3 Depth 2
 ; LA64-NEXT:    srl.w $a5, $a3, $a0
 ; LA64-NEXT:    bstrpick.d $a6, $a5, 15, 0
 ; LA64-NEXT:    sltu $a6, $a6, $a1
 ; LA64-NEXT:    addi.d $a5, $a5, 1
 ; LA64-NEXT:    xori $a6, $a6, 1
; LA64-NEXT:    ret
   ret i16 %result
 }
 
 define i32 @atomicrmw_uinc_wrap_i32(ptr %ptr, i32 %val) {
 ; LA64-LABEL: atomicrmw_uinc_wrap_i32:
 ; LA64:       # %bb.0:
 ; LA64-NEXT:    ld.w $a3, $a0, 0
 ; LA64-NEXT:    bstrpick.d $a2, $a1, 31, 0
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB2_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB2_3 Depth 2
 ; LA64-NEXT:    bstrpick.d $a1, $a3, 31, 0
 ; LA64-NEXT:    sltu $a1, $a1, $a2
 ; LA64-NEXT:    xori $a1, $a1, 1
 ; LA64-NEXT:    addi.d $a4, $a3, 1
 ; LA64-NEXT:    masknez $a4, $a4, $a1
; LA64-NEXT:    ret
   %result = atomicrmw uinc_wrap ptr %ptr, i32 %val seq_cst
   ret i32 %result
 }
 
 define i64 @atomicrmw_uinc_wrap_i64(ptr %ptr, i64 %val) {
 ; LA64-LABEL: atomicrmw_uinc_wrap_i64:
 ; LA64:       # %bb.0:
 ; LA64-NEXT:    ld.d $a2, $a0, 0
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB3_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB3_3 Depth 2
 ; LA64-NEXT:    move $a3, $a2
 ; LA64-NEXT:    sltu $a2, $a2, $a1
 ; LA64-NEXT:    xori $a2, $a2, 1
 ; LA64-NEXT:    addi.d $a4, $a3, 1
 ; LA64-NEXT:    masknez $a4, $a4, $a2

 ; LA64-NEXT:    and $a2, $a0, $a2
 ; LA64-NEXT:    ld.w $a3, $a2, 0
 ; LA64-NEXT:    slli.d $a0, $a0, 3
 ; LA64-NEXT:    ori $a4, $zero, 255
 ; LA64-NEXT:    sll.w $a4, $a4, $a0
 ; LA64-NEXT:    andi $a0, $a0, 24
 ; LA64-NEXT:    nor $a4, $a4, $zero
 ; LA64-NEXT:    andi $a5, $a1, 255
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB4_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB4_3 Depth 2
 ; LA64-NEXT:    srl.w $a6, $a3, $a0
 ; LA64-NEXT:    andi $a7, $a6, 255
 ; LA64-NEXT:    sltu $t0, $a5, $a7
 ; LA64-NEXT:    addi.d $a6, $a6, -1
 ; LA64-NEXT:    masknez $a6, $a6, $t0

 ; LA64-NEXT:    ld.w $a3, $a2, 0
 ; LA64-NEXT:    slli.d $a0, $a0, 3
 ; LA64-NEXT:    lu12i.w $a4, 15
 ; LA64-NEXT:    ori $a4, $a4, 4095
 ; LA64-NEXT:    sll.w $a4, $a4, $a0
 ; LA64-NEXT:    andi $a0, $a0, 24
 ; LA64-NEXT:    nor $a4, $a4, $zero
 ; LA64-NEXT:    bstrpick.d $a5, $a1, 15, 0
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB5_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB5_3 Depth 2
 ; LA64-NEXT:    srl.w $a6, $a3, $a0
 ; LA64-NEXT:    bstrpick.d $a7, $a6, 15, 0
 ; LA64-NEXT:    sltu $t0, $a5, $a7
 ; LA64-NEXT:    addi.d $a6, $a6, -1
 ; LA64-NEXT:    masknez $a6, $a6, $t0
; LA64-NEXT:    ret
   ret i16 %result
 }
 
 define i32 @atomicrmw_udec_wrap_i32(ptr %ptr, i32 %val) {
 ; LA64-LABEL: atomicrmw_udec_wrap_i32:
 ; LA64:       # %bb.0:
 ; LA64-NEXT:    ld.w $a4, $a0, 0
 ; LA64-NEXT:    bstrpick.d $a3, $a1, 31, 0
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB6_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB6_3 Depth 2
 ; LA64-NEXT:    bstrpick.d $a2, $a4, 31, 0
 ; LA64-NEXT:    sltu $a5, $a3, $a2
 ; LA64-NEXT:    addi.d $a6, $a4, -1
 ; LA64-NEXT:    masknez $a6, $a6, $a5
 ; LA64-NEXT:    maskeqz $a5, $a1, $a5
; LA64-NEXT:    ret
   %result = atomicrmw udec_wrap ptr %ptr, i32 %val seq_cst
   ret i32 %result
 }
 
 define i64 @atomicrmw_udec_wrap_i64(ptr %ptr, i64 %val) {
 ; LA64-LABEL: atomicrmw_udec_wrap_i64:
 ; LA64:       # %bb.0:
 ; LA64-NEXT:    ld.d $a2, $a0, 0
+; LA64-NEXT:    .p2align 4, , 16
 ; LA64-NEXT:  .LBB7_1: # %atomicrmw.start
 ; LA64-NEXT:    # =>This Loop Header: Depth=1
 ; LA64-NEXT:    # Child Loop BB7_3 Depth 2
 ; LA64-NEXT:    move $a3, $a2
 ; LA64-NEXT:    sltu $a2, $a1, $a2
 ; LA64-NEXT:    addi.d $a4, $a3, -1
 ; LA64-NEXT:    masknez $a4, $a4, $a2
 ; LA64-NEXT:    maskeqz $a2, $a1, $a2

llvm/test/CodeGen/LoongArch/ir-instruction/atomicrmw-fp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=loongarch64 --mattr=+f,-d < %s | FileCheck %s --check-prefix=LA64F
 ; RUN: llc --mtriple=loongarch64 --mattr=+d < %s | FileCheck %s --check-prefix=LA64D
 
 ;; Add more test cases after supporting different AtomicOrdering.
 
 define float @float_fadd_acquire(ptr %p) nounwind {
 ; LA64F-LABEL: float_fadd_acquire:
 ; LA64F:       # %bb.0:
 ; LA64F-NEXT:    fld.s $fa0, $a0, 0
 ; LA64F-NEXT:    addi.w $a1, $zero, 1
 ; LA64F-NEXT:    movgr2fr.w $fa1, $a1
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB0_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Loop Header: Depth=1
 ; LA64F-NEXT:    # Child Loop BB0_3 Depth 2
 ; LA64F-NEXT:    ffint.s.w $fa2, $fa1
 ; LA64F-NEXT:    fadd.s $fa2, $fa0, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a2, $fa0
 ; LA64F-NEXT:  .LBB0_3: # %atomicrmw.start

 ; LA64F-NEXT:  # %bb.2: # %atomicrmw.end
 ; LA64F-NEXT:    ret
 ;
 ; LA64D-LABEL: float_fadd_acquire:
 ; LA64D:       # %bb.0:
 ; LA64D-NEXT:    fld.s $fa0, $a0, 0
 ; LA64D-NEXT:    addi.w $a1, $zero, 1
 ; LA64D-NEXT:    movgr2fr.w $fa1, $a1
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB0_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Loop Header: Depth=1
 ; LA64D-NEXT:    # Child Loop BB0_3 Depth 2
 ; LA64D-NEXT:    ffint.s.w $fa2, $fa1
 ; LA64D-NEXT:    fadd.s $fa2, $fa0, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a2, $fa0
 ; LA64D-NEXT:  .LBB0_3: # %atomicrmw.start

 
 define float @float_fsub_acquire(ptr %p) nounwind {
 ; LA64F-LABEL: float_fsub_acquire:
 ; LA64F:       # %bb.0:
 ; LA64F-NEXT:    fld.s $fa0, $a0, 0
 ; LA64F-NEXT:    pcalau12i $a1, %pc_hi20(.LCPI1_0)
 ; LA64F-NEXT:    addi.d $a1, $a1, %pc_lo12(.LCPI1_0)
 ; LA64F-NEXT:    fld.s $fa1, $a1, 0
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB1_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Loop Header: Depth=1
 ; LA64F-NEXT:    # Child Loop BB1_3 Depth 2
 ; LA64F-NEXT:    fadd.s $fa2, $fa0, $fa1
 ; LA64F-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a2, $fa0
 ; LA64F-NEXT:  .LBB1_3: # %atomicrmw.start
 ; LA64F-NEXT:    # Parent Loop BB1_1 Depth=1

 ; LA64F-NEXT:    ret
 ;
 ; LA64D-LABEL: float_fsub_acquire:
 ; LA64D:       # %bb.0:
 ; LA64D-NEXT:    fld.s $fa0, $a0, 0
 ; LA64D-NEXT:    pcalau12i $a1, %pc_hi20(.LCPI1_0)
 ; LA64D-NEXT:    addi.d $a1, $a1, %pc_lo12(.LCPI1_0)
 ; LA64D-NEXT:    fld.s $fa1, $a1, 0
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB1_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Loop Header: Depth=1
 ; LA64D-NEXT:    # Child Loop BB1_3 Depth 2
 ; LA64D-NEXT:    fadd.s $fa2, $fa0, $fa1
 ; LA64D-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a2, $fa0
 ; LA64D-NEXT:  .LBB1_3: # %atomicrmw.start
 ; LA64D-NEXT:    # Parent Loop BB1_1 Depth=1

 }
 
 define float @float_fmin_acquire(ptr %p) nounwind {
 ; LA64F-LABEL: float_fmin_acquire:
 ; LA64F:       # %bb.0:
 ; LA64F-NEXT:    fld.s $fa0, $a0, 0
 ; LA64F-NEXT:    addi.w $a1, $zero, 1
 ; LA64F-NEXT:    movgr2fr.w $fa1, $a1
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB2_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Loop Header: Depth=1
 ; LA64F-NEXT:    # Child Loop BB2_3 Depth 2
 ; LA64F-NEXT:    ffint.s.w $fa2, $fa1
 ; LA64F-NEXT:    fmax.s $fa3, $fa0, $fa0
 ; LA64F-NEXT:    fmin.s $fa2, $fa3, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a2, $fa0

 ; LA64F-NEXT:  # %bb.2: # %atomicrmw.end
 ; LA64F-NEXT:    ret
 ;
 ; LA64D-LABEL: float_fmin_acquire:
 ; LA64D:       # %bb.0:
 ; LA64D-NEXT:    fld.s $fa0, $a0, 0
 ; LA64D-NEXT:    addi.w $a1, $zero, 1
 ; LA64D-NEXT:    movgr2fr.w $fa1, $a1
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB2_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Loop Header: Depth=1
 ; LA64D-NEXT:    # Child Loop BB2_3 Depth 2
 ; LA64D-NEXT:    ffint.s.w $fa2, $fa1
 ; LA64D-NEXT:    fmax.s $fa3, $fa0, $fa0
 ; LA64D-NEXT:    fmin.s $fa2, $fa3, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a2, $fa0

 }
 
 define float @float_fmax_acquire(ptr %p) nounwind {
 ; LA64F-LABEL: float_fmax_acquire:
 ; LA64F:       # %bb.0:
 ; LA64F-NEXT:    fld.s $fa0, $a0, 0
 ; LA64F-NEXT:    addi.w $a1, $zero, 1
 ; LA64F-NEXT:    movgr2fr.w $fa1, $a1
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB3_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Loop Header: Depth=1
 ; LA64F-NEXT:    # Child Loop BB3_3 Depth 2
 ; LA64F-NEXT:    ffint.s.w $fa2, $fa1
 ; LA64F-NEXT:    fmax.s $fa3, $fa0, $fa0
 ; LA64F-NEXT:    fmax.s $fa2, $fa3, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64F-NEXT:    movfr2gr.s $a2, $fa0

 ; LA64F-NEXT:  # %bb.2: # %atomicrmw.end
 ; LA64F-NEXT:    ret
 ;
 ; LA64D-LABEL: float_fmax_acquire:
 ; LA64D:       # %bb.0:
 ; LA64D-NEXT:    fld.s $fa0, $a0, 0
 ; LA64D-NEXT:    addi.w $a1, $zero, 1
 ; LA64D-NEXT:    movgr2fr.w $fa1, $a1
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB3_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Loop Header: Depth=1
 ; LA64D-NEXT:    # Child Loop BB3_3 Depth 2
 ; LA64D-NEXT:    ffint.s.w $fa2, $fa1
 ; LA64D-NEXT:    fmax.s $fa3, $fa0, $fa0
 ; LA64D-NEXT:    fmax.s $fa2, $fa3, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a1, $fa2
 ; LA64D-NEXT:    movfr2gr.s $a2, $fa0

 ; LA64F-NEXT:    ld.wu $a0, $a0, 0
 ; LA64F-NEXT:    ld.wu $a1, $fp, 4
 ; LA64F-NEXT:    slli.d $a1, $a1, 32
 ; LA64F-NEXT:    or $a0, $a1, $a0
 ; LA64F-NEXT:    ori $s0, $zero, 8
 ; LA64F-NEXT:    addi.d $s1, $sp, 8
 ; LA64F-NEXT:    addi.d $s2, $sp, 0
 ; LA64F-NEXT:    ori $s3, $zero, 2
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB4_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64F-NEXT:    st.d $a0, $sp, 8
 ; LA64F-NEXT:    lu52i.d $a1, $zero, 1023
 ; LA64F-NEXT:    bl %plt(__adddf3)
 ; LA64F-NEXT:    st.d $a0, $sp, 0
 ; LA64F-NEXT:    move $a0, $s0
 ; LA64F-NEXT:    move $a1, $fp

 ; LA64D-NEXT:    or $a0, $a1, $a0
 ; LA64D-NEXT:    movgr2fr.d $fa0, $a0
 ; LA64D-NEXT:    addi.d $a0, $zero, 1
 ; LA64D-NEXT:    movgr2fr.d $fs0, $a0
 ; LA64D-NEXT:    ori $s0, $zero, 8
 ; LA64D-NEXT:    addi.d $s1, $sp, 16
 ; LA64D-NEXT:    addi.d $s2, $sp, 8
 ; LA64D-NEXT:    ori $s3, $zero, 2
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB4_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 16
 ; LA64D-NEXT:    ffint.d.l $fa1, $fs0
 ; LA64D-NEXT:    fadd.d $fa0, $fa0, $fa1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 8
 ; LA64D-NEXT:    move $a0, $s0
 ; LA64D-NEXT:    move $a1, $fp

 ; LA64F-NEXT:    ld.wu $a0, $a0, 0
 ; LA64F-NEXT:    ld.wu $a1, $fp, 4
 ; LA64F-NEXT:    slli.d $a1, $a1, 32
 ; LA64F-NEXT:    or $a0, $a1, $a0
 ; LA64F-NEXT:    ori $s0, $zero, 8
 ; LA64F-NEXT:    addi.d $s1, $sp, 8
 ; LA64F-NEXT:    addi.d $s2, $sp, 0
 ; LA64F-NEXT:    ori $s3, $zero, 2
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB5_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64F-NEXT:    st.d $a0, $sp, 8
 ; LA64F-NEXT:    lu52i.d $a1, $zero, -1025
 ; LA64F-NEXT:    bl %plt(__adddf3)
 ; LA64F-NEXT:    st.d $a0, $sp, 0
 ; LA64F-NEXT:    move $a0, $s0
 ; LA64F-NEXT:    move $a1, $fp

 ; LA64D-NEXT:    movgr2fr.d $fa0, $a0
 ; LA64D-NEXT:    pcalau12i $a0, %pc_hi20(.LCPI5_0)
 ; LA64D-NEXT:    addi.d $a0, $a0, %pc_lo12(.LCPI5_0)
 ; LA64D-NEXT:    fld.d $fs0, $a0, 0
 ; LA64D-NEXT:    ori $s0, $zero, 8
 ; LA64D-NEXT:    addi.d $s1, $sp, 16
 ; LA64D-NEXT:    addi.d $s2, $sp, 8
 ; LA64D-NEXT:    ori $s3, $zero, 2
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB5_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 16
 ; LA64D-NEXT:    fadd.d $fa0, $fa0, $fs0
 ; LA64D-NEXT:    fst.d $fa0, $sp, 8
 ; LA64D-NEXT:    move $a0, $s0
 ; LA64D-NEXT:    move $a1, $fp
 ; LA64D-NEXT:    move $a2, $s1

 ; LA64F-NEXT:    ld.wu $a0, $a0, 0
 ; LA64F-NEXT:    ld.wu $a1, $fp, 4
 ; LA64F-NEXT:    slli.d $a1, $a1, 32
 ; LA64F-NEXT:    or $a0, $a1, $a0
 ; LA64F-NEXT:    ori $s0, $zero, 8
 ; LA64F-NEXT:    addi.d $s1, $sp, 8
 ; LA64F-NEXT:    addi.d $s2, $sp, 0
 ; LA64F-NEXT:    ori $s3, $zero, 2
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB6_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64F-NEXT:    st.d $a0, $sp, 8
 ; LA64F-NEXT:    lu52i.d $a1, $zero, 1023
 ; LA64F-NEXT:    bl %plt(fmin)
 ; LA64F-NEXT:    st.d $a0, $sp, 0
 ; LA64F-NEXT:    move $a0, $s0
 ; LA64F-NEXT:    move $a1, $fp

 ; LA64D-NEXT:    or $a0, $a1, $a0
 ; LA64D-NEXT:    movgr2fr.d $fa0, $a0
 ; LA64D-NEXT:    addi.d $a0, $zero, 1
 ; LA64D-NEXT:    movgr2fr.d $fs0, $a0
 ; LA64D-NEXT:    ori $s0, $zero, 8
 ; LA64D-NEXT:    addi.d $s1, $sp, 16
 ; LA64D-NEXT:    addi.d $s2, $sp, 8
 ; LA64D-NEXT:    ori $s3, $zero, 2
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB6_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 16
 ; LA64D-NEXT:    fmax.d $fa0, $fa0, $fa0
 ; LA64D-NEXT:    ffint.d.l $fa1, $fs0
 ; LA64D-NEXT:    fmin.d $fa0, $fa0, $fa1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 8
 ; LA64D-NEXT:    move $a0, $s0

 ; LA64F-NEXT:    ld.wu $a0, $a0, 0
 ; LA64F-NEXT:    ld.wu $a1, $fp, 4
 ; LA64F-NEXT:    slli.d $a1, $a1, 32
 ; LA64F-NEXT:    or $a0, $a1, $a0
 ; LA64F-NEXT:    ori $s0, $zero, 8
 ; LA64F-NEXT:    addi.d $s1, $sp, 8
 ; LA64F-NEXT:    addi.d $s2, $sp, 0
 ; LA64F-NEXT:    ori $s3, $zero, 2
+; LA64F-NEXT:    .p2align 4, , 16
 ; LA64F-NEXT:  .LBB7_1: # %atomicrmw.start
 ; LA64F-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64F-NEXT:    st.d $a0, $sp, 8
 ; LA64F-NEXT:    lu52i.d $a1, $zero, 1023
 ; LA64F-NEXT:    bl %plt(fmax)
 ; LA64F-NEXT:    st.d $a0, $sp, 0
 ; LA64F-NEXT:    move $a0, $s0
 ; LA64F-NEXT:    move $a1, $fp

 ; LA64D-NEXT:    or $a0, $a1, $a0
 ; LA64D-NEXT:    movgr2fr.d $fa0, $a0
 ; LA64D-NEXT:    addi.d $a0, $zero, 1
 ; LA64D-NEXT:    movgr2fr.d $fs0, $a0
 ; LA64D-NEXT:    ori $s0, $zero, 8
 ; LA64D-NEXT:    addi.d $s1, $sp, 16
 ; LA64D-NEXT:    addi.d $s2, $sp, 8
 ; LA64D-NEXT:    ori $s3, $zero, 2
+; LA64D-NEXT:    .p2align 4, , 16
 ; LA64D-NEXT:  .LBB7_1: # %atomicrmw.start
 ; LA64D-NEXT:    # =>This Inner Loop Header: Depth=1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 16
 ; LA64D-NEXT:    fmax.d $fa0, $fa0, $fa0
 ; LA64D-NEXT:    ffint.d.l $fa1, $fs0
 ; LA64D-NEXT:    fmax.d $fa0, $fa0, $fa1
 ; LA64D-NEXT:    fst.d $fa0, $sp, 8
 ; LA64D-NEXT:    move $a0, $s0

llvm/test/CodeGen/LoongArch/ir-instruction/br.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=loongarch32 < %s | FileCheck %s --check-prefixes=ALL,LA32
 ; RUN: llc --mtriple=loongarch64 < %s | FileCheck %s --check-prefixes=ALL,LA64
 
 define void @foo() noreturn nounwind {
 ; ALL-LABEL: foo:
 ; ALL:       # %bb.0: # %entry
+; ALL-NEXT:    .p2align 4, , 16
 ; ALL-NEXT:  .LBB0_1: # %loop
 ; ALL-NEXT:    # =>This Inner Loop Header: Depth=1
 ; ALL-NEXT:    b .LBB0_1
 entry:
   br label %loop
 loop:
   br label %loop
 }

llvm/test/CodeGen/LoongArch/preferred-alignments.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc --mtriple=loongarch64 < %s | FileCheck --check-prefix=LA464 %s
+; RUN: llc --mtriple=loongarch64 --mcpu=la464 < %s | FileCheck --check-prefix=LA464 %s
+
+define signext i32 @sum(ptr noalias nocapture noundef readonly %0, i32 noundef signext %1) {
+; LA464-LABEL: sum:
+; LA464:       # %bb.0:
+; LA464-NEXT:    ori $a2, $zero, 1
+; LA464-NEXT:    blt $a1, $a2, .LBB0_4
+; LA464-NEXT:  # %bb.1:
+; LA464-NEXT:    bstrpick.d $a2, $a1, 31, 0
+; LA464-NEXT:    move $a1, $zero
+; LA464-NEXT:    .p2align 4, , 16
+; LA464-NEXT:  .LBB0_2: # =>This Inner Loop Header: Depth=1
+; LA464-NEXT:    ld.w $a3, $a0, 0
+; LA464-NEXT:    add.d $a1, $a3, $a1
+; LA464-NEXT:    addi.d $a0, $a0, 4
+; LA464-NEXT:    addi.d $a2, $a2, -1
+; LA464-NEXT:    bnez $a2, .LBB0_2
+; LA464-NEXT:  # %bb.3:
+; LA464-NEXT:    addi.w $a0, $a1, 0
+; LA464-NEXT:    ret
+; LA464-NEXT:  .LBB0_4:
+; LA464-NEXT:    move $a1, $zero
+; LA464-NEXT:    addi.w $a0, $a1, 0
+; LA464-NEXT:    ret
+  %3 = icmp sgt i32 %1, 0
+  br i1 %3, label %4, label %6
+
+4:                                                ; preds = %2
+  %5 = zext i32 %1 to i64
+  br label %8
+
+6:                                                ; preds = %8, %2
+  %7 = phi i32 [ 0, %2 ], [ %13, %8 ]
+  ret i32 %7
+
+8:                                                ; preds = %4, %8
+  %9 = phi i64 [ 0, %4 ], [ %14, %8 ]
+  %10 = phi i32 [ 0, %4 ], [ %13, %8 ]
+  %11 = getelementptr inbounds i32, ptr %0, i64 %9
+  %12 = load i32, ptr %11, align 4
+  %13 = add nsw i32 %12, %10
+  %14 = add nuw nsw i64 %9, 1
+  %15 = icmp eq i64 %14, %5
+  br i1 %15, label %6, label %8
+}