[RISCV] Ensure PseudoLA* can be hoisted

[jrtc27]

Summary

Since we mark the pseudos as mayLoad but do not provide any MMOs,
isSafeToMove conservatively returns false, stopping MachineLICM from
hoisting the instructions. PseudoLA_TLS_GD does not actually expand to a
load, so stop marking that as mayLoad to allow it to be hoisted, and for
the others make sure to add MMOs during lowering to indicate they're GOT
loads and thus can be freely moved.

Fixes https://github.com/llvm/llvm-project/issues/54372

[craig.topper]

llvm/test/CodeGen/RISCV/machinelicm-got.ll

8 ↗

(On Diff #415286)

Is this TODO stale?

[jrtc27]

llvm/test/CodeGen/RISCV/machinelicm-got.ll

8 ↗

(On Diff #415286)

Oops, retroactively added it to the queued-up precommit (not committed yet, pending any review comments here of the tests themselves) but forgot to remove it here

[MaskRay]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

3762

Is it possible to construct a test that will compile to inferior assembly if MOInvariant is unset?

[MaskRay]

llvm/test/CodeGen/RISCV/machinelicm-got.ll

1 ↗

(On Diff #415286)

machinelicm-got.ll may not be the best name. I think it makes sense to place non-GOT tests in this file. For example test_lla doesn't use got.

[arichardson]

LGTM

llvm/lib/Target/RISCV/RISCVInstrInfo.td

1327

This could be split into a separate commit but committing it at the same time also seems fine to me.

[jrtc27]

• Renamed test case
• Delete TODO as fixed

[jrtc27]

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

3762

I just copied what AMDGPU did. Other backends seem to bother less with all the attributes, I suspect they don't really matter even though technically correct.

llvm/test/CodeGen/RISCV/machinelicm-got.ll

1 ↗

(On Diff #415286)

Yeah, originally it was just a PseudoLA test but then I later added PseudoLA_TLS_* and figured I should add PseudoLLA for completeness to show that one was already fine. Now renamed.

[kamaub]

I have confirmed that this patch cause a miscompile on our sanitizer-ppc64be-linux #4255 bot, please investigate and fix this as soon as possible, thank you.

[jrtc27]

In D121654#3390865, @kamaub wrote:

I have confirmed that this patch cause a miscompile on our sanitizer-ppc64be-linux #4255 bot, please investigate and fix this as soon as possible, thank you.

The log says:

../../../../lib/libLLVMRISCVCodeGen.a(RISCVISelDAGToDAG.cpp.o): In function `selectImm(llvm::SelectionDAG*, llvm::SDLoc const&, llvm::MVT, long, llvm::RISCVSubtarget const&)':
RISCVISelDAGToDAG.cpp:(.text._ZL9selectImmPN4llvm12SelectionDAGERKNS_5SDLocENS_3MVTElRKNS_14RISCVSubtargetE+0x3d8): undefined reference to `llvm::SDValue llvm::RISCVTargetLowering::getAddr<llvm::ConstantPoolSDNode>(llvm::ConstantPoolSDNode*, llvm::SelectionDAG&, bool) const'
collect2: error: ld returned 1 exit status

Other buildbots are totally fine, as are my local builds. I really do not see how on earth _this_ change can cause the function I'm editing to no longer exist, unless the host toolchain is just broken.

[jrtc27]

My guess as to what the problem is is that this has perturbed inlining heuristics such that the ConstantPoolSDNode template instantiation no longer exists in the output object file, as we don't explicitly instantiate any of them, only implicitly via their uses?

[jrtc27]

Does:

diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index f52965dab759..299c68da7c2f 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -3769,6 +3769,15 @@ SDValue RISCVTargetLowering::getAddr(NodeTy *N, SelectionDAG &DAG,
   }
 }
 
+template SDValue RISCVTargetLowering::getAddr<GlobalAddressSDNode>(
+    GlobalAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<BlockAddressSDNode>(
+    BlockAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<ConstantPoolSDNode>(
+    ConstantPoolSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<JumpTableSDNode>(
+    JumpTableSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+
 SDValue RISCVTargetLowering::lowerGlobalAddress(SDValue Op,
                                                 SelectionDAG &DAG) const {
   SDLoc DL(Op);

fix the error for you?

[kamaub]

In D121654#3390916, @jrtc27 wrote:

Does:

diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index f52965dab759..299c68da7c2f 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -3769,6 +3769,15 @@ SDValue RISCVTargetLowering::getAddr(NodeTy *N, SelectionDAG &DAG,
   }
 }
 
+template SDValue RISCVTargetLowering::getAddr<GlobalAddressSDNode>(
+    GlobalAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<BlockAddressSDNode>(
+    BlockAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<ConstantPoolSDNode>(
+    ConstantPoolSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<JumpTableSDNode>(
+    JumpTableSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+
 SDValue RISCVTargetLowering::lowerGlobalAddress(SDValue Op,
                                                 SelectionDAG &DAG) const {
   SDLoc DL(Op);

fix the error for you?

I'm happy to report that this patch fixes the error:
I built f5ddcf25d67f3dbdbb486fbcf48c5247d4f59d6a without this change and I saw the failure then I built with this change and it completed without error.

[kamaub]

In D121654#3390888, @jrtc27 wrote:

In D121654#3390865, @kamaub wrote:

I have confirmed that this patch cause a miscompile on our sanitizer-ppc64be-linux #4255 bot, please investigate and fix this as soon as possible, thank you.

The log says:

../../../../lib/libLLVMRISCVCodeGen.a(RISCVISelDAGToDAG.cpp.o): In function `selectImm(llvm::SelectionDAG*, llvm::SDLoc const&, llvm::MVT, long, llvm::RISCVSubtarget const&)':
RISCVISelDAGToDAG.cpp:(.text._ZL9selectImmPN4llvm12SelectionDAGERKNS_5SDLocENS_3MVTElRKNS_14RISCVSubtargetE+0x3d8): undefined reference to `llvm::SDValue llvm::RISCVTargetLowering::getAddr<llvm::ConstantPoolSDNode>(llvm::ConstantPoolSDNode*, llvm::SelectionDAG&, bool) const'
collect2: error: ld returned 1 exit status

Other buildbots are totally fine, as are my local builds. I really do not see how on earth _this_ change can cause the function I'm editing to no longer exist, unless the host toolchain is just broken.

I thought it was very strange as well, which is why I took so long to report it but the only thing that fixes the error is to revert the changeset.

[kamaub]

In D121654#3390916, @jrtc27 wrote:

Does:

diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index f52965dab759..299c68da7c2f 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -3769,6 +3769,15 @@ SDValue RISCVTargetLowering::getAddr(NodeTy *N, SelectionDAG &DAG,
   }
 }
 
+template SDValue RISCVTargetLowering::getAddr<GlobalAddressSDNode>(
+    GlobalAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<BlockAddressSDNode>(
+    BlockAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<ConstantPoolSDNode>(
+    ConstantPoolSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<JumpTableSDNode>(
+    JumpTableSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+
 SDValue RISCVTargetLowering::lowerGlobalAddress(SDValue Op,
                                                 SelectionDAG &DAG) const {
   SDLoc DL(Op);

fix the error for you?

I'd be happy to review and approve a patch with this change but I think the fastest resolution to getting the bot back to green would be to revert and then recommit with this change added. The bot has been broken for over 24 hours so we would like to get it back online as soon as possible, thank you for your quick response. :)

[jrtc27]

In D121654#3391091, @kamaub wrote:

In D121654#3390916, @jrtc27 wrote:

Does:

diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index f52965dab759..299c68da7c2f 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -3769,6 +3769,15 @@ SDValue RISCVTargetLowering::getAddr(NodeTy *N, SelectionDAG &DAG,
   }
 }
 
+template SDValue RISCVTargetLowering::getAddr<GlobalAddressSDNode>(
+    GlobalAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<BlockAddressSDNode>(
+    BlockAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<ConstantPoolSDNode>(
+    ConstantPoolSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<JumpTableSDNode>(
+    JumpTableSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+
 SDValue RISCVTargetLowering::lowerGlobalAddress(SDValue Op,
                                                 SelectionDAG &DAG) const {
   SDLoc DL(Op);

fix the error for you?

I'd be happy to review and approve a patch with this change but I think the fastest resolution to getting the bot back to green would be to revert and then recommit with this change added. The bot has been broken for over 24 hours so we would like to get it back online as soon as possible, thank you for your quick response. :)

I've just committed the fix as 63ea7797dd5bb77cc7b2904e20a5b779d30d4f2d; strange how this exposes it but it's been lurking ready to break since 41454ab25645 / D114950.

[kamaub]

In D121654#3391141, @jrtc27 wrote:

In D121654#3391091, @kamaub wrote:

In D121654#3390916, @jrtc27 wrote:

Does:

diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index f52965dab759..299c68da7c2f 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -3769,6 +3769,15 @@ SDValue RISCVTargetLowering::getAddr(NodeTy *N, SelectionDAG &DAG,
   }
 }
 
+template SDValue RISCVTargetLowering::getAddr<GlobalAddressSDNode>(
+    GlobalAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<BlockAddressSDNode>(
+    BlockAddressSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<ConstantPoolSDNode>(
+    ConstantPoolSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+template SDValue RISCVTargetLowering::getAddr<JumpTableSDNode>(
+    JumpTableSDNode *N, SelectionDAG &DAG, bool IsLocal) const;
+
 SDValue RISCVTargetLowering::lowerGlobalAddress(SDValue Op,
                                                 SelectionDAG &DAG) const {
   SDLoc DL(Op);

fix the error for you?

I'd be happy to review and approve a patch with this change but I think the fastest resolution to getting the bot back to green would be to revert and then recommit with this change added. The bot has been broken for over 24 hours so we would like to get it back online as soon as possible, thank you for your quick response. :)

I've just committed the fix as 63ea7797dd5bb77cc7b2904e20a5b779d30d4f2d; strange how this exposes it but it's been lurking ready to break since 41454ab25645 / D114950.

Thank you very much, the bots is green now :)

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

if (IsLocal)
       // Use PC-relative addressing to access the symbol. This generates the
       // pattern (PseudoLLA sym), which expands to (addi (auipc %pcrel_hi(sym))
       // %pcrel_lo(auipc)).
       return SDValue(DAG.getMachineNode(RISCV::PseudoLLA, DL, Ty, Addr), 0);
 
     // Use PC-relative addressing to access the GOT for this symbol, then load
     // the address from the GOT. This generates the pattern (PseudoLA sym),
     // which expands to (ld (addi (auipc %got_pcrel_hi(sym)) %pcrel_lo(auipc))).
-    return SDValue(DAG.getMachineNode(RISCV::PseudoLA, DL, Ty, Addr), 0);
+    SDValue Load =
+        SDValue(DAG.getMachineNode(RISCV::PseudoLA, DL, Ty, Addr), 0);
+    MachineFunction &MF = DAG.getMachineFunction();
+    MachineMemOperand *MemOp = MF.getMachineMemOperand(
+        MachinePointerInfo::getGOT(MF),
+        MachineMemOperand::MOLoad | MachineMemOperand::MODereferenceable |
+            MachineMemOperand::MOInvariant,
+        LLT(Ty.getSimpleVT()), Align(Ty.getFixedSizeInBits() / 8));
+    DAG.setNodeMemRefs(cast<MachineSDNode>(Load.getNode()), {MemOp});
+    return Load;
   }
 
   switch (getTargetMachine().getCodeModel()) {
   default:
     report_fatal_error("Unsupported code model for lowering");
   case CodeModel::Small: {
     // Generate a sequence for accessing addresses within the first 2 GiB of
     // address space. This generates the pattern (addi (lui %hi(sym)) %lo(sym)).
SDValue RISCVTargetLowering::getStaticTLSAddr(GlobalAddressSDNode *N,
   if (UseGOT) {
     // Use PC-relative addressing to access the GOT for this TLS symbol, then
     // load the address from the GOT and add the thread pointer. This generates
     // the pattern (PseudoLA_TLS_IE sym), which expands to
     // (ld (auipc %tls_ie_pcrel_hi(sym)) %pcrel_lo(auipc)).
     SDValue Addr = DAG.getTargetGlobalAddress(GV, DL, Ty, 0, 0);
     SDValue Load =
         SDValue(DAG.getMachineNode(RISCV::PseudoLA_TLS_IE, DL, Ty, Addr), 0);
+    MachineFunction &MF = DAG.getMachineFunction();
+    MachineMemOperand *MemOp = MF.getMachineMemOperand(
+        MachinePointerInfo::getGOT(MF),
+        MachineMemOperand::MOLoad | MachineMemOperand::MODereferenceable |
+            MachineMemOperand::MOInvariant,
+        LLT(Ty.getSimpleVT()), Align(Ty.getFixedSizeInBits() / 8));
+    DAG.setNodeMemRefs(cast<MachineSDNode>(Load.getNode()), {MemOp});
 
     // Add the thread pointer.
     SDValue TPReg = DAG.getRegister(RISCV::X4, XLenVT);
     return DAG.getNode(ISD::ADD, DL, Ty, Load, TPReg);
   }
 
   // Generate a sequence for accessing the address relative to the thread
   // pointer, with the appropriate adjustment for the thread pointer offset.

llvm/lib/Target/RISCV/RISCVInstrInfo.td

 def PseudoLA : Pseudo<(outs GPR:$dst), (ins bare_symbol:$src), [],
                       "la", "$dst, $src">;
 
 let hasSideEffects = 0, mayLoad = 1, mayStore = 0, Size = 8, isCodeGenOnly = 0,
     isAsmParserOnly = 1 in
 def PseudoLA_TLS_IE : Pseudo<(outs GPR:$dst), (ins bare_symbol:$src), [],
                              "la.tls.ie", "$dst, $src">;
 
-let hasSideEffects = 0, mayLoad = 1, mayStore = 0, Size = 8, isCodeGenOnly = 0,
+let hasSideEffects = 0, mayLoad = 0, mayStore = 0, Size = 8, isCodeGenOnly = 0,
     isAsmParserOnly = 1 in
 def PseudoLA_TLS_GD : Pseudo<(outs GPR:$dst), (ins bare_symbol:$src), [],
                              "la.tls.gd", "$dst, $src">;
 
 
 /// Sign/Zero Extends
 
 // There are single-instruction versions of these in Zbb, so disable these

llvm/test/CodeGen/RISCV/machinelicm-address-pseudos.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=riscv32 -relocation-model=pic -verify-machineinstrs < %s \
 ; RUN:   | FileCheck -check-prefixes=RV32I %s
 ; RUN: llc -mtriple=riscv64 -relocation-model=pic -verify-machineinstrs < %s \
 ; RUN:   | FileCheck -check-prefixes=RV64I %s
 
 ; Verifies that MachineLICM can hoist address generation pseudos out of loops.
-; TODO: Does not currently work for anything other than PseudoLLA.
 
 @l = protected global i32 0, align 4
 
 define void @test_lla(i32 signext %n) {
 ; RV32I-LABEL: test_lla:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    li a1, 0
 ; RV32I-NEXT:  .LBB0_3: # %entry
ret:
   ret void
 }
 
 @g = global i32 0, align 4
 
 define void @test_la(i32 signext %n) {
 ; RV32I-LABEL: test_la:
 ; RV32I:       # %bb.0: # %entry
-; RV32I-NEXT:    li a1, 0
+; RV32I-NEXT:  .LBB1_3: # %entry
+; RV32I-NEXT:    # Label of block must be emitted
+; RV32I-NEXT:    auipc a1, %got_pcrel_hi(g)
+; RV32I-NEXT:    lw a1, %pcrel_lo(.LBB1_3)(a1)
+; RV32I-NEXT:    li a2, 0
 ; RV32I-NEXT:  .LBB1_1: # %loop
 ; RV32I-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV32I-NEXT:  .LBB1_3: # %loop
-; RV32I-NEXT:    # in Loop: Header=BB1_1 Depth=1
-; RV32I-NEXT:    # Label of block must be emitted
-; RV32I-NEXT:    auipc a2, %got_pcrel_hi(g)
-; RV32I-NEXT:    lw a2, %pcrel_lo(.LBB1_3)(a2)
-; RV32I-NEXT:    lw a2, 0(a2)
-; RV32I-NEXT:    addi a1, a1, 1
-; RV32I-NEXT:    blt a1, a0, .LBB1_1
+; RV32I-NEXT:    lw a3, 0(a1)
+; RV32I-NEXT:    addi a2, a2, 1
+; RV32I-NEXT:    blt a2, a0, .LBB1_1
 ; RV32I-NEXT:  # %bb.2: # %ret
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: test_la:
 ; RV64I:       # %bb.0: # %entry
-; RV64I-NEXT:    li a1, 0
+; RV64I-NEXT:  .LBB1_3: # %entry
+; RV64I-NEXT:    # Label of block must be emitted
+; RV64I-NEXT:    auipc a1, %got_pcrel_hi(g)
+; RV64I-NEXT:    ld a1, %pcrel_lo(.LBB1_3)(a1)
+; RV64I-NEXT:    li a2, 0
 ; RV64I-NEXT:  .LBB1_1: # %loop
 ; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV64I-NEXT:  .LBB1_3: # %loop
-; RV64I-NEXT:    # in Loop: Header=BB1_1 Depth=1
-; RV64I-NEXT:    # Label of block must be emitted
-; RV64I-NEXT:    auipc a2, %got_pcrel_hi(g)
-; RV64I-NEXT:    ld a2, %pcrel_lo(.LBB1_3)(a2)
-; RV64I-NEXT:    lw a2, 0(a2)
-; RV64I-NEXT:    addiw a1, a1, 1
-; RV64I-NEXT:    blt a1, a0, .LBB1_1
+; RV64I-NEXT:    lw a3, 0(a1)
+; RV64I-NEXT:    addiw a2, a2, 1
+; RV64I-NEXT:    blt a2, a0, .LBB1_1
 ; RV64I-NEXT:  # %bb.2: # %ret
 ; RV64I-NEXT:    ret
 entry:
   br label %loop
 
 loop:
   %i = phi i32 [ %inc, %loop ], [ 0, %entry ]
   %0 = load volatile i32, i32* @g, align 4
   %inc = add nuw nsw i32 %i, 1
   %cmp = icmp slt i32 %inc, %n
   br i1 %cmp, label %loop, label %ret
 
 ret:
   ret void
 }
 
 @ie = external thread_local(initialexec) global i32
 
 define void @test_la_tls_ie(i32 signext %n) {
 ; RV32I-LABEL: test_la_tls_ie:
 ; RV32I:       # %bb.0: # %entry
-; RV32I-NEXT:    li a1, 0
-; RV32I-NEXT:  .LBB2_1: # %loop
-; RV32I-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV32I-NEXT:  .LBB2_3: # %loop
-; RV32I-NEXT:    # in Loop: Header=BB2_1 Depth=1
+; RV32I-NEXT:  .LBB2_3: # %entry
 ; RV32I-NEXT:    # Label of block must be emitted
 ; RV32I-NEXT:    auipc a2, %tls_ie_pcrel_hi(ie)
 ; RV32I-NEXT:    lw a2, %pcrel_lo(.LBB2_3)(a2)
+; RV32I-NEXT:    li a1, 0
 ; RV32I-NEXT:    add a2, a2, tp
-; RV32I-NEXT:    lw a2, 0(a2)
+; RV32I-NEXT:  .LBB2_1: # %loop
+; RV32I-NEXT:    # =>This Inner Loop Header: Depth=1
+; RV32I-NEXT:    lw a3, 0(a2)
 ; RV32I-NEXT:    addi a1, a1, 1
 ; RV32I-NEXT:    blt a1, a0, .LBB2_1
 ; RV32I-NEXT:  # %bb.2: # %ret
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: test_la_tls_ie:
 ; RV64I:       # %bb.0: # %entry
-; RV64I-NEXT:    li a1, 0
-; RV64I-NEXT:  .LBB2_1: # %loop
-; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV64I-NEXT:  .LBB2_3: # %loop
-; RV64I-NEXT:    # in Loop: Header=BB2_1 Depth=1
+; RV64I-NEXT:  .LBB2_3: # %entry
 ; RV64I-NEXT:    # Label of block must be emitted
 ; RV64I-NEXT:    auipc a2, %tls_ie_pcrel_hi(ie)
 ; RV64I-NEXT:    ld a2, %pcrel_lo(.LBB2_3)(a2)
+; RV64I-NEXT:    li a1, 0
 ; RV64I-NEXT:    add a2, a2, tp
-; RV64I-NEXT:    lw a2, 0(a2)
+; RV64I-NEXT:  .LBB2_1: # %loop
+; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
+; RV64I-NEXT:    lw a3, 0(a2)
 ; RV64I-NEXT:    addiw a1, a1, 1
 ; RV64I-NEXT:    blt a1, a0, .LBB2_1
 ; RV64I-NEXT:  # %bb.2: # %ret
 ; RV64I-NEXT:    ret
 entry:
   br label %loop
 
 loop:

 
 define void @test_la_tls_gd(i32 signext %n) nounwind {
 ; RV32I-LABEL: test_la_tls_gd:
 ; RV32I:       # %bb.0: # %entry
 ; RV32I-NEXT:    addi sp, sp, -16
 ; RV32I-NEXT:    sw ra, 12(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s0, 8(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    sw s1, 4(sp) # 4-byte Folded Spill
+; RV32I-NEXT:    sw s2, 0(sp) # 4-byte Folded Spill
 ; RV32I-NEXT:    mv s0, a0
-; RV32I-NEXT:    li s1, 0
+; RV32I-NEXT:    li s2, 0
+; RV32I-NEXT:  .LBB3_3: # %entry
+; RV32I-NEXT:    # Label of block must be emitted
+; RV32I-NEXT:    auipc s1, %tls_gd_pcrel_hi(gd)
+; RV32I-NEXT:    addi s1, s1, %pcrel_lo(.LBB3_3)
 ; RV32I-NEXT:  .LBB3_1: # %loop
 ; RV32I-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV32I-NEXT:  .LBB3_3: # %loop
-; RV32I-NEXT:    # in Loop: Header=BB3_1 Depth=1
-; RV32I-NEXT:    # Label of block must be emitted
-; RV32I-NEXT:    auipc a0, %tls_gd_pcrel_hi(gd)
-; RV32I-NEXT:    addi a0, a0, %pcrel_lo(.LBB3_3)
+; RV32I-NEXT:    mv a0, s1
 ; RV32I-NEXT:    call __tls_get_addr@plt
 ; RV32I-NEXT:    lw a0, 0(a0)
-; RV32I-NEXT:    addi s1, s1, 1
-; RV32I-NEXT:    blt s1, s0, .LBB3_1
+; RV32I-NEXT:    addi s2, s2, 1
+; RV32I-NEXT:    blt s2, s0, .LBB3_1
 ; RV32I-NEXT:  # %bb.2: # %ret
 ; RV32I-NEXT:    lw ra, 12(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    lw s0, 8(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    lw s1, 4(sp) # 4-byte Folded Reload
+; RV32I-NEXT:    lw s2, 0(sp) # 4-byte Folded Reload
 ; RV32I-NEXT:    addi sp, sp, 16
 ; RV32I-NEXT:    ret
 ;
 ; RV64I-LABEL: test_la_tls_gd:
 ; RV64I:       # %bb.0: # %entry
 ; RV64I-NEXT:    addi sp, sp, -32
 ; RV64I-NEXT:    sd ra, 24(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s0, 16(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    sd s1, 8(sp) # 8-byte Folded Spill
+; RV64I-NEXT:    sd s2, 0(sp) # 8-byte Folded Spill
 ; RV64I-NEXT:    mv s0, a0
-; RV64I-NEXT:    li s1, 0
+; RV64I-NEXT:    li s2, 0
+; RV64I-NEXT:  .LBB3_3: # %entry
+; RV64I-NEXT:    # Label of block must be emitted
+; RV64I-NEXT:    auipc s1, %tls_gd_pcrel_hi(gd)
+; RV64I-NEXT:    addi s1, s1, %pcrel_lo(.LBB3_3)
 ; RV64I-NEXT:  .LBB3_1: # %loop
 ; RV64I-NEXT:    # =>This Inner Loop Header: Depth=1
-; RV64I-NEXT:  .LBB3_3: # %loop
-; RV64I-NEXT:    # in Loop: Header=BB3_1 Depth=1
-; RV64I-NEXT:    # Label of block must be emitted
-; RV64I-NEXT:    auipc a0, %tls_gd_pcrel_hi(gd)
-; RV64I-NEXT:    addi a0, a0, %pcrel_lo(.LBB3_3)
+; RV64I-NEXT:    mv a0, s1
 ; RV64I-NEXT:    call __tls_get_addr@plt
 ; RV64I-NEXT:    lw a0, 0(a0)
-; RV64I-NEXT:    addiw s1, s1, 1
-; RV64I-NEXT:    blt s1, s0, .LBB3_1
+; RV64I-NEXT:    addiw s2, s2, 1
+; RV64I-NEXT:    blt s2, s0, .LBB3_1
 ; RV64I-NEXT:  # %bb.2: # %ret
 ; RV64I-NEXT:    ld ra, 24(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld s0, 16(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    ld s1, 8(sp) # 8-byte Folded Reload
+; RV64I-NEXT:    ld s2, 0(sp) # 8-byte Folded Reload
 ; RV64I-NEXT:    addi sp, sp, 32
 ; RV64I-NEXT:    ret
 entry:
   br label %loop
 
 loop:
   %i = phi i32 [ %inc, %loop ], [ 0, %entry ]
   %0 = load volatile i32, i32* @gd, align 4
   %inc = add nuw nsw i32 %i, 1
   %cmp = icmp slt i32 %inc, %n
   br i1 %cmp, label %loop, label %ret
 
 ret:
   ret void
 }

llvm/test/CodeGen/RISCV/mir-target-flags.ll

 @t_le = external thread_local(localexec) global i32
 
 declare i32 @callee(i32) nounwind
 
 define i32 @caller(i32 %a) nounwind {
 ; RV32-SMALL-LABEL: name: caller
 ; RV32-SMALL:      target-flags(riscv-hi) @g_e
 ; RV32-SMALL-NEXT: target-flags(riscv-lo) @g_e
-; RV32-SMALL-NEXT: target-flags(riscv-hi) @g_i
-; RV32-SMALL-NEXT: target-flags(riscv-lo) @g_i
 ; RV32-SMALL:      target-flags(riscv-tls-got-hi) @t_un
 ; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo) %bb.1
+; RV32-SMALL:      target-flags(riscv-hi) @g_i
+; RV32-SMALL-NEXT: target-flags(riscv-lo) @g_i
 ; RV32-SMALL:      target-flags(riscv-tls-got-hi) @t_ld
 ; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo) %bb.2
 ; RV32-SMALL:      target-flags(riscv-tls-got-hi) @t_ie
 ; RV32-SMALL-NEXT: target-flags(riscv-pcrel-lo) %bb.3
 ; RV32-SMALL:      target-flags(riscv-tprel-hi) @t_le
 ; RV32-SMALL-NEXT: target-flags(riscv-tprel-add) @t_le
 ; RV32-SMALL-NEXT: target-flags(riscv-tprel-lo) @t_le
 ; RV32-SMALL:      target-flags(riscv-plt) @callee