[SelectionDAG] Add pcsections recursively on SDNode values

[martin-fink]

Summary

When adding pcsetions to SDNodes, recursively add them to all values of
the node as well.

[SelectionDAG] Add missing setValue calls in visitIntrinsicCall

This commit introduces missing setValue calls in SelectionDAGBuilder for
mem-transfer intrinsic calls. These setValue calls are required in order to
propagate pcsections metadata from IR to MIR.

[melver]

llvm/test/CodeGen/AArch64/SelectionDAG/arm64-pcsections-selectiondag.ll

1 ↗

(On Diff #486543)

Similar to x86 comment:

I think we want to say -global-isel=0, to ensure GlobalISel isn't used?

Also, just call it similar to the x86 one 'pcsections-memtransfer.ll', i.e. it'll end up in 'CodeGen/AArch64/pcsections-memtransfer.ll'.

87–100 ↗

(On Diff #486543)

I see no pcsections, is it still missing?

109–122 ↗

(On Diff #486543)

I see no pcsections, is it still missing?

131–144 ↗

(On Diff #486543)

I see no pcsections, is it still missing?

llvm/test/CodeGen/X86/SelectionDAG/x64-pcsections-memtransfer.ll

1 ↗

(On Diff #486543)

I think creating a SelectionDAG subdir is unnecessary (you're the first to create it). I think we can just dump it in X86/

Also, I think we need to explicitly make sure global-isel isn't used as it might be enabled in future, at which point it doesn't test SelectionDAG anymore.

E.g. see llvm/test/CodeGen/X86/switch-bit-test-unreachable-default.ll

I'd name this test simply 'pcsections-memtransfer.ll'

[melver]

Also update commit message to say something about "mem transfer" intrinsics.

[arsenm]

llvm/test/CodeGen/AArch64/SelectionDAG/arm64-pcsections-selectiondag.ll

1 ↗

(On Diff #486543)

Just because it's a test for sdag doesn't mean it also shouldn't pass for globalisel. I wouldn't explicitly turn it off

[martin-fink]

llvm/test/CodeGen/AArch64/SelectionDAG/arm64-pcsections-selectiondag.ll

87–100 ↗

(On Diff #486543)

I think I've overlooked that pcsections are not passed to MIR for the .element.unordered.atomic instrinsics on AArch64.
As the issue for that probably lies somewhere else and not in the scope of this commit (the pcsections are passed to MIR on X86, for example), I would suggest that I (for now) remove these three functions from the test in this commit and have a look at them later on.
What do you think about that?

[martin-fink]

[SelectionDAG] Add missing setValue calls in visitIntrinsicCall

This commit introduces missing setValue calls in SelectionDAGBuilder for
mem-transfer intrinsic calls. These setValue calls are required in order to
propagate pcsections metadata from IR to MIR.

[melver]

llvm/test/CodeGen/AArch64/SelectionDAG/arm64-pcsections-selectiondag.ll

87–100 ↗

(On Diff #486543)

I think it's somewhere in the AArch64-specific bits of SelectionDAG.

I think if you want this working properly, it should be fixed.

Otherwise, the support for the mem-transfer functions seems incomplete and whatever you intend to use !pcsections for, will be half working only.

[martin-fink]

Recursively set pcsections on SDNodes to make sure pcsections are propagated to all generated instructions.

[martin-fink]

Updating D141048: [SelectionDAG] Add pcsections recursively on SDNode values

Use recursive function instead of lambda

[martin-fink]

Remove unneeded import

[martin-fink]

Rebase main

[melver]

Nice!

Minor comments below. Otherwise LGTM.

llvm/include/llvm/CodeGen/SelectionDAG.h

2266–2267

This looks like DFS - this is typically called the visited set. So I'd just call it "Visited".

2346

No need for different name, just "addPCSections" - the fact it has different arguments means it's overloaded and there's no conflict.

Given it's private and has documentation above it, it seems cleaner.

2346

This is a rather large non-trivial function now, I'd now define it in the header, but move it to the .cpp file (like most functions here).

[melver]

llvm/include/llvm/CodeGen/SelectionDAG.h

2346

... I'd _not_ define it in the header ...

[martin-fink]

Made the following changes based on @melver's comments:

• Rename addPcsectionsr
• Rename Once to Visited
• Move function out of header file

[melver]

Thanks!
I'll land this for you.

[melver]

llvm/include/llvm/CodeGen/SelectionDAG.h

2347

I think this function wants to receive a reference to the set - I'll fix it up for you.

llvm/include/llvm/CodeGen/SelectionDAG.h

public:
   }
   /// Return HeapAllocSite associated with Node, or nullptr if none exists.
   MDNode *getHeapAllocSite(const SDNode *Node) const {
     auto I = SDEI.find(Node);
     return I != SDEI.end() ? I->second.HeapAllocSite : nullptr;
   }
   /// Set PCSections to be associated with Node.
   void addPCSections(const SDNode *Node, MDNode *MD) {
-    SDEI[Node].PCSections = MD;
+    SmallPtrSet<const llvm::SDNode *, 32> Visited;
+    addPCSections(Node, MD, Visited);
   }
   /// Return PCSections associated with Node, or nullptr if none exists.
   MDNode *getPCSections(const SDNode *Node) const {
     auto It = SDEI.find(Node);
     return It != SDEI.end() ? It->second.PCSections : nullptr;
   }
   /// Set NoMergeSiteInfo to be associated with Node if NoMerge is true.
   void addNoMergeSiteInfo(const SDNode *Node, bool NoMerge) {
private:
   SDNode *FindNodeOrInsertPos(const FoldingSetNodeID &ID, void *&InsertPos);
 
   /// Look up the node specified by ID in CSEMap.  If it exists, return it.  If
   /// not, return the insertion token that will make insertion faster.  Performs
   /// additional processing for constant nodes.
   SDNode *FindNodeOrInsertPos(const FoldingSetNodeID &ID, const SDLoc &DL,
                               void *&InsertPos);
 
+  /// Recursively set PCSections to be associated with Node and all its values.
+  void addPCSections(const SDNode *Node, MDNode *MD,
+                     SmallPtrSet<const llvm::SDNode *, 32> &Visited);
+
   /// Maps to auto-CSE operations.
   std::vector<CondCodeSDNode*> CondCodeNodes;
 
   std::vector<SDNode*> ValueTypeNodes;
   std::map<EVT, SDNode*, EVT::compareRawBits> ExtendedValueTypeNodes;
   StringMap<SDNode*> ExternalSymbols;
 
   std::map<std::pair<std::string, unsigned>, SDNode *> TargetExternalSymbols;

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

default:
       if (DL.getIROrder() && DL.getIROrder() < N->getIROrder())
         N->setDebugLoc(DL.getDebugLoc());
       break;
     }
   }
   return N;
 }
 
+void SelectionDAG::addPCSections(
+    const SDNode *Node, MDNode *MD,
+    SmallPtrSet<const llvm::SDNode *, 32> &Visited) {
+  // If we've been here before, return now.
+  if (!Visited.insert(Node).second)
+    return;
+
+  SDEI[Node].PCSections = MD;
+
+  for (const SDValue &Op : Node->op_values())
+    addPCSections(Op.getNode(), MD, Visited);
+}
+
 void SelectionDAG::clear() {
   allnodes_clear();
   OperandRecycler.clear(OperandAllocator);
   OperandAllocator.Reset();
   CSEMap.clear();
 
   ExtendedValueTypeNodes.clear();
   ExternalSymbols.clear();

llvm/test/CodeGen/AArch64/pcsections.ll

 ; RUN: llc -O0 < %s | FileCheck %s --check-prefixes=CHECK,CHECK-UNOPT,DEFCM
 ; RUN: llc -O1 < %s | FileCheck %s --check-prefixes=CHECK,CHECK-OPT,DEFCM
 ; RUN: llc -O2 < %s | FileCheck %s --check-prefixes=CHECK,CHECK-OPT,DEFCM
 ; RUN: llc -O3 < %s | FileCheck %s --check-prefixes=CHECK,CHECK-OPT,DEFCM
-; RUN: llc -O1 -code-model=large < %s | FileCheck %s --check-prefixes=CHECK,CHECK-OPT,LARGE
+; RUN: llc -O1 -code-model=large < %s | FileCheck %s --check-prefixes=CHECK,LARGE
 
 target triple = "aarch64-unknown-linux-gnu"
 
 @foo = dso_local global i64 0, align 8
 @bar = dso_local global i64 0, align 8
 
 define i64 @multiple() !pcsections !0 {
 ; CHECK-LABEL: multiple:
 ; CHECK:       .Lfunc_begin0:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK:       .Lpcsection0:
+; --
+; LARGE-NEXT:  movz
+; LARGE:  .Lpcsection1:
+; LARGE-NEXT:  movk
+; LARGE:  .Lpcsection2:
+; LARGE-NEXT:  movk
+; LARGE:  .Lpcsection3:
+; LARGE-NEXT:  movk
+; LARGE:  .Lpcsection4:
+; --
 ; CHECK-NEXT:    ldr
 ; CHECK-NEXT:    ret
 ; CHECK:       .section section_no_aux,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base0:
+; --
+; DEFCM-NEXT:  .Lpcsection_base0:
 ; DEFCM-NEXT:  .word  .Lfunc_begin0-.Lpcsection_base0
-; LARGE-NEXT:  .xword .Lfunc_begin0-.Lpcsection_base0
-; CHECK-NEXT:  .word  .Lfunc_end0-.Lfunc_begin0
-; CHECK-NEXT:  .section section_aux_42,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base1:
+; DEFCM-NEXT:  .word  .Lfunc_end0-.Lfunc_begin0
+; DEFCM-NEXT:  .section section_aux_42,"awo",@progbits,.text
+; DEFCM-NEXT:  .Lpcsection_base1:
 ; DEFCM-NEXT:  .word  .Lpcsection0-.Lpcsection_base1
-; LARGE-NEXT:  .xword .Lpcsection0-.Lpcsection_base1
-; CHECK-NEXT:  .word  42
-; CHECK-NEXT:  .section section_aux_21264,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base2:
+; DEFCM-NEXT:  .word  42
+; DEFCM-NEXT:  .section section_aux_21264,"awo",@progbits,.text
+; DEFCM-NEXT:  .Lpcsection_base2:
 ; DEFCM-NEXT:  .word  .Lpcsection0-.Lpcsection_base2
-; LARGE-NEXT:  .xword .Lpcsection0-.Lpcsection_base2
-; CHECK-NEXT:  .word  21264
-; CHECK-NEXT:  .text
+; DEFCM-NEXT:  .word  21264
+; --
+; LARGE-NEXT: .Lpcsection_base0:
+; LARGE-NEXT:   .xword  .Lfunc_begin0-.Lpcsection_base0
+; LARGE-NEXT:   .word .Lfunc_end0-.Lfunc_begin0
+; LARGE-NEXT:   .section  section_aux_42,"awo",@progbits,.text
+; LARGE-NEXT: .Lpcsection_base1:
+; LARGE-NEXT:   .xword  .Lpcsection0-.Lpcsection_base1
+; LARGE-NEXT: .Lpcsection_base2:
+; LARGE-NEXT:   .xword  .Lpcsection1-.Lpcsection_base2
+; LARGE-NEXT: .Lpcsection_base3:
+; LARGE-NEXT:   .xword  .Lpcsection2-.Lpcsection_base3
+; LARGE-NEXT: .Lpcsection_base4:
+; LARGE-NEXT:   .xword  .Lpcsection3-.Lpcsection_base4
+; LARGE-NEXT: .Lpcsection_base5:
+; LARGE-NEXT:   .xword  .Lpcsection4-.Lpcsection_base5
+; LARGE-NEXT:   .word 42
+; LARGE-NEXT:   .section  section_aux_21264,"awo",@progbits,.text
+; LARGE-NEXT: .Lpcsection_base6:
+; LARGE-NEXT:   .xword  .Lpcsection0-.Lpcsection_base6
+; LARGE-NEXT: .Lpcsection_base7:
+; LARGE-NEXT:   .xword  .Lpcsection1-.Lpcsection_base7
+; LARGE-NEXT: .Lpcsection_base8:
+; LARGE-NEXT:   .xword  .Lpcsection2-.Lpcsection_base8
+; LARGE-NEXT: .Lpcsection_base9:
+; LARGE-NEXT:   .xword  .Lpcsection3-.Lpcsection_base9
+; LARGE-NEXT: .Lpcsection_base10:
+; LARGE-NEXT:   .xword  .Lpcsection4-.Lpcsection_base10
+; LARGE-NEXT:   .word 21264
+; --
+; CHECK-NEXT:   .text
 entry:
   %0 = load i64, ptr @bar, align 8, !pcsections !1
   ret i64 %0
 }
 
 define i64 @test_simple_atomic() {
 ; CHECK-LABEL: test_simple_atomic:
-; CHECK:       .Lpcsection1:
-; CHECK-NEXT:    ldr
-; CHECK-NOT:   .Lpcsection2
-; CHECK:         ldr
+; --
+; DEFCM:       .Lpcsection1:
+; DEFCM-NEXT:    ldr
+; DEFCM-NOT:   .Lpcsection2
+; DEFCM:         ldr
+; --
+; LARGE:  .Lpcsection5:
+; LARGE-NEXT:   movz
+; LARGE-NEXT:   movz
+; LARGE:  .Lpcsection6:
+; LARGE-NEXT:   movk
+; LARGE-NEXT:   movk
+; LARGE:  .Lpcsection7:
+; LARGE-NEXT:   movk
+; LARGE-NEXT:   movk
+; LARGE:  .Lpcsection8:
+; LARGE-NEXT:   movk
+; LARGE-NEXT:   movk
+; LARGE:  .Lpcsection9:
+; LARGE-NEXT:   ldr
+; LARGE-NEXT:   ldr
+; --
 ; CHECK:         add
 ; CHECK-NEXT:    ret
 ; CHECK:       .section section_no_aux,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base3:
+; --
+; DEFCM-NEXT:  .Lpcsection_base3:
 ; DEFCM-NEXT:  .word  .Lpcsection1-.Lpcsection_base3
-; LARGE-NEXT:  .xword .Lpcsection1-.Lpcsection_base3
+; --
+; LARGE-NEXT:  .Lpcsection_base11:
+; LARGE-NEXT:  .xword .Lpcsection5-.Lpcsection_base11
+; LARGE-NEXT:  .Lpcsection_base12:
+; LARGE-NEXT:  .xword .Lpcsection6-.Lpcsection_base12
+; LARGE-NEXT:  .Lpcsection_base13:
+; LARGE-NEXT:  .xword .Lpcsection7-.Lpcsection_base13
+; LARGE-NEXT:  .Lpcsection_base14:
+; LARGE-NEXT:  .xword .Lpcsection8-.Lpcsection_base14
+; LARGE-NEXT:  .Lpcsection_base15:
+; LARGE-NEXT:  .xword .Lpcsection9-.Lpcsection_base15
+; --
 ; CHECK-NEXT:  .text
 entry:
   %0 = load atomic i64, ptr @foo monotonic, align 8, !pcsections !0
   %1 = load i64, ptr @bar, align 8
   %add = add nsw i64 %1, %0
   ret i64 %add
 }
 

 ; CHECK-UNOPT-NEXT:  cmp
 ; CHECK-UNOPT:     .Lpcsection8:
 ; CHECK-UNOPT-NEXT:  stlxr
 ; CHECK-UNOPT:     .Lpcsection9:
 ; CHECK-UNOPT-NEXT:  cbnz
 ; CHECK-UNOPT:     .Lpcsection13:
 ; CHECK-UNOPT-NEXT:  b
 ; ---
+; LARGE:      .Lpcsection10:
+; LARGE-NEXT:   movz
+; LARGE-NEXT: .Lpcsection11:
+; LARGE-NEXT:   movk
+; LARGE-NEXT: .Lpcsection12:
+; LARGE-NEXT:   movk
+; LARGE-NEXT: .Lpcsection13:
+; LARGE-NEXT:   movk
+; LARGE:      .Lpcsection14:
+; LARGE-NEXT:   ldxr
+; LARGE-NEXT: .Lpcsection15:
+; LARGE-NEXT:   add
+; LARGE-NEXT: .Lpcsection16:
+; LARGE-NEXT:   stxr
+; LARGE-NEXT: .Lpcsection17:
+; LARGE-NEXT:   cbnz
+; ---
 ; CHECK-NOT:   .Lpcsection
 ; CHECK:         ldr
 ; CHECK:         ret
 ; CHECK:       .section section_no_aux,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base4:
-; DEFCM-NEXT:  .word  .Lpcsection2-.Lpcsection_base4
-; LARGE-NEXT:  .xword .Lpcsection2-.Lpcsection_base4
-; CHECK-NEXT:  .Lpcsection_base5:
-; DEFCM-NEXT:  .word  .Lpcsection3-.Lpcsection_base5
-; LARGE-NEXT:  .xword .Lpcsection3-.Lpcsection_base5
-; CHECK-NEXT:  .Lpcsection_base6:
-; DEFCM-NEXT:  .word  .Lpcsection4-.Lpcsection_base6
-; LARGE-NEXT:  .xword .Lpcsection4-.Lpcsection_base6
-; CHECK-NEXT:  .Lpcsection_base7:
-; DEFCM-NEXT:  .word  .Lpcsection5-.Lpcsection_base7
-; LARGE-NEXT:  .xword .Lpcsection5-.Lpcsection_base7
-; CHECK-UNOPT: .word  .Lpcsection13-.Lpcsection_base15
+; ---
+; CHECK-OPT-NEXT: .Lpcsection_base4:
+; CHECK-OPT-NEXT:   .word .Lpcsection2-.Lpcsection_base4
+; CHECK-OPT-NEXT: .Lpcsection_base5:
+; CHECK-OPT-NEXT:   .word .Lpcsection3-.Lpcsection_base5
+; CHECK-OPT-NEXT: .Lpcsection_base6:
+; CHECK-OPT-NEXT:   .word .Lpcsection4-.Lpcsection_base6
+; CHECK-OPT-NEXT: .Lpcsection_base7:
+; CHECK-OPT-NEXT:   .word .Lpcsection5-.Lpcsection_base7
+; ---
+; CHECK-UNOPT-NEXT: .Lpcsection_base4:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection2-.Lpcsection_base4
+; CHECK-UNOPT-NEXT: .Lpcsection_base5:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection3-.Lpcsection_base5
+; CHECK-UNOPT-NEXT: .Lpcsection_base6:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection4-.Lpcsection_base6
+; CHECK-UNOPT-NEXT: .Lpcsection_base7:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection5-.Lpcsection_base7
+; CHECK-UNOPT-NEXT: .Lpcsection_base8:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection6-.Lpcsection_base8
+; CHECK-UNOPT-NEXT: .Lpcsection_base9:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection7-.Lpcsection_base9
+; CHECK-UNOPT-NEXT: .Lpcsection_base10:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection8-.Lpcsection_base10
+; CHECK-UNOPT-NEXT: .Lpcsection_base11:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection9-.Lpcsection_base11
+; CHECK-UNOPT-NEXT: .Lpcsection_base12:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection10-.Lpcsection_base12
+; CHECK-UNOPT-NEXT: .Lpcsection_base13:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection11-.Lpcsection_base13
+; CHECK-UNOPT-NEXT: .Lpcsection_base14:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection12-.Lpcsection_base14
+; CHECK-UNOPT-NEXT: .Lpcsection_base15:
+; CHECK-UNOPT-NEXT:   .word .Lpcsection13-.Lpcsection_base15
+; ---
+; LARGE-NEXT: .Lpcsection_base16:
+; LARGE-NEXT:   .xword  .Lpcsection10-.Lpcsection_base16
+; LARGE-NEXT: .Lpcsection_base17:
+; LARGE-NEXT:   .xword  .Lpcsection11-.Lpcsection_base17
+; LARGE-NEXT: .Lpcsection_base18:
+; LARGE-NEXT:   .xword  .Lpcsection12-.Lpcsection_base18
+; LARGE-NEXT: .Lpcsection_base19:
+; LARGE-NEXT:   .xword  .Lpcsection13-.Lpcsection_base19
+; LARGE-NEXT: .Lpcsection_base20:
+; LARGE-NEXT:   .xword  .Lpcsection14-.Lpcsection_base20
+; LARGE-NEXT: .Lpcsection_base21:
+; LARGE-NEXT:   .xword  .Lpcsection15-.Lpcsection_base21
+; LARGE-NEXT: .Lpcsection_base22:
+; LARGE-NEXT:   .xword  .Lpcsection16-.Lpcsection_base22
+; LARGE-NEXT: .Lpcsection_base23:
+; LARGE-NEXT:   .xword  .Lpcsection17-.Lpcsection_base23
 ; CHECK-NEXT:  .text
 entry:
   %0 = atomicrmw add ptr @foo, i64 1 monotonic, align 8, !pcsections !0
   %1 = load i64, ptr @bar, align 8
   %inc = add nsw i64 %1, 1
   store i64 %inc, ptr @bar, align 8
   %add = add nsw i64 %1, %0
   ret i64 %add
 }
 
 !0 = !{!"section_no_aux"}
 !1 = !{!"section_aux_42", !2, !"section_aux_21264", !3}
 !2 = !{i32 42}
 !3 = !{i32 21264}

llvm/test/CodeGen/X86/pcsections.ll

 entry:
   ret void
 }
 
 define i64 @multiple() !pcsections !0 {
 ; CHECK-LABEL: multiple:
 ; CHECK-NEXT:  .Lfunc_begin2
 ; CHECK:       # %bb.0: # %entry
-; CHECK:       .Lpcsection0:
-; CHECK-NEXT:    movq
+; --
+; DEFCM:       .Lpcsection0:
+; DEFCM-NEXT:    movq
+; --
+; LARGE:       .Lpcsection0:
+; LARGE-NEXT:    movabsq
+; LARGE-NEXT:  .Lpcsection1:
+; LARGE-NEXT:    movq
+; --
 ; CHECK-NEXT:    retq
 ; CHECK-NEXT:  .Lfunc_end2:
-; CHECK:       .section section_no_aux,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base2:
-; DEFCM-NEXT:  .long  .Lfunc_begin2-.Lpcsection_base2
-; LARGE-NEXT:  .quad  .Lfunc_begin2-.Lpcsection_base2
-; CHECK-NEXT:  .long  .Lfunc_end2-.Lfunc_begin2
-; CHECK-NEXT:  .section section_aux_42,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base3:
-; DEFCM-NEXT:  .long  .Lpcsection0-.Lpcsection_base3
-; LARGE-NEXT:  .quad  .Lpcsection0-.Lpcsection_base3
-; CHECK-NEXT:  .long  42
-; CHECK-NEXT:  .section section_aux_21264,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base4:
-; DEFCM-NEXT:  .long  .Lpcsection0-.Lpcsection_base4
-; LARGE-NEXT:  .quad  .Lpcsection0-.Lpcsection_base4
-; CHECK-NEXT:  .long  21264
+; CHECK:       .section section_no_aux,"awo",@progbits,.text
+; --
+; DEFCM-NEXT:  .Lpcsection_base2:
+; DEFCM-NEXT:  .long .Lfunc_begin2-.Lpcsection_base2
+; DEFCM-NEXT:  .long .Lfunc_end2-.Lfunc_begin2
+; DEFCM-NEXT:  .section section_aux_42,"awo",@progbits,.text
+; DEFCM-NEXT:  .Lpcsection_base3:
+; DEFCM-NEXT:  .long .Lpcsection0-.Lpcsection_base3
+; DEFCM-NEXT:  .long 42
+; DEFCM-NEXT:  .section section_aux_21264,"awo",@progbits,.text
+; DEFCM-NEXT:  .Lpcsection_base4:
+; DEFCM-NEXT:  .long .Lpcsection0-.Lpcsection_base4
+; DEFCM-NEXT:  .long 21264
+; --
+; LARGE:       .Lpcsection_base2:
+; LARGE-NEXT:  .quad .Lfunc_begin2-.Lpcsection_base2
+; LARGE-NEXT:  .long .Lfunc_end2-.Lfunc_begin2
+; LARGE-NEXT:  .section section_aux_42,"awo",@progbits,.text
+; LARGE-NEXT:  .Lpcsection_base3:
+; LARGE-NEXT:  .quad .Lpcsection0-.Lpcsection_base3
+; LARGE-NEXT:  .Lpcsection_base4:
+; LARGE-NEXT:  .quad .Lpcsection1-.Lpcsection_base4
+; LARGE-NEXT:  .long 42
+; LARGE-NEXT:  .section section_aux_21264,"awo",@progbits,.text
+; LARGE-NEXT:  .Lpcsection_base5:
+; LARGE-NEXT:  .quad .Lpcsection0-.Lpcsection_base5
+; LARGE-NEXT:  .Lpcsection_base6:
+; LARGE-NEXT:  .quad .Lpcsection1-.Lpcsection_base6
+; LARGE-NEXT:  .long 21264
+; --
 ; CHECK-NEXT:  .text
 entry:
   %0 = load i64, ptr @bar, align 8, !pcsections !2
   ret i64 %0
 }
 
 define i64 @test_simple_atomic() {
 ; CHECK-LABEL: test_simple_atomic:
-; CHECK:       .Lpcsection1:
-; CHECK-NEXT:    movq
+; --
+; DEFCM:       .Lpcsection1:
+; DEFCM-NEXT:    movq
+; --
+; LARGE:       .Lpcsection2:
+; LARGE-NEXT:    movabsq
+; LARGE-NEXT:  .Lpcsection3:
+; LARGE-NEXT:    movq
+; --
 ; CHECK-NOT:   .Lpcsection
 ; CHECK:         addq
 ; CHECK-NEXT:    retq
 ; CHECK-NEXT:  .Lfunc_end3:
-; CHECK:       .section section_no_aux,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base5:
-; DEFCM-NEXT:  .long  .Lpcsection1-.Lpcsection_base5
-; LARGE-NEXT:  .quad  .Lpcsection1-.Lpcsection_base5
+; CHECK:       .section section_no_aux,"awo",@progbits,.text
+; --
+; DEFCM-NEXT:  .Lpcsection_base5:
+; DEFCM-NEXT:  .long .Lpcsection1-.Lpcsection_base5
+; --
+; LARGE-NEXT:  .Lpcsection_base7:
+; LARGE-NEXT:  .quad .Lpcsection2-.Lpcsection_base7
+; LARGE-NEXT:  .Lpcsection_base8:
+; LARGE-NEXT:  .quad .Lpcsection3-.Lpcsection_base8
+; --
 ; CHECK-NEXT:  .text
 entry:
   %0 = load atomic i64, ptr @foo monotonic, align 8, !pcsections !0
   %1 = load i64, ptr @bar, align 8
   %add = add nsw i64 %1, %0
   ret i64 %add
 }
 
 define i64 @test_complex_atomic() {
 ; CHECK-LABEL: test_complex_atomic:
-; CHECK:         movl $1
-; CHECK-NEXT:  .Lpcsection2:
-; CHECK-NEXT:    lock xaddq
+; --
+; DEFCM:      .Lpcsection2:
+; DEFCM-NEXT:   movl
+; DEFCM-NEXT: .Lpcsection3:
+; DEFCM-NEXT:   lock
+; --
+; LARGE:       .Lpcsection4:
+; LARGE-NEXT:    movabsq
+; LARGE-NEXT:  .Lpcsection5:
+; LARGE-NEXT:    movl
+; LARGE-NEXT:  .Lpcsection6:
+; LARGE-NEXT:    lock
+; --
 ; CHECK-NOT:   .Lpcsection
 ; CHECK:         movq
 ; CHECK:         addq
 ; CHECK:         retq
 ; CHECK-NEXT:  .Lfunc_end4:
-; CHECK:       .section section_no_aux,"awo",@progbits,.text
-; CHECK-NEXT:  .Lpcsection_base6:
-; DEFCM-NEXT:  .long  .Lpcsection2-.Lpcsection_base6
-; LARGE-NEXT:  .quad  .Lpcsection2-.Lpcsection_base6
+; CHECK:       .section section_no_aux,"awo",@progbits,.text
+; --
+; DEFCM-NEXT:  .Lpcsection_base6:
+; DEFCM-NEXT:  .long .Lpcsection2-.Lpcsection_base6
+; DEFCM-NEXT:  .Lpcsection_base7:
+; DEFCM-NEXT:  .long .Lpcsection3-.Lpcsection_base7
+; --
+; LARGE-NEXT:  .Lpcsection_base9:
+; LARGE-NEXT:  .quad .Lpcsection4-.Lpcsection_base9
+; LARGE-NEXT:  .Lpcsection_base10:
+; LARGE-NEXT:  .quad .Lpcsection5-.Lpcsection_base10
+; LARGE-NEXT:  .Lpcsection_base11:
+; LARGE-NEXT:  .quad .Lpcsection6-.Lpcsection_base11
+; --
 ; CHECK-NEXT:  .text
 entry:
   %0 = atomicrmw add ptr @foo, i64 1 monotonic, align 8, !pcsections !0
   %1 = load i64, ptr @bar, align 8
   %inc = add nsw i64 %1, 1
   store i64 %inc, ptr @bar, align 8
   %add = add nsw i64 %1, %0
   ret i64 %add
 }
 
 !0 = !{!"section_no_aux"}
 !1 = !{!"section_aux", !3}
 !2 = !{!"section_aux_42", !4, !"section_aux_21264", !5}
 !3 = !{i32 10, i32 20, i32 30}
 !4 = !{i32 42}
 !5 = !{i32 21264}