[X86] Introduce a large data threshold for the medium code model

[aeubanks]

Summary

Currently clang's medium code model treats all data as large, putting them in a large data section and using more expensive instruction sequences to access them.

Following gcc's -mlarge-data-threshold, which allows putting data under a certain size in a normal data section as opposed to a large data section. This allows using cheaper code sequences to access some portion of data in the binary (which will be implemented in LLVM in a future patch).

And under the medium codel mode, only put data above the large data threshold into large data sections, not all data.

[aeubanks]

depends on D148836

[jyknight]

large-data-threshold is ABI, and needs to have a default value which is the same as GCC, 65535. (And users should be discouraged from changing it.)

In many common cases you can get away with using different values in different object files, because cross-object references are going via PLT/GOT when the compiler doesn't know the definition is in-DSO. But that's definitely not a 100% solution -- e.g. __attribute__((visibility("hidden"))) or ODR-data defined in a -fPIE object is known to be defined in-DSO, so will use a pc-relative reference.

llvm/lib/Target/TargetMachine.cpp

47

W also must handle unknown-sized objects as large here.

E.g. clang -mcmodel=medium -fPIC on

__attribute__((visibility("hidden"))) extern int x[];
int bar(void) { return x[0]; }

must refer to x via GOTOFF not pc-relative, because it _could_ be defined as large.

[aeubanks]

check size 0

[aeubanks]

In D149288#4343075, @jyknight wrote:

large-data-threshold is ABI, and needs to have a default value which is the same as GCC, 65535. (And users should be discouraged from changing it.)

In many common cases you can get away with using different values in different object files, because cross-object references are going via PLT/GOT when the compiler doesn't know the definition is in-DSO. But that's definitely not a 100% solution -- e.g. __attribute__((visibility("hidden"))) or ODR-data defined in a -fPIE object is known to be defined in-DSO, so will use a pc-relative reference.

Yes, changing the default value and adding a clang flag will come in a future patch; this patch preserves the current behavior by default

llvm/lib/Target/TargetMachine.cpp

47

ah ok, although that's not observable in this patch since we only use this for determining which section to place a symbol in, not how we reference a symbol (that's D150297). I'll make the change here but add tests in the other patch

[MaskRay]

llvm/lib/CodeGen/CommandFlags.cpp

169

To match GCC, we can use 65536

Consider applying this to large code model as well.

https://gcc.gnu.org/pipermail/gcc-patches/2023-May/619695.html

llvm/lib/Target/TargetMachine.cpp

51

I think GCC's condition is > instead of >=.

[tkoeppe]

llvm/tools/llc/llc.cpp

594

Can we avoid the implicit check for 0 here and spell out ; LDT > 0? Does LLVM require C++17?

[MaskRay]

llvm/tools/llc/llc.cpp

594

Building llvm-project requires c++17 (libomptarget may be an exception).

We can use if (auto LDT = ...; LDT && LDT > 0) only if the default is 0.

[aeubanks]

instead of >=

llvm/lib/CodeGen/CommandFlags.cpp

169

changes to this value will come in a later patch, and same for making the large code model respect this value

llvm/tools/llc/llc.cpp

594

yeah LLVM uses C++17

codegen::getExplicitLargeDataThreshold returns a std::optional, so we're checking if the user explicitly overrode the value, not if the value is 0. we would want to set the value to 0 if the user explicitly requested it

[tkoeppe]

llvm/tools/llc/llc.cpp

594

Ah yes, sorry, I forgot that it's an optional<int>, not just an int. In that case all I meant was for the check to not be implicit, i.e. I'd recommend:

for (auto LDT = ...; LDT.has_vaiue()) {
  // use *LDT
}

Just so there's one less implicit thing one needs to keep in mind when reading this.

[MaskRay]

llvm/lib/CodeGen/CommandFlags.cpp

169

This seems strange. Introducing this variable and setting it to an appropriate should come in the same patch, no ?

I don't want clang -mcmodel=medium emitted data sections for small objects to change from .data to .ldata, then to .data again.

[aeubanks]

use std::optional<uint64_t> instead of auto

llvm/lib/CodeGen/CommandFlags.cpp

169

this patch doesn't change the current behavior of the medium code model in clang since a large data threshold of zero treats all data as large, which is the current behavior. I'd rather have the implementation in this patch then a separate small patch to actually change the value, especially since not all of the large data threshold functionality is done yet (e.g. D150297)

llvm/tools/llc/llc.cpp

594

I think this is the standard idiom for checking if a std::optional has a value, not calling has_value().
I did change the type to be explicit though, instead of auto

[rnk]

What about storing some kind of "large data attribute" on the global itself in the IR? That would help make sure the medium code model flags don't have to be passed to ThinLTO backend action compilations, for example, and give intuitive semantics to linking together objects compiled in the large, medium, and small code models.

The other consideration is making this feature easy for other frontends (Rust) to use, and your solution does make that easy. The best I can come up with is an LLVM pass or helper function in LLVM that annotates IR globals appropriately given a size threshold from the frontend.

[aeubanks]

In D149288#4388728, @rnk wrote:

What about storing some kind of "large data attribute" on the global itself in the IR? That would help make sure the medium code model flags don't have to be passed to ThinLTO backend action compilations, for example, and give intuitive semantics to linking together objects compiled in the large, medium, and small code models.

apparently a module metadata for the code model for passing along to LTO backends was already done a while back in https://reviews.llvm.org/D52322

and I verified that you don't need to pass -mcmodel to lld with ThinLTO

[efriedma]

Right, the model should be encoded in the IR for LTO. So as long as we consider the threshold an ABI constraint, and we can accurately compute whether a global has indefinite size, we don't need to tag individual globals.

Tagging individual globals could be useful as an extension, I guess, but it doesn't need to be in this patch.

(Note there's a weird interaction between -mcmodel=medium and the gcc extension that allows initializing flexible arrays: it's possible to define a "small" global that's actually larger than the threshold, and gcc will still treat it as small. But the interaction is consistent with the way this patch works.)

[rnk]

One of our use cases for having a "large" attribute for global data is to have a way to mark instrumentation data as large. We would use this for ASan global metadata and PGO name data, which are not accessed frequently during program execution, but contribute significantly to binary data size.

Regardless, I'm in favor of using a simple module flag for this. It makes it easy for non-Clang frontends. The explicit "large" attribute can come later.

[aeubanks]

I have a patch that introduces the large data threshold module metadata, but I'd like to separate that patch to keep patches smallish. It has some dependencies so I need to do stacked patches but we're deprecating new patches on phab and GH doesn't support stacked patch reviews yet. Can we move forward with this one first?

[rnk]

llvm/tools/llc/llc.cpp

494

Should the threshold be a TargetOption? Would that avoid the need for the new TargetMachine field?

[aeubanks]

llvm/tools/llc/llc.cpp

494

it should go along with the code model since they're tied together. I do think putting all these options (code model, relocation model) in TargetOptions makes sense, it fits with lots of existing options there. if we put all these options into TargetOptions it'd simplify the TargetMachine constructors. if people think this makes sense I can do that cleanup first

[rnk]

lgtm

llvm/tools/llc/llc.cpp

494

I think what I'd like to do is to minimize the number of changes to the createTargetMachine prototype, since it's called in many places. It has many optional parameters, and I suspect that most callers accept the defaults:

TargetMachine *createTargetMachine(
    StringRef TT, StringRef CPU, StringRef Features,
    const TargetOptions &Options,
    std::optional<Reloc::Model> RM, // No default, but is std::optional
    std::optional<CodeModel::Model> CM = std::nullopt, // Optional, with a default.
    CodeGenOpt::Level OL = CodeGenOpt::Default, // has a default, this will change to an enum class
    bool JIT = false // Boolean optional parameters are unreadable.
) const {

Maybe the first change is to move the relocation and code model into target options. We probably have to leave the optimization level alone, since there are multiple callers of TargetMachine::setOptLevel, and it's not really an immutable option, it varies from function to function.

That said, if you want to land this now, and defer that work, I'm OK with that, this change doesn't affect the createTargetMachine prototype, it's the CodeGenLevel one that does.

[MaskRay]

llvm/lib/CodeGen/CommandFlags.cpp

168

Perhaps we should say x86-64 medium/large code model.
My https://gcc.gnu.org/pipermail/gcc-patches/2023-September/630207.html is still pending after many pings, but I think we can go ahead as the obvious choice.

[aeubanks]

llvm/lib/CodeGen/CommandFlags.cpp

168

we can change this to also say "large" once LLVM actually respects it in the large code model

llvm/include/llvm/CodeGen/CommandFlags.h

 Reloc::Model getRelocModel();
 std::optional<Reloc::Model> getExplicitRelocModel();
 
 ThreadModel::Model getThreadModel();
 
 CodeModel::Model getCodeModel();
 std::optional<CodeModel::Model> getExplicitCodeModel();
 
+uint64_t getLargeDataThreshold();
+std::optional<uint64_t> getExplicitLargeDataThreshold();
+
 llvm::ExceptionHandling getExceptionModel();
 
 std::optional<CodeGenFileType> getExplicitFileType();
 
 CodeGenFileType getFileType();
 
 FramePointerKind getFramePointerUsage();
 

llvm/include/llvm/Target/TargetMachine.h

protected: // Can only create subclasses.
   /// Triple string, CPU name, and target feature strings the TargetMachine
   /// instance is created with.
   Triple TargetTriple;
   std::string TargetCPU;
   std::string TargetFS;
 
   Reloc::Model RM = Reloc::Static;
   CodeModel::Model CMModel = CodeModel::Small;
+  uint64_t LargeDataThreshold = 0;
   CodeGenOptLevel OptLevel = CodeGenOptLevel::Default;
 
   /// Contains target specific asm information.
   std::unique_ptr<const MCAsmInfo> AsmInfo;
   std::unique_ptr<const MCRegisterInfo> MRI;
   std::unique_ptr<const MCInstrInfo> MII;
   std::unique_ptr<const MCSubtargetInfo> STI;
 
public:
   CodeModel::Model getCodeModel() const { return CMModel; }
 
   /// Returns the maximum code size possible under the code model.
   uint64_t getMaxCodeSize() const;
 
   /// Set the code model.
   void setCodeModel(CodeModel::Model CM) { CMModel = CM; }
 
-  bool isLargeData() const;
+  void setLargeDataThreshold(uint64_t LDT) { LargeDataThreshold = LDT; }
+  bool isLargeData(const GlobalVariable *GV) const;
 
   bool isPositionIndependent() const;
 
   bool shouldAssumeDSOLocal(const Module &M, const GlobalValue *GV) const;
 
   /// Returns true if this target uses emulated TLS.
   bool useEmulatedTLS() const;
 

llvm/lib/CodeGen/CommandFlags.cpp

#define CGOPT_EXP(TY, NAME)                                                    \
   }
 
 CGOPT(std::string, MArch)
 CGOPT(std::string, MCPU)
 CGLIST(std::string, MAttrs)
 CGOPT_EXP(Reloc::Model, RelocModel)
 CGOPT(ThreadModel::Model, ThreadModel)
 CGOPT_EXP(CodeModel::Model, CodeModel)
+CGOPT_EXP(uint64_t, LargeDataThreshold)
 CGOPT(ExceptionHandling, ExceptionModel)
 CGOPT_EXP(CodeGenFileType, FileType)
 CGOPT(FramePointerKind, FramePointerUsage)
 CGOPT(bool, EnableUnsafeFPMath)
 CGOPT(bool, EnableNoInfsFPMath)
 CGOPT(bool, EnableNoNaNsFPMath)
 CGOPT(bool, EnableNoSignedZerosFPMath)
 CGOPT(bool, EnableApproxFuncFPMath)
static cl::opt<CodeModel::Model> CodeModel(
       "code-model", cl::desc("Choose code model"),
       cl::values(clEnumValN(CodeModel::Tiny, "tiny", "Tiny code model"),
                  clEnumValN(CodeModel::Small, "small", "Small code model"),
                  clEnumValN(CodeModel::Kernel, "kernel", "Kernel code model"),
                  clEnumValN(CodeModel::Medium, "medium", "Medium code model"),
                  clEnumValN(CodeModel::Large, "large", "Large code model")));
   CGBINDOPT(CodeModel);
 
+  static cl::opt<uint64_t> LargeDataThreshold(
+      "large-data-threshold",
+      cl::desc("Choose large data threshold for x86_64 medium code model"),
+      cl::init(0));
+  CGBINDOPT(LargeDataThreshold);
+
   static cl::opt<ExceptionHandling> ExceptionModel(
       "exception-model", cl::desc("exception model"),
       cl::init(ExceptionHandling::None),
       cl::values(
           clEnumValN(ExceptionHandling::None, "default",
                      "default exception handling model"),
           clEnumValN(ExceptionHandling::DwarfCFI, "dwarf",
                      "DWARF-like CFI based exception handling"),

llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

if (Kind.isMergeableCString()) {
 
     std::string SizeSpec = ".rodata.str" + utostr(EntrySize) + ".";
     Name = SizeSpec + utostr(Alignment.value());
   } else if (Kind.isMergeableConst()) {
     Name = ".rodata.cst";
     Name += utostr(EntrySize);
   } else {
     bool IsLarge = false;
-    if (isa<GlobalVariable>(GO))
-      IsLarge = TM.isLargeData();
+    if (auto *GV = dyn_cast<GlobalVariable>(GO))
+      IsLarge = TM.isLargeData(GV);
     Name = getSectionPrefixForGlobal(Kind, IsLarge);
   }
 
   bool HasPrefix = false;
   if (const auto *F = dyn_cast<Function>(GO)) {
     if (std::optional<StringRef> Prefix = F->getSectionPrefix()) {
       raw_svector_ostream(Name) << '.' << *Prefix;
       HasPrefix = true;
static MCSectionELF *selectELFSectionForGlobal(
 
   StringRef Group = "";
   bool IsComdat = false;
   if (const Comdat *C = getELFComdat(GO)) {
     Flags |= ELF::SHF_GROUP;
     Group = C->getName();
     IsComdat = C->getSelectionKind() == Comdat::Any;
   }
-  if (isa<GlobalVariable>(GO)) {
-    if (TM.isLargeData()) {
+  if (auto *GV = dyn_cast<GlobalVariable>(GO)) {
+    if (TM.isLargeData(GV)) {
       assert(TM.getTargetTriple().getArch() == Triple::x86_64);
       Flags |= ELF::SHF_X86_64_LARGE;
     }
   }
 
   // Get the section entry size based on the kind.
   unsigned EntrySize = getEntrySizeForKind(Kind);
 

llvm/lib/Target/TargetMachine.cpp

TargetMachine::TargetMachine(const Target &T, StringRef DataLayoutString,
                              const TargetOptions &Options)
     : TheTarget(T), DL(DataLayoutString), TargetTriple(TT),
       TargetCPU(std::string(CPU)), TargetFS(std::string(FS)), AsmInfo(nullptr),
       MRI(nullptr), MII(nullptr), STI(nullptr), RequireStructuredCFG(false),
       O0WantsFastISel(false), DefaultOptions(Options), Options(Options) {}
 
 TargetMachine::~TargetMachine() = default;
 
-bool TargetMachine::isLargeData() const {
+bool TargetMachine::isLargeData(const GlobalVariable *GV) const {
   if (getTargetTriple().getArch() != Triple::x86_64)
     return false;
   // Large data under the large code model still needs to be thought about, so
   // restrict this to medium.
   if (getCodeModel() != CodeModel::Medium)
     return false;
-  return true;
+  const DataLayout &DL = GV->getParent()->getDataLayout();
+  uint64_t Size = DL.getTypeSizeInBits(GV->getValueType()) / 8;
+  return Size == 0 || Size > LargeDataThreshold;
 }
 
 bool TargetMachine::isPositionIndependent() const {
   return getRelocationModel() == Reloc::PIC_;
 }
 
 /// Reset the target options based on the function's attributes.
 /// setFunctionAttributes should have made the raw attribute value consistent

llvm/test/CodeGen/X86/code-model-elf-sections.ll

 ; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=small -o %t
 ; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=SMALL
 ; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=medium -o %t
 ; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=LARGE
+; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=medium -large-data-threshold=79 -o %t
+; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=LARGE
+; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=medium -large-data-threshold=80 -o %t
+; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=SMALL
 ; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=large -o %t
 ; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=SMALL
 
 ; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=small -data-sections -o %t
 ; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=SMALL-DS
 ; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=medium -data-sections -o %t
 ; RUN: llvm-readelf -S %t | FileCheck %s --check-prefix=LARGE-DS
 ; RUN: llc < %s -relocation-model=pic -filetype=obj -code-model=large -data-sections -o %t

llvm/tools/llc/llc.cpp

if (V.consumeInteger(10, Num) || Num == 0 ||
           (V.consume_front(".") && !V.consumeInteger(10, Num) && V.empty()))) {
       WithColor::error(errs(), argv[0])
           << "invalid -binutils-version, accepting 'none' or major.minor\n";
       return 1;
     }
   }
   TargetOptions Options;
   auto InitializeOptions = [&](const Triple &TheTriple) {
     Options = codegen::InitTargetOptionsFromCodeGenFlags(TheTriple);
 
     if (Options.XCOFFReadOnlyPointers) {
       if (!TheTriple.isOSAIX())
         reportError("-mxcoff-roptr option is only supported on AIX",
                     InputFilename);
 
       // Since the storage mapping class is specified per csect,
       // without using data sections, it is less effective to use read-only
if (!M) {
       return 1;
     }
     if (!TargetTriple.empty())
       M->setTargetTriple(Triple::normalize(TargetTriple));
 
     std::optional<CodeModel::Model> CM_IR = M->getCodeModel();
     if (!CM && CM_IR)
       Target->setCodeModel(*CM_IR);
+    if (std::optional<uint64_t> LDT = codegen::getExplicitLargeDataThreshold())
+      Target->setLargeDataThreshold(*LDT);
   } else {
     TheTriple = Triple(Triple::normalize(TargetTriple));
     if (TheTriple.getTriple().empty())
       TheTriple.setTriple(sys::getDefaultTargetTriple());
 
     // Get the target specific parser.
     std::string Error;
     TheTarget =