[amdgpu] Reimplement LDS lowering

[JonChesterfield]

Summary

Renames the current lowering scheme to "module" and introduces two new
ones, "kernel" and "table", plus a "hybrid" that chooses between those three
on a per-variable basis.

Unit tests are set up to pass with the default lowering of "module" or "hybrid"
with this patch defaulting to "module", which will be a less dramatic codegen
change relative to the current. This reflects the sparsity of test coverage for
the table lowering method. Hybrid is better than module in every respect and
will be default in a subsequent patch.

[JonChesterfield]

• remove if (verbose) debugging hook

[JonChesterfield]

This could (and should) be prettier, and there should be lit tests for the (gated behind the default =module, so inactive here) table lowering pass. This has been through epsdb in the hybrid mode and set up to force all lowerings through the table lookup so there's a credible chance it works correctly.

One path forward is:

• land this in the current somewhat unpolished state and possibly toggle the default in rocm to hybrid immediately to unblock the inlining experimentation
• a follow up patch changes the default to hybrid along with a bunch of lit tests showing how the table lookup behaves
• I refactor this implementation from behind more paranoid test coverage, in particular to walk the functions list less often

Alternatively I'll iterate on it in phab until the inessential complexity is reduced enough. It's a bigger patch than I'd have liked, but at least the complicated backend lowering landed separately in D125060. This is mostly an IR transform.

[JonChesterfield]

Structure lends itself relatively well to refactoring the current lowering as "module", then adding kernel and table as separate patches, then hybrid as a final one. That's significant work for the author and does little to de-risk this, but would make each review easier to do.

[arsenm]

I think we need to get some documentation explaining the LDS lowering flow and the options. Not sure where the best place for it; I guess AMDGPUUsage?

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

87

I think you're supposed to use StringLiteral these days for some reason

98

Don't see the point of this string copy, GV.getName() gives you a StringRef to begin with

126

Capitalize

128

Why not just ==?

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h

111–112

These aren't really gaining anything from being part of AMDGPUMachineFunction, but I guess it's where the other stuff already lives

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

6015

Could just early return on each of these instead of or'ing

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

14–49

I somehow missed this giant comment in the first pass. Perhaps this should move to AMDGPUUsage

161–190

You're re-inventing cl::opt with clEnumVal

419–420

Comment the bool operand meanings

612

Don't need llvm:: (not sure why this seems to be the most popular function for people to put it on)

642

Can you add an assert somewhere that the maximum LDS size fits in 16-bits? I'm also wondering if we should just make LDS pointers 16-bit with 4 byte alignment, or add a 16-bit address space for this

654

Early return and reduce indentation

680

Use new style member initializers

987–993

This whole thing is just return isa<Instruction>(U.getUser())

1037

Don't need llvm::

llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp

171

Is this actually reachable?

[nlopes]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

534

Please use PoisonValue here if possible. We are trying to remove undef values, and thus we want to avoid adding new uses that can be easily replaced with poison.
Thank you!

[JonChesterfield]

• Initial tests

[JonChesterfield]

• Add test for lowering via table

[JonChesterfield]

• addressing easier comments from Matt

[JonChesterfield]

Was sidetracked by trying to work out why global-variable-relocs.ll and waitcnt-looptest.ll have started failing, but neither do anything with LDS (no addrspace(3) vars plus lowering pass makes no change to the IR), and rolling back to main doesn't fix them. Buildbot is green so that's weird, attributing to local quirk / non-determinism.

Comments responded to above, lit test coverage could still be better.

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

161–190

So I was. Interesting. Changed to use clEnumValN (to preserve the namespacing at the use sites).

Sadly the test case lds-reject-unknown-lowering-kind.ll - updated with the new error string - is now refusing to run under check-llvm despite running fine in a shell. That coverage seems less important when using a ready-made parser so I've deleted the test.

534

Whether this can be easily replaced with poison depends on how the rest of the toolchain deals with poison. The string replacement is fine, it's whether it then blows up under testing due to interaction with the rest of the compiler that encourages caution.

The semantics of poison fit this behaviour - if this value is read, the compiler has a bug or runtime behaviour has diverged from that compiler's expectation. I would therefore like to change from undef to poison after this lands, so that if it does fail testing we have a much easier time triaging.

612

Had it in a few other places too, on various types and on sort. Dropped all of them, then put it back on sort to resolve ambiguity with std::sort (which doesn't make masses of sense to me but will stay on task)

642

The backend currently errors on that in user-facing fashion. A check here will be slightly less accurate than in the back end because this runs before promote alloca puts more things into LDS and I'm not sure we have more information to provide a better report either - what sort of error message do you have in mind?

I suppose the current code path would truncate the offsets to i16 before the back end crashes, but that seems harmless. We'd have slightly cleaner tests with a native sixteen bit type but there's not much in it.

llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp

171

Definitely yes. It's hit when lowering the lookup table for the table fallback.

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

128

iirc that did the wrong thing with a string literal, as opposed to a StringLiteral

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h

111–112

I'd guess they're here because the dynamic lds alignment is tracked through the instance, but with that now mostly factored out they could be moved under some utility header.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

161–190

I'm not seeing this change here?

387

Don't need std::string, but this is also unused?

393

Don't need std::string, but this is also unused?

520

Type *I16

534

Just change to poison. Nothing we're doing in the backend is aware enough of the difference (currently poison just gets selected to the same implicit_def anyway, although that's kind of broken)

575

Not sure if this is just copying something existing, but I think it would be preferably to consistently use . as the word separator instead of mixing in _

642

I don't mean here, I mean somewhere. Promote alloca also shouldn't be pushing the usage up into something unsupportable

657

New line

791

llvm_unreachable

819–822

Probably should promote this to a DiagnosticInfoUnsupported

[JonChesterfield]

• Change lowering kind flag to use clEnumValN

[JonChesterfield]

• dead variables removed, undef to poison, rename offset_table to offset.table

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

575

OK, so the rest of the similar things are dot delimited, but I_write_in_snake_case by default so that's how this happens. Will rename it to llvm.amdgcn.lds.offset.table

657

Ambiguous comment, let me know if you meant something other than a newline between the declaration and the early return

[JonChesterfield]

• Use i32 for table instead of i16, still need to drop the inttoptr

[JonChesterfield]

• minor cleanup

[JonChesterfield]

Something of a speed bump. https://reviews.llvm.org/D131246 breaks the table and hybrid implementation here. Fails check-openmp reliably. "illegal SGPR to VGPR copy".

Stack trace from a debug build, which oddly printed it twice. This is from the check-openmp test offloading_success.c which is the simplest one. save-temps unfortunately generates IR which doesn't trigger the same failure.

 #0 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) $HOME/llvm-project/llvm/lib/Support/Unix/Signals.inc:569:11
 #1 PrintStackTraceSignalHandler(void*) $HOME/llvm-project/llvm/lib/Support/Unix/Signals.inc:636:1
 #2 llvm::sys::RunSignalHandlers() $HOME/llvm-project/llvm/lib/Support/Signals.cpp:102:5
 #3 SignalHandler(int) $HOME/llvm-project/llvm/lib/Support/Unix/Signals.inc:407:1
 #4 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x13140)
 #5 raise ./signal/../sysdeps/unix/sysv/linux/raise.c:51:1
 #6 abort ./stdlib/abort.c:81:7
 #7 llvm::report_fatal_error(llvm::Twine const&, bool) $HOME/llvm-project/llvm/lib/Support/ErrorHandling.cpp:125:5
 #8 $HOME/llvm-project/llvm/lib/Support/ErrorHandling.cpp:83:3
 #9 llvm::AMDGPUMCCodeEmitter::getBinaryCodeForInstr(llvm::MCInst const&, llvm::SmallVectorImpl<llvm::MCFixup>&, llvm::APInt&, llvm::APInt&, llvm::MCSubtargetInfo const&) const $HOME/llvm-build-debug/llvm/lib/Target/AMDGPU/AMDGPUGenMCCodeEmitter.inc:63586:10
#10 (anonymous namespace)::SIMCCodeEmitter::encodeInstruction(llvm::MCInst const&, llvm::raw_ostream&, llvm::SmallVectorImpl<llvm::MCFixup>&, llvm::MCSubtargetInfo const&) const $HOME/llvm-project/llvm/lib/Target/AMDGPU/MCTargetDesc/SIMCCodeEmitter.cpp:321:3
#11 llvm::MCELFStreamer::emitInstToData(llvm::MCInst const&, llvm::MCSubtargetInfo const&) $HOME/llvm-project/llvm/lib/MC/MCELFStreamer.cpp:554:22
#12 llvm::MCObjectStreamer::emitInstructionImpl(llvm::MCInst const&, llvm::MCSubtargetInfo const&) $HOME/llvm-project/llvm/lib/MC/MCObjectStreamer.cpp:440:5
#13 llvm::MCObjectStreamer::emitInstruction(llvm::MCInst const&, llvm::MCSubtargetInfo const&) $HOME/llvm-project/llvm/lib/MC/MCObjectStreamer.cpp:419:3
#14 llvm::AsmPrinter::EmitToStreamer(llvm::MCStreamer&, llvm::MCInst const&) $HOME/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:404:1
#15 llvm::AMDGPUAsmPrinter::emitInstruction(llvm::MachineInstr const*) $HOME/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp:275:5
#16 llvm::AsmPrinter::emitFunctionBody() $HOME/llvm-project/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:1660:13
#17 llvm::AMDGPUAsmPrinter::runOnMachineFunction(llvm::MachineFunction&) $HOME/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp:515:28
#18 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) $HOME/llvm-project/llvm/lib/CodeGen/MachineFunctionPass.cpp:91:8
#19 llvm::FPPassManager::runOnFunction(llvm::Function&) $HOME/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1430:23
#20 llvm::FPPassManager::runOnModule(llvm::Module&) $HOME/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1476:16
#21 (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) $HOME/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1545:23
#22 llvm::legacy::PassManagerImpl::run(llvm::Module&) $HOME/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:535:16
#23 llvm::legacy::PassManager::run(llvm::Module&) $HOME/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1672:3
#24 codegen(llvm::lto::Config const&, llvm::TargetMachine*, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex const&) $HOME/llvm-project/llvm/lib/LTO/LTOBackend.cpp:413:7
#25 llvm::lto::backend(llvm::lto::Config const&, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex&) $HOME/llvm-project/llvm/lib/LTO/LTOBackend.cpp:510:5
#26 llvm::lto::LTO::runRegularLTO(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int)>) $HOME/llvm-project/llvm/lib/LTO/LTO.cpp:1144:13
#27 llvm::lto::LTO::run(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int)>, std::function<llvm::Expected<std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int)>> (unsigned int, llvm::StringRef)>) $HOME/llvm-project/llvm/lib/LTO/LTO.cpp:1040:18
#28 (anonymous namespace)::linkBitcodeFiles(llvm::SmallVectorImpl<llvm::object::OffloadFile>&, llvm::SmallVectorImpl<llvm::StringRef>&, llvm::opt::ArgList const&) $HOME/llvm-project/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp:861:19
#29 (anonymous namespace)::linkAndWrapDeviceFiles(llvm::SmallVectorImpl<llvm::object::OffloadFile>&, llvm::opt::InputArgList const&) $HOME/llvm-project/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp:1140:15
#30 main $HOME/llvm-project/clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp:1369:10
#31 __libc_start_main ./csu/../csu/libc-start.c:308:16
#32 _start ($HOME/llvm-build-debug/llvm/./bin/clang-linker-wrapper+0x457d3a)

I'm unclear whether this is a latent problem exposed by D131246 or a novel failure mode. Since this patch works as written - a benefit from defaulting to module - I'm hoping we can land this, then fix that lowering bug, then change the default to hybrid. Sound reasonable?

[jmmartinez]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

497

I might be wrong, but I think the copy-elision is done for the newly created pair, but not for the arguments used for create it.

I'd rather do:

return {std::move(direct_map_kernel), std::move(indirect_map_kernel)};

Without move: https://godbolt.org/z/f5Ghr8M1o
With move: https://godbolt.org/z/WeG7Y8nY8

511

Is there a missing const& for the Variables argument ?

537

Could you use a const& for kernels ?

775–778

In the case HybridModuleRoot is null, the reference HybridModuleRootKernels will point to a temporary that has already been destroyed.

1233

What do you think about taking an r-value reference for LDSVarsToTransformArg. That would avoid the copy into the local variable LDSVarsToTransform.

llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp

176–177

Cosmetic: There is a shorter ConstantInt::get that takes directly an APInt and an LLVMContext.

[foad]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

511

Isn't std::vector<GlobalVariable *> const & is just a long way of saying ArrayRef<GlobalVariable *>?

[jmmartinez]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

511

+1

[JonChesterfield]

• addressing comments

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

522

SmallVector

554

Use NumberKernels from above?

555

Extra line

592

getFirstNonPHIOrDbgOrAlloca

606

Extra line

609

The constant expression case matters?

In particular instruction uses of addrspacecasted globals come up a lot. The constant initializer case is harder

[JonChesterfield]

• address Matt's comments

[JonChesterfield]

@arsenm thanks, all addressed. Constantexpr were a recurring pain for this pass because their uniqueness interferes with the callgraph walking so are unconditionally rewritten into instructions at the top of run().

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

14–49

reluctant to document this in user facing fashion - there's a credible risk of it being improved in the future, and none of the flags here are intended to be used on a per application basis

497

sure

511

arrayref is good, thanks

537

got some mileage out of ArrayRef for these arguments, thanks

609

Constant expressions involving LDS variables are unconditionally expanded before this point by eliminateConstantExprUsesOfLDSFromAllInstructions.

Primarily because one constantexpr can be reachable from multiple kernels through instructions which are only reachable from distinct kernels, so eliminating them up front makes the analysis more precise as well as the rest of this pass much simpler.

Constant initializer is indeed harder and is presently out of scope - LDS variables have undef as their initialiser, and non-LDS variables have program scope which is broadly resistant to per-kernel-execution initialisation with the address of LDS variables.

An exception for that can be carved out for invariant LDS addresses (e.g. if we want an attribute to say a given global is always at offset 42, whatever kernel is involved), but we don't have a concept for that implemented yet beyond the emergent feature of the back end calculating offsets into a per-kernel struct.

642

changed it to i32 for now, can always back it back down to i16 with more error reporting later if we want to. Stopped short of dropping inttoptr as it's difficult to test with D131246 applied

775–778

Today I learned temporary lifetime extension doesn't work through the ternary operator. Sad times. Also using nullptr as the key into the map wins me a segfault, so gone with ugly workaround pending a better idea.

819–822

i think this is OK as is - it's an undocumented command line flag, where "kernel" is commented as only applicable in narrow cases - and the report_fatal_error path is otherwise only reachable on a bug in this pass

1233

this was refactoring noise, inlined replaceLDSVariablesWithStructVec into replaceLDSVariablesWithStruct to remove

[arsenm]

LGTM with some nits

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

979

Capitalize

1271

Junk comment?

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

140

Capitalize

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

6007

Newlines

6013

More early return?

[JonChesterfield]

• remove obsolete comment

[JonChesterfield]

I can't undo what arc has done here. Apologies for the mass spam.

[philnik]

In D138141#3974603, @JonChesterfield wrote:

I can't undo what arc has done here. Apologies for the mass spam.

You have to update the diff first and then remove the people. Otherwise Herald will just re-add them.

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

SDValue AMDGPUTargetLowering::LowerGlobalAddress(AMDGPUMachineFunction* MFI,
                                                  SelectionDAG &DAG) const {
 
   const DataLayout &DL = DAG.getDataLayout();
   GlobalAddressSDNode *G = cast<GlobalAddressSDNode>(Op);
   const GlobalValue *GV = G->getGlobal();
 
   if (G->getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS ||
       G->getAddressSpace() == AMDGPUAS::REGION_ADDRESS) {
+
+    if (G->getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
+      // special case handling for kernel block variable
+      // it's allocated in the kernel at a predictable address
+      // so that uses of it from functions and globals can be
+      // resolved here
+      // This only works if the current function is called from the kernel
+      // with the corresponding global
+      if (const GlobalVariable *GV2 = dyn_cast<const GlobalVariable>(GV)) {
+        if (MFI->isKnownAddressLDSGlobal(*GV2)) {
+          unsigned offset = MFI->calculateKnownAddressOfLDSGlobal(*GV2);
+
+          return DAG.getConstant(offset + G->getOffset(), SDLoc(Op),
+                                 Op.getValueType());
+        }
+      }
+    }
+
     if (!MFI->isModuleEntryFunction() &&
         !GV->getName().equals("llvm.amdgcn.module.lds")) {
       SDLoc DL(Op);
       const Function &Fn = DAG.getMachineFunction().getFunction();
       DiagnosticInfoUnsupported BadLDSDecl(
         Fn, "local memory global used by non-kernel function",
         DL.getDebugLoc(), DS_Warning);
       DAG.getContext()->diagnose(BadLDSDecl);

llvm/lib/Target/AMDGPU/AMDGPULowerModuleLDSPass.cpp

 //===-- AMDGPULowerModuleLDSPass.cpp ------------------------------*- C++ -*-=//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This pass eliminates LDS uses from non-kernel functions.
+// This pass eliminates local data store, LDS, uses from non-kernel functions.
+// LDS is contiguous memory allocated per kernel execution.
 //
-// The strategy is to create a new struct with a field for each LDS variable
-// and allocate that struct at the same address for every kernel. Uses of the
-// original LDS variables are then replaced with compile time offsets from that
-// known address. AMDGPUMachineFunction allocates the LDS global.
+// Background.
 //
-// Local variables with constant annotation or non-undef initializer are passed
+// The programming model is global variables, or equivalently function local
+// static variables, accessible from kernels or other functions. For uses from
+// kernels this is straightforward - assign an integer to the kernel for the
+// memory required by all the variables combined, allocate them within that.
+// For uses from functions there are performance tradeoffs to choose between.
+//
+// This model means the GPU runtime can specify the amount of memory allocated.
+// If this is more than the kernel assumed, the excess can be made available
+// using a language specific feature, which IR represents as a variable with
+// no initializer. This feature is not yet implemented for non-kernel functions.
+// This lowering could be extended to handle that use case, but would probably
+// require closer integration with promoteAllocaToLDS.
+//
+// Consequences of this GPU feature:
+// - memory is limited and exceeding it halts compilation
+// - a global accessed by one kernel exists independent of other kernels
+// - a global exists independent of simultaneous execution of the same kernel
+// - the address of the global may be different from different kernels as they
+//   do not alias, which permits only allocating variables they use
+// - if the address is allowed to differ, functions need help to find it
+//
+// Uses from kernels are implemented here by grouping them in a per-kernel
+// struct instance. This duplicates the variables, accurately modelling their
+// aliasing properties relative to a single global representation. It also
+// permits control over alignment via padding.
+//
+// Uses from functions are more complicated and the primary purpose of this
+// IR pass. Several different lowering are chosen between to meet requirements
+// to avoid allocating any LDS where it is not necessary, as that impacts
+// occupancy and may fail the compilation, while not imposing overhead on a
+// feature whose primary advantage over global memory is performance. The basic
+// design goal is to avoid one kernel imposing overhead on another.
+//
+// Implementation.
+//
+// LDS variables with constant annotation or non-undef initializer are passed
 // through unchanged for simplification or error diagnostics in later passes.
+// Non-undef initializers are not yet implemented for LDS.
+//
+// LDS variables that are always allocated at the same address can be found
+// by lookup at that address. Otherwise runtime information/cost is required.
+//
+// The simplest strategy possible is to group all LDS variables in a single
+// struct and allocate that struct in every kernel such that the original
+// variables are always at the same address. LDS is however a limited resource
+// so this strategy is unusable in practice. It is not implemented here.
+//
+// Strategy | Precise allocation | Zero runtime cost | General purpose |
+//  --------+--------------------+-------------------+-----------------+
+//   Module |                 No |               Yes |             Yes |
+//    Table |                Yes |                No |             Yes |
+//   Kernel |                Yes |               Yes |              No |
+//   Hybrid |                Yes |           Partial |             Yes |
 //
-// To reduce the memory overhead variables that are only used by kernels are
-// excluded from this transform. The analysis to determine whether a variable
-// is only used by a kernel is cheap and conservative so this may allocate
-// a variable in every kernel when it was not strictly necessary to do so.
+// Module spends LDS memory to save cycles. Table spends cycles and global
+// memory to save LDS. Kernel is as fast as kernel allocation but only works
+// for variables that are known reachable from a single kernel. Hybrid picks
+// between all three. When forced to choose between LDS and cycles it minimises
+// LDS use.
+
+// The "module" lowering implemented here finds LDS variables which are used by
+// non-kernel functions and creates a new struct with a field for each of those
+// LDS variables. Variables that are only used from kernels are excluded.
+// Kernels that do not use this struct are annoteated with the attribute
+// amdgpu-elide-module-lds which allows the back end to elide the allocation.
 //
-// A possible future refinement is to specialise the structure per-kernel, so
-// that fields can be elided based on more expensive analysis.
+// The "table" lowering implemented here has three components.
+// First kernels are assigned a unique integer identifier which is available in
+// functions it calls through the intrinsic amdgcn_lds_kernel_id. The integer
+// is passed through a specific SGPR, thus works with indirect calls.
+// Second, each kernel allocates LDS variables independent of other kernels and
+// writes the addresses it chose for each variable into an array in consistent
+// order. If the kernel does not allocate a given variable, it writes undef to
+// the corresponding array location. These arrays are written to a constant
+// table in the order matching the kernel unique integer identifier.
+// Third, uses from non-kernel functions are replaced with a table lookup using
+// the intrinsic function to find the address of the variable.
+//
+// "Kernel" lowering is only applicable for variables that are unambiguously
+// reachable from exactly one kernel. For those cases, accesses to the variable
+// can be lowered to ConstantExpr address of a struct instance specific to that
+// one kernel. This is zero cost in space and in compute. It will raise a fatal
+// error on any variable that might be reachable from multiple kernels and is
+// thus most easily used as part of the hybrid lowering strategy.
+//
+// Hybrid lowering is a mixture of the above. It uses the zero cost kernel
+// lowering where it can. It lowers the variable accessed by the greatest
+// number of kernels using the module strategy as that is free for the first
+// variable. Any futher variables that can be lowered with the module strategy
+// without incurring LDS memory overhead are. The remaining ones are lowered
+// via table.
+//
+// Consequences
+// - No heuristics or user controlled magic numbers, hybrid is the right choice
+// - Kernels that don't use functions (or have had them all inlined) are not
+//   affected by any lowering for kernels that do.
+// - Kernels that don't make indirect function calls are not affected by those
+//   that do.
+// - Variables which are used by lots of kernels, e.g. those injected by a
+//   language runtime in most kernels, are expected to have no overhead
+// - Implementations that instantiate templates per-kernel where those templates
+//   use LDS are expected to hit the "Kernel" lowering strategy
+// - The runtime properties impose a cost in compiler implementation complexity
 //
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "Utils/AMDGPUBaseInfo.h"
 #include "Utils/AMDGPUMemoryUtils.h"
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SetOperations.h"
 #include "llvm/ADT/SetVector.h"
+#include "llvm/ADT/StringSwitch.h"
 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicsAMDGPU.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/OptimizedStructLayout.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
+
 #include <tuple>
 #include <vector>
 
+#include <cstdio>
+
 #define DEBUG_TYPE "amdgpu-lower-module-lds"
 
 using namespace llvm;
 
 static cl::opt<bool> SuperAlignLDSGlobals(
     "amdgpu-super-align-lds-globals",
     cl::desc("Increase alignment of LDS if it is not on align boundary"),
     cl::init(true), cl::Hidden);
 
+static struct LoweringKind {
+  static constexpr const char name[] = "amdgpu-lower-module-lds-strategy";
+
+  enum Opts { module, table, kernel, hybrid, unknown };
+
+  Opts Value = Opts::module;
+  operator Opts() const { return Value; }
+
+  void operator=(const std::string &Arg) {
+    Value = StringSwitch<Opts>(Arg)
+                .Case("module", Opts::module)
+                .Case("table", Opts::table)
+                .Case("kernel", Opts::kernel)
+                .Case("hybrid", Opts::hybrid)
+                .Default(Opts::unknown);
+
+    if (Value == Opts::unknown)
+      report_fatal_error(Twine("Invalid argument '" + Arg + "' to " + name));
+  }
+} LoweringKindLoc;
+
+static cl::opt<LoweringKind, true, cl::parser<std::string>>
+    LowerFunctionLDSStrategy(
+        LoweringKind::name,
+        cl::desc("Specify lowering strategy for function LDS access:"
+                 "\ntable   Lower via table lookup"
+                 "\nmodule  Lower via module struct"
+                 "\nkernel  Lower variables reachable from one kernel, "
+                 "otherwise abort"
+                 "\nhybrid  Lower via mixture of above strategies"),
+        cl::location(LoweringKindLoc), cl::Hidden);
+
 namespace {
+
+static const bool verbose = false;
+
+bool isKernelLDS(const Function *F) {
+  // Some weirdness here. AMDGPU::isKernelCC does not call into
+  // AMDGPU::isKernel with the calling conv, it instead calls into
+  // isModuleEntryFunction which returns true for more calling conventions
+  // than AMDGPU::isKernel does. There's a FIXME on AMDGPU::isKernel.
+  // There's also a test that checks that the LDS lowering does not hit on
+  // a graphics shader, denoted amdgpu_ps, so stay with the limited case.
+  // Putting LDS in the name of the function to draw attention to this.
+  return AMDGPU::isKernel(F->getCallingConv());
+}
+
 class AMDGPULowerModuleLDS : public ModulePass {
 
   static void removeFromUsedList(Module &M, StringRef Name,
                                  SmallPtrSetImpl<Constant *> &ToRemove) {
     GlobalVariable *GV = M.getNamedGlobal(Name);
     if (!GV || ToRemove.empty()) {
       return;
     }
if (!Init.empty()) {
           ArrayType::get(Type::getInt8PtrTy(M.getContext()), Init.size());
       GV =
           new llvm::GlobalVariable(M, ATy, false, GlobalValue::AppendingLinkage,
                                    ConstantArray::get(ATy, Init), Name);
       GV->setSection("llvm.metadata");
     }
   }
 
-  static void
-  removeFromUsedLists(Module &M,
-                      const std::vector<GlobalVariable *> &LocalVars) {
+  static void removeFromUsedLists(Module &M,
+                                  const DenseSet<GlobalVariable *> &LocalVars) {
     // The verifier rejects used lists containing an inttoptr of a constant
     // so remove the variables from these lists before replaceAllUsesWith
 
     SmallPtrSet<Constant *, 32> LocalVarsSet;
     for (GlobalVariable *LocalVar : LocalVars)
       if (Constant *C = dyn_cast<Constant>(LocalVar->stripPointerCasts()))
         LocalVarsSet.insert(C);
     removeFromUsedList(M, "llvm.used", LocalVarsSet);

 
 public:
   static char ID;
 
   AMDGPULowerModuleLDS() : ModulePass(ID) {
     initializeAMDGPULowerModuleLDSPass(*PassRegistry::getPassRegistry());
   }
 
+  using FunctionVariableMap = DenseMap<Function *, DenseSet<GlobalVariable *>>;
+
+  using VariableFunctionMap = DenseMap<GlobalVariable *, DenseSet<Function *>>;
+
+  static void getUsesOfLDSByFunction(CallGraph const &CG, Module &M,
+                                     FunctionVariableMap &kernels,
+                                     FunctionVariableMap &functions) {
+
+    // Get uses from the current function, excluding uses by called functions
+    // Two output variables to avoid walking the globals list twice
+    for (auto &GV : M.globals()) {
+      if (!AMDGPU::isLDSVariableToLower(GV)) {
+        continue;
+      }
+
+      std::string varname = GV.getName().str();
+      if (verbose)
+        fprintf(stderr, "Found LDS variable: %s\n", varname.c_str());
+
+      SmallVector<User *, 16> Stack(GV.users());
+      for (User *V : GV.users()) {
+        if (auto *I = dyn_cast<Instruction>(V)) {
+          Function *F = I->getFunction();
+          std::string funcname = F->getName().str();
+          if (isKernelLDS(F)) {
+            if (verbose)
+              fprintf(stderr, "  Var %s used by kernel %s\n", varname.c_str(),
+                      funcname.c_str());
+            kernels[F].insert(&GV);
+          } else {
+            functions[F].insert(&GV);
+            if (verbose)
+              fprintf(stderr, "  Var %s used by function %s\n", varname.c_str(),
+                      funcname.c_str());
+          }
+        }
+      }
+    }
+  }
+
+  struct LDSUsesInfoTy {
+    FunctionVariableMap direct_access;
+    FunctionVariableMap indirect_access;
+  };
+
+  static LDSUsesInfoTy getTransitiveUsesOfLDS(CallGraph const &CG, Module &M) {
+
+    FunctionVariableMap direct_map_kernel;
+    FunctionVariableMap direct_map_function;
+    getUsesOfLDSByFunction(CG, M, direct_map_kernel, direct_map_function);
+
+    // Collect variables that are used by functions whose address has escaped
+    DenseSet<GlobalVariable *> VariablesReachableThroughFunctionPointer;
+    for (Function &F : M.functions()) {
+      if (!isKernelLDS(&F))
+        if (F.hasAddressTaken(nullptr, false, false, true /* ignore llvmused */,
+                              false)) {
+          set_union(VariablesReachableThroughFunctionPointer,
+                    direct_map_function[&F]);
+        }
+    }
+
+    auto functionMakesUnknownCall = [&](const Function *F) -> bool {
+      assert(!F->isDeclaration());
+      for (CallGraphNode::CallRecord R : *CG[F]) {
+        if (!R.second->getFunction()) {
+          return true;
+        }
+      }
+      return false;
+    };
+
+    // Work out which variables are reachable through function calls
+    FunctionVariableMap transitive_map_function = direct_map_function;
+
+    // If the function makes any unknown call, assume the worst case that it can
+    // access all variables accessed by functions whose address escaped
+    for (Function &F : M.functions()) {
+      if (!F.isDeclaration() && functionMakesUnknownCall(&F)) {
+        if (!isKernelLDS(&F)) {
+          set_union(transitive_map_function[&F],
+                    VariablesReachableThroughFunctionPointer);
+        }
+      }
+    }
+
+    // Direct implementation of collecting all variables reachable from each
+    // function
+    for (Function &Func : M.functions()) {
+      if (Func.isDeclaration() || isKernelLDS(&Func))
+        continue;
+
+      DenseSet<Function *> seen; // catches cycles
+      SmallVector<Function *, 4> wip{&Func};
+
+      while (!wip.empty()) {
+        Function *F = wip.pop_back_val();
+
+        // Can accelerate this by referring to transitive map for functions that
+        // have already been computed, with more care than this
+        set_union(transitive_map_function[&Func], direct_map_function[F]);
+
+        for (CallGraphNode::CallRecord R : *CG[F]) {
+          Function *ith = R.second->getFunction();
+          if (ith) {
+            if (!seen.contains(ith)) {
+              seen.insert(ith);
+              wip.push_back(ith);
+            }
+          }
+        }
+      }
+    }
+
+    // direct_map_kernel lists which variables are used by the kernel
+    // find the variables which are used through a function call
+    FunctionVariableMap indirect_map_kernel;
+
+    for (Function &Func : M.functions()) {
+      if (Func.isDeclaration() || !isKernelLDS(&Func))
+        continue;
+
+      for (CallGraphNode::CallRecord R : *CG[&Func]) {
+        Function *ith = R.second->getFunction();
+        if (ith) {
+          set_union(indirect_map_kernel[&Func], transitive_map_function[ith]);
+        } else {
+          set_union(indirect_map_kernel[&Func],
+                    VariablesReachableThroughFunctionPointer);
+        }
+      }
+    }
+
+    return {direct_map_kernel, indirect_map_kernel};
+  }
+
+  struct LDSVariableReplacement {
+    GlobalVariable *SGV = nullptr;
+    DenseMap<GlobalVariable *, Constant *> LDSVarsToConstantGEP;
+  };
+
+  // remap from lds global to a constantexpr gep to where it has been moved to
+  // for each kernel
+  // an array with an element for each kernel containing where the corresponding
+  // variable was remapped to
+
+  static Constant *getAddressesOfVariablesInKernel(
+      LLVMContext &Ctx, std::vector<GlobalVariable *> Variables,
+      DenseMap<GlobalVariable *, Constant *> &LDSVarsToConstantGEP) {
+    // Create a ConstantArray containing the address of each Variable within the
+    // kernel corresponding to LDSVarsToConstantGEP, or undef if that kernel
+    // does not allocate it
+
+    auto I16 = Type::getInt16Ty(Ctx);
+    auto I32 = Type::getInt32Ty(Ctx);
+
+    ArrayType *KernelOffsetsType = ArrayType::get(I16, Variables.size());
+
+    std::vector<Constant *> Elements;
+    for (size_t i = 0; i < Variables.size(); i++) {
+      GlobalVariable *GV = Variables[i];
+      if (LDSVarsToConstantGEP.count(GV) != 0) {
+
+        auto elt = ConstantExpr::getPtrToInt(LDSVarsToConstantGEP[GV], I32);
+
+        // creates the same thing as getPtrToInt to I16
+        auto trunc = ConstantExpr::getTrunc(elt, I16);
+
+        Elements.push_back(trunc);
+      } else {
+        Elements.push_back(UndefValue::get(I16));
+      }
+    }
+    return ConstantArray::get(KernelOffsetsType, Elements);
+  }
+
+  static GlobalVariable *buildLookupTable(
+      Module &M, std::vector<GlobalVariable *> const &Variables,
+      std::vector<Function *> kernels,
+      DenseMap<Function *, LDSVariableReplacement> &KernelToReplacement) {
+    if (Variables.empty()) {
+      return nullptr;
+    }
+    LLVMContext &Ctx = M.getContext();
+
+    size_t NumberVariables = Variables.size();
+    size_t NumberKernels = kernels.size();
+
+    ArrayType *KernelOffsetsType =
+        ArrayType::get(Type::getInt16Ty(Ctx), NumberVariables);
+
+    ArrayType *AllKernelsOffsetsType =
+        ArrayType::get(KernelOffsetsType, NumberKernels);
+
+    std::vector<Constant *> overallConstantExprElts(kernels.size());
+    for (size_t i = 0; i < kernels.size(); i++) {
+
+      LDSVariableReplacement Replacement = KernelToReplacement[kernels[i]];
+
+      overallConstantExprElts[i] = getAddressesOfVariablesInKernel(
+          Ctx, Variables, Replacement.LDSVarsToConstantGEP);
+    }
+
+    Constant *init =
+        ConstantArray::get(AllKernelsOffsetsType, overallConstantExprElts);
+
+    return new GlobalVariable(
+        M, AllKernelsOffsetsType, true, GlobalValue::InternalLinkage, init,
+        "llvm.amdgcn.lds_offset_table", nullptr, GlobalValue::NotThreadLocal,
+        AMDGPUAS::CONSTANT_ADDRESS);
+  }
+
+  void replaceUsesInInstructionsWithTableLookup(
+      Module &M, std::vector<GlobalVariable *> const &ModuleScopeVariables,
+      GlobalVariable *LookupTable) {
+
+    LLVMContext &Ctx = M.getContext();
+    IRBuilder<> Builder(Ctx);
+    Type *I16 = Type::getInt16Ty(Ctx);
+    Type *I32 = Type::getInt32Ty(Ctx);
+
+    // Accesses from a function use the amdgcn_lds_kernel_id intrinsic which
+    // lowers to a read from a live in register. Emit it once in the entry
+    // block to spare deduplicating it later.
+
+    DenseMap<Function *, Value *> tableKernelIndexCache;
+    auto getTableKernelIndex = [&](Function *F) -> Value * {
+      if (tableKernelIndexCache.count(F) == 0) {
+        LLVMContext &Ctx = M.getContext();
+        Type *I32 = Type::getInt32Ty(Ctx);
+        FunctionType *FTy = FunctionType::get(I32, {});
+        Function *Decl =
+            Intrinsic::getDeclaration(&M, Intrinsic::amdgcn_lds_kernel_id, {});
+
+        llvm::BasicBlock::iterator it =
+            F->getEntryBlock().getFirstInsertionPt();
+        Instruction &i = *it;
+        Builder.SetInsertPoint(&i);
+
+        tableKernelIndexCache[F] = Builder.CreateCall(FTy, Decl, {});
+      }
+
+      return tableKernelIndexCache[F];
+    };
+
+    for (size_t Index = 0; Index < ModuleScopeVariables.size(); Index++) {
+      auto *GV = ModuleScopeVariables[Index];
+
+      for (Use &U : llvm::make_early_inc_range(GV->uses())) {
+
+        auto *I = dyn_cast<Instruction>(U.getUser());
+        if (!I) {
+          continue;
+        }
+        Value *tableKernelIndex = getTableKernelIndex(I->getFunction());
+
+        // So if the phi uses this value multiple times, what does this look
+        // like?
+        if (auto *Phi = dyn_cast<PHINode>(I)) {
+          BasicBlock *BB = Phi->getIncomingBlock(U);
+          Builder.SetInsertPoint(&(*(BB->getFirstInsertionPt())));
+        } else {
+          Builder.SetInsertPoint(I);
+        }
+
+        Value *GEPIdx[3] = {
+            ConstantInt::get(I32, 0),
+            tableKernelIndex,
+            ConstantInt::get(I32, Index),
+        };
+
+        Value *Address = Builder.CreateInBoundsGEP(
+            LookupTable->getValueType(), LookupTable, GEPIdx, GV->getName());
+
+        Value *loaded = Builder.CreateLoad(I16, Address);
+        Value *extended = Builder.CreateZExt(loaded, I32);
+
+        Value *replacement =
+            Builder.CreateIntToPtr(extended, GV->getType(), GV->getName());
+
+        U.set(replacement);
+      }
+    }
+  }
+
+  static DenseSet<Function *> kernelsThatIndirectlyAccessAnyOfPassedVariables(
+      Module &M, LDSUsesInfoTy &LDSUsesInfo,
+      DenseSet<GlobalVariable *> const &VariableSet) {
+    DenseSet<Function *> KernelSet;
+
+    if (!VariableSet.empty()) {
+      for (Function &Func : M.functions()) {
+        if (!Func.isDeclaration() && isKernelLDS(&Func)) {
+          for (GlobalVariable *GV : LDSUsesInfo.indirect_access[&Func]) {
+            if (VariableSet.contains(GV)) {
+              KernelSet.insert(&Func);
+              break;
+            }
+          }
+        }
+      }
+    }
+
+    return KernelSet;
+  }
+
+  static GlobalVariable *
+  chooseBestVariableForModuleStrategy(const DataLayout &DL,
+                                      VariableFunctionMap &LDSVars) {
+    // Find the global variable with the most indirect uses from kernels
+
+    struct props {
+      GlobalVariable *GV;
+      size_t UserCount;
+      size_t Size;
+
+      props() : props(nullptr, 0, 0) {}
+
+      props(GlobalVariable *GV, uint64_t UserCount, uint64_t AllocSize)
+          : GV(GV), UserCount(UserCount), Size(AllocSize) {}
+
+      bool operator<(const props &other) const {
+        // Fewer users makes module scope variable less attractive
+        if (UserCount < other.UserCount) {
+          return true;
+        }
+        if (UserCount > other.UserCount) {
+          return false;
+        }
+
+        // Bigger makes module scope variable less attractive
+        if (Size < other.Size) {
+          return false;
+        }
+
+        if (Size > other.Size) {
+          return true;
+        }
+
+        // arbitrary but consistent
+        return GV->getName() < other.GV->getName();
+      }
+    };
+
+    props MostUsed;
+
+    for (auto &K : LDSVars) {
+      GlobalVariable *GV = K.first;
+      if (K.second.size() <= 1) {
+        // A variable reachable by only one kernel is best lowered with kernel
+        // strategy
+        continue;
+      }
+      props candidate(GV, K.second.size(),
+                      DL.getTypeAllocSize(GV->getValueType()).getFixedValue());
+      if (MostUsed < candidate)
+        MostUsed = candidate;
+    }
+
+    return MostUsed.GV;
+  }
+
   bool runOnModule(Module &M) override {
     LLVMContext &Ctx = M.getContext();
     CallGraph CG = CallGraph(M);
     bool Changed = superAlignLDSGlobals(M);
 
     Changed |= eliminateConstantExprUsesOfLDSFromAllInstructions(M);
 
-    // Move variables used by functions into amdgcn.module.lds
-    std::vector<GlobalVariable *> ModuleScopeVariables =
-        AMDGPU::findLDSVariablesToLower(M, nullptr);
-    if (!ModuleScopeVariables.empty()) {
-      std::string VarName = "llvm.amdgcn.module.lds";
+    Changed = true; // todo: narrow this down
 
-      GlobalVariable *SGV;
-      DenseMap<GlobalVariable *, Constant *> LDSVarsToConstantGEP;
-      std::tie(SGV, LDSVarsToConstantGEP) =
-          createLDSVariableReplacement(M, VarName, ModuleScopeVariables);
+    assert(LoweringKindLoc != LoweringKind::Opts::unknown);
+
+    // For each kernel, what variables does it access directly or through
+    // callees
+    LDSUsesInfoTy LDSUsesInfo = getTransitiveUsesOfLDS(CG, M);
+
+    // For each variable accessed through callees, which kernels access it
+    VariableFunctionMap LDSToKernelsThatNeedToAccessItIndirectly;
+    for (auto &K : LDSUsesInfo.indirect_access) {
+      Function *F = K.first;
+      assert(isKernelLDS(F));
+      for (GlobalVariable *GV : K.second) {
+        if (verbose)
+          fprintf(stderr, "Recording LDSToKernels[%s] -> %s\n",
+                  GV->getName().str().c_str(), F->getName().str().c_str());
+        LDSToKernelsThatNeedToAccessItIndirectly[GV].insert(F);
+      }
+    }
+
+    // Partition variables into the different strategies
+    DenseSet<GlobalVariable *> ModuleScopeVariables;
+    DenseSet<GlobalVariable *> TableLookupVariables;
+    DenseSet<GlobalVariable *> KernelAccessVariables;
+
+    {
+      GlobalVariable *HybridModuleRoot =
+          LoweringKindLoc != LoweringKind::Opts::hybrid
+              ? nullptr
+              : chooseBestVariableForModuleStrategy(
+                    M.getDataLayout(),
+                    LDSToKernelsThatNeedToAccessItIndirectly);
+
+      DenseSet<Function *> const &HybridModuleRootKernels =
+          HybridModuleRoot
+              ? LDSToKernelsThatNeedToAccessItIndirectly[HybridModuleRoot]
+              : DenseSet<Function *>{};
+
+      for (auto &K : LDSToKernelsThatNeedToAccessItIndirectly) {
+        // Each iteration of this loop assigns exactly one global variable to
+        // exactly one of the implementation strategies.
+
+        GlobalVariable *GV = K.first;
+        assert(AMDGPU::isLDSVariableToLower(*GV));
+        assert(K.second.size() != 0);
+
+        switch (LoweringKindLoc) {
+        default:
+        case LoweringKind::Opts::unknown:
+          __builtin_unreachable(); // would report fatal error from LoweringKind
+
+        case LoweringKind::Opts::module:
+          ModuleScopeVariables.insert(GV);
+          break;
+
+        case LoweringKind::Opts::table:
+          TableLookupVariables.insert(GV);
+          break;
+
+        case LoweringKind::Opts::kernel:
+          if (K.second.size() == 1) {
+            KernelAccessVariables.insert(GV);
+          } else {
+            report_fatal_error("Cannot lower LDS to kernel access as it is "
+                               "reachable from multiple kernels");
+          }
+          break;
+
+        case LoweringKind::Opts::hybrid: {
+          if (GV == HybridModuleRoot) {
+            assert(K.second.size() != 1);
+            ModuleScopeVariables.insert(GV);
+          } else if (K.second.size() == 1) {
+            KernelAccessVariables.insert(GV);
+          } else if (set_is_subset(K.second, HybridModuleRootKernels)) {
+            ModuleScopeVariables.insert(GV);
+          } else {
+            TableLookupVariables.insert(GV);
+          }
+          break;
+        }
+        }
+      }
+
+      assert(ModuleScopeVariables.size() + TableLookupVariables.size() +
+                 KernelAccessVariables.size() ==
+             LDSToKernelsThatNeedToAccessItIndirectly.size());
+    }
 
-      appendToCompilerUsed(
-          M, {static_cast<GlobalValue *>(
+    // Variables have now been partitioned into the three lowering strategies.
+    if (verbose) {
+      for (auto &GV : ModuleScopeVariables) {
+        fprintf(stderr, "Module: %s\n", GV->getName().str().c_str());
+      }
+      for (auto &GV : TableLookupVariables) {
+        fprintf(stderr, "Table:  %s\n", GV->getName().str().c_str());
+      }
+      for (auto &GV : KernelAccessVariables) {
+        fprintf(stderr, "Direct: %s\n", GV->getName().str().c_str());
+      }
+    }
+
+    // If the kernel accesses a variable that is going to be stored in the
+    // module instance through a call then that kernel needs to allocate the
+    // module instance
+    DenseSet<Function *> KernelsThatAllocateModuleLDS =
+        kernelsThatIndirectlyAccessAnyOfPassedVariables(M, LDSUsesInfo,
+                                                        ModuleScopeVariables);
+    DenseSet<Function *> KernelsThatAllocateTableLDS =
+        kernelsThatIndirectlyAccessAnyOfPassedVariables(M, LDSUsesInfo,
+                                                        TableLookupVariables);
+
+    if (!ModuleScopeVariables.empty()) {
+      LDSVariableReplacement ModuleScopeReplacement =
+          createLDSVariableReplacement(M, "llvm.amdgcn.module.lds",
+                                       ModuleScopeVariables);
+
+      appendToCompilerUsed(M,
+                           {static_cast<GlobalValue *>(
                                ConstantExpr::getPointerBitCastOrAddrSpaceCast(
-                     cast<Constant>(SGV), Type::getInt8PtrTy(Ctx)))});
+                                   cast<Constant>(ModuleScopeReplacement.SGV),
+                                   Type::getInt8PtrTy(Ctx)))});
 
+      // historic
       removeFromUsedLists(M, ModuleScopeVariables);
-      replaceLDSVariablesWithStruct(M, ModuleScopeVariables, SGV,
-                                    LDSVarsToConstantGEP,
-                                    [](Use &) { return true; });
-
-      // This ensures the variable is allocated when called functions access it.
-      // It also lets other passes, specifically PromoteAlloca, accurately
-      // calculate how much LDS will be used by the kernel after lowering.
 
+      // Replace all uses of module scope variable from non-kernel functions
+      replaceLDSVariablesWithStruct(
+          M, ModuleScopeVariables, ModuleScopeReplacement, [&](Use &U) {
+            Instruction *I = dyn_cast<Instruction>(U.getUser());
+            if (!I) {
+              return false;
+            }
+            Function *F = I->getFunction();
+            return !isKernelLDS(F);
+          });
+
+      // Replace uses of module scope variable from kernel functions that
+      // allocate the module scope variable, otherwise leave them unchanged
+      // Record on each kernel whether the module scope global is used by it
+
+      LLVMContext &Ctx = M.getContext();
       IRBuilder<> Builder(Ctx);
+
       for (Function &Func : M.functions()) {
-        if (!Func.isDeclaration() && AMDGPU::isKernelCC(&Func)) {
-          const CallGraphNode *N = CG[&Func];
-          const bool CalleesRequireModuleLDS = N->size() > 0;
-
-          if (CalleesRequireModuleLDS) {
-            // If a function this kernel might call requires module LDS,
-            // annotate the kernel to let later passes know it will allocate
-            // this structure, even if not apparent from the IR.
-            markUsedByKernel(Builder, &Func, SGV);
-          } else {
-            // However if we are certain this kernel cannot call a function that
-            // requires module LDS, annotate the kernel so the backend can elide
-            // the allocation without repeating callgraph walks.
-            Func.addFnAttr("amdgpu-elide-module-lds");
-          }
-        }
-      }
+        if (Func.isDeclaration() || !isKernelLDS(&Func))
+          continue;
+
+        if (KernelsThatAllocateModuleLDS.contains(&Func)) {
+          replaceLDSVariablesWithStruct(
+              M, ModuleScopeVariables, ModuleScopeReplacement, [&](Use &U) {
+                Instruction *I = dyn_cast<Instruction>(U.getUser());
+                if (!I) {
+                  return false;
+                }
+                Function *F = I->getFunction();
+                return F == &Func;
+              });
+
+          markUsedByKernel(Builder, &Func, ModuleScopeReplacement.SGV);
+
 
-      Changed = true;
-    }
-
-    // Move variables used by kernels into per-kernel instances
-    for (Function &F : M.functions()) {
-      if (F.isDeclaration())
+    // Create a struct for each kernel for the non-module-scope variables
+    DenseMap<Function *, LDSVariableReplacement> KernelToReplacement;
+    for (Function &Func : M.functions()) {
+      if (Func.isDeclaration() || !isKernelLDS(&Func))
         continue;
 
-      // Only lower compute kernels' LDS.
-      if (!AMDGPU::isKernel(F.getCallingConv()))
-        continue;
+      DenseSet<GlobalVariable *> KernelUsedVariables;
+      for (auto &v : LDSUsesInfo.direct_access[&Func]) {
+        if (verbose)
+          fprintf(stderr, "Kernel %s uses variable %s itself\n",
+                  Func.getName().str().c_str(), v->getName().str().c_str());
+        KernelUsedVariables.insert(v);
+      }
+      for (auto &v : LDSUsesInfo.indirect_access[&Func]) {
+        if (verbose)
+          fprintf(stderr, "Kernel %s uses variable %s indirectly\n",
+                  Func.getName().str().c_str(), v->getName().str().c_str());
+        KernelUsedVariables.insert(v);
+      }
+
+      // Variables allocated in module lds must all resolve to that struct,
+      // not to the per-kernel instance.
+      if (KernelsThatAllocateModuleLDS.contains(&Func)) {
+        for (GlobalVariable *v : ModuleScopeVariables) {
+          if (verbose)
+            fprintf(
+                stderr,
+                "Kernel %s not directly allocating module.lds variable %s\n",
+                Func.getName().str().c_str(), v->getName().str().c_str());
 
-      std::vector<GlobalVariable *> KernelUsedVariables =
-          AMDGPU::findLDSVariablesToLower(M, &F);
+          KernelUsedVariables.erase(v);
+        }
+      }
+
+      if (KernelUsedVariables.empty()) {
+        // Either used no LDS, or all the LDS it used was also in module
+        continue;
+      }
 
-      if (!KernelUsedVariables.empty()) {
-        // The association between kernel function and LDS struct is done by
-        // symbol name, which only works if the function in question has a name
-        // This is not expected to be a problem in practice as kernels are
-        // called by name making anonymous ones (which are named by the backend)
-        // difficult to use. This does mean that llvm test cases need
-        // to name the kernels.
-        if (!F.hasName()) {
-          report_fatal_error("Anonymous kernels cannot use LDS variables");
-        }
+      // symbol name, which only works if the function in question has a
+      // name This is not expected to be a problem in practice as kernels
+      // are called by name making anonymous ones (which are named by the
+      // backend) difficult to use. This does mean that llvm test cases need
+      if (!Func.hasName()) {
 
-        std::string VarName =
-            (Twine("llvm.amdgcn.kernel.") + F.getName() + ".lds").str();
-        GlobalVariable *SGV;
-        DenseMap<GlobalVariable *, Constant *> LDSVarsToConstantGEP;
-        std::tie(SGV, LDSVarsToConstantGEP) =
-            createLDSVariableReplacement(M, VarName, KernelUsedVariables);
+          (Twine("llvm.amdgcn.kernel.") + Func.getName() + ".lds").str();
+
+      auto Replacement =
 
-        removeFromUsedLists(M, KernelUsedVariables);
-        replaceLDSVariablesWithStruct(
-            M, KernelUsedVariables, SGV, LDSVarsToConstantGEP, [&F](Use &U) {
-              Instruction *I = dyn_cast<Instruction>(U.getUser());
-              return I && I->getFunction() == &F;
-            });
-        Changed = true;
+      // remove preserves existing codegen
+      KernelToReplacement[&Func] = Replacement;
+
+      // Rewrite uses within kernel to the new struct
+          M, KernelUsedVariables, Replacement, [&Func](Use &U) {
+            return I && I->getFunction() == &Func;
+    }
+
+    // Lower zero cost accesses to the kernel instances just created
+    for (auto &GV : KernelAccessVariables) {
+      auto &funcs = LDSToKernelsThatNeedToAccessItIndirectly[GV];
+      assert(funcs.size() == 1); // Only one kernel can access it
+      LDSVariableReplacement Replacement =
+          KernelToReplacement[*(funcs.begin())];
+
+      DenseSet<GlobalVariable *> Vec;
+      Vec.insert(GV);
+
+      if (verbose)
+        fprintf(stderr, "Lowering kernel access variable %s\n",
+                GV->getName().str().c_str());
+
+      replaceLDSVariablesWithStruct(M, Vec, Replacement, [](Use &U) {
+        Instruction *I = dyn_cast<Instruction>(U.getUser());
+        if (!I) {
+          return false;
+        }
+
+        assert(!isKernelLDS(I->getFunction()));
+        return true;
+      });
+    }
+
+    if (!KernelsThatAllocateTableLDS.empty()) {
+      // Collect the kernels that allocate table lookup LDS
+      std::vector<Function *> OrderedKernels;
+      {
+        for (Function &Func : M.functions()) {
+          if (Func.isDeclaration())
+            continue;
+          if (!isKernelLDS(&Func))
+            continue;
+
+          if (KernelsThatAllocateTableLDS.contains(&Func)) {
+            assert(Func.hasName()); // else fatal error earlier
+            OrderedKernels.push_back(&Func);
+          }
+        }
+
+        // Put them in an arbitrary but reproducible order
+        llvm::sort(OrderedKernels.begin(), OrderedKernels.end(),
+                   [](const Function *lhs, const Function *rhs) -> bool {
+                     return lhs->getName() < rhs->getName();
+                   });
+
+        // Annotate the kernels with their order in this vector
+        LLVMContext &Ctx = M.getContext();
+        IRBuilder<> Builder(Ctx);
+
+        if (OrderedKernels.size() > UINT32_MAX) {
+          // 32 bit keeps it in one SGPR. > 2**32 kernels won't fit on the GPU
+          report_fatal_error("Unimplemented LDS lowering for > 2**32 kernels");
+        }
+
+        for (size_t i = 0; i < OrderedKernels.size(); i++) {
+          llvm::Metadata *AttrMDArgs[1] = {
+              llvm::ConstantAsMetadata::get(Builder.getInt32(i)),
+          };
+          OrderedKernels[i]->setMetadata("llvm.amdgcn.lds.kernel.id",
+                                         llvm::MDNode::get(Ctx, AttrMDArgs));
+
+          markUsedByKernel(Builder, OrderedKernels[i],
+                           KernelToReplacement[OrderedKernels[i]].SGV);
         }
       }
 
+      // The order must be consistent between lookup table and accesses to
+      // lookup table
+      std::vector<GlobalVariable *> TableLookupVariablesOrdered(
+          TableLookupVariables.begin(), TableLookupVariables.end());
+      llvm::sort(TableLookupVariablesOrdered.begin(),
+                 TableLookupVariablesOrdered.end(),
+                 [](const GlobalVariable *lhs, const GlobalVariable *rhs) {
+                   return lhs->getName() < rhs->getName();
+                 });
+
+      GlobalVariable *LookupTable = buildLookupTable(
+          M, TableLookupVariablesOrdered, OrderedKernels, KernelToReplacement);
+      replaceUsesInInstructionsWithTableLookup(M, TableLookupVariablesOrdered,
+                                               LookupTable);
+    }
+
     for (auto &GV : make_early_inc_range(M.globals()))
       if (AMDGPU::isLDSVariableToLower(GV)) {
+
+        // probably want to remove from used lists
         GV.removeDeadConstantUsers();
         if (GV.use_empty())
           GV.eraseFromParent();
       }
 
     return Changed;
   }
 
for (auto &GV : M.globals()) {
       if (Alignment != AMDGPU::getAlign(DL, &GV)) {
         Changed = true;
         GV.setAlignment(Alignment);
       }
     }
     return Changed;
   }
 
-  std::tuple<GlobalVariable *, DenseMap<GlobalVariable *, Constant *>>
-  createLDSVariableReplacement(
+  static LDSVariableReplacement createLDSVariableReplacement(
       Module &M, std::string VarName,
-      std::vector<GlobalVariable *> const &LDSVarsToTransform) {
+      DenseSet<GlobalVariable *> const &LDSVarsToTransform) {
     // Create a struct instance containing LDSVarsToTransform and map from those
     // variables to ConstantExprGEP
     // Variables may be introduced to meet alignment requirements. No aliasing
     // metadata is useful for these as they have no uses. Erased before return.
 
     LLVMContext &Ctx = M.getContext();
     const DataLayout &DL = M.getDataLayout();
     assert(!LDSVarsToTransform.empty());
for (size_t I = 0; I < LocalVars.size(); I++) {
       if (IsPaddingField[I]) {
         assert(GV->use_empty());
         GV->eraseFromParent();
       } else {
         Map[GV] = GEP;
       }
     }
     assert(Map.size() == LDSVarsToTransform.size());
-    return std::make_tuple(SGV, std::move(Map));
+    return {SGV, std::move(Map)};
   }
 
   template <typename PredicateTy>
   void replaceLDSVariablesWithStruct(
-      Module &M, std::vector<GlobalVariable *> const &LDSVarsToTransform,
-      GlobalVariable *SGV,
-      DenseMap<GlobalVariable *, Constant *> &LDSVarsToConstantGEP,
-      PredicateTy Predicate) {
+      Module &M, DenseSet<GlobalVariable *> const &LDSVarsToTransform,
+      LDSVariableReplacement Replacement, PredicateTy Predicate) {
+    std::vector<GlobalVariable *> LDSVarsToTransformVec(
+        LDSVarsToTransform.begin(), LDSVarsToTransform.end());
+    // This indirection makes a mess of the lit tests checking alias information
+    // TODO: Put the variables in a constant order before assigning the aliasing
+    // info
+    replaceLDSVariablesWithStructVec(M, LDSVarsToTransformVec, Replacement,
+                                     Predicate);
+  }
+
+  template <typename PredicateTy>
+  void replaceLDSVariablesWithStructVec(
+      Module &M, std::vector<GlobalVariable *> const &LDSVarsToTransformArg,
+      LDSVariableReplacement Replacement, PredicateTy Predicate) {
     LLVMContext &Ctx = M.getContext();
     const DataLayout &DL = M.getDataLayout();
 
+    // A hack... need to insert the aliasing info in a predictable order for
+    // lit tests. TODO: Have them in a reasonable order already, ideally the
+    // same order they get allocated
+    std::vector<GlobalVariable *> LDSVarsToTransform(
+        LDSVarsToTransformArg.begin(), LDSVarsToTransformArg.end());
+    llvm::sort(LDSVarsToTransform.begin(), LDSVarsToTransform.end(),
+               [](const GlobalVariable *lhs, const GlobalVariable *rhs) {
+                 return lhs->getName() < rhs->getName();
+               });
+
     // Create alias.scope and their lists. Each field in the new structure
     // does not alias with all other fields.
     SmallVector<MDNode *> AliasScopes;
     SmallVector<Metadata *> NoAliasList;
     const size_t NumberVars = LDSVarsToTransform.size();
     if (NumberVars > 1) {
       MDBuilder MDB(Ctx);
       AliasScopes.reserve(NumberVars);
       MDNode *Domain = MDB.createAnonymousAliasScopeDomain();
       for (size_t I = 0; I < NumberVars; I++) {
         MDNode *Scope = MDB.createAnonymousAliasScope(Domain);
         AliasScopes.push_back(Scope);
       }
       NoAliasList.append(&AliasScopes[1], AliasScopes.end());
     }
 
     // Replace uses of ith variable with a constantexpr to the corresponding
     // field of the instance that will be allocated by AMDGPUMachineFunction
     for (size_t I = 0; I < NumberVars; I++) {
       GlobalVariable *GV = LDSVarsToTransform[I];
-      Constant *GEP = LDSVarsToConstantGEP[GV];
+      Constant *GEP = Replacement.LDSVarsToConstantGEP[GV];
 
       GV->replaceUsesWithIf(GEP, Predicate);
-      if (GV->use_empty()) {
-        GV->eraseFromParent();
-      }
+      // trunk does a eraseFromParent here
 
       APInt APOff(DL.getIndexTypeSizeInBits(GEP->getType()), 0);
       GEP->stripAndAccumulateInBoundsConstantOffsets(DL, APOff);
       uint64_t Offset = APOff.getZExtValue();
 
-      Align A = commonAlignment(SGV->getAlign().valueOrOne(), Offset);
+      Align A =
+          commonAlignment(Replacement.SGV->getAlign().valueOrOne(), Offset);
 
       if (I)
         NoAliasList[I - 1] = AliasScopes[I - 1];
       MDNode *NoAlias =
           NoAliasList.empty() ? nullptr : MDNode::get(Ctx, NoAliasList);
       MDNode *AliasScope =
           AliasScopes.empty() ? nullptr : MDNode::get(Ctx, {AliasScopes[I]});
 

llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp

 //===- AMDGPUMCInstLower.cpp - Lower AMDGPU MachineInstr to an MCInst -----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file
 /// Code to lower AMDGPU MachineInstrs to their corresponding MCInst.
 //
 //===----------------------------------------------------------------------===//
 //
 
 #include "AMDGPUMCInstLower.h"
 #include "AMDGPUAsmPrinter.h"
+#include "AMDGPUMachineFunction.h"
 #include "AMDGPUTargetMachine.h"
 #include "MCTargetDesc/AMDGPUInstPrinter.h"
 #include "MCTargetDesc/AMDGPUMCTargetDesc.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalVariable.h"

 bool AMDGPUAsmPrinter::lowerOperand(const MachineOperand &MO,
                                     MCOperand &MCOp) const {
   const GCNSubtarget &STI = MF->getSubtarget<GCNSubtarget>();
   AMDGPUMCInstLower MCInstLowering(OutContext, STI, *this);
   return MCInstLowering.lowerOperand(MO, MCOp);
 }
 
 const MCExpr *AMDGPUAsmPrinter::lowerConstant(const Constant *CV) {
+
+  // Intercept LDS variables with known addresses
+  if (const GlobalVariable *GV = dyn_cast<GlobalVariable>(CV)) {
+    if (AMDGPUMachineFunction::isKnownAddressLDSGlobal(*GV)) {
+      unsigned offset =
+          AMDGPUMachineFunction::calculateKnownAddressOfLDSGlobal(*GV);
+
+      Constant *C = Constant::getIntegerValue(
+          IntegerType::get(CV->getContext(), 32), APInt(32, offset));
+      return AsmPrinter::lowerConstant(C);
+    }
+  }
+
   if (const MCExpr *E = lowerAddrSpaceCast(TM, CV, OutContext))
     return E;
   return AsmPrinter::lowerConstant(CV);
 }
 
 void AMDGPUAsmPrinter::emitInstruction(const MachineInstr *MI) {
   // FIXME: Enable feature predicate checks once all the test pass.
   // AMDGPU_MC::verifyInstructionPredicates(MI->getOpcode(),

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.h

public:
 
   bool needsWaveLimiter() const {
     return WaveLimiter;
   }
 
   unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV) {
     return allocateLDSGlobal(DL, GV, DynLDSAlign);
   }
+
   unsigned allocateLDSGlobal(const DataLayout &DL, const GlobalVariable &GV,
                              Align Trailing);
 
   void allocateKnownAddressLDSGlobal(const Function &F);
 
   // A kernel function may have an associated LDS allocation, and a kernel-scope
   // LDS allocation must have an associated kernel function
+
+  // LDS allocation should have an associated kernel function
+  static const Function *
+  getKernelLDSFunctionFromGlobal(const GlobalVariable &GV);
   static const GlobalVariable *
   getKernelLDSGlobalFromFunction(const Function &F);
 
+  // Module or kernel scope LDS variable
+  static bool isKnownAddressLDSGlobal(const GlobalVariable &GV);
+  static unsigned calculateKnownAddressOfLDSGlobal(const GlobalVariable &GV);
+
   static Optional<uint32_t> getLDSKernelIdMetadata(const Function &F);
 
   Align getDynLDSAlign() const { return DynLDSAlign; }
 
   void setDynLDSAlign(const DataLayout &DL, const GlobalVariable &GV);
 };
 
 }
 #endif

llvm/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp

if (GV.getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
     // FIXME: Apply alignment of dynamic GDS
     GDSSize = StaticGDSSize;
   }
 
   Entry.first->second = Offset;
   return Offset;
 }
 
+static const char ModuleLDSName[] = "llvm.amdgcn.module.lds";
+
+bool AMDGPUMachineFunction::isKnownAddressLDSGlobal(const GlobalVariable &GV) {
+  auto name = GV.getName();
+  return name.equals(ModuleLDSName) ||
+         (name.startswith("llvm.amdgcn.kernel.") && name.endswith(".lds"));
+}
+
+const Function *AMDGPUMachineFunction::getKernelLDSFunctionFromGlobal(
+    const GlobalVariable &GV) {
+  const Module &M = *GV.getParent();
+  std::string NS{GV.getName()};
+  StringRef N(NS);
+  if (N.consume_front("llvm.amdgcn.kernel.") && N.consume_back(".lds")) {
+    return M.getFunction(N);
+  }
+  return nullptr;
+}
+
 const GlobalVariable *
 AMDGPUMachineFunction::getKernelLDSGlobalFromFunction(const Function &F) {
   const Module *M = F.getParent();
   std::string KernelLDSName = "llvm.amdgcn.kernel.";
   KernelLDSName += F.getName();
   KernelLDSName += ".lds";
   return M->getNamedGlobal(KernelLDSName);
 }
 
 // This kernel calls no functions that require the module lds struct
 static bool canElideModuleLDS(const Function &F) {
   return F.hasFnAttribute("amdgpu-elide-module-lds");
 }
 
+unsigned AMDGPUMachineFunction::calculateKnownAddressOfLDSGlobal(
+    const GlobalVariable &GV) {
+  // module.lds, then alignment padding, then kernel.lds, then other variables
+  // if any
+
+  assert(isKnownAddressLDSGlobal(GV));
+  unsigned offset = 0;
+
+  if (GV.getName().equals(ModuleLDSName)) {
+    return 0;
+  }
+
+  const Module *M = GV.getParent();
+  const DataLayout &DL = M->getDataLayout();
+
+  const GlobalVariable *GVM = M->getNamedGlobal(ModuleLDSName);
+  const Function *f = getKernelLDSFunctionFromGlobal(GV);
+
+  // Account for module.lds if allocated for this function
+  if (GVM && f && !canElideModuleLDS(*f)) {
+    // allocator aligns this to var align, but it's zero to begin with
+    offset += DL.getTypeAllocSize(GVM->getValueType());
+  }
+
+  // No dynamic LDS alignment done by allocateModuleLDSGlobal
+  offset = alignTo(
+      offset, DL.getValueOrABITypeAlignment(GV.getAlign(), GV.getValueType()));
+
+  return offset;
+}
+
 void AMDGPUMachineFunction::allocateKnownAddressLDSGlobal(const Function &F) {
   const Module *M = F.getParent();
 
   // This function is called before allocating any other LDS so that it can
   // reliably put values at known addresses. Consequently, dynamic LDS, if
   // present, will not yet have been allocated
 
   assert(getDynLDSAlign() == Align() && "dynamic LDS not yet allocated");
if (isModuleEntryFunction()) {
     //   llvm.amdgcn.module.lds
     // }
     // alignment padding
     // {
     //   llvm.amdgcn.kernel.some-name.lds
     // }
     // other variables, e.g. dynamic lds, allocated after this call
 
-    const GlobalVariable *GV = M->getNamedGlobal("llvm.amdgcn.module.lds");
+    const GlobalVariable *GV = M->getNamedGlobal(ModuleLDSName);
     const GlobalVariable *KV = getKernelLDSGlobalFromFunction(F);
 
     if (GV && !canElideModuleLDS(F)) {
+      assert(isKnownAddressLDSGlobal(*GV));
       unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *GV, Align());
       (void)Offset;
-      assert(Offset == 0 &&
+      assert(Offset == calculateKnownAddressOfLDSGlobal(*GV) &&
              "Module LDS expected to be allocated before other LDS");
     }
 
     if (KV) {
       // The per-kernel offset is deterministic because it is allocated
       // before any other non-module LDS variables.
+      assert(isKnownAddressLDSGlobal(*KV));
       unsigned Offset = allocateLDSGlobal(M->getDataLayout(), *KV, Align());
       (void)Offset;
+      assert(Offset == calculateKnownAddressOfLDSGlobal(*KV) &&
+             "Kernel LDS expected to be immediately after module LDS");
     }
   }
 }
 
 Optional<uint32_t>
 AMDGPUMachineFunction::getLDSKernelIdMetadata(const Function &F) {
   auto MD = F.getMetadata("llvm.amdgcn.lds.kernel.id");
   if (MD && MD->getNumOperands() == 1) {

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

SDValue SITargetLowering::lowerBUILD_VECTOR(SDValue Op,
 
   SDValue Or = DAG.getNode(ISD::OR, SL, MVT::i32, Lo, ShlHi);
   return DAG.getNode(ISD::BITCAST, SL, VT, Or);
 }
 
 bool
 SITargetLowering::isOffsetFoldingLegal(const GlobalAddressSDNode *GA) const {
   // We can fold offsets for anything that doesn't require a GOT relocation.
-  return (GA->getAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS ||
+  bool CanFold = (GA->getAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS ||
                   GA->getAddressSpace() == AMDGPUAS::CONSTANT_ADDRESS ||
                   GA->getAddressSpace() == AMDGPUAS::CONSTANT_ADDRESS_32BIT) &&
                   !shouldEmitGOTReloc(GA->getGlobal());
+
+  // Some LDS variables have compile time known addresses
+  if (GA->getAddressSpace() == AMDGPUAS::LOCAL_ADDRESS) {
+    if (const GlobalVariable *GV =
+        dyn_cast<const GlobalVariable>(GA->getGlobal())) {
+      CanFold |= AMDGPUMachineFunction::isKnownAddressLDSGlobal(*GV);
+    }
+  }
+
+  return CanFold;
 }
 
 static SDValue
 buildPCRelGlobalAddress(SelectionDAG &DAG, const GlobalValue *GV,
                         const SDLoc &DL, int64_t Offset, EVT PtrVT,
                         unsigned GAFlags = SIInstrInfo::MO_NONE) {
   assert(isInt<32>(Offset + 4) && "32-bit offset is expected!");
   // In order to support pc-relative addressing, the PC_ADD_REL_OFFSET SDNode is

llvm/test/CodeGen/AMDGPU/GlobalISel/dropped_debug_info_assert.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-; RUN: llc -global-isel -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -stop-after=instruction-select -o - %s | FileCheck %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -stop-after=instruction-select --amdgpu-lower-module-lds-strategy=module -o - %s | FileCheck %s
 ; Make sure there are no assertions on dropped debug info
 
 declare void @callee()
 
 define amdgpu_kernel void @call_debug_loc() {
   ; CHECK-LABEL: name: call_debug_loc
   ; CHECK: bb.1.entry:
   ; CHECK-NEXT:   liveins: $sgpr14, $sgpr15, $sgpr16, $vgpr0, $vgpr1, $vgpr2, $sgpr4_sgpr5, $sgpr6_sgpr7, $sgpr8_sgpr9, $sgpr10_sgpr11

llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-inline-asm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx908 -O0 -global-isel -stop-after=irtranslator -verify-machineinstrs -o - %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx908 -O0 -global-isel -stop-after=irtranslator -verify-machineinstrs --amdgpu-lower-module-lds-strategy=module -o - %s | FileCheck %s
 
 define amdgpu_kernel void @asm_convergent() convergent{
   ; CHECK-LABEL: name: asm_convergent
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   INLINEASM &s_barrier, 33 /* sideeffect isconvergent attdialect */, !0
   ; CHECK-NEXT:   S_ENDPGM 0
   call void asm sideeffect "s_barrier", ""() convergent, !srcloc !0
   ret void

llvm/test/CodeGen/AMDGPU/addrspacecast-known-non-null.ll

-; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -o - %s | FileCheck %s
-; RUN: llc -global-isel -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -o - %s | FileCheck %s
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 --amdgpu-lower-module-lds-strategy=module -o - %s | FileCheck %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 --amdgpu-lower-module-lds-strategy=module -o - %s | FileCheck %s
 
 ; Test that a null check is not emitted for lowered addrspacecast
 
 
 define void @flat_user(i8* %ptr) {
   store i8 0, i8* %ptr
   ret void
 }
define void @cast_alloca() {
   ret void
 }
 
 @lds = internal unnamed_addr addrspace(3) global i8 undef, align 4
 
 ; CHECK-LABEL: {{^}}cast_lds_gv:
 ; CHECK: s_getreg_b32 [[GETREG:s[0-9]+]], hwreg(HW_REG_SH_MEM_BASES, 16, 16)
 ; CHECK: s_lshl_b32 [[APERTURE:s[0-9]+]], [[GETREG]], 16
-; CHECK: v_mov_b32_e32 v0, 0
 ; CHECK: v_mov_b32_e32 v1, [[APERTURE]]
 ; CHECK-NOT: v0
 ; CHECK-NOT: v1
 define void @cast_lds_gv() {
   %cast = addrspacecast i8 addrspace(3)* @lds to i8*
   call void @flat_user(i8* %cast)
   ret void
 }

llvm/test/CodeGen/AMDGPU/ds_write2.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --force-update
-; RUN: llc -mtriple=amdgcn--amdpal -mcpu=bonaire -verify-machineinstrs -mattr=+load-store-opt < %s | FileCheck -enable-var-scope --check-prefix=CI %s
-; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs -mattr=+load-store-opt,-unaligned-access-mode < %s | FileCheck -enable-var-scope -check-prefixes=GFX9,GFX9-ALIGNED %s
-; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs -mattr=+load-store-opt,+unaligned-access-mode < %s | FileCheck -enable-var-scope -check-prefixes=GFX9,GFX9-UNALIGNED %s
+; RUN: llc -mtriple=amdgcn--amdpal -mcpu=bonaire -verify-machineinstrs -mattr=+load-store-opt --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -enable-var-scope --check-prefix=CI %s
+; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs -mattr=+load-store-opt,-unaligned-access-mode --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -enable-var-scope -check-prefixes=GFX9,GFX9-ALIGNED %s
+; RUN: llc -mtriple=amdgcn--amdpal -mcpu=gfx900 -verify-machineinstrs -mattr=+load-store-opt,+unaligned-access-mode --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -enable-var-scope -check-prefixes=GFX9,GFX9-UNALIGNED %s
 
 @lds = addrspace(3) global [512 x float] undef, align 4
 @lds.f64 = addrspace(3) global [512 x double] undef, align 8
 
 define amdgpu_kernel void @simple_write2_one_val_f32(float addrspace(1)* %C, float addrspace(1)* %in) #0 {
 ; CI-LABEL: simple_write2_one_val_f32:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x2

llvm/test/CodeGen/AMDGPU/hsa.ll

-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 | FileCheck --check-prefix=HSA %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 -mattr=-flat-for-global | FileCheck --check-prefix=HSA-CI %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo --amdhsa-code-object-version=2 | FileCheck --check-prefix=HSA %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo --amdhsa-code-object-version=2 -mattr=-flat-for-global | FileCheck --check-prefix=HSA-VI %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri -filetype=obj --amdhsa-code-object-version=2 | llvm-readobj -S --sd --syms - | FileCheck --check-prefix=ELF %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 | llvm-mc -filetype=obj -triple amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 | llvm-readobj -S --sd --syms - | FileCheck %s --check-prefix=ELF
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W32 %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W64 %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W32 %s
-; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W64 %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=HSA %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 -mattr=-flat-for-global --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=HSA-CI %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo --amdhsa-code-object-version=2 --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=HSA %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=carrizo --amdhsa-code-object-version=2 -mattr=-flat-for-global --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=HSA-VI %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri -filetype=obj --amdhsa-code-object-version=2 --amdgpu-lower-module-lds-strategy=module | llvm-readobj -S --sd --syms - | FileCheck --check-prefix=ELF %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 --amdgpu-lower-module-lds-strategy=module | llvm-mc -filetype=obj -triple amdgcn--amdhsa -mcpu=kaveri --amdhsa-code-object-version=2 | llvm-readobj -S --sd --syms - | FileCheck %s --check-prefix=ELF
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W32 %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=-wavefrontsize32,+wavefrontsize64 --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W64 %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1100 -mattr=+wavefrontsize32,-wavefrontsize64 --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W32 %s
+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=gfx1100 -mattr=-wavefrontsize32,+wavefrontsize64 --amdgpu-lower-module-lds-strategy=module | FileCheck --check-prefix=GFX10 --check-prefix=GFX10-W64 %s
 
 ; The SHT_NOTE section contains the output from the .hsa_code_object_*
 ; directives.
 
 ; ELF: Section {
 ; ELF: Name: .text
 ; ELF: Type: SHT_PROGBITS (0x1)
 ; ELF: Flags [ (0x6)

llvm/test/CodeGen/AMDGPU/lds-reject-unknown-lowering-kind.ll

+; RUN: not --crash opt -S -amdgpu-lower-module-lds -amdgpu-lower-module-lds-strategy=other < %s 2>&1 | FileCheck %s
+
+target triple="amdgcn--"
+
+; CHECK: LLVM ERROR: Invalid argument 'other' to amdgpu-lower-module-lds-strategy
+define amdgpu_kernel void @k() {
+  ret void
+}

llvm/test/CodeGen/AMDGPU/lds-size.ll

 ; RUN: llc -global-isel=0 -mtriple=amdgcn-amd-amdhsa --amdhsa-code-object-version=2 < %s | FileCheck -check-prefix=ALL -check-prefix=HSA %s
 ; RUN: llc -global-isel=1 -mtriple=amdgcn-amd-amdhsa --amdhsa-code-object-version=2 < %s | FileCheck -check-prefix=ALL -check-prefix=HSA %s
 ; RUN: llc -march=r600 -mcpu=redwood < %s | FileCheck -check-prefix=ALL -check-prefix=EG %s
 
 ; This test makes sure we do not double count global values when they are
 ; used in different basic blocks.
 
 ; GCN: .long 47180
 ; GCN-NEXT: .long 32900
 
 ; EG: .long 166120
-; EG-NEXT: .long 1
+; EG-NEXT: .long 0
 ; ALL: {{^}}test:
 
 ; HSA: granulated_lds_size = 0
 ; HSA: workgroup_group_segment_byte_size = 4
 
 ; GCN: ; LDSByteSize: 4 bytes/workgroup (compile time only)
 @lds = internal unnamed_addr addrspace(3) global i32 undef, align 4
 

llvm/test/CodeGen/AMDGPU/local-memory.amdgcn.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck %s -check-prefixes=GCN,SI
-; RUN: llc -march=amdgcn -mcpu=bonaire -verify-machineinstrs < %s | FileCheck %s -check-prefixes=GCN,CI
+; RUN: llc -march=amdgcn -mcpu=verde -verify-machineinstrs --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s -check-prefixes=GCN,SI
+; RUN: llc -march=amdgcn -mcpu=bonaire -verify-machineinstrs --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s -check-prefixes=GCN,CI
 
 @local_memory.local_mem = internal unnamed_addr addrspace(3) global [128 x i32] undef, align 4
 
 define amdgpu_kernel void @local_memory(i32 addrspace(1)* %out) #0 {
 ; GCN-LABEL: local_memory:
 ; GCN:       ; %bb.0: ; %entry
 ; GCN-NEXT:    v_lshlrev_b32_e32 v1, 2, v0
 ; GCN-NEXT:    s_mov_b32 m0, -1

llvm/test/CodeGen/AMDGPU/local-memory.r600.ll

 ; RUN: llc -march=r600 -mcpu=redwood < %s | FileCheck -check-prefix=EG -check-prefix=FUNC %s
 
 @local_memory.local_mem = internal unnamed_addr addrspace(3) global [128 x i32] undef, align 4
 
 ; Check that the LDS size emitted correctly
 ; EG: .long 166120
-; EG-NEXT: .long 128
+; EG-NEXT: .long 0
 
 ; FUNC-LABEL: {{^}}local_memory:
 
 ; EG: LDS_WRITE
 
 ; GROUP_BARRIER must be the last instruction in a clause
 ; EG: GROUP_BARRIER
 ; EG-NEXT: ALU clause
entry:
   ret void
 }
 
 @local_memory_two_objects.local_mem0 = internal unnamed_addr addrspace(3) global [4 x i32] undef, align 4
 @local_memory_two_objects.local_mem1 = internal unnamed_addr addrspace(3) global [4 x i32] undef, align 4
 
 ; Check that the LDS size emitted correctly
 ; EG: .long 166120
-; EG-NEXT: .long 8
+; EG-NEXT: .long 0
 ; GCN: .long 47180
 ; GCN-NEXT: .long 32900
 
 ; FUNC-LABEL: {{^}}local_memory_two_objects:
 
 ; We would like to check the lds writes are using different
 ; addresses, but due to variations in the scheduler, we can't do
 ; this consistently on evergreen GPUs.

llvm/test/CodeGen/AMDGPU/lower-kernel-and-module-lds.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 @lds.size.1.align.1 = internal unnamed_addr addrspace(3) global [1 x i8] undef, align 1
 @lds.size.2.align.2 = internal unnamed_addr addrspace(3) global [2 x i8] undef, align 2
 @lds.size.4.align.4 = internal unnamed_addr addrspace(3) global [4 x i8] undef, align 4
 @lds.size.8.align.8 = internal unnamed_addr addrspace(3) global [8 x i8] undef, align 8
 @lds.size.16.align.16 = internal unnamed_addr addrspace(3) global [16 x i8] undef, align 16
 
 ; CHECK: %llvm.amdgcn.module.lds.t = type { [8 x i8], [1 x i8] }
-; CHECK: %llvm.amdgcn.kernel.k0.lds.t = type { [16 x i8], [4 x i8], [2 x i8] }
+; CHECK: %llvm.amdgcn.kernel.k0.lds.t = type { [16 x i8], [4 x i8], [2 x i8], [1 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k1.lds.t = type { [16 x i8], [4 x i8], [2 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k2.lds.t = type { [2 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k3.lds.t = type { [4 x i8] }
 
 ;.
 ; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 8
 ; CHECK: @llvm.compiler.used = appending global [1 x i8*] [i8* addrspacecast (i8 addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 0, i32 0) to i8*)], section "llvm.metadata"
 ; CHECK: @llvm.amdgcn.kernel.k0.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k0.lds.t undef, align 16
 ; CHECK: @llvm.amdgcn.kernel.k1.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k1.lds.t undef, align 16
 ; CHECK: @llvm.amdgcn.kernel.k2.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k2.lds.t undef, align 2
 ; CHECK: @llvm.amdgcn.kernel.k3.lds = internal addrspace(3) global %llvm.amdgcn.kernel.k3.lds.t undef, align 4
 ;.
 define amdgpu_kernel void @k0() #0 {
 ; CHECK-LABEL: @k0(
-; CHECK-NEXT:    %lds.size.1.align.1.bc = bitcast [1 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i8 addrspace(3)*
-; CHECK-NEXT:    store i8 1, i8 addrspace(3)* %lds.size.1.align.1.bc, align 8
+; CHECK-NEXT:    %lds.size.1.align.1.bc = bitcast [1 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, %llvm.amdgcn.kernel.k0.lds.t addrspace(3)* @llvm.amdgcn.kernel.k0.lds, i32 0, i32 3) to i8 addrspace(3)*
+; CHECK-NEXT:    store i8 1, i8 addrspace(3)* %lds.size.1.align.1.bc, align 2, !alias.scope !0, !noalias !3
 ; CHECK-NEXT:    %lds.size.2.align.2.bc = bitcast [2 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, %llvm.amdgcn.kernel.k0.lds.t addrspace(3)* @llvm.amdgcn.kernel.k0.lds, i32 0, i32 2) to i8 addrspace(3)*
 ; CHECK-NEXT:    store i8 2, i8 addrspace(3)* %lds.size.2.align.2.bc, align 4
 ; CHECK-NEXT:    %lds.size.4.align.4.bc = bitcast [4 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, %llvm.amdgcn.kernel.k0.lds.t addrspace(3)* @llvm.amdgcn.kernel.k0.lds, i32 0, i32 1) to i8 addrspace(3)*
 ; CHECK-NEXT:    store i8 4, i8 addrspace(3)* %lds.size.4.align.4.bc, align 16
 ; CHECK-NEXT:    %lds.size.16.align.16.bc = bitcast [16 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.k0.lds.t, %llvm.amdgcn.kernel.k0.lds.t addrspace(3)* @llvm.amdgcn.kernel.k0.lds, i32 0, i32 0) to i8 addrspace(3)*
 ; CHECK-NEXT:    store i8 16, i8 addrspace(3)* %lds.size.16.align.16.bc, align 16
 ; CHECK-NEXT:    ret void
 ;

 ; CHECK-NEXT:    ret void
 ;
   %lds.size.4.align.4.bc = bitcast [4 x i8] addrspace(3)* @lds.size.4.align.4 to i8 addrspace(3)*
   store i8 4, i8 addrspace(3)* %lds.size.4.align.4.bc, align 4
 
   ret void
 }
 
+
+define amdgpu_kernel void @calls_f0() {
+  call void @f0()
+  ret void
+}
+
 define void @f0() {
 ; CHECK-LABEL: @f0(
-; CHECK-NEXT:    %lds.size.1.align.1.bc = bitcast [1 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i8 addrspace(3)*
+; CHECK:         %lds.size.1.align.1.bc = bitcast [1 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i8 addrspace(3)*
 ; CHECK-NEXT:    store i8 1, i8 addrspace(3)* %lds.size.1.align.1.bc, align 8
 ; CHECK-NEXT:    %lds.size.8.align.8.bc = bitcast [8 x i8] addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 0) to i8 addrspace(3)*
 ; CHECK-NEXT:    store i8 8, i8 addrspace(3)* %lds.size.8.align.8.bc, align 8
 ; CHECK-NEXT:    ret void
 ;
   %lds.size.1.align.1.bc = bitcast [1 x i8] addrspace(3)* @lds.size.1.align.1 to i8 addrspace(3)*
   store i8 1, i8 addrspace(3)* %lds.size.1.align.1.bc, align 1
 
   %lds.size.8.align.8.bc = bitcast [8 x i8] addrspace(3)* @lds.size.8.align.8 to i8 addrspace(3)*
   store i8 8, i8 addrspace(3)* %lds.size.8.align.8.bc, align 4
 
   ret void
 }
 
 attributes #0 = { "amdgpu-elide-module-lds" }
 ; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }

llvm/test/CodeGen/AMDGPU/lower-kernel-lds-constexpr.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: -p --check-globals
-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 @lds.1 = internal unnamed_addr addrspace(3) global [2 x i8] undef, align 1
 
 ; CHECK: %llvm.amdgcn.kernel.k0.lds.t = type { [2 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k1.lds.t = type { [2 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k2.lds.t = type { i32 }
 ; CHECK: %llvm.amdgcn.kernel.k3.lds.t = type { [32 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k4.lds.t = type { [2 x i8] }

llvm/test/CodeGen/AMDGPU/lower-kernel-lds-super-align.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=true < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_ON %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=true < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_ON %s
-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=false < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_OFF %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=false < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_OFF %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=true --amdgpu-lower-module-lds-strategy=module < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_ON %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=true --amdgpu-lower-module-lds-strategy=module < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_ON %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=false --amdgpu-lower-module-lds-strategy=module < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_OFF %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-super-align-lds-globals=false --amdgpu-lower-module-lds-strategy=module < %s | FileCheck --check-prefixes=CHECK,SUPER-ALIGN_OFF %s
 
 ; CHECK: %llvm.amdgcn.kernel.k1.lds.t = type { [32 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k2.lds.t = type { i16, [2 x i8], i16 }
 ; CHECK: %llvm.amdgcn.kernel.k3.lds.t = type { [32 x i64], [32 x i32] }
 ; CHECK: %llvm.amdgcn.kernel.k4.lds.t = type { [2 x i32 addrspace(3)*] }
 
 ; SUPER-ALIGN_ON: @lds.unused = addrspace(3) global i32 undef, align 4
 ; SUPER-ALIGN_OFF: @lds.unused = addrspace(3) global i32 undef, align 2

llvm/test/CodeGen/AMDGPU/lower-kernel-lds.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 @lds.size.1.align.1 = internal unnamed_addr addrspace(3) global [1 x i8] undef, align 1
 @lds.size.2.align.2 = internal unnamed_addr addrspace(3) global [2 x i8] undef, align 2
 @lds.size.4.align.4 = internal unnamed_addr addrspace(3) global [4 x i8] undef, align 4
 @lds.size.16.align.16 = internal unnamed_addr addrspace(3) global [16 x i8] undef, align 16
 
 ; CHECK: %llvm.amdgcn.kernel.k0.lds.t = type { [16 x i8], [4 x i8], [2 x i8], [1 x i8] }
 ; CHECK: %llvm.amdgcn.kernel.k1.lds.t = type { [16 x i8], [4 x i8], [2 x i8] }

llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-memcpy.ll

-; RUN: llc -march=amdgcn -mcpu=gfx900 -O3 < %s | FileCheck -check-prefix=GCN %s
-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -O3 --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -check-prefix=GCN %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 %vec_type = type { %vec_base }
 %vec_base = type { %union.anon }
 %union.anon = type { %"vec_base<char, 3>::n_vec_" }
 %"vec_base<char, 3>::n_vec_" = type { [3 x i8] }
 
 $_f1 = comdat any
 $_f2 = comdat any

llvm/test/CodeGen/AMDGPU/lower-lds-struct-aa-merge.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 @a = internal unnamed_addr addrspace(3) global [64 x i32] undef, align 4
 @b = internal unnamed_addr addrspace(3) global [64 x i32] undef, align 4
 
 ; CHECK-LABEL: @no_clobber_ds_load_stores_x2_preexisting_aa
 ; CHECK: store i32 1, i32 addrspace(3)* %0, align 16, !tbaa !0, !noalias !5
 ; CHECK: %val.a = load i32, i32 addrspace(3)* %gep.a, align 4, !tbaa !0, !noalias !5
 ; CHECK: store i32 2, i32 addrspace(3)* %1, align 16, !tbaa !0, !noalias !5

llvm/test/CodeGen/AMDGPU/lower-module-lds-constantexpr-phi.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 @var = addrspace(3) global i32 undef, align 4
 
 ; Regression test. Duplicate constantexpr in phi nodes shall not emit broken IR
 define amdgpu_kernel void @func(i32 %c) {
 ; CHECK-LABEL: @func(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    switch i32 [[C:%.*]], label [[RETURN:%.*]] [

llvm/test/CodeGen/AMDGPU/lower-module-lds-constantexpr.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 ; CHECK: %llvm.amdgcn.module.lds.t = type { float, float }
+; CHECK: %llvm.amdgcn.kernel.timestwo.lds.t = type { float, float }
 
 @a_func = addrspace(3) global float undef, align 4
 
-; CHECK: %llvm.amdgcn.kernel.timestwo.lds.t = type { float }
-
 @kern = addrspace(3) global float undef, align 4
 
 ; @a_func is only used from a non-kernel function so is rewritten
 ; CHECK-NOT: @a_func
 ; @b_both is used from a non-kernel function so is rewritten
 ; CHECK-NOT: @b_both
 ; sorted both < func, so @b_both at null and @a_func at 4
 @b_both = addrspace(3) global float undef, align 4

 entry:
   store i32 %x, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @b_both to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @b_both to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   ret void
 }
 
 ; CHECK-LABEL: @timestwo() #0
 ; CHECK-NOT: call void @llvm.donothing()
 
-
-; CHECK:      %1 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*
+; CHECK:      %1 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
 ; CHECK:      %2 = addrspacecast i32 addrspace(3)* %1 to i32*
 ; CHECK:      %3 = ptrtoint i32* %2 to i64
-; CHECK:      %4 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
+; CHECK:      %4 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 1) to i32 addrspace(3)*
 ; CHECK:      %5 = addrspacecast i32 addrspace(3)* %4 to i32*
 ; CHECK:      %6 = ptrtoint i32* %5 to i64
 ; CHECK:      %7 = add i64 %3, %6
 ; CHECK:      %8 = inttoptr i64 %7 to i32*
 ; CHECK:      %ld = load i32, i32* %8, align 4
 ; CHECK:      %mul = mul i32 %ld, 2
-; CHECK:      %9 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
+; CHECK:      %9 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 1) to i32 addrspace(3)*
 ; CHECK:      %10 = addrspacecast i32 addrspace(3)* %9 to i32*
 ; CHECK:      %11 = ptrtoint i32* %10 to i64
-; CHECK:      %12 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 1) to i32 addrspace(3)*
+; CHECK:      %12 = bitcast float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.kernel.timestwo.lds.t, %llvm.amdgcn.kernel.timestwo.lds.t addrspace(3)* @llvm.amdgcn.kernel.timestwo.lds, i32 0, i32 0) to i32 addrspace(3)*
 ; CHECK:      %13 = addrspacecast i32 addrspace(3)* %12 to i32*
 ; CHECK:      %14 = ptrtoint i32* %13 to i64
 ; CHECK:      %15 = add i64 %11, %14
 ; CHECK:      %16 = inttoptr i64 %15 to i32*
 ; CHECK:      store i32 %mul, i32* %16, align 4
 ; CHECK:      ret void
 define amdgpu_kernel void @timestwo() {
   %ld = load i32, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @b_both to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @kern to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   %mul = mul i32 %ld, 2
   store i32 %mul, i32* inttoptr (i64 add (i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @kern to i32 addrspace(3)*) to i32*) to i64), i64 ptrtoint (i32* addrspacecast (i32 addrspace(3)* bitcast (float addrspace(3)* @b_both to i32 addrspace(3)*) to i32*) to i64)) to i32*), align 4
   ret void
 }
 
+; CHECK-LABEL: @through_functions()
+define amdgpu_kernel void @through_functions() {
+  %ld = call i32 @get_func()
+  %mul = mul i32 %ld, 4
+  call void @set_func(i32 %mul)
+  ret void
+}
+
 attributes #0 = { "amdgpu-elide-module-lds" }
 ; CHECK: attributes #0 = { "amdgpu-elide-module-lds" }

llvm/test/CodeGen/AMDGPU/lower-module-lds-inactive.ll

 ; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
 ; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
 
 ; Variables that are not lowered by this pass are left unchanged
 ; CHECK-NOT: asm
 ; CHECK-NOT: llvm.amdgcn.module.lds
 ; CHECK-NOT: llvm.amdgcn.module.lds.t
 
-; var1, var2 would be transformed were they used from a non-kernel function
-; CHECK-NOT: @var1 =
+; var1 is removed, var2 stays because it's in compiler.used
+; CHECK-NOT: @var1
 ; CHECK: @var2 = addrspace(3) global float undef
 @var1 = addrspace(3) global i32 undef
 @var2 = addrspace(3) global float undef
 
 ; constant variables are left to the optimizer / error diagnostics
 ; CHECK: @const_undef = addrspace(3) constant i32 undef
 ; CHECK: @const_with_init = addrspace(3) constant i64 8
 @const_undef = addrspace(3) constant i32 undef

llvm/test/CodeGen/AMDGPU/lower-module-lds-offsets.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck -check-prefix=OPT %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck -check-prefix=OPT %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -check-prefix=OPT %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -check-prefix=OPT %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -check-prefix=GCN %s
 
 ; Check that module LDS is allocated at address 0 and kernel starts its
 ; allocation past module LDS when a call is present.
 
 @lds.size.1.align.1 = internal unnamed_addr addrspace(3) global [1 x i8] undef, align 1
 @lds.size.16.align.16 = internal unnamed_addr addrspace(3) global [16 x i8] undef, align 16
 
 ; GCN-LABEL: {{^}}k0:

llvm/test/CodeGen/AMDGPU/lower-module-lds-used-list.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 ; Check new struct is added to compiler.used and that the replaced variable is removed
 
 ; CHECK: %llvm.amdgcn.module.lds.t = type { float }
 ; CHECK: @ignored = addrspace(1) global i64 0
 ; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 8
 
 ; CHECK-NOT: @tolower

 
 @llvm.used = appending global [2 x i8*] [i8* addrspacecast (i8 addrspace(3)* bitcast (float addrspace(3)* @tolower to i8 addrspace(3)*) to i8*), i8* addrspacecast (i8 addrspace(1)* bitcast (i64 addrspace(1)* @ignored to i8 addrspace(1)*) to i8*)], section "llvm.metadata"
 
 ; @ignored still in list, @tolower removed, llvm.amdgcn.module.lds appended
 ; CHECK: @llvm.compiler.used = appending global [2 x i8*] [i8* addrspacecast (i8 addrspace(1)* bitcast (i64 addrspace(1)* @ignored to i8 addrspace(1)*) to i8*), i8* addrspacecast (i8 addrspace(3)* bitcast (%llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds to i8 addrspace(3)*) to i8*)], section "llvm.metadata"
 
 @llvm.compiler.used = appending global [2 x i8*] [i8* addrspacecast (i8 addrspace(3)* bitcast (float addrspace(3)* @tolower to i8 addrspace(3)*) to i8*), i8* addrspacecast (i8 addrspace(1)* bitcast (i64 addrspace(1)* @ignored to i8 addrspace(1)*) to i8*)], section "llvm.metadata"
 
+
+; Functions that are not called are ignored by the lowering
+define amdgpu_kernel void @call_func() {
+  call void @func()
+  ret void
+}
+
 ; CHECK-LABEL: @func()
 ; CHECK: %dec = atomicrmw fsub float addrspace(3)* getelementptr inbounds (%llvm.amdgcn.module.lds.t, %llvm.amdgcn.module.lds.t addrspace(3)* @llvm.amdgcn.module.lds, i32 0, i32 0), float 1.000000e+00 monotonic, align 8
 define void @func() {
   %dec = atomicrmw fsub float addrspace(3)* @tolower, float 1.0 monotonic
   %unused0 = atomicrmw add i64 addrspace(1)* @ignored, i64 1 monotonic
   ret void
 }

llvm/test/CodeGen/AMDGPU/lower-module-lds.ll

-; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds < %s | FileCheck %s
-; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: opt -S -mtriple=amdgcn-- -passes=amdgpu-lower-module-lds --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
 
 ; Padding to meet alignment, so references to @var1 replaced with gep ptr, 0, 2
 ; No i64 as addrspace(3) types with initializers are ignored. Likewise no addrspace(4).
 ; CHECK: %llvm.amdgcn.module.lds.t = type { float, [4 x i8], i32 }
 
-; Variables removed by pass
+; Variable removed by pass
 ; CHECK-NOT: @var0
-; CHECK-NOT: @var1
 
 @var0 = addrspace(3) global float undef, align 8
 @var1 = addrspace(3) global i32 undef, align 8
 
-@ptr =  addrspace(1) global i32 addrspace(3)* @var1, align 4
+; The invalid use by the global is left unchanged
+; CHECK: @var1 = addrspace(3) global i32 undef, align 8
+; CHECK: @ptr = addrspace(1) global i32 addrspace(3)* @var1, align 4 
+@ptr = addrspace(1) global i32 addrspace(3)* @var1, align 4
 
 ; A variable that is unchanged by pass
 ; CHECK: @with_init = addrspace(3) global i64 0
 @with_init = addrspace(3) global i64 0
 
 ; Instance of new type, aligned to max of element alignment
 ; CHECK: @llvm.amdgcn.module.lds = internal addrspace(3) global %llvm.amdgcn.module.lds.t undef, align 8
 

llvm/test/CodeGen/AMDGPU/module-lds-false-sharing.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=CHECK,GFX9 %s
-; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=CHECK,GFX10 %s
-; RUN: llc -global-isel -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=CHECK,G_GFX9 %s
-; RUN: llc -global-isel -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=CHECK,G_GFX10 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -enable-var-scope -check-prefixes=CHECK,GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -enable-var-scope -check-prefixes=CHECK,GFX10 %s
+; RUN: llc -global-isel -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -enable-var-scope -check-prefixes=CHECK,G_GFX9 %s
+; RUN: llc -global-isel -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s --amdgpu-lower-module-lds-strategy=module | FileCheck -enable-var-scope -check-prefixes=CHECK,G_GFX10 %s
 
 ; Test case looks at the allocated offset of @used_by_both. It's at zero when
 ; allocated by itself, but at 8 when allocated in combination with the double.
 ; Redundantly also checks LDSByteSize.
 @used_by_both = addrspace(3) global i32 undef
 @used_by_kernel = addrspace(3) global i32 undef
 @used_by_function = addrspace(3) global double undef
 

 ; G_GFX10-NEXT:    s_swappc_b64 s[30:31], s[4:5]
 ; G_GFX10-NEXT:    s_endpgm
   store i32 0, i32 addrspace(3)* @used_by_both
   call void @nonkernel()
   ret void
 }
 ; CHECK: ; LDSByteSize: 16 bytes
 
-; Kernel only needs to allocate the i32 it uses, but because that i32 was
-; also used by a non-kernel function it was block allocated along with
-; the double used by the non-kernel function, this kernel allocates 16 bytes
-; and the accesses to the integer are at offset 8
+; Previous lowering was less efficient here than necessary as the i32 used
+; by the kernel is also used by an unrelated non-kernel function. Codegen
+; is now the same as nocall_ideal.
 define amdgpu_kernel void @nocall_false_sharing() {
-; GFX9-LABEL: nocall_false_sharing:
-; GFX9:       ; %bb.0:
-; GFX9-NEXT:    v_mov_b32_e32 v0, 0
-; GFX9-NEXT:    ds_write_b32 v0, v0 offset:8
-; GFX9-NEXT:    s_endpgm
-;
-; GFX10-LABEL: nocall_false_sharing:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    v_mov_b32_e32 v0, 0
-; GFX10-NEXT:    ds_write_b32 v0, v0 offset:8
-; GFX10-NEXT:    s_endpgm
-;
-; G_GFX9-LABEL: nocall_false_sharing:
-; G_GFX9:       ; %bb.0:
-; G_GFX9-NEXT:    v_mov_b32_e32 v0, 0
-; G_GFX9-NEXT:    v_mov_b32_e32 v1, 8
-; G_GFX9-NEXT:    ds_write_b32 v1, v0
-; G_GFX9-NEXT:    s_endpgm
-;
-; G_GFX10-LABEL: nocall_false_sharing:
-; G_GFX10:       ; %bb.0:
-; G_GFX10-NEXT:    v_mov_b32_e32 v0, 0
-; G_GFX10-NEXT:    v_mov_b32_e32 v1, 8
-; G_GFX10-NEXT:    ds_write_b32 v1, v0
-; G_GFX10-NEXT:    s_endpgm
+; CHECK-LABEL: nocall_false_sharing:
+; CHECK:       ; %bb.0:
+; CHECK-NEXT:    v_mov_b32_e32 v0, 0
+; CHECK-NEXT:    ds_write_b32 v0, v0
+; CHECK-NEXT:    s_endpgm
   store i32 0, i32 addrspace(3)* @used_by_both
   ret void
 }
-; CHECK: ; LDSByteSize: 16 bytes
-
+; CHECK: ; LDSByteSize: 4 bytes
 
 
 define void @nonkernel() {
 ; GFX9-LABEL: nonkernel:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, v0

llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -march=amdgcn -mcpu=gfx900 -amdgpu-aa -amdgpu-aa-wrapper -amdgpu-annotate-uniform -S < %s | FileCheck %s
-; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+; RUN: opt -march=amdgcn -mcpu=gfx900 -amdgpu-aa -amdgpu-aa-wrapper -amdgpu-annotate-uniform -S --amdgpu-lower-module-lds-strategy=module < %s | FileCheck %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -check-prefix=GCN %s
 
 ; Check that barrier or fence in between of loads is not considered a clobber
 ; for the purpose of converting vector loads into scalar.
 
 @LDS = linkonce_odr hidden local_unnamed_addr addrspace(3) global i32 undef
 
 ; GCN-LABEL: {{^}}simple_barrier:
 ; GCN: s_load_dword s

llvm/test/CodeGen/AMDGPU/promote-alloca-stored-pointer-value.ll

-; RUN: llc -march=amdgcn -mattr=+promote-alloca,+max-private-element-size-4 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
-; RUN: llc -march=amdgcn -mattr=-promote-alloca,+max-private-element-size-4 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mattr=+promote-alloca,+max-private-element-size-4 -verify-machineinstrs --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mattr=-promote-alloca,+max-private-element-size-4 -verify-machineinstrs --amdgpu-lower-module-lds-strategy=module < %s | FileCheck -check-prefix=GCN %s
 
 ; Pointer value is stored in a candidate for LDS usage.
 
 ; GCN-LABEL: {{^}}stored_lds_pointer_value:
 ; GCN: buffer_store_dword v
 define amdgpu_kernel void @stored_lds_pointer_value(float addrspace(5)* addrspace(1)* %ptr) #0 {
   %tmp = alloca float, addrspace(5)
   store float 0.0, float  addrspace(5)*%tmp