[RFC] Fix TLS and Coroutine

[lxfind]

Summary

This patch is to address https://bugs.llvm.org/show_bug.cgi?id=47835.
A relevant discussion regarding pthread_self and TLS can be found here: http://lists.llvm.org/pipermail/llvm-dev/2020-November/146766.html.

A coroutine may suspend and resume on a different thread, and hence the address of a thread_local variable may change after coroutine suspension.
In the existing design, getting the address of a TLS variable is through a direct reference, like @tls_variable. Such kind of value can be
arbitrarily moved around/replaced in the IR within the same function. This will lead to incorrect caching of TLS variable address in coroutines across suspension points.
To fix it, we have to turn the TLS address access into an intrinsics call, so that it will not be simply CSE-ed.
After CoroSplit, we no longer have coroutines, and hence can safely lower the TLS intrinsics back into references.

Note:
The current placement of the LowerThreadLocalIntrinsicPass may not be ideal. I am not quite sure how to organize it. Suggestions welcome!
Testing isn't sufficient, and there may also be failing tests. I will add/fix more tests if this patch is along the right direction.

[hoy]

llvm/include/llvm/IR/Intrinsics.td

1311

With the intrinsic, can TLS variable reference in the same coroutine or regular routine be DCE-ed anymore?

[hoy]

llvm/include/llvm/IR/Intrinsics.td

1311

Sorry, I meant CSE-ed.

[lxfind]

llvm/include/llvm/IR/Intrinsics.td

1311

Since the intrinsics does not have readnone attribute, it won't be CSE-ed before CoroSplit.
However after CoroSplit, it will be lowered back to the direct reference of the TLS, and will be CSE-ed by latter passes.
I can add a test function to demonstrate that too.

[hoy]

llvm/include/llvm/IR/Intrinsics.td

1311

Sounds good. Can you please point out what optimization passes CSE-ed tls reference without this implementation? I'm wondering if those optimizations can be postponed to after CoroSplit.

[lxfind]

llvm/include/llvm/IR/Intrinsics.td

1311

To clarify, it wasn't just CSE that would merge the references of the same TLS.
For instance, without this patch, a reference to "tls_variable" will just be "@tls_variable". For code like this:

@tls_variable = internal thread_local global i32 0, align 4

define i32* @foo(){
  ret i32* @tls_variable
}

define void @bar() {
  %tls1 = call i32* @foo()
  ..coro.suspend..
  %tls2 = call i32* @foo()
  %cond = icmp eq i32* %tls1, %tls2
}

When foo() is inlined into bar(), all uses of %tls1 will be replaced with @tls_variable.

[hoy]

llvm/include/llvm/IR/Intrinsics.td

1311

Thanks for the explanation. I have a dumb question. Why isn't corosplit placed at the very beginning of the pipeline?

[lxfind]

llvm/include/llvm/IR/Intrinsics.td

1311

The coroutine frame size is determined during CoroSplit. So if CoroSplit happens too early without any optimizations, the frame size will always be very big and there is no chance to optimize it.
This is indeed a fundamental trade-off. If CoroSplit happens much earlier then it will be immune to this kind of problem.

[nhaehnle]

llvm/include/llvm/IR/Intrinsics.td

1284–1285

Unrelated change

[lxfind]

Fix all failing tests

clang/lib/CodeGen/CGExpr.cpp

LValue CodeGenFunction::EmitLoadOfPointerLValue(Address PtrAddr,
   Address Addr = EmitLoadOfPointer(PtrAddr, PtrTy, &BaseInfo, &TBAAInfo);
   return MakeAddrLValue(Addr, PtrTy->getPointeeType(), BaseInfo, TBAAInfo);
 }
 
 static LValue EmitGlobalVarDeclLValue(CodeGenFunction &CGF,
                                       const Expr *E, const VarDecl *VD) {
   QualType T = E->getType();
 
-  // If it's thread_local, emit a call to its wrapper function instead.
+  // If it's a dynamic thread_local, and the ABI requires a wrapper function,
+  // emit a call to its wrapper function instead.
   if (VD->getTLSKind() == VarDecl::TLS_Dynamic &&
       CGF.CGM.getCXXABI().usesThreadWrapperFunction(VD))
     return CGF.CGM.getCXXABI().EmitThreadLocalVarDeclLValue(CGF, VD, T);
   // Check if the variable is marked as declare target with link clause in
   // device codegen.
   if (CGF.getLangOpts().OpenMPIsDevice) {
     Address Addr = emitDeclTargetVarDeclLValue(CGF, VD, T);
     if (Addr.isValid())
       return CGF.MakeAddrLValue(Addr, T, AlignmentSource::Decl);
   }
 
+  bool ShouldEmitPrivateCopy = CGF.getLangOpts().OpenMP &&
+                               !CGF.getLangOpts().OpenMPSimd &&
+                               VD->hasAttr<OMPThreadPrivateDeclAttr>();
   llvm::Value *V = CGF.CGM.GetAddrOfGlobalVar(VD);
+  if (VD->getTLSKind() != VarDecl::TLS_None && !ShouldEmitPrivateCopy &&
+      CGF.getLangOpts().Coroutines)
+    V = CGF.Builder.CreateThreadLocal(V);
   llvm::Type *RealVarTy = CGF.getTypes().ConvertTypeForMem(VD->getType());
   V = EmitBitCastOfLValueToProperType(CGF, V, RealVarTy);
   CharUnits Alignment = CGF.getContext().getDeclAlign(VD);
   Address Addr(V, Alignment);
   // Emit reference to the private copy of the variable if it is an OpenMP
   // threadprivate variable.
-  if (CGF.getLangOpts().OpenMP && !CGF.getLangOpts().OpenMPSimd &&
-      VD->hasAttr<OMPThreadPrivateDeclAttr>()) {
+  if (ShouldEmitPrivateCopy) {
     return EmitThreadPrivateVarDeclLValue(CGF, VD, T, Addr, RealVarTy,
                                           E->getExprLoc());
   }
   LValue LV = VD->getType()->isReferenceType() ?
       CGF.EmitLoadOfReferenceLValue(Addr, VD->getType(),
                                     AlignmentSource::Decl) :
       CGF.MakeAddrLValue(Addr, T, AlignmentSource::Decl);
   setObjCGCLValueClass(CGF.getContext(), E, LV);

clang/lib/CodeGen/ItaniumCXXABI.cpp

if (HasConstantInitialization) {
 
       Builder.SetInsertPoint(InitBB);
       Builder.CreateCall(InitFnTy, Init);
       Builder.CreateBr(ExitBB);
 
       Builder.SetInsertPoint(ExitBB);
     }
 
+    llvm::Value *Val = Var;
+    if (CGM.getLangOpts().Coroutines)
+      Val = Builder.CreateThreadLocal(Val);
     // For a reference, the result of the wrapper function is a pointer to
     // the referenced object.
-    llvm::Value *Val = Var;
     if (VD->getType()->isReferenceType()) {
       CharUnits Align = CGM.getContext().getDeclAlign(VD);
       Val = Builder.CreateAlignedLoad(Val, Align);
     }
     if (Val->getType() != Wrapper->getReturnType())
       Val = Builder.CreatePointerBitCastOrAddrSpaceCast(
           Val, Wrapper->getReturnType(), "");
     Builder.CreateRet(Val);

clang/test/CodeGen/lto-newpm-pipeline.c

 // RUN:   -check-prefix=CHECK-FULL-OPTIMIZED
 // RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -o /dev/null -fexperimental-new-pass-manager -fdebug-pass-manager -flto=thin -Oz %s 2>&1 | FileCheck %s \
 // RUN:   -check-prefix=CHECK-THIN-OPTIMIZED
 
 // CHECK-FULL-O0: Starting llvm::Module pass manager run.
 // CHECK-FULL-O0: Running pass: AlwaysInlinerPass
 // CHECK-FULL-O0-NEXT: Running analysis: InnerAnalysisManagerProxy
 // CHECK-FULL-O0-NEXT: Running analysis: ProfileSummaryAnalysis
+// CHECK-FULL-O0-NEXT: Running pass: LowerThreadLocalIntrinsicPass
 // CHECK-FULL-O0-NEXT: Running pass: CanonicalizeAliasesPass
 // CHECK-FULL-O0-NEXT: Running pass: NameAnonGlobalPass
 // CHECK-FULL-O0-NEXT: Running pass: BitcodeWriterPass
 // CHECK-FULL-O0: Finished llvm::Module pass manager run.
 
 // CHECK-THIN-O0: Starting llvm::Module pass manager run.
 // CHECK-THIN-O0: Running pass: AlwaysInlinerPass
 // CHECK-THIN-O0-NEXT: Running analysis: InnerAnalysisManagerProxy
 // CHECK-THIN-O0-NEXT: Running analysis: ProfileSummaryAnalysis
+// CHECK-THIN-O0-NEXT: Running pass: LowerThreadLocalIntrinsicPass
 // CHECK-THIN-O0-NEXT: Running pass: CanonicalizeAliasesPass
 // CHECK-THIN-O0-NEXT: Running pass: NameAnonGlobalPass
 // CHECK-THIN-O0-NEXT: Running pass: ThinLTOBitcodeWriterPass
 // CHECK-THIN-O0: Finished llvm::Module pass manager run.
 
 // TODO: The LTO pre-link pipeline currently invokes
 //       buildPerModuleDefaultPipeline(), which contains LoopVectorizePass.
 //       This may change as the pipeline gets implemented.

clang/test/CodeGenCXX/cxx2a-thread-local-constinit.cpp

 // LINUX-LABEL: define linkonce_odr {{.*}} @_ZTW1a()
 // LINUX: br i1
 // LINUX: call void @_ZTH1a()
 // LINUX: }
 // DARWIN-NOT: define {{.*}}@_ZTW1a()
 
 // CHECK-LABEL: define i32 @_Z5get_bv()
 // CHECK-NOT: call
-// CHECK: load i32, i32* @b
+// CHECK: %[[TMP:.+]] = bitcast i8* bitcast (i32* @b to i8*) to i32*
+// CHECK-NEXT: load i32, i32* %[[TMP]]
 // CHECK-NOT: call
 // CHECK: }
 int get_b() { return b; }
 
 // CHECK-NOT: define {{.*}} @_ZTW1b()
 
 extern thread_local int c;
 
 // CHECK-LABEL: define i32 @_Z5get_cv()
 // LINUX: call {{(cxx_fast_tlscc )?}}i32* @_ZTW1c()
 // CHECK: load i32, i32* %
 // CHECK: }
 int get_c() { return c; }
 
 // Note: use of 'c' does not trigger initialization of 'd', because 'c' has a
 // constant initializer.
 // DARWIN-LABEL: define cxx_fast_tlscc {{.*}} @_ZTW1c()
 // LINUX-LABEL: define weak_odr {{.*}} @_ZTW1c()
 // CHECK-NOT: br i1
 // CHECK-NOT: call
-// CHECK: ret i32* @c
+// CHECK: %[[TMP:.+]] = bitcast i8* bitcast (i32* @c to i8*) to i32*
+// CHECK: ret i32* %[[TMP]]
 // CHECK: }
 
 thread_local int c = 0;
 
 int d_init();
 
 // CHECK: define {{.*}}[[D_INIT:@__cxx_global_var_init[^(]*]](
 // CHECK: call {{.*}} @_Z6d_initv()

clang/test/CodeGenCoroutines/coro-tls.cpp

+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fcoroutines-ts -std=c++14 -O3 -emit-llvm %s -o - | FileCheck %s
+
+#include "Inputs/coroutine.h"
+
+namespace coro = std::experimental::coroutines_v1;
+
+struct awaitable {
+  bool await_ready() { return false; }
+  void await_suspend(coro::coroutine_handle<> h);
+  void await_resume() {}
+};
+awaitable switch_to_new_thread();
+
+struct task {
+  struct promise_type {
+    task get_return_object() { return {}; }
+    coro::suspend_never initial_suspend() { return {}; }
+    coro::suspend_never final_suspend() noexcept { return {}; }
+    void return_void() {}
+    void unhandled_exception() {}
+  };
+};
+
+void check(int *i, int *j);
+
+thread_local int tls_variable = 0;
+
+bool non_coroutine() {
+  auto *i = &tls_variable;
+  auto *j = &tls_variable;
+  return i == j;
+}
+
+// CHECK-LABEL: define zeroext i1 @_Z13non_coroutinev()
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    ret i1 true
+
+
+task resuming_on_new_thread() {
+  auto *i = &tls_variable;
+  co_await switch_to_new_thread();
+  auto *j = &tls_variable;
+  check(i, j);
+}
+
+// This test checks that two arguments passed to "check" will be different.
+// The first one will be a value loaded from the frame, and the second is
+// the current address of tsl_variable.
+
+// CHECK-LABEL: define internal fastcc void @_Z22resuming_on_new_threadv.resume
+// CHECK:         %[[RELOAD_ADDR:.+reload.addr]] = getelementptr inbounds %_Z22resuming_on_new_threadv.Frame, %_Z22resuming_on_new_threadv.Frame* %FramePtr
+// CHECK:         %[[TMP:.+]] = bitcast i8** %[[RELOAD_ADDR]] to i32**
+// CHECK:         %[[RELOAD:.+]] = load i32*, i32** %[[TMP]]
+// CHECK:         tail call void @_Z5checkPiS_(i32* %[[RELOAD]], i32* nonnull @tls_variable)

llvm/include/llvm/IR/IRBuilder.h

#endif
   /// If the pointer isn't i8* it will be converted.
   CallInst *CreateLifetimeEnd(Value *Ptr, ConstantInt *Size = nullptr);
 
   /// Create a call to invariant.start intrinsic.
   ///
   /// If the pointer isn't i8* it will be converted.
   CallInst *CreateInvariantStart(Value *Ptr, ConstantInt *Size = nullptr);
 
+  /// Create a threadlocal intrinsic.
+  CallInst *CreateThreadLocal(Value *Ptr);
+
   /// Create a call to Masked Load intrinsic
   LLVM_ATTRIBUTE_DEPRECATED(
       CallInst *CreateMaskedLoad(Value *Ptr, unsigned Alignment, Value *Mask,
                                  Value *PassThru = nullptr,
                                  const Twine &Name = ""),
       "Use the version that takes Align instead") {
     return CreateMaskedLoad(Ptr, assumeAligned(Alignment), Mask, PassThru,
                             Name);

llvm/include/llvm/IR/Intrinsics.td

 // This instruction has no actual effect, though it is treated by the optimizer
 // has having opaque side effects. This may be inserted into loops to ensure
 // that they are not removed even if they turn out to be empty, for languages
 // which specify that infinite loops must be preserved.
 def int_sideeffect : DefaultAttrsIntrinsic<[], [], [IntrInaccessibleMemOnly, IntrWillReturn]>;
 
 // The pseudoprobe intrinsic works as a place holder to the block it probes.
 // Like the sideeffect intrinsic defined above, this intrinsic is treated by the 
 // optimizer as having opaque side effects so that it won't be get rid of or moved 
 // out of the block it probes.
 def int_pseudoprobe : Intrinsic<[], [llvm_i64_ty, llvm_i64_ty, llvm_i32_ty],
                                     [IntrInaccessibleMemOnly, IntrWillReturn]>;
 
 // Intrinsics to support half precision floating point format
 let IntrProperties = [IntrNoMem, IntrWillReturn] in {
 def int_convert_to_fp16   : DefaultAttrsIntrinsic<[llvm_i16_ty], [llvm_anyfloat_ty]>;
 def int_convert_from_fp16 : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [llvm_i16_ty]>;
 }
 
 // Clear cache intrinsic, default to ignore (ie. emit nothing)
 // maps to void __clear_cache() on supporting platforms
 def int_clear_cache : Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty],
                                 [], "llvm.clear_cache">;
 
 // Intrinsic to detect whether its argument is a constant.
 def int_is_constant : DefaultAttrsIntrinsic<[llvm_i1_ty], [llvm_any_ty],
                                 [IntrNoMem, IntrWillReturn, IntrConvergent],
                                 "llvm.is.constant">;
 
 // Intrinsic to mask out bits of a pointer.
 def int_ptrmask: DefaultAttrsIntrinsic<[llvm_anyptr_ty], [LLVMMatchType<0>, llvm_anyint_ty],
                            [IntrNoMem, IntrSpeculatable, IntrWillReturn]>;
 
+
+// Intrinsic to obtain the address of a thread_local variable.
+def int_threadlocal : Intrinsic<[llvm_ptr_ty], [llvm_ptr_ty]>;
+
 //===---------------- Vector Predication Intrinsics --------------===//
 
 // Speculatable Binary operators
 let IntrProperties = [IntrSpeculatable, IntrNoMem, IntrNoSync, IntrWillReturn] in {
   def int_vp_add : DefaultAttrsIntrinsic<[ llvm_anyvector_ty ],
                              [ LLVMMatchType<0>,
                                LLVMMatchType<0>,
                                LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,

llvm/include/llvm/InitializePasses.h

 void initializeLowerConstantIntrinsicsPass(PassRegistry&);
 void initializeLowerEmuTLSPass(PassRegistry&);
 void initializeLowerExpectIntrinsicPass(PassRegistry&);
 void initializeLowerGuardIntrinsicLegacyPassPass(PassRegistry&);
 void initializeLowerWidenableConditionLegacyPassPass(PassRegistry&);
 void initializeLowerIntrinsicsPass(PassRegistry&);
 void initializeLowerInvokeLegacyPassPass(PassRegistry&);
 void initializeLowerSwitchLegacyPassPass(PassRegistry &);
+void initializeLowerThreadLocalIntrinsicLegacyPassPass(PassRegistry &);
 void initializeLowerTypeTestsPass(PassRegistry&);
 void initializeLowerMatrixIntrinsicsLegacyPassPass(PassRegistry &);
 void initializeLowerMatrixIntrinsicsMinimalLegacyPassPass(PassRegistry &);
 void initializeMIRCanonicalizerPass(PassRegistry &);
 void initializeMIRNamerPass(PassRegistry &);
 void initializeMIRPrintingPassPass(PassRegistry&);
 void initializeMachineBlockFrequencyInfoPass(PassRegistry&);
 void initializeMachineBlockPlacementPass(PassRegistry&);

llvm/include/llvm/Transforms/Scalar.h

 //
 // LowerMatrixIntrinsicsMinimal - Lower matrix intrinsics to vector operations
 //                               (lightweight, does not require extra analysis)
 //
 Pass *createLowerMatrixIntrinsicsMinimalPass();
 
 //===----------------------------------------------------------------------===//
 //
+// createLowerThreadLocalIntrinsic - Lower threadlocal intrinsics to direct
+//                                   reference of the thread_local variable.
+//
+Pass *createLowerThreadLocalIntrinsicPass();
+
+//===----------------------------------------------------------------------===//
+//
 // LowerWidenableCondition - Lower widenable condition to i1 true.
 //
 Pass *createLowerWidenableConditionPass();
 
 //===----------------------------------------------------------------------===//
 //
 // MergeICmps - Merge integer comparison chains into a memcmp
 //

llvm/include/llvm/Transforms/Scalar/LowerThreadLocalIntrinsic.h

+//===--- LowerThreadLocalIntrinsic.h - Lower the threadlocal intrinsic
+//---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers the llvm.threadlocal intrinsic to a direct reference to the
+// thread local variable.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_TRANSFORMS_SCALAR_LOWERTHREADLOCALINTRINSIC_H
+#define LLVM_TRANSFORMS_SCALAR_LOWERTHREADLOCALINTRINSIC_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct LowerThreadLocalIntrinsicPass
+    : PassInfoMixin<LowerThreadLocalIntrinsicPass> {
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
+  static bool isRequired() { return true; }
+};
+
+} // namespace llvm
+
+#endif // LLVM_TRANSFORMS_SCALAR_LOWERTHREADLOCALINTRINSIC_H

llvm/lib/IR/IRBuilder.cpp

CallInst *IRBuilderBase::CreateInvariantStart(Value *Ptr, ConstantInt *Size) {
   // Fill in the single overloaded type: memory object type.
   Type *ObjectPtr[1] = {Ptr->getType()};
   Module *M = BB->getParent()->getParent();
   Function *TheFn =
       Intrinsic::getDeclaration(M, Intrinsic::invariant_start, ObjectPtr);
   return createCallHelper(TheFn, Ops, this);
 }
 
+CallInst *IRBuilderBase::CreateThreadLocal(Value *Ptr) {
+  return CreateIntrinsic(
+      llvm::Intrinsic::threadlocal, llvm::None,
+      {CreatePointerBitCastOrAddrSpaceCast(Ptr, getInt8PtrTy())});
+}
+
 CallInst *
 IRBuilderBase::CreateAssumption(Value *Cond,
                                 ArrayRef<OperandBundleDef> OpBundles) {
   assert(Cond->getType() == getInt1Ty() &&
          "an assumption condition must be of type i1");
 
   Value *Ops[] = { Cond };
   Module *M = BB->getParent()->getParent();

llvm/lib/Passes/PassBuilder.cpp

 #include "llvm/Transforms/Scalar/LoopUnrollAndJamPass.h"
 #include "llvm/Transforms/Scalar/LoopUnrollPass.h"
 #include "llvm/Transforms/Scalar/LoopVersioningLICM.h"
 #include "llvm/Transforms/Scalar/LowerAtomic.h"
 #include "llvm/Transforms/Scalar/LowerConstantIntrinsics.h"
 #include "llvm/Transforms/Scalar/LowerExpectIntrinsic.h"
 #include "llvm/Transforms/Scalar/LowerGuardIntrinsic.h"
 #include "llvm/Transforms/Scalar/LowerMatrixIntrinsics.h"
+#include "llvm/Transforms/Scalar/LowerThreadLocalIntrinsic.h"
 #include "llvm/Transforms/Scalar/LowerWidenableCondition.h"
 #include "llvm/Transforms/Scalar/MakeGuardsExplicit.h"
 #include "llvm/Transforms/Scalar/MemCpyOptimizer.h"
 #include "llvm/Transforms/Scalar/MergeICmps.h"
 #include "llvm/Transforms/Scalar/MergedLoadStoreMotion.h"
 #include "llvm/Transforms/Scalar/NaryReassociate.h"
 #include "llvm/Transforms/Scalar/NewGVN.h"
 #include "llvm/Transforms/Scalar/PartiallyInlineLibCalls.h"
for (auto &C : PipelineStartEPCallbacks)
     C(MPM, Level);
 
   if (PGOOpt && PGOOpt->DebugInfoForProfiling)
     MPM.addPass(createModuleToFunctionPassAdaptor(AddDiscriminatorsPass()));
 
   // Add the core simplification pipeline.
   MPM.addPass(buildModuleSimplificationPipeline(Level, ThinLTOPhase::None));
 
+  MPM.addPass(LowerThreadLocalIntrinsicPass());
+
   // Now add the optimization pipeline.
   MPM.addPass(buildModuleOptimizationPipeline(Level, LTOPreLink));
 
   // Emit annotation remarks.
   addAnnotationRemarksPass(MPM);
 
   if (LTOPreLink)
     addRequiredLTOPreLinkPasses(MPM);
if (PTO.Coroutines) {
     CGSCCPassManager CGPM(DebugLogging);
     CGPM.addPass(CoroSplitPass());
     CGPM.addPass(createCGSCCToFunctionPassAdaptor(CoroElidePass()));
     MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(std::move(CGPM)));
 
     MPM.addPass(createModuleToFunctionPassAdaptor(CoroCleanupPass()));
   }
 
+  MPM.addPass(LowerThreadLocalIntrinsicPass());
+
   for (auto &C : OptimizerLastEPCallbacks)
     C(MPM, Level);
 
   if (LTOPreLink)
     addRequiredLTOPreLinkPasses(MPM);
 
   return MPM;
 }

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

if (PerformThinLTO) {
       // Drop available_externally and unreferenced globals. This is necessary
       // with ThinLTO in order to avoid leaving undefined references to dead
       // globals in the object file.
       MPM.add(createEliminateAvailableExternallyPass());
       MPM.add(createGlobalDCEPass());
     }
 
     addExtensionsToPM(EP_EnabledOnOptLevel0, MPM);
+    MPM.add(createLowerThreadLocalIntrinsicPass());
 
     if (PrepareForLTO || PrepareForThinLTO) {
       MPM.add(createCanonicalizeAliasesPass());
       // Rename anon globals to be able to export them in the summary.
       // This has to be done after we add the extensions to the pass manager
       // as there could be passes (e.g. Adddress sanitizer) which introduce
       // new unnamed globals.
       MPM.add(createNameAnonGlobalPass());
void PassManagerBuilder::populateModulePassManager(
 
   addExtensionsToPM(EP_CGSCCOptimizerLate, MPM);
   addFunctionSimplificationPasses(MPM);
 
   // FIXME: This is a HACK! The inliner pass above implicitly creates a CGSCC
   // pass manager that we are specifically trying to avoid. To prevent this
   // we must insert a no-op module pass to reset the pass manager.
   MPM.add(createBarrierNoopPass());
+  MPM.add(createLowerThreadLocalIntrinsicPass());
 
   if (RunPartialInlining)
     MPM.add(createPartialInliningPass());
 
   if (OptLevel > 1 && !PrepareForLTO && !PrepareForThinLTO)
     // Remove avail extern fns and globals definitions if we aren't
     // compiling an object file for later LTO. For LTO we want to preserve
     // these so they are eligible for inlining at link-time. Note if they

llvm/lib/Transforms/Scalar/CMakeLists.txt

add_llvm_component_library(LLVMScalarOpts
   LoopUnrollAndJamPass.cpp
   LoopUnswitch.cpp
   LoopVersioningLICM.cpp
   LowerAtomic.cpp
   LowerConstantIntrinsics.cpp
   LowerExpectIntrinsic.cpp
   LowerGuardIntrinsic.cpp
   LowerMatrixIntrinsics.cpp
+  LowerThreadLocalIntrinsic.cpp
   LowerWidenableCondition.cpp
   MakeGuardsExplicit.cpp
   MemCpyOptimizer.cpp
   MergeICmps.cpp
   MergedLoadStoreMotion.cpp
   NaryReassociate.cpp
   NewGVN.cpp
   PartiallyInlineLibCalls.cpp

llvm/lib/Transforms/Scalar/LowerThreadLocalIntrinsic.cpp

+//===- LowerThreadLocalIntrinsic.cpp - Lower the threadlocal intrinsic
+//---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers the llvm.threadlocal intrinsic to a direct reference to the
+// thread local variable.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Scalar/LowerThreadLocalIntrinsic.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Transforms/Scalar.h"
+
+using namespace llvm;
+
+static bool lowerThreadLocalIntrinsic(Module &M) {
+  // Check if we can cheaply rule out the possibility of not having any work to
+  // do.
+  Function *ThreadLocalDecl =
+      M.getFunction(Intrinsic::getName(Intrinsic::threadlocal));
+  if (!ThreadLocalDecl || ThreadLocalDecl->use_empty())
+    return false;
+
+  for (auto Itr = ThreadLocalDecl->users().begin(),
+            E = ThreadLocalDecl->users().end();
+       Itr != E;) {
+    Instruction *I = cast<Instruction>(*Itr);
+    ++Itr;
+    I->replaceAllUsesWith(I->getOperand(0));
+    I->eraseFromParent();
+  }
+
+  ThreadLocalDecl->eraseFromParent();
+
+  return true;
+}
+
+PreservedAnalyses
+LowerThreadLocalIntrinsicPass::run(Module &M, ModuleAnalysisManager &AM) {
+  if (lowerThreadLocalIntrinsic(M))
+    return PreservedAnalyses::none();
+
+  return PreservedAnalyses::all();
+}
+
+namespace {
+struct LowerThreadLocalIntrinsicLegacyPass : public ModulePass {
+  static char ID;
+  LowerThreadLocalIntrinsicLegacyPass() : ModulePass(ID) {
+    initializeLowerThreadLocalIntrinsicLegacyPassPass(
+        *PassRegistry::getPassRegistry());
+  }
+
+  bool runOnModule(Module &M) override;
+};
+} // namespace
+
+bool LowerThreadLocalIntrinsicLegacyPass::runOnModule(Module &M) {
+  return lowerThreadLocalIntrinsic(M);
+}
+
+char LowerThreadLocalIntrinsicLegacyPass::ID = 0;
+INITIALIZE_PASS(LowerThreadLocalIntrinsicLegacyPass,
+                "lower-threadlocal-intrinsic", "Lower ThreadLocal Intrinsics",
+                false, false)
+
+Pass *llvm::createLowerThreadLocalIntrinsicPass() {
+  return new LowerThreadLocalIntrinsicLegacyPass();
+}

llvm/test/CodeGen/AMDGPU/opt-pipeline.ll

 ; GCN-O1-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-NEXT:       Function Alias Analysis Results
 ; GCN-O1-NEXT:       Natural Loop Information
 ; GCN-O1-NEXT:       Lazy Branch Probability Analysis
 ; GCN-O1-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O1-NEXT:       Optimization Remark Emitter
 ; GCN-O1-NEXT:       Combine redundant instructions
 ; GCN-O1-NEXT:     A No-Op Barrier Pass
+; GCN-O1-NEXT:     Lower ThreadLocal Intrinsics
 ; GCN-O1-NEXT:     CallGraph Construction
 ; GCN-O1-NEXT:     Deduce function attributes in RPO
 ; GCN-O1-NEXT:     Global Variable Optimizer
 ; GCN-O1-NEXT:       FunctionPass Manager
 ; GCN-O1-NEXT:         Dominator Tree Construction
 ; GCN-O1-NEXT:         Natural Loop Information
 ; GCN-O1-NEXT:         Post-Dominator Tree Construction
 ; GCN-O1-NEXT:         Branch Probability Analysis

 ; GCN-O2-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; GCN-O2-NEXT:       Function Alias Analysis Results
 ; GCN-O2-NEXT:       Natural Loop Information
 ; GCN-O2-NEXT:       Lazy Branch Probability Analysis
 ; GCN-O2-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O2-NEXT:       Optimization Remark Emitter
 ; GCN-O2-NEXT:       Combine redundant instructions
 ; GCN-O2-NEXT:     A No-Op Barrier Pass
+; GCN-O2-NEXT:     Lower ThreadLocal Intrinsics
 ; GCN-O2-NEXT:     Eliminate Available Externally Globals
 ; GCN-O2-NEXT:     CallGraph Construction
 ; GCN-O2-NEXT:     Deduce function attributes in RPO
 ; GCN-O2-NEXT:     Global Variable Optimizer
 ; GCN-O2-NEXT:       FunctionPass Manager
 ; GCN-O2-NEXT:         Dominator Tree Construction
 ; GCN-O2-NEXT:         Natural Loop Information
 ; GCN-O2-NEXT:         Post-Dominator Tree Construction

 ; GCN-O3-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; GCN-O3-NEXT:       Function Alias Analysis Results
 ; GCN-O3-NEXT:       Natural Loop Information
 ; GCN-O3-NEXT:       Lazy Branch Probability Analysis
 ; GCN-O3-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O3-NEXT:       Optimization Remark Emitter
 ; GCN-O3-NEXT:       Combine redundant instructions
 ; GCN-O3-NEXT:     A No-Op Barrier Pass
+; GCN-O3-NEXT:     Lower ThreadLocal Intrinsics
 ; GCN-O3-NEXT:     Eliminate Available Externally Globals
 ; GCN-O3-NEXT:     CallGraph Construction
 ; GCN-O3-NEXT:     Deduce function attributes in RPO
 ; GCN-O3-NEXT:     Global Variable Optimizer
 ; GCN-O3-NEXT:       FunctionPass Manager
 ; GCN-O3-NEXT:         Dominator Tree Construction
 ; GCN-O3-NEXT:         Natural Loop Information
 ; GCN-O3-NEXT:         Post-Dominator Tree Construction

llvm/test/Other/new-pass-manager.ll

 ; RUN:     | FileCheck %s --check-prefix=CHECK-O0 --check-prefix=%llvmcheckext
 ; CHECK-O0: Starting llvm::Module pass manager run
 ; CHECK-O0-NEXT: Running pass: AlwaysInlinerPass
 ; CHECK-O0-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}>
 ; CHECK-O0-NEXT: Running analysis: ProfileSummaryAnalysis
 ; CHECK-EXT-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-EXT-NEXT: Running pass: {{.*}}Bye
 ; CHECK-EXT-NEXT: Finished llvm::Function pass manager run.
+; CHECK-O0-NEXT: Running pass: LowerThreadLocalIntrinsicPass
 ; CHECK-O0-NEXT: Finished llvm::Module pass manager run
 
 ; RUN: opt -disable-output -disable-verify -debug-pass-manager \
 ; RUN:     -passes='repeat<3>(no-op-module)' %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-REPEAT-MODULE-PASS
 ; CHECK-REPEAT-MODULE-PASS: Starting llvm::Module pass manager run
 ; CHECK-REPEAT-MODULE-PASS-NEXT: Running pass: RepeatedPass
 ; CHECK-REPEAT-MODULE-PASS-NEXT: Starting llvm::Module pass manager run

llvm/test/Other/new-pm-O0-defaults.ll

 ; RUN:     | FileCheck %s --check-prefixes=CHECK,CHECK-LTO
 
 ; CHECK: Starting llvm::Module pass manager run.
 ; CHECK-DEFAULT-NEXT: Running pass: AlwaysInlinerPass
 ; CHECK-DEFAULT-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-DEFAULT-NEXT: Running analysis: ProfileSummaryAnalysis
 ; CHECK-MATRIX-NEXT: Running pass: LowerMatrixIntrinsicsPass
 ; CHECK-MATRIX-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-DEFAULT-NEXT: Running pass: LowerThreadLocalIntrinsicPass
 ; CHECK-PRE-LINK-NEXT: Running pass: CanonicalizeAliasesPass
 ; CHECK-PRE-LINK-NEXT: Running pass: NameAnonGlobalPass
 ; CHECK-THINLTO-NEXT: Running pass: Annotation2MetadataPass
 ; CHECK-LTO-NEXT: Running pass: Annotation2MetadataPass
 ; CHECK-LTO-NEXT: Running pass: WholeProgramDevirtPass
 ; CHECK-LTO-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-LTO-NEXT: Running pass: LowerTypeTestsPass
 ; CHECK-LTO-NEXT: Running pass: LowerTypeTestsPass

llvm/test/Other/new-pm-defaults.ll

 ; CHECK-EP-SCALAR-LATE-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-PEEPHOLE-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O-NEXT: Finished llvm::Function pass manager run.
 ; CHECK-EP-CGSCC-LATE-NEXT: Running pass: NoOpCGSCCPass
 ; CHECK-O-NEXT: Finished CGSCC pass manager run.
 ; CHECK-O-NEXT: Finished llvm::Module pass manager run.
+; CHECK-O-NEXT: Running pass: LowerThreadLocalIntrinsicPass
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: GlobalDCEPass
 ; CHECK-DEFAULT-NEXT: Running pass: EliminateAvailableExternallyPass
 ; CHECK-LTO-NOT: Running pass: EliminateAvailableExternallyPass
 ; CHECK-O-NEXT: Running pass: ReversePostOrderFunctionAttrsPass
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: Float2IntPass

llvm/test/Other/opt-O2-pipeline.ll

 ; CHECK-NEXT:         Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Lazy Branch Probability Analysis
 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:     A No-Op Barrier Pass
+; CHECK-NEXT:     Lower ThreadLocal Intrinsics
 ; CHECK-NEXT:     Eliminate Available Externally Globals
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Post-Dominator Tree Construction

llvm/test/Other/opt-O3-pipeline-enable-matrix.ll

 ; CHECK-NEXT:         Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Lazy Branch Probability Analysis
 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:     A No-Op Barrier Pass
+; CHECK-NEXT:     Lower ThreadLocal Intrinsics
 ; CHECK-NEXT:     Eliminate Available Externally Globals
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Post-Dominator Tree Construction

llvm/test/Other/opt-O3-pipeline.ll

 ; CHECK-NEXT:         Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Lazy Branch Probability Analysis
 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:     A No-Op Barrier Pass
+; CHECK-NEXT:     Lower ThreadLocal Intrinsics
 ; CHECK-NEXT:     Eliminate Available Externally Globals
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Post-Dominator Tree Construction

llvm/test/Other/opt-Os-pipeline.ll

 ; CHECK-NEXT:         Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Lazy Branch Probability Analysis
 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:     A No-Op Barrier Pass
+; CHECK-NEXT:     Lower ThreadLocal Intrinsics
 ; CHECK-NEXT:     Eliminate Available Externally Globals
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:         Natural Loop Information
 ; CHECK-NEXT:         Post-Dominator Tree Construction

llvm/test/Other/pass-pipelines.ll

 ; and run of LICM here.
 ; CHECK-O2-NOT: Manager
 ; CHECK-O2: Loop Pass Manager
 ; CHECK-O2-NEXT: Loop Invariant Code Motion
 ; CHECK-O2-NOT: Manager
 ; Next we break out of the main Function passes inside the CGSCC pipeline with
 ; a barrier pass.
 ; CHECK-O2: A No-Op Barrier Pass
+; CHECK-O2-NEXT: Lower ThreadLocal Intrinsics
 ; CHECK-O2-NEXT: Eliminate Available Externally
 ; Inferring function attribute should be right after the CGSCC pipeline, before
 ; any other optimizations/analyses.
 ; CHECK-O2-NEXT: CallGraph
 ; CHECK-O2-NEXT: Deduce function attributes in RPO
 ; CHECK-O2-NOT: Manager
 ; Reduce the size of the IR ASAP after the inliner.
 ; CHECK-O2-NEXT: Global Variable Optimizer