[Clang] Correct when Itanium ABI guard variables are set for non-block variables with static or thread storage duration.

[tahonermann]

Summary

Previously, Itanium ABI guard variables were set after initialization was complete for non-block declared variables with static and thread storage duration. That resulted in initialization of such variables being restarted in cases where the variable was referenced while it was still under construction. Per C++20 [class.cdtor]p2, such references are permitted (though the value obtained by such an access is unspecified). The late initialization resulted in recursive reinitialization loops for cases like this:

template<typename T>
struct ct {
  struct mc {
    mc() { ct<T>::smf(); }
    void mf() const {}
  };
  thread_local static mc tlsdm;
  static void smf() { tlsdm.mf(); }
};
template<typename T>
thread_local typename ct<T>::mc ct<T>::tlsdm;
int main() {
  ct<int>::smf();
}

With this change, guard variables are set before initialization is started so as to avoid such reinitialization loops.

Fixes https://github.com/llvm/llvm-project/issues/57828

[erichkeane]

So this logically looks/sounds right to me (and I have no code concerns)... But I'm also notoriously bad at anything threading related. I'd love it if one of the other reviewers could sanity check this for me.

[aaron.ballman]

LGTM!

[rjmccall]

We can't set the flag if initialization is aborted by an exception, which is not ruled out by the use of thread-unsafe statics. So this is not a correct change.

More basically, I don't think "recursive calls try to re-initialize" is actually worse behavior than "recursive calls access an uninitialized object". And both seem to be allowed.

The real problem here is that everyone understands this flag to be a request to generate a cheap code sequence for testing initialization, but it's not actually possible to do the maximally cheap sequence and have it still be safe against recursive entry; you need to generate guards and diagnose the failure.

[hubert.reinterpretcast]

In D135919#3858808, @rjmccall wrote:

We can't set the flag if initialization is aborted by an exception, which is not ruled out by the use of thread-unsafe statics. So this is not a correct change.

@tahonermann, the case from https://github.com/llvm/llvm-project/issues/57828 is not for a block-scope variable, but the case in the patch description is...

[rjmccall]

Sorry, I responded as if you were modifying the guards of thread-unsafe global variables, but you're actually modifying the guards of thread-local variables. I apologize for the confusion. However, my substantive points pretty much all still hold:

• we have to generate code that behaves correctly in the presence of exceptions
• providing access to an uninitialized variable is worse than recursively re-entering initialization; in either case, the program is incorrect, and the latter is much more likely to cause an immediate failure at runtime
• the real fix requires a more elaborate code sequence to reliably generate a runtime failure

The major difference is that this is the standard behavior of the compiler and therefore not something we can just throw our hands up about.

Also, this needs Itanium discussion because it can affect interoperation.

[tahonermann]

the case from https://github.com/llvm/llvm-project/issues/57828 is not for a block-scope variable, but the case in the patch description is...

Thanks, Hubert. Yes, I found that the reported issue occurred for any case where thread safe guard variables are not required. I chose the block-scope variable example for the patch summary because I felt it better presented the issue.

Your question inspired me to do some additional testing though and I see both gcc and icc also exhibit the re-initialization behavior for that case, but not the case reported in https://github.com/llvm/llvm-project/issues/57828.

we have to generate code that behaves correctly in the presence of exceptions

Thank you, John. Indeed, I didn't consider exception handling. And I locally confirmed that the changes break cases like https://godbolt.org/z/M8Penaxa1.

providing access to an uninitialized variable is worse than recursively re-entering initialization; in either case, the program is incorrect, and the latter is much more likely to cause an immediate failure at runtime

My understanding is that, as long as the value of the uninitialized variable is not used, the program is not incorrect.

the real fix requires a more elaborate code sequence to reliably generate a runtime failure
Also, this needs Itanium discussion because it can affect interoperation.

I'll do some more analysis of how gcc and icc handle these scenarios and report back. If they appear to handle these cases right, perhaps Clang can be updated to match without the need for further ABI discussion.

[hubert.reinterpretcast]

In D135919#3859491, @tahonermann wrote:

the case from https://github.com/llvm/llvm-project/issues/57828 is not for a block-scope variable, but the case in the patch description is...

Thanks, Hubert. Yes, I found that the reported issue occurred for any case where thread safe guard variables are not required. I chose the block-scope variable example for the patch summary because I felt it better presented the issue.

Your question inspired me to do some additional testing though and I see both gcc and icc also exhibit the re-initialization behavior for that case, but not the case reported in https://github.com/llvm/llvm-project/issues/57828.

For the block-scope case, https://eel.is/c++draft/stmt.dcl#3: If control re-enters the declaration recursively while the variable is being initialized, the behavior is undefined.

[rjmccall]

In D135919#3859520, @hubert.reinterpretcast wrote:

In D135919#3859491, @tahonermann wrote:

the case from https://github.com/llvm/llvm-project/issues/57828 is not for a block-scope variable, but the case in the patch description is...

Thanks, Hubert. Yes, I found that the reported issue occurred for any case where thread safe guard variables are not required. I chose the block-scope variable example for the patch summary because I felt it better presented the issue.

Your question inspired me to do some additional testing though and I see both gcc and icc also exhibit the re-initialization behavior for that case, but not the case reported in https://github.com/llvm/llvm-project/issues/57828.

For the block-scope case, https://eel.is/c++draft/stmt.dcl#3: If control re-enters the declaration recursively while the variable is being initialized, the behavior is undefined.

Ah, right, so in principle we can just ignore this possibility. It would still be good to have some mode that dynamically diagnoses it, though, maybe under UBSan. We could probably get the Itanium ABI to agree to a standard guard value that means "currently being initialized" which implementations could use if they want to diagnose this case.

For non-block variables, this is a deferred initialization, as set out in [basic.start.dynamic]p7 (https://eel.is/c++draft/basic.start.dynamic#7). I don't think there's an analogous restriction about recursion, so only the general object lifetime rules apply, [basic.life]p7 (https://eel.is/c++draft/basic.life#7). A recursive use during initialization would be a use of the "original object" prior to the completion of initialization. Under the lifetime rules, the acceptability of that depends on what's being done, but generally, anything that accesses memory through it is either UB or yields an unspecified value. That does not permit us to do arbitrary things whenever we see a recursive reference, though, unless we actually do an analysis of that reference, which we are not doing here.

Note also that, for non-block variables, the exceptions concern does not arise because [basic.start.dynamic]p8 (https://eel.is/c++draft/basic.start.dynamic#8) says that exceptions during initialization trigger std::terminate.

I think that suggests that setting the guard to 1 prior to initialization is actually *required* for the deferred initialization of thread-locals, because otherwise we are interfering with the evaluation of uses of the name that are potentially valid prior to initialization, such as storing the address of the variable somewhere without accessing it. And if the use isn't valid and is one of the UB/unspecified cases, well, it's too bad we can't catch that immediately, but it's no worse than what would happen with static storage duration.

So if you alter this patch to only apply to non-block variables, I think we can move forward with it.

[rjmccall]

Please add a comment something like this:

The semantics of dynamic initialization of thread-local variables depends subtly on whether they are block-scope. The initialization of block-scope thread locals can be aborted with an exception and later retried (<cite>), and recursive entry to the initialization has undefined behavior (<cite>). For non-block thread locals, exceptions lead to termination (<cite>), and recursive references to the variables are governed only by the lifetime rules (<cite>), which means they’re perfectly fine as long as they avoid touching memory. As a result, we need to avoid marking block-scope variables as initialized until after initialization completes (or be prepared revert the mark after an exception), but we need to mark non-block-scope variables as initialized immediately so that non-memory-accessing uses will succeed.

[tahonermann]

Please add a comment something like this:

Done. Thank you for the prior review comments. This is again ready for review.

[tahonermann]

Rebased.

[tahonermann]

@rjmccall, could you please take a look at the updated changes? I think I've addressed the issues you raised.

[rjmccall]

Thanks, this looks great.

clang/lib/CodeGen/ItaniumCXXABI.cpp

if (!threadsafe || MaxInlineWidthInBits) {
 
     // Check if the first byte of the guard variable is zero.
     CGF.EmitCXXGuardedInitBranch(NeedsInit, InitCheckBlock, EndBlock,
                                  CodeGenFunction::GuardKind::VariableGuard, &D);
 
     CGF.EmitBlock(InitCheckBlock);
   }
 
+  // The semantics of dynamic initialization of variables with static or thread
+  // storage duration depends on whether they are declared at block-scope. The
+  // initialization of such variables at block-scope can be aborted with an
+  // exception and later retried (per C++20 [stmt.dcl]p4), and recursive entry
+  // to their initialization has undefined behavior (also per C++20
+  // [stmt.dcl]p4). For such variables declared at non-block scope, exceptions
+  // lead to termination (per C++20 [except.terminate]p1), and recursive
+  // references to the variables are governed only by the lifetime rules (per
+  // C++20 [class.cdtor]p2), which means such references are perfectly fine as
+  // long as they avoid touching memory. As a result, block-scope variables must
+  // not be marked as initialized until after initialization completes (unless
+  // the mark is reverted following an exception), but non-block-scope variables
+  // must be marked prior to initialization so that recursive accesses during
+  // initialization do not restart initialization.
+
   // Variables used when coping with thread-safe statics and exceptions.
   if (threadsafe) {
     // Call __cxa_guard_acquire.
     llvm::Value *V
       = CGF.EmitNounwindRuntimeCall(getGuardAcquireFn(CGM, guardPtrTy), guard);
 
     llvm::BasicBlock *InitBlock = CGF.createBasicBlock("init");
 
     Builder.CreateCondBr(Builder.CreateIsNotNull(V, "tobool"),
                          InitBlock, EndBlock);
 
     // Call __cxa_guard_abort along the exceptional edge.
     CGF.EHStack.pushCleanup<CallGuardAbort>(EHCleanup, guard);
 
     CGF.EmitBlock(InitBlock);
+  } else if (!D.isLocalVarDecl()) {
+    // For non-local variables, store 1 into the first byte of the guard
+    // variable before the object initialization begins so that references
+    // to the variable during initialization don't restart initialization.
+    Builder.CreateStore(llvm::ConstantInt::get(CGM.Int8Ty, 1),
+                        Builder.CreateElementBitCast(guardAddr, CGM.Int8Ty));
   }
 
   // Emit the initializer and add a global destructor if appropriate.
   CGF.EmitCXXGlobalVarDeclInit(D, var, shouldPerformInit);
 
   if (threadsafe) {
     // Pop the guard-abort cleanup if we pushed one.
     CGF.PopCleanupBlock();
 
     // Call __cxa_guard_release.  This cannot throw.
     CGF.EmitNounwindRuntimeCall(getGuardReleaseFn(CGM, guardPtrTy),
                                 guardAddr.getPointer());
-  } else {
-    // Store 1 into the first byte of the guard variable after initialization is
-    // complete.
+  } else if (D.isLocalVarDecl()) {
+    // For local variables, store 1 into the first byte of the guard variable
+    // after the object initialization completes so that initialization is
+    // retried if initialization is interrupted by an exception.
     Builder.CreateStore(llvm::ConstantInt::get(CGM.Int8Ty, 1),
                         Builder.CreateElementBitCast(guardAddr, CGM.Int8Ty));
   }
 
   CGF.EmitBlock(EndBlock);
 }
 
 /// Register a global destructor using __cxa_atexit.

clang/test/CodeGenCXX/aix-static-init-temp-spec-and-inline-var.cpp

 
 // CHECK: define internal void @__cxx_global_var_init.2() [[ATTR:#[0-9]+]] {
 // CHECK: entry:
 // CHECK:   %0 = load i8, ptr @_ZGVN5test21AIvE8instanceE, align 8
 // CHECK:   %guard.uninitialized = icmp eq i8 %0, 0
 // CHECK:   br i1 %guard.uninitialized, label %init.check, label %init.end
 
 // CHECK: init.check:
+// CHECK:   store i8 1, ptr @_ZGVN5test21AIvE8instanceE, align 8
 // CHECK:   call void @_ZN5test21AIvEC1Ev(ptr {{[^,]*}} @_ZN5test21AIvE8instanceE)
 // CHECK:   %1 = call i32 @atexit(ptr @__dtor__ZN5test21AIvE8instanceE)
-// CHECK:   store i8 1, ptr @_ZGVN5test21AIvE8instanceE, align 8
 // CHECK:   br label %init.end
 
 // CHECK: init.end:
 // CHECK:   ret void
 // CHECK: }
 
 // CHECK: define internal void @__dtor__ZN5test21AIvE8instanceE() [[ATTR:#[0-9]+]] {
 // CHECK: entry:

 
 // CHECK: define internal void @__cxx_global_var_init.4() [[ATTR:#[0-9]+]] {
 // CHECK: entry:
 // CHECK:   %0 = load i8, ptr @_ZGVN5test12t1IiEE, align 8
 // CHECK:   %guard.uninitialized = icmp eq i8 %0, 0
 // CHECK:   br i1 %guard.uninitialized, label %init.check, label %init.end
 
 // CHECK: init.check:
+// CHECK:   store i8 1, ptr @_ZGVN5test12t1IiEE, align 8
 // CHECK32: call void @_ZN5test15Test1C1Ei(ptr {{[^,]*}} @_ZN5test12t1IiEE, i32 noundef 2)
 // CHECK64: call void @_ZN5test15Test1C1Ei(ptr {{[^,]*}} @_ZN5test12t1IiEE, i32 noundef signext 2)
 // CHECK:   %1 = call i32 @atexit(ptr @__dtor__ZN5test12t1IiEE)
-// CHECK:   store i8 1, ptr @_ZGVN5test12t1IiEE, align 8
 // CHECK:   br label %init.end
 
 // CHECK: init.end:
 // CHECK:   ret void
 // CHECK: }
 
 // CHECK: define internal void @__dtor__ZN5test12t1IiEE() [[ATTR:#[0-9]+]] {
 // CHECK: entry:

clang/test/CodeGenCXX/cxx11-thread-local.cpp

 
 // LINUX_AIX: define internal void @[[VF_M_INIT]]()
 // DARWIN: define internal cxx_fast_tlscc void @[[VF_M_INIT]]()
 // CHECK-NOT: comdat
 // CHECK: load i8, i8* bitcast (i64* @_ZGVN1VIfE1mE to i8*)
 // CHECK: %[[VF_M_INITIALIZED:.*]] = icmp eq i8 %{{.*}}, 0
 // CHECK: br i1 %[[VF_M_INITIALIZED]],
 // need init:
+// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1VIfE1mE to i8*)
 // CHECK: call{{.*}} i32 @_Z1gv()
 // CHECK: [[VFM_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @_ZN1VIfE1mE)
 // CHECK: store i32 %{{.*}}, i32* [[VFM_ADDR]], align 4
-// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1VIfE1mE to i8*)
 // CHECK: br label
 
 // LINUX_AIX: define internal void @[[XF_M_INIT]]()
 // DARWIN: define internal cxx_fast_tlscc void @[[XF_M_INIT]]()
 // CHECK-NOT: comdat
 // CHECK: load i8, i8* bitcast (i64* @_ZGVN1XIfE1mE to i8*)
 // CHECK: %[[XF_M_INITIALIZED:.*]] = icmp eq i8 %{{.*}}, 0
 // CHECK: br i1 %[[XF_M_INITIALIZED]],
 // need init:
+// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1XIfE1mE to i8*)
 // AIX-NOT: br
 // LINUX: call {{.*}}__cxa_thread_atexit
 // AIX: call {{.*}}__pt_atexit_np
 // DARWIN: call {{.*}}_tlv_atexit
-// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1XIfE1mE to i8*)
 // CHECK: br label
 
 // LINUX: declare i32 @__cxa_thread_atexit(void (i8*)*, i8*, i8*)
 // AIX: declare i32 @__pt_atexit_np(i32, i32 (i32, ...)*, ...)
 // DARWIN: declare i32 @_tlv_atexit(void (i8*)*, i8*, i8*)
 
 // DARWIN: declare cxx_fast_tlscc noundef i32* @_ZTWN1VIcE1mE()
 // LINUX_AIX: define linkonce_odr hidden noundef i32* @_ZTWN1VIcE1mE() {{#[0-9]+}}{{( comdat)?}} {

 
 // LINUX_AIX: define internal void @[[V_M_INIT]]()
 // DARWIN: define internal cxx_fast_tlscc void @[[V_M_INIT]]()
 // CHECK-NOT: comdat
 // CHECK: load i8, i8* bitcast (i64* @_ZGVN1VIiE1mE to i8*)
 // CHECK: %[[V_M_INITIALIZED:.*]] = icmp eq i8 %{{.*}}, 0
 // CHECK: br i1 %[[V_M_INITIALIZED]],
 // need init:
+// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1VIiE1mE to i8*)
 // CHECK: call{{.*}} i32 @_Z1gv()
 // CHECK: [[VEM_ADDR:%.+]] = call align 4 i32* @llvm.threadlocal.address.p0i32(i32* align 4 @_ZN1VIiE1mE)
 // CHECK: store i32 %{{.*}}, i32* [[VEM_ADDR]], align 4
-// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1VIiE1mE to i8*)
 // CHECK: br label
 
 // LINUX_AIX: define internal void @[[X_M_INIT]]()
 // DARWIN: define internal cxx_fast_tlscc void @[[X_M_INIT]]()
 // CHECK-NOT: comdat
 // CHECK: load i8, i8* bitcast (i64* @_ZGVN1XIiE1mE to i8*)
 // CHECK: %[[X_M_INITIALIZED:.*]] = icmp eq i8 %{{.*}}, 0
 // CHECK: br i1 %[[X_M_INITIALIZED]],
 // need init:
+// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1XIiE1mE to i8*)
 // LINUX: call {{.*}}__cxa_thread_atexit
 // AIX: call {{.*}}__pt_atexit_np
 // DARWIN: call {{.*}}_tlv_atexit
-// CHECK: store i8 1, i8* bitcast (i64* @_ZGVN1XIiE1mE to i8*)
 // CHECK: br label
 
 // CHECK: define {{.*}}@[[GLOBAL_INIT:.*]]()
 // CHECK: call void @[[C_INIT]]()
 // CHECK: call void @[[E_INIT]]()
 
 
 // CHECK: define {{.*}}@__tls_init()

clang/test/CodeGenCXX/cxx1y-variable-template.cpp

 // variable with internal linkage.
 namespace PR42111 {
   int f();
   namespace { template <int = 0> thread_local int n = f(); }
   // CHECK: define {{.*}}@[[PR42111_CTOR]](
   // CHECK: load {{.*}} @_ZGVN7PR4211112_GLOBAL__N_11nILi0EEE
   // CHECK: icmp eq i8 {{.*}}, 0
   // CHECK: br i1
+  // CHECK: store i8 1, ptr @_ZGVN7PR4211112_GLOBAL__N_11nILi0EEE
   // CHECK: call noundef i32 @_ZN7PR421111fEv(
   // CHECK: [[N_ADDR:%.+]] = call align 4 ptr @llvm.threadlocal.address.p0(ptr align 4 @_ZN7PR4211112_GLOBAL__N_11nILi0EEE)
   // CHECK: store i32 {{.*}}, ptr [[N_ADDR]]
-  // CHECK: store i8 1, ptr @_ZGVN7PR4211112_GLOBAL__N_11nILi0EEE
   int g() { return n<> + n<>; }
 }

clang/test/CodeGenCXX/global-init.cpp

 namespace test4 {
   struct A {
     A();
   };
   extern int foo();
 
   // This needs an initialization function and guard variables.
   // CHECK: load i8, i8* bitcast (i64* @_ZGVN5test41xE to i8*)
-  // CHECK: [[CALL:%.*]] = call noundef i32 @_ZN5test43fooEv
+  // CHECK: store i8 1, i8* bitcast (i64* @_ZGVN5test41xE to i8*)
+  // CHECK-NEXT: [[CALL:%.*]] = call noundef i32 @_ZN5test43fooEv
   // CHECK-NEXT: store i32 [[CALL]], i32* @_ZN5test41xE
-  // CHECK-NEXT: store i8 1, i8* bitcast (i64* @_ZGVN5test41xE to i8*)
   __attribute__((weak)) int x = foo();
 }
 
 namespace PR5974 {
   struct A { int a; };
   struct B { int b; };
   struct C : A, B { int c; };
 

clang/test/CodeGenCXX/static-data-member.cpp

namespace test3 {
   template struct A<int>;
 
   // CHECK-LABEL: define internal void @__cxx_global_var_init.1() {{.*}} comdat($_ZN5test31AIiE1xE)
   // MACHO-LABEL: define internal void @__cxx_global_var_init.1()
   // MACHO-NOT: comdat
   // CHECK:      [[GUARDBYTE:%.*]] = load i8, ptr @_ZGVN5test31AIiE1xE
   // CHECK-NEXT: [[UNINITIALIZED:%.*]] = icmp eq i8 [[GUARDBYTE]], 0
   // CHECK-NEXT: br i1 [[UNINITIALIZED]]
-  // CHECK:      [[TMP:%.*]] = call noundef i32 @_ZN5test33fooEv()
+  // CHECK:      store i8 1, ptr @_ZGVN5test31AIiE1xE
+  // CHECK-NEXT: [[TMP:%.*]] = call noundef i32 @_ZN5test33fooEv()
   // CHECK-NEXT: store i32 [[TMP]], ptr @_ZN5test31AIiE1xE, align 4
-  // CHECK-NEXT: store i8 1, ptr @_ZGVN5test31AIiE1xE
   // CHECK-NEXT: br label
   // CHECK:      ret void
 }
 
 // Test that we can fold member lookup expressions which resolve to static data
 // members.
 namespace test4 {
   struct A {

clang/test/OpenMP/threadprivate_codegen.cpp

 //
 // CHECK1-LABEL: define {{[^@]+}}@__cxx_global_var_init.8
 // CHECK1-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) {
 // CHECK1-NEXT:  entry:
 // CHECK1-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK1-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0
 // CHECK1-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]]
 // CHECK1:       init.check:
+// CHECK1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK1-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
 // CHECK1-NEXT:    call void @__kmpc_threadprivate_register(ptr @[[GLOB1]], ptr @_ZN2STI2S4E2stE, ptr @.__kmpc_global_ctor_..9, ptr null, ptr @.__kmpc_global_dtor_..10)
 // CHECK1-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23)
 // CHECK1-NEXT:    [[TMP2:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]]
-// CHECK1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK1-NEXT:    br label [[INIT_END]]
 // CHECK1:       init.end:
 // CHECK1-NEXT:    ret void
 //
 //
 // CHECK1-LABEL: define {{[^@]+}}@.__kmpc_global_ctor_..9
 // CHECK1-SAME: (ptr noundef [[TMP0:%.*]]) #[[ATTR0]] {
 // CHECK1-NEXT:  entry:

 //
 // CHECK2-LABEL: define {{[^@]+}}@__cxx_global_var_init.8
 // CHECK2-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) {
 // CHECK2-NEXT:  entry:
 // CHECK2-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK2-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0
 // CHECK2-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]]
 // CHECK2:       init.check:
+// CHECK2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK2-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB1]])
 // CHECK2-NEXT:    call void @__kmpc_threadprivate_register(ptr @[[GLOB1]], ptr @_ZN2STI2S4E2stE, ptr @.__kmpc_global_ctor_..9, ptr null, ptr @.__kmpc_global_dtor_..10)
 // CHECK2-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23)
 // CHECK2-NEXT:    [[TMP2:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]]
-// CHECK2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK2-NEXT:    br label [[INIT_END]]
 // CHECK2:       init.end:
 // CHECK2-NEXT:    ret void
 //
 //
 // CHECK2-LABEL: define {{[^@]+}}@.__kmpc_global_ctor_..9
 // CHECK2-SAME: (ptr noundef [[TMP0:%.*]]) #[[ATTR0]] {
 // CHECK2-NEXT:  entry:

 //
 // SIMD1-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // SIMD1-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) {
 // SIMD1-NEXT:  entry:
 // SIMD1-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8
 // SIMD1-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0
 // SIMD1-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]]
 // SIMD1:       init.check:
+// SIMD1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // SIMD1-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23)
 // SIMD1-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]]
-// SIMD1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // SIMD1-NEXT:    br label [[INIT_END]]
 // SIMD1:       init.end:
 // SIMD1-NEXT:    ret void
 //
 //
 // SIMD1-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // SIMD1-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 {
 // SIMD1-NEXT:  entry:

 //
 // SIMD2-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // SIMD2-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) !dbg [[DBG233:![0-9]+]] {
 // SIMD2-NEXT:  entry:
 // SIMD2-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG234:![0-9]+]]
 // SIMD2-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0, !dbg [[DBG234]]
 // SIMD2-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]], !dbg [[DBG234]]
 // SIMD2:       init.check:
+// SIMD2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG234]]
 // SIMD2-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23), !dbg [[DBG235:![0-9]+]]
 // SIMD2-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]], !dbg [[DBG234]]
-// SIMD2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG234]]
 // SIMD2-NEXT:    br label [[INIT_END]], !dbg [[DBG234]]
 // SIMD2:       init.end:
 // SIMD2-NEXT:    ret void, !dbg [[DBG237:![0-9]+]]
 //
 //
 // SIMD2-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // SIMD2-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 !dbg [[DBG238:![0-9]+]] {
 // SIMD2-NEXT:  entry:

 //
 // CHECK-TLS1-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // CHECK-TLS1-SAME: () #[[ATTR0]] {
 // CHECK-TLS1-NEXT:  entry:
 // CHECK-TLS1-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK-TLS1-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0
 // CHECK-TLS1-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]]
 // CHECK-TLS1:       init.check:
+// CHECK-TLS1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK-TLS1-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23)
 // CHECK-TLS1-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_thread_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]]
-// CHECK-TLS1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK-TLS1-NEXT:    br label [[INIT_END]]
 // CHECK-TLS1:       init.end:
 // CHECK-TLS1-NEXT:    ret void
 //
 //
 // CHECK-TLS1-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // CHECK-TLS1-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 {
 // CHECK-TLS1-NEXT:  entry:

 //
 // CHECK-TLS2-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // CHECK-TLS2-SAME: () #[[ATTR7]] {
 // CHECK-TLS2-NEXT:  entry:
 // CHECK-TLS2-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK-TLS2-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0
 // CHECK-TLS2-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]]
 // CHECK-TLS2:       init.check:
+// CHECK-TLS2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK-TLS2-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23)
 // CHECK-TLS2-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_thread_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR4]]
-// CHECK-TLS2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // CHECK-TLS2-NEXT:    br label [[INIT_END]]
 // CHECK-TLS2:       init.end:
 // CHECK-TLS2-NEXT:    ret void
 //
 //
 // CHECK-TLS2-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // CHECK-TLS2-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR2]] comdat align 2 {
 // CHECK-TLS2-NEXT:  entry:

 //
 // CHECK-TLS3-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // CHECK-TLS3-SAME: () #[[ATTR0]] !dbg [[DBG290:![0-9]+]] {
 // CHECK-TLS3-NEXT:  entry:
 // CHECK-TLS3-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG291:![0-9]+]]
 // CHECK-TLS3-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0, !dbg [[DBG291]]
 // CHECK-TLS3-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]], !dbg [[DBG291]]
 // CHECK-TLS3:       init.check:
+// CHECK-TLS3-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG291]]
 // CHECK-TLS3-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23), !dbg [[DBG292:![0-9]+]]
 // CHECK-TLS3-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_thread_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]], !dbg [[DBG291]]
-// CHECK-TLS3-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG291]]
 // CHECK-TLS3-NEXT:    br label [[INIT_END]], !dbg [[DBG291]]
 // CHECK-TLS3:       init.end:
 // CHECK-TLS3-NEXT:    ret void, !dbg [[DBG294:![0-9]+]]
 //
 //
 // CHECK-TLS3-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // CHECK-TLS3-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 !dbg [[DBG295:![0-9]+]] {
 // CHECK-TLS3-NEXT:  entry:

 //
 // CHECK-TLS4-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // CHECK-TLS4-SAME: () #[[ATTR7]] !dbg [[DBG290:![0-9]+]] {
 // CHECK-TLS4-NEXT:  entry:
 // CHECK-TLS4-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG291:![0-9]+]]
 // CHECK-TLS4-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0, !dbg [[DBG291]]
 // CHECK-TLS4-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]], !dbg [[DBG291]]
 // CHECK-TLS4:       init.check:
+// CHECK-TLS4-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG291]]
 // CHECK-TLS4-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23), !dbg [[DBG292:![0-9]+]]
 // CHECK-TLS4-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_thread_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR5]], !dbg [[DBG291]]
-// CHECK-TLS4-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG291]]
 // CHECK-TLS4-NEXT:    br label [[INIT_END]], !dbg [[DBG291]]
 // CHECK-TLS4:       init.end:
 // CHECK-TLS4-NEXT:    ret void, !dbg [[DBG294:![0-9]+]]
 //
 //
 // CHECK-TLS4-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // CHECK-TLS4-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR3]] comdat align 2 !dbg [[DBG295:![0-9]+]] {
 // CHECK-TLS4-NEXT:  entry:

 //
 // SIMD3-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // SIMD3-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) {
 // SIMD3-NEXT:  entry:
 // SIMD3-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8
 // SIMD3-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0
 // SIMD3-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]]
 // SIMD3:       init.check:
+// SIMD3-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // SIMD3-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23)
 // SIMD3-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]]
-// SIMD3-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8
 // SIMD3-NEXT:    br label [[INIT_END]]
 // SIMD3:       init.end:
 // SIMD3-NEXT:    ret void
 //
 //
 // SIMD3-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // SIMD3-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 {
 // SIMD3-NEXT:  entry:

 //
 // SIMD4-LABEL: define {{[^@]+}}@__cxx_global_var_init.3
 // SIMD4-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) !dbg [[DBG233:![0-9]+]] {
 // SIMD4-NEXT:  entry:
 // SIMD4-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG234:![0-9]+]]
 // SIMD4-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0, !dbg [[DBG234]]
 // SIMD4-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]], !dbg [[DBG234]]
 // SIMD4:       init.check:
+// SIMD4-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG234]]
 // SIMD4-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23), !dbg [[DBG235:![0-9]+]]
 // SIMD4-NEXT:    [[TMP1:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR3]], !dbg [[DBG234]]
-// SIMD4-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG234]]
 // SIMD4-NEXT:    br label [[INIT_END]], !dbg [[DBG234]]
 // SIMD4:       init.end:
 // SIMD4-NEXT:    ret void, !dbg [[DBG237:![0-9]+]]
 //
 //
 // SIMD4-LABEL: define {{[^@]+}}@_ZN2S4C1Ei
 // SIMD4-SAME: (ptr noundef nonnull align 4 dereferenceable(8) [[THIS:%.*]], i32 noundef [[A:%.*]]) unnamed_addr #[[ATTR1]] comdat align 2 !dbg [[DBG238:![0-9]+]] {
 // SIMD4-NEXT:  entry:

 //
 // DEBUG1-LABEL: define {{[^@]+}}@__cxx_global_var_init.8
 // DEBUG1-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) !dbg [[DBG333:![0-9]+]] {
 // DEBUG1-NEXT:  entry:
 // DEBUG1-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG334:![0-9]+]]
 // DEBUG1-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0, !dbg [[DBG334]]
 // DEBUG1-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]], !dbg [[DBG334]]
 // DEBUG1:       init.check:
+// DEBUG1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG334]]
 // DEBUG1-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB41:[0-9]+]]), !dbg [[DBG334]]
 // DEBUG1-NEXT:    call void @__kmpc_threadprivate_register(ptr @[[GLOB41]], ptr @_ZN2STI2S4E2stE, ptr @.__kmpc_global_ctor_..9, ptr null, ptr @.__kmpc_global_dtor_..10), !dbg [[DBG334]]
 // DEBUG1-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23), !dbg [[DBG335:![0-9]+]]
 // DEBUG1-NEXT:    [[TMP2:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR4]], !dbg [[DBG334]]
-// DEBUG1-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG334]]
 // DEBUG1-NEXT:    br label [[INIT_END]], !dbg [[DBG334]]
 // DEBUG1:       init.end:
 // DEBUG1-NEXT:    ret void, !dbg [[DBG337:![0-9]+]]
 //
 //
 // DEBUG1-LABEL: define {{[^@]+}}@.__kmpc_global_ctor_..9
 // DEBUG1-SAME: (ptr noundef [[TMP0:%.*]]) #[[ATTR0]] !dbg [[DBG338:![0-9]+]] {
 // DEBUG1-NEXT:  entry:

 //
 // DEBUG2-LABEL: define {{[^@]+}}@__cxx_global_var_init.7
 // DEBUG2-SAME: () #[[ATTR0]] comdat($_ZN2STI2S4E2stE) !dbg [[DBG284:![0-9]+]] {
 // DEBUG2-NEXT:  entry:
 // DEBUG2-NEXT:    [[TMP0:%.*]] = load i8, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG285:![0-9]+]]
 // DEBUG2-NEXT:    [[GUARD_UNINITIALIZED:%.*]] = icmp eq i8 [[TMP0]], 0, !dbg [[DBG285]]
 // DEBUG2-NEXT:    br i1 [[GUARD_UNINITIALIZED]], label [[INIT_CHECK:%.*]], label [[INIT_END:%.*]], !dbg [[DBG285]]
 // DEBUG2:       init.check:
+// DEBUG2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG285]]
 // DEBUG2-NEXT:    [[TMP1:%.*]] = call i32 @__kmpc_global_thread_num(ptr @[[GLOB41:[0-9]+]]), !dbg [[DBG285]]
 // DEBUG2-NEXT:    call void @__kmpc_threadprivate_register(ptr @[[GLOB41]], ptr @_ZN2STI2S4E2stE, ptr @.__kmpc_global_ctor_..8, ptr null, ptr @.__kmpc_global_dtor_..9), !dbg [[DBG285]]
 // DEBUG2-NEXT:    call void @_ZN2S4C1Ei(ptr noundef nonnull align 4 dereferenceable(8) @_ZN2STI2S4E2stE, i32 noundef 23), !dbg [[DBG286:![0-9]+]]
 // DEBUG2-NEXT:    [[TMP2:%.*]] = call i32 @__cxa_atexit(ptr @_ZN2S4D1Ev, ptr @_ZN2STI2S4E2stE, ptr @__dso_handle) #[[ATTR4]], !dbg [[DBG285]]
-// DEBUG2-NEXT:    store i8 1, ptr @_ZGVN2STI2S4E2stE, align 8, !dbg [[DBG285]]
 // DEBUG2-NEXT:    br label [[INIT_END]], !dbg [[DBG285]]
 // DEBUG2:       init.end:
 // DEBUG2-NEXT:    ret void, !dbg [[DBG288:![0-9]+]]
 //
 //
 // DEBUG2-LABEL: define {{[^@]+}}@.__kmpc_global_ctor_..8
 // DEBUG2-SAME: (ptr noundef [[TMP0:%.*]]) #[[ATTR0]] !dbg [[DBG289:![0-9]+]] {
 // DEBUG2-NEXT:  entry: