Clang/driver: emulated TLS mode.

[chh]

Summary

This is the clang and driver changes for emulated TLS implemented in LLVM in http://reviews.llvm.org/D10522.

[chh]

ping.

[aaron.ballman]

I do not know enough about TLS modes to comment on much of this. The
attribute-specific stuff looks good to me, with one exception:

Index: docs/AttributeReference.rst

• docs/AttributeReference.rst

+++ docs/AttributeReference.rst
@@ -743,6 +743,7 @@

• local-dynamic
• initial-exec
• local-exec

+* emulated

Please do not modify AttributeReference.rst directly; just AttrDocs.td.

Thanks!

~Aaron

[chh]

This is for bug https://llvm.org/bugs/show_bug.cgi?id=23566

[chh]

Aaron, I uploaded a second patch which does not change AttributeReference.rst.
Thanks.

[joerg]

Why do we need this? Shouldn't the backend translate whatever TLS access it finds into the emulation? I really would like to see some context for this change.

[chh]

This patch should keep the default clang/llvm configuration for all targets, but add the necessary -ftls-model=emulated flag for Android users that need TLS.
Please see also the comments in the llvm bug:
https://llvm.org/bugs/show_bug.cgi?id=23566#c4

[chh]

I see your question now. At the beginning I was looking for a way to set up
some configuration flag too for *-*-android targets to tell LLVM how to
lower TLS code. Trying to recognize "android" environment in LLVM seems too
restricted, as the emulated model could be useful to any target.

The existing -ftls-model looks like a natural hook to me.
This allows any target to support multiple TLS models and let users choose
which model to compile. The attribute((tls_model(...)) syntax seems
also a nice feature to keep, so I added the "emulated" mode into both
-ftls-model and tls_model attributes. As you can see in this patch and the
other LLVM patch, I was testing multiple models at the same time.

[chh]

Here is Joerg's comment sent to llvm-commits@ but somehow did not reach here:

Date: Tue, 30 Jun 2015 13:03:13 +0200
From: Joerg Sonnenberger <[e-mail]>
To: [e-mail]
Subject: Re: [PATCH] Clang/driver: emulated TLS mode.
Message-ID: <[e-mail]>
Content-Type: text/plain; charset=us-ascii

On Mon, Jun 29, 2015 at 01:45:25PM -0700, Chih-hung Hsieh wrote:

I see your question now. At the beginning I was looking for a way to set up
some configuration flag too for *-*-android targets to tell LLVM how to
lower TLS code. Trying to recognize "android" environment in LLVM seems too
restricted, as the emulated model could be useful to any target.

Having a codegen flag to enable the translation of TLS access via the
emulation pass would be an option, but for that the pass should exist
first.

The existing -ftls-model looks like a natural hook to me.
This allows any target to support multiple TLS models and let users choose
which model to compile. The attribute((tls_model(...)) syntax seems
also a nice feature to keep, so I added the "emulated" mode into both
-ftls-model and tls_model attributes. As you can see in this patch and the
other LLVM patch, I was testing multiple models at the same time.

I don't think multiple TLS models in the sense of "native" vs "emultls"
is something that makes sense. TLS emulation is pretty much a crude hack
to workaround limitations either in the OS environment (Android) or the
toolchain (for NetBSD: VAX and Sun2). Given all the associated
complications, it would be insane to force the emulation path when
native TLS exists. Note that GCC supports different dialects for TLS
access on some platforms like ARM, but those are more like optimised
sequences.

[chh]

I agree that this is a less optimal, or you can say backward or "temporary", solution in llvm to support emutls. The important benefit now is to allow switching of a few core Android modules from gcc to clang/llvm sooner, as we cannot fix Android linker in time or upgrade all old Android devices for new TLS applications. Note that this is not only an Arm target issue but all major Android targets.

But Android's linker might have ELF TLS support in the future. I see no alternative for llvm to support emutls for current Android and there might be a time in the future Android users will need choices to continue using emutls (with gcc or llvm) or native TLS with llvm.

As emutls is target independent, in http://reviews.llvm.org/D10522, it is added as a new IL TLS mode. Of course it could also be completely implement in clang front-end and be translated to IL that contains no reference to special TLS get_address IL. That seems a more complicated implementation to me, or did I miss something?
Is there a similar front-end translation pass in clang that I can learn or reuse for emutls?

[rnk]

How does GCC do expose this model? Do they support __attribute__((tls_model("emulated"))), or is it a backend option like Joerg is suggesting?

If we go the backend option route, it doesn't have to be specific to Android, we can have some target predicate that can change in the future like useGCCEmulatedTLS().

test/CodeGenCXX/cxx11-thread-local.cpp

3

You can avoid duplicating the CHECK lines by giving FileCheck multiple prefixes:

| FileCheck --check-prefix=COMMON --check-prefix=EMU %s

[chh]

Thanks for the suggestion about FileCheck prefixes.
I will simplify the code soon.

Gcc has configuration flags to enable or disable TLS.
It's emutls model/pass can be selected through configuration:

https://github.com/gcc-mirror/gcc/blob/master/gcc/tree-emutls.c

I have not found a compile time flag to select emulated vs native TLS models,
although that should be doable.

When Android gcc is configured to use the emulated TLS model,
TLS attributes like attribute((tls_model("local-dynamic")))
are silently ignored. The emulated model is used.
Lacking warning is bad, and it might be accepted only
when a target really cannot support both native and native TLS.

Android supports only emulated TLS now, but should be able to support both
in the future. To replace Android gcc with llvm, I would really like to have
a compile time flag instead of two copies of Android llvm to support/test
both TLS models. The -ftls-model seems to be the flag to add such a feature.
Then it follows to allow 5 tls_model attributes.

I see that clang's -ftls-model flag sets default, but does not overwrite
attribute((tls_model("local-dynamic"))). So I think an easy
implementation is to do the same for the -ftls-model=emulated flag.

The alternative is a new compile-time flag to enable emulated TLS and overrides all other 4 native tls_model attributes, and give warnings. This new flag will be needed at both clang/driver level and llvm IL level, right? Is that what you want? Any suggestion for the name or mechanism?

[yaron.keren]

mingw on Windows uses emutls, so an option to do the same makes sense, maybe even the default to keep compatibility.

[rnk]

I don't think it's necessary to warn on the local_exec, etc models. Those flags still provide information about how visible the TLS is, we just don't use it.

It sounds like we should have a -f flag, like -femulated-tls, that controls what we do. The default for Android and MinGW would be to enable it, and that could change over time.

Chandler thinks that this emulation should be done in Clang IRGen, but Nick and I disagree that this should happen late in LLVM. I suggested it could be an IR codegen preparation pass, but David Majnemer disagreed because then it's harder to CSE the call to compute the address of the TLS. Then again, the IR lowering you already have doesn't support CSE, and I think that's fine.

[chh]

Reid, -femulated-tls and the defaults sound good to me.

I don't know much of clang's other passes yet. I made the emulated mode as similar to other 4 modes as possible and it seemed to be the easiest approach for me.

Could you tell me the best way to pass the -femulated-tls to llvm backend?

With -femulated-tls, we don't need attribute tls_model("emulated").
(1) If other tls_model attributes found by clang are passed to llvm, e.g., as thread_local(localexec) attribute, llvm will need a separate flag to process them like thread_local(emulated) attribute. Would this be a per compilation unit flag?
(2) If clang converts all other tls_model attributes and the default to thread_local(emulated) for llvm, we can use most of http://reviews.llvm.org/D10522 as is, and llvm can handle 5 TLS modes in one compilation unit.

[rnk]

In D10524#205066, @chh wrote:

Could you tell me the best way to pass the -femulated-tls to llvm backend?

Because of LTO, we generally don't have a place for codegen options to live on the module. Probably a string attribute "emulated-tls" or "tls-emulation" on the GlobalVariable would be the way to go.

[chh]

So in IR, a global variable can have string attributes like

thread_local emulated-tls
thread_local(localdynamic) emulated-tls
thread_local(initialexec) emulated-tls
thread_local(localexec) emulated-tls

I can give it a try.
Not sure yet if that will have shorter or longer patch.

[chh]

Diff 3 simplified the clang front-end and driver changes:

• Only -femulated-tls and -fno-emulated-tls flags are added. Default is -fno-emulated-tls mode.

• Unit tests are modified to check:
  • C uninitialized global variables are common, but uninitialized thread local variables are not.
  • The -femulated-tls and -fno-emulated-tls flags are passed from clang to cc1.

[rnk]

include/clang/Driver/Options.td

515

This doesn't need to be a cc1 option, we can have just -femulated-tls for cc1 and remove the negative one in the driver.

lib/Driver/Tools.cpp

3955–3956

Generally we try to canonicalize flags in the driver rather than in cc1. Rather than doing this, how about something like the following:

// Emulated TLS is enabled by default on Android, and can be enabled manually with -femulated-tls.
bool EmulatedTLSDefault = Triple.getEnvironment() == llvm::Triple::Android;
if (Args.hasFlag(options::OPT_femulated_tls, options::OPT_fno_emulated_tls, EmulatedTLSDefault))
  CmdArgs.push_back("-femulated-tls");

test/CodeGen/tls-model.c

7

We try to use %clang_cc1 outside of driver tests. We can check for the -femulated-tls flag easily in the driver, and then use the cc1 interface directly.

8–17

We only need the first test for CodeGen.

[chh]

I might guess wrong the suggested changes.
Please check if the new diff is what you suggested.
Thanks.

[rnk]

lgtm

test/CodeGenCXX/cxx11-thread-local.cpp

2

I also meant for this comment to apply here:
"We try to use %clang_cc1 outside of driver tests. We can check for the -femulated-tls flag easily in the driver, and then use the cc1 interface directly."

[chh]

test/CodeGenCXX/cxx11-thread-local.cpp

2

Sure. I will fix that before submit.

docs/UsersManual.rst

.. option:: -ftls-model=[model]
 
    Valid values are: ``global-dynamic``, ``local-dynamic``,
    ``initial-exec`` and ``local-exec``. The default value is
    ``global-dynamic``. The compiler may use a different model if the
    selected model is not supported by the target, or if a more
    efficient model can be used. The TLS model can be overridden per
    variable using the ``tls_model`` attribute.
 
+.. option:: -femulated-tls
+
+   Select emulated TLS model, which overrides all -ftls-model choices.
+
+   In emulated TLS mode, all access to TLS variables are converted to
+   calls to __emutls_get_address in the runtime library.
+
 .. option:: -mhwdiv=[values]
 
    Select the ARM modes (arm or thumb) that support hardware division
    instructions.
 
    Valid values are: ``arm``, ``thumb`` and ``arm,thumb``.
    This option is used to indicate which mode (arm or thumb) supports
    hardware division instructions. This only applies to the ARM

include/clang/Driver/Options.td

 def fno_dwarf_directory_asm : Flag<["-"], "fno-dwarf-directory-asm">, Group<f_Group>, Flags<[CC1Option]>;
 def felide_constructors : Flag<["-"], "felide-constructors">, Group<f_Group>;
 def fno_elide_type : Flag<["-"], "fno-elide-type">, Group<f_Group>,
     Flags<[CC1Option]>,
     HelpText<"Do not elide types when printing diagnostics">;
 def feliminate_unused_debug_symbols : Flag<["-"], "feliminate-unused-debug-symbols">, Group<f_Group>;
 def femit_all_decls : Flag<["-"], "femit-all-decls">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Emit all declarations, even if unused">;
+def femulated_tls : Flag<["-"], "femulated-tls">, Group<f_Group>, Flags<[CC1Option]>,
+  HelpText<"Use emutls functions to access thread_local variables">;
+def fno_emulated_tls : Flag<["-"], "fno-emulated-tls">, Group<f_Group>, Flags<[CC1Option]>;
 def fencoding_EQ : Joined<["-"], "fencoding=">, Group<f_Group>;
 def ferror_limit_EQ : Joined<["-"], "ferror-limit=">, Group<f_Group>, Flags<[CoreOption]>;
 def fexceptions : Flag<["-"], "fexceptions">, Group<f_Group>, Flags<[CC1Option, CoreOption]>,
   HelpText<"Enable support for exception handling">;
 def fexcess_precision_EQ : Joined<["-"], "fexcess-precision=">,
     Group<clang_ignored_gcc_optimization_f_Group>;
 def : Flag<["-"], "fexpensive-optimizations">, Group<clang_ignored_gcc_optimization_f_Group>;
 def : Flag<["-"], "fno-expensive-optimizations">, Group<clang_ignored_gcc_optimization_f_Group>;

include/clang/Frontend/CodeGenOptions.def

 CODEGENOPT(DisableTailCalls  , 1, 0) ///< Do not emit tail calls.
 CODEGENOPT(EmitDeclMetadata  , 1, 0) ///< Emit special metadata indicating what
                                      ///< Decl* various IR entities came from. 
                                      ///< Only useful when running CodeGen as a
                                      ///< subroutine.
 CODEGENOPT(EmitGcovArcs      , 1, 0) ///< Emit coverage data files, aka. GCDA.
 CODEGENOPT(EmitGcovNotes     , 1, 0) ///< Emit coverage "notes" files, aka GCNO.
 CODEGENOPT(EmitOpenCLArgMetadata , 1, 0) ///< Emit OpenCL kernel arg metadata.
+CODEGENOPT(EmulatedTLS       , 1, 0) ///< Set when -femulated-tls is enabled.
 /// \brief FP_CONTRACT mode (on/off/fast).
 ENUM_CODEGENOPT(FPContractMode, FPContractModeKind, 2, FPC_On)
 CODEGENOPT(ForbidGuardVariables , 1, 0) ///< Issue errors if C++ guard variables
                                         ///< are required.
 CODEGENOPT(FunctionSections  , 1, 0) ///< Set when -ffunction-sections is enabled.
 CODEGENOPT(InstrumentFunctions , 1, 0) ///< Set when -finstrument-functions is
                                        ///< enabled.
 CODEGENOPT(InstrumentForProfiling , 1, 0) ///< Set when -pg is enabled.

lib/CodeGen/BackendUtil.cpp

TargetMachine *EmitAssemblyHelper::CreateTargetMachine(bool MustCreateTM) {
   Options.NoNaNsFPMath = CodeGenOpts.NoNaNsFPMath;
   Options.NoZerosInBSS = CodeGenOpts.NoZeroInitializedInBSS;
   Options.UnsafeFPMath = CodeGenOpts.UnsafeFPMath;
   Options.StackAlignmentOverride = CodeGenOpts.StackAlignment;
   Options.PositionIndependentExecutable = LangOpts.PIELevel != 0;
   Options.FunctionSections = CodeGenOpts.FunctionSections;
   Options.DataSections = CodeGenOpts.DataSections;
   Options.UniqueSectionNames = CodeGenOpts.UniqueSectionNames;
+  Options.EmulatedTLS = CodeGenOpts.EmulatedTLS;
 
   Options.MCOptions.MCRelaxAll = CodeGenOpts.RelaxAll;
   Options.MCOptions.MCSaveTempLabels = CodeGenOpts.SaveTempLabels;
   Options.MCOptions.MCUseDwarfDirectory = !CodeGenOpts.NoDwarfDirectoryAsm;
   Options.MCOptions.MCNoExecStack = CodeGenOpts.NoExecStack;
   Options.MCOptions.MCFatalWarnings = CodeGenOpts.FatalWarnings;
   Options.MCOptions.AsmVerbose = CodeGenOpts.AsmVerbose;
   Options.MCOptions.ABIName = TargetOpts.ABI;

lib/Driver/Tools.cpp

if (Args.hasFlag(options::OPT_ffreestanding, options::OPT_fhosted, false) ||
     CmdArgs.push_back("-ffreestanding");
 
   // Forward -f (flag) options which we can pass directly.
   Args.AddLastArg(CmdArgs, options::OPT_femit_all_decls);
   Args.AddLastArg(CmdArgs, options::OPT_fheinous_gnu_extensions);
   Args.AddLastArg(CmdArgs, options::OPT_fstandalone_debug);
   Args.AddLastArg(CmdArgs, options::OPT_fno_standalone_debug);
   Args.AddLastArg(CmdArgs, options::OPT_fno_operator_names);
+  Args.AddLastArg(CmdArgs, options::OPT_femulated_tls);
+  Args.AddLastArg(CmdArgs, options::OPT_fno_emulated_tls);
   // AltiVec language extensions aren't relevant for assembling.
   if (!isa<PreprocessJobAction>(JA) || Output.getType() != types::TY_PP_Asm)
     Args.AddLastArg(CmdArgs, options::OPT_faltivec);
   Args.AddLastArg(CmdArgs, options::OPT_fdiagnostics_show_template_tree);
   Args.AddLastArg(CmdArgs, options::OPT_fno_elide_type);
 
   // Forward flags for OpenMP
   if (Args.hasFlag(options::OPT_fopenmp, options::OPT_fopenmp_EQ,

lib/Frontend/CompilerInvocation.cpp

if (Method == ~0U) {
       Diags.Report(diag::err_drv_invalid_value) << A->getAsString(Args) << Name;
       Success = false;
     } else {
       Opts.setObjCDispatchMethod(
         static_cast<CodeGenOptions::ObjCDispatchMethodKind>(Method));
     }
   }
 
+  Opts.EmulatedTLS =
+      Args.hasFlag(OPT_femulated_tls, OPT_fno_emulated_tls, false);
+
   if (Arg *A = Args.getLastArg(OPT_ftlsmodel_EQ)) {
     StringRef Name = A->getValue();
     unsigned Model = llvm::StringSwitch<unsigned>(Name)
         .Case("global-dynamic", CodeGenOptions::GeneralDynamicTLSModel)
         .Case("local-dynamic", CodeGenOptions::LocalDynamicTLSModel)
         .Case("initial-exec", CodeGenOptions::InitialExecTLSModel)
         .Case("local-exec", CodeGenOptions::LocalExecTLSModel)
         .Default(~0U);

test/CodeGen/tls-model.c

 // RUN: %clang_cc1 %s -triple x86_64-pc-linux-gnu -emit-llvm -o - | FileCheck %s -check-prefix=CHECK-GD
 // RUN: %clang_cc1 %s -triple x86_64-pc-linux-gnu -ftls-model=global-dynamic -emit-llvm -o - | FileCheck %s -check-prefix=CHECK-GD
 // RUN: %clang_cc1 %s -triple x86_64-pc-linux-gnu -ftls-model=local-dynamic -emit-llvm -o - | FileCheck %s -check-prefix=CHECK-LD
 // RUN: %clang_cc1 %s -triple x86_64-pc-linux-gnu -ftls-model=initial-exec -emit-llvm -o - | FileCheck %s -check-prefix=CHECK-IE
 // RUN: %clang_cc1 %s -triple x86_64-pc-linux-gnu -ftls-model=local-exec -emit-llvm -o - | FileCheck %s -check-prefix=CHECK-LE
+//
+// RUN: %clang %s -target x86_64-pc-linux-gnu -v -femulated-tls -S -emit-llvm -o - 2>&1 | \
+// RUN:     FileCheck %s -check-prefix=EMU -check-prefix=CHECK-GD
+// RUN: %clang %s -target x86_64-pc-linux-gnu -v -fno-emulated-tls -S -emit-llvm -o - 2>&1 | \
+// RUN:     FileCheck %s -check-prefix=NOEMU -check-prefix=CHECK-GD
+// RUN: %clang %s -target x86_64-linux-android -v -S -emit-llvm -o - 2>&1 | \
+// RUN:     FileCheck %s -check-prefix=DEFAULT -check-prefix=CHECK-GD
 
+// EMU: -cc1 {{.*}} -femulated-tls
+// NOEMU: -cc1 {{.*}} -fno-emulated-tls
+// DEFAULT-NOT: -cc1 {{.*}} -femulated-tls
+// DEFAULT-NOT: -cc1 {{.*}} -fno-emulated-tls
+
+int z1 = 0;
+int z2;
 int __thread x;
 int f() {
   static int __thread y;
   return y++;
 }
 int __thread __attribute__((tls_model("initial-exec"))) z;
 
+// Note that unlike normal C uninitialized global variables,
+// uninitialized TLS variables do NOT have COMMON linkage.
+
+// CHECK-GD: @z1 = global i32 0
 // CHECK-GD: @f.y = internal thread_local global i32 0
+// CHECK-GD: @z2 = common global i32 0
 // CHECK-GD: @x = thread_local global i32 0
 // CHECK-GD: @z = thread_local(initialexec) global i32 0
 
+// CHECK-LD: @z1 = global i32 0
 // CHECK-LD: @f.y = internal thread_local(localdynamic) global i32 0
+// CHECK-LD: @z2 = common global i32 0
 // CHECK-LD: @x = thread_local(localdynamic) global i32 0
 // CHECK-LD: @z = thread_local(initialexec) global i32 0
 
+// CHECK-IE: @z1 = global i32 0
 // CHECK-IE: @f.y = internal thread_local(initialexec) global i32 0
+// CHECK-IE: @z2 = common global i32 0
 // CHECK-IE: @x = thread_local(initialexec) global i32 0
 // CHECK-IE: @z = thread_local(initialexec) global i32 0
 
+// CHECK-LE: @z1 = global i32 0
 // CHECK-LE: @f.y = internal thread_local(localexec) global i32 0
+// CHECK-LE: @z2 = common global i32 0
 // CHECK-LE: @x = thread_local(localexec) global i32 0
 // CHECK-LE: @z = thread_local(initialexec) global i32 0

test/CodeGenCXX/cxx11-thread-local.cpp

 // RUN: %clang_cc1 -std=c++11 -emit-llvm %s -o - -triple x86_64-linux-gnu | FileCheck %s
+// RUN: %clang -std=c++11 -v -femulated-tls -S -emit-llvm %s -o - \
+// RUN:     -target x86_64-linux-gnu 2>&1 | FileCheck --check-prefix=CHECK --check-prefix=EMU %s
+
+// Check if the -femulated-tls option is passed to cc1.
+// EMU: -cc1 {{.*}} -femulated-tls
 
 int f();
 int g();
 
 // CHECK: @a = thread_local global i32 0
 thread_local int a = f();
 extern thread_local int b;
 // CHECK: @c = global i32 0