[CUDA] Added --cuda-noopt-device-debug option to control ptxas' debug info generation.

[echristo]

Wow really? That's ... wow.

OK

[jlebar]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

I think this is would be very surprising to users. -g does not usually have a large performance impact, so -O2 -g does not generally mean "generate slow code," as far as I know. I'm concerned that this will result in people accidentally compiling with ptxas -O0 (which is why I didn't do it like this to begin with).

Can we accomplish this in a more explicit way?

[echristo]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

Other than warning I'm not sure what we can do, we could do that instead and make everyone use O0 that wants debug info in their ptx?

[hfinkel]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

I'd rather we refuse to do anything (i.e. produce an error) than silently remove either optimizations or -g. Do we have a way to separately specify the optimization level for host and device code? If not, looks like we should add one.

[tra]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

NVCC has -G option to control ptxas' debug options. If it's present, ptxas optimizations are disabled. I could add a similar option. "-gcuda-device" perhaps?

[tra]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

@hfinkel: separate option should work. Any suggestions for a good name? -gcuda-device sounds awkward.

@echristo: "-O0" is not going to work for everyone in practice due to ptxas limitations. For instance on some thrust files ptxas runs out of memory on all non-inlined functions in unoptimized code. Compiling with -O2 is one way to work around that, but I do want device-side debug info!

[echristo]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

We can do that, I'd have warned because it doesn't seem like something we should hard error on, but I can see that perspective, i.e. we asked for "incompatible" options.

And no, we don't currently have a way to do that. We can try to come up with a driver interface.

10707 ↗

(On Diff #47558)

Huh? I'm not sure what you're doing here then with turning off optimizations in the presence of debug info requests. Your commentary made it sound like you can't have both.

[hfinkel]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

@hfinkel: separate option should work. Any suggestions for a good name? -gcuda-device sounds awkward.

I agree it sounds awkward, but I currently have no better suggestion.

[tra]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

@echristo: I assumed you were talking about compiler when you said "make everyone use O0". I need compiler to optimize code with -O2, but run ptxas with -g and -O0 which this patch makes possible.

I'll update the patch to add an option to control device-side debug info explicitly.

[echristo]

lib/Driver/Tools.cpp

10707 ↗

(On Diff #47558)

OK.

[tra]

Added --cuda-noopt-device-debug option to control ptxas' debug info generation.

The idea was to make it painfully obvious that unlike regular -g this option 8s going to affect performance of device-side code.
The option name is a mouthful, so feel free to suggest better alternatives.

[jlebar]

lib/Driver/Tools.cpp

10696 ↗

(On Diff #47680)

Nit: ignore the compiler's

cfe/trunk/include/clang/Driver/Options.td

 def c : Flag<["-"], "c">, Flags<[DriverOption]>,
   HelpText<"Only run preprocess, compile, and assemble steps">;
 def cuda_device_only : Flag<["--"], "cuda-device-only">,
   HelpText<"Do device-side CUDA compilation only">;
 def cuda_gpu_arch_EQ : Joined<["--"], "cuda-gpu-arch=">,
   Flags<[DriverOption, HelpHidden]>, HelpText<"CUDA GPU architecture">;
 def cuda_host_only : Flag<["--"], "cuda-host-only">,
   HelpText<"Do host-side CUDA compilation only">;
+def cuda_noopt_device_debug : Flag<["--"], "cuda-noopt-device-debug">,
+  HelpText<"Enable device-side debug info generation. Disables ptxas optimizations.">;
 def cuda_path_EQ : Joined<["--"], "cuda-path=">, Group<i_Group>,
   HelpText<"CUDA installation path">;
 def dA : Flag<["-"], "dA">, Group<d_Group>;
 def dD : Flag<["-"], "dD">, Group<d_Group>, Flags<[CC1Option]>,
   HelpText<"Print macro definitions in -E mode in addition to normal output">;
 def dM : Flag<["-"], "dM">, Group<d_Group>, Flags<[CC1Option]>,
   HelpText<"Print macro definitions in -E mode instead of normal output">;
 def dead__strip : Flag<["-"], "dead_strip">;

cfe/trunk/lib/Driver/Tools.cpp

const auto &TC =
       static_cast<const toolchains::CudaToolChain &>(getToolChain());
   assert(TC.getTriple().isNVPTX() && "Wrong platform");
 
   std::vector<std::string> gpu_archs =
       Args.getAllArgValues(options::OPT_march_EQ);
   assert(gpu_archs.size() == 1 && "Exactly one GPU Arch required for ptxas.");
   const std::string& gpu_arch = gpu_archs[0];
 
-
   ArgStringList CmdArgs;
   CmdArgs.push_back(TC.getTriple().isArch64Bit() ? "-m64" : "-m32");
-
+  if (Args.getLastArg(options::OPT_cuda_noopt_device_debug)) {
+    // ptxas does not accept -g option if optimization is enabled, so
+    // we ignore the compiler's -O* options if we want debug info.
+    CmdArgs.push_back("-g");
+    CmdArgs.push_back("--dont-merge-basicblocks");
+    CmdArgs.push_back("--return-at-end");
+  } else if (Arg *A = Args.getLastArg(options::OPT_O_Group)) {
     // Map the -O we received to -O{0,1,2,3}.
     //
-  // TODO: Perhaps we should map host -O2 to ptxas -O3. -O3 is ptxas's default,
-  // so it may correspond more closely to the spirit of clang -O2.
-  if (Arg *A = Args.getLastArg(options::OPT_O_Group)) {
+    // TODO: Perhaps we should map host -O2 to ptxas -O3. -O3 is ptxas's
+    // default, so it may correspond more closely to the spirit of clang -O2.
+
     // -O3 seems like the least-bad option when -Osomething is specified to
     // clang but it isn't handled below.
     StringRef OOpt = "3";
     if (A->getOption().matches(options::OPT_O4) ||
         A->getOption().matches(options::OPT_Ofast))
       OOpt = "3";
     else if (A->getOption().matches(options::OPT_O0))
       OOpt = "0";
if (Args.getLastArg(options::OPT_cuda_noopt_device_debug)) {
     }
     CmdArgs.push_back(Args.MakeArgString(llvm::Twine("-O") + OOpt));
   } else {
     // If no -O was passed, pass -O0 to ptxas -- no opt flag should correspond
     // to no optimizations, but ptxas's default is -O3.
     CmdArgs.push_back("-O0");
   }
 
-  // Don't bother passing -g to ptxas: It's enabled by default at -O0, and
-  // not supported at other optimization levels.
-
   CmdArgs.push_back("--gpu-name");
   CmdArgs.push_back(Args.MakeArgString(gpu_arch));
   CmdArgs.push_back("--output-file");
   CmdArgs.push_back(Args.MakeArgString(Output.getFilename()));
   for (const auto& II : Inputs)
     CmdArgs.push_back(Args.MakeArgString(II.getFilename()));
 
   for (const auto& A : Args.getAllArgValues(options::OPT_Xcuda_ptxas))

cfe/trunk/test/Driver/cuda-external-tools.cu

 // RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix OPT2 %s
 // RUN: %clang -### -target x86_64-linux-gnu -O3 -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix OPT3 %s
 // RUN: %clang -### -target x86_64-linux-gnu -O4 -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix OPT3 %s
 // RUN: %clang -### -target x86_64-linux-gnu -Ofast -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix OPT3 %s
 
+// With debugging enabled, ptxas should be run with with no ptxas optimizations.
+// RUN: %clang -### -target x86_64-linux-gnu --cuda-noopt-device-debug -O2 -c %s 2>&1 \
+// RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix DBG %s
+
 // Regular compile without -O.  This should result in us passing -O0 to ptxas.
 // RUN: %clang -### -target x86_64-linux-gnu -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix OPT0 %s
 
 // Regular compiles with -Os and -Oz.  For lack of a better option, we map
 // these to ptxas -O3.
 // RUN: %clang -### -target x86_64-linux-gnu -Os -c %s 2>&1 \
 // RUN: | FileCheck -check-prefix ARCH64 -check-prefix SM20 -check-prefix OPT2 %s

 // SM20: "-o" "[[PTXFILE:[^"]*]]"
 // SM35: "-o" "[[PTXFILE:[^"]*]]"
 
 // Match the call to ptxas (which assembles PTX to SASS).
 // CHECK:ptxas
 // ARCH64: "-m64"
 // ARCH32: "-m32"
 // OPT0: "-O0"
+// OPT0-NOT: "-g"
 // OPT1: "-O1"
+// OPT1-NOT: "-g"
 // OPT2: "-O2"
+// OPT2-NOT: "-g"
 // OPT3: "-O3"
+// OPT3-NOT: "-g"
+// DBG: "-g" "--dont-merge-basicblocks" "--return-at-end"
 // SM20: "--gpu-name" "sm_20"
 // SM35: "--gpu-name" "sm_35"
 // SM20: "--output-file" "[[CUBINFILE:[^"]*]]"
 // SM35: "--output-file" "[[CUBINFILE:[^"]*]]"
 // PTXAS-EXTRA: "-foo1"
 // PTXAS-EXTRA-SAME: "-foo2"
 // CHECK-SAME: "[[PTXFILE]]"