[CUDA] Add section to docs about controlling fp optimizations.

[jlebar]

Clarify that -fp-contract='fast' default is just for CUDA device code.

[tra]

docs/CompileCudaWithLLVM.rst

160

It's actually for both host and device. The idea was to keep host & device behavior as close to each other as we can.

[jlebar]

Update per tra's comment that fp-contract=fast is set on both the host and device.

[jlebar]

Remove unrelated changes accidentally included in last revision.

[jlebar]

Thanks, Art!

[rnk]

lgtm

docs/CompileCudaWithLLVM.rst

166–168

Do we actually know how to honor the C11 semantics?

[jlebar]

docs/CompileCudaWithLLVM.rst

166–168

My understanding from talking to Chandler was that this was the intent of "on". But there is a good chance I misunderstood.

docs/CompileCudaWithLLVM.rst

.. code-block:: c++
 
   #if defined(__clang__) && defined(__CUDA__) && defined(__CUDA_ARCH__)
     // clang compiling CUDA code, device mode.
   #endif
 
 Both clang and nvcc define ``__CUDACC__`` during CUDA compilation.  You can
 detect NVCC specifically by looking for ``__NVCC__``.
 
+Flags that control numerical code
+=================================
+
+If you're using GPUs, you probably care about making numerical code run fast.
+GPU hardware allows for more control over numerical operations than most CPUs,
+but this results in more compiler options for you to juggle.
+
+Flags you may wish to tweak include:
+
+* ``-ffp-contract={on,off,fast}`` (default to ``fast`` for CUDA device code)
+  Controls whether the compiler emits fused multiply-add operations.
+
+  * ``off``: never emit fma operations, and prevent ptxas from fusing multiply
+    and add instructions.
+  * ``on``: fuse multiplies and adds within a single statement, but never
+    across statements (C11 semantics).  Prevent ptxas from fusing other
+    multiplies and adds.
+  * ``fast``: fuse multiplies and adds wherever profitable, even across
+    statements.  Don't prevent ptxas from fusing additional multiplies and
+    adds.
+
+  Fused multiply-add instructions can be much faster than the unfused
+  equivalents, but because the intermediate result in an fma is not rounded,
+  this flag can affect numerical code.
+
+* ``-fcuda-flush-denormals-to-zero`` (default: off) When this is enabled,
+  floating point operations may flush `denormal
+  <https://en.wikipedia.org/wiki/Denormal_number>`_ inputs and/or outputs to 0.
+  Operations on denormal numbers are often much slower than the same operations
+  on normal numbers.
+
+* ``-fcuda-approx-transcendentals`` (default: off) When this is enabled, the
+  compiler may emit calls to faster, approximate versions of transcendental
+  functions, instead of using the slower, fully IEEE-compliant versions.  For
+  example, this flag allows clang to emit the ptx ``sin.approx.f32``
+  instruction.
+
+  This is implied by ``-ffast-math``.
+
 Optimizations
 =============
 
 CPU and GPU have different design philosophies and architectures. For example, a
 typical CPU has branch prediction, out-of-order execution, and is superscalar,
 whereas a typical GPU has none of these. Due to such differences, an
 optimization pipeline well-tuned for CPUs may be not suitable for GPUs.