[Clang] [Docs] Add HLSLSupport page

[beanz]

Summary

This document is a first-stab at addressing some of the questions about
HLSL support in Clang.

[aaron.ballman]

Thanks for getting a start on this documentation, I appreciate it!

clang/docs/HLSLSupport.rst

12

I can't wait to read "is a new initiative underway" four years from now. :-D (We may want to prevent that by making it a less temporal statement.)

13

20–24

34

47–49

60

I think it would be good to also mention what happens when HLSL is in conflict with either C or C++, at a high level.

74

Probably also worth pointing out that most HLSL parsing is expected to be the usual C and C++ code paths, or is that not your expectation?

122–123

Presumably it also uses pointer types in the AST for things like array and function decay?

144–146

It's not clear whether the behavior will vary for all types or just vector types. Also, does this apply in preprocessor conditionals the same as runtime expressions?

151–153

Is it worth supporting at all (I know you want source compatibility...)? Type system changes are generally expensive and invasive, largely because changing the type system in C++ is complicated. For example, does this qualifier impact overload sets or template specializations? Does it get name mangled? That sort of thing.

167

169

176

186

From C99:
VLAs?
_Complex/_Imaginary?

From C11:
Threads/Atomics?
Language features like _Generic (or [[]] attributes in C2x)?

Also, it's not clear whether you expect these unsupported features to be diagnosed or supported as extensions in HLSL, etc.

190

No dynamic_cast or typeid?

194

Inline namespaces?

195

I presume...

How about placement new expressions?

198

Same question here as above with C on whether we expect to support those as extensions or diagnose them as invalid.

[tschuett]

Thanks. I let Aaron go over the details.

[kuhar]

Looks good to me overall, I just left some local comments. Please take my writing suggestions with a pinch of salt, English is my second language.

clang/docs/HLSLSupport.rst

14

ubernit: some of the idiosyncrasies -> some idiosyncrasies

23–24

What do you mean by source compatibility in this case? In terms of the input source language (HLSL), or the source code of both implementations?

Perhaps we could make it more direct? E.g., 'This requires us to add the HLSL language support to Clang.'

29–32

It's great you made this so explicit!

37

ubernit: how about just 'details'?

51–53

Is DXC the reference implementation?

56–57

This is a bit awkward, as if the language HSLS created files on disk on something :P.

Maybe something like: We will maintain separation between HSLS-specific code and the rest of Clang as much as possible (e.g., ParseHSLS.cpp, ...)

91–92

Are all attributes guaranteed to be preserved? I thought some might get dropped by opt.

129

nit: With member function support*,* HLSL also

136

nit: so *it* is worth?

138

accommodate

188

These are C++ language features. I assume that all library features are also out of the window?

[kuhar]

clang/docs/HLSLSupport.rst

191

How about goto and labels? Irreducible control flow? Are infinite loops valid?

[beanz]

I'll do an edit pass tonight and update. I've also gone through and replied to some of the questions and comments inline.

Thank you!

clang/docs/HLSLSupport.rst

23–24

I see the confusion there. I mean in terms of the input source, not the implementation. We do vend the compiler as a library too with a COM API to invoke the compiler on in-memory sources (which is slightly terrifying to me).

I don't yet know how much flexibility we have to radically change the API, but that shouldn't have any impact on the implementations in clang because we'll do it in a wrapper library.

51–53

Yes, with the caveat that it doesn't actually support older language versions (for that we have FXC).

74

Yea, that is the expectation.

91–92

Some attributes do get dropped in optimization. We are going to need to evaluate a bit case by case. DXC (and DXIL) puts a lot of information in metadata, which is a pain to work with through compiler layers. We actually have a set of helpers to read data in and out of the IR metadata... I'm trying to avoid needing any of that.

122–123

That is my plan for in Clang. In DXC they prevent array->pointer decay in the AST which IMO causes more problems than it solves, especially since the IR decays to pointers anyways.

144–146

It only applies in runtime expressions, and does apply to all types not just vectors. Vectors are the motivation for it because short circuiting would be unwieldy for vector types.

With HLSL 2021, operators follow C short circuit rules and are not allowed to operate on vector types, instead there are builtin functions to handle the vector cases.

151–153

Unfortunately we do need to implement it to some degree. In DXC it is implemented as an attribute rather than a qualifier, so it shouldn't impact overloads, template specialization or mangling.

I call it a qualifier here because it is syntactically more like a qualifier, but I'm honestly not sure how we should implement it. In DXC, we emit IR metadata denoting the underlying value as "precise", then in an IR pass propagate disabling fast math.

186

Yea... none of those are supported either.

C11 atomics might actually be interesting because I _think_ you could make them work.

There's also probably no good reason why _Generic wouldn't work.

My plan is to bring in [[]] attributes in HLSL 202x, so those get to stay :).

188

Yep, I was focused on language, but should mention the library.

190

Yep, no C++ casting at all (although we've discussed adding static and reinterpret casts).

191

No support for goto or labels. I think infinite loops are undefined behavior, but our resident HLSL language expert is on vacation and I can't find documentation.

194

Not supported, also not a feature I was aware of in C++... Learn something new every day...

195

Also not supported.

198

I didn't really answer that above. I am inclined to do a mix of both. Anything that we can support and lower I'd like to treat as extensions. Some things we just can't.

We don't currently have any way to allocate memory dynamically, so there's really no way to support new/delete. RTTI and exceptions are another case where we just don't have the support we would need in the driver specifications.

[beanz]

Updating with edits based on review feedback.

[aaron.ballman]

clang/docs/HLSLSupport.rst

151–153

Unfortunately we do need to implement it to some degree. In DXC it is implemented as an attribute rather than a qualifier, so it shouldn't impact overloads, template specialization or mangling.

Okay, I kind of figured you needed it.

I call it a qualifier here because it is syntactically more like a qualifier, but I'm honestly not sure how we should implement it. In DXC, we emit IR metadata denoting the underlying value as "precise", then in an IR pass propagate disabling fast math.

This smells more like an attribute than a qualifier to me, but I'd like to see if I understand first (please ignore if I get syntax wrong):

// Initialization
precise float f = 1.0 / 3.0 * sinf(3.14f); // value computation is precise
// Assignment
f = 1.0 / 3.0 * sinf(3.14f); // value computation is NOT precise (not a declaration)?
// RHS of expression
float f2 = f * 2.0; // value computation is NOT precise (f2 is not marked precise)?

198

Okay, that seems reasonable enough. Should we issue "this is a Clang extension" diagnostics in the places where you're supporting something as an extension? Also, I presume you intend to add error diagnostics for all the cases that are not supported?

[beanz]

clang/docs/HLSLSupport.rst

151–153

Yea, your syntax and behavior is correct with the added complication that precise upward propagates. The upward propagation is the part that is wonky and problematic.

float f = 1.0 * sinf(3.14f); // precise
precise float f2 = f / 2.0f; // precise
float f3 = f * f2; // not precise

198

I think issuing clang extension diagnostics will be super valuable to our users. My expectation is that it will take a few years to transition all of our users from DXC to Clang, and in the meantime if they are supporting both compilers in their codebases diagnostics for extension use will be immensely helpful.

For unsupported features, I think the immovable priority is that we can't succeed compiling and generate invalid code (in our primary case that would be code that can't be run under one of our driver specifications).

Having nice, targeted diagnostics for each unsupported case is the ideal, but there are cases (like with Microsoft attribute parsing on templates), where we fail the compile with diagnostics that are "good enough" or even just in parity to DXC which we may take as an opportunity for future improvement and not a requirement.

Does that make sense? I'm trying not to back myself into a corner of having perfect be the enemy of "better than what we have today".

[kuhar]

LGTM

clang/docs/HLSLSupport.rst

224

nit, for consistency with the other list items: make goto verbatim

226

nit: also verbatim?

238

ubernit: & -> and, for consistency with the other list items

[beanz]

Updating based on feedback :)

[aaron.ballman]

clang/docs/HLSLSupport.rst

151–153

Sorry if I'm being dense, but I'm still lost. Why is f precise in your example?

But more interestingly, is this a property of the *initialization* of the variable of that type only, or is it a property of the type itself where arbitrary expressions involving that type need to care whether it's precise or not?

If it's just about the initialization, then I think a declaration attribute is the clear winner for the semantic support. If it needs to be tracked through arbitrary expressions involving the type, it's probably better as a qualifier sort of like the nullability ones. e.g.,

template <typename Ty>
void func(Ty Val) {
  Val += 1.0f // Does this get precise semantics?
}

int main() {
  precise float f = 1.0f;
  func(f);
}

198

I think issuing clang extension diagnostics will be super valuable to our users.

+1, I think it'd be useful as well, for exactly that situation.

For unsupported features, I think the immovable priority is that we can't succeed compiling and generate invalid code (in our primary case that would be code that can't be run under one of our driver specifications).

+1, definitely agreed.

Having nice, targeted diagnostics for each unsupported case is the ideal, but there are cases (like with Microsoft attribute parsing on templates), where we fail the compile with diagnostics that are "good enough" or even just in parity to DXC which we may take as an opportunity for future improvement and not a requirement.

Does that make sense? I'm trying not to back myself into a corner of having perfect be the enemy of "better than what we have today".

Yes, this makes sense. Mostly, I want to ensure that code which is invalid causes errors in the frontend rather than errors in the backend or at runtime when it comes to unsupported features. So it's not so much about what diagnostics are generated as it is about ensuring diagnostics are generated where necessary.

[beanz]

clang/docs/HLSLSupport.rst

151–153

precise behaves unlike anything I've ever encountered in a programming language before. It neither follows the type nor the declared value.

precise never propagates down (so a use of a precise value does not make subsequent things precise. It does propagate up to values that contribute to the precise, which is just... weird.

I don't know whether HLSL or GLSL (or some other SL) did it first, but GLSL has a spec which states:

The precise qualifier ensures that operations contributing to a variable's value are done in their stated order and with operator consistency.

The GLSL spec expands into a whole lot more details and examples:
https://www.khronos.org/registry/OpenGL/specs/gl/GLSLangSpec.4.60.pdf

HLSL's behavior should match GLSL (and some other vendor-specific languages), but frequently doesn't.

198

I think You and I are 100% on the same page here. Fail early with actionable errors, rather than asserting in the backend ;P

[aaron.ballman]

LGTM!

clang/docs/HLSLSupport.rst

151–153

(I need to shower after learning this cursed knowledge, and I fear I will never be clean again.)

Thank you for the discussion on this (both here and on IRC). It sounds like we could get away with using a declaration attribute and then punt all the complexity back to the backend which handles it entirely from the initialization of the marked-precise variable. Not ideal, but far better than trying to model this in the frontend.

198

Excellent!

clang/docs/HLSLSupport.rst

+============
+HLSL Support
+============
+
+.. contents::
+   :local:
+
+Introduction
+============
+
+HLSL Support is under active development in the Clang codebase. This document
+describes the high level goals of the project, the guiding principles, as well
+as some idiosyncrasies of the HLSL language and how we intend to support them in
+Clang.
+
+Project Goals
+=============
+
+The long term goal of this project is to enable Clang to function as a
+replacement for the `DirectXShaderCompiler (DXC)
+<https://github.com/microsoft/DirectXShaderCompiler/>`_ in all its supported
+use cases. Accomplishing that goal will require Clang to be able to process most
+existing HLSL programs with a high degree of source compatibility.
+
+Non-Goals
+---------
+
+HLSL ASTs do not need to be compatible between DXC and Clang. We do not expect
+identical code generation or that features will resemble DXC's implementation or
+architecture. In fact, we explicitly expect to deviate from DXC's implementation
+in key ways.
+
+Guiding Principles
+==================
+
+This document lacks details for architectural decisions that are not yet
+finalized. Our top priorities are quality, maintainability, and flexibility. In
+accordance with community standards we are expecting a high level of test
+coverage, and we will engineer our solutions with long term maintenance in mind.
+We are also working to limit modifications to the Clang C++ code paths and
+share as much functionality as possible.
+
+Architectural Direction
+=======================
+
+HLSL support in Clang is expressed as C++ minus unsupported C and C++ features.
+This is different from how other Clang languages are implemented. Most languages
+in Clang are additive on top of C.
+
+HLSL is not a formally or fully specified language, and while our goals require
+a high level of source compatibility, implementations can vary and we have some
+flexibility to be more or less permissive in some cases. For modern HLSL DXC is
+the reference implementation.
+
+The HLSL effort prioritizes following similar patterns for other languages,
+drivers, runtimes and targets. Specifically, We will maintain separation between
+HSLS-specific code and the rest of Clang as much as possible following patterns
+in use in Clang code today (i.e. ParseHLSL.cpp, SemaHLSL.cpp, CGHLSL*.cpp...).
+We will use inline checks on language options where the code is simple and
+isolated, and prefer HLSL-specific implementation files for any code of
+reasonable complexity.
+
+In places where the HLSL language is in conflict with C and C++, we will seek to
+make minimally invasive changes guarded under the HLSL language options. We will
+seek to make HLSL language support as minimal a maintenance burden as possible.
+
+DXC Driver
+----------
+
+A DXC driver mode will provide command-line compatibility with DXC, supporting
+DXC's options and flags. The DXC driver is HLSL-specific and will create an
+HLSLToolchain which will provide the basis to support targeting both DirectX and
+Vulkan.
+
+Parser
+------
+
+Following the examples of other parser extensions HLSL will add a ParseHLSL.cpp
+file to contain the implementations of HLSL-specific extensions to the Clang
+parser. The HLSL grammar shares most of its structure with C and C++, so we will
+use the existing C/C++ parsing code paths.
+
+Sema
+----
+
+HLSL's Sema implementation will also provide an ``ExternalSemaSource``. In DXC,
+an ``ExternalSemaSource`` is used to provide definitions for HLSL built-in data
+types and built-in templates. Clang is already designed to allow an attached
+``ExternalSemaSource`` to lazily complete data types, which is a **huge**
+performance win for HLSL.
+
+CodeGen
+-------
+
+Like OpenCL, HLSL relies on capturing a lot of information into IR metadata.
+*hand wave* *hand wave* *hand wave* As a design principle here we want our IR to
+be idiomatic Clang IR as much as possible. We will use IR attributes wherever we
+can, and use metadata as sparingly as possible. One example of a difference from
+DXC already implemented in Clang is the use of target triples to communicate
+shader model versions and shader stages.
+
+Our HLSL CodeGen implementation should also have an eye toward generating IR
+that will map directly to targets other than DXIL. While IR itself is generally
+not re-targetable, we want to share the Clang CodeGen implementation for HLSL
+with other GPU graphics targets like SPIR-V and possibly other GPU and even CPU
+targets.
+
+HLSL Language
+=============
+
+The HLSL language is insufficiently documented, and not formally specified.
+Documentation is available on `Microsoft's website
+<https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl>`_.
+The language syntax is similar enough to C and C++ that carefully written C and
+C++ code is valid HLSL. HLSL has some key differences from C & C++ which we will
+need to handle in Clang.
+
+HLSL is not a conforming or valid extension or superset of C or C++. The
+language has key incompatibilities with C and C++, both syntactically and
+semantically.
+
+An Aside on GPU Languages
+-------------------------
+
+Due to HLSL being a GPU targeted language HLSL is a Single Program Multiple Data
+(SPMD) language relying on the implicit parallelism provided by GPU hardware.
+Some language features in HLSL enable programmers to take advantage of the
+parallel nature of GPUs in a hardware abstracted language.
+
+HLSL also prohibits some features of C and C++ which can have catastrophic
+performance or are not widely supportable on GPU hardware or drivers. As an
+example, register spilling is often excessively expensive on GPUs, so HLSL
+requires all functions to be inlined during code generation, and does not
+support a runtime calling convention.
+
+Pointers & References
+---------------------
+
+HLSL does not support referring to values by address. Semantically all variables
+are value-types and behave as such. HLSL disallows the pointer dereference
+operators (unary ``*``, and ``->``), as well as the address of operator (unary
+&). While HLSL disallows pointers and references in the syntax, HLSL does use
+reference types in the AST, and we intend to use pointer decay in the AST in
+the Clang implementation.
+
+HLSL ``this`` Keyword
+---------------------
+
+HLSL does support member functions, and (in HLSL 2021) limited operator
+overloading. With member function support, HLSL also has a ``this`` keyword. The
+``this`` keyword is an example of one of the places where HLSL relies on
+references in the AST, because ``this`` is a reference.
+
+Bitshifts
+---------
+
+In deviation from C, HLSL bitshifts are defined to mask the shift count by the
+size of the type. In DXC, the semantics of LLVM IR were altered to accommodate
+this, in Clang we intend to generate the mask explicitly in the IR. In cases
+where the shift value is constant, this will be constant folded appropriately,
+in other cases we can clean it up in the DXIL target.
+
+Non-short Circuiting Logical Operators
+--------------------------------------
+
+In HLSL 2018 and earlier, HLSL supported logical operators (and the ternary
+operator) on vector types. This behavior required that operators not short
+circuit. The non-short circuiting behavior applies to all data types until HLSL
+2021. In HLSL 2021, logical and ternary operators do not support vector types
+instead builtin functions ``and``, ``or`` and ``select`` are available, and
+operators short circuit matching C behavior.
+
+Precise Qualifier
+-----------------
+
+HLSL has a ``precise`` qualifier that behaves unlike anything else in the C
+language. The support for this qualifier in DXC is buggy, so our bar for
+compatibility is low.
+
+The ``precise`` qualifier applies in the inverse direction from normal
+qualifiers. Rather than signifying that the declaration containing ``precise``
+qualifier be precise, it signifies that the operations contributing to the
+declaration's value be ``precise``. Additionally, ``precise`` is a misnomer:
+values attributed as ``precise`` comply with IEEE-754 floating point semantics,
+and are prevented from optimizations which could decrease *or increase*
+precision.
+
+Differences in Templates
+------------------------
+
+HLSL uses templates to define builtin types and methods, but disallowed
+user-defined templates until HLSL 2021. HLSL also allows omitting empty template
+parameter lists when all template parameters are defaulted. This is an ambiguous
+syntax in C++, but Clang detects the case and issues a diagnostic. This makes
+supporting the case in Clang minimally invasive.
+
+Vector Extensions
+-----------------
+
+HLSL uses the OpenCL vector extensions, and also provides C++-style constructors
+for vectors that are not supported by Clang.
+
+Standard Library
+----------------
+
+HLSL does not support the C or C++ standard libraries. Like OpenCL, HLSL
+describes its own library of built in types, complex data types, and functions.
+
+Unsupported C & C++ Features
+----------------------------
+
+HLSL does not support all features of C and C++. In implementing HLSL in Clang
+use of some C and C++ features will produce diagnostics under HLSL, and others
+will be supported as language extensions. In general, any C or C++ feature that
+can be supported by the DXIL and SPIR-V code generation targets could be treated
+as a clang HLSL extension. Features that cannot be lowered to DXIL or SPIR-V,
+must be diagnosed as errors.
+
+HLSL does not support the following C features:
+
+* Pointers
+* References
+* ``goto`` or labels
+* Variable Length Arrays
+* ``_Complex`` and ``_Imaginary``
+* C Threads or Atomics (or Obj-C blocks)
+* ``union`` types `(in progress for HLSL 202x) <https://github.com/microsoft/DirectXShaderCompiler/pull/4132>`_
+* Most features C11 and later
+
+HLSL does not support the following C++ features:
+
+* RTTI
+* Exceptions
+* Multiple inheritance
+* Access specifiers
+* Anonymous or inline namespaces
+* ``new`` & ``delete`` operators in all of their forms (array, placement, etc)
+* Constructors and destructors
+* Any use of the ``virtual`` keyword
+* Most features C++11 and later

clang/docs/index.rst

.. toctree::
    SafeStack
    ShadowCallStack
    SourceBasedCodeCoverage
    Modules
    MSVCCompatibility
    OpenCLSupport
    OpenMPSupport
    SYCLSupport
+   HLSLSupport
    ThinLTO
    APINotes
    CommandGuide/index
    FAQ
 
 Using Clang as a Library
 ========================