Start support for HLSL `RWBuffer`

[beanz]

Summary

Most of the change here is fleshing out the HLSLExternalSemaSource with
builder implementations to build the builtin types. Eventually, I may
move some of this code into tablegen or a more managable declarative
file but I want to get the AST generation logic ready first.

This code adds two new types into the HLSL AST, hlsl::Resource and
hlsl::RWBuffer. The Resource type is just a wrapper around a handle
identifier, and is largely unused in source. It will morph a bit over
time as I work on getting the source compatability correct, but for now
it is a reasonable stand-in. The RWBuffer type is not ready for use.
I'm posting this change for review because it adds a lot of
infrastructure code and is testable.

There is one change to clang code outside the HLSL-specific logic here,
which addresses a behavior change introduced a long time ago in
967d438439ac. That change resulted in unintentionally breaking
situations where an incomplete template declaration was provided from
an AST source, and needed to be completed later by the external AST.
That situation doesn't happen in the normal AST importer flow, but can
happen when an AST source provides incomplete declarations of
templates. The solution is to annotate template specializations of
incomplete types with the HasExternalLexicalSource bit from the base
template.

Depends on D128012.

[python3kgae]

clang/include/clang/Basic/Builtins.def

1703 ↗

(On Diff #439940)

Is it possible to return a ptr with special address space so we know it is from a resource?

clang/test/AST/HLSL/ResourceStruct.hlsl

2

lib profile starts from shader model 6.3.

[beanz]

clang/include/clang/Basic/Builtins.def

1703 ↗

(On Diff #439940)

Yes, although I'm going to actually hold back the builtins from this patch. I don't think these are quite what we want to generate.

[beanz]

Updating based on PR feedback and removing the new builtins.

[beanz]

Updating with some minor tweaks.

[bogner]

Fairly straightforward. LGTM with a couple of nits

clang/include/clang/Sema/HLSLExternalSemaSource.h

48

This should be "///" for doxygen

clang/test/AST/HLSL/RWBuffer-AST.hlsl

3

Might be worth putting a comment at the top of this test explaining the difference between the two test modes here

[uabelho]

Hello,

Compiling with gcc I get the following warning with this patch:

[1832/2330] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/HLSLExternalSemaSource.cpp.o
In file included from ../../clang/include/clang/Sema/ExternalSemaSource.h:15,
                 from ../../clang/include/clang/Sema/HLSLExternalSemaSource.h:17,
                 from ../../clang/lib/Sema/HLSLExternalSemaSource.cpp:12:
../../clang/include/clang/AST/ExternalASTSource.h:211:16: warning: 'virtual void clang::ExternalASTSource::CompleteType(clang::ObjCInterfaceDecl*)' was hidden [-Woverloaded-virtual]
  211 |   virtual void CompleteType(ObjCInterfaceDecl *Class);
      |                ^~~~~~~~~~~~
In file included from ../../clang/lib/Sema/HLSLExternalSemaSource.cpp:12:
../../clang/include/clang/Sema/HLSLExternalSemaSource.h:49:8: warning:   by 'virtual void clang::HLSLExternalSemaSource::CompleteType(clang::TagDecl*)' [-Woverloaded-virtual]
   49 |   void CompleteType(TagDecl *Tag) override;
      |        ^~~~~~~~~~~~

clang/include/clang/Sema/HLSLExternalSemaSource.h

 //===--- HLSLExternalSemaSource.h - HLSL Sema Source ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file defines the HLSLExternalSemaSource interface.
 //
 //===----------------------------------------------------------------------===//
 #ifndef CLANG_SEMA_HLSLEXTERNALSEMASOURCE_H
 #define CLANG_SEMA_HLSLEXTERNALSEMASOURCE_H
 
+#include "llvm/ADT/DenseMap.h"
+
 #include "clang/Sema/ExternalSemaSource.h"
 
 namespace clang {
 class NamespaceDecl;
 class Sema;
 
 class HLSLExternalSemaSource : public ExternalSemaSource {
   Sema *SemaPtr = nullptr;
   NamespaceDecl *HLSLNamespace;
+  CXXRecordDecl *ResourceDecl;
+
+  using CompletionFunction = std::function<void(CXXRecordDecl *)>;
+  llvm::DenseMap<CXXRecordDecl *, CompletionFunction> Completions;
 
   void defineHLSLVectorAlias();
+  void defineTrivialHLSLTypes();
+  void forwardDeclareHLSLTypes();
+
+  void completeBufferType(CXXRecordDecl *Record);
 
 public:
   ~HLSLExternalSemaSource() override;
 
   /// Initialize the semantic source with the Sema instance
   /// being used to perform semantic analysis on the abstract syntax
   /// tree.
   void InitializeSema(Sema &S) override;
 
   /// Inform the semantic consumer that Sema is no longer available.
   void ForgetSema() override { SemaPtr = nullptr; }
+
+  /// Complete an incomplete HLSL builtin type
+  void CompleteType(TagDecl *Tag) override;
 };
 
 } // namespace clang
 
 #endif // CLANG_SEMA_HLSLEXTERNALSEMASOURCE_H

clang/lib/AST/DeclTemplate.cpp

ClassTemplateSpecializationDecl::Create(ASTContext &Context, TagKind TK,
                                         ArrayRef<TemplateArgument> Args,
                                    ClassTemplateSpecializationDecl *PrevDecl) {
   auto *Result =
       new (Context, DC) ClassTemplateSpecializationDecl(
           Context, ClassTemplateSpecialization, TK, DC, StartLoc, IdLoc,
           SpecializedTemplate, Args, PrevDecl);
   Result->setMayHaveOutOfDateDef(false);
 
+  // If the template decl is incomplete, copy the external lexical storage from
+  // the base template. This allows instantiations of incomplete types to
+  // complete using the external AST if the template's declaration came from an
+  // external AST.
+  if (!SpecializedTemplate->getTemplatedDecl()->isCompleteDefinition())
+    Result->setHasExternalLexicalStorage(
+      SpecializedTemplate->getTemplatedDecl()->hasExternalLexicalStorage());
+
   Context.getTypeDeclType(Result, PrevDecl);
   return Result;
 }
 
 ClassTemplateSpecializationDecl *
 ClassTemplateSpecializationDecl::CreateDeserialized(ASTContext &C,
                                                     unsigned ID) {
   auto *Result =

clang/lib/Sema/HLSLExternalSemaSource.cpp

 //===----------------------------------------------------------------------===//
 
 #include "clang/Sema/HLSLExternalSemaSource.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/DeclCXX.h"
 #include "clang/Basic/AttrKinds.h"
 #include "clang/Sema/Sema.h"
 
+#include <functional>
+
 using namespace clang;
 
+namespace {
+
+struct TemplateParameterListBuilder;
+
+struct BuiltinTypeDeclBuilder {
+  CXXRecordDecl *Record = nullptr;
+  ClassTemplateDecl *Template = nullptr;
+  NamespaceDecl *HLSLNamespace = nullptr;
+
+  BuiltinTypeDeclBuilder(CXXRecordDecl *R) : Record(R) {
+    Record->startDefinition();
+    Template = Record->getDescribedClassTemplate();
+  }
+
+  BuiltinTypeDeclBuilder(Sema &S, NamespaceDecl *Namespace, StringRef Name)
+      : HLSLNamespace(Namespace) {
+    ASTContext &AST = S.getASTContext();
+    IdentifierInfo &II = AST.Idents.get(Name, tok::TokenKind::identifier);
+
+    Record = CXXRecordDecl::Create(AST, TagDecl::TagKind::TTK_Class,
+                                   HLSLNamespace, SourceLocation(),
+                                   SourceLocation(), &II, nullptr, true);
+    Record->setImplicit(true);
+    Record->setLexicalDeclContext(HLSLNamespace);
+    Record->setHasExternalLexicalStorage();
+
+    // Don't let anyone derive from built-in types
+    Record->addAttr(FinalAttr::CreateImplicit(AST, SourceRange(),
+                                              AttributeCommonInfo::AS_Keyword,
+                                              FinalAttr::Keyword_final));
+  }
+
+  ~BuiltinTypeDeclBuilder() {
+    if (HLSLNamespace && !Template)
+      HLSLNamespace->addDecl(Record);
+  }
+
+  BuiltinTypeDeclBuilder &
+  addTemplateArgumentList(llvm::ArrayRef<NamedDecl *> TemplateArgs) {
+    ASTContext &AST = Record->getASTContext();
+
+    auto *ParamList =
+        TemplateParameterList::Create(AST, SourceLocation(), SourceLocation(),
+                                      TemplateArgs, SourceLocation(), nullptr);
+    Template = ClassTemplateDecl::Create(
+        AST, Record->getDeclContext(), SourceLocation(),
+        DeclarationName(Record->getIdentifier()), ParamList, Record);
+    Record->setDescribedClassTemplate(Template);
+    Template->setImplicit(true);
+    Template->setLexicalDeclContext(Record->getDeclContext());
+    Record->getDeclContext()->addDecl(Template);
+
+    // Requesting the class name specialization will fault in required types.
+    QualType T = Template->getInjectedClassNameSpecialization();
+    T = AST.getInjectedClassNameType(Record, T);
+    return *this;
+  }
+
+  BuiltinTypeDeclBuilder &
+  addMemberVariable(StringRef Name, QualType Type,
+                    AccessSpecifier Access = AccessSpecifier::AS_private) {
+    assert(Record->isBeingDefined() &&
+           "Definition must be started before adding members!");
+    ASTContext &AST = Record->getASTContext();
+
+    IdentifierInfo &II = AST.Idents.get(Name, tok::TokenKind::identifier);
+    TypeSourceInfo *MemTySource =
+        AST.getTrivialTypeSourceInfo(Type, SourceLocation());
+    auto *Field = FieldDecl::Create(
+        AST, Record, SourceLocation(), SourceLocation(), &II, Type, MemTySource,
+        nullptr, false, InClassInitStyle::ICIS_NoInit);
+    Field->setAccess(Access);
+    Field->setImplicit(true);
+    Record->addDecl(Field);
+    return *this;
+  }
+
+  BuiltinTypeDeclBuilder &
+  addHandleMember(AccessSpecifier Access = AccessSpecifier::AS_private) {
+    return addMemberVariable("h", Record->getASTContext().VoidPtrTy, Access);
+  }
+
+  BuiltinTypeDeclBuilder &startDefinition() {
+    Record->startDefinition();
+    return *this;
+  }
+
+  BuiltinTypeDeclBuilder &completeDefinition() {
+    assert(Record->isBeingDefined() &&
+           "Definition must be started before completing it.");
+
+    Record->completeDefinition();
+    return *this;
+  }
+
+  TemplateParameterListBuilder addTemplateArgumentList();
+};
+
+struct TemplateParameterListBuilder {
+  BuiltinTypeDeclBuilder &Builder;
+  ASTContext &AST;
+  llvm::SmallVector<NamedDecl *> Params;
+
+  TemplateParameterListBuilder(BuiltinTypeDeclBuilder &RB)
+      : Builder(RB), AST(RB.Record->getASTContext()) {}
+
+  ~TemplateParameterListBuilder() { finalizeTemplateArgs(); }
+
+  TemplateParameterListBuilder &
+  addTypeParameter(StringRef Name, QualType DefaultValue = QualType()) {
+    unsigned Position = static_cast<unsigned>(Params.size());
+    auto *Decl = TemplateTypeParmDecl::Create(
+        AST, Builder.Record->getDeclContext(), SourceLocation(),
+        SourceLocation(), /* TemplateDepth */ 0, Position,
+        &AST.Idents.get(Name, tok::TokenKind::identifier), /* Typename */ false,
+        /* ParameterPack */ false);
+    if (!DefaultValue.isNull())
+      Decl->setDefaultArgument(AST.getTrivialTypeSourceInfo(DefaultValue));
+
+    Params.emplace_back(Decl);
+    return *this;
+  }
+
+  BuiltinTypeDeclBuilder &finalizeTemplateArgs() {
+    if (Params.empty())
+      return Builder;
+    auto *ParamList =
+        TemplateParameterList::Create(AST, SourceLocation(), SourceLocation(),
+                                      Params, SourceLocation(), nullptr);
+    Builder.Template = ClassTemplateDecl::Create(
+        AST, Builder.Record->getDeclContext(), SourceLocation(),
+        DeclarationName(Builder.Record->getIdentifier()), ParamList,
+        Builder.Record);
+    Builder.Record->setDescribedClassTemplate(Builder.Template);
+    Builder.Template->setImplicit(true);
+    Builder.Template->setLexicalDeclContext(Builder.Record->getDeclContext());
+    Builder.Record->getDeclContext()->addDecl(Builder.Template);
+    Params.clear();
+
+    QualType T = Builder.Template->getInjectedClassNameSpecialization();
+    T = AST.getInjectedClassNameType(Builder.Record, T);
+
+    return Builder;
+  }
+};
+
+TemplateParameterListBuilder BuiltinTypeDeclBuilder::addTemplateArgumentList() {
+  return TemplateParameterListBuilder(*this);
+}
+} // namespace
+
 HLSLExternalSemaSource::~HLSLExternalSemaSource() {}
 
 void HLSLExternalSemaSource::InitializeSema(Sema &S) {
   SemaPtr = &S;
   ASTContext &AST = SemaPtr->getASTContext();
   IdentifierInfo &HLSL = AST.Idents.get("hlsl", tok::TokenKind::identifier);
   HLSLNamespace =
       NamespaceDecl::Create(AST, AST.getTranslationUnitDecl(), false,
                             SourceLocation(), SourceLocation(), &HLSL, nullptr);
   HLSLNamespace->setImplicit(true);
   AST.getTranslationUnitDecl()->addDecl(HLSLNamespace);
-  defineHLSLVectorAlias();
+  defineTrivialHLSLTypes();
+  forwardDeclareHLSLTypes();
 
   // This adds a `using namespace hlsl` directive. In DXC, we don't put HLSL's
   // built in types inside a namespace, but we are planning to change that in
   // the near future. In order to be source compatible older versions of HLSL
   // will need to implicitly use the hlsl namespace. For now in clang everything
   // will get added to the namespace, and we can remove the using directive for
   // future language versions to match HLSL's evolution.
   auto *UsingDecl = UsingDirectiveDecl::Create(
auto *Template =
       TypeAliasTemplateDecl::Create(AST, HLSLNamespace, SourceLocation(),
                                     Record->getIdentifier(), ParamList, Record);
 
   Record->setDescribedAliasTemplate(Template);
   Template->setImplicit(true);
   Template->setLexicalDeclContext(Record->getDeclContext());
   HLSLNamespace->addDecl(Template);
 }
+
+void HLSLExternalSemaSource::defineTrivialHLSLTypes() {
+  defineHLSLVectorAlias();
+
+  ResourceDecl = BuiltinTypeDeclBuilder(*SemaPtr, HLSLNamespace, "Resource")
+                     .startDefinition()
+                     .addHandleMember(AccessSpecifier::AS_public)
+                     .completeDefinition()
+                     .Record;
+}
+
+void HLSLExternalSemaSource::forwardDeclareHLSLTypes() {
+  CXXRecordDecl *Decl;
+  Decl = BuiltinTypeDeclBuilder(*SemaPtr, HLSLNamespace, "RWBuffer")
+             .addTemplateArgumentList()
+             .addTypeParameter("element_type", SemaPtr->getASTContext().FloatTy)
+             .finalizeTemplateArgs()
+             .Record;
+  Completions.insert(std::make_pair(
+      Decl, std::bind(&HLSLExternalSemaSource::completeBufferType, this,
+                      std::placeholders::_1)));
+}
+
+void HLSLExternalSemaSource::CompleteType(TagDecl *Tag) {
+  if (!isa<CXXRecordDecl>(Tag))
+    return;
+  auto Record = cast<CXXRecordDecl>(Tag);
+
+  // If this is a specialization, we need to get the underlying templated
+  // declaration and complete that.
+  if (auto TDecl = dyn_cast<ClassTemplateSpecializationDecl>(Record))
+    Record = TDecl->getSpecializedTemplate()->getTemplatedDecl();
+  auto It = Completions.find(Record);
+  if (It == Completions.end())
+    return;
+  It->second(Record);
+}
+
+void HLSLExternalSemaSource::completeBufferType(CXXRecordDecl *Record) {
+  BuiltinTypeDeclBuilder(Record).addHandleMember().completeDefinition();
+}

clang/test/AST/HLSL/RWBuffer-AST.hlsl

+// RUN: %clang_cc1 -triple dxil-pc-shadermodel6.0-library -x hlsl -fsyntax-only -ast-dump -DEMPTY %s | FileCheck -check-prefix=EMPTY %s 
+// RUN: %clang_cc1 -triple dxil-pc-shadermodel6.0-library -x hlsl -fsyntax-only -ast-dump %s | FileCheck %s 
+
+
+// This test tests two different AST generations. The "EMPTY" test mode verifies
+// the AST generated by forward declaration of the HLSL types which happens on
+// initializing the HLSL external AST with an AST Context.
+
+// The non-empty mode has a use that requires the RWBuffer type be complete,
+// which results in the AST being populated by the external AST source. That
+// case covers the full implementation of the template declaration and the
+// instantiated specialization.
+
+// EMPTY: ClassTemplateDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit RWBuffer
+// EMPTY-NEXT: TemplateTypeParmDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> class depth 0 index 0 element_type
+// EMPTY-NEXT: TemplateArgument type 'float'
+// EMPTY-NEXT: BuiltinType 0x{{[0-9A-Fa-f]+}} 'float'
+// EMPTY-NEXT: CXXRecordDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit <undeserialized declarations> class RWBuffer
+// EMPTY-NEXT: FinalAttr 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> Implicit final
+
+// There should be no more occurrances of RWBuffer
+// EMPTY-NOT: RWBuffer
+
+#ifndef EMPTY
+
+RWBuffer<float> Buffer;
+
+#endif
+
+// CHECK: CXXRecordDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit <undeserialized declarations> class Resource definition
+// CHECK: FinalAttr 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> Implicit final
+// CHECK-NEXT: FieldDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit h 'void *'
+
+// CHECK: ClassTemplateDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit RWBuffer
+// CHECK-NEXT: TemplateTypeParmDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> class depth 0 index 0 element_type
+// CHECK-NEXT: TemplateArgument type 'float'
+// CHECK-NEXT: BuiltinType 0x{{[0-9A-Fa-f]+}} 'float'
+// CHECK-NEXT: CXXRecordDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit class RWBuffer definition
+
+// CHECK: FinalAttr 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> Implicit final
+// CHECK-NEXT: FieldDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit h 'void *'
+// CHECK: ClassTemplateSpecializationDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> class RWBuffer definition
+
+// CHECK: TemplateArgument type 'float'
+// CHECK-NEXT: BuiltinType 0x{{[0-9A-Fa-f]+}} 'float'
+// CHECK-NEXT: FinalAttr 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> Implicit final
+// CHECK-NEXT: FieldDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc>  implicit h 'void *'

clang/test/AST/HLSL/ResourceStruct.hlsl

+// RUN: %clang_cc1 -triple dxil-pc-shadermodel6.3-library -x hlsl -fsyntax-only -ast-dump %s | FileCheck %s 
+
+// CHECK: NamespaceDecl {{.*}} implicit hlsl
+// CHECK: CXXRecordDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc> implicit <undeserialized declarations> class Resource definition
+// CHECK-NEXT: DefinitionData
+// CHECK-NEXT: DefaultConstructor exists trivial needs_implicit
+// CHECK-NEXT: CopyConstructor simple trivial has_const_param needs_implicit implicit_has_const_param
+// CHECK-NEXT: MoveConstructor exists simple trivial needs_implicit
+// CHECK-NEXT: CopyAssignment simple trivial has_const_param needs_implicit implicit_has_const_param
+// CHECK-NEXT: MoveAssignment exists simple trivial needs_implicit
+// CHECK-NEXT: Destructor simple irrelevant trivial needs_implicit
+// CHECK-NEXT: FinalAttr 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> Implicit final
+// CHECK-NEXT: FieldDecl 0x{{[0-9A-Fa-f]+}} <<invalid sloc>> <invalid sloc>
+// implicit h 'void *'

clang/test/SemaHLSL/BuiltIns/RWBuffers.hlsl

+// RUN: %clang_cc1 -triple dxil-pc-shadermodel6.0-compute -x hlsl -fsyntax-only -verify %s
+
+Resource ResourceDescriptorHeap[5];
+typedef vector<float, 3> float3;
+
+RWBuffer<float3> Buffer;
+
+[numthreads(1,1,1)]
+void main() {
+  (void)Buffer.h; // expected-error {{'h' is a private member of 'hlsl::RWBuffer<float __attribute__((ext_vector_type(3)))>'}}
+  // expected-note@* {{implicitly declared private here}}
+}