This is an archive of the discontinued LLVM Phabricator instance.

[HLSL] Support cbuffer/tbuffer for hlsl.
ClosedPublic

Authored by python3kgae on Jul 15 2022, 10:47 AM.

Details

Summary

This is first part for support cbuffer/tbuffer.

The format for cbuffer/tbuffer is
BufferType [Name] [: register(b#)] { VariableDeclaration [: packoffset(c#.xyzw)]; ... };

More details at https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-constants

New keyword 'cbuffer' and 'tbuffer' are added.
New AST node HLSLBufferDecl is added.
Build AST for simple cbuffer/tbuffer without attribute support.

The special thing is variables declared inside cbuffer is exposed into global scope.
So isTransparentContext should return true for HLSLBuffer.

Diff Detail

Event Timeline

python3kgae created this revision.Jul 15 2022, 10:47 AM
Herald added a project: Restricted Project. · View Herald TranscriptJul 15 2022, 10:47 AM
Herald added a subscriber: mgorny. · View Herald Transcript
python3kgae requested review of this revision.Jul 15 2022, 10:47 AM
Herald added a project: Restricted Project. · View Herald TranscriptJul 15 2022, 10:47 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
beanz added inline comments.Jul 15 2022, 12:48 PM
clang/include/clang/Parse/Parser.h
2821

nit: maybe ParseHLSLBuffer, I don't think ParseCTBuffer makes it apparent what this is doing, also everything else is HLSLBuffer

clang/test/SemaHLSL/cbuffer_tbuffer.hlsl
1 ↗(On Diff #445057)

This looks like it should be an AST test, not SEMA.

We should also have parser tests for malformed parse cases:

cbuffer { ... };
cbuffer missing_definition;
int cbuffer; // using a keyword as a variable name
cbuffer; // lots wrong here...
cbuffer missing_semicolon { int woo; }

It looks like this patch doesn't handle making the buffer variables constant. Having not looked at that I don't know how big of a change that is, so it might be okay as a subsequent patch, but we'll need that support to handle cases like:

cbuffer cb{
int x;
};

[numthreads(1,1,1)]
void main(int GI : SV_GROUPINDEX) {
  x = GI; // error: buffer content is const
}

Code cleanup and add test for error case.

python3kgae marked 2 inline comments as done.Jul 15 2022, 5:28 PM
python3kgae added inline comments.
clang/test/SemaHLSL/cbuffer_tbuffer.hlsl
1 ↗(On Diff #445057)

Marking constant variables const better be done when supporting global constant buffer. Will add it in another patch.

beanz added a comment.Aug 8 2022, 7:56 AM

Since this is adding a new AST node (hopefully the only one we need in HLSL) it would be nice to get @aaron.ballman to take a look here too to make sure this isn't too insane.

clang/lib/AST/DeclBase.cpp
1197

This could be merged into the return line below, which would make the function easier to understand.

clang/test/SemaHLSL/cb_error.hlsl
13

We should duplicate these cases for tbuffer

python3kgae marked 3 inline comments as done.

Add more comment about no name conflict for cbuffer.

aaron.ballman requested changes to this revision.Aug 18 2022, 11:23 AM

There's a whole ton of test coverage missing from this (no actual sema tests, no AST dumping tests for text or JSON) not to mention a ton of situational tests that are missing. For example, can you put one of these declarations inside of a struct (so it's a member declaration)? How about declaring one as a local variable? Tests that it properly diagnoses use of a template, as in template <typename Ty> cbuffer a { Ty f; };? Are there restrictions on what can be declared within one of these buffers (can I declare an inline function in one?)? All those sorts of situations and more should be tested explicitly.

clang/include/clang/AST/Decl.h
4675

Are these redeclarable declarations? I assume not based on previous discussions, so mostly just double checking.

4690–4693

Speculative question: would it make more sense to add CreateCBuffer and CreateTBuffer static methods rather than use a bool parameter?

4696

Nope. :-D

4699–4700

Unused and unimplemented.

clang/lib/AST/Decl.cpp
5228–5230

So semantically these are all in the global namespace no matter where they're spelled? e.g., if you put one inside of a namespace... it's still globally accessible?

clang/lib/AST/DeclBase.cpp
1197–1198
clang/lib/AST/DeclPrinter.cpp
465–469

Let's make this better while we're touching it.

1660–1664
clang/lib/AST/TextNodeDumper.cpp
2390

You're missing similar changes for the JSONNodeDumper.

clang/lib/Parse/ParseDecl.cpp
1793

So these can form a declaration group? e.g.,

cbuffer a { int b; }, b { float f; };

I don't see any test coverage for this situation if that is supported. If it's not supported, I think this should be a return statement instead.

clang/lib/Parse/ParseHLSL.cpp
48–52

This looks like it's going to have some pretty bad error recovery behavior if something is amiss:

cbuffer a {
  oh no!
  this isn't even valid HLSL code
  despite seeming totally reasonable
  once you understand that HLSL
  is so flaming weird.
};
clang/lib/Sema/SemaHLSL.cpp
28

I don't see any tests for this.

34

(The interface was deprecated recently and this is the replacement API.)

clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
877

Is this actually unreachable?

clang/test/AST/HLSL/cbuffer_tbuffer.hlsl
8

Something's amiss -- this declaration doesn't have a semicolon, is that expected? Because the SemaHLSL tests do have terminating semicolons and no diagnostics about it being superfluous.

clang/test/SemaHLSL/cb_error.hlsl
4–22

None of these are Sema tests, they're all parsing tests. I think the content should be moved to the parsing test directory and sema tests should be added. For example, we should clearly test using the same identifiers for these buffers and comment about why that's valid and not an error. We should also have tests that demonstrate name lookup behavior, e.g., what happens with this:

cbuffer a {
  int x;
};

int main() {
  return sizeof(a);
}
This revision now requires changes to proceed.Aug 18 2022, 11:23 AM
python3kgae marked 10 inline comments as done.

Add more comment about no name conflict for cbuffer.
Add Serialize and JSON dump for HLSLBufferDecl.

python3kgae marked 5 inline comments as done.Aug 25 2022, 11:52 AM
python3kgae added inline comments.
clang/lib/AST/Decl.cpp
5228–5230

It should not.
I was doing this to hoist nested buffer.
I'll drop the nested buffer support and make namespace work first.
Nested buffer support will be done in another review.

clang/lib/Parse/ParseDecl.cpp
1793

No, it cannot form a declaration group.
If early return, still need to convert to Decl group to match the return type of ParseDeclaration.

clang/lib/Parse/ParseHLSL.cpp
48–52

Add more checks for the Tok here.

clang/lib/Sema/SemaHLSL.cpp
28

Removed. Not see any difference without it when -Wdocumentation is on.

clang/lib/Sema/SemaTemplateInstantiateDecl.cpp
877

Yeah. template on cbuffer/tbuffer is not allowed.

clang/test/AST/HLSL/cbuffer_tbuffer.hlsl
8

It is expected that cbuffer doesn't need a semicolon.

Thanks for the update, this is heading in a good direction! I think there's still a lot of test coverage missing. Consider:

namespace Name {
cbuffer a {
  int x;
}
}

struct S {
  cbuffer what {
    int y;
  }
};

void func() {
  tbuffer derp {
    int z;
  }

  decltype(derp) another {
    int a;
  }
}
clang/include/clang/AST/Decl.h
4690–4693

I'm still on the fence about this. On the one hand, being explicit is far better than hoping the reader understand the bool argument at the call site. On the other hand, using the bool parameter means less branching at the call sites. So I'm leaning towards leaving this as-is, but if @beanz has other opinions, I don't yet have a strong feeling.

clang/include/clang/Basic/DiagnosticParseKinds.td
1608

You know what kind of buffer it's within so we can be more precise.

clang/lib/AST/JSONNodeDumper.cpp
904

Matches the naming style for most everything else in the file.

clang/lib/Parse/ParseHLSL.cpp
20–21

Parameter is unused, I presume that's intentional rather than accidental?

25
33
50–54
59

The approach of using a switch and handling individual keywords specially strikes me as being quite fragile. I think a better approach is to split this loop out into its own function and model it (at least somewhat) after ParseStructUnionBody().

The current approach worries me because I have no idea why there's a giant block of invalid things or what should be added to that as we add new declarations to the compiler (and certainly nobody is going to think to come update this function when adding support for new kinds of declarations.

64–65
71–74

There's no way to reach this, so we shouldn't need the [[fallthrough]] here, right?

103–104

Why are type aliases prohibited?

106–107

Why are static assertions prohibited?

122–123
  1. GCC supports HLSL?
  1. This seems wrong because it will result in:
static template <typename Ty> Ty Val{}; // Prohibited
static void func(); // Allowed
inline void other(); // Prohibited
static inline void haha(); // Allowed

As mentioned above, I think we should use general parsing mechanisms here and then look at the AST node generated to see whether it's an allowed one or not.

134

The declarator context looks wrong to me -- I don't see anything prohibiting the user from doing:

namespace N {
cbuffer {
  ...
}
}
134–135

You're parsing things and then dropping them on the floor?

clang/test/ParserHLSL/invalid_inside_cb.hlsl
21–25

These aren't declarations or even really close (maybe if you squint and think about Objective-C?) so the diagnostic here is really weird. I'd expect to be some sort of syntax error because we have no idea what to parse.

python3kgae marked 14 inline comments as done.

Parse first and report error on the generated AST.

python3kgae marked 7 inline comments as done.Sep 7 2022, 3:43 PM
python3kgae added inline comments.
clang/lib/Parse/ParseHLSL.cpp
59

Changed to ParseExternalDeclaration then validate that only Function/Record/Var Declarations are in the result.

103–104

cbuffer is a legacy feature for HLSL while type alias is a new feature for HLSL2021.
The plan is to keep the legacy features as is.

134

namespace N {

cbuffer A {
}

}
is supported in HLSL.

cbuffer A {

namespace N {
}

} is tricky to support because the namespace N decl inside cbuffer needs to be accessed by things outside the cbuffer. This is not supported now and I hope we don't need to support it in the future.

I'll add test for the supported case.

134–135

The goal is to add these things to HLSLBuffer DeclarationContext.
ActOnStartHLSLBuffer should already make sure that.

(I didn't have the chance to do a complete review, but I did a partial one and spotted some things)

clang/lib/Parse/ParseHLSL.cpp
79–80

Just double-checking, but this allows [[]] style attributes as well as [] style attributes, but not __attribute__ or __declspec style attributes, is that intended?

clang/lib/Sema/SemaHLSL.cpp
32–33

cast will assert if given nullptr or the cast is invalid, so this should be equivalent.

python3kgae marked 4 inline comments as done.

Use cast to simplify code.

clang/lib/Parse/ParseHLSL.cpp
79–80

That is what dxc currently support.
It may change in the future. But for now, only [[]] and [] are supported.

beanz added inline comments.Sep 8 2022, 11:29 AM
clang/lib/Parse/ParseHLSL.cpp
79–80

Well... worth noting, HLSL doesn't actually support C++11 attributes, but that is almost certainly going to change in the near future, so we might as well support them from the start in Clang.

aaron.ballman added inline comments.Sep 9 2022, 5:04 AM
clang/lib/Parse/ParseHLSL.cpp
79–80

Ah, that is good to know @beanz -- we should think carefully about this situation because there are some tradeoffs to consider.

  1. It's pretty weird to support half of the Microsoft attribute syntax (and the half we barely have any attribute support for, at that). Is there a reason to not support __declspec as well? (For example, are there concerns about things like using those attributes to do dllexport or specify a COMDAT section, etc?) In fact, if we're going to support vendor attributes like [[clang::overloadable]], it's a bit weird that we then prohibit the same attribute from being spelled __attribute__((overloadable)), is there a particular reason to not extend to all attributes?
  2. Supporting [] style attributes means that attribute order is important. We cannot use MaybeParseAttributes() to parse attribute specifiers in any order because [] causes ambiguities under some circumstances. So you're stuck with the order you have -- [[]] attributes first, [] attributes second. Is that ordering reasonable?

And for the patch itself -- there are no test cases demonstrating what happens when using attributes on things within one of these buffers. I expect many things to be quite reasonable, like using [[deprecated]], but are the attributes which impact codegen reasonable as well? (Like naked functions, returns twice, disable tail calls, etc)

beanz added inline comments.Sep 9 2022, 7:27 AM
clang/lib/Parse/ParseHLSL.cpp
79–80

@aaron.ballman I think those are all good questions. Generally HLSL has used Microsoft attribute syntax, and I've started extending the Clang support to be more robust, but still have more work to do.

More on this patch, I want to take a step back.

I think @python3kgae copied this code from DXC, but I don't think it is ever used. I don't think we have any attributes in the language that are valid with cbuffer or tbuffer subjects. We certainly don't have any attributes implemented in clang that would be valid on these targets.

That makes me think we should remove since it should be dead and unreachable and untestable code.

Since these HLSL buffer decls are an older (although frequently used) HLSL feature, I think our general preference is to not extend new feature support to them, and instead to encourage users to use the newer buffer types.

Does that sound reasonable?

aaron.ballman added inline comments.Sep 9 2022, 7:49 AM
clang/lib/Parse/ParseHLSL.cpp
79–80

We certainly don't have any attributes implemented in clang that would be valid on these targets.

Despite knowing nothing about HLSL, I feel like pushing back a little bit here: deprecated, nodiscard, maybe_unused, and many others seem like they'd not only be valid on the target but perhaps useful to users.

Does that sound reasonable?

I'm totally fine with that approach; we can debate attributes later. :-)

beanz added inline comments.Sep 9 2022, 8:13 AM
clang/lib/Parse/ParseHLSL.cpp
79–80

Despite knowing nothing about HLSL, I feel like pushing back a little bit here: deprecated, nodiscard, maybe_unused, and many others seem like they'd not only be valid on the target but perhaps useful to users.

Okay... you got me here. I hadn't considered deprecated but can see a use for it. I don't think the other two apply, but I'll concede there may be more general clang attributes that do have uses.

If we can postpone this discussion though I think we can do some background and get a better feeling for what attributes we should and shouldn't support, and maybe consider the syntax a bit carefully too.

If I'm reading this correctly the DXC-supported syntax is:

cbuffer A { ... } [some_attribute]

(note: DXC doesn't really support CXX11 attributes, just the MS syntax)

If this syntax is really unreachable in DXC (which I believe it is), it might be better to shift the syntax to be more like C++ class and struct attributes:

[[some_attribute]]
cbuffer A {...}

I think that would be more familiar and understandable to users, especially as buffer declarations are sometimes hundreds of lines long.

Remove attribute for first cbuffer commit.

python3kgae added inline comments.Sep 9 2022, 9:02 AM
clang/lib/Parse/ParseHLSL.cpp
79–80

Removed attribute parsing.
Will add it back when real HLSL attributes like packoffset are supported.

This revision is now accepted and ready to land.Sep 9 2022, 10:49 AM

Rebase for the PCH fix.

This revision was landed with ongoing or failed builds.Sep 21 2022, 10:08 AM
This revision was automatically updated to reflect the committed changes.
kazu added a subscriber: kazu.Sep 21 2022, 10:31 AM

This patch seems to cause several warnings:

clang/include/clang/AST/Decl.h:4696:15: error: 'getSourceRange' overrides a member function but is not marked 'override' [-Werror,-Wsuggest-override]
clang/lib/Parse/ParseHLSL.cpp:76:20: error: unused variable 'Loc' [-Werror,-Wunused-variable]
clang/lib/CodeGen/CGDecl.cpp:48:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

I could fix the first two, but I don't know what should be done for the last one. Would you mind taking a look?

FWIW, I'm using clang-14.0.0.6 as the hot compiler with a cmake flag -DLLVM_ENABLE_WERROR=On.

This patch seems to cause several warnings:

clang/include/clang/AST/Decl.h:4696:15: error: 'getSourceRange' overrides a member function but is not marked 'override' [-Werror,-Wsuggest-override]
clang/lib/Parse/ParseHLSL.cpp:76:20: error: unused variable 'Loc' [-Werror,-Wunused-variable]
clang/lib/CodeGen/CGDecl.cpp:48:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

I could fix the first two, but I don't know what should be done for the last one. Would you mind taking a look?

FWIW, I'm using clang-14.0.0.6 as the hot compiler with a cmake flag -DLLVM_ENABLE_WERROR=On.

Working on it.
Building with clang now.

This patch seems to cause several warnings:

clang/include/clang/AST/Decl.h:4696:15: error: 'getSourceRange' overrides a member function but is not marked 'override' [-Werror,-Wsuggest-override]
clang/lib/Parse/ParseHLSL.cpp:76:20: error: unused variable 'Loc' [-Werror,-Wunused-variable]
clang/lib/CodeGen/CGDecl.cpp:48:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

I could fix the first two, but I don't know what should be done for the last one. Would you mind taking a look?

FWIW, I'm using clang-14.0.0.6 as the hot compiler with a cmake flag -DLLVM_ENABLE_WERROR=On.

Fixed with https://github.com/llvm/llvm-project/commit/a7e3de2450f5b62e7dfe8999443f15be5dfec0b0
Thanks for the help.
I will switch to clang in the future.

kazu added a comment.Sep 21 2022, 11:27 AM

This patch seems to cause several warnings:

clang/include/clang/AST/Decl.h:4696:15: error: 'getSourceRange' overrides a member function but is not marked 'override' [-Werror,-Wsuggest-override]
clang/lib/Parse/ParseHLSL.cpp:76:20: error: unused variable 'Loc' [-Werror,-Wunused-variable]
clang/lib/CodeGen/CGDecl.cpp:48:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

I could fix the first two, but I don't know what should be done for the last one. Would you mind taking a look?

FWIW, I'm using clang-14.0.0.6 as the hot compiler with a cmake flag -DLLVM_ENABLE_WERROR=On.

Fixed with https://github.com/llvm/llvm-project/commit/a7e3de2450f5b62e7dfe8999443f15be5dfec0b0
Thanks for the help.
I will switch to clang in the future.

Thank you so much for fixing this quickly!

By the way, after your fix, I am getting:

clang/tools/libclang/CIndex.cpp:6648:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

Sorry for not listing all the warnings in my first message. (I thought I listed them all...) Should we add something like this?

case Decl::HLSLBuffer:
  return clang_getNullCursor();

This patch seems to cause several warnings:

clang/include/clang/AST/Decl.h:4696:15: error: 'getSourceRange' overrides a member function but is not marked 'override' [-Werror,-Wsuggest-override]
clang/lib/Parse/ParseHLSL.cpp:76:20: error: unused variable 'Loc' [-Werror,-Wunused-variable]
clang/lib/CodeGen/CGDecl.cpp:48:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

I could fix the first two, but I don't know what should be done for the last one. Would you mind taking a look?

FWIW, I'm using clang-14.0.0.6 as the hot compiler with a cmake flag -DLLVM_ENABLE_WERROR=On.

Fixed with https://github.com/llvm/llvm-project/commit/a7e3de2450f5b62e7dfe8999443f15be5dfec0b0
Thanks for the help.
I will switch to clang in the future.

Thank you so much for fixing this quickly!

By the way, after your fix, I am getting:

clang/tools/libclang/CIndex.cpp:6648:11: error: enumeration value 'HLSLBuffer' not handled in switch [-Werror,-Wswitch]

Sorry for not listing all the warnings in my first message. (I thought I listed them all...) Should we add something like this?

case Decl::HLSLBuffer:
  return clang_getNullCursor();

Yeah. return clang_getNullCursor(); looks fine.
I'm running a full build now. Hope it can catch all the issue.