Page MenuHomePhabricator

[flang][driver] Refactor boolean options
ClosedPublic

Authored by awarzynski on Jul 13 2021, 2:48 AM.

Details

Summary

For boolean options, e.g. -fxor-operator/-fno-xor-operator, we ought
to be using TableGen multi-classes. This way, we only have to write one
definition to have both forms auto-generated. This patch refactors all of
Flang's boolean options to use two new multi-classes: OptInFC1FFOption
and OptOutFC1FFOption.

With the new approach, "empty" help text is now replaced with an empty
string. When running flang-new --help, that's considered as non-empty
help messages, which is then printed. This means that with this patch,
flang-new --help will start printing e.g. -fno-backslash, even
though there is no actual help text to print for this option (apart from
the emtpy string "").

Diff Detail

Event Timeline

awarzynski created this revision.Jul 13 2021, 2:48 AM
awarzynski requested review of this revision.Jul 13 2021, 2:48 AM
Herald added a project: Restricted Project. · View Herald TranscriptJul 13 2021, 2:48 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
awarzynski added inline comments.Jul 13 2021, 2:52 AM
clang/include/clang/Driver/Options.td
246

@jansvoboda11 , is this the right approach here? I'd like use BoolFOption in Flang, but we don't need KeyPathAndMacro just yet. We may do in the future.

jansvoboda11 added inline comments.Jul 13 2021, 4:37 AM
clang/include/clang/Driver/Options.td
246

There was an EmptyKPM class present in the code before, but only as an implementation detail of the marshalling system. People started using it with BoolFOption to conveniently declare pairs Clang driver flags. I'm not a fan since the whole BooFOption is designed around the keypath and its marshalling. Putting a "null" keypath in BoolFOption and then talking about its defaults and the effect of each flag seems highly unintuitive to me and it's essentially dead code. I'd prefer a different approach in this patch unless you have plans to switch to the marshalling infrastructure and remove EmptyKPM in the short term.

Would something similar to OptInFFlag and OptOutFFlag work for you? You could generalize it (remove Flags<[CC1Option]>), make use of it for Flang's options directly and forward the current Opt{In,Out}FFlag to the generalization while specifying the CC1Option. WDYT?

awarzynski added inline comments.Jul 13 2021, 5:57 AM
clang/include/clang/Driver/Options.td
246

Thanks for the quick feedback! Initially when I saw CC1Option in CC1Option/OptOutFFlag, I immediately decided to explore alternatives. But I agree with everything that you said and am happy to generalize these multiclasses (instead of adding EmptyKPM). I don't see why it wouldn't work for what we currently need in Flang. I should be able to update this patch tomorrow.

Btw, what's the key benefit of the marshaling infra? Is that for being able to restore the original compiler invocation from instances of CompilerInvocation? We are still busy with other things, but definitely don't want Flang to miss out here.

PeteSteinfeld requested changes to this revision.Jul 13 2021, 7:02 AM

This built OK, but when I run check-flang, I see errors for .../Driver/driver-help.f90 and .../Driver/driver-help-hidden.f90. I'm building on a Linux system using GNU g++ version 9.3. Here's an excerpt from the log file:

FAIL: Flang :: Driver/driver-help.f90 (4 of 712)
******************** TEST 'Flang :: Driver/driver-help.f90' FAILED ********************
Script:
--
: 'RUN: at line 6';   /local/home/psteinfeld/main/andy/flang/build/bin/flang-new -help 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help.f90 --check-prefix=HELP
: 'RUN: at line 7';   not /local/home/psteinfeld/main/andy/flang/build/bin/flang-new -helps 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help.f90 --check-prefix=ERROR
: 'RUN: at line 12';   /local/home/psteinfeld/main/andy/flang/build/bin/flang-new -fc1 -help 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help.f90 --check-prefix=HELP-FC1
: 'RUN: at line 13';   not /local/home/psteinfeld/main/andy/flang/build/bin/flang-new -fc1 -helps 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help.f90 --check-prefix=ERROR
--
Exit Code: 1

Command Output (stderr):
--
/local/home/psteinfeld/main/andy/flang/test/Driver/driver-help.f90:43:14: error: HELP-NEXT: expected string not found in input
! HELP-NEXT: -fno-backslash
             ^
<stdin>:27:54: note: scanning from here
 -flogical-abbreviations Enable logical abbreviations
                                                     ^
<stdin>:28:2: note: possible intended match here
 -fno-color-diagnostics Disable colors in diagnostics
 ^

Input file: <stdin>
Check file: /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help.f90

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           .
           .
           .
          22:  -fimplicit-none No implicit typing allowed unless overridden by IMPLICIT statements 
          23:  -finput-charset=<value> Specify the default character set for source files 
          24:  -fintrinsic-modules-path <dir> 
          25:  Specify where to find the compiled intrinsic modules 
          26:  -flarge-sizes Use INTEGER(KIND=8) for the result type in size-related intrinsics 
          27:  -flogical-abbreviations Enable logical abbreviations 
next:43'0                                                          X error: no match found
          28:  -fno-color-diagnostics Disable colors in diagnostics 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
next:43'1      ?                                                     possible intended match
          29:  -fopenacc Enable OpenACC 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
          30:  -fopenmp Parse OpenMP pragmas and generate parallel code. 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          31:  -fxor-operator Enable .XOR. as a synonym of .NEQV. 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          32:  -help Display available options 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          33:  -I <dir> Add directory to the end of the list of include search paths 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           .
           .
           .
>>>>>>

--

********************
FAIL: Flang :: Driver/driver-help-hidden.f90 (5 of 712)
******************** TEST 'Flang :: Driver/driver-help-hidden.f90' FAILED ********************
Script:
--
: 'RUN: at line 6';   /local/home/psteinfeld/main/andy/flang/build/bin/flang-new --help-hidden 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help-hidden.f90
: 'RUN: at line 7';   not /local/home/psteinfeld/main/andy/flang/build/bin/flang-new  -help-hidden 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help-hidden.f90 --check-prefix=ERROR-FLANG
: 'RUN: at line 12';   not /local/home/psteinfeld/main/andy/flang/build/bin/flang-new -fc1 --help-hidden 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help-hidden.f90 --check-prefix=ERROR-FLANG-FC1
: 'RUN: at line 13';   not /local/home/psteinfeld/main/andy/flang/build/bin/flang-new -fc1  -help-hidden 2>&1 | /local/home/psteinfeld/main/clean/install/bin/FileCheck /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help-hidden.f90 --check-prefix=ERROR-FLANG-FC1
--
Exit Code: 1

Command Output (stderr):
--
/local/home/psteinfeld/main/andy/flang/test/Driver/driver-help-hidden.f90:43:15: error: CHECK-NEXT: expected string not found in input
! CHECK-NEXT: -fno-backslash
              ^
<stdin>:27:54: note: scanning from here
 -flogical-abbreviations Enable logical abbreviations
                                                     ^
<stdin>:28:2: note: possible intended match here
 -fno-color-diagnostics Disable colors in diagnostics
 ^

Input file: <stdin>
Check file: /local/home/psteinfeld/main/andy/flang/test/Driver/driver-help-hidden.f90

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           .
           .
           .
          22:  -fimplicit-none No implicit typing allowed unless overridden by IMPLICIT statements 
          23:  -finput-charset=<value> Specify the default character set for source files 
          24:  -fintrinsic-modules-path <dir> 
          25:  Specify where to find the compiled intrinsic modules 
          26:  -flarge-sizes Use INTEGER(KIND=8) for the result type in size-related intrinsics 
          27:  -flogical-abbreviations Enable logical abbreviations 
next:43'0                                                          X error: no match found
          28:  -fno-color-diagnostics Disable colors in diagnostics 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
next:43'1      ?                                                     possible intended match
          29:  -fopenacc Enable OpenACC 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~
          30:  -fopenmp Parse OpenMP pragmas and generate parallel code. 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          31:  -fxor-operator Enable .XOR. as a synonym of .NEQV. 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          32:  -help Display available options 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          33:  -I <dir> Add directory to the end of the list of include search paths 
next:43'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           .
           .
           .
>>>>>>

--

********************
This revision now requires changes to proceed.Jul 13 2021, 7:02 AM
jansvoboda11 added inline comments.Jul 13 2021, 8:24 AM
clang/include/clang/Driver/Options.td
246

Cool, happy to help.

We're using marshalling for "implicitly discovered, explicitly built Clang modules" via clang-scan-deps. We're able to change the CompilerInvocation of the original TU for the purpose of explicit build of the discovered modules, serialize it to clang -cc1 command-line and report it to the build system.

Apologies, it has taken me a bit longer to get back to this. @jansvoboda11 , now I'm realising the key disadvantage of using OptInFFlag/OptOutFFlag - it's impossible to express the opt-in/opt-out semantics in TableGen. In fact, only the contents of clang -cc1 --help are being tweaked by using OptInFFlag/OptOutFFlag. Basically:

  • for OptInFFlang, only -ffoo is printed with -clang -cc1 --help (the help text for -ffno-foo becomes irrelevant)
  • for OptOutFFlang, only -fno-foo is printed with -clang -cc1 --help (the help text for -ffoo is irrelevant)

IIUC, this is achieved through flags (CC1Option vs []). In Flang, we rely on the fact that "no help text? don't include it in flang-new -fc1 --help" instead. This behavior is implemented in clangDriver and applies to all drivers that use it (so this is nothing specific to Flang). This way, we can always mark Flang options with the relevant flags (e.g. FC1Option). In other words, we can claim them and make it clear that other drivers can ignore them. In general, I think that Options.td would become a bit cleaner and easier to reason about if all options specified all their flags. That wouldn't be easy to achieve though!

I'm just about send an updated patch. It takes into account the points that I raised above ^^^. It also fixes the failing tests reported by @PeteSteinfeld. Does this new approach work for Clang (I quickly tested clang -cc1 --help and see no difference)?

It would be nicer to have the opt-in/opt-out semantics for Flang in TableGen through something similar to BoolFOption. No need for this just now - one step at a time :)

Switch from BoolFOption to OptInFFlag/OptOutFFlag

I've refactored OptInFFlag/OptOutFFlag a tiny bit and created specialisations for clang -cc1 (OptInCC1FFlag/OptOutCC1FFlag). I have added some comments in Options.td to clarify the semantics of OptInFFlag/OptOutFFlag. Does this make sense to you?

PeteSteinfeld accepted this revision.Jul 20 2021, 7:57 AM

All builds, tests, and looks good.

This revision is now accepted and ready to land.Jul 20 2021, 7:57 AM
jansvoboda11 added a comment.EditedJul 26 2021, 9:50 AM

Sorry, I'm not sure I follow.

Apologies, it has taken me a bit longer to get back to this. @jansvoboda11 , now I'm realising the key disadvantage of using OptInFFlag/OptOutFFlag - it's impossible to express the opt-in/opt-out semantics in TableGen.

What do you mean exactly? I think the semantics are expressed pretty well from Clang's point of view:

  • OptInFFlag -> users can opt-in into the behavior by specifying the positive flag in clang -cc1 (e.g. -fmodules),
  • OptOutFFlag -> users can opt-out of the behavior by specifying the negative flag in clang -cc1 (e.g. -fno-rtti).

In fact, only the contents of clang -cc1 --help are being tweaked by using OptInFFlag/OptOutFFlag.

That's incorrect, OptInFFlag and OptOutFFlag both set HelpText for the positive and negative driver flags.

Basically:

  • for OptInFFlang, only -ffoo is printed with -clang -cc1 --help (the help text for -ffno-foo becomes irrelevant)
  • for OptOutFFlang, only -fno-foo is printed with -clang -cc1 --help (the help text for -ffoo is irrelevant)

IIUC, this is achieved through flags (CC1Option vs []).

Correct.

In Flang, we rely on the fact that "no help text? don't include it in flang-new -fc1 --help" instead. This behavior is implemented in clangDriver and applies to all drivers that use it (so this is nothing specific to Flang).

I see. Is this intentional?

This way, we can always mark Flang options with the relevant flags (e.g. FC1Option). In other words, we can claim them and make it clear that other drivers can ignore them.

I don't understand this. Can you elaborate? Are you relying on the absence/presence of a help text to mark flags with FC1Option? When does this happen? (In Options.td? The TableGen backend? The consumer of Options.inc?)

In general, I think that Options.td would become a bit cleaner and easier to reason about if all options specified all their flags. That wouldn't be easy to achieve though!

I think that the way we define flags for the Clang driver and -cc1 in one go is already bit confusing. But it's convenient, so we roll with it. On top of that, we have the marshalling infrastructure, that explicitly only applies to the -cc1 side of things. If we introduce EmptyKPM to the mix and say "Marshalling only applies to -cc1, but sometimes not at all", things stop making sense in my opinion. People will copy-paste it and be confused things don't work.

I'm just about send an updated patch. It takes into account the points that I raised above ^^^. It also fixes the failing tests reported by @PeteSteinfeld. Does this new approach work for Clang (I quickly tested clang -cc1 --help and see no difference)?

I'd need to double-check if we particularly care about showing/hiding flags without help text.

It would be nicer to have the opt-in/opt-out semantics for Flang in TableGen through something similar to BoolFOption. No need for this just now - one step at a time :)

I suggest to find another means of achieving the semantics you want. You can always copy the BoolFOption implementation and remove the marshalling stuff, if that's really the semantics you want.

jansvoboda11 requested changes to this revision.Jul 26 2021, 9:50 AM
This revision now requires changes to proceed.Jul 26 2021, 9:50 AM

Thank you for your detailed reply!

Sorry, I'm not sure I follow.

OK, let me try to clarify. I've tried to split this into threads.

(flag == OptionFlag )

SEMANTICS

@jansvoboda11 , now I'm realising the key disadvantage of using OptInFFlag/OptOutFFlag - it's impossible to express the opt-in/opt-out semantics in TableGen.

What do you mean exactly?

BoolOption (from which BoolFOption derives) has e.g. the default value and a bunch of asserts to define the relation between -ffoo and -fno-foo. So effectively, the opt-in/opt-out semantics are enforced here. OptInFFlag/OptOutFFlag have none of this. It's just a syntactic sugar to automate the generation of -ffoo and -fno-foo. That's fine, we can add such checks inside the driver.

CC1Option

I think the semantics are expressed pretty well from Clang's point of view:

OK, that's fair enough. But what about the output from clang --help and clang -cc1 --help? Let's look at -feliminate-unused-debug-types (the only instance of OptOutFFLag):

# Only -fno-eliminate-unusued-debug-types is printed
clang -cc1 --help | grep eliminate-unused
  -fno-eliminate-unused-debug-types
# Both -fno-eliminate-unusued-debug-types and -feliminate-unusued-debug-types printed?
clang --help | grep eliminate-unused
  -feliminate-unused-debug-types
  -fno-eliminate-unused-debug-types

To me this behavior is counter-intuitive. So are these clang options? clang -cc1 options? Both? What's the point of adding CC1Option if both clang and clang -cc1 accept them. The definition of CC1Option from Options.td:

// CC1Option - This option should be accepted by clang -cc1.
def CC1Option : OptionFlag;

OK, so currently -fno-eliminate-unused-debug-types is a CC1Option and _should_ be accepted by clang -cc1. How about -feliminate-unused-debug-types?

clang -cc1 -feliminate-unused-debug-types file.c
error: unknown argument: '-feliminate-unused-debug-types'

Makes sense, but:

clang -c -feliminate-unused-debug-types file.c
clang-13: warning: argument unused during compilation: '-feliminate-unused-debug-types' [-Wunused-command-line-argument]

Shouldn't it be rejected by clang with an error as well? Shouldn't CC1Option be used consistently for -feliminate-unused-debug-types and -fno-eliminate-unused-debug-types?

In fact, only the contents of clang -cc1 --help are being tweaked by using OptInFFlag/OptOutFFlag.

That's incorrect, OptInFFlag and OptOutFFlag both set HelpText for the positive and negative driver flags.

Yes, but in clang --cc1 --help, only the variant with CC1Option is being printed and the other one is ignored. So why bother adding the other one?

Btw, now I see that -cc1 does reject -feliminate-unused-debug-types, so it's not only about the contents of clang -cc1 --help. I was wrong, sorry about that and thanks for pointing this out!

Content of --help

In Flang, we rely on the fact that "no help text? don't include it in flang-new -fc1 --help" instead. This behavior is implemented in clangDriver and applies to all drivers that use it (so this is nothing specific to Flang).

I see. Is this intentional?

This way, we can always mark Flang options with the relevant flags (e.g. FC1Option). In other words, we can claim them and make it clear that other drivers can ignore them.

I don't understand this. Can you elaborate? Are you relying on the absence/presence of a help text to mark flags with FC1Option? When does this happen? (In Options.td? The TableGen backend? The consumer of Options.inc?)

See the implementation of printHelp from OptTable.cpp. Basically: "No help text? Not printing". We rely on this not to print options supported by Flang, but for which we have no help text. We only used that for boolean options (so we have a help text for e.g. -ffoo, but not for -fno-foo). This is a minor thing and I am not concerned about loosing it.

As for FC1Option (and FlangOption), we always add these explicitly in Options.td. This helps to identify options that are either shared or Flang-only and to avoid polluting clang --help with Flang/Fortran specific options. And that's what we rely on when calling printHelp.

I'd need to double-check if we particularly care about showing/hiding flags without help text.

We should. clang --help should not be printing any Flang options, and vice-versa. I'm under the impression that the contents of clang --help and clang --cc1 --help got a bit out of control. We did discuss this briefly on cfe-dev. I guess that with ~1000 options supported by Clang, this is rarely noticed. Does anyone use clang --help these days? But in Flang, we have only ~50 options and if we are not careful, flang --help would be printing hundreds of options that are not supported (and in fact, will never be).

We try to be strict about what is and what is not included in flang --help as that's currently our main end-user-facing documentation. We do this by adding a Flang flag (e.g. FlangOption or FC1Option) to every option that we support. Options that don't contain flags (e.g. feliminate-unused-debug-types) are very problematic for us. What is Flang meant to do with such options? This is an issue for every tool that relies on Options.td. This also works the other way - if we don't flag Flang options, clang --help and clang -cc1 --help will start printing options added by us. Same mechanism applies to auto-completion for command line options or for finding suggestions when an option is misspelled.

Splitting Options.td

In general, I think that Options.td would become a bit cleaner and easier to reason about if all options specified all their flags. That wouldn't be easy to achieve though!

I think that the way we define flags for the Clang driver and -cc1 in one go is already bit confusing. But it's convenient, so we roll with it.

I find it very hard to reason about things in Options.td. IMHO, we should have a separate Options.td for generic driver options (shared between Clang and Flang), and then separate Options.td for cc1 and fc1. This was brought up when we discussed implementing the Flang driver in terms of clangDriver. It would help in forcing every driver to clearly define what options it supports. Also, the long term goal is to move clangDriver out of Clang, at which point that's going to be required anyway.

IMO, OptInFFlang/OptOutFFlag are generic enough and should be shared.

Next steps

It would be nicer to have the opt-in/opt-out semantics for Flang in TableGen through something similar to BoolFOption. No need for this just now - one step at a time :)

I suggest to find another means of achieving the semantics you want. You can always copy the BoolFOption implementation and remove the marshalling stuff, if that's really the semantics you want.

The updated OptInFFlang/OptOutFFlag work just fine for us. Flang does not need EmptyKPM and BoolFOption just now. I'm not suggesting any changes here.

As there's only a handful of options using OptInFFlang/OptOutFFlag right now, I was hoping that the current revision would be non-controversial. We definitely need to use FC1Option (or other Flang flag) for both -ffoo and -fno-foo. Otherwise, clang --help will start printing/supporting the version without a flag. If adding a flag to both variants is not acceptable in -cc1, then I suggest splitting OptInFFlag/OptOutFFlag into

  • OptInCC1FFlang/OptOutCC1FFlag, and
  • OptInFC1FFlang/OptOutFC1FFlag.

As for the help text, what would be best for Clang and -cc1? Add it to both -ffoo and -fno-foo? Just one of them?

Thanks for reviewing this and all your comments! I hope that we can figure something out that will work for both Clang and Flang.

OK, that's fair enough. But what about the output from clang --help and clang -cc1 --help? Let's look at -feliminate-unused-debug-types (the only instance of OptOutFFLag):

# Only -fno-eliminate-unusued-debug-types is printed
clang -cc1 --help | grep eliminate-unused
  -fno-eliminate-unused-debug-types
# Both -fno-eliminate-unusued-debug-types and -feliminate-unusued-debug-types printed?
clang --help | grep eliminate-unused
  -feliminate-unused-debug-types
  -fno-eliminate-unused-debug-types

To me this behavior is counter-intuitive. So are these clang options? clang -cc1 options? Both? What's the point of adding CC1Option if both clang and clang -cc1 accept them. The definition of CC1Option from Options.td:

// CC1Option - This option should be accepted by clang -cc1.
def CC1Option : OptionFlag;

OK, so currently -fno-eliminate-unused-debug-types is a CC1Option and _should_ be accepted by clang -cc1. How about -feliminate-unused-debug-types?

clang -cc1 -feliminate-unused-debug-types file.c
error: unknown argument: '-feliminate-unused-debug-types'

Makes sense, but:

clang -c -feliminate-unused-debug-types file.c
clang-13: warning: argument unused during compilation: '-feliminate-unused-debug-types' [-Wunused-command-line-argument]

Shouldn't it be rejected by clang with an error as well? Shouldn't CC1Option be used consistently for -feliminate-unused-debug-types and -fno-eliminate-unused-debug-types?

In fact, only the contents of clang -cc1 --help are being tweaked by using OptInFFlag/OptOutFFlag.

That's incorrect, OptInFFlag and OptOutFFlag both set HelpText for the positive and negative driver flags.

Yes, but in clang --cc1 --help, only the variant with CC1Option is being printed and the other one is ignored. So why bother adding the other one?

Btw, now I see that -cc1 does reject -feliminate-unused-debug-types, so it's not only about the contents of clang -cc1 --help. I was wrong, sorry about that and thanks for pointing this out!

The clang driver always accepts both the positive and negative options, and it always only considers the argument that appeared last on the command-line (using ArgList::getLastArg(Pos, Neg)). This allows users to take a random clang driver command-line, append -feliminate-unused-debug-types and get the desired behavior, even though the original command-line might have contained -fno-eliminate-unused-debug-types.

For the pair of options you mentioned, the clang driver knows they are "opt-out" in clang -cc1. The behavior is enabled by default in -cc1 and can be disabled/opted-out through the negative -fno-eliminate-unused-debug-types. The positive option is not accepted by clang -cc1, since explicitly enabling a behavior that's already enabled by default is redundant.

If driver sees -fno-eliminate-unused-debug-types as the last option, it will forward it to clang -cc1. If it sees -feliminate-unused-debug-types, it will ignore it.

Does this make sense?

If neither the positive nor negative options have CC1Option, they are driver-only options. If you add CC1Option to both, both are accepted by clang -cc1. But the most common scenario is that only one of the two options is marked with CC1Option.

The HelpText doesn't play a role in option being accepted or not in clang or clang -cc1.

Content of --help

In Flang, we rely on the fact that "no help text? don't include it in flang-new -fc1 --help" instead. This behavior is implemented in clangDriver and applies to all drivers that use it (so this is nothing specific to Flang).

I see. Is this intentional?

This way, we can always mark Flang options with the relevant flags (e.g. FC1Option). In other words, we can claim them and make it clear that other drivers can ignore them.

I don't understand this. Can you elaborate? Are you relying on the absence/presence of a help text to mark flags with FC1Option? When does this happen? (In Options.td? The TableGen backend? The consumer of Options.inc?)

See the implementation of printHelp from OptTable.cpp. Basically: "No help text? Not printing". We rely on this not to print options supported by Flang, but for which we have no help text. We only used that for boolean options (so we have a help text for e.g. -ffoo, but not for -fno-foo). This is a minor thing and I am not concerned about loosing it.

Okay, this makes sense, thanks! I think we're in agreement that the current distinction between HelpText<""> and no HelpText is not particularly useful when printing help.

As for FC1Option (and FlangOption), we always add these explicitly in Options.td. This helps to identify options that are either shared or Flang-only and to avoid polluting clang --help with Flang/Fortran specific options. And that's what we rely on when calling printHelp.

I'd need to double-check if we particularly care about showing/hiding flags without help text.

We should. clang --help should not be printing any Flang options, and vice-versa.

I agree we shouldn't mix Clang and Flang options together in -help. But to make sure we're on the same page: this doesn't have to do anything with the presence/absence of HelpText, this is controlled by FlangOption, FC1Option, CC1Option etc.

I'm under the impression that the contents of clang --help and clang --cc1 --help got a bit out of control. We did discuss this briefly on cfe-dev. I guess that with ~1000 options supported by Clang, this is rarely noticed. Does anyone use clang --help these days? But in Flang, we have only ~50 options and if we are not careful, flang --help would be printing hundreds of options that are not supported (and in fact, will never be).

We try to be strict about what is and what is not included in flang --help as that's currently our main end-user-facing documentation. We do this by adding a Flang flag (e.g. FlangOption or FC1Option) to every option that we support. Options that don't contain flags (e.g. feliminate-unused-debug-types) are very problematic for us. What is Flang meant to do with such options? This is an issue for every tool that relies on Options.td. This also works the other way - if we don't flag Flang options, clang --help and clang -cc1 --help will start printing options added by us. Same mechanism applies to auto-completion for command line options or for finding suggestions when an option is misspelled.

Splitting Options.td

In general, I think that Options.td would become a bit cleaner and easier to reason about if all options specified all their flags. That wouldn't be easy to achieve though!

I think that the way we define flags for the Clang driver and -cc1 in one go is already bit confusing. But it's convenient, so we roll with it.

I find it very hard to reason about things in Options.td. IMHO, we should have a separate Options.td for generic driver options (shared between Clang and Flang), and then separate Options.td for cc1 and fc1. This was brought up when we discussed implementing the Flang driver in terms of clangDriver. It would help in forcing every driver to clearly define what options it supports. Also, the long term goal is to move clangDriver out of Clang, at which point that's going to be required anyway.

It's cool to see people are thinking about improving Options.td!

IMO, OptInFFlang/OptOutFFlag are generic enough and should be shared.

Next steps

It would be nicer to have the opt-in/opt-out semantics for Flang in TableGen through something similar to BoolFOption. No need for this just now - one step at a time :)

I suggest to find another means of achieving the semantics you want. You can always copy the BoolFOption implementation and remove the marshalling stuff, if that's really the semantics you want.

The updated OptInFFlang/OptOutFFlag work just fine for us. Flang does not need EmptyKPM and BoolFOption just now. I'm not suggesting any changes here.

Ah, okay. Can you remove EmptyKPM from this patch then?

As there's only a handful of options using OptInFFlang/OptOutFFlag right now, I was hoping that the current revision would be non-controversial. We definitely need to use FC1Option (or other Flang flag) for both -ffoo and -fno-foo. Otherwise, clang --help will start printing/supporting the version without a flag. If adding a flag to both variants is not acceptable in -cc1, then I suggest splitting OptInFFlag/OptOutFFlag into

  • OptInCC1FFlang/OptOutCC1FFlag, and
  • OptInFC1FFlang/OptOutFC1FFlag.

Agreed. What was controversial from my point of view was the addition of EmptyKPM and the wording that I interpreted as "the content of HelpText influences whether a flag is accepted by driver or the front-end".

As for the help text, what would be best for Clang and -cc1? Add it to both -ffoo and -fno-foo? Just one of them?

Add it to both TableGen definitions. Whether each of the options will be printed only in clang -help or also clang -cc1 -help is determined by the presence of CC1Option.

Thanks for reviewing this and all your comments! I hope that we can figure something out that will work for both Clang and Flang.

No problem!

The clang driver always accepts both the positive and negative options, and it always only considers the argument that appeared last on the command-line (using ArgList::getLastArg(Pos, Neg)). This allows users to take a random clang driver command-line, append -feliminate-unused-debug-types and get the desired behavior, even though the original command-line might have contained -fno-eliminate-unused-debug-types.

For the pair of options you mentioned, the clang driver knows they are "opt-out" in clang -cc1. The behavior is enabled by default in -cc1 and can be disabled/opted-out through the negative -fno-eliminate-unused-debug-types. The positive option is not accepted by clang -cc1, since explicitly enabling a behavior that's already enabled by default is redundant.

If driver sees -fno-eliminate-unused-debug-types as the last option, it will forward it to clang -cc1. If it sees -feliminate-unused-debug-types, it will ignore it.

Does this make sense?

Yes. Thanks for the context and the rationale! This is much clearer now and indeed, makes sense. It's hard to extract this stuff from Options.td so I really appreciate you taking the time to explain it.

The HelpText doesn't play a role in option being accepted or not in clang or clang -cc1.

We are 100% in agreement with regard to HelpText.

I think we're in agreement that the current distinction between HelpText<""> and no HelpText is not particularly useful when printing help.

Yes. Updating printHelp wouldn't be too difficult. Would you be in favor?

I agree we shouldn't mix Clang and Flang options together in -help. But to make sure we're on the same page: this doesn't have to do anything with the presence/absence of HelpText, this is controlled by FlangOption, FC1Option, CC1Option etc.

Agreed.

I've experimented with a few more approaches and feel that the cleanest approach would be to:

  • rename OptOutFFlag/OptInFFlag as OptOutCC1FFlag and OptInCC1FFlag
  • introduce OptOutFC1FFlag and OptInFC1FFlag.

We will end-up with a bit of duplication in Options.td, but the long term goal is to split it into multiple files anyway. Also, I think that in this case code re-use would lead to a rather convoluted implementation. I'm will send an updated patch shortly.

Remove EmptyKPM, introduce OptOutFC1FFlag and OptInFC1FFlag

awarzynski retitled this revision from [flang][driver] Switch to `BoolFOption` for boolean options to [flang][driver] Refactor boolean options.Aug 4 2021, 12:41 PM
awarzynski edited the summary of this revision. (Show Details)
jansvoboda11 accepted this revision.Aug 5 2021, 12:58 AM

Yes. Updating printHelp wouldn't be too difficult. Would you be in favor?

Yes, I think that would make stuff a bit easier to understand.

I've experimented with a few more approaches and feel that the cleanest approach would be to:

  • rename OptOutFFlag/OptInFFlag as OptOutCC1FFlag and OptInCC1FFlag
  • introduce OptOutFC1FFlag and OptInFC1FFlag.

We will end-up with a bit of duplication in Options.td, but the long term goal is to split it into multiple files anyway. Also, I think that in this case code re-use would lead to a rather convoluted implementation. I'm will send an updated patch shortly.

That sounds good to me.

Thanks for seeing this through! LGTM.

This revision is now accepted and ready to land.Aug 5 2021, 12:58 AM
This revision was landed with ongoing or failed builds.Aug 5 2021, 3:21 AM
This revision was automatically updated to reflect the committed changes.