Enable fexec-charset option

[abhina.sreeskantharajan]

Summary

This patch enables the fexec-charset option to control the execution charset of string literals. It sets the default internal charset, system charset, and execution charset for z/OS and UTF-8 for all other platforms.
This patch depends on https://reviews.llvm.org/D88741

[rsmith]

I'm overall pretty happy about how clean and non-invasive the changes required here are. But please make sure you don't change the encodings of u8"..." / u"..." / U"..." literals; those need to stay as UTF-8 / UTF-16 / UTF-32. Also, we should have a story for how the wide execution character set is controlled -- is it derived from the narrow execution character set, or can the two be changed independently, or ...?

We should use the original source form of the string literal when pretty-printing a StringLiteral or CharacterLiteral; there are a bunch of UTF-8 assumptions baked into StmtPrinter that will need revisiting. And we'll need to modify the handful of places that put the contents of StringLiterals into diagnostics (#warning, #error, static_assert) and make them use a different ConversionState, since our assumption is that diagnostic output should be in UTF-8.

clang/include/clang/Lex/LiteralTranslator.h

29–31

It is not acceptable to use global state for this per-compilation information; this will behave badly if multiple independent Clang compilations are performed by different threads in the same process, for example.

32

Similarly, use of a global cache here will require you guard it with a mutex.

As an alternative, how about we move all this state to be per-instance state, and store an instance of LiteralTranslator on the Preprocessor?

clang/lib/Lex/LiteralSupport.cpp

231–239

Is it correct, in general, to do character-at-a-time translation here, when processing a string literal? I would expect there to be some (stateful) target character sets where that's not correct.

236

Reinterpreting an unsigned* as a char* like this is not correct on big-endian, and is way too "clever" on little-endian. Please create an actual char object to hold the value and pass that in instead.

238

What should happen if the result doesn't fit into an unsigned? This also appears to be making problematic assumptions about the endianness of the host. If we really want to pack multiple bytes of encoded output into a single unsigned result value (which itself seems dubious), we should do so with an endianness that doesn't depend on the host.

1321

Shouldn't this depend on the kind of literal? We should have no converter for UTF8/UTF16/UTF32 literals, should use the wide execution character set for L... literals, and the narrow execution character set otherwise. (It looks like this patch doesn't properly distinguish the narrow and wide execution character sets?)

clang/test/Driver/cl-options.c

214

Checking for "don't produce exactly this one spelling of this one diagnostic" is not a useful test; if we started warning on this again, there's a good chance the warning would be spelled differently, so your test does not do a good job of determining whether the code under test is bad (it passes in most bad states as well as in the good state). ...-NOT: error and ...-NOT: warning would be a bit better, if this is worth testing.

clang/test/Driver/clang_f_opts.c

213

Again, this is not a useful test.

[tahonermann]

clang/include/clang/Driver/Options.td

3583–3584

How about substituting "character set", "character encoding", or "charset" for "codepage"?

This doesn't state what names are recognized. The ones provided by the system iconv() implementation (as is the case for gcc)? Or all names and aliases specified by the IANA character set registry?

The set of recognized names can be a superset of the names that are actually supported.

clang/include/clang/Lex/LiteralSupport.h

192

Does the conversion state need to be persisted as a data member? The literal is consumed in the constructor.

244

Same concern here with respect to persisting the conversion state as a data member.

246

This static data member will presumably need to be lifted to per-instance state as Richard mentioned elsewhere.

clang/lib/Driver/ToolChains/Clang.cpp

5969–5977

I think it would be preferable to diagnose an unrecognized character encoding name here if possible. The current changes will result in an unrecognized name (as opposed to one that is unsupported for the target) being diagnosed for each compiler instance.

clang/lib/Frontend/CompilerInvocation.cpp

3573

I wouldn't expect the cast to std::string to be needed here.

clang/lib/Lex/LiteralSupport.cpp

231–239

For stateful encodings, I can imagine that state would have to be transitioned to the initial state before translating the escape sequence. I suspect support for stateful encodings is not a goal at this time.

234

What should happen if ResultChar >= 0x100? IBM-1047 does have representation for other UTF-8 characters. Regardless, it seems ResultChar should be converted to something.

1742–1745

UCNs will require conversion here.

llvm/lib/Support/Triple.cpp

1028–1029

No support for targeting the z/OS Enhanced ASCII run-time?

[abhina.sreeskantharajan]

Thanks for your quick reviews! I haven't addressed all the comments yet but I plan to address all of them. I put up this patch early because it has a few major changes:

• moves LiteralTranslator class to Preprocessor instead of being a static global class
• add isUTFLiteral() function to detect strings like u8"..." and stop translation
• translate wide string literals to the system charset for now (we don't have an implementation plan for -fwide-charset right now)
• remove tests that check fexec-charset will not accept non-UTF charsets

[abhina.sreeskantharajan]

In D93031#2447230, @rsmith wrote:

I'm overall pretty happy about how clean and non-invasive the changes required here are. But please make sure you don't change the encodings of u8"..." / u"..." / U"..." literals; those need to stay as UTF-8 / UTF-16 / UTF-32. Also, we should have a story for how the wide execution character set is controlled -- is it derived from the narrow execution character set, or can the two be changed independently, or ...?

We should use the original source form of the string literal when pretty-printing a StringLiteral or CharacterLiteral; there are a bunch of UTF-8 assumptions baked into StmtPrinter that will need revisiting. And we'll need to modify the handful of places that put the contents of StringLiterals into diagnostics (#warning, #error, static_assert) and make them use a different ConversionState, since our assumption is that diagnostic output should be in UTF-8.

Yes, these are some of the complications we will need to visit in later patches. We may need to somehow save the original string or reverse the translation.

clang/include/clang/Driver/Options.td

3583–3584

I've updated the description from codepage to charset.

It's hard to specify what charsets are supported because iconv library differs between targets, so the list will not be the same on every platform.

clang/include/clang/Lex/LiteralTranslator.h

32

Thanks, I've added an instance of LiteralTranslator to Preprocessor instead and use that when the Preprocessor is available. There is one constructor of StringLiteralParser that does not pass Preprocessor as an argument, so I had to create a LiteralTranslator instance there as well.

clang/lib/Driver/ToolChains/Clang.cpp

5969–5977

Since we do not know what charsets are supported by the iconv library on the target platform, we don't know what charsets are actually invalid until we try creating a CharSetConverter.

clang/lib/Frontend/CompilerInvocation.cpp

3573

Without that cast, I get the following build error:

error: no viable overloaded '='

clang/test/Driver/cl-options.c

214

You're right, I made a change just to make the testcase pass. I think this testcase is no longer needed because fexec-charset should be able to accept all charset names. We won't be able to diagnose invalid charset names until we actually try creating the CharSetConverter.

llvm/lib/Support/Triple.cpp

1028–1029

We plan to support both modes in the future, but we want the default to still be IBM-1047 (EBCDIC).

[tahonermann]

clang/include/clang/Driver/Options.td

3583–3584

Being dependent on the host iconv library seems fine by me; that is the case for gcc today. I suggest making that explicit here:

def fexec_charset : Separate<["-"], "fexec-charset">, MetaVarName<"<charset>">,
  HelpText<"Set the execution <charset> for string and character literals.  Supported character encodings include XXX and those supported by the host iconv library.">;

clang/lib/Driver/ToolChains/Clang.cpp

5969–5977

Understood, but what would be the harm in performing a lookup (constructing a CharSetConverter) here?

clang/lib/Frontend/CompilerInvocation.cpp

3573

Ok, rather than a cast, I suggest:

Opts.ExecCharset = Value.str();

clang/lib/Lex/LiteralSupport.cpp

1321–1322

Converting wide character literals to the system encoding doesn't seem right to me. For z/OS, this should presumably convert to the wide EBCDIC encoding, but for all other supported platforms, the wide execution character set is either UTF-16 or UTF-32 depending on the size of wchar_t (which may be influenced by the -fshort-wchar option).

1587–1591

The stored TranslationState should not be completely ignored for wide and UTF string literals. The standard permits things like the following.

#pragma rigoot L"bozit"
#pragma rigoot u"bozit"
_Pragma(L"rigoot bozit")
_Pragma(u8"rigoot bozit")

For at least the _Pragma(L"...") case, the C++ standard states the L is ignored, but it doesn't say anything about other encoding prefixes.

1588–1589

Converting wide string literals to the system encoding doesn't seem right to me. For z/OS, this should presumably convert to the wide EBCDIC encoding, but for all other supported platforms, the wide execution character set is either UTF-16 or UTF-32 depending on the size of wchar_t (which may be influenced by the -fshort-wchar option).

[tahonermann]

clang/test/CodeGen/systemz-charset.c

5

const char * please :)

25

Add validation of UCNs. Something like:

const char *UcnCharacters = "\u00E2\u00AC\U000000DF";
// CHECK: c"\42\B0\59\00"

[abhina.sreeskantharajan]

Thanks for your patience, I've addressed some more comments. Here is the summary of the changes in this patch:

• add translation for UCN strings, update testcase
• fix helptext for fexec-charset option in Options.td
• check for invalid charsets when parsing driver options.
• fix up char conversion code

[abhina.sreeskantharajan]

clang/include/clang/Driver/Options.td

3583–3584

I've updated the HelpText with your suggested description.

clang/include/clang/Lex/LiteralSupport.h

192

Thanks, I've removed this.

244

If this member is removed in StringLiteralParser, we will need to pass the State to multiple functions in StringLiteralParser like init(). Would this solution be preferable to keeping a data member?

clang/lib/Driver/ToolChains/Clang.cpp

5969–5977

I initially thought it will be a performance issue if we are creating the Converter twice, once here and once in the Preprocessor. But I do think its a good idea to diagnose this early. I've modified the code to diagnose and error here.

clang/lib/Frontend/CompilerInvocation.cpp

3573

Thanks, I've applied this change.

clang/lib/Lex/LiteralSupport.cpp

231–239

Right, stateful encodings may be a problem we will need to revisit later as well.

234

This is no longer valid, thanks for catching that. We were initially translating to ASCII instead of UTF-8 so we needed to guard against larger characters. I've removed this guard since the internal charset is UTF-8.

236

Thanks, I've created a char instead.

238

This may be a problem we need to revisit since ResultChar is expecting a char.

1742–1745

I've added code to translate UCN characters and have updated the testcase as well.

[abhina.sreeskantharajan]

clang/lib/Lex/LiteralSupport.cpp

1321–1322

Since we don't implement -fwide-exec-charset yet, what do you think should be the default behaviour for the interim?

[tahonermann]

clang/include/clang/Lex/LiteralSupport.h

244

I think so, yes. Data members should be used to reflect the state of the object, not as a convenient mechanism to avoid passing arguments.

clang/lib/Driver/ToolChains/Clang.cpp

5981–5983

Thank you for adding this.

clang/lib/Lex/LiteralSupport.cpp

234

Conversion can fail here, particularly in the scenario corresponding to the default switch case above; ResultChar could contain, for example, a lead byte of a UTF-8 sequence. Something sensible should be done here; either rejecting the code with an error or substituting ? (in the execution encoding) seems appropriate to me.

235

As Richard previously noted, this memcpy() needs to be addressed. The intended behavior here is not clear. Are there valid scenarios in which the conversion will produce a sequence of more than one code units? I believe the input is limited to ASCII characters and invalid code units (e.g., a lead byte of a UTF-8 sequence) and in the latter case, an error and/or substitution of a ? (in the execution encoding) seem like acceptable behaviors to me.

1321–1322

Perhaps an Internal compiler error to indicate that appropriate support is not yet in place?

clang/test/CodeGen/systemz-charset.c

17–24

const char* here too please.

clang/test/CodeGen/systemz-charset.cpp

1–25 ↗

(On Diff #313112)

This is good. I suggest adding escape sequences and UCNs to validate that they are not converted to IBM-1047.

clang/test/Driver/clang_f_opts.c

212–214

This looks good. Can tests also be added to validate that the UTF-8, ISO8895-1, and IBM-1047 option arguments are properly recognized?

[abhina.sreeskantharajan]

Thanks for the review! I've addressed most of the comments but I still need to work on the translation issues in CharLiteralParser that was kindly pointed out by Tom and Richard. Here are the summary of changes in this patch:

• Removed TranslationState as a member of StringLiteralParser and pass it as an argument instead
• Added an assertion for wide character translation instead of translating them to the system charset
• Invalid char escapes are changed to '?' and then translated
• Updated testcases as requested

[abhina.sreeskantharajan]

clang/include/clang/Lex/LiteralSupport.h

244

Thanks, I've removed this member.

clang/lib/Lex/LiteralSupport.cpp

1321–1322

Thanks for the suggestion. I've added assertions for wide character translation before we do any translation.

1588–1589

I've now added an assertion when translating wide characters.

clang/test/CodeGen/systemz-charset.cpp

1–25 ↗

(On Diff #313112)

Good idea, I added those testcases as per your suggestion.

[abhina.sreeskantharajan]

clang/lib/Lex/LiteralSupport.cpp

234

Thanks, I added the substitution with the '?' character for invalid escapes.

[abhina.sreeskantharajan]

This patch replaces the memcpy in CharLiteralParser with an assignment. I've added an assertion for cases where the character size increases after translation.

[abhina.sreeskantharajan]

clang/lib/Lex/LiteralSupport.cpp

235

I replaced memcpy with an assignment. Please let me know if there is a better solution.

238

I added an assertion for this case where the size of the character increases after translation. I've also removed the memcpy to avoid endianness issues.

[abhina.sreeskantharajan]

clang/lib/Lex/LiteralSupport.cpp

1587–1591

Please correct me if I'm wrong, these Pragma strings are not parsed through StringLiteralParser, they are parsed in clang/lib/Lex/Pragma.cpp in this function.
void Preprocessor::Handle_Pragma(Token &Tok)
So if they require translation, it would need to be done in that function.

[abhina.sreeskantharajan]

ping :)
Is there any more feedback on the implementation inside ProcessCharEscape()?

[tahonermann]

Hi, Abhina. Sorry for the delay getting back to you. I added some more comments.

clang/include/clang/Lex/LiteralSupport.h

191

Is the default argument for TranslationState actually used anywhere? I'm skeptical that a default argument provides a benefit here.

Actually, this diff doesn't include any changes to construct a CharLiteralParser with an explicit argument. It seems this argument isn't actually needed.

The only places I see objects of CharLiteralParser type constructed are in EvaluateValue() in clang/lib/Lex/PPExpressions.cpp and Sema::ActOnCharacterConstant() in clang/lib/Sema/SemaExpr.cpp.

249–250

I don't think a LiteralTranslator object is actually needed in this case. The only use of this constructor that I see is in ModuleMapParser::consumeToken() in clang/lib/Lex/ModuleMap.cpp and, in that case, I don't think any translation is necessary.

This suggests that TranslationState is not needed for this constructor either; NoTranslation can be passed to init().

262–263

I don't think getOffsetOfStringByte() should require a ConversionState parameter. If I understand it correctly, this function should be operating on the string in the internal encoding, never in a converted encoding.

clang/include/clang/Lex/LiteralTranslator.h

20–24

Some naming suggestions... The enumeration is not used to record a state, but rather to indicate an action to take. Also, use of both "conversion" and "translation" could be confusing, so I suggest sticking with one. Perhaps:

enum class LiteralConversion {
  None,
  ToSystemCharset,
  ToExecCharset
};

31–32

I don't know the LLVM style guides well, but I suspect a class with all public members should be defined using struct and not include access specifiers.

36

Given the converter setters and accessors below, ExecCharsetTables should be a private member.

38

getConversionTable() is logically const. Perhaps ExecCharsetTables should be mutable.

From a terminology stand point, this function is misnamed. It doesn't return a table, it returns a converter for an encoding. I suggest:

llvm::CharSetConverter *getCharSetConverter(const char *Encoding) const;

39

findOrCreateExecCharsetTable() seems oddly named since it doesn't return whatever it finds or creates. It seems like this function would be more useful if it returned a llvm::CharSetConverter pointer with nullptr indicating lookup/creation failed.

This function seems like it should be an implementation detail of the class, not a public interface.

42–44

setTranslationTables() is awkward. It is effectively operating as a constructor for the class, but isn't called at object construction and it does work that goes beyond initialization.

45

I suggest trying a design more like this:

class LiteralTranslator {
  std::string SystemEncoding;
  std::string ExecutionEncoding;
public:
  LiteralTranslator(llvm::StringRef SystemEncoding,
                    llvm::StringRef ExecutionEncoding);

  // Retrieve the name for the system encoding.
  llvm::StringRef getSystemEncoding() const;
  // Retrieve the name for the execution encoding.
  llvm::StringRef getExecutionEncoding() const;

  // Retrieve a converter for converting from the internal encoding (UTF-8)
  // to the system encoding.
  llvm::CharSetConverter* getSystemEncodingConverter() const;
  // Retrieve a converter for converting from the internal encoding (UTF-8)
  // to the execution encoding.
  llvm::CharSetConverter* getExecutionEncodingConverter() const;
};

LiteralTranslator createLiteralTranslatorFromOptions(const clang::LangOptions &Opts,
                                                     const clang::TargetInfo &TInfo,
                                                     clang::DiagnosticsEngine &Diags);

clang/include/clang/Lex/Preprocessor.h

145 ↗

(On Diff #314115)

I don't see a reason for LT to be a pointer. Can it be made a reference or, better, a non-reference, non-pointer data member?

clang/lib/Lex/LiteralSupport.cpp

1320–1322

Per the comment associated with the constructor declaration, I don't think the new constructor parameter is needed; translation to execution character set is always desired for non-UTF character literals. I think this can be something like:

llvm::CharSetConverter *Converter = nullptr;
if (! isUTFLiteral(Kind)) {
  assert(LT);
  Converter = LT->getCharConversionTable(TranslateToExecCharset);
}

1587–1591

Ah, ok, good. There are other cases where a string literal is not used to produce a string literal object. See https://wg21.link/p2314 for a table. You may want to audit for those cases.

clang/lib/Lex/Preprocessor.cpp

88 ↗

(On Diff #314115)

Per comments elsewhere, please try to make LT a non-pointer non-reference data member.

[jansvoboda11]

clang/include/clang/Driver/Options.td

3583

Could you switch to the option marshalling infrastructure? https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option
Adding MarshallingInfoString<LangOpts<"ExecCharset">> here should do the trick. You can then delete the option parsing in CompilerInvocation.cpp.

[abhina.sreeskantharajan]

Thanks for the feedback! I haven't addressed all the comments yet but I've made major renaming changes and hope to get feedback on it.

[abhina.sreeskantharajan]

clang/include/clang/Lex/LiteralSupport.h

191

You're right, we don't have any cases that use this arg yet so we can remove it.

249–250

Thanks, I've removed it.

clang/include/clang/Lex/Preprocessor.h

145 ↗

(On Diff #314115)

Thanks, I've changed it to a non-reference non-pointer member.

clang/lib/Lex/LiteralSupport.cpp

1320–1322

I can't add an assertion here because LT might not be created in the case of the second StringLiteralParser constructor which does not pass the Preprocessor. But I have added the remaining changes.

[abhina.sreeskantharajan]

Hi Tom, @tahonermann I renamed the LiteralTranslator class to LiteralConverter.cpp and have renamed a lot of the functions. Let me know what you think. I agree that the setConverters function is awkward, the problem stems from initializing the member early in Preprocessor but only being able to create the Converters once we know the target host later in the compilation process.

[rsmith]

clang/include/clang/Lex/Preprocessor.h

145 ↗

(On Diff #327470)

Please give this a longer name. Abbreviation names should only be used in fairly small scopes where it's easy to look up what they refer to.

Also: why LT? What does the T stand for?

clang/lib/Driver/ToolChains/Clang.cpp

5975

Looping over all the arguments is a little unusual. Normally we'd get the last argument value and only check that one. Do you need to pass more than one value onto the frontend?

clang/lib/Lex/LiteralSupport.cpp

234

This is a regression. Our prior behavior for unknown escapes was to leave the character alone. We should still do that wherever possible -- eg, \q should produce q -- and take fallback action only if the character is unencodable. Producing a ? seems unlikely to ever be what anyone wants; producing a hard error would seem preferable.

235

Can you avoid using std::string here? Eg, pass a StringRef, extending the converter to be able to take one if necessary.

238

Is there any guarantee the assertion will not fail?

1361–1362

Why is this case not possible?

1365

What assurance do we have that 1 output character is correct? I would expect we need to reject with a diagnostic if the character doesn't fit in one converted character.

1692–1694

Do we need to convert the newline character too?

Perhaps for raw string literals it'd be better to do the normal processing here and then convert the entire string at once?

clang/test/Driver/cl-options.c

213

Please use the given spelling of the flag in the diagnostic. (You can ask the argument how it was spelled.)

[abhina.sreeskantharajan]

Addressing some more comments. Updating the argument parsing, lit tests, some more renaming.

[abhina.sreeskantharajan]

clang/include/clang/Lex/Preprocessor.h

145 ↗

(On Diff #327470)

Thanks for catching this. This was a change I missed when renaming LiteralTranslator to LiteralConverter. I've added a longer name.

clang/lib/Driver/ToolChains/Clang.cpp

5975

Thanks, I've changed it back to get the LastArg only and use the spelling of the argument to fix the diagnostic error message in the driver lit tests.

clang/lib/Lex/LiteralSupport.cpp

234

Hi @tahonermann, do you also agree we should use the original behaviour or give a hard error instead?

1361–1362

This case should be handled when fwide-exec-charset option is implemented. Until then, we thought it was best to emit a error message that wide literal translation is not supported.

[abhina.sreeskantharajan]

Add assertion, add testcase for multi-line raw string

[abhina.sreeskantharajan]

clang/lib/Lex/LiteralSupport.cpp

1365

Right, I'll add a similar assertion to the one we have above.

1692–1694

Yes, we need to convert newlines as well. I think the current behaviour is already converting multi line raw strings correctly. I'll add a testcase for this.

[abhina.sreeskantharajan]

clang/include/clang/Lex/LiteralTranslator.h

31–32

I've made these private.

38

I've renamed this function to getConverter

[abhina.sreeskantharajan]

Rebase + fix CharLiteralParser endian issue by saving the char to a char variable first and then creating a StringRef

[abhina.sreeskantharajan]

Accidentally added dependent patch in this one. Removing that

[abhina.sreeskantharajan]

Rebase + set size of char as 1 when creating a StringRef to fix lit failure

[ThePhD]

Just a tiny comment: could you please make sure the name of the resolved encoding is also propagated to InitPreprocessor.cpp that sets the __clang_literal_encoding__ macro? (https://github.com/llvm/llvm-project/blob/main/clang/lib/Frontend/InitPreprocessor.cpp#L784)

[abhina.sreeskantharajan]

Thanks for catching that. This sets the clang_literal_encoding to Opts.ExecCharset or defaults to SystemCharset.

[cor3ntin]

We should use the original source form of the string literal when pretty-printing a StringLiteral or CharacterLiteral; there are a bunch of UTF-8 assumptions baked into StmtPrinter that will need revisiting. And we'll need to modify the handful of places that put the contents of StringLiterals into diagnostics (#warning, #error, static_assert) and make them use a different ConversionState, since our assumption is that diagnostic output should be in UTF-8.

Yes, these are some of the complications we will need to visit in later patches. We may need to somehow save the original string or reverse the translation.

The operation is destructive and therefore cannot be reverted.
So I do believe the correct behavior here would indeed be to keep the original spelling around - with *some* of phase 5 applied (replacement of UCNs and replacement of numeric escape sequences).
An alternative would be to do the conversion lazily when the strings are evaluated, rather than during lexing, although that might be more involved

[joerg]

"Keeping the original spelling around" would assume that the input is not using a stateful encoding. That seems worse as assumption than giving the canonical output in UTF-8 and shifting the problem to the user's editor?

[cor3ntin]

In D93031#2706988, @joerg wrote:

"Keeping the original spelling around" would assume that the input is not using a stateful encoding. That seems worse as assumption than giving the canonical output in UTF-8 and shifting the problem to the user's editor?

Right, terrible choice of words
s/original spelling/the concatenated, non-encoded string literal, in UTF-8

[abhina.sreeskantharajan]

In D93031#2706660, @cor3ntin wrote:

We should use the original source form of the string literal when pretty-printing a StringLiteral or CharacterLiteral; there are a bunch of UTF-8 assumptions baked into StmtPrinter that will need revisiting. And we'll need to modify the handful of places that put the contents of StringLiterals into diagnostics (#warning, #error, static_assert) and make them use a different ConversionState, since our assumption is that diagnostic output should be in UTF-8.

Yes, these are some of the complications we will need to visit in later patches. We may need to somehow save the original string or reverse the translation.

The operation is destructive and therefore cannot be reverted.
So I do believe the correct behavior here would indeed be to keep the original spelling around - with *some* of phase 5 applied (replacement of UCNs and replacement of numeric escape sequences).
An alternative would be to do the conversion lazily when the strings are evaluated, rather than during lexing, although that might be more involved

Thanks for the input! I agree doing the conversion lazily will help avoid hitting these issues since we push translation to a later stage but as you mentioned it will be more involved. I think keeping the original spelling might be the best solution. We can make a extra member in StringLiteralParser to save the string prior to translation. But we would need to go through each use of StringLiteralParser and save the original encoding (possibly print it in the .ll file along with the translated string or as an attribute?). Let me know what you think.

[barannikov88]

@abhina.sreeskantharajan
What is the status of this patch?

[abhina.sreeskantharajan]

In D93031#4308764, @barannikov88 wrote:

@abhina.sreeskantharajan
What is the status of this patch?

Hello, I was waiting for the CharSetConverter patch to land. Now that this patch has landed https://reviews.llvm.org/D148821 to add limited EBCDIC <-> UTF-8 conversion support, I have started to refactor my patch to use this instead. This implementation also heavily relies on iconv support which is still being discussed in the CharSet Converter RFC here https://discourse.llvm.org/t/rfc-adding-a-charset-converter-to-the-llvm-support-library/69795/16

[barannikov88]

In D93031#4309546, @abhina.sreeskantharajan wrote:

In D93031#4308764, @barannikov88 wrote:

@abhina.sreeskantharajan
What is the status of this patch?

Hello, I was waiting for the CharSetConverter patch to land. Now that this patch has landed https://reviews.llvm.org/D148821 to add limited EBCDIC <-> UTF-8 conversion support, I have started to refactor my patch to use this instead. This implementation also heavily relies on iconv support which is still being discussed in the CharSet Converter RFC here https://discourse.llvm.org/t/rfc-adding-a-charset-converter-to-the-llvm-support-library/69795/16

Thanks! I was beginning to think it is forgotten/abandoned.

[abhina.sreeskantharajan]

I have opened a new patch https://reviews.llvm.org/D153419 and am closing this revision

clang/include/clang/Basic/LangOptions.h

public:
   /// Name of the IR file that contains the result of the OpenMP target
   /// host code generation.
   std::string OMPHostIRFile;
 
   /// Indicates whether the front-end is explicitly told that the
   /// input is a header file (i.e. -x c-header).
   bool IsHeaderFile = false;
 
+  /// Name of the exec charset to convert the internal charset to.
+  std::string ExecCharset;
+
   LangOptions();
 
   // Define accessors/mutators for language options of enumeration type.
 #define LANGOPT(Name, Bits, Default, Description)
 #define ENUM_LANGOPT(Name, Type, Bits, Default, Description) \
   Type get##Name() const { return static_cast<Type>(Name); } \
   void set##Name(Type Value) { Name = static_cast<unsigned>(Value); }
 #include "clang/Basic/LangOptions.def"

clang/include/clang/Driver/Options.td

 let Flags = [CC1Option, NoDriverOption] in {
 
 //===----------------------------------------------------------------------===//
 // Target Options
 //===----------------------------------------------------------------------===//
 
 let Flags = [CC1Option, CC1AsOption, NoDriverOption] in {
 
+def fexec_charset : Separate<["-"], "fexec-charset">, MetaVarName<"<codepage>">,
+  HelpText<"Set the execution <codepage> for string and character literals">;
 def target_cpu : Separate<["-"], "target-cpu">,
   HelpText<"Target a specific cpu type">;
 def tune_cpu : Separate<["-"], "tune-cpu">,
   HelpText<"Tune for a specific cpu type">;
 def target_feature : Separate<["-"], "target-feature">,
   HelpText<"Target specific attributes">;
 def triple : Separate<["-"], "triple">,
   HelpText<"Specify target triple (e.g. i686-apple-darwin9)">,

clang/include/clang/Frontend/CompilerInstance.h

public:
 
   /// setSourceManager - Replace the current source manager.
   void setSourceManager(SourceManager *Value);
 
   /// }
   /// @name Preprocessor
   /// {
 
+  static void SetTranslationTables(LangOptions &Opts, clang::TargetInfo &TInfo,
+                                   clang::DiagnosticsEngine &Diags);
+
   bool hasPreprocessor() const { return PP != nullptr; }
 
   /// Return the current preprocessor.
   Preprocessor &getPreprocessor() const {
     assert(PP && "Compiler instance has no preprocessor!");
     return *PP;
   }
 

clang/include/clang/Lex/LiteralSupport.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LEX_LITERALSUPPORT_H
 #define LLVM_CLANG_LEX_LITERALSUPPORT_H
 
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Basic/TokenKinds.h"
+#include "clang/Lex/LiteralTranslator.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/CharSet.h"
 #include "llvm/Support/DataTypes.h"
 
 namespace clang {
 
 class DiagnosticsEngine;
 class Preprocessor;
 class Token;
 class SourceLocation;

 class CharLiteralParser {
   uint64_t Value;
   tok::TokenKind Kind;
   bool IsMultiChar;
   bool HadError;
   SmallString<32> UDSuffixBuf;
   unsigned UDSuffixOffset;
 public:
-  CharLiteralParser(const char *begin, const char *end,
-                    SourceLocation Loc, Preprocessor &PP,
-                    tok::TokenKind kind);
+  CharLiteralParser(const char *begin, const char *end, SourceLocation Loc,
+                    Preprocessor &PP, tok::TokenKind kind,
+                    ConversionState translationState = TranslateToExecCharset);
+  ConversionState TranslationState;
 
   bool hadError() const { return HadError; }
   bool isAscii() const { return Kind == tok::char_constant; }
   bool isWide() const { return Kind == tok::wide_char_constant; }
   bool isUTF8() const { return Kind == tok::utf8_char_constant; }
   bool isUTF16() const { return Kind == tok::utf16_char_constant; }
   bool isUTF32() const { return Kind == tok::utf32_char_constant; }
   bool isMultiChar() const { return IsMultiChar; }
class StringLiteralParser {
   unsigned CharByteWidth;
   tok::TokenKind Kind;
   SmallString<512> ResultBuf;
   char *ResultPtr; // cursor
   SmallString<32> UDSuffixBuf;
   unsigned UDSuffixToken;
   unsigned UDSuffixOffset;
 public:
-  StringLiteralParser(ArrayRef<Token> StringToks,
-                      Preprocessor &PP, bool Complain = true);
-  StringLiteralParser(ArrayRef<Token> StringToks,
-                      const SourceManager &sm, const LangOptions &features,
-                      const TargetInfo &target,
-                      DiagnosticsEngine *diags = nullptr)
+  StringLiteralParser(
+      ArrayRef<Token> StringToks, Preprocessor &PP, bool Complain = true,
+      ConversionState translationState = TranslateToExecCharset);
+  StringLiteralParser(ArrayRef<Token> StringToks, const SourceManager &sm,
+                      const LangOptions &features, const TargetInfo &target,
+                      DiagnosticsEngine *diags = nullptr,
+                      ConversionState translation = TranslateToExecCharset)
       : SM(sm), Features(features), Target(target), Diags(diags),
         MaxTokenLength(0), SizeBound(0), CharByteWidth(0), Kind(tok::unknown),
-      ResultPtr(ResultBuf.data()), hadError(false), Pascal(false) {
+        ResultPtr(ResultBuf.data()), hadError(false), Pascal(false),
+        TranslationState(translation) {
     init(StringToks);
   }
 
-
   bool hadError;
   bool Pascal;
+  ConversionState TranslationState;
+
+  static LiteralTranslator Translator;
 
   StringRef GetString() const {
     return StringRef(ResultBuf.data(), GetStringLength());
   }
   unsigned GetStringLength() const { return ResultPtr-ResultBuf.data(); }
 
   unsigned GetNumStringChars() const {
     return GetStringLength() / CharByteWidth;
   }
   /// getOffsetOfStringByte - This function returns the offset of the
   /// specified byte of the string data represented by Token.  This handles
   /// advancing over escape sequences in the string.
   ///
   /// If the Diagnostics pointer is non-null, then this will do semantic
   /// checking of the string literal and emit errors and warnings.
   unsigned getOffsetOfStringByte(const Token &TheTok, unsigned ByteNo) const;
 
   bool isAscii() const { return Kind == tok::string_literal; }
   bool isWide() const { return Kind == tok::wide_string_literal; }
   bool isUTF8() const { return Kind == tok::utf8_string_literal; }
   bool isUTF16() const { return Kind == tok::utf16_string_literal; }
   bool isUTF32() const { return Kind == tok::utf32_string_literal; }
   bool isPascal() const { return Pascal; }
 
   StringRef getUDSuffix() const { return UDSuffixBuf; }

clang/include/clang/Lex/LiteralTranslator.h

+//===--- clang/Lex/LiteralTranslator.h - Translator for Literals -*- C++ -*-==//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LEX_LITERALTRANSLATOR_H
+#define LLVM_CLANG_LEX_LITERALTRANSLATOR_H
+
+#include "llvm/ADT/StringMap.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/CharSet.h"
+
+enum ConversionState {
+  NoTranslation,
+  TranslateToSystemCharset,
+  TranslateToExecCharset
+};
+
+enum CharsetTableStatusCode {
+  CharsetTableOk = 1,
+  InvalidCharsetTable,
+};
+
+class LiteralTranslator {
+public:
+  static llvm::StringRef InternalCharset;
+  static llvm::StringRef SystemCharset;
+  static llvm::StringRef ExecCharset;
+  static llvm::StringMap<llvm::CharSetConverter> ExecCharsetTables;
+
+  static llvm::CharSetConverter *getConversionTable(const char *Codepage);
+  static CharsetTableStatusCode findOrCreateExecCharsetTable(const char *To);
+  llvm::CharSetConverter *
+  getCharConversionTable(ConversionState TranslationState);
+};
+
+#endif

clang/lib/Driver/ToolChains/Clang.cpp

   // -finput_charset=UTF-8 is default. Reject others
   if (Arg *inputCharset = Args.getLastArg(options::OPT_finput_charset_EQ)) {
     StringRef value = inputCharset->getValue();
     if (!value.equals_lower("utf-8"))
       D.Diag(diag::err_drv_invalid_value) << inputCharset->getAsString(Args)
                                           << value;
   }
 
-  // -fexec_charset=UTF-8 is default. Reject others
-  if (Arg *execCharset = Args.getLastArg(options::OPT_fexec_charset_EQ)) {
-    StringRef value = execCharset->getValue();
-    if (!value.equals_lower("utf-8"))
-      D.Diag(diag::err_drv_invalid_value) << execCharset->getAsString(Args)
-                                          << value;
+  // Pass all -fexec-charset options to cc1.
+  std::vector<std::string> vList =
+      Args.getAllArgValues(options::OPT_fexec_charset_EQ);
+  // Set the default fexec-charset as the system charset.
+  CmdArgs.push_back("-fexec-charset");
+  CmdArgs.push_back(Args.MakeArgString(Triple.getSystemCharset()));
+  for (auto it = vList.begin(), ie = vList.end(); it != ie; ++it) {
+    CmdArgs.push_back("-fexec-charset");
+    CmdArgs.push_back(Args.MakeArgString(*it));
   }
 
   RenderDiagnosticsOptions(D, Args, CmdArgs);
 
   // -fno-asm-blocks is default.
   if (Args.hasFlag(options::OPT_fasm_blocks, options::OPT_fno_asm_blocks,
                    false))
     CmdArgs.push_back("-fasm-blocks");
 
   // -fgnu-inline-asm is default.
   if (!Args.hasFlag(options::OPT_fgnu_inline_asm,
                     options::OPT_fno_gnu_inline_asm, true))
     CmdArgs.push_back("-fno-gnu-inline-asm");
 

clang/lib/Frontend/CompilerInstance.cpp

 //===--- CompilerInstance.cpp ---------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/AST/ASTConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/DiagnosticDriver.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/LangStandard.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/Stack.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Basic/Version.h"
 #include "clang/Config/config.h"
 #include "clang/Frontend/ChainedDiagnosticConsumer.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/LogDiagnosticPrinter.h"
 #include "clang/Frontend/SerializedDiagnosticPrinter.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Frontend/VerifyDiagnosticConsumer.h"
 #include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/LiteralTranslator.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
 #include "clang/Serialization/InMemoryModuleCache.h"
 #include "llvm/ADT/Statistic.h"
CompilerInstance::getModuleDepCollector() const {
   return ModuleDepCollector;
 }
 
 void CompilerInstance::setModuleDepCollector(
     std::shared_ptr<ModuleDependencyCollector> Collector) {
   ModuleDepCollector = std::move(Collector);
 }
 
+void CompilerInstance::SetTranslationTables(LangOptions &Opts,
+                                            TargetInfo &TInfo,
+                                            DiagnosticsEngine &Diags) {
+  using namespace llvm;
+  LiteralTranslator::SystemCharset = TInfo.getTriple().getSystemCharset();
+
+  // Create translation table between internal and system charset
+  if (!LiteralTranslator::InternalCharset.equals(
+          LiteralTranslator::SystemCharset)) {
+    LiteralTranslator::findOrCreateExecCharsetTable(
+        LiteralTranslator::SystemCharset.data());
+  }
+  // Create translation table between internal and exec charset specified
+  // in fexec-charset option.
+  if (Opts.ExecCharset.empty())
+    LiteralTranslator::ExecCharset = LiteralTranslator::InternalCharset;
+  else {
+    LiteralTranslator::ExecCharset = Opts.ExecCharset;
+    if (LiteralTranslator::InternalCharset.equals(
+            LiteralTranslator::ExecCharset))
+      return;
+    CharsetTableStatusCode RC = LiteralTranslator::findOrCreateExecCharsetTable(
+        LiteralTranslator::ExecCharset.data());
+
+    if (RC != CharsetTableOk)
+      Diags.Report(diag::err_drv_invalid_value)
+          << "-fexec-charset" << LiteralTranslator::ExecCharset;
+  }
+}
+
 static void collectHeaderMaps(const HeaderSearch &HS,
                               std::shared_ptr<ModuleDependencyCollector> MDC) {
   SmallVector<std::string, 4> HeaderMapFileNames;
   HS.getHeaderMapFileNames(HeaderMapFileNames);
   for (auto &Name : HeaderMapFileNames)
     MDC->addFile(Name);
 }
 
if (!Act.PrepareToExecute(*this))
     return false;
 
   // Create the target instance.
   setTarget(TargetInfo::CreateTargetInfo(getDiagnostics(),
                                          getInvocation().TargetOpts));
   if (!hasTarget())
     return false;
 
+  SetTranslationTables(getLangOpts(), getTarget(), getDiagnostics());
+
   // Create TargetInfo for the other side of CUDA/OpenMP/SYCL compilation.
   if ((getLangOpts().CUDA || getLangOpts().OpenMPIsDevice ||
        getLangOpts().SYCLIsDevice) &&
       !getFrontendOpts().AuxTriple.empty()) {
     auto TO = std::make_shared<TargetOptions>();
     TO->Triple = llvm::Triple::normalize(getFrontendOpts().AuxTriple);
     if (getFrontendOpts().AuxTargetCPU)
       TO->CPU = getFrontendOpts().AuxTargetCPU.getValue();

clang/lib/Frontend/CompilerInvocation.cpp

#include "clang/Basic/LangStandards.def"
 
   Opts.CompatibilityQualifiedIdBlockParamTypeChecking =
       Args.hasArg(OPT_fcompatibility_qualified_id_block_param_type_checking);
 
   Opts.RelativeCXXABIVTables =
       Args.hasFlag(OPT_fexperimental_relative_cxx_abi_vtables,
                    OPT_fno_experimental_relative_cxx_abi_vtables,
                    /*default=*/false);
+
+  if (Arg *ExecCharset = Args.getLastArg(OPT_fexec_charset)) {
+    StringRef Value = ExecCharset->getValue();
+    Opts.ExecCharset = (std::string)Value;
+  }
 }
 
 static bool isStrictlyPreprocessorAction(frontend::ActionKind Action) {
   switch (Action) {
   case frontend::ASTDeclList:
   case frontend::ASTDump:
   case frontend::ASTPrint:
   case frontend::ASTView:

clang/lib/Lex/CMakeLists.txt

 # TODO: Add -maltivec when ARCH is PowerPC.
 
 set(LLVM_LINK_COMPONENTS support)
 
 add_clang_library(clangLex
   DependencyDirectivesSourceMinimizer.cpp
   HeaderMap.cpp
   HeaderSearch.cpp
   Lexer.cpp
   LiteralSupport.cpp
+  LiteralTranslator.cpp
   MacroArgs.cpp
   MacroInfo.cpp
   ModuleMap.cpp
   PPCaching.cpp
   PPCallbacks.cpp
   PPConditionalDirectiveRecord.cpp
   PPDirectives.cpp
   PPExpressions.cpp

clang/lib/Lex/LiteralSupport.cpp

 
 /// ProcessCharEscape - Parse a standard C escape sequence, which can occur in
 /// either a character or a string literal.
 static unsigned ProcessCharEscape(const char *ThisTokBegin,
                                   const char *&ThisTokBuf,
                                   const char *ThisTokEnd, bool &HadError,
                                   FullSourceLoc Loc, unsigned CharWidth,
                                   DiagnosticsEngine *Diags,
-                                  const LangOptions &Features) {
+                                  const LangOptions &Features,
+                                  llvm::CharSetConverter *Converter) {
   const char *EscapeBegin = ThisTokBuf;
 
   // Skip the '\' char.
   ++ThisTokBuf;
 
   // We know that this character can't be off the end of the buffer, because
   // that would have been \", which would not have been the end of string.
   unsigned ResultChar = *ThisTokBuf++;
+  bool Translate = true;
   switch (ResultChar) {
   // These map to themselves.
   case '\\': case '\'': case '"': case '?': break;
 
     // These have fixed mappings.
   case 'a':
     // TODO: K&R: the meaning of '\\a' is different in traditional C
     ResultChar = 7;
case 'r':
     break;
   case 't':
     ResultChar = 9;
     break;
   case 'v':
     ResultChar = 11;
     break;
   case 'x': { // Hex escape.
+    Translate = false;
     ResultChar = 0;
     if (ThisTokBuf == ThisTokEnd || !isHexDigit(*ThisTokBuf)) {
       if (Diags)
         Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
              diag::err_hex_escape_no_digits) << "x";
       HadError = true;
       break;
     }
if (Overflow && Diags)   // Too many digits to fit in
       Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
            diag::err_escape_too_large) << 0;
     break;
   }
   case '0': case '1': case '2': case '3':
   case '4': case '5': case '6': case '7': {
     // Octal escapes.
     --ThisTokBuf;
+    Translate = false;
     ResultChar = 0;
 
     // Octal escapes are a series of octal digits with maximum length 3.
     // "\0123" is a two digit sequence equal to "\012" "3".
     unsigned NumDigits = 0;
     do {
       ResultChar <<= 3;
       ResultChar |= *ThisTokBuf++ - '0';
if (isPrintable(ResultChar))
         << std::string(1, ResultChar);
     else
       Diag(Diags, Features, Loc, ThisTokBegin, EscapeBegin, ThisTokBuf,
            diag::ext_unknown_escape)
         << "x" + llvm::utohexstr(ResultChar);
     break;
   }
 
+  if (Translate && Converter) {
+    // ResultChar is either UTF-8 or ASCII literal and can only be converted
+    // to EBCDIC on z/OS if the character can be represented in one byte.
+    if (ResultChar < 0x100) {
+      SmallString<8> ResultCharConv;
+      Converter->convert(StringRef((char *)&ResultChar), ResultCharConv);
+      void *Pointer = &ResultChar;
+      memcpy(Pointer, ResultCharConv.data(), sizeof(unsigned));
+    }
+  }
   return ResultChar;
 }
 
 static void appendCodePoint(unsigned Codepoint,
                             llvm::SmallVectorImpl<char> &Str) {
   char ResultBuf[4];
   char *ResultPtr = ResultBuf;
   bool Res = llvm::ConvertCodePointToUTF8(Codepoint, ResultPtr);

 ///         \u hex-quad
 ///         \U hex-quad hex-quad
 ///       hex-quad:
 ///         hex-digit hex-digit hex-digit hex-digit
 /// \endverbatim
 ///
 CharLiteralParser::CharLiteralParser(const char *begin, const char *end,
                                      SourceLocation Loc, Preprocessor &PP,
-                                     tok::TokenKind kind) {
+                                     tok::TokenKind kind,
+                                     ConversionState translationState) {
   // At this point we know that the character matches the regex "(L|u|U)?'.*'".
   HadError = false;
-
+  TranslationState = translationState;
   Kind = kind;
 
   const char *TokBegin = begin;
 
   // Skip over wide character determinant.
   if (Kind != tok::char_constant)
     ++begin;
   if (Kind == tok::utf8_char_constant)
CharLiteralParser::CharLiteralParser(const char *begin, const char *end,
   } else if (tok::utf16_char_constant == Kind) {
     largest_character_for_kind = 0xFFFF;
   } else if (tok::utf32_char_constant == Kind) {
     largest_character_for_kind = 0x10FFFF;
   } else {
     largest_character_for_kind = 0x7Fu;
   }
 
+  llvm::CharSetConverter *Converter =
+      StringLiteralParser::Translator.getCharConversionTable(TranslationState);
+
   while (begin != end) {
     // Is this a span of non-escape characters?
     if (begin[0] != '\\') {
       char const *start = begin;
       do {
         ++begin;
       } while (begin != end && *begin != '\\');
 
if (begin[0] != '\\') {
           HadError = true;
         }
       } else {
         for (; tmp_out_start < buffer_begin; ++tmp_out_start) {
           if (*tmp_out_start > largest_character_for_kind) {
             HadError = true;
             PP.Diag(Loc, diag::err_character_too_large);
           }
+          if (!HadError && Converter) {
+            SmallString<1> ConvertedChar;
+            Converter->convert(StringRef((char *)tmp_out_start), ConvertedChar);
+            memmove((void *)tmp_out_start, ConvertedChar.data(), 1);
+          }
         }
       }
 
       continue;
     }
     // Is this a Universal Character Name escape?
     if (begin[1] == 'u' || begin[1] == 'U') {
       unsigned short UcnLen = 0;
       if (!ProcessUCNEscape(TokBegin, begin, end, *buffer_begin, UcnLen,
                             FullSourceLoc(Loc, PP.getSourceManager()),
                             &PP.getDiagnostics(), PP.getLangOpts(), true)) {
         HadError = true;
       } else if (*buffer_begin > largest_character_for_kind) {
         HadError = true;
         PP.Diag(Loc, diag::err_character_too_large);
       }
 
       ++buffer_begin;
       continue;
     }
     unsigned CharWidth = getCharWidth(Kind, PP.getTargetInfo());
     uint64_t result =
         ProcessCharEscape(TokBegin, begin, end, HadError,
-                        FullSourceLoc(Loc,PP.getSourceManager()),
-                        CharWidth, &PP.getDiagnostics(), PP.getLangOpts());
+                          FullSourceLoc(Loc, PP.getSourceManager()), CharWidth,
+                          &PP.getDiagnostics(), PP.getLangOpts(), nullptr);
     *buffer_begin++ = result;
   }
 
   unsigned NumCharsSoFar = buffer_begin - &codepoint_buffer.front();
 
   if (NumCharsSoFar > 1) {
     if (isWide())
       PP.Diag(Loc, diag::warn_extraneous_char_constant);

 ///         hexadecimal-escape-sequence hexadecimal-digit
 ///       universal-character-name:
 ///         \u hex-quad
 ///         \U hex-quad hex-quad
 ///       hex-quad:
 ///         hex-digit hex-digit hex-digit hex-digit
 /// \endverbatim
 ///
-StringLiteralParser::
-StringLiteralParser(ArrayRef<Token> StringToks,
-                    Preprocessor &PP, bool Complain)
+LiteralTranslator StringLiteralParser::Translator;
+
+StringLiteralParser::StringLiteralParser(ArrayRef<Token> StringToks,
+                                         Preprocessor &PP, bool Complain,
+                                         ConversionState translationState)
     : SM(PP.getSourceManager()), Features(PP.getLangOpts()),
-    Target(PP.getTargetInfo()), Diags(Complain ? &PP.getDiagnostics() :nullptr),
-    MaxTokenLength(0), SizeBound(0), CharByteWidth(0), Kind(tok::unknown),
-    ResultPtr(ResultBuf.data()), hadError(false), Pascal(false) {
+      Target(PP.getTargetInfo()),
+      Diags(Complain ? &PP.getDiagnostics() : nullptr), MaxTokenLength(0),
+      SizeBound(0), CharByteWidth(0), Kind(tok::unknown),
+      ResultPtr(ResultBuf.data()), hadError(false), Pascal(false),
+      TranslationState(translationState) {
   init(StringToks);
 }
 
 void StringLiteralParser::init(ArrayRef<Token> StringToks){
   // The literal token may have come from an invalid source location (e.g. due
   // to a PCH error), in which case the token length will be 0.
   if (StringToks.empty() || StringToks[0].getLength() < 2)
     return DiagnoseLexingError(SourceLocation());
void StringLiteralParser::init(ArrayRef<Token> StringToks){
   // Loop over all the strings, getting their spelling, and expanding them to
   // wide strings as appropriate.
   ResultPtr = &ResultBuf[0];   // Next byte to fill in.
 
   Pascal = false;
 
   SourceLocation UDSuffixTokLoc;
 
+  ConversionState State = TranslationState;
+  llvm::CharSetConverter *Converter = Translator.getCharConversionTable(State);
+
   for (unsigned i = 0, e = StringToks.size(); i != e; ++i) {
     const char *ThisTokBuf = &TokenBuf[0];
     // Get the spelling of the token, which eliminates trigraphs, etc.  We know
     // that ThisTokBuf points to a buffer that is big enough for the whole token
     // and 'spelled' tokens can only shrink.
     bool StringInvalid = false;
     unsigned ThisTokLen =
       Lexer::getSpelling(StringToks[i], ThisTokBuf, SM, Features,
                          &StringInvalid);
     if (StringInvalid)
if (ThisTokBuf[0] == 'R') {
         size_t CRLFPos = RemainingTokenSpan.find("\r\n");
         StringRef BeforeCRLF = RemainingTokenSpan.substr(0, CRLFPos);
         StringRef AfterCRLF = RemainingTokenSpan.substr(CRLFPos);
 
         // Copy everything before the \r\n sequence into the string literal.
         if (CopyStringFragment(StringToks[i], ThisTokBegin, BeforeCRLF))
           hadError = true;
 
+        if (!hadError && Converter) {
+          SmallString<256> CpConv;
+          int ResultLength = BeforeCRLF.size() * CharByteWidth;
+          unsigned char *Cp = (unsigned char *)ResultPtr - ResultLength;
+          Converter->convert(StringRef((char *)Cp, ResultLength), CpConv);
+          memmove(Cp, CpConv.data(), ResultLength);
+        }
         // Point into the \n inside the \r\n sequence and operate on the
         // remaining portion of the literal.
         RemainingTokenSpan = AfterCRLF.substr(1);
       }
     } else {
       if (ThisTokBuf[0] != '"') {
         // The file may have come from PCH and then changed after loading the
         // PCH; Fail gracefully.
         return DiagnoseLexingError(StringToks[i].getLocation());
       }
       ++ThisTokBuf; // skip "
if (ThisTokBuf[0] == 'R') {
       while (ThisTokBuf != ThisTokEnd) {
         // Is this a span of non-escape characters?
         if (ThisTokBuf[0] != '\\') {
           const char *InStart = ThisTokBuf;
           do {
             ++ThisTokBuf;
           } while (ThisTokBuf != ThisTokEnd && ThisTokBuf[0] != '\\');
 
+          int Length = ThisTokBuf - InStart;
           // Copy the character span over.
           if (CopyStringFragment(StringToks[i], ThisTokBegin,
                                  StringRef(InStart, ThisTokBuf - InStart)))
             hadError = true;
+
+          if (!hadError && Converter) {
+            SmallString<256> CpConv;
+            int ResultLength = Length * CharByteWidth;
+            unsigned char *Cp = (unsigned char *)ResultPtr - ResultLength;
+            Converter->convert(StringRef((char *)Cp, ResultLength), CpConv);
+            memmove(Cp, CpConv.data(), ResultLength);
+          }
           continue;
         }
         // Is this a Universal Character Name escape?
         if (ThisTokBuf[1] == 'u' || ThisTokBuf[1] == 'U') {
           EncodeUCNEscape(ThisTokBegin, ThisTokBuf, ThisTokEnd,
                           ResultPtr, hadError,
                           FullSourceLoc(StringToks[i].getLocation(), SM),
                           CharByteWidth, Diags, Features);
           continue;
         }
         // Otherwise, this is a non-UCN escape character.  Process it.
         unsigned ResultChar =
             ProcessCharEscape(ThisTokBegin, ThisTokBuf, ThisTokEnd, hadError,
                               FullSourceLoc(StringToks[i].getLocation(), SM),
-                            CharByteWidth*8, Diags, Features);
+                              CharByteWidth * 8, Diags, Features, Converter);
 
         if (CharByteWidth == 4) {
           // FIXME: Make the type of the result buffer correct instead of
           // using reinterpret_cast.
           llvm::UTF32 *ResultWidePtr = reinterpret_cast<llvm::UTF32*>(ResultPtr);
           *ResultWidePtr = ResultChar;
           ResultPtr += 4;
         } else if (CharByteWidth == 2) {
if (SpellingPtr[0] == 'R') {
     ++SpellingPtr;
     return SpellingPtr - SpellingStart + ByteNo;
   }
 
   // Skip over the leading quote
   assert(SpellingPtr[0] == '"' && "Should be a string literal!");
   ++SpellingPtr;
 
+  ConversionState State = TranslationState;
+  llvm::CharSetConverter *Converter = Translator.getCharConversionTable(State);
+
   // Skip over bytes until we find the offset we're looking for.
   while (ByteNo) {
     assert(SpellingPtr < SpellingEnd && "Didn't find byte offset!");
 
     // Step over non-escapes simply.
     if (*SpellingPtr != '\\') {
       ++SpellingPtr;
       --ByteNo;
if (SpellingPtr[1] == 'u' || SpellingPtr[1] == 'U') {
       if (Len > ByteNo) {
         // ByteNo is somewhere within the escape sequence.
         SpellingPtr = EscapePtr;
         break;
       }
       ByteNo -= Len;
     } else {
       ProcessCharEscape(SpellingStart, SpellingPtr, SpellingEnd, HadError,
-                        FullSourceLoc(Tok.getLocation(), SM),
-                        CharByteWidth*8, Diags, Features);
+                        FullSourceLoc(Tok.getLocation(), SM), CharByteWidth * 8,
+                        Diags, Features, Converter);
       --ByteNo;
     }
     assert(!HadError && "This method isn't valid on erroneous strings");
   }
 
   return SpellingPtr-SpellingStart;
 }
 
 /// Determine whether a suffix is a valid ud-suffix. We avoid treating reserved
 /// suffixes as ud-suffixes, because the diagnostic experience is better if we
 /// treat it as an invalid suffix.
 bool StringLiteralParser::isValidUDSuffix(const LangOptions &LangOpts,
                                           StringRef Suffix) {
   return NumericLiteralParser::isValidUDSuffix(LangOpts, Suffix) ||
          Suffix == "sv";
 }

clang/lib/Lex/LiteralTranslator.cpp

+//===--- LiteralTranslator.cpp - Translator for String Literals -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Lex/LiteralTranslator.h"
+
+using namespace llvm;
+
+StringRef LiteralTranslator::InternalCharset = "UTF-8";
+StringRef LiteralTranslator::SystemCharset = "";
+StringRef LiteralTranslator::ExecCharset = "";
+llvm::StringMap<llvm::CharSetConverter> LiteralTranslator::ExecCharsetTables;
+
+llvm::CharSetConverter *
+LiteralTranslator::getConversionTable(const char *Codepage) {
+  auto TableIter = LiteralTranslator::ExecCharsetTables.find(Codepage);
+  if (TableIter != LiteralTranslator::ExecCharsetTables.end())
+    return &TableIter->second;
+  else
+    return nullptr;
+}
+
+CharsetTableStatusCode
+LiteralTranslator::findOrCreateExecCharsetTable(const char *To) {
+  const char *From = LiteralTranslator::InternalCharset.data();
+  llvm::CharSetConverter *Converter = LiteralTranslator::getConversionTable(To);
+
+  if (Converter)
+    return CharsetTableOk;
+
+  ErrorOr<CharSetConverter> ErrorOrConverter =
+      llvm::CharSetConverter::create(From, To);
+  if (!ErrorOrConverter)
+    return InvalidCharsetTable;
+  LiteralTranslator::ExecCharsetTables.insert_or_assign(
+      StringRef(To), std::move(*ErrorOrConverter));
+  return CharsetTableOk;
+}
+
+llvm::CharSetConverter *
+LiteralTranslator::getCharConversionTable(ConversionState TranslationState) {
+  StringRef CodePage;
+  if (TranslationState == TranslateToSystemCharset)
+    CodePage = LiteralTranslator::SystemCharset;
+  else if (TranslationState == TranslateToExecCharset)
+    CodePage = LiteralTranslator::ExecCharset;
+  else
+    CodePage = LiteralTranslator::InternalCharset;
+  return getConversionTable(CodePage.data());
+}

clang/test/CodeGen/systemz-charset.c

+// RUN: %clang_cc1 %s -emit-llvm -triple s390x-none-zos -fexec-charset IBM-1047 -o - | FileCheck %s
+// RUN: %clang %s -emit-llvm -S -target s390x-ibm-zos -o - | FileCheck %s
+
+char *UpperCaseLetters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
+// CHECK: c"\C1\C2\C3\C4\C5\C6\C7\C8\C9\D1\D2\D3\D4\D5\D6\D7\D8\D9\E2\E3\E4\E5\E6\E7\E8\E9\00"
+
+char *LowerCaseLetters = "abcdefghijklmnopqrstuvwxyz";
+//CHECK: c"\81\82\83\84\85\86\87\88\89\91\92\93\94\95\96\97\98\99\A2\A3\A4\A5\A6\A7\A8\A9\00"
+
+char *Digits = "0123456789";
+// CHECK: c"\F0\F1\F2\F3\F4\F5\F6\F7\F8\F9\00"
+
+char *SpecialCharacters = " .<(+|&!$*);^-/,%%_>`:#@=";
+// CHECK: c"@KLMNOPZ[\\]^_`akllmnyz{|~\00"
+
+char *EscapeCharacters = "\a\b\f\n\r\t\v\\\'\"\?";
+//CHECK: c"/\16\0C\15\0D\05\0B\E0}\7Fo\00"
+
+char *HexCharacters = "\x12\x13\x14";
+//CHECK: c"\12\13\14\00"
+
+char *OctalCharacters = "\141\142\143";
+//CHECK: c"abc\00"
+
+char singleChar = 'a';
+//CHECK: i8 -127

clang/test/Driver/cl-options.c

 // RUN: %clang_cl /E /EP /showIncludes -### -- %s 2>&1 | FileCheck -check-prefix=showIncludes_E %s
 // RUN: %clang_cl /EP /P /showIncludes -### -- %s 2>&1 | FileCheck -check-prefix=showIncludes_E %s
 // showIncludes_E-NOT: warning: argument unused during compilation: '--show-includes'
 
 // /source-charset: should warn on everything except UTF-8.
 // RUN: %clang_cl /source-charset:utf-16 -### -- %s 2>&1 | FileCheck -check-prefix=source-charset-utf-16 %s
 // source-charset-utf-16: invalid value 'utf-16' in '/source-charset:utf-16'
 
-// /execution-charset: should warn on everything except UTF-8.
-// RUN: %clang_cl /execution-charset:utf-16 -### -- %s 2>&1 | FileCheck -check-prefix=execution-charset-utf-16 %s
-// execution-charset-utf-16: invalid value 'utf-16' in '/execution-charset:utf-16'
-//
+// /execution-charset: should not warn on character sets.
+// RUN: %clang_cl /execution-charset:iso8859-1 -### -- %s 2>&1 | FileCheck -check-prefix=execution-charset-iso8859-1 %s
+// execution-charset-iso8859-1-NOT: invalid value 'iso8859-1' in '/execution-charset:iso8859-1'
+
 // RUN: %clang_cl /Umymacro -### -- %s 2>&1 | FileCheck -check-prefix=U %s
 // RUN: %clang_cl /U mymacro -### -- %s 2>&1 | FileCheck -check-prefix=U %s
 // U: "-U" "mymacro"
 
 // RUN: %clang_cl /validate-charset -### -- %s 2>&1 | FileCheck -check-prefix=validate-charset %s
 // validate-charset: -Winvalid-source-encoding
 
 // RUN: %clang_cl /validate-charset- -### -- %s 2>&1 | FileCheck -check-prefix=validate-charset_ %s

clang/test/Driver/clang_f_opts.c

 // CHECK-MAX-O: -O3
 
 // RUN: %clang -S -O20 -o /dev/null %s 2>&1 | FileCheck -check-prefix=CHECK-INVALID-O %s
 // CHECK-INVALID-O: warning: optimization level '-O20' is not supported; using '-O3' instead
 
 // RUN: %clang -### -S -finput-charset=iso-8859-1 -o /dev/null %s 2>&1 | FileCheck -check-prefix=CHECK-INVALID-CHARSET %s
 // CHECK-INVALID-CHARSET: error: invalid value 'iso-8859-1' in '-finput-charset=iso-8859-1'
 
-// RUN: %clang -### -S -fexec-charset=iso-8859-1 -o /dev/null %s 2>&1 | FileCheck -check-prefix=CHECK-INVALID-INPUT-CHARSET %s
-// CHECK-INVALID-INPUT-CHARSET: error: invalid value 'iso-8859-1' in '-fexec-charset=iso-8859-1'
+// RUN: %clang -### -S -fexec-charset=iso8859-1 -o /dev/null %s 2>&1 | FileCheck -check-prefix=CHECK-VALID-INPUT-CHARSET %s
+// CHECK-VALID-INPUT-CHARSET-NOT: error: invalid value 'iso8859-1' in '-fexec-charset=iso8859-1'
 
 // Test that we don't error on these.
 // RUN: %clang -### -S -Werror                                                \
 // RUN:     -falign-functions -falign-functions=2 -fno-align-functions        \
 // RUN:     -fasynchronous-unwind-tables -fno-asynchronous-unwind-tables      \
 // RUN:     -fbuiltin -fno-builtin                                            \
 // RUN:     -fdiagnostics-show-location=once                                  \
 // RUN:     -ffloat-store -fno-float-store                                    \
 // RUN:     -feliminate-unused-debug-types -fno-eliminate-unused-debug-types  \

llvm/include/llvm/ADT/Triple.h

public:
   /// component of the triple, or "" if empty.
   StringRef getEnvironmentName() const;
 
   /// getOSAndEnvironmentName - Get the operating system and optional
   /// environment components as a single string (separated by a '-'
   /// if the environment component is present).
   StringRef getOSAndEnvironmentName() const;
 
+  /// getSystemCharset - Get the system charset of the triple.
+  StringRef getSystemCharset() const;
+
   /// @}
   /// @name Convenience Predicates
   /// @{
 
   /// Test whether the architecture is 64-bit
   ///
   /// Note that this tests for 64-bit pointer width, and nothing else. Note
   /// that we intentionally expose only three predicates, 64-bit, 32-bit, and

llvm/lib/Support/Triple.cpp

StringRef Triple::getEnvironmentName() const {
   return Tmp.split('-').second;                      // Strip third component
 }
 
 StringRef Triple::getOSAndEnvironmentName() const {
   StringRef Tmp = StringRef(Data).split('-').second; // Strip first component
   return Tmp.split('-').second;                      // Strip second component
 }
 
+// System charset on z/OS is IBM-1047 and UTF-8 otherwise
+StringRef Triple::getSystemCharset() const {
+  if (getOS() == llvm::Triple::ZOS)
+    return "IBM-1047";
+  return "UTF-8";
+}
+
 static unsigned EatNumber(StringRef &Str) {
   assert(!Str.empty() && Str[0] >= '0' && Str[0] <= '9' && "Not a number");
   unsigned Result = 0;
 
   do {
     // Consume the leading digit.
     Result = Result*10 + (Str[0] - '0');