[SystemZ/z/OS] Add utility class for char set conversion.

[Kai]

Summary

This class adds support for conversion between different character
sets. The conversion between EBCDIC-1047 and Latin-1/UTF-8 is always
available. If the iconv library is installed, then all conversions
of the iconv library can be used, too.

Use of iconv functions is not as standardized as wanted.
Challenges found so far:

• On some functions, the iconv*() functions are part of the C library. On other systems, a separate library must be linked in.
• There are systems with multiple, incompatible implementations of iconv functionality.
• Not each system provides an implementation.
• Mapping of EBCDIC-1047 to ASCII/Latin-1/UTF-8 and vice versa is done differently on different platforms.
• Each implementation supports different mappings.

As result, the implementation provides a thin wrapper around the
iconv functionality, including a fixed conversion for EBCDIC-1047.

[echristo]

Drive by as it's years since I've dealt with any of this:

Seems ok to me. Be good to send your RFC as well to clang-dev to get a view into -finput-charset translations as well if we wanted to go there.

[Kai]

In D88741#2308899, @echristo wrote:

Seems ok to me. Be good to send your RFC as well to clang-dev to get a view into -finput-charset translations as well if we wanted to go there.

Sure, done.

[ctetreau]

llvm/lib/Support/CharSet.cpp

27

These names leak the iconv dependency, and can be error prone if somebody types them wrong. We should have an enum class for the set of char sets that can be converted. If we keep the iconv dependency, it's trivial to map CharSet::IBM1047 -> "IBM-1047", but the compiler can catch the error if you mistype the enum name.

159

If we replace the name strings with enums, then it should be possible to make this function total and remove the ErrorOr

191

If I'm understanding this correctly, having iconv provides the possibility of supporting conversions other than to and from ascii, utf8, and ebcdic? I'm concerned that this is going to create a ton of bug reports of the form "CharSetConverter::create returned an error on my machine, but not my coworker's machine!" which will be closed as "operator error, install iconv". I feel like there should be a set of conversions supported by CharSetConverter, and they should work regardless of the presense of iconv.

From messages I've seen on the mailing lists, it sounds like there is license uncertainty with linking iconv. Maybe it's best to just not have this?

[ThePhD]

Minor addition for this change; if this does go through, llvm-specific macros in C-like languages should expose the chosen name. I submitted a patch and a feature request to MSVC and GCC for their own implementation-specific macros as well:

• https://developercommunity.visualstudio.com/content/idea/1160821/-compiler-feature-macro-for-narrow-literal-foo-enc.html
• https://gcc.gnu.org/pipermail/gcc-patches/2020-October/555801.html

It doesn't need to be synchronized with any other compiler's name or exposed feature: it just needs to exist. This allows folks interested in helping people write portable code that uses the execution character set preserve the invariants of their code by allowing them to inspect the name and act meaningfully for names they recognize.

[echristo]

Seems reasonable to me.

-eric

[Kai]

Sorry for taking so long - I was on vacation.

• Added an enumeration for the basic charsets
• Added an named constructor just for the basic charsets
• Updated the test cases

I like the addition of the enumeration. It simplifies the source in some parts.
However, I did not drop the function which uses iconv for conversion. This will be needed for some further extensions.

[fanbo-meng]

llvm/lib/Support/CharSet.cpp

90

The naming here with "UTF" is ambiguous as it can mean UTF8, UTF16, UTF32. Using UTF8 with the enum names here would be better.

[jhenderson]

Drive-by comment as I'm reviewing a downstream patch - I think it's preferable to use llvm::Error and llvm::Expected instead of std::error_code, for reporting errors, as it allows you to provide more contextual information (e.g. why the conversion failed, which characters were invalid etc).

[abhina.sreeskantharajan]

The CI on Windows is failing because it seems that the iconv header exists but the path is unknown. Is it possible to set a variable to Iconv_INCLUDE_DIRS and use that instead of writing out <iconv.h> ? I'm seeing this failure on the CI on my fexec-charset patch as well.

[Kai]

Renamed enum members to include 8 for UTF8.

[Kai]

llvm/lib/Support/CharSet.cpp

90

Renamed.

[Kai]

Added the path of the <iconv.h> header as additional include path,

[Kai]

Use the right CMake variable Iconv_INCLUDE_DIRS.

[Kai]

In D88741#2613823, @abhina.sreeskantharajan wrote:

The CI on Windows is failing because it seems that the iconv header exists but the path is unknown. Is it possible to set a variable to Iconv_INCLUDE_DIRS and use that instead of writing out <iconv.h> ? I'm seeing this failure on the CI on my fexec-charset patch as well.

Iconv_INCLUDE_DIRS is now in the header file search path. This hopefully fixes the compile error.

[Kai]

llvm/lib/Support/CharSet.cpp

191

This can be viewed as the same as compiling a file: it fails on my machine if I do not have the same file in the same place as my coworker. This problem seems acceptable for the clang -fexec-charset patch.

iconv is a POSIX specification (POSIX iconv.h), so there is no license problem.

[Kai]

• Introduces build option LLVM_ENABLE_ICONV to optionally turn building with iconv off
• Add include path to the iconv header when building with iconv enabled and not part of C library This hopefully fixes the build failure on the Windows bot.

[Kai]

Fixed a problem in the CMake files.
This should finally fix the problem seen on the Windows builder.

[Kai]

Fix wrong use of #ifdef

[Kai]

Rebased on latest main.

[hubert.reinterpretcast]

llvm/include/llvm/Support/CharSet.h

48

Just noting that https://wg21.link/p1885 proposes enumeration names (and values) for these.

llvm/lib/Support/CharSet.cpp

31–36

There is a normalization process for character set names described by https://wg21.link/p1885 (please refer to the source material directly as well: https://www.unicode.org/reports/tr22/tr22-8.html#Charset_Alias_Matching).

ISO8859-1 is neither the primary name nor one of the aliases defined for this charset in the IANA character set registry.

103

The comment should say, "only valid sequences encoding UCS scalar values in the range [U+0080, U+00FF] can be decoded".

105

The API contract between the table conversion and the iconv conversion is inconsistent. iconv conversion performs an implementation-defined conversion for valid input characters that does not have a representation in the output codeset. This implementation fails the conversion instead.

114

The coding guidelines say to use preincrement in cases where postincrement is not needed.

141

This looks like a use case for resize_for_overwrite.

152

Missing overflow detection for when the most significant bit of Capacity is already set.

153

Same comment about resize_for_overwrite.

156

The coding guidelines have been updated to discourage mixing use and omission of braces for the constituent parts of an if/else chain.

195

Use a C-style cast if necessary, but reinterpret_cast is wrong here.

[hubert.reinterpretcast]

llvm/lib/Support/CharSet.cpp

175–178

This is a public constructor. It would be appropriate to check that CSFrom and CSTo aren't both CS_IBM1047...

179–183

Because of the [U+0000, U+00FF] limitation of the convertWithTable converter, we should not be getting here if there is a no-op conversion from UTF-8 to UTF-8. Either a no-op converter should be returned, or a request for such a converter is erroneous (report_fatal_error here would be friendlier by failing fast).

[hubert.reinterpretcast]

llvm/include/llvm/Support/CharSet.h

98

The expectations around null termination and shift state should be documented. Note that https://wg21.link/p2029 recommends that numeric escapes do not trigger changes in shift state, so avoiding automatic transitions to the initial shift state at the end of the (potential mid-string) translation is probably the right thing to do. Using a null StringRef as the method to cause a return to the initial shift state would be consistent with the iconv interface.

107

Same comment about null termination and shift state.

[hubert.reinterpretcast]

llvm/unittests/Support/CharSetTest.cpp

130

The other cases of "identity conversion" look like they would have suspicious behaviour. If they do, then this test is insufficient.

191

There is no representation in the testing of stateful encodings.

Reasonable tests (separately for ISO-2022-JP and IBM-939) include:

• "Returning to the initial shift state" when in the initial shift state generates an empty output sequence.
• "Returning to the initial shift state" after the previous conversion ended with a character that requires a shift from the initial shift state generates a non-empty output sequence.

[Kai]

Changes based on review comments:

• Renamed/moved enumeration members for charset names
• Using a charset alias algorithm to make using charset names easier
• Some source code style updates

[Kai]

llvm/include/llvm/Support/CharSet.h

48

Good point, thanks. It makes a lot of sense to have the names similar to proposed standards.

llvm/lib/Support/CharSet.cpp

31–36

Thanks! I added the normalization algorithm, as I think that it makes using the converter easier.

Obviously, the ISO8859-1 name comes from the z/OS implementation of iconv, which accepts 'ISO8859-1` or 819 but not the registered name ISO-8859-1.

103

Changed.

114

Thanks for catching - my fault.

156

Thanks again, I was not aware of this change.

[ZarkoCA]

llvm/include/llvm/Support/CharSet.h

33

I think enums should start with an uppercase letter as per the style guide.

43

nit: may be more clear to add the ending comment.

[hubert.reinterpretcast]

I've looked at the new changes and verified that the recent updates are an improvement. There remain unaddressed comments.

llvm/include/llvm/Support/CharSet.h

33

Thanks @ZarkoCA. The coding standards don't state this, but the existing practice is that the exception (from the coding standards) about following the C++ STL naming convention in some cases is more broadly applicable for components with a corresponding standard (or proposed standard) facility like this one.

[Kai]

• Update table-based algorithm to map Unicode characters outside the range [U+0080, U+00FF] to the SUB character. This is the same behavior as iconv.
• Add a new identify transformation which does not alter UTF-8 sequences
• Use resize_for_overwrite for memory allocation

[Kai]

I've looked at the new changes and verified that the recent updates are an improvement. There remain unaddressed comments.

Yep, I know.

[Kai]

llvm/include/llvm/Support/CharSet.h

33

To be precise, the exception is documented here: https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly (at the end of the paragraph).

llvm/lib/Support/CharSet.cpp

105

Changed implementation to have same behavior as iconv.

141

Changed, thanks.

153

Changed, thanks.

179–183

Added a no-op converter.

195

Changed.

[hubert.reinterpretcast]

llvm/lib/Support/CharSet.cpp

105

Thanks for making the change. Thinking about it a bit more, I am wondering if there should be a "policy" available for the class to support selecting one behaviour or the other (or at least identifying that there were characters with no output codeset representation encountered). For example, iconv conversions that would not round-trip are identifiable via non-error non-zero return codes (indicating the number of input characters that did not have a representation in the output codeset).

This information/policy will be useful for implementing the diagnostics required if https://wg21.link/p1854 is adopted.

[hubert.reinterpretcast]

llvm/lib/Support/CharSet.cpp

135

The comment should say that the encoded value is greater than 0xFF or the encoding is overlong. Overlong encodings are still an error. If the conversion facility has no use case for "lossy" translation, then reverting to the previous code for the UTF-8 to ISO-8859-1 decoding (and fixing the iconv path to check the return code) would save effort on getting the error handling right.

It is also an error if the encoded value ends up being not a UCS scalar value (either because it is greater then 0x10FFFF or because it encodes a surrogate code point).

139

I suggest renaming this to SeqLength and changing the comment above to say "the rest of the sequence is skipped" (if indeed that aspect survives the changes to either implement error detection or abandon lossy translation).

140

An error should also be flagged when SkipLen == 1.

[hubert.reinterpretcast]

llvm/lib/Support/CharSet.cpp

179

My first comment regarding resize_for_overwrite was for this line.

213

This is consistent with convertWithTable but not convertWithIconv with respect to whether Result is being appended to as opposed to it having its contents replaced. All three should be consistent.

llvm/unittests/Support/CharSetTest.cpp

43

Comment should indicate that this is the substitution character.

[Kai]

• only support reversible conversions
• make sure that all converters behave in the same way
• updated comments

[Kai]

llvm/include/llvm/Support/CharSet.h

98

I updated the comment (and also the implementation) that an empty string resets the shift state. However, this feels a bit clumsy, and leaks a lot of the details of the iconv interface. So I wonder if a new method, e.g. resetState() would not be the better approach.

llvm/lib/Support/CharSet.cpp

135

Ah, I see your point. I reverted the error handling here, and check the return code from iconv() instead below. There is no real use case for it. I

152

Changed the handling to detect overflows.

179

Sorry, I noted this later, too.

213

Thanks for catching this! All three converters should now behave in the same way, that is, replacing the content.

[hubert.reinterpretcast]

llvm/CMakeLists.txt

380

Minor nit: s/for for/for;

llvm/include/llvm/Support/CharSet.h

43

Minor comment still to address here.

77

Minor nit: s/occuor/occur/;

91

If the current object had a cleanup action on entry to the move assignment operator, then that cleanup action should be run prior to replacing the value.

98

I think that empty string is maybe not unique enough for this operation, so yes, a new method is probably better. However, it may be useful to understand how using StringRef("", 1u) as the input fits in.

108

Here too: s/occuor/occur/;

llvm/lib/Support/CharSet.cpp

136

164

It may be useful for the client of the interface to be able to retrieve the shift sequence (if any) to the initial shift state from the current internal output shift state. Although that use case might be achieved by using StringRef("", 1u).

[hubert.reinterpretcast]

llvm/lib/Support/CharSet.cpp

198–200

I don't think there's any "partially converted input characters" to flush. There's only shift sequences that may be unnecessary or unwanted.

204–206

I think forcing return to the initial shift state at this level is too rigid.

As mentioned in https://reviews.llvm.org/D88741?id=379158#inline-1064326, the recommended behaviour for numeric escapes involves no return to the initial shift state prior to the injection of the values specified.

Whether Clang follows that recommendation or not for one platform or the other is a higher level question that should be had in the context of a client of this facility. The facility itself should leave the possibility of resuming with the prior translation's consequent shift state.

[hubert.reinterpretcast]

llvm/include/llvm/Support/CharSet.h

103

I think "string" is too ambiguous. Additional sentences to clarify would help:
Converts the contents of a StringRef, which conventionally does not include a terminating null character. No additional null termination of the result is attempted.

128

Can refer to the other function to note clarifications re: null termination.

[hubert.reinterpretcast]

@Kai, I've completed my review of the current state of this patch. I am out-of-office for three weeks. I hope my comments are clear enough to address and for other reviewers to confirm action/response on.

llvm/unittests/Support/CharSetTest.cpp

191

I suggest trying to write these tests.

[Kai]

Fix a couple of review comments.

[Kai]

llvm/CMakeLists.txt

380

Changed.

llvm/include/llvm/Support/CharSet.h

43

Comment added.

77

Fixed. Thanks for catching!

91

Call to cleanup added.

108

Fixed, too.

llvm/lib/Support/CharSet.cpp

136

Changed.

[hubert.reinterpretcast]

The new changes are improvements. The comments regarding null termination and shift state are still pending resolution.

[Kai]

Rewrite based on review comments

• New class hierarchy using pimpl idiom
• New handling for flushing

[cor3ntin]

Thanks for working on this.

Before starting a more in depth review on that, I think this is big enough that we want to see an RFC with wider consensus and interest from the community as far as maintaining this.
I know of https://discourse.llvm.org/t/rfc-adding-a-char-set-converter-to-support-library/56592/2 and https://discourse.llvm.org/t/rfc-adding-a-char-set-converter-to-support-library/56574 , but neither gather a lot of comments.

I would like to better understand the set of circumstances in which the system iconv's IBM1047 transcoder would not be suitable for use by llvm, on the systems that make use of that encoding.
Maintaining these tables won't be free. Are the discrepancies regarding LF handling something you expect to cause issues in practice?

I am of the opinion that we should rely on iconv as much as possible, as i do not think maintaining conversion table will be a good use of our resources, however i think people might have widely different opinions and you will want
to make sure there is community buy-in on that point.

We should note that GCC has had good success using iconv, and I don't think the platform-specific availability of encondings has been an issue in practice but it's worth addressing. I know some people mentioned that point previously.
IMO, Any user confusion in that regard could be addressed by providing a sufficiently high-quality diagnostic in the presence of unsupported encoding.

If we do use iconv though, i would like us to have a better understanding of use cases, The patch currently links iconv to all llvm libraries, which might be overkill if the only project using it is Clang, and I wonder how that affects packaging
on linux distributions.

Speaking of, maybe the LLVM foundation will be able to provide legal opinions on linking to iconv on various platforms. Maybe something @tstellar would be able to help with. I would not expect particular challenges but we
should to make sure.

@aaron.ballman also reminded me that if iconv is generally available on posix-ish platforms, it is usually not available on Windows. Building iconv-like facilities on top of win32 APIs may be challenging, but we might want to think about
ways to allow or facilitate the use of iconv in Windows build. At least i would like us to give some consideration to that point, Indeed, Windows would, logically be the other big motivation to teach Clang about encodings beyond UTF-8.

Are there any other supported platforms for which iconv availability would be a concern?

I think all of these points should be addressed in an RFC so that we have a clear plan moving forward.

On a slightly more technical aspects, I'm concerned about the attempt at providing genericity in the CharSetConverterTable class - as Hubert alluded to, a table of bytes only works for specific, stateless, single byte encoding that have a reasonable mapping to one another.
If we do agree that we want a homegrown IBM1047<->UTF-8 conversion class, maybe we should have a class that does only that instead of trying to future proof and expect to add more tables. As i hope that we won't try to support more encoding without iconv.

CharSetConverterIdentity will never be a very efficient thing to use as implemented as it will perform a copy that may not be be useful. Have you consider either asserting that the encoding are distincts, or providing an error in that case instead of making a no-op copying converter?

llvm/include/llvm/Support/CharSet.h

94

Please use std::unique_ptr here, that way you won't have to manually manage memory in the destructor and move assignment operator

[Kai]

• Use std::unique_ptr
• clang-format the file

[aaron.ballman]

In D88741#4232746, @cor3ntin wrote:

Thanks for working on this.

Before starting a more in depth review on that, I think this is big enough that we want to see an RFC with wider consensus and interest from the community as far as maintaining this.
I know of https://discourse.llvm.org/t/rfc-adding-a-char-set-converter-to-support-library/56592/2 and https://discourse.llvm.org/t/rfc-adding-a-char-set-converter-to-support-library/56574 , but neither gather a lot of comments.

+1, neither of those RFCs really showed a strong community consensus behind the idea, so I think another RFC would be appropriate. Please be sure to address the items in https://clang.llvm.org/get_involved.html#criteria explicitly in the RFC where appropriate.

I am of the opinion that we should rely on iconv as much as possible, as i do not think maintaining conversion table will be a good use of our resources, however i think people might have widely different opinions and you will want
to make sure there is community buy-in on that point.

We should note that GCC has had good success using iconv, and I don't think the platform-specific availability of encondings has been an issue in practice but it's worth addressing. I know some people mentioned that point previously.
IMO, Any user confusion in that regard could be addressed by providing a sufficiently high-quality diagnostic in the presence of unsupported encoding.

If we do use iconv though, i would like us to have a better understanding of use cases, The patch currently links iconv to all llvm libraries, which might be overkill if the only project using it is Clang, and I wonder how that affects packaging
on linux distributions.

Speaking of, maybe the LLVM foundation will be able to provide legal opinions on linking to iconv on various platforms. Maybe something @tstellar would be able to help with. I would not expect particular challenges but we
should to make sure.

Agreed -- any time we bring in a third-party dependency, we should be going through an explicit license review just to make sure we're in the clear.

@aaron.ballman also reminded me that if iconv is generally available on posix-ish platforms, it is usually not available on Windows. Building iconv-like facilities on top of win32 APIs may be challenging, but we might want to think about
ways to allow or facilitate the use of iconv in Windows build. At least i would like us to give some consideration to that point, Indeed, Windows would, logically be the other big motivation to teach Clang about encodings beyond UTF-8.

This is my big concern. iconv isn't installed on Windows by default, so this means we either need to dynamically link against iconv (which means Windows users now have extra dependencies they need to deal with and this can negatively impact downstream projects that ship an iconv version because of versioning differences) or we need to statically link against iconv (which means Windows users now have a significantly larger binary to deal with, and it still can negatively impact downstream projects that also statically link against iconv). That said, I also don't think we want to try to replace iconv with our own set of APIs. This should be explicitly discussed in the new RFC.

Are there any other supported platforms for which iconv availability would be a concern?

I think all of these points should be addressed in an RFC so that we have a clear plan moving forward.

+1

[cor3ntin]

@aaron.ballman because of the LGPL nature of iconv we night be restricted in our ability to link iconv statistically (but IANAL).
However, we might actually have options there. ICU is generally available on windows https://learn.microsoft.com/en-us/windows/win32/intl/international-components-for-unicode--icu- - that's something that could be investiguated.
ICU converter API is similar to that of iconv https://unicode-org.github.io/icu/userguide/conversion/converters.html#creating-a-converter so, it could be surfaced the same way (ie through a name), both libs use a (potentially superset) of IANA names

[Everybody0523]

CharSetConverterIdentity will never be a very efficient thing to use as implemented as it will perform a copy that may not be be useful. Have you consider either asserting that the encoding are distincts, or providing an error in that case instead of making a no-op copying converter?

We did consider that, but ultimately decided against it. The reason for that is so we are able to create a create method that is guaranteed to return a functional converter (ie. is guaranteed to not throw an error). However, if it is greatly preferable to have it assert/error in the event of an "identity" conversion, we can make that change too.

[Everybody0523]

Based on comments here and on the new RFC, we've decided to remove the use of iconv and limit this patch to supporting conversions between EBCDIC and UTF-8.

[Kai]

Strip down implementation to just having 2 functions to convert between EBCDIC-1047 and UTF-8.

[Kai]

Fix error in unit test configuration.

llvm/CMakeLists.txt

 if (CMAKE_SYSTEM_NAME MATCHES "OS390")
   option(LLVM_ENABLE_THREADS "Use threads if available." OFF)
 else()
   option(LLVM_ENABLE_THREADS "Use threads if available." ON)
 endif()
 
 set(LLVM_ENABLE_ZLIB "ON" CACHE STRING "Use zlib for compression/decompression if available. Can be ON, OFF, or FORCE_ON")
 
+set(LLVM_ENABLE_ICONV "ON" CACHE STRING "Use iconv for for character set conversion if available. Can be ON, OFF, or FORCE_ON")
+
 set(LLVM_Z3_INSTALL_DIR "" CACHE STRING "Install directory of the Z3 solver.")
 
 option(LLVM_ENABLE_Z3_SOLVER
   "Enable Support for the Z3 constraint solver in LLVM."
   ${LLVM_ENABLE_Z3_SOLVER_DEFAULT}
 )
 
 if (LLVM_ENABLE_Z3_SOLVER)

llvm/cmake/config-ix.cmake

else()
   set(LLVM_ENABLE_TERMINFO 0)
 endif()
 
 check_library_exists(xar xar_open "" LLVM_HAVE_LIBXAR)
 if(LLVM_HAVE_LIBXAR)
   set(XAR_LIB xar)
 endif()
 
+# Check for iconv.
+if(LLVM_ENABLE_ICONV)
+  if(LLVM_ENABLE_ICONV STREQUAL FORCE_ON)
+    find_package(Iconv REQUIRED)
+  elseif(NOT LLVM_USE_SANITIZER MATCHES "Memory.*")
+    find_package(Iconv)
+  endif()
+  if(Iconv_FOUND)
+    # Check if iconv we found is usable; for example, we may have found a 32-bit
+    # library on a 64-bit system which would result in a link-time failure.
+    cmake_push_check_state()
+    list(APPEND CMAKE_REQUIRED_INCLUDES ${Iconv_INCLUDE_DIRS})
+    list(APPEND CMAKE_REQUIRED_LIBRARIES ${Iconv_LIBRARIES})
+    check_symbol_exists(iconv_open iconv.h HAVE_ICONV)
+    cmake_pop_check_state()
+    if(LLVM_ENABLE_ICONV STREQUAL FORCE_ON AND NOT HAVE_ICONV)
+      message(FATAL_ERROR "Failed to configure iconv")
+    endif()
+  endif()
+  set(LLVM_ENABLE_ICONV "${HAVE_ICONV}")
+endif()
+
 # function checks
 check_symbol_exists(arc4random "stdlib.h" HAVE_DECL_ARC4RANDOM)
 find_package(Backtrace)
 set(HAVE_BACKTRACE ${Backtrace_FOUND})
 set(BACKTRACE_HEADER ${Backtrace_HEADER})
 
 # Prevent check_symbol_exists from using API that is not supported for a given
 # deployment target.

llvm/cmake/modules/LLVMConfig.cmake.in

 
 set(LLVM_ENABLE_UNWIND_TABLES @LLVM_ENABLE_UNWIND_TABLES@)
 
 set(LLVM_ENABLE_ZLIB @LLVM_ENABLE_ZLIB@)
 if(LLVM_ENABLE_ZLIB)
   find_package(ZLIB)
 endif()
 
+set(LLVM_ENABLE_ICONV @LLVM_ENABLE_ICONV@)
+if(LLVM_ENABLE_ICONV)
+  find_package(Iconv)
+endif()
+
 set(LLVM_ENABLE_LIBXML2 @LLVM_ENABLE_LIBXML2@)
 if(LLVM_ENABLE_LIBXML2)
   find_package(LibXml2)
 endif()
 
 set(LLVM_WITH_Z3 @LLVM_WITH_Z3@)
 
 set(LLVM_ENABLE_DIA_SDK @LLVM_ENABLE_DIA_SDK@)

llvm/docs/CMake.rst

 **LLVM_ENABLE_FFI**:BOOL
   Indicates whether the LLVM Interpreter will be linked with the Foreign Function
   Interface library (libffi) in order to enable calling external functions.
   If the library or its headers are installed in a custom
   location, you can also set the variables FFI_INCLUDE_DIR and
   FFI_LIBRARY_DIR to the directories where ffi.h and libffi.so can be found,
   respectively. Defaults to OFF.
 
+**LLVM_ENABLE_ICONV**:BOOL
+  Enable building with iconv library to support conversion between different
+  character sets.
+  Defaults to ON.
+
 **LLVM_ENABLE_IDE**:BOOL
   Tell the build system that an IDE is being used. This in turn disables the
   creation of certain convenience build system targets, such as the various
   ``install-*`` and ``check-*`` targets, since IDEs don't always deal well with
   a large number of targets. This is usually autodetected, but it can be
   configured manually to explicitly control the generation of those targets. One
   scenario where a manual override may be desirable is when using Visual Studio
   2017's CMake integration, which would not be detected as an IDE otherwise.

llvm/include/llvm/Config/config.h.cmake

 
 /* Target triple LLVM will generate code for by default */
 /* Doesn't use `cmakedefine` because it is allowed to be empty. */
 #define LLVM_DEFAULT_TARGET_TRIPLE "${LLVM_DEFAULT_TARGET_TRIPLE}"
 
 /* Define if zlib compression is available */
 #cmakedefine01 LLVM_ENABLE_ZLIB
 
+/* Define if iconv library is available */
+#cmakedefine01 LLVM_ENABLE_ICONV
+
 /* Define if overriding target triple is enabled */
 #cmakedefine LLVM_TARGET_TRIPLE_ENV "${LLVM_TARGET_TRIPLE_ENV}"
 
 /* LLVM version information */
 #cmakedefine LLVM_VERSION_INFO "${LLVM_VERSION_INFO}"
 
 /* Whether tools show host and target info when invoked with --version */
 #cmakedefine01 LLVM_VERSION_PRINTER_SHOW_HOST_TARGET_INFO

llvm/include/llvm/Support/CharSet.h

+//===-- CharSet.h - Utility class to convert between char sets ----*- C++ -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file provides a utility class to convert between different character
+/// set encodings.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_CHARSET_H
+#define LLVM_SUPPORT_CHARSET_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/config.h"
+#include "llvm/Support/ErrorOr.h"
+
+#include <functional>
+#include <string>
+#include <system_error>
+
+namespace llvm {
+
+template <typename T> class SmallVectorImpl;
+
+/// Utility class to convert between different character set encodings.
+/// The class always supports converting between EBCDIC 1047 and Latin-1/UTF-8.
+/// If the iconv library is available, then arbitrary conversions are supported.
+/// TODO Add Windows support.
+class CharSetConverter {
+public:
+  using ConverterFunc =
+      std::function<std::error_code(StringRef, SmallVectorImpl<char> &)>;
+  using CleanupFunc = std::function<void(void)>;
+
+private:
+  ConverterFunc Convert;
+  CleanupFunc Cleanup;
+
+public:
+  enum CharSetNames {
+    /// UTF-8 character set encoding.
+    CS_UTF8,
+
+    /// ISO 8859-1 (Latin-1) character set encoding.
+    CS_LATIN1,
+
+    /// IBM EBCDIC 1047 character set encoding.
+    CS_IBM1047
+  };
+
+private:
+  CharSetConverter(ConverterFunc Convert, CleanupFunc Cleanup)
+      : Convert(Convert), Cleanup(Cleanup) {}
+
+public:
+  /// Creates a CharSetConverter instance.
+  /// \param[in] CSFrom name of the source character encoding
+  /// \param[in] CSTo name of the target character encoding
+  /// \return a CharSetConverter instance
+  static CharSetConverter create(CharSetNames CSFrom, CharSetNames CSTo);
+
+  /// Creates a CharSetConverter instance.
+  /// Returns std::errc::invalid_argument in case the requested conversion is
+  /// not supported.
+  /// \param[in] CSFrom name of the source character encoding
+  /// \param[in] CSTo name of the target character encoding
+  /// \return a CharSetConverter instance or an error code
+  static ErrorOr<CharSetConverter> create(StringRef CSFrom, StringRef CSTo);
+
+  CharSetConverter(const CharSetConverter &) = delete;
+  CharSetConverter &operator=(const CharSetConverter &) = delete;
+
+  CharSetConverter(CharSetConverter &&Other) {
+    this->Convert = Other.Convert;
+    this->Cleanup = Other.Cleanup;
+    Other.Cleanup = nullptr;
+  }
+
+  CharSetConverter &operator=(CharSetConverter &&Other) {
+    this->Convert = Other.Convert;
+    this->Cleanup = Other.Cleanup;
+    Other.Cleanup = nullptr;
+    return *this;
+  }
+
+  ~CharSetConverter() {
+    if (Cleanup)
+      Cleanup();
+  }
+
+  /// Converts a string.
+  /// \param[in] Source source string
+  /// \param[in,out] Result container for converted string
+  /// \return error code in case something went wrong
+  std::error_code convert(StringRef Source,
+                          SmallVectorImpl<char> &Result) const {
+    return Convert(Source, Result);
+  }
+
+  /// Converts a string.
+  /// \param[in] Source source string
+  /// \param[in,out] Result container for converted string
+  /// \return error code in case something went wrong
+  std::error_code convert(const std::string &Source,
+                          SmallVectorImpl<char> &Result) const {
+    return convert(StringRef(Source), Result);
+  }
+};
+
+} // namespace llvm
+
+#endif

llvm/lib/Support/CMakeLists.txt

if (MSVC)
   set (delayload_flags delayimp -delayload:shell32.dll -delayload:ole32.dll)
 endif()
 
 # Link Z3 if the user wants to build it.
 if(LLVM_WITH_Z3)
   set(system_libs ${system_libs} ${Z3_LIBRARIES})
 endif()
 
+# Link iconv library if it is an external library.
+if(LLVM_ENABLE_ICONV AND NOT Iconv_IS_BUILT_IN)
+  set(system_libs ${system_libs} ${Iconv_LIBRARIES})
+endif()
+
 # Override the C runtime allocator on Windows and embed it into LLVM tools & libraries
 if(LLVM_INTEGRATED_CRT_ALLOC)
   if (CMAKE_BUILD_TYPE AND NOT ${LLVM_USE_CRT_${uppercase_CMAKE_BUILD_TYPE}} MATCHES "^(MT|MTd)$")
     message(FATAL_ERROR "LLVM_INTEGRATED_CRT_ALLOC only works with /MT or /MTd. Use LLVM_USE_CRT_${uppercase_CMAKE_BUILD_TYPE} to set the appropriate option.")
   endif()
 
   string(REGEX REPLACE "(/|\\\\)$" "" LLVM_INTEGRATED_CRT_ALLOC "${LLVM_INTEGRATED_CRT_ALLOC}")
 
add_llvm_component_library(LLVMSupport
   BinaryStreamReader.cpp
   BinaryStreamRef.cpp
   BinaryStreamWriter.cpp
   BlockFrequency.cpp
   BranchProbability.cpp
   BuryPointer.cpp
   CachePruning.cpp
   circular_raw_ostream.cpp
+  CharSet.cpp
   Chrono.cpp
   COM.cpp
   CodeGenCoverage.cpp
   CommandLine.cpp
   Compression.cpp
   CRC.cpp
   ConvertUTF.cpp
   ConvertUTFWrapper.cpp
if(LLVM_INTEGRATED_CRT_ALLOC MATCHES "snmalloc$")
     add_definitions(-D_SILENCE_CXX17_ITERATOR_BASE_CLASS_DEPRECATION_WARNING)
     if ("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang" AND
         "${CMAKE_SYSTEM_PROCESSOR}" MATCHES "x86_64")
       set_property(TARGET LLVMSupport PROPERTY COMPILE_FLAGS "-mcx16")
     endif()
   endif()
 endif()
 
+# Add include dir for iconv library if it is an external library.
+if(LLVM_ENABLE_ICONV AND NOT Iconv_IS_BUILT_IN)
+  target_include_directories(LLVMSupport SYSTEM
+    PRIVATE
+    ${Iconv_INCLUDE_DIRS}
+    )
+endif()
+
 if(LLVM_WITH_Z3)
   target_include_directories(LLVMSupport SYSTEM
     PRIVATE
     ${Z3_INCLUDE_DIR}
     )
 endif()

llvm/lib/Support/CharSet.cpp

+//===-- CharSet.cpp - Utility class to convert between char sets --*- C++ -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file provides utility classes to convert between different character
+/// set encoding.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/CharSet.h"
+#include "llvm/ADT/SmallVector.h"
+#include <algorithm>
+#include <system_error>
+
+#if LLVM_ENABLE_ICONV
+#include <iconv.h>
+#endif
+
+using namespace llvm;
+
+namespace {
+
+// Maps the charset name to enum constant if possible.
+Optional<CharSetConverter::CharSetNames> getKnownCharSet(StringRef CSName) {
+#define CSNAME(CS, STR)                                                        \
+  if (CSName == STR)                                                           \
+  return CS
+  CSNAME(CharSetConverter::CS_UTF8, "UTF-8");
+  CSNAME(CharSetConverter::CS_LATIN1, "ISO8859-1");
+  CSNAME(CharSetConverter::CS_IBM1047, "IBM-1047");
+#undef CSNAME
+  return None;
+}
+
+// Character conversion between Enhanced ASCII and EBCDIC (IBM-1047).
+const unsigned char ISO88591ToIBM1047[256] = {
+    0x00, 0x01, 0x02, 0x03, 0x37, 0x2d, 0x2e, 0x2f, 0x16, 0x05, 0x15, 0x0b,
+    0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x3c, 0x3d, 0x32, 0x26,
+    0x18, 0x19, 0x3f, 0x27, 0x1c, 0x1d, 0x1e, 0x1f, 0x40, 0x5a, 0x7f, 0x7b,
+    0x5b, 0x6c, 0x50, 0x7d, 0x4d, 0x5d, 0x5c, 0x4e, 0x6b, 0x60, 0x4b, 0x61,
+    0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8, 0xf9, 0x7a, 0x5e,
+    0x4c, 0x7e, 0x6e, 0x6f, 0x7c, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
+    0xc8, 0xc9, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xe2,
+    0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xad, 0xe0, 0xbd, 0x5f, 0x6d,
+    0x79, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x91, 0x92,
+    0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6,
+    0xa7, 0xa8, 0xa9, 0xc0, 0x4f, 0xd0, 0xa1, 0x07, 0x20, 0x21, 0x22, 0x23,
+    0x24, 0x25, 0x06, 0x17, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x09, 0x0a, 0x1b,
+    0x30, 0x31, 0x1a, 0x33, 0x34, 0x35, 0x36, 0x08, 0x38, 0x39, 0x3a, 0x3b,
+    0x04, 0x14, 0x3e, 0xff, 0x41, 0xaa, 0x4a, 0xb1, 0x9f, 0xb2, 0x6a, 0xb5,
+    0xbb, 0xb4, 0x9a, 0x8a, 0xb0, 0xca, 0xaf, 0xbc, 0x90, 0x8f, 0xea, 0xfa,
+    0xbe, 0xa0, 0xb6, 0xb3, 0x9d, 0xda, 0x9b, 0x8b, 0xb7, 0xb8, 0xb9, 0xab,
+    0x64, 0x65, 0x62, 0x66, 0x63, 0x67, 0x9e, 0x68, 0x74, 0x71, 0x72, 0x73,
+    0x78, 0x75, 0x76, 0x77, 0xac, 0x69, 0xed, 0xee, 0xeb, 0xef, 0xec, 0xbf,
+    0x80, 0xfd, 0xfe, 0xfb, 0xfc, 0xba, 0xae, 0x59, 0x44, 0x45, 0x42, 0x46,
+    0x43, 0x47, 0x9c, 0x48, 0x54, 0x51, 0x52, 0x53, 0x58, 0x55, 0x56, 0x57,
+    0x8c, 0x49, 0xcd, 0xce, 0xcb, 0xcf, 0xcc, 0xe1, 0x70, 0xdd, 0xde, 0xdb,
+    0xdc, 0x8d, 0x8e, 0xdf};
+
+const unsigned char IBM1047ToISO88591[256] = {
+    0x00, 0x01, 0x02, 0x03, 0x9c, 0x09, 0x86, 0x7f, 0x97, 0x8d, 0x8e, 0x0b,
+    0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x9d, 0x0a, 0x08, 0x87,
+    0x18, 0x19, 0x92, 0x8f, 0x1c, 0x1d, 0x1e, 0x1f, 0x80, 0x81, 0x82, 0x83,
+    0x84, 0x85, 0x17, 0x1b, 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x05, 0x06, 0x07,
+    0x90, 0x91, 0x16, 0x93, 0x94, 0x95, 0x96, 0x04, 0x98, 0x99, 0x9a, 0x9b,
+    0x14, 0x15, 0x9e, 0x1a, 0x20, 0xa0, 0xe2, 0xe4, 0xe0, 0xe1, 0xe3, 0xe5,
+    0xe7, 0xf1, 0xa2, 0x2e, 0x3c, 0x28, 0x2b, 0x7c, 0x26, 0xe9, 0xea, 0xeb,
+    0xe8, 0xed, 0xee, 0xef, 0xec, 0xdf, 0x21, 0x24, 0x2a, 0x29, 0x3b, 0x5e,
+    0x2d, 0x2f, 0xc2, 0xc4, 0xc0, 0xc1, 0xc3, 0xc5, 0xc7, 0xd1, 0xa6, 0x2c,
+    0x25, 0x5f, 0x3e, 0x3f, 0xf8, 0xc9, 0xca, 0xcb, 0xc8, 0xcd, 0xce, 0xcf,
+    0xcc, 0x60, 0x3a, 0x23, 0x40, 0x27, 0x3d, 0x22, 0xd8, 0x61, 0x62, 0x63,
+    0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0xab, 0xbb, 0xf0, 0xfd, 0xfe, 0xb1,
+    0xb0, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, 0x70, 0x71, 0x72, 0xaa, 0xba,
+    0xe6, 0xb8, 0xc6, 0xa4, 0xb5, 0x7e, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78,
+    0x79, 0x7a, 0xa1, 0xbf, 0xd0, 0x5b, 0xde, 0xae, 0xac, 0xa3, 0xa5, 0xb7,
+    0xa9, 0xa7, 0xb6, 0xbc, 0xbd, 0xbe, 0xdd, 0xa8, 0xaf, 0x5d, 0xb4, 0xd7,
+    0x7b, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0xad, 0xf4,
+    0xf6, 0xf2, 0xf3, 0xf5, 0x7d, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50,
+    0x51, 0x52, 0xb9, 0xfb, 0xfc, 0xf9, 0xfa, 0xff, 0x5c, 0xf7, 0x53, 0x54,
+    0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0xb2, 0xd4, 0xd6, 0xd2, 0xd3, 0xd5,
+    0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0xb3, 0xdb,
+    0xdc, 0xd9, 0xda, 0x9f};
+
+enum { NoUTF8 = 0x0, SrcIsUTF8 = 0x1, DstIsUTF8 = 0x2 };
+
+std::error_code convertWithTable(const unsigned char *Table, unsigned Flags,
+                                 StringRef Source,
+                                 SmallVectorImpl<char> &Result) {
+  const unsigned char *Ptr =
+      reinterpret_cast<const unsigned char *>(Source.data());
+  size_t Length = Source.size();
+  while (Length--) {
+    unsigned char Ch = *Ptr++;
+    // Handle UTF-8 2-byte-sequences in input.
+    if (Flags & SrcIsUTF8) {
+      if (Ch >= 128) {
+        // Only two-byte sequences can be decoded.
+        if (Ch != 0xc2 && Ch != 0xc3)
+          return std::make_error_code(std::errc::illegal_byte_sequence);
+        // Is buffer truncated?
+        if (!Length)
+          return std::make_error_code(std::errc::invalid_argument);
+        unsigned char Ch2 = *Ptr++;
+        // Is second byte well-formed?
+        if ((Ch2 & 0xc0) != 0x80)
+          return std::make_error_code(std::errc::illegal_byte_sequence);
+        Ch = Ch2 | (Ch << 6);
+        Length--;
+      }
+    }
+    // Translate the character.
+    Ch = Table ? Table[Ch] : Ch;
+    // Handle UTF-8 2-byte-sequences in output.
+    if (Flags & DstIsUTF8) {
+      if (Ch >= 128) {
+        // First byte prefixed with either 0xc2 or 0xc3.
+        Result.push_back(static_cast<char>(0xc0 | (Ch >> 6)));
+        // Second byte is either the same as the ASCII byte or ASCII byte -64.
+        Ch = Ch & 0xbf;
+      }
+    }
+    Result.push_back(static_cast<char>(Ch));
+  }
+  return std::error_code();
+}
+
+#if LLVM_ENABLE_ICONV
+std::error_code convertWithIconv(iconv_t ConvDesc, StringRef Source,
+                                 SmallVectorImpl<char> &Result) {
+  // Setup the input.
+  size_t InputLength = Source.size();
+  char *Input = const_cast<char *>(Source.data());
+  // Setup the output. We directly write into the SmallVector.
+  size_t Capacity = Result.capacity();
+  Result.resize(Capacity);
+  char *Output = static_cast<char *>(Result.data());
+  size_t OutputLength = Capacity;
+
+  while (iconv(ConvDesc, &Input, &InputLength, &Output, &OutputLength) ==
+         static_cast<size_t>(-1)) {
+    if (errno == E2BIG) {
+      // No space left in output buffer. Double the size of the underlying
+      // memory in the SmallVectorImpl, adjust pointer and length and continue
+      // the conversion.
+      const size_t Used = Capacity - OutputLength;
+      Capacity *= 2;
+      Result.resize(Capacity);
+      Output = static_cast<char *>(Result.data()) + Used;
+      OutputLength = Capacity - Used;
+    } else
+      // Some other error occured.
+      return std::error_code(errno, std::generic_category());
+  }
+
+  // Re-adjust size to actual size.
+  Result.resize(Capacity - OutputLength);
+  return std::error_code();
+}
+#endif
+} // namespace
+
+CharSetConverter CharSetConverter::create(CharSetNames CSFrom,
+                                          CharSetNames CSTo) {
+  unsigned Flags = NoUTF8;
+  if (CSFrom == CS_UTF8)
+    Flags |= SrcIsUTF8;
+  if (CSTo == CS_UTF8)
+    Flags |= DstIsUTF8;
+  const unsigned char *Table = nullptr;
+  if (CSFrom == CS_IBM1047)
+    Table = IBM1047ToISO88591;
+  if (CSTo == CS_IBM1047)
+    Table = ISO88591ToIBM1047;
+  return CharSetConverter{
+      [Table, Flags](StringRef Source, SmallVectorImpl<char> &Result) {
+        return convertWithTable(Table, Flags, Source, Result);
+      },
+      nullptr};
+}
+
+ErrorOr<CharSetConverter> CharSetConverter::create(StringRef CSFrom,
+                                                   StringRef CSTo) {
+  Optional<CharSetConverter::CharSetNames> From = getKnownCharSet(CSFrom);
+  Optional<CharSetConverter::CharSetNames> To = getKnownCharSet(CSTo);
+  if (From && To)
+    return create(*From, *To);
+#if LLVM_ENABLE_ICONV
+  iconv_t ConvDesc = iconv_open(CSTo.str().c_str(), CSFrom.str().c_str());
+  if (ConvDesc == reinterpret_cast<iconv_t>(-1))
+    return std::error_code(errno, std::generic_category());
+  return CharSetConverter{
+      [ConvDesc](StringRef Source, SmallVectorImpl<char> &Result) {
+        return convertWithIconv(ConvDesc, Source, Result);
+      },
+      [ConvDesc]() { iconv_close(ConvDesc); }};
+#endif
+  return std::make_error_code(std::errc::invalid_argument);
+}
 No newline at end of file

llvm/unittests/Support/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 add_llvm_unittest(SupportTests
   AlignmentTest.cpp
   AlignOfTest.cpp
   AllocatorTest.cpp
   AnnotationsTest.cpp
   ARMAttributeParser.cpp
   ArrayRecyclerTest.cpp
   Base64Test.cpp
   BinaryStreamTest.cpp
   BlockFrequencyTest.cpp
   BranchProbabilityTest.cpp
   CachePruningTest.cpp
+  CharSetTest.cpp
   CrashRecoveryTest.cpp
   Casting.cpp
   CheckedArithmeticTest.cpp
   Chrono.cpp
   CommandLineTest.cpp
   CompressionTest.cpp
   ConvertUTFTest.cpp
   CRCTest.cpp

llvm/unittests/Support/CharSetTest.cpp

+//===- unittests/Support/CharSetTest.cpp - Charset conversion tests -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/CharSet.h"
+#include "llvm/ADT/SmallString.h"
+#include "gtest/gtest.h"
+using namespace llvm;
+
+namespace {
+
+// String "Hello World!"
+static const char HelloA[] =
+    "\x48\x65\x6C\x6C\x6F\x20\x57\x6F\x72\x6C\x64\x21\x0a";
+static const char HelloE[] =
+    "\xC8\x85\x93\x93\x96\x40\xE6\x96\x99\x93\x84\x5A\x15";
+
+// String "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+static const char ABCStrA[] =
+    "\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4A\x4B\x4C\x4D\x4E\x4F\x50\x51\x52"
+    "\x53\x54\x55\x56\x57\x58\x59\x5A\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6A"
+    "\x6B\x6C\x6D\x6E\x6F\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7A";
+static const char ABCStrE[] =
+    "\xC1\xC2\xC3\xC4\xC5\xC6\xC7\xC8\xC9\xD1\xD2\xD3\xD4\xD5\xD6\xD7\xD8\xD9"
+    "\xE2\xE3\xE4\xE5\xE6\xE7\xE8\xE9\x81\x82\x83\x84\x85\x86\x87\x88\x89\x91"
+    "\x92\x93\x94\x95\x96\x97\x98\x99\xA2\xA3\xA4\xA5\xA6\xA7\xA8\xA9";
+
+// String "¡¢£AÄÅÆEÈÉÊaàáâãäeèéêë"
+static const char AccentUTF[] =
+    "\xc2\xa1\xc2\xa2\xc2\xa3\x41\xc3\x84\xc3\x85\xc3\x86\x45\xc3\x88\xc3\x89"
+    "\xc3\x8a\x61\xc3\xa0\xc3\xa1\xc3\xa2\xc3\xa3\xc3\xa4\x65\xc3\xa8\xc3\xa9"
+    "\xc3\xaa\xc3\xab";
+static const char AccentE[] = "\xaa\x4a\xb1\xc1\x63\x67\x9e\xc5\x74\x71\x72"
+                              "\x81\x44\x45\x42\x46\x43\x85\x54\x51\x52\x53";
+
+TEST(CharSet, FromASCII) {
+  // Hello string.
+  StringRef Src(HelloA);
+  SmallString<64> Dst;
+
+  CharSetConverter Conv = CharSetConverter::create(
+      CharSetConverter::CS_LATIN1, CharSetConverter::CS_IBM1047);
+  std::error_code EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(HelloE, static_cast<std::string>(Dst).c_str());
+
+  // ABC string.
+  Src = ABCStrA;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(ABCStrE, static_cast<std::string>(Dst).c_str());
+}
+
+TEST(CharSet, ToASCII) {
+  // Hello string.
+  StringRef Src(HelloE);
+  SmallString<64> Dst;
+
+  CharSetConverter Conv = CharSetConverter::create(CharSetConverter::CS_IBM1047,
+                                                   CharSetConverter::CS_LATIN1);
+  std::error_code EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(HelloA, static_cast<std::string>(Dst).c_str());
+
+  // ABC string.
+  Src = ABCStrE;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(ABCStrA, static_cast<std::string>(Dst).c_str());
+}
+
+TEST(CharSet, FromUTF8) {
+  // Hello string.
+  StringRef Src(HelloA);
+  SmallString<64> Dst;
+
+  CharSetConverter Conv = CharSetConverter::create(
+      CharSetConverter::CS_UTF8, CharSetConverter::CS_IBM1047);
+  std::error_code EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(HelloE, static_cast<std::string>(Dst).c_str());
+
+  // ABC string.
+  Src = ABCStrA;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(ABCStrE, static_cast<std::string>(Dst).c_str());
+
+  // Accent string.
+  Src = AccentUTF;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(AccentE, static_cast<std::string>(Dst).c_str());
+}
+
+TEST(CharSet, ToUTF8) {
+  // Hello string.
+  StringRef Src(HelloE);
+  SmallString<64> Dst;
+
+  CharSetConverter Conv = CharSetConverter::create(CharSetConverter::CS_IBM1047,
+                                                   CharSetConverter::CS_UTF8);
+  std::error_code EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(HelloA, static_cast<std::string>(Dst).c_str());
+
+  // ABC string.
+  Src = ABCStrE;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(ABCStrA, static_cast<std::string>(Dst).c_str());
+
+  // Accent string.
+  Src = AccentE;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(AccentUTF, static_cast<std::string>(Dst).c_str());
+}
+
+TEST(CharSet, Identity) {
+  // Hello string.
+  StringRef Src(HelloA);
+  SmallString<64> Dst;
+
+  CharSetConverter Conv = CharSetConverter::create(CharSetConverter::CS_LATIN1,
+                                                   CharSetConverter::CS_LATIN1);
+  std::error_code EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(HelloA, static_cast<std::string>(Dst).c_str());
+
+  // ABC string.
+  Src = ABCStrA;
+  Dst.clear();
+  EC = Conv.convert(Src, Dst);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(ABCStrA, static_cast<std::string>(Dst).c_str());
+}
+
+TEST(CharSet, RoundTrip) {
+  ErrorOr<CharSetConverter> ConvToUTF16 =
+      CharSetConverter::create("IBM-1047", "UTF-16");
+  // Stop test if conversion is not supported (no underlying iconv support).
+  if (!ConvToUTF16) {
+    ASSERT_EQ(ConvToUTF16.getError(),
+              std::make_error_code(std::errc::invalid_argument));
+    return;
+  }
+  ErrorOr<CharSetConverter> ConvToUTF32 =
+      CharSetConverter::create("UTF-16", "UTF-32");
+  // Stop test if conversion is not supported (no underlying iconv support).
+  if (!ConvToUTF32) {
+    ASSERT_EQ(ConvToUTF32.getError(),
+              std::make_error_code(std::errc::invalid_argument));
+    return;
+  }
+  ErrorOr<CharSetConverter> ConvToEBCDIC =
+      CharSetConverter::create("UTF-32", "IBM-1047");
+  // Stop test if conversion is not supported (no underlying iconv support).
+  if (!ConvToEBCDIC) {
+    ASSERT_EQ(ConvToEBCDIC.getError(),
+              std::make_error_code(std::errc::invalid_argument));
+    return;
+  }
+
+  // Setup source string.
+  char SrcStr[256];
+  for (size_t I = 0; I < 256; ++I)
+    SrcStr[I] = (I + 1) % 256;
+
+  SmallString<99> Dst1Str, Dst2Str, Dst3Str;
+
+  std::error_code EC = ConvToUTF16->convert(StringRef(SrcStr), Dst1Str);
+  EXPECT_TRUE(!EC);
+  EC = ConvToUTF32->convert(Dst1Str, Dst2Str);
+  EXPECT_TRUE(!EC);
+  EC = ConvToEBCDIC->convert(Dst2Str, Dst3Str);
+  EXPECT_TRUE(!EC);
+  EXPECT_STREQ(SrcStr, static_cast<std::string>(Dst3Str).c_str());
+}
+
+} // namespace