[PATCH] More UTF string conversion wrappers

[cameron314]

Summary

(And a missing string conversion for getenv on Win32, too.)

These new string conversion wrappers convert between std::string (of UTF-8 bytes) and std::wstring, which is particularly useful for Win32 interop.

The motivation behind this is to support a patch I'm preparing for LLDB (the first version of which is at D17107) that makes heavy use of these helper functions for better Unicode support on Windows.

[cameron314]

This patch is applicable as is to rL262844 (the tip a short while ago).

[zturner]

Sorry, I didn't see this one come through before.

Adding a few other LLVM windows people, as they are probably more qualified to review something on the LLVM side than I am.

[aaron.ballman]

llvm/trunk/include/llvm/Support/ConvertUTF.h

204

& and * bind to the identifier instead of the type in our coding conventions (here and elsewhere). Run the patch through clang-format and it should fix all of those up automatically, as well as other formatting issues in the patch.

216

Inconsistent capitalization of the function name.

llvm/trunk/lib/Support/CommandLine.cpp

793

I feel like maybe we shouldn't silently eat this sort of failure, but should either report a fatal error or assert. I don't feel strongly about it, however.

799

Same for this failure.

llvm/trunk/lib/Support/ConvertUTFWrapper.cpp

140

This function overload does not appear to be used, is it required?

[cameron314]

llvm/trunk/include/llvm/Support/ConvertUTF.h

204

Oops, missed these, will fix!

216

It was already inconsistent. I had to keep ConvertUTF8toWide (with the capital 'C' and lower 't') because it already existed and I was adding overloads.

However, all the other functions in this file (and LLVM generally, I believe) follow the camelcase-starting-with-lower style for global functions. So I tried to pick the lesser of two evils ;-)

llvm/trunk/lib/Support/CommandLine.cpp

793

Hmm, good point. I'll add asserts.

llvm/trunk/lib/Support/ConvertUTFWrapper.cpp

140

Hmm, you're right. It was used by an earlier version of my patch for LLDB, but no longer. I'll remove it.

[aaron.ballman]

Some small nits, but basically looking good.

llvm/trunk/lib/Support/CommandLine.cpp

793

Can you turn them into assert(false && "Message about what went wrong");?

llvm/trunk/lib/Support/ConvertUTFWrapper.cpp

190

You should run clang-format over this as well, the formatting isn't correct (here and elsewhere in the file).

243

Use llvm_unreachable?

[cameron314]

llvm/trunk/lib/Support/CommandLine.cpp

793

Will do.

llvm/trunk/lib/Support/ConvertUTFWrapper.cpp

190

Will do. I was hoping to avoid this since clang-format changes a lot of code that was already present as well. I'll extract just the parts that apply to my patch, I suppose.

243

Aha, didn't know it existed :-)

[aaron.ballman]

llvm/trunk/lib/Support/ConvertUTFWrapper.cpp

190

You can run clang-format over the patch file directly, too. http://clang.llvm.org/docs/ClangFormat.html#script-for-patch-reformatting

[aaron.ballman]

LGTM!

[cameron314]

Great, thanks for the review!
I don't have commit access, can someone commit this on my behalf?

llvm/trunk/include/llvm/Support/ConvertUTF.h

  * ResultPtr will not be changed, and ErrorPtr will be set to the location of
  * the first character which could not be converted.
  * \return true on success.
  */
 bool ConvertUTF8toWide(unsigned WideCharWidth, llvm::StringRef Source,
                        char *&ResultPtr, const UTF8 *&ErrorPtr);
 
 /**
+* Converts a UTF-8 StringRef to a std::wstring.
+* \return true on success.
+*/
+bool ConvertUTF8toWide(llvm::StringRef Source, std::wstring &Result);
+
+/**
+* Converts a UTF-8 C-string to a std::wstring.
+* \return true on success.
+*/
+bool ConvertUTF8toWide(const char *Source, std::wstring &Result);
+
+/**
+* Converts a std::wstring to a UTF-8 encoded std::string.
+* \return true on success.
+*/
+bool convertWideToUTF8(const std::wstring &Source, std::string &Result);
+
+
+/**
  * Convert an Unicode code point to UTF8 sequence.
  *
  * \param Source a Unicode code point.
  * \param [in,out] ResultPtr pointer to the output buffer, needs to be at least
  * \c UNI_MAX_UTF8_BYTES_PER_CODE_POINT bytes.  On success \c ResultPtr is
  * updated one past end of the converted sequence.
  *
  * \returns true on success.

  *
  * \param [in] SrcBytes A buffer of what is assumed to be UTF-16 encoded text.
  * \param [out] Out Converted UTF-8 is stored here on success.
  * \returns true on success
  */
 bool convertUTF16ToUTF8String(ArrayRef<char> SrcBytes, std::string &Out);
 
 /**
+* Converts a UTF16 string into a UTF8 std::string.
+*
+* \param [in] Src A buffer of UTF-16 encoded text.
+* \param [out] Out Converted UTF-8 is stored here on success.
+* \returns true on success
+*/
+bool convertUTF16ToUTF8String(ArrayRef<UTF16> Src, std::string &Out);
+
+/**
  * Converts a UTF-8 string into a UTF-16 string with native endianness.
  *
  * \returns true on success
  */
 bool convertUTF8ToUTF16String(StringRef SrcUTF8,
                               SmallVectorImpl<UTF16> &DstUTF16);
 
 } /* end namespace llvm */
 
 #endif
 
 /* --------------------------------------------------------------------- */
 
 #endif

llvm/trunk/lib/Support/CommandLine.cpp

 ///
 void cl::ParseEnvironmentOptions(const char *progName, const char *envVar,
                                  const char *Overview) {
   // Check args.
   assert(progName && "Program name not specified");
   assert(envVar && "Environment variable name missing");
 
   // Get the environment variable they want us to parse options out of.
+#ifdef _WIN32
+  std::wstring wenvVar;
+  if (!llvm::ConvertUTF8toWide(envVar, wenvVar)) {
+    assert(false);
+    return;
+  }
+  const wchar_t *wenvValue = _wgetenv(wenvVar.c_str());
+  if (!wenvValue)
+    return;
+  std::string envValueBuffer;
+  if (!llvm::convertWideToUTF8(wenvValue, envValueBuffer)) {
+    assert(false);
+    return;
+  }
+  const char *envValue = envValueBuffer.c_str();
+#else
   const char *envValue = getenv(envVar);
   if (!envValue)
     return;
+#endif
 
   // Get program's "name", which we wouldn't know without the caller
   // telling us.
   SmallVector<const char *, 20> newArgv;
   BumpPtrAllocator A;
   StringSaver Saver(A);
   newArgv.push_back(Saver.save(progName));
 

llvm/trunk/lib/Support/ConvertUTFWrapper.cpp

bool ConvertUTF8toWide(unsigned WideCharWidth, llvm::StringRef Source,
   } else if (WideCharWidth == 2) {
     const UTF8 *sourceStart = (const UTF8*)Source.data();
     // FIXME: Make the type of the result buffer correct instead of
     // using reinterpret_cast.
     UTF16 *targetStart = reinterpret_cast<UTF16*>(ResultPtr);
     ConversionFlags flags = strictConversion;
     result = ConvertUTF8toUTF16(
         &sourceStart, sourceStart + Source.size(),
-        &targetStart, targetStart + 2*Source.size(), flags);
+        &targetStart, targetStart + Source.size(), flags);
     if (result == conversionOK)
       ResultPtr = reinterpret_cast<char*>(targetStart);
     else
       ErrorPtr = sourceStart;
   } else if (WideCharWidth == 4) {
     const UTF8 *sourceStart = (const UTF8*)Source.data();
     // FIXME: Make the type of the result buffer correct instead of
     // using reinterpret_cast.
     UTF32 *targetStart = reinterpret_cast<UTF32*>(ResultPtr);
     ConversionFlags flags = strictConversion;
     result = ConvertUTF8toUTF32(
         &sourceStart, sourceStart + Source.size(),
-        &targetStart, targetStart + 4*Source.size(), flags);
+        &targetStart, targetStart + Source.size(), flags);
     if (result == conversionOK)
       ResultPtr = reinterpret_cast<char*>(targetStart);
     else
       ErrorPtr = sourceStart;
   }
   assert((result != targetExhausted)
          && "ConvertUTF8toUTFXX exhausted target buffer");
   return result == conversionOK;
bool convertUTF16ToUTF8String(ArrayRef<char> SrcBytes, std::string &Out) {
   }
 
   Out.resize(reinterpret_cast<char *>(Dst) - &Out[0]);
   Out.push_back(0);
   Out.pop_back();
   return true;
 }
 
+bool convertUTF16ToUTF8String(ArrayRef<UTF16> Src, std::string &Out)
+{
+  return convertUTF16ToUTF8String(
+      llvm::ArrayRef<char>(reinterpret_cast<const char *>(Src.data()),
+      Src.size() * sizeof(UTF16)), Out);
+}
+
 bool convertUTF8ToUTF16String(StringRef SrcUTF8,
                               SmallVectorImpl<UTF16> &DstUTF16) {
   assert(DstUTF16.empty());
 
   // Avoid OOB by returning early on empty input.
   if (SrcUTF8.empty()) {
     DstUTF16.push_back(0);
     DstUTF16.pop_back();
     return true;
bool convertUTF8ToUTF16String(StringRef SrcUTF8,
   }
 
   DstUTF16.resize(Dst - &DstUTF16[0]);
   DstUTF16.push_back(0);
   DstUTF16.pop_back();
   return true;
 }
 
+static_assert(sizeof(wchar_t) == 1 || sizeof(wchar_t) == 2 || sizeof(wchar_t) == 4,
+    "Expected wchar_t to be 1, 2, or 4 bytes");
+
+template<typename TResult>
+static inline bool ConvertUTF8toWideInternal(
+  llvm::StringRef Source, TResult &Result) {
+  // Even in the case of UTF-16, the number of bytes in a UTF-8 string is
+  // at least as large as the number of elements in the resulting wide
+  // string, because surrogate pairs take at least 4 bytes in UTF-8.
+  Result.resize(Source.size() + 1);
+  char *ResultPtr = reinterpret_cast<char *>(&Result[0]);
+  const UTF8 *ErrorPtr;
+  if (!ConvertUTF8toWide(sizeof(wchar_t), Source, ResultPtr, ErrorPtr))
+  {
+    Result.clear();
+    return false;
+  }
+  Result.resize(reinterpret_cast<wchar_t *>(ResultPtr) - &Result[0]);
+  return true;
+}
+
+bool ConvertUTF8toWide(llvm::StringRef Source, std::wstring &Result) {
+  return ConvertUTF8toWideInternal(Source, Result);
+}
+
+bool ConvertUTF8toWide(const char *Source, std::wstring &Result) {
+  if (!Source)
+  {
+    Result.clear();
+    return true;
+  }
+  return ConvertUTF8toWide(llvm::StringRef(Source), Result);
+}
+
+bool convertWideToUTF8(const std::wstring &Source, std::string &Result)
+{
+  if (sizeof(wchar_t) == 1) {
+    const UTF8 *Start = reinterpret_cast<const UTF8 *>(Source.data());
+    const UTF8 *End = reinterpret_cast<const UTF8 *>(Source.data() + Source.size());
+    if (!isLegalUTF8String(&Start, End))
+      return false;
+    Result.resize(Source.size());
+    memcpy(&Result[0], Source.data(), Source.size());
+    return true;
+  }
+  else if (sizeof(wchar_t) == 2) {
+    return convertUTF16ToUTF8String(llvm::ArrayRef<UTF16>(
+      reinterpret_cast<const UTF16 *>(Source.data()), Source.size()), Result);
+  }
+  else if (sizeof(wchar_t) == 4) {
+    const UTF32 *Start = reinterpret_cast<const UTF32 *>(Source.data());
+    const UTF32 *End = reinterpret_cast<const UTF32 *>(Source.data() + Source.size());
+    Result.resize(UNI_MAX_UTF8_BYTES_PER_CODE_POINT * Source.size());
+    UTF8 *ResultPtr = reinterpret_cast<UTF8 *>(&Result[0]);
+    UTF8 *ResultEnd = reinterpret_cast<UTF8 *>(&Result[0] + Result.size());
+    if (ConvertUTF32toUTF8(&Start, End, &ResultPtr, ResultEnd, strictConversion) == conversionOK) {
+      Result.resize(reinterpret_cast<char *>(ResultPtr) - &Result[0]);
+      return true;
+    }
+    else {
+      Result.clear();
+      return false;
+    }
+  }
+  else {
+    // Unreachable code
+    return false;
+  }
+}
+
 } // end namespace llvm
 

llvm/trunk/unittests/Support/ConvertUTFTest.cpp

 TEST(ConvertUTFTest, OddLengthInput) {
   std::string Result;
   bool Success = convertUTF16ToUTF8String(makeArrayRef("xxxxx", 5), Result);
   EXPECT_FALSE(Success);
 }
 
 TEST(ConvertUTFTest, Empty) {
   std::string Result;
-  bool Success = convertUTF16ToUTF8String(None, Result);
+  bool Success = convertUTF16ToUTF8String(llvm::ArrayRef<char>(None), Result);
   EXPECT_TRUE(Success);
   EXPECT_TRUE(Result.empty());
 }
 
 TEST(ConvertUTFTest, HasUTF16BOM) {
   bool HasBOM = hasUTF16ByteOrderMark(makeArrayRef("\xff\xfe", 2));
   EXPECT_TRUE(HasBOM);
   HasBOM = hasUTF16ByteOrderMark(makeArrayRef("\xfe\xff", 2));
   EXPECT_TRUE(HasBOM);
   HasBOM = hasUTF16ByteOrderMark(makeArrayRef("\xfe\xff ", 3));
   EXPECT_TRUE(HasBOM); // Don't care about odd lengths.
   HasBOM = hasUTF16ByteOrderMark(makeArrayRef("\xfe\xff\x00asdf", 6));
   EXPECT_TRUE(HasBOM);
 
   HasBOM = hasUTF16ByteOrderMark(None);
   EXPECT_FALSE(HasBOM);
   HasBOM = hasUTF16ByteOrderMark(makeArrayRef("\xfe", 1));
   EXPECT_FALSE(HasBOM);
 }
 
+TEST(ConvertUTFTest, UTF16WrappersForConvertUTF16ToUTF8String) {
+  // Src is the look of disapproval.
+  static const char Src[] = "\xff\xfe\xa0\x0c_\x00\xa0\x0c";
+  ArrayRef<UTF16> SrcRef = makeArrayRef((const UTF16 *)Src, 4);
+  std::string Result;
+  bool Success = convertUTF16ToUTF8String(SrcRef, Result);
+  EXPECT_TRUE(Success);
+  std::string Expected("\xe0\xb2\xa0_\xe0\xb2\xa0");
+  EXPECT_EQ(Expected, Result);
+}
+
+TEST(ConvertUTFTest, ConvertUTF8toWide) {
+  // Src is the look of disapproval.
+  static const char Src[] = "\xe0\xb2\xa0_\xe0\xb2\xa0";
+  std::wstring Result;
+  bool Success = ConvertUTF8toWide((const char*)Src, Result);
+  EXPECT_TRUE(Success);
+  std::wstring Expected(L"\x0ca0_\x0ca0");
+  EXPECT_EQ(Expected, Result);
+  Result.clear();
+  Success = ConvertUTF8toWide(StringRef(Src, 7), Result);
+  EXPECT_TRUE(Success);
+  EXPECT_EQ(Expected, Result);
+}
+
+TEST(ConvertUTFTest, convertWideToUTF8) {
+  // Src is the look of disapproval.
+  static const wchar_t Src[] = L"\x0ca0_\x0ca0";
+  std::string Result;
+  bool Success = convertWideToUTF8(Src, Result);
+  EXPECT_TRUE(Success);
+  std::string Expected("\xe0\xb2\xa0_\xe0\xb2\xa0");
+  EXPECT_EQ(Expected, Result);
+}
+
 struct ConvertUTFResultContainer {
   ConversionResult ErrorCode;
   std::vector<unsigned> UnicodeScalars;
 
   ConvertUTFResultContainer(ConversionResult ErrorCode)
       : ErrorCode(ErrorCode) {}
 
   ConvertUTFResultContainer