[libcxx] Fix definition of regex_traits::__regex_word on big-endian glibc systems

[dsanders]

Summary

On glibc, the bits used for the various character classes is endian dependant
(see _ISbit() in ctypes.h) but __regex_word does not account for this and uses
a spare bit that isn't spare on big-endian. On big-endian, it overlaps with the
bit for graphic characters which causes '-', '@', etc. to be considered a word
character.

Fixed this by defining the value using _ISbit(15) on glibc systems.

Fixes PR26476.

[dsanders]

include/regex

980

The static_cast is necessary to silence a false-positive warning on little-endian. _ISbit(15) expands to:

((15) < 8 ? ((1 << (15)) << 8) : ((1 << (15)) >> 8))

which simplifies to:

0 ? 0x800000 : 0x80

Clang warns about the truncation of the 0x800000 to char_class_type (unsigned short) even though this value doesn't matter.

[mclow.lists]

In r260527, I added some tests to catch this if it happens again.

If those tests fail w/o this patch and succeed with, then I'm happy with applying it.

[dsanders]

In D17132#349993, @mclow.lists wrote:

In r260527, I added some tests to catch this if it happens again.

If those tests fail w/o this patch and succeed with, then I'm happy with applying it.

The tests in r260527, don't fail without this patch because it doesn't check 'graph'. It also doesn't fail when I add it. The reason for this is that std::ctype_base::graph isn't the same as _ISgraph:

std::regex_traits<char>::__regex_word = 128
std::ctype_base::graph = 1036
_ISgraph = 128

ctype_base is defining graph by combining alnum and punct instead of using the _IS* macro like the other bits.

I'm not sure where it's getting the definition of classic_table. Presumably it's coming from glibc because adding a #error next to the optional one in libcxx doesn't cause build failures.

[mclow.lists]

ctype_base is defining graph by combining alnum and punct instead of using the _IS* macro like the other bits.

Because that's how the standard (22.4.1) shows to do it :-)

I'm not sure where it's getting the definition of classic_table. Presumably it's coming from glibc ...

Exactly. The classic_table is there for C libraries (like Bionic) that don't have their own rune table.

[dsanders]

I haven't fully proven this yet (because I haven't found the table), but I think that C and C++ are sharing the same table and we're colliding with a bit that only makes sense to C. This would explain why your tests don't fail but regex_word still collides with _ISgraph in the table.

I've confirmed that glibc is testing for _ISgraph, and libstdc++ defines ctype_base as per the standard. I just need to find the table.

[dsanders]

The table is apparently inside locale-archive and it looks like C and C++ both share this table. At this point I'm satisfied that the table also contains information that C++ isn't interested in and it's this data that regex_word collides with.

Are you happy to lift the condition that your tests fail without my patch?

[dsanders]

Sorry for the early ping but I need to fix this for rc3 (which Hans is hoping to tag mid-week) and I'm stuck as long as the requirement that the C++ tests in r260527 fail without my patch is in place.

I'm happy to make this '#ifdef mips' if that helps.

[mclow.lists]

Let's wrap it in a MIPS ifdef for the moment, and then, post-release we can see if that can be relaxed.

Given that, LGTM.

[dsanders]

Thanks. I'll commit using a MIPS ifdef in the morning and merge once it's through the buildbots.

include/regex

 struct _LIBCPP_TYPE_VIS_ONLY regex_traits
 {
 public:
     typedef _CharT                  char_type;
     typedef basic_string<char_type> string_type;
     typedef locale                  locale_type;
     typedef ctype_base::mask        char_class_type;
 
+#if defined(__GLIBC__)
+    static const char_class_type __regex_word = static_cast<char_class_type>(_ISbit(15));
+#else
     static const char_class_type __regex_word = 0x80;
+#endif
+
 private:
     locale __loc_;
     const ctype<char_type>* __ct_;
     const collate<char_type>* __col_;
 
 public:
     regex_traits();