[libc] Improve memcmp latency and codegen

[gchatelet]

Summary

This is based on ideas from @nafi to:

• use a branchless version of 'cmp' for 'uint32_t',
• completely resolve the lexicographic comparison through vector operations when wide types are available. We also get rid of byte reloads and serializing '__builtin_ctzll'.

I did not include the suggestion to replace comparisons of 'uint16_t'
with two 'uint8_t' as it did not seem to help the codegen. This can
be revisited in sub-sequent patches.

The code been rewritten to reduce nested function calls, making the
job of the inliner easier and preventing harmful code duplication.

[gchatelet]

libc/src/string/CMakeLists.txt

464

I'll submit this as a separate patch.

[gchatelet]

• rebase
• Simplifying uint32_t neq<uint64_t>

[gchatelet]

@nafi3000
I've created a trimmed down version of this code to play with the codegen : https://godbolt.org/z/rdEG5nY1q
I defaulted the compile option so that you can compare with the code we discussed earlier aka -O3 -march=haswell -mprefer-vector-width=128 -mno-avx2

Let me know what you think.
Also phabricator is sometimes confusing and you need to hit the Submit button at the bottom of the page if you've made inline comments, otherwise I won't see them.
Ping me offline if needed.

[nafi3000]

libc/src/string/memory_utils/bcmp_implementations.h

142

OOC, how about using head_tail (2 loads) instead of BcmpSequence of 3 loads? E.g.

generic::Bcmp<uint32_t>::head_tail(p1, p2, 7)

libc/src/string/memory_utils/op_generic.h

608

OOC, have you ever tried other values? E.g. how about:

return a > b ? 0x7fffffff : 0x80000000;

or if it does not compile due to the MemcmpReturnType, then:

return a > b ? static_cast<int32_t>(0x7fffffff) : static_cast<int32_t>(0x80000000);

-1 and 1 are 2 values apart.
0x7fffffff and 0x80000000 are 1 value apart.

assembly diff: https://www.diffchecker.com/LMVfxJ1D/

xor eax, eax	
cmp rcx, r8	
sbb eax, eax	
or eax, 1

vs

cmp r8, rcx
mov eax, -2147483648
sbb eax, 0

In theory... the former should take 3 cycles (or waiting for sbb, sbb waiting for cmp) while the latter should take 2 cycles (cmp and mov should happen in parallel, sbb happening after the cmp), right?

libc/src/string/memory_utils/op_x86.h

70–71

[optional nit] Maybe not in this diff, but eventually we can programmatically generate the sequences here and at lines 129 and 160 below.

202–203

Would it make sense to factor out this part to another function?
This is used here and for cmp<uint32_t>.

233

Ditto, -1 : 1 vs 0x80000000 : 0x7fffffff

libc/src/string/memory_utils/x86_64/memcmp_implementations.h

75

Ditto. Similar to the bcmp comment, how about using head_tail (2 loads) instead of MemcmpSequence of 3 loads? E.g.

generic::Memcmp<uint32_t>::head_tail(p1, p2, 7)

x86 asm diff:
https://www.diffchecker.com/XQNu3lGN/

libc/test/src/string/memory_utils/op_tests.cpp

220

Do we need to add generic::BcmpSequence<uint32_t, uint8_t> and generic::BcmpSequence<uint32_t, uint16_t> here too? I am interpreting the above list as:
8, 1, 2, 4, 1+1, 1+1+1, 2+1, 4+2+1

[nafi3000]

libc/src/string/memory_utils/op_generic.h

407–417

I wonder if it is better to use cmp<T> only for the last comparison. Motivation is that for non-last compare blocks we need to check the comparison result anyway (e.g. line 470 above) to decide whether to load and compare the next block in the sequence. Isn't it better to compute this decision (0 or non-0) as early as possible instead of computing the full cmp result (0, <0 or >0)?

E.g.

if constexpr (sizeof...(TS) == 0) {
  if constexpr (cmp_is_expensive<T>::value) {
    if (eq<T>(p1, p2, 0))
      return MemcmpReturnType::ZERO();
    return cmp_neq<T>(p1, p2, 0);
  } else {
    return cmp<T>(p1, p2, 0);
  }
} else {
  if (!eq<T>(p1, p2, 0))
    return cmp_neq<T>(p1, p2, 0);
  return MemcmpSequence<TS...>::block(p1 + sizeof(T), p2 + sizeof(T));
}

And, for the last block, I wonder if we can invariably call cmp<T> instead. What is better would depend on data. E.g. for __m512i, cmp<T> is faster if there is at least 1 byte mismatch in the last 64 bytes.

[gchatelet]

• Address most comments

[gchatelet]

libc/src/string/memory_utils/bcmp_implementations.h

142

Yeah there are a bunch of options here. Usually I want to use head_tail to cover a range of sizes as it clearly diminishes the overall code size (all sizes from 5 to 8 with only two loads per pointer).
Depending on how often those sizes appear in the size distribution it might be useful to special case the code.
I'm tempted to keep the previous logic to prevent regressions and make this a separate change. WDYT?

libc/src/string/memory_utils/op_generic.h

608

Nice one. Picking other values was on my TODO list but I never thought it through.

I had a look at codegen for armv8 it uses cinv instead of cneg but it seems to be neutral in terms of performance.
https://godbolt.org/z/Y9aGq5sPd

For x86 in theory this should be better yes 👍.
https://godbolt.org/z/69Gefhqef

For RISC-V it seems it's worse as it generates a branch.
https://godbolt.org/z/bvqMeMjMP
This seems to be in line with what I found on stackoverflow

Now I tried it on the full implementation and the additional branch seems to be outlined leading to the same code size. I don't know the impact on the code speed but since we don't yet have an optimized version of memcmp for RISC-V we can happily revisit the function later on.
I've created a separate function so we can keep track of the rationale.

libc/src/string/memory_utils/op_x86.h

70–71

AFAICT we can't do it with the intel intrinsics as they are real functions expecting a certain number of arguments.
It may be possible to generate them with GCC and clang vector extensions though and then convert them back to Intel types.

I gave it a try but it's brittle on clang and fails on GCC
https://godbolt.org/z/Ms7fW5nP3

Not sure it's worth it.

202–203

Yes, I've done so for the uint64_t so let's factor this out for uint32_t as well.

libc/test/src/string/memory_utils/op_tests.cpp

220

Done. More coverage doesn't hurt : )

[gchatelet]

• Rebase
• Simplifying uint32_t neq<uint64_t>
• Address most comments
• Fix bazel build

[lntue]

libc/src/string/memory_utils/bcmp_implementations.h

92

Currently we only check for SSE4.2 in our CMake build https://github.com/llvm/llvm-project/blob/main/libc/cmake/modules/LLVMLibCCheckCpuFeatures.cmake#L9

Do you want to change or add SSE4.1 to the list instead?

[gchatelet]

libc/src/string/memory_utils/bcmp_implementations.h

92

Technically this code only needs SSE4.1 but I don't think it's worth discriminating between the two.
https://en.wikipedia.org/wiki/SSE4#SSE4_subsets

Looking at Steam hardware survey (click the Other Settings line at the bottom of the page) the share of cpu having SSE4.1 but not SSE4.2 is about 0.23%.

• SSE4.1 : 99.36%
• SSE4.2 : 99.13%

So we can basically only discriminate between SSE2 and SSE4.2 and call it a day : )

libc/src/string/memory_utils/op_generic.h

407–417

The current benchmark works for all memory functions and is only able to assess the throughput of functions under a particular size distribution.
As-is, it is not a good tool to evaluate functions that may return early (bcmp and memcmp) and for which latency is important.

I'll work on adding a latency benchmark based on the one you provided me earlier. Once it's done I think we will be in a better position to decide which strategy is better.

SGTY?

[gchatelet]

• rebase and merge RISCV changes

[gchatelet]

• rebase and add a TODO to explore more optimization once we have proper latency benchmarking.

[gchatelet]

@nafi3000 I've been busy and haven't had time to work on the benchmark..
I will land this change as-is if you accept this revision (Menu : Add Action... -> Accept Revision). We can iterate on this through subsequent patches. WDYT?

[nafi3000]

libc/src/string/memory_utils/bcmp_implementations.h

142

Separate change SGTM.

libc/src/string/memory_utils/op_generic.h

407–417

Sounds good.

libc/src/string/memory_utils/op_x86.h

70–71

We can use _mm_load_si128 instead of _mm_set_epi8. Snappy code has some example:
https://github.com/google/snappy/blob/main/snappy.cc
Search for pattern_generation_masks.

Anyway, this can be addressed in a separate diff. And like you mention, it may not be worth it. In snappy, we actually need a array of such shuffle masks. But in this case we just need one.

[gchatelet]

Reopening to fix aarch64 and riscv

[gchatelet]

• Fix aarch64 and RISCV implementations

[gchatelet]

• Specialize types per architecture

[gchatelet]

• Forgot to add libc/src/string/memory_utils/aarch64/memcmp_implementations.h

[gchatelet]

• Add type specialization for RISCV

[gchatelet]

I still need to check that the ARM platform is not impacted by this patch. I'll land it once it's done.

[gchatelet]

• Add missing namespace for RISCV
• Also include riscv32

[lntue]

libc/src/string/CMakeLists.txt

573

If you drop the requirement to AVX, should the compile option be -march=sandybridge instead?

[gchatelet]

• Disable non uint8_t type tests for ARM platform

[gchatelet]

• Use sandybrige with AVX

[gchatelet]

This seems to be good to go. @nafi3000 do you want to have a final look before I submit?

[nafi3000]

libc/src/string/memory_utils/utils.h

157–159

nit: s/uint64_t/int64_t/ and s/uint32_t/int32_t/ in the comments.

160

For the explanation, please consider whether we can add some version of the following points:

For the int64_t to int32_t conversion we want the following properties:
- int32_t[31:31] == 1 iff diff < 0
- int32_t[31:0] == 0 iff diff == 0

We also observe that:
- When diff < 0: diff[63:32] == 0xffffffff and diff[31:0] != 0
- When diff > 0: diff[63:32] == 0 and diff[31:0] != 0
- When diff == 0: diff[63:32] == 0 and diff[31:0] == 0
- https://godbolt.org/z/8W7qWP6e5
- This implies that we can only look at diff[32:32] for determining the sign bit for the returned int32_t.

So, we do the following:
- int32_t[31:31] = diff[32:32]
- int32_t[30:0] = diff[31:0] == 0 ? 0 : non-0.

And, we can achieve the above by the expression below. We could have also used (diff64 >> 1) | (diff64 & 0x1) but (diff64 & 0xFFFF) is faster than (diff64 & 0x1). https://godbolt.org/z/j3b569rW1

We can also add all these in a separate diff.

[gchatelet]

• Fix typos and added explanation for int64_t to int32_t conversion in cmp_uint32_t

[gchatelet]

libc/src/string/memory_utils/utils.h

160

The explanation is fantastic, I copied it verbatim.

[gchatelet]

Reopening it to fix aarch64 build bot breakage
https://lab.llvm.org/buildbot/#/builders/223/builds/21703/steps/4/logs/stdio

[gchatelet]

• Prevent pulling the limits.h header which turns out to define PTHREAD_STACK_MIN on aarch64

[gchatelet]

• Prevent pulling the limits.h header which turns out to define PTHREAD_STACK_MIN on aarch64

[gchatelet]

Upon investigation the patch seems correct but some libraries need to be updated to conform to memcmp semantic.
One example is sqlite3 which reverses ordering by negating the result of memcmp. Since signed arithmetic is not symmetric (e.g., uint8_t ∈ [-128, 127]) negating does not negate when value is INT_MIN (e.g., godbolt).

[gchatelet]

• use -5/5 instead of INT_MIN/INT_MAX for uint64 not equal comparison

[gchatelet]

• modify comment

[lntue]

libc/src/string/memory_utils/utils.h

199–202

I wonder what's the tradeoffs between this and what is generated for 1 and -1? If this is better, then the compiler should just use this for 1 and -1 also, right?

[gchatelet]

libc/src/string/memory_utils/utils.h

199–202

I wonder what's the tradeoffs between this and what is generated for 1 and -1? If this is better, then the compiler should just use this for 1 and -1 also, right?

x86 does not have conditional negate and codegen for returning 1 and -1 has higher latency.

xor     eax, eax
cmp     rdi, rsi <- serializing
sbb     eax, eax <- dep on previous instruction
or      eax, 1   <- dep on previous instruction

I think the tradeoff is around register pressure, in the -1 / 1 case we just need eax at the expense of a longer dependency chain.
In the -5 / 5 case we need ecx on top of eax but the dependency chain is shorter and then latency is reduced. Since latency matters for memcmp it makes more sense to use this construct.

Now TBH I haven't measured that the overall generated code is better but I'll run a few tests before landing.

https://godbolt.org/z/Gqahv7r7e

[xbolva00]

libc/src/string/memory_utils/utils.h

195

So they have UB in their codebases. They should really fix instead of workarounds like this one.

[gchatelet]

libc/src/string/memory_utils/utils.h

195

So they have UB in their codebases. They should really fix instead of workarounds like this one.

Yeah I agree, I've been pushing for this but we have many instances of this bug (not only in sqlite3.c) and they're quite painful to chase down. They usually show up quite far away from the actual memcmp call. Fixing all of them will take time but we'll release the optimized version eventually /me hope.

[nafi3000]

libc/src/string/memory_utils/utils.h

199–202

The compiler could have also used edi or esi instead of ecx. Would that cause slightly lower register pressure? E.g. why is it not doing something like:

cmp rdi, rsi
mov edi, -5
mov eax, 5
cmovb eax, edi

[gchatelet]

libc/src/string/memory_utils/utils.h

199–202

The compiler could have also used edi or esi instead of ecx. Would that cause slightly lower register pressure? E.g. why is it not doing something like:

cmp rdi, rsi
mov edi, -5
mov eax, 5
cmovb eax, edi

Not exactly sure why, it may first use available registers (greedy algorithm) and then tries extra hard to reuse but only is it necessary?

[gchatelet]

• rebase for reland

libc/src/__support/macros/properties/architectures.h

 #if (defined(__arm__) || defined(_M_ARM))
 #define LIBC_TARGET_ARCH_IS_ARM
 #endif
 
 #if defined(__aarch64__) || defined(__arm64__) || defined(_M_ARM64)
 #define LIBC_TARGET_ARCH_IS_AARCH64
 #endif
 
+#if (defined(LIBC_TARGET_ARCH_IS_AARCH64) || defined(LIBC_TARGET_ARCH_IS_ARM))
+#define LIBC_TARGET_ARCH_IS_ANY_ARM
+#endif
+
 #if defined(__riscv) && (__riscv_xlen == 64)
 #define LIBC_TARGET_ARCH_IS_RISCV64
 #endif
 
 #if defined(__riscv) && (__riscv_xlen == 32)
 #define LIBC_TARGET_ARCH_IS_RISCV32
 #endif
 
-#if (defined(LIBC_TARGET_ARCH_IS_AARCH64) || defined(LIBC_TARGET_ARCH_IS_ARM))
-#define LIBC_TARGET_ARCH_IS_ANY_ARM
+#if (defined(LIBC_TARGET_ARCH_IS_RISCV64) ||                                   \
+     defined(LIBC_TARGET_ARCH_IS_RISCV32))
+#define LIBC_TARGET_ARCH_IS_ANY_RISCV
 #endif
 
 #endif // LLVM_LIBC_SUPPORT_MACROS_PROPERTIES_ARCHITECTURES_H

libc/src/string/CMakeLists.txt

function(add_implementation name impl_name)
 
   if("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
     # Note that '-mllvm' needs to be prefixed with 'SHELL:' to prevent CMake flag deduplication.
     foreach(opt IN LISTS ADD_IMPL_MLLVM_COMPILE_OPTIONS)
       list(APPEND ADD_IMPL_COMPILE_OPTIONS "SHELL:-mllvm ${opt}")
     endforeach()
   endif()
 
+  if("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU")
+    # Prevent warning when passing x86 SIMD types as template arguments.
+    # e.g. "warning: ignoring attributes on template argument ‘__m128i’ [-Wignored-attributes]"
+    list(APPEND ADD_IMPL_COMPILE_OPTIONS "-Wno-ignored-attributes")
+  endif()
+
   add_entrypoint_object(${impl_name}
     NAME ${name}
     SRCS ${ADD_IMPL_SRCS}
     HDRS ${ADD_IMPL_HDRS}
     DEPENDS ${ADD_IMPL_DEPENDS}
     COMPILE_OPTIONS -O3 ${ADD_IMPL_COMPILE_OPTIONS}
   )
   get_fq_target_name(${impl_name} fq_target_name)
   set_target_properties(${fq_target_name} PROPERTIES REQUIRE_CPU_FEATURES "${ADD_IMPL_REQUIRE}")
   set_property(GLOBAL APPEND PROPERTY "${name}_implementations" "${fq_target_name}")
 endfunction()
 
 # ------------------------------------------------------------------------------
 # bcmp
DEPENDS
       libc.include.string
     ${ARGN}
   )
 endfunction()
 
 if(${LIBC_TARGET_ARCHITECTURE_IS_X86})
   add_memcpy(memcpy_x86_64_opt_sse2   COMPILE_OPTIONS -march=k8             REQUIRE SSE2)
   add_memcpy(memcpy_x86_64_opt_sse4   COMPILE_OPTIONS -march=nehalem        REQUIRE SSE4_2)
-  add_memcpy(memcpy_x86_64_opt_avx2   COMPILE_OPTIONS -march=haswell        REQUIRE AVX2)
+  add_memcpy(memcpy_x86_64_opt_avx    COMPILE_OPTIONS -march=sandybridge    REQUIRE AVX)
   add_memcpy(memcpy_x86_64_opt_avx512 COMPILE_OPTIONS -march=skylake-avx512 REQUIRE AVX512F)
   add_memcpy(memcpy_opt_host          COMPILE_OPTIONS ${LIBC_COMPILE_OPTIONS_NATIVE})
   add_memcpy(memcpy)
 elseif(${LIBC_TARGET_ARCHITECTURE_IS_AARCH64})
   # Disable tail merging as it leads to lower performance.
   add_memcpy(memcpy_opt_host          COMPILE_OPTIONS ${LIBC_COMPILE_OPTIONS_NATIVE}
                                       MLLVM_COMPILE_OPTIONS "-tail-merge-threshold=0")
   add_memcpy(memcpy                   MLLVM_COMPILE_OPTIONS "-tail-merge-threshold=0")

libc/src/string/memory_utils/CMakeLists.txt

HDRS
     x86_64/memcpy_implementations.h
   DEPS
     libc.src.__support.common
     libc.src.__support.CPP.bit
     libc.src.__support.CPP.cstddef
     libc.src.__support.CPP.type_traits
     libc.src.__support.macros.config
     libc.src.__support.macros.optimization
+    libc.src.__support.macros.properties.architectures
 )
 
 add_header_library(
   memcpy_implementation
   HDRS
     memcpy_implementations.h
   DEPS
     .memory_utils

libc/src/string/memory_utils/aarch64/memcmp_implementations.h

 #include "src/string/memory_utils/op_generic.h"
 #include "src/string/memory_utils/utils.h" // MemcmpReturnType
 
 namespace __llvm_libc {
 
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_generic_gt16(CPtr p1, CPtr p2, size_t count) {
   if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = generic::Memcmp<16>::block(p1, p2))
+    if (auto value = generic::Memcmp<uint8x16_t>::block(p1, p2))
       return value;
     align_to_next_boundary<16, Arg::P1>(p1, p2, count);
   }
-  return generic::Memcmp<16>::loop_and_tail(p1, p2, count);
+  return generic::Memcmp<uint8x16_t>::loop_and_tail(p1, p2, count);
 }
 
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_aarch64_neon_gt16(CPtr p1, CPtr p2, size_t count) {
   if (LIBC_UNLIKELY(count >= 128)) { // [128, ∞]
-    if (auto value = generic::Memcmp<16>::block(p1, p2))
+    if (auto value = generic::Memcmp<uint8x16_t>::block(p1, p2))
       return value;
     align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-    return generic::Memcmp<32>::loop_and_tail(p1, p2, count);
+    return generic::Memcmp<uint8x16x2_t>::loop_and_tail(p1, p2, count);
   }
-  if (generic::Bcmp<16>::block(p1, p2)) // [16, 16]
-    return generic::Memcmp<16>::block(p1, p2);
+  if (generic::Bcmp<uint8x16_t>::block(p1, p2)) // [16, 16]
+    return generic::Memcmp<uint8x16_t>::block(p1, p2);
   if (count < 32) // [17, 31]
-    return generic::Memcmp<16>::tail(p1, p2, count);
-  if (generic::Bcmp<16>::block(p1 + 16, p2 + 16)) // [32, 32]
-    return generic::Memcmp<16>::block(p1 + 16, p2 + 16);
+    return generic::Memcmp<uint8x16_t>::tail(p1, p2, count);
+  if (generic::Bcmp<uint8x16_t>::block(p1 + 16, p2 + 16)) // [32, 32]
+    return generic::Memcmp<uint8x16_t>::block(p1 + 16, p2 + 16);
   if (count < 64) // [33, 63]
-    return generic::Memcmp<32>::tail(p1, p2, count);
+    return generic::Memcmp<uint8x16x2_t>::tail(p1, p2, count);
   // [64, 127]
-  return generic::Memcmp<16>::loop_and_tail(p1 + 32, p2 + 32, count - 32);
+  return generic::Memcmp<uint8x16_t>::loop_and_tail(p1 + 32, p2 + 32,
+                                                    count - 32);
 }
 
 LIBC_INLINE MemcmpReturnType inline_memcmp_aarch64(CPtr p1, CPtr p2,
                                                    size_t count) {
   if (count == 0)
     return MemcmpReturnType::ZERO();
   if (count == 1)
-    return generic::Memcmp<1>::block(p1, p2);
+    return generic::Memcmp<uint8_t>::block(p1, p2);
   if (count == 2)
-    return generic::Memcmp<2>::block(p1, p2);
+    return generic::Memcmp<uint16_t>::block(p1, p2);
   if (count == 3)
-    return generic::Memcmp<3>::block(p1, p2);
+    return generic::MemcmpSequence<uint16_t, uint8_t>::block(p1, p2);
   if (count <= 8)
-    return generic::Memcmp<4>::head_tail(p1, p2, count);
+    return generic::Memcmp<uint32_t>::head_tail(p1, p2, count);
   if (count <= 16)
-    return generic::Memcmp<8>::head_tail(p1, p2, count);
+    return generic::Memcmp<uint64_t>::head_tail(p1, p2, count);
   if constexpr (aarch64::kNeon)
     return inline_memcmp_aarch64_neon_gt16(p1, p2, count);
   else
     return inline_memcmp_generic_gt16(p1, p2, count);
 }
 } // namespace __llvm_libc
 
 #endif // LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H

libc/src/string/memory_utils/bcmp_implementations.h

 #define LLVM_LIBC_SRC_STRING_MEMORY_UTILS_BCMP_IMPLEMENTATIONS_H
 
 #include "src/__support/common.h"
 #include "src/__support/macros/optimization.h" // LIBC_UNLIKELY LIBC_LOOP_NOUNROLL
 #include "src/__support/macros/properties/architectures.h"
 #include "src/string/memory_utils/op_aarch64.h"
 #include "src/string/memory_utils/op_builtin.h"
 #include "src/string/memory_utils/op_generic.h"
+#include "src/string/memory_utils/op_riscv.h"
 #include "src/string/memory_utils/op_x86.h"
 
 #include <stddef.h> // size_t
 
 namespace __llvm_libc {
 
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
-inline_bcmp_byte_per_byte(CPtr p1, CPtr p2, size_t offset, size_t count) {
-  LIBC_LOOP_NOUNROLL
-  for (; offset < count; ++offset)
-    if (p1[offset] != p2[offset])
-      return BcmpReturnType::NONZERO();
-  return BcmpReturnType::ZERO();
+inline_bcmp_byte_per_byte(CPtr p1, CPtr p2, size_t count, size_t offset = 0) {
+  return generic::Bcmp<uint8_t>::loop_and_tail_offset(p1, p2, count, offset);
 }
 
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
 inline_bcmp_aligned_access_64bit(CPtr p1, CPtr p2, size_t count) {
   constexpr size_t kAlign = sizeof(uint64_t);
   if (count <= 2 * kAlign)
-    return inline_bcmp_byte_per_byte(p1, p2, 0, count);
+    return inline_bcmp_byte_per_byte(p1, p2, count);
   size_t bytes_to_p1_align = distance_to_align_up<kAlign>(p1);
-  if (auto value = inline_bcmp_byte_per_byte(p1, p2, 0, bytes_to_p1_align))
+  if (auto value = inline_bcmp_byte_per_byte(p1, p2, bytes_to_p1_align))
     return value;
   size_t offset = bytes_to_p1_align;
   size_t p2_alignment = distance_to_align_down<kAlign>(p2 + offset);
   for (; offset < count - kAlign; offset += kAlign) {
     uint64_t a;
     if (p2_alignment == 0)
       a = load64_aligned<uint64_t>(p2, offset);
     else if (p2_alignment == 4)
       a = load64_aligned<uint32_t, uint32_t>(p2, offset);
     else if (p2_alignment == 2)
       a = load64_aligned<uint16_t, uint16_t, uint16_t, uint16_t>(p2, offset);
     else
       a = load64_aligned<uint8_t, uint16_t, uint16_t, uint16_t, uint8_t>(
           p2, offset);
     uint64_t b = load64_aligned<uint64_t>(p1, offset);
     if (a != b)
       return BcmpReturnType::NONZERO();
   }
-  return inline_bcmp_byte_per_byte(p1, p2, offset, count);
+  return inline_bcmp_byte_per_byte(p1, p2, count, offset);
 }
 
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
 inline_bcmp_aligned_access_32bit(CPtr p1, CPtr p2, size_t count) {
   constexpr size_t kAlign = sizeof(uint32_t);
   if (count <= 2 * kAlign)
-    return inline_bcmp_byte_per_byte(p1, p2, 0, count);
+    return inline_bcmp_byte_per_byte(p1, p2, count);
   size_t bytes_to_p1_align = distance_to_align_up<kAlign>(p1);
-  if (auto value = inline_bcmp_byte_per_byte(p1, p2, 0, bytes_to_p1_align))
+  if (auto value = inline_bcmp_byte_per_byte(p1, p2, bytes_to_p1_align))
     return value;
   size_t offset = bytes_to_p1_align;
   size_t p2_alignment = distance_to_align_down<kAlign>(p2 + offset);
   for (; offset < count - kAlign; offset += kAlign) {
     uint32_t a;
     if (p2_alignment == 0)
       a = load32_aligned<uint32_t>(p2, offset);
     else if (p2_alignment == 2)
       a = load32_aligned<uint16_t, uint16_t>(p2, offset);
     else
       a = load32_aligned<uint8_t, uint16_t, uint8_t>(p2, offset);
     uint32_t b = load32_aligned<uint32_t>(p1, offset);
     if (a != b)
       return BcmpReturnType::NONZERO();
   }
-  return inline_bcmp_byte_per_byte(p1, p2, offset, count);
+  return inline_bcmp_byte_per_byte(p1, p2, count, offset);
 }
 
 #if defined(LIBC_TARGET_ARCH_IS_X86) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
 inline_bcmp_generic_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (count < 256)
-    return generic::Bcmp<16>::loop_and_tail(p1, p2, count);
-  if (auto value = generic::Bcmp<64>::block(p1, p2))
-    return value;
-  align_to_next_boundary<64, Arg::P1>(p1, p2, count);
-  return generic::Bcmp<64>::loop_and_tail(p1, p2, count);
+  return generic::Bcmp<uint64_t>::loop_and_tail_align_above(256, p1, p2, count);
 }
 #endif // defined(LIBC_TARGET_ARCH_IS_X86) ||
        // defined(LIBC_TARGET_ARCH_IS_AARCH64)
 
 #if defined(LIBC_TARGET_ARCH_IS_X86)
+#if defined(__SSE4_1__)
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
-inline_bcmp_x86_sse2_gt16(CPtr p1, CPtr p2, size_t count) {
+inline_bcmp_x86_sse41_gt16(CPtr p1, CPtr p2, size_t count) {
   if (count <= 32)
-    return x86::sse2::Bcmp<16>::head_tail(p1, p2, count);
-  if (count < 256)
-    return x86::sse2::Bcmp<16>::loop_and_tail(p1, p2, count);
-  if (auto value = x86::sse2::Bcmp<16>::block(p1, p2))
-    return value;
-  align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-  return x86::sse2::Bcmp<64>::loop_and_tail(p1, p2, count);
+    return generic::Bcmp<__m128i>::head_tail(p1, p2, count);
+  return generic::Bcmp<__m128i>::loop_and_tail_align_above(256, p1, p2, count);
 }
+#endif // __SSE4_1__
 
+#if defined(__AVX__)
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
-inline_bcmp_x86_avx2_gt16(CPtr p1, CPtr p2, size_t count) {
+inline_bcmp_x86_avx_gt16(CPtr p1, CPtr p2, size_t count) {
   if (count <= 32)
-    return x86::sse2::Bcmp<16>::head_tail(p1, p2, count);
+    return generic::Bcmp<__m128i>::head_tail(p1, p2, count);
   if (count <= 64)
-    return x86::avx2::Bcmp<32>::head_tail(p1, p2, count);
-  if (count <= 128)
-    return x86::avx2::Bcmp<64>::head_tail(p1, p2, count);
-  if (LIBC_UNLIKELY(count >= 256)) {
-    if (auto value = x86::avx2::Bcmp<64>::block(p1, p2))
-      return value;
-    align_to_next_boundary<64, Arg::P1>(p1, p2, count);
-  }
-  return x86::avx2::Bcmp<64>::loop_and_tail(p1, p2, count);
+    return generic::Bcmp<__m256i>::head_tail(p1, p2, count);
+  return generic::Bcmp<__m256i>::loop_and_tail_align_above(256, p1, p2, count);
 }
+#endif // __AVX__
 
+#if defined(__AVX512BW__)
 [[maybe_unused]] LIBC_INLINE BcmpReturnType
 inline_bcmp_x86_avx512bw_gt16(CPtr p1, CPtr p2, size_t count) {
   if (count <= 32)
-    return x86::sse2::Bcmp<16>::head_tail(p1, p2, count);
+    return generic::Bcmp<__m128i>::head_tail(p1, p2, count);
   if (count <= 64)
-    return x86::avx2::Bcmp<32>::head_tail(p1, p2, count);
+    return generic::Bcmp<__m256i>::head_tail(p1, p2, count);
   if (count <= 128)
-    return x86::avx512bw::Bcmp<64>::head_tail(p1, p2, count);
-  if (LIBC_UNLIKELY(count >= 256)) {
-    if (auto value = x86::avx512bw::Bcmp<64>::block(p1, p2))
-      return value;
-    align_to_next_boundary<64, Arg::P1>(p1, p2, count);
-  }
-  return x86::avx512bw::Bcmp<64>::loop_and_tail(p1, p2, count);
+    return generic::Bcmp<__m512i>::head_tail(p1, p2, count);
+  return generic::Bcmp<__m512i>::loop_and_tail_align_above(256, p1, p2, count);
 }
+#endif // __AVX512BW__
 
 [[maybe_unused]] LIBC_INLINE BcmpReturnType inline_bcmp_x86(CPtr p1, CPtr p2,
                                                             size_t count) {
   if (count == 0)
     return BcmpReturnType::ZERO();
   if (count == 1)
-    return generic::Bcmp<1>::block(p1, p2);
+    return generic::Bcmp<uint8_t>::block(p1, p2);
   if (count == 2)
-    return generic::Bcmp<2>::block(p1, p2);
-  if (count <= 4)
-    return generic::Bcmp<2>::head_tail(p1, p2, count);
-  if (count <= 8)
-    return generic::Bcmp<4>::head_tail(p1, p2, count);
+    return generic::Bcmp<uint16_t>::block(p1, p2);
+  if (count == 3)
+    return generic::BcmpSequence<uint16_t, uint8_t>::block(p1, p2);
+  if (count == 4)
+    return generic::Bcmp<uint32_t>::block(p1, p2);
+  if (count == 5)
+    return generic::BcmpSequence<uint32_t, uint8_t>::block(p1, p2);
+  if (count == 6)
+    return generic::BcmpSequence<uint32_t, uint16_t>::block(p1, p2);
+  if (count == 7)
+    return generic::BcmpSequence<uint32_t, uint16_t, uint8_t>::block(p1, p2);
+  if (count == 8)
+    return generic::Bcmp<uint64_t>::block(p1, p2);
   if (count <= 16)
-    return generic::Bcmp<8>::head_tail(p1, p2, count);
-  if constexpr (x86::kAvx512BW)
-    return inline_bcmp_x86_avx512bw_gt16(p1, p2, count);
-  else if constexpr (x86::kAvx2)
-    return inline_bcmp_x86_avx2_gt16(p1, p2, count);
-  else if constexpr (x86::kSse2)
-    return inline_bcmp_x86_sse2_gt16(p1, p2, count);
-  else
-    return inline_bcmp_generic_gt16(p1, p2, count);
+    return generic::Bcmp<uint64_t>::head_tail(p1, p2, count);
+#if defined(__AVX512BW__)
+#elif defined(__AVX__)
+  return inline_bcmp_x86_avx_gt16(p1, p2, count);
+#elif defined(__SSE4_1__)
+  return inline_bcmp_x86_sse41_gt16(p1, p2, count);
+#else
+#endif
 }
 #endif // defined(LIBC_TARGET_ARCH_IS_X86)
 
 #if defined(LIBC_TARGET_ARCH_IS_AARCH64)
 [[maybe_unused]] LIBC_INLINE BcmpReturnType inline_bcmp_aarch64(CPtr p1,
                                                                 CPtr p2,
                                                                 size_t count) {
   if (LIBC_LIKELY(count <= 32)) {
     if (LIBC_UNLIKELY(count >= 16)) {
       return aarch64::Bcmp<16>::head_tail(p1, p2, count);
     }
     switch (count) {
     case 0:
       return BcmpReturnType::ZERO();
     case 1:
-      return generic::Bcmp<1>::block(p1, p2);
+      return generic::Bcmp<uint8_t>::block(p1, p2);
     case 2:
-      return generic::Bcmp<2>::block(p1, p2);
+      return generic::Bcmp<uint16_t>::block(p1, p2);
     case 3:
-      return generic::Bcmp<2>::head_tail(p1, p2, count);
+      return generic::Bcmp<uint16_t>::head_tail(p1, p2, count);
     case 4:
-      return generic::Bcmp<4>::block(p1, p2);
+      return generic::Bcmp<uint32_t>::block(p1, p2);
     case 5:
     case 6:
     case 7:
-      return generic::Bcmp<4>::head_tail(p1, p2, count);
+      return generic::Bcmp<uint32_t>::head_tail(p1, p2, count);
     case 8:
-      return generic::Bcmp<8>::block(p1, p2);
+      return generic::Bcmp<uint64_t>::block(p1, p2);
     case 9:
     case 10:
     case 11:
     case 12:
     case 13:
     case 14:
     case 15:
-      return generic::Bcmp<8>::head_tail(p1, p2, count);
+      return generic::Bcmp<uint64_t>::head_tail(p1, p2, count);
     }
   }
 
   if (count <= 64)
     return aarch64::Bcmp<32>::head_tail(p1, p2, count);
 
   // Aligned loop if > 256, otherwise normal loop
   if (LIBC_UNLIKELY(count > 256)) {
#if defined(LIBC_TARGET_ARCH_IS_X86)
   return inline_bcmp_x86(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
   return inline_bcmp_aarch64(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
   return inline_bcmp_aligned_access_64bit(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_RISCV32)
   return inline_bcmp_aligned_access_32bit(p1, p2, count);
 #else
-  return inline_bcmp_byte_per_byte(p1, p2, 0, count);
+  return inline_bcmp_byte_per_byte(p1, p2, count);
 #endif
 }
 
 LIBC_INLINE int inline_bcmp(const void *p1, const void *p2, size_t count) {
   return static_cast<int>(inline_bcmp(reinterpret_cast<CPtr>(p1),
                                       reinterpret_cast<CPtr>(p2), count));
 }
 
 } // namespace __llvm_libc
 
 #endif // LLVM_LIBC_SRC_STRING_MEMORY_UTILS_BCMP_IMPLEMENTATIONS_H

libc/src/string/memory_utils/memcmp_implementations.h

 //===-- Implementation of memcmp ------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIBC_SRC_STRING_MEMORY_UTILS_MEMCMP_IMPLEMENTATIONS_H
 #define LLVM_LIBC_SRC_STRING_MEMORY_UTILS_MEMCMP_IMPLEMENTATIONS_H
 
 #include "src/__support/common.h"
 #include "src/__support/macros/optimization.h" // LIBC_UNLIKELY LIBC_LOOP_NOUNROLL
 #include "src/__support/macros/properties/architectures.h"
 #include "src/string/memory_utils/op_generic.h"
+#include "src/string/memory_utils/op_riscv.h"
 #include "src/string/memory_utils/utils.h" // CPtr MemcmpReturnType
 
 #include <stddef.h> // size_t
 
 #if defined(LIBC_TARGET_ARCH_IS_X86)
 #include "src/string/memory_utils/x86_64/memcmp_implementations.h"
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
 #include "src/string/memory_utils/aarch64/memcmp_implementations.h"
 #endif
 
 namespace __llvm_libc {
 
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_byte_per_byte(CPtr p1, CPtr p2, size_t offset, size_t count) {
-  LIBC_LOOP_NOUNROLL
-  for (; offset < count; ++offset)
-    if (auto value = generic::Memcmp<1>::block(p1 + offset, p2 + offset))
-      return value;
-  return MemcmpReturnType::ZERO();
+inline_memcmp_byte_per_byte(CPtr p1, CPtr p2, size_t count, size_t offset = 0) {
+  return generic::Memcmp<uint8_t>::loop_and_tail_offset(p1, p2, count, offset);
 }
 
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_aligned_access_64bit(CPtr p1, CPtr p2, size_t count) {
   constexpr size_t kAlign = sizeof(uint64_t);
   if (count <= 2 * kAlign)
-    return inline_memcmp_byte_per_byte(p1, p2, 0, count);
+    return inline_memcmp_byte_per_byte(p1, p2, count);
   size_t bytes_to_p1_align = distance_to_align_up<kAlign>(p1);
-  if (auto value = inline_memcmp_byte_per_byte(p1, p2, 0, bytes_to_p1_align))
+  if (auto value = inline_memcmp_byte_per_byte(p1, p2, bytes_to_p1_align))
     return value;
   size_t offset = bytes_to_p1_align;
   size_t p2_alignment = distance_to_align_down<kAlign>(p2 + offset);
   for (; offset < count - kAlign; offset += kAlign) {
     uint64_t b;
     if (p2_alignment == 0)
       b = load64_aligned<uint64_t>(p2, offset);
     else if (p2_alignment == 4)
       b = load64_aligned<uint32_t, uint32_t>(p2, offset);
     else if (p2_alignment == 2)
       b = load64_aligned<uint16_t, uint16_t, uint16_t, uint16_t>(p2, offset);
     else
       b = load64_aligned<uint8_t, uint16_t, uint16_t, uint16_t, uint8_t>(
           p2, offset);
     uint64_t a = load64_aligned<uint64_t>(p1, offset);
-    if (a != b) {
-      // TODO use cmp_neq_uint64_t from D148717 once it's submitted.
-      return Endian::to_big_endian(a) < Endian::to_big_endian(b) ? -1 : 1;
-    }
+    if (a != b)
+      return cmp_neq_uint64_t(Endian::to_big_endian(a),
+                              Endian::to_big_endian(b));
   }
-  return inline_memcmp_byte_per_byte(p1, p2, offset, count);
+  return inline_memcmp_byte_per_byte(p1, p2, count, offset);
 }
 
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_aligned_access_32bit(CPtr p1, CPtr p2, size_t count) {
   constexpr size_t kAlign = sizeof(uint32_t);
   if (count <= 2 * kAlign)
-    return inline_memcmp_byte_per_byte(p1, p2, 0, count);
+    return inline_memcmp_byte_per_byte(p1, p2, count);
   size_t bytes_to_p1_align = distance_to_align_up<kAlign>(p1);
-  if (auto value = inline_memcmp_byte_per_byte(p1, p2, 0, bytes_to_p1_align))
+  if (auto value = inline_memcmp_byte_per_byte(p1, p2, bytes_to_p1_align))
     return value;
   size_t offset = bytes_to_p1_align;
   size_t p2_alignment = distance_to_align_down<kAlign>(p2 + offset);
   for (; offset < count - kAlign; offset += kAlign) {
     uint32_t b;
     if (p2_alignment == 0)
       b = load32_aligned<uint32_t>(p2, offset);
     else if (p2_alignment == 2)
       b = load32_aligned<uint16_t, uint16_t>(p2, offset);
     else
       b = load32_aligned<uint8_t, uint16_t, uint8_t>(p2, offset);
     uint32_t a = load32_aligned<uint32_t>(p1, offset);
-    if (a != b) {
-      // TODO use cmp_uint32_t from D148717 once it's submitted.
-      // We perform the difference as an uint64_t.
-      const int64_t diff = static_cast<int64_t>(Endian::to_big_endian(a)) -
-                           static_cast<int64_t>(Endian::to_big_endian(b));
-      // And reduce the uint64_t into an uint32_t.
-      return static_cast<int32_t>((diff >> 1) | (diff & 0xFFFF));
-    }
+    if (a != b)
+      return cmp_uint32_t(Endian::to_big_endian(a), Endian::to_big_endian(b));
   }
-  return inline_memcmp_byte_per_byte(p1, p2, offset, count);
+  return inline_memcmp_byte_per_byte(p1, p2, count, offset);
 }
 
 LIBC_INLINE MemcmpReturnType inline_memcmp(CPtr p1, CPtr p2, size_t count) {
 #if defined(LIBC_TARGET_ARCH_IS_X86)
   return inline_memcmp_x86(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
   return inline_memcmp_aarch64(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_RISCV64)
   return inline_memcmp_aligned_access_64bit(p1, p2, count);
 #elif defined(LIBC_TARGET_ARCH_IS_RISCV32)
   return inline_memcmp_aligned_access_32bit(p1, p2, count);
 #else
-  return inline_memcmp_byte_per_byte(p1, p2, 0, count);
+  return inline_memcmp_byte_per_byte(p1, p2, count);
 #endif
 }
 
 LIBC_INLINE int inline_memcmp(const void *p1, const void *p2, size_t count) {
   return static_cast<int>(inline_memcmp(reinterpret_cast<CPtr>(p1),
                                         reinterpret_cast<CPtr>(p2), count));
 }
 
 } // namespace __llvm_libc
 
 #endif // LLVM_LIBC_SRC_STRING_MEMORY_UTILS_MEMCMP_IMPLEMENTATIONS_H

libc/src/string/memory_utils/memmove_implementations.h

for (ptrdiff_t offset = count - 1; offset >= 0; --offset)
       builtin::Memcpy<1>::block(dst + offset, src + offset);
   }
 }
 
 LIBC_INLINE void inline_memmove(Ptr dst, CPtr src, size_t count) {
 #if defined(LIBC_TARGET_ARCH_IS_X86) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
 #if defined(LIBC_TARGET_ARCH_IS_X86)
 #if defined(__AVX512F__)
-  using uint128_t = uint8x16_t;
-  using uint256_t = uint8x32_t;
-  using uint512_t = uint8x64_t;
+  using uint128_t = generic_v128;
+  using uint256_t = generic_v256;
+  using uint512_t = generic_v512;
 #elif defined(__AVX__)
-  using uint128_t = uint8x16_t;
-  using uint256_t = uint8x32_t;
-  using uint512_t = cpp::array<uint8x32_t, 2>;
+  using uint128_t = generic_v128;
+  using uint256_t = generic_v256;
+  using uint512_t = cpp::array<generic_v256, 2>;
 #elif defined(__SSE2__)
-  using uint128_t = uint8x16_t;
-  using uint256_t = cpp::array<uint8x16_t, 2>;
-  using uint512_t = cpp::array<uint8x16_t, 4>;
+  using uint128_t = generic_v128;
+  using uint256_t = cpp::array<generic_v128, 2>;
+  using uint512_t = cpp::array<generic_v128, 4>;
 #else
   using uint128_t = cpp::array<uint64_t, 2>;
   using uint256_t = cpp::array<uint64_t, 4>;
   using uint512_t = cpp::array<uint64_t, 8>;
 #endif
 #elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
   static_assert(aarch64::kNeon, "aarch64 supports vector types");
-  using uint128_t = uint8x16_t;
-  using uint256_t = uint8x32_t;
-  using uint512_t = uint8x64_t;
+  using uint128_t = generic_v128;
+  using uint256_t = generic_v256;
+  using uint512_t = generic_v512;
 #endif
   if (count == 0)
     return;
   if (count == 1)
     return generic::Memmove<uint8_t>::block(dst, src);
   if (count <= 4)
     return generic::Memmove<uint16_t>::head_tail(dst, src, count);
   if (count <= 8)

libc/src/string/memory_utils/memset_implementations.h

for (; offset < count - kAlign; offset += kAlign)
     store64_aligned<uint64_t>(generic::splat<uint64_t>(value), dst, offset);
   inline_memset_byte_per_byte(dst, offset, value, count);
 }
 
 #if defined(LIBC_TARGET_ARCH_IS_X86)
 [[maybe_unused]] LIBC_INLINE static void
 inline_memset_x86(Ptr dst, uint8_t value, size_t count) {
 #if defined(__AVX512F__)
-  using uint128_t = uint8x16_t;
-  using uint256_t = uint8x32_t;
-  using uint512_t = uint8x64_t;
+  using uint128_t = generic_v128;
+  using uint256_t = generic_v256;
+  using uint512_t = generic_v512;
 #elif defined(__AVX__)
-  using uint128_t = uint8x16_t;
-  using uint256_t = uint8x32_t;
-  using uint512_t = cpp::array<uint8x32_t, 2>;
+  using uint128_t = generic_v128;
+  using uint256_t = generic_v256;
+  using uint512_t = cpp::array<generic_v256, 2>;
 #elif defined(__SSE2__)
-  using uint128_t = uint8x16_t;
-  using uint256_t = cpp::array<uint8x16_t, 2>;
-  using uint512_t = cpp::array<uint8x16_t, 4>;
+  using uint128_t = generic_v128;
+  using uint256_t = cpp::array<generic_v128, 2>;
+  using uint512_t = cpp::array<generic_v128, 4>;
 #else
   using uint128_t = cpp::array<uint64_t, 2>;
   using uint256_t = cpp::array<uint64_t, 4>;
   using uint512_t = cpp::array<uint64_t, 8>;
 #endif
 
   if (count == 0)
     return;
#endif
   return generic::Memset<uint256_t>::loop_and_tail(dst, value, count);
 }
 #endif // defined(LIBC_TARGET_ARCH_IS_X86)
 
 #if defined(LIBC_TARGET_ARCH_IS_AARCH64)
 [[maybe_unused]] LIBC_INLINE static void
 inline_memset_aarch64(Ptr dst, uint8_t value, size_t count) {
   static_assert(aarch64::kNeon, "aarch64 supports vector types");
-  using uint128_t = uint8x16_t;
-  using uint256_t = uint8x32_t;
-  using uint512_t = uint8x64_t;
+  using uint128_t = generic_v128;
+  using uint256_t = generic_v256;
+  using uint512_t = generic_v512;
   if (count == 0)
     return;
   if (count <= 3) {
     generic::Memset<uint8_t>::block(dst, value);
     if (count > 1)
       generic::Memset<uint16_t>::tail(dst, value, count);
     return;
   }

libc/src/string/memory_utils/op_aarch64.h

#endif
 
   LIBC_INLINE static void loop_and_tail(Ptr dst, uint8_t value, size_t count) {
     size_t offset = 0;
     do {
       block(dst + offset, value);
       offset += SIZE;
     } while (offset < count - SIZE);
     // Unaligned store, we can't use 'dc zva' here.
-    generic::Memset<uint8x64_t>::tail(dst, value, count);
+    generic::Memset<generic_v512>::tail(dst, value, count);
   }
 };
 
 LIBC_INLINE static bool hasZva() {
   uint64_t zva_val;
   asm("mrs %[zva_val], dczid_el0" : [zva_val] "=r"(zva_val));
   // DC ZVA is permitted if DZP, bit [4] is zero.
   // BS, bits [3:0] is log2 of the block count in words.
do {
       offset += SIZE;
     } while (offset < count - SIZE);
     return tail(p1, p2, count);
   }
 };
 
 } // namespace __llvm_libc::aarch64
 
+namespace __llvm_libc::generic {
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint16_t
+template <> struct cmp_is_expensive<uint16_t> : public cpp::false_type {};
+template <> LIBC_INLINE bool eq<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint16_t>(p1, offset) == load<uint16_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint16_t>(p1, offset) ^ load<uint16_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return static_cast<int32_t>(load_be<uint16_t>(p1, offset)) -
+         static_cast<int32_t>(load_be<uint16_t>(p2, offset));
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint32_t
+template <> struct cmp_is_expensive<uint32_t> : cpp::false_type {};
+template <>
+LIBC_INLINE uint32_t neq<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint32_t>(p1, offset) ^ load<uint32_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load_be<uint32_t>(p1, offset);
+  const auto b = load_be<uint32_t>(p2, offset);
+  return a > b ? 1 : a < b ? -1 : 0;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint64_t
+template <> struct cmp_is_expensive<uint64_t> : cpp::false_type {};
+template <>
+LIBC_INLINE uint32_t neq<uint64_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint64_t>(p1, offset) != load<uint64_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint64_t>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load_be<uint64_t>(p1, offset);
+  const auto b = load_be<uint64_t>(p2, offset);
+  if (a != b)
+    return a > b ? 1 : -1;
+  return MemcmpReturnType::ZERO();
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint8x16_t
+template <> struct is_vector<uint8x16_t> : cpp::true_type {};
+template <> struct cmp_is_expensive<uint8x16_t> : cpp::false_type {};
+template <>
+LIBC_INLINE uint32_t neq<uint8x16_t>(CPtr p1, CPtr p2, size_t offset) {
+  for (size_t i = 0; i < 2; ++i) {
+    auto a = load<uint64_t>(p1, offset);
+    auto b = load<uint64_t>(p2, offset);
+    uint32_t cond = a != b;
+    if (cond)
+      return cond;
+    offset += sizeof(uint64_t);
+  }
+  return 0;
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint8x16_t>(CPtr p1, CPtr p2, size_t offset) {
+  for (size_t i = 0; i < 2; ++i) {
+    auto a = load_be<uint64_t>(p1, offset);
+    auto b = load_be<uint64_t>(p2, offset);
+    if (a != b)
+      return cmp_neq_uint64_t(a, b);
+    offset += sizeof(uint64_t);
+  }
+  return MemcmpReturnType::ZERO();
+}
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint8x16x2_t
+template <> struct is_vector<uint8x16x2_t> : cpp::true_type {};
+template <> struct cmp_is_expensive<uint8x16x2_t> : cpp::false_type {};
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint8x16x2_t>(CPtr p1, CPtr p2,
+                                               size_t offset) {
+  for (size_t i = 0; i < 4; ++i) {
+    auto a = load_be<uint64_t>(p1, offset);
+    auto b = load_be<uint64_t>(p2, offset);
+    if (a != b)
+      return cmp_neq_uint64_t(a, b);
+    offset += sizeof(uint64_t);
+  }
+  return MemcmpReturnType::ZERO();
+}
+} // namespace __llvm_libc::generic
+
 #endif // LIBC_TARGET_ARCH_IS_AARCH64
 
 #endif // LLVM_LIBC_SRC_STRING_MEMORY_UTILS_OP_AARCH64_H

libc/src/string/memory_utils/op_generic.h

 #include "src/__support/common.h"
 #include "src/__support/endian.h"
 #include "src/__support/macros/optimization.h"
 #include "src/string/memory_utils/op_builtin.h"
 #include "src/string/memory_utils/utils.h"
 
 #include <stdint.h>
 
+static_assert((UINTPTR_MAX == 4294967295U) ||
+                  (UINTPTR_MAX == 18446744073709551615UL),
+              "We currently only support 32- or 64-bit platforms");
+
+#if defined(UINT64_MAX)
+#define LLVM_LIBC_HAS_UINT64
+#endif
+
 namespace __llvm_libc {
 // Compiler types using the vector attributes.
-using uint8x1_t = uint8_t __attribute__((__vector_size__(1)));
-using uint8x2_t = uint8_t __attribute__((__vector_size__(2)));
-using uint8x4_t = uint8_t __attribute__((__vector_size__(4)));
-using uint8x8_t = uint8_t __attribute__((__vector_size__(8)));
-using uint8x16_t = uint8_t __attribute__((__vector_size__(16)));
-using uint8x32_t = uint8_t __attribute__((__vector_size__(32)));
-using uint8x64_t = uint8_t __attribute__((__vector_size__(64)));
+using generic_v128 = uint8_t __attribute__((__vector_size__(16)));
+using generic_v256 = uint8_t __attribute__((__vector_size__(32)));
+using generic_v512 = uint8_t __attribute__((__vector_size__(64)));
 } // namespace __llvm_libc
 
 namespace __llvm_libc::generic {
+
 // We accept three types of values as elements for generic operations:
-// - scalar : unsigned integral types
-// - vector : compiler types using the vector attributes
+// - scalar : unsigned integral types,
+// - vector : compiler types using the vector attributes or platform builtins,
 // - array  : a cpp::array<T, N> where T is itself either a scalar or a vector.
 // The following traits help discriminate between these cases.
-template <typename T>
-constexpr bool is_scalar_v = cpp::is_integral_v<T> && cpp::is_unsigned_v<T>;
 
-template <typename T>
-constexpr bool is_vector_v =
-    cpp::details::is_unqualified_any_of<T, uint8x1_t, uint8x2_t, uint8x4_t,
-                                        uint8x8_t, uint8x16_t, uint8x32_t,
-                                        uint8x64_t>();
+template <typename T> struct is_scalar : cpp::false_type {};
+template <> struct is_scalar<uint8_t> : cpp::true_type {};
+template <> struct is_scalar<uint16_t> : cpp::true_type {};
+template <> struct is_scalar<uint32_t> : cpp::true_type {};
+#ifdef LLVM_LIBC_HAS_UINT64
+template <> struct is_scalar<uint64_t> : cpp::true_type {};
+#endif // LLVM_LIBC_HAS_UINT64
+template <typename T> constexpr bool is_scalar_v = is_scalar<T>::value;
+
+template <typename T> struct is_vector : cpp::false_type {};
+template <> struct is_vector<generic_v128> : cpp::true_type {};
+template <> struct is_vector<generic_v256> : cpp::true_type {};
+template <> struct is_vector<generic_v512> : cpp::true_type {};
+template <typename T> constexpr bool is_vector_v = is_vector<T>::value;
 
 template <class T> struct is_array : cpp::false_type {};
 template <class T, size_t N> struct is_array<cpp::array<T, N>> {
   static constexpr bool value = is_scalar_v<T> || is_vector_v<T>;
 };
 template <typename T> constexpr bool is_array_v = is_array<T>::value;
 
 template <typename T>
 constexpr bool is_element_type_v =
     is_scalar_v<T> || is_vector_v<T> || is_array_v<T>;
 
-//
+// Helper struct to retrieve the number of elements of an array.
 template <class T> struct array_size {};
 template <class T, size_t N>
 struct array_size<cpp::array<T, N>> : cpp::integral_constant<size_t, N> {};
 template <typename T> constexpr size_t array_size_v = array_size<T>::value;
 
 // Generic operations for the above type categories.
 
 template <typename T> T load(CPtr src) {
else if constexpr (is_vector_v<T>) {
     T Out;
     // This for loop is optimized out for vector types.
     for (size_t i = 0; i < sizeof(T); ++i)
       Out[i] = value;
     return Out;
   }
 }
 
-static_assert((UINTPTR_MAX == 4294967295U) ||
-                  (UINTPTR_MAX == 18446744073709551615UL),
-              "We currently only support 32- or 64-bit platforms");
-
-#if defined(LIBC_TARGET_ARCH_IS_X86_64) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
-#define LLVM_LIBC_HAS_UINT64
-#endif
-
-namespace details {
-// Checks that each type is sorted in strictly decreasing order of size.
-// i.e. sizeof(First) > sizeof(Second) > ... > sizeof(Last)
-template <typename First> constexpr bool is_decreasing_size() {
-  return sizeof(First) == 1;
-}
-template <typename First, typename Second, typename... Next>
-constexpr bool is_decreasing_size() {
-  if constexpr (sizeof...(Next) > 0)
-    return sizeof(First) > sizeof(Second) && is_decreasing_size<Next...>();
-  else
-    return sizeof(First) > sizeof(Second) && is_decreasing_size<Second>();
-}
-
-template <size_t Size, typename... Ts> struct Largest;
-template <size_t Size> struct Largest<Size> : cpp::type_identity<uint8_t> {};
-template <size_t Size, typename T, typename... Ts>
-struct Largest<Size, T, Ts...> {
-  using next = Largest<Size, Ts...>;
-  using type = cpp::conditional_t<(Size >= sizeof(T)), T, typename next::type>;
-};
-
-} // namespace details
-
-// 'SupportedTypes' holds a list of natively supported types.
-// The types are instanciations of ScalarType or VectorType.
-// They should be ordered in strictly decreasing order.
-// The 'TypeFor<Size>' type retrieves is the largest supported type that can
-// handle 'Size' bytes. e.g.
-//
-// using ST = SupportedTypes<ScalarType<uint16_t>, ScalarType<uint8_t>>;
-// using Type = ST::TypeFor<10>;
-// static_assert(cpp:is_same_v<Type, ScalarType<uint16_t>>);
-
-template <typename First, typename... Ts> struct SupportedTypes {
-  static_assert(details::is_decreasing_size<First, Ts...>());
-
-  using MaxType = First;
-
-  template <size_t Size>
-  using TypeFor = typename details::Largest<Size, First, Ts...>::type;
-};
-
-// Map from sizes to structures offering static load, store and splat methods.
-// Note: On platforms lacking vector support, we use the ArrayType below and
-// decompose the operation in smaller pieces.
-
-// Lists a generic native types to use for Memset and Memmove operations.
-// TODO: Inject the native types within Memset and Memmove depending on the
-// target architectures and derive MaxSize from it.
-using NativeTypeMap = SupportedTypes<uint8x64_t, //
-                                     uint8x32_t, //
-                                     uint8x16_t,
-#if defined(LLVM_LIBC_HAS_UINT64)
-                                     uint64_t, // Not available on 32bit
-#endif
-                                     uint32_t, //
-                                     uint16_t, //
-                                     uint8_t>;
-
-namespace details {
-
-// Helper to test if a type is void.
-template <typename T> inline constexpr bool is_void_v = cpp::is_same_v<T, void>;
-
-// In case the 'Size' is not supported we can fall back to a sequence of smaller
-// operations using the largest natively supported type.
-template <size_t Size, size_t MaxSize> static constexpr bool useArrayType() {
-  return (Size > MaxSize) && ((Size % MaxSize) == 0) &&
-         !details::is_void_v<NativeTypeMap::TypeFor<MaxSize>>;
-}
-
-// Compute the type to handle an operation of 'Size' bytes knowing that the
-// underlying platform only support native types up to MaxSize bytes.
-template <size_t Size, size_t MaxSize>
-using getTypeFor = cpp::conditional_t<
-    useArrayType<Size, MaxSize>(),
-    cpp::array<NativeTypeMap::TypeFor<MaxSize>, Size / MaxSize>,
-    NativeTypeMap::TypeFor<Size>>;
-
-} // namespace details
-
 ///////////////////////////////////////////////////////////////////////////////
 // Memset
 ///////////////////////////////////////////////////////////////////////////////
 
 template <typename T> struct Memset {
+  static_assert(is_element_type_v<T>);
   static constexpr size_t SIZE = sizeof(T);
 
   LIBC_INLINE static void block(Ptr dst, uint8_t value) {
-    static_assert(is_element_type_v<T>);
     if constexpr (is_scalar_v<T> || is_vector_v<T>) {
       store<T>(dst, splat<T>(value));
     } else if constexpr (is_array_v<T>) {
       using value_type = typename T::value_type;
       const auto Splat = splat<value_type>(value);
       for (size_t I = 0; I < array_size_v<T>; ++I)
         store<value_type>(dst + (I * sizeof(value_type)), Splat);
     }
LIBC_INLINE static void loop_and_tail(Ptr dst, uint8_t value, size_t count) {
     tail(dst, value, count);
   }
 };
 
 template <typename T, typename... TS> struct MemsetSequence {
   static constexpr size_t SIZE = (sizeof(T) + ... + sizeof(TS));
   LIBC_INLINE static void block(Ptr dst, uint8_t value) {
     Memset<T>::block(dst, value);
-    if constexpr (sizeof...(TS) > 0) {
+    if constexpr (sizeof...(TS) > 0)
       return MemsetSequence<TS...>::block(dst + sizeof(T), value);
-    }
-  }
 };
 
 ///////////////////////////////////////////////////////////////////////////////
 // Memmove
 ///////////////////////////////////////////////////////////////////////////////
 
 template <typename T> struct Memmove {
+  static_assert(is_element_type_v<T>);
   static constexpr size_t SIZE = sizeof(T);
 
   LIBC_INLINE static void block(Ptr dst, CPtr src) {
     store<T>(dst, load<T>(src));
   }
 
   LIBC_INLINE static void head_tail(Ptr dst, CPtr src, size_t count) {
     const size_t offset = count - SIZE;
do {
       block(dst + offset, src + offset);
       offset -= SIZE;
     } while (offset >= 0);
     store<T>(dst, head_value);
   }
 };
 
 ///////////////////////////////////////////////////////////////////////////////
-// Bcmp
+// Low level operations for Bcmp and Memcmp that operate on memory locations.
 ///////////////////////////////////////////////////////////////////////////////
-template <size_t Size> struct Bcmp {
-  static constexpr size_t SIZE = Size;
-  static constexpr size_t MaxSize = LLVM_LIBC_IS_DEFINED(LLVM_LIBC_HAS_UINT64)
-                                        ? sizeof(uint64_t)
-                                        : sizeof(uint32_t);
-
-  template <typename T> LIBC_INLINE static uint32_t load_xor(CPtr p1, CPtr p2) {
-    static_assert(sizeof(T) <= sizeof(uint32_t));
-    return load<T>(p1) ^ load<T>(p2);
-  }
 
-  template <typename T>
-  LIBC_INLINE static uint32_t load_not_equal(CPtr p1, CPtr p2) {
-    return load<T>(p1) != load<T>(p2);
-  }
+// Same as load above but with an offset to the pointer.
+// Making the offset explicit hints the compiler to use relevant addressing mode
+// consistently.
+template <typename T> LIBC_INLINE static T load(CPtr ptr, size_t offset) {
+  return ::__llvm_libc::load<T>(ptr + offset);
+}
+
+// Same as above but also makes sure the loaded value is in big endian format.
+// This is useful when implementing lexicograhic comparisons as big endian
+// scalar comparison directly maps to lexicographic byte comparisons.
+template <typename T> LIBC_INLINE static T load_be(CPtr ptr, size_t offset) {
+  return Endian::to_big_endian(load<T>(ptr, offset));
+}
+
+// Equality: returns true iff values at locations (p1 + offset) and (p2 +
+// offset) compare equal.
+template <typename T> static bool eq(CPtr p1, CPtr p2, size_t offset);
+
+// Not equals: returns non-zero iff values at locations (p1 + offset) and (p2 +
+// offset) differ.
+template <typename T> static uint32_t neq(CPtr p1, CPtr p2, size_t offset);
+
+// Lexicographic comparison:
+// - returns 0 iff values at locations (p1 + offset) and (p2 + offset) compare
+//   equal.
+// - returns a negative value if value at location (p1 + offset) is
+//   lexicographically less than value at (p2 + offset).
+// - returns a positive value if value at location (p1 + offset) is
+//   lexicographically greater than value at (p2 + offset).
+static MemcmpReturnType cmp(CPtr p1, CPtr p2, size_t offset);
 
-  LIBC_INLINE static BcmpReturnType block(CPtr p1, CPtr p2) {
-    if constexpr (Size == 1) {
-      return load_xor<uint8_t>(p1, p2);
-    } else if constexpr (Size == 2) {
-      return load_xor<uint16_t>(p1, p2);
-    } else if constexpr (Size == 4) {
-      return load_xor<uint32_t>(p1, p2);
-    } else if constexpr (Size == 8) {
-      return load_not_equal<uint64_t>(p1, p2);
-    } else if constexpr (details::useArrayType<Size, MaxSize>()) {
-      for (size_t offset = 0; offset < Size; offset += MaxSize)
-        if (auto value = Bcmp<MaxSize>::block(p1 + offset, p2 + offset))
-          return value;
+// Lexicographic comparison of non-equal values:
+// - returns a negative value if value at location (p1 + offset) is
+//   lexicographically less than value at (p2 + offset).
+// - returns a positive value if value at location (p1 + offset) is
+//   lexicographically greater than value at (p2 + offset).
+template <typename T>
+static MemcmpReturnType cmp_neq(CPtr p1, CPtr p2, size_t offset);
+
+///////////////////////////////////////////////////////////////////////////////
+// Memcmp implementation
+//
+// When building memcmp, not all types are considered equals.
+//
+// For instance, the lexicographic comparison of two uint8_t can be implemented
+// as a simple subtraction, but for wider operations the logic can be much more
+// involving, especially on little endian platforms.
+//
+// For such wider types it is a good strategy to test for equality first and
+// only do the expensive lexicographic comparison if necessary.
+//
+// Decomposing the algorithm like this for wider types allows us to have
+// efficient implementation of higher order functions like 'head_tail' or
+// 'loop_and_tail'.
+///////////////////////////////////////////////////////////////////////////////
+
+// Type traits to decide whether we can use 'cmp' directly or if we need to
+// split the computation.
+template <typename T> struct cmp_is_expensive;
+
+template <typename T> struct Memcmp {
+  static_assert(is_element_type_v<T>);
+  static constexpr size_t SIZE = sizeof(T);
+
+private:
+  LIBC_INLINE static MemcmpReturnType block_offset(CPtr p1, CPtr p2,
+                                                   size_t offset) {
+    if constexpr (cmp_is_expensive<T>::value) {
+      if (!eq<T>(p1, p2, offset))
+        return cmp_neq<T>(p1, p2, offset);
+      return MemcmpReturnType::ZERO();
     } else {
-      deferred_static_assert("Unimplemented Size");
+      return cmp<T>(p1, p2, offset);
     }
-    return BcmpReturnType::ZERO();
   }
 
-  LIBC_INLINE static BcmpReturnType tail(CPtr p1, CPtr p2, size_t count) {
-    return block(p1 + count - SIZE, p2 + count - SIZE);
+public:
+  LIBC_INLINE static MemcmpReturnType block(CPtr p1, CPtr p2) {
+    return block_offset(p1, p2, 0);
   }
 
-  LIBC_INLINE static BcmpReturnType head_tail(CPtr p1, CPtr p2, size_t count) {
-    return block(p1, p2) | tail(p1, p2, count);
+  LIBC_INLINE static MemcmpReturnType tail(CPtr p1, CPtr p2, size_t count) {
+    return block_offset(p1, p2, count - SIZE);
   }
 
-  LIBC_INLINE static BcmpReturnType loop_and_tail(CPtr p1, CPtr p2,
-                                                  size_t count) {
-    static_assert(Size > 1, "a loop of size 1 does not need tail");
-    size_t offset = 0;
-    do {
-      if (auto value = block(p1 + offset, p2 + offset))
+  LIBC_INLINE static MemcmpReturnType head_tail(CPtr p1, CPtr p2,
+    if constexpr (cmp_is_expensive<T>::value) {
+      if (!eq<T>(p1, p2, 0))
+        return cmp_neq<T>(p1, p2, 0);
+    } else {
+      if (const auto value = cmp<T>(p1, p2, 0))
         return value;
-      offset += SIZE;
-    } while (offset < count - SIZE);
+    }
     return tail(p1, p2, count);
   }
-};
 
-///////////////////////////////////////////////////////////////////////////////
-// Memcmp
-///////////////////////////////////////////////////////////////////////////////
-template <size_t Size> struct Memcmp {
-  static constexpr size_t SIZE = Size;
-  static constexpr size_t MaxSize = LLVM_LIBC_IS_DEFINED(LLVM_LIBC_HAS_UINT64)
-                                        ? sizeof(uint64_t)
-                                        : sizeof(uint32_t);
-
-  template <typename T> LIBC_INLINE static T load_be(CPtr ptr) {
-    return Endian::to_big_endian(load<T>(ptr));
+  LIBC_INLINE static MemcmpReturnType loop_and_tail(CPtr p1, CPtr p2,
+                                                    size_t count) {
+    return loop_and_tail_offset(p1, p2, count, 0);
   }
 
-  template <typename T>
-  LIBC_INLINE static MemcmpReturnType load_be_diff(CPtr p1, CPtr p2) {
-    return load_be<T>(p1) - load_be<T>(p2);
+  LIBC_INLINE static MemcmpReturnType
+  loop_and_tail_offset(CPtr p1, CPtr p2, size_t count, size_t offset) {
+    if constexpr (SIZE > 1) {
+      const size_t limit = count - SIZE;
+      LIBC_LOOP_NOUNROLL
+      for (; offset < limit; offset += SIZE) {
+        if constexpr (cmp_is_expensive<T>::value) {
+          if (!eq<T>(p1, p2, offset))
+            return cmp_neq<T>(p1, p2, offset);
+        } else {
+          if (const auto value = cmp<T>(p1, p2, offset))
+            return value;
+        }
+      }
+      return block_offset(p1, p2, limit); // tail
+    } else {
+      // No need for a tail operation when SIZE == 1.
+      LIBC_LOOP_NOUNROLL
+      for (; offset < count; offset += SIZE)
+        if (auto value = cmp<T>(p1, p2, offset))
+          return value;
+      return MemcmpReturnType::ZERO();
+    }
   }
 
-  template <typename T>
-  LIBC_INLINE static MemcmpReturnType load_be_cmp(CPtr p1, CPtr p2) {
-    const auto la = load_be<T>(p1);
-    const auto lb = load_be<T>(p2);
-    return la > lb ? 1 : la < lb ? -1 : 0;
+  LIBC_INLINE static MemcmpReturnType
+  loop_and_tail_align_above(size_t threshold, CPtr p1, CPtr p2, size_t count) {
+    const AlignHelper<sizeof(T)> helper(p1);
+    if (LIBC_UNLIKELY(count >= threshold) && helper.not_aligned()) {
+      if (auto value = block(p1, p2))
+        return value;
+      adjust(helper.offset(), p1, p2, count);
+    }
+    return loop_and_tail(p1, p2, count);
   }
+};
 
+template <typename T, typename... TS> struct MemcmpSequence {
+  static constexpr size_t SIZE = (sizeof(T) + ... + sizeof(TS));
   LIBC_INLINE static MemcmpReturnType block(CPtr p1, CPtr p2) {
-    if constexpr (Size == 1) {
-      return load_be_diff<uint8_t>(p1, p2);
-    } else if constexpr (Size == 2) {
-      return load_be_diff<uint16_t>(p1, p2);
-    } else if constexpr (Size == 4) {
-      return load_be_cmp<uint32_t>(p1, p2);
-    } else if constexpr (Size == 8) {
-      return load_be_cmp<uint64_t>(p1, p2);
-    } else if constexpr (details::useArrayType<Size, MaxSize>()) {
-      for (size_t offset = 0; offset < Size; offset += MaxSize)
-        if (Bcmp<MaxSize>::block(p1 + offset, p2 + offset))
-          return Memcmp<MaxSize>::block(p1 + offset, p2 + offset);
-      return MemcmpReturnType::ZERO();
-    } else if constexpr (Size == 3) {
-      if (auto value = Memcmp<2>::block(p1, p2))
-        return value;
-      return Memcmp<1>::block(p1 + 2, p2 + 2);
+    // TODO: test suggestion in
+    // https://reviews.llvm.org/D148717?id=515724#inline-1446890
+    // once we have a proper way to check memory operation latency.
+    if constexpr (cmp_is_expensive<T>::value) {
+      if (!eq<T>(p1, p2, 0))
+        return cmp_neq<T>(p1, p2, 0);
     } else {
-      deferred_static_assert("Unimplemented Size");
+      if (auto value = cmp<T>(p1, p2, 0))
+        return value;
     }
+    if constexpr (sizeof...(TS) > 0)
+      return MemcmpSequence<TS...>::block(p1 + sizeof(T), p2 + sizeof(T));
+    else
+      return MemcmpReturnType::ZERO();
   }
+};
 
-  LIBC_INLINE static MemcmpReturnType tail(CPtr p1, CPtr p2, size_t count) {
-    return block(p1 + count - SIZE, p2 + count - SIZE);
+///////////////////////////////////////////////////////////////////////////////
+// Bcmp
+///////////////////////////////////////////////////////////////////////////////
+template <typename T> struct Bcmp {
+  static_assert(is_element_type_v<T>);
+  static constexpr size_t SIZE = sizeof(T);
+
+  LIBC_INLINE static BcmpReturnType block(CPtr p1, CPtr p2) {
+    return neq<T>(p1, p2, 0);
   }
 
-  LIBC_INLINE static MemcmpReturnType head_tail(CPtr p1, CPtr p2,
-                                                size_t count) {
-    if (auto value = block(p1, p2))
+  LIBC_INLINE static BcmpReturnType tail(CPtr p1, CPtr p2, size_t count) {
+    const size_t tail_offset = count - SIZE;
+    return neq<T>(p1, p2, tail_offset);
+  }
+
+  LIBC_INLINE static BcmpReturnType head_tail(CPtr p1, CPtr p2, size_t count) {
+    if (const auto value = neq<T>(p1, p2, 0))
       return value;
     return tail(p1, p2, count);
   }
 
-  LIBC_INLINE static MemcmpReturnType loop_and_tail(CPtr p1, CPtr p2,
-                                                    size_t count) {
-    static_assert(Size > 1, "a loop of size 1 does not need tail");
-    size_t offset = 0;
-    do {
-      if (auto value = block(p1 + offset, p2 + offset))
+  LIBC_INLINE static BcmpReturnType loop_and_tail(CPtr p1, CPtr p2,
+    return loop_and_tail_offset(p1, p2, count, 0);
+  }
+
+  LIBC_INLINE static BcmpReturnType
+  loop_and_tail_offset(CPtr p1, CPtr p2, size_t count, size_t offset) {
+    if constexpr (SIZE > 1) {
+      const size_t limit = count - SIZE;
+      LIBC_LOOP_NOUNROLL
+      for (; offset < limit; offset += SIZE)
+        if (const auto value = neq<T>(p1, p2, offset))
           return value;
-      offset += SIZE;
-    } while (offset < count - SIZE);
       return tail(p1, p2, count);
+    } else {
+      // No need for a tail operation when SIZE == 1.
+      LIBC_LOOP_NOUNROLL
+      for (; offset < count; offset += SIZE)
+        if (const auto value = neq<T>(p1, p2, offset))
+          return value;
+      return BcmpReturnType::ZERO();
+    }
+  }
+
+  LIBC_INLINE static BcmpReturnType
+  loop_and_tail_align_above(size_t threshold, CPtr p1, CPtr p2, size_t count) {
+    static_assert(SIZE > 1,
+                  "No need to align when processing one byte at a time");
+    const AlignHelper<sizeof(T)> helper(p1);
+    if (LIBC_UNLIKELY(count >= threshold) && helper.not_aligned()) {
+      if (auto value = block(p1, p2))
+        return value;
+      adjust(helper.offset(), p1, p2, count);
+    }
+    return loop_and_tail(p1, p2, count);
   }
 };
 
+template <typename T, typename... TS> struct BcmpSequence {
+  static constexpr size_t SIZE = (sizeof(T) + ... + sizeof(TS));
+  LIBC_INLINE static BcmpReturnType block(CPtr p1, CPtr p2) {
+    if (auto value = neq<T>(p1, p2, 0))
+      return value;
+    if constexpr (sizeof...(TS) > 0)
+      return BcmpSequence<TS...>::block(p1 + sizeof(T), p2 + sizeof(T));
+    else
+      return BcmpReturnType::ZERO();
+  }
+};
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint8_t
+template <> struct cmp_is_expensive<uint8_t> : public cpp::false_type {};
+template <> LIBC_INLINE bool eq<uint8_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint8_t>(p1, offset) == load<uint8_t>(p2, offset);
+}
+template <> LIBC_INLINE uint32_t neq<uint8_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint8_t>(p1, offset) ^ load<uint8_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint8_t>(CPtr p1, CPtr p2, size_t offset) {
+  return static_cast<int32_t>(load<uint8_t>(p1, offset)) -
+         static_cast<int32_t>(load<uint8_t>(p2, offset));
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint8_t>(CPtr p1, CPtr p2, size_t offset);
 } // namespace __llvm_libc::generic
 
 #endif // LLVM_LIBC_SRC_STRING_MEMORY_UTILS_OP_GENERIC_H

libc/src/string/memory_utils/op_riscv.h

+//===-- RISC-V implementation of memory function building blocks ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides x86 specific building blocks to compose memory functions.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_LIBC_SRC_STRING_MEMORY_UTILS_OP_RISCV_H
+#define LLVM_LIBC_SRC_STRING_MEMORY_UTILS_OP_RISCV_H
+
+#include "src/__support/macros/properties/architectures.h"
+
+#if defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
+
+#include "src/__support/common.h"
+#include "src/string/memory_utils/op_generic.h"
+
+namespace __llvm_libc::generic {
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint16_t
+template <> struct cmp_is_expensive<uint16_t> : public cpp::false_type {};
+template <> LIBC_INLINE bool eq<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint16_t>(p1, offset) == load<uint16_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint16_t>(p1, offset) ^ load<uint16_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return static_cast<int32_t>(load_be<uint16_t>(p1, offset)) -
+         static_cast<int32_t>(load_be<uint16_t>(p2, offset));
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint16_t>(CPtr p1, CPtr p2, size_t offset);
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint32_t
+template <> struct cmp_is_expensive<uint32_t> : public cpp::false_type {};
+template <> LIBC_INLINE bool eq<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint32_t>(p1, offset) == load<uint32_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint32_t>(p1, offset) ^ load<uint32_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load_be<uint32_t>(p1, offset);
+  const auto b = load_be<uint32_t>(p2, offset);
+  return cmp_uint32_t(a, b);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint32_t>(CPtr p1, CPtr p2, size_t offset);
+
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint64_t
+template <> struct cmp_is_expensive<uint64_t> : public cpp::true_type {};
+template <> LIBC_INLINE bool eq<uint64_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint64_t>(p1, offset) == load<uint64_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint64_t>(CPtr p1, CPtr p2, size_t offset) {
+  return !eq<uint64_t>(p1, p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint64_t>(CPtr p1, CPtr p2, size_t offset);
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint64_t>(CPtr p1, CPtr p2,
+                                               size_t offset) {
+  const auto a = load_be<uint64_t>(p1, offset);
+  const auto b = load_be<uint64_t>(p2, offset);
+  return cmp_neq_uint64_t(a, b);
+}
+
+} // namespace __llvm_libc::generic
+
+#endif // LIBC_TARGET_ARCH_IS_ANY_RISCV
+#endif // LLVM_LIBC_SRC_STRING_MEMORY_UTILS_OP_RISCV_H

libc/src/string/memory_utils/op_x86.h

 #endif
 #if !defined(__SSE2__) && (defined(_MSC_VER) || defined(__SCE__))
 #define _mm_movemask_epi8(A) 0
 #endif
 
 namespace __llvm_libc::x86 {
 
 // A set of constants to check compile time features.
-static inline constexpr bool kSse2 = LLVM_LIBC_IS_DEFINED(__SSE2__);
-static inline constexpr bool kAvx = LLVM_LIBC_IS_DEFINED(__AVX__);
-static inline constexpr bool kAvx2 = LLVM_LIBC_IS_DEFINED(__AVX2__);
-static inline constexpr bool kAvx512F = LLVM_LIBC_IS_DEFINED(__AVX512F__);
-static inline constexpr bool kAvx512BW = LLVM_LIBC_IS_DEFINED(__AVX512BW__);
+static LIBC_INLINE constexpr bool kSse2 = LLVM_LIBC_IS_DEFINED(__SSE2__);
+static LIBC_INLINE constexpr bool kSse41 = LLVM_LIBC_IS_DEFINED(__SSE4_1__);
+static LIBC_INLINE constexpr bool kAvx = LLVM_LIBC_IS_DEFINED(__AVX__);
+static LIBC_INLINE constexpr bool kAvx2 = LLVM_LIBC_IS_DEFINED(__AVX2__);
+static LIBC_INLINE constexpr bool kAvx512F = LLVM_LIBC_IS_DEFINED(__AVX512F__);
+static LIBC_INLINE constexpr bool kAvx512BW =
+    LLVM_LIBC_IS_DEFINED(__AVX512BW__);
 
 ///////////////////////////////////////////////////////////////////////////////
 // Memcpy repmovsb implementation
 struct Memcpy {
   static void repmovsb(void *dst, const void *src, size_t count) {
     asm volatile("rep movsb" : "+D"(dst), "+S"(src), "+c"(count) : : "memory");
   }
 };
 
-///////////////////////////////////////////////////////////////////////////////
-// Bcmp
+} // namespace __llvm_libc::x86
 
-// Base implementation for the Bcmp specializations.
-//  - BlockSize is either 16, 32 or 64 depending on the available compile time
-// features, it is used to switch between "single native operation" or a
-// "sequence of native operations".
-//  - BlockBcmp is the function that implements the bcmp logic.
-template <size_t Size, size_t BlockSize, auto BlockBcmp> struct BcmpImpl {
-  static constexpr size_t SIZE = Size;
-  LIBC_INLINE static BcmpReturnType block(CPtr p1, CPtr p2) {
-    if constexpr (Size == BlockSize) {
-      return BlockBcmp(p1, p2);
-    } else if constexpr (Size % BlockSize == 0) {
-      for (size_t offset = 0; offset < Size; offset += BlockSize)
-        if (auto value = BlockBcmp(p1 + offset, p2 + offset))
-          return value;
-    } else {
-      deferred_static_assert("SIZE not implemented");
-    }
-    return BcmpReturnType::ZERO();
-  }
-
-  LIBC_INLINE static BcmpReturnType tail(CPtr p1, CPtr p2, size_t count) {
-    return block(p1 + count - Size, p2 + count - Size);
-  }
-
-  LIBC_INLINE static BcmpReturnType head_tail(CPtr p1, CPtr p2, size_t count) {
-    return block(p1, p2) | tail(p1, p2, count);
-  }
-
-  LIBC_INLINE static BcmpReturnType loop_and_tail(CPtr p1, CPtr p2,
-                                                  size_t count) {
-    static_assert(Size > 1, "a loop of size 1 does not need tail");
-    size_t offset = 0;
-    do {
-      if (auto value = block(p1 + offset, p2 + offset))
-        return value;
-      offset += Size;
-    } while (offset < count - Size);
-    return tail(p1, p2, count);
-  }
-};
+namespace __llvm_libc::generic {
 
-namespace sse2 {
-LIBC_INLINE BcmpReturnType bcmp16(CPtr p1, CPtr p2) {
-#if defined(__SSE2__)
-  using T = char __attribute__((__vector_size__(16)));
-  // A mask indicating which bytes differ after loading 16 bytes from p1 and p2.
-  const int mask =
-      _mm_movemask_epi8(cpp::bit_cast<__m128i>(load<T>(p1) != load<T>(p2)));
-  return static_cast<uint32_t>(mask);
-#else
-  (void)p1;
-  (void)p2;
-  return BcmpReturnType::ZERO();
-#endif // defined(__SSE2__)
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint16_t
+template <> struct cmp_is_expensive<uint16_t> : public cpp::false_type {};
+template <> LIBC_INLINE bool eq<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint16_t>(p1, offset) == load<uint16_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint16_t>(p1, offset) ^ load<uint16_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint16_t>(CPtr p1, CPtr p2, size_t offset) {
+  return static_cast<int32_t>(load_be<uint16_t>(p1, offset)) -
+         static_cast<int32_t>(load_be<uint16_t>(p2, offset));
 }
-template <size_t Size> using Bcmp = BcmpImpl<Size, 16, bcmp16>;
-} // namespace sse2
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint16_t>(CPtr p1, CPtr p2, size_t offset);
 
-namespace avx2 {
-LIBC_INLINE BcmpReturnType bcmp32(CPtr p1, CPtr p2) {
-#if defined(__AVX2__)
-  using T = char __attribute__((__vector_size__(32)));
-  // A mask indicating which bytes differ after loading 32 bytes from p1 and p2.
-  const int mask =
-      _mm256_movemask_epi8(cpp::bit_cast<__m256i>(load<T>(p1) != load<T>(p2)));
-  // _mm256_movemask_epi8 returns an int but it is to be interpreted as a 32-bit
-  // mask.
-  return static_cast<uint32_t>(mask);
-#else
-  (void)p1;
-  (void)p2;
-  return BcmpReturnType::ZERO();
-#endif // defined(__AVX2__)
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint32_t
+template <> struct cmp_is_expensive<uint32_t> : public cpp::false_type {};
+template <> LIBC_INLINE bool eq<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint32_t>(p1, offset) == load<uint32_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint32_t>(p1, offset) ^ load<uint32_t>(p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint32_t>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load_be<uint32_t>(p1, offset);
+  const auto b = load_be<uint32_t>(p2, offset);
+  return cmp_uint32_t(a, b);
 }
-template <size_t Size> using Bcmp = BcmpImpl<Size, 32, bcmp32>;
-} // namespace avx2
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint32_t>(CPtr p1, CPtr p2, size_t offset);
 
-namespace avx512bw {
-LIBC_INLINE BcmpReturnType bcmp64(CPtr p1, CPtr p2) {
-#if defined(__AVX512BW__)
-  using T = char __attribute__((__vector_size__(64)));
-  // A mask indicating which bytes differ after loading 64 bytes from p1 and p2.
-  const uint64_t mask = _mm512_cmpneq_epi8_mask(
-      cpp::bit_cast<__m512i>(load<T>(p1)), cpp::bit_cast<__m512i>(load<T>(p2)));
-  const bool mask_is_set = mask != 0;
-  return static_cast<uint32_t>(mask_is_set);
-#else
-  (void)p1;
-  (void)p2;
-  return BcmpReturnType::ZERO();
-#endif // defined(__AVX512BW__)
-}
-template <size_t Size> using Bcmp = BcmpImpl<Size, 64, bcmp64>;
-} // namespace avx512bw
-
-// Assuming that the mask is non zero, the index of the first mismatching byte
-// is the number of trailing zeros in the mask. Trailing zeros and not leading
-// zeros because the x86 architecture is little endian.
-LIBC_INLINE MemcmpReturnType char_diff_no_zero(CPtr p1, CPtr p2,
-                                               uint64_t mask) {
-  const size_t diff_index = __builtin_ctzll(mask);
-  const int16_t ca = cpp::to_integer<uint8_t>(p1[diff_index]);
-  const int16_t cb = cpp::to_integer<uint8_t>(p2[diff_index]);
-  return ca - cb;
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for uint64_t
+template <> struct cmp_is_expensive<uint64_t> : public cpp::true_type {};
+template <> LIBC_INLINE bool eq<uint64_t>(CPtr p1, CPtr p2, size_t offset) {
+  return load<uint64_t>(p1, offset) == load<uint64_t>(p2, offset);
+}
+template <>
+LIBC_INLINE uint32_t neq<uint64_t>(CPtr p1, CPtr p2, size_t offset) {
+  return !eq<uint64_t>(p1, p2, offset);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp<uint64_t>(CPtr p1, CPtr p2, size_t offset);
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<uint64_t>(CPtr p1, CPtr p2,
+                                               size_t offset) {
+  const auto a = load_be<uint64_t>(p1, offset);
+  const auto b = load_be<uint64_t>(p2, offset);
+  return cmp_neq_uint64_t(a, b);
 }
 
 ///////////////////////////////////////////////////////////////////////////////
-// Memcmp
-
-// Base implementation for the Memcmp specializations.
-//  - BlockSize is either 16, 32 or 64 depending on the available compile time
-// features, it is used to switch between "single native operation" or a
-// "sequence of native operations".
-//  - BlockMemcmp is the function that implements the memcmp logic.
-//  - BlockBcmp is the function that implements the bcmp logic.
-template <size_t Size, size_t BlockSize, auto BlockMemcmp, auto BlockBcmp>
-struct MemcmpImpl {
-  static constexpr size_t SIZE = Size;
-  LIBC_INLINE static MemcmpReturnType block(CPtr p1, CPtr p2) {
-    if constexpr (Size == BlockSize) {
-      return BlockMemcmp(p1, p2);
-    } else if constexpr (Size % BlockSize == 0) {
-      for (size_t offset = 0; offset < Size; offset += BlockSize)
-        if (auto value = BlockBcmp(p1 + offset, p2 + offset))
-          return BlockMemcmp(p1 + offset, p2 + offset);
-    } else {
-      deferred_static_assert("SIZE not implemented");
-    }
-    return MemcmpReturnType::ZERO();
-  }
-
-  LIBC_INLINE static MemcmpReturnType tail(CPtr p1, CPtr p2, size_t count) {
-    return block(p1 + count - Size, p2 + count - Size);
-  }
-
-  LIBC_INLINE static MemcmpReturnType head_tail(CPtr p1, CPtr p2,
-                                                size_t count) {
-    if (auto value = block(p1, p2))
-      return value;
-    return tail(p1, p2, count);
-  }
-
-  LIBC_INLINE static MemcmpReturnType loop_and_tail(CPtr p1, CPtr p2,
-                                                    size_t count) {
-    static_assert(Size > 1, "a loop of size 1 does not need tail");
-    size_t offset = 0;
-    do {
-      if (auto value = block(p1 + offset, p2 + offset))
-        return value;
-      offset += Size;
-    } while (offset < count - Size);
-    return tail(p1, p2, count);
-  }
-};
+// Specializations for __m128i
+#if defined(__SSE4_1__)
+template <> struct is_vector<__m128i> : cpp::true_type {};
+template <> struct cmp_is_expensive<__m128i> : cpp::true_type {};
+LIBC_INLINE __m128i bytewise_max(__m128i a, __m128i b) {
+  return _mm_max_epu8(a, b);
+}
+LIBC_INLINE __m128i bytewise_reverse(__m128i value) {
+  return _mm_shuffle_epi8(value, _mm_set_epi8(0, 1, 2, 3, 4, 5, 6, 7, //
+                                              8, 9, 10, 11, 12, 13, 14, 15));
+}
+LIBC_INLINE uint16_t big_endian_cmp_mask(__m128i max, __m128i value) {
+  return _mm_movemask_epi8(bytewise_reverse(_mm_cmpeq_epi8(max, value)));
+}
+template <> LIBC_INLINE bool eq<__m128i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m128i>(p1, offset);
+  const auto b = load<__m128i>(p2, offset);
+  const auto xored = _mm_xor_si128(a, b);
+  return _mm_testz_si128(xored, xored) == 1; // 1 iff xored == 0
+}
+template <> LIBC_INLINE uint32_t neq<__m128i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m128i>(p1, offset);
+  const auto b = load<__m128i>(p2, offset);
+  const auto xored = _mm_xor_si128(a, b);
+  return _mm_testz_si128(xored, xored) == 0; // 0 iff xored != 0
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<__m128i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m128i>(p1, offset);
+  const auto b = load<__m128i>(p2, offset);
+  const auto vmax = bytewise_max(a, b);
+  const auto le = big_endian_cmp_mask(vmax, b);
+  const auto ge = big_endian_cmp_mask(vmax, a);
+  static_assert(cpp::is_same_v<cpp::remove_cv_t<decltype(le)>, uint16_t>);
+  return static_cast<int32_t>(ge) - static_cast<int32_t>(le);
+#endif // __SSE4_1__
 
-namespace sse2 {
-LIBC_INLINE MemcmpReturnType memcmp16(CPtr p1, CPtr p2) {
-#if defined(__SSE2__)
-  using T = char __attribute__((__vector_size__(16)));
-  // A mask indicating which bytes differ after loading 16 bytes from p1 and p2.
-  if (int mask =
-          _mm_movemask_epi8(cpp::bit_cast<__m128i>(load<T>(p1) != load<T>(p2))))
-    return char_diff_no_zero(p1, p2, mask);
-  return MemcmpReturnType::ZERO();
-#else
-  (void)p1;
-  (void)p2;
-  return MemcmpReturnType::ZERO();
-#endif // defined(__SSE2__)
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for __m256i
+#if defined(__AVX__)
+template <> struct is_vector<__m256i> : cpp::true_type {};
+template <> struct cmp_is_expensive<__m256i> : cpp::true_type {};
+template <> LIBC_INLINE bool eq<__m256i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m256i>(p1, offset);
+  const auto b = load<__m256i>(p2, offset);
+  const auto xored = _mm256_castps_si256(
+      _mm256_xor_ps(_mm256_castsi256_ps(a), _mm256_castsi256_ps(b)));
+  return _mm256_testz_si256(xored, xored) == 1; // 1 iff xored == 0
+}
+template <> LIBC_INLINE uint32_t neq<__m256i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m256i>(p1, offset);
+  const auto b = load<__m256i>(p2, offset);
+  const auto xored = _mm256_castps_si256(
+      _mm256_xor_ps(_mm256_castsi256_ps(a), _mm256_castsi256_ps(b)));
+  return _mm256_testz_si256(xored, xored) == 0; // 0 iff xored != 0
 }
-template <size_t Size> using Memcmp = MemcmpImpl<Size, 16, memcmp16, bcmp16>;
-} // namespace sse2
+#endif // __AVX__
 
-namespace avx2 {
-LIBC_INLINE MemcmpReturnType memcmp32(CPtr p1, CPtr p2) {
 #if defined(__AVX2__)
-  using T = char __attribute__((__vector_size__(32)));
-  // A mask indicating which bytes differ after loading 32 bytes from p1 and p2.
-  if (int mask = _mm256_movemask_epi8(
-          cpp::bit_cast<__m256i>(load<T>(p1) != load<T>(p2))))
-    return char_diff_no_zero(p1, p2, mask);
-  return MemcmpReturnType::ZERO();
-#else
-  (void)p1;
-  (void)p2;
-  return MemcmpReturnType::ZERO();
-#endif // defined(__AVX2__)
+LIBC_INLINE __m256i bytewise_max(__m256i a, __m256i b) {
+  return _mm256_max_epu8(a, b);
 }
-template <size_t Size> using Memcmp = MemcmpImpl<Size, 32, memcmp32, bcmp32>;
-} // namespace avx2
+LIBC_INLINE __m256i bytewise_reverse(__m256i value) {
+  return _mm256_shuffle_epi8(value,
+                             _mm256_set_epi8(0, 1, 2, 3, 4, 5, 6, 7,         //
+                                             8, 9, 10, 11, 12, 13, 14, 15,   //
+                                             16, 17, 18, 19, 20, 21, 22, 23, //
+                                             24, 25, 26, 27, 28, 29, 30, 31));
+}
+LIBC_INLINE uint32_t big_endian_cmp_mask(__m256i max, __m256i value) {
+  return _mm256_movemask_epi8(bytewise_reverse(_mm256_cmpeq_epi8(max, value)));
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<__m256i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m256i>(p1, offset);
+  const auto b = load<__m256i>(p2, offset);
+  const auto vmax = bytewise_max(a, b);
+  const auto le = big_endian_cmp_mask(vmax, b);
+  const auto ge = big_endian_cmp_mask(vmax, a);
+  static_assert(cpp::is_same_v<cpp::remove_cv_t<decltype(le)>, uint32_t>);
+  return cmp_uint32_t(ge, le);
+}
+#endif // __AVX2__
 
-namespace avx512bw {
-LIBC_INLINE MemcmpReturnType memcmp64(CPtr p1, CPtr p2) {
+///////////////////////////////////////////////////////////////////////////////
+// Specializations for __m512i
 #if defined(__AVX512BW__)
-  using T = char __attribute__((__vector_size__(64)));
-  // A mask indicating which bytes differ after loading 64 bytes from p1 and p2.
-  if (uint64_t mask =
-          _mm512_cmpneq_epi8_mask(cpp::bit_cast<__m512i>(load<T>(p1)),
-                                  cpp::bit_cast<__m512i>(load<T>(p2))))
-    return char_diff_no_zero(p1, p2, mask);
-  return MemcmpReturnType::ZERO();
-#else
-  (void)p1;
-  (void)p2;
-  return MemcmpReturnType::ZERO();
-#endif // defined(__AVX512BW__)
+template <> struct is_vector<__m512i> : cpp::true_type {};
+template <> struct cmp_is_expensive<__m512i> : cpp::true_type {};
+LIBC_INLINE __m512i bytewise_max(__m512i a, __m512i b) {
+  return _mm512_max_epu8(a, b);
+}
+LIBC_INLINE __m512i bytewise_reverse(__m512i value) {
+  return _mm512_shuffle_epi8(value,
+                             _mm512_set_epi8(0, 1, 2, 3, 4, 5, 6, 7,         //
+                                             8, 9, 10, 11, 12, 13, 14, 15,   //
+                                             16, 17, 18, 19, 20, 21, 22, 23, //
+                                             24, 25, 26, 27, 28, 29, 30, 31, //
+                                             32, 33, 34, 35, 36, 37, 38, 39, //
+                                             40, 41, 42, 43, 44, 45, 46, 47, //
+                                             48, 49, 50, 51, 52, 53, 54, 55, //
+                                             56, 57, 58, 59, 60, 61, 62, 63));
+}
+LIBC_INLINE uint64_t big_endian_cmp_mask(__m512i max, __m512i value) {
+  return _mm512_cmpeq_epi8_mask(bytewise_reverse(max), bytewise_reverse(value));
+}
+template <> LIBC_INLINE bool eq<__m512i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m512i>(p1, offset);
+  const auto b = load<__m512i>(p2, offset);
+  return _mm512_cmpneq_epi8_mask(a, b) == 0;
+}
+template <> LIBC_INLINE uint32_t neq<__m512i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m512i>(p1, offset);
+  const auto b = load<__m512i>(p2, offset);
+  const uint64_t xored = _mm512_cmpneq_epi8_mask(a, b);
+  return (xored >> 32) | (xored & 0xFFFFFFFF);
+}
+template <>
+LIBC_INLINE MemcmpReturnType cmp_neq<__m512i>(CPtr p1, CPtr p2, size_t offset) {
+  const auto a = load<__m512i>(p1, offset);
+  const auto b = load<__m512i>(p2, offset);
+  const auto vmax = bytewise_max(a, b);
+  const auto le = big_endian_cmp_mask(vmax, b);
+  const auto ge = big_endian_cmp_mask(vmax, a);
+  static_assert(cpp::is_same_v<cpp::remove_cv_t<decltype(le)>, uint64_t>);
+  return cmp_neq_uint64_t(ge, le);
 }
-template <size_t Size> using Memcmp = MemcmpImpl<Size, 64, memcmp64, bcmp64>;
-} // namespace avx512bw
+#endif // __AVX512BW__
 
-} // namespace __llvm_libc::x86
+} // namespace __llvm_libc::generic
 
 #endif // LIBC_TARGET_ARCH_IS_X86_64
 
 #endif // LLVM_LIBC_SRC_STRING_MEMORY_UTILS_OP_X86_H

libc/src/string/memory_utils/utils.h

 //===-- Memory utils --------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIBC_SRC_MEMORY_UTILS_UTILS_H
 #define LLVM_LIBC_SRC_MEMORY_UTILS_UTILS_H
 
 #include "src/__support/CPP/bit.h"
 #include "src/__support/CPP/cstddef.h"
+#include "src/__support/CPP/limits.h" // cpp::numeric_limits
 #include "src/__support/CPP/type_traits.h"
 #include "src/__support/endian.h"
 #include "src/__support/macros/attributes.h" // LIBC_INLINE
 #include "src/__support/macros/config.h"     // LIBC_HAS_BUILTIN
+#include "src/__support/macros/properties/architectures.h"
 
 #include <stddef.h> // size_t
 #include <stdint.h> // intptr_t / uintptr_t
 
 namespace __llvm_libc {
 
 // Allows compile time error reporting in `if constexpr` branches.
 template <bool flag = false>

 
 private:
   T value;
 };
 
 using MemcmpReturnType = StrictIntegralType<int32_t>;
 using BcmpReturnType = StrictIntegralType<uint32_t>;
 
+// This implements the semantic of 'memcmp' returning a negative value when 'a'
+// is less than 'b', '0' when 'a' equals 'b' and a positive number otherwise.
+LIBC_INLINE MemcmpReturnType cmp_uint32_t(uint32_t a, uint32_t b) {
+  // We perform the difference as an uint64_t.
+  const int64_t diff = static_cast<int64_t>(a) - static_cast<int64_t>(b);
+  // And reduce the uint64_t into an uint32_t.
+  // TODO: provide a detailed explanation.
+  return static_cast<int32_t>((diff >> 1) | (diff & 0xFFFF));
+}
+
+// Returns a negative value if 'a' is less than 'b' and a positive value
+// otherwise. This implements the semantic of 'memcmp' when we know that 'a' and
+// 'b' differ.
+LIBC_INLINE MemcmpReturnType cmp_neq_uint64_t(uint64_t a, uint64_t b) {
+#if defined(LIBC_TARGET_ARCH_IS_X86_64)
+  // On x86, we choose the returned values so that they are just one unit appart
+  // as this allows for better code generation.
+  static constexpr int32_t POSITIVE = cpp::numeric_limits<int32_t>::max();
+  static constexpr int32_t NEGATIVE = cpp::numeric_limits<int32_t>::min();
+  static_assert(cpp::bit_cast<uint32_t>(NEGATIVE) -
+                    cpp::bit_cast<uint32_t>(POSITIVE) ==
+                1);
+#else
+  // On RISC-V we simply use '1' and '-1' as it leads to branchless code.
+  // On ARMv8, both strategies lead to the same performance.
+  static constexpr int32_t POSITIVE = 1;
+  static constexpr int32_t NEGATIVE = -1;
+#endif
+  static_assert(POSITIVE > 0);
+  static_assert(NEGATIVE < 0);
+  return a < b ? NEGATIVE : POSITIVE;
+}
+
 // Loads bytes from memory (possibly unaligned) and materializes them as
 // type.
 template <typename T> LIBC_INLINE T load(CPtr ptr) {
   T Out;
   memcpy_inline<sizeof(T)>(&Out, ptr);
   return Out;
 }
 
 // Stores a value of type T in memory (possibly unaligned).
 template <typename T> LIBC_INLINE void store(Ptr ptr, T value) {
   memcpy_inline<sizeof(T)>(ptr, &value);
 }
 
 // On architectures that do not allow for unaligned access we perform several
 // aligned accesses and recombine them through shifts and logicals operations.
 // For instance, if we know that the pointer is 2-byte aligned we can decompose
 // a 64-bit operation into four 16-bit operations.
 
 // Loads a 'ValueType' by decomposing it into several loads that are assumed to
 // be aligned.
 // e.g. load_aligned<uint32_t, uint16_t, uint16_t>(ptr);
 template <typename ValueType, typename T, typename... TS>
 ValueType load_aligned(CPtr src) {
   static_assert(sizeof(ValueType) >= (sizeof(T) + ... + sizeof(TS)));
void align_to_next_boundary(T1 *__restrict &p1, T2 *__restrict &p2,
   if constexpr (AlignOn == Arg::P1)
     align_p1_to_next_boundary<SIZE>(p1, p2, count);
   else if constexpr (AlignOn == Arg::P2)
     align_p1_to_next_boundary<SIZE>(p2, p1, count); // swapping p1 and p2.
   else
     deferred_static_assert("AlignOn must be either Arg::P1 or Arg::P2");
 }
 
+template <size_t SIZE> struct AlignHelper {
+  AlignHelper(CPtr ptr) : offset_(distance_to_next_aligned<SIZE>(ptr)) {}
+
+  LIBC_INLINE bool not_aligned() const { return offset_ != SIZE; }
+  LIBC_INLINE uintptr_t offset() const { return offset_; }
+
+private:
+  uintptr_t offset_;
+};
+
 } // namespace __llvm_libc
 
 #endif // LLVM_LIBC_SRC_MEMORY_UTILS_UTILS_H

libc/src/string/memory_utils/x86_64/memcmp_implementations.h

 #include "src/string/memory_utils/op_generic.h"
 #include "src/string/memory_utils/op_x86.h"
 #include "src/string/memory_utils/utils.h" // MemcmpReturnType
 
 namespace __llvm_libc {
 
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_generic_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = generic::Memcmp<16>::block(p1, p2))
-      return value;
-    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-  }
-  return generic::Memcmp<16>::loop_and_tail(p1, p2, count);
+  return generic::Memcmp<uint64_t>::loop_and_tail_align_above(384, p1, p2,
+                                                              count);
 }
 
+#if defined(__SSE4_1__)
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
-inline_memcmp_x86_sse2_gt16(CPtr p1, CPtr p2, size_t count) {
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = x86::sse2::Memcmp<16>::block(p1, p2))
-      return value;
-    align_to_next_boundary<16, Arg::P1>(p1, p2, count);
-  }
-  return x86::sse2::Memcmp<16>::loop_and_tail(p1, p2, count);
+inline_memcmp_x86_sse41_gt16(CPtr p1, CPtr p2, size_t count) {
+  return generic::Memcmp<__m128i>::loop_and_tail_align_above(384, p1, p2,
+                                                             count);
 }
+#endif // __SSE4_1__
 
+#if defined(__AVX2__)
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_x86_avx2_gt16(CPtr p1, CPtr p2, size_t count) {
   if (count <= 32)
-    return x86::sse2::Memcmp<16>::head_tail(p1, p2, count);
+    return generic::Memcmp<__m128i>::head_tail(p1, p2, count);
   if (count <= 64)
-    return x86::avx2::Memcmp<32>::head_tail(p1, p2, count);
-  if (count <= 128)
-    return x86::avx2::Memcmp<64>::head_tail(p1, p2, count);
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = x86::avx2::Memcmp<32>::block(p1, p2))
-      return value;
-    align_to_next_boundary<32, Arg::P1>(p1, p2, count);
-  }
-  return x86::avx2::Memcmp<32>::loop_and_tail(p1, p2, count);
+    return generic::Memcmp<__m256i>::head_tail(p1, p2, count);
+  return generic::Memcmp<__m256i>::loop_and_tail_align_above(384, p1, p2,
+                                                             count);
 }
+#endif // __AVX2__
 
+#if defined(__AVX512BW__)
 [[maybe_unused]] LIBC_INLINE MemcmpReturnType
 inline_memcmp_x86_avx512bw_gt16(CPtr p1, CPtr p2, size_t count) {
   if (count <= 32)
-    return x86::sse2::Memcmp<16>::head_tail(p1, p2, count);
+    return generic::Memcmp<__m128i>::head_tail(p1, p2, count);
   if (count <= 64)
-    return x86::avx2::Memcmp<32>::head_tail(p1, p2, count);
+    return generic::Memcmp<__m256i>::head_tail(p1, p2, count);
   if (count <= 128)
-    return x86::avx512bw::Memcmp<64>::head_tail(p1, p2, count);
-  if (LIBC_UNLIKELY(count >= 384)) {
-    if (auto value = x86::avx512bw::Memcmp<64>::block(p1, p2))
-      return value;
-    align_to_next_boundary<64, Arg::P1>(p1, p2, count);
-  }
-  return x86::avx512bw::Memcmp<64>::loop_and_tail(p1, p2, count);
+    return generic::Memcmp<__m512i>::head_tail(p1, p2, count);
+  return generic::Memcmp<__m512i>::loop_and_tail_align_above(384, p1, p2,
+                                                             count);
 }
+#endif // __AVX512BW__
 
 LIBC_INLINE MemcmpReturnType inline_memcmp_x86(CPtr p1, CPtr p2, size_t count) {
-
   if (count == 0)
     return MemcmpReturnType::ZERO();
   if (count == 1)
-    return generic::Memcmp<1>::block(p1, p2);
+    return generic::Memcmp<uint8_t>::block(p1, p2);
   if (count == 2)
-    return generic::Memcmp<2>::block(p1, p2);
+    return generic::Memcmp<uint16_t>::block(p1, p2);
   if (count == 3)
-    return generic::Memcmp<3>::block(p1, p2);
-  if (count <= 8)
-    return generic::Memcmp<4>::head_tail(p1, p2, count);
+    return generic::MemcmpSequence<uint16_t, uint8_t>::block(p1, p2);
+  if (count == 4)
+    return generic::Memcmp<uint32_t>::block(p1, p2);
+  if (count == 5)
+    return generic::MemcmpSequence<uint32_t, uint8_t>::block(p1, p2);
+  if (count == 6)
+    return generic::MemcmpSequence<uint32_t, uint16_t>::block(p1, p2);
+  if (count == 7)
+    return generic::Memcmp<uint32_t>::head_tail(p1, p2, 7);
+  if (count == 8)
+    return generic::Memcmp<uint64_t>::block(p1, p2);
   if (count <= 16)
-    return generic::Memcmp<8>::head_tail(p1, p2, count);
-  if constexpr (x86::kAvx512BW)
-    return inline_memcmp_x86_avx512bw_gt16(p1, p2, count);
-  else if constexpr (x86::kAvx2)
-    return inline_memcmp_x86_avx2_gt16(p1, p2, count);
-  else if constexpr (x86::kSse2)
-    return inline_memcmp_x86_sse2_gt16(p1, p2, count);
-  else
-    return inline_memcmp_generic_gt16(p1, p2, count);
+    return generic::Memcmp<uint64_t>::head_tail(p1, p2, count);
+#if defined(__AVX512BW__)
+#elif defined(__AVX2__)
+#elif defined(__SSE4_1__)
+  return inline_memcmp_x86_sse41_gt16(p1, p2, count);
+#else
+#endif
 }
+
 } // namespace __llvm_libc
 
 #endif // LIBC_SRC_STRING_MEMORY_UTILS_X86_64_MEMCMP_IMPLEMENTATIONS_H

libc/test/src/string/memory_utils/op_tests.cpp

 //===-- Unittests for op_ files -------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "memory_check_utils.h"
 #include "src/string/memory_utils/op_aarch64.h"
 #include "src/string/memory_utils/op_builtin.h"
-#include "src/string/memory_utils/op_generic.h"
+#include "src/string/memory_utils/op_generic.h" // LLVM_LIBC_HAS_UINT64
+#include "src/string/memory_utils/op_riscv.h"
 #include "src/string/memory_utils/op_x86.h"
 #include "test/UnitTest/Test.h"
 
-#if defined(LIBC_TARGET_ARCH_IS_X86_64) || defined(LIBC_TARGET_ARCH_IS_AARCH64)
-#define LLVM_LIBC_HAS_UINT64
-#endif
-
 namespace __llvm_libc {
 
 template <typename T> struct has_head_tail {
   template <typename C> static char sfinae(decltype(&C::head_tail));
   template <typename C> static uint16_t sfinae(...);
   static constexpr bool value = sizeof(sfinae<T>(0)) == sizeof(char);
 };
 
#ifdef LLVM_LIBC_HAS_BUILTIN_MEMSET_INLINE
     builtin::Memset<16>, //
     builtin::Memset<32>, //
     builtin::Memset<64>,
 #endif
 #ifdef LLVM_LIBC_HAS_UINT64
     generic::Memset<uint64_t>, generic::Memset<cpp::array<uint64_t, 2>>,
 #endif
 #ifdef __AVX512F__
-    generic::Memset<uint8x64_t>, generic::Memset<cpp::array<uint8x64_t, 2>>,
+    generic::Memset<generic_v512>, generic::Memset<cpp::array<generic_v512, 2>>,
 #endif
 #ifdef __AVX__
-    generic::Memset<uint8x32_t>, generic::Memset<cpp::array<uint8x32_t, 2>>,
+    generic::Memset<generic_v256>, generic::Memset<cpp::array<generic_v256, 2>>,
 #endif
 #ifdef __SSE2__
-    generic::Memset<uint8x16_t>, generic::Memset<cpp::array<uint8x16_t, 2>>,
+    generic::Memset<generic_v128>, generic::Memset<cpp::array<generic_v128, 2>>,
 #endif
     generic::Memset<uint32_t>, generic::Memset<cpp::array<uint32_t, 2>>, //
     generic::Memset<uint16_t>, generic::Memset<cpp::array<uint16_t, 2>>, //
     generic::Memset<uint8_t>, generic::Memset<cpp::array<uint8_t, 2>>,   //
     generic::MemsetSequence<uint8_t, uint8_t>,                           //
     generic::MemsetSequence<uint16_t, uint8_t>,                          //
     generic::MemsetSequence<uint32_t, uint16_t, uint8_t>                 //
     >;
if constexpr (kSize > 1) {
         auto dst = DstBuffer.span().subspan(0, size);
         ASSERT_TRUE((CheckMemset<LoopImpl>(dst, value, size)));
       }
     }
   }
 }
 
 using BcmpImplementations = testing::TypeList<
-#ifdef __SSE2__
-    x86::sse2::Bcmp<16>,  //
-    x86::sse2::Bcmp<32>,  //
-    x86::sse2::Bcmp<64>,  //
-    x86::sse2::Bcmp<128>, //
-#endif
+#ifdef LIBC_TARGET_ARCH_IS_X86_64
+#ifdef __SSE4_1__
+    generic::Bcmp<__m128i>,
+#endif // __SSE4_1__
 #ifdef __AVX2__
-    x86::avx2::Bcmp<32>,  //
-    x86::avx2::Bcmp<64>,  //
-    x86::avx2::Bcmp<128>, //
-#endif
+    generic::Bcmp<__m256i>,
+#endif // __AVX2__
 #ifdef __AVX512BW__
-    x86::avx512bw::Bcmp<64>,  //
-    x86::avx512bw::Bcmp<128>, //
-#endif
+    generic::Bcmp<__m512i>,
+#endif // __AVX512BW__
+
+#endif // LIBC_TARGET_ARCH_IS_X86_64
 #ifdef LIBC_TARGET_ARCH_IS_AARCH64
     aarch64::Bcmp<16>, //
-    aarch64::Bcmp<32>, //
+    aarch64::Bcmp<32>,
 #endif
+#ifndef LIBC_TARGET_ARCH_IS_ARM // Removing non uint8_t types for ARM
+    generic::Bcmp<uint16_t>,
+    generic::Bcmp<uint32_t>, //
 #ifdef LLVM_LIBC_HAS_UINT64
-    generic::Bcmp<8>, //
-#endif
-    generic::Bcmp<1>,  //
-    generic::Bcmp<2>,  //
-    generic::Bcmp<4>,  //
-    generic::Bcmp<16>, //
-    generic::Bcmp<32>, //
-    generic::Bcmp<64>  //
-    >;
+    generic::Bcmp<uint64_t>,
+#endif // LLVM_LIBC_HAS_UINT64
+    generic::BcmpSequence<uint16_t, uint8_t>,
+    generic::BcmpSequence<uint32_t, uint8_t>,  //
+    generic::BcmpSequence<uint32_t, uint16_t>, //
+    generic::BcmpSequence<uint32_t, uint16_t, uint8_t>,
+#endif // LIBC_TARGET_ARCH_IS_ARM
+    generic::BcmpSequence<uint8_t, uint8_t>,
+    generic::BcmpSequence<uint8_t, uint8_t, uint8_t>, //
+    generic::Bcmp<uint8_t>>;
 
 // Adapt CheckBcmp signature to op implementation signatures.
 template <auto FnImpl>
 int CmpAdaptor(cpp::span<char> p1, cpp::span<char> p2, size_t size) {
   return (int)FnImpl(as_byte(p1), as_byte(p2), size);
 }
 template <size_t Size, auto FnImpl>
 int CmpBlockAdaptor(cpp::span<char> p1, cpp::span<char> p2, size_t size) {
   return (int)FnImpl(as_byte(p1), as_byte(p2));
 }
 
 TYPED_TEST(LlvmLibcOpTest, Bcmp, BcmpImplementations) {
   using Impl = ParamType;
   constexpr size_t kSize = Impl::SIZE;
   { // Test block operation
     static constexpr auto BlockImpl = CmpBlockAdaptor<kSize, Impl::block>;
     Buffers Buffer1(kSize);
     Buffers Buffer2(kSize);
     for (auto span1 : Buffer1.spans()) {
       Randomize(span1);
       for (auto span2 : Buffer2.spans())
         ASSERT_TRUE((CheckBcmp<BlockImpl>(span1, span2, kSize)));
     }
   }
-  { // Test head tail operations from kSize to 2 * kSize.
+  if constexpr (has_head_tail<Impl>::value) {
+    // Test head tail operations from kSize to 2 * kSize.
     static constexpr auto HeadTailImpl = CmpAdaptor<Impl::head_tail>;
     Buffer Buffer1(2 * kSize);
     Buffer Buffer2(2 * kSize);
     Randomize(Buffer1.span());
     for (size_t size = kSize; size < 2 * kSize; ++size) {
       auto span1 = Buffer1.span().subspan(0, size);
       auto span2 = Buffer2.span().subspan(0, size);
       ASSERT_TRUE((CheckBcmp<HeadTailImpl>(span1, span2, size)));
     }
   }
-  { // Test loop operations from kSize to 3 * kSize.
+  if constexpr (has_loop_and_tail<Impl>::value) {
+    // Test loop operations from kSize to 3 * kSize.
     if constexpr (kSize > 1) {
       static constexpr auto LoopImpl = CmpAdaptor<Impl::loop_and_tail>;
       Buffer Buffer1(3 * kSize);
       Buffer Buffer2(3 * kSize);
       Randomize(Buffer1.span());
       for (size_t size = kSize; size < 3 * kSize; ++size) {
         auto span1 = Buffer1.span().subspan(0, size);
         auto span2 = Buffer2.span().subspan(0, size);
         ASSERT_TRUE((CheckBcmp<LoopImpl>(span1, span2, size)));
       }
     }
   }
 }
 
 using MemcmpImplementations = testing::TypeList<
+#ifdef LIBC_TARGET_ARCH_IS_X86_64
 #ifdef __SSE2__
-    x86::sse2::Memcmp<16>,  //
-    x86::sse2::Memcmp<32>,  //
-    x86::sse2::Memcmp<64>,  //
-    x86::sse2::Memcmp<128>, //
+    generic::Memcmp<__m128i>, //
 #endif
 #ifdef __AVX2__
-    x86::avx2::Memcmp<32>,  //
-    x86::avx2::Memcmp<64>,  //
-    x86::avx2::Memcmp<128>, //
+    generic::Memcmp<__m256i>, //
 #endif
 #ifdef __AVX512BW__
-    x86::avx512bw::Memcmp<64>,  //
-    x86::avx512bw::Memcmp<128>, //
+    generic::Memcmp<__m512i>, //
 #endif
-#ifdef LLVM_LIBC_HAS_UINT64
-    generic::Memcmp<8>, //
+#endif // LIBC_TARGET_ARCH_IS_X86_64
+#ifdef LIBC_TARGET_ARCH_IS_AARCH64
+    generic::Memcmp<uint8x16_t>, //
+    generic::Memcmp<uint8x16x2_t>,
 #endif
-    generic::Memcmp<1>,  //
-    generic::Memcmp<2>,  //
-    generic::Memcmp<3>,  //
-    generic::Memcmp<4>,  //
-    generic::Memcmp<16>, //
-    generic::Memcmp<32>, //
-    generic::Memcmp<64>  //
-    >;
+#ifndef LIBC_TARGET_ARCH_IS_ARM // Removing non uint8_t types for ARM
+    generic::Memcmp<uint16_t>,
+    generic::Memcmp<uint32_t>, //
+#ifdef LLVM_LIBC_HAS_UINT64
+    generic::Memcmp<uint64_t>,
+#endif // LLVM_LIBC_HAS_UINT64
+    generic::MemcmpSequence<uint16_t, uint8_t>,
+    generic::MemcmpSequence<uint32_t, uint16_t, uint8_t>, //
+#endif // LIBC_TARGET_ARCH_IS_ARM
+    generic::MemcmpSequence<uint8_t, uint8_t>,
+    generic::MemcmpSequence<uint8_t, uint8_t, uint8_t>,
+    generic::Memcmp<uint8_t>>;
 
 TYPED_TEST(LlvmLibcOpTest, Memcmp, MemcmpImplementations) {
   using Impl = ParamType;
   constexpr size_t kSize = Impl::SIZE;
   { // Test block operation
     static constexpr auto BlockImpl = CmpBlockAdaptor<kSize, Impl::block>;
     Buffers Buffer1(kSize);
     Buffers Buffer2(kSize);
     for (auto span1 : Buffer1.spans()) {
       Randomize(span1);
       for (auto span2 : Buffer2.spans())
         ASSERT_TRUE((CheckMemcmp<BlockImpl>(span1, span2, kSize)));
     }
   }
-  { // Test head tail operations from kSize to 2 * kSize.
+  if constexpr (has_head_tail<Impl>::value) {
+    // Test head tail operations from kSize to 2 * kSize.
     static constexpr auto HeadTailImpl = CmpAdaptor<Impl::head_tail>;
     Buffer Buffer1(2 * kSize);
     Buffer Buffer2(2 * kSize);
     Randomize(Buffer1.span());
     for (size_t size = kSize; size < 2 * kSize; ++size) {
       auto span1 = Buffer1.span().subspan(0, size);
       auto span2 = Buffer2.span().subspan(0, size);
       ASSERT_TRUE((CheckMemcmp<HeadTailImpl>(span1, span2, size)));
     }
   }
-  { // Test loop operations from kSize to 3 * kSize.
+  if constexpr (has_loop_and_tail<Impl>::value) {
+    // Test loop operations from kSize to 3 * kSize.
     if constexpr (kSize > 1) {
       static constexpr auto LoopImpl = CmpAdaptor<Impl::loop_and_tail>;
       Buffer Buffer1(3 * kSize);
       Buffer Buffer2(3 * kSize);
       Randomize(Buffer1.span());
       for (size_t size = kSize; size < 3 * kSize; ++size) {
         auto span1 = Buffer1.span().subspan(0, size);
         auto span2 = Buffer2.span().subspan(0, size);
         ASSERT_TRUE((CheckMemcmp<LoopImpl>(span1, span2, size)));
       }
     }
   }
 }
 
 } // namespace __llvm_libc

utils/bazel/llvm-project-overlay/libc/BUILD.bazel

 ]
 
 libc_support_library(
     name = "string_memory_utils",
     hdrs = [
         "src/string/memory_utils/op_aarch64.h",
         "src/string/memory_utils/op_builtin.h",
         "src/string/memory_utils/op_generic.h",
+        "src/string/memory_utils/op_riscv.h",
         "src/string/memory_utils/op_x86.h",
         "src/string/memory_utils/utils.h",
     ],
     textual_hdrs = [
         "src/string/memory_utils/aarch64/memcmp_implementations.h",
         "src/string/memory_utils/aarch64/memcpy_implementations.h",
         "src/string/memory_utils/bcmp_implementations.h",
         "src/string/memory_utils/bzero_implementations.h",
         "src/string/memory_utils/memcmp_implementations.h",
         "src/string/memory_utils/memcpy_implementations.h",
         "src/string/memory_utils/memmove_implementations.h",
         "src/string/memory_utils/memset_implementations.h",
         "src/string/memory_utils/strcmp_implementations.h",
         "src/string/memory_utils/strstr_implementations.h",
         "src/string/memory_utils/memmem_implementations.h",
         "src/string/memory_utils/x86_64/memcmp_implementations.h",
         "src/string/memory_utils/x86_64/memcpy_implementations.h",
     ],
     deps = [
         ":__support_common",
         ":__support_cpp_array",
         ":__support_cpp_bit",
         ":__support_cpp_cstddef",
+        ":__support_cpp_limits",
         ":__support_cpp_type_traits",
         ":__support_macros_attributes",
         ":__support_macros_config",
         ":__support_macros_optimization",
         ":__support_macros_properties_architectures",
         ":__support_macros_properties_cpu_features",
         ":libc_root",
     ],