[clang] Add support for __builtin_memset_inline

[gchatelet]

Summary

In the same spirit as D73543 and in reply to https://reviews.llvm.org/D126768#3549920 this patch is adding support for __builtin_memset_inline.

The idea is to get support from the compiler to easily write efficient memory function implementations.

This patch could be split in two:

• one for the LLVM part adding the llvm.memset.inline.* intrinsics.
• and another one for the Clang part providing the instrinsic as a builtin.

[gchatelet]

• remove unused variable and rebase

[rsmith]

clang/lib/CodeGen/CGBuilder.h

53

There are a lot of unrelated whitespace changes (clang-format, I assume) in this file. Can you produce a separate patch with just the whitespace changes and go ahead and check that in first?

clang/lib/Sema/SemaChecking.cpp

2280–2302

This custom checking seems unnecessary and also insufficient: you're not enforcing that the arguments have the right types. We should convert the arguments to this builtin in the same way we'd convert arguments for a normally-declared function, which you can achieve by removing the "t" flag from the builtin definition and removing all of this logic (except for the nonnull checking, which I don't think we have a way to model in Builtins.def yet).

Please do the same to __builtin_memcpy_inline, which is wrong in the same way. (For example, __builtin_memcpy_inline(1, 2, 3) crashes in code generation because it fails to perform proper conversions and type-checking on its arguments.)

[gchatelet]

• Rebase over formatted CGBuilder.h

[gchatelet]

• Fix sema checking

[gchatelet]

clang/lib/Sema/SemaChecking.cpp

2280–2302

Thx!

[courbet]

clang/test/Sema/builtins-memcpy-inline.cpp

11 ↗

(On Diff #433982)

can you split this off (and the corresponding code) to a separate patch ?

llvm/docs/LangRef.rst

13898

take*

13898–13899

"an extra isvolatile argument"

13899

pointer*

llvm/include/llvm/IR/IntrinsicInst.h

988

update comment

llvm/include/llvm/IR/Intrinsics.td

654

"version" ? or "semantics"

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

7342

There is nothing preveting a target from emitting a call to memset here when AlwaysInline is true. Actually, X86SelectionDAGInfo does just that (this very patch is adding /* AlwaysInline */ false, to the getMemset call that handles the trailing bytes). It happens that because trailing bytes are typically small and therefore inline. it happens to work, but this should be verified somehow (or, maybe easier, AlwaysInline should be passed to EmitTargetCodeForMemset so it can do the right thing).

7352

Not that this is not strictly required to return a valid SDValue: Even with an "infinite" Limit in TLI.findOptimalMemOpLowering, a target that overrides this function could decide to just return false. I'm not sure what we want to do in this case.
So I think we should document findOptimalMemOpLowering to mention that the default implementation always returns a valid memop lowering if Limit is UINT_MAX and that target that decide to not provide a memop lowering *must* emit a valid one in EmitTargetCodeForMemset. Another option would be to call the generic TargetLowering::findOptimalMemOpLowering when the target declines to generate eithe ra memop lowering or target-specific code.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5928

memset

llvm/test/CodeGen/X86/memset-inline.ll

2

Add tests for other targets ?

22

I could see memset deciding to inline 129 bytes in the future. What about using a more absurdly large number ?

[gchatelet]

• rebase

clang/test/Sema/builtins-memcpy-inline.cpp

11 ↗

(On Diff #433982)

Done in https://github.com/llvm/llvm-project/commit/d8b540cd312cf924b1904047b56cf8a1cea333f9

[gchatelet]

• Address comments, add more codegen tests, rebase

[gchatelet]

Pending D127279 before making sure that all EmitTargetCodeForMemset implementations are enforcing AlwaysInline.

[gchatelet]

• Address comments

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

7342

I think this is OK now, I had to tweak the ARM selection dag as well.

7352

This one is more tricky. I've added a comment on the function and bypassed problematic cases but this is far from ideal.
The greedy nature of the algorithm makes it hard to understand the appropriate behavior.

I think this is worth stepping back and redesigning the abstraction in a subsequent patch.

[courbet]

llvm/include/llvm/CodeGen/SelectionDAGTargetInfo.h

79–80

Add comment about honoring AlwaysInline ?

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6990

"Makes sure no function call is generated", because technically the code can generate an accelerator call (e.g. rep stos)

7352

Let's assert that the returned value is non-default before returning.

[gchatelet]

• Address comments

[courbet]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

7355

maybe add when AlwaysInline to make the issue more obvious.

[gchatelet]

• Update assert message

[gchatelet]

rebase

[aganea]

Hello @gchatelet! This patches introduces a regression in a two-stage build using rpmalloc on Windows. Bisection lead me here. Would you have a chance to take a look please? Thanks in advance.

To reproduce, I used the following script (rename to make_llvm.bat)

make_llvm.txt2 KBDownload

C:\git\llvm-project>make_llvm.bat stage1
...
C:\git\llvm-project>ninja clang lld -C stage1
...
C:\git\llvm-project>make_llvm.bat stage2_rpmalloc
...
C:\git\llvm-project>ninja lld -C stage2_rpmalloc

generates:

[158/2714] Building C object lib\Support\CMakeFiles\LLVMSupport.dir\C_\git\rpmalloc\rpmalloc\rpmalloc.c.obj
FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/C_/git/rpmalloc/rpmalloc/rpmalloc.c.obj
C:\git\llvm-project\stage1\bin\clang-cl.exe  /nologo -DENABLE_OVERRIDE -DENABLE_PRELOAD -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\git\llvm-project\stage2_rpmalloc\lib\Support -IC:\git\llvm-project\llvm\lib\Support -IC:\git\llvm-project\stage2_rpmalloc\include -IC:\git\llvm-project\llvm\include /GS- /D_ITERATOR_DEBUG_LEVEL=0 -Xclang -O3 -fstrict-aliasing -march=native -flto=thin -fwhole-program-vtables -fuse-ld=lld /Zc:inline /Zi -gcodeview-ghash /Zc:strictStrings /Oi /W4  -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation /Gw -flto=thin /MT /O2 /Ob2 /DNDEBUG /showIncludes /Folib\Support\CMakeFiles\LLVMSupport.dir\C_\git\rpmalloc\rpmalloc\rpmalloc.c.obj /Fdlib\Support\CMakeFiles\LLVMSupport.dir\LLVMSupport.pdb -c -- C:\git\rpmalloc\rpmalloc\rpmalloc.c
Assertion failed: isa<To>(Val) && "cast<Ty>() argument of incompatible type!", file C:/git/llvm-project/llvm/include\llvm/Support/Casting.h, line 578
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: C:\\git\\llvm-project\\stage1\\bin\\clang-cl.exe /nologo -DENABLE_OVERRIDE -DENABLE_PRELOAD -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\\git\\llvm-project\\stage2_rpmalloc\\lib\\Support -IC:\\git\\llvm-project\\llvm\\lib\\Support -IC:\\git\\llvm-project\\stage2_rpmalloc\\include -IC:\\git\\llvm-project\\llvm\\include /GS- /D_ITERATOR_DEBUG_LEVEL=0 -Xclang -O3 -fstrict-aliasing -march=native -flto=thin -fwhole-program-vtables -fuse-ld=lld /Zc:inline /Zi -gcodeview-ghash /Zc:strictStrings /Oi /W4 -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation /Gw -flto=thin /MT /O2 /Ob2 /DNDEBUG /showIncludes /Folib\\Support\\CMakeFiles\\LLVMSupport.dir\\C_\\git\\rpmalloc\\rpmalloc\\rpmalloc.c.obj /Fdlib\\Support\\CMakeFiles\\LLVMSupport.dir\\LLVMSupport.pdb -c -- C:\\git\\rpmalloc\\rpmalloc\\rpmalloc.c
1.      <eof> parser at end of file
2.      Optimizer
 #0 0x00007ff7f877ca86 HandleAbort C:\git\llvm-project\llvm\lib\Support\Windows\Signals.inc:418:0
 #1 0x00007ff7fcdfc6bb raise C:\git\llvm-project\stage1\minkernel\crts\ucrt\src\appcrt\misc\signal.cpp:547:0
 #2 0x00007ff7fcdf1ff4 abort C:\git\llvm-project\stage1\minkernel\crts\ucrt\src\appcrt\startup\abort.cpp:71:0
 #3 0x00007ff7fcdf2672 common_assert_to_stderr<wchar_t> C:\git\llvm-project\stage1\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp:186:0
 #4 0x00007ff7fcdf251a _wassert C:\git\llvm-project\stage1\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp:443:0
 #5 0x00007ff7f9b48e7b llvm::VNCoercion::analyzeLoadFromClobberingMemInst(class llvm::Type *, class llvm::Value *, class llvm::MemIntrinsic *, class llvm::DataLayout const &) C:\git\llvm-project\llvm\lib\Transforms\Utils\VNCoercion.cpp:381:0
 #6 0x00007ff7f85a1ec4 llvm::GVNPass::AnalyzeLoadAvailability(class llvm::LoadInst *, class llvm::MemDepResult, class llvm::Value *, struct llvm::gvn::AvailableValue &) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:1211:0
 #7 0x00007ff7f85a8a5c llvm::GVNPass::processLoad(class llvm::LoadInst *) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:2068:0
 #8 0x00007ff7f85aa465 llvm::GVNPass::processInstruction(class llvm::Instruction *) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:2483:0
 #9 0x00007ff7f85ab3ed llvm::GVNPass::processBlock(class llvm::BasicBlock *) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:2682:0
#10 0x00007ff7f85aadd0 llvm::GVNPass::iterateOnFunction(class llvm::Function &) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:3027:0
#11 0x00007ff7f85a099d llvm::GVNPass::runImpl(class llvm::Function &, class llvm::AssumptionCache &, class llvm::DominatorTree &, class llvm::TargetLibraryInfo const &, class llvm::AAResults &, class llvm::MemoryDependenceResults *, class llvm::LoopInfo *, class llvm::OptimizationRemarkEmitter *, class llvm::MemorySSA *) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:2627:0
#12 0x00007ff7f85a0595 llvm::GVNPass::run(class llvm::Function &, class llvm::AnalysisManager<class llvm::Function> &) C:\git\llvm-project\llvm\lib\Transforms\Scalar\GVN.cpp:732:0
#13 0x00007ff7fb2f5771 llvm::detail::PassModel<class llvm::Function, class llvm::GVNPass, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::Function>>::run(class llvm::Function &, class llvm::AnalysisManager<class llvm::Function> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#14 0x00007ff7f8eba44f llvm::PassManager<class llvm::Function, class llvm::AnalysisManager<class llvm::Function>>::run(class llvm::Function &, class llvm::AnalysisManager<class llvm::Function> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManager.h:522:0
#15 0x00007ff7f7faed21 llvm::detail::PassModel<class llvm::Function, class llvm::PassManager<class llvm::Function, class llvm::AnalysisManager<class llvm::Function>>, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::Function>>::run(class llvm::Function &, class llvm::AnalysisManager<class llvm::Function> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#16 0x00007ff7f8ec1d7e llvm::CGSCCToFunctionPassAdaptor::run(class llvm::LazyCallGraph::SCC &, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &> &, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &) C:\git\llvm-project\llvm\lib\Analysis\CGSCCPassManager.cpp:554:0
#17 0x00007ff7f7fb094d llvm::detail::PassModel<class llvm::LazyCallGraph::SCC, class llvm::CGSCCToFunctionPassAdaptor, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &>, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &>::run(class llvm::LazyCallGraph::SCC &, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &> &, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#18 0x00007ff7f8ebc8c1 llvm::PassManager<class llvm::LazyCallGraph::SCC, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &>, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &>::run(class llvm::LazyCallGraph::SCC &, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &> &, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &) C:\git\llvm-project\llvm\lib\Analysis\CGSCCPassManager.cpp:90:0
#19 0x00007ff7f989597d llvm::detail::PassModel<class llvm::LazyCallGraph::SCC, class llvm::PassManager<class llvm::LazyCallGraph::SCC, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &>, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &>, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &>, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &>::run(class llvm::LazyCallGraph::SCC &, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &> &, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#20 0x00007ff7f8ebffe1 llvm::DevirtSCCRepeatedPass::run(class llvm::LazyCallGraph::SCC &, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &> &, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &) C:\git\llvm-project\llvm\lib\Analysis\CGSCCPassManager.cpp:417:0
#21 0x00007ff7f9895e4d llvm::detail::PassModel<class llvm::LazyCallGraph::SCC, class llvm::DevirtSCCRepeatedPass, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &>, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &>::run(class llvm::LazyCallGraph::SCC &, class llvm::AnalysisManager<class llvm::LazyCallGraph::SCC, class llvm::LazyCallGraph &> &, class llvm::LazyCallGraph &, struct llvm::CGSCCUpdateResult &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#22 0x00007ff7f8ebeeb0 llvm::ModuleToPostOrderCGSCCPassAdaptor::run(class llvm::Module &, class llvm::AnalysisManager<class llvm::Module> &) C:\git\llvm-project\llvm\lib\Analysis\CGSCCPassManager.cpp:283:0
#23 0x00007ff7f9895b91 llvm::detail::PassModel<class llvm::Module, class llvm::ModuleToPostOrderCGSCCPassAdaptor, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::Module>>::run(class llvm::Module &, class llvm::AnalysisManager<class llvm::Module> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#24 0x00007ff7f8eb9744 llvm::PassManager<class llvm::Module, class llvm::AnalysisManager<class llvm::Module>>::run(class llvm::Module &, class llvm::AnalysisManager<class llvm::Module> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManager.h:522:0
#25 0x00007ff7f989504c llvm::ModuleInlinerWrapperPass::run(class llvm::Module &, class llvm::AnalysisManager<class llvm::Module> &) C:\git\llvm-project\llvm\lib\Transforms\IPO\Inliner.cpp:1150:0
#26 0x00007ff7fb2f6621 llvm::detail::PassModel<class llvm::Module, class llvm::ModuleInlinerWrapperPass, class llvm::PreservedAnalyses, class llvm::AnalysisManager<class llvm::Module>>::run(class llvm::Module &, class llvm::AnalysisManager<class llvm::Module> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManagerInternal.h:88:0
#27 0x00007ff7f8eb9744 llvm::PassManager<class llvm::Module, class llvm::AnalysisManager<class llvm::Module>>::run(class llvm::Module &, class llvm::AnalysisManager<class llvm::Module> &) C:\git\llvm-project\llvm\include\llvm\IR\PassManager.h:522:0
#28 0x00007ff7f9cd6282 `anonymous namespace'::EmitAssemblyHelper::RunOptimizationPipeline C:\git\llvm-project\clang\lib\CodeGen\BackendUtil.cpp:965:0
#29 0x00007ff7f9cce34e clang::EmitBackendOutput(class clang::DiagnosticsEngine &, class clang::HeaderSearchOptions const &, class clang::CodeGenOptions const &, class clang::TargetOptions const &, class clang::LangOptions const &, class llvm::StringRef, class llvm::Module *, enum clang::BackendAction, class std::unique_ptr<class llvm::raw_pwrite_stream, struct std::default_delete<class llvm::raw_pwrite_stream>>) C:\git\llvm-project\clang\lib\CodeGen\BackendUtil.cpp:1181:0
#30 0x00007ff7fa0b26ae clang::BackendConsumer::HandleTranslationUnit(class clang::ASTContext &) C:\git\llvm-project\clang\lib\CodeGen\CodeGenAction.cpp:379:0
#31 0x00007ff7fbaf08e4 clang::ParseAST(class clang::Sema &, bool, bool) C:\git\llvm-project\clang\lib\Parse\ParseAST.cpp:182:0
#32 0x00007ff7fa017775 clang::FrontendAction::Execute(void) C:\git\llvm-project\clang\lib\Frontend\FrontendAction.cpp:1032:0
#33 0x00007ff7f88eca63 clang::CompilerInstance::ExecuteAction(class clang::FrontendAction &) C:\git\llvm-project\clang\lib\Frontend\CompilerInstance.cpp:1033:0
#34 0x00007ff7f8989667 clang::ExecuteCompilerInvocation(class clang::CompilerInstance *) C:\git\llvm-project\clang\lib\FrontendTool\ExecuteCompilerInvocation.cpp:266:0
#35 0x00007ff7f7f184c9 cc1_main(class llvm::ArrayRef<char const *>, char const *, void *) C:\git\llvm-project\clang\tools\driver\cc1_main.cpp:248:0
#36 0x00007ff7f7f14fc3 ExecuteCC1Tool C:\git\llvm-project\clang\tools\driver\driver.cpp:317:0
#37 0x00007ff7f9dd7236 llvm::function_ref<void ()>::callback_fn<`lambda at C:/git/llvm-project/clang/lib/Driver/Job.cpp:407:22'> C:\git\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45:0
#38 0x00007ff7f873317e llvm::CrashRecoveryContext::RunSafely(class llvm::function_ref<(void)>) C:\git\llvm-project\llvm\lib\Support\CrashRecoveryContext.cpp:234:0
#39 0x00007ff7f9dd6eed clang::driver::CC1Command::Execute(class llvm::ArrayRef<class llvm::Optional<class llvm::StringRef>>, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> *, bool *) const C:\git\llvm-project\clang\lib\Driver\Job.cpp:407:0
#40 0x00007ff7f88a9d3a clang::driver::Compilation::ExecuteCommand(class clang::driver::Command const &, class clang::driver::Command const *&) const C:\git\llvm-project\clang\lib\Driver\Compilation.cpp:196:0
#41 0x00007ff7f88aa20a clang::driver::Compilation::ExecuteJobs(class clang::driver::JobList const &, class llvm::SmallVectorImpl<struct std::pair<int, class clang::driver::Command const *>> &) const C:\git\llvm-project\clang\lib\Driver\Compilation.cpp:249:0
#42 0x00007ff7f88c3107 clang::driver::Driver::ExecuteCompilation(class clang::driver::Compilation &, class llvm::SmallVectorImpl<struct std::pair<int, class clang::driver::Command const *>> &) C:\git\llvm-project\clang\lib\Driver\Driver.cpp:1709:0
#43 0x00007ff7f7f148b1 clang_main(int, char **) C:\git\llvm-project\clang\tools\driver\driver.cpp:513:0
#44 0x00007ff7fcdde008 __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288:0
#45 0x00007ff8f4944ed0 (C:\Windows\System32\KERNEL32.DLL+0x14ed0)
#46 0x00007ff8f529e40b (C:\Windows\SYSTEM32\ntdll.dll+0x7e40b)
clang-cl: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 15.0.0 (https://github.com/llvm/llvm-project.git 38637ee477541370a90b37f149069d8e5c0c2efd)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\git\llvm-project\stage1\bin
clang-cl: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-cl: note: diagnostic msg: C:\Users\aganea\AppData\Local\Temp\2\rpmalloc-70541f.c
clang-cl: note: diagnostic msg: C:\Users\aganea\AppData\Local\Temp\2\rpmalloc-70541f.sh
clang-cl: note: diagnostic msg:

********************

The reproducer:

rpmalloc-70541f.zip907 KBDownload

Using rpmalloc at this revision: https://github.com/mjansson/rpmalloc/commit/f56e2f6794eab5c280b089c90750c681679fde92

[efriedma]

It's not really a regression, just a bug in the feature. The rpmalloc code is checking __has_builtin(__builtin_memset_inline), so older versions of clang aren't seeing the same source code.

[thieta]

We have been building a two stage clang and our internal projects with this version of rpmalloc and clang 15.x a while now and I haven't see the issue you see @aganea. I don't think this patch is the problem as @efriedma suggested, could be in rpmalloc but since I have used this successfully already I am not sure about that either.

[courbet]

I think there's a bug in the code though:

// If this is memset, we just need to see if the offset is valid in the size
  // of the memset..
  if (MI->getIntrinsicID() == Intrinsic::memset)

should really be:

// If this is memset, we just need to see if the offset is valid in the size
  // of the memset..
  if (isa<MemSetInst>(MI))

[gchatelet]

In D126903#3884873, @courbet wrote:

I think there's a bug in the code though:

// If this is memset, we just need to see if the offset is valid in the size
  // of the memset..
  if (MI->getIntrinsicID() == Intrinsic::memset)

should really be:

// If this is memset, we just need to see if the offset is valid in the size
  // of the memset..
  if (isa<MemSetInst>(MI))

Yes this seems a likely culprit.

[gchatelet]

It should be fixed now @aganea . Can you check?

[aganea]

In D126903#3885114, @gchatelet wrote:

It should be fixed now @aganea . Can you check?

Works now, thanks again for the quick response. Should D136752 be cherry-picked into the 15.0 branch?

In D126903#3884748, @thieta wrote:

We have been building a two stage clang and our internal projects with this version of rpmalloc and clang 15.x a while now and I haven't see the issue you see @aganea. I don't think this patch is the problem as @efriedma suggested, could be in rpmalloc but since I have used this successfully already I am not sure about that either.

Is your internal build using the rpmalloc develop branch?

[gchatelet]

In D126903#3885269, @aganea wrote:

In D126903#3885114, @gchatelet wrote:

It should be fixed now @aganea . Can you check?

Works now, thanks again for the quick response. Should D136752 be cherry-picked into the 15.0 branch?

Yes done here : https://github.com/llvm/llvm-project/issues/58627

clang/docs/LanguageExtensions.rst

 
 This is useful to implement a custom version of ``memcpy``, implement a
 ``libc`` memcpy or work around the absence of a ``libc``.
 
 Note that the `size` argument must be a compile time constant.
 
 Note that this intrinsic cannot yet be called in a ``constexpr`` context.
 
+Guaranteed inlined memset
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+. code-block:: c
+
+  void __builtin_memset_inline(void *dst, int value, size_t size);
+
+
+``__builtin_memset_inline`` has been designed as a building block for efficient
+``memset`` implementations. It is identical to ``__builtin_memset`` but also
+guarantees not to call any external functions. See LLVM IR `llvm.memset.inline
+<https://llvm.org/docs/LangRef.html#llvm-memset-inline-intrinsic>`_ intrinsic
+for more information.
+
+This is useful to implement a custom version of ``memset``, implement a
+``libc`` memset or work around the absence of a ``libc``.
+
+Note that the `size` argument must be a compile time constant.
+
+Note that this intrinsic cannot yet be called in a ``constexpr`` context.
 
 Atomic Min/Max builtins with memory ordering
 --------------------------------------------
 
 There are two atomic builtins with min/max in-memory comparison and swap.
 The syntax and semantics are similar to GCC-compatible __atomic_* builtins.
 
 * ``__atomic_fetch_min``

clang/include/clang/Basic/Builtins.def

 BUILTIN(__builtin_malloc, "v*z", "nF")
 BUILTIN(__builtin_memchr, "v*vC*iz", "nF")
 BUILTIN(__builtin_memcmp, "ivC*vC*z", "nF")
 BUILTIN(__builtin_memcpy, "v*v*vC*z", "nF")
 BUILTIN(__builtin_memcpy_inline, "vv*vC*Iz", "n")
 BUILTIN(__builtin_memmove, "v*v*vC*z", "nF")
 BUILTIN(__builtin_mempcpy, "v*v*vC*z", "nF")
 BUILTIN(__builtin_memset, "v*v*iz", "nF")
+BUILTIN(__builtin_memset_inline, "vv*iIz", "n")
 BUILTIN(__builtin_printf, "icC*.", "Fp:0:")
 BUILTIN(__builtin_stpcpy, "c*c*cC*", "nF")
 BUILTIN(__builtin_stpncpy, "c*c*cC*z", "nF")
 BUILTIN(__builtin_strcasecmp, "icC*cC*", "nF")
 BUILTIN(__builtin_strcat, "c*c*cC*", "nF")
 BUILTIN(__builtin_strchr, "c*cC*i", "nF")
 BUILTIN(__builtin_strcmp, "icC*cC*", "nF")
 BUILTIN(__builtin_strcpy, "c*c*cC*", "nF")

clang/lib/CodeGen/CGBuilder.h

 class CGBuilderTy : public CGBuilderBaseTy {
   /// Storing a reference to the type cache here makes it a lot easier
   /// to build natural-feeling, target-specific IR.
   const CodeGenTypeCache &TypeCache;
 
 public:
   CGBuilderTy(const CodeGenTypeCache &TypeCache, llvm::LLVMContext &C)
       : CGBuilderBaseTy(C), TypeCache(TypeCache) {}
   CGBuilderTy(const CodeGenTypeCache &TypeCache, llvm::LLVMContext &C,
               const llvm::ConstantFolder &F,
               const CGBuilderInserterTy &Inserter)
       : CGBuilderBaseTy(C, F, Inserter), TypeCache(TypeCache) {}
   CGBuilderTy(const CodeGenTypeCache &TypeCache, llvm::Instruction *I)
       : CGBuilderBaseTy(I), TypeCache(TypeCache) {}
   CGBuilderTy(const CodeGenTypeCache &TypeCache, llvm::BasicBlock *BB)
       : CGBuilderBaseTy(BB), TypeCache(TypeCache) {}
 
public:
 
   using CGBuilderBaseTy::CreateMemSet;
   llvm::CallInst *CreateMemSet(Address Dest, llvm::Value *Value,
                                llvm::Value *Size, bool IsVolatile = false) {
     return CreateMemSet(Dest.getPointer(), Value, Size,
                         Dest.getAlignment().getAsAlign(), IsVolatile);
   }
 
+  using CGBuilderBaseTy::CreateMemSetInline;
+  llvm::CallInst *CreateMemSetInline(Address Dest, llvm::Value *Value,
+                                     uint64_t Size) {
+    return CreateMemSetInline(Dest.getPointer(),
+                              Dest.getAlignment().getAsAlign(), Value,
+                              getInt64(Size));
+  }
+
   using CGBuilderBaseTy::CreatePreserveStructAccessIndex;
   Address CreatePreserveStructAccessIndex(Address Addr, unsigned Index,
                                           unsigned FieldIndex,
                                           llvm::MDNode *DbgInfo) {
     llvm::StructType *ElTy = cast<llvm::StructType>(Addr.getElementType());
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
     const llvm::StructLayout *Layout = DL.getStructLayout(ElTy);
     auto Offset = CharUnits::fromQuantity(Layout->getElementOffset(Index));

clang/lib/CodeGen/CGBuiltin.cpp

case Builtin::BI__builtin_memset: {
     Value *ByteVal = Builder.CreateTrunc(EmitScalarExpr(E->getArg(1)),
                                          Builder.getInt8Ty());
     Value *SizeVal = EmitScalarExpr(E->getArg(2));
     EmitNonNullArgCheck(RValue::get(Dest.getPointer()), E->getArg(0)->getType(),
                         E->getArg(0)->getExprLoc(), FD, 0);
     Builder.CreateMemSet(Dest, ByteVal, SizeVal, false);
     return RValue::get(Dest.getPointer());
   }
+  case Builtin::BI__builtin_memset_inline: {
+    Address Dest = EmitPointerWithAlignment(E->getArg(0));
+    Value *ByteVal =
+        Builder.CreateTrunc(EmitScalarExpr(E->getArg(1)), Builder.getInt8Ty());
+    uint64_t Size =
+        E->getArg(2)->EvaluateKnownConstInt(getContext()).getZExtValue();
+    EmitNonNullArgCheck(RValue::get(Dest.getPointer()), E->getArg(0)->getType(),
+                        E->getArg(0)->getExprLoc(), FD, 0);
+    Builder.CreateMemSetInline(Dest, ByteVal, Size);
+    return RValue::get(nullptr);
+  }
   case Builtin::BI__builtin___memset_chk: {
     // fold __builtin_memset_chk(x, y, cst1, cst2) to memset iff cst1<=cst2.
     Expr::EvalResult SizeResult, DstSizeResult;
     if (!E->getArg(2)->EvaluateAsInt(SizeResult, CGM.getContext()) ||
         !E->getArg(3)->EvaluateAsInt(DstSizeResult, CGM.getContext()))
       break;
     llvm::APSInt Size = SizeResult.Val.getInt();
     llvm::APSInt DstSize = DstSizeResult.Val.getInt();

clang/lib/Sema/SemaChecking.cpp

case Builtin::BI__builtin_memcpy_inline: {
     if (SizeOp->isValueDependent())
       break;
     if (!SizeOp->EvaluateKnownConstInt(Context).isZero()) {
       CheckNonNullArgument(*this, TheCall->getArg(0), TheCall->getExprLoc());
       CheckNonNullArgument(*this, TheCall->getArg(1), TheCall->getExprLoc());
     }
     break;
   }
+  case Builtin::BI__builtin_memset_inline: {
+    clang::Expr *SizeOp = TheCall->getArg(2);
+    // We warn about filling to `nullptr` pointers when `size` is greater than
+    // 0. When `size` is value dependent we cannot evaluate its value so we bail
+    // out.
+    if (SizeOp->isValueDependent())
+      break;
+    if (!SizeOp->EvaluateKnownConstInt(Context).isZero())
+      CheckNonNullArgument(*this, TheCall->getArg(0), TheCall->getExprLoc());
+    break;
+  }
 #define BUILTIN(ID, TYPE, ATTRS)
 #define ATOMIC_BUILTIN(ID, TYPE, ATTRS) \
   case Builtin::BI##ID: \
     return SemaAtomicOpsOverloaded(TheCallResult, AtomicExpr::AO##ID);
 #include "clang/Basic/Builtins.def"
   case Builtin::BI__annotation:
     if (SemaBuiltinMSVCAnnotation(*this, TheCall))
       return ExprError();
     break;
   case Builtin::BI__builtin_annotation:
     if (SemaBuiltinAnnotation(*this, TheCall))
       return ExprError();
     break;
   case Builtin::BI__builtin_addressof:
     if (SemaBuiltinAddressof(*this, TheCall))
       return ExprError();
     break;
   case Builtin::BI__builtin_function_start:
     if (SemaBuiltinFunctionStart(*this, TheCall))
       return ExprError();

clang/test/CodeGen/builtins-memset-inline.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: x86-registered-target
+// RUN: %clang_cc1 -no-opaque-pointers -triple x86_64-unknown-linux -emit-llvm %s -o - | FileCheck %s
+
+// CHECK-LABEL: define{{.*}} void @test_memset_inline_0(i8* noundef %dst, i8 noundef signext %value)
+void test_memset_inline_0(void *dst, char value) {
+  // CHECK:    call void @llvm.memset.inline.p0i8.i64(i8* align 1 %0, i8 %2, i64 0, i1 false)
+  __builtin_memset_inline(dst, value, 0);
+}
+
+// CHECK-LABEL: define{{.*}} void @test_memset_inline_1(i8* noundef %dst, i8 noundef signext %value)
+void test_memset_inline_1(void *dst, char value) {
+  // CHECK:    call void @llvm.memset.inline.p0i8.i64(i8* align 1 %0, i8 %2, i64 1, i1 false)
+  __builtin_memset_inline(dst, value, 1);
+}
+
+// CHECK-LABEL: define{{.*}} void @test_memset_inline_4(i8* noundef %dst, i8 noundef signext %value)
+void test_memset_inline_4(void *dst, char value) {
+  // CHECK:    call void @llvm.memset.inline.p0i8.i64(i8* align 1 %0, i8 %2, i64 4, i1 false)
+  __builtin_memset_inline(dst, value, 4);
+}

clang/test/Sema/builtins-memset-inline.cpp

+// RUN: %clang_cc1 -fsyntax-only -verify %s
+
+#define NULL ((char *)0)
+
+#if __has_builtin(__builtin_memset_inline)
+#warning defined as expected
+// expected-warning@-1 {{defined as expected}}
+#endif
+
+void test_memset_inline_invalid_arg_types() {
+  __builtin_memset_inline(1, 2, 3); // expected-error {{cannot initialize a parameter of type 'void *' with an rvalue of type 'int'}}
+}
+
+void test_memset_inline_null_dst(void *ptr) {
+  __builtin_memset_inline(NULL, 1, 4); // expected-warning {{null passed to a callee that requires a non-null argument}}
+}
+
+void test_memset_inline_null_buffer_is_ok_if_size_is_zero(void *ptr, char value) {
+  __builtin_memset_inline(NULL, value, /*size */ 0);
+}
+
+void test_memset_inline_non_constant_size(void *dst, char value, unsigned size) {
+  __builtin_memset_inline(dst, value, size); // expected-error {{argument to '__builtin_memset_inline' must be a constant integer}}
+}
+
+template <unsigned size>
+void test_memset_inline_template(void *dst, char value) {
+  // we do not try to evaluate size in non intantiated templates.
+  __builtin_memset_inline(dst, value, size);
+}
+
+void test_memset_inline_implicit_conversion(void *ptr, char value) {
+  char a[5];
+  __builtin_memset_inline(a, value, 5);
+}
+
+void test_memset_inline_num_args(void *dst, char value) {
+  __builtin_memset_inline();                    // expected-error {{too few arguments to function call}}
+  __builtin_memset_inline(dst, value, 4, NULL); // expected-error {{too many arguments to function call}}
+}

llvm/docs/LangRef.rst

 the argument.
 
 If ``<len>`` is 0, it is no-op modulo the behavior of attributes attached to
 the arguments.
 If ``<len>`` is not a well-defined value, the behavior is undefined.
 If ``<len>`` is not zero, ``<dest>`` should be well-defined, otherwise the
 behavior is undefined.
 
+.. _int_memset_inline:
+
+'``llvm.memset.inline``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+This is an overloaded intrinsic. You can use ``llvm.memset.inline`` on any
+integer bit width and for different address spaces. Not all targets
+support all bit widths however.
+
+::
+
+      declare void @llvm.memset.inline.p0i8.p0i8.i32(i8* <dest>, i8 <val>,
+                                                     i32 <len>,
+                                                     i1 <isvolatile>)
+      declare void @llvm.memset.inline.p0i8.p0i8.i64(i8* <dest>, i8 <val>,
+                                                     i64 <len>,
+                                                     i1 <isvolatile>)
+
+Overview:
+"""""""""
+
+The '``llvm.memset.inline.*``' intrinsics fill a block of memory with a
+particular byte value and guarantees that no external functions are called.
+
+Note that, unlike the standard libc function, the ``llvm.memset.inline.*``
+intrinsics do not return a value, take an extra isvolatile argument and the
+pointer can be in specified address spaces.
+
+Arguments:
+""""""""""
+
+The first argument is a pointer to the destination to fill, the second
+is the byte value with which to fill it, the third argument is a constant
+integer argument specifying the number of bytes to fill, and the fourth
+is a boolean indicating a volatile access.
+
+The :ref:`align <attr_align>` parameter attribute can be provided
+for the first argument.
+
+If the ``isvolatile`` parameter is ``true``, the ``llvm.memset.inline`` call is
+a :ref:`volatile operation <volatile>`. The detailed access behavior is not
+very cleanly specified and it is unwise to depend on it.
+
+Semantics:
+""""""""""
+
+The '``llvm.memset.inline.*``' intrinsics fill "len" bytes of memory starting
+at the destination location. If the argument is known to be
+aligned to some boundary, this can be specified as an attribute on
+the argument.
+
+``len`` must be a constant expression.
+If ``<len>`` is 0, it is no-op modulo the behavior of attributes attached to
+the arguments.
+If ``<len>`` is not a well-defined value, the behavior is undefined.
+If ``<len>`` is not zero, ``<dest>`` should be well-defined, otherwise the
+behavior is undefined.
+
+The behavior of '``llvm.memset.inline.*``' is equivalent to the behavior of
+'``llvm.memset.*``', but the generated code is guaranteed not to call any
+external functions.
+
 '``llvm.sqrt.*``' Intrinsic
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 This is an overloaded intrinsic. You can use ``llvm.sqrt`` on any
 floating-point or vector of floating-point type. Not all targets support

llvm/include/llvm/CodeGen/SelectionDAG.h

#endif
 
   SDValue getMemmove(SDValue Chain, const SDLoc &dl, SDValue Dst, SDValue Src,
                      SDValue Size, Align Alignment, bool isVol, bool isTailCall,
                      MachinePointerInfo DstPtrInfo,
                      MachinePointerInfo SrcPtrInfo,
                      const AAMDNodes &AAInfo = AAMDNodes());
 
   SDValue getMemset(SDValue Chain, const SDLoc &dl, SDValue Dst, SDValue Src,
-                    SDValue Size, Align Alignment, bool isVol, bool isTailCall,
+                    SDValue Size, Align Alignment, bool isVol,
+                    bool AlwaysInline, bool isTailCall,
                     MachinePointerInfo DstPtrInfo,
                     const AAMDNodes &AAInfo = AAMDNodes());
 
   SDValue getAtomicMemcpy(SDValue Chain, const SDLoc &dl, SDValue Dst,
                           unsigned DstAlign, SDValue Src, unsigned SrcAlign,
                           SDValue Size, Type *SizeTy, unsigned ElemSz,
                           bool isTailCall, MachinePointerInfo DstPtrInfo,
                           MachinePointerInfo SrcPtrInfo);

llvm/include/llvm/CodeGen/SelectionDAGTargetInfo.h

virtual SDValue EmitTargetCodeForMemmove(
     return SDValue();
   }
 
   /// Emit target-specific code that performs a memset.
   /// This can be used by targets to provide code sequences for cases
   /// that don't fit the target's parameters for simple stores and can be more
   /// efficient than using a library call. This function can return a null
   /// SDValue if the target declines to use custom code and a different
-  /// lowering strategy should be used.
+  /// lowering strategy should be used. Note that if AlwaysInline is true the
+  /// function has to return a valid SDValue.
   virtual SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                           SDValue Chain, SDValue Op1,
                                           SDValue Op2, SDValue Op3,
                                           Align Alignment, bool isVolatile,
+                                          bool AlwaysInline,
                                           MachinePointerInfo DstPtrInfo) const {
     return SDValue();
   }
 
   /// Emit target-specific code that performs a memcmp/bcmp, in cases where that is
   /// faster than a libcall. The first returned SDValue is the result of the
   /// memcmp and the second is the chain. Both SDValues can be null if a normal
   /// libcall should be used.

llvm/include/llvm/CodeGen/TargetLowering.h

bool CombineTo(SDValue O, SDValue N) {
       Old = O;
       New = N;
       return true;
     }
   };
 
   /// Determines the optimal series of memory ops to replace the memset / memcpy.
   /// Return true if the number of memory ops is below the threshold (Limit).
+  /// Note that this is always the case when Limit is ~0.
   /// It returns the types of the sequence of memory ops to perform
   /// memset / memcpy by reference.
   virtual bool
   findOptimalMemOpLowering(std::vector<EVT> &MemOps, unsigned Limit,
                            const MemOp &Op, unsigned DstAS, unsigned SrcAS,
                            const AttributeList &FuncAttributes) const;
 
   /// Check to see if the specified operand of the specified instruction is a

llvm/include/llvm/IR/IRBuilder.h

return CreateMemSet(Ptr, Val, getInt64(Size), Align, isVolatile,
                         TBAATag, ScopeTag, NoAliasTag);
   }
 
   CallInst *CreateMemSet(Value *Ptr, Value *Val, Value *Size, MaybeAlign Align,
                          bool isVolatile = false, MDNode *TBAATag = nullptr,
                          MDNode *ScopeTag = nullptr,
                          MDNode *NoAliasTag = nullptr);
 
+  CallInst *CreateMemSetInline(Value *Dst, MaybeAlign DstAlign, Value *Val,
+                               Value *Size, bool IsVolatile = false,
+                               MDNode *TBAATag = nullptr,
+                               MDNode *ScopeTag = nullptr,
+                               MDNode *NoAliasTag = nullptr);
+
   /// Create and insert an element unordered-atomic memset of the region of
   /// memory starting at the given pointer to the given value.
   ///
   /// If the pointer isn't an i8*, it will be converted. If a TBAA tag is
   /// specified, it will be added to the instruction. Likewise with alias.scope
   /// and noalias tags.
   CallInst *CreateElementUnorderedAtomicMemSet(Value *Ptr, Value *Val,
                                                uint64_t Size, Align Alignment,

llvm/include/llvm/IR/IntrinsicInst.h

public:
   void setVolatile(Constant *V) { setArgOperand(ARG_VOLATILE, V); }
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
     switch (I->getIntrinsicID()) {
     case Intrinsic::memcpy:
     case Intrinsic::memmove:
     case Intrinsic::memset:
+    case Intrinsic::memset_inline:
     case Intrinsic::memcpy_inline:
       return true;
     default:
       return false;
     }
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
-/// This class wraps the llvm.memset intrinsic.
+/// This class wraps the llvm.memset and llvm.memset.inline intrinsics.
 class MemSetInst : public MemSetBase<MemIntrinsic> {
 public:
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const IntrinsicInst *I) {
-    return I->getIntrinsicID() == Intrinsic::memset;
+    switch (I->getIntrinsicID()) {
+    case Intrinsic::memset:
+    case Intrinsic::memset_inline:
+      return true;
+    default:
+      return false;
+    }
+  }
+  static bool classof(const Value *V) {
+    return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
+  }
+};
+
+/// This class wraps the llvm.memset.inline intrinsic.
+class MemSetInlineInst : public MemSetInst {
+public:
+  ConstantInt *getLength() const {
+    return cast<ConstantInt>(MemSetInst::getLength());
+  }
+  // Methods for support type inquiry through isa, cast, and dyn_cast:
+  static bool classof(const IntrinsicInst *I) {
+    return I->getIntrinsicID() == Intrinsic::memset_inline;
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
 /// This class wraps the llvm.memcpy/memmove intrinsics.
 class MemTransferInst : public MemTransferBase<MemIntrinsic> {
public:
   }
 
   static bool classof(const IntrinsicInst *I) {
     switch (I->getIntrinsicID()) {
     case Intrinsic::memcpy:
     case Intrinsic::memcpy_inline:
     case Intrinsic::memmove:
     case Intrinsic::memset:
+    case Intrinsic::memset_inline:
     case Intrinsic::memcpy_element_unordered_atomic:
     case Intrinsic::memmove_element_unordered_atomic:
     case Intrinsic::memset_element_unordered_atomic:
       return true;
     default:
       return false;
     }
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
   }
 };
 
 /// This class represents any memset intrinsic
 // i.e. llvm.element.unordered.atomic.memset
 // and  llvm.memset
 class AnyMemSetInst : public MemSetBase<AnyMemIntrinsic> {
 public:
   static bool classof(const IntrinsicInst *I) {
     switch (I->getIntrinsicID()) {
     case Intrinsic::memset:
+    case Intrinsic::memset_inline:
     case Intrinsic::memset_element_unordered_atomic:
       return true;
     default:
       return false;
     }
   }
   static bool classof(const Value *V) {
     return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));

llvm/include/llvm/IR/Intrinsics.td

 def int_memset  : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_i8_ty, llvm_anyint_ty,
                              llvm_i1_ty],
                             [IntrWriteMem, IntrArgMemOnly, IntrWillReturn,
                              IntrNoFree,
                              NoCapture<ArgIndex<0>>, WriteOnly<ArgIndex<0>>,
                              ImmArg<ArgIndex<3>>]>;
 
+// Memset version that is guaranteed to be inlined.
+// In particular this means that the generated code is not allowed to call any
+// external function.
+// The third argument (specifying the size) must be a constant.
+def int_memset_inline
+    : Intrinsic<[],
+      [llvm_anyptr_ty, llvm_i8_ty, llvm_anyint_ty, llvm_i1_ty],
+      [IntrWriteMem, IntrArgMemOnly, IntrWillReturn, IntrNoFree,
+       NoCapture<ArgIndex<0>>, WriteOnly<ArgIndex<0>>, 
+       ImmArg<ArgIndex<2>>, ImmArg<ArgIndex<3>>]>;
+
 // FIXME: Add version of these floating point intrinsics which allow non-default
 // rounding modes and FP exception handling.
 
 let IntrProperties = [IntrNoMem, IntrSpeculatable, IntrWillReturn] in {
   def int_fma  : DefaultAttrsIntrinsic<[llvm_anyfloat_ty],
                            [LLVMMatchType<0>, LLVMMatchType<0>,
                             LLVMMatchType<0>]>;
   def int_fmuladd : DefaultAttrsIntrinsic<[llvm_anyfloat_ty],

llvm/lib/Analysis/Lint.cpp

case Intrinsic::memmove: {
       break;
     }
     case Intrinsic::memset: {
       MemSetInst *MSI = cast<MemSetInst>(&I);
       visitMemoryReference(I, MemoryLocation::getForDest(MSI),
                            MSI->getDestAlign(), nullptr, MemRef::Write);
       break;
     }
+    case Intrinsic::memset_inline: {
+      MemSetInlineInst *MSII = cast<MemSetInlineInst>(&I);
+      visitMemoryReference(I, MemoryLocation::getForDest(MSII),
+                           MSII->getDestAlign(), nullptr, MemRef::Write);
+      break;
+    }
 
     case Intrinsic::vastart:
       Check(I.getParent()->getParent()->isVarArg(),
             "Undefined behavior: va_start called in a non-varargs function",
             &I);
 
       visitMemoryReference(I, MemoryLocation::getForArgument(&I, 0, TLI), None,
                            nullptr, MemRef::Read | MemRef::Write);

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

 /// \param DAG Selection DAG where lowered code is placed.
 /// \param dl Link to corresponding IR location.
 /// \param Chain Control flow dependency.
 /// \param Dst Pointer to destination memory location.
 /// \param Src Value of byte to write into the memory.
 /// \param Size Number of bytes to write.
 /// \param Alignment Alignment of the destination in bytes.
 /// \param isVol True if destination is volatile.
+/// \param AlwaysInline Makes sure no function call is generated.
 /// \param DstPtrInfo IR information on the memory pointer.
 /// \returns New head in the control flow, if lowering was successful, empty
 /// SDValue otherwise.
 ///
 /// The function tries to replace 'llvm.memset' intrinsic with several store
 /// operations and value calculation code. This is usually profitable for small
-/// memory size.
+/// memory size or when the semantic requires inlining.
 static SDValue getMemsetStores(SelectionDAG &DAG, const SDLoc &dl,
                                SDValue Chain, SDValue Dst, SDValue Src,
                                uint64_t Size, Align Alignment, bool isVol,
-                               MachinePointerInfo DstPtrInfo,
+                               bool AlwaysInline, MachinePointerInfo DstPtrInfo,
                                const AAMDNodes &AAInfo) {
   // Turn a memset of undef to nop.
   // FIXME: We need to honor volatile even is Src is undef.
   if (Src.isUndef())
     return Chain;
 
   // Expand memset to a series of load/store ops if the size operand
   // falls below a certain threshold.
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   std::vector<EVT> MemOps;
   bool DstAlignCanChange = false;
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   bool OptSize = shouldLowerMemFuncForSize(MF, DAG);
   FrameIndexSDNode *FI = dyn_cast<FrameIndexSDNode>(Dst);
   if (FI && !MFI.isFixedObjectIndex(FI->getIndex()))
     DstAlignCanChange = true;
   bool IsZeroVal =
       isa<ConstantSDNode>(Src) && cast<ConstantSDNode>(Src)->isZero();
+  unsigned Limit = AlwaysInline ? ~0 : TLI.getMaxStoresPerMemset(OptSize);
+
   if (!TLI.findOptimalMemOpLowering(
-          MemOps, TLI.getMaxStoresPerMemset(OptSize),
+          MemOps, Limit,
           MemOp::Set(Size, DstAlignCanChange, Alignment, IsZeroVal, isVol),
           DstPtrInfo.getAddrSpace(), ~0u, MF.getFunction().getAttributes()))
     return SDValue();
 
   if (DstAlignCanChange) {
     Type *Ty = MemOps[0].getTypeForEVT(*DAG.getContext());
     Align NewAlign = DAG.getDataLayout().getABITypeAlign(Ty);
     if (NewAlign > Alignment) {
CLI.setDebugLoc(dl)
       .setTailCall(isTailCall);
 
   std::pair<SDValue, SDValue> CallResult = TLI->LowerCallTo(CLI);
   return CallResult.second;
 }
 
 SDValue SelectionDAG::getMemset(SDValue Chain, const SDLoc &dl, SDValue Dst,
                                 SDValue Src, SDValue Size, Align Alignment,
-                                bool isVol, bool isTailCall,
+                                bool isVol, bool AlwaysInline, bool isTailCall,
                                 MachinePointerInfo DstPtrInfo,
                                 const AAMDNodes &AAInfo) {
   // Check to see if we should lower the memset to stores first.
   // For cases within the target-specified limits, this is the best choice.
   ConstantSDNode *ConstantSize = dyn_cast<ConstantSDNode>(Size);
   if (ConstantSize) {
     // Memset with size zero? Just return the original chain.
     if (ConstantSize->isZero())
       return Chain;
 
     SDValue Result = getMemsetStores(*this, dl, Chain, Dst, Src,
                                      ConstantSize->getZExtValue(), Alignment,
-                                     isVol, DstPtrInfo, AAInfo);
+                                     isVol, false, DstPtrInfo, AAInfo);
 
     if (Result.getNode())
       return Result;
   }
 
   // Then check to see if we should lower the memset with target-specific
   // code. If the target chooses to do this, this is the next best.
   if (TSI) {
     SDValue Result = TSI->EmitTargetCodeForMemset(
-        *this, dl, Chain, Dst, Src, Size, Alignment, isVol, DstPtrInfo);
+        *this, dl, Chain, Dst, Src, Size, Alignment, isVol, AlwaysInline, DstPtrInfo);
     if (Result.getNode())
       return Result;
   }
 
+  // If we really need inline code and the target declined to provide it,
+  // use a (potentially long) sequence of loads and stores.
+  if (AlwaysInline) {
+    assert(ConstantSize && "AlwaysInline requires a constant size!");
+    SDValue Result = getMemsetStores(*this, dl, Chain, Dst, Src,
+                                     ConstantSize->getZExtValue(), Alignment,
+                                     isVol, true, DstPtrInfo, AAInfo);
+    assert(Result &&
+           "getMemsetStores must return a valid sequence when AlwaysInline");
+    return Result;
+  }
+
   checkAddrSpaceIsValidForLibcall(TLI, DstPtrInfo.getAddrSpace());
 
   // Emit a library call.
   auto &Ctx = *getContext();
   const auto& DL = getDataLayout();
 
   TargetLowering::CallLoweringInfo CLI(*this);
   // FIXME: pass in SDLoc

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::memset: {
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     SDValue Op3 = getValue(I.getArgOperand(2));
     // @llvm.memset defines 0 and 1 to both mean no alignment.
     Align Alignment = MSI.getDestAlign().valueOrOne();
     bool isVol = MSI.isVolatile();
     bool isTC = I.isTailCall() && isInTailCallPosition(I, DAG.getTarget());
     SDValue Root = isVol ? getRoot() : getMemoryRoot();
-    SDValue MS = DAG.getMemset(Root, sdl, Op1, Op2, Op3, Alignment, isVol, isTC,
+    SDValue MS = DAG.getMemset(
+        Root, sdl, Op1, Op2, Op3, Alignment, isVol, /* AlwaysInline */ false,
+        isTC, MachinePointerInfo(I.getArgOperand(0)), I.getAAMetadata());
+    updateDAGForMaybeTailCall(MS);
+    return;
+  }
+  case Intrinsic::memset_inline: {
+    const auto &MSII = cast<MemSetInlineInst>(I);
+    SDValue Dst = getValue(I.getArgOperand(0));
+    SDValue Value = getValue(I.getArgOperand(1));
+    SDValue Size = getValue(I.getArgOperand(2));
+    assert(isa<ConstantSDNode>(Size) && "memset_inline needs constant size");
+    // @llvm.memset defines 0 and 1 to both mean no alignment.
+    Align DstAlign = MSII.getDestAlign().valueOrOne();
+    bool isVol = MSII.isVolatile();
+    bool isTC = I.isTailCall() && isInTailCallPosition(I, DAG.getTarget());
+    SDValue Root = isVol ? getRoot() : getMemoryRoot();
+    SDValue MC = DAG.getMemset(Root, sdl, Dst, Value, Size, DstAlign, isVol,
+                               /* AlwaysInline */ true, isTC,
                                MachinePointerInfo(I.getArgOperand(0)),
                                I.getAAMetadata());
-    updateDAGForMaybeTailCall(MS);
+    updateDAGForMaybeTailCall(MC);
     return;
   }
   case Intrinsic::memmove: {
     const auto &MMI = cast<MemMoveInst>(I);
     SDValue Op1 = getValue(I.getArgOperand(0));
     SDValue Op2 = getValue(I.getArgOperand(1));
     SDValue Op3 = getValue(I.getArgOperand(2));
     // @llvm.memmove defines 0 and 1 to both mean no alignment.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

CLI.setDebugLoc(dl)
       .setSExtResult(signExtend)
       .setZExtResult(zeroExtend);
   return LowerCallTo(CLI);
 }
 
 bool TargetLowering::findOptimalMemOpLowering(
     std::vector<EVT> &MemOps, unsigned Limit, const MemOp &Op, unsigned DstAS,
     unsigned SrcAS, const AttributeList &FuncAttributes) const {
-  if (Op.isMemcpyWithFixedDstAlign() && Op.getSrcAlign() < Op.getDstAlign())
+  if (Limit != ~unsigned(0) && Op.isMemcpyWithFixedDstAlign() &&
+      Op.getSrcAlign() < Op.getDstAlign())
     return false;
 
   EVT VT = getOptimalMemOpType(Op, FuncAttributes);
 
   if (VT == MVT::Other) {
     // Use the largest integer type whose alignment constraints are satisfied.
     // We only need to check DstAlign here as SrcAlign is always greater or
     // equal to DstAlign (or zero).

llvm/lib/IR/IRBuilder.cpp

if (ScopeTag)
     CI->setMetadata(LLVMContext::MD_alias_scope, ScopeTag);
 
   if (NoAliasTag)
     CI->setMetadata(LLVMContext::MD_noalias, NoAliasTag);
 
   return CI;
 }
 
+CallInst *IRBuilderBase::CreateMemSetInline(Value *Dst, MaybeAlign DstAlign,
+                                            Value *Val, Value *Size,
+                                            bool IsVolatile, MDNode *TBAATag,
+                                            MDNode *ScopeTag,
+                                            MDNode *NoAliasTag) {
+  Dst = getCastedInt8PtrValue(Dst);
+  Value *Ops[] = {Dst, Val, Size, getInt1(IsVolatile)};
+  Type *Tys[] = {Dst->getType(), Size->getType()};
+  Module *M = BB->getParent()->getParent();
+  Function *TheFn = Intrinsic::getDeclaration(M, Intrinsic::memset_inline, Tys);
+
+  CallInst *CI = createCallHelper(TheFn, Ops, this);
+
+  if (DstAlign)
+    cast<MemSetInlineInst>(CI)->setDestAlignment(*DstAlign);
+
+  // Set the TBAA info if present.
+  if (TBAATag)
+    CI->setMetadata(LLVMContext::MD_tbaa, TBAATag);
+
+  if (ScopeTag)
+    CI->setMetadata(LLVMContext::MD_alias_scope, ScopeTag);
+
+  if (NoAliasTag)
+    CI->setMetadata(LLVMContext::MD_noalias, NoAliasTag);
+
+  return CI;
+}
+
 CallInst *IRBuilderBase::CreateElementUnorderedAtomicMemSet(
     Value *Ptr, Value *Val, Value *Size, Align Alignment, uint32_t ElementSize,
     MDNode *TBAATag, MDNode *ScopeTag, MDNode *NoAliasTag) {
 
   Ptr = getCastedInt8PtrValue(Ptr);
   Value *Ops[] = {Ptr, Val, Size, getInt32(ElementSize)};
   Type *Tys[] = {Ptr->getType(), Size->getType()};
   Module *M = BB->getParent()->getParent();

llvm/lib/IR/Verifier.cpp

case Intrinsic::dbg_value: // llvm.dbg.value
     visitDbgIntrinsic("value", cast<DbgVariableIntrinsic>(Call));
     break;
   case Intrinsic::dbg_label: // llvm.dbg.label
     visitDbgLabelIntrinsic("label", cast<DbgLabelInst>(Call));
     break;
   case Intrinsic::memcpy:
   case Intrinsic::memcpy_inline:
   case Intrinsic::memmove:
-  case Intrinsic::memset: {
+  case Intrinsic::memset:
+  case Intrinsic::memset_inline: {
     const auto *MI = cast<MemIntrinsic>(&Call);
     auto IsValidAlignment = [&](unsigned Alignment) -> bool {
       return Alignment == 0 || isPowerOf2_32(Alignment);
     };
     Check(IsValidAlignment(MI->getDestAlignment()),
           "alignment of arg 0 of memory intrinsic must be 0 or a power of 2",
           Call);
     if (const auto *MTI = dyn_cast<MemTransferInst>(MI)) {

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.h

SDValue EmitTargetCodeForMemcpy(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
                                   bool isVolatile, bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo,
                                   MachinePointerInfo SrcPtrInfo) const override;
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
-                                  bool isVolatile,
+                                  bool isVolatile, bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo) const override;
   SDValue
   EmitTargetCodeForMemmove(SelectionDAG &DAG, const SDLoc &dl, SDValue Chain,
                            SDValue Dst, SDValue Src, SDValue Size,
                            Align Alignment, bool isVolatile,
                            MachinePointerInfo DstPtrInfo,
                            MachinePointerInfo SrcPtrInfo) const override;
 
   SDValue EmitTargetCodeForSetTag(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Op1, SDValue Op2,
                                   MachinePointerInfo DstPtrInfo,
                                   bool ZeroData) const override;
 };
 }
 
 #endif

llvm/lib/Target/AArch64/AArch64SelectionDAGInfo.cpp

SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemcpy(
   if (STI.hasMOPS())
     return EmitMOPS(AArch64ISD::MOPS_MEMCOPY, DAG, DL, Chain, Dst, Src, Size,
                     Alignment, isVolatile, DstPtrInfo, SrcPtrInfo);
   return SDValue();
 }
 
 SDValue AArch64SelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
-    SDValue Size, Align Alignment, bool isVolatile,
+    SDValue Size, Align Alignment, bool isVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo) const {
   const AArch64Subtarget &STI =
       DAG.getMachineFunction().getSubtarget<AArch64Subtarget>();
 
   if (STI.hasMOPS()) {
     return EmitMOPS(AArch64ISD::MOPS_MEMSET, DAG, dl, Chain, Dst, Src, Size,
                     Alignment, isVolatile, DstPtrInfo, MachinePointerInfo{});
   }

llvm/lib/Target/ARM/ARMSelectionDAGInfo.h

EmitTargetCodeForMemmove(SelectionDAG &DAG, const SDLoc &dl, SDValue Chain,
                            Align Alignment, bool isVolatile,
                            MachinePointerInfo DstPtrInfo,
                            MachinePointerInfo SrcPtrInfo) const override;
 
   // Adjust parameters for memset, see RTABI section 4.3.4
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Op1, SDValue Op2,
                                   SDValue Op3, Align Alignment, bool isVolatile,
+                                  bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo) const override;
 
   SDValue EmitSpecializedLibcall(SelectionDAG &DAG, const SDLoc &dl,
                                  SDValue Chain, SDValue Dst, SDValue Src,
                                  SDValue Size, unsigned Align,
                                  RTLIB::Libcall LC) const;
 };
 
 }
 
 #endif

llvm/lib/Target/ARM/ARMSelectionDAGInfo.cpp

SDValue ARMSelectionDAGInfo::EmitTargetCodeForMemmove(
     SDValue Size, Align Alignment, bool isVolatile,
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
   return EmitSpecializedLibcall(DAG, dl, Chain, Dst, Src, Size,
                                 Alignment.value(), RTLIB::MEMMOVE);
 }
 
 SDValue ARMSelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Src,
-    SDValue Size, Align Alignment, bool isVolatile,
+    SDValue Size, Align Alignment, bool isVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo) const {
 
   const ARMSubtarget &Subtarget =
       DAG.getMachineFunction().getSubtarget<ARMSubtarget>();
 
   ConstantSDNode *ConstantSize = dyn_cast<ConstantSDNode>(Size);
 
   // Generate TP loop for llvm.memset
   if (Subtarget.hasMVEIntegerOps() &&
       shouldGenerateInlineTPLoop(Subtarget, DAG, ConstantSize, Alignment,
                                  false)) {
     Src = DAG.getSplatBuildVector(MVT::v16i8, dl,
                                   DAG.getNode(ISD::TRUNCATE, dl, MVT::i8, Src));
     return DAG.getNode(ARMISD::MEMSETLOOP, dl, MVT::Other, Chain, Dst, Src,
                        DAG.getZExtOrTrunc(Size, dl, MVT::i32));
   }
 
+  if (!AlwaysInline)
     return EmitSpecializedLibcall(DAG, dl, Chain, Dst, Src, Size,
                                   Alignment.value(), RTLIB::MEMSET);
+
+  return SDValue();
 }

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

else
     return AM.Scale == 0 || AM.Scale == 1;
 }
 
 bool SystemZTargetLowering::findOptimalMemOpLowering(
     std::vector<EVT> &MemOps, unsigned Limit, const MemOp &Op, unsigned DstAS,
     unsigned SrcAS, const AttributeList &FuncAttributes) const {
   const int MVCFastLen = 16;
 
+  if (Limit != ~unsigned(0)) {
     // Don't expand Op into scalar loads/stores in these cases:
     if (Op.isMemcpy() && Op.allowOverlap() && Op.size() <= MVCFastLen)
-    return false;  // Small memcpy: Use MVC
+      return false; // Small memcpy: Use MVC
     if (Op.isMemset() && Op.size() - 1 <= MVCFastLen)
-    return false;  // Small memset (first byte with STC/MVI): Use MVC
+      return false; // Small memset (first byte with STC/MVI): Use MVC
     if (Op.isZeroMemset())
-    return false;  // Memset zero: Use XC
+      return false; // Memset zero: Use XC
+  }
 
   return TargetLowering::findOptimalMemOpLowering(MemOps, Limit, Op, DstAS,
                                                   SrcAS, FuncAttributes);
 }
 
 EVT SystemZTargetLowering::getOptimalMemOpType(const MemOp &Op,
                                    const AttributeList &FuncAttributes) const {
   return Subtarget.hasVector() ? MVT::v2i64 : MVT::Other;

llvm/lib/Target/SystemZ/SystemZSelectionDAGInfo.h

SDValue EmitTargetCodeForMemcpy(SelectionDAG &DAG, const SDLoc &DL,
                                   SDValue Size, Align Alignment,
                                   bool IsVolatile, bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo,
                                   MachinePointerInfo SrcPtrInfo) const override;
 
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &DL,
                                   SDValue Chain, SDValue Dst, SDValue Byte,
                                   SDValue Size, Align Alignment,
-                                  bool IsVolatile,
+                                  bool IsVolatile, bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo) const override;
 
   std::pair<SDValue, SDValue>
   EmitTargetCodeForMemcmp(SelectionDAG &DAG, const SDLoc &DL, SDValue Chain,
                           SDValue Src1, SDValue Src2, SDValue Size,
                           MachinePointerInfo Op1PtrInfo,
                           MachinePointerInfo Op2PtrInfo) const override;
 

llvm/lib/Target/SystemZ/SystemZSelectionDAGInfo.cpp

static SDValue memsetStore(SelectionDAG &DAG, const SDLoc &DL, SDValue Chain,
   return DAG.getStore(
       Chain, DL, DAG.getConstant(StoreVal, DL, MVT::getIntegerVT(Size * 8)),
       Dst, DstPtrInfo, Align);
 }
 
 SDValue SystemZSelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst,
     SDValue Byte, SDValue Size, Align Alignment, bool IsVolatile,
-    MachinePointerInfo DstPtrInfo) const {
+    bool AlwaysInline, MachinePointerInfo DstPtrInfo) const {
   EVT PtrVT = Dst.getValueType();
 
   if (IsVolatile)
     return SDValue();
 
   auto *CByte = dyn_cast<ConstantSDNode>(Byte);
   if (auto *CSize = dyn_cast<ConstantSDNode>(Size)) {
     uint64_t Bytes = CSize->getZExtValue();

llvm/lib/Target/WebAssembly/WebAssemblySelectionDAGInfo.h

public:
   EmitTargetCodeForMemmove(SelectionDAG &DAG, const SDLoc &dl, SDValue Chain,
                            SDValue Op1, SDValue Op2, SDValue Op3,
                            Align Alignment, bool isVolatile,
                            MachinePointerInfo DstPtrInfo,
                            MachinePointerInfo SrcPtrInfo) const override;
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &DL,
                                   SDValue Chain, SDValue Op1, SDValue Op2,
                                   SDValue Op3, Align Alignment, bool IsVolatile,
+                                  bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo) const override;
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/WebAssembly/WebAssemblySelectionDAGInfo.cpp

SDValue WebAssemblySelectionDAGInfo::EmitTargetCodeForMemmove(
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
   return EmitTargetCodeForMemcpy(DAG, DL, Chain, Op1, Op2, Op3,
                                  Alignment, IsVolatile, false,
                                  DstPtrInfo, SrcPtrInfo);
 }
 
 SDValue WebAssemblySelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst, SDValue Val,
-    SDValue Size, Align Alignment, bool IsVolatile,
+    SDValue Size, Align Alignment, bool IsVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo) const {
   auto &ST = DAG.getMachineFunction().getSubtarget<WebAssemblySubtarget>();
   if (!ST.hasBulkMemory())
     return SDValue();
 
   SDValue MemIdx = DAG.getConstant(0, DL, MVT::i32);
   auto LenMVT = ST.hasAddr64() ? MVT::i64 : MVT::i32;
   // Only low byte matters for val argument, so anyext the i8
   return DAG.getNode(WebAssemblyISD::MEMORY_FILL, DL, MVT::Other, Chain, MemIdx,
                      Dst, DAG.getAnyExtOrTrunc(Val, DL, MVT::i32),
                      DAG.getZExtOrTrunc(Size, DL, LenMVT));
 }

llvm/lib/Target/X86/X86SelectionDAGInfo.h

bool isBaseRegConflictPossible(SelectionDAG &DAG,
                                  ArrayRef<MCPhysReg> ClobberSet) const;
 
 public:
   explicit X86SelectionDAGInfo() = default;
 
   SDValue EmitTargetCodeForMemset(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
-                                  bool isVolatile,
+                                  bool isVolatile, bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo) const override;
 
   SDValue EmitTargetCodeForMemcpy(SelectionDAG &DAG, const SDLoc &dl,
                                   SDValue Chain, SDValue Dst, SDValue Src,
                                   SDValue Size, Align Alignment,
                                   bool isVolatile, bool AlwaysInline,
                                   MachinePointerInfo DstPtrInfo,
                                   MachinePointerInfo SrcPtrInfo) const override;
 };
 
 }
 
 #endif

llvm/lib/Target/X86/X86SelectionDAGInfo.cpp

bool X86SelectionDAGInfo::isBaseRegConflictPossible(
 
   const X86RegisterInfo *TRI = static_cast<const X86RegisterInfo *>(
       DAG.getSubtarget().getRegisterInfo());
   return llvm::is_contained(ClobberSet, TRI->getBaseRegister());
 }
 
 SDValue X86SelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &dl, SDValue Chain, SDValue Dst, SDValue Val,
-    SDValue Size, Align Alignment, bool isVolatile,
+    SDValue Size, Align Alignment, bool isVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo) const {
   ConstantSDNode *ConstantSize = dyn_cast<ConstantSDNode>(Size);
   const X86Subtarget &Subtarget =
       DAG.getMachineFunction().getSubtarget<X86Subtarget>();
 
 #ifndef NDEBUG
   // If the base register might conflict with our physical registers, bail out.
   const MCPhysReg ClobberSet[] = {X86::RCX, X86::RAX, X86::RDI,
if (BytesLeft) {
     EVT AddrVT = Dst.getValueType();
     EVT SizeVT = Size.getValueType();
 
     Chain =
         DAG.getMemset(Chain, dl,
                       DAG.getNode(ISD::ADD, dl, AddrVT, Dst,
                                   DAG.getConstant(Offset, dl, AddrVT)),
                       Val, DAG.getConstant(BytesLeft, dl, SizeVT), Alignment,
-                      isVolatile, false, DstPtrInfo.getWithOffset(Offset));
+                      isVolatile, AlwaysInline,
+                      /* isTailCall */ false, DstPtrInfo.getWithOffset(Offset));
   }
 
   // TODO: Use a Tokenfactor, as in memcpy, instead of a single chain.
   return Chain;
 }
 
 /// Emit a single REP MOVS{B,W,D,Q} instruction.
 static SDValue emitRepmovs(const X86Subtarget &Subtarget, SelectionDAG &DAG,

llvm/test/CodeGen/AArch64/memset-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu -mattr=-neon | FileCheck %s --check-prefixes=ALL,GPR
+; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu -mattr=neon  | FileCheck %s --check-prefixes=ALL,NEON
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+declare void @llvm.memset.inline.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @memset_1(i8* %a, i8 %value) nounwind {
+; ALL-LABEL: memset_1:
+; ALL:       // %bb.0:
+; ALL-NEXT:    strb w1, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 1, i1 0)
+  ret void
+}
+
+define void @memset_2(i8* %a, i8 %value) nounwind {
+; ALL-LABEL: memset_2:
+; ALL:       // %bb.0:
+; ALL-NEXT:    bfi w1, w1, #8, #24
+; ALL-NEXT:    strh w1, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 2, i1 0)
+  ret void
+}
+
+define void @memset_4(i8* %a, i8 %value) nounwind {
+; ALL-LABEL: memset_4:
+; ALL:       // %bb.0:
+; ALL-NEXT:    mov w8, #16843009
+; ALL-NEXT:    and w9, w1, #0xff
+; ALL-NEXT:    mul w8, w9, w8
+; ALL-NEXT:    str w8, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 4, i1 0)
+  ret void
+}
+
+define void @memset_8(i8* %a, i8 %value) nounwind {
+; ALL-LABEL: memset_8:
+; ALL:       // %bb.0:
+; ALL-NEXT:    // kill: def $w1 killed $w1 def $x1
+; ALL-NEXT:    mov x8, #72340172838076673
+; ALL-NEXT:    and x9, x1, #0xff
+; ALL-NEXT:    mul x8, x9, x8
+; ALL-NEXT:    str x8, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 8, i1 0)
+  ret void
+}
+
+define void @memset_16(i8* %a, i8 %value) nounwind {
+; ALL-LABEL: memset_16:
+; ALL:       // %bb.0:
+; ALL-NEXT:    // kill: def $w1 killed $w1 def $x1
+; ALL-NEXT:    mov x8, #72340172838076673
+; ALL-NEXT:    and x9, x1, #0xff
+; ALL-NEXT:    mul x8, x9, x8
+; ALL-NEXT:    stp x8, x8, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 16, i1 0)
+  ret void
+}
+
+define void @memset_32(i8* %a, i8 %value) nounwind {
+; GPR-LABEL: memset_32:
+; GPR:       // %bb.0:
+; GPR-NEXT:    // kill: def $w1 killed $w1 def $x1
+; GPR-NEXT:    mov x8, #72340172838076673
+; GPR-NEXT:    and x9, x1, #0xff
+; GPR-NEXT:    mul x8, x9, x8
+; GPR-NEXT:    stp x8, x8, [x0, #16]
+; GPR-NEXT:    stp x8, x8, [x0]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: memset_32:
+; NEON:       // %bb.0:
+; NEON-NEXT:    dup v0.16b, w1
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 32, i1 0)
+  ret void
+}
+
+define void @memset_64(i8* %a, i8 %value) nounwind {
+; GPR-LABEL: memset_64:
+; GPR:       // %bb.0:
+; GPR-NEXT:    // kill: def $w1 killed $w1 def $x1
+; GPR-NEXT:    mov x8, #72340172838076673
+; GPR-NEXT:    and x9, x1, #0xff
+; GPR-NEXT:    mul x8, x9, x8
+; GPR-NEXT:    stp x8, x8, [x0, #48]
+; GPR-NEXT:    stp x8, x8, [x0, #32]
+; GPR-NEXT:    stp x8, x8, [x0, #16]
+; GPR-NEXT:    stp x8, x8, [x0]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: memset_64:
+; NEON:       // %bb.0:
+; NEON-NEXT:    dup v0.16b, w1
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    stp q0, q0, [x0, #32]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 64, i1 0)
+  ret void
+}
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @aligned_memset_16(i8* align 16 %a, i8 %value) nounwind {
+; ALL-LABEL: aligned_memset_16:
+; ALL:       // %bb.0:
+; ALL-NEXT:    // kill: def $w1 killed $w1 def $x1
+; ALL-NEXT:    mov x8, #72340172838076673
+; ALL-NEXT:    and x9, x1, #0xff
+; ALL-NEXT:    mul x8, x9, x8
+; ALL-NEXT:    stp x8, x8, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 16 %a, i8 %value, i64 16, i1 0)
+  ret void
+}
+
+define void @aligned_memset_32(i8* align 32 %a, i8 %value) nounwind {
+; GPR-LABEL: aligned_memset_32:
+; GPR:       // %bb.0:
+; GPR-NEXT:    // kill: def $w1 killed $w1 def $x1
+; GPR-NEXT:    mov x8, #72340172838076673
+; GPR-NEXT:    and x9, x1, #0xff
+; GPR-NEXT:    mul x8, x9, x8
+; GPR-NEXT:    stp x8, x8, [x0, #16]
+; GPR-NEXT:    stp x8, x8, [x0]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: aligned_memset_32:
+; NEON:       // %bb.0:
+; NEON-NEXT:    dup v0.16b, w1
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 32 %a, i8 %value, i64 32, i1 0)
+  ret void
+}
+
+define void @aligned_memset_64(i8* align 64 %a, i8 %value) nounwind {
+; GPR-LABEL: aligned_memset_64:
+; GPR:       // %bb.0:
+; GPR-NEXT:    // kill: def $w1 killed $w1 def $x1
+; GPR-NEXT:    mov x8, #72340172838076673
+; GPR-NEXT:    and x9, x1, #0xff
+; GPR-NEXT:    mul x8, x9, x8
+; GPR-NEXT:    stp x8, x8, [x0, #48]
+; GPR-NEXT:    stp x8, x8, [x0, #32]
+; GPR-NEXT:    stp x8, x8, [x0, #16]
+; GPR-NEXT:    stp x8, x8, [x0]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: aligned_memset_64:
+; NEON:       // %bb.0:
+; NEON-NEXT:    dup v0.16b, w1
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    stp q0, q0, [x0, #32]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 64 %a, i8 %value, i64 64, i1 0)
+  ret void
+}
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @bzero_1(i8* %a) nounwind {
+; ALL-LABEL: bzero_1:
+; ALL:       // %bb.0:
+; ALL-NEXT:    strb wzr, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 1, i1 0)
+  ret void
+}
+
+define void @bzero_2(i8* %a) nounwind {
+; ALL-LABEL: bzero_2:
+; ALL:       // %bb.0:
+; ALL-NEXT:    strh wzr, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 2, i1 0)
+  ret void
+}
+
+define void @bzero_4(i8* %a) nounwind {
+; ALL-LABEL: bzero_4:
+; ALL:       // %bb.0:
+; ALL-NEXT:    str wzr, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 4, i1 0)
+  ret void
+}
+
+define void @bzero_8(i8* %a) nounwind {
+; ALL-LABEL: bzero_8:
+; ALL:       // %bb.0:
+; ALL-NEXT:    str xzr, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 8, i1 0)
+  ret void
+}
+
+define void @bzero_16(i8* %a) nounwind {
+; ALL-LABEL: bzero_16:
+; ALL:       // %bb.0:
+; ALL-NEXT:    stp xzr, xzr, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 16, i1 0)
+  ret void
+}
+
+define void @bzero_32(i8* %a) nounwind {
+; GPR-LABEL: bzero_32:
+; GPR:       // %bb.0:
+; GPR-NEXT:    adrp x8, .LCPI15_0
+; GPR-NEXT:    ldr q0, [x8, :lo12:.LCPI15_0]
+; GPR-NEXT:    stp q0, q0, [x0]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: bzero_32:
+; NEON:       // %bb.0:
+; NEON-NEXT:    movi v0.2d, #0000000000000000
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 32, i1 0)
+  ret void
+}
+
+define void @bzero_64(i8* %a) nounwind {
+; GPR-LABEL: bzero_64:
+; GPR:       // %bb.0:
+; GPR-NEXT:    adrp x8, .LCPI16_0
+; GPR-NEXT:    ldr q0, [x8, :lo12:.LCPI16_0]
+; GPR-NEXT:    stp q0, q0, [x0]
+; GPR-NEXT:    stp q0, q0, [x0, #32]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: bzero_64:
+; NEON:       // %bb.0:
+; NEON-NEXT:    movi v0.2d, #0000000000000000
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    stp q0, q0, [x0, #32]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 64, i1 0)
+  ret void
+}
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @aligned_bzero_16(i8* %a) nounwind {
+; ALL-LABEL: aligned_bzero_16:
+; ALL:       // %bb.0:
+; ALL-NEXT:    stp xzr, xzr, [x0]
+; ALL-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 16 %a, i8 0, i64 16, i1 0)
+  ret void
+}
+
+define void @aligned_bzero_32(i8* %a) nounwind {
+; GPR-LABEL: aligned_bzero_32:
+; GPR:       // %bb.0:
+; GPR-NEXT:    adrp x8, .LCPI18_0
+; GPR-NEXT:    ldr q0, [x8, :lo12:.LCPI18_0]
+; GPR-NEXT:    stp q0, q0, [x0]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: aligned_bzero_32:
+; NEON:       // %bb.0:
+; NEON-NEXT:    movi v0.2d, #0000000000000000
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 32 %a, i8 0, i64 32, i1 0)
+  ret void
+}
+
+define void @aligned_bzero_64(i8* %a) nounwind {
+; GPR-LABEL: aligned_bzero_64:
+; GPR:       // %bb.0:
+; GPR-NEXT:    adrp x8, .LCPI19_0
+; GPR-NEXT:    ldr q0, [x8, :lo12:.LCPI19_0]
+; GPR-NEXT:    stp q0, q0, [x0]
+; GPR-NEXT:    stp q0, q0, [x0, #32]
+; GPR-NEXT:    ret
+;
+; NEON-LABEL: aligned_bzero_64:
+; NEON:       // %bb.0:
+; NEON-NEXT:    movi v0.2d, #0000000000000000
+; NEON-NEXT:    stp q0, q0, [x0]
+; NEON-NEXT:    stp q0, q0, [x0, #32]
+; NEON-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 64 %a, i8 0, i64 64, i1 0)
+  ret void
+}

llvm/test/CodeGen/AArch64/memset-vs-memset-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu | FileCheck %s
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+declare void @llvm.memset.inline.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+
+define void @test1(i8* %a, i8 %value) nounwind {
+; CHECK-LABEL: test1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $w1 killed $w1 def $x1
+; CHECK-NEXT:    mov x8, #72340172838076673
+; CHECK-NEXT:    and x9, x1, #0xff
+; CHECK-NEXT:    mul x8, x9, x8
+; CHECK-NEXT:    str x8, [x0]
+; CHECK-NEXT:    ret
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 8, i1 0)
+  ret void
+}
+
+define void @regular_memset_calls_external_function(i8* %a, i8 %value) nounwind {
+; CHECK-LABEL: regular_memset_calls_external_function:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov w2, #1024
+; CHECK-NEXT:    b memset
+  tail call void @llvm.memset.p0i8.i64(i8* %a, i8 %value, i64 1024, i1 0)
+  ret void
+}
+
+define void @inlined_set_doesnt_call_external_function(i8* %a, i8 %value) nounwind {
+; CHECK-LABEL: inlined_set_doesnt_call_external_function:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.16b, w1
+; CHECK-NEXT:    stp q0, q0, [x0]
+; CHECK-NEXT:    stp q0, q0, [x0, #32]
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 1024, i1 0)
+  ret void
+}

llvm/test/CodeGen/X86/memset-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=sse2,-sse4.2 | FileCheck %s --check-prefixes=GPR,SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=sse4.2,-avx  | FileCheck %s --check-prefixes=GPR,SSE4
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=avx,-avx512f | FileCheck %s --check-prefixes=GPR,AVX
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=avx512f      | FileCheck %s --check-prefixes=GPR,AVX512
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+declare void @llvm.memset.inline.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @memset_1(i8* %a, i8 %value) nounwind {
+; GPR-LABEL: memset_1:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movb %sil, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 1, i1 0)
+  ret void
+}
+
+define void @memset_2(i8* %a, i8 %value) nounwind {
+; GPR-LABEL: memset_2:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movzbl %sil, %eax
+; GPR-NEXT:    shll $8, %esi
+; GPR-NEXT:    orl %esi, %eax
+; GPR-NEXT:    movw %ax, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 2, i1 0)
+  ret void
+}
+
+define void @memset_4(i8* %a, i8 %value) nounwind {
+; GPR-LABEL: memset_4:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movzbl %sil, %eax
+; GPR-NEXT:    imull $16843009, %eax, %eax # imm = 0x1010101
+; GPR-NEXT:    movl %eax, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 4, i1 0)
+  ret void
+}
+
+define void @memset_8(i8* %a, i8 %value) nounwind {
+; GPR-LABEL: memset_8:
+; GPR:       # %bb.0:
+; GPR-NEXT:    # kill: def $esi killed $esi def $rsi
+; GPR-NEXT:    movzbl %sil, %eax
+; GPR-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
+; GPR-NEXT:    imulq %rax, %rcx
+; GPR-NEXT:    movq %rcx, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 8, i1 0)
+  ret void
+}
+
+define void @memset_16(i8* %a, i8 %value) nounwind {
+; SSE2-LABEL: memset_16:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    # kill: def $esi killed $esi def $rsi
+; SSE2-NEXT:    movzbl %sil, %eax
+; SSE2-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
+; SSE2-NEXT:    imulq %rax, %rcx
+; SSE2-NEXT:    movq %rcx, 8(%rdi)
+; SSE2-NEXT:    movq %rcx, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: memset_16:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    movd %esi, %xmm0
+; SSE4-NEXT:    pxor %xmm1, %xmm1
+; SSE4-NEXT:    pshufb %xmm1, %xmm0
+; SSE4-NEXT:    movdqu %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: memset_16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovd %esi, %xmm0
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vmovdqu %xmm0, (%rdi)
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: memset_16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %esi, %xmm0
+; AVX512-NEXT:    vpbroadcastb %xmm0, %xmm0
+; AVX512-NEXT:    vmovdqu %xmm0, (%rdi)
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 16, i1 0)
+  ret void
+}
+
+define void @memset_32(i8* %a, i8 %value) nounwind {
+; SSE2-LABEL: memset_32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    # kill: def $esi killed $esi def $rsi
+; SSE2-NEXT:    movzbl %sil, %eax
+; SSE2-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
+; SSE2-NEXT:    imulq %rax, %rcx
+; SSE2-NEXT:    movq %rcx, 24(%rdi)
+; SSE2-NEXT:    movq %rcx, 16(%rdi)
+; SSE2-NEXT:    movq %rcx, 8(%rdi)
+; SSE2-NEXT:    movq %rcx, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: memset_32:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    movd %esi, %xmm0
+; SSE4-NEXT:    pxor %xmm1, %xmm1
+; SSE4-NEXT:    pshufb %xmm1, %xmm0
+; SSE4-NEXT:    movdqu %xmm0, 16(%rdi)
+; SSE4-NEXT:    movdqu %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: memset_32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovd %esi, %xmm0
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vmovdqu %xmm0, 16(%rdi)
+; AVX-NEXT:    vmovdqu %xmm0, (%rdi)
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: memset_32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %esi, %xmm0
+; AVX512-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX512-NEXT:    vmovdqu %ymm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 32, i1 0)
+  ret void
+}
+
+define void @memset_64(i8* %a, i8 %value) nounwind {
+; SSE2-LABEL: memset_64:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    # kill: def $esi killed $esi def $rsi
+; SSE2-NEXT:    movzbl %sil, %eax
+; SSE2-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
+; SSE2-NEXT:    imulq %rax, %rcx
+; SSE2-NEXT:    movq %rcx, 56(%rdi)
+; SSE2-NEXT:    movq %rcx, 48(%rdi)
+; SSE2-NEXT:    movq %rcx, 40(%rdi)
+; SSE2-NEXT:    movq %rcx, 32(%rdi)
+; SSE2-NEXT:    movq %rcx, 24(%rdi)
+; SSE2-NEXT:    movq %rcx, 16(%rdi)
+; SSE2-NEXT:    movq %rcx, 8(%rdi)
+; SSE2-NEXT:    movq %rcx, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: memset_64:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    movd %esi, %xmm0
+; SSE4-NEXT:    pxor %xmm1, %xmm1
+; SSE4-NEXT:    pshufb %xmm1, %xmm0
+; SSE4-NEXT:    movdqu %xmm0, 48(%rdi)
+; SSE4-NEXT:    movdqu %xmm0, 32(%rdi)
+; SSE4-NEXT:    movdqu %xmm0, 16(%rdi)
+; SSE4-NEXT:    movdqu %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: memset_64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovd %esi, %xmm0
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX-NEXT:    vmovups %ymm0, 32(%rdi)
+; AVX-NEXT:    vmovups %ymm0, (%rdi)
+; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: memset_64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movzbl %sil, %eax
+; AVX512-NEXT:    imull $16843009, %eax, %eax # imm = 0x1010101
+; AVX512-NEXT:    vpbroadcastd %eax, %zmm0
+; AVX512-NEXT:    vmovdqu64 %zmm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 64, i1 0)
+  ret void
+}
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @aligned_memset_16(i8* align 16 %a, i8 %value) nounwind {
+; SSE2-LABEL: aligned_memset_16:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movd %esi, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: aligned_memset_16:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    movd %esi, %xmm0
+; SSE4-NEXT:    pxor %xmm1, %xmm1
+; SSE4-NEXT:    pshufb %xmm1, %xmm0
+; SSE4-NEXT:    movdqa %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: aligned_memset_16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovd %esi, %xmm0
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, (%rdi)
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: aligned_memset_16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %esi, %xmm0
+; AVX512-NEXT:    vpbroadcastb %xmm0, %xmm0
+; AVX512-NEXT:    vmovdqa %xmm0, (%rdi)
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 16 %a, i8 %value, i64 16, i1 0)
+  ret void
+}
+
+define void @aligned_memset_32(i8* align 32 %a, i8 %value) nounwind {
+; SSE2-LABEL: aligned_memset_32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movd %esi, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, 16(%rdi)
+; SSE2-NEXT:    movdqa %xmm0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: aligned_memset_32:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    movd %esi, %xmm0
+; SSE4-NEXT:    pxor %xmm1, %xmm1
+; SSE4-NEXT:    pshufb %xmm1, %xmm0
+; SSE4-NEXT:    movdqa %xmm0, 16(%rdi)
+; SSE4-NEXT:    movdqa %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: aligned_memset_32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovd %esi, %xmm0
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vmovdqa %xmm0, 16(%rdi)
+; AVX-NEXT:    vmovdqa %xmm0, (%rdi)
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: aligned_memset_32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd %esi, %xmm0
+; AVX512-NEXT:    vpbroadcastb %xmm0, %ymm0
+; AVX512-NEXT:    vmovdqa %ymm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 32 %a, i8 %value, i64 32, i1 0)
+  ret void
+}
+
+define void @aligned_memset_64(i8* align 64 %a, i8 %value) nounwind {
+; SSE2-LABEL: aligned_memset_64:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movd %esi, %xmm0
+; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,0,0,0,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,0,0]
+; SSE2-NEXT:    movdqa %xmm0, 48(%rdi)
+; SSE2-NEXT:    movdqa %xmm0, 32(%rdi)
+; SSE2-NEXT:    movdqa %xmm0, 16(%rdi)
+; SSE2-NEXT:    movdqa %xmm0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: aligned_memset_64:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    movd %esi, %xmm0
+; SSE4-NEXT:    pxor %xmm1, %xmm1
+; SSE4-NEXT:    pshufb %xmm1, %xmm0
+; SSE4-NEXT:    movdqa %xmm0, 48(%rdi)
+; SSE4-NEXT:    movdqa %xmm0, 32(%rdi)
+; SSE4-NEXT:    movdqa %xmm0, 16(%rdi)
+; SSE4-NEXT:    movdqa %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: aligned_memset_64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovd %esi, %xmm0
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpshufb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; AVX-NEXT:    vmovaps %ymm0, 32(%rdi)
+; AVX-NEXT:    vmovaps %ymm0, (%rdi)
+; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: aligned_memset_64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movzbl %sil, %eax
+; AVX512-NEXT:    imull $16843009, %eax, %eax # imm = 0x1010101
+; AVX512-NEXT:    vpbroadcastd %eax, %zmm0
+; AVX512-NEXT:    vmovdqa64 %zmm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 64 %a, i8 %value, i64 64, i1 0)
+  ret void
+}
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @bzero_1(i8* %a) nounwind {
+; GPR-LABEL: bzero_1:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movb $0, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 1, i1 0)
+  ret void
+}
+
+define void @bzero_2(i8* %a) nounwind {
+; GPR-LABEL: bzero_2:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movw $0, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 2, i1 0)
+  ret void
+}
+
+define void @bzero_4(i8* %a) nounwind {
+; GPR-LABEL: bzero_4:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movl $0, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 4, i1 0)
+  ret void
+}
+
+define void @bzero_8(i8* %a) nounwind {
+; GPR-LABEL: bzero_8:
+; GPR:       # %bb.0:
+; GPR-NEXT:    movq $0, (%rdi)
+; GPR-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 8, i1 0)
+  ret void
+}
+
+define void @bzero_16(i8* %a) nounwind {
+; SSE2-LABEL: bzero_16:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movq $0, 8(%rdi)
+; SSE2-NEXT:    movq $0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: bzero_16:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    xorps %xmm0, %xmm0
+; SSE4-NEXT:    movups %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: bzero_16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    vmovups %xmm0, (%rdi)
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: bzero_16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmovups %xmm0, (%rdi)
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 16, i1 0)
+  ret void
+}
+
+define void @bzero_32(i8* %a) nounwind {
+; SSE2-LABEL: bzero_32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movq $0, 24(%rdi)
+; SSE2-NEXT:    movq $0, 16(%rdi)
+; SSE2-NEXT:    movq $0, 8(%rdi)
+; SSE2-NEXT:    movq $0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: bzero_32:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    xorps %xmm0, %xmm0
+; SSE4-NEXT:    movups %xmm0, 16(%rdi)
+; SSE4-NEXT:    movups %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: bzero_32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    vmovups %ymm0, (%rdi)
+; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: bzero_32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmovups %ymm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 32, i1 0)
+  ret void
+}
+
+define void @bzero_64(i8* %a) nounwind {
+; SSE2-LABEL: bzero_64:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movq $0, 56(%rdi)
+; SSE2-NEXT:    movq $0, 48(%rdi)
+; SSE2-NEXT:    movq $0, 40(%rdi)
+; SSE2-NEXT:    movq $0, 32(%rdi)
+; SSE2-NEXT:    movq $0, 24(%rdi)
+; SSE2-NEXT:    movq $0, 16(%rdi)
+; SSE2-NEXT:    movq $0, 8(%rdi)
+; SSE2-NEXT:    movq $0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: bzero_64:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    xorps %xmm0, %xmm0
+; SSE4-NEXT:    movups %xmm0, 48(%rdi)
+; SSE4-NEXT:    movups %xmm0, 32(%rdi)
+; SSE4-NEXT:    movups %xmm0, 16(%rdi)
+; SSE4-NEXT:    movups %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: bzero_64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    vmovups %ymm0, 32(%rdi)
+; AVX-NEXT:    vmovups %ymm0, (%rdi)
+; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: bzero_64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmovups %zmm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 0, i64 64, i1 0)
+  ret void
+}
+
+; /////////////////////////////////////////////////////////////////////////////
+
+define void @aligned_bzero_16(i8* %a) nounwind {
+; SSE2-LABEL: aligned_bzero_16:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    xorps %xmm0, %xmm0
+; SSE2-NEXT:    movaps %xmm0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: aligned_bzero_16:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    xorps %xmm0, %xmm0
+; SSE4-NEXT:    movaps %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: aligned_bzero_16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    vmovaps %xmm0, (%rdi)
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: aligned_bzero_16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmovaps %xmm0, (%rdi)
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 16 %a, i8 0, i64 16, i1 0)
+  ret void
+}
+
+define void @aligned_bzero_32(i8* %a) nounwind {
+; SSE2-LABEL: aligned_bzero_32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    xorps %xmm0, %xmm0
+; SSE2-NEXT:    movaps %xmm0, 16(%rdi)
+; SSE2-NEXT:    movaps %xmm0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: aligned_bzero_32:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    xorps %xmm0, %xmm0
+; SSE4-NEXT:    movaps %xmm0, 16(%rdi)
+; SSE4-NEXT:    movaps %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: aligned_bzero_32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    vmovaps %ymm0, (%rdi)
+; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: aligned_bzero_32:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmovaps %ymm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 32 %a, i8 0, i64 32, i1 0)
+  ret void
+}
+
+define void @aligned_bzero_64(i8* %a) nounwind {
+; SSE2-LABEL: aligned_bzero_64:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    xorps %xmm0, %xmm0
+; SSE2-NEXT:    movaps %xmm0, 48(%rdi)
+; SSE2-NEXT:    movaps %xmm0, 32(%rdi)
+; SSE2-NEXT:    movaps %xmm0, 16(%rdi)
+; SSE2-NEXT:    movaps %xmm0, (%rdi)
+; SSE2-NEXT:    retq
+;
+; SSE4-LABEL: aligned_bzero_64:
+; SSE4:       # %bb.0:
+; SSE4-NEXT:    xorps %xmm0, %xmm0
+; SSE4-NEXT:    movaps %xmm0, 48(%rdi)
+; SSE4-NEXT:    movaps %xmm0, 32(%rdi)
+; SSE4-NEXT:    movaps %xmm0, 16(%rdi)
+; SSE4-NEXT:    movaps %xmm0, (%rdi)
+; SSE4-NEXT:    retq
+;
+; AVX-LABEL: aligned_bzero_64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX-NEXT:    vmovaps %ymm0, 32(%rdi)
+; AVX-NEXT:    vmovaps %ymm0, (%rdi)
+; AVX-NEXT:    vzeroupper
+; AVX-NEXT:    retq
+;
+; AVX512-LABEL: aligned_bzero_64:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AVX512-NEXT:    vmovaps %zmm0, (%rdi)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* align 64 %a, i8 0, i64 64, i1 0)
+  ret void
+}

llvm/test/CodeGen/X86/memset-vs-memset-inline.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=core2 | FileCheck %s
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+declare void @llvm.memset.inline.p0i8.i64(i8* nocapture, i8, i64, i1) nounwind
+
+define void @test1(i8* %a, i8 %value) nounwind {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT:    movzbl %sil, %eax
+; CHECK-NEXT:    movabsq $72340172838076673, %rcx # imm = 0x101010101010101
+; CHECK-NEXT:    imulq %rax, %rcx
+; CHECK-NEXT:    movq %rcx, (%rdi)
+; CHECK-NEXT:    retq
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 8, i1 0)
+  ret void
+}
+
+define void @regular_memset_calls_external_function(i8* %a, i8 %value) nounwind {
+; CHECK-LABEL: regular_memset_calls_external_function:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movl $1024, %edx # imm = 0x400
+; CHECK-NEXT:    jmp memset@PLT # TAILCALL
+  tail call void @llvm.memset.p0i8.i64(i8* %a, i8 %value, i64 1024, i1 0)
+  ret void
+}
+
+define void @inlined_set_doesnt_call_external_function(i8* %a, i8 %value) nounwind {
+; CHECK-LABEL: inlined_set_doesnt_call_external_function:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $esi killed $esi def $rsi
+; CHECK-NEXT:    movzbl %sil, %ecx
+; CHECK-NEXT:    movabsq $72340172838076673, %rax # imm = 0x101010101010101
+; CHECK-NEXT:    imulq %rcx, %rax
+; CHECK-NEXT:    movq %rax, 1016(%rdi)
+; CHECK-NEXT:    movq %rax, 1008(%rdi)
+  tail call void @llvm.memset.inline.p0i8.i64(i8* %a, i8 %value, i64 1024, i1 0)
+  ret void
+}

llvm/test/Other/lint.ll

 ; RUN: opt -basic-aa -lint -disable-output < %s 2>&1 | FileCheck %s
 ; RUN: opt -aa-pipeline=basic-aa -passes=lint -disable-output < %s 2>&1 | FileCheck %s
 target datalayout = "e-p:64:64:64"
 
 declare fastcc void @bar()
 declare void @llvm.stackrestore(i8*)
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind
 declare void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind
+declare void @llvm.memset.p0i8.i8.i64(i8* nocapture, i8, i64, i1) nounwind
+declare void @llvm.memset.inline.p0i8.i8.i64(i8* nocapture, i8, i64, i1) nounwind
 declare void @has_sret(i8* sret(i8) %p)
 declare void @has_noaliases(i32* noalias %p, i32* %q)
 declare void @one_arg(i32)
 
 @CG = constant i32 7
 @CG2 = constant i32 7
 @E = external global i8
 

 
 ; CHECK: Write to read-only memory
 call void @llvm.memcpy.p0i8.p0i8.i64(i8* bitcast (i32* @CG to i8*), i8* bitcast (i32* @CG2 to i8*), i64 1, i1 0)
 ; CHECK: Write to read-only memory
 call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* bitcast (i32* @CG to i8*), i8* bitcast (i32* @CG2 to i8*), i64 1, i1 0)
 ; CHECK: Unusual: noalias argument aliases another argument
 call void @llvm.memcpy.p0i8.p0i8.i64(i8* bitcast (i32* @CG to i8*), i8* bitcast (i32* @CG to i8*), i64 1, i1 0)
 
+; CHECK: Write to read-only memory
+call void @llvm.memset.p0i8.i8.i64(i8* bitcast (i32* @CG to i8*), i8 1, i64 1, i1 0)
+; CHECK: Write to read-only memory
+call void @llvm.memset.inline.p0i8.i8.i64(i8* bitcast (i32* @CG to i8*), i8 1, i64 1, i1 0)
+
 ; CHECK: Undefined behavior: Buffer overflow
   %wider = bitcast i8* %buf to i16*
   store i16 0, i16* %wider
 ; CHECK: Undefined behavior: Buffer overflow
   %inner = getelementptr {i8, i8}, {i8, i8}* %buf2, i32 0, i32 1
   %wider2 = bitcast i8* %inner to i16*
   store i16 0, i16* %wider2
 ; CHECK: Undefined behavior: Buffer overflow

llvm/test/Verifier/intrinsic-immarg.ll

 define void @memset(i8* %dest, i8 %val, i1 %is.volatile) {
   ; CHECK: immarg operand has non-immediate parameter
   ; CHECK-NEXT: i1 %is.volatile
   ; CHECK-NEXT: call void @llvm.memset.p0i8.i32(i8* %dest, i8 %val, i32 8, i1 %is.volatile)
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 %val, i32 8, i1 %is.volatile)
   ret void
 }
 
+declare void @llvm.memset.inline.p0i8.i32(i8* nocapture, i8, i32, i1)
+define void @memset_inline_is_volatile(i8* %dest, i8 %value, i1 %is.volatile) {
+  ; CHECK: immarg operand has non-immediate parameter
+  ; CHECK-NEXT: i1 %is.volatile
+  ; CHECK-NEXT: call void @llvm.memset.inline.p0i8.i32(i8* %dest, i8 %value, i32 8, i1 %is.volatile)
+  call void @llvm.memset.inline.p0i8.i32(i8* %dest, i8 %value, i32 8, i1 %is.volatile)
+  ret void
+}
+
+define void @memset_inline_variable_size(i8* %dest, i8 %value, i32 %size) {
+  ; CHECK: immarg operand has non-immediate parameter
+  ; CHECK-NEXT: i32 %size
+  ; CHECK-NEXT: call void @llvm.memset.inline.p0i8.i32(i8* %dest, i8 %value, i32 %size, i1 true)
+  call void @llvm.memset.inline.p0i8.i32(i8* %dest, i8 %value, i32 %size, i1 true)
+  ret void
+}
+
 
 declare i64 @llvm.objectsize.i64.p0i8(i8*, i1, i1, i1)
 define void @objectsize(i8* %ptr, i1 %a, i1 %b, i1 %c) {
   ; CHECK: immarg operand has non-immediate parameter
   ; CHECK-NEXT: i1 %a
   ; CHECK-NEXT: %val0 = call i64 @llvm.objectsize.i64.p0i8(i8* %ptr, i1 %a, i1 false, i1 false)
   %val0 = call i64 @llvm.objectsize.i64.p0i8(i8* %ptr, i1 %a, i1 false, i1 false)
 

llvm/test/Verifier/memset-inline.ll

+; RUN: not opt -verify < %s 2>&1 | FileCheck %s
+
+; CHECK: alignment is not a power of two 
+
+define void @foo(i8* %P, i8 %value) {
+  call void @llvm.memset.inline.p0i8.i32(i8* align 3 %P, i8 %value, i32 4, i1 false)
+  ret void
+}
+declare void @llvm.memset.inline.p0i8.i32(i8* nocapture, i8, i32, i1) nounwind