[SCEV] Fix and validate ValueExprMap/ExprValueMap consistency

[nikic]

Summary

This adds validation for consistency of ValueExprMap and ExprValueMap, and fixes identified issues:

• Addrec construction directly wrote to ValueExprMap in a few places, without updating ExprValueMap. Add a helper to ensures they stay consistent. The adjustment in forgetSymbolicName() explicitly drops the old value from the map, so that we don't rely on it being overwritten.
• forgetMemoizedResultsImpl() was dropping the SCEV from ExprValueMap, but not dropping the corresponding entries from ValueExprMap.

[nikic]

llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll

1

This test changes because more instructions get reused by SCEV expander.

[mkazantsev]

Can we split out refactoring + verifier changes (that expose some failues) and fixes for those failures? I'd like to see the tests and situations where they fail to make sure these functional fixes really address the problems. If those failures aren't too numerous or disrupting, please do.

[nikic]

In D113349#3114647, @mkazantsev wrote:

Can we split out refactoring + verifier changes (that expose some failues) and fixes for those failures? I'd like to see the tests and situations where they fail to make sure these functional fixes really address the problems. If those failures aren't too numerous or disrupting, please do.

The new verification makes pretty much all tests that verify SCEV fail. (This is a surprisingly low number, basically only LCSSA, LoopFuse and LoopUnrollAndJam tests, which use either verify<scalar-evolution> or explicitly call SE.verify(). Apparently the -verify-scev option doesn't do anything under the new pass manager...)

[nikic]

It may be worth mentioning that currently mismatches in these maps aren't a correctness problem -- worst that will happen is that SCEVExpander misses a CSE opportunity. However, I want to use these maps in a way where correctness is important in the future, thus the validation now.

[nikic]

ping

[nikic]

rebase

[mkazantsev]

lg!

[nikic]

Reduction from asan bootstrap for the assertion failure (opt -loop-reduce):

define void @test() {
entry:
  br label %loop.header

loop.header:
  %ptr = phi i64* [ %ptr.next, %loop.latch ], [ null, %entry ]
  br i1 true, label %loop.exit, label %loop2.preheader

loop.exit:
  ret void

loop2.preheader:
  br label %loop2.header

loop2.header:
  %ptr2 = phi i64* [ %ptr, %loop2.preheader ], [ %ptr2.next, %loop2.header ]
  %ptr2.next = getelementptr inbounds i64, i64* %ptr2, i64 1
  br i1 false, label %loop2.header, label %loop2.cont

loop2.cont:
  %ptr2.i8 = bitcast i64* %ptr2 to i8*
  %v = load i8, i8* %ptr2.i8
  %c = icmp ne i8 %v, 0
  br i1 %c, label %loop.exit, label %loop.latch

loop.latch:
  %ptr.next = getelementptr inbounds i64, i64* %ptr, i64 1
  br label %loop.header
}

[sberg]

I commented inline at https://reviews.llvm.org/rG2b160e95c89f28a30e2481e6131073659d866a8c#1040993 "Reland [SCEV] Fix and validate ValueExprMap/ExprValueMap consistency":

I see this assert [in ScalarEvolution::insertValueToMap at llvm/lib/Analysis/ScalarEvolution.cpp:4105] trigger for

$ cat test.cc
struct S {
    virtual void f1();
    void f2(int *);
    int * a1;
    int n1;
};
void S::f2(int * a2) {
    int n2 = 0;
    for (int i = 0; i != 2; ++i) {
        int n3 = n2;
        for (; n2 < n1; ++n2) {
            if (a1[n2]) {
                break;
            }
        }
        *a2 = 0;
        for (; n2 < n1; ++n2) {
            a1[n2];
        }
    }
}

$ clang++ -fsanitize=undefined -O -c test.cc
# Driver args: "--driver-mode=g++" "-fsanitize=undefined" "-O" "-c" "test.cc"
# Original command:  "~/llvm/inst/bin/clang-14" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "--mrelax-relocations" "-disable-free" "-clear-ast-before-backend" "-main-file-name" "test.cc" "-mrelocation-model" "static" "-mframe-pointer=none" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=~/lo/core" "-resource-dir" "~/llvm/inst/lib/clang/14.0.0" "-internal-isystem" "/usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11" "-internal-isystem" "/usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/x86_64-redhat-linux" "-internal-isystem" "/usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/backward" "-internal-isystem" "~/llvm/inst/lib/clang/14.0.0/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/gcc/x86_64-redhat-linux/11/../../../../x86_64-redhat-linux/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-O1" "-fdeprecated-macro" "-fdebug-compilation-dir=~/lo/core" "-ferror-limit" "19" "-fsanitize=alignment,array-bounds,bool,builtin,enum,float-cast-overflow,function,integer-divide-by-zero,nonnull-attribute,null,object-size,pointer-overflow,return,returns-nonnull-attribute,shift-base,shift-exponent,signed-integer-overflow,unreachable,vla-bound,vptr" "-fsanitize-recover=alignment,array-bounds,bool,builtin,enum,float-cast-overflow,function,integer-divide-by-zero,nonnull-attribute,null,object-size,pointer-overflow,returns-nonnull-attribute,shift-base,shift-exponent,signed-integer-overflow,vla-bound,vptr" "-fgnuc-version=4.2.1" "-fcxx-exceptions" "-fexceptions" "-fcolor-diagnostics" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "test.o" "-x" "c++" "test.cc"
 "~/llvm/inst/bin/clang-14" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "--mrelax-relocations" "-disable-free" "-clear-ast-before-backend" "-main-file-name" "test.cc" "-mrelocation-model" "static" "-mframe-pointer=none" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=~/lo/core" "-O1" "-fdeprecated-macro" "-fdebug-compilation-dir=~/lo/core" "-ferror-limit" "19" "-fsanitize=alignment,array-bounds,bool,builtin,enum,float-cast-overflow,function,integer-divide-by-zero,nonnull-attribute,null,object-size,pointer-overflow,return,returns-nonnull-attribute,shift-base,shift-exponent,signed-integer-overflow,unreachable,vla-bound,vptr" "-fsanitize-recover=alignment,array-bounds,bool,builtin,enum,float-cast-overflow,function,integer-divide-by-zero,nonnull-attribute,null,object-size,pointer-overflow,returns-nonnull-attribute,shift-base,shift-exponent,signed-integer-overflow,vla-bound,vptr" "-fgnuc-version=4.2.1" "-fcxx-exceptions" "-fexceptions" "-fcolor-diagnostics" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-x" "c++" "test-580877.cpp"
test.cc:18:18: warning: expression result unused [-Wunused-value]
            a1[n2];
            ~~ ~~^
clang++: ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4105: void llvm::ScalarEvolution::insertValueToMap(llvm::Value *, const llvm::SCEV *): Assertion `It->second == S' failed.
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: ~/llvm/inst/bin/clang++ -fsanitize=undefined -O -c test.cc
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'Function Pass Manager' on module 'test.cc'.
4.	Running pass 'Loop Pass Manager' on function '@_ZN1S2f2EPi'
5.	Running pass 'Canonicalize Freeze Instructions in Loops' on basic block '%cont4'
 #0 0x00000000029923a3 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) ~/github.com/llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:565:13
 #1 0x0000000002990580 llvm::sys::RunSignalHandlers() ~/github.com/llvm/llvm-project/llvm/lib/Support/Signals.cpp:98:18
 #2 0x000000000290cc33 (anonymous namespace)::CrashRecoveryContextImpl::HandleCrash(int, unsigned long) ~/github.com/llvm/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:76:5
 #3 0x000000000290cdf8 CrashRecoverySignalHandler(int) ~/github.com/llvm/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:390:1
 #4 0x00007f71e2b50760 __restore_rt (/lib64/libc.so.6+0x42760)
 #5 0x00007f71e2b9d85c __pthread_kill_implementation (/lib64/libc.so.6+0x8f85c)
 #6 0x00007f71e2b506b6 gsignal (/lib64/libc.so.6+0x426b6)
 #7 0x00007f71e2b3a7d3 abort (/lib64/libc.so.6+0x2c7d3)
 #8 0x00007f71e2b3a6fb _nl_load_domain.cold (/lib64/libc.so.6+0x2c6fb)
 #9 0x00007f71e2b493a6 (/lib64/libc.so.6+0x3b3a6)
#10 0x0000000001c0ad70 llvm::CallbackVH::CallbackVH(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/include/llvm/IR/ValueHandle.h:396:55
#11 0x0000000001c0ad70 llvm::ScalarEvolution::SCEVCallbackVH::SCEVCallbackVH(llvm::Value*, llvm::ScalarEvolution*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:12428:5
#12 0x0000000001c0ad70 llvm::ScalarEvolution::insertValueToMap(llvm::Value*, llvm::SCEV const*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4100:26
#13 0x0000000001c0f1ed llvm::ScalarEvolution::createSimpleAffineAddRec(llvm::PHINode*, llvm::Value*, llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:3
#14 0x0000000001c10040 llvm::ScalarEvolution::createAddRecFromPHI(llvm::PHINode*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:17
#15 0x0000000001c11614 llvm::ScalarEvolution::createNodeForPHI(llvm::PHINode*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:5638:19
#16 0x0000000001c0c472 llvm::ScalarEvolution::createSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:12
#17 0x0000000001c04b78 llvm::ScalarEvolution::getSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4116:9
#18 0x0000000001c0c4d7 llvm::Value::getType() const ~/github.com/llvm/llvm-project/llvm/include/llvm/IR/Value.h:255:34
#19 0x0000000001c0c4d7 llvm::ScalarEvolution::createSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:7105:58
#20 0x0000000001c04b78 llvm::ScalarEvolution::getSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4116:9
#21 0x0000000001c1007a llvm::isa_impl_cl<llvm::SCEVAddExpr, llvm::SCEV const*>::doit(llvm::SCEV const*) ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Casting.h:104:5
#22 0x0000000001c1007a llvm::isa_impl_wrap<llvm::SCEVAddExpr, llvm::SCEV const*, llvm::SCEV const*>::doit(llvm::SCEV const* const&) ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Casting.h:131:12
#23 0x0000000001c1007a llvm::isa_impl_wrap<llvm::SCEVAddExpr, llvm::SCEV const* const, llvm::SCEV const*>::doit(llvm::SCEV const* const&) ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Casting.h:121:12
#24 0x0000000001c1007a bool llvm::isa<llvm::SCEVAddExpr, llvm::SCEV const*>(llvm::SCEV const* const&) ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Casting.h:142:10
#25 0x0000000001c1007a llvm::cast_retty<llvm::SCEVAddExpr, llvm::SCEV const*>::ret_type llvm::dyn_cast<llvm::SCEVAddExpr, llvm::SCEV const>(llvm::SCEV const*) ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Casting.h:345:10
#26 0x0000000001c1007a llvm::ScalarEvolution::createAddRecFromPHI(llvm::PHINode*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:5375:32
#27 0x0000000001c11614 llvm::ScalarEvolution::createNodeForPHI(llvm::PHINode*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:5638:19
#28 0x0000000001c0c472 llvm::ScalarEvolution::createSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:12
#29 0x0000000001c04b78 llvm::ScalarEvolution::getSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4116:9
#30 0x0000000001c0f1b8 llvm::ScalarEvolution::createSimpleAffineAddRec(llvm::PHINode*, llvm::Value*, llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:26
#31 0x0000000001c10040 llvm::ScalarEvolution::createAddRecFromPHI(llvm::PHINode*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:17
#32 0x0000000001c11614 llvm::ScalarEvolution::createNodeForPHI(llvm::PHINode*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:5638:19
#33 0x0000000001c0c472 llvm::ScalarEvolution::createSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:0:12
#34 0x0000000001c04b78 llvm::ScalarEvolution::getSCEV(llvm::Value*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/ScalarEvolution.cpp:4116:9
#35 0x0000000001cca7d4 llvm::InductionDescriptor::isInductionPHI(llvm::PHINode*, llvm::Loop const*, llvm::ScalarEvolution*, llvm::InductionDescriptor&, llvm::SCEV const*, llvm::SmallVectorImpl<llvm::Instruction*>*) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/IVDescriptors.cpp:0:43
#36 0x00000000029bdb90 (anonymous namespace)::CanonicalizeFreezeInLoopsImpl::run() ~/github.com/llvm/llvm-project/llvm/lib/Transforms/Utils/CanonicalizeFreezeInLoops.cpp:133:9
#37 0x00000000029bee0d (anonymous namespace)::CanonicalizeFreezeInLoops::runOnLoop(llvm::Loop*, llvm::LPPassManager&) ~/github.com/llvm/llvm-project/llvm/lib/Transforms/Utils/CanonicalizeFreezeInLoops.cpp:225:1
#38 0x0000000001b8431b llvm::LPPassManager::runOnFunction(llvm::Function&) ~/github.com/llvm/llvm-project/llvm/lib/Analysis/LoopPass.cpp:209:13
#39 0x00000000021f37e8 llvm::FPPassManager::runOnFunction(llvm::Function&) ~/github.com/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1439:27
#40 0x00000000021f9c91 llvm::FPPassManager::runOnModule(llvm::Module&) ~/github.com/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1485:13
#41 0x00000000021f3e78 (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) ~/github.com/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:0:27
#42 0x00000000021f3e78 llvm::legacy::PassManagerImpl::run(llvm::Module&) ~/github.com/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:542:44
#43 0x0000000002c0e6a7 (anonymous namespace)::EmitAssemblyHelper::RunCodegenPipeline(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >&, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile> >&) ~/github.com/llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1520:1
#44 0x0000000002c0e6a7 (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) ~/github.com/llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1548:3
#45 0x0000000002c0e6a7 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) ~/github.com/llvm/llvm-project/clang/lib/CodeGen/BackendUtil.cpp:1710:15
#46 0x000000000384bd1e std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >::~unique_ptr() /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:360:6
#47 0x000000000384bd1e clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) ~/github.com/llvm/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:370:7
#48 0x00000000042c80a4 __gnu_cxx::__normal_iterator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> >*, std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> >, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> > > > >::__normal_iterator(std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> >* const&) /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1011:20
#49 0x00000000042c80a4 std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> >, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> > > >::begin() /usr/bin/../lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_vector.h:812:16
#50 0x00000000042c80a4 void clang::finalize<std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> >, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> > > > >(std::vector<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> >, std::allocator<std::unique_ptr<clang::TemplateInstantiationCallback, std::default_delete<clang::TemplateInstantiationCallback> > > >&, clang::Sema const&) ~/github.com/llvm/llvm-project/clang/include/clang/Sema/TemplateInstCallback.h:54:16
#51 0x00000000042c80a4 clang::ParseAST(clang::Sema&, bool, bool) ~/github.com/llvm/llvm-project/clang/lib/Parse/ParseAST.cpp:178:3
#52 0x000000000320ac61 clang::FrontendAction::Execute() ~/github.com/llvm/llvm-project/clang/lib/Frontend/FrontendAction.cpp:971:10
#53 0x0000000003191d5f llvm::Error::getPtr() const ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Error.h:273:42
#54 0x0000000003191d5f llvm::Error::operator bool() ~/github.com/llvm/llvm-project/llvm/include/llvm/Support/Error.h:236:16
#55 0x0000000003191d5f clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) ~/github.com/llvm/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1030:23
#56 0x00000000032c7fa9 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) ~/github.com/llvm/llvm-project/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:261:25
#57 0x00000000009dfecb cc1_main(llvm::ArrayRef<char const*>, char const*, void*) ~/github.com/llvm/llvm-project/clang/tools/driver/cc1_main.cpp:246:15
#58 0x00000000009dd9ca ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) ~/github.com/llvm/llvm-project/clang/tools/driver/driver.cpp:317:12
#59 0x000000000306bc62 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool*) const::$_1::operator()() const ~/github.com/llvm/llvm-project/clang/lib/Driver/Job.cpp:405:30
#60 0x000000000306bc62 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool*) const::$_1>(long) ~/github.com/llvm/llvm-project/llvm/include/llvm/ADT/STLExtras.h:177:12
#61 0x000000000290cb47 llvm::function_ref<void ()>::operator()() const ~/github.com/llvm/llvm-project/llvm/include/llvm/ADT/STLExtras.h:0:12
#62 0x000000000290cb47 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) ~/github.com/llvm/llvm-project/llvm/lib/Support/CrashRecoveryContext.cpp:425:3
#63 0x000000000306b517 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool*) const ~/github.com/llvm/llvm-project/clang/lib/Driver/Job.cpp:405:7
#64 0x000000000303f763 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const ~/github.com/llvm/llvm-project/clang/lib/Driver/Compilation.cpp:196:15
#65 0x000000000303fae7 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) const ~/github.com/llvm/llvm-project/clang/lib/Driver/Compilation.cpp:249:13
#66 0x0000000003053f9e llvm::SmallVectorBase<unsigned int>::empty() const ~/github.com/llvm/llvm-project/llvm/include/llvm/ADT/SmallVector.h:73:47
#67 0x0000000003053f9e clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) ~/github.com/llvm/llvm-project/clang/lib/Driver/Driver.cpp:1555:23
#68 0x00000000009dd22d main ~/github.com/llvm/llvm-project/clang/tools/driver/driver.cpp:489:21
#69 0x00007f71e2b3b560 __libc_start_call_main (/lib64/libc.so.6+0x2d560)
#70 0x00007f71e2b3b60c __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2d60c)
#71 0x00000000009da9a5 _start (~/llvm/inst/bin/clang+++0x9da9a5)

with It->second->dump() being

{(sext i32 {%n2.067,+,1}<nsw><%cont4> to i64),+,1}<nsw><%cont24>

and S->dump() being

{{(sext i32 %n2.067 to i64),+,1}<nsw><%cont4>,+,1}<nsw><%cont24>

(There may be an easier reproducer not needing -fsanitize=undefined; I only noticed that I still had it on the command line late into coming up with a minimal reproducer.)

[nikic]

@sberg Thanks for the report! Here's a reduction for opt -passes='print<scalar-evolution>':

define void @test(i32* %p) {
entry:
  br label %loop.header

loop.header:
  %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop.latch ]
  br label %loop2

loop2:
  %iv2 = phi i32 [ %iv, %loop.header ], [ %iv2.next, %loop2 ]
  %iv2.next = add i32 %iv2, 1
  %v = load i32, i32* %p
  %cmp = icmp slt i32 %iv2, %v
  br i1 %cmp, label %loop2, label %loop2.end

loop2.end:
  %iv2.ext = sext i32 %iv2 to i64
  br label %loop3

loop3:
  %iv3 = phi i64 [ %iv2.ext, %loop2.end ], [ %iv3.next, %loop3 ]
  %iv3.next = add nsw i64 %iv3, 1
  br i1 false, label %loop3, label %loop.latch

loop.latch:
  %iv.next = trunc i64 %iv3 to i32
  br label %loop.header
}

I'll probably just drop the assert -- while it should be the same in an ideal world, lazy nowrap flag updates mean that it can be a different (but equivalent) SCEV expression, as is the case here.

[nikic]

Assert dropped in https://github.com/llvm/llvm-project/commit/77dd579827f2e7574be4bbf3f94a48930e7b094f.

[uabelho]

I wrote
https://github.com/llvm/llvm-project/issues/53905
that started happening with this patch.

llvm/include/llvm/Analysis/ScalarEvolution.h

private:
   void forgetMemoizedResultsImpl(const SCEV *S);
 
   /// Return an existing SCEV for V if there is one, otherwise return nullptr.
   const SCEV *getExistingSCEV(Value *V);
 
   /// Erase Value from ValueExprMap and ExprValueMap.
   void eraseValueFromMap(Value *V);
 
+  /// Insert V to S mapping into ValueExprMap and ExprValueMap.
+  void insertValueToMap(Value *V, const SCEV *S);
+
   /// Return false iff given SCEV contains a SCEVUnknown with NULL value-
   /// pointer.
   bool checkValidity(const SCEV *S) const;
 
   /// Return true if `ExtendOpTy`({`Start`,+,`Step`}) can be proved to be
   /// equal to {`ExtendOpTy`(`Start`),+,`ExtendOpTy`(`Step`)}.  This is
   /// equivalent to proving no signed (resp. unsigned) wrap in
   /// {`Start`,+,`Step`} if `ExtendOpTy` is `SCEVSignExtendExpr`

llvm/lib/Analysis/ScalarEvolution.cpp

if (I != ValueExprMap.end()) {
     if (Offset != nullptr) {
       if (auto *SV = getSCEVValues(Stripped))
         SV->remove({V, Offset});
     }
     ValueExprMap.erase(V);
   }
 }
 
+void ScalarEvolution::insertValueToMap(Value *V, const SCEV *S) {
+  assert(ValueExprMap.find_as(V) == ValueExprMap.end() &&
+         "Must not be in value map yet");
+  ValueExprMap.insert({SCEVCallbackVH(V, this), S});
+  ExprValueMap[S].insert({V, nullptr});
+}
+
 /// Return an existing SCEV if it exists, otherwise analyze the expression and
 /// create a new one.
 const SCEV *ScalarEvolution::getSCEV(Value *V) {
   assert(isSCEVable(V->getType()) && "Value is not SCEVable!");
 
   const SCEV *S = getExistingSCEV(V);
   if (S == nullptr) {
     S = createSCEV(V);
if (It != ValueExprMap.end()) {
 
       // SCEVUnknown for a PHI either means that it has an unrecognized
       // structure, it's a PHI that's in the progress of being computed
       // by createNodeForPHI, or it's a single-value PHI. In the first case,
       // additional loop trip count information isn't going to change anything.
       // In the second case, createNodeForPHI will perform the necessary
       // updates on its own when it gets to that point. In the third, we do
       // want to forget the SCEVUnknown.
-      if (!isa<PHINode>(I) ||
-          !isa<SCEVUnknown>(Old) ||
-          (I != PN && Old == SymName)) {
+      if (!isa<PHINode>(I) || !isa<SCEVUnknown>(Old) || Old == SymName) {
         eraseValueFromMap(It->first);
         ToForget.push_back(Old);
       }
     }
 
     PushDefUseChildren(I, Worklist, Visited);
   }
   forgetMemoizedResults(ToForget);
const SCEV *ScalarEvolution::createSimpleAffineAddRec(PHINode *PN,
   if (BO->IsNUW)
     Flags = setFlags(Flags, SCEV::FlagNUW);
   if (BO->IsNSW)
     Flags = setFlags(Flags, SCEV::FlagNSW);
 
   const SCEV *StartVal = getSCEV(StartValueV);
   const SCEV *PHISCEV = getAddRecExpr(StartVal, Accum, L, Flags);
 
-  ValueExprMap[SCEVCallbackVH(PN, this)] = PHISCEV;
+  // The recursive queries above may have already computed the SCEV.
+  // However, it must be the same as the one computed here.
+  auto It = ValueExprMap.find_as(PN);
+  if (It != ValueExprMap.end())
+    assert(It->second == PHISCEV);
+  else
+    insertValueToMap(PN, PHISCEV);
 
   // We can add Flags to the post-inc expression only if we
   // know that it is *undefined behavior* for BEValueV to
   // overflow.
   if (auto *BEInst = dyn_cast<Instruction>(BEValueV))
     if (isLoopInvariant(Accum, L) && isAddRecNeverPoison(BEInst, L))
       (void)getAddRecExpr(getAddExpr(StartVal, Accum), Accum, L, Flags);
 
const SCEV *ScalarEvolution::createAddRecFromPHI(PHINode *PN) {
 
   // First, try to find AddRec expression without creating a fictituos symbolic
   // value for PN.
   if (auto *S = createSimpleAffineAddRec(PN, BEValueV, StartValueV))
     return S;
 
   // Handle PHI node value symbolically.
   const SCEV *SymbolicName = getUnknown(PN);
-  ValueExprMap.insert({SCEVCallbackVH(PN, this), SymbolicName});
+  insertValueToMap(PN, SymbolicName);
 
   // Using this symbolic name for the PHI, analyze the value coming around
   // the back-edge.
   const SCEV *BEValue = getSCEV(BEValueV);
 
   // NOTE: If BEValue is loop invariant, we know that the PHI node just
   // has a special value for the first iteration of the loop.
 
if (FoundIndex != Add->getNumOperands()) {
 
         const SCEV *StartVal = getSCEV(StartValueV);
         const SCEV *PHISCEV = getAddRecExpr(StartVal, Accum, L, Flags);
 
         // Okay, for the entire analysis of this edge we assumed the PHI
         // to be symbolic.  We now need to go back and purge all of the
         // entries for the scalars that use the symbolic expression.
         forgetSymbolicName(PN, SymbolicName);
-        ValueExprMap[SCEVCallbackVH(PN, this)] = PHISCEV;
+        insertValueToMap(PN, PHISCEV);
 
         // We can add Flags to the post-inc expression only if we
         // know that it is *undefined behavior* for BEValueV to
         // overflow.
         if (auto *BEInst = dyn_cast<Instruction>(BEValueV))
           if (isLoopInvariant(Accum, L) && isAddRecNeverPoison(BEInst, L))
             (void)getAddRecExpr(getAddExpr(StartVal, Accum), Accum, L, Flags);
 
if (const SCEVAddExpr *Add = dyn_cast<SCEVAddExpr>(BEValue)) {
     if (Shifted != getCouldNotCompute() &&
         Start != getCouldNotCompute()) {
       const SCEV *StartVal = getSCEV(StartValueV);
       if (Start == StartVal) {
         // Okay, for the entire analysis of this edge we assumed the PHI
         // to be symbolic.  We now need to go back and purge all of the
         // entries for the scalars that use the symbolic expression.
         forgetSymbolicName(PN, SymbolicName);
-        ValueExprMap[SCEVCallbackVH(PN, this)] = Shifted;
+        insertValueToMap(PN, Shifted);
         return Shifted;
       }
     }
   }
 
   // Remove the temporary PHI node SCEV that has been inserted while intending
   // to create an AddRecExpr for this PHI node. We can not keep this temporary
   // as it will prevent later (possibly simpler) SCEV expressions to be added

 }
 
 void ScalarEvolution::forgetMemoizedResultsImpl(const SCEV *S) {
   ValuesAtScopes.erase(S);
   LoopDispositions.erase(S);
   BlockDispositions.erase(S);
   UnsignedRanges.erase(S);
   SignedRanges.erase(S);
-  ExprValueMap.erase(S);
   HasRecMap.erase(S);
   MinTrailingZerosCache.erase(S);
+
+  auto ExprIt = ExprValueMap.find(S);
+  if (ExprIt != ExprValueMap.end()) {
+    for (auto &ValueAndOffset : ExprIt->second) {
+      if (ValueAndOffset.second == nullptr) {
+        auto ValueIt = ValueExprMap.find_as(ValueAndOffset.first);
+        if (ValueIt != ValueExprMap.end())
+          ValueExprMap.erase(ValueIt);
+      }
+    }
+    ExprValueMap.erase(ExprIt);
+  }
 }
 
 void
 ScalarEvolution::getUsedLoops(const SCEV *S,
                               SmallPtrSetImpl<const Loop *> &LoopsUsed) {
   struct FindUsedLoops {
     FindUsedLoops(SmallPtrSetImpl<const Loop *> &LoopsUsed)
         : LoopsUsed(LoopsUsed) {}
void ScalarEvolution::verify() const {
   SmallVector<Loop *, 32> Worklist(LI.begin(), LI.end());
   while (!Worklist.empty()) {
     Loop *L = Worklist.pop_back_val();
     if (ValidLoops.contains(L))
       continue;
     ValidLoops.insert(L);
     Worklist.append(L->begin(), L->end());
   }
-  // Check for SCEV expressions referencing invalid/deleted loops.
   for (auto &KV : ValueExprMap) {
-    auto *AR = dyn_cast<SCEVAddRecExpr>(KV.second);
-    if (!AR)
-      continue;
+    // Check for SCEV expressions referencing invalid/deleted loops.
+    if (auto *AR = dyn_cast<SCEVAddRecExpr>(KV.second)) {
       assert(ValidLoops.contains(AR->getLoop()) &&
              "AddRec references invalid loop");
     }
 
+    // Check that the value is also part of the reverse map.
+    auto It = ExprValueMap.find(KV.second);
+    if (It == ExprValueMap.end() || !It->second.contains({KV.first, nullptr})) {
+      dbgs() << "Value " << *KV.first
+             << " is in ValueExprMap but not in ExprValueMap\n";
+      std::abort();
+    }
+  }
+
+  for (const auto &KV : ExprValueMap) {
+    for (const auto &ValueAndOffset : KV.second) {
+      if (ValueAndOffset.second != nullptr)
+        continue;
+
+      auto It = ValueExprMap.find_as(ValueAndOffset.first);
+      if (It == ValueExprMap.end()) {
+        dbgs() << "Value " << *ValueAndOffset.first
+               << " is in ExprValueMap but not in ValueExprMap\n";
+        std::abort();
+      }
+      if (It->second != KV.first) {
+        dbgs() << "Value " << *ValueAndOffset.first
+               << " mapped to " << *It->second
+               << " rather than " << *KV.first << "\n";
+        std::abort();
+      }
+    }
+  }
+
   // Verify intergity of SCEV users.
   for (const auto &S : UniqueSCEVs) {
     SmallVector<const SCEV *, 4> Ops;
     collectUniqueOps(&S, Ops);
     for (const auto *Op : Ops) {
       // We do not store dependencies of constants.
       if (isa<SCEVConstant>(Op))
         continue;

llvm/test/CodeGen/PowerPC/more-dq-form-prepare.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mcpu=pwr10 < %s | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-p:64:64-n32:64-v256:256:256-v512:512:512"
 target triple = "powerpc64le-unknown-linux-gnu"
 
 %_elem_type_of_a = type <{ double }>
 %_elem_type_of_x = type <{ double }>
 
 define void @foo(i32* %.m, i32* %.n, [0 x %_elem_type_of_a]* %.a, [0 x %_elem_type_of_x]* %.x, i32* %.l, <2 x double>* %.vy01, <2 x double>* %.vy02, <2 x double>* %.vy03, <2 x double>* %.vy04, <2 x double>* %.vy05, <2 x double>* %.vy06, <2 x double>* %.vy07, <2 x double>* %.vy08, <2 x double>* %.vy09, <2 x double>* %.vy0a, <2 x double>* %.vy0b, <2 x double>* %.vy0c, <2 x double>* %.vy21, <2 x double>* %.vy22, <2 x double>* %.vy23, <2 x double>* %.vy24, <2 x double>* %.vy25, <2 x double>* %.vy26, <2 x double>* %.vy27, <2 x double>* %.vy28, <2 x double>* %.vy29, <2 x double>* %.vy2a, <2 x double>* %.vy2b, <2 x double>* %.vy2c) {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    stdu 1, -608(1)
-; CHECK-NEXT:    .cfi_def_cfa_offset 608
+; CHECK-NEXT:    stdu 1, -592(1)
+; CHECK-NEXT:    .cfi_def_cfa_offset 592
 ; CHECK-NEXT:    .cfi_offset r14, -192
 ; CHECK-NEXT:    .cfi_offset r15, -184
 ; CHECK-NEXT:    .cfi_offset r16, -176
 ; CHECK-NEXT:    .cfi_offset r17, -168
 ; CHECK-NEXT:    .cfi_offset r18, -160
 ; CHECK-NEXT:    .cfi_offset r19, -152
 ; CHECK-NEXT:    .cfi_offset r20, -144
 ; CHECK-NEXT:    .cfi_offset r21, -136

 ; CHECK-NEXT:    .cfi_offset v24, -320
 ; CHECK-NEXT:    .cfi_offset v25, -304
 ; CHECK-NEXT:    .cfi_offset v26, -288
 ; CHECK-NEXT:    .cfi_offset v27, -272
 ; CHECK-NEXT:    .cfi_offset v28, -256
 ; CHECK-NEXT:    .cfi_offset v29, -240
 ; CHECK-NEXT:    .cfi_offset v30, -224
 ; CHECK-NEXT:    .cfi_offset v31, -208
-; CHECK-NEXT:    lwz 0, 0(4)
-; CHECK-NEXT:    std 14, 416(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 15, 424(1) # 8-byte Folded Spill
-; CHECK-NEXT:    cmpwi 0, 1
-; CHECK-NEXT:    std 16, 432(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 17, 440(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 18, 448(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 19, 456(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 20, 464(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 21, 472(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 22, 480(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 23, 488(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 24, 496(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 25, 504(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 26, 512(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 27, 520(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 28, 528(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 29, 536(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 30, 544(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 31, 552(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stfd 26, 560(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stfd 27, 568(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stfd 28, 576(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stfd 29, 584(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stfd 30, 592(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stfd 31, 600(1) # 8-byte Folded Spill
-; CHECK-NEXT:    stxv 52, 224(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 53, 240(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 54, 256(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 55, 272(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 56, 288(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 57, 304(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 58, 320(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 59, 336(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 60, 352(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 61, 368(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 62, 384(1) # 16-byte Folded Spill
-; CHECK-NEXT:    stxv 63, 400(1) # 16-byte Folded Spill
+; CHECK-NEXT:    lwz 4, 0(4)
+; CHECK-NEXT:    std 14, 400(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 15, 408(1) # 8-byte Folded Spill
+; CHECK-NEXT:    cmpwi 4, 1
+; CHECK-NEXT:    std 16, 416(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 17, 424(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 18, 432(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 19, 440(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 20, 448(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 21, 456(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 22, 464(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 23, 472(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 24, 480(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 25, 488(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 26, 496(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 27, 504(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 28, 512(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 29, 520(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 30, 528(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 31, 536(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stfd 26, 544(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stfd 27, 552(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stfd 28, 560(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stfd 29, 568(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stfd 30, 576(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stfd 31, 584(1) # 8-byte Folded Spill
+; CHECK-NEXT:    stxv 52, 208(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 53, 224(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 54, 240(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 55, 256(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 56, 272(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 57, 288(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 58, 304(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 59, 320(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 60, 336(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 61, 352(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 62, 368(1) # 16-byte Folded Spill
+; CHECK-NEXT:    stxv 63, 384(1) # 16-byte Folded Spill
 ; CHECK-NEXT:    blt 0, .LBB0_7
 ; CHECK-NEXT:  # %bb.1: # %_loop_1_do_.lr.ph
 ; CHECK-NEXT:    lwz 3, 0(3)
 ; CHECK-NEXT:    cmpwi 3, 1
 ; CHECK-NEXT:    blt 0, .LBB0_7
 ; CHECK-NEXT:  # %bb.2: # %_loop_1_do_.preheader
-; CHECK-NEXT:    mr 23, 5
-; CHECK-NEXT:    ld 5, 704(1)
+; CHECK-NEXT:    mr 21, 5
+; CHECK-NEXT:    ld 5, 848(1)
+; CHECK-NEXT:    lwa 0, 0(7)
 ; CHECK-NEXT:    addi 3, 3, 1
-; CHECK-NEXT:    ld 4, 728(1)
+; CHECK-NEXT:    mr 14, 7
+; CHECK-NEXT:    mr 22, 6
 ; CHECK-NEXT:    mr 11, 10
-; CHECK-NEXT:    mr 10, 6
-; CHECK-NEXT:    std 8, 40(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 9, 48(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lwa 7, 0(7)
-; CHECK-NEXT:    ld 29, 840(1)
+; CHECK-NEXT:    ld 18, 736(1)
+; CHECK-NEXT:    ld 17, 728(1)
+; CHECK-NEXT:    std 8, 32(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 9, 40(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    cmpldi 3, 9
-; CHECK-NEXT:    ld 27, 832(1)
-; CHECK-NEXT:    ld 28, 856(1)
-; CHECK-NEXT:    std 5, 112(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 4, 120(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 1, 0(5)
+; CHECK-NEXT:    ld 19, 744(1)
+; CHECK-NEXT:    ld 20, 752(1)
+; CHECK-NEXT:    std 5, 200(1) # 8-byte Folded Spill
+; CHECK-NEXT:    ld 5, 840(1)
+; CHECK-NEXT:    std 17, 80(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 18, 88(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 36, 0(18)
+; CHECK-NEXT:    std 19, 96(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 20, 104(1) # 8-byte Folded Spill
+; CHECK-NEXT:    lxv 13, 0(19)
+; CHECK-NEXT:    lxv 12, 0(20)
+; CHECK-NEXT:    ld 30, 832(1)
+; CHECK-NEXT:    ld 2, 824(1)
+; CHECK-NEXT:    ld 12, 816(1)
+; CHECK-NEXT:    ld 29, 808(1)
+; CHECK-NEXT:    std 2, 176(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 30, 184(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 29, 160(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 12, 168(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 5, 192(1) # 8-byte Folded Spill
 ; CHECK-NEXT:    li 5, 9
-; CHECK-NEXT:    ld 30, 848(1)
-; CHECK-NEXT:    lxv 0, 0(4)
-; CHECK-NEXT:    sldi 4, 7, 3
-; CHECK-NEXT:    add 4, 4, 23
-; CHECK-NEXT:    sldi 16, 7, 2
-; CHECK-NEXT:    sldi 15, 7, 1
-; CHECK-NEXT:    ld 17, 760(1)
-; CHECK-NEXT:    std 27, 192(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 29, 200(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 6, 0(29)
-; CHECK-NEXT:    ld 26, 824(1)
-; CHECK-NEXT:    ld 25, 816(1)
-; CHECK-NEXT:    ld 24, 808(1)
-; CHECK-NEXT:    std 30, 208(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 28, 216(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 25, 176(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 26, 184(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 24, 168(1) # 8-byte Folded Spill
-; CHECK-NEXT:    iselgt 3, 3, 5
-; CHECK-NEXT:    ld 5, 752(1)
-; CHECK-NEXT:    addi 14, 4, 32
-; CHECK-NEXT:    sldi 4, 7, 4
-; CHECK-NEXT:    add 29, 7, 15
-; CHECK-NEXT:    ld 22, 800(1)
-; CHECK-NEXT:    ld 21, 792(1)
-; CHECK-NEXT:    ld 20, 784(1)
-; CHECK-NEXT:    std 22, 160(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 20, 144(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 21, 152(1) # 8-byte Folded Spill
-; CHECK-NEXT:    addi 6, 3, -2
-; CHECK-NEXT:    add 3, 7, 16
-; CHECK-NEXT:    add 4, 4, 23
-; CHECK-NEXT:    ld 19, 776(1)
-; CHECK-NEXT:    ld 18, 768(1)
-; CHECK-NEXT:    lxv 4, 0(8)
-; CHECK-NEXT:    lxv 2, 0(11)
-; CHECK-NEXT:    std 18, 128(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 19, 136(1) # 8-byte Folded Spill
-; CHECK-NEXT:    addi 12, 4, 32
-; CHECK-NEXT:    rldicl 2, 6, 61, 3
-; CHECK-NEXT:    sldi 6, 3, 3
-; CHECK-NEXT:    ld 4, 736(1)
-; CHECK-NEXT:    ld 31, 720(1)
-; CHECK-NEXT:    std 11, 56(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 31, 64(1) # 8-byte Folded Spill
-; CHECK-NEXT:    add 11, 23, 6
-; CHECK-NEXT:    ld 6, 744(1)
-; CHECK-NEXT:    ld 8, 712(1)
-; CHECK-NEXT:    std 5, 96(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 17, 104(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 39, 0(5)
-; CHECK-NEXT:    sldi 5, 7, 5
-; CHECK-NEXT:    lxv 5, 0(30)
-; CHECK-NEXT:    lxv 7, 0(28)
-; CHECK-NEXT:    lxv 3, 0(9)
-; CHECK-NEXT:    addi 2, 2, 1
-; CHECK-NEXT:    add 30, 23, 5
-; CHECK-NEXT:    sldi 5, 29, 3
-; CHECK-NEXT:    add 28, 23, 5
-; CHECK-NEXT:    ld 5, 864(1)
+; CHECK-NEXT:    ld 28, 800(1)
+; CHECK-NEXT:    ld 27, 792(1)
+; CHECK-NEXT:    ld 26, 784(1)
+; CHECK-NEXT:    ld 25, 776(1)
+; CHECK-NEXT:    ld 24, 768(1)
+; CHECK-NEXT:    ld 23, 760(1)
+; CHECK-NEXT:    ld 16, 720(1)
+; CHECK-NEXT:    ld 15, 712(1)
+; CHECK-NEXT:    ld 6, 704(1)
+; CHECK-NEXT:    ld 7, 696(1)
+; CHECK-NEXT:    ld 10, 688(1)
 ; CHECK-NEXT:    lxv 43, 0(8)
-; CHECK-NEXT:    lxv 42, 0(31)
-; CHECK-NEXT:    lxv 38, 0(17)
-; CHECK-NEXT:    std 4, 72(1) # 8-byte Folded Spill
-; CHECK-NEXT:    std 6, 80(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 41, 0(4)
-; CHECK-NEXT:    lxv 40, 0(6)
-; CHECK-NEXT:    lxv 33, 0(18)
-; CHECK-NEXT:    lxv 32, 0(19)
-; CHECK-NEXT:    std 5, 88(1) # 8-byte Folded Spill
-; CHECK-NEXT:    lxv 37, 0(20)
-; CHECK-NEXT:    lxv 36, 0(21)
-; CHECK-NEXT:    lxv 13, 0(22)
-; CHECK-NEXT:    lxv 12, 0(24)
-; CHECK-NEXT:    lxv 11, 0(25)
-; CHECK-NEXT:    lxv 9, 0(26)
-; CHECK-NEXT:    lxv 8, 0(27)
-; CHECK-NEXT:    lxv 10, 0(5)
-; CHECK-NEXT:    mulli 27, 7, 48
-; CHECK-NEXT:    mulli 26, 7, 6
-; CHECK-NEXT:    li 25, 1
-; CHECK-NEXT:    li 24, 0
-; CHECK-NEXT:    mr 5, 23
+; CHECK-NEXT:    std 11, 48(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 6, 56(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 27, 144(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 28, 152(1) # 8-byte Folded Spill
+; CHECK-NEXT:    mr 8, 7
+; CHECK-NEXT:    std 25, 128(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 26, 136(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 15, 64(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 16, 72(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 23, 112(1) # 8-byte Folded Spill
+; CHECK-NEXT:    std 24, 120(1) # 8-byte Folded Spill
+; CHECK-NEXT:    iselgt 3, 3, 5
+; CHECK-NEXT:    sldi 5, 0, 3
+; CHECK-NEXT:    add 5, 5, 21
+; CHECK-NEXT:    lxv 42, 0(9)
+; CHECK-NEXT:    lxv 41, 0(11)
+; CHECK-NEXT:    lxv 40, 0(10)
+; CHECK-NEXT:    lxv 39, 0(7)
+; CHECK-NEXT:    mulli 11, 0, 48
+; CHECK-NEXT:    addi 14, 5, 32
+; CHECK-NEXT:    sldi 5, 0, 4
+; CHECK-NEXT:    addi 3, 3, -2
+; CHECK-NEXT:    lxv 38, 0(6)
+; CHECK-NEXT:    lxv 33, 0(15)
+; CHECK-NEXT:    lxv 32, 0(16)
+; CHECK-NEXT:    lxv 37, 0(17)
+; CHECK-NEXT:    add 5, 5, 21
+; CHECK-NEXT:    lxv 11, 0(23)
+; CHECK-NEXT:    lxv 10, 0(24)
+; CHECK-NEXT:    lxv 8, 0(25)
+; CHECK-NEXT:    lxv 6, 0(26)
+; CHECK-NEXT:    rldicl 3, 3, 61, 3
+; CHECK-NEXT:    li 26, 0
+; CHECK-NEXT:    mr 25, 21
+; CHECK-NEXT:    addi 31, 5, 32
+; CHECK-NEXT:    mulli 5, 0, 40
+; CHECK-NEXT:    lxv 5, 0(27)
+; CHECK-NEXT:    lxv 3, 0(28)
+; CHECK-NEXT:    lxv 1, 0(29)
+; CHECK-NEXT:    lxv 0, 0(12)
+; CHECK-NEXT:    mulli 28, 0, 6
+; CHECK-NEXT:    addi 3, 3, 1
+; CHECK-NEXT:    li 27, 1
+; CHECK-NEXT:    add 18, 21, 5
+; CHECK-NEXT:    sldi 5, 0, 5
+; CHECK-NEXT:    lxv 2, 0(2)
+; CHECK-NEXT:    lxv 4, 0(30)
+; CHECK-NEXT:    sldi 2, 0, 1
+; CHECK-NEXT:    add 19, 21, 5
+; CHECK-NEXT:    mulli 5, 0, 24
+; CHECK-NEXT:    add 20, 21, 5
+; CHECK-NEXT:    ld 5, 192(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lxv 9, 0(5)
+; CHECK-NEXT:    ld 5, 200(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lxv 7, 0(5)
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  .LBB0_3: # %_loop_2_do_.lr.ph
 ; CHECK-NEXT:    # =>This Loop Header: Depth=1
 ; CHECK-NEXT:    # Child Loop BB0_4 Depth 2
-; CHECK-NEXT:    maddld 6, 26, 24, 3
-; CHECK-NEXT:    mtctr 2
-; CHECK-NEXT:    sldi 6, 6, 3
-; CHECK-NEXT:    add 22, 23, 6
-; CHECK-NEXT:    maddld 6, 26, 24, 16
-; CHECK-NEXT:    sldi 6, 6, 3
-; CHECK-NEXT:    add 21, 23, 6
-; CHECK-NEXT:    maddld 6, 26, 24, 29
-; CHECK-NEXT:    sldi 6, 6, 3
-; CHECK-NEXT:    add 20, 23, 6
-; CHECK-NEXT:    maddld 6, 26, 24, 15
-; CHECK-NEXT:    sldi 6, 6, 3
-; CHECK-NEXT:    add 19, 23, 6
-; CHECK-NEXT:    maddld 6, 26, 24, 7
-; CHECK-NEXT:    sldi 6, 6, 3
-; CHECK-NEXT:    add 18, 23, 6
-; CHECK-NEXT:    mulld 6, 26, 24
-; CHECK-NEXT:    sldi 6, 6, 3
-; CHECK-NEXT:    add 17, 23, 6
-; CHECK-NEXT:    mr 6, 10
+; CHECK-NEXT:    maddld 5, 28, 26, 2
+; CHECK-NEXT:    mr 6, 22
+; CHECK-NEXT:    mr 29, 20
+; CHECK-NEXT:    mr 30, 19
+; CHECK-NEXT:    mr 12, 18
+; CHECK-NEXT:    mtctr 3
+; CHECK-NEXT:    sldi 5, 5, 3
+; CHECK-NEXT:    add 24, 21, 5
+; CHECK-NEXT:    maddld 5, 28, 26, 0
+; CHECK-NEXT:    sldi 5, 5, 3
+; CHECK-NEXT:    add 23, 21, 5
+; CHECK-NEXT:    mr 5, 25
 ; CHECK-NEXT:    .p2align 5
 ; CHECK-NEXT:  .LBB0_4: # %_loop_2_do_
 ; CHECK-NEXT:    # Parent Loop BB0_3 Depth=1
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    lxvp 34, 0(6)
-; CHECK-NEXT:    lxvp 44, 0(17)
-; CHECK-NEXT:    xvmaddadp 4, 45, 35
-; CHECK-NEXT:    lxvp 46, 0(18)
-; CHECK-NEXT:    xvmaddadp 3, 47, 35
-; CHECK-NEXT:    lxvp 48, 0(19)
-; CHECK-NEXT:    lxvp 50, 0(20)
-; CHECK-NEXT:    lxvp 62, 0(21)
-; CHECK-NEXT:    lxvp 60, 0(22)
+; CHECK-NEXT:    lxvp 44, 0(5)
+; CHECK-NEXT:    xvmaddadp 43, 45, 35
+; CHECK-NEXT:    lxvp 46, 0(23)
+; CHECK-NEXT:    xvmaddadp 42, 47, 35
+; CHECK-NEXT:    lxvp 48, 0(24)
+; CHECK-NEXT:    lxvp 50, 0(29)
+; CHECK-NEXT:    lxvp 62, 0(30)
+; CHECK-NEXT:    lxvp 60, 0(12)
 ; CHECK-NEXT:    lxvp 58, 32(6)
-; CHECK-NEXT:    lxvp 56, 32(17)
-; CHECK-NEXT:    lxvp 54, 32(18)
-; CHECK-NEXT:    lxvp 52, 32(19)
-; CHECK-NEXT:    lxvp 30, 32(20)
-; CHECK-NEXT:    lxvp 28, 32(21)
-; CHECK-NEXT:    lxvp 26, 32(22)
-; CHECK-NEXT:    xvmaddadp 2, 49, 35
-; CHECK-NEXT:    xvmaddadp 1, 51, 35
-; CHECK-NEXT:    xvmaddadp 43, 63, 35
-; CHECK-NEXT:    xvmaddadp 42, 61, 35
-; CHECK-NEXT:    xvmaddadp 0, 44, 34
-; CHECK-NEXT:    xvmaddadp 41, 46, 34
-; CHECK-NEXT:    xvmaddadp 40, 48, 34
-; CHECK-NEXT:    xvmaddadp 39, 50, 34
-; CHECK-NEXT:    xvmaddadp 38, 62, 34
-; CHECK-NEXT:    xvmaddadp 33, 60, 34
-; CHECK-NEXT:    xvmaddadp 32, 57, 59
-; CHECK-NEXT:    xvmaddadp 37, 55, 59
-; CHECK-NEXT:    xvmaddadp 36, 53, 59
-; CHECK-NEXT:    xvmaddadp 13, 31, 59
-; CHECK-NEXT:    xvmaddadp 12, 29, 59
-; CHECK-NEXT:    xvmaddadp 11, 27, 59
-; CHECK-NEXT:    xvmaddadp 9, 56, 58
-; CHECK-NEXT:    xvmaddadp 8, 54, 58
-; CHECK-NEXT:    xvmaddadp 6, 52, 58
-; CHECK-NEXT:    xvmaddadp 5, 30, 58
-; CHECK-NEXT:    xvmaddadp 7, 28, 58
-; CHECK-NEXT:    xvmaddadp 10, 26, 58
+; CHECK-NEXT:    lxvp 56, 32(5)
+; CHECK-NEXT:    lxvp 54, 32(23)
+; CHECK-NEXT:    lxvp 52, 32(24)
+; CHECK-NEXT:    lxvp 30, 32(29)
+; CHECK-NEXT:    lxvp 28, 32(30)
+; CHECK-NEXT:    lxvp 26, 32(12)
+; CHECK-NEXT:    xvmaddadp 41, 49, 35
+; CHECK-NEXT:    xvmaddadp 40, 51, 35
+; CHECK-NEXT:    xvmaddadp 39, 63, 35
+; CHECK-NEXT:    xvmaddadp 38, 61, 35
+; CHECK-NEXT:    xvmaddadp 33, 44, 34
+; CHECK-NEXT:    xvmaddadp 32, 46, 34
+; CHECK-NEXT:    xvmaddadp 37, 48, 34
+; CHECK-NEXT:    xvmaddadp 36, 50, 34
+; CHECK-NEXT:    xvmaddadp 13, 62, 34
+; CHECK-NEXT:    xvmaddadp 12, 60, 34
+; CHECK-NEXT:    xvmaddadp 11, 57, 59
+; CHECK-NEXT:    xvmaddadp 10, 55, 59
+; CHECK-NEXT:    xvmaddadp 8, 53, 59
+; CHECK-NEXT:    xvmaddadp 6, 31, 59
+; CHECK-NEXT:    xvmaddadp 5, 29, 59
+; CHECK-NEXT:    xvmaddadp 3, 27, 59
+; CHECK-NEXT:    xvmaddadp 1, 56, 58
+; CHECK-NEXT:    xvmaddadp 0, 54, 58
+; CHECK-NEXT:    xvmaddadp 2, 52, 58
+; CHECK-NEXT:    xvmaddadp 4, 30, 58
+; CHECK-NEXT:    xvmaddadp 9, 28, 58
+; CHECK-NEXT:    xvmaddadp 7, 26, 58
 ; CHECK-NEXT:    addi 6, 6, 64
-; CHECK-NEXT:    addi 17, 17, 64
-; CHECK-NEXT:    addi 18, 18, 64
-; CHECK-NEXT:    addi 19, 19, 64
-; CHECK-NEXT:    addi 20, 20, 64
-; CHECK-NEXT:    addi 21, 21, 64
-; CHECK-NEXT:    addi 22, 22, 64
+; CHECK-NEXT:    addi 5, 5, 64
+; CHECK-NEXT:    addi 23, 23, 64
+; CHECK-NEXT:    addi 24, 24, 64
+; CHECK-NEXT:    addi 29, 29, 64
+; CHECK-NEXT:    addi 30, 30, 64
+; CHECK-NEXT:    addi 12, 12, 64
 ; CHECK-NEXT:    bdnz .LBB0_4
 ; CHECK-NEXT:  # %bb.5: # %_loop_2_endl_
 ; CHECK-NEXT:    #
-; CHECK-NEXT:    addi 25, 25, 6
-; CHECK-NEXT:    add 5, 5, 27
-; CHECK-NEXT:    add 14, 14, 27
-; CHECK-NEXT:    add 11, 11, 27
-; CHECK-NEXT:    add 12, 12, 27
-; CHECK-NEXT:    add 30, 30, 27
-; CHECK-NEXT:    add 28, 28, 27
-; CHECK-NEXT:    addi 24, 24, 1
-; CHECK-NEXT:    cmpld 25, 0
+; CHECK-NEXT:    addi 27, 27, 6
+; CHECK-NEXT:    add 25, 25, 11
+; CHECK-NEXT:    add 14, 14, 11
+; CHECK-NEXT:    add 18, 18, 11
+; CHECK-NEXT:    add 31, 31, 11
+; CHECK-NEXT:    add 19, 19, 11
+; CHECK-NEXT:    add 20, 20, 11
+; CHECK-NEXT:    addi 26, 26, 1
+; CHECK-NEXT:    cmpld 27, 4
 ; CHECK-NEXT:    ble 0, .LBB0_3
 ; CHECK-NEXT:  # %bb.6: # %_loop_1_loopHeader_._return_bb_crit_edge.loopexit
+; CHECK-NEXT:    ld 3, 32(1) # 8-byte Folded Reload
+; CHECK-NEXT:    stxv 43, 0(3)
 ; CHECK-NEXT:    ld 3, 40(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 4, 0(3)
+; CHECK-NEXT:    stxv 42, 0(3)
 ; CHECK-NEXT:    ld 3, 48(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 3, 0(3)
+; CHECK-NEXT:    stxv 41, 0(3)
 ; CHECK-NEXT:    ld 3, 56(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 2, 0(3)
-; CHECK-NEXT:    ld 3, 112(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 1, 0(3)
+; CHECK-NEXT:    stxv 40, 0(10)
+; CHECK-NEXT:    stxv 39, 0(8)
+; CHECK-NEXT:    stxv 38, 0(3)
 ; CHECK-NEXT:    ld 3, 64(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 43, 0(8)
-; CHECK-NEXT:    stxv 42, 0(3)
-; CHECK-NEXT:    ld 3, 120(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 0, 0(3)
+; CHECK-NEXT:    stxv 33, 0(3)
 ; CHECK-NEXT:    ld 3, 72(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 41, 0(3)
+; CHECK-NEXT:    stxv 32, 0(3)
 ; CHECK-NEXT:    ld 3, 80(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 40, 0(3)
+; CHECK-NEXT:    stxv 37, 0(3)
+; CHECK-NEXT:    ld 3, 88(1) # 8-byte Folded Reload
+; CHECK-NEXT:    stxv 36, 0(3)
 ; CHECK-NEXT:    ld 3, 96(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 39, 0(3)
+; CHECK-NEXT:    stxv 13, 0(3)
 ; CHECK-NEXT:    ld 3, 104(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 38, 0(3)
+; CHECK-NEXT:    stxv 12, 0(3)
+; CHECK-NEXT:    ld 3, 112(1) # 8-byte Folded Reload
+; CHECK-NEXT:    stxv 11, 0(3)
+; CHECK-NEXT:    ld 3, 120(1) # 8-byte Folded Reload
+; CHECK-NEXT:    stxv 10, 0(3)
 ; CHECK-NEXT:    ld 3, 128(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 33, 0(3)
+; CHECK-NEXT:    stxv 8, 0(3)
 ; CHECK-NEXT:    ld 3, 136(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 32, 0(3)
+; CHECK-NEXT:    stxv 6, 0(3)
 ; CHECK-NEXT:    ld 3, 144(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 37, 0(3)
+; CHECK-NEXT:    stxv 5, 0(3)
 ; CHECK-NEXT:    ld 3, 152(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 36, 0(3)
+; CHECK-NEXT:    stxv 3, 0(3)
 ; CHECK-NEXT:    ld 3, 160(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 13, 0(3)
+; CHECK-NEXT:    stxv 1, 0(3)
 ; CHECK-NEXT:    ld 3, 168(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 12, 0(3)
+; CHECK-NEXT:    stxv 0, 0(3)
 ; CHECK-NEXT:    ld 3, 176(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 11, 0(3)
+; CHECK-NEXT:    stxv 2, 0(3)
 ; CHECK-NEXT:    ld 3, 184(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 9, 0(3)
+; CHECK-NEXT:    stxv 4, 0(3)
 ; CHECK-NEXT:    ld 3, 192(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 8, 0(3)
+; CHECK-NEXT:    stxv 9, 0(3)
 ; CHECK-NEXT:    ld 3, 200(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 6, 0(3)
-; CHECK-NEXT:    ld 3, 208(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 5, 0(3)
-; CHECK-NEXT:    ld 3, 216(1) # 8-byte Folded Reload
 ; CHECK-NEXT:    stxv 7, 0(3)
-; CHECK-NEXT:    ld 3, 88(1) # 8-byte Folded Reload
-; CHECK-NEXT:    stxv 10, 0(3)
 ; CHECK-NEXT:  .LBB0_7: # %_return_bb
-; CHECK-NEXT:    lxv 63, 400(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 62, 384(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 61, 368(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 60, 352(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 59, 336(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 58, 320(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 57, 304(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 56, 288(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 55, 272(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 54, 256(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 53, 240(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lxv 52, 224(1) # 16-byte Folded Reload
-; CHECK-NEXT:    lfd 31, 600(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 30, 592(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 29, 584(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 28, 576(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 27, 568(1) # 8-byte Folded Reload
-; CHECK-NEXT:    lfd 26, 560(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 31, 552(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 30, 544(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 29, 536(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 28, 528(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 27, 520(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 26, 512(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 25, 504(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 24, 496(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 23, 488(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 22, 480(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 21, 472(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 20, 464(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 19, 456(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 18, 448(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 17, 440(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 16, 432(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 15, 424(1) # 8-byte Folded Reload
-; CHECK-NEXT:    ld 14, 416(1) # 8-byte Folded Reload
-; CHECK-NEXT:    addi 1, 1, 608
+; CHECK-NEXT:    lxv 63, 384(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 62, 368(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 61, 352(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 60, 336(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 59, 320(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 58, 304(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 57, 288(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 56, 272(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 55, 256(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 54, 240(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 53, 224(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lxv 52, 208(1) # 16-byte Folded Reload
+; CHECK-NEXT:    lfd 31, 584(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 30, 576(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 29, 568(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 28, 560(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 27, 552(1) # 8-byte Folded Reload
+; CHECK-NEXT:    lfd 26, 544(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 31, 536(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 30, 528(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 29, 520(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 28, 512(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 27, 504(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 26, 496(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 25, 488(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 24, 480(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 23, 472(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 22, 464(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 21, 456(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 20, 448(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 19, 440(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 18, 432(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 17, 424(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 16, 416(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 15, 408(1) # 8-byte Folded Reload
+; CHECK-NEXT:    ld 14, 400(1) # 8-byte Folded Reload
+; CHECK-NEXT:    addi 1, 1, 592
 ; CHECK-NEXT:    blr
 entry:
   %_val_l_ = load i32, i32* %.l, align 4
   %_conv = sext i32 %_val_l_ to i64
   %_mult_tmp = shl nsw i64 %_conv, 3
   %_sub_tmp4 = sub nuw nsw i64 -8, %_mult_tmp
   %_val_n_ = load i32, i32* %.n, align 4
   %_leq_tmp.not116 = icmp slt i32 %_val_n_, 1