[GVN] teach ConstantFolding correct handling of non-integral addrspace casts

[vtjnash]

Summary

Here we teach the ConstantFolding analysis pass that it is not legal to replace a load of a bitcast constant (having a non-integral addrspace) with a bitcast of the value of that constant (with a different non-integral addrspace).

But also teach it that certain bit patterns are always known and convertable (a fact it already uses elsewhere). This requires us to also fix a globalopt test, since, after this change, LLVM is able to realize that the test actually is a valid transform (NULL is always a known bit-pattern) and so it doesn't need to emit the failure remarks for it.

Also simplify some of the negative tests for transforms by avoiding a type change in their bitcast, and add positive versions of the same tests, to show that they otherwise should work.

[reames]

test/Transforms/GVN/non-integral-pointers.ll

184 ↗

(On Diff #191978)

Please revert the type change on this test. Feel free to add a separate test with the i64 type.

Actually, your test diffs in general are confusing. Please land your tests, and then rebase on top of them so that changes are visible. Please do NOT change existing tests without cause.

[vtjnash]

bump. please review, thanks!

[loladiro]

This looks fine to me now. @reames if you have a chance, I'd appreciate confirmation that your concerns are addressed as well.

[reames]

lib/Analysis/ConstantFolding.cpp

101 ↗

(On Diff #196652)

This line is incorrect. a) It volates the specification of the function (return ConstantExpr::getBitCast(C, DestTy);). b) for the ASs not to match this would have to be an addrspace cast, not a bitcast. Which shouldn't reach here.

lib/Transforms/Utils/VNCoercion.cpp

324 ↗

(On Diff #196652)

there's probably something missing here. You're removing an early exit without introducing a new one. Given we still need to bail in at least some cases, that's highly suspicious?

[vtjnash]

FoldBitCast was being instructed to create invalid IR in several test cases. We happened to fold away them away before the verifier noticed, so this adds an explicit check, and corrects the caller FoldReinterpretLoadFromConstPtr to form valid IR when handling vectors of pointers with this method.

[vtjnash]

lib/Analysis/ConstantFolding.cpp

101 ↗

(On Diff #196652)

Good call. That makes sense. Several cases in the test suite do accidentally reach here. I'll fix the caller to ensure it only passes in valid inputs, and make this an assert here instead.

lib/Transforms/Utils/VNCoercion.cpp

324 ↗

(On Diff #196652)

It's really hard to write a correct early-exit here, as we need duplicate a substantial chunk of ConstantFoldLoadFromConstPtr (as the TODO below hints at) to prove which cases it knows how to handle correctly. This instead now leans on ConstantFolding to only create and return valid IR. I think dropping this makes it easier to test also, since we don't need to show that both the early-return here and the fall-through cases would give the right answers.

[vtjnash]

Bump. Fixed the other bug in handling vectors that you noted, and added an assert to catch that misuse.

[reames]

lib/Analysis/ConstantFolding.cpp

96 ↗

(On Diff #207207)

Please separate and land this assert.

332 ↗

(On Diff #207207)

This may accidentally change semantics for struct typed loads, I'm not sure.

543 ↗

(On Diff #207207)

I'm not following the need for this change.

Strong suggestion: Make the smallest possible change you can, and post that for review. You've changed the patch here multiple times which is making it challenging to give you meaningful review across revisions. Take the time to pull out helper functions if relevant, add asserts, whatever. Just find a way to make the correctness of each step self documenting.

[vtjnash]

[GVN] teach ConstantFolding correct handling of non-integral addrspace casts

[vtjnash]

[GVN] teach ConstantFolding correct handling of non-integral addrspace casts

[vtjnash]

lib/Analysis/ConstantFolding.cpp

332 ↗

(On Diff #207207)

I don't see how that could happen: the while loop below repeated slices up C (aka SrcTy, aka SrcSize) until SrcSize==DestSize. If SrcSize < DestSize, then transitivity ensures that was never true. That should make sense too from a desired behavior standpoint, since if we're casting a small constant to a larger one, we're appending unknown (not necessarily undef) bits.

Since we need to catch the "isNullValue" obvious cases below to avoid regressing capabilities (and failing the GVN tests) while introducing the DL.isNonIntegralPointerType correctness check, we need this early-exit first to check the same size precondition that would get implied by the while loop.

543 ↗

(On Diff #207207)

It's the bug you mentioned above (this line ... violates the specification of the function). I've split off this fix into https://reviews.llvm.org/D65057.

[vtjnash]

bump: this should be ready to review now.

[vtjnash]

ping. With the other commits split out and landed, this is now much smaller at moving the isNonIntegralPointerType check into the correct place. This is still good to go from my perspective.

[reames]

LGTM after in person discussion.

[vtjnash]

Merged as 19f01a484760d9a29f4f414b30da110dd6550191, e244f86f4dfd9e8982940d09294af522d3809d7f, and 2c7a07b59d5da54eba8e3e030e1cc040a88ecf58

[nathanchance]

This commits causes a failed assertion when building the Linux kernel for arm32:

$ make -skj"$(nproc)" ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 O=out/arm distclean defconfig net/core/fib_rules.o
...
clang-11: /home/nathan/cbl/github/tc-build/llvm-project/llvm/include/llvm/Support/Casting.h:269: typename cast_retty<X, Y *>::ret_type llvm::cast(Y *) [X = llvm::VectorType, Y = llvm::Type]: Assertion `isa<X>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11 -cc1 -triple armv7-unknown-linux-gnueabi -S -disable-free -main-file-name fib_rules.c -mrelocation-model static -meabi gnu -fno-delete-null-pointer-checks -mllvm -warn-stack-size=1024 -mframe-pointer=none -relaxed-aliasing -fmath-errno -fno-rounding-math -no-integrated-as -mconstructor-aliases -munwind-tables -target-cpu generic -target-feature +soft-float -target-feature +soft-float-abi -target-feature -vfp2 -target-feature -vfp2sp -target-feature -vfp3 -target-feature -vfp3d16 -target-feature -vfp3d16sp -target-feature -vfp3sp -target-feature -fp16 -target-feature -vfp4 -target-feature -vfp4d16 -target-feature -vfp4d16sp -target-feature -vfp4sp -target-feature -fp-armv8 -target-feature -fp-armv8d16 -target-feature -fp-armv8d16sp -target-feature -fp-armv8sp -target-feature -fullfp16 -target-feature -fp64 -target-feature -d32 -target-feature -neon -target-feature -crypto -target-feature -dotprod -target-feature -fp16fml -target-feature -mve -target-feature -mve.fp -target-feature -fpregs -target-abi aapcs-linux -msoft-float -mfloat-abi soft -mllvm -arm-global-merge=false -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -nostdsysteminc -nobuiltininc -resource-dir /home/nathan/cbl/github/tc-build/build/llvm/stage1/lib/clang/11.0.0 -dependency-file net/core/.fib_rules.o.d -MT net/core/fib_rules.o -isystem /home/nathan/cbl/github/tc-build/build/llvm/stage1/lib/clang/11.0.0/include -include /home/nathan/src/linux/include/linux/kconfig.h -include /home/nathan/src/linux/include/linux/compiler_types.h -I /home/nathan/src/linux/arch/arm/include -I ./arch/arm/include/generated -I /home/nathan/src/linux/include -I ./include -I /home/nathan/src/linux/arch/arm/include/uapi -I ./arch/arm/include/generated/uapi -I /home/nathan/src/linux/include/uapi -I ./include/generated/uapi -D __KERNEL__ -D __LINUX_ARM_ARCH__=7 -U arm -I /home/nathan/src/linux/net/core -I ./net/core -D KBUILD_MODFILE="net/core/fib_rules" -D KBUILD_BASENAME="fib_rules" -D KBUILD_MODNAME="fib_rules" -fmacro-prefix-map=/home/nathan/src/linux/= -O2 -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -Werror=unknown-warning-option -Wno-frame-address -Wno-address-of-packed-member -Wno-format-invalid-specifier -Wno-gnu -Wno-unused-const-variable -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-array-bounds -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -std=gnu89 -fno-dwarf-directory-asm -fdebug-compilation-dir /home/nathan/src/linux/out/arm -ferror-limit 19 -fwrapv -stack-protector 2 -fno-signed-char -fwchar-type=short -fno-signed-wchar -fgnuc-version=4.2.1 -vectorize-loops -vectorize-slp -o /tmp/fib_rules-aafd14.s -x c /home/nathan/src/linux/net/core/fib_rules.c 
1.	<eof> parser at end of file
2.	Per-function optimization
3.	Running pass 'Early CSE' on function '@fib_rule_matchall'
 #0 0x0000000002b0e584 PrintStackTraceSignalHandler(void*) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x2b0e584)
 #1 0x0000000002b0c25e llvm::sys::RunSignalHandlers() (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x2b0c25e)
 #2 0x0000000002b0e8a5 SignalHandler(int) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x2b0e8a5)
 #3 0x00007fa6d14863c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x153c0)
 #4 0x00007fa6d0f4b18b raise (/lib/x86_64-linux-gnu/libc.so.6+0x4618b)
 #5 0x00007fa6d0f2a859 abort (/lib/x86_64-linux-gnu/libc.so.6+0x25859)
 #6 0x00007fa6d0f2a729 (/lib/x86_64-linux-gnu/libc.so.6+0x25729)
 #7 0x00007fa6d0f3bf36 (/lib/x86_64-linux-gnu/libc.so.6+0x36f36)
 #8 0x00000000024043cc (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x24043cc)
 #9 0x0000000001e775b1 llvm::ConstantFoldLoadFromConstPtr(llvm::Constant*, llvm::Type*, llvm::DataLayout const&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x1e775b1)
#10 0x0000000001e78cf6 llvm::ConstantFoldInstruction(llvm::Instruction*, llvm::DataLayout const&, llvm::TargetLibraryInfo const*) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x1e78cf6)
#11 0x0000000001e66b08 llvm::SimplifyInstruction(llvm::Instruction*, llvm::SimplifyQuery const&, llvm::OptimizationRemarkEmitter*) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x1e66b08)
#12 0x00000000028b58a2 (anonymous namespace)::EarlyCSE::processNode(llvm::DomTreeNodeBase<llvm::BasicBlock>*) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x28b58a2)
#13 0x00000000028b4530 (anonymous namespace)::EarlyCSE::run() (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x28b4530)
#14 0x00000000028bd0d8 (anonymous namespace)::EarlyCSELegacyCommonPass<false>::runOnFunction(llvm::Function&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x28bd0d8)
#15 0x00000000024b5291 llvm::FPPassManager::runOnFunction(llvm::Function&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x24b5291)
#16 0x00000000024b48f9 llvm::legacy::FunctionPassManagerImpl::run(llvm::Function&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x24b48f9)
#17 0x00000000024bb778 llvm::legacy::FunctionPassManager::run(llvm::Function&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x24bb778)
#18 0x0000000002d15565 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x2d15565)
#19 0x0000000003505a9f clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x3505a9f)
#20 0x0000000003bb9c03 clang::ParseAST(clang::Sema&, bool, bool) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x3bb9c03)
#21 0x000000000346a183 clang::FrontendAction::Execute() (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x346a183)
#22 0x00000000033ca993 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x33ca993)
#23 0x00000000035001d3 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x35001d3)
#24 0x00000000018d637d cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x18d637d)
#25 0x00000000018d44ec ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x18d44ec)
#26 0x00000000018d4251 main (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x18d4251)
#27 0x00007fa6d0f2c0b3 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b3)
#28 0x00000000018d125e _start (/home/nathan/cbl/github/tc-build/build/llvm/stage1/bin/clang-11+0x18d125e)
clang-11: error: unable to execute command: Aborted
clang-11: error: clang frontend command failed due to signal (use -v to see invocation)
ClangBuiltLinux clang version 11.0.0 (https://github.com/llvm/llvm-project 2c7a07b59d5da54eba8e3e030e1cc040a88ecf58)
Target: arm-unknown-linux-gnueabi
Thread model: posix
InstalledDir: /home/nathan/cbl/github/tc-build/build/llvm/stage1/bin
clang-11: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-11: note: diagnostic msg: /tmp/fib_rules-ed4e47.c
clang-11: note: diagnostic msg: /tmp/fib_rules-ed4e47.sh
clang-11: note: diagnostic msg: 

********************
...

A reduction of the problematic file spat out:

struct {
  int a
} const b = {~0};
c() { d(b); }

The full preprocessed file plus interestingness test are available here: https://github.com/nathanchance/creduce-files/tree/22e52e0720fe63d50e01a1a9fb11475ad3ee07c3/D59730

[vtjnash]

Thanks, that's a great reduction. I'll fix this shortly.

llvm/lib/Analysis/ConstantFolding.cpp

bool llvm::IsConstantOffsetFromGlobal(Constant *C, GlobalValue *&GV,
   Offset = TmpOffset;
   return true;
 }
 
 Constant *llvm::ConstantFoldLoadThroughBitcast(Constant *C, Type *DestTy,
                                          const DataLayout &DL) {
   do {
     Type *SrcTy = C->getType();
+    uint64_t DestSize = DL.getTypeSizeInBits(DestTy);
+    uint64_t SrcSize = DL.getTypeSizeInBits(SrcTy);
+    if (SrcSize < DestSize)
+      return nullptr;
 
     // If the type sizes are the same and a cast is legal, just directly
     // cast the constant.
-    if (DL.getTypeSizeInBits(DestTy) == DL.getTypeSizeInBits(SrcTy)) {
+    // But be careful not to coerce non-integral pointers illegally.
+    if (SrcSize == DestSize) {
       Instruction::CastOps Cast = Instruction::BitCast;
       // If we are going from a pointer to int or vice versa, we spell the cast
       // differently.
       if (SrcTy->isIntegerTy() && DestTy->isPointerTy())
         Cast = Instruction::IntToPtr;
       else if (SrcTy->isPointerTy() && DestTy->isIntegerTy())
         Cast = Instruction::PtrToInt;
 

llvm/test/Transforms/GVN/non-integral-pointers.ll

 ;
   entry:
   %loc.bc = bitcast i8 addrspace(4)* addrspace(4)* %loc to <2 x i32> addrspace(4)*
   store <2 x i32> zeroinitializer, <2 x i32> addrspace(4)* %loc.bc
   %ref = load i8 addrspace(4)*, i8 addrspace(4)* addrspace(4)* %loc
   ret i8 addrspace(4)* %ref
 }
 
+
+
 @NonZeroConstant = constant <4 x i64> <i64 3, i64 3, i64 3, i64 3>
+@NonZeroConstant2 = constant <4 x i64 addrspace(4)*> <
+  i64 addrspace(4)* getelementptr (i64, i64 addrspace(4)* null, i32 3),
+  i64 addrspace(4)* getelementptr (i64, i64 addrspace(4)* null, i32 3),
+  i64 addrspace(4)* getelementptr (i64, i64 addrspace(4)* null, i32 3),
+  i64 addrspace(4)* getelementptr (i64, i64 addrspace(4)* null, i32 3)>
 @ZeroConstant = constant <4 x i64> zeroinitializer
 
 
 ; Can't forward as the load might be dead.  (Pretend we wrote out the alwaysfalse idiom above.)
 define i8 addrspace(4)* @neg_forward_memcopy(i8 addrspace(4)* addrspace(4)* %loc) {
 ; CHECK-LABEL: @neg_forward_memcopy(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast i8 addrspace(4)* addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
 ; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64>* @NonZeroConstant to i8*), i64 8, i1 false)
 ; CHECK-NEXT:    [[REF:%.*]] = load i8 addrspace(4)*, i8 addrspace(4)* addrspace(4)* [[LOC]], align 8
 ; CHECK-NEXT:    ret i8 addrspace(4)* [[REF]]
 ;
 entry:
   %loc.bc = bitcast i8 addrspace(4)* addrspace(4)* %loc to i8 addrspace(4)*
   %src.bc = bitcast <4 x i64>* @NonZeroConstant to i8*
   call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 %loc.bc, i8* %src.bc, i64 8, i1 false)
   %ref = load i8 addrspace(4)*, i8 addrspace(4)* addrspace(4)* %loc
   ret i8 addrspace(4)* %ref
 }
 
+define i64 addrspace(4)* @neg_forward_memcopy2(i64 addrspace(4)* addrspace(4)* %loc) {
+; CHECK-LABEL: @neg_forward_memcopy2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast i64 addrspace(4)* addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
+; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64>* @NonZeroConstant to i8*), i64 8, i1 false)
+; CHECK-NEXT:    [[REF:%.*]] = load i64 addrspace(4)*, i64 addrspace(4)* addrspace(4)* [[LOC]]
+; CHECK-NEXT:    ret i64 addrspace(4)* [[REF]]
+;
+entry:
+  %loc.bc = bitcast i64 addrspace(4)* addrspace(4)* %loc to i8 addrspace(4)*
+  %src.bc = bitcast <4 x i64>* @NonZeroConstant to i8*
+  call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 %loc.bc, i8* %src.bc, i64 8, i1 false)
+  %ref = load i64 addrspace(4)*, i64 addrspace(4)* addrspace(4)* %loc
+  ret i64 addrspace(4)* %ref
+}
+
+; TODO: missed optimization
+define i8 addrspace(4)* @forward_memcopy(i8 addrspace(4)* addrspace(4)* %loc) {
+; CHECK-LABEL: @forward_memcopy(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast i8 addrspace(4)* addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
+; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64 addrspace(4)*>* @NonZeroConstant2 to i8*), i64 8, i1 false)
+; CHECK-NEXT:    [[REF:%.*]] = load i8 addrspace(4)*, i8 addrspace(4)* addrspace(4)* %loc
+; CHECK-NEXT:    ret i8 addrspace(4)* [[REF]]
+;
+entry:
+  %loc.bc = bitcast i8 addrspace(4)* addrspace(4)* %loc to i8 addrspace(4)*
+  %src.bc = bitcast <4 x i64 addrspace(4)*>* @NonZeroConstant2 to i8*
+  call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 %loc.bc, i8* %src.bc, i64 8, i1 false)
+  %ref = load i8 addrspace(4)*, i8 addrspace(4)* addrspace(4)* %loc
+  ret i8 addrspace(4)* %ref
+}
+
 define <1 x i8 addrspace(4)*> @neg_forward_memcpy_vload(<1 x i8 addrspace(4)*> addrspace(4)* %loc) {
 ; CHECK-LABEL: @neg_forward_memcpy_vload(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast <1 x i8 addrspace(4)*> addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
 ; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64>* @NonZeroConstant to i8*), i64 8, i1 false)
 ; CHECK-NEXT:    [[REF:%.*]] = load <1 x i8 addrspace(4)*>, <1 x i8 addrspace(4)*> addrspace(4)* [[LOC]], align 8
 ; CHECK-NEXT:    ret <1 x i8 addrspace(4)*> [[REF]]
 ;
 entry:
   %loc.bc = bitcast <1 x i8 addrspace(4)*> addrspace(4)* %loc to i8 addrspace(4)*
   %src.bc = bitcast <4 x i64>* @NonZeroConstant to i8*
   call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 %loc.bc, i8* %src.bc, i64 8, i1 false)
   %ref = load <1 x i8 addrspace(4)*>, <1 x i8 addrspace(4)*> addrspace(4)* %loc
   ret <1 x i8 addrspace(4)*> %ref
 }
 
+define <4 x i64 addrspace(4)*> @neg_forward_memcpy_vload2(<4 x i64 addrspace(4)*> addrspace(4)* %loc) {
+; CHECK-LABEL: @neg_forward_memcpy_vload2(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast <4 x i64 addrspace(4)*> addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
+; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64>* @NonZeroConstant to i8*), i64 32, i1 false)
+; CHECK-NEXT:    [[REF:%.*]] = load <4 x i64 addrspace(4)*>, <4 x i64 addrspace(4)*> addrspace(4)* [[LOC]]
+; CHECK-NEXT:    ret <4 x i64 addrspace(4)*> [[REF]]
+;
+entry:
+  %loc.bc = bitcast <4 x i64 addrspace(4)*> addrspace(4)* %loc to i8 addrspace(4)*
+  %src.bc = bitcast <4 x i64>* @NonZeroConstant to i8*
+  call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 %loc.bc, i8* %src.bc, i64 32, i1 false)
+  %ref = load <4 x i64 addrspace(4)*>, <4 x i64 addrspace(4)*> addrspace(4)* %loc
+  ret <4 x i64 addrspace(4)*> %ref
+}
+
+define <4 x i64> @neg_forward_memcpy_vload3(<4 x i64> addrspace(4)* %loc) {
+; CHECK-LABEL: @neg_forward_memcpy_vload3(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast <4 x i64> addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
+; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64 addrspace(4)*>* @NonZeroConstant2 to i8*), i64 32, i1 false)
+; CHECK-NEXT:    [[REF:%.*]] = load <4 x i64>, <4 x i64> addrspace(4)* [[LOC]]
+; CHECK-NEXT:    ret <4 x i64> [[REF]]
+;
+entry:
+  %loc.bc = bitcast <4 x i64> addrspace(4)* %loc to i8 addrspace(4)*
+  %src.bc = bitcast <4 x i64 addrspace(4)*>* @NonZeroConstant2 to i8*
+  call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 %loc.bc, i8* %src.bc, i64 32, i1 false)
+  %ref = load <4 x i64>, <4 x i64> addrspace(4)* %loc
+  ret <4 x i64> %ref
+}
 
 ; Can forward since we can do so w/o breaking types
 ; TODO: missed optimization
 define i8 addrspace(4)* @forward_memcpy_zero(i8 addrspace(4)* addrspace(4)* %loc) {
 ; CHECK-LABEL: @forward_memcpy_zero(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[LOC_BC:%.*]] = bitcast i8 addrspace(4)* addrspace(4)* [[LOC:%.*]] to i8 addrspace(4)*
 ; CHECK-NEXT:    call void @llvm.memcpy.p4i8.p0i8.i64(i8 addrspace(4)* align 4 [[LOC_BC]], i8* bitcast (<4 x i64>* @ZeroConstant to i8*), i64 8, i1 false)

llvm/test/Transforms/GlobalOpt/evaluate-call-errors.ll

define linkonce_odr void @_ZN1SC2Ev(%struct.S*) unnamed_addr align 2 {
   store i32 %6, i32* %7, align 4
   %8 = getelementptr inbounds %struct.Foo, %struct.Foo* %3, i32 0, i32 0
   %9 = load i32, i32* %8, align 4
   store i32 %9, i32* %5, align 4
   ret void
 }
 
 define internal %struct.Foo* @_ZL3foov() {
-  ret %struct.Foo* null
+  ret %struct.Foo* getelementptr (%struct.Foo, %struct.Foo *null, i32 1)
 }
 
 define linkonce_odr void @_ZN1QC2Ev(%struct.Q*) unnamed_addr align 2 {
   %2 = alloca %struct.Q*, align 8
   store %struct.Q* %0, %struct.Q** %2, align 8
   %3 = load %struct.Q*, %struct.Q** %2, align 8
   %4 = getelementptr inbounds %struct.Q, %struct.Q* %3, i32 0, i32 0
-  %5 = call i32 bitcast (i32 (i32)* @_ZL3baz3Foo to i32 (%struct.Foo*)*)(%struct.Foo* null)
+  %5 = call i32 bitcast (i32 (i32)* @_ZL3baz3Foo to i32 (%struct.Foo*)*)(%struct.Foo* getelementptr (%struct.Foo, %struct.Foo *null, i32 1))
   store i32 %5, i32* %4, align 4
   ret void
 }
 
 define internal i32 @_ZL3baz3Foo(i32) {
   %2 = alloca %struct.Foo, align 4
   %3 = getelementptr inbounds %struct.Foo, %struct.Foo* %2, i32 0, i32 0
   store i32 %0, i32* %3, align 4