[InstCombine] Don't blow up in foldICmpWithCastAndCast on vector icmp instructions.

[dneilson]

Summary

Presently, InstCombiner::foldICmpWithCastAndCast() implicitly assumes that it is
only invoked with icmp instructions of integer type. If that assumption is broken,
and it is called with an icmp of vector type, then it fails (asserts/crashes).

This patch addresses the deficiency. It allows it to simplify
icmp (ptrtoint x), (ptrtoint/c) of vector type into a compare of the inputs,
much as is done when the type is integer.

[anna]

lib/Transforms/InstCombine/InstCombineCompares.cpp

3411 ↗

(On Diff #136890)

Instead of capturing everything, let's capture just DL.

[dneilson]

lib/Transforms/InstCombine/InstCombineCompares.cpp

3411 ↗

(On Diff #136890)

This is not capturing everything, it is only capturing the this pointer.

Per: http://en.cppreference.com/w/cpp/language/lambda

[&] captures all automatic variables **used in the body** of the lambda by reference and current object by reference if exists (emphasis mine).

There are no automatics used in the body of this lambda, so only the current object reference (i.e. the this pointer) is captured.

I could change [&] to [this] to make it more explicit, if desired...

[anna]

lib/Transforms/InstCombine/InstCombineCompares.cpp

3411 ↗

(On Diff #136890)

But why do you need to capture this? Basically, this should do right:

const auto& CompatibleSizes = [&DL](Type* SrcTy, Type* DestTy) -> bool {
....

It makes it clearer that the only capture required is DL which is used in the lambda.

[dneilson]

lib/Transforms/InstCombine/InstCombineCompares.cpp

3411 ↗

(On Diff #136890)

Non-obvious detail: DL is a member of the InstCombiner class, and not an automatic.

You can't capture members. Can only capture reference to current object (i.e. this), and automatics.

[anna]

lgtm

lib/Transforms/InstCombine/InstCombineCompares.cpp

3411 ↗

(On Diff #136890)

ah I didn't see that.

llvm/trunk/lib/Transforms/InstCombine/InstCombineCompares.cpp

Instruction *InstCombiner::foldICmpWithCastAndCast(ICmpInst &ICmp) {
   const CastInst *LHSCI = cast<CastInst>(ICmp.getOperand(0));
   Value *LHSCIOp        = LHSCI->getOperand(0);
   Type *SrcTy     = LHSCIOp->getType();
   Type *DestTy    = LHSCI->getType();
   Value *RHSCIOp;
 
   // Turn icmp (ptrtoint x), (ptrtoint/c) into a compare of the input if the
   // integer type is the same size as the pointer type.
+  const auto& CompatibleSizes = [&](Type* SrcTy, Type* DestTy) -> bool {
+    if (isa<VectorType>(SrcTy)) {
+      SrcTy = cast<VectorType>(SrcTy)->getElementType();
+      DestTy = cast<VectorType>(DestTy)->getElementType();
+    }
+    return DL.getPointerTypeSizeInBits(SrcTy) == DestTy->getIntegerBitWidth();
+  };
   if (LHSCI->getOpcode() == Instruction::PtrToInt &&
-      DL.getPointerTypeSizeInBits(SrcTy) == DestTy->getIntegerBitWidth()) {
+      CompatibleSizes(SrcTy, DestTy)) {
     Value *RHSOp = nullptr;
     if (auto *RHSC = dyn_cast<PtrToIntOperator>(ICmp.getOperand(1))) {
       Value *RHSCIOp = RHSC->getOperand(0);
       if (RHSCIOp->getType()->getPointerAddressSpace() ==
           LHSCIOp->getType()->getPointerAddressSpace()) {
         RHSOp = RHSC->getOperand(0);
         // If the pointer types don't match, insert a bitcast.
         if (LHSCIOp->getType() != RHSOp->getType())

llvm/trunk/test/Transforms/InstCombine/icmp.ll

 ; CHECK-NEXT:    ret i1 true
 ;
   %cmp = icmp sgt i32* %a, inttoptr (i64 -1 to i32*)
   %maskcond = icmp eq i32* %a, null
   tail call void @llvm.assume(i1 %maskcond)
   ret i1 %cmp
 }
 
+; Don't crash by assuming the compared values are integers.
+define <2 x i1> @PR36583(<2 x i8*>)  {
+; CHECK-LABEL: @PR36583(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[RES:%.*]] = icmp eq <2 x i8*> %0, zeroinitializer
+; CHECK-NEXT:    ret <2 x i1> [[RES]]
+;
+entry:
+  %cast = ptrtoint <2 x i8*> %0 to <2 x i64>
+  %res = icmp eq <2 x i64> %cast, zeroinitializer
+  ret <2 x i1> %res
+}