[CodeGen] Add a boolean flag to `Address::getPointer` and `Lvalue::getPointer` that indicates whether the pointer is known not to be null

[ahatanak]

Summary

The flag will be used for the arm64e work we plan to upstream in the future (see https://lists.llvm.org/pipermail/llvm-dev/2019-October/136091.html). Passing KnownNonNull_t::True allows clang to avoid emitting null checks on arm64e. Currently the flag has no effect on code generation on other targets.

This is the first patch and there are many more places where KnownNonNull_t::True can be passed. I plan to fix those places in future patches.

[efriedma]

I'm having a little trouble following what the meaning of an "Address" is with this patch. The Pointer member refers to an encoded value, and calling getPointer() emits IR to decode it? Could use a few comments to explain the API.

[rjmccall]

In D142584#4081345, @efriedma wrote:

I'm having a little trouble following what the meaning of an "Address" is with this patch. The Pointer member refers to an encoded value, and calling getPointer() emits IR to decode it? Could use a few comments to explain the API.

That's something of a future direction w.r.t this patch, but it's reasonable to talk about it, yeah.

Pointer authentication includes the ability to have signed data pointers. Function pointers can only be used opaquely in C, but data pointers are subject to a lot of operations that boil down to pointer arithmetic, both as part of l-value accesses (such as signedPtr->array[i] = 5) and just in abstract pointer-typed expressions (such as &ptr->array[i]). For l-values that are ultimately accessed, we really want to make sure that we do the authentication as close as possible to the actual access rather than authenticating it immediately and then having a raw, attackable pointer sitting around while we set up the access. For pointer expressions in general, if they ultimately need to produce another signed pointer (e.g. we assign into a __ptrauth-qualified l-value), for similar reasons we really want to set up all the arithmetic ahead of time for a secure auth-add-and-resign sequence instead of authenticating and having raw, attackable pointers sitting around as we do the arithmetic. We could do all that with a specialized pointer-with-signature emitter that covers literally every kind of pointer arithmetic, but that requires duplication of a *lot* of code, especially because we have a lot of sanitizers and other special logic also tied to those operations. It seemed much less invasive to add a state to Address where it can be a pair of a signed pointer and an offset, then make a few basic GEP-formation operations accumulate into the offset when the pointer is in that state. Of course, converting an Address into a normal pointer then becomes a non-trivial operation that needs a CGF.

Pointer authentication doesn't sign null pointers, so we have to do some explicit extra checks as part of this when the pointer is possibly null, and this patch is trying to make that easier to avoid.

clang/lib/CodeGen/Address.h

26

The usual idiom for this kind of boolean enum is that you just use an unscoped enum (not enum class), and you spell the enumerators something that reads well where it'll be used. So you'd make the enumerators KnownNotNull and (I guess) something like NotKnownNotNull or NullnessUnknown.

113

Apologies if this rehashes a conversation we had earlier, but does it make more sense to track this internally in Address? Some Address sources definitely known that they're non-null, and the places where we only know that *contextually* could definitely just pass it down to e.g. EmitPointerWithAlignment. That would make it a less invasive change for all the clients of getPointer().

[ahatanak]

clang/lib/CodeGen/Address.h

113

Yes, tracking it in Address and LValue is another way to fix the problem. The implementation in my local branch currently adds an IsKnownNonNull flag to the constructor of Address and method setKnownNonNull to both Address and LValue. The method is called when the pointer is known not to be null contextually.

[rjmccall]

clang/lib/CodeGen/Address.h

113

Okay. It might make sense to pass this into EmitPointerWithAlignment contextually instead of just relying on changing it afterwards, since it's possible that some of the work there can be optimized when the pointer is contextually not allowed to be null. (For example, C++ derived-to-base conversions, although I think we might optimize those already by using a different CastKind when there's a contextual requirement to be non-null (e.g. when it's the this argument of a method call). But I'm sure there are situations that doesn't kick in that would be optimized, like if you static_cast the pointer before calling a method on it.)

[ahatanak]

Add a bit to Address and LValue that tracks whether the pointer is known not to be null.

[ahatanak]

There are more places where we know contextually the pointer can't be null and can set the bit to KnownNonNull or set the bit of an LValue or Address using an existing Address's KnownNonNull bit.

[ahatanak]

Fix a few typos that were causing compile errors.

[ahatanak]

clang/lib/CodeGen/Address.h

147

This isn't safe. We cannot tell whether the new pointer is non-null or not based on whether it's currently non-null.

[ahatanak]

Pass a KnownNonNull_t flag to Address::withPointer.

[rjmccall]

This seems reasonable to me. Eli, are your questions answered?

[ahatanak]

@efriedma do you have any comments?

[efriedma]

The approach here makes sense.

clang/lib/CodeGen/Address.h

65–66

Do you need to do something with IsKnownNonNull in the Alignment.isZero() case?

68–69

This comment isn't right. The max alignment is, as far as I can tell, 1<<32 exactly. (But there's something weird going on with very large values... somehow int a[1LL<<32] __attribute((aligned(1ULL<<32))) = {}; ignores the alignment.)

clang/lib/CodeGen/CGBuilder.h

164

Do address-space casts preserve non-null?

277

Do non-inbounds GEPs preserve nonnull?

clang/lib/CodeGen/CGValue.h

181

Do we not have spare bits in the bitfield?

[ahatanak]

Address review comments.

clang/lib/CodeGen/Address.h

65–66

We can probably get away with not doing anything here because currently the alignment is zero only when Address::invalid is called.

68–69

The following function generated by tablegen (and a few others directly or indirectly calling the function) returns a 32-bit int, but it should be returning a 64-bit int.

https://github.com/llvm/llvm-project/blob/main/clang/utils/TableGen/ClangAttrEmitter.cpp#L532

clang/lib/CodeGen/CGBuilder.h

164

I think we can preserve non-null here. We just can't assume that KnownNonNull indicates the pointer is non-zero when the address space isn't the default address space.

[ahatanak]

clang/lib/CodeGen/CGExpr.cpp

1723

KnownNonNull is being passed here because the address of a load must be non-null. Do we have to check that the function doesn't have null_pointer_is_valid here?

[ahatanak]

• Set the KnownNonNull bit correctly when Address::withAlignment is called.
• Pass NotKnownNonNull to the call to Addr.withPointer when the address of thread local storage is created.

[ahatanak]

clang/lib/CodeGen/CGExpr.cpp

1723

I don't think we should pass KnownNonNull unless we know for sure the address of the global variable is non-null, which isn't always true (for example, when it's extern weak).

We can set KnownNonNull below this line if we know NullPointerIsValid is false.

[efriedma]

LGTM

clang/lib/CodeGen/Address.h

68–69

Filed https://github.com/llvm/llvm-project/issues/60752 so we don't lose track of this.

clang/lib/CodeGen/CGExpr.cpp

1723

If you're assuming something is non-null based on the fact that loads from null trap, then yes, you need to check NullPointerIsValid.

If you want to see the current logic LLVM uses to figure out whether a global can be null, see evaluateICmpRelation in llvm/lib/IR/ConstantFold.cpp. But in this context, you probably don't need that.

[ahatanak]

Fix type and add comment. Remove unnecessary cast.

[ahatanak]

clang/lib/CodeGen/Address.h

68–69

I just realized we can't reduce the number of bits used for alignment here as we need 6 bits for alignment of 1 << 32.

Should we allocate additional memory when AlignLog is either 31 or 32? If the 5-bit alignment is equal to 0b11111, it would mean that there is an out-of-line storage large enough to hold the alignment and any other extra information that is needed. I think https://reviews.llvm.org/D117262#3267899 proposes a similar idea.

[efriedma]

clang/lib/CodeGen/Address.h

68–69

How much does sizeof(Address) actually matter, anyway? If it's going to get that nasty to implement the packing, I'm not sure it's worth the effort to optimize.

[ahatanak]

clang/lib/CodeGen/Address.h

68–69

I'm not sure, but apparently https://reviews.llvm.org/D117262 was needed to reduce the memory usage.

I don't see a significant increase in stack usage (a little over 1%) in the files in clang/lib/CodeGen when I build clang with -fstack-usage.

[ahatanak]

clang/lib/CodeGen/Address.h

68–69

@aeubanks do we have to use the specialization AddressImpl<T, true>?

https://github.com/llvm/llvm-project/blob/main/clang/lib/CodeGen/Address.h#L31

[aeubanks]

clang/lib/CodeGen/Address.h

68–69

looking back at the internal bug motivating the patch, it was because some we had to lower some thresholds in a stress test, so that in it of itself isn't super important. but Address is used a lot and it's nice to keep resource usage down (1% isn't trivial). does this noticeably affect max RSS? could run a potential patch through llvm-compile-time-tracker (I can help with that) to see the impact

[ahatanak]

clang/lib/CodeGen/Address.h

68–69

Note that 1% is the increase in the files in clang/lib/CodeGen. The files in other directories (e.g., clang/lib/Sema) aren't affected by the change as they don't use Address.

I see an increase of 0.33% in max RSS when I run the tests in clang/test/CodeGen*, but I'm not sure how reliable that is.

How do you run the potential patch through llvm-compile-time-tracker? The patch just changes the type to AddressImpl<void, false> A; on this line:

https://github.com/llvm/llvm-project/blob/main/clang/lib/CodeGen/Address.h#L92

[aeubanks]

clang/lib/CodeGen/Address.h

68–69

https://llvm-compile-time-tracker.com/compare.php?from=998ad085e865f2e5acc589d6bee0e3379042da2e&to=5de4a1989c474f37ac03f20ccb0aef50f6e3b854&stat=max-rss

seems fine to me

clang/lib/CodeGen/Address.h

 #include "clang/AST/CharUnits.h"
 #include "llvm/ADT/PointerIntPair.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/Support/MathExtras.h"
 
 namespace clang {
 namespace CodeGen {
 
+// Indicates whether a pointer is known not to be null.
+enum KnownNonNull_t { NotKnownNonNull, KnownNonNull };
+
 // We try to save some space by using 6 bits over two PointerIntPairs to store
 // the alignment. However, some arches don't support 3 bits in a PointerIntPair
 // so we fallback to storing the alignment separately.
 template <typename T, bool = alignof(llvm::Value *) >= 8> class AddressImpl {};
 
 template <typename T> class AddressImpl<T, false> {
-  llvm::Value *Pointer;
+  llvm::PointerIntPair<llvm::Value *, 1, bool> PointerAndKnownNonNull;
   llvm::Type *ElementType;
   CharUnits Alignment;
 
 public:
   AddressImpl(llvm::Value *Pointer, llvm::Type *ElementType,
-              CharUnits Alignment)
-      : Pointer(Pointer), ElementType(ElementType), Alignment(Alignment) {}
-  llvm::Value *getPointer() const { return Pointer; }
+              CharUnits Alignment, KnownNonNull_t IsKnownNonNull)
+      : PointerAndKnownNonNull(Pointer, IsKnownNonNull),
+        ElementType(ElementType), Alignment(Alignment) {}
+  llvm::Value *getPointer() const {
+    return PointerAndKnownNonNull.getPointer();
+  }
   llvm::Type *getElementType() const { return ElementType; }
   CharUnits getAlignment() const { return Alignment; }
+  KnownNonNull_t isKnownNonNull() const {
+    return (KnownNonNull_t)PointerAndKnownNonNull.getInt();
+  }
+  void setKnownNonNull() { PointerAndKnownNonNull.setInt(true); }
 };
 
 template <typename T> class AddressImpl<T, true> {
-  // Int portion stores upper 3 bits of the log of the alignment.
+  // Int portion stores the non-null bit and the upper 2 bits of the log of the
+  // alignment.
   llvm::PointerIntPair<llvm::Value *, 3, unsigned> Pointer;
   // Int portion stores lower 3 bits of the log of the alignment.
   llvm::PointerIntPair<llvm::Type *, 3, unsigned> ElementType;
 
 public:
   AddressImpl(llvm::Value *Pointer, llvm::Type *ElementType,
-              CharUnits Alignment)
+              CharUnits Alignment, KnownNonNull_t IsKnownNonNull)
       : Pointer(Pointer), ElementType(ElementType) {
-    if (Alignment.isZero())
+    if (Alignment.isZero()) {
+      this->Pointer.setInt(IsKnownNonNull << 2);
       return;
-    // Currently the max supported alignment is much less than 1 << 63 and is
+    }
+    // Currently the max supported alignment is exactly 1 << 32 and is
     // guaranteed to be a power of 2, so we can store the log of the alignment
-    // into 6 bits.
+    // into 5 bits.
     assert(Alignment.isPowerOfTwo() && "Alignment cannot be zero");
     auto AlignLog = llvm::Log2_64(Alignment.getQuantity());
-    assert(AlignLog < (1 << 6) && "cannot fit alignment into 6 bits");
-    this->Pointer.setInt(AlignLog >> 3);
+    assert(AlignLog < (1 << 5) && "cannot fit alignment into 5 bits");
+    this->Pointer.setInt(IsKnownNonNull << 2 | AlignLog >> 3);
     this->ElementType.setInt(AlignLog & 7);
   }
   llvm::Value *getPointer() const { return Pointer.getPointer(); }
   llvm::Type *getElementType() const { return ElementType.getPointer(); }
   CharUnits getAlignment() const {
-    unsigned AlignLog = (Pointer.getInt() << 3) | ElementType.getInt();
+    unsigned AlignLog = ((Pointer.getInt() & 0x3) << 3) | ElementType.getInt();
     return CharUnits::fromQuantity(CharUnits::QuantityType(1) << AlignLog);
   }
+  KnownNonNull_t isKnownNonNull() const {
+    return (KnownNonNull_t)(!!(Pointer.getInt() & 0x4));
+  }
+  void setKnownNonNull() { Pointer.setInt(Pointer.getInt() | 0x4); }
 };
 
 /// An aligned address.
 class Address {
   AddressImpl<void> A;
 
 protected:
-  Address(std::nullptr_t) : A(nullptr, nullptr, CharUnits::Zero()) {}
+  Address(std::nullptr_t)
+      : A(nullptr, nullptr, CharUnits::Zero(), NotKnownNonNull) {}
 
 public:
-  Address(llvm::Value *Pointer, llvm::Type *ElementType, CharUnits Alignment)
-      : A(Pointer, ElementType, Alignment) {
+  Address(llvm::Value *Pointer, llvm::Type *ElementType, CharUnits Alignment,
+          KnownNonNull_t IsKnownNonNull = NotKnownNonNull)
+      : A(Pointer, ElementType, Alignment, IsKnownNonNull) {
     assert(Pointer != nullptr && "Pointer cannot be null");
     assert(ElementType != nullptr && "Element type cannot be null");
     assert(llvm::cast<llvm::PointerType>(Pointer->getType())
                ->isOpaqueOrPointeeTypeMatches(ElementType) &&
            "Incorrect pointer element type");
   }
 
   static Address invalid() { return Address(nullptr); }
   bool isValid() const { return A.getPointer() != nullptr; }
 
   llvm::Value *getPointer() const {
     assert(isValid());
     return A.getPointer();
   }
 
   /// Return the type of the pointer value.
   llvm::PointerType *getType() const {
     return llvm::cast<llvm::PointerType>(getPointer()->getType());
   }
 
public:
   /// Return the alignment of this pointer.
   CharUnits getAlignment() const {
     assert(isValid());
     return A.getAlignment();
   }
 
   /// Return address with different pointer, but same element type and
   /// alignment.
-  Address withPointer(llvm::Value *NewPointer) const {
-    return Address(NewPointer, getElementType(), getAlignment());
+  Address withPointer(llvm::Value *NewPointer,
+                      KnownNonNull_t IsKnownNonNull) const {
+    return Address(NewPointer, getElementType(), getAlignment(),
+                   IsKnownNonNull);
   }
 
   /// Return address with different alignment, but same pointer and element
   /// type.
   Address withAlignment(CharUnits NewAlignment) const {
     return Address(getPointer(), getElementType(), NewAlignment);
   }
+
+  /// Whether the pointer is known not to be null.
+  KnownNonNull_t isKnownNonNull() const {
+    assert(isValid());
+    return A.isKnownNonNull();
+  }
+
+  /// Set the non-null bit.
+  Address setKnownNonNull() {
+    assert(isValid());
+    A.setKnownNonNull();
+    return *this;
+  }
 };
 
 /// A specialization of Address that requires the address to be an
 /// LLVM Constant.
 class ConstantAddress : public Address {
   ConstantAddress(std::nullptr_t) : Address(nullptr) {}
 
 public:

clang/lib/CodeGen/CGBuilder.h

public:
   }
 
   using CGBuilderBaseTy::CreateAddrSpaceCast;
   Address CreateAddrSpaceCast(Address Addr, llvm::Type *Ty,
                               const llvm::Twine &Name = "") {
     assert(cast<llvm::PointerType>(Ty)->isOpaqueOrPointeeTypeMatches(
                Addr.getElementType()) &&
            "Should not change the element type");
-    return Addr.withPointer(CreateAddrSpaceCast(Addr.getPointer(), Ty, Name));
+    return Addr.withPointer(CreateAddrSpaceCast(Addr.getPointer(), Ty, Name),
+                            Addr.isKnownNonNull());
   }
 
   /// Cast the element type of the given address to a different type,
   /// preserving information like the alignment and address space.
   Address CreateElementBitCast(Address Addr, llvm::Type *Ty,
                                const llvm::Twine &Name = "") {
     auto *PtrTy = Ty->getPointerTo(Addr.getAddressSpace());
     return Address(CreateBitCast(Addr.getPointer(), PtrTy, Name), Ty,
-                   Addr.getAlignment());
+                   Addr.getAlignment(), Addr.isKnownNonNull());
   }
 
   using CGBuilderBaseTy::CreatePointerBitCastOrAddrSpaceCast;
   Address CreatePointerBitCastOrAddrSpaceCast(Address Addr, llvm::Type *Ty,
                                               llvm::Type *ElementTy,
                                               const llvm::Twine &Name = "") {
     llvm::Value *Ptr =
         CreatePointerBitCastOrAddrSpaceCast(Addr.getPointer(), Ty, Name);
-    return Address(Ptr, ElementTy, Addr.getAlignment());
+    return Address(Ptr, ElementTy, Addr.getAlignment(), Addr.isKnownNonNull());
   }
 
   /// Given
   ///   %addr = {T1, T2...}* ...
   /// produce
   ///   %name = getelementptr inbounds %addr, i32 0, i32 index
   ///
   /// This API assumes that drilling into a struct like this is always an
   /// inbounds operation.
   using CGBuilderBaseTy::CreateStructGEP;
   Address CreateStructGEP(Address Addr, unsigned Index,
                           const llvm::Twine &Name = "") {
     llvm::StructType *ElTy = cast<llvm::StructType>(Addr.getElementType());
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
     const llvm::StructLayout *Layout = DL.getStructLayout(ElTy);
     auto Offset = CharUnits::fromQuantity(Layout->getElementOffset(Index));
 
     return Address(
         CreateStructGEP(Addr.getElementType(), Addr.getPointer(), Index, Name),
         ElTy->getElementType(Index),
-        Addr.getAlignment().alignmentAtOffset(Offset));
+        Addr.getAlignment().alignmentAtOffset(Offset), Addr.isKnownNonNull());
   }
 
   /// Given
   ///   %addr = [n x T]* ...
   /// produce
   ///   %name = getelementptr inbounds %addr, i64 0, i64 index
   /// where i64 is actually the target word size.
   ///
   /// This API assumes that drilling into an array like this is always
   /// an inbounds operation.
   Address CreateConstArrayGEP(Address Addr, uint64_t Index,
                               const llvm::Twine &Name = "") {
     llvm::ArrayType *ElTy = cast<llvm::ArrayType>(Addr.getElementType());
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
     CharUnits EltSize =
         CharUnits::fromQuantity(DL.getTypeAllocSize(ElTy->getElementType()));
 
     return Address(
         CreateInBoundsGEP(Addr.getElementType(), Addr.getPointer(),
                           {getSize(CharUnits::Zero()), getSize(Index)}, Name),
         ElTy->getElementType(),
-        Addr.getAlignment().alignmentAtOffset(Index * EltSize));
+        Addr.getAlignment().alignmentAtOffset(Index * EltSize),
+        Addr.isKnownNonNull());
   }
 
   /// Given
   ///   %addr = T* ...
   /// produce
   ///   %name = getelementptr inbounds %addr, i64 index
   /// where i64 is actually the target word size.
   Address CreateConstInBoundsGEP(Address Addr, uint64_t Index,
                                  const llvm::Twine &Name = "") {
     llvm::Type *ElTy = Addr.getElementType();
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
     CharUnits EltSize = CharUnits::fromQuantity(DL.getTypeAllocSize(ElTy));
 
     return Address(CreateInBoundsGEP(Addr.getElementType(), Addr.getPointer(),
                                      getSize(Index), Name),
-                   ElTy,
-                   Addr.getAlignment().alignmentAtOffset(Index * EltSize));
+                   ElTy, Addr.getAlignment().alignmentAtOffset(Index * EltSize),
+                   Addr.isKnownNonNull());
   }
 
   /// Given
   ///   %addr = T* ...
   /// produce
   ///   %name = getelementptr inbounds %addr, i64 index
   /// where i64 is actually the target word size.
   Address CreateConstGEP(Address Addr, uint64_t Index,
                          const llvm::Twine &Name = "") {
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
     CharUnits EltSize =
         CharUnits::fromQuantity(DL.getTypeAllocSize(Addr.getElementType()));
 
     return Address(CreateGEP(Addr.getElementType(), Addr.getPointer(),
                              getSize(Index), Name),
                    Addr.getElementType(),
-                   Addr.getAlignment().alignmentAtOffset(Index * EltSize));
+                   Addr.getAlignment().alignmentAtOffset(Index * EltSize),
+                   NotKnownNonNull);
   }
 
   /// Create GEP with single dynamic index. The address alignment is reduced
   /// according to the element size.
   using CGBuilderBaseTy::CreateGEP;
   Address CreateGEP(Address Addr, llvm::Value *Index,
                     const llvm::Twine &Name = "") {
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
     CharUnits EltSize =
         CharUnits::fromQuantity(DL.getTypeAllocSize(Addr.getElementType()));
 
     return Address(
         CreateGEP(Addr.getElementType(), Addr.getPointer(), Index, Name),
         Addr.getElementType(),
-        Addr.getAlignment().alignmentOfArrayElement(EltSize));
+        Addr.getAlignment().alignmentOfArrayElement(EltSize), NotKnownNonNull);
   }
 
   /// Given a pointer to i8, adjust it by a given constant offset.
   Address CreateConstInBoundsByteGEP(Address Addr, CharUnits Offset,
                                      const llvm::Twine &Name = "") {
     assert(Addr.getElementType() == TypeCache.Int8Ty);
     return Address(CreateInBoundsGEP(Addr.getElementType(), Addr.getPointer(),
                                      getSize(Offset), Name),
                    Addr.getElementType(),
-                   Addr.getAlignment().alignmentAtOffset(Offset));
+                   Addr.getAlignment().alignmentAtOffset(Offset),
+                   Addr.isKnownNonNull());
   }
   Address CreateConstByteGEP(Address Addr, CharUnits Offset,
                              const llvm::Twine &Name = "") {
     assert(Addr.getElementType() == TypeCache.Int8Ty);
     return Address(CreateGEP(Addr.getElementType(), Addr.getPointer(),
                              getSize(Offset), Name),
                    Addr.getElementType(),
-                   Addr.getAlignment().alignmentAtOffset(Offset));
+                   Addr.getAlignment().alignmentAtOffset(Offset),
+                   NotKnownNonNull);
   }
 
   using CGBuilderBaseTy::CreateConstInBoundsGEP2_32;
   Address CreateConstInBoundsGEP2_32(Address Addr, unsigned Idx0, unsigned Idx1,
                                      const llvm::Twine &Name = "") {
     const llvm::DataLayout &DL = BB->getParent()->getParent()->getDataLayout();
 
     auto *GEP = cast<llvm::GetElementPtrInst>(CreateConstInBoundsGEP2_32(
         Addr.getElementType(), Addr.getPointer(), Idx0, Idx1, Name));
     llvm::APInt Offset(
         DL.getIndexSizeInBits(Addr.getType()->getPointerAddressSpace()), 0,
         /*isSigned=*/true);
     if (!GEP->accumulateConstantOffset(DL, Offset))
       llvm_unreachable("offset of GEP with constants is always computable");
     return Address(GEP, GEP->getResultElementType(),
                    Addr.getAlignment().alignmentAtOffset(
-                       CharUnits::fromQuantity(Offset.getSExtValue())));
+                       CharUnits::fromQuantity(Offset.getSExtValue())),
+                   Addr.isKnownNonNull());
   }
 
   using CGBuilderBaseTy::CreateMemCpy;
   llvm::CallInst *CreateMemCpy(Address Dest, Address Src, llvm::Value *Size,
                                bool IsVolatile = false) {
     return CreateMemCpy(Dest.getPointer(), Dest.getAlignment().getAsAlign(),
                         Src.getPointer(), Src.getAlignment().getAsAlign(), Size,
                         IsVolatile);
Address CreatePreserveStructAccessIndex(Address Addr, unsigned Index,
     return Address(CreatePreserveStructAccessIndex(ElTy, Addr.getPointer(),
                                                    Index, FieldIndex, DbgInfo),
                    ElTy->getElementType(Index),
                    Addr.getAlignment().alignmentAtOffset(Offset));
   }
 
   using CGBuilderBaseTy::CreateLaunderInvariantGroup;
   Address CreateLaunderInvariantGroup(Address Addr) {
-    return Addr.withPointer(CreateLaunderInvariantGroup(Addr.getPointer()));
+    return Addr.withPointer(CreateLaunderInvariantGroup(Addr.getPointer()),
+                            Addr.isKnownNonNull());
   }
 };
 
 } // end namespace CodeGen
 } // end namespace clang
 
 #endif

clang/lib/CodeGen/CGCall.cpp

case ABIArgInfo::InAlloca: {
       ArgVals.push_back(ParamValue::forIndirect(V));
       break;
     }
 
     case ABIArgInfo::Indirect:
     case ABIArgInfo::IndirectAliased: {
       assert(NumIRArgs == 1);
       Address ParamAddr = Address(Fn->getArg(FirstIRArg), ConvertTypeForMem(Ty),
-                                  ArgI.getIndirectAlign());
+                                  ArgI.getIndirectAlign(), KnownNonNull);
 
       if (!hasScalarEvaluationKind(Ty)) {
         // Aggregates and complex variables are accessed by reference. All we
         // need to do is realign the value, if requested. Also, if the address
         // may be aliased, copy it to ensure that the parameter variable is
         // mutable and has a unique adress, as C requires.
         Address V = ParamAddr;
         if (ArgI.getIndirectRealign() || ArgI.isIndirectAliased()) {

clang/lib/CodeGen/CGClass.cpp

Address CodeGenFunction::LoadCXXThisAddress() {
   if (CXXThisAlignment.isZero()) {
     // Just use the best known alignment for the parent.
     // TODO: if we're currently emitting a complete-object ctor/dtor,
     // we can always use the complete-object alignment.
     CXXThisAlignment = CGM.getClassPointerAlignment(MD->getParent());
   }
 
   llvm::Type *Ty = ConvertType(MD->getThisType()->getPointeeType());
-  return Address(LoadCXXThis(), Ty, CXXThisAlignment);
+  return Address(LoadCXXThis(), Ty, CXXThisAlignment, KnownNonNull);
 }
 
 /// Emit the address of a field using a member data pointer.
 ///
 /// \param E Only used for emergency diagnostics
 Address
 CodeGenFunction::EmitCXXMemberDataPointerAddress(const Expr *E, Address base,
                                                  llvm::Value *memberPtr,
Address CodeGenFunction::GetAddressOfBaseClass(
   if (NullCheckValue) {
     llvm::BasicBlock *notNullBB = Builder.GetInsertBlock();
     Builder.CreateBr(endBB);
     EmitBlock(endBB);
 
     llvm::PHINode *PHI = Builder.CreatePHI(BasePtrTy, 2, "cast.result");
     PHI->addIncoming(Value.getPointer(), notNullBB);
     PHI->addIncoming(llvm::Constant::getNullValue(BasePtrTy), origBB);
-    Value = Value.withPointer(PHI);
+    Value = Value.withPointer(PHI, NotKnownNonNull);
   }
 
   return Value;
 }
 
 Address
 CodeGenFunction::GetAddressOfDerivedClass(Address BaseAddr,
                                           const CXXRecordDecl *Derived,

clang/lib/CodeGen/CGDecl.cpp

if (Arg.isIndirect()) {
     auto SrcLangAS = getLangOpts().OpenCL ? LangAS::opencl_private : AllocaAS;
     auto DestLangAS =
         getLangOpts().OpenCL ? LangAS::opencl_private : LangAS::Default;
     if (SrcLangAS != DestLangAS) {
       assert(getContext().getTargetAddressSpace(SrcLangAS) ==
              CGM.getDataLayout().getAllocaAddrSpace());
       auto DestAS = getContext().getTargetAddressSpace(DestLangAS);
       auto *T = DeclPtr.getElementType()->getPointerTo(DestAS);
-      DeclPtr = DeclPtr.withPointer(getTargetHooks().performAddrSpaceCast(
-          *this, V, SrcLangAS, DestLangAS, T, true));
+      DeclPtr =
+          DeclPtr.withPointer(getTargetHooks().performAddrSpaceCast(
+                                  *this, V, SrcLangAS, DestLangAS, T, true),
+                              DeclPtr.isKnownNonNull());
     }
 
     // Push a destructor cleanup for this parameter if the ABI requires it.
     // Don't push a cleanup in a thunk for a method that will also emit a
     // cleanup.
     if (Ty->isRecordType() && !CurFuncIsThunk &&
         Ty->castAs<RecordType>()->getDecl()->isParamDestroyedInCallee()) {
       if (QualType::DestructionKind DtorKind =

clang/lib/CodeGen/CGException.cpp

if (auto *ParentAlloca = dyn_cast<llvm::AllocaInst>(ParentVar.getPointer())) {
     RecoverCall->setArgOperand(1, ParentFP);
     RecoverCall->insertBefore(AllocaInsertPt);
   }
 
   // Bitcast the variable, rename it, and insert it in the local decl map.
   llvm::Value *ChildVar =
       Builder.CreateBitCast(RecoverCall, ParentVar.getType());
   ChildVar->setName(ParentVar.getName());
-  return ParentVar.withPointer(ChildVar);
+  return ParentVar.withPointer(ChildVar, KnownNonNull);
 }
 
 void CodeGenFunction::EmitCapturedLocals(CodeGenFunction &ParentCGF,
                                          const Stmt *OutlinedStmt,
                                          bool IsFilter) {
   // Find all captures in the Stmt.
   CaptureFinder Finder(ParentCGF, ParentCGF.CXXABIThisDecl);
   Finder.Visit(OutlinedStmt);

clang/lib/CodeGen/CGExpr.cpp

 /// CreateTempAlloca - This creates a alloca and inserts it into the entry
 /// block.
 Address CodeGenFunction::CreateTempAllocaWithoutCast(llvm::Type *Ty,
                                                      CharUnits Align,
                                                      const Twine &Name,
                                                      llvm::Value *ArraySize) {
   auto Alloca = CreateTempAlloca(Ty, Name, ArraySize);
   Alloca->setAlignment(Align.getAsAlign());
-  return Address(Alloca, Ty, Align);
+  return Address(Alloca, Ty, Align, KnownNonNull);
 }
 
 /// CreateTempAlloca - This creates a alloca and inserts it into the entry
 /// block. The alloca is casted to default address space if necessary.
 Address CodeGenFunction::CreateTempAlloca(llvm::Type *Ty, CharUnits Align,
                                           const Twine &Name,
                                           llvm::Value *ArraySize,
                                           Address *AllocaAddr) {
if (getASTAllocaAddressSpace() != LangAS::Default) {
     // builder.
     if (!ArraySize)
       Builder.SetInsertPoint(getPostAllocaInsertPoint());
     V = getTargetHooks().performAddrSpaceCast(
         *this, V, getASTAllocaAddressSpace(), LangAS::Default,
         Ty->getPointerTo(DestAddrSpace), /*non-null*/ true);
   }
 
-  return Address(V, Ty, Align);
+  return Address(V, Ty, Align, KnownNonNull);
 }
 
 /// CreateTempAlloca - This creates an alloca and inserts it into the entry
 /// block if \p ArraySize is nullptr, otherwise inserts it at the current
 /// insertion point of the builder.
 llvm::AllocaInst *CodeGenFunction::CreateTempAlloca(llvm::Type *Ty,
                                                     const Twine &Name,
                                                     llvm::Value *ArraySize) {
Address CodeGenFunction::CreateMemTemp(QualType Ty, CharUnits Align,
 
   if (Ty->isConstantMatrixType()) {
     auto *ArrayTy = cast<llvm::ArrayType>(Result.getElementType());
     auto *VectorTy = llvm::FixedVectorType::get(ArrayTy->getElementType(),
                                                 ArrayTy->getNumElements());
 
     Result = Address(
         Builder.CreateBitCast(Result.getPointer(), VectorTy->getPointerTo()),
-        VectorTy, Result.getAlignment());
+        VectorTy, Result.getAlignment(), KnownNonNull);
   }
   return Result;
 }
 
 Address CodeGenFunction::CreateMemTempWithoutCast(QualType Ty, CharUnits Align,
                                                   const Twine &Name) {
   return CreateTempAllocaWithoutCast(ConvertTypeForMem(Ty), Align, Name);
 }
void CodeGenModule::EmitExplicitCastExprType(const ExplicitCastExpr *E,
   if (CGDebugInfo *DI = getModuleDebugInfo())
     DI->EmitExplicitCastType(E->getType());
 }
 
 //===----------------------------------------------------------------------===//
 //                         LValue Expression Emission
 //===----------------------------------------------------------------------===//
 
-/// EmitPointerWithAlignment - Given an expression of pointer type, try to
-/// derive a more accurate bound on the alignment of the pointer.
-Address CodeGenFunction::EmitPointerWithAlignment(const Expr *E,
-                                                  LValueBaseInfo *BaseInfo,
-                                                  TBAAAccessInfo *TBAAInfo) {
+static Address EmitPointerWithAlignment(const Expr *E, LValueBaseInfo *BaseInfo,
+                                        TBAAAccessInfo *TBAAInfo,
+                                        KnownNonNull_t IsKnownNonNull,
+                                        CodeGenFunction &CGF) {
   // We allow this with ObjC object pointers because of fragile ABIs.
   assert(E->getType()->isPointerType() ||
          E->getType()->isObjCObjectPointerType());
   E = E->IgnoreParens();
 
   // Casts:
   if (const CastExpr *CE = dyn_cast<CastExpr>(E)) {
     if (const auto *ECE = dyn_cast<ExplicitCastExpr>(CE))
-      CGM.EmitExplicitCastExprType(ECE, this);
+      CGF.CGM.EmitExplicitCastExprType(ECE, &CGF);
 
     switch (CE->getCastKind()) {
     // Non-converting casts (but not C's implicit conversion from void*).
     case CK_BitCast:
     case CK_NoOp:
     case CK_AddressSpaceConversion:
       if (auto PtrTy = CE->getSubExpr()->getType()->getAs<PointerType>()) {
         if (PtrTy->getPointeeType()->isVoidType())
           break;
 
         LValueBaseInfo InnerBaseInfo;
         TBAAAccessInfo InnerTBAAInfo;
-        Address Addr = EmitPointerWithAlignment(CE->getSubExpr(),
-                                                &InnerBaseInfo,
-                                                &InnerTBAAInfo);
+        Address Addr = CGF.EmitPointerWithAlignment(
+            CE->getSubExpr(), &InnerBaseInfo, &InnerTBAAInfo, IsKnownNonNull);
         if (BaseInfo) *BaseInfo = InnerBaseInfo;
         if (TBAAInfo) *TBAAInfo = InnerTBAAInfo;
 
         if (isa<ExplicitCastExpr>(CE)) {
           LValueBaseInfo TargetTypeBaseInfo;
           TBAAAccessInfo TargetTypeTBAAInfo;
-          CharUnits Align = CGM.getNaturalPointeeTypeAlignment(
+          CharUnits Align = CGF.CGM.getNaturalPointeeTypeAlignment(
               E->getType(), &TargetTypeBaseInfo, &TargetTypeTBAAInfo);
           if (TBAAInfo)
-            *TBAAInfo = CGM.mergeTBAAInfoForCast(*TBAAInfo,
-                                                 TargetTypeTBAAInfo);
+            *TBAAInfo =
+                CGF.CGM.mergeTBAAInfoForCast(*TBAAInfo, TargetTypeTBAAInfo);
           // If the source l-value is opaque, honor the alignment of the
           // casted-to type.
           if (InnerBaseInfo.getAlignmentSource() != AlignmentSource::Decl) {
             if (BaseInfo)
               BaseInfo->mergeForCast(TargetTypeBaseInfo);
-            Addr = Address(Addr.getPointer(), Addr.getElementType(), Align);
+            Addr = Address(Addr.getPointer(), Addr.getElementType(), Align,
+                           IsKnownNonNull);
           }
         }
 
-        if (SanOpts.has(SanitizerKind::CFIUnrelatedCast) &&
+        if (CGF.SanOpts.has(SanitizerKind::CFIUnrelatedCast) &&
             CE->getCastKind() == CK_BitCast) {
           if (auto PT = E->getType()->getAs<PointerType>())
-            EmitVTablePtrCheckForCast(PT->getPointeeType(), Addr,
+            CGF.EmitVTablePtrCheckForCast(PT->getPointeeType(), Addr,
                                           /*MayBeNull=*/true,
                                           CodeGenFunction::CFITCK_UnrelatedCast,
                                           CE->getBeginLoc());
         }
 
-        llvm::Type *ElemTy = ConvertTypeForMem(E->getType()->getPointeeType());
-        Addr = Builder.CreateElementBitCast(Addr, ElemTy);
+        llvm::Type *ElemTy =
+            CGF.ConvertTypeForMem(E->getType()->getPointeeType());
+        Addr = CGF.Builder.CreateElementBitCast(Addr, ElemTy);
         if (CE->getCastKind() == CK_AddressSpaceConversion)
-          Addr = Builder.CreateAddrSpaceCast(Addr, ConvertType(E->getType()));
+          Addr = CGF.Builder.CreateAddrSpaceCast(Addr,
+                                                 CGF.ConvertType(E->getType()));
         return Addr;
       }
       break;
 
     // Array-to-pointer decay.
     case CK_ArrayToPointerDecay:
-      return EmitArrayToPointerDecay(CE->getSubExpr(), BaseInfo, TBAAInfo);
+      return CGF.EmitArrayToPointerDecay(CE->getSubExpr(), BaseInfo, TBAAInfo);
 
     // Derived-to-base conversions.
     case CK_UncheckedDerivedToBase:
     case CK_DerivedToBase: {
       // TODO: Support accesses to members of base classes in TBAA. For now, we
       // conservatively pretend that the complete object is of the base class
       // type.
       if (TBAAInfo)
-        *TBAAInfo = CGM.getTBAAAccessInfo(E->getType());
-      Address Addr = EmitPointerWithAlignment(CE->getSubExpr(), BaseInfo);
+        *TBAAInfo = CGF.CGM.getTBAAAccessInfo(E->getType());
+      Address Addr = CGF.EmitPointerWithAlignment(
+          CE->getSubExpr(), BaseInfo, nullptr,
+          (KnownNonNull_t)(IsKnownNonNull ||
+                           CE->getCastKind() == CK_UncheckedDerivedToBase));
       auto Derived = CE->getSubExpr()->getType()->getPointeeCXXRecordDecl();
-      return GetAddressOfBaseClass(Addr, Derived,
-                                   CE->path_begin(), CE->path_end(),
-                                   ShouldNullCheckClassCastValue(CE),
-                                   CE->getExprLoc());
+      return CGF.GetAddressOfBaseClass(
+          Addr, Derived, CE->path_begin(), CE->path_end(),
+          CGF.ShouldNullCheckClassCastValue(CE), CE->getExprLoc());
     }
 
     // TODO: Is there any reason to treat base-to-derived conversions
     // specially?
     default:
       break;
     }
   }
 
   // Unary &.
   if (const UnaryOperator *UO = dyn_cast<UnaryOperator>(E)) {
     if (UO->getOpcode() == UO_AddrOf) {
-      LValue LV = EmitLValue(UO->getSubExpr());
+      LValue LV = CGF.EmitLValue(UO->getSubExpr(), IsKnownNonNull);
       if (BaseInfo) *BaseInfo = LV.getBaseInfo();
       if (TBAAInfo) *TBAAInfo = LV.getTBAAInfo();
-      return LV.getAddress(*this);
+      return LV.getAddress(CGF);
     }
   }
 
   // std::addressof and variants.
   if (auto *Call = dyn_cast<CallExpr>(E)) {
     switch (Call->getBuiltinCallee()) {
     default:
       break;
     case Builtin::BIaddressof:
     case Builtin::BI__addressof:
     case Builtin::BI__builtin_addressof: {
-      LValue LV = EmitLValue(Call->getArg(0));
+      LValue LV = CGF.EmitLValue(Call->getArg(0), IsKnownNonNull);
       if (BaseInfo) *BaseInfo = LV.getBaseInfo();
       if (TBAAInfo) *TBAAInfo = LV.getTBAAInfo();
-      return LV.getAddress(*this);
+      return LV.getAddress(CGF);
     }
     }
   }
 
   // TODO: conditional operators, comma.
 
   // Otherwise, use the alignment of the type.
   CharUnits Align =
-      CGM.getNaturalPointeeTypeAlignment(E->getType(), BaseInfo, TBAAInfo);
-  llvm::Type *ElemTy = ConvertTypeForMem(E->getType()->getPointeeType());
-  return Address(EmitScalarExpr(E), ElemTy, Align);
+      CGF.CGM.getNaturalPointeeTypeAlignment(E->getType(), BaseInfo, TBAAInfo);
+  llvm::Type *ElemTy = CGF.ConvertTypeForMem(E->getType()->getPointeeType());
+  return Address(CGF.EmitScalarExpr(E), ElemTy, Align, IsKnownNonNull);
+}
+
+/// EmitPointerWithAlignment - Given an expression of pointer type, try to
+/// derive a more accurate bound on the alignment of the pointer.
+Address CodeGenFunction::EmitPointerWithAlignment(
+    const Expr *E, LValueBaseInfo *BaseInfo, TBAAAccessInfo *TBAAInfo,
+    KnownNonNull_t IsKnownNonNull) {
+  Address Addr =
+      ::EmitPointerWithAlignment(E, BaseInfo, TBAAInfo, IsKnownNonNull, *this);
+  if (IsKnownNonNull && !Addr.isKnownNonNull())
+    Addr.setKnownNonNull();
+  return Addr;
 }
 
 llvm::Value *CodeGenFunction::EmitNonNullRValueCheck(RValue RV, QualType T) {
   llvm::Value *V = RV.getScalarVal();
   if (auto MPT = T->getAs<MemberPointerType>())
     return CGM.getCXXABI().EmitMemberPointerIsNotNull(*this, V, MPT);
   return Builder.CreateICmpNE(V, llvm::Constant::getNullValue(V->getType()));
 }

 /// If this returns a bitfield reference, nothing about the pointee type of the
 /// LLVM value is known: For example, it may not be a pointer to an integer.
 ///
 /// If this returns a normal address, and if the lvalue's C type is fixed size,
 /// this method guarantees that the returned pointer type will point to an LLVM
 /// type of the same size of the lvalue's type.  If the lvalue has a variable
 /// length type, this is not possible.
 ///
-LValue CodeGenFunction::EmitLValue(const Expr *E) {
+LValue CodeGenFunction::EmitLValue(const Expr *E,
+                                   KnownNonNull_t IsKnownNonNull) {
+  LValue LV = EmitLValueHelper(E, IsKnownNonNull);
+  if (IsKnownNonNull && !LV.isKnownNonNull())
+    LV.setKnownNonNull();
+  return LV;
+}
+
+LValue CodeGenFunction::EmitLValueHelper(const Expr *E,
+                                         KnownNonNull_t IsKnownNonNull) {
   ApplyDebugLocation DL(*this, E);
   switch (E->getStmtClass()) {
   default: return EmitUnsupportedLValue(E, "l-value expression");
 
   case Expr::ObjCPropertyRefExprClass:
     llvm_unreachable("cannot emit a property reference directly");
 
   case Expr::ObjCSelectorExprClass:
case Expr::CompoundAssignOperatorClass: {
     return EmitComplexCompoundAssignmentLValue(cast<CompoundAssignOperator>(E));
   }
   case Expr::CallExprClass:
   case Expr::CXXMemberCallExprClass:
   case Expr::CXXOperatorCallExprClass:
   case Expr::UserDefinedLiteralClass:
     return EmitCallExprLValue(cast<CallExpr>(E));
   case Expr::CXXRewrittenBinaryOperatorClass:
-    return EmitLValue(cast<CXXRewrittenBinaryOperator>(E)->getSemanticForm());
+    return EmitLValue(cast<CXXRewrittenBinaryOperator>(E)->getSemanticForm(),
+                      IsKnownNonNull);
   case Expr::VAArgExprClass:
     return EmitVAArgExprLValue(cast<VAArgExpr>(E));
   case Expr::DeclRefExprClass:
     return EmitDeclRefLValue(cast<DeclRefExpr>(E));
   case Expr::ConstantExprClass: {
     const ConstantExpr *CE = cast<ConstantExpr>(E);
     if (llvm::Value *Result = ConstantEmitter(*this).tryEmitConstantExpr(CE)) {
       QualType RetType = cast<CallExpr>(CE->getSubExpr()->IgnoreImplicit())
                              ->getCallReturnType(getContext())
                              ->getPointeeType();
       return MakeNaturalAlignAddrLValue(Result, RetType);
     }
-    return EmitLValue(cast<ConstantExpr>(E)->getSubExpr());
+    return EmitLValue(cast<ConstantExpr>(E)->getSubExpr(), IsKnownNonNull);
   }
   case Expr::ParenExprClass:
-    return EmitLValue(cast<ParenExpr>(E)->getSubExpr());
+    return EmitLValue(cast<ParenExpr>(E)->getSubExpr(), IsKnownNonNull);
   case Expr::GenericSelectionExprClass:
-    return EmitLValue(cast<GenericSelectionExpr>(E)->getResultExpr());
+    return EmitLValue(cast<GenericSelectionExpr>(E)->getResultExpr(),
+                      IsKnownNonNull);
   case Expr::PredefinedExprClass:
     return EmitPredefinedLValue(cast<PredefinedExpr>(E));
   case Expr::StringLiteralClass:
     return EmitStringLiteralLValue(cast<StringLiteral>(E));
   case Expr::ObjCEncodeExprClass:
     return EmitObjCEncodeExprLValue(cast<ObjCEncodeExpr>(E));
   case Expr::PseudoObjectExprClass:
     return EmitPseudoObjectLValue(cast<PseudoObjectExpr>(E));
   case Expr::InitListExprClass:
     return EmitInitListLValue(cast<InitListExpr>(E));
   case Expr::CXXTemporaryObjectExprClass:
   case Expr::CXXConstructExprClass:
     return EmitCXXConstructLValue(cast<CXXConstructExpr>(E));
   case Expr::CXXBindTemporaryExprClass:
     return EmitCXXBindTemporaryLValue(cast<CXXBindTemporaryExpr>(E));
   case Expr::CXXUuidofExprClass:
     return EmitCXXUuidofLValue(cast<CXXUuidofExpr>(E));
   case Expr::LambdaExprClass:
     return EmitAggExprToLValue(E);
 
   case Expr::ExprWithCleanupsClass: {
     const auto *cleanups = cast<ExprWithCleanups>(E);
     RunCleanupsScope Scope(*this);
-    LValue LV = EmitLValue(cleanups->getSubExpr());
+    LValue LV = EmitLValue(cleanups->getSubExpr(), IsKnownNonNull);
     if (LV.isSimple()) {
       // Defend against branches out of gnu statement expressions surrounded by
       // cleanups.
       Address Addr = LV.getAddress(*this);
       llvm::Value *V = Addr.getPointer();
       Scope.ForceCleanup({&V});
-      return LValue::MakeAddr(Addr.withPointer(V), LV.getType(), getContext(),
-                              LV.getBaseInfo(), LV.getTBAAInfo());
+      return LValue::MakeAddr(Addr.withPointer(V, Addr.isKnownNonNull()),
+                              LV.getType(), getContext(), LV.getBaseInfo(),
+                              LV.getTBAAInfo());
     }
     // FIXME: Is it possible to create an ExprWithCleanups that produces a
     // bitfield lvalue or some other non-simple lvalue?
     return LV;
   }
 
   case Expr::CXXDefaultArgExprClass: {
     auto *DAE = cast<CXXDefaultArgExpr>(E);
     CXXDefaultArgExprScope Scope(*this, DAE);
-    return EmitLValue(DAE->getExpr());
+    return EmitLValue(DAE->getExpr(), IsKnownNonNull);
   }
   case Expr::CXXDefaultInitExprClass: {
     auto *DIE = cast<CXXDefaultInitExpr>(E);
     CXXDefaultInitExprScope Scope(*this, DIE);
-    return EmitLValue(DIE->getExpr());
+    return EmitLValue(DIE->getExpr(), IsKnownNonNull);
   }
   case Expr::CXXTypeidExprClass:
     return EmitCXXTypeidLValue(cast<CXXTypeidExpr>(E));
 
   case Expr::ObjCMessageExprClass:
     return EmitObjCMessageExprLValue(cast<ObjCMessageExpr>(E));
   case Expr::ObjCIvarRefExprClass:
     return EmitObjCIvarRefLValue(cast<ObjCIvarRefExpr>(E));
case Expr::MemberExprClass:
     return EmitMemberExpr(cast<MemberExpr>(E));
   case Expr::CompoundLiteralExprClass:
     return EmitCompoundLiteralLValue(cast<CompoundLiteralExpr>(E));
   case Expr::ConditionalOperatorClass:
     return EmitConditionalOperatorLValue(cast<ConditionalOperator>(E));
   case Expr::BinaryConditionalOperatorClass:
     return EmitConditionalOperatorLValue(cast<BinaryConditionalOperator>(E));
   case Expr::ChooseExprClass:
-    return EmitLValue(cast<ChooseExpr>(E)->getChosenSubExpr());
+    return EmitLValue(cast<ChooseExpr>(E)->getChosenSubExpr(), IsKnownNonNull);
   case Expr::OpaqueValueExprClass:
     return EmitOpaqueValueLValue(cast<OpaqueValueExpr>(E));
   case Expr::SubstNonTypeTemplateParmExprClass:
-    return EmitLValue(cast<SubstNonTypeTemplateParmExpr>(E)->getReplacement());
+    return EmitLValue(cast<SubstNonTypeTemplateParmExpr>(E)->getReplacement(),
+                      IsKnownNonNull);
   case Expr::ImplicitCastExprClass:
   case Expr::CStyleCastExprClass:
   case Expr::CXXFunctionalCastExprClass:
   case Expr::CXXStaticCastExprClass:
   case Expr::CXXDynamicCastExprClass:
   case Expr::CXXReinterpretCastExprClass:
   case Expr::CXXConstCastExprClass:
   case Expr::CXXAddrspaceCastExprClass:

 llvm::Value *CodeGenFunction::EmitLoadOfScalar(Address Addr, bool Volatile,
                                                QualType Ty,
                                                SourceLocation Loc,
                                                LValueBaseInfo BaseInfo,
                                                TBAAAccessInfo TBAAInfo,
                                                bool isNontemporal) {
   if (auto *GV = dyn_cast<llvm::GlobalValue>(Addr.getPointer()))
     if (GV->isThreadLocal())
-      Addr = Addr.withPointer(Builder.CreateThreadLocalAddress(GV));
+      Addr =
+          Addr.withPointer(Builder.CreateThreadLocalAddress(GV), KnownNonNull);
 
   if (const auto *ClangVecTy = Ty->getAs<VectorType>()) {
     // Boolean vectors use `iN` as storage type.
     if (ClangVecTy->isExtVectorBoolType()) {
       llvm::Type *ValTy = ConvertType(Ty);
       unsigned ValNumElems =
           cast<llvm::FixedVectorType>(ValTy)->getNumElements();
       // Load the `iP` storage object (P is the padded vector size).

 
 void CodeGenFunction::EmitStoreOfScalar(llvm::Value *Value, Address Addr,
                                         bool Volatile, QualType Ty,
                                         LValueBaseInfo BaseInfo,
                                         TBAAAccessInfo TBAAInfo,
                                         bool isInit, bool isNontemporal) {
   if (auto *GV = dyn_cast<llvm::GlobalValue>(Addr.getPointer()))
     if (GV->isThreadLocal())
-      Addr = Addr.withPointer(Builder.CreateThreadLocalAddress(GV));
+      Addr =
+          Addr.withPointer(Builder.CreateThreadLocalAddress(GV), KnownNonNull);
 
   llvm::Type *SrcTy = Value->getType();
   if (const auto *ClangVecTy = Ty->getAs<VectorType>()) {
     auto *VecTy = dyn_cast<llvm::FixedVectorType>(SrcTy);
     if (VecTy && ClangVecTy->isExtVectorBoolType()) {
       auto *MemIntTy = cast<llvm::IntegerType>(Addr.getElementType());
       // Expand to the memory bit width.
       unsigned MemNumElems = MemIntTy->getPrimitiveSizeInBits();
if (const auto *VD = dyn_cast<VarDecl>(ND)) {
 
     // No other cases for now.
     } else {
       llvm_unreachable("DeclRefExpr for Decl not entered in LocalDeclMap?");
     }
 
     // Handle threadlocal function locals.
     if (VD->getTLSKind() != VarDecl::TLS_None)
-      addr =
-          addr.withPointer(Builder.CreateThreadLocalAddress(addr.getPointer()));
+      addr = addr.withPointer(
+          Builder.CreateThreadLocalAddress(addr.getPointer()), KnownNonNull);
 
     // Check for OpenMP threadprivate variables.
     if (getLangOpts().OpenMP && !getLangOpts().OpenMPSimd &&
         VD->hasAttr<OMPThreadPrivateDeclAttr>()) {
       return EmitThreadPrivateVarDeclLValue(
           *this, VD, T, addr, getTypes().ConvertTypeForMem(VD->getType()),
           E->getExprLoc());
     }
CGCallee CodeGenFunction::EmitCallee(const Expr *E) {
   // Otherwise, we have an indirect reference.
   llvm::Value *calleePtr;
   QualType functionType;
   if (auto ptrType = E->getType()->getAs<PointerType>()) {
     calleePtr = EmitScalarExpr(E);
     functionType = ptrType->getPointeeType();
   } else {
     functionType = E->getType();
-    calleePtr = EmitLValue(E).getPointer(*this);
+    calleePtr = EmitLValue(E, KnownNonNull).getPointer(*this);
   }
   assert(functionType->isFunctionType());
 
   GlobalDecl GD;
   if (const auto *VD =
           dyn_cast_or_null<VarDecl>(E->getReferencedDeclOfCallee()))
     GD = GlobalDecl(VD);
 

clang/lib/CodeGen/CGExprCXX.cpp

CodeGenFunction::EmitCXXMemberPointerCallExpr(const CXXMemberCallExpr *E,
   const auto *MPT = MemFnExpr->getType()->castAs<MemberPointerType>();
   const auto *FPT = MPT->getPointeeType()->castAs<FunctionProtoType>();
   const auto *RD =
       cast<CXXRecordDecl>(MPT->getClass()->castAs<RecordType>()->getDecl());
 
   // Emit the 'this' pointer.
   Address This = Address::invalid();
   if (BO->getOpcode() == BO_PtrMemI)
-    This = EmitPointerWithAlignment(BaseExpr);
+    This = EmitPointerWithAlignment(BaseExpr, nullptr, nullptr, KnownNonNull);
   else
-    This = EmitLValue(BaseExpr).getAddress(*this);
+    This = EmitLValue(BaseExpr, KnownNonNull).getAddress(*this);
 
   EmitTypeCheck(TCK_MemberCall, E->getExprLoc(), This.getPointer(),
                 QualType(MPT->getClass(), 0));
 
   // Get the member function pointer.
   llvm::Value *MemFnPtr = EmitScalarExpr(MemFnExpr);
 
   // Ask the ABI to load the callee.  Note that This is modified.
void CodeGenFunction::EmitCXXDeleteExpr(const CXXDeleteExpr *E) {
   // keep the branch. This might be worth revisiting for a -O0 code size win.
   llvm::BasicBlock *DeleteNotNull = createBasicBlock("delete.notnull");
   llvm::BasicBlock *DeleteEnd = createBasicBlock("delete.end");
 
   llvm::Value *IsNull = Builder.CreateIsNull(Ptr.getPointer(), "isnull");
 
   Builder.CreateCondBr(IsNull, DeleteEnd, DeleteNotNull);
   EmitBlock(DeleteNotNull);
+  Ptr.setKnownNonNull();
 
   QualType DeleteTy = E->getDestroyedType();
 
   // A destroying operator delete overrides the entire operation of the
   // delete expression.
   if (E->getOperatorDelete()->isDestroyingOperatorDelete()) {
     EmitDestroyingObjectDelete(*this, E, Ptr, DeleteTy);
     EmitBlock(DeleteEnd);
while (const ConstantArrayType *Arr
       DeleteTy = Arr->getElementType();
 
       // 2. GEP to the first element of the array.
       GEP.push_back(Zero);
     }
 
     Ptr = Address(Builder.CreateInBoundsGEP(Ptr.getElementType(),
                                             Ptr.getPointer(), GEP, "del.first"),
-                  ConvertTypeForMem(DeleteTy), Ptr.getAlignment());
+                  ConvertTypeForMem(DeleteTy), Ptr.getAlignment(),
+                  Ptr.isKnownNonNull());
   }
 
   assert(ConvertTypeForMem(DeleteTy) == Ptr.getElementType());
 
   if (E->isArrayForm()) {
     EmitArrayDelete(*this, E, Ptr, DeleteTy);
     EmitBlock(DeleteEnd);
   } else {

clang/lib/CodeGen/CGNonTrivialStruct.cpp

 
 template <size_t N, size_t... Ints>
 static std::array<Address, N> getParamAddrs(std::index_sequence<Ints...> IntSeq,
                                             std::array<CharUnits, N> Alignments,
                                             FunctionArgList Args,
                                             CodeGenFunction *CGF) {
   return std::array<Address, N>{
       {Address(CGF->Builder.CreateLoad(CGF->GetAddrOfLocalVar(Args[Ints])),
-               CGF->VoidPtrTy, Alignments[Ints])...}};
+               CGF->VoidPtrTy, Alignments[Ints], KnownNonNull)...}};
 }
 
 // Template classes that are used as bases for classes that emit special
 // functions.
 template <class Derived> struct GenFuncBase {
   template <size_t N>
   void visitStruct(QualType FT, const FieldDecl *FD, CharUnits CurStructOffset,
                    std::array<Address, N> Addrs) {

clang/lib/CodeGen/CGOpenMPRuntime.cpp

if (Tmp.isValid()) {
     Addr = CGF.Builder.CreatePointerBitCastOrAddrSpaceCast(
         Addr, Tmp.getElementType());
     CGF.Builder.CreateStore(Addr, Tmp);
     return MostTopTmp;
   }
 
   Addr = CGF.Builder.CreatePointerBitCastOrAddrSpaceCast(
       Addr, OriginalBaseAddress.getType());
-  return OriginalBaseAddress.withPointer(Addr);
+  return OriginalBaseAddress.withPointer(Addr, NotKnownNonNull);
 }
 
 static const VarDecl *getBaseDecl(const Expr *Ref, const DeclRefExpr *&DE) {
   const VarDecl *OrigVD = nullptr;
   if (const auto *OASE = dyn_cast<OMPArraySectionExpr>(Ref)) {
     const Expr *Base = OASE->getBase()->IgnoreParenImpCasts();
     while (const auto *TempOASE = dyn_cast<OMPArraySectionExpr>(Base))
       Base = TempOASE->getBase()->IgnoreParenImpCasts();
void CGOpenMPRuntime::emitUpdateClause(CodeGenFunction &CGF, LValue DepobjLVal,
   // The basic structure here is a while-do loop.
   llvm::BasicBlock *BodyBB = CGF.createBasicBlock("omp.body");
   llvm::BasicBlock *DoneBB = CGF.createBasicBlock("omp.done");
   llvm::BasicBlock *EntryBB = CGF.Builder.GetInsertBlock();
   CGF.EmitBlock(BodyBB);
   llvm::PHINode *ElementPHI =
       CGF.Builder.CreatePHI(Begin.getType(), 2, "omp.elementPast");
   ElementPHI->addIncoming(Begin.getPointer(), EntryBB);
-  Begin = Begin.withPointer(ElementPHI);
+  Begin = Begin.withPointer(ElementPHI, KnownNonNull);
   Base = CGF.MakeAddrLValue(Begin, KmpDependInfoTy, Base.getBaseInfo(),
                             Base.getTBAAInfo());
   // deps[i].flags = NewDepKind;
   RTLDependenceKindTy DepKind = translateDependencyKind(NewDepKind);
   LValue FlagsLVal = CGF.EmitLValueForField(
       Base, *std::next(KmpDependInfoRD->field_begin(),
                        static_cast<unsigned int>(RTLDependInfoFields::Flags)));
   CGF.EmitStoreOfScalar(

clang/lib/CodeGen/CGValue.h

enum {
     Simple,       // This is a normal l-value, use getAddress().
     VectorElt,    // This is a vector element l-value (V[i]), use getVector*
     BitField,     // This is a bitfield l-value, use getBitfield*.
     ExtVectorElt, // This is an extended vector subset, use getExtVectorComp
     GlobalReg,    // This is a register l-value, use getGlobalReg()
     MatrixElt     // This is a matrix element, use getVector*
   } LVType;
 
   llvm::Value *V;
   llvm::Type *ElementType;
 
   union {
     // Index into a vector subscript: V[i]
     llvm::Value *VectorIdx;
 
     // ExtVector element subset: V.xyx
     llvm::Constant *VectorElts;
class LValue {
   // Lvalue has ARC imprecise lifetime.  We store this inverted to try
   // to make the default bitfield pattern all-zeroes.
   bool ImpreciseLifetime : 1;
 
   // This flag shows if a nontemporal load/stores should be used when accessing
   // this lvalue.
   bool Nontemporal : 1;
 
+  bool IsKnownNonNull : 1l;
+
   LValueBaseInfo BaseInfo;
   TBAAAccessInfo TBAAInfo;
 
   Expr *BaseIvarExp;
 
 private:
   void Initialize(QualType Type, Qualifiers Quals, CharUnits Alignment,
                   LValueBaseInfo BaseInfo, TBAAAccessInfo TBAAInfo) {
public:
   LangAS getAddressSpace() const { return Quals.getAddressSpace(); }
 
   CharUnits getAlignment() const { return CharUnits::fromQuantity(Alignment); }
   void setAlignment(CharUnits A) { Alignment = A.getQuantity(); }
 
   LValueBaseInfo getBaseInfo() const { return BaseInfo; }
   void setBaseInfo(LValueBaseInfo Info) { BaseInfo = Info; }
 
+  KnownNonNull_t isKnownNonNull() const {
+    return (KnownNonNull_t)IsKnownNonNull;
+  }
+  LValue setKnownNonNull() {
+    IsKnownNonNull = true;
+    return *this;
+  }
+
   // simple lvalue
   llvm::Value *getPointer(CodeGenFunction &CGF) const {
     assert(isSimple());
     return V;
   }
   Address getAddress(CodeGenFunction &CGF) const {
-    return Address(getPointer(CGF), ElementType, getAlignment());
+    return Address(getPointer(CGF), ElementType, getAlignment(),
+                   (KnownNonNull_t)isKnownNonNull());
   }
   void setAddress(Address address) {
     assert(isSimple());
     V = address.getPointer();
     ElementType = address.getElementType();
     Alignment = address.getAlignment().getQuantity();
+    IsKnownNonNull = address.isKnownNonNull();
   }
 
   // vector elt lvalue
   Address getVectorAddress() const {
-    return Address(getVectorPointer(), ElementType, getAlignment());
+    return Address(getVectorPointer(), ElementType, getAlignment(),
+                   (KnownNonNull_t)isKnownNonNull());
   }
   llvm::Value *getVectorPointer() const {
     assert(isVectorElt());
     return V;
   }
   llvm::Value *getVectorIdx() const {
     assert(isVectorElt());
     return VectorIdx;
   }
 
   Address getMatrixAddress() const {
-    return Address(getMatrixPointer(), ElementType, getAlignment());
+    return Address(getMatrixPointer(), ElementType, getAlignment(),
+                   (KnownNonNull_t)isKnownNonNull());
   }
   llvm::Value *getMatrixPointer() const {
     assert(isMatrixElt());
     return V;
   }
   llvm::Value *getMatrixIdx() const {
     assert(isMatrixElt());
     return VectorIdx;
   }
 
   // extended vector elements.
   Address getExtVectorAddress() const {
-    return Address(getExtVectorPointer(), ElementType, getAlignment());
+    return Address(getExtVectorPointer(), ElementType, getAlignment(),
+                   (KnownNonNull_t)isKnownNonNull());
   }
   llvm::Value *getExtVectorPointer() const {
     assert(isExtVectorElt());
     return V;
   }
   llvm::Constant *getExtVectorElts() const {
     assert(isExtVectorElt());
     return VectorElts;
   }
 
   // bitfield lvalue
   Address getBitFieldAddress() const {
-    return Address(getBitFieldPointer(), ElementType, getAlignment());
+    return Address(getBitFieldPointer(), ElementType, getAlignment(),
+                   (KnownNonNull_t)isKnownNonNull());
   }
   llvm::Value *getBitFieldPointer() const { assert(isBitField()); return V; }
   const CGBitFieldInfo &getBitFieldInfo() const {
     assert(isBitField());
     return *BitFieldInfo;
   }
 
   // global register lvalue
   llvm::Value *getGlobalReg() const { assert(isGlobalReg()); return V; }
 
   static LValue MakeAddr(Address address, QualType type, ASTContext &Context,
                          LValueBaseInfo BaseInfo, TBAAAccessInfo TBAAInfo) {
     Qualifiers qs = type.getQualifiers();
     qs.setObjCGCAttr(Context.getObjCGCAttrKind(type));
 
     LValue R;
     R.LVType = Simple;
     assert(address.getPointer()->getType()->isPointerTy());
     R.V = address.getPointer();
     R.ElementType = address.getElementType();
+    R.IsKnownNonNull = address.isKnownNonNull();
     R.Initialize(type, qs, address.getAlignment(), BaseInfo, TBAAInfo);
     return R;
   }
 
   static LValue MakeVectorElt(Address vecAddress, llvm::Value *Idx,
                               QualType type, LValueBaseInfo BaseInfo,
                               TBAAAccessInfo TBAAInfo) {
     LValue R;
     R.LVType = VectorElt;
     R.V = vecAddress.getPointer();
     R.ElementType = vecAddress.getElementType();
     R.VectorIdx = Idx;
+    R.IsKnownNonNull = vecAddress.isKnownNonNull();
     R.Initialize(type, type.getQualifiers(), vecAddress.getAlignment(),
                  BaseInfo, TBAAInfo);
     return R;
   }
 
   static LValue MakeExtVectorElt(Address vecAddress, llvm::Constant *Elts,
                                  QualType type, LValueBaseInfo BaseInfo,
                                  TBAAAccessInfo TBAAInfo) {
     LValue R;
     R.LVType = ExtVectorElt;
     R.V = vecAddress.getPointer();
     R.ElementType = vecAddress.getElementType();
     R.VectorElts = Elts;
+    R.IsKnownNonNull = vecAddress.isKnownNonNull();
     R.Initialize(type, type.getQualifiers(), vecAddress.getAlignment(),
                  BaseInfo, TBAAInfo);
     return R;
   }
 
   /// Create a new object to represent a bit-field access.
   ///
   /// \param Addr - The base address of the bit-field sequence this
   /// bit-field refers to.
   /// \param Info - The information describing how to perform the bit-field
   /// access.
   static LValue MakeBitfield(Address Addr, const CGBitFieldInfo &Info,
                              QualType type, LValueBaseInfo BaseInfo,
                              TBAAAccessInfo TBAAInfo) {
     LValue R;
     R.LVType = BitField;
     R.V = Addr.getPointer();
     R.ElementType = Addr.getElementType();
     R.BitFieldInfo = &Info;
+    R.IsKnownNonNull = Addr.isKnownNonNull();
     R.Initialize(type, type.getQualifiers(), Addr.getAlignment(), BaseInfo,
                  TBAAInfo);
     return R;
   }
 
   static LValue MakeGlobalReg(llvm::Value *V, CharUnits alignment,
                               QualType type) {
     LValue R;
     R.LVType = GlobalReg;
     R.V = V;
     R.ElementType = nullptr;
+    R.IsKnownNonNull = true;
     R.Initialize(type, type.getQualifiers(), alignment,
                  LValueBaseInfo(AlignmentSource::Decl), TBAAAccessInfo());
     return R;
   }
 
   static LValue MakeMatrixElt(Address matAddress, llvm::Value *Idx,
                               QualType type, LValueBaseInfo BaseInfo,
                               TBAAAccessInfo TBAAInfo) {
     LValue R;
     R.LVType = MatrixElt;
     R.V = matAddress.getPointer();
     R.ElementType = matAddress.getElementType();
     R.VectorIdx = Idx;
+    R.IsKnownNonNull = matAddress.isKnownNonNull();
     R.Initialize(type, type.getQualifiers(), matAddress.getAlignment(),
                  BaseInfo, TBAAInfo);
     return R;
   }
 
   RValue asAggregateRValue(CodeGenFunction &CGF) const {
     return RValue::getAggregate(getAddress(CGF), isVolatileQualified());
   }
public:
   static AggValueSlot forAddr(Address addr,
                               Qualifiers quals,
                               IsDestructed_t isDestructed,
                               NeedsGCBarriers_t needsGC,
                               IsAliased_t isAliased,
                               Overlap_t mayOverlap,
                               IsZeroed_t isZeroed = IsNotZeroed,
                        IsSanitizerChecked_t isChecked = IsNotSanitizerChecked) {
+    if (addr.isValid())
+      addr.setKnownNonNull();
     return AggValueSlot(addr, quals, isDestructed, needsGC, isZeroed, isAliased,
                         mayOverlap, isChecked);
   }
 
   static AggValueSlot
   forLValue(const LValue &LV, CodeGenFunction &CGF, IsDestructed_t isDestructed,
             NeedsGCBarriers_t needsGC, IsAliased_t isAliased,
             Overlap_t mayOverlap, IsZeroed_t isZeroed = IsNotZeroed,

clang/lib/CodeGen/CodeGenFunction.h

public:
 
     llvm::Value *getDirectValue() const {
       assert(!isIndirect());
       return Value;
     }
 
     Address getIndirectAddress() const {
       assert(isIndirect());
-      return Address(Value, ElementType, CharUnits::fromQuantity(Alignment));
+      return Address(Value, ElementType, CharUnits::fromQuantity(Alignment),
+                     KnownNonNull);
     }
   };
 
   /// EmitParmDecl - Emit a ParmVarDecl or an ImplicitParamDecl.
   void EmitParmDecl(const VarDecl &D, ParamValue Arg, unsigned ArgNo);
 
   /// protectFromPeepholes - Protect a value that we're intending to
   /// store to the side, but which will probably be used later, from
public:
   /// the LLVM value is known: For example, it may not be a pointer to an
   /// integer.
   ///
   /// If this returns a normal address, and if the lvalue's C type is fixed
   /// size, this method guarantees that the returned pointer type will point to
   /// an LLVM type of the same size of the lvalue's type.  If the lvalue has a
   /// variable length type, this is not possible.
   ///
-  LValue EmitLValue(const Expr *E);
+  LValue EmitLValue(const Expr *E,
+                    KnownNonNull_t IsKnownNonNull = NotKnownNonNull);
 
+private:
+  LValue EmitLValueHelper(const Expr *E, KnownNonNull_t IsKnownNonNull);
+
+public:
   /// Same as EmitLValue but additionally we generate checking code to
   /// guard against undefined behavior.  This is only suitable when we know
   /// that the address will be used to access the object.
   LValue EmitCheckedLValue(const Expr *E, TypeCheckKind TCK);
 
   RValue convertTempToRValue(Address addr, QualType type,
                              SourceLocation Loc);
 
public:
   ///
   /// One reasonable way to use this information is when there's a language
   /// guarantee that the pointer must be aligned to some stricter value, and
   /// we're simply trying to ensure that sufficiently obvious uses of under-
   /// aligned objects don't get miscompiled; for example, a placement new
   /// into the address of a local variable.  In such a case, it's quite
   /// reasonable to just ignore the returned alignment when it isn't from an
   /// explicit source.
-  Address EmitPointerWithAlignment(const Expr *Addr,
-                                   LValueBaseInfo *BaseInfo = nullptr,
-                                   TBAAAccessInfo *TBAAInfo = nullptr);
+  Address
+  EmitPointerWithAlignment(const Expr *Addr, LValueBaseInfo *BaseInfo = nullptr,
+                           TBAAAccessInfo *TBAAInfo = nullptr,
+                           KnownNonNull_t IsKnownNonNull = NotKnownNonNull);
 
   /// If \p E references a parameter with pass_object_size info or a constant
   /// array size modifier, emit the object size divided by the size of \p EltTy.
   /// Otherwise return null.
   llvm::Value *LoadPassedObjectSize(const Expr *E, QualType EltTy);
 
   void EmitSanitizerStatReport(llvm::SanitizerStatKind SSK);
 

clang/lib/CodeGen/CodeGenFunction.cpp

if (RetTy->isVoidType()) {
     if (!endsWithReturn(D))
       ++NumReturnExprs;
   } else if (CurFnInfo->getReturnInfo().getKind() == ABIArgInfo::Indirect) {
     // Indirect return; emit returned value directly into sret slot.
     // This reduces code size, and affects correctness in C++.
     auto AI = CurFn->arg_begin();
     if (CurFnInfo->getReturnInfo().isSRetAfterThis())
       ++AI;
-    ReturnValue = Address(&*AI, ConvertType(RetTy),
-                          CurFnInfo->getReturnInfo().getIndirectAlign());
+    ReturnValue =
+        Address(&*AI, ConvertType(RetTy),
+                CurFnInfo->getReturnInfo().getIndirectAlign(), KnownNonNull);
     if (!CurFnInfo->getReturnInfo().getIndirectByVal()) {
       ReturnValuePointer =
           CreateDefaultAlignTempAlloca(Int8PtrTy, "result.ptr");
       Builder.CreateStore(Builder.CreatePointerBitCastOrAddrSpaceCast(
                               ReturnValue.getPointer(), Int8PtrTy),
                           ReturnValuePointer);
     }
   } else if (CurFnInfo->getReturnInfo().getKind() == ABIArgInfo::InAlloca &&
              !hasScalarEvaluationKind(CurFnInfo->getReturnType())) {
     // Load the sret pointer from the argument struct and return into that.
     unsigned Idx = CurFnInfo->getReturnInfo().getInAllocaFieldIndex();
     llvm::Function::arg_iterator EI = CurFn->arg_end();
     --EI;
     llvm::Value *Addr = Builder.CreateStructGEP(
         CurFnInfo->getArgStruct(), &*EI, Idx);
     llvm::Type *Ty =
         cast<llvm::GetElementPtrInst>(Addr)->getResultElementType();
     ReturnValuePointer = Address(Addr, Ty, getPointerAlign());
     Addr = Builder.CreateAlignedLoad(Ty, Addr, getPointerAlign(), "agg.result");
-    ReturnValue =
-        Address(Addr, ConvertType(RetTy), CGM.getNaturalTypeAlignment(RetTy));
+    ReturnValue = Address(Addr, ConvertType(RetTy),
+                          CGM.getNaturalTypeAlignment(RetTy), KnownNonNull);
   } else {
     ReturnValue = CreateIRTemp(RetTy, "retval");
 
     // Tell the epilog emitter to autorelease the result.  We do this
     // now so that various specialized functions can suppress it
     // during their IR-generation.
     if (getLangOpts().ObjCAutoRefCount &&
         !CurFnInfo->isReturnsRetained() &&