[RFC] Introduce `__attribute__((nontemporal))`.

[mzolotukhin]

Summary

Currently there is no way to generate nontemporal memory accesses for some
architectures, e.g. for AArch64. In contrast to x86, it doesn't have special
intrinsics for this, and the suggested solution is using such attribute (see ARM
ACLE 2.0, section 13.1.6). The attribute would result in generating
'!nontemporal' attribute in IR, which then will (hopefully) live through
optimizations till backend, where it will be lowered to a non-temporal
instruction (for AArch64 - to STNP). I have committed a couple of patches for
vectorizers to preserve this attribute, and it seems that no other
transformation removes it.

So, is introducing a new type attribute a right approach for this problem?

Also, since I don't have much experience in front-end, I'd appreciate any help
with the patch itself to get it ready to be committed. Specifically, I currently
have following questions:

1. What tests should I add (examples would be appreciated)?
2. How does one implements constraints on how the attribute can be used, what

should be the constraints in this case, and how to properly implement them?

3. How can I check if I covered all places where this attribute might be used in

codegen? I.e. I seem to cover array-subscript and pointer-dereference
expressions, which is probaly the only cases I care about, but I easily could
miss something.

Any other feedback is also welcome!

Thanks,
Michael

[majnemer]

What does it mean to have the attribute applied to non-pointer types like int __attribute__((nontemporal)) i; ? The ACLE doesn't say but making it erroneous might make sense. Perhaps it would be good to have a semantic test which uses __attribute__((nontemporal)).

[aaron.ballman]

This doesn't seem like a fundamental property of a type, to me. If I understand properly, this has more to do with specific instances of memory access. By making it part of the type, you run into sticky situations that become hard to resolve, such as with templates in C++.

~Aaron

[mzolotukhin]

Hi all,

Thanks for the feedback, please find my answers below:

What does it mean to have the attribute applied to non-pointer types like int attribute((nontemporal)) i; ? The ACLE doesn't say but making it erroneous might make sense. Perhaps it would be good to have a semantic test which uses attribute((nontemporal)).

David,
That's a good idea. Actually, I don't know how we should behave in such cases, but probably just giving an error should be fine. And should we handle references in a similar manner (int __attribute__((nontemporal)) &i)? I'll update the patch correspondingly if we decide to go with type attributes.

This seems like a property of an operation, rather than a property of a type. Have you considered adding a __builtin_nontemporal_store builtin as an alternative?

Richard,
Yes, I've considered a builitin as an alternative. In fact, I started with it as it was easier to implement, but then decided to switch to type attribute due to the following reasons:

1. ARM ACLE 2.0 mentions attribute. Though it's not a final version of the document, AFAIU, I still preferred to use it as an argument for type-attribute.
2. Once we introduce a builtin, we'll have to support it forever (otherwise we could break someone's code). With the attribute the burden is much smaller, as we can just start ignoring it at any point if we need to - all the code will remain correct and compilable.
3. We'll need to have an intrinsic for every type + separate intrinsics for loads and stores. If we use the type attribute, one fits all.
4. While it's true, that this is more type of operation, than a type, I think in real use-cases a user would rarely need to use it on a single operation. I.e. nontemporal operations are usually used for processing bulk volumes of data, and probably this data is almost always is processed as a whole. That's why I think it's fine to mark the entire 'data' as nontemporal. And, if a user then wants to work with a small subset of it, she can use a usual (not nontemporal) pointer to it.
5. Personally, I find the code using attributes more elegant than using builtins. Compare:

void foo(float *__attribute__((nontemporal)) dst,
         float *__attribute__((nontemporal)) src1,
         float *__attribute__((nontemporal)) src2) {
  *dst = *src1 + *src2;
}

and

void foo(float *dst, float *src1, float *src2) {
  float s1 = __builtin_nontemporal_load(src1);
  float s2 = __builtin_nontemporal_load(src2);
  __builtin_nontemporal_store(s1 + s2, dst);
}

But that said, in the end I'm open to other alternatives (including builtins), and this thread is just an attempt to find the best option.

This doesn't seem like a fundamental property of a type, to me. If I understand properly, this has more to do with specific instances of memory access. By making it part of the type, you run into sticky situations that become hard to resolve, such as with templates in C++.

Aaron,
As far as I understand, type attributes doesn't result in such complications (as opposed to type qualifiers, e.g. __restrict__). That is, it doesn't change the canonical type, it only adds some 'sugar' to it. I.e. float *__attribute__((nontemporal)) and float * would behave as the same type in templates and names mangling. Please correct me if I'm wrong here.

Thanks,
Michael

[aaron.ballman]

In D12221#230187, @mzolotukhin wrote:

Aaron,
As far as I understand, type attributes doesn't result in such complications (as opposed to type qualifiers, e.g. __restrict__). That is, it doesn't change the canonical type, it only adds some 'sugar' to it. I.e. float *__attribute__((nontemporal)) and float * would behave as the same type in templates and names mangling. Please correct me if I'm wrong here.

You are correct in that type attributes do not change the canonical type, but I perhaps didn't explain the complications properly. For instance, if I wanted to store a std::vector of these nontemporal type objects, I could not do so because the type attribute information would be lost. By using a builtin, I could instead push the temporality decision to the operation on the vector objects.

~Aaron

[mzolotukhin]

Oh, I see. So, you meant something like this?

void foo(std::vector<float * __attribute__((nontemporal))> av, float * b, int N) {
  for (auto a: av)      // << `a` doesn't have nontemporal attribute here
    for (int i = 0; i < N; i++)
      a[i] = b[i]+1;
}

One can easily work around it by using an explicit type here (float * __attribute__((nontemporal)) instead of auto), but I agree that disappeared attribute might be a surprise for the user. Do you think it would be a frequent case?

BTW, there are other type attributes, which also suffer from the same issue, e.g. vector_size. What was the rationale of making them type attributes?

[aaron.ballman]

In D12221#230263, @mzolotukhin wrote:

Oh, I see. So, you meant something like this?

void foo(std::vector<float * __attribute__((nontemporal))> av, float * b, int N) {
  for (auto a: av)      // << `a` doesn't have nontemporal attribute here
    for (int i = 0; i < N; i++)
      a[i] = b[i]+1;
}

One can easily work around it by using an explicit type here (float * __attribute__((nontemporal)) instead of auto), but I agree that disappeared attribute might be a surprise for the user. Do you think it would be a frequent case?

Yes, that's along the lines of what I was thinking. There are also other questions, as to whether a user would expect this code to work or not:

template <typename Ty>
void f(Ty *ptr);

template <typename Ty>
void f(Ty * attribute((nontemporal)) ptr);

I honestly don't know enough about nontemporal object usage patterns to really have a gut feeling for what patterns are likely to appear in the wild.

BTW, there are other type attributes, which also suffer from the same issue, e.g. vector_size. What was the rationale of making them type attributes?

The usual rationale is that these attributes are targeting C code more than C++, or that the C++ use cases that would be strange to a user are unlikely to happen with realistic code. The discussion that's come up in the past when we touch on these is that Clang could perhaps use a pluggable type system that allows for more fine-grained control on whether an attribute participates as part of a type or not. A production-quality pluggable type system is a pretty large undertaking, and it's a bit research-y at this point, so I'm not proposing anything like that.

Similar questions that help decide is whether you should be able to overload on the type attribute, specialize templates on it, type identity, etc.

~Aaron

[cfe-commits]

• msg-1588-168.txt159 BDownload

[mzolotukhin]

Thanks for the feedback everyone!
I think at this point I'll try to return to builtins then. In my original patch I didn't have type overloading, so I'll need some time to add this. We can return back to type attributes later if we'd like to.

And do I understand it correctly, that we are talking about target-independent builtins?

[mzolotukhin]

Hi,

I implemented builtin-based version in D12313 - could you please take a look?

Thanks,
Michael

[mzolotukhin]

We decided to go with an alternative way - with builtins instead of type attributes. The corresponding patch is D12313, and it's already reviewed and committed.

include/clang/AST/Type.h

public:
 
   /// Determine the nullability of the given type.
   ///
   /// Note that nullability is only captured as sugar within the type
   /// system, not as part of the canonical type, so nullability will
   /// be lost by canonicalization and desugaring.
   Optional<NullabilityKind> getNullability(const ASTContext &context) const;
 
+  /// Determine the non-temporality of the given type.
+  ///
+  /// The non-temporal attribute is only captured as sugar within the type
+  /// system, not as part of the cacnonical type, so it will be lost by
+  /// canonicalization and desugaring.
+  bool hasNonTemporalAttr(const ASTContext &context) const;
+
   /// Determine whether the given type can have a nullability
   /// specifier applied to it, i.e., if it is any kind of pointer type
   /// or a dependent type that could instantiate to any kind of
   /// pointer type.
   bool canHaveNullability() const;
 
   /// Retrieve the set of substitutions required when accessing a member
   /// of the Objective-C receiver type that is declared in the given context.
enum Kind {
     attr_ptr32,
     attr_ptr64,
     attr_sptr,
     attr_uptr,
     attr_nonnull,
     attr_nullable,
     attr_null_unspecified,
     attr_objc_kindof,
+    attr_nontemporal,
   };
 
 private:
   QualType ModifiedType;
   QualType EquivalentType;
 
   friend class ASTContext; // creates these
 

include/clang/Basic/Attr.td

def TypeNullable : TypeAttr {
   let Documentation = [TypeNullableDocs];
 }
 
 def TypeNullUnspecified : TypeAttr {
   let Spellings = [Keyword<"_Null_unspecified">];
   let Documentation = [TypeNullUnspecifiedDocs];
 }
 
+def TypeNonTemporal : TypeAttr {
+  let Spellings = [GCC<"nontemporal">];
+  let Documentation = [Undocumented];
+}
+
 def ObjCKindOf : TypeAttr {
   let Spellings = [Keyword<"__kindof">];
   let Documentation = [Undocumented];
 }
 
 def AssumeAligned : InheritableAttr {
   let Spellings = [GCC<"assume_aligned">];
   let Subjects = SubjectList<[ObjCMethod, Function]>;

lib/AST/Type.cpp

bool AttributedType::isCallingConv() const {
   case attr_neon_polyvector_type:
   case attr_objc_gc:
   case attr_objc_ownership:
   case attr_noreturn:
   case attr_nonnull:
   case attr_nullable:
   case attr_null_unspecified:
   case attr_objc_kindof:
+  case attr_nontemporal:
     return false;
 
   case attr_pcs:
   case attr_pcs_vfp:
   case attr_cdecl:
   case attr_fastcall:
   case attr_stdcall:
   case attr_thiscall:
LinkageInfo Type::getLinkageAndVisibility() const {
   if (!isCanonicalUnqualified())
     return computeLinkageInfo(getCanonicalTypeInternal());
 
   LinkageInfo LV = computeLinkageInfo(this);
   assert(LV.getLinkage() == getLinkage());
   return LV;
 }
 
+bool Type::hasNonTemporalAttr(const ASTContext &context) const {
+  QualType type(this, 0);
+  do {
+    // Check whether this is an attributed type with nullability
+    // information.
+    if (auto attributed = dyn_cast<AttributedType>(type.getTypePtr())) {
+      if (attributed->getAttrKind() == AttributedType::attr_nontemporal)
+        return true;
+    }
+
+    // Desugar the type. If desugaring does nothing, we're done.
+    QualType desugared = type.getSingleStepDesugaredType(context);
+    if (desugared.getTypePtr() == type.getTypePtr())
+      return false;
+
+    type = desugared;
+  } while (true);
+}
+
 Optional<NullabilityKind> Type::getNullability(const ASTContext &context) const {
   QualType type(this, 0);
   do {
     // Check whether this is an attributed type with nullability
     // information.
     if (auto attributed = dyn_cast<AttributedType>(type.getTypePtr())) {
       if (auto nullability = attributed->getImmediateNullability())
         return nullability;

lib/AST/TypePrinter.cpp

void TypePrinter::printAttributedAfter(const AttributedType *T,
   switch (T->getAttrKind()) {
   default: llvm_unreachable("This attribute should have been handled already");
   case AttributedType::attr_address_space:
     OS << "address_space(";
     OS << T->getEquivalentType().getAddressSpace();
     OS << ')';
     break;
 
+  case AttributedType::attr_nontemporal: {
+    OS << "nontemporal";
+    break;
+  }
+
   case AttributedType::attr_vector_size: {
     OS << "__vector_size__(";
     if (const VectorType *vector =T->getEquivalentType()->getAs<VectorType>()) {
       OS << vector->getNumElements();
       OS << " * sizeof(";
       print(vector->getElementType(), OS, StringRef());
       OS << ')';
     }

lib/CodeGen/CGExpr.cpp

CodeGenFunction::tryEmitAsConstant(DeclRefExpr *refExpr) {
   return ConstantEmission::forValue(C);
 }
 
 llvm::Value *CodeGenFunction::EmitLoadOfScalar(LValue lvalue,
                                                SourceLocation Loc) {
   return EmitLoadOfScalar(lvalue.getAddress(), lvalue.isVolatile(),
                           lvalue.getAlignment().getQuantity(),
                           lvalue.getType(), Loc, lvalue.getTBAAInfo(),
-                          lvalue.getTBAABaseType(), lvalue.getTBAAOffset());
+                          lvalue.getTBAABaseType(), lvalue.getTBAAOffset(),
+                          lvalue.getNT());
 }
 
 static bool hasBooleanRepresentation(QualType Ty) {
   if (Ty->isBooleanType())
     return true;
 
   if (const EnumType *ET = Ty->getAs<EnumType>())
     return ET->getDecl()->getIntegerType()->isBooleanType();
llvm::MDNode *CodeGenFunction::getRangeForLoadFromType(QualType Ty) {
   return MDHelper.createRange(Min, End);
 }
 
 llvm::Value *CodeGenFunction::EmitLoadOfScalar(llvm::Value *Addr, bool Volatile,
                                                unsigned Alignment, QualType Ty,
                                                SourceLocation Loc,
                                                llvm::MDNode *TBAAInfo,
                                                QualType TBAABaseType,
-                                               uint64_t TBAAOffset) {
+                                               uint64_t TBAAOffset,
+                                               bool isNonTemporal) {
   // For better performance, handle vector loads differently.
   if (Ty->isVectorType()) {
     llvm::Value *V;
     const llvm::Type *EltTy =
     cast<llvm::PointerType>(Addr->getType())->getElementType();
 
     const auto *VTy = cast<llvm::VectorType>(EltTy);
 
if (Ty->isAtomicType() || typeIsSuitableForInlineAtomic(Ty, Volatile)) {
     return EmitAtomicLoad(lvalue, Loc).getScalarVal();
   }
 
   llvm::LoadInst *Load = Builder.CreateLoad(Addr);
   if (Volatile)
     Load->setVolatile(true);
   if (Alignment)
     Load->setAlignment(Alignment);
+  if (isNonTemporal) {
+    llvm::LLVMContext &C = Load->getContext();
+    llvm::Module *M = Load->getModule();
+    SmallVector<llvm::Metadata *, 1> Elts;
+    Elts.push_back(llvm::ConstantAsMetadata::get(
+        llvm::ConstantInt::get(llvm::Type::getInt32Ty(C), 1)));
+    llvm::MDNode *Node = llvm::MDNode::get(C, Elts);
+    Load->setMetadata(M->getMDKindID("nontemporal"), Node);
+  }
+
   if (TBAAInfo) {
     llvm::MDNode *TBAAPath = CGM.getTBAAStructTagInfo(TBAABaseType, TBAAInfo,
                                                       TBAAOffset);
     if (TBAAPath)
       CGM.DecorateInstruction(Load, TBAAPath, false/*ConvertTypeToTag*/);
   }
 
   bool NeedsBoolCheck =
llvm::Value *CodeGenFunction::EmitFromMemory(llvm::Value *Value, QualType Ty) {
 
   return Value;
 }
 
 void CodeGenFunction::EmitStoreOfScalar(llvm::Value *Value, llvm::Value *Addr,
                                         bool Volatile, unsigned Alignment,
                                         QualType Ty, llvm::MDNode *TBAAInfo,
                                         bool isInit, QualType TBAABaseType,
-                                        uint64_t TBAAOffset) {
+                                        uint64_t TBAAOffset,
+                                        bool isNonTemporal) {
 
   // Handle vectors differently to get better performance.
   if (Ty->isVectorType()) {
     llvm::Type *SrcTy = Value->getType();
     auto *VecTy = cast<llvm::VectorType>(SrcTy);
     // Handle vec3 special.
     if (VecTy->getNumElements() == 3) {
       // Our source is a vec3, do a shuffle vector to make it a vec4.
EmitAtomicStore(RValue::get(Value),
                                      getContext(), TBAAInfo),
                     isInit);
     return;
   }
 
   llvm::StoreInst *Store = Builder.CreateStore(Value, Addr, Volatile);
   if (Alignment)
     Store->setAlignment(Alignment);
+  if (isNonTemporal) {
+    llvm::LLVMContext &C = Store->getContext();
+    llvm::Module *M = Store->getModule();
+    SmallVector<llvm::Metadata *, 1> Elts;
+    Elts.push_back(llvm::ConstantAsMetadata::get(
+        llvm::ConstantInt::get(llvm::Type::getInt32Ty(C), 1)));
+    llvm::MDNode *Node = llvm::MDNode::get(C, Elts);
+    Store->setMetadata(M->getMDKindID("nontemporal"), Node);
+  }
+
   if (TBAAInfo) {
     llvm::MDNode *TBAAPath = CGM.getTBAAStructTagInfo(TBAABaseType, TBAAInfo,
                                                       TBAAOffset);
     if (TBAAPath)
       CGM.DecorateInstruction(Store, TBAAPath, false/*ConvertTypeToTag*/);
   }
 }
 
 void CodeGenFunction::EmitStoreOfScalar(llvm::Value *value, LValue lvalue,
                                         bool isInit) {
   EmitStoreOfScalar(value, lvalue.getAddress(), lvalue.isVolatile(),
                     lvalue.getAlignment().getQuantity(), lvalue.getType(),
                     lvalue.getTBAAInfo(), isInit, lvalue.getTBAABaseType(),
-                    lvalue.getTBAAOffset());
+                    lvalue.getTBAAOffset(), lvalue.getNT());
 }
 
 /// EmitLoadOfLValue - Given an expression that represents a value lvalue, this
 /// method emits the address of the lvalue, then loads the result as an rvalue,
 /// returning the rvalue.
 RValue CodeGenFunction::EmitLoadOfLValue(LValue LV, SourceLocation Loc) {
   if (LV.isObjCWeak()) {
     // load of a __weak object.
if (LV.isSimple()) {
     // Everything needs a load.
     return RValue::get(EmitLoadOfScalar(LV, Loc));
   }
 
   if (LV.isVectorElt()) {
     llvm::LoadInst *Load = Builder.CreateLoad(LV.getVectorAddr(),
                                               LV.isVolatileQualified());
     Load->setAlignment(LV.getAlignment().getQuantity());
+    if (LV.getNT()) {
+      llvm::LLVMContext &C = Load->getContext();
+      llvm::Module *M = Load->getModule();
+      SmallVector<llvm::Metadata *, 1> Elts;
+      Elts.push_back(llvm::ConstantAsMetadata::get(
+          llvm::ConstantInt::get(llvm::Type::getInt32Ty(C), 1)));
+      llvm::MDNode *Node = llvm::MDNode::get(C, Elts);
+      Load->setMetadata(M->getMDKindID("nontemporal"), Node);
+    }
+
     return RValue::get(Builder.CreateExtractElement(Load, LV.getVectorIdx(),
                                                     "vecext"));
   }
 
   // If this is a reference to a subset of the elements of a vector, either
   // shuffle the input or extract/insert them as appropriate.
   if (LV.isExtVectorElt())
     return EmitLoadOfExtVectorElementLValue(LV);
LValue CodeGenFunction::EmitUnaryOpLValue(const UnaryOperator *E) {
 
   QualType ExprTy = getContext().getCanonicalType(E->getSubExpr()->getType());
   switch (E->getOpcode()) {
   default: llvm_unreachable("Unknown unary operator lvalue!");
   case UO_Deref: {
     QualType T = E->getSubExpr()->getType()->getPointeeType();
     assert(!T.isNull() && "CodeGenFunction::EmitUnaryOpLValue: Illegal type");
 
-    LValue LV = MakeNaturalAlignAddrLValue(EmitScalarExpr(E->getSubExpr()), T);
+    LValue LV = MakeNaturalAlignAddrLValue(
+        EmitScalarExpr(E->getSubExpr()), T,
+        E->getSubExpr()->IgnoreImpCasts()->getType()->hasNonTemporalAttr(
+            getContext()));
     LV.getQuals().setAddressSpace(ExprTy.getAddressSpace());
 
     // We should not generate __weak write barrier on indirect reference
     // of a pointer to object; as in void foo (__weak id *param); *param = 0;
     // But, we continue to generate __strong write barrier on indirect write
     // into a pointer to object.
     if (getLangOpts().ObjC1 &&
         getLangOpts().getGC() != LangOptions::NonGC &&
else if (const VariableArrayType *vla =
     else
       Address = Builder.CreateInBoundsGEP(Base, Idx, "arrayidx");
   }
 
   QualType T = E->getBase()->getType()->getPointeeType();
   assert(!T.isNull() &&
          "CodeGenFunction::EmitArraySubscriptExpr(): Illegal base type");
 
+  bool isNonTemporal =
+      E->getBase()->IgnoreImpCasts()->getType()->hasNonTemporalAttr(
+          getContext());
 
   // Limit the alignment to that of the result type.
   LValue LV;
   if (!ArrayAlignment.isZero()) {
     CharUnits Align = getContext().getTypeAlignInChars(T);
     ArrayAlignment = std::min(Align, ArrayAlignment);
-    LV = MakeAddrLValue(Address, T, ArrayAlignment);
+    LV = MakeAddrLValue(Address, T, ArrayAlignment, isNonTemporal);
   } else {
-    LV = MakeNaturalAlignAddrLValue(Address, T);
+    LV = MakeNaturalAlignAddrLValue(Address, T, isNonTemporal);
   }
 
   LV.getQuals().setAddressSpace(E->getBase()->getType().getAddressSpace());
 
   if (getLangOpts().ObjC1 &&
       getLangOpts().getGC() != LangOptions::NonGC) {
     LV.setNonGC(!E->isOBJCGCCandidate(getContext()));
     setObjCGCLValueClass(getContext(), E, LV);

lib/CodeGen/CGValue.h

class LValue {
 
   // 'const' is unused here
   Qualifiers Quals;
 
   // The alignment to use when accessing this lvalue.  (For vector elements,
   // this is the alignment of the whole vector.)
   int64_t Alignment;
 
+  // Non-temporality attribute to use when accessing this lvalue.
+  bool isNonTemporal : 1;
+
   // objective-c's ivar
   bool Ivar:1;
   
   // objective-c's ivar is an array
   bool ObjIsArray:1;
 
   // LValue is non-gc'able for any reason, including being a parameter or local
   // variable.
class LValue {
   uint64_t TBAAOffset;
 
   /// TBAAInfo - TBAA information to attach to dereferences of this LValue.
   llvm::MDNode *TBAAInfo;
 
 private:
   void Initialize(QualType Type, Qualifiers Quals,
                   CharUnits Alignment,
-                  llvm::MDNode *TBAAInfo = nullptr) {
+                  llvm::MDNode *TBAAInfo = nullptr,
+                  bool isNonTemporal = false) {
     this->Type = Type;
     this->Quals = Quals;
     this->Alignment = Alignment.getQuantity();
     assert(this->Alignment == Alignment.getQuantity() &&
            "Alignment exceeds allowed max!");
 
+    this->isNonTemporal = isNonTemporal;
+
     // Initialize Objective-C flags.
     this->Ivar = this->ObjIsArray = this->NonGC = this->GlobalObjCRef = false;
     this->ImpreciseLifetime = false;
     this->ThreadLocalRef = false;
     this->BaseIvarExp = nullptr;
 
     // Initialize fields for TBAA.
     this->TBAABaseType = Type;
public:
 
   const Qualifiers &getQuals() const { return Quals; }
   Qualifiers &getQuals() { return Quals; }
 
   unsigned getAddressSpace() const { return Quals.getAddressSpace(); }
 
   CharUnits getAlignment() const { return CharUnits::fromQuantity(Alignment); }
   void setAlignment(CharUnits A) { Alignment = A.getQuantity(); }
+  bool getNT() const { return isNonTemporal; }
+  void setNT(bool nt) { isNonTemporal = nt; }
 
   // simple lvalue
   llvm::Value *getAddress() const { assert(isSimple()); return V; }
   void setAddress(llvm::Value *address) {
     assert(isSimple());
     V = address;
   }
 
const CGBitFieldInfo &getBitFieldInfo() const {
     return *BitFieldInfo;
   }
 
   // global register lvalue
   llvm::Value *getGlobalReg() const { assert(isGlobalReg()); return V; }
 
   static LValue MakeAddr(llvm::Value *address, QualType type,
                          CharUnits alignment, ASTContext &Context,
-                         llvm::MDNode *TBAAInfo = nullptr) {
+                         llvm::MDNode *TBAAInfo = nullptr,
+                         bool isNonTemporal = false) {
     Qualifiers qs = type.getQualifiers();
     qs.setObjCGCAttr(Context.getObjCGCAttrKind(type));
 
     LValue R;
     R.LVType = Simple;
     assert(address->getType()->isPointerTy());
     R.V = address;
-    R.Initialize(type, qs, alignment, TBAAInfo);
+    R.Initialize(type, qs, alignment, TBAAInfo, isNonTemporal);
     return R;
   }
 
-  static LValue MakeVectorElt(llvm::Value *Vec, llvm::Value *Idx,
-                              QualType type, CharUnits Alignment) {
+  static LValue MakeVectorElt(llvm::Value *Vec, llvm::Value *Idx, QualType type,
+                              CharUnits Alignment, bool isNonTemporal = false) {
     LValue R;
     R.LVType = VectorElt;
     R.V = Vec;
     R.VectorIdx = Idx;
     R.Initialize(type, type.getQualifiers(), Alignment);
     return R;
   }
 
   static LValue MakeExtVectorElt(llvm::Value *Vec, llvm::Constant *Elts,
-                                 QualType type, CharUnits Alignment) {
+                                 QualType type, CharUnits Alignment,
+                                 bool isNonTemporal = false) {
     LValue R;
     R.LVType = ExtVectorElt;
     R.V = Vec;
     R.VectorElts = Elts;
     R.Initialize(type, type.getQualifiers(), Alignment);
     return R;
   }
 

lib/CodeGen/CodeGenFunction.h

#endif
   /// specified stmt yet.
   void ErrorUnsupported(const Stmt *S, const char *Type);
 
   //===--------------------------------------------------------------------===//
   //                                  Helpers
   //===--------------------------------------------------------------------===//
 
   LValue MakeAddrLValue(llvm::Value *V, QualType T,
-                        CharUnits Alignment = CharUnits()) {
+                        CharUnits Alignment = CharUnits(),
+                        bool isNonTemporal = false) {
     return LValue::MakeAddr(V, T, Alignment, getContext(),
-                            CGM.getTBAAInfo(T));
+                            CGM.getTBAAInfo(T), isNonTemporal);
   }
 
-  LValue MakeNaturalAlignAddrLValue(llvm::Value *V, QualType T);
+  LValue MakeNaturalAlignAddrLValue(llvm::Value *V, QualType T,
+                                    bool isNonTemporal = false);
 
   /// CreateTempAlloca - This creates a alloca and inserts it into the entry
   /// block. The caller is responsible for setting an appropriate alignment on
   /// the alloca.
   llvm::AllocaInst *CreateTempAlloca(llvm::Type *Ty,
                                      const Twine &Name = "tmp");
 
   /// InitTempAlloca - Provide an initial value for the given alloca.
public:
   /// EmitLoadOfScalar - Load a scalar value from an address, taking
   /// care to appropriately convert from the memory representation to
   /// the LLVM value representation.
   llvm::Value *EmitLoadOfScalar(llvm::Value *Addr, bool Volatile,
                                 unsigned Alignment, QualType Ty,
                                 SourceLocation Loc,
                                 llvm::MDNode *TBAAInfo = nullptr,
                                 QualType TBAABaseTy = QualType(),
-                                uint64_t TBAAOffset = 0);
+                                uint64_t TBAAOffset = 0,
+                                bool isNonTemporal = false);
 
   /// EmitLoadOfScalar - Load a scalar value from an address, taking
   /// care to appropriately convert from the memory representation to
   /// the LLVM value representation.  The l-value must be a simple
   /// l-value.
   llvm::Value *EmitLoadOfScalar(LValue lvalue, SourceLocation Loc);
 
   /// EmitStoreOfScalar - Store a scalar value to an address, taking
   /// care to appropriately convert from the memory representation to
   /// the LLVM value representation.
   void EmitStoreOfScalar(llvm::Value *Value, llvm::Value *Addr,
                          bool Volatile, unsigned Alignment, QualType Ty,
                          llvm::MDNode *TBAAInfo = nullptr, bool isInit = false,
                          QualType TBAABaseTy = QualType(),
-                         uint64_t TBAAOffset = 0);
+                         uint64_t TBAAOffset = 0, bool isNonTemporal = false);
 
   /// EmitStoreOfScalar - Store a scalar value to an address, taking
   /// care to appropriately convert from the memory representation to
   /// the LLVM value representation.  The l-value must be a simple
   /// l-value.  The isInit flag indicates whether this is an initialization.
   /// If so, atomic qualifiers are ignored and the store is always non-atomic.
   void EmitStoreOfScalar(llvm::Value *value, LValue lvalue, bool isInit=false);
 

lib/CodeGen/CodeGenFunction.cpp

CodeGenFunction::~CodeGenFunction() {
   if (FirstBlockInfo)
     destroyBlockInfos(FirstBlockInfo);
 
   if (getLangOpts().OpenMP) {
     CGM.getOpenMPRuntime().functionFinished(*this);
   }
 }
 
-LValue CodeGenFunction::MakeNaturalAlignAddrLValue(llvm::Value *V, QualType T) {
+LValue CodeGenFunction::MakeNaturalAlignAddrLValue(llvm::Value *V, QualType T,
+                                                   bool isNonTemporal) {
   CharUnits Alignment;
   if (CGM.getCXXABI().isTypeInfoCalculable(T)) {
     Alignment = getContext().getTypeAlignInChars(T);
     unsigned MaxAlign = getContext().getLangOpts().MaxTypeAlign;
     if (MaxAlign && Alignment.getQuantity() > MaxAlign &&
         !getContext().isAlignmentRequired(T))
       Alignment = CharUnits::fromQuantity(MaxAlign);
   }
-  return LValue::MakeAddr(V, T, Alignment, getContext(), CGM.getTBAAInfo(T));
+  return LValue::MakeAddr(V, T, Alignment, getContext(), CGM.getTBAAInfo(T),
+                          isNonTemporal);
 }
 
 llvm::Type *CodeGenFunction::ConvertTypeForMem(QualType T) {
   return CGM.getTypes().ConvertTypeForMem(T);
 }
 
 llvm::Type *CodeGenFunction::ConvertType(QualType T) {
   return CGM.getTypes().ConvertType(T);

lib/Sema/SemaType.cpp

MS_TYPE_ATTRS_CASELIST:
       // Microsoft type attributes cannot go after the declarator-id.
       continue;
 
     NULLABILITY_TYPE_ATTRS_CASELIST:
       // Nullability specifiers cannot go after the declarator-id.
 
     // Objective-C __kindof does not get distributed.
     case AttributeList::AT_ObjCKindOf:
+    case AttributeList::AT_TypeNonTemporal:
       continue;
 
     default:
       break;
     }
   } while ((attr = next));
 }
 
static AttributeList::Kind getAttrListKind(AttributedType::Kind kind) {
   case AttributedType::attr_sptr:
     return AttributeList::AT_SPtr;
   case AttributedType::attr_uptr:
     return AttributeList::AT_UPtr;
   case AttributedType::attr_nonnull:
     return AttributeList::AT_TypeNonNull;
   case AttributedType::attr_nullable:
     return AttributeList::AT_TypeNullable;
+  case AttributedType::attr_nontemporal:
+    return AttributeList::AT_TypeNonTemporal;
   case AttributedType::attr_null_unspecified:
     return AttributeList::AT_TypeNullUnspecified;
   case AttributedType::attr_objc_kindof:
     return AttributeList::AT_ObjCKindOf;
   }
   llvm_unreachable("unexpected attribute kind!");
 }
 
NULLABILITY_TYPE_ATTRS_CASELIST:
               attr.isContextSensitiveKeywordAttribute())) {
           attr.setInvalid();
         }
 
         attr.setUsedAsTypeAttr();
       }
       break;
 
+    case AttributeList::AT_TypeNonTemporal:
+      type = state.getSema().Context.getAttributedType(
+          AttributedType::attr_nontemporal, type, type);
+      attr.setUsedAsTypeAttr();
+      break;
+
     case AttributeList::AT_ObjCKindOf:
       // '__kindof' must be part of the decl-specifiers.
       switch (TAL) {
       case TAL_DeclSpec:
         break;
 
       case TAL_DeclChunk:
       case TAL_DeclName:

test/CodeGen/nontemporal.cpp

+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -emit-llvm -o - %s | FileCheck %s
+typedef float * PtrT;
+typedef float * __attribute__((nontemporal)) NonTemporalPtrT;
+
+// CHECK-LABEL: @_Z3fooPfS_S_i
+void foo(NonTemporalPtrT a, NonTemporalPtrT b, PtrT c, int N) {
+// CHECK-DAG:  [[VALUE_B:%.+]] = load float, float* %{{[0-9a-z._]+}}, align 4, !nontemporal ![[NT:[0-9]+]]
+// CHECK-DAG:  [[VALUE_C:%.+]] = load float, float* %{{[0-9a-z._]+}}, align 4{{$}}
+// CHECK:      [[VALUE_SUB:%.+]] = fsub float [[VALUE_B]], [[VALUE_C]]
+// CHECK:      store float [[VALUE_SUB]], float* %{{[0-9a-z._]+}}, align 4, !nontemporal ![[NT]]
+  a[N] = b[N] - c[N];
+}
+
+// CHECK-LABEL: @_Z4foo2PfS_S_
+void foo2(NonTemporalPtrT a, NonTemporalPtrT b, PtrT c) {
+// CHECK-DAG:  [[VALUE_B:%.+]] = load float, float* %{{[0-9]+}}, align 4, !nontemporal ![[NT:[0-9]+]]
+// CHECK-DAG:  [[VALUE_C:%.+]] = load float, float* %{{[0-9]+}}, align 4{{$}}
+// CHECK:      [[VALUE_SUB:%.+]] = fsub float [[VALUE_B]], [[VALUE_C]]
+// CHECK:      store float [[VALUE_SUB]], float* %{{[0-9]+}}, align 4, !nontemporal ![[NT]]
+  *a = *b - *c;
+}