[X86] Add xgetbv xsetbv intrinsics

[guyblank]

Summary

[X86] Add xgetbv xsetbv intrinsics

[delena]

#include <x86intrin.h>in the test is not clear for me. Does it mean that you broke backward compatibility?

lib/CodeGen/CGBuiltin.cpp

6779

remove {}

lib/Headers/intrin.h

905

I'm not sure that we can move it from one file to another. And what was wrong with current implementation.

[guyblank]

the <x86intrin.h> include is because i added calls to the intrinsics themselves in the test, no just the builtins.

lib/Headers/intrin.h

905

it can't be left here since it will conflict with non-windows implementation.

my impression was that it is generally better to use "regular" lowering flow, over using inline asm.

[craig.topper]

This change seems consistent with similar being done in r250158 when the other xsave intrinsics were added. intrin.h includes x86intrin.h which in turn includes xsaveintrin.h. So this just makes xgetbv/xsetbv available on non-microsoft platforms.

lib/Headers/intrin.h

289

Should _XCR_XFEATURE_ENABLE_MASK be moved?

Also what is that "static inline" bound to? It looks like when _XCR_XFEATURE_ENABLE_MASK was added, the "static inline" was separated from the declaration of _xgetbv.

[guyblank]

ping

[guyblank]

If there aren't any further objections, I'd like go ahead with the commit.

[craig.topper]

LGTM

[rnk]

Reverted in r278814, it appears to break usage of _xgetbv on Windows:
https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin%28dll%29/builds/5846/steps/compile/logs/stdio
../../base/cpu.cc(194,10): error: use of undeclared identifier '_xgetbv'

(_xgetbv(0) & 6) == 6 /* XSAVE enabled by kernel */;

You removed the test in ms-intrin.cpp that would have prevented this breakage, too.

[guyblank]

Sorry about that, forgot that i changed the ms_intrin test.

about the failure, I think that xsaveintrin.h is not being included because it requires the xsave feature - which should be on if the target supports it.
do you know what in which target the failure occurred? also, can you direct me to the source code for the failure?

thanks

[rnk]

The source isn't that interesting, it includes intrin.h and immintrin.h before using _xgetbv.

I think the issue is that Nico added the _MSC_VER check to intrin.h in http://reviews.llvm.org/D20291:

#if !defined(_MSC_VER) || has_feature(modules) || defined(XSAVE__)
#include <xsaveintrin.h>
#endif

Nico is on vacation right now, but we might want to reconsider that. We really only need to block the AVX512 intrinsics which are unreasonably large.

[guyblank]

Still, XSAVE should have been defined when compiling for a target that supports the feature.

But anyway, the xsaveintrin.h is quite small so always including it shouldn't be an issue.
Are you ok with me removing the #if just for this header file, or would you like to wait for Nico?

[rnk]

Hm, resending my comments because it doesn't appear to work from email. I swear it used to...

In D21959#519179, @guyblank wrote:

Still, XSAVE should have been defined when compiling for a target that supports the feature.

That's not how MSVC intrinsics (or icc intrinsics, right?) are supposed to work, though. In MSVC, all intrinsics are always available, regardless of subtarget options. Unfortunately, Clang is not very compatible in this area, because of the way that we map the vector intrinsics to our generic vector IR and then codegen that. In this case, we don't have any generic instruction to map to, so I think we should try to be compatible here.

But anyway, the xsaveintrin.h is quite small so always including it shouldn't be an issue.
Are you ok with me removing the #if just for this header file, or would you like to wait for Nico?

I think removing the _MSC_VER check is probably OK, but will the new LLVM intrinsic you added generate correct code when the xsave feature is disabled, as I expect it is in Chromium?

[guyblank]

removing the MSC_VER check will not be enough, the feature guards from the intrinsic and the builtin need to be removed to make it work. not sure if this is the right way to go, any thoughts on this?

[rnk]

Now might be the time to solve the larger problem of wider intrinsic availability. Like I mentioned, all these intrinsics really ought to be available all the time, regardless of CPU subtarget.

include/clang/Basic/BuiltinsX86.def

 TARGET_BUILTIN(__builtin_ia32_fxrstor, "vv*", "", "fxsr")
 TARGET_BUILTIN(__builtin_ia32_fxrstor64, "vv*", "", "fxsr")
 TARGET_BUILTIN(__builtin_ia32_fxsave, "vv*", "", "fxsr")
 TARGET_BUILTIN(__builtin_ia32_fxsave64, "vv*", "", "fxsr")
 
 // XSAVE
 TARGET_BUILTIN(__builtin_ia32_xsave, "vv*ULLi", "", "xsave")
 TARGET_BUILTIN(__builtin_ia32_xsave64, "vv*ULLi", "", "xsave")
+TARGET_BUILTIN(__builtin_ia32_xgetbv, "ULLiUi", "", "xsave")
+TARGET_BUILTIN(__builtin_ia32_xsetbv, "vUiULLi", "", "xsave")
 TARGET_BUILTIN(__builtin_ia32_xrstor, "vv*ULLi", "", "xsave")
 TARGET_BUILTIN(__builtin_ia32_xrstor64, "vv*ULLi", "", "xsave")
 TARGET_BUILTIN(__builtin_ia32_xsaveopt, "vv*ULLi", "", "xsaveopt")
 TARGET_BUILTIN(__builtin_ia32_xsaveopt64, "vv*ULLi", "", "xsaveopt")
 TARGET_BUILTIN(__builtin_ia32_xrstors, "vv*ULLi", "", "xsaves")
 TARGET_BUILTIN(__builtin_ia32_xrstors64, "vv*ULLi", "", "xsaves")
 TARGET_BUILTIN(__builtin_ia32_xsavec, "vv*ULLi", "", "xsavec")
 TARGET_BUILTIN(__builtin_ia32_xsavec64, "vv*ULLi", "", "xsavec")

lib/CodeGen/CGBuiltin.cpp

Value *CodeGenFunction::EmitX86BuiltinExpr(unsigned BuiltinID,
   case X86::BI__builtin_ia32_xrstor64:
   case X86::BI__builtin_ia32_xsaveopt:
   case X86::BI__builtin_ia32_xsaveopt64:
   case X86::BI__builtin_ia32_xrstors:
   case X86::BI__builtin_ia32_xrstors64:
   case X86::BI__builtin_ia32_xsavec:
   case X86::BI__builtin_ia32_xsavec64:
   case X86::BI__builtin_ia32_xsaves:
-  case X86::BI__builtin_ia32_xsaves64: {
+  case X86::BI__builtin_ia32_xsaves64:
+  case X86::BI__builtin_ia32_xsetbv: {
     Intrinsic::ID ID;
 #define INTRINSIC_X86_XSAVE_ID(NAME) \
     case X86::BI__builtin_ia32_##NAME: \
       ID = Intrinsic::x86_##NAME; \
       break
     switch (BuiltinID) {
     default: llvm_unreachable("Unsupported intrinsic!");
     INTRINSIC_X86_XSAVE_ID(xsave);
     INTRINSIC_X86_XSAVE_ID(xsave64);
     INTRINSIC_X86_XSAVE_ID(xrstor);
     INTRINSIC_X86_XSAVE_ID(xrstor64);
     INTRINSIC_X86_XSAVE_ID(xsaveopt);
     INTRINSIC_X86_XSAVE_ID(xsaveopt64);
     INTRINSIC_X86_XSAVE_ID(xrstors);
     INTRINSIC_X86_XSAVE_ID(xrstors64);
     INTRINSIC_X86_XSAVE_ID(xsavec);
     INTRINSIC_X86_XSAVE_ID(xsavec64);
     INTRINSIC_X86_XSAVE_ID(xsaves);
     INTRINSIC_X86_XSAVE_ID(xsaves64);
+    INTRINSIC_X86_XSAVE_ID(xsetbv);
     }
 #undef INTRINSIC_X86_XSAVE_ID
     Value *Mhi = Builder.CreateTrunc(
       Builder.CreateLShr(Ops[1], ConstantInt::get(Int64Ty, 32)), Int32Ty);
     Value *Mlo = Builder.CreateTrunc(Ops[1], Int32Ty);
     Ops[1] = Mhi;
     Ops.push_back(Mlo);
     return Builder.CreateCall(CGM.getIntrinsic(ID), Ops);
   }
+  case X86::BI__builtin_ia32_xgetbv: {
+    return Builder.CreateCall(CGM.getIntrinsic(Intrinsic::x86_xgetbv), Ops);
+  }
   case X86::BI__builtin_ia32_storedqudi128_mask:
   case X86::BI__builtin_ia32_storedqusi128_mask:
   case X86::BI__builtin_ia32_storedquhi128_mask:
   case X86::BI__builtin_ia32_storedquqi128_mask:
   case X86::BI__builtin_ia32_storeupd128_mask:
   case X86::BI__builtin_ia32_storeups128_mask:
   case X86::BI__builtin_ia32_storedqudi256_mask:
   case X86::BI__builtin_ia32_storedqusi256_mask:

lib/Headers/intrin.h

 void _Store_HLERelease(long volatile *, long);
 void _Store64_HLERelease(__int64 volatile *, __int64);
 void _StorePointer_HLERelease(void *volatile *, void *);
 static __inline__
 void _WriteBarrier(void);
 unsigned __int32 xbegin(void);
 void _xend(void);
 static __inline__
 #define _XCR_XFEATURE_ENABLED_MASK 0
-unsigned __int64 __cdecl _xgetbv(unsigned int);
-void __cdecl _xsetbv(unsigned int, unsigned __int64);
 
 /* These additional intrinsics are turned on in x64/amd64/x86_64 mode. */
 #ifdef __x86_64__
 void __addgsbyte(unsigned long, unsigned char);
 void __addgsdword(unsigned long, unsigned long);
 void __addgsqword(unsigned long, unsigned __int64);
 void __addgsword(unsigned long, unsigned short);
 static __inline__
__cpuid(int __info[4], int __level) {
   __asm__ ("cpuid" : "=a"(__info[0]), "=b" (__info[1]), "=c"(__info[2]), "=d"(__info[3])
                    : "a"(__level));
 }
 static __inline__ void __DEFAULT_FN_ATTRS
 __cpuidex(int __info[4], int __level, int __ecx) {
   __asm__ ("cpuid" : "=a"(__info[0]), "=b" (__info[1]), "=c"(__info[2]), "=d"(__info[3])
                    : "a"(__level), "c"(__ecx));
 }
-static __inline__ unsigned __int64 __cdecl __DEFAULT_FN_ATTRS
-_xgetbv(unsigned int __xcr_no) {
-  unsigned int __eax, __edx;
-  __asm__ ("xgetbv" : "=a" (__eax), "=d" (__edx) : "c" (__xcr_no));
-  return ((unsigned __int64)__edx << 32) | __eax;
-}
 static __inline__ void __DEFAULT_FN_ATTRS
 __halt(void) {
   __asm__ volatile ("hlt");
 }
 #endif
 
 /*----------------------------------------------------------------------------*\
 |* Privileged intrinsics
 \*----------------------------------------------------------------------------*/

lib/Headers/xsaveintrin.h

_xsave(void *__p, unsigned long long __m) {
   return __builtin_ia32_xsave(__p, __m);
 }
 
 static __inline__ void __DEFAULT_FN_ATTRS
 _xrstor(void *__p, unsigned long long __m) {
   return __builtin_ia32_xrstor(__p, __m);
 }
 
+static __inline__ unsigned long long __DEFAULT_FN_ATTRS
+_xgetbv(unsigned int __a) {
+  return  __builtin_ia32_xgetbv(__a);
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS
+_xsetbv(unsigned int __a, unsigned long long __b) {
+  __builtin_ia32_xsetbv(__a, __b);
+}
+
 #ifdef __x86_64__
 static __inline__ void __DEFAULT_FN_ATTRS
 _xsave64(void *__p, unsigned long long __m) {
   return __builtin_ia32_xsave64(__p, __m);
 }
 
 static __inline__ void __DEFAULT_FN_ATTRS
 _xrstor64(void *__p, unsigned long long __m) {
   return __builtin_ia32_xrstor64(__p, __m);
 }
+
 #endif
 
 #undef __DEFAULT_FN_ATTRS
 
 #endif

test/CodeGen/builtins-x86.c

#endif
   tmp_Ui = __builtin_ia32_stmxcsr();
   (void)__builtin_ia32_fxsave(tmp_vp);
   (void)__builtin_ia32_fxsave64(tmp_vp);
   (void)__builtin_ia32_fxrstor(tmp_vp);
   (void)__builtin_ia32_fxrstor64(tmp_vp);
 
   (void)__builtin_ia32_xsave(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xsave64(tmp_vp, tmp_ULLi);
+  tmp_ULLi = __builtin_ia32_xgetbv(tmp_Ui);
+  (void)__builtin_ia32_xsetbv(tmp_Ui, tmp_ULLi);
   (void)__builtin_ia32_xrstor(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xrstor64(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xsaveopt(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xsaveopt64(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xrstors(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xrstors64(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xsavec(tmp_vp, tmp_ULLi);
   (void)__builtin_ia32_xsavec64(tmp_vp, tmp_ULLi);

test/CodeGen/x86_32-xsave.c

 // RUN: %clang_cc1 %s -DTEST_XSAVE -O0 -triple=i686-unknown-unknown -target-feature +xsave -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVE
 // RUN: %clang_cc1 %s -DTEST_XSAVE -O0 -triple=i686-unknown-unknown -target-feature +xsave -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVE
 
+// RUN: %clang_cc1 %s -DTEST_XGETBV -O0 -triple=i686-unknown-unknown -target-feature +xsave -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XGETBV
+// RUN: %clang_cc1 %s -DTEST_XSETBV -O0 -triple=i686-unknown-unknown -target-feature +xsave -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSETBV
+
 // RUN: %clang_cc1 %s -DTEST_XSAVEOPT -O0 -triple=i686-unknown-unknown -target-feature +xsave -target-feature +xsaveopt -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEOPT
 // RUN: %clang_cc1 %s -DTEST_XSAVEOPT -O0 -triple=i686-unknown-unknown -target-feature +xsave -target-feature +xsaveopt -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEOPT
 
 // RUN: %clang_cc1 %s -DTEST_XSAVEC -O0 -triple=i686-unknown-unknown -target-feature +xsave -target-feature +xsavec -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEC
 // RUN: %clang_cc1 %s -DTEST_XSAVEC -O0 -triple=i686-unknown-unknown -target-feature +xsave -target-feature +xsavec -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEC
 
 // RUN: %clang_cc1 %s -DTEST_XSAVES -O0 -triple=i686-unknown-unknown -target-feature +xsave -target-feature +xsaves -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVES
 // RUN: %clang_cc1 %s -DTEST_XSAVES -O0 -triple=i686-unknown-unknown -target-feature +xsave -target-feature +xsaves -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVES
 
+// Don't include mm_malloc.h, it's system specific.
+#define __MM_MALLOC_H
+#include <x86intrin.h>
+
 void test() {
   unsigned long long tmp_ULLi;
+  unsigned int       tmp_Ui;
   void*              tmp_vp;
 
 #ifdef TEST_XSAVE
 // XSAVE: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 4
 // XSAVE: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVE: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVE: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVE: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32
 // XSAVE: call void @llvm.x86.xsave(i8* [[tmp_vp_1]], i32 [[high32_1]], i32 [[low32_1]])
   (void)__builtin_ia32_xsave(tmp_vp, tmp_ULLi);
 
 // XSAVE: [[tmp_vp_3:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 4
 // XSAVE: [[tmp_ULLi_3:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVE: [[high64_3:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_3]], 32
 // XSAVE: [[high32_3:%[0-9a-zA-z]+]] = trunc i64 [[high64_3]] to i32
 // XSAVE: [[low32_3:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_3]] to i32
 // XSAVE: call void @llvm.x86.xrstor(i8* [[tmp_vp_3]], i32 [[high32_3]], i32 [[low32_3]])
   (void)__builtin_ia32_xrstor(tmp_vp, tmp_ULLi);
+  
+// XSAVE: call void @llvm.x86.xsave
+  (void)_xsave(tmp_vp, tmp_ULLi);
+  
+// XSAVE: call void @llvm.x86.xrstor
+  (void)_xrstor(tmp_vp, tmp_ULLi);
 #endif
 
 #ifdef TEST_XSAVEOPT
 // XSAVEOPT: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 4
 // XSAVEOPT: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVEOPT: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVEOPT: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVEOPT: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32
 // XSAVEOPT: call void @llvm.x86.xsaveopt(i8* [[tmp_vp_1]], i32 [[high32_1]], i32 [[low32_1]])
   (void)__builtin_ia32_xsaveopt(tmp_vp, tmp_ULLi);
+  
+// XSAVEOPT: call void @llvm.x86.xsaveopt
+  (void)_xsaveopt(tmp_vp, tmp_ULLi);
 #endif
 
 #ifdef TEST_XSAVEC
 // XSAVEC: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 4
 // XSAVEC: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVEC: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVEC: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVEC: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32
 // XSAVEC: call void @llvm.x86.xsavec(i8* [[tmp_vp_1]], i32 [[high32_1]], i32 [[low32_1]])
   (void)__builtin_ia32_xsavec(tmp_vp, tmp_ULLi);
+ 
+// XSAVEC: call void @llvm.x86.xsavec 
+  (void)_xsavec(tmp_vp, tmp_ULLi);
 #endif
 
 #ifdef TEST_XSAVES
 // XSAVES: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 4
 // XSAVES: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVES: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVES: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVES: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32
 // XSAVES: call void @llvm.x86.xsaves(i8* [[tmp_vp_1]], i32 [[high32_1]], i32 [[low32_1]])
   (void)__builtin_ia32_xsaves(tmp_vp, tmp_ULLi);
 
 // XSAVES: [[tmp_vp_3:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 4
 // XSAVES: [[tmp_ULLi_3:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVES: [[high64_3:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_3]], 32
 // XSAVES: [[high32_3:%[0-9a-zA-z]+]] = trunc i64 [[high64_3]] to i32
 // XSAVES: [[low32_3:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_3]] to i32
 // XSAVES: call void @llvm.x86.xrstors(i8* [[tmp_vp_3]], i32 [[high32_3]], i32 [[low32_3]])
   (void)__builtin_ia32_xrstors(tmp_vp, tmp_ULLi);
+  
+// XSAVES: call void @llvm.x86.xsaves
+  (void)_xsaves(tmp_vp, tmp_ULLi); 
+
+// XSAVES: call void @llvm.x86.xrstors
+  (void)_xrstors(tmp_vp, tmp_ULLi);
 #endif
-}
+
+#ifdef TEST_XGETBV
+// XGETBV: [[tmp_Ui:%[0-9a-zA-z]+]] = load i32, i32* %tmp_Ui, align 4
+// XGETBV: call i64 @llvm.x86.xgetbv(i32 [[tmp_Ui]])
+  tmp_ULLi = __builtin_ia32_xgetbv(tmp_Ui);
+  
+// XGETBV: call i64 @llvm.x86.xgetbv
+  tmp_ULLi = _xgetbv(tmp_Ui);
+#endif
+
+#ifdef TEST_XSETBV
+// XSETBV: [[tmp_Ui:%[0-9a-zA-z]+]] = load i32, i32* %tmp_Ui, align 4
+// XSETBV: [[tmp_ULLi_3:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
+// XSETBV: [[high64_3:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_3]], 32
+// XSETBV: [[high32_3:%[0-9a-zA-z]+]] = trunc i64 [[high64_3]] to i32
+// XSETBV: [[low32_3:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_3]] to i32
+// XSETBV: call void @llvm.x86.xsetbv(i32 [[tmp_Ui]], i32 [[high32_3]], i32 [[low32_3]])
+  (void)__builtin_ia32_xsetbv(tmp_Ui, tmp_ULLi);
+  
+  // XSETBV: call void @llvm.x86.xsetbv
+  (void)_xsetbv(tmp_Ui, tmp_ULLi);
+#endif
+
+}
 No newline at end of file

test/CodeGen/x86_64-xsave.c

 // RUN: %clang_cc1 %s -DTEST_XSAVE -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVE
 // RUN: %clang_cc1 %s -DTEST_XSAVE -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVE
 
+// RUN: %clang_cc1 %s -DTEST_XGETBV -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XGETBV
+// RUN: %clang_cc1 %s -DTEST_XSETBV -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSETBV
+
 // RUN: %clang_cc1 %s -DTEST_XSAVEOPT -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -target-feature +xsaveopt -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEOPT
 // RUN: %clang_cc1 %s -DTEST_XSAVEOPT -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -target-feature +xsaveopt -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEOPT
 
 // RUN: %clang_cc1 %s -DTEST_XSAVEC -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -target-feature +xsavec -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEC
 // RUN: %clang_cc1 %s -DTEST_XSAVEC -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -target-feature +xsavec -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVEC
 
 // RUN: %clang_cc1 %s -DTEST_XSAVES -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -target-feature +xsaves -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVES
 // RUN: %clang_cc1 %s -DTEST_XSAVES -O0 -triple=x86_64-unknown-unknown -target-feature +xsave -target-feature +xsaves -fno-signed-char -emit-llvm -o - -Werror | FileCheck %s --check-prefix=XSAVES
 
+// Don't include mm_malloc.h, it's system specific.
+#define __MM_MALLOC_H
+#include <x86intrin.h>
+
+
 void test() {
   unsigned long long tmp_ULLi;
+  unsigned int       tmp_Ui;
   void*              tmp_vp;
 
 #ifdef TEST_XSAVE
 // XSAVE: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVE: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVE: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVE: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVE: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32

 
 // XSAVE: [[tmp_vp_4:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVE: [[tmp_ULLi_4:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVE: [[high64_4:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_4]], 32
 // XSAVE: [[high32_4:%[0-9a-zA-z]+]] = trunc i64 [[high64_4]] to i32
 // XSAVE: [[low32_4:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_4]] to i32
 // XSAVE: call void @llvm.x86.xrstor64(i8* [[tmp_vp_4]], i32 [[high32_4]], i32 [[low32_4]])
   (void)__builtin_ia32_xrstor64(tmp_vp, tmp_ULLi);
+  
+// XSAVE: call void @llvm.x86.xsave
+  (void)_xsave(tmp_vp, tmp_ULLi);
+  
+// XSAVE: call void @llvm.x86.xsave64
+  (void)_xsave64(tmp_vp, tmp_ULLi);
+  
+// XSAVE: call void @llvm.x86.xrstor
+  (void)_xrstor(tmp_vp, tmp_ULLi);
+  
+// XSAVE: call void @llvm.x86.xrstor64
+  (void)_xrstor64(tmp_vp, tmp_ULLi);
 #endif
 
 #ifdef TEST_XSAVEOPT
 // XSAVEOPT: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVEOPT: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVEOPT: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVEOPT: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVEOPT: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32
 // XSAVEOPT: call void @llvm.x86.xsaveopt(i8* [[tmp_vp_1]], i32 [[high32_1]], i32 [[low32_1]])
   (void)__builtin_ia32_xsaveopt(tmp_vp, tmp_ULLi);
 
 // XSAVEOPT: [[tmp_vp_2:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVEOPT: [[tmp_ULLi_2:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVEOPT: [[high64_2:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_2]], 32
 // XSAVEOPT: [[high32_2:%[0-9a-zA-z]+]] = trunc i64 [[high64_2]] to i32
 // XSAVEOPT: [[low32_2:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_2]] to i32
 // XSAVEOPT: call void @llvm.x86.xsaveopt64(i8* [[tmp_vp_2]], i32 [[high32_2]], i32 [[low32_2]])
   (void)__builtin_ia32_xsaveopt64(tmp_vp, tmp_ULLi);
+  
+// XSAVEOPT: call void @llvm.x86.xsaveopt
+  (void)_xsaveopt(tmp_vp, tmp_ULLi);
+  
+// XSAVEOPT: call void @llvm.x86.xsaveopt64
+  (void)_xsaveopt64(tmp_vp, tmp_ULLi);
 #endif
 
 #ifdef TEST_XSAVEC
 // XSAVEC: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVEC: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVEC: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVEC: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVEC: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32
 // XSAVEC: call void @llvm.x86.xsavec(i8* [[tmp_vp_1]], i32 [[high32_1]], i32 [[low32_1]])
   (void)__builtin_ia32_xsavec(tmp_vp, tmp_ULLi);
 
 // XSAVEC: [[tmp_vp_2:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVEC: [[tmp_ULLi_2:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVEC: [[high64_2:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_2]], 32
 // XSAVEC: [[high32_2:%[0-9a-zA-z]+]] = trunc i64 [[high64_2]] to i32
 // XSAVEC: [[low32_2:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_2]] to i32
 // XSAVEC: call void @llvm.x86.xsavec64(i8* [[tmp_vp_2]], i32 [[high32_2]], i32 [[low32_2]])
   (void)__builtin_ia32_xsavec64(tmp_vp, tmp_ULLi);
+  
+// XSAVEC: call void @llvm.x86.xsavec 
+  (void)_xsavec(tmp_vp, tmp_ULLi);
+  
+// XSAVEC: call void @llvm.x86.xsavec64
+  (void)_xsavec64(tmp_vp, tmp_ULLi);
 #endif
 
 #ifdef TEST_XSAVES
 // XSAVES: [[tmp_vp_1:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVES: [[tmp_ULLi_1:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVES: [[high64_1:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_1]], 32
 // XSAVES: [[high32_1:%[0-9a-zA-z]+]] = trunc i64 [[high64_1]] to i32
 // XSAVES: [[low32_1:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_1]] to i32

 
 // XSAVES: [[tmp_vp_4:%[0-9a-zA-z]+]] = load i8*, i8** %tmp_vp, align 8
 // XSAVES: [[tmp_ULLi_4:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
 // XSAVES: [[high64_4:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_4]], 32
 // XSAVES: [[high32_4:%[0-9a-zA-z]+]] = trunc i64 [[high64_4]] to i32
 // XSAVES: [[low32_4:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_4]] to i32
 // XSAVES: call void @llvm.x86.xrstors64(i8* [[tmp_vp_4]], i32 [[high32_4]], i32 [[low32_4]])
   (void)__builtin_ia32_xrstors64(tmp_vp, tmp_ULLi);
+  
+// XSAVES: call void @llvm.x86.xsaves
+  (void)_xsaves(tmp_vp, tmp_ULLi); 
+  
+// XSAVES: call void @llvm.x86.xsaves64
+  (void)_xsaves64(tmp_vp, tmp_ULLi); 
+
+// XSAVES: call void @llvm.x86.xrstors
+  (void)_xrstors(tmp_vp, tmp_ULLi);
+  
+// XSAVES: call void @llvm.x86.xrstors64
+  (void)_xrstors64(tmp_vp, tmp_ULLi);
+#endif
+
+#ifdef TEST_XGETBV
+// XGETBV: [[tmp_Ui:%[0-9a-zA-z]+]] = load i32, i32* %tmp_Ui, align 4
+// XGETBV: call i64 @llvm.x86.xgetbv(i32 [[tmp_Ui]])
+  tmp_ULLi = __builtin_ia32_xgetbv(tmp_Ui);
+  
+// XGETBV: call i64 @llvm.x86.xgetbv
+  tmp_ULLi = _xgetbv(tmp_Ui);
+#endif
+
+#ifdef TEST_XSETBV
+// XSETBV: [[tmp_Ui:%[0-9a-zA-z]+]] = load i32, i32* %tmp_Ui, align 4
+// XSETBV: [[tmp_ULLi_3:%[0-9a-zA-z]+]] = load i64, i64* %tmp_ULLi, align 8
+// XSETBV: [[high64_3:%[0-9a-zA-z]+]] = lshr i64 [[tmp_ULLi_3]], 32
+// XSETBV: [[high32_3:%[0-9a-zA-z]+]] = trunc i64 [[high64_3]] to i32
+// XSETBV: [[low32_3:%[0-9a-zA-z]+]] = trunc i64 [[tmp_ULLi_3]] to i32
+// XSETBV: call void @llvm.x86.xsetbv(i32 [[tmp_Ui]], i32 [[high32_3]], i32 [[low32_3]])
+  (void)__builtin_ia32_xsetbv(tmp_Ui, tmp_ULLi);
+  
+  // XSETBV: call void @llvm.x86.xsetbv
+  (void)_xsetbv(tmp_Ui, tmp_ULLi);
 #endif
 }