[X86] Don't allow floating-point return types when SSE[12] is disabled

[rnk]

Summary

The following program hits a fatal_error in the X86 backend, when the
program is compiled with -mno-sse or -mno-sse2, which is understandable
due to the calling convention:

float f() { return 0.5f; };

since the error occurs in the backend, there are stack traces and bug
report messages that are generated.

This patch allows clang to avoid crashing and report a proper diagnostic.

No tests are included in this patch, since it was already tested in llvm's codegen test suite.

[RKSimon]

Please can you add the context to the diff?

[thegameg]

In D27522#615861, @RKSimon wrote:

Please can you add the context to the diff?

I'm sorry, I'm not sure I understand what you mean by "context". Should I include the comments in the .patch? Should I add the comments added in the revision as comments in the code?

[RKSimon]

In D27522#615894, @thegameg wrote:

In D27522#615861, @RKSimon wrote:

Please can you add the context to the diff?

I'm sorry, I'm not sure I understand what you mean by "context". Should I include the comments in the .patch? Should I add the comments added in the revision as comments in the code?

http://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

Use svn diff --diff-cmd=diff -x -U999999 to create patches that include surrounding context so its easier to work out how the diff fits into the rest of the source.

Cheers, Simon

[thegameg]

More context added, thanks RKSimon.

[joerg]

There are two issues here. First of all, it is missing a test case. The second issue is that I somewhat don't like the local diagnose_fatal_error definition, I wonder if SDAG should provide such a helper in general.

[thegameg]

In D27522#630412, @joerg wrote:

There are two issues here. First of all, it is missing a test case.

There are already tests on this, but now it shows up in clang as just a diagnostic instead of a crash.

nosse-error1.ll
nosse-error2.ll

The second issue is that I somewhat don't like the local diagnose_fatal_error definition, I wonder if SDAG should provide such a helper in general.

I see. Should I just inline the code for each call?

Also, should FastISel be handling this too ?

[rnk]

I'll clean this up and land it with a clang-side test. We get too many issues about this.

[rnk]

• clean up, add test

lib/Target/X86/X86ISelLowering.cpp

 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCExpr.h"

              "rather than promotion."),
     cl::Hidden);
 
+namespace {
+void diagnose_fatal_error(SelectionDAG &DAG, const SDLoc &dl, const char *msg) {
+  MachineFunction &MF = DAG.getMachineFunction();
+  DAG.getContext()->diagnose(
+      DiagnosticInfoUnsupported(*MF.getFunction(), msg, dl.getDebugLoc()));
+}
+}
+
 X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
                                      const X86Subtarget &STI)
     : TargetLowering(TM), Subtarget(STI) {

     // or SSE or MMX vectors.
     if ((ValVT == MVT::f32 || ValVT == MVT::f64 ||
          VA.getLocReg() == X86::XMM0 || VA.getLocReg() == X86::XMM1) &&
-          (Subtarget.is64Bit() && !Subtarget.hasSSE1())) {
-      report_fatal_error("SSE register return with SSE disabled");
+      diagnose_fatal_error(DAG, dl, "SSE register return with SSE disabled");
+      VA.convertToReg(X86::FP0); // Set reg to FP0, avoid hitting asserts.
     }
     // Likewise we can't return F64 values with SSE1 only.  gcc does so, but
     // llvm-gcc has never done it right and no one has noticed, so this
     // should be OK for now.
-    if (ValVT == MVT::f64 &&
-        (Subtarget.is64Bit() && !Subtarget.hasSSE2()))
-      report_fatal_error("SSE2 register return with SSE2 disabled");
+    if (ValVT == MVT::f64 && (Subtarget.is64Bit() && !Subtarget.hasSSE2())) {
+      diagnose_fatal_error(DAG, dl, "SSE2 register return with SSE2 disabled");
+      VA.convertToReg(X86::FP0); // Set reg to FP0, avoid hitting asserts.
+    }
 
     // Returns in ST0/ST1 are handled specially: these are pushed as operands to
     // the RET instruction and handled by the FP Stackifier.

     // If this is x86-64, and we disabled SSE, we can't return FP values
     if ((CopyVT == MVT::f32 || CopyVT == MVT::f64 || CopyVT == MVT::f128) &&
         ((Is64Bit || Ins[InsIndex].Flags.isInReg()) && !Subtarget.hasSSE1())) {
-      report_fatal_error("SSE register return with SSE disabled");
+      diagnose_fatal_error(DAG, dl, "SSE register return with SSE disabled");
+      VA.convertToReg(X86::FP0); // Set reg to FP0, avoid hitting asserts.
     }
 
     // If we prefer to use the value in xmm registers, copy it out as f80 and