Automatic variable initialization

[jfb]

Summary

Add an option to initialize automatic variables with either a pattern or with
zeroes. The default is still that automatic variables are uninitialized. Also
add attributes to request uninitialized on a per-variable basis, mainly to disable
initialization of large stack arrays when deemed too expensive.

This isn't meant to change the semantics of C and C++. Rather, it's meant to be
a last-resort when programmers inadvertently have some undefined behavior in
their code. This patch aims to make undefined behavior hurt less, which
security-minded people will be very happy about. Notably, this means that
there's no inadvertent information leak when:

• The compiler re-uses stack slots, and a value is used uninitialized.
• The compiler re-uses a register, and a value is used uninitialized.
• Stack structs / arrays / unions with padding are copied.

This patch only addresses stack and register information leaks. There's many
more infoleaks that we could address, and much more undefined behavior that
could be tamed. Let's keep this patch focused, and I'm happy to address related
issues elsewhere.

To keep the patch simple, only some undef is removed for now, see
replaceUndef. The padding-related infoleaks are therefore not all gone yet.
This will be addressed in a follow-up, mainly because addressing padding-related
leaks should be a stand-alone option which is implied by variable
initialization.

There are three options when it comes to automatic variable initialization:

0. Uninitialized

  This is C and C++'s default. It's not changing. Depending on code
  generation, a programmer who runs into undefined behavior by using an
  uninialized automatic variable may observe any previous value (including
  program secrets), or any value which the compiler saw fit to materialize on
  the stack or in a register (this could be to synthesize an immediate, to
  refer to code or data locations, to generate cookies, etc).

1. Pattern initialization

  This is the recommended initialization approach. Pattern initialization's
  goal is to initialize automatic variables with values which will likely
  transform logic bugs into crashes down the line, are easily recognizable in
  a crash dump, without being values which programmers can rely on for useful
  program semantics. At the same time, pattern initialization tries to
  generate code which will optimize well. You'll find the following details in
  `patternFor`:

  - Integers are initialized with repeated 0xAA bytes (infinite scream).
  - Vectors of integers are also initialized with infinite scream.
  - Pointers are initialized with infinite scream on 64-bit platforms because
    it's an unmappable pointer value on architectures I'm aware of. Pointers
    are initialize to 0x000000AA (small scream) on 32-bit platforms because
    32-bit platforms don't consistently offer unmappable pages. When they do
    it's usually the zero page. As people try this out, I expect that we'll
    want to allow different platforms to customize this, let's do so later.
  - Vectors of pointers are initialized the same way pointers are.
  - Floating point values and vectors are initialized with a negative quiet NaN
    with repeated 0xFF payload (e.g. 0xffffffff and 0xffffffffffffffff). NaNs are nice
    (here, anways) because they propagate on arithmetic, making it more likely
    that entire computations become NaN when a single uninitialized value
    sneaks in.
  - Arrays are initialized to their homogeneous elements' initialization
    value, repeated. Stack-based Variable-Length Arrays (VLAs) are
    runtime-initialized to the allocated size (no effort is made for negative
    size, but zero-sized VLAs are untouched even if technically undefined).
  - Structs are initialized to their heterogeneous element's initialization
    values. Zero-size structs are initialized as 0xAA since they're allocated
    a single byte.
  - Unions are initialized using the initialization for the largest member of
    the union.

  Expect the values used for pattern initialization to change over time, as we
  refine heuristics (both for performance and security). The goal is truly to
  avoid injecting semantics into undefined behavior, and we should be
  comfortable changing these values when there's a worthwhile point in doing
  so.

  Why so much infinite scream? Repeated byte patterns tend to be easy to
  synthesize on most architectures, and otherwise memset is usually very
  efficient. For values which aren't entirely repeated byte patterns, LLVM
  will often generate code which does memset + a few stores.

2. Zero initialization

  Zero initialize all values. This has the unfortunate side-effect of
  providing semantics to otherwise undefined behavior, programs therefore
  might start to rely on this behavior, and that's sad. However, some
  programmers believe that pattern initialization is too expensive for them,
  and data might show that they're right. The only way to make these
  programmers wrong is to offer zero-initialization as an option, figure out
  where they are right, and optimize the compiler into submission. Until the
  compiler provides acceptable performance for all security-minded code, zero
  initialization is a useful (if blunt) tool.

I've been asked for a fourth initialization option: user-provided byte value.
This might be useful, and can easily be added later.

Why is an out-of band initialization mecanism desired? We could instead use
-Wuninitialized! Indeed we could, but then we're forcing the programmer to
provide semantics for something which doesn't actually have any (it's
uninitialized!). It's then unclear whether int derp = 0; lends meaning to 0,
or whether it's just there to shut that warning up. It's also way easier to use
a compiler flag than it is to manually and intelligently initialize all values
in a program.

Why not just rely on static analysis? Because it cannot reason about all dynamic
code paths effectively, and it has false positives. It's a great tool, could get
even better, but it's simply incapable of catching all uses of uninitialized
values.

Why not just rely on memory sanitizer? Because it's not universally available,
has a 3x performance cost, and shouldn't be deployed in production. Again, it's
a great tool, it'll find the dynamic uses of uninitialized variables that your
test coverage hits, but it won't find the ones that you encounter in production.

What's the performance like? Not too bad! Previous publications [0] have cited
2.7 to 4.5% averages. We've commmitted a few patches over the last few months to
address specific regressions, both in code size and performance. In all cases,
the optimizations are generally useful, but variable initialization benefits
from them a lot more than regular code does. We've got a handful of other
optimizations in mind, but the code is in good enough shape and has found enough
latent issues that it's a good time to get the change reviewed, checked in, and
have others kick the tires. We'll continue reducing overheads as we try this out
on diverse codebases.

Is it a good idea? Security-minded folks think so, and apparently so does the
Microsoft Visual Studio team [1] who say "Between 2017 and mid 2018, this
feature would have killed 49 MSRC cases that involved uninitialized struct data
leaking across a trust boundary. It would have also mitigated a number of bugs
involving uninitialized struct data being used directly.". They seem to use pure
zero initialization, and claim to have taken the overheads down to within noise.
Don't just trust Microsoft though, here's another relevant person asking for
this [2]. It's been proposed for GCC [3] and LLVM [4] before.

What are the caveats? A few!

• Variables declared in unreachable code, and used later, aren't initialized. This goto, Duff's device, other objectionable uses of switch. This should instead be a hard-error in any serious codebase.
• Volatile stack variables are still weird. That's pre-existing, it's really the language's fault and this patch keeps it weird. We should deprecate volatile [5].
• As noted above, padding isn't fully handled yet.

I don't think these caveats make the patch untenable because they can be
addressed separately.

Should this be on by default? Maybe, in some circumstances. It's a conversation
we can have when we've tried it out sufficiently, and we're confident that we've
eliminated enough of the overheads that most codebases would want to opt-in.
Let's keep our precious undefined behavior until that point in time.

How do I use it:

1. On the command-line:
   
   -ftrivial-auto-var-init=uninitialized (the default) -ftrivial-auto-var-init=pattern -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
2. Using an attribute:
   
   int dont_initialize_me __attribute((uninitialized));
   
   [0]: https://users.elis.ugent.be/~jsartor/researchDocs/OOPSLA2011Zero-submit.pdf [1]: https://twitter.com/JosephBialek/status/1062774315098112001 [2]: https://outflux.net/slides/2018/lss/danger.pdf [3]: https://gcc.gnu.org/ml/gcc-patches/2014-06/msg00615.html [4]: https://github.com/AndroidHardeningArchive/platform_external_clang/commit/776a0955ef6686d23a82d2e6a3cbd4a6a882c31c [5]: http://wg21.link/p1152

rdar://problem/39131435

I've also posted an RFC to cfe-dev: http://lists.llvm.org/pipermail/cfe-dev/2018-November/060172.html

[t.p.northover]

This isn't meant to change the semantics of C and C++.

As I said in the cfe-dev thread, that's exactly what it does. Specifically I think this is essentially defining a new dialect of C++, which I have massive concerns about. Additionally, as much as we might claim it's a transitional measure, we all know that's not how it'll be used in practice.

[kcc]

exciting. adding more folks FYI and to help review

[jfb]

In D54604#1300544, @t.p.northover wrote:

This isn't meant to change the semantics of C and C++.

As I said in the cfe-dev thread, that's exactly what it does. Specifically I think this is essentially defining a new dialect of C++, which I have massive concerns about. Additionally, as much as we might claim it's a transitional measure, we all know that's not how it'll be used in practice.

(cross-posting from cfe-dev)

I suspected that you’d disagree with zero-initialization. :-)

I tried to outline why I think it’s important:

• Security-minded people think the need it. They might be right.
• We need data to prove that they don’t need it.

Forgive my use of silly rhetoric: I’m sure you agree that more data is good! Here’s what I propose, since you cared enough to voice your opinion: let’s worth together to narrow the performance gap. I’ve got a handful of optimization opportunities, and you know the backend better than I do. Once we’ve addressed the issues we know about, I’m sure other types of codebases will surface other performance issues, let’s address those too.

Once people who think they need zero initialization are proven wrong though compiler optimizations, we can deprecate (and eventually remove / repurpose) the flag. If they’re not proven wrong… we’ll have learned something.

Sounds fair?

[lebedev.ri]

There is also D54473 [sanitizers] Initial implementation for -fsanitize=init-locals.

[kcc]

Would it be possible to extend test/Sema/uninit-variables.c to verify that the new option
doesn't break -Wuninitialized?

[jfb]

• Make sure uninit-variables.c doesn't break.

[jfb]

In D54604#1301807, @kcc wrote:

Would it be possible to extend test/Sema/uninit-variables.c to verify that the new option
doesn't break -Wuninitialized?

Done.

[pcc]

include/clang/Driver/ToolChain.h

355

I wouldn't introduce this function because nothing is overriding it (for now, at least).

lib/CodeGen/CGDecl.cpp

968

*repeated

977

Do you have test coverage for vectors?

978

Is it necessary to qualify cast with llvm::? It doesn't seem to be qualified elsewhere in this file. (Same below.)

1145

Maybe use User::operands() to enumerate operands here?

1627

Is it possible to rewrite this function using early returns to avoid all the indentation below for the VLA case? I'd also put the code for non-VLAs first to make this function easier to understand (on my first reading I somehow got the impression that this function is *only* handling VLAs). I'm thinking maybe something like:

if (trivialAutoVarInit == LangOptions::TrivialAutoVarInitKind::Uninitialized)
  return;

if (!getContext().getTypeSizeInChars(type).isZero()) {
  // handle non-VLAs
}

auto *VlaType = dyn_cast_or_null<VariableArrayType>(getContext().getAsArrayType(type));
if (!VlaType)
  return;

// handle VLAs

1669

I think we need to check that the size is non-zero here. Otherwise we're going to clobber whatever comes after the VLA if its size is zero (I know that C forbids zero-sized VLAs, but I believe we support them as a GNU extension).

test/CodeGenCXX/trivial-auto-var-init.cpp

6

Since this is your only test involving structs and arrays, you probably want to test them beyond the fact that they don't contain undef. I also believe that this is testing for the absence of undef before the first label and not the total absence of undef in the output.

[jfb]

• Make sure uninit-variables.c doesn't break.
• Address Peter's comments, improve tests.

[jfb]

I've addressed @pcc's comments. I'll now update the patch to mandate a -Xclang option for zero-init (otherwise it'll warn and pattern-init instead), and remove documentation for zero-init. I'm also thinking of changing float init to negative NaN with infinite scream payload.

include/clang/Driver/ToolChain.h

355

I expect to do so for particular targets, so it's only a question of time, and for now it makes internal testing easier (I can build my compiler and override it for testing, before sending that upstream). I'd rather do it now, since it reduces merge conflicts as we kick the tires internally.

lib/CodeGen/CGDecl.cpp

977

test/CodeGenCXX/auto-var-init.cpp has those tests. I was lazy with this patch and hadn't updated the tests yet, in case people asked me to change the values. I've done so in this update.

1669

Indeed! Thanks for catching that. I thought I'd done it, and it turns out on O0 it wasn't doing anything bad, and neither was it in O2 because the GEPs were marked inbounds so the codegen magically turned into a memset based on size!

test/CodeGenCXX/trivial-auto-var-init.cpp

6

I was lazy with testing in case the design changed significantly. The updated test explains what I was going for here :)

[jfb]

• Add an ugly option to enable zero init

[jfb]

I added an option that's required when using clang directly (*not* -cc1!) If you use -ftrivial-auto-var-init=zero without -enable-trivial-auto-var-init-zero-knowning-it-will-be-removed-from-clang in clang it'll generate a warning and change initialization to pattern initialization instead. That's roughly along the lines people were going for on cfe-dev.

[jfb]

• Update warning-flags.c

[dexonsmith]

include/clang/Basic/DiagnosticDriverKinds.td

412

s/knowning/knowing/ gives -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang

[jfb]

• Fix typo

[jfb]

• Make sure uninit-variables.c doesn't break.
• Address Peter's comments, improve tests.
• Add an ugly option to enable zero init
• Update warning-flags.c
• Fix typo
• Use negative NaN with repeated 0xFF payload for all floating-point types. This is potentially faster to initialize than pure positive no-payload NaNs.

[rsmith]

For the record: I'm OK with this direction. (I somewhat prefer removing the -enable-long-wordy-thing option and instead automatically disabling the zero option for clang release builds, but I'm OK with either approach.)

include/clang/Basic/Attr.td

3119–3127

I'm unconvinced that we should add this attribute; it seems insufficiently motivated to me, and harmful to portability. I think we should have some way of saying "leave this uninitialized", but zero- and pattern-initialization really seem like things that should simply be written in the source code in the normal way.

I think it would be reasonable to have a __uninitialized__ keyword or similar that can be used as the initializer of an explicitly-uninitialized variable. If you want an attribute for this (for compatibility with other compilers), [[clang::uninitialized]] also seems reasonable. But absent a strong motivation, I do not think we should have an attribute that specifies a particular initial value.

include/clang/Basic/AttrDocs.td

3646–3650

If we keep the attribute, we should not document the zero mode. We don't need to make it discoverable.

include/clang/Basic/DiagnosticDriverKinds.td

409–412

I think this should be an error instead of a warning. We shouldn't silently do something different from what was requested.

[jfb]

• Don't document 'zero'.
• Make drv_trivial_auto_var_init_zero_disabled an error instead of a warning.
• Change the parameter: we only have [[clang::uninitialized]] now.

[jfb]

In D54604#1327893, @rsmith wrote:

For the record: I'm OK with this direction. (I somewhat prefer removing the -enable-long-wordy-thing option and instead automatically disabling the zero option for clang release builds, but I'm OK with either approach.)

I'm inclined to go with what @kcc wants, because I want the data he'll gather. For my own use, I don't really care. One thing I could do: check __DATE__ and generate a warning if it's above a certain value and zero is used? i.e. check the date at which the compiler was compiled (not when it's used!) and error out if it was compiled after a certain point in time? I'm throwing out options that could makes folks more comfortable with zero...

include/clang/Basic/Attr.td

3119–3127

I definitely think that we want *some* attribute: the usecase is primarily to turn on automatic initialization for entire projects, and then turning it off manually where you measure unacceptable runtime costs (until you can do something better, or until the compiler fixes its perf problem). For example, large stack arrays would be something that you'd want to slap this attribute on.

That being said, I agree that __uninitialized__ or [[clang::uninitialized]] would serve this purpose. I figured that mirroring the command-line options would be more desirable, but I'm not sure anymore. I've updated the patch to have [[clang::uninitialized]].

I don't think we want user code to write it in the source: that's forcing user code to add semantics which their code otherwise doesn't have. It's existing practice, but IMO it's bad.

[pcc]

Mostly looking good, a few style nits.

include/clang/Basic/DiagnosticDriverKinds.td

409

s/Warning/Error/ here? Then I think you won't need to adjust the list of warnings without warning flags.

lib/CodeGen/CGDecl.cpp

993

No else after return (same below)

1037

Remove braces

1169

No else after return, no braces.

test/CodeGenCXX/trivial-auto-var-init.cpp

181

PATTERN:

197

PATTERN:

[jfb]

• Address @pcc's comments.

[jfb]

In D54604#1331711, @pcc wrote:

Mostly looking good, a few style nits.

Thanks! I think I addressed all of your comments.

[jfb]

• Rebase

[rsmith]

Thanks, you've addressed my comments to my satisfaction. (Not approving as I've not done a detailed review of the implementation, only of the approach, but it looks like your review with @pcc is nearly complete.)

In D54604#1330356, @jfb wrote:

In D54604#1327893, @rsmith wrote:

For the record: I'm OK with this direction. (I somewhat prefer removing the -enable-long-wordy-thing option and instead automatically disabling the zero option for clang release builds, but I'm OK with either approach.)

I'm inclined to go with what @kcc wants, because I want the data he'll gather. For my own use, I don't really care. One thing I could do: check __DATE__ and generate a warning if it's above a certain value and zero is used? i.e. check the date at which the compiler was compiled (not when it's used!) and error out if it was compiled after a certain point in time? I'm throwing out options that could makes folks more comfortable with zero...

If you want to go there, I'd be OK with that. I don't personally think it's necessary.

[pcc]

LGTM

test/CodeGenCXX/trivial-auto-var-init-attribute.cpp

14

This test should be updated with the new attribute spelling, and should also test initialization. The tests below can be removed as well.

[jfb]

• Update attribute test.

[pcc]

cfe/trunk/include/clang/Driver/Options.td

1657 ↗

(On Diff #178590)

I noticed that this introduces a parsing ambiguity on the command line: the flag -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang can be interpreted either as this flag or as -e nable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang, meaning "tell the linker to use that long symbol name as the program's entry point". Worth fixing (e.g. by renaming to --enable-...)? It's probably a good idea to rename it every so often just to keep people on their toes anyway.

[jfb]

cfe/trunk/include/clang/Driver/Options.td

1657 ↗

(On Diff #178590)

Don't we have the same issue with:

include/clang/Driver/CC1Options.td:def emit_html : Flag<["-"], "emit-html">,
include/clang/Driver/CC1Options.td:def emit_module : Flag<["-"], "emit-module">,
include/clang/Driver/CC1Options.td:def emit_module_interface : Flag<["-"], "emit-module-interface">,
include/clang/Driver/CC1Options.td:def emit_header_module : Flag<["-"], "emit-header-module">,
include/clang/Driver/CC1Options.td:def emit_pch : Flag<["-"], "emit-pch">,
include/clang/Driver/CC1Options.td:def emit_llvm_bc : Flag<["-"], "emit-llvm-bc">,
include/clang/Driver/CC1Options.td:def emit_llvm_only : Flag<["-"], "emit-llvm-only">,
include/clang/Driver/CC1Options.td:def emit_codegen_only : Flag<["-"], "emit-codegen-only">,
include/clang/Driver/CC1Options.td:def emit_obj : Flag<["-"], "emit-obj">,
include/clang/Driver/CC1Options.td:def emit_llvm_uselists : Flag<["-"], "emit-llvm-uselists">,
include/clang/Driver/CC1Options.td:def enable_split_dwarf : Flag<["-"], "enable-split-dwarf">,
include/clang/Driver/CC1Options.td:def enable_split_dwarf_EQ : Joined<["-"], "enable-split-dwarf=">,
include/clang/Driver/CC1Options.td:def error_on_deserialized_pch_decl : Separate<["-"], "error-on-deserialized-decl">,
include/clang/Driver/CC1Options.td:def error_on_deserialized_pch_decl_EQ : Joined<["-"], "error-on-deserialized-decl=">,
include/clang/Driver/Options.td:def emit_ast : Flag<["-"], "emit-ast">,
include/clang/Driver/Options.td:def emit_llvm : Flag<["-"], "emit-llvm">, Flags<[CC1Option]>, Group<Action_Group>,
include/clang/Driver/Options.td:def exported__symbols__list : Separate<["-"], "exported_symbols_list">;
include/clang/Driver/Options.td:def e : JoinedOrSeparate<["-"], "e">, Group<Link_Group>;
include/clang/Driver/Options.td:def enable_trivial_var_init_zero : Joined<["-"], "enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang">,

?

[pcc]

cfe/trunk/include/clang/Driver/Options.td

1657 ↗

(On Diff #178590)

Just the ones in Options.td, I believe (since this is a driver-only flag).

[glider]

Not sure if a similar problem was mentioned already or not, but the following program:

void alloc_sock(int *err);
int try_send();

int send(int len) {
  int err;
  if (len) {
    err = -1; 
    alloc_sock(&err);
    err = try_send();
  }
  return -1; 
}

produces a redundant store of 0xaaaaaaaa to err:

0000000000000000 <send>:
void alloc_sock(int *err);
int try_send();

int send(int len) {
   0:   50                      push   %rax
  int err;
   1:   c7 44 24 04 aa aa aa    movl   $0xaaaaaaaa,0x4(%rsp)
   8:   aa 
  if (len) {
   9:   85 ff                   test   %edi,%edi
   b:   74 1d                   je     2a <send+0x2a>
    err = -1;
   d:   c7 44 24 04 ff ff ff    movl   $0xffffffff,0x4(%rsp)
  14:   ff 
  15:   48 8d 7c 24 04          lea    0x4(%rsp),%rdi
    alloc_sock(&err);
  1a:   e8 00 00 00 00          callq  1f <send+0x1f>
                        1b: R_X86_64_PLT32      alloc_sock-0x4
    err = try_send();
  1f:   31 c0                   xor    %eax,%eax
  21:   e8 00 00 00 00          callq  26 <send+0x26>
                        22: R_X86_64_PLT32      try_send-0x4
  26:   89 44 24 04             mov    %eax,0x4(%rsp)
  }
  return -1;
  2a:   b8 ff ff ff ff          mov    $0xffffffff,%eax
  2f:   59                      pop    %rcx
  30:   c3                      retq

Not sure whether this isn't a bug in DSE though.

[glider]

(Overall, since this is an "unsupported" feature, is it ok to file Bugzilla bugs for it?)

[rjmccall]

I think it's absolutely fair game to file bugs for this.

[jfb]

In D54604#1375514, @glider wrote:

Not sure if a similar problem was mentioned already or not, but the following program:

Yes please file a bug. Seems like the optimizer should sink the store enough that it appears only once on each path.

[glider]

I think doing separate stores of 0xAA into struct members instead of copying part of .rodata over it will let us do a much better job in optimizing away redundant stores.
I've opened https://bugs.llvm.org/show_bug.cgi?id=40605 to track that.

include/clang/Basic/Attr.td

def AlwaysDestroy : InheritableAttr {
   let Documentation = [AlwaysDestroyDocs];
 }
 
 def SpeculativeLoadHardening : InheritableAttr {
   let Spellings = [Clang<"speculative_load_hardening">];
   let Subjects = SubjectList<[Function, ObjCMethod], ErrorDiag>;
   let Documentation = [SpeculativeLoadHardeningDocs];
 }
+
+def TrivialAutoInit : InheritableAttr {
+  let Spellings = [Clang<"trivial_auto_init", 1>];
+  let Args = [EnumArgument<"TrivialAutoVarInit", "TrivialAutoVarInitKind",
+                           ["uninitialized", "zero", "pattern"],
+                           ["Uninitialized", "Zero", "Pattern"]>];
+  let Subjects = SubjectList<[LocalVar]>;
+  let Documentation = [TrivialAutoInitDocs];
+}

include/clang/Basic/AttrDocs.td

def NoDestroyDocs : Documentation {
   let Content = [{
 The ``no_destroy`` attribute specifies that a variable with static or thread
 storage duration shouldn't have its exit-time destructor run. Annotating every
 static and thread duration variable with this attribute is equivalent to
 invoking clang with -fno-c++-static-destructors.
   }];
 }
 
+def TrivialAutoInitDocs : Documentation {
+  let Category = DocCatVariable;
+  let Content = [{
+The command-line parameter ``-ftrivial-auto-var-init=*`` can be used to
+automatically initialize trivial automatic stack variables. By default, trivial
+automatic stack variables are uninitialized. This attribute is used to override
+the command-line parameter, and accepts a single parameter that must be one of
+the following: ``uninitialized``, ``zero``, or ``pattern``.
+  }];
+}
+
 def GnuInlineDocs : Documentation {
   let Category = DocCatFunction;
   let Content = [{
 The ``gnu_inline`` changes the meaning of ``extern inline`` to use GNU inline
 semantics, meaning:
 
 * If any declaration that is declared ``inline`` is not declared ``extern``,
 then the ``inline`` keyword is just a hint. In particular, an out-of-line

include/clang/Basic/DiagnosticDriverKinds.td

 
 def warn_drv_experimental_isel_incomplete_opt : Warning<
   "-fexperimental-isel support is incomplete for this architecture at the current optimization level">,
   InGroup<ExperimentalISel>;
 
 def warn_drv_moutline_unsupported_opt : Warning<
   "The '%0' architecture does not support -moutline; flag ignored">,
   InGroup<OptionIgnored>;
+
+def warn_drv_trivial_auto_var_init_zero_disabled : Warning<
+  "-ftrivial-auto-var-init=zero hasn't been enabled, using pattern initialization instead. "
+  "Enable it at your own peril for benchmarking purpose only with "
+  "-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang">;
+
 }

include/clang/Basic/LangOptions.h

 /// enabled, which controls the dialect of C or C++ that is accepted.
 class LangOptions : public LangOptionsBase {
 public:
   using Visibility = clang::Visibility;
 
   enum GCMode { NonGC, GCOnly, HybridGC };
   enum StackProtectorMode { SSPOff, SSPOn, SSPStrong, SSPReq };
 
+  // Automatic variables live on the stack, and when trivial they're usually
+  // uninitialized because it's undefined behavior to use them without
+  // initializing them.
+  enum class TrivialAutoVarInitKind { Uninitialized, Zero, Pattern };
+
   enum SignedOverflowBehaviorTy {
     // Default C standard behavior.
     SOB_Undefined,
 
     // -fwrapv
     SOB_Defined,
 
     // -ftrapv

include/clang/Basic/LangOptions.def

 
 ENUM_LANGOPT(GC, GCMode, 2, NonGC, "Objective-C Garbage Collection mode")
 ENUM_LANGOPT(ValueVisibilityMode, Visibility, 3, DefaultVisibility,
              "value symbol visibility")
 ENUM_LANGOPT(TypeVisibilityMode, Visibility, 3, DefaultVisibility,
              "type symbol visibility")
 ENUM_LANGOPT(StackProtector, StackProtectorMode, 2, SSPOff,
              "stack protector mode")
+ENUM_LANGOPT(TrivialAutoVarInit, TrivialAutoVarInitKind, 2, TrivialAutoVarInitKind::Uninitialized,
+             "trivial automatic variable initialization")
 ENUM_LANGOPT(SignedOverflowBehavior, SignedOverflowBehaviorTy, 2, SOB_Undefined,
              "signed integer overflow handling")
 
 BENIGN_LANGOPT(ArrowDepth, 32, 256,
                "maximum number of operator->s to follow")
 BENIGN_LANGOPT(InstantiationDepth, 32, 1024,
                "maximum template instantiation depth")
 BENIGN_LANGOPT(ConstexprCallDepth, 32, 512,

include/clang/Driver/Options.td

def fno_signed_char : Flag<["-"], "fno-signed-char">, Group<f_Group>,
     Flags<[CC1Option]>, HelpText<"Char is unsigned">;
 def fsplit_stack : Flag<["-"], "fsplit-stack">, Group<f_Group>;
 def fstack_protector_all : Flag<["-"], "fstack-protector-all">, Group<f_Group>,
   HelpText<"Force the usage of stack protectors for all functions">;
 def fstack_protector_strong : Flag<["-"], "fstack-protector-strong">, Group<f_Group>,
   HelpText<"Use a strong heuristic to apply stack protectors to functions">;
 def fstack_protector : Flag<["-"], "fstack-protector">, Group<f_Group>,
   HelpText<"Enable stack protectors for functions potentially vulnerable to stack smashing">;
+def ftrivial_auto_var_init : Joined<["-"], "ftrivial-auto-var-init=">, Group<f_Group>,
+  Flags<[CC1Option]>, HelpText<"Initialize trivial automatic stack variables: uninitialized (default)"
+  " | pattern">, Values<"uninitialized,pattern">;
+def enable_trivial_var_init_zero : Joined<["-"], "enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang">,
+  Flags<[CC1Option]>,
+  HelpText<"Trivial automatic variable initialization to zero is only here for benchmarks, it'll eventually be removed, and I'm OK with that because I'm only using it to benchmark">;
 def fstandalone_debug : Flag<["-"], "fstandalone-debug">, Group<f_Group>, Flags<[CoreOption]>,
   HelpText<"Emit full debug info for all types used by the program">;
 def fno_standalone_debug : Flag<["-"], "fno-standalone-debug">, Group<f_Group>, Flags<[CoreOption]>,
   HelpText<"Limit debug information produced to reduce size of debug binary">;
 def flimit_debug_info : Flag<["-"], "flimit-debug-info">, Flags<[CoreOption]>, Alias<fno_standalone_debug>;
 def fno_limit_debug_info : Flag<["-"], "fno-limit-debug-info">, Flags<[CoreOption]>, Alias<fstandalone_debug>;
 def fdebug_macro : Flag<["-"], "fdebug-macro">, Group<f_Group>, Flags<[CoreOption]>,
   HelpText<"Emit macro debug information">;

include/clang/Driver/ToolChain.h

 //===- ToolChain.h - Collections of tools for one platform ------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_DRIVER_TOOLCHAIN_H
 #define LLVM_CLANG_DRIVER_TOOLCHAIN_H
 
+#include "clang/Basic/DebugInfoOptions.h"
 #include "clang/Basic/LLVM.h"
+#include "clang/Basic/LangOptions.h"
 #include "clang/Basic/Sanitizers.h"
-#include "clang/Basic/DebugInfoOptions.h"
 #include "clang/Driver/Action.h"
 #include "clang/Driver/Multilib.h"
 #include "clang/Driver/Types.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/MC/MCTargetOptions.h"
public:
   virtual bool useRelaxRelocations() const;
 
   /// GetDefaultStackProtectorLevel - Get the default stack protector level for
   /// this tool chain (0=off, 1=on, 2=strong, 3=all).
   virtual unsigned GetDefaultStackProtectorLevel(bool KernelOrKext) const {
     return 0;
   }
 
+  /// Get the default trivial automatic variable initialization.
+  virtual LangOptions::TrivialAutoVarInitKind
+  GetDefaultTrivialAutoVarInit() const {
+    return LangOptions::TrivialAutoVarInitKind::Uninitialized;
+  }
+
   /// GetDefaultLinker - Get the default linker to use.
   virtual const char *getDefaultLinker() const { return "ld"; }
 
   /// GetDefaultRuntimeLibType - Get the default runtime library variant to use.
   virtual RuntimeLibType GetDefaultRuntimeLibType() const {
     return ToolChain::RLT_Libgcc;
   }
 

lib/CodeGen/CGDecl.cpp

 static llvm::Value *shouldUseMemSetToInitialize(llvm::Constant *Init,
                                                 uint64_t GlobalSize) {
   uint64_t SizeLimit = 32;
   if (GlobalSize <= SizeLimit)
     return nullptr;
   return llvm::isBytewiseValue(Init);
 }
 
+static llvm::Constant *patternFor(CodeGenModule &CGM, llvm::Type *Ty) {
+  // The following value is a guaranteed unmappable pointer value and has a
+  // repeated byte-pattern which makes it easier to synthesize. We use it for
+  // pointers as well as integers so that aggregates are likely to be
+  // initialized with this repeated value.
+  constexpr uint64_t LargeValue = 0xAAAAAAAAAAAAAAAAull;
+  // For 32-bit platforms it's a bit trickier because, across systems, only the
+  // zero page can reasonably be expected to be unmapped, and even then we need
+  // a very low address. We use a smaller value, and that value sadly doesn't
+  // have a repeated byte-pattern. We don't use it for integers.
+  constexpr uint32_t SmallValue = 0x000000AA;
+  if (Ty->isIntOrIntVectorTy()) {
+    unsigned BitWidth = cast<llvm::IntegerType>(
+                            Ty->isVectorTy() ? Ty->getVectorElementType() : Ty)
+                            ->getBitWidth();
+    if (BitWidth <= 64)
+      return llvm::ConstantInt::get(Ty, LargeValue);
+    return llvm::ConstantInt::get(
+        Ty, llvm::APInt::getSplat(BitWidth, llvm::APInt(64, LargeValue)));
+  } else if (Ty->isPtrOrPtrVectorTy()) {
+    auto *PtrTy = cast<llvm::PointerType>(
+        Ty->isVectorTy() ? Ty->getVectorElementType() : Ty);
+    unsigned PtrWidth = CGM.getContext().getTargetInfo().getPointerWidth(
+        PtrTy->getAddressSpace());
+    llvm::Type *IntTy = llvm::IntegerType::get(CGM.getLLVMContext(), PtrWidth);
+    uint64_t IntValue;
+    switch (PtrWidth) {
+    default:
+      llvm_unreachable("pattern initialization of unsupported pointer width");
+    case 64:
+      IntValue = LargeValue;
+      break;
+    case 32:
+      IntValue = SmallValue;
+      break;
+    }
+    auto *Int = llvm::ConstantInt::get(IntTy, IntValue);
+    return llvm::ConstantExpr::getIntToPtr(Int, PtrTy);
+  } else if (Ty->isFPOrFPVectorTy()) {
+    return llvm::ConstantFP::getNaN(Ty);
+  } else if (Ty->isArrayTy()) {
+    // Note: this doesn't touch tail padding (at the end of an object, before
+    // the next array object). It is instead handled by replaceUndef.
+    auto *ArrTy = cast<llvm::ArrayType>(Ty);
+    llvm::SmallVector<llvm::Constant *, 8> Element(
+        ArrTy->getNumElements(), patternFor(CGM, ArrTy->getElementType()));
+    return llvm::ConstantArray::get(ArrTy, Element);
+  }
+
+  // Note: this doesn't touch struct padding. It will initialize as much union
+  // padding as is required for the largest type in the union. Padding is
+  // instead handled by replaceUndef. Stores to structs with volatile members
+  // don't have a volatile qualifier when initialized according to C++. This is
+  // fine because stack-based volatiles don't really have volatile semantics
+  // anyways, and the initialization shouldn't be observable.
+  auto *StructTy = cast<llvm::StructType>(Ty);
+  llvm::SmallVector<llvm::Constant *, 8> Struct(StructTy->getNumElements());
+  for (unsigned El = 0; El != Struct.size(); ++El) {
+    Struct[El] = patternFor(CGM, StructTy->getElementType(El));
+  }
+  return llvm::ConstantStruct::get(StructTy, Struct);
+}
+
 static Address createUnnamedGlobalFrom(CodeGenModule &CGM, const VarDecl &D,
                                        CGBuilderTy &Builder,
                                        llvm::Constant *Constant,
                                        CharUnits Align) {
   auto FunctionName = [&](const DeclContext *DC) -> std::string {
     if (const auto *FD = dyn_cast<FunctionDecl>(DC)) {
       if (const auto *CC = dyn_cast<CXXConstructorDecl>(FD))
         return CC->getNameAsString();
       if (const auto *CD = dyn_cast<CXXDestructorDecl>(FD))
         return CD->getNameAsString();
       return CGM.getMangledName(FD);
     } else if (const auto *OM = dyn_cast<ObjCMethodDecl>(DC)) {
       return OM->getNameAsString();
     } else if (isa<BlockDecl>(DC)) {
       return "<block>";
     } else if (isa<CapturedDecl>(DC)) {
       return "<captured>";
if (SrcPtr.getType() != BP)
     SrcPtr = Builder.CreateBitCast(SrcPtr, BP);
   return SrcPtr;
 }
 
 static void emitStoresForConstant(CodeGenModule &CGM, const VarDecl &D,
                                   Address Loc, bool isVolatile,
                                   CGBuilderTy &Builder,
                                   llvm::Constant *constant) {
+  auto *Ty = constant->getType();
+  bool isScalar = Ty->isIntOrIntVectorTy() || Ty->isPtrOrPtrVectorTy() ||
+                  Ty->isFPOrFPVectorTy();
+  if (isScalar) {
+    Builder.CreateStore(constant, Loc, isVolatile);
+    return;
+  }
+
   auto *Int8Ty = llvm::IntegerType::getInt8Ty(CGM.getLLVMContext());
   auto *IntPtrTy = CGM.getDataLayout().getIntPtrType(CGM.getLLVMContext());
 
   // If the initializer is all or mostly the same, codegen with bzero / memset
   // then do a few stores afterward.
-  uint64_t ConstantSize =
-      CGM.getDataLayout().getTypeAllocSize(constant->getType());
+  uint64_t ConstantSize = CGM.getDataLayout().getTypeAllocSize(Ty);
   auto *SizeVal = llvm::ConstantInt::get(IntPtrTy, ConstantSize);
   if (shouldUseBZeroPlusStoresToInitialize(constant, ConstantSize)) {
     Builder.CreateMemSet(Loc, llvm::ConstantInt::get(Int8Ty, 0), SizeVal,
                          isVolatile);
 
     bool valueAlreadyCorrect =
         constant->isNullValue() || isa<llvm::UndefValue>(constant);
     if (!valueAlreadyCorrect) {
-      Loc = Builder.CreateBitCast(
-          Loc, constant->getType()->getPointerTo(Loc.getAddressSpace()));
+      Loc = Builder.CreateBitCast(Loc, Ty->getPointerTo(Loc.getAddressSpace()));
       emitStoresForInitAfterBZero(CGM, constant, Loc, isVolatile, Builder);
     }
     return;
   }
 
   llvm::Value *Pattern = shouldUseMemSetToInitialize(constant, ConstantSize);
   if (Pattern) {
     uint64_t Value = 0x00;
     if (!isa<llvm::UndefValue>(Pattern)) {
       const llvm::APInt &AP = cast<llvm::ConstantInt>(Pattern)->getValue();
       assert(AP.getBitWidth() <= 8);
       Value = AP.getLimitedValue();
     }
     Builder.CreateMemSet(Loc, llvm::ConstantInt::get(Int8Ty, Value), SizeVal,
                          isVolatile);
     return;
   }
 
   Builder.CreateMemCpy(
       Loc,
       createUnnamedGlobalFrom(CGM, D, Builder, constant, Loc.getAlignment()),
       SizeVal, isVolatile);
 }
 
+static void emitStoresForZeroInit(CodeGenModule &CGM, const VarDecl &D,
+                                  Address Loc, bool isVolatile,
+                                  CGBuilderTy &Builder) {
+  llvm::Type *ElTy = Loc.getElementType();
+  llvm::Constant *constant = llvm::Constant::getNullValue(ElTy);
+  emitStoresForConstant(CGM, D, Loc, isVolatile, Builder, constant);
+}
+
+static void emitStoresForPatternInit(CodeGenModule &CGM, const VarDecl &D,
+                                     Address Loc, bool isVolatile,
+                                     CGBuilderTy &Builder) {
+  llvm::Type *ElTy = Loc.getElementType();
+  llvm::Constant *constant = patternFor(CGM, ElTy);
+  assert(!isa<llvm::UndefValue>(constant));
+  emitStoresForConstant(CGM, D, Loc, isVolatile, Builder, constant);
+}
+
+static bool containsUndef(llvm::Constant *constant) {
+  auto *Ty = constant->getType();
+  if (isa<llvm::UndefValue>(constant))
+    return true;
+  if (Ty->isStructTy() || Ty->isArrayTy() || Ty->isVectorTy())
+    for (llvm::Use &Op : constant->operands())
+      if (containsUndef(cast<llvm::Constant>(Op)))
+        return true;
+  return false;
+}
+
+static llvm::Constant *replaceUndef(llvm::Constant *constant) {
+  // FIXME: when doing pattern initialization, replace undef with 0xAA instead.
+  // FIXME: also replace padding between values by creating a new struct type
+  //        which has no padding.
+  auto *Ty = constant->getType();
+  if (isa<llvm::UndefValue>(constant))
+    return llvm::Constant::getNullValue(Ty);
+  if (!(Ty->isStructTy() || Ty->isArrayTy() || Ty->isVectorTy()))
+    return constant;
+  if (!containsUndef(constant))
+    return constant;
+  llvm::SmallVector<llvm::Constant *, 8> Values(constant->getNumOperands());
+  for (unsigned Op = 0, NumOp = constant->getNumOperands(); Op != NumOp; ++Op) {
+    auto *OpValue = cast<llvm::Constant>(constant->getOperand(Op));
+    Values[Op] = replaceUndef(OpValue);
+  }
+  if (Ty->isStructTy()) {
+    return llvm::ConstantStruct::get(cast<llvm::StructType>(Ty), Values);
+  } else if (Ty->isArrayTy()) {
+    return llvm::ConstantArray::get(cast<llvm::ArrayType>(Ty), Values);
+  }
+  assert(Ty->isVectorTy());
+  return llvm::ConstantVector::get(Values);
+}
+
 /// EmitAutoVarDecl - Emit code and set up an entry in LocalDeclMap for a
 /// variable declaration with auto, register, or no storage class specifier.
 /// These turn into simple stack objects, or GlobalValues depending on target.
 void CodeGenFunction::EmitAutoVarDecl(const VarDecl &D) {
   AutoVarEmission emission = EmitAutoVarAlloca(D);
   EmitAutoVarInit(emission);
   EmitAutoVarCleanups(emission);
 }
void CodeGenFunction::EmitAutoVarInit(const AutoVarEmission &emission) {
 
   // If this was emitted as a global constant, we're done.
   if (emission.wasEmittedAsGlobal()) return;
 
   const VarDecl &D = *emission.Variable;
   auto DL = ApplyDebugLocation::CreateDefaultArtificial(*this, D.getLocation());
   QualType type = D.getType();
 
+  bool isVolatile = type.isVolatileQualified();
+
   // If this local has an initializer, emit it now.
   const Expr *Init = D.getInit();
 
   // If we are at an unreachable point, we don't need to emit the initializer
   // unless it contains a label.
   if (!HaveInsertPoint()) {
     if (!Init || !ContainsLabel(Init)) return;
     EnsureInsertPoint();
if (!Init &&
           QualType::PDIK_Struct) {
     LValue Dst = MakeAddrLValue(emission.getAllocatedAddress(), type);
     if (emission.IsEscapingByRef)
       drillIntoBlockVariable(*this, Dst, &D);
     defaultInitNonTrivialCStructVar(Dst);
     return;
   }
 
-  if (isTrivialInitializer(Init))
-    return;
-
   // Check whether this is a byref variable that's potentially
   // captured and moved by its own initializer.  If so, we'll need to
   // emit the initializer first, then copy into the variable.
-  bool capturedByInit = emission.IsEscapingByRef && isCapturedBy(D, Init);
+  bool capturedByInit =
+      Init && emission.IsEscapingByRef && isCapturedBy(D, Init);
 
   Address Loc =
       capturedByInit ? emission.Addr : emission.getObjectAddress(*this);
 
+  // Note: constexpr already initializes everything correctly.
+  LangOptions::TrivialAutoVarInitKind trivialAutoVarInit;
+  if (D.isConstexpr()) {
+    trivialAutoVarInit = LangOptions::TrivialAutoVarInitKind::Uninitialized;
+  } else if (const TrivialAutoInitAttr *Attr =
+                 D.getAttr<TrivialAutoInitAttr>()) {
+    switch (Attr->getTrivialAutoVarInit()) {
+    case TrivialAutoInitAttr::Uninitialized:
+      trivialAutoVarInit = LangOptions::TrivialAutoVarInitKind::Uninitialized;
+      break;
+    case TrivialAutoInitAttr::Zero:
+      trivialAutoVarInit = LangOptions::TrivialAutoVarInitKind::Zero;
+      break;
+    case TrivialAutoInitAttr::Pattern:
+      trivialAutoVarInit = LangOptions::TrivialAutoVarInitKind::Pattern;
+      break;
+    }
+  } else {
+    trivialAutoVarInit = getContext().getLangOpts().getTrivialAutoVarInit();
+  }
+
+  auto initializeWhatIsTechnicallyUninitialized = [&]() {
+    if (trivialAutoVarInit ==
+        LangOptions::TrivialAutoVarInitKind::Uninitialized)
+      return;
+
+    CharUnits Size = getContext().getTypeSizeInChars(type);
+    if (!Size.isZero()) {
+      switch (trivialAutoVarInit) {
+      case LangOptions::TrivialAutoVarInitKind::Uninitialized:
+        llvm_unreachable("Uninitialized handled above");
+      case LangOptions::TrivialAutoVarInitKind::Zero:
+        emitStoresForZeroInit(CGM, D, Loc, isVolatile, Builder);
+        break;
+      case LangOptions::TrivialAutoVarInitKind::Pattern:
+        emitStoresForPatternInit(CGM, D, Loc, isVolatile, Builder);
+        break;
+      }
+      return;
+    }
+
+    // VLAs look zero-sized to getTypeInfo. We can't emit constant stores to
+    // them, so emit a memcpy with the VLA size to initialize each element.
+    // Technically zero-sized or negative-sized VLAs are undefined, and UBSan
+    // will catch that code, but there exists code which generates zero-sized
+    // VLAs. Be nice and initialize whatever they requested.
+    const VariableArrayType *VlaType =
+        dyn_cast_or_null<VariableArrayType>(getContext().getAsArrayType(type));
+    if (!VlaType)
+      return;
+    auto VlaSize = getVLASize(VlaType);
+    auto SizeVal = VlaSize.NumElts;
+    CharUnits EltSize = getContext().getTypeSizeInChars(VlaSize.Type);
+    switch (trivialAutoVarInit) {
+    case LangOptions::TrivialAutoVarInitKind::Uninitialized:
+      llvm_unreachable("Uninitialized handled above");
+
+    case LangOptions::TrivialAutoVarInitKind::Zero:
+      if (!EltSize.isOne())
+        SizeVal = Builder.CreateNUWMul(SizeVal, CGM.getSize(EltSize));
+      Builder.CreateMemSet(Loc, llvm::ConstantInt::get(Int8Ty, 0), SizeVal,
+                           isVolatile);
+      break;
+
+    case LangOptions::TrivialAutoVarInitKind::Pattern: {
+      llvm::Type *ElTy = Loc.getElementType();
+      llvm::Constant *Constant = patternFor(CGM, ElTy);
+      CharUnits ConstantAlign = getContext().getTypeAlignInChars(VlaSize.Type);
+      llvm::BasicBlock *SetupBB = createBasicBlock("vla-setup.loop");
+      llvm::BasicBlock *LoopBB = createBasicBlock("vla-init.loop");
+      llvm::BasicBlock *ContBB = createBasicBlock("vla-init.cont");
+      llvm::Value *IsZeroSizedVLA = Builder.CreateICmpEQ(
+          SizeVal, llvm::ConstantInt::get(SizeVal->getType(), 0),
+          "vla.iszerosized");
+      Builder.CreateCondBr(IsZeroSizedVLA, ContBB, SetupBB);
+      EmitBlock(SetupBB);
+      if (!EltSize.isOne())
+        SizeVal = Builder.CreateNUWMul(SizeVal, CGM.getSize(EltSize));
+      llvm::Value *BaseSizeInChars =
+          llvm::ConstantInt::get(IntPtrTy, EltSize.getQuantity());
+      Address Begin = Builder.CreateElementBitCast(Loc, Int8Ty, "vla.begin");
+      llvm::Value *End =
+          Builder.CreateInBoundsGEP(Begin.getPointer(), SizeVal, "vla.end");
+      llvm::BasicBlock *OriginBB = Builder.GetInsertBlock();
+      EmitBlock(LoopBB);
+      llvm::PHINode *Cur = Builder.CreatePHI(Begin.getType(), 2, "vla.cur");
+      Cur->addIncoming(Begin.getPointer(), OriginBB);
+      CharUnits CurAlign = Loc.getAlignment().alignmentOfArrayElement(EltSize);
+      Builder.CreateMemCpy(
+          Address(Cur, CurAlign),
+          createUnnamedGlobalFrom(CGM, D, Builder, Constant, ConstantAlign),
+          BaseSizeInChars, isVolatile);
+      llvm::Value *Next =
+          Builder.CreateInBoundsGEP(Int8Ty, Cur, BaseSizeInChars, "vla.next");
+      llvm::Value *Done = Builder.CreateICmpEQ(Next, End, "vla-init.isdone");
+      Builder.CreateCondBr(Done, ContBB, LoopBB);
+      Cur->addIncoming(Next, LoopBB);
+      EmitBlock(ContBB);
+    } break;
+    }
+  };
+
+  if (isTrivialInitializer(Init)) {
+    initializeWhatIsTechnicallyUninitialized();
+    return;
+  }
+
   llvm::Constant *constant = nullptr;
   if (emission.IsConstantAggregate || D.isConstexpr()) {
     assert(!capturedByInit && "constant init contains a capturing block?");
     constant = ConstantEmitter(*this).tryEmitAbstractForInitializer(D);
+    if (constant && trivialAutoVarInit !=
+                        LangOptions::TrivialAutoVarInitKind::Uninitialized)
+      constant = replaceUndef(constant);
   }
 
   if (!constant) {
+    initializeWhatIsTechnicallyUninitialized();
     LValue lv = MakeAddrLValue(Loc, type);
     lv.setNonGC(true);
     return EmitExprAsInit(Init, &D, lv, capturedByInit);
   }
 
   if (!emission.IsConstantAggregate) {
     // For simple scalar/complex initialization, store the value directly.
     LValue lv = MakeAddrLValue(Loc, type);
     lv.setNonGC(true);
     return EmitStoreThroughLValue(RValue::get(constant), lv, true);
   }
 
-  // If this is a simple aggregate initialization, we can optimize it
-  // in various ways.
-  bool isVolatile = type.isVolatileQualified();
-
   llvm::Type *BP = CGM.Int8Ty->getPointerTo(Loc.getAddressSpace());
   if (Loc.getType() != BP)
     Loc = Builder.CreateBitCast(Loc, BP);
 
   emitStoresForConstant(CGM, D, Loc, isVolatile, Builder, constant);
 }
 
 /// Emit an expression as an initializer for an object (variable, field, etc.)

lib/Driver/ToolChains/Clang.cpp

if (Str.startswith("ssp-buffer-size=")) {
         // FIXME: Verify the argument is a valid integer.
         CmdArgs.push_back(Args.MakeArgString(Str.drop_front(16)));
       }
       A->claim();
     }
   }
 }
 
+static void RenderTrivialAutoVarInitOptions(const Driver &D,
+                                            const ToolChain &TC,
+                                            const ArgList &Args,
+                                            ArgStringList &CmdArgs) {
+  auto DefaultTrivialAutoVarInit = TC.GetDefaultTrivialAutoVarInit();
+  StringRef TrivialAutoVarInit = "";
+
+  for (const Arg *A : Args) {
+    switch (A->getOption().getID()) {
+    default:
+      continue;
+    case options::OPT_ftrivial_auto_var_init: {
+      A->claim();
+      StringRef Val = A->getValue();
+      if (Val == "uninitialized" || Val == "zero" || Val == "pattern")
+        TrivialAutoVarInit = Val;
+      else
+        D.Diag(diag::err_drv_unsupported_option_argument)
+            << A->getOption().getName() << Val;
+      break;
+    }
+    }
+  }
+
+  if (TrivialAutoVarInit.empty())
+    switch (DefaultTrivialAutoVarInit) {
+    case LangOptions::TrivialAutoVarInitKind::Uninitialized:
+      break;
+    case LangOptions::TrivialAutoVarInitKind::Pattern:
+      TrivialAutoVarInit = "pattern";
+      break;
+    case LangOptions::TrivialAutoVarInitKind::Zero:
+      TrivialAutoVarInit = "zero";
+      break;
+    }
+
+  if (!TrivialAutoVarInit.empty()) {
+    if (TrivialAutoVarInit == "zero" && !Args.hasArg(options::OPT_enable_trivial_var_init_zero)) {
+      TrivialAutoVarInit = "pattern";
+      D.Diag(diag::warn_drv_trivial_auto_var_init_zero_disabled);
+    }
+    CmdArgs.push_back(
+        Args.MakeArgString("-ftrivial-auto-var-init=" + TrivialAutoVarInit));
+  }
+}
+
 static void RenderOpenCLOptions(const ArgList &Args, ArgStringList &CmdArgs) {
   const unsigned ForwardedArguments[] = {
       options::OPT_cl_opt_disable,
       options::OPT_cl_strict_aliasing,
       options::OPT_cl_single_precision_constant,
       options::OPT_cl_finite_math_only,
       options::OPT_cl_kernel_arg_info,
       options::OPT_cl_unsafe_math_optimizations,
#endif
 
   Args.AddLastArg(CmdArgs, options::OPT_pthread);
 
   if (Args.hasFlag(options::OPT_mspeculative_load_hardening, options::OPT_mno_speculative_load_hardening,
                    false))
     CmdArgs.push_back(Args.MakeArgString("-mspeculative-load-hardening"));
 
   RenderSSPOptions(TC, Args, CmdArgs, KernelOrKext);
+  RenderTrivialAutoVarInitOptions(D, TC, Args, CmdArgs);
 
   // Translate -mstackrealign
   if (Args.hasFlag(options::OPT_mstackrealign, options::OPT_mno_stackrealign,
                    false))
     CmdArgs.push_back(Args.MakeArgString("-mstackrealign"));
 
   if (Args.hasArg(options::OPT_mstack_alignment)) {
     StringRef alignment = Args.getLastArgValue(options::OPT_mstack_alignment);

lib/Frontend/CompilerInvocation.cpp

Diags.Report(diag::err_drv_invalid_value)
       << Args.getLastArg(OPT_stack_protector)->getAsString(Args) << SSP;
     break;
   case 0: Opts.setStackProtector(LangOptions::SSPOff); break;
   case 1: Opts.setStackProtector(LangOptions::SSPOn);  break;
   case 2: Opts.setStackProtector(LangOptions::SSPStrong); break;
   case 3: Opts.setStackProtector(LangOptions::SSPReq); break;
   }
 
+  if (Arg *A = Args.getLastArg(OPT_ftrivial_auto_var_init)) {
+    StringRef Val = A->getValue();
+    if (Val == "uninitialized")
+      Opts.setTrivialAutoVarInit(
+          LangOptions::TrivialAutoVarInitKind::Uninitialized);
+    else if (Val == "zero")
+      Opts.setTrivialAutoVarInit(LangOptions::TrivialAutoVarInitKind::Zero);
+    else if (Val == "pattern")
+      Opts.setTrivialAutoVarInit(LangOptions::TrivialAutoVarInitKind::Pattern);
+    else
+      Diags.Report(diag::err_drv_invalid_value) << A->getAsString(Args) << Val;
+  }
+
   // Parse -fsanitize= arguments.
   parseSanitizerKinds("-fsanitize=", Args.getAllArgValues(OPT_fsanitize_EQ),
                       Diags, Opts.Sanitize);
   // -fsanitize-address-field-padding=N has to be a LangOpt, parse it here.
   Opts.SanitizeAddressFieldPadding =
       getLastArgIntValue(Args, OPT_fsanitize_address_field_padding, 0, Diags);
   Opts.SanitizerBlacklistFiles = Args.getAllArgValues(OPT_fsanitize_blacklist);
 

lib/Sema/SemaDeclAttr.cpp

static void handleDestroyAttr(Sema &S, Decl *D, const ParsedAttr &A) {
   }
 
   if (A.getKind() == ParsedAttr::AT_AlwaysDestroy)
     handleSimpleAttributeWithExclusions<AlwaysDestroyAttr, NoDestroyAttr>(S, D, A);
   else
     handleSimpleAttributeWithExclusions<NoDestroyAttr, AlwaysDestroyAttr>(S, D, A);
 }
 
+static void handleTrivialAutoInitAttr(Sema &S, Decl *D, const ParsedAttr &AL) {
+  if (AL.getNumArgs() != 1) {
+    S.Diag(AL.getLoc(), diag::err_attribute_wrong_number_arguments) << AL << 1;
+    return;
+  }
+  StringRef Str;
+  SourceLocation ArgLoc;
+  if (!S.checkStringLiteralArgumentAttr(AL, 0, Str, &ArgLoc))
+    return;
+  TrivialAutoInitAttr::TrivialAutoVarInitKind Kind;
+  if (!TrivialAutoInitAttr::ConvertStrToTrivialAutoVarInitKind(Str, Kind)) {
+    S.Diag(AL.getLoc(), diag::warn_attribute_type_not_supported)
+        << AL << Str << ArgLoc;
+    return;
+  }
+  assert(cast<VarDecl>(D)->getStorageDuration() == SD_Automatic &&
+         "trivial_auto_init is only valid on automatic duration variables");
+  unsigned Index = AL.getAttributeSpellingListIndex();
+  D->addAttr(::new (S.Context)
+                 TrivialAutoInitAttr(AL.getLoc(), S.Context, Kind, Index));
+}
+
 //===----------------------------------------------------------------------===//
 // Top Level Sema Entry Points
 //===----------------------------------------------------------------------===//
 
 /// ProcessDeclAttribute - Apply the specific attribute to the specified decl if
 /// the attribute applies to decls.  If the attribute is a type attribute, just
 /// silently ignore it if a GNU attribute.
 static void ProcessDeclAttribute(Sema &S, Scope *scope, Decl *D,
static void ProcessDeclAttribute(Sema &S, Scope *scope, Decl *D,
   case ParsedAttr::AT_Reinitializes:
     handleSimpleAttribute<ReinitializesAttr>(S, D, AL);
     break;
 
   case ParsedAttr::AT_AlwaysDestroy:
   case ParsedAttr::AT_NoDestroy:
     handleDestroyAttr(S, D, AL);
     break;
+
+  case ParsedAttr::AT_TrivialAutoInit:
+    handleTrivialAutoInitAttr(S, D, AL);
+    break;
   }
 }
 
 /// ProcessDeclAttributeList - Apply all the decl attributes in the specified
 /// attribute list to the specified decl, ignoring any type attributes.
 void Sema::ProcessDeclAttributeList(Scope *S, Decl *D,
                                     const ParsedAttributesView &AttrList,
                                     bool IncludeCXX11Attributes) {

test/CodeGenCXX/auto-var-init.cpp

 // RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks -ftrivial-auto-var-init=pattern %s -emit-llvm -o - | FileCheck %s -check-prefix=PATTERN
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks -ftrivial-auto-var-init=zero %s -emit-llvm -o - | FileCheck %s -check-prefix=ZERO
 
 template<typename T> void used(T &) noexcept;
 
 #define TEST_UNINIT(NAME, TYPE)                 \
   using type_##NAME = TYPE;                     \
   void test_##NAME##_uninit() {                 \
     type_##NAME uninit;                         \
     used(uninit);                               \

 
 #define TEST_CUSTOM(NAME, TYPE, ...)            \
   using type_##NAME = TYPE;                     \
   void test_##NAME##_custom() {                 \
     type_##NAME custom __VA_ARGS__;             \
     used(custom);                               \
   }
 
+// None of the synthesized globals should contain `undef`.
+// PATTERN-NOT: undef
+// ZERO-NOT: undef
+
+// PATTERN: @__const.test_empty_uninit.uninit = private unnamed_addr constant %struct.empty { i8 -86 }, align 1
 struct empty {};
+// PATTERN: @__const.test_small_uninit.uninit = private unnamed_addr constant %struct.small { i8 -86 }, align 1
+// PATTERN: @__const.test_small_custom.custom = private unnamed_addr constant %struct.small { i8 42 }, align 1
+// ZERO: @__const.test_small_custom.custom = private unnamed_addr constant %struct.small { i8 42 }, align 1
 struct small { char c; };
+// PATTERN: @__const.test_smallinit_uninit.uninit = private unnamed_addr constant %struct.smallinit { i8 -86 }, align 1
+// PATTERN: @__const.test_smallinit_braces.braces = private unnamed_addr constant %struct.smallinit { i8 -86 }, align 1
+// PATTERN: @__const.test_smallinit_custom.custom = private unnamed_addr constant %struct.smallinit { i8 -86 }, align 1
 struct smallinit { char c = 42; };
+// PATTERN: @__const.test_smallpartinit_uninit.uninit = private unnamed_addr constant %struct.smallpartinit { i8 -86, i8 -86 }, align 1
+// PATTERN: @__const.test_smallpartinit_braces.braces = private unnamed_addr constant %struct.smallpartinit { i8 -86, i8 -86 }, align 1
+// PATTERN: @__const.test_smallpartinit_custom.custom = private unnamed_addr constant %struct.smallpartinit { i8 -86, i8 -86 }, align 1
 struct smallpartinit { char c = 42, d; };
+// PATTERN: @__const.test_nullinit_uninit.uninit = private unnamed_addr constant %struct.nullinit { i8* inttoptr (i64 -6148914691236517206 to i8*) }, align 8
+// PATTERN: @__const.test_nullinit_braces.braces = private unnamed_addr constant %struct.nullinit { i8* inttoptr (i64 -6148914691236517206 to i8*) }, align 8
+// PATTERN: @__const.test_nullinit_custom.custom = private unnamed_addr constant %struct.nullinit { i8* inttoptr (i64 -6148914691236517206 to i8*) }, align 8
 struct nullinit { char* null = nullptr; };
+// PATTERN: @__const.test_padded_uninit.uninit = private unnamed_addr constant %struct.padded { i8 -86, i32 -1431655766 }, align 4
+// PATTERN: @__const.test_padded_custom.custom = private unnamed_addr constant %struct.padded { i8 42, i32 13371337 }, align 4
+// ZERO: @__const.test_padded_custom.custom = private unnamed_addr constant %struct.padded { i8 42, i32 13371337 }, align 4
 struct padded { char c; int i; };
+// PATTERN: @__const.test_paddednullinit_uninit.uninit = private unnamed_addr constant %struct.paddednullinit { i8 -86, i32 -1431655766 }, align 4
+// PATTERN: @__const.test_paddednullinit_braces.braces = private unnamed_addr constant %struct.paddednullinit { i8 -86, i32 -1431655766 }, align 4
+// PATTERN: @__const.test_paddednullinit_custom.custom = private unnamed_addr constant %struct.paddednullinit { i8 -86, i32 -1431655766 }, align 4
 struct paddednullinit { char c = 0; int i = 0; };
+// PATTERN: @__const.test_bitfield_uninit.uninit = private unnamed_addr constant %struct.bitfield { i8 -86, [3 x i8] c"\AA\AA\AA" }, align 4
+// PATTERN: @__const.test_bitfield_custom.custom = private unnamed_addr constant %struct.bitfield { i8 20, [3 x i8] zeroinitializer }, align 4
+// ZERO: @__const.test_bitfield_custom.custom = private unnamed_addr constant %struct.bitfield { i8 20, [3 x i8] zeroinitializer }, align 4
 struct bitfield { int i : 4; int j : 2; };
+// PATTERN: @__const.test_bitfieldaligned_uninit.uninit = private unnamed_addr constant %struct.bitfieldaligned { i8 -86, [3 x i8] c"\AA\AA\AA", i8 -86, [3 x i8] c"\AA\AA\AA" }, align 4
+// PATTERN: @__const.test_bitfieldaligned_custom.custom = private unnamed_addr constant %struct.bitfieldaligned { i8 4, [3 x i8] zeroinitializer, i8 1, [3 x i8] zeroinitializer }, align 4
+// ZERO: @__const.test_bitfieldaligned_custom.custom = private unnamed_addr constant %struct.bitfieldaligned { i8 4, [3 x i8] zeroinitializer, i8 1, [3 x i8] zeroinitializer }, align 4
 struct bitfieldaligned { int i : 4; int : 0; int j : 2; };
 struct big { unsigned a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z; };
+// PATTERN: @__const.test_arraytail_uninit.uninit = private unnamed_addr constant %struct.arraytail { i32 -1431655766, [0 x i32] zeroinitializer }, align 4
+// PATTERN: @__const.test_arraytail_custom.custom = private unnamed_addr constant %struct.arraytail { i32 57005, [0 x i32] zeroinitializer }, align 4
+// ZERO: @__const.test_arraytail_custom.custom = private unnamed_addr constant %struct.arraytail { i32 57005, [0 x i32] zeroinitializer }, align 4
 struct arraytail { int i; int arr[]; };
+// PATTERN: @__const.test_int1_uninit.uninit = private unnamed_addr constant [1 x i32] [i32 -1431655766], align 4
+// PATTERN: @__const.test_int1_custom.custom = private unnamed_addr constant [1 x i32] [i32 858993459], align 4
+// ZERO: @__const.test_int1_custom.custom = private unnamed_addr constant [1 x i32] [i32 858993459], align 4
+// PATTERN: @__const.test_bool4_uninit.uninit = private unnamed_addr constant [4 x i8] c"\AA\AA\AA\AA", align 1
+// PATTERN: @__const.test_bool4_custom.custom = private unnamed_addr constant [4 x i8] c"\01\01\01\01", align 1
+// ZERO: @__const.test_bool4_custom.custom = private unnamed_addr constant [4 x i8] c"\01\01\01\01", align 1
+// PATTERN: @__const.test_intptr4_uninit.uninit = private unnamed_addr constant [4 x i32*] [i32* inttoptr (i64 -6148914691236517206 to i32*), i32* inttoptr (i64 -6148914691236517206 to i32*), i32* inttoptr (i64 -6148914691236517206 to i32*), i32* inttoptr (i64 -6148914691236517206 to i32*)], align 16
+// PATTERN: @__const.test_intptr4_custom.custom = private unnamed_addr constant [4 x i32*] [i32* inttoptr (i64 572662306 to i32*), i32* inttoptr (i64 572662306 to i32*), i32* inttoptr (i64 572662306 to i32*), i32* inttoptr (i64 572662306 to i32*)], align 16
+// ZERO: @__const.test_intptr4_custom.custom = private unnamed_addr constant [4 x i32*] [i32* inttoptr (i64 572662306 to i32*), i32* inttoptr (i64 572662306 to i32*), i32* inttoptr (i64 572662306 to i32*), i32* inttoptr (i64 572662306 to i32*)], align 16
+// PATTERN: @__const.test_tailpad4_uninit.uninit = private unnamed_addr constant [4 x %struct.tailpad] [%struct.tailpad { i16 -21846, i8 -86 }, %struct.tailpad { i16 -21846, i8 -86 }, %struct.tailpad { i16 -21846, i8 -86 }, %struct.tailpad { i16 -21846, i8 -86 }], align 16
+// PATTERN: @__const.test_tailpad4_custom.custom = private unnamed_addr constant [4 x %struct.tailpad] [%struct.tailpad { i16 257, i8 1 }, %struct.tailpad { i16 257, i8 1 }, %struct.tailpad { i16 257, i8 1 }, %struct.tailpad { i16 257, i8 1 }], align 16
+// ZERO: @__const.test_tailpad4_custom.custom = private unnamed_addr constant [4 x %struct.tailpad] [%struct.tailpad { i16 257, i8 1 }, %struct.tailpad { i16 257, i8 1 }, %struct.tailpad { i16 257, i8 1 }, %struct.tailpad { i16 257, i8 1 }], align 16
 struct tailpad { short s; char c; };
+// PATTERN: @__const.test_atomicnotlockfree_uninit.uninit = private unnamed_addr constant %struct.notlockfree { [4 x i64] [i64 -6148914691236517206, i64 -6148914691236517206, i64 -6148914691236517206, i64 -6148914691236517206] }, align 8
 struct notlockfree { long long a[4]; };
+// PATTERN: @__const.test_atomicpadded_uninit.uninit = private unnamed_addr constant %struct.padded { i8 -86, i32 -1431655766 }, align 8
+// PATTERN: @__const.test_atomictailpad_uninit.uninit = private unnamed_addr constant %struct.tailpad { i16 -21846, i8 -86 }, align 4
+// PATTERN: @__const.test_complexfloat_uninit.uninit = private unnamed_addr constant { float, float } { float 0x7FF8000000000000, float 0x7FF8000000000000 }, align 4
+// PATTERN: @__const.test_complexfloat_braces.braces = private unnamed_addr constant { float, float } { float 0x7FF8000000000000, float 0x7FF8000000000000 }, align 4
+// PATTERN: @__const.test_complexfloat_custom.custom = private unnamed_addr constant { float, float } { float 0x7FF8000000000000, float 0x7FF8000000000000 }, align 4
+// PATTERN: @__const.test_complexdouble_uninit.uninit = private unnamed_addr constant { double, double } { double 0x7FF8000000000000, double 0x7FF8000000000000 }, align 8
+// PATTERN: @__const.test_complexdouble_braces.braces = private unnamed_addr constant { double, double } { double 0x7FF8000000000000, double 0x7FF8000000000000 }, align 8
+// PATTERN: @__const.test_complexdouble_custom.custom = private unnamed_addr constant { double, double } { double 0x7FF8000000000000, double 0x7FF8000000000000 }, align 8
+// PATTERN: @__const.test_semivolatile_uninit.uninit = private unnamed_addr constant %struct.semivolatile { i32 -1431655766, i32 -1431655766 }, align 4
+// PATTERN: @__const.test_semivolatile_custom.custom = private unnamed_addr constant %struct.semivolatile { i32 1145324612, i32 1145324612 }, align 4
 struct semivolatile { int i; volatile int vi; };
+// PATTERN: @__const.test_semivolatileinit_uninit.uninit = private unnamed_addr constant %struct.semivolatileinit { i32 -1431655766, i32 -1431655766 }, align 4
+// PATTERN: @__const.test_semivolatileinit_braces.braces = private unnamed_addr constant %struct.semivolatileinit { i32 -1431655766, i32 -1431655766 }, align 4
+// PATTERN: @__const.test_semivolatileinit_custom.custom = private unnamed_addr constant %struct.semivolatileinit { i32 -1431655766, i32 -1431655766 }, align 4
+// ZERO: @__const.test_semivolatile_custom.custom = private unnamed_addr constant %struct.semivolatile { i32 1145324612, i32 1145324612 }, align 4
 struct semivolatileinit { int i = 0x11111111; volatile int vi = 0x11111111; };
+// PATTERN: @__const.test_base_uninit.uninit = private unnamed_addr constant %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) }, align 8
+// PATTERN: @__const.test_base_braces.braces = private unnamed_addr constant %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) }, align 8
 struct base { virtual ~base(); };
+// PATTERN: @__const.test_derived_uninit.uninit = private unnamed_addr constant %struct.derived { %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) } }, align 8
+// PATTERN: @__const.test_derived_braces.braces = private unnamed_addr constant %struct.derived { %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) } }, align 8
 struct derived : public base {};
+// PATTERN: @__const.test_virtualderived_uninit.uninit = private unnamed_addr constant %struct.virtualderived { %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) }, %struct.derived { %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) } } }, align 8
+// PATTERN: @__const.test_virtualderived_braces.braces = private unnamed_addr constant %struct.virtualderived { %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) }, %struct.derived { %struct.base { i32 (...)** inttoptr (i64 -6148914691236517206 to i32 (...)**) } } }, align 8
 struct virtualderived : public virtual base, public virtual derived {};
+// PATTERN: @__const.test_matching_uninit.uninit = private unnamed_addr constant %union.matching { i32 -1431655766 }, align 4
+// PATTERN: @__const.test_matching_custom.custom = private unnamed_addr constant { float } { float 6.145500e+04 }, align 4
 union matching { int i; float f; };
+// PATTERN: @__const.test_matchingreverse_uninit.uninit = private unnamed_addr constant %union.matchingreverse { float 0x7FF8000000000000 }, align 4
+// PATTERN: @__const.test_matchingreverse_custom.custom = private unnamed_addr constant { i32 } { i32 61455 }, align 4
+// ZERO: @__const.test_matching_custom.custom = private unnamed_addr constant { float } { float 6.145500e+04 }, align 4
 union matchingreverse { float f; int i; };
+// PATTERN: @__const.test_unmatched_uninit.uninit = private unnamed_addr constant %union.unmatched { i32 -1431655766 }, align 4
+// PATTERN: @__const.test_unmatched_custom.custom = private unnamed_addr constant %union.unmatched { i32 1001242351 }, align 4
+// ZERO: @__const.test_matchingreverse_custom.custom = private unnamed_addr constant { i32 } { i32 61455 }, align 4
 union unmatched { char c; int i; };
+// PATTERN: @__const.test_unmatchedreverse_uninit.uninit = private unnamed_addr constant %union.unmatchedreverse { i32 -1431655766 }, align 4
+// PATTERN: @__const.test_unmatchedreverse_custom.custom = private unnamed_addr constant { i8, [3 x i8] } { i8 42, [3 x i8] zeroinitializer }, align 4
+// ZERO: @__const.test_unmatched_custom.custom = private unnamed_addr constant %union.unmatched { i32 1001242351 }, align 4
 union unmatchedreverse { int i; char c; };
+// PATTERN: @__const.test_unmatchedfp_uninit.uninit = private unnamed_addr constant %union.unmatchedfp { double 0x7FF8000000000000 }, align 8
+// PATTERN: @__const.test_unmatchedfp_custom.custom = private unnamed_addr constant %union.unmatchedfp { double 0x400921FB54442D18 }, align 8
+// ZERO: @__const.test_unmatchedreverse_custom.custom = private unnamed_addr constant { i8, [3 x i8] } { i8 42, [3 x i8] zeroinitializer }, align 4
+// ZERO: @__const.test_unmatchedfp_custom.custom = private unnamed_addr constant %union.unmatchedfp { double 0x400921FB54442D18 }, align 8
 union unmatchedfp { float f; double d; };
 enum emptyenum {};
 enum smallenum { VALUE };
 
 extern "C" {
 
 TEST_UNINIT(char, char);
 // CHECK-LABEL: @test_char_uninit()
 // CHECK:       %uninit = alloca i8, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_char_uninit()
+// PATTERN: store i8 -86, i8* %uninit, align 1
+// ZERO-LABEL: @test_char_uninit()
+// ZERO: store i8 0, i8* %uninit, align 1
 
 TEST_BRACES(char, char);
 // CHECK-LABEL: @test_char_braces()
 // CHECK:       %braces = alloca i8, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i8 0, i8* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(uchar, unsigned char);
 // CHECK-LABEL: @test_uchar_uninit()
 // CHECK:       %uninit = alloca i8, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_uchar_uninit()
+// PATTERN: store i8 -86, i8* %uninit, align 1
+// ZERO-LABEL: @test_uchar_uninit()
+// ZERO: store i8 0, i8* %uninit, align 1
 
 TEST_BRACES(uchar, unsigned char);
 // CHECK-LABEL: @test_uchar_braces()
 // CHECK:       %braces = alloca i8, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i8 0, i8* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(schar, signed char);
 // CHECK-LABEL: @test_schar_uninit()
 // CHECK:       %uninit = alloca i8, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_schar_uninit()
+// PATTERN: store i8 -86, i8* %uninit, align 1
+// ZERO-LABEL: @test_schar_uninit()
+// ZERO: store i8 0, i8* %uninit, align 1
 
 TEST_BRACES(schar, signed char);
 // CHECK-LABEL: @test_schar_braces()
 // CHECK:       %braces = alloca i8, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i8 0, i8* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(wchar_t, wchar_t);
 // CHECK-LABEL: @test_wchar_t_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_wchar_t_uninit()
+// PATTERN: store i32 -1431655766, i32* %uninit, align
+// ZERO-LABEL: @test_wchar_t_uninit()
+// ZERO: store i32 0, i32* %uninit, align
 
 TEST_BRACES(wchar_t, wchar_t);
 // CHECK-LABEL: @test_wchar_t_braces()
 // CHECK:       %braces = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 0, i32* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(short, short);
 // CHECK-LABEL: @test_short_uninit()
 // CHECK:       %uninit = alloca i16, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_short_uninit()
+// PATTERN: store i16 -21846, i16* %uninit, align
+// ZERO-LABEL: @test_short_uninit()
+// ZERO: store i16 0, i16* %uninit, align
 
 TEST_BRACES(short, short);
 // CHECK-LABEL: @test_short_braces()
 // CHECK:       %braces = alloca i16, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i16 0, i16* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(ushort, unsigned short);
 // CHECK-LABEL: @test_ushort_uninit()
 // CHECK:       %uninit = alloca i16, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_ushort_uninit()
+// PATTERN: store i16 -21846, i16* %uninit, align
+// ZERO-LABEL: @test_ushort_uninit()
+// ZERO: store i16 0, i16* %uninit, align
 
 TEST_BRACES(ushort, unsigned short);
 // CHECK-LABEL: @test_ushort_braces()
 // CHECK:       %braces = alloca i16, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i16 0, i16* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(int, int);
 // CHECK-LABEL: @test_int_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_int_uninit()
+// PATTERN: store i32 -1431655766, i32* %uninit, align
+// ZERO-LABEL: @test_int_uninit()
+// ZERO: store i32 0, i32* %uninit, align
 
 TEST_BRACES(int, int);
 // CHECK-LABEL: @test_int_braces()
 // CHECK:       %braces = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 0, i32* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(unsigned, unsigned);
 // CHECK-LABEL: @test_unsigned_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_unsigned_uninit()
+// PATTERN: store i32 -1431655766, i32* %uninit, align
+// ZERO-LABEL: @test_unsigned_uninit()
+// ZERO: store i32 0, i32* %uninit, align
 
 TEST_BRACES(unsigned, unsigned);
 // CHECK-LABEL: @test_unsigned_braces()
 // CHECK:       %braces = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 0, i32* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(long, long);
 // CHECK-LABEL: @test_long_uninit()
 // CHECK:       %uninit = alloca i64, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_long_uninit()
+// PATTERN: store i64 -6148914691236517206, i64* %uninit, align
+// ZERO-LABEL: @test_long_uninit()
+// ZERO: store i64 0, i64* %uninit, align
 
 TEST_BRACES(long, long);
 // CHECK-LABEL: @test_long_braces()
 // CHECK:       %braces = alloca i64, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i64 0, i64* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(ulong, unsigned long);
 // CHECK-LABEL: @test_ulong_uninit()
 // CHECK:       %uninit = alloca i64, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_ulong_uninit()
+// PATTERN: store i64 -6148914691236517206, i64* %uninit, align
+// ZERO-LABEL: @test_ulong_uninit()
+// ZERO: store i64 0, i64* %uninit, align
 
 TEST_BRACES(ulong, unsigned long);
 // CHECK-LABEL: @test_ulong_braces()
 // CHECK:       %braces = alloca i64, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i64 0, i64* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(longlong, long long);
 // CHECK-LABEL: @test_longlong_uninit()
 // CHECK:       %uninit = alloca i64, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_longlong_uninit()
+// PATTERN: store i64 -6148914691236517206, i64* %uninit, align
+// ZERO-LABEL: @test_longlong_uninit()
+// ZERO: store i64 0, i64* %uninit, align
 
 TEST_BRACES(longlong, long long);
 // CHECK-LABEL: @test_longlong_braces()
 // CHECK:       %braces = alloca i64, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i64 0, i64* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(ulonglong, unsigned long long);
 // CHECK-LABEL: @test_ulonglong_uninit()
 // CHECK:       %uninit = alloca i64, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_ulonglong_uninit()
+// PATTERN: store i64 -6148914691236517206, i64* %uninit, align
+// ZERO-LABEL: @test_ulonglong_uninit()
+// ZERO: store i64 0, i64* %uninit, align
 
 TEST_BRACES(ulonglong, unsigned long long);
 // CHECK-LABEL: @test_ulonglong_braces()
 // CHECK:       %braces = alloca i64, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i64 0, i64* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(int128, __int128);
 // CHECK-LABEL: @test_int128_uninit()
 // CHECK:       %uninit = alloca i128, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_int128_uninit()
+// PATTERN: store i128 -113427455640312821154458202477256070486, i128* %uninit, align
+// ZERO-LABEL: @test_int128_uninit()
+// ZERO: store i128 0, i128* %uninit, align
 
 TEST_BRACES(int128, __int128);
 // CHECK-LABEL: @test_int128_braces()
 // CHECK:       %braces = alloca i128, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i128 0, i128* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(uint128, unsigned __int128);
 // CHECK-LABEL: @test_uint128_uninit()
 // CHECK:       %uninit = alloca i128, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_uint128_uninit()
+// PATTERN: store i128 -113427455640312821154458202477256070486, i128* %uninit, align
+// ZERO-LABEL: @test_uint128_uninit()
+// ZERO: store i128 0, i128* %uninit, align
 
 TEST_BRACES(uint128, unsigned __int128);
 // CHECK-LABEL: @test_uint128_braces()
 // CHECK:       %braces = alloca i128, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i128 0, i128* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 
 TEST_UNINIT(fp16, __fp16);
 // CHECK-LABEL: @test_fp16_uninit()
 // CHECK:       %uninit = alloca half, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_fp16_uninit()
+// PATTERN: store half 0xH7E00, half* %uninit, align
+// ZERO-LABEL: @test_fp16_uninit()
+// ZERO: store half 0xH0000, half* %uninit, align
 
 TEST_BRACES(fp16, __fp16);
 // CHECK-LABEL: @test_fp16_braces()
 // CHECK:       %braces = alloca half, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store half 0xH0000, half* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(float, float);
 // CHECK-LABEL: @test_float_uninit()
 // CHECK:       %uninit = alloca float, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_float_uninit()
+// PATTERN: store float 0x7FF8000000000000, float* %uninit, align
+// ZERO-LABEL: @test_float_uninit()
+// ZERO: store float 0.000000e+00, float* %uninit, align
 
 TEST_BRACES(float, float);
 // CHECK-LABEL: @test_float_braces()
 // CHECK:       %braces = alloca float, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store float 0.000000e+00, float* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(double, double);
 // CHECK-LABEL: @test_double_uninit()
 // CHECK:       %uninit = alloca double, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_double_uninit()
+// PATTERN: store double 0x7FF8000000000000, double* %uninit, align
+// ZERO-LABEL: @test_double_uninit()
+// ZERO: store double 0.000000e+00, double* %uninit, align
 
 TEST_BRACES(double, double);
 // CHECK-LABEL: @test_double_braces()
 // CHECK:       %braces = alloca double, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store double 0.000000e+00, double* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(longdouble, long double);
 // CHECK-LABEL: @test_longdouble_uninit()
 // CHECK:       %uninit = alloca x86_fp80, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_longdouble_uninit()
+// PATTERN: store x86_fp80 0xK7FFFC000000000000000, x86_fp80* %uninit, align
+// ZERO-LABEL: @test_longdouble_uninit()
+// ZERO: store x86_fp80 0xK00000000000000000000, x86_fp80* %uninit, align
 
 TEST_BRACES(longdouble, long double);
 // CHECK-LABEL: @test_longdouble_braces()
 // CHECK:       %braces = alloca x86_fp80, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store x86_fp80 0xK00000000000000000000, x86_fp80* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 
 TEST_UNINIT(intptr, int*);
 // CHECK-LABEL: @test_intptr_uninit()
 // CHECK:       %uninit = alloca i32*, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_intptr_uninit()
+// PATTERN: store i32* inttoptr (i64 -6148914691236517206 to i32*), i32** %uninit, align
+// ZERO-LABEL: @test_intptr_uninit()
+// ZERO: store i32* null, i32** %uninit, align
 
 TEST_BRACES(intptr, int*);
 // CHECK-LABEL: @test_intptr_braces()
 // CHECK:       %braces = alloca i32*, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32* null, i32** %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(intptrptr, int**);
 // CHECK-LABEL: @test_intptrptr_uninit()
 // CHECK:       %uninit = alloca i32**, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_intptrptr_uninit()
+// PATTERN: store i32** inttoptr (i64 -6148914691236517206 to i32**), i32*** %uninit, align
+// ZERO-LABEL: @test_intptrptr_uninit()
+// ZERO: store i32** null, i32*** %uninit, align
 
 TEST_BRACES(intptrptr, int**);
 // CHECK-LABEL: @test_intptrptr_braces()
 // CHECK:       %braces = alloca i32**, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32** null, i32*** %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(function, void(*)());
 // CHECK-LABEL: @test_function_uninit()
 // CHECK:       %uninit = alloca void ()*, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_function_uninit()
+// PATTERN: store void ()* inttoptr (i64 -6148914691236517206 to void ()*), void ()** %uninit, align
+// ZERO-LABEL: @test_function_uninit()
+// ZERO: store void ()* null, void ()** %uninit, align
 
 TEST_BRACES(function, void(*)());
 // CHECK-LABEL: @test_function_braces()
 // CHECK:       %braces = alloca void ()*, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store void ()* null, void ()** %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(bool, bool);
 // CHECK-LABEL: @test_bool_uninit()
 // CHECK:       %uninit = alloca i8, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_bool_uninit()
+// PATTERN: store i8 -86, i8* %uninit, align 1
+// ZERO-LABEL: @test_bool_uninit()
+// ZERO: store i8 0, i8* %uninit, align 1
 
 TEST_BRACES(bool, bool);
 // CHECK-LABEL: @test_bool_braces()
 // CHECK:       %braces = alloca i8, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i8 0, i8* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 
 TEST_UNINIT(empty, empty);
 // CHECK-LABEL: @test_empty_uninit()
 // CHECK:       %uninit = alloca %struct.empty, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_empty_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_empty_uninit.uninit
+// ZERO-LABEL: @test_empty_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(empty, empty);
 // CHECK-LABEL: @test_empty_braces()
 // CHECK:       %braces = alloca %struct.empty, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(small, small);
 // CHECK-LABEL: @test_small_uninit()
 // CHECK:       %uninit = alloca %struct.small, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_small_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_small_uninit.uninit
+// ZERO-LABEL: @test_small_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(small, small);
 // CHECK-LABEL: @test_small_braces()
 // CHECK:       %braces = alloca %struct.small, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 1, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
-  TEST_CUSTOM(small, small, { 42 });
 // CHECK-LABEL: @test_small_custom()
 // CHECK:       %custom = alloca %struct.small, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(smallinit, smallinit);
 // CHECK-LABEL: @test_smallinit_uninit()

 // CHECK-NEXT:  store i8 100, i8* %[[C]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(smallpartinit, smallpartinit);
 // CHECK-LABEL: @test_smallpartinit_uninit()
 // CHECK:       %uninit = alloca %struct.smallpartinit, align
 // CHECK-NEXT:  call void @{{.*}}smallpartinit{{.*}}%uninit)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_smallpartinit_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_smallpartinit_uninit.uninit
+// ZERO-LABEL: @test_smallpartinit_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(smallpartinit, smallpartinit);
 // CHECK-LABEL: @test_smallpartinit_braces()
 // CHECK:       %braces = alloca %struct.smallpartinit, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[C:[^ ]*]] = getelementptr inbounds %struct.smallpartinit, %struct.smallpartinit* %braces, i32 0, i32 0
 // CHECK-NEXT:  store i8 42, i8* %[[C]], align [[ALIGN]]
 // CHECK-NEXT:  %[[D:[^ ]*]] = getelementptr inbounds %struct.smallpartinit, %struct.smallpartinit* %braces, i32 0, i32 1
 // CHECK-NEXT:  store i8 0, i8* %[[D]], align [[ALIGN]]

 // CHECK-NEXT:  %[[N:[^ ]*]] = getelementptr inbounds %struct.nullinit, %struct.nullinit* %custom, i32 0, i32 0
 // CHECK-NEXT:  store i8* getelementptr inbounds {{.*}}, i8** %[[N]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(padded, padded);
 // CHECK-LABEL: @test_padded_uninit()
 // CHECK:       %uninit = alloca %struct.padded, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_padded_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_padded_uninit.uninit
+// ZERO-LABEL: @test_padded_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(padded, padded);
 // CHECK-LABEL: @test_padded_braces()
 // CHECK:       %braces = alloca %struct.padded, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 8, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(padded, padded, { 42, 13371337 });
 // CHECK-LABEL: @test_padded_custom()
 // CHECK:       %custom = alloca %struct.padded, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(paddednullinit, paddednullinit);
 // CHECK-LABEL: @test_paddednullinit_uninit()
 // CHECK:       %uninit = alloca %struct.paddednullinit, align
 // CHECK-NEXT:  call void @{{.*}}paddednullinit{{.*}}%uninit)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_paddednullinit_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_paddednullinit_uninit.uninit
+// ZERO-LABEL: @test_paddednullinit_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(paddednullinit, paddednullinit);
 // CHECK-LABEL: @test_paddednullinit_braces()
 // CHECK:       %braces = alloca %struct.paddednullinit, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[C:[^ ]*]] = getelementptr inbounds %struct.paddednullinit, %struct.paddednullinit* %braces, i32 0, i32 0
 // CHECK-NEXT:  store i8 0, i8* %[[C]], align [[ALIGN]]
 // CHECK-NEXT:  %[[I:[^ ]*]] = getelementptr inbounds %struct.paddednullinit, %struct.paddednullinit* %braces, i32 0, i32 1
 // CHECK-NEXT:  store i32 0, i32* %[[I]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(paddednullinit, paddednullinit, { 42, 13371337 });
 // CHECK-LABEL: @test_paddednullinit_custom()
 // CHECK:       %custom = alloca %struct.paddednullinit, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[C:[^ ]*]] = getelementptr inbounds %struct.paddednullinit, %struct.paddednullinit* %custom, i32 0, i32 0
 // CHECK-NEXT:  store i8 42, i8* %[[C]], align [[ALIGN]]
 // CHECK-NEXT:  %[[I:[^ ]*]] = getelementptr inbounds %struct.paddednullinit, %struct.paddednullinit* %custom, i32 0, i32 1
 // CHECK-NEXT:  store i32 13371337, i32* %[[I]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(bitfield, bitfield);
 // CHECK-LABEL: @test_bitfield_uninit()
 // CHECK:       %uninit = alloca %struct.bitfield, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_bitfield_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_bitfield_uninit.uninit
+// ZERO-LABEL: @test_bitfield_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(bitfield, bitfield);
 // CHECK-LABEL: @test_bitfield_braces()
 // CHECK:       %braces = alloca %struct.bitfield, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(bitfield, bitfield, { 4, 1 });
 // CHECK-LABEL: @test_bitfield_custom()
 // CHECK:       %custom = alloca %struct.bitfield, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(bitfieldaligned, bitfieldaligned);
 // CHECK-LABEL: @test_bitfieldaligned_uninit()
 // CHECK:       %uninit = alloca %struct.bitfieldaligned, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_bitfieldaligned_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_bitfieldaligned_uninit.uninit
+// ZERO-LABEL: @test_bitfieldaligned_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(bitfieldaligned, bitfieldaligned);
 // CHECK-LABEL: @test_bitfieldaligned_braces()
 // CHECK:       %braces = alloca %struct.bitfieldaligned, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(bitfieldaligned, bitfieldaligned, { 4, 1  });
 // CHECK-LABEL: @test_bitfieldaligned_custom()
 // CHECK:       %custom = alloca %struct.bitfieldaligned, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(big, big);
 // CHECK-LABEL: @test_big_uninit()
 // CHECK:       %uninit = alloca %struct.big, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_big_uninit()
+// PATTERN: call void @llvm.memset{{.*}}, i8 -86,
+// ZERO-LABEL: @test_big_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(big, big);
 // CHECK-LABEL: @test_big_braces()
 // CHECK:       %braces = alloca %struct.big, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 104, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(big, big, { 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA, 0xAAAAAAAA });
 // CHECK-LABEL: @test_big_custom()
 // CHECK:       %custom = alloca %struct.big, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 -86, i64 104, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(arraytail, arraytail);
 // CHECK-LABEL: @test_arraytail_uninit()
 // CHECK:       %uninit = alloca %struct.arraytail, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_arraytail_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_arraytail_uninit.uninit
+// ZERO-LABEL: @test_arraytail_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(arraytail, arraytail);
 // CHECK-LABEL: @test_arraytail_braces()
 // CHECK:       %braces = alloca %struct.arraytail, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 4, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(arraytail, arraytail, { 0xdead });
 // CHECK-LABEL: @test_arraytail_custom()
 // CHECK:       %custom = alloca %struct.arraytail, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 TEST_UNINIT(int0, int[0]);
 // CHECK-LABEL: @test_int0_uninit()
 // CHECK:       %uninit = alloca [0 x i32], align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_int0_uninit()
+// PATTERN:       %uninit = alloca [0 x i32], align
+// PATTERN-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// ZERO-LABEL: @test_int0_uninit()
+// ZERO:       %uninit = alloca [0 x i32], align
+// ZERO-NEXT:  call void @{{.*}}used{{.*}}%uninit)
 
 TEST_BRACES(int0, int[0]);
 // CHECK-LABEL: @test_int0_braces()
 // CHECK:       %braces = alloca [0 x i32], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 0, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(int1, int[1]);
 // CHECK-LABEL: @test_int1_uninit()
 // CHECK:       %uninit = alloca [1 x i32], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_int1_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_int1_uninit.uninit
+// ZERO-LABEL: @test_int1_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(int1, int[1]);
 // CHECK-LABEL: @test_int1_braces()
 // CHECK:       %braces = alloca [1 x i32], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 4, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(int1, int[1], { 0x33333333 });
 // CHECK-LABEL: @test_int1_custom()
 // CHECK:       %custom = alloca [1 x i32], align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(int64, int[64]);
 // CHECK-LABEL: @test_int64_uninit()
 // CHECK:       %uninit = alloca [64 x i32], align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_int64_uninit()
+// PATTERN: call void @llvm.memset{{.*}}, i8 -86,
+// ZERO-LABEL: @test_int64_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(int64, int[64]);
 // CHECK-LABEL: @test_int64_braces()
 // CHECK:       %braces = alloca [64 x i32], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 256, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(int64, int[64], = { 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111, 0x11111111 });
 // CHECK-LABEL: @test_int64_custom()
 // CHECK:       %custom = alloca [64 x i32], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 17, i64 256, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(bool4, bool[4]);
 // CHECK-LABEL: @test_bool4_uninit()
 // CHECK:       %uninit = alloca [4 x i8], align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_bool4_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_bool4_uninit.uninit
+// ZERO-LABEL: @test_bool4_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(bool4, bool[4]);
 // CHECK-LABEL: @test_bool4_braces()
 // CHECK:       %braces = alloca [4 x i8], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 4, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(bool4, bool[4], { true, true, true, true });
 // CHECK-LABEL: @test_bool4_custom()
 // CHECK:       %custom = alloca [4 x i8], align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(intptr4, int*[4]);
 // CHECK-LABEL: @test_intptr4_uninit()
 // CHECK:       %uninit = alloca [4 x i32*], align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_intptr4_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_intptr4_uninit.uninit
+// ZERO-LABEL: @test_intptr4_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(intptr4, int*[4]);
 // CHECK-LABEL: @test_intptr4_braces()
 // CHECK:       %braces = alloca [4 x i32*], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 32, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
   TEST_CUSTOM(intptr4, int*[4], = { (int*)0x22222222, (int*)0x22222222, (int*)0x22222222, (int*)0x22222222 });
 // CHECK-LABEL: @test_intptr4_custom()
 // CHECK:       %custom = alloca [4 x i32*], align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(tailpad4, tailpad[4]);
 // CHECK-LABEL: @test_tailpad4_uninit()
 // CHECK:       %uninit = alloca [4 x %struct.tailpad], align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_tailpad4_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_tailpad4_uninit.uninit
+// ZERO-LABEL: @test_tailpad4_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(tailpad4, tailpad[4]);
 // CHECK-LABEL: @test_tailpad4_braces()
 // CHECK:       %braces = alloca [4 x %struct.tailpad], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 16, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(tailpad4, tailpad[4], { {257, 1}, {257, 1}, {257, 1}, {257, 1} });
 // CHECK-LABEL: @test_tailpad4_custom()
 // CHECK:       %custom = alloca [4 x %struct.tailpad], align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(tailpad9, tailpad[9]);
 // CHECK-LABEL: @test_tailpad9_uninit()
 // CHECK:       %uninit = alloca [9 x %struct.tailpad], align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_tailpad9_uninit()
+// PATTERN: call void @llvm.memset{{.*}}, i8 -86,
+// ZERO-LABEL: @test_tailpad9_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(tailpad9, tailpad[9]);
 // CHECK-LABEL: @test_tailpad9_braces()
 // CHECK:       %braces = alloca [9 x %struct.tailpad], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 36, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(tailpad9, tailpad[9], { {257, 1}, {257, 1}, {257, 1}, {257, 1}, {257, 1}, {257, 1}, {257, 1}, {257, 1}, {257, 1} });
 // CHECK-LABEL: @test_tailpad9_custom()
 // CHECK:       %custom = alloca [9 x %struct.tailpad], align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 1, i64 36, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 TEST_UNINIT(atomicbool, _Atomic(bool));
 // CHECK-LABEL: @test_atomicbool_uninit()
 // CHECK:       %uninit = alloca i8, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_atomicbool_uninit()
+// PATTERN: store i8 -86, i8* %uninit, align 1
+// ZERO-LABEL: @test_atomicbool_uninit()
+// ZERO: store i8 0, i8* %uninit, align 1
 
 TEST_UNINIT(atomicint, _Atomic(int));
 // CHECK-LABEL: @test_atomicint_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_atomicint_uninit()
+// PATTERN: store i32 -1431655766, i32* %uninit, align 4
+// ZERO-LABEL: @test_atomicint_uninit()
+// ZERO: store i32 0, i32* %uninit, align 4
 
 TEST_UNINIT(atomicdouble, _Atomic(double));
 // CHECK-LABEL: @test_atomicdouble_uninit()
 // CHECK:       %uninit = alloca double, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_atomicdouble_uninit()
+// PATTERN: store double 0x7FF8000000000000, double* %uninit, align 8
+// ZERO-LABEL: @test_atomicdouble_uninit()
+// ZERO: store double 0.000000e+00, double* %uninit, align 8
 
 TEST_UNINIT(atomicnotlockfree, _Atomic(notlockfree));
 // CHECK-LABEL: @test_atomicnotlockfree_uninit()
 // CHECK:       %uninit = alloca %struct.notlockfree, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_atomicnotlockfree_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_atomicnotlockfree_uninit.uninit
+// ZERO-LABEL: @test_atomicnotlockfree_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_UNINIT(atomicpadded, _Atomic(padded));
 // CHECK-LABEL: @test_atomicpadded_uninit()
 // CHECK:       %uninit = alloca %struct.padded, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_atomicpadded_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_atomicpadded_uninit.uninit
+// ZERO-LABEL: @test_atomicpadded_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_UNINIT(atomictailpad, _Atomic(tailpad));
 // CHECK-LABEL: @test_atomictailpad_uninit()
 // CHECK:       %uninit = alloca %struct.tailpad, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_atomictailpad_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_atomictailpad_uninit.uninit
+// ZERO-LABEL: @test_atomictailpad_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 
 TEST_UNINIT(complexfloat, _Complex float);
 // CHECK-LABEL: @test_complexfloat_uninit()
 // CHECK:       %uninit = alloca { float, float }, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_complexfloat_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_complexfloat_uninit.uninit
+// ZERO-LABEL: @test_complexfloat_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(complexfloat, _Complex float);
 // CHECK-LABEL: @test_complexfloat_braces()
 // CHECK:       %braces = alloca { float, float }, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[R:[^ ]*]] = getelementptr inbounds { float, float }, { float, float }* %braces, i32 0, i32 0
 // CHECK-NEXT:  %[[I:[^ ]*]] = getelementptr inbounds { float, float }, { float, float }* %braces, i32 0, i32 1
 // CHECK-NEXT:  store float 0.000000e+00, float* %[[R]], align [[ALIGN]]
 // CHECK-NEXT:  store float 0.000000e+00, float* %[[I]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(complexfloat, _Complex float, { 3.1415926535897932384626433, 3.1415926535897932384626433 });
 // CHECK-LABEL: @test_complexfloat_custom()
 // CHECK:       %custom = alloca { float, float }, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[R:[^ ]*]] = getelementptr inbounds { float, float }, { float, float }* %custom, i32 0, i32 0
 // CHECK-NEXT:  %[[I:[^ ]*]] = getelementptr inbounds { float, float }, { float, float }* %custom, i32 0, i32 1
 // CHECK-NEXT:  store float 0x400921FB60000000, float* %[[R]], align [[ALIGN]]
 // CHECK-NEXT:  store float 0x400921FB60000000, float* %[[I]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(complexdouble, _Complex double);
 // CHECK-LABEL: @test_complexdouble_uninit()
 // CHECK:       %uninit = alloca { double, double }, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_complexdouble_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_complexdouble_uninit.uninit
+// ZERO-LABEL: @test_complexdouble_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(complexdouble, _Complex double);
 // CHECK-LABEL: @test_complexdouble_braces()
 // CHECK:       %braces = alloca { double, double }, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[R:[^ ]*]] = getelementptr inbounds { double, double }, { double, double }* %braces, i32 0, i32 0
 // CHECK-NEXT:  %[[I:[^ ]*]] = getelementptr inbounds { double, double }, { double, double }* %braces, i32 0, i32 1
 // CHECK-NEXT:  store double 0.000000e+00, double* %[[R]], align [[ALIGN]]
 // CHECK-NEXT:  store double 0.000000e+00, double* %[[I]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(complexdouble, _Complex double, { 3.1415926535897932384626433, 3.1415926535897932384626433 });
 // CHECK-LABEL: @test_complexdouble_custom()
 // CHECK:       %custom = alloca { double, double }, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  %[[R:[^ ]*]] = getelementptr inbounds { double, double }, { double, double }* %custom, i32 0, i32 0
 // CHECK-NEXT:  %[[I:[^ ]*]] = getelementptr inbounds { double, double }, { double, double }* %custom, i32 0, i32 1
 // CHECK-NEXT:  store double 0x400921FB54442D18, double* %[[R]], align [[ALIGN]]
 // CHECK-NEXT:  store double 0x400921FB54442D18, double* %[[I]], align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 TEST_UNINIT(volatileint, volatile int);
 // CHECK-LABEL: @test_volatileint_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_volatileint_uninit()
+// PATTERN: store volatile i32 -1431655766, i32* %uninit, align 4
+// ZERO-LABEL: @test_volatileint_uninit()
+// ZERO: store volatile i32 0, i32* %uninit, align 4
 
 TEST_BRACES(volatileint, volatile int);
 // CHECK-LABEL: @test_volatileint_braces()
 // CHECK:       %braces = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store volatile i32 0, i32* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(semivolatile, semivolatile);
 // CHECK-LABEL: @test_semivolatile_uninit()
 // CHECK:       %uninit = alloca %struct.semivolatile, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_semivolatile_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_semivolatile_uninit.uninit
+// ZERO-LABEL: @test_semivolatile_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(semivolatile, semivolatile);
 // CHECK-LABEL: @test_semivolatile_braces()
 // CHECK:       %braces = alloca %struct.semivolatile, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 8, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 

 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 TEST_UNINIT(base, base);
 // CHECK-LABEL: @test_base_uninit()
 // CHECK:       %uninit = alloca %struct.base, align
 // CHECK-NEXT:  call void @{{.*}}base{{.*}}%uninit)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_base_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_base_uninit.uninit
+// ZERO-LABEL: @test_base_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(base, base);
 // CHECK-LABEL: @test_base_braces()
 // CHECK:       %braces = alloca %struct.base, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 8, i1 false)
 // CHECK-NEXT:  call void @{{.*}}base{{.*}}%braces)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(derived, derived);
 // CHECK-LABEL: @test_derived_uninit()
 // CHECK:       %uninit = alloca %struct.derived, align
 // CHECK-NEXT:  call void @{{.*}}derived{{.*}}%uninit)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_derived_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_derived_uninit.uninit
+// ZERO-LABEL: @test_derived_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(derived, derived);
 // CHECK-LABEL: @test_derived_braces()
 // CHECK:       %braces = alloca %struct.derived, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 8, i1 false)
 // CHECK-NEXT:  call void @{{.*}}derived{{.*}}%braces)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_UNINIT(virtualderived, virtualderived);
 // CHECK-LABEL: @test_virtualderived_uninit()
 // CHECK:       %uninit = alloca %struct.virtualderived, align
 // CHECK-NEXT:  call void @{{.*}}virtualderived{{.*}}%uninit)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_virtualderived_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_virtualderived_uninit.uninit
+// ZERO-LABEL: @test_virtualderived_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(virtualderived, virtualderived);
 // CHECK-LABEL: @test_virtualderived_braces()
 // CHECK:       %braces = alloca %struct.virtualderived, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 16, i1 false)
 // CHECK-NEXT:  call void @{{.*}}virtualderived{{.*}}%braces)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 
 TEST_UNINIT(matching, matching);
 // CHECK-LABEL: @test_matching_uninit()
 // CHECK:       %uninit = alloca %union.matching, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_matching_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_matching_uninit.uninit
+// ZERO-LABEL: @test_matching_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(matching, matching);
 // CHECK-LABEL: @test_matching_braces()
 // CHECK:       %braces = alloca %union.matching, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 4, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(matching, matching, { .f = 0xf00f });
 // CHECK-LABEL: @test_matching_custom()
 // CHECK:       %custom = alloca %union.matching, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(matchingreverse, matchingreverse);
 // CHECK-LABEL: @test_matchingreverse_uninit()
 // CHECK:       %uninit = alloca %union.matchingreverse, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_matchingreverse_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_matchingreverse_uninit.uninit
+// ZERO-LABEL: @test_matchingreverse_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(matchingreverse, matchingreverse);
 // CHECK-LABEL: @test_matchingreverse_braces()
 // CHECK:       %braces = alloca %union.matchingreverse, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 4, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(matchingreverse, matchingreverse, { .i = 0xf00f });
 // CHECK-LABEL: @test_matchingreverse_custom()
 // CHECK:       %custom = alloca %union.matchingreverse, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(unmatched, unmatched);
 // CHECK-LABEL: @test_unmatched_uninit()
 // CHECK:       %uninit = alloca %union.unmatched, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_unmatched_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_unmatched_uninit.uninit
+// ZERO-LABEL: @test_unmatched_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(unmatched, unmatched);
 // CHECK-LABEL: @test_unmatched_braces()
 // CHECK:       %braces = alloca %union.unmatched, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(unmatched, unmatched, { .i = 0x3badbeef });
 // CHECK-LABEL: @test_unmatched_custom()
 // CHECK:       %custom = alloca %union.unmatched, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(unmatchedreverse, unmatchedreverse);
 // CHECK-LABEL: @test_unmatchedreverse_uninit()
 // CHECK:       %uninit = alloca %union.unmatchedreverse, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_unmatchedreverse_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_unmatchedreverse_uninit.uninit
+// ZERO-LABEL: @test_unmatchedreverse_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(unmatchedreverse, unmatchedreverse);
 // CHECK-LABEL: @test_unmatchedreverse_braces()
 // CHECK:       %braces = alloca %union.unmatchedreverse, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memset{{.*}}(i8* align [[ALIGN]] %{{.*}}, i8 0, i64 4, i1 false)
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(unmatchedreverse, unmatchedreverse, { .c = 42  });
 // CHECK-LABEL: @test_unmatchedreverse_custom()
 // CHECK:       %custom = alloca %union.unmatchedreverse, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(unmatchedfp, unmatchedfp);
 // CHECK-LABEL: @test_unmatchedfp_uninit()
 // CHECK:       %uninit = alloca %union.unmatchedfp, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_unmatchedfp_uninit()
+// PATTERN: call void @llvm.memcpy{{.*}} @__const.test_unmatchedfp_uninit.uninit
+// ZERO-LABEL: @test_unmatchedfp_uninit()
+// ZERO: call void @llvm.memset{{.*}}, i8 0,
 
 TEST_BRACES(unmatchedfp, unmatchedfp);
 // CHECK-LABEL: @test_unmatchedfp_braces()
 // CHECK:       %braces = alloca %union.unmatchedfp, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(unmatchedfp, unmatchedfp, { .d = 3.1415926535897932384626433 });
 // CHECK-LABEL: @test_unmatchedfp_custom()
 // CHECK:       %custom = alloca %union.unmatchedfp, align
 // CHECK-NEXT:  bitcast
 // CHECK-NEXT:  call void @llvm.memcpy
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 TEST_UNINIT(emptyenum, emptyenum);
 // CHECK-LABEL: @test_emptyenum_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_emptyenum_uninit()
+// PATTERN: store i32 -1431655766, i32* %braces, align 4
+// ZERO-LABEL: @test_emptyenum_uninit()
+// ZERO: store i32 0, i32* %braces, align 4
 
 TEST_BRACES(emptyenum, emptyenum);
 // CHECK-LABEL: @test_emptyenum_braces()
 // CHECK:       %braces = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 0, i32* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(emptyenum, emptyenum, { (emptyenum)42 });
 // CHECK-LABEL: @test_emptyenum_custom()
 // CHECK:       %custom = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 42, i32* %custom, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(smallenum, smallenum);
 // CHECK-LABEL: @test_smallenum_uninit()
 // CHECK:       %uninit = alloca i32, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_smallenum_uninit()
+// PATTERN: store i32 -1431655766, i32* %braces, align 4
+// ZERO-LABEL: @test_smallenum_uninit()
+// ZERO: store i32 0, i32* %braces, align 4
 
 TEST_BRACES(smallenum, smallenum);
 // CHECK-LABEL: @test_smallenum_braces()
 // CHECK:       %braces = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 0, i32* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(smallenum, smallenum, { (smallenum)42 });
 // CHECK-LABEL: @test_smallenum_custom()
 // CHECK:       %custom = alloca i32, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store i32 42, i32* %custom, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 TEST_UNINIT(intvec16, int  __attribute__((vector_size(16))));
 // CHECK-LABEL: @test_intvec16_uninit()
 // CHECK:       %uninit = alloca <4 x i32>, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_intvec16_uninit()
+// PATTERN: store <4 x i32> <i32 -1431655766, i32 -1431655766, i32 -1431655766, i32 -1431655766>, <4 x i32>* %uninit, align 16
+// ZERO-LABEL: @test_intvec16_uninit()
+// ZERO: store <4 x i32> zeroinitializer, <4 x i32>* %uninit, align 16
 
 TEST_BRACES(intvec16, int  __attribute__((vector_size(16))));
 // CHECK-LABEL: @test_intvec16_braces()
 // CHECK:       %braces = alloca <4 x i32>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x i32> zeroinitializer, <4 x i32>* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
-  TEST_CUSTOM(intvec16, int  __attribute__((vector_size(16))), { 0x44444444, 0x44444444, 0x44444444, 0x44444444 });
 // CHECK-LABEL: @test_intvec16_custom()
 // CHECK:       %custom = alloca <4 x i32>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x i32> <i32 1145324612, i32 1145324612, i32 1145324612, i32 1145324612>, <4 x i32>* %custom, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(longlongvec32, long long  __attribute__((vector_size(32))));
 // CHECK-LABEL: @test_longlongvec32_uninit()
 // CHECK:       %uninit = alloca <4 x i64>, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_longlongvec32_uninit()
+// PATTERN: store <4 x i64> <i64 -6148914691236517206, i64 -6148914691236517206, i64 -6148914691236517206, i64 -6148914691236517206>, <4 x i64>* %uninit, align 32
+// ZERO-LABEL: @test_longlongvec32_uninit()
+// ZERO: store <4 x i64> zeroinitializer, <4 x i64>* %uninit, align 32
 
 TEST_BRACES(longlongvec32, long long  __attribute__((vector_size(32))));
 // CHECK-LABEL: @test_longlongvec32_braces()
 // CHECK:       %braces = alloca <4 x i64>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x i64> zeroinitializer, <4 x i64>* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(longlongvec32, long long  __attribute__((vector_size(32))), { 0x3333333333333333, 0x3333333333333333, 0x3333333333333333, 0x3333333333333333 });
 // CHECK-LABEL: @test_longlongvec32_custom()
 // CHECK:       %custom = alloca <4 x i64>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x i64> <i64 3689348814741910323, i64 3689348814741910323, i64 3689348814741910323, i64 3689348814741910323>, <4 x i64>* %custom, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(floatvec16, float  __attribute__((vector_size(16))));
 // CHECK-LABEL: @test_floatvec16_uninit()
 // CHECK:       %uninit = alloca <4 x float>, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_floatvec16_uninit()
+// PATTERN: store <4 x float> <float 0x7FF8000000000000, float 0x7FF8000000000000, float 0x7FF8000000000000, float 0x7FF8000000000000>, <4 x float>* %uninit, align 16
+// ZERO-LABEL: @test_floatvec16_uninit()
+// ZERO: store <4 x float> zeroinitializer, <4 x float>* %uninit, align 16
 
 TEST_BRACES(floatvec16, float  __attribute__((vector_size(16))));
 // CHECK-LABEL: @test_floatvec16_braces()
 // CHECK:       %braces = alloca <4 x float>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x float> zeroinitializer, <4 x float>* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(floatvec16, float  __attribute__((vector_size(16))), { 3.1415926535897932384626433, 3.1415926535897932384626433, 3.1415926535897932384626433, 3.1415926535897932384626433 });
 // CHECK-LABEL: @test_floatvec16_custom()
 // CHECK:       %custom = alloca <4 x float>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x float> <float 0x400921FB60000000, float 0x400921FB60000000, float 0x400921FB60000000, float 0x400921FB60000000>, <4 x float>* %custom, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 TEST_UNINIT(doublevec32, double  __attribute__((vector_size(32))));
 // CHECK-LABEL: @test_doublevec32_uninit()
 // CHECK:       %uninit = alloca <4 x double>, align
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%uninit)
+// PATTERN-LABEL: @test_doublevec32_uninit()
+// PATTERN: store <4 x double> <double 0x7FF8000000000000, double 0x7FF8000000000000, double 0x7FF8000000000000, double 0x7FF8000000000000>, <4 x double>* %uninit, align 32
+// ZERO-LABEL: @test_doublevec32_uninit()
+// ZERO: store <4 x double> zeroinitializer, <4 x double>* %uninit, align 32
 
 TEST_BRACES(doublevec32, double  __attribute__((vector_size(32))));
 // CHECK-LABEL: @test_doublevec32_braces()
 // CHECK:       %braces = alloca <4 x double>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x double> zeroinitializer, <4 x double>* %braces, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%braces)
 
 TEST_CUSTOM(doublevec32, double  __attribute__((vector_size(32))), { 3.1415926535897932384626433, 3.1415926535897932384626433, 3.1415926535897932384626433, 3.1415926535897932384626433 });
 // CHECK-LABEL: @test_doublevec32_custom()
 // CHECK:       %custom = alloca <4 x double>, align [[ALIGN:[0-9]*]]
 // CHECK-NEXT:  store <4 x double> <double 0x400921FB54442D18, double 0x400921FB54442D18, double 0x400921FB54442D18, double 0x400921FB54442D18>, <4 x double>* %custom, align [[ALIGN]]
 // CHECK-NEXT:  call void @{{.*}}used{{.*}}%custom)
 
 
 } // extern "C"

test/CodeGenCXX/trivial-auto-var-init-attribute.cpp

+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks %s -emit-llvm -o - | FileCheck %s -check-prefix=UNINIT
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks -ftrivial-auto-var-init=pattern %s -emit-llvm -o - | FileCheck %s -check-prefix=PATTERN
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks -ftrivial-auto-var-init=zero %s -emit-llvm -o - | FileCheck %s -check-prefix=ZERO
+
+template<typename T> void used(T &) noexcept;
+
+extern "C" {
+
+// UNINIT-LABEL:  test_attribute_uninitialized
+// ZERO-LABEL:    test_attribute_uninitialized
+// PATTERN-LABEL: test_attribute_uninitialized
+void test_attribute_uninitialized() {
+  [[clang::trivial_auto_init("uninitialized")]] int i;
+  used(i);
+}
+
+// UNINIT-LABEL:  test_attribute_zero
+// ZERO-LABEL:    test_attribute_zero
+// PATTERN-LABEL: test_attribute_zero
+void test_attribute_zero() {
+  [[clang::trivial_auto_init("zero")]] int i;
+  used(i);
+}
+
+// UNINIT-LABEL:  test_attribute_pattern
+// ZERO-LABEL:    test_attribute_pattern
+// PATTERN-LABEL: test_attribute_pattern
+void test_attribute_pattern() {
+  [[clang::trivial_auto_init("pattern")]] int i;
+  used(i);
+}
+
+} // extern "C"

test/CodeGenCXX/trivial-auto-var-init.cpp

+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks %s -emit-llvm -o - | FileCheck %s -check-prefix=UNINIT
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks -ftrivial-auto-var-init=pattern %s -emit-llvm -o - | FileCheck %s -check-prefix=PATTERN
+// RUN: %clang_cc1 -triple x86_64-unknown-unknown -fblocks -ftrivial-auto-var-init=zero %s -emit-llvm -o - | FileCheck %s -check-prefix=ZERO
+
+// None of the synthesized globals should contain `undef`.
+// PATTERN-NOT: undef
+// ZERO-NOT: undef
+
+template<typename T> void used(T &) noexcept;
+
+extern "C" {
+
+// UNINIT-LABEL:  test_selfinit(
+// ZERO-LABEL:    test_selfinit(
+// ZERO: store i32 0, i32* %self, align 4
+// PATTERN-LABEL: test_selfinit(
+// PATTERN: store i32 -1431655766, i32* %self, align 4
+void test_selfinit() {
+  int self = self + 1;
+  used(self);
+}
+
+// UNINIT-LABEL:  test_block(
+// ZERO-LABEL:    test_block(
+// ZERO: store i32 0, i32* %block, align 4
+// PATTERN-LABEL: test_block(
+// PATTERN: store i32 -1431655766, i32* %block, align 4
+void test_block() {
+  __block int block;
+  used(block);
+}
+
+// This type of code is currently not handled by zero / pattern initialization.
+// The test will break when that is fixed.
+// UNINIT-LABEL:  test_goto_unreachable_value(
+// ZERO-LABEL:    test_goto_unreachable_value(
+// ZERO-NOT: store {{.*}}%oops
+// PATTERN-LABEL: test_goto_unreachable_value(
+// PATTERN-NOT: store {{.*}}%oops
+void test_goto_unreachable_value() {
+  goto jump;
+  int oops;
+ jump:
+  used(oops);
+}
+
+// This type of code is currently not handled by zero / pattern initialization.
+// The test will break when that is fixed.
+// UNINIT-LABEL:  test_goto(
+// ZERO-LABEL:    test_goto(
+// ZERO: if.then:
+// ZERO: br label %jump
+// ZERO: store i32 0, i32* %oops, align 4
+// ZERO: br label %jump
+// ZERO: jump:
+// PATTERN-LABEL: test_goto(
+// PATTERN: if.then:
+// PATTERN: br label %jump
+// PATTERN: store i32 -1431655766, i32* %oops, align 4
+// PATTERN: br label %jump
+// PATTERN: jump:
+void test_goto(int i) {
+  if (i)
+    goto jump;
+  int oops;
+ jump:
+  used(oops);
+}
+
+// This type of code is currently not handled by zero / pattern initialization.
+// The test will break when that is fixed.
+// UNINIT-LABEL:  test_switch(
+// ZERO-LABEL:    test_switch(
+// ZERO:      sw.bb:
+// ZERO-NEXT: store i32 0, i32* %oops, align 4
+// ZERO:      sw.bb1:
+// ZERO-NEXT: call void @{{.*}}used
+// PATTERN-LABEL: test_switch(
+// PATTERN:      sw.bb:
+// PATTERN-NEXT: store i32 -1431655766, i32* %oops, align 4
+// PATTERN:      sw.bb1:
+// PATTERN-NEXT: call void @{{.*}}used
+void test_switch(int i) {
+  switch (i) {
+  case 0:
+    int oops;
+    break;
+  case 1:
+    used(oops);
+  }
+}
+
+// UNINIT-LABEL:  test_vla(
+// ZERO-LABEL:    test_vla(
+// ZERO:  %[[SIZE:[0-9]+]] = mul nuw i64 %{{.*}}, 4
+// ZERO:  call void @llvm.memset{{.*}}(i8* align 16 %{{.*}}, i8 0, i64 %[[SIZE]], i1 false)
+// PATTERN-LABEL: test_vla(
+// PATTERN:  %vla.iszerosized = icmp eq i64 %{{.*}}, 0
+// PATTERN:  br i1 %vla.iszerosized, label %vla-init.cont, label %vla-setup.loop
+// PATTERN: vla-setup.loop:
+// PATTERN:  %[[SIZE:[0-9]+]] = mul nuw i64 %{{.*}}, 4
+// PATTERN:  %vla.begin = bitcast i32* %vla to i8*
+// PATTERN:  %vla.end = getelementptr inbounds i8, i8* %vla.begin, i64 %[[SIZE]]
+// PATTERN:  br label %vla-init.loop
+// PATTERN: vla-init.loop:
+// PATTERN:  %vla.cur = phi i8* [ %vla.begin, %vla-setup.loop ], [ %vla.next, %vla-init.loop ]
+// PATTERN:  call void @llvm.memcpy{{.*}} %vla.cur, {{.*}}@__const.test_vla.vla
+// PATTERN:  %vla.next = getelementptr inbounds i8, i8* %vla.cur, i64 4
+// PATTERN:  %vla-init.isdone = icmp eq i8* %vla.next, %vla.end
+// PATTERN:  br i1 %vla-init.isdone, label %vla-init.cont, label %vla-init.loop
+// PATTERN: vla-init.cont:
+// PATTERN:  call void @{{.*}}used
+void test_vla(int size) {
+  // Variable-length arrays can't have a zero size according to C11 6.7.6.2/5.
+  // Neither can they be negative-sized.
+  //
+  // We don't use the former fact because some code creates zero-sized VLAs and
+  // doesn't use them. clang makes these share locations with other stack
+  // values, which leads to initialization of the wrong values.
+  //
+  // We rely on the later fact because it generates better code.
+  //
+  // Both cases are caught by UBSan.
+  int vla[size];
+  int *ptr = vla;
+  used(ptr);
+}
+
+// UNINIT-LABEL:  test_struct_vla(
+// ZERO-LABEL:    test_struct_vla(
+// ZERO:  %[[SIZE:[0-9]+]] = mul nuw i64 %{{.*}}, 16
+// ZERO:  call void @llvm.memset{{.*}}(i8* align 16 %{{.*}}, i8 0, i64 %[[SIZE]], i1 false)
+// PATTERN-LABEL: test_struct_vla(
+// PATTERN:  %vla.iszerosized = icmp eq i64 %{{.*}}, 0
+// PATTERN:  br i1 %vla.iszerosized, label %vla-init.cont, label %vla-setup.loop
+// PATTERN: vla-setup.loop:
+// PATTERN:  %[[SIZE:[0-9]+]] = mul nuw i64 %{{.*}}, 16
+// PATTERN:  %vla.begin = bitcast %struct.anon* %vla to i8*
+// PATTERN:  %vla.end = getelementptr inbounds i8, i8* %vla.begin, i64 %[[SIZE]]
+// PATTERN:  br label %vla-init.loop
+// PATTERN: vla-init.loop:
+// PATTERN:  %vla.cur = phi i8* [ %vla.begin, %vla-setup.loop ], [ %vla.next, %vla-init.loop ]
+// PATTERN:  call void @llvm.memcpy{{.*}} %vla.cur, {{.*}}@__const.test_struct_vla.vla
+// PATTERN:  %vla.next = getelementptr inbounds i8, i8* %vla.cur, i64 16
+// PATTERN:  %vla-init.isdone = icmp eq i8* %vla.next, %vla.end
+// PATTERN:  br i1 %vla-init.isdone, label %vla-init.cont, label %vla-init.loop
+// PATTERN: vla-init.cont:
+// PATTERN:  call void @{{.*}}used
+void test_struct_vla(int size) {
+  // Same as above, but with a struct that doesn't just memcpy.
+  struct {
+    float f;
+    char c;
+    void *ptr;
+  } vla[size];
+  void *ptr = static_cast<void*>(vla);
+  used(ptr);
+}
+
+// UNINIT-LABEL:  test_zsa(
+// ZERO-LABEL:    test_zsa(
+// ZERO: %zsa = alloca [0 x i32], align 4
+// ZERO-NOT: %zsa
+// ZERO:  call void @{{.*}}used
+// PATTERN-LABEL: test_zsa(
+// PATTERN: %zsa = alloca [0 x i32], align 4
+// PATTERN-NOT: %zsa
+// PATTERN:  call void @{{.*}}used
+void test_zsa(int size) {
+  // Technically not valid, but as long as clang accepts them we should do
+  // something sensible (i.e. not store to the zero-size array).
+  int zsa[0];
+  used(zsa);
+}
+  
+// UNINIT-LABEL:  test_huge_uninit(
+// ZERO-LABEL:    test_huge_uninit(
+// ZERO: call void @llvm.memset{{.*}}, i8 0, i64 65536,
+// PATTERN-LABEL: test_huge_uninit(
+// PATTTERN: call void @llvm.memset{{.*}}, i8 -86, i64 65536,
+void test_huge_uninit() {
+  // We can't emit this as an inline constant to a store instruction because
+  // SDNode hits an internal size limit.
+  char big[65536];
+  used(big);
+}
+
+// UNINIT-LABEL:  test_huge_small_init(
+// ZERO-LABEL:    test_huge_small_init(
+// ZERO: call void @llvm.memset{{.*}}, i8 0, i64 65536,
+// ZERO: store i8 97,
+// ZERO: store i8 98,
+// ZERO: store i8 99,
+// ZERO: store i8 100,
+// PATTERN-LABEL: test_huge_small_init(
+// PATTTERN: call void @llvm.memset{{.*}}, i8 -86, i64 65536,
+// PATTERN: store i8 97,
+// PATTERN: store i8 98,
+// PATTERN: store i8 99,
+// PATTERN: store i8 100,
+void test_huge_small_init() {
+  char big[65536] = { 'a', 'b', 'c', 'd' };
+  used(big);
+}
+
+// UNINIT-LABEL:  test_huge_larger_init(
+// ZERO-LABEL:    test_huge_larger_init(
+// ZERO:  call void @llvm.memcpy{{.*}} @__const.test_huge_larger_init.big, {{.*}}, i64 65536,
+// PATTERN-LABEL: test_huge_larger_init(
+// PATTERN:  call void @llvm.memcpy{{.*}} @__const.test_huge_larger_init.big, {{.*}}, i64 65536,
+void test_huge_larger_init() {
+  char big[65536] = { 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' };
+  used(big);
+}
+
+} // extern "C"

test/Driver/clang_f_opts.c

 // RUN: %clang -### -S -fno-delete-null-pointer-checks -fdelete-null-pointer-checks %s 2>&1 | FileCheck -check-prefix=CHECK-NULL-POINTER-CHECKS %s
 // CHECK-NO-NULL-POINTER-CHECKS: "-fno-delete-null-pointer-checks"
 // CHECK-NULL-POINTER-CHECKS-NOT: "-fno-delete-null-pointer-checks"
 
 // RUN: %clang -### -S -fomit-frame-pointer -pg %s 2>&1 | FileCheck -check-prefix=CHECK-NO-MIX-OMIT-FP-PG %s
 // RUN: %clang -### -S -fomit-frame-pointer -fno-omit-frame-pointer -pg %s 2>&1 | FileCheck -check-prefix=CHECK-MIX-NO-OMIT-FP-PG %s
 // CHECK-NO-MIX-OMIT-FP-PG: '-fomit-frame-pointer' not allowed with '-pg'
 // CHECK-MIX-NO-OMIT-FP-PG-NOT: '-fomit-frame-pointer' not allowed with '-pg'
+
+// RUN: %clang -### -S -ftrivial-auto-var-init=uninitialized %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-UNINIT %s
+// RUN: %clang -### -S -ftrivial-auto-var-init=pattern %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-PATTERN %s
+// RUN: %clang -### -S -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-ZERO-GOOD %s
+// RUN: %clang -### -S -ftrivial-auto-var-init=zero %s 2>&1 | FileCheck -check-prefix=CHECK-TRIVIAL-ZERO-BAD %s
+// CHECK-TRIVIAL-UNINIT-NOT: hasn't been enabled
+// CHECK-TRIVIAL-PATTERN-NOT: hasn't been enabled
+// CHECK-TRIVIAL-ZERO-GOOD-NOT: hasn't been enabled
+// CHECK-TRIVIAL-ZERO-BAD: hasn't been enabled

test/Misc/warning-flags.c

(2) It prevents us adding new warnings to Clang that have no -W flag.  All
     new warnings should have -W flags.
 
     If you add a new warning without a flag, this test will fail.  To fix
     this test, simply add a warning group to that warning.
 
 
 The list of warnings below should NEVER grow.  It should gradually shrink to 0.
 
-CHECK: Warnings without flags (75):
+CHECK: Warnings without flags (76):
 CHECK-NEXT:   ext_excess_initializers
 CHECK-NEXT:   ext_excess_initializers_in_char_array_initializer
 CHECK-NEXT:   ext_expected_semi_decl_list
 CHECK-NEXT:   ext_explicit_specialization_storage_class
 CHECK-NEXT:   ext_initializer_string_for_char_array_too_long
 CHECK-NEXT:   ext_missing_declspec
 CHECK-NEXT:   ext_missing_whitespace_after_macro_name
 CHECK-NEXT:   ext_new_paren_array_nonconst

 CHECK-NEXT:   warn_conv_to_base_not_used
 CHECK-NEXT:   warn_conv_to_self_not_used
 CHECK-NEXT:   warn_conv_to_void_not_used
 CHECK-NEXT:   warn_delete_array_type
 CHECK-NEXT:   warn_double_const_requires_fp64
 CHECK-NEXT:   warn_drv_assuming_mfloat_abi_is
 CHECK-NEXT:   warn_drv_clang_unsupported
 CHECK-NEXT:   warn_drv_pch_not_first_include
+CHECK-NEXT:   warn_drv_trivial_auto_var_init_zero_disabled
 CHECK-NEXT:   warn_dup_category_def
 CHECK-NEXT:   warn_enum_value_overflow
 CHECK-NEXT:   warn_expected_qualified_after_typename
 CHECK-NEXT:   warn_extraneous_char_constant
 CHECK-NEXT:   warn_fe_cc_log_diagnostics_failure
 CHECK-NEXT:   warn_fe_cc_print_header_failure
 CHECK-NEXT:   warn_fe_macro_contains_embedded_newline
 CHECK-NEXT:   warn_ignoring_ftabstop_value

test/Sema/attr-trivial_auto_init.c

+// RUN: %clang_cc1 %s -verify -fsyntax-only
+
+void good() {
+    int dont_initialize_me __attribute((trivial_auto_init("uninitialized")));
+    int zero_me __attribute((trivial_auto_init("zero")));
+    int pattern_me __attribute((trivial_auto_init("pattern")));
+}
+
+void bad() {
+    int im_bad __attribute((trivial_auto_init)); // expected-error {{'trivial_auto_init' attribute takes one argument}}
+    int im_baaad __attribute((trivial_auto_init("uninitialized", "zero"))); // expected-error {{'trivial_auto_init' attribute takes one argument}}
+    static int come_on __attribute((trivial_auto_init("uninitialized"))); // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+    int you_know __attribute((trivial_auto_init("pony"))); // expected-warning {{'trivial_auto_init' attribute argument not supported: pony}}
+    int and_the_whole_world_has_to __attribute((trivial_auto_init(uninitialized))); // expected-error {{'trivial_auto_init' attribute requires a string}}
+}
+
+extern int answer_right_now __attribute((trivial_auto_init("uninitialized"))); // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+int just_to_tell_you_once_again __attribute((trivial_auto_init("uninitialized"))); // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+static int whos_bad __attribute((trivial_auto_init("uninitialized"))); // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+
+void the_word_is_out() __attribute((trivial_auto_init("uninitialized"))) {} // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+void youre_doin_wrong(__attribute((trivial_auto_init("uninitialized"))) int a) {} // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+
+struct GonnaLockYouUp {
+  __attribute((trivial_auto_init("uninitialized"))) int before_too_long; // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}
+} __attribute((trivial_auto_init("uninitialized"))); // expected-warning {{'trivial_auto_init' attribute only applies to local variables}}

test/Sema/uninit-variables.c

 // RUN: %clang_cc1 -fsyntax-only -Wuninitialized -Wconditional-uninitialized -fsyntax-only -fblocks %s -verify
+// RUN: %clang_cc1 -fsyntax-only -Wuninitialized -Wconditional-uninitialized -ftrivial-auto-var-init=pattern -fsyntax-only -fblocks %s -verify
 
 typedef __typeof(sizeof(int)) size_t;
 void *malloc(size_t);
 
 int test1() {
   int x; // expected-note{{initialize the variable 'x' to silence this warning}}
   return x; // expected-warning{{variable 'x' is uninitialized when used here}}
 }