FileCheck [12/12]: Support use of var defined on same line

[thopre]

Summary

This patch is part of a patch series to add support for FileCheck
numeric expressions. This specific patch adds support for using a
numeric variable with empty numeric expression defined on the same line
either directly or indirectly.

The regex language used to represent a constraint corresponding to a
given CHECK directive does not allow to express a numeric relation. It
is therefore necessary to initially match any number in the appropriate
matching format and check the constraints are verified in a follow-up
step. Such a process must then be repeated as long as the constraints
fail because of cases with a check directive like #N: #N+1
where the input contains lines such as:
10 12
10 11

The first line would match the regex "[0-9]+ [0-9]+" generated for the
CHECK directive but the constraint would fail. The second line would
then need to be tried since it both matches successfully and verifies
the constraints of the CHECK directive. Note that this process would
still fail to match if the input was:
10 12 13
because the regex would match 10 and 12 which fail to verify the
constraints and the process would then skip to the next line since there
is no way to ask the regex engine if the line had other possible
matches. This caveat is documented.

Copyright:

• Linaro (changes up to diff 183612 of revision D55940)
• GraphCore (changes in later versions of revision D55940 and in new revision created off D55940)

[thopre]

Use camel casing for new functions

[thopre]

Rebase

[thopre]

Make numeric value constructor explicit

[thopre]

Rebase on latest changes

[thopre]

Rebase on latest changes

[thopre]

Rebase on latest unittest changes

[thopre]

Rebase on top of latest changes

[thopre]

Rebase on top of latest changes

[thopre]

Rebase on top of latest changes

[thopre]

Rebase on top of latest changes

[thopre]

Rebase on top of latest changes

[thopre]

Rebase on top of latest changes

[thopre]

Rebase on top of latest changes

[thopre]

rebase on top of latest changes

[thopre]

rebase on top of latest changes

[thopre]

Rebase on top of latest changes in patch series

[thopre]

Rebase on top of latest changes in the series

[thopre]

Rebase on top of latest changes in patch series

[thopre]

Codebase has changed considerably since last time this review was updated. Patch will need some reworking and before that I want to build consensus on whether the feature should be done given the caveats it has.

[jhenderson]

I don't have a clear solution to the problems you're describing, but I have already run into one or two places where I wanted the behaviour of being able to use a numeric variable on the same line it was defined. CHECK-SAME is a valid workaround, but has its own issues (for example, it doesn't inherently ensure the next match starts immediately after the previous one, and is just simply more verbose). My instinct is that the right approach is to ignore the constraint initially, match a number, and then verify that it is the right number, failing if not, without further search attempts. I think it's relatively easy to workaround that issue in most cases, with additional checks elsewhere, although I can't be positive that is the case.

[thopre]

In D60392#2033357, @jhenderson wrote:

I don't have a clear solution to the problems you're describing, but I have already run into one or two places where I wanted the behaviour of being able to use a numeric variable on the same line it was defined. CHECK-SAME is a valid workaround, but has its own issues (for example, it doesn't inherently ensure the next match starts immediately after the previous one, and is just simply more verbose). My instinct is that the right approach is to ignore the constraint initially, match a number, and then verify that it is the right number, failing if not, without further search attempts. I think it's relatively easy to workaround that issue in most cases, with additional checks elsewhere, although I can't be positive that is the case.

What worries me is the effect on a CHECK-NOT directive since it could lead to the directive being fulfilled in a case where the intent was to forbid it. Consider the following:

CHECK-NOT: [[#VAR:]] [[#VAR+1]]

Say the author of the test was looking a a bug where the output being checked was 10 11 14. The test would pass. Now if the bug wasn't fully fixed, perhaps this doesn't happen anymore but we can have cases like 10 13 14. The above directive will turn into ([[:digit:]]+) ([[:digit:]]+) which will match 10 and 13 and will then check if they are consecutive numbers. They are not and thus the CHECK-NOT is satisfied even though there was 13 14 later on the line. Of course here we could retry after the first number matched but that doesn't work in the general case where regex can appear anywhere (e.g. #VAR: {{.*}} #VAR+1 where the input is now 10 8 11).

Perhaps the solution is to forbid using a variable defined on the same line for a CHECK-NOT directive? In all other cases we'll get a false negative (directive not satisfied) instead of a false positive (directive satisfied).

[jhenderson]

In D60392#2033995, @thopre wrote:

Perhaps the solution is to forbid using a variable defined on the same line for a CHECK-NOT directive? In all other cases we'll get a false negative (directive not satisfied) instead of a false positive (directive satisfied).

Hmm, you make a fair argument, and I think this suggestion might work. I'll need to give it more thought though. Perhaps it needs a mailing list thread to get more eyes on this?

[thopre]

In D60392#2035853, @jhenderson wrote:

In D60392#2033995, @thopre wrote:

Perhaps the solution is to forbid using a variable defined on the same line for a CHECK-NOT directive? In all other cases we'll get a false negative (directive not satisfied) instead of a false positive (directive satisfied).

Hmm, you make a fair argument, and I think this suggestion might work. I'll need to give it more thought though. Perhaps it needs a mailing list thread to get more eyes on this?

That was my intention. I'm focusing on D60390 for now, then I'll start a thread.

[thopre]

Status update: I've started manually rebasing this patch on top of latest FileCheck. Hopefully I'll update this diff in about 2 weeks. Sorry for the delay.

[thopre]

In D60392#2682536, @thopre wrote:

Status update: I've started manually rebasing this patch on top of latest FileCheck. Hopefully I'll update this diff in about 2 weeks. Sorry for the delay.

Sorry, I dropped the ball on that one but recently picked it up again and have a patch ready. It needs clean up but it's feature complete and all tests are already written. Please hang in there.

[thopre]

Rebase and fix issues with variables defined from an expression and with
verification failure failing to restore initial values of variables.

[jhenderson]

Sorry for ignoring this for a couple of weeks - it fell off my radar a bit, when I was swamped with deadline-realted tasks.

I recommend doing the llvm/test/FileCheck/numeric-expressions/numeric-expressions.txt renaming in a separate patch, as it's not really related to this patch.

I've not looked at the testing in depth yet, and I'm a little concerned that I'm very cold on the surrounding code. I think it would be useful if someone else could take a look too (@jdenny maybe?).

llvm/docs/CommandGuide/FileCheck.rst

650–663

Where this refers to "line" should it be "CHECK" directive?

Also, I think it would be valuable to give an example.

654–655

llvm/lib/FileCheck/FileCheck.cpp

382 ↗

(On Diff #381083)

My general rule is that consumeError should always have a comment explaining why you can just "ignore" the error and continue. In this case, I guess you're not ignoring the error per se, but you are throwing away the context, so a comment would still be useful.

411–415 ↗

(On Diff #381083)

Do you need this change? Can't you just return the result of EvalBinop directly?

449–450 ↗

(On Diff #381083)

1184 ↗

(On Diff #381083)

TODO that needs action still?

1307 ↗

(On Diff #381083)

(or possibly "using variables with empty expressions")

1311–1312 ↗

(On Diff #381083)

(or "using variables")

Should this definitely be "line"?

Perhaps change "Known issue" to FIXME and file a bug (bug filing to happen once this patch has landed, or shortly beforehand, with the FIXME to contain a link to that bug).

1322 ↗

(On Diff #381083)

1323–1324 ↗

(On Diff #381083)

1385–1386 ↗

(On Diff #381083)

1403 ↗

(On Diff #381083)

1424 ↗

(On Diff #381083)

1425–1426 ↗

(On Diff #381083)

1431 ↗

(On Diff #381083)

1433 ↗

(On Diff #381083)

No need for braces for single line if.

1438 ↗

(On Diff #381083)

Any reason we can't just use a for loop here?

1487 ↗

(On Diff #381083)

llvm/lib/FileCheck/FileCheckImpl.h

269 ↗

(On Diff #381083)

292–294 ↗

(On Diff #381083)

Seems like TentativeValue and StrTentativeValue are unnecessary? Could you not just set Value and StrValue in this context? If you need to distinguish a tentative versus concrete definition, add a boolean member or similar that states isTentative.

520 ↗

(On Diff #381083)

578 ↗

(On Diff #381083)

Nit: add new line after this method declaration.

[jdenny]

In D60392#3137020, @jhenderson wrote:

I've not looked at the testing in depth yet, and I'm a little concerned that I'm very cold on the surrounding code. I think it would be useful if someone else could take a look too (@jdenny maybe?).

Sorry for the delay, but I need a bit more time. I'll plan to take a look next week. Thanks for working on this.

[jdenny]

I understand the desire to be able to define and reference a variable within the same directive. However, as I understand it, this patch adds that ability in a way that introduces a new subtlety to FileCheck: the semantics of numeric variable references are then not always consistent. If you assume the wrong semantics, you can end up with a false pass of a test. I understand you've disabled the case of CHECK-NOT, but it does not appear to be the only case that can lead to false passes.

Example

Let's say we want to check that the very last value of y on an input line is equal to the first value of y plus 3. Thus, the following input line is then erroneous:

x=20, y=30, z=40, x=21, y=31, z=41, x=22, y=33, z=42, x=23, y=32, z=43,

Check version 1

CHECK: y=[[#VAR:]],{{.*}}y=[[#VAR+3]],

The behavior is:

• y=[[#VAR:]], matches y=30,.
• {{.*}} is maximal munch.
• So y=[[#VAR+3]], matches the last y= expression, which is y=32,.

The directive then fails because 32 is not 30+3. OK, that's the exact behavior I was going for. And now I think I understand how numeric variable references work.

(If the above is not the behavior you expect after applying this patch, let me know. Every example I tried after applying this patch produced seg faults or assertion failures, so I couldn't verify.)

Check version 2

Let's say I later realize I'd rather use VAR as defined to 30 by some earlier directive, so I rewrite check version 1 as follows:

CHECK: y=[[#VAR]],{{.*}}y=[[#VAR+3]],

The behavior is:

• y=[[#VAR]], is visibly differently than y=[[#VAR:]],, and its semantics are different too: it now matches a previously defined value instead of capturing a new value. Nevertheless, it still matches y=30,. So far, so good.
• {{.*}} is still maximal munch.
• Given that previous parts of the pattern match as before and VAR ends up with the same value as before, shouldn't the semantics of y=[[#VAR+3]], be the same as before? No. It can only match y=33, now, and so that's what it does.

The directive then accepts the erroneous input. That's pretty subtle.

Check version 3

Or, let's say I decide that splitting check version 1 into two directives would look better:

     CHECK: y=[[#VAR:]],
CHECK-SAME: {{.*}}y=[[#VAR+3]],

That's equivalent to version 1, right? Normally, I think it would be. Not in this case. Again, y=[[#VAR+3]], now has a different behavior: it only matches y=33,, and the erroneous input is now accepted.

Conclusion

I'm leery of letting the semantics of numeric variable references vary depending on where the variable happens to have been defined. It's too subtle.

I think it would be better to introduce the desired functionality with a different syntax to reflect the semantic shift. For example:

CHECK: y=[[#VAR1:]],{{.*}}y=[[#VAR2:]],[[!#VAR2 == VAR1 + 3]]

That introduces the concept of a post-match variable assertion. It would always appear at the end of a pattern to indicate it's evaluated last. It would always be delimited by [[! and ]], which hopefully doesn't collide with any existing syntax.

There's probably a better syntax I haven't thought of. My main point is to demonstrate how a different syntax could eliminate the inconsistent behavior I am concerned about.

[jdenny]

Or maybe a CHECK-ASSERT would be clearer and more generally useful:

CHECK: y=[[#VAR1:]],{{.*}}y=[[#VAR2:]]
CHECK-ASSERT: #VAR2 == VAR1 + 3

[thopre]

Hi Joel,

Thanks for your feedback, good job catching that!

In D60392#3181763, @jdenny wrote:

I understand the desire to be able to define and reference a variable within the same directive. However, as I understand it, this patch adds that ability in a way that introduces a new subtlety to FileCheck: the semantics of numeric variable references are then not always consistent. If you assume the wrong semantics, you can end up with a false pass of a test. I understand you've disabled the case of CHECK-NOT, but it does not appear to be the only case that can lead to false passes.

Example

Let's say we want to check that the very last value of y on an input line is equal to the first value of y plus 3. Thus, the following input line is then erroneous:

x=20, y=30, z=40, x=21, y=31, z=41, x=22, y=33, z=42, x=23, y=32, z=43,

Check version 1

CHECK: y=[[#VAR:]],{{.*}}y=[[#VAR+3]],

The behavior is:

• y=[[#VAR:]], matches y=30,.
• {{.*}} is maximal munch.
• So y=[[#VAR+3]], matches the last y= expression, which is y=32,.

The directive then fails because 32 is not 30+3. OK, that's the exact behavior I was going for. And now I think I understand how numeric variable references work.

(If the above is not the behavior you expect after applying this patch, let me know. Every example I tried after applying this patch produced seg faults or assertion failures, so I couldn't verify.)

I think the real issue here is using a wildcard pattern. I feel most often that not the user actually intended that. I remember someone asking for being able to use variables defined on the same line because using CHECK-SAME was different as pointed in your example below. See I think this is the behaviour people usually expect, judging from https://bugs.llvm.org/show_bug.cgi?id=30198 also about wildcard being a risky business.

Check version 2

Let's say I later realize I'd rather use VAR as defined to 30 by some earlier directive, so I rewrite check version 1 as follows:

CHECK: y=[[#VAR]],{{.*}}y=[[#VAR+3]],

The behavior is:

• y=[[#VAR]], is visibly differently than y=[[#VAR:]],, and its semantics are different too: it now matches a previously defined value instead of capturing a new value. Nevertheless, it still matches y=30,. So far, so good.
• {{.*}} is still maximal munch.
• Given that previous parts of the pattern match as before and VAR ends up with the same value as before, shouldn't the semantics of y=[[#VAR+3]], be the same as before? No. It can only match y=33, now, and so that's what it does.

The directive then accepts the erroneous input. That's pretty subtle.

I think the main issue is the regex was badly written in the first place but I agree this patch is making things worse by having inconsistent behaviour. Someone ought to have written either of those two regex instead depending on intended pattern to be matched:

CHECK: y=[[#VAR:]],{{.*}}y=[[#VAR+3]],{{[^y]*$}}

or

CHECK: y=[[#VAR:]],{{[^y]*}}y=[[#VAR+3]],

I like the suggestion of CHECK-ASSERT or a new syntax. I'll think about it and in the meantime anyone with an opinion on this feel free to chime in.

[jdenny]

In D60392#3182279, @thopre wrote:

Hi Joel,

Hi Thomas.

I like the suggestion of CHECK-ASSERT or a new syntax. I'll think about it

Thanks for considering that.

and in the meantime anyone with an opinion on this feel free to chime in.

Yes, it will be good to get other opinions. @jhenderson?

[jhenderson]

In D60392#3184072, @jdenny wrote:

In D60392#3182279, @thopre wrote:

Hi Joel,

Hi Thomas.

I like the suggestion of CHECK-ASSERT or a new syntax. I'll think about it

Thanks for considering that.

and in the meantime anyone with an opinion on this feel free to chime in.

Yes, it will be good to get other opinions. @jhenderson?

I think @jdenny has hit the nail on the head with regards to a fairly major problem with this patch. I don't think we want to introduce subtleties in behaviour like has been highlighted. Sorry for not considering this before.

Regarding the CHECK-ASSERT suggestion, do we need it? I was under the impression one of the earlier patches added a syntax like [[#VAR2, VAR1 + 3:]] or something to that effect, which achieves the same thing, right? Maybe I'm missing something though.

[thopre]

In D60392#3184688, @jhenderson wrote:

Regarding the CHECK-ASSERT suggestion, do we need it? I was under the impression one of the earlier patches added a syntax like [[#VAR2, VAR1 + 3:]] or something to that effect, which achieves the same thing, right? Maybe I'm missing something though.

You mean [[#VAR2:VAR1+3]]? Why does that prevent the need for something like CHECK-ASSERT? You still can only refer to a variable defined in an earlier line.

[jhenderson]

In D60392#3184854, @thopre wrote:

In D60392#3184688, @jhenderson wrote:

Regarding the CHECK-ASSERT suggestion, do we need it? I was under the impression one of the earlier patches added a syntax like [[#VAR2, VAR1 + 3:]] or something to that effect, which achieves the same thing, right? Maybe I'm missing something though.

You mean [[#VAR2:VAR1+3]]? Why does that prevent the need for something like CHECK-ASSERT? You still can only refer to a variable defined in an earlier line.

Looking back at the example, I guess it doesn't, although you could use CHECK-SAME to achieve the same effect, I think. More generally, I'm not sure how much I like the idea of checks that don't actually match anything, but I could be persuaded, especially if we can leverage the same functionality to support the CHECK-DEFINE concept I proposed previously (but never got as far as finishing off).

[jdenny]

In D60392#3188307, @jhenderson wrote:

In D60392#3184854, @thopre wrote:

In D60392#3184688, @jhenderson wrote:

Regarding the CHECK-ASSERT suggestion, do we need it? I was under the impression one of the earlier patches added a syntax like [[#VAR2, VAR1 + 3:]] or something to that effect, which achieves the same thing, right? Maybe I'm missing something though.

You mean [[#VAR2:VAR1+3]]? Why does that prevent the need for something like CHECK-ASSERT? You still can only refer to a variable defined in an earlier line.

Looking back at the example, I guess it doesn't, although you could use CHECK-SAME to achieve the same effect, I think. More generally, I'm not sure how much I like the idea of checks that don't actually match anything, but I could be persuaded, especially if we can leverage the same functionality to support the CHECK-DEFINE concept I proposed previously (but never got as far as finishing off).

I too would like a CHECK-DEFINE. Conceptually at least, I imagine CHECK-DEFINE or CHECK-ASSERT would match an empty string at the start of its search range (that is, end of the previous match or beginning of input if none). Giving it a match location in this way should help with diagnostics, such as -dump-input.

llvm/docs/CommandGuide/FileCheck.rst

 
 A numeric variable can also be defined to the result of a numeric expression,
 in which case the numeric expression constraint is checked and if verified the
 variable is assigned to the value. The unified syntax for both defining numeric
 variables and checking a numeric expression is thus
 ``[[#%<fmtspec>,<NUMVAR>: <constraint> <expr>]]`` with each element as
 described previously.
 
 In the same way as for pattern variables, ``--enable-var-scope`` only consider
 global numeric variables that start with ``$`` and undefined local variables at
 the beginning of each CHECK-LABEL block.
 
-Important note: In its current implementation, a numeric expression cannot use
-a numeric variable with a non-empty numeric expression constraint defined on
-the same line.
+Important note: In its current implementation, a numeric expression using
+numeric variable defined on the same line can fail to match even though an
+input line satisfies it.  This happens when there is more than one possible
+match in the line that should match when ignoring the constraint and
+considering only the matching format (ie. matching an unsigned number with
+[[:digit:]]+ and the first match does not satisfy the constraint.  This is
+due to those numeric expressions constraints not being able to be expressed at
+the regular expression level and the regular expression engine not offering the
+possibility of getting all possible matches.
 
 FileCheck Pseudo Numeric Variables
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Sometimes there's a need to verify output which contains line numbers of the
 match file, e.g. when testing compiler diagnostics.  This introduces a certain
 fragility of the match file structure, as "``CHECK:``" lines contain absolute
 line numbers in the same file, which have to be updated whenever line numbers
 change due to text addition or deletion.

llvm/include/llvm/Support/FileCheck.h

union {
     int64_t SignedValue;
     uint64_t UnsignedValue;
   };
 
   /// Whether value is signed (and thus is stored in SignedValue) or not (in
   /// which case it is stored in UnsignedValue).
   bool Signed;
 
-  /// Whether this holds an actual value. Examples of where this would be
-  /// false:
-  /// - underflow or overflow of one of the binary operation in the expression;
-  /// - unset variable's value.
+  /// Whether this holds an actual value. Examples of where this would be false
+  /// include:
+  /// - underflow or overflow of one of the binary operation in the expression
+  /// - value of undefined variable
   bool Valid;
 
+  /// Whether this is a tentative value, ie other expressions on the same CHECK
+  /// line that might be using that value or other values that this value was
+  /// computed from have not yet been verified to match. True if any of the
+  /// operand involved in its computation had a tentative value.
+  bool Tentative;
+
 public:
   /// Constructor for an invalid value.
   FileCheckNumExprVal() : Valid(false) {}
 
   /// Constructor for a signed value.
   FileCheckNumExprVal(int64_t Val)
-      : SignedValue(Val), Signed(true), Valid(true) {}
+      : SignedValue(Val), Signed(true), Valid(true), Tentative(false) {}
 
   /// Constructor for an unsigned value.
   FileCheckNumExprVal(uint64_t Val)
-      : UnsignedValue(Val), Signed(false), Valid(true) {}
+      : UnsignedValue(Val), Signed(false), Valid(true), Tentative(false) {}
 
   /// Define equality to be true only if both values are valid and they have
   /// the same signedness and corresponding value.  Tentative bit is ignored to
   /// allow the evaluation of a numeric expression using variables with
   /// tentative value to compare equal to a matched value.
   bool operator==(const FileCheckNumExprVal &other);
   bool operator!=(const FileCheckNumExprVal &other) {
     return !(*this == other);
   }
 
   bool isSigned() const { return Signed; }
 
   bool isValid() const { return Valid; }
 
+  /// Whether this is a tentative value, ie other expressions on the same CHECK
+  /// line that might be using that value or other values that this value was
+  /// computed from have not yet been verified to match. True if any of the
+  /// operand involved in its computation had a tentative value.
+  bool isTentative() const { return Tentative; }
+
   /// Return the signed value. Must only be called if value is signed in the
   /// first place.
   int64_t getSignedValue() const {
     assert(Signed);
     return SignedValue;
   }
 
   /// Return the unsigned value. Must only be called if value is unsigned in
   /// the first place.
   uint64_t getUnsignedValue() const {
     assert(!Signed);
     return UnsignedValue;
   }
 
+  /// Change whether this value is tentative or not.
+  bool toggleTentative() { return Tentative = !Tentative; }
+
   /// Convert value to a signed value, or mark value invalid if not possible
   /// (original value was not within range for a signed integer).
   void convertSigned();
 
   /// Convert value to an unsigned value, or mark value invalid if not possible
   /// (original value was not within range for an unsigned integer).
   void convertUnsigned();
 

 private:
   /// Pointer to AST of the numeric expression.
   std::shared_ptr<FileCheckNumExprAST> AST;
 
   /// Matching format, i.e format (e.g. hex upper case letters) to use for the
   /// value when matching it.
   FileCheckNumExprFmt Fmt;
 
+  /// Whether the value of this numeric expression can be known at match time
+  /// (ie. when substituting it in the regular expression). If False, the
+  /// value is known after matching has happened.
+  bool MatchTimeKnown;
+
 public:
   /// Generic constructor for a numeric expression whose equality constraint is
-  /// represented by \p AST and matching format is \p Fmt. If matching format
-  /// is unset (ie. no explicit or implicit matching format), set it to default
-  /// one (unsigned decimal integer).
+  /// represented by \p AST, whose matching format is \p Fmt and which is known
+  /// at match time if \p MatchTimeKnown is true. If matching format is unset
+  /// (i.e. no explicit or implicit matching format), set it to default one
+  /// (unsigned decimal integer).
   FileCheckNumExpr(std::shared_ptr<FileCheckNumExprAST> AST,
-                   FileCheckNumExprFmt Fmt);
+                   FileCheckNumExprFmt Fmt, bool MatchTimeKnown);
 
   /// Return pointer to AST of the numeric expression. Pointer is guaranteed to
   /// be valid as long as this object is.
   FileCheckNumExprAST *getAST() const { return AST.get(); }
 
+  /// Return whether the value of this numeric expression can be known at match
+  /// time (ie. when substituting it in the regular expression). If False, the
+  /// value is known after matching has happened.
+  bool isMatchTimeKnown() const { return MatchTimeKnown; }
+
   /// Return effective format of this numeric expression, ie (i) its explicit
   /// format if any, (ii) its implicit format if any or (iii) the default
   /// format.
   FileCheckNumExprFmt getEffectiveFmt() { return Fmt; }
+
+  /// Verify that the matched value in \p MatchedValue satisfies the constraint
+  /// expressed by this expression. Return true if constraint is not satisfied.
+  bool verifyConstraint(FileCheckNumExprVal MatchedValue) const;
 };
 
 /// Class representing a numeric variable with a given value in the AST of a
 /// numeric expression. Each definition of a variable gets its own instance of
 /// this class, variable uses share the same instance as the respective
 /// definition.
 class FileCheckNumExprVar : public FileCheckNumExprAST {
 private:
public:
   bool isMatchTimeKnown() const;
 
   /// Return implicit format of this numeric variable.
   FileCheckNumExprFmt getImplicitFmt() const;
 
   /// Append numeric variable's name to UndefVarNames if undefined.
   void appendUndefVarNames(std::vector<StringRef> &UndefVarNames) const;
 
-  /// Set value of this numeric variable if not defined. Return whether
-  /// variable was already defined.
-  bool setValue(FileCheckNumExprVal Value);
+  /// Set value of this numeric variable if not defined. Value is set as a
+  /// tentative value if \p Tentative is true. Return whether variable was
+  /// already definitely (i.e. not tentatively) defined.
+  bool setValue(FileCheckNumExprVal Value, bool Tentative);
+
+  /// Mark value as definitive (i.e. not tentative). Return whether value was
+  /// already definitive.
+  bool commitValue();
 
   /// Clear value of this numeric variable. Return whether value was already
   /// undefined.
   bool clearValue();
 
   /// Return line number where this variable is defined.
   unsigned getDefLineNumber() { return DefLineNumber; }
 };
public:
 
   int getCount() const { return Count; }
   FileCheckType &setCount(int C);
 
   std::string getDescription(StringRef Prefix) const;
 };
 } // namespace Check
 
-/// Structure representing the definition of a numeric variable in a pattern.
-/// It holds the parenthesized capture number and the pointer to the class
-/// instance holding the value and matching format of the numeric variable
-/// being defined.
+/// Structure representing a numeric expression whose constraint needs to be
+/// verified after matching (either a variable definition with empty numeric
+/// expression or a numeric expression using directly or indirectly such a
+/// variable. It holds the parenthesized capture number, a pointer to the class
+/// instance holding the numeric expression to verify and in case of a numeric
+/// variable definition a pointer to the class instance holding the value and
+/// matching format of the numeric variable being defined.
 struct FileCheckNumExprMatch {
+  /// Pointer to class instance representing the numeric expression to verify.
+  std::shared_ptr<FileCheckNumExpr> NumExpr;
+
   /// Pointer to class instance holding the value and matching format of the
   /// numeric variable being defined.
   std::shared_ptr<FileCheckNumExprVar> NumVarDef;
 
-  /// Parenthesized capture number for this numeric variable definition.
+  /// Parenthesized capture number for the numeric expression that needs
+  /// verifying.
   unsigned CaptureParen;
 };
 
 /// Class holding the FileCheckPattern global state, shared by all patterns:
 /// tables holding values of variables and whether they are defined or not at
 /// any given time in the matching process.
 class FileCheckPatternContext {
   friend class FileCheckPattern;
class FileCheckPattern {
   /// E.g. for the pattern "foo[[bar:.*]]baz[[bar]]quux[[bar:.*]]",
   /// VariableDefs will map "bar" to 2 corresponding to the second definition
   /// of "bar".
   ///
   /// Note: uses std::map rather than StringMap to be able to get the key when
   /// iterating over values.
   std::map<StringRef, unsigned> VariableDefs;
 
-  /// Holds the capture parentheses number and pointer to corresponding
-  /// FileCheckNumExprVar class instance of all numeric variable definitions.
-  /// Used to set the matched value of all those variables.
-  std::map<StringRef, FileCheckNumExprMatch> NumericVariableDefs;
+  /// Holds the capture parentheses number, pointer to corresponding
+  /// FileCheckNumExpr class instance of the numeric expression to verify and
+  /// pointer to FileCheckNumExprVar class instance of variable defined from
+  /// this numeric expression if any for all numeric expressions that needs to
+  /// be verified after matching.
+  std::vector<FileCheckNumExprMatch> CapturedNumericExpressions;
 
   /// Pointer to a class instance holding the global state shared by all
   /// patterns:
   /// - tables with the values of live pattern and numeric variables at
   ///   the start of any given CHECK line;
   /// - table holding whether a pattern variable has been defined at any given
   ///   point during the parsing phase.
   FileCheckPatternContext *Context;
private:
   size_t FindRegexVarEnd(StringRef Str, SourceMgr &SM);
 
   /// Numeric expression parsing helpers.
   enum AllowedOperand { LegacyVar, LegacyLiteral, Any };
   bool parseNumericVariable(StringRef &Expr, StringRef &Name, bool IsDefinition,
                             bool AcceptFail, const SourceMgr &SM) const;
   std::shared_ptr<FileCheckNumExprAST>
   parseNumericOperand(StringRef &Expr, enum AllowedOperand AO,
-                      const SourceMgr &SM) const;
+                      bool &MatchTimeKnown, const SourceMgr &SM) const;
   std::shared_ptr<FileCheckNumExprAST>
   parseFileCheckBinop(StringRef &Expr, std::shared_ptr<FileCheckNumExprAST> Opl,
-                      bool Legacy, const SourceMgr &SM) const;
+                      bool Legacy, bool &MatchTimeKnown,
+                      const SourceMgr &SM) const;
 };
 
 //===----------------------------------------------------------------------===//
 /// Summary of a FileCheck diagnostic.
 //===----------------------------------------------------------------------===//
 
 struct FileCheckDiag {
   /// What is the FileCheck directive for this diagnostic?

llvm/lib/Support/FileCheck.cpp

if (Op1.Signed) {
     // Op1 < Op2.
     if (Val1 < Val2)
       return FileCheckNumExprVal();
     return FileCheckNumExprVal(Val1 - Val2);
   }
 }
 
 /// Generic constructor for a numeric expression whose equality constraint is
-/// represented by \p AST and matching format is \p Fmt. If matching format is
-/// unset (ie. no explicit or implicit matching format), set it to default one
+/// represented by \p AST, whose matching format is \p Fmt and which is known
+/// at match time if \p MatchTimeKnown is true. If matching format is unset
+/// (i.e. no explicit or implicit matching format), set it to default one
 /// (unsigned decimal integer).
 FileCheckNumExpr::FileCheckNumExpr(std::shared_ptr<FileCheckNumExprAST> AST,
-                                   FileCheckNumExprFmt Fmt)
-    : AST(AST) {
+                                   FileCheckNumExprFmt Fmt, bool MatchTimeKnown)
+    : AST(AST), MatchTimeKnown(MatchTimeKnown) {
   if (Fmt.Valid)
     this->Fmt = Fmt;
   else
     this->Fmt = FmtUnsigned;
 }
 
 /// Evaluate the value of the expression represented by this AST. Therefore,
 /// return this variable's value or the value of its associated numeric
 /// expression if any.
 FileCheckNumExprVal FileCheckNumExprVar::eval() const {
   if (Value.isValid())
     return Value;
 
   assert(NumExpr != nullptr);
   if (NumExpr->getAST() != nullptr)
     return NumExpr->getAST()->eval();
 
   return FileCheckNumExprVal();
 }
 
 /// Return whether this variable's value is known at match time, when
 /// performing the substitutions.
 bool FileCheckNumExprVar::isMatchTimeKnown() const {
-  return NumExpr != nullptr && NumExpr->getAST() != nullptr;
+  return Value.isValid() || NumExpr->isMatchTimeKnown();
 }
 
 /// Return implicit format of this numeric variable.
 FileCheckNumExprFmt FileCheckNumExprVar::getImplicitFmt() const {
   if (NumExpr == nullptr)
     return FmtNone;
 
   return NumExpr->getEffectiveFmt();
 }
 
 /// Append numeric variable's name to UndefVarNames if undefined.
 void FileCheckNumExprVar::appendUndefVarNames(
     std::vector<StringRef> &UndefVarNames) const {
   if (!Value.isValid())
     UndefVarNames.emplace_back(Name);
 }
 
-/// Set value of this numeric variable if not defined. Return whether the
-/// variable was already defined.
-bool FileCheckNumExprVar::setValue(FileCheckNumExprVal Value) {
-  if (this->Value.isValid())
+/// Set value of this numeric variable if not defined. Value is set as a
+/// tentative value if \p Tentative is true. Return whether variable was
+/// already definitely (i.e. not tentatively) defined.
+bool FileCheckNumExprVar::setValue(FileCheckNumExprVal Value, bool Tentative) {
+  if (this->Value.isValid() && !this->Value.isTentative())
     return true;
+
+  assert(NumExpr != nullptr && "Setting value of parse time variable");
+  assert(!Value.isTentative() && "Setting value from tentative value");
   assert(Value.isValid() && "Setting invalid value");
+  if (Tentative)
+    Value.toggleTentative();
   this->Value = Value;
   return false;
 }
 
+/// Mark value as definitive (i.e. not tentative). Return whether value was
+/// already definitive.
+bool FileCheckNumExprVar::commitValue() {
+  // Value is not a valid tentative value.
+  if (!Value.isValid() || !Value.isTentative())
+    return true;
+
+  Value.toggleTentative();
+  return false;
+}
+
 /// Clear value of this numeric variable. Return whether value was already
 /// undefined.
 bool FileCheckNumExprVar::clearValue() {
   if (!Value.isValid())
     return true;
   Value = FileCheckNumExprVal();
   return false;
 }
FileCheckNumExprVal FileCheckASTBinop::eval() const {
   if (!LVal.isSigned() || !RVal.isSigned()) {
     LVal.convertUnsigned();
     RVal.convertUnsigned();
   }
 
   if (!LVal.isValid() || !RVal.isValid())
     return FileCheckNumExprVal();
 
-  return EvalBinop(LVal, RVal);
+  FileCheckNumExprVal Val = EvalBinop(LVal, RVal);
+
+  // Mark result as tentative if any of the operand was tentative. Assert
+  // result was not tentative beforehand to have that check only here.
+  assert((!Val.isValid() || !Val.isTentative()) &&
+         "Binary operation returned tentative value");
+  if (Val.isValid() && (LVal.isTentative() || RVal.isTentative()))
+    Val.toggleTentative();
+  return Val;
+}
+
+/// Verify that the matched value in \p MatchedValue satisfies the constraint
+/// expressed by this expression. Return true if constraint is not satisfied.
+bool FileCheckNumExpr::verifyConstraint(
+    FileCheckNumExprVal MatchedValue) const {
+  // Nothing to verify: numeric variable definition with empty numeric
+  // expression.
+  if (AST == nullptr)
+    return false;
+
+  // Will fail if evaluated value or MatchedValue are invalid.
+  return AST->eval() != MatchedValue;
 }
 
 /// Return implicit format of this AST, FmtConflict if implicit formats of the
 /// AST's components conflict and Fmt none if the AST has no implicit format
 /// (e.g. AST is made of a single literal).
 FileCheckNumExprFmt FileCheckASTBinop::getImplicitFmt() const {
   FileCheckNumExprFmt LFmt = Opl->getImplicitFmt();
   FileCheckNumExprFmt RFmt = Opr->getImplicitFmt();
if (IsDefinition) {
     // last definition of the variable before this use.
     auto git = Context->GlobalNumericVariableTable.find(Name);
     if (git == Context->GlobalNumericVariableTable.end()) {
       SM.PrintMessage(SMLoc::getFromPointer(Name.data()), SourceMgr::DK_Error,
                       "Using undefined numeric variable '" + Name + "'");
       return true;
     }
 
-    std::shared_ptr<FileCheckNumExprVar> NumVar = git->second;
-    if (!IsPseudo && NumVar->getDefLineNumber() == LineNumber &&
-        !NumVar->isMatchTimeKnown()) {
-      SM.PrintMessage(SMLoc::getFromPointer(Name.data()), SourceMgr::DK_Error,
-                      "Numeric variable '" + NumVar->getName() +
-                          "' defined on the same line");
-      return true;
-    }
-
     return false;
   }
 }
 
 /// Parse \p Expr for use of a numeric operand. Accept both literal values and
 /// numeric variables, depending on the value of \p AllowedOperandFlag. Return
 /// the class representing that operand in the AST of the numeric expression or
-/// nullptr if parsing fails in which case errors are reported on \p SM.
+/// nullptr if parsing fails in which case errors are reported on \p SM. Also
+/// return in MatchTimeKnown if the operand's value is known at match time.
 std::shared_ptr<FileCheckNumExprAST>
 FileCheckPattern::parseNumericOperand(StringRef &Expr, enum AllowedOperand AO,
+                                      bool &MatchTimeKnown,
                                       const SourceMgr &SM) const {
 
   // Try to parse as a numeric variable use.
   if (AO == LegacyVar || AO == Any) {
     StringRef Name;
     // Error reporting done in ParseNumericVariable.
     if (!parseNumericVariable(Expr, Name, false /*IsDefinition*/,
                               true /*AcceptFail*/, SM)) {
       std::shared_ptr<FileCheckNumExprVar> NumVar =
           Context->GlobalNumericVariableTable.find(Name)->second;
+      MatchTimeKnown =
+          NumVar->getDefLineNumber() < LineNumber || NumVar->isMatchTimeKnown();
       return NumVar;
     }
   }
 
   // Otherwise, parse it as a literal.
   if (AO != LegacyLiteral && AO != Any)
     return nullptr;
   unsigned Radix = (AO == LegacyLiteral) ? 10 : 0;
   int64_t SignedLiteralValue;
   uint64_t UnsignedLiteralValue;
   StringRef SaveExpr = Expr;
   // Accept both signed and unsigned literal.
   if (AO != LegacyLiteral && !Expr.consumeInteger(Radix, SignedLiteralValue)) {
+    MatchTimeKnown = true;
     return std::make_shared<FileCheckNumExprLiteral>(SignedLiteralValue);
   } else {
     Expr = SaveExpr;
     if (!Expr.consumeInteger(Radix, UnsignedLiteralValue)) {
+      MatchTimeKnown = true;
       return std::make_shared<FileCheckNumExprLiteral>(UnsignedLiteralValue);
     } else
       return nullptr;
   }
 }
 
-/// Parse \p Expr for a binary operation. The left operand of this binary
-/// operation is given in \p Opl and \p Legacy indicates whether we are parsing
-/// a legacy numeric expression. Return the class representing this binary
-/// operation in the AST of the numeric expression or nullptr if parsing fails
-/// in which case errors are reported on \p SM.
-std::shared_ptr<FileCheckNumExprAST>
-FileCheckPattern::parseFileCheckBinop(StringRef &Expr,
-                                      std::shared_ptr<FileCheckNumExprAST> Opl,
-                                      bool Legacy, const SourceMgr &SM) const {
+/// Parse \p Expr for a binary operation, as a legacy numeric expression if
+/// \p Legacy is true. The left operand of this binary operation is given in
+/// \p Opl with \p MatchTimeKnown holding whether the left operand's value is
+/// known at match time. Return the class representing this binary operation
+/// in the AST of the numeric expression or nullptr if parsing fails in which
+/// case errors are reported on \p SM. Also return in MatchTimeKnown if the
+/// binary operation can be evaluated at match time.
+std::shared_ptr<FileCheckNumExprAST> FileCheckPattern::parseFileCheckBinop(
+    StringRef &Expr, std::shared_ptr<FileCheckNumExprAST> Opl, bool Legacy,
+    bool &MatchTimeKnown, const SourceMgr &SM) const {
   skipWhitespace(Expr);
   if (Expr.empty())
     return Opl;
 
   // Check if this is a supported operation and selection function to perform
   // it.
   SMLoc oploc = SMLoc::getFromPointer(Expr.data());
   char Operator = next(Expr);
std::shared_ptr<FileCheckNumExprAST> FileCheckPattern::parseFileCheckBinop(
   skipWhitespace(Expr);
   if (Expr.empty()) {
     SM.PrintMessage(SMLoc::getFromPointer(Expr.data()), SourceMgr::DK_Error,
                     "Missing operand in numeric expression");
     return nullptr;
   }
   // Second operand in legacy numeric expression is a literal.
   enum AllowedOperand AO = Legacy ? LegacyLiteral : Any;
-  std::shared_ptr<FileCheckNumExprAST> Opr = parseNumericOperand(Expr, AO, SM);
+  bool OprMatchTimeKnown;
+  std::shared_ptr<FileCheckNumExprAST> Opr =
+      parseNumericOperand(Expr, AO, OprMatchTimeKnown, SM);
   if (Opr == nullptr) {
     SM.PrintMessage(SMLoc::getFromPointer(Expr.data()), SourceMgr::DK_Error,
                     "Invalid operand format '" + Expr + "'");
     return nullptr;
   }
   skipWhitespace(Expr);
   if (!Expr.empty()) {
     SM.PrintMessage(SMLoc::getFromPointer(Expr.data()), SourceMgr::DK_Error,
                     "Unexpected characters at end of numeric expression '" +
                         Expr + "'");
     return nullptr;
   }
 
+  MatchTimeKnown = MatchTimeKnown && OprMatchTimeKnown;
   return std::make_shared<FileCheckASTBinop>(EvalBinop, Opl, Opr);
 }
 
 /// Parse \p Expr for a numeric expression. Parameter \p Legacy indicates
 /// whether Expr should be a legacy numeric expression. Return the class
 /// representing the AST of numeric expression or nullptr if parsing fails
 /// in which case errors are reported on \p SM. Set \p NumVarDef to the
 /// pointer to the class representing the variable defined to this numeric
std::shared_ptr<FileCheckNumExpr> FileCheckPattern::parseNumericExpression(
   skipWhitespace(Expr);
   StringRef ConstraintExpr = Expr;
   if (skip(Expr, "==", true /*Optional*/))
     return nullptr;
 
   // Parse numeric expression itself.
   skipWhitespace(Expr);
   StringRef UseExpr = Expr;
+  bool MatchTimeKnown = false;
   if (Expr.empty()) {
     if (Expr.size() != ConstraintExpr.size()) {
       SM.PrintMessage(
           SMLoc::getFromPointer(Expr.data()), SourceMgr::DK_Error,
           "Empty numeric expression should not have a matching constraint");
       return nullptr;
     }
   } else {
     // First operand in legacy numeric expression is the @LINE pseudo variable.
     enum AllowedOperand AO = Legacy ? LegacyVar : Any;
-    NumExprAST = parseNumericOperand(Expr, AO, SM);
+    NumExprAST = parseNumericOperand(Expr, AO, MatchTimeKnown, SM);
     while (NumExprAST != nullptr && !Expr.empty()) {
-      NumExprAST = parseFileCheckBinop(Expr, NumExprAST, Legacy, SM);
+      NumExprAST =
+          parseFileCheckBinop(Expr, NumExprAST, Legacy, MatchTimeKnown, SM);
       // Legacy numeric expressions only allow 2 operands.
       if (Legacy && !Expr.empty()) {
         SM.PrintMessage(SMLoc::getFromPointer(Expr.data()), SourceMgr::DK_Error,
                         "Unexpected characters at end of numeric expression '" +
                             Expr + "'");
         return nullptr;
       }
     }
std::shared_ptr<FileCheckNumExpr> FileCheckPattern::parseNumericExpression(
   else if (ImplicitFmt.Conflict) {
     SM.PrintMessage(
         SMLoc::getFromPointer(UseExpr.data()), SourceMgr::DK_Error,
         "Variables with conflicting format specifier: need an explicit one");
     return nullptr;
   } else
     Fmt = ImplicitFmt;
 
-  auto NumExpr = std::make_shared<FileCheckNumExpr>(NumExprAST, Fmt);
+  auto NumExpr =
+      std::make_shared<FileCheckNumExpr>(NumExprAST, Fmt, MatchTimeKnown);
 
   // Parse numeric variable definition.
   if (!DefExpr.empty()) {
     skipWhitespace(DefExpr);
     StringRef Name;
     if (parseNumericVariable(DefExpr, Name, true /*IsDefinition*/,
                              false /*AcceptFail*/, SM))
       // Invalid variable definition. Error reporting done in parsing function.
if (PatternStr.startswith("[[")) {
         return true;
       }
 
       MatchStr = MatchStr.substr(0, End);
       PatternStr = PatternStr.substr(End + 4 + (int)IsNumExpr);
 
       bool IsVarDef;
       bool SubstNeeded;
+      bool CaptureNeeded;
       bool Legacy = false;
       StringRef DefName;
       StringRef SubstStr;
       StringRef MatchRegexp = StringRef();
       unsigned SubstInsertIdx = RegExStr.size();
       std::shared_ptr<FileCheckNumExprVar> NumVarDef;
       std::shared_ptr<FileCheckNumExpr> NumExpr;
 
if (PatternStr.startswith("[[")) {
           SM.PrintMessage(SMLoc::getFromPointer(MatchStr.data()),
                           SourceMgr::DK_Error, "Invalid variable name");
           return true;
         }
 
         StringRef Name = MatchStr.substr(0, TrailIdx);
         StringRef Trailer = MatchStr.substr(TrailIdx);
         size_t DefSepIdx = Trailer.find(":");
-        IsVarDef = (DefSepIdx != StringRef::npos);
+        CaptureNeeded = IsVarDef = (DefSepIdx != StringRef::npos);
         Legacy = IsNumExpr = IsPseudo;
         SubstNeeded = !IsVarDef;
 
         if (IsVarDef) {
           if ((IsPseudo || !Trailer.consume_front(":"))) {
             SM.PrintMessage(SMLoc::getFromPointer(MatchStr.data()),
                             SourceMgr::DK_Error,
                             "Invalid name in pattern variable definition");
if (PatternStr.startswith("[[")) {
 
       // Parse numeric expression.
       if (IsNumExpr) {
         NumExpr = parseNumericExpression(MatchStr, NumVarDef, Legacy, SM);
         if (NumExpr == nullptr)
           return true;
         IsNumExpr = true;
         IsVarDef = NumVarDef != nullptr;
-        SubstNeeded = NumExpr->getAST() != nullptr;
         if (IsVarDef)
           DefName = NumVarDef->getName();
+        SubstNeeded = NumExpr->isMatchTimeKnown();
+        CaptureNeeded = IsVarDef || !NumExpr->isMatchTimeKnown();
         if (SubstNeeded)
           SubstStr = MatchStr;
         else {
           FileCheckNumExprFmt Fmt = NumExpr->getEffectiveFmt();
           MatchRegexp = Fmt.getWildcardRegex();
         }
       }
 
-      // Handle variable definition: [[<def>:(...)]] and [[#(...)<def>:(...)]].
-      if (IsVarDef) {
+      // Handle variable definition: [[<def>:(...)]] and [[#(...)<def>:(...)]]
+      // and numeric expression needing verifying.
+      if (CaptureNeeded) {
         RegExStr += '(';
         ++SubstInsertIdx;
 
         if (IsNumExpr) {
-          struct FileCheckNumExprMatch NumExprDef = {NumVarDef, CurParen};
-          NumericVariableDefs[DefName] = NumExprDef;
+          struct FileCheckNumExprMatch NumExprDef = {NumExpr, NumVarDef,
+                                                     CurParen};
+          CapturedNumericExpressions.emplace_back(NumExprDef);
           // This store is done here rather than in Match() to allow
           // ParseNumericVariable() to get the pointer to the class instance
           // of the right variable definition corresponding to a given numeric
           // variable use.
           Context->GlobalNumericVariableTable[DefName] = NumVarDef;
         } else {
           VariableDefs[DefName] = CurParen;
           // Mark pattern variable as defined to detect collision between
           // pattern and numeric variable in ParseNumericVariable and
           // DefineCmdlineVariables when the latter is created later than the
           // former. We cannot reuse GlobalVariableTable for that by populating
           // it with an empty string since we would then loose the ability to
           // detect use of undefined variable in Match().
           Context->DefinedVariableTable[DefName] = true;
         }
 
         ++CurParen;
       }
 
       if (!MatchRegexp.empty() && AddRegExToRegEx(MatchRegexp, CurParen, SM))
         return true;
 
-      if (IsVarDef)
+      if (CaptureNeeded)
         RegExStr += ')';
 
       // Handle variable use: [[foo]] and [[#<foo expr>]].
       if (SubstNeeded) {
         // Handle use of pattern variables that were defined earlier on the
         // same line by emitting a backreference. Numeric expressions do not
         // support using a numeric variable defined on the same line.
         if (!IsNumExpr && VariableDefs.find(SubstStr) != VariableDefs.end()) {
size_t FileCheckPattern::match(StringRef Buffer, size_t &MatchLen,
   }
 
   // Regex match.
 
   // If there are variable uses, we need to create a temporary string with the
   // actual value.
   StringRef RegExToMatch = RegExStr;
   std::string TmpStr;
+
   if (!Substs.empty()) {
     TmpStr = RegExStr;
 
     unsigned InsertOffset = 0;
     // Substitute all pattern variables and numeric expressions whose value is
     // known just now. Use of pattern variables defined on the same line are
-    // handled by back-references.
+    // handled by back-references. For numeric expressions using variable
+    // with empty numeric expression defined on the same line, RegExStr is
+    // filled with a suitable wildcard pattern and the constraint is then
+    // verified against the matched value.
+    //
+    // Known issue: Due to the constraints for numeric expressions using
+    // variable defined in earlier line not being expressed at the regular
+    // expression level, the wrong values might be matched, leading to a
+    // constraint check failure. An example of this is what happens when
+    // checking the line "1 3 4" against the directive
+    // "CHECK: [[#N:]] [[#N+1]]" which gets substituted by
+    // "[[:digit:]]+ [[:digit:]]+". This would match 1 3 which then fails the
+    // constraint checks.
+    //
+    // One possible solution would be to augment the regular expression engine
+    // to be able to return all matches for a given pattern. FileCheck could
+    // then test all possibility and fail the check only if none of them
+    // satisfy all numeric expressions. This would work both for numeric
+    // expression using a numeric variable defined on the same line as well as
+    // numeric expressions with a comparison rather than equality constraint.
     for (const auto &Subst : Substs) {
       // Substitute and check for failure (e.g. use of undefined variable).
       llvm::Optional<std::string> OptValue = Subst.getSubstitute();
       if (!OptValue.hasValue()) {
         SM.PrintMessage(SMLoc::getFromPointer(Subst.getSubstString().data()),
                         SourceMgr::DK_Error,
                         "Unable to substitute variable or numeric expression");
         return StringRef::npos;
       }
       std::string Value = OptValue.getValue();
 
       // Plop it into the regex at the adjusted offset.
       TmpStr.insert(TmpStr.begin() + Subst.getIndex() + InsertOffset,
                     Value.begin(), Value.end());
       InsertOffset += Value.size();
     }
 
     // Match the newly constructed regex.
     RegExToMatch = TmpStr;
   }
 
+  bool AllNumExprSatisfied;
   SmallVector<StringRef, 4> MatchInfo;
+  do {
+    AllNumExprSatisfied = true;
     if (!Regex(RegExToMatch, Regex::Newline).match(Buffer, &MatchInfo))
       return StringRef::npos;
 
     // Successful regex match.
     assert(!MatchInfo.empty() && "Didn't get any match");
-  StringRef FullMatch = MatchInfo[0];
 
     // If this defines any pattern variables, remember their values.
     for (const auto &VariableDef : VariableDefs) {
       assert(VariableDef.second < MatchInfo.size() && "Internal paren error");
       Context->GlobalVariableTable[VariableDef.first] =
           MatchInfo[VariableDef.second];
     }
 
-  // If this defines any numeric variables, remember their values.
-  for (const auto &NumericVariableDef : NumericVariableDefs) {
-    assert(NumericVariableDef.second.CaptureParen < MatchInfo.size() &&
+    // Check the matched value satisfies the numeric expression constraint and
+    // store it in the numeric variable being defined if any.
+    for (const auto &CapturedNumericExpression : CapturedNumericExpressions) {
+      assert(CapturedNumericExpression.CaptureParen < MatchInfo.size() &&
              "Internal paren error");
-    unsigned CaptureParen = NumericVariableDef.second.CaptureParen;
-    std::shared_ptr<FileCheckNumExprVar> NumVarDef =
-        NumericVariableDef.second.NumVarDef;
+      unsigned CaptureParen = CapturedNumericExpression.CaptureParen;
+      StringRef MatchedStr = MatchInfo[CaptureParen];
+      FileCheckNumExpr *NumExpr = CapturedNumericExpression.NumExpr.get();
+      FileCheckNumExprVar *NumVarDef =
+          CapturedNumericExpression.NumVarDef.get();
 
       StringRef MatchedValue = MatchInfo[CaptureParen];
-    assert(NumVarDef->getNumExpr() != nullptr);
-    FileCheckNumExprFmt Fmt = NumVarDef->getNumExpr()->getEffectiveFmt();
+      FileCheckNumExprFmt Fmt = NumExpr->getEffectiveFmt();
       FileCheckNumExprVal Value = Fmt.valueFromStringRepr(MatchedValue);
+
+      // Get numeric value from the matched string according to matching format
+      // of the numeric expression. Error out if this fails.
       if (!Value.isValid()) {
-      SM.PrintMessage(SMLoc::getFromPointer(MatchedValue.data()),
-                      SourceMgr::DK_Error, "Unable to represent numeric value");
+        SM.PrintMessage(SMLoc::getFromPointer(MatchedStr.data()),
+                        SourceMgr::DK_Error,
+                        "Unable to represent numeric value " + MatchedStr);
+        return StringRef::npos;
       }
-    if (NumVarDef->setValue(Value))
-      assert(false && "Numeric variable redefined");
+
+      // Now verify the constraint of the numeric expression.
+      if (NumExpr->verifyConstraint(Value)) {
+        AllNumExprSatisfied = false;
+        // Try matching again from the line following the one matched. There
+        // might be other possible matches in the current line but we have no
+        // way to ask for them to the regular expression engine. However this
+        // hazard can be alleviated if the user write a CHECK pattern that uses
+        // numeric expression for all numeric values the line to match contains.
+        Buffer = Buffer.substr(Buffer.find_first_of('\n')).substr(1);
+        break;
+      }
+
+      // Store verified matched value. This will set variable defined by the
+      // expression if any.
+      if (NumVarDef != nullptr &&
+          NumVarDef->setValue(Value, true /*Tentative*/))
+        assert(false && "Value of numeric variable already set");
+    }
+  } while (!AllNumExprSatisfied && !Buffer.empty());
+
+  // All constraints satisfied, make variable definitions definitive.
+  if (AllNumExprSatisfied) {
+    for (const auto &CapturedNumericExpression : CapturedNumericExpressions) {
+      FileCheckNumExprVar *NumVarDef =
+          CapturedNumericExpression.NumVarDef.get();
+      if (NumVarDef != nullptr)
+        NumVarDef->commitValue();
     }
+    // No line with all constraints satisfied found, return failure to match.
+  } else
+    return StringRef::npos;
 
   // Like CHECK-NEXT, CHECK-EMPTY's match range is considered to start after
   // the required preceding newline, which is consumed by the pattern in the
   // case of CHECK-EMPTY but not CHECK-NEXT.
   size_t MatchStartSkip = CheckTy == Check::CheckEmpty;
+  StringRef FullMatch = MatchInfo[0];
   MatchLen = FullMatch.size() - MatchStartSkip;
   return FullMatch.data() - Buffer.data() + MatchStartSkip;
 }
 
 /// Computes an arbitrary estimate for the quality of matching this pattern at
 /// the start of \p Buffer; a distance of zero should correspond to a perfect
 /// match.
 unsigned FileCheckPattern::ComputeMatchDistance(StringRef Buffer) const {
unsigned FileCheckPattern::ComputeMatchDistance(StringRef Buffer) const {
   // Only compare up to the first line in the buffer, or the string size.
   StringRef BufferPrefix = Buffer.substr(0, ExampleString.size());
   BufferPrefix = BufferPrefix.split('\n').first;
   return BufferPrefix.edit_distance(ExampleString);
 }
 
 /// Print value of successful substitutions or name of undefined pattern or
 /// numeric variables preventing such a successful substitution.
+///
+/// Note: Numeric expression using numeric variables defined on the same line
+/// are not shown because we only have substitutions for numeric expressions
+/// whose value can be known at match time.
 void FileCheckPattern::PrintSubsts(const SourceMgr &SM, StringRef Buffer,
                                    SMRange MatchRange) const {
   // Print info what we know about substitutions. This covers both uses of
-  // pattern variables and numeric subsitutions.
+  // pattern variables and numeric subsitutions as long as they only use
+  // variables defined in earlier lines.
   if (!Substs.empty()) {
     for (const auto &Subst : Substs) {
       SmallString<256> Msg;
       raw_svector_ostream OS(Msg);
       bool IsNumExpr = Subst.isNumExpr();
       llvm::Optional<std::string> OptMatchedValue = Subst.getSubstitute();
 
       // Substitution failed or is not known at match time, print undefined
       // variables it uses.
       if (!OptMatchedValue.hasValue()) {
         std::vector<StringRef> UndefVarNames;
         Subst.getUndefVarNames(UndefVarNames);
         if (UndefVarNames.empty())
           continue;
         for (auto UndefVarName : UndefVarNames) {
           OS << "uses undefined variable \"";
           OS.write_escaped(UndefVarName) << "\"";
         }
       } else {
-        // Substitution succeeded. Print substituted value.
+        // Substitution succeeded and numeric expression satisfied its
+        // constraint. Print substituted value.
         std::string MatchedValue = OptMatchedValue.getValue();
         if (IsNumExpr)
           OS << "with numeric expression \"";
         else
           OS << "with variable \"";
         OS.write_escaped(Subst.getSubstString()) << "\" equal to \"";
         OS.write_escaped(MatchedValue) << "\"";
       }
if (CmdlineDef[0] == '#') {
       FileCheckNumExprVal Value = NumExpr->getAST()->eval();
       if (!Value.isValid()) {
         SM.PrintMessage(SMLoc::getFromPointer(CmdlineDefRef.data()),
                         SourceMgr::DK_Error,
                         "Unable to represent numeric value");
         ErrorFound = true;
         continue;
       }
-      NumVarDef->setValue(Value);
+      NumVarDef->setValue(Value, false /*Tentative*/);
 
       // Record this variable definition.
       GlobalNumericVariableTable[NumVarDef->getName()] = NumVarDef;
     } else {
       // Pattern variable definition
       std::string Prefix = "-D";
       std::string DiagCmdline = Prefix + CmdlineDef;
       std::unique_ptr<MemoryBuffer> CmdLine =

llvm/test/FileCheck/numeric-expression.txt

 ; RUN: not FileCheck -DPATVAR=foobar -check-prefixes CONFLICT,CONFLICT4 -input-file %s %s 2>&1 | FileCheck --strict-whitespace -check-prefix INPUT-NUM-CONFLICT %s
 ; RUN: not FileCheck -DPATVAR=foobar -D#PATVAR=42 -check-prefix CONFLICT -input-file %s %s 2>&1 | FileCheck --strict-whitespace -check-prefix CLI-NUM-CONFLICT %s
 ; RUN: not FileCheck -check-prefix DEF-EXPR-FAIL -input-file %s %s
 ; RUN: not FileCheck -check-prefixes CHECK,FMT-CONFLICT -input-file %s %s 2>&1 | FileCheck --strict-whitespace -check-prefix FMT-CONFLICT-MSG %s
 ; RUN: not FileCheck -check-prefix OVERFLOW -input-file %s %s 2>&1 | FileCheck -check-prefix OVERFLOW-MSG %s
 ; RUN: not FileCheck -check-prefix UNDERFLOW -input-file %s %s 2>&1 | FileCheck -check-prefix UNDERFLOW-MSG %s
 ; RUN: not FileCheck -check-prefix EMPTY-NUMEXPR-CONSTRAINT -input-file %s %s 2>&1 | FileCheck -check-prefix EMPTY-NUMEXPR-CONSTRAINT-MSG %s
 ; RUN: not FileCheck -check-prefix EMPTY-NUMDEF-CONSTRAINT -input-file %s %s 2>&1 | FileCheck -check-prefix EMPTY-NUMDEF-CONSTRAINT-MSG %s
+; RUN: not FileCheck -check-prefix SAME-LINE-USE-FAIL -input-file %s %s
 
 
 ; We ensure we attemt to match all lines with digits by using CHECK-NEXT
 ; directives for the checks to ensure each line of input is covered.
 
 
 ; Numeric variable definition with default matching format
 DEF DEFAULT FMT

 ; CHECK-NEXT: [[# %x, VAR1]]
 ; CHECK-NEXT: [[# %X, VAR1]]
 ; CHECK-NEXT: [[# %u, VAR3]]
 ; CHECK-NEXT: [[# %d, VAR3]]
 ; CHECK-NEXT: [[# %u, VAR4]]
 ; CHECK-NEXT: [[# %d, VAR4]]
 
 
+; Numeric expression with explicit and default matching format and implicit
+; matching rule using variables defined on same line
+USE SAME EXPL FMT IMPL MATCH
+11 10
+-11 C
+b c
+b 12 c
+C B
+B 10
+; CHECK-LABEL: USE SAME EXPL FMT IMPL MATCH
+; CHECK-NEXT: [[#VAR5:]] [[#%u,VAR5-1]]
+; CHECK-NEXT: [[#%d,VAR6:]] [[#%X,VAR6+23]]
+; CHECK-NEXT: [[#%x,VAR7:]] [[#%x,VAR7+1]]
+; CHECK-NEXT: [[#%x,VAR8:]] [[#%u,VAR8+1]] [[#%x,VAR8+1]]
+; CHECK-NEXT: [[#%X,VAR9:]] [[#%X,VAR9-1]]
+; CHECK-NEXT: [[#%X,VAR10:]] [[#%d,VAR10-1]]
+
+
+; Numeric expression with implicit matching format and implicit matching rule
+; using variables defined on same line
+USE SAME IMPL FMT IMPL MATCH
+11 10
+-11 -4
+b c
+b 12 c
+C B
+; CHECK-LABEL: USE SAME IMPL FMT IMPL MATCH
+; CHECK-NEXT: [[#VAR5:]] [[#VAR5-1]]
+; CHECK-NEXT: [[#%d,VAR6:]] [[#VAR6+7]]
+; CHECK-NEXT: [[#%x,VAR7:]] [[#VAR7+1]]
+; CHECK-NEXT: [[#%x,VAR8:]] [[#%u,VAR8+1]] [[#VAR8+1]]
+; CHECK-NEXT: [[#%X,VAR9:]] [[#VAR9-1]]
+
+
+; Numeric expression with implicit matching format and implicit matching rule
+; using variables defined on same line satisfying constraint is not the first
+; line encountered.
+REDEF SAME IMPL FMT IMPL MATCH NOT FIRST
+SAME LINE USE IMPL FMT IMPL MATCH NOT FIRST
+1 3
+1 2
+; CHECK-LABEL: SAME LINE USE IMPL FMT IMPL MATCH NOT FIRST
+; CHECK: [[#N:]] [[#N+1]]
+
+
+; Numeric expression with explicit matching format and implicit matching rule
+; using variables redefined on same line
+REDEF EXPL FMT IMPL MATCH
+21
+22 -31 -32
+AC/DC dc db
+; CHECK-LABEL: REDEF EXPL FMT IMPL MATCH
+; CHECK-NEXT: [[#VAR11:]]
+; CHECK-NEXT: [[#%u,VAR11+1]] [[#%d,VAR11:]] [[#%d,VAR11-1]]
+; CHECK-NEXT: [[#%X,VAR12:]]/[[#%X,VAR12+48]] [[#%x,VAR12:]] [[#%x,VAR12-1]]
+
+
+; Numeric expression with implicit matching format and implicit matching rule
+; using variables redefined on same line
+REDEF IMPL FMT IMPL MATCH
+21
+22 -31 -32
+AC/DC dc db
+; CHECK-LABEL: REDEF IMPL FMT IMPL MATCH
+; CHECK-NEXT: [[#VAR11:]]
+; CHECK-NEXT: [[#VAR11+1]] [[#%d,VAR11:]] [[#VAR11-1]]
+; CHECK-NEXT: [[#%X,VAR12:]]/[[#VAR12+0x30]] [[#%x,VAR12:]] [[#VAR12-1]]
+
+
 ; Numeric variable definition with unsupported matching format
 DEF INVALID FMT
 INVVAR1=a
 INVVAR2=11
 ; ERR-LABEL: DEF INVALID FMT
 ; INVALID-FMT-SPEC1-NEXT: INVVAR1=[[#%c,INVVAR1:]]
 ; INVALID-FMT-SPEC2-NEXT: INVVAR2=[[#%hhd,INVVAR2:]]
 ; INVALID-FMT-SPEC-MSG1: numeric-expression.txt:[[#@LINE-2]]:39: error: Invalid format specifier in numeric expression

 ; Empty numeric expression with matching constraint
 EMPTY NUMEXPR DEF WITH CONSTRAINT
 18
 ; EMPTY-NUMDEF-CONSTRAINT-LABEL: EMPTY NUMEXPR CONSTRAINT
 ; EMPTY-NUMDEF-CONSTRAINT-NEXT: [[#VARDEF: ==]]
 ; EMPTY-NUMDEF-CONSTRAINT-MSG: numeric-expression.txt:[[#@LINE-1]]:46: error: Empty numeric expression should not have a matching constraint
 ; EMPTY-NUMDEF-CONSTRAINT-MSG-NEXT: {{E}}MPTY-NUMDEF-CONSTRAINT-NEXT: {{\[\[#VARDEF: ==\]\]}}
 ; EMPTY-NUMDEF-CONSTRAINT-MSG-NEXT: {{^                                             \^$}}
+
+
+; Numeric expression with implicit matching format and implicit matching rule
+; using variables defined on same line satisfying constraint is not the first
+; line encountered.
+SAME LINE USE IMPL FMT IMPL MATCH FAIL CONSTRAINT
+1 3
+1 4
+; SAME-LINE-USE-FAIL-LABEL: SAME LINE USE IMPL FMT IMPL MATCH FAIL CONSTRAINT
+; SAME-LINE-USE-FAIL-NEXT: [[#N:]] [[#N+1]]

llvm/test/FileCheck/verbose.txt

 VV-NEXT: verbose.txt:[[@LINE-17]]:12: remark: {{C}}HECK-NOT: excluded string not found in input
 VV-NEXT: {{C}}HECK-NOT: raboof{{$}}
 VV-NEXT:  {{^           \^$}}
 VV-NEXT: verbose.txt:[[@LINE-22]]:1: note: scanning from here
 VV-NEXT: {{^}}bar{{$}}
 VV-NEXT: {{^}}^{{$}}
 
 NUMVAR=42
-NUMVAR - 1:41
+NUMVAR:42 NUMVAR=12 NUMVAR:12
 CHECK: NUMVAR=[[#NUMVAR:]]
 CHECK-NOT: [[#NUMVAR + 1]]
-CHECK-NEXT: NUMVAR - 1:[[#NUMVAR - 1]]
+CHECK-NEXT: NUMVAR:[[#NUMVAR]] NUMVAR=[[#NUMVAR:]] NUMVAR:[[#NUMVAR]]
 
 V:      verbose.txt:[[#@LINE-4]]:8: remark: {{C}}HECK: expected string found in input
 V-NEXT: {{C}}HECK: {{NUMVAR=[[][[]#NUMVAR:[]][]]$}}
 V-NEXT: {{^       \^$}}
 V-NEXT: verbose.txt:[[#@LINE-9]]:1: note: found here
 V-NEXT: {{^}}NUMVAR=42{{$}}
 V-NEXT: {{^}}^~~~~~~~~{{$}}
 
 V-NEXT: verbose.txt:[[#@LINE-9]]:13: remark: {{C}}HECK-NEXT: expected string found in input
-V-NEXT: {{C}}HECK-NEXT: {{NUMVAR - 1:[[][[]#NUMVAR - 1[]][]]$}}
-V-NEXT: {{^            \^$}}
-V-NEXT: verbose.txt:[[#@LINE-15]]:1: note: found here
-V-NEXT: {{^}}NUMVAR - 1:41{{$}}
-V-NEXT: {{^}}^~~~~~~~~~~~~{{$}}
-V-NEXT: verbose.txt:[[#@LINE-18]]:1: note: with numeric expression "NUMVAR - 1" equal to "41"
-V-NEXT: {{^}}NUMVAR - 1:41{{$}}
-V-NEXT: {{^}}^~~~~~~~~~~~~{{$}}
+V-NEXT: {{C}}HECK-NEXT: {{NUMVAR:[[][[]#NUMVAR[]][]] NUMVAR=[[][[]#NUMVAR:[]][]] NUMVAR:[[][[]#NUMVAR[]][]]$}}
+V-NEXT:  {{^            \^$}}
+V-NEXT: verbose.txt:[[#VARLIN1:@LINE-15]]:1: note: found here
+V-NEXT: {{^}}NUMVAR:42 NUMVAR=12 NUMVAR:12{{$}}
+V-NEXT: {{^}}^~~~~~~~~~~~~~~~~~~~~~~~~~~~~{{$}}
+V-NEXT: verbose.txt:[[#VARLIN1]]:1: note: with numeric expression "NUMVAR" equal to "42"
+V-NEXT: {{^}}NUMVAR:42 NUMVAR=12 NUMVAR:12{{$}}
+V-NEXT: {{^}}^~~~~~~~~~~~~~~~~~~~~~~~~~~~~{{$}}
 
 VV-NEXT: verbose.txt:[[#@LINE-20]]:12: remark: {{C}}HECK-NOT: excluded string not found in input
 VV-NEXT: {{C}}HECK-NOT: {{[[][[]#NUMVAR [+] 1[]][]]$}}
 VV-NEXT:  {{^           \^$}}
 VV-NEXT: verbose.txt:[[#@LINE-25]]:1: note: scanning from here
-VV-NEXT: {{^}}NUMVAR - 1:41{{$}}
+VV-NEXT: {{^}}NUMVAR:42 NUMVAR=12 NUMVAR:12{{$}}
 VV-NEXT: {{^}}^{{$}}
 
 before empty
 
 after empty
 CHECK: before empty
 CHECK-EMPTY:
 CHECK-NEXT: after empty