given string contains a substring that matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form. <del> The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf- jsonpath-iregexp. </del> Its first argument is a string that is searched for at least one substring that matches the iregexp contained in the string that is the second argument.  The result is "true" if such a substring exists, "false" otherwise.  2.6.7. 2.7.

Also, I noticed a minor read-flow issue and reorganized a paragraph to match the other section.
I think this relates to . An expression boolean (e.g. the result of ) is not a JSON value. This is something that we decided long ago. However Table 13 describes it as " or ". This is wrong. If we allow to be a subtype of , then paths like and will be valid and produce no parsing errors. In (see also ), we decided that this wouldn't be the case. only operates as a boolean, and only operates as a value. and MUST be distinct in order for this to work. This issue is about the role a function plays within an expression. That role is determined by its return value.
Since we don't allow as a function argument, I don't think has a type.
I don't see how that's relevant. I'm not talking about argument types. I'm talking about return types and the role of functions within expressions. returns an expression boolean (not a JSON boolean), which is valid where booleans are valid in expressions, i.e. not in comparisons. returns a JSON-like value, which is valid where values are valid in expressions, i.e. only in comparisons.
This distinction is implied because of the decisions that we've already made in the issue and PR linked above that we aren't going to support type coercion that a JSON- is distinct from expression-true
From : If we allow to be a subtype of , then the above decisions are contradicted.
In the interim we discussed that a function can return one of three kinds of data: a value (e.g. ) a boolean (e.g. ) a nodelist (e.g. a hypothetical function) Value functions can only be used in a comparison (i.e. as a ). This wasn't disputed. I argue that boolean functions can only be used as operands in a logical expression. Effectively they are . The value that they return cannot be translated/lowered/casted/converted/coerced into the JSON literals and and so they are not . (This discussion prompted , as linked above.) We don't have any functions that return a nodelist currently. The hypothetical function (or something like it) that takes a nodelist and returns that nodelist with duplicates removed was proposed during the interim. A use case could be for finding elements with more than x distinct values, like . The key here, though, is that this would only ever be used as: an argument for another function (as above), which isn't part of this discussion a , which is the same as the boolean return scenario (because nodelists aren't ). The role of this return type is the same as a boolean return type. What I want to highlight from this discussion is that a boolean isn't a value, so Table 13 is wrong when it lists as a subtype for as a subtype for See also URL There was an interesting scenario that was mentioned in the interim: . (This may need to be moved to another issue, but it came up during this discussion, so it's here. For now, I just want to explore the use case.) This finds all elements where contains the correct result of whether matches , for example in this data: (The inverse is arguably more useful, but that aside.) Importantly, this is only possible if the result of can be translated/lowered/casted/converted/coerced into the JSON literals and to be considered as . (Again, see .) I can see a use case for something similar in the test suite actually. Let's assume we have an function that takes a path and some data and returns the nodelist result of evaluting the path against the data. (We'll disregard any security or other practical issues of doing that for this purpose of this example.) The test suite is represented as an object with a single key that contains an array of scenarios. Each scenario looks something like this: Now suppose we wanted to query this document for all of the test cases which are incorrect. We could use this path: Ideally, this would return an empty nodelist for a compliant implementation. Something like this would be a great CI sanity check for the test suite. (This exact path would imply that nodelists are , which they're not, but we could work around that by introducing a function or something. This problem doesn't apply to the example since that's attempting to compare things that are more "boolean"-y.)
I'm not sure what "it" is referring to, but Table 13 says that has abstract instances and . Table 13 doesn't mention expression booleans.
You can't use Table 13 in your argument. The basis of my claim is that the table is wrong. "It" in my comment is the "expression boolean," or what we started calling TestBoolean.
I am trying to understand the second paragraph in the of this issue. After substituting "expression boolean" for "it", the paragraph reads: I don't understand how Table 13 says anything at all about expression booleans. Please can you explain?
... pedantry... okay. It should be I'm saying is what we're now calling TestBoolean, which is not a JSON value.
I disagree that represents a "test boolean". The spec makes this clear: The tricky thing, which is what I suspect we should be focussing on, is in the following: >A test expression either tests the existence of a node designated by an embedded query (see ) or tests the result of a function expression (see ). In the latter case, if the function expression is of type or one of its subtypes, it tests whether the result is true; [...] Note that the special case in this paragraph only applies where the return type of the function expression is defined to be (or one of its subtypes). The special case is not triggered by a return value being a boolean value (such as ). In other words, it can be determined before the query is executed.
I think this is an important point -- while JSON values are dynamically typed (there aren't really JSON types anyway), the OptionalBooleanType is statically confined to have three members: Nothing, Value(false), Value(true). So it is compatible with an OptionalValueType recipient, and with being used in a comparable. It is also compatible with being used in a test-expr.
(And the point is that the function expression type system is statically typed.)
You keep using the spec as evidence. I'm arguing that the spec is wrong. You can't use it as evidence. I'm saying that the spec shouldn't define this way. How is compatible with if it represents a JSON value? This is the argument in . You're saying that a value of (//) can be used in a . This means that can be used in a because is , which implies is a valid expression. We've already decided that this is an invalid expression, as linked above, so we have a contradiction. The only way to resolve this contradiction is to make (and by extension ) akin to 's TestBoolean, not the JSON values and . Taking this further, cannot be a subtype of and by extension cannot be a subtype of . These are disjoint and operate under different roles in expressions. Additionally, if truly does represent JSON and , why make a type for it and not make and ? This definition is an incomplete type system. Further, why make the type system stop at functions? We've defined these types for functions, but then they don't extend into the expressions in which they exist, when it's clearly beneficial to do so (even if it requires more work).
Hi Greg, we now know that you want TestBooleans to be distinct from JSON value Booleans. The only argument that you present is that you want them to be different. This is fine; intuition often is useful in designing things. However, other people have the inverse intuition, so we won't get around discussing this with more technical arguments. The purpose of the type system we have been designing is not to replace or augment the JSON type system (which doesn't exist, I'd argue, but that is a different discussion). The purpose is to be able to statically check how functions fit together, and how arguments for functions can be derived from JSONPath filter expressions and how returns from functions can feed back into JSONPath filter expressions. It turns out we have three places where function expressions fit: In a test-expr, in a filter-path, and in a comparable. We wouldn't need to list filter-path here; for some reason we have made this one of the two things that can go into a function argument, along with comparable. This anomaly in the grammar can probably be fixed. The current definition of the function expression type system is clouded by the fact that there is both a subtype relationship and a compatibility relationship (more specifically, a type can be used in place of another type, which involves coercion of the value). We care about: nodelists, which can also stand in for a test (by checking whether the nodelist is empty) single nodes (which almost always need to be optional), which can stand in for a value values, which can also be used as comparable; these are often, but not always Optional. Functions like match or search return a special kind of optional value, which can be used in test expressions, and as a comparable, and as an argument for a function that expects just such a an instance and/or a JSONValue . You don't want them to be used in comparables or as function arguments where a JSONValue can be used. Why. (Note that you can't argue from the grammar, as that doesn't express the function argument/return value type system. Of course, you can ask why literal true and literal false cannot be used in a logical expression, and I'd say that this is a little wart, but it also isn't particularly useful.)
No, I've presented quite a lot of logical evidence that shows they MUST be different. What we have now is rife with contradiction. I agree with this stated purpose. We have failed to fulfil this purpose. What we have failed to do is identify when it's appropriate for a function to appear in each of these places. An individual function needs to be identified as valid as either or (XOR if you prefer) , but it can't be valid as both. My evidence is , which (as currently defined) is capable of returning JSON or . These are s. Everyone is saying that can be used as a , but JSON and cannot be used as . (That's the contradiction.) Because has the potential to return these JSON values, the function MUST be restricted to locations where those values are valid, namely . This would mean that MUST be invalid. If returns here, it reduces to , which we have declared invalid in in order to resolve an ambiguity. How are you not seeing this very obvious contradiction?
The solution to making valid is to make it return what calls a "TestBoolean," optional or not. But that means it's no longer valid as a and or even become invalid.
Note that the first half of this sentence is about the type system, and the second appears to be about the grammar (which just doesn't allow literals as test-expr). I tried to clean up the grammar (without changing it) over in so that may become more obvious.
The grammar and the type system need to align.
That is a desirable, but there are some other desirables in conflict with that. The grammar can very well disallow confusing expressions that do not violate the type system.
Like what? What desirable can be so important that it overrides a logical contradiction?
I'm saying the type system is allowing something that the grammar disallows.
What is obvious to you is far from obvious to me. My starting point is that the aspect of the spec being scrutinised by this issue is actually consistent and maybe just needs tweaking or explaining better. I've to point out what I believe to be the root cause of the confusion. (I am now essentially AFK for a while, so apologies for any delayed responses.)
The underlying requirement is that we want to convert Nodelists into Booleans when used in a test-expr (true if non-empty). We want to convert OptionalNodes into OptionalValues (by looking up the JSON value) when used in a comparable. We somehow need to map this into the function-expr type system as well. Since a function-argument says whether it is a nodelist or a value, this transfers one to one. Smearing up the type system by equating subtyping with coercion does not help, though. I would prefer to have the conversion from a Nodelist to an OptionalNode be explicit (possibly supported by a function that does just that). Then we don't have an unclear situation when a Nodelist is offered as an argument to a function that declares this as a Boolean.
NAME / NAME I think I have a good explanation of the problem. In this post, I will be using angle brackets to denote a nodelist to distinguish it from a JSON array and (code formatting) to indicate JSON literal values true and false (italics) to indicate the operands of a logical operation I would also like to use a couple definitions for expressions: A comparison context is any that involves a comparison operator, like and . A logical context is any that either can be used as the final result of an expression or involves a logical operator, , , or . These are the only two contexts which exists in our expressions. Further, it's important to recognize that the result of a comparison is used in a logical context. We decided that in a logical context was to be interpreted as an existence test. This means that if an item had an property, regardless of its value, the result of would be a logical true to select both items in To illustrate why, consider an implementation that interprets the value at as the result of the expression. It would consider the JSON as logical true and select the first item, but it would consider JSON as logical false and not select the second. With this behavior, there is be no way to select the node with , so we decided this was to always be interpreted as an existence test. This showed that JSON values, even and , only have meaning in a comparison context. Thus we made a distinction between the JSON literals and and the logical values true and false. We then codified this decision by making JSON literals (, , ) and other raw JSON values (strings, numbers) only valid in a comparison context using the ABNF. These functions are defined to return , which is defined by Table 13 to be the JSON literals and and the value . Because they return JSON values, and following from the decisions above, they can only be used in a comparison context. That is, they must be compared to another JSON value using a comparison operator to yield a logical result; they cannot be used directly in a logical context. But we also want these functions to be valid in both a comparison context and a logical context. This is the contradiction. They way they're defined, they cannot be used in both contexts. My original suggestion was to have the functions instead return a logical true or false instead of the JSON values. However, this only reverses the contraction rather than solving it. This weekend, I thought about how to define them so that they work in both contexts. I started by analyzing a construct that already does: a path. can exist in both a comparison context and a logical context, and it always returns a nodelist. How that nodelist is interpreted in each context determines the different behaviors it exhibits. To illustrate, let's take a look at what happens for in a comparison context with (1) and in a logical context (2) when evaluating Most importantly, always returns and , respectively, for these items, regardless of the context. In a comparison context, the single value is extracted from and -compared with , and the node is selected. is converted to and -compared with , and the node is not selected. In a logical context, contains a node, so it evaluates to logical true, and the node is selected. contains no nodes, so it evaluates to logical false, and the node is not selected. returns the same nodelist for both contexts because it's unaware of that context. It's just a path evaluating data. While always returns the same nodelist, how that nodelist is interpreted changes with context. Specifically, in a comparison context, is interpreted as JSON . in a logical context, is interpreted as logical true. In order to make these functions valid in both contexts, they need to return something that is valid in both contexts, and the only thing we have that works in both contexts is a nodelist, as demonstrated by . Let's perform our analysis again for the comparison context with (1) and the logical context with (2). Theoretically, these should select the same nodes, namely nodes that DO NOT match. Starting with the affirmative case where the function identifies a match, let's say that returns . In a comparison context, the single value is extracted from and -compared with , and the node is not selected. In a logical context, has nodes so is evaluated to logical true and negated by , and the node is not selected. Good so far. Both paths return the same thing for cases where a match is found. Now, let's look at the negative case where the function does not identify a match. For this, let's say that the function returns . In a comparison context, the single value is extracted from and -compared with , and the node is selected. In a logical context, has nodes so is evaluated to logical true and negated by , and the node is NOT selected. Uh, oh. We have differing results. This doesn't work. Let's try returning for the negative case instead. In a comparison context, is converted to and -compared with , and the node is NOT selected. In a logical context, has no nodes so is evaluated to logical false and negated by , and the node is selected. Hm... that doesn't work either. In fact, there is NO SINGLE VALUE that can return to make these paths select the same nodes for the non-matching case. The only recourse, then is to restrict to only one of: comparison context: logical context: and can't support both. Doing this would mean that when appears outside of its decided context, it will not be considered well-formed, requiring a parsing error (or compilation error, if you prefer). Note that doesn't preclude some other function from returning a nodelist in a manner that is consistent between contexts.
NAME thanks for the detailed analysis. The current spec defines a special case which produces different behaviour. Section 2.5.5 says (emphasis added): So, when a function returning an (or one of its subtypes) is used in a test expression, this is not treated as an existence test. Instead, the value returned by the function is compared against . So, for the three possible return values, yields true whereas and yield false. On the other hand, when a function returning an (or one of its subtypes) is used in a comparison, the value returned by the function is compared to the other side of the comparison. Thus and produce identical nodelists. The crucial thing to note, and which avoids the above contradiction, is that a function returning an (or one of its subtypes) is treated differently depending on where it is used. This difference is determined at parse/compile time. During execution, the return value of the function is treated accordingly.
I appreciate that the spec says this explicitly. However, I maintain that this special case needs to be removed. This contextual difference in behavior represents an inconsistency in the overall grammar of expressions. Why are functions treated specially? The underlying mechanics should operate the same, and they don't. The spec very explicitly says that a JSON / is not valid in a logical context... unless it comes from a function. Why?! Why special-case functions? A function (or any expression component) should only be valid where its return type is valid, that is, where a value of that return type is valid. I should be able to replace the function with a value of its return type and still have a valid expression. But if I do that for , I get which is explicitly invalid. Therefore we have an inconsistent grammar. That statement from the spec you quoted is merely a bandage trying to cover up a more serious problem. In order to have a consistent grammar, this sort of value substitution MUST work. Finally, as the only person who has even attempted to implement this, I find it confusing and difficult, which should be more than enough reason to change it.
We don’t agree. They are not treated specially. It is just a case that only applies to functions. No. It doesn’t provide a way to put JSON literals there, so the issue only occurs for function returns. (Because it is the only case.) (If talking about the type system:). Yes. Maybe we should say “well-typed” instead of “valid”... Where you can’t notate that value, this is moot. It is not invalid, it is not well-formed. I don’t follow. You could say that. The original problem is that NAME is not saying whether it talks about the node(s) or about the value(s) there. We have made sure this is well-defined in all cases: in a test-expr, it means (the existence) of nodes, and in a comparison-expr (where we restrict the paths to singular ones) it means the value (or Nothing if no node). Function expressions straddle this boundary, so something needed to be done. Hence the type system, which is not about values. I don’t follow. Maybe we can learn from this exercise that a mental model that conflates values and types doesn’t work too well here. Grüße, Carsten
THAT'S WHAT A SPECIAL CASE IS! A special case is precisely a case that only applies to one thing. Precisely! JSON literals can't go there because the values are invalid there. That's not a reason. You're saying functions are special-cased because they're special cases. They shouldn't be. They should be treated like everything else in the expression. Special casing yields an inconsistent grammar. It's inconsistent because the same grammar can behave differently. and have the same grammar: and both act in a logical capacity (i.e. ). They should behave the same: return a nodelist which is analyzed for contents. But they don't; they are inconsistent in their behavior. There's no reason for them to behave differently except that the spec says that they do. When evaluating an expression such as , I must first analyze the components, , , and . From those, I get values which can be operated on by and . Those values MUST be of the correct type for those operators. The operators, though, have no knowledge of what produced the values. But you're saying that needs to know if is a function or a path, and depending on which, it changes its behavior. That's inconsistent behavior. should do one thing, and it should do that thing without context. For the purpose of this discussion, "valid" means that it doesn't produce an error and can be evaluated. I'm not making a distinction between being ABNF-valid and well-formed. I don't understand what you mean by "notating" the value. The example I gave shows that what would be considered a valid expression () is actually invalid when you consider the return type . The problem is that, within a single context, you're defining one behavior for a path and another for a function. Functions should behave the same as paths in the same contexts. The spec disagrees with you: "A type is a set of instances." The type system is very much about values. (Note that the spec doesn't define "instance." This section is the only place the word is used. This is the example I gave. always finds a match. According to the spec, it should return JSON . Thus the expectation is that will return a nodelist containing all of the nodes of the subject data. Because the return value is always the same, I should be able to substitute the return value into the expression and yield another valid (and well-formed) expression that yields the same result. For example, in , I can substitue a logical true (not JSON ) for and get . Performing that substitution on yields (that's the JSON literal ). This is an invalid expression. It produces an error. Becuase and do not produce the same outcome, we have an inconsistency. It follows that if we are to have a consistent grammar and is invalid (explicitly via the ABNF), must also be invalid. This kind of substitution exists in every programming language and every mathematical convention I've ever seen. It is the very definition of logical consistency. It is foolish of us to define JSON Path expressions in a way that operates in a different manner. As mentioned, I'm not conflating values and types. The spec clearly defines a type as a set of instances/values. I understand the difference. To accuse me of misunderstanding this is rude, especially knowing that I come from .Net and C#, which are very strictly typed. You're very clearly holding onto keeping this for some reason despite the many logical arguments I've made against it. I don't understand why. Is it pride? Is it unwillingness to change? Is it laziness? For the benefit of the specification and its wide adoption, I urge you to actually consider my arguments and remove this inconsistency from the spec.
these are the two options I present. I don't see a way to have a consistent typing system and allow and in both logical and comparative contexts. The other option I can see is just doing away with the typing system altogether and just checking syntax (and maybe argument count). Then functions can return whatever they like and the expression evaluation system will just run. If it encounters something invalid, then it doesn't select that node, just like it would if returned a string in .
NAME you have refused to acknowledge that there is anything wrong with the current solution of special-casing function in test expressions, and yet you have also failed to explain why such a special case is acceptable. The only argument you have presented is to quote the spec and say, "The document provides a solution for this." As I'm arguing that the document (and therefore the solution) is wrong, the document cannot be used as evidence. There is a logical and syntactical inconsistency in the solution that the document presents ("in the document"). I have repeatedly offered multiple analyses, explanations, logical arguments, and examples detailing why the current solution is inconsistent and contradictory. Yet it seems you're not reading them. You have not in any way refuted my objections with why my analyses are incorrect or how the current solution is consistent and non-contradictory (I re-read the issue to be sure), yet you continue to oppose me. I ask you to provide such reasoning or stand down and accept one of my proposals.
Yes. Of course. I had problems responding to your critique because it doesn't seem to me to be aligned with the document. So the logical next step is to improve the document editorially, and then see whether your critique still applies. Unfortunately, I don't have much time this week, as I'm part of a small group that teaches an intensive course.. I can't really ask the others to do my work while I tend to this. So I ask for a little more patience.
Co-chair hat on. Clearly we are having a communication problem here. Greg is making two claims: The language/meaning of the spec contains unacceptable special-casing and/or inconsistency The current version of the spec is unreasonably hard to implement. On (1.) he has failed to convince the editors; Carsten has suggested that there is room for editorial improvements to clear the air. My notes: I'm inclined to give the editors a chance to see if they can reduce the discomfort and find us a path forward. I confess that after repeated readings of the spec and the discussion, I'm finding it hard to figure out what is what. In particular I find the language describing the type system to be really opaque. I see no particular reason why the input type repertoire and output type repertoire for functions need to be identical. See my other issue about the return values for and . In particular I suspect that strictly limiting the output type repertoire might simplify the lives of spec readers and implementors too. Options such as limiting return types to JSON primitive types (, , number, string, ) or alternately a strictly-constrained set of node-lists may be worth considering? I'm probably more concerned about Greg's concern (2.), that the current spec is hard to implement. I have painful personal experience with beautifully crafted specs that are unreasonably hard to code up. Greg, perhaps a little more on the difficulty would be useful? It does seem to me that any implementation is going to have some special-casing around the functions, since they are a different kind of beast than the rest of the spec. If we are unable to come to rough consensus on how to specify the function-extension framework, one plausible path forward is simply to remove it. I think that would be well within the scope of our charter.
NAME thank you for taking the time to review this issue. I've been asking for this and haven't received anthing except "the spec says you're wrong." (See below) I agree with your statement, but aligning function input with function output is not what this discussion is about. This is about how functions should only be valid where values of their return types would be valid. E.g. A function which returns a JSON or should only be valid where a JSON or literal would be valid. This makes for a consistent type system. For the current document, this is not the case. I've tried to explain it above by describing how typed systems should work, but maybe I can rewrite my arguments from an implementation point of view. I'll work on this and post back. I might be happy removing the type system (I still need to explore that), but I think we need to keep functions. And that means that we still need to address what happens when a function appears in a place it's not (shouldn't be) expected, like a function that returns a JSON or appearing as an operand for when JSON or themselves cannot appear there. As mentioned in your , we need functions to fill in the gaps for "traditional" syntaxes that don't fit within the spec's syntax. You're right, though: removing functions would make this issue moot, but we'd also lose some of the functionality that users are used to when working with JSON Path. To address the general response that I've received, I need to remove JSON Path from the argument and make the dispute absurd. Suppose we have the image below, and we want to write a document that describes how to reproduce this image. ! There are two scenarios I want to explore: The document says, "The sky is red." I see this and open an issue saying that the specification is wrong, that it should say that the sky is blue. In this scenario, I am correct in saying the spec is wrong. The document says, "The sky is blue," and I open an issue saying that the specification is wrong and suggest that it should say that the sky is red. In this scenario, I am incorrect. In both cases, I lay out an argument. And in both cases, the response should be, "Let's check the source image," not "No, the spec says that the sky is [color]. It's fine the way it is." We check the source image, and the issue is resolved. My point is that if I'm saying the spec is wrong, the spec can't be used as evidence to say that the spec is right. The evidence has to come from an external source.
The editors did not have much time in the last week, so that's why you didn't get a detailed explanation. Responding to each of these reminders doesn't increase the speed at which I can work. I'll submit a PR with an improved description of function expressions that should make the whole thing moot. But I can't do this in zero time. I do have empathy for your impatience. We can choose to do so, we can choose not to do so. The only reason you have given so far is that you want that to be the case. I believe that allowing JSON literals in the test-expr grammar would be very confusing. But we could. It just isn't needed. It is indeed hard to introduce a useful function extension without a type system. The requirement on the type system I'm trying to fulfill is that it meshes with the properties of the non-extended parts of JSONPath, in particular that we can examine well-formed expressions whether they also are well-typed, and that this can be done independently of the actual JSON data that will be fed to the expression. This works best with a type system that is entirely static. Yes. No. Again, that equivalence is a preference from you that I don't share. I'm not discussing your blue sky, because we do agree that you do have that preference. We just don't agree about that preference. If you have a little time left, can you please check ? I don't think Glyn is getting to this.
Understood. I await your detailed comments. There was no such reminder until recently. No, I have given several very detailed logically reasoned explanations backing up my case. Yes, it would. I'm not advocating for this. I'm advocating that because "allowing JSON literals in the test-expr grammar would be very confusing," we also shouldn't allow functions that return JSON literals in test-expr grammar. This works best with a type system that is entirely static. I 100% agree with this. This is not a preference. It is an aspect of typed systems in general. Functions may only appear where their return values may appear.
I'm AFK until Tuesday, but I wonder if Greg's concerns would be addressed if it was possible to support referential transparency where a function could be replaced, per call (tricky to implement in general), with (a concrete representation of) its return value without changing the result of the query.
(1) We would need literals for the whole type system. We can't notate Nothing or other node lists at the moment. (2) We would need to tag the literal with the function return type that is intended.
Not sure we would need all those literals. Suppose there was a node whose value was the same as the return value of a function. One referential transparency rule would be that replacing the function call with the path results in the same query result. But I take your point about possibly needing to tag values with types. This makes me wonder how we position the type system as a natural extension of the syntactic rules for non-function expressions.
I agree that we don't need representation of all values in the syntax. For example, JSON arrays and objects are values that are not allowed in the syntax. Here's how I see the type landscape. We have three types: values, boolean, and nodelists. Values include any and all JSON values: arrays, objects, strings, numbers, and the three literals , , and . We don't need any more precise typing. These can all be considered "values." Also included as a value is , which we designated to represent the absence of a value, e.g. the value of in the object . Strings, numbers, and the three JSON literals can be expressed in the syntax. JSON objects, JSON arrays, and have no representation in the syntax. Booleans are the result of any operation, e.g. , , or . Expressions in their entirety must also result in a boolean. There are two states (I'll use "state" to not confuse it with "value") for a boolean: true and false. The relation operators, and et al., take two values (from above) as arguments and return a boolean. The logical operators, , , and , take two (one for ) booleans as arguments and return a boolean. There is no relationship between the boolean states true and false and the JSON literals and . There is no representation for booleans in the syntax. Nodelists result from path evaluation. There is no representation for nodelists in the syntax. A nodelist is convertible to a Value if it has at most a single node. If a nodelist contains a node, the result is the value of that node. If a nodelist contains no nodes, the resulting value is . To enable syntax checking for paths in expressions, we have defined "Singular Path" as an identifiable syntax to ensure that a path can return at most one node and thus may be converted to a Value. A nodelist is always convertible to a Boolean. The resulting state is whether it contains nodes: true if it contains nodes false otherwise Setting up the type system this way covers the entire expression syntax. If we use V for a value, B for a boolean, and SNL and MNL for single nodelist and multiple nodelist respectively, the following are all valid expressions. V == V B && B B V != V B !B !B && V < V SNL == V MNL && B SNL && B MNL SNL The following are not valid expressions. V B && V B == V !V MNL == V Note that values and booleans each have their place in the syntax, and one cannot be substituted for the other because there is no mapping between them. This is all well-formed and consistent. Now, when we add functions, we need to add them in such a way that they fit into this system. The way to ensure that is to type the function based on its return. This way, we know where in the expression the function is valid. If a function returns a value, it may only be used where a value may be used, i.e. relations. If a function returns a boolean, it may only be used where a boolean may be used, i.e. logical operations and entire expressions. If a function returns a nodelist, it may only be used where a nodelist may be used, i.e. anywhere. While the first two are fairly straightforward, a function returning a nodelist presents an inconsistency: a function returning a nodelist can appear in an expression as a value only if it contains a single node, but we have no syntactic way to ensure that the nodelist it returns has at most a single value. To address this inconsistency, we need to update our conversion rules for nodelists. Instead of: We now use: We also keep the singular path requirement in the syntax because identifying paths that are guaranteed to result in zero-or-one -length nodelists is easy. This also means that MNL == V is now valid, but because of the singular path requirement it can only occur if the MNL is returned by a function. Finally, because of the conversion rules from nodelist to value and from nodelist to boolean, the same function return may behave differently (and perhaps unexpectedly) when a nodelist function appears as value vs as a boolean. I believe this is okay but I advise that we explicitly call it out as a note, possibly with an example as well.
(Note that this removes all of the "optional" types as well.)
NAME here's what I came up with regarding my experience trying to implement the current document. In implementing expressions, I developed a typed binary tree structure with three kinds of nodes: Value nodes - These represent values and can only be used in comparisons. Logic nodes - These represent logical outcomes and come in two flavors: Comparison nodes - These take two values and an operator. Logical operation nodes - These take one or two logical outcomes and an operator. Nodelist nodes - These represent nodelists and (prior to functions) are only returned by paths. (Note, this isn't actually my code, but it's similar enough to what I have.) Functions need to fit into this structure, too, so we create function types for each type of node. With this, I can now represent expressions like . As you can see, everything is typed nicely. Everything gets what it expects. Logic get other logic and comparisons get values. Now, according to the current document, and should return a boolean: which means that with my model, they can't appear in a comparison. But the document shows examples where it's being used both in a test expression and in a comparison: Something is off. I considered that my model was wrong for a while, but I've made many similar expression-parsing apps previously (it has been a special project of mine since high school), and I've never run into a case where a value makes sense where logic should be. So I started to explore the type system more deeply. The many comments above have resulted from this exploration. This is certainly the most loosely typed system I've ever parsed, but it's still typed, so it should behave accordingly. In JSON Path, we say that is valid and returns false because the string and the number are both typed as "values" and those values are not equal. (In C#, they're not even comparable because they're different types, and compilation will fail.) However, (which is perfectly legal in C#) is not valid for JSON Path because (disregarding the ABNF; I'm just discussing the theory here) the logical produced by is not a "value" and can't be used with the operator. We encoded this into the ABNF because of this reasoning. So this is the first problem (and what this issue was created for): if a is to be used as a "logical", it cannot be a subtype of . They cannot be relatable at all because we decided previously that JSON and literals were merely constant values, not "logicals." This also applies to functions. If a function is defined to return a , then it can only appear in an expression where a "logical" would appear. Conversely, if a function is defined to return a , then it can only appear in an expression where a "value" would appear. This means must be invalid. The same logic applies, though in reverse, if you take to mean "JSON or ," and is the invalid one in this case. So we can't have both. Traditionally, a "function" takes a number of arguments and deterministically and in a context-free manner produces the same result. The document, however says this for when the function exists in a test expression: and also this: This means that the function changes behavior depending on the context, meaning it's not really a "function." If you want to claim that there is an implicit conversion from the or that the function returns to the "logical" required by the existence test, then it follows that the literals themselves should also receive that implicit conversion, meaning should be valid, and we already decided that it's not for the reasons I mentioned before. It always comes down to an intentional separation between "logicals" and JSON /, and the type system as it exists tries to blur that line. If we want to re-hash that discussion, I'm sure we'll come to the same conclusion that they are not the same, nor can they be converted between. To do so would introduce ambiguities and remove functionality. (This was the reasoning behind the initial decision.)
For filter expressions, we have made a distinction between the JSON literals / and the result of a comparison (i.e. is different from ). However the examples for the function seem to conflict with this idea. The returns for and are described as I think that this implies the latter would be valid, but the former would imply that the function can operate as an existence test akin to , which doesn't make sense. I don't think the function is testing whether a match exists, but whether the value is a match. The only way I can see the former being valid is if the first argument allowed for a nodelist, but then that would make the latter feel weird (and ). It could be defined so that: if a singular path is used for the first argument, then it MUST be compared to a literal if a plural (?) path is used for the first argument, then it MUST NOT be compared to a literal (it MUST be an existence test) This would resolve the ambiguity and allow for parsing to handle things, but context-sensitive grammar is tricky.
I'm not sure you saw: (https://ietf-wg-URL)
I'm making the point that this is confusing. Why allow both behaviors for functions when we've clearly defined that JSON literals are not treated this way (for which we have good reason) but both can be part of expressions?
In re-reading this, I see the difference between returning a which may be a boolean and returning a . Since we make that distinction, I would now expect that would be invalid as it's comparing a to a , which we've disallowed. must only operate on s.
However... (I'm working on implementing this) this does mean that we effectively have two kinds of functions: value functions (e.g. ) which return a boolean functions (e.g. ) which return a From a user perspective, I think I prefer this as it allows the syntax without the . However such functions would never be supported as a . They could be operands of and , but never or . From an implementor perspective, it's a bit more that I have to consider because I have to make two kinds of functions. Whereas if simply returned a that contains a boolean (like returns a that contains an integer), then I only have to make a "value function." I think the user perspective takes precedence here, even though it's a bit more work for me.
LGTM, thanks.

given string contains a substring that matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form.  Its first argument is a string that is searched for at least one substring that matches the iregexp contained in the string that is the second argument.  The result is "true" if such a substring exists, "false" otherwise. <ins> The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf- jsonpath-iregexp. </ins> 2.6.7. 2.7.


1.4.1. The root node identifier "$" refers to the root node of the argument, <del> i.e., to the argument as a whole.  Every JSONPath expression begins with the root node identifier. </del>  The current node identifier "@" refers to the current node in the context of the evaluation of a filter expression (described later). 2.4.1. <del> Every JSONPath query MUST begin with the root identifier "$". </del>  2.4.2.

Fixes URL
Section 1.4 says: >A JSONPath expression consists of an identifier followed by a series of zero or more segments each of which contains one or more selectors. So this includes relative paths (starting with ). However, section 1.4.1 contradicts this: Similarly, section 2.4 says: >Every JSONPath query MUST begin with the root identifier . (Remember that, according to our terminology, "query" is a short name for a JSONPath expression.) A possible solution is for sections 1.4.1 and 2.4 to talk about JSONPath expressions/queries which do not appear inside a filter expression.

1.4.1. The root node identifier "$" refers to the root node of the argument,  <ins> i.e., to the argument as a whole. </ins> The current node identifier "@" refers to the current node in the context of the evaluation of a filter expression (described later). 2.4.1.  <ins> Every JSONPath query (except those inside filter expressions, see filter-selector) MUST begin with the root identifier "$". </ins> 2.4.2.


Each entry in the registry must include: <del> Initial entries in this sub-registry are as listed in pre-reg: </del>  4.

Fixes URL NAME Any advice on how to format the table so that line length warnings are not produced (by or )?
(1) we could shorten unwieldy monsters such as or (2) we could strip the columns "Change Controller" and "Reference" and just add to the caption that the column value for all of these are "IESG" and "Section 2.6 of RFCthis".
I made a commit for (2). I'd still like to tame the type names as well..
The Function Extensions section of the IANA Considerations section should use the defined type system terminology rather than talking about "kinds", "values", and "nodelists".

Each entry in the registry must include:  <ins> Initial entries in this sub-registry are as listed in pre-reg; the Column "Change Controller" always has the value "IESG" and the column "Reference" always has the value "fnex of RFCthis": </ins> 4.


a comparison using the operator "==" yields true if and only if the comparison is between: <del> equal primitive values, </del>  equal arrays, that is arrays of the same length where each element of the first array is equal to the corresponding

Fixes URL
The ABNF is not currently subject to the I-JSON range restriction that applies to array indices and array slice parameters, but I don't see why it shouldn't be.
I don't understand -- the grammar cannot express this restriction. Indices and slice parameters are integers, not numbers, BTW.
I was thinking of the integer subset of . It seems strange to have to cope with arbitrary precision integers there, but not for array indices and slices.
As for the grammar expressing the restriction, this is a question of validity rather than well-formedness.
Ah, so this would be a text fix.
Indeed. Sorry for the ABNF red herring. I was trying to be precise about which "number" I was talking about.
I guess we'll have to worry about decimal as well as integers in , for consistency. I wonder if there's a suitable precedent like the I-JSON range?
Hmm, is not really discussed in the unnumbered (argh) section "comparisons". (There is text for .)
How about this text?
(and in case anyone thinks we could get away with comparing the string forms of the numbers, remember scientific notation .)
Oh, I see now. We should mention the I-JSON restrictions for numbers, in the comparison section. That seems like a reasonable way to fix this issue.
the complement of "both interoperate" is "at least one doesn't interoperate"...

a comparison using the operator "==" yields true if and only if the comparison is between:  <ins> numbers expected to interoperate as per RFC7493 that compare equal using normal mathematical equality, numbers at least one of which is not expected to interoperate as per I-JSON, where the numbers compare equal using an implementation specific equality, equal primitive values which are not numbers, </ins> equal arrays, that is arrays of the same length where each element of the first array is equal to the corresponding


of a value and make that available for further processing in the filter expression: <del> Its only argument is a value (possibly taken from a singular path as in the example above).  The result also is a value, an unsigned integer. </del>  If the argument value is a string, the result is the number of Unicode scalar values in the string. entirety of, see search below) a given string matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form. <del> Its first argument is a string that is matched against the iregexp contained in the string that is the second argument.  The result is "true" if the string matches the iregexp and "false" otherwise. </del>  The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf- given string contains a substring that matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form. <del> Its first argument is a string that is searched for at least one substring that matches the iregexp contained in the string that is the second argument.  The result is "true" if such a substring exists, "false" otherwise. </del>  The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf-

Fixes URL
NAME I pushed cb1cf124ab to clarify optional arguments.
NAME raised this in a PR review.
I think the answer is "yes" because that allows the function to be used on a Singular Path rather than on just a literal.
Yes. I would expect any function that can take a path must allow any value.
That should be enough to remove this double-take.

of a value and make that available for further processing in the filter expression:  <ins> Its only argument is an optional value (possibly taken from a singular path as in the example above).  The result also is an optional value, an unsigned integer. </ins> If the argument value is a string, the result is the number of Unicode scalar values in the string. entirety of, see search below) a given string matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form.  <ins> Its first argument is an optional string that is matched against the iregexp contained in the string that is the second argument.  The result is "true" if the string matches the iregexp and "false" otherwise. </ins> The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf- given string contains a substring that matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form.  <ins> Its first argument is an optional string that is searched for at least one substring that matches the iregexp contained in the string that is the second argument.  The result is "true" if such a substring exists, "false" otherwise. </ins> The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf-


A type is a set of instances.  A type is a subtype of another type if its set of instances (possibly after coercion) is a subset of the set <del> of instances of the other type. </del>  tbl-types defines the available types in terms of abstract instances, where "n" denotes a node, "v" denotes a value, and "nl" denotes a <del> non-empty nodelist.  The table also lists the subtypes of each type. </del>  Notes:

Fixes URL
is a subtype by coercion of , but Table 13 does not include this subtype relationship.
Not sure I like the concept of "subtype by coercion". One can be used in place of the other as it can be coerced, but that is not usually what we'd call a subtype.
That's a different point really. Table 13 already includes at least one subtype by coercion, so this issue adds another subtype by coercion.
NAME How would you like to proceed with "subtype by coercion"? If we want to explore alternatives, I think it deserves a separate, non-editorial issue.
From the spec, just below : I disagree with the idea that coercion implies subtypes. If I have integer and floating point numeric types, I can cast from one to the other, but that doesn't mean that either is a subtype of the other. C# provides "implicit" casts where the compiler can infer a type cast from context. For example, I think this is closer to what we're dealing with here. We're looking at compatibility and conversion between types, not polymorphism (as "subtype" implies).
If I'm reading the right, it looks like the term is being used to mean different things depending on which column it's in. In column 1 ("Type") or column 3 ("Subtypes"), it's referring to the type system we're creating. In column 2 ("Abstract Instances"), it's referring to the underlying JSON(ish) value kind, i.e. , , , etc, plus . I think having the same word to define these things can be confusing for the reader, and maybe we need something else in one of these places to indicate the distinction. We've already done this for the type to distinguish it from the value.
I'm not sure there is a really good alternative. We could use for abstract instances, but in the rest of the spec, "value" implies a JSON value, so this doesn't seem quite right.
If "value" implies a JSON value (corresponding to the middle column), then we should change the others (the type name). It really is confusing.
In the interim, we decided to append the type names with "Type". I will create a PR for this.
I'm going to wait for to complete before going forward with this as they change the same chunk of text.
Thanks. That's why I am trying to tidy up and get it merged ASAP.
(Modulo the coercion entry on OptionalValue, as discussed.)

A type is a set of instances.  A type is a subtype of another type if its set of instances (possibly after coercion) is a subset of the set  <ins> of instances of the other type.  The subtype relationship is transitive: for any A, B, and C if B is a subtype of A and C is a subtype of B, then C is a subtype of A. </ins> tbl-types defines the available types in terms of abstract instances, where "n" denotes a node, "v" denotes a value, and "nl" denotes a  <ins> non-empty nodelist.  The table also lists the (immediate) subtypes of each type. </ins> Notes:


A test expression either tests the existence of a node designated by an embedded query (see extest) or tests the result of a function expression (see fnex).  In the latter case, if the function <del> expression is of type "OptionalBoolean" or one of its subtypes, it tests whether the result is "true"; if the function expression is of type "OptionalNodes" or one of its subtypes, it tests whether the result is different from "Nothing". </del>  Parentheses MAY be used within "boolean-expr" for grouping. Notes: <del> "OptionalNodeOrValue" is an abstraction of a "comparable" (which may appear on either side of a comparison or as a function </del>  argument). <del> "OptionalNode" is an abstraction of a Singular Path. </del>  <del> "Value" is an abstraction of a primitive value. </del>  <del> "Boolean" is an abstraction of a primitive value that is either "true" or "false". </del>  <del> "OptionalValue" is an abstraction of a primitive value that may alternatively be absent ("Nothing"). </del>  <del> "OptionalNodes" is an abstraction of a "filter-path" (which </del>  appears in a test expression or as a function argument). The abstract instances above can be obtained from the concrete The following subtype relationships depend on coercion: <del> "OptionalNode" is a subtype of "OptionalValue" via coercion since the "OptionalNode" instance "Node(n)" can be coerced to the "OptionalValue" instance "Value(v)", where "v" is the value of the node "n". </del>  <del> "OptionalNode" is a subtype of "OptionalNodes" via coercion since the "OptionalNode" instance "Node(n)" can be coerced to the "OptionalNodes" instance "Nodes(l)", where "l" is a nodelist consisting of just the node "n". </del>  The type correctness of function expressions can now be defined in terms of this type system. true: If it occurs as a "filter-path" in a test expression, the function <del> is defined to have result type "OptionalNodes" or one of its subtypes, or to have result type "OptionalBoolean" or one of its subtypes. </del>  If it occurs as a "comparable" in a comparison, the function is <del> defined to have result type "OptionalNodeOrValue" or one of its subtypes. </del>  For it and any function expression it contains, each argument of the function matches the defined type of the argument according to subtype. The argument is a Singular Path and the defined type of the <del> argument is "OptionalNode" or any type of which "OptionalNode" is a subtype. </del>  The argument is a "filter-path" or a Singular Path and the <del> defined type of the argument is "OptionalNodes". </del>  2.6.3.

This is literally just a renaming of the names in the Type column of Table 13. I also noticed that the description doesn't account for non-nodelist inputs. I'll open another issue for that.
If I'm reading the right, it looks like the term is being used to mean different things depending on which column it's in. In column 1 ("Type") or column 3 ("Subtypes"), it's referring to the type system we're creating. In column 2 ("Abstract Instances"), it's referring to the underlying JSON(ish) value kind, i.e. , , , etc, plus . I think having the same word to define these things can be confusing for the reader, and maybe we need something else in one of these places to indicate the distinction. We've already done this for the type to distinguish it from the value.
I'm not sure there is a really good alternative. We could use for abstract instances, but in the rest of the spec, "value" implies a JSON value, so this doesn't seem quite right.
If "value" implies a JSON value (corresponding to the middle column), then we should change the others (the type name). It really is confusing.
In the interim, we decided to append the type names with "Type". I will create a PR for this.
I'm going to wait for to complete before going forward with this as they change the same chunk of text.
Thanks. That's why I am trying to tidy up and get it merged ASAP.
Thanks!

A test expression either tests the existence of a node designated by an embedded query (see extest) or tests the result of a function expression (see fnex).  In the latter case, if the function  <ins> expression is of type "OptionalBooleanType" or one of its subtypes, it tests whether the result is "true"; if the function expression is of type "OptionalNodesType" or one of its subtypes, it tests whether the result is different from "Nothing". </ins> Parentheses MAY be used within "boolean-expr" for grouping. Notes:  <ins> "OptionalNodeOrValueType" is an abstraction of a "comparable" (which may appear on either side of a comparison or as a function </ins> argument).  <ins> "OptionalNodeType" is an abstraction of a Singular Path. </ins>  <ins> "ValueType" is an abstraction of a primitive value. </ins>  <ins> "BooleanType" is an abstraction of a primitive value that is either "true" or "false". </ins>  <ins> "OptionalValueType" is an abstraction of a primitive value that may alternatively be absent ("Nothing"). </ins>  <ins> "OptionalNodesType" is an abstraction of a "filter-path" (which </ins> appears in a test expression or as a function argument). The abstract instances above can be obtained from the concrete The following subtype relationships depend on coercion:  <ins> "OptionalNodeType" is a subtype of "OptionalValueType" via coercion since the "OptionalNodeType" instance "Node(n)" can be coerced to the "OptionalValueType" instance "Value(v)", where "v" is the value of the node "n". </ins>  <ins> "OptionalNodeType" is a subtype of "OptionalNodesType" via coercion since the "OptionalNodeType" instance "Node(n)" can be coerced to the "OptionalNodesType" instance "Nodes(l)", where "l" is a nodelist consisting of just the node "n". </ins> The type correctness of function expressions can now be defined in terms of this type system. true: If it occurs as a "filter-path" in a test expression, the function  <ins> is defined to have result type "OptionalNodesType" or one of its subtypes, or to have result type "OptionalBooleanType" or one of its subtypes. </ins> If it occurs as a "comparable" in a comparison, the function is  <ins> defined to have result type "OptionalNodeOrValueType" or one of its subtypes. </ins> For it and any function expression it contains, each argument of the function matches the defined type of the argument according to subtype. The argument is a Singular Path and the defined type of the  <ins> argument is "OptionalNodeType" or any type of which "OptionalNodeType" is a subtype. </ins> The argument is a "filter-path" or a Singular Path and the  <ins> defined type of the argument is "OptionalNodesType". </ins> 2.6.3.


integer, that gives the number of nodes in the nodelist.  Note that there is no deduplication of the nodelist.  2.6.5. The "match" function extension provides a way to check whether (the

Fixes URL
All of the other functions we've defined account for input types that they don't expect, and they return in those cases. We should update to do so as well.
count() has declared type OptionalNodesType for its Argument 1. How would it get anything but that?
includes as a potential value.
Yes. As in Lisp, there is a weird alignment between () and nil... I'm not even sure we need NodesType to have an "Optional" aspect. But if we do, I agree we have to say what count makes out of Nothing (either 0 or Nothing), and shape the return type accordingly.
PR fixes this issue in terms of the current type system in which is represented by a non-empty nodelist and is represented by a( path producing a)n empty nodelist.
My argument for a distinction (related to ). When we first started discussing path expressions (, I think), we decided that was disallowed as a complete expression, e.g. . This was intended to disambiguate from being either an existence test or a check that the value in is (e.g. ). As a result of this decision, the literal was only valid as a ; that is, it can only be used with a comparison operator, and a path such as is syntactically invalid. It also allowed the path to return a node for even though the value that's returned by is . Without this distinction, would return JSON , be evaluated as Test-false, and no node would be returned. This means that there must be some distinction between the result of and the literal , although the colloquial meaning for both is "true." In the interim call, I gave the example that if these are the same, then should be valid. Here's why. If is the same thing as the literal , then I can do substitution and a valid expression should result. Since is a valid expression, it is implied that is a valid expression by substituting back the other way. But the ABNF forbids such an expression, so we have a contradiction, which implies our original assumption (" is the same thing as the literal ") must have been wrong. Pedantically, the literal is just that: a literal. It's no different than the literal . It shouldn't carry any meaning of truthiness behind it, but it is commonly used that way. In terms of our expressions, it should be used as a mere token. is asking if the value in is the JSON token . The comparison returns a "true" or "false" result, an actual boolean value rather than just the JSON token, which can then be interpreted as the final result of the expression, or used as an operand for a logical operator, which itself returns a boolean to be interpreted as the final result of the expression.
I also want to explore the idea of "lowering" a TestBoolean to a JSONBoolean so that it can be used in a comparison. This idea is that we can go from a TestBoolean "true"/"false" to the JSONBoolean /, respectively, but not the other direction. This does retrict the literals and from being used in a logical expression (e.g. ). However, if TestBoolean can be lowered to function as JSONBoolean, then can be lowered to and can be lowered to , still implying that is valid. I think the only recourse here is to keep them distinct ideas.
I agree there should be a clear distinction between JSON booleans and the truth values associated with and . I believe the WG consensus supports this, as does the spec.
Good. Progress. No. The spec does not support this. The spec says that a function which returns can be used in both and . But you just said above that there needs to be a distinction between these. See my response to NAME in URL as to why cannot represent JSON booleans. I don't feel like making that argument again.
(Please also see the discussion in and , which is quite useful to find the root cause of this disagreement.)

integer, that gives the number of nodes in the nodelist.  Note that there is no deduplication of the nodelist. <ins> If the argument is "Nothing" (which corresponds to an empty nodelist), the result is the value "0". </ins> 2.6.5. The "match" function extension provides a way to check whether (the