optional step value that moves the position from the start to the end. <del> Filter expressions "?<boolean expr>" select certain children of an </del>  object or array, as in: 1.4.4. )-1]. <del> Uses of function extensions must be correctly typed, as described in fnex. </del>  A JSONPath implementation MUST raise an error for any query which is not well-formed and valid.  The well-formedness and the validity of 2.5.5. <del> 2.5.5.1. </del>  <del> The filter selector has the form "?<expr>".  It iterates over structured values, i.e., arrays and objects. </del>  <del> During the iteration process the node of each array element or object member value being visited is known as the current node.  A boolean expression, usually involving the current node, is evaluated and the current node is selected if and only if the expression yields true. </del>  <del> As the expression is composed of side-effect free components, the order of evaluation does not need to be (and is not) defined. Similarly, for conjunction ("&&") and disjunction ("||") (defined later), both a short-circuiting and a fully evaluating implementation will lead to the same result; both implementation strategies are therefore valid. </del>  The current node is accessible via the current node identifier "@". This identifier addresses the current node of the filter-selector filter-selectors enclosing the filter-selector that is directly enclosing the identifier).  A test expression either tests the existence of a node designated by an embedded query (see extest) or tests the result of a function <del> expression (see fnex).  In the latter case, if the function expression is of type "ST(OptionalBooleanType)" (see typesys), it tests whether the result is "true"; if the function expression is of type "ST(OptionalNodesType)", it tests whether the result is different from "Nothing". Parentheses MAY be used within "boolean-expr" for grouping. Comparisons are between primitive values (that is, numbers, strings, "true", "false", and "null"), Singular Paths, each of which selects at most one node, and function expressions (see fnex) that return a primitive value or at most one node. Alphabetic characters in ABNF are case-insensitive, so "e" can be either "e" or "E". </del>  "true", "false", and "null" are lower-case only (case-sensitive). 2.5.5.2.1. <del> A path by itself in a Boolean context is an existence test which </del>  yields true if the path selects at least one node and yields false if the path does not select any nodes. "filter-path" or a "comparable". Any function expressions in a query must be well-formed (by <del> conforming to the above ABNF) and correctly typed, otherwise the JSONPath implementation MUST raise an error (see synsem-overview). To define which function expressions are correctly typed, a type system is first introduced. </del>  2.6.1. <del> Each argument and result of a function extension must have a declared type. </del>  <del> A type is a set of instances.  A type is a subtype of another type if its set of instances (possibly after coercion) is a subset of the set of instances of the other type.  The subtype relationship is transitive: for any A, B, and C if B is a subtype of A and C is a subtype of B, then C is a subtype of A.  We denote a type "T" and the transitive closure of all its subtypes as "ST(T)". </del>  tbl-types defines the available types in terms of abstract instances, where "n" denotes a node, "v" denotes a value, and "nl" denotes a <del> non-empty nodelist.  The table also lists the (immediate) subtypes of each type. </del>  Notes: <del> "OptionalNodeOrValueType" is an abstraction of a "comparable" (which may appear on either side of a comparison or as a function argument). </del>  <del> "OptionalNodeType" is an abstraction of a Singular Path. </del>  <del> "ValueType" is an abstraction of a primitive value. </del>  <del> "BooleanType" is an abstraction of a primitive value that is either "true" or "false". </del>  <del> "OptionalValueType" is an abstraction of a primitive value that may alternatively be absent ("Nothing"). </del>  <del> "OptionalNodesType" is an abstraction of a "filter-path" (which appears in a test expression or as a function argument). </del>  <del> The abstract instances above can be obtained from the concrete representations in tbl-typerep. </del>  <del> The following subtype relationships depend on coercion: </del>  <del> "OptionalNodeType" is a subtype of "OptionalValueType" via coercion since the "OptionalNodeType" instance "Node(n)" can be coerced to the "OptionalValueType" instance "Value(v)", where "v" is the value of the node "n". </del>  <del> "OptionalNodeType" is a subtype of "OptionalNodesType" via coercion since the "OptionalNodeType" instance "Node(n)" can be coerced to the "OptionalNodesType" instance "Nodes(l)", where "l" is a nodelist consisting of just the node "n". </del>  <del> The type correctness of function expressions can now be defined in </del>  terms of this type system. <del> 2.6.2. </del>  <del> A function expression is correctly typed if all the following are true: </del>  If it occurs directly in a test expression, the function is <del> defined to have a result type in "ST(OptionalNodesType)".  or to have a result type in "ST(OptionalBooleanType)". </del>  <del> If it occurs as a "comparable" in a comparison, the function is defined to have a result type in "ST(OptionalNodeOrValueType)". </del>  <del> For it and any function expression it contains, each argument of the function matches the defined type of the argument according to one of the following rules: </del>  <del> The argument is a function expression with defined result type that is the same as, or a subtype of, the defined type of the argument. </del>  The argument is a literal primitive value and the defined type <del> of the argument is "Value" or any type of which "Value" is a subtype. </del>  <del> The argument is a Singular Path and the defined type of the argument is "OptionalNodeType" or any type of which "OptionalNodeType" is a subtype. </del>  <del> The argument is a "filter-path" (which includes Singular Paths) and the defined type of the argument is "OptionalNodesType". </del>  <del> 2.6.3. </del>  The "length" function extension provides a way to compute the length of a value and make that available for further processing in the filter expression: <del> Its only argument is an optional value (possibly taken from a singular path as in the example above).  The result also is an optional value, an unsigned integer. </del>  If the argument value is a string, the result is the number of Unicode scalar values in the string. For any other argument value, the result is "Nothing". <del> 2.6.4. </del>  The "count" function extension provides a way to obtain the number of nodes in a nodelist and make that available for further processing in integer, that gives the number of nodes in the nodelist.  Note that there is no deduplication of the nodelist. <del> If the argument is "Nothing" (which corresponds to an empty nodelist), the result is the value "0". 2.6.5. </del>  The "match" function extension provides a way to check whether (the entirety of, see search below) a given string matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form. <del> Its first argument is an optional string that is matched against the iregexp contained in the string that is the second argument.  The result is "true" if the string matches the iregexp and "false" otherwise. </del>  <del> The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf- jsonpath-iregexp. 2.6.6. </del>  The "search" function extension provides a way to check whether a given string contains a substring that matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form. <del> Its first argument is an optional string that is searched for at </del>  least one substring that matches the iregexp contained in the string <del> that is the second argument.  The result is "true" if such a substring exists, "false" otherwise. </del>  <del> The result is "Nothing" if the first argument is not a string or the second argument is not a string conforming to I-D.draft-ietf- jsonpath-iregexp. 2.6.7. </del>  2.7.

General simplification and cleanup of the type system used in filter selectors. Intended to supersede , which this is based on. Intended to supersede .
So I think what we want to do is get this PR consistent, and then open the floodgates on 3 vs. 4 types. At least I now know that you can implement this spec in a day :-)
I think that likely depends on the language.
Indeed. 708def9
From the ABNF, paths that return multiple nodes are not allowed in comparisons. I agree with this, and this PR is not changing it. From the type system, as described here, functions that return multiple nodelists are allowed in comparisons. I think this is fine with the defined conversions (also see below). To change this, we would need to (one of): disallow nodelist functions in comparisons at all define a new type that represents a single-node nodelist that is suitable for comparisons (there's also a third option) I'm okay with this. I'm not sure if this would limit any functionality, though. It would limit functions that return to logicals because the nodelist is still convertible to (or "has meaning in," or however you want to say it) a logical. The I put up for the hypothetical function were contrived to endure NAME tests. I certainly can't think of a practical use case to put a nodelist in a comparison. This does raise the question of whether to even have , though. If a function that returns is only valid as a logical, then it might as well return . A function that returns a single-node nodelist would be valid in a comparison, but a function that returns would not. This is really an extension of disallowing nodelist functions in comparisons, so all of that logic still applies (i.e. is still useful?). Conversely, a function that returns the new single-node nodelist type would only be useful in a comparison (otherwise, we run into the same inconsistencies that I've been highlighting this whole time). Because it's only useful in a comparison, it might as well just return the value of the single node, which is . So why have it? Only allow nodelist functions to be parameters for other functions, i.e. disallow them from both logicals and comparisons. This would only allow something like , which is arguably useful, but both and would be invalid. We cannot avoid type conversions. They are implicit in the grammar. To prove this, I'll exclude functions. All paths return nodelists. They don't return logicals, and they don't return values. Singular paths, specifically, return a nodelist with a single value. Paths are always allowed to act as logicals. That is, there is a conversion from a nodelist to a logical. Namely, if a nodelist has one or more nodes, it is treated as (converted to) ; if it has no nodes, it is treated as (converted to) . If I were to write as a test expression explicitly (in a C-ish syntax), it would be: where I'm "casting" the nodelist returned by to . Paths are allowed to act as values if they're singular. That is, there is a conversion from a nodelist to a value if the nodelist contains at most a single node. If the nodelist were to contain more than one node, the conversion would be undefined, and a runtime error would occur. To avoid the potential error, the ABNF applies an additional restriction to identify paths which are guaranteed to return at most a single node, and we can catch it at parse time. Writing explicitly would be: where I'm "casting" the nodelist returned by to a . We have specified these conversions by instead describing their behavior, but that doesn't detract from the fact that they're there. Type conversions are unavoidable. We might as well state them. I think it helps us in the end.
I don't think it's fine as it introduces (more) comparisons with surprising results, such as: which both yield true. Not so. Such functions can be passed as arguments to other functions, e.g. . This is currently my preferred option. Again, for passing as an argument to another function, e.g. . Please note that I'm not proposing any change of semantics, merely an editorial change to improve readability. I suggest we defer discussion of the (de)merits of such a change until/unless we have the proposed text before us. If I attempt such a change, I'll take into account the way some readers may think of it in terms of type conversion.
Clearly, this needs to be decided, but we should first complete this PR so we have a stable basis for that.
I added issue so we can continue on this after completing .
NAME I'm not sure what you are proposing. From my perspective, represents the only blocker to merging .
In the spirit of continuous improvement, we can merge and work on from that as a stable basis. This is not a "blocker", just another work item.
URL Followup issue from
If we want to know whether a function will return a singular nodelist or potentially a full one, we need to add to the declared types. We could define explicit conversion functions from nodelists and/or the new singular nodelist type to JSON values. So would become , making it explicit that we are not comparing the nodelists.
I think (type conversions) needs to be explored (and probably settled) before we can rightfully address this.
I had considered this as well, but with the conversions, I think they just muddy up the syntax.
To me could be taken a number of different ways. Either the node lists are the same or each element has an and a that must be equal. It could also mean some exists that equals some . Wouldn't it be better to use a nested selector (sorry if my terminology is incorrect) such as ?
I just noticed the double-dot. I'm not sure what you would mean by . This isn't valid (enforced by ABNF), and I don't think anyone is arguing that it should be. For , I think the type conversions are obvious. For something like , I'm not sure what to do. With the current conversions, you have three cases: (please forgive the incorrect syntax) both independently give no nodes or multiple nodes, implying returning true both give one node, meaning returning whatever the result of that is (could be true or false) one gives either no nodes or multiple nodes, and the other gives one node, meaning returning false I think the one we're hung up on here is the first case because and also in most cases . Moreover, we don't have a mechanism to compare multiple-node nodelists at all.
I see your line of thinking is to compare the node lists for a boolean result. There are case where a comparison is clear for lists of one or less. If that option was to be allowed and not just return false in all cases then I would think that the lists could only consistently be compared as to the contained nodes in any order since the ancestor () results could be in any order if one of the ancestor nodes was an object which is generally considered to be unordered.
is merely editorial, so it shouldn't block this issue.
PR is my preferred solution to this issue. PR 420 encapsulates the surprising behavior, which is the subject of this issue, in the function. The use of the function in queries becomes an explicit and conscious choice. The following alternative solutions have been suggested: PR proposes that all type conversions be explicit. This adds some precision to the spec and makes the well-typedness rules very compact, but at the cost of usability. The "Conversion example" section in PR 410 highlights the potential surprises remaining in the approach of PR 410. PR builds on PR 410 by adding a . This further impacts usability, as highlighted by the example query . PR 414 provides a type equivalent of a singular nodelist (the result of evaluating a Singular Path). is used either as a variant of in test expressions or as a way of obtaining a value in a comparison. Unlike Singular Paths, which are a syntactic device for avoiding multi-nodelist comparisons, does not appear to pull its weight in terms of specification precision vs the extra complexity/verbosity involved. PR also adds the , but without the usability downsides of the type conversion functions of PRs 410/414. The above critique of the applies to PR 412 too. PR 412 co-opts some implicit conversion rules in the rest of the spec into the function type system, making well-typedness more complex to define and understand.
I notice that the match() function extension has return type This seems wrong to me? A regexp is provided and either there is a match or there isn't. if the argument yields nothing then yields false. How can be returned? (Same is true for obviously).
was introduced as a return for the cases where the inputs weren't the right kind of values, possibly returned by a singular path.
Well, if they're not the right kind of values, then they don't match the regex, right? Thus false.
I mean, we could be pedantic and rename the function but that's what I thought it meant.
It is actually useful to distinguish the three cases (invalid/nonexistent argument, valid but does not match, valid but does match). I'm currently looking at 2.5.5 and trying to make sure that we say all the right things about function-exprs, too.
Um, I disagree, and neither of us has brought forward any evidence. Let's assume the function means I think that as a developer, that is dead easy to understand and use. We already have excellent predicates to test separately for existence, if that's what we care about. Also, I think the stuff is one of the thing that's making the function type system hard to understand.
So you want to map all for Boolean functions cases into , as far as I can tell. That is indeed simpler. Not so easy with functions that return a general JSON value. No, is not better than .
More specifically: Since we have explicit ways to test for existence, and we have boolean combinations, I'm perfectly comfortable with supporting a semantic of "exists and is the right type and X" for some functions. Because it handles two very common use cases very nicely: I really mean "it exists and is the right type and matches" I'm processing fixed-type messages and I know for sure it what fields exist and what their types are. In neither of those cases do I care about the possibility, and I think eliminating might simplify some of our other problems.
This discussion actually depends heavily on the outcome of other discussions. If a is a , then a return is valuable because it can be compared with . If a is a , then a return is unnecessary as it's treated as a "false" expression (the result of ). The other discussions open right now assert that a function can't do both of these and that functions (as a whole) need to be restricted to fit into one of these categories.
I totally agree ... can enter discussion this evening at the earliest ...
So it seems that we'll have Nothing as the value added to the type that otherwise has JSON values, but keep logical (≠ JSON Boolean) and nodelist without such an addition. Let's see how that works out...
I support a (much) simpler type system if we can find one that works and which doesn't compromise the rest of the spec.
I appreciate that everyone in the past few comments has effectively come to the same conclusion, and that it's exactly what represents. I'm happy to close my other PRs if this is the agreed-upon direction.
My argument for a distinction (related to ). When we first started discussing path expressions (, I think), we decided that was disallowed as a complete expression, e.g. . This was intended to disambiguate from being either an existence test or a check that the value in is (e.g. ). As a result of this decision, the literal was only valid as a ; that is, it can only be used with a comparison operator, and a path such as is syntactically invalid. It also allowed the path to return a node for even though the value that's returned by is . Without this distinction, would return JSON , be evaluated as Test-false, and no node would be returned. This means that there must be some distinction between the result of and the literal , although the colloquial meaning for both is "true." In the interim call, I gave the example that if these are the same, then should be valid. Here's why. If is the same thing as the literal , then I can do substitution and a valid expression should result. Since is a valid expression, it is implied that is a valid expression by substituting back the other way. But the ABNF forbids such an expression, so we have a contradiction, which implies our original assumption (" is the same thing as the literal ") must have been wrong. Pedantically, the literal is just that: a literal. It's no different than the literal . It shouldn't carry any meaning of truthiness behind it, but it is commonly used that way. In terms of our expressions, it should be used as a mere token. is asking if the value in is the JSON token . The comparison returns a "true" or "false" result, an actual boolean value rather than just the JSON token, which can then be interpreted as the final result of the expression, or used as an operand for a logical operator, which itself returns a boolean to be interpreted as the final result of the expression.
I also want to explore the idea of "lowering" a TestBoolean to a JSONBoolean so that it can be used in a comparison. This idea is that we can go from a TestBoolean "true"/"false" to the JSONBoolean /, respectively, but not the other direction. This does retrict the literals and from being used in a logical expression (e.g. ). However, if TestBoolean can be lowered to function as JSONBoolean, then can be lowered to and can be lowered to , still implying that is valid. I think the only recourse here is to keep them distinct ideas.
I agree there should be a clear distinction between JSON booleans and the truth values associated with and . I believe the WG consensus supports this, as does the spec.
Good. Progress. No. The spec does not support this. The spec says that a function which returns can be used in both and . But you just said above that there needs to be a distinction between these. See my response to NAME in URL as to why cannot represent JSON booleans. I don't feel like making that argument again.
(Please also see the discussion in and , which is quite useful to find the root cause of this disagreement.)
I think this relates to . An expression boolean (e.g. the result of ) is not a JSON value. This is something that we decided long ago. However Table 13 describes it as " or ". This is wrong. If we allow to be a subtype of , then paths like and will be valid and produce no parsing errors. In (see also ), we decided that this wouldn't be the case. only operates as a boolean, and only operates as a value. and MUST be distinct in order for this to work. This issue is about the role a function plays within an expression. That role is determined by its return value.
Since we don't allow as a function argument, I don't think has a type.
I don't see how that's relevant. I'm not talking about argument types. I'm talking about return types and the role of functions within expressions. returns an expression boolean (not a JSON boolean), which is valid where booleans are valid in expressions, i.e. not in comparisons. returns a JSON-like value, which is valid where values are valid in expressions, i.e. only in comparisons.
This distinction is implied because of the decisions that we've already made in the issue and PR linked above that we aren't going to support type coercion that a JSON- is distinct from expression-true
From : If we allow to be a subtype of , then the above decisions are contradicted.
In the interim we discussed that a function can return one of three kinds of data: a value (e.g. ) a boolean (e.g. ) a nodelist (e.g. a hypothetical function) Value functions can only be used in a comparison (i.e. as a ). This wasn't disputed. I argue that boolean functions can only be used as operands in a logical expression. Effectively they are . The value that they return cannot be translated/lowered/casted/converted/coerced into the JSON literals and and so they are not . (This discussion prompted , as linked above.) We don't have any functions that return a nodelist currently. The hypothetical function (or something like it) that takes a nodelist and returns that nodelist with duplicates removed was proposed during the interim. A use case could be for finding elements with more than x distinct values, like . The key here, though, is that this would only ever be used as: an argument for another function (as above), which isn't part of this discussion a , which is the same as the boolean return scenario (because nodelists aren't ). The role of this return type is the same as a boolean return type. What I want to highlight from this discussion is that a boolean isn't a value, so Table 13 is wrong when it lists as a subtype for as a subtype for See also URL There was an interesting scenario that was mentioned in the interim: . (This may need to be moved to another issue, but it came up during this discussion, so it's here. For now, I just want to explore the use case.) This finds all elements where contains the correct result of whether matches , for example in this data: (The inverse is arguably more useful, but that aside.) Importantly, this is only possible if the result of can be translated/lowered/casted/converted/coerced into the JSON literals and to be considered as . (Again, see .) I can see a use case for something similar in the test suite actually. Let's assume we have an function that takes a path and some data and returns the nodelist result of evaluting the path against the data. (We'll disregard any security or other practical issues of doing that for this purpose of this example.) The test suite is represented as an object with a single key that contains an array of scenarios. Each scenario looks something like this: Now suppose we wanted to query this document for all of the test cases which are incorrect. We could use this path: Ideally, this would return an empty nodelist for a compliant implementation. Something like this would be a great CI sanity check for the test suite. (This exact path would imply that nodelists are , which they're not, but we could work around that by introducing a function or something. This problem doesn't apply to the example since that's attempting to compare things that are more "boolean"-y.)
I'm not sure what "it" is referring to, but Table 13 says that has abstract instances and . Table 13 doesn't mention expression booleans.
You can't use Table 13 in your argument. The basis of my claim is that the table is wrong. "It" in my comment is the "expression boolean," or what we started calling TestBoolean.
I am trying to understand the second paragraph in the of this issue. After substituting "expression boolean" for "it", the paragraph reads: I don't understand how Table 13 says anything at all about expression booleans. Please can you explain?
... pedantry... okay. It should be I'm saying is what we're now calling TestBoolean, which is not a JSON value.
I disagree that represents a "test boolean". The spec makes this clear: The tricky thing, which is what I suspect we should be focussing on, is in the following: >A test expression either tests the existence of a node designated by an embedded query (see ) or tests the result of a function expression (see ). In the latter case, if the function expression is of type or one of its subtypes, it tests whether the result is true; [...] Note that the special case in this paragraph only applies where the return type of the function expression is defined to be (or one of its subtypes). The special case is not triggered by a return value being a boolean value (such as ). In other words, it can be determined before the query is executed.
I think this is an important point -- while JSON values are dynamically typed (there aren't really JSON types anyway), the OptionalBooleanType is statically confined to have three members: Nothing, Value(false), Value(true). So it is compatible with an OptionalValueType recipient, and with being used in a comparable. It is also compatible with being used in a test-expr.
(And the point is that the function expression type system is statically typed.)
You keep using the spec as evidence. I'm arguing that the spec is wrong. You can't use it as evidence. I'm saying that the spec shouldn't define this way. How is compatible with if it represents a JSON value? This is the argument in . You're saying that a value of (//) can be used in a . This means that can be used in a because is , which implies is a valid expression. We've already decided that this is an invalid expression, as linked above, so we have a contradiction. The only way to resolve this contradiction is to make (and by extension ) akin to 's TestBoolean, not the JSON values and . Taking this further, cannot be a subtype of and by extension cannot be a subtype of . These are disjoint and operate under different roles in expressions. Additionally, if truly does represent JSON and , why make a type for it and not make and ? This definition is an incomplete type system. Further, why make the type system stop at functions? We've defined these types for functions, but then they don't extend into the expressions in which they exist, when it's clearly beneficial to do so (even if it requires more work).
Hi Greg, we now know that you want TestBooleans to be distinct from JSON value Booleans. The only argument that you present is that you want them to be different. This is fine; intuition often is useful in designing things. However, other people have the inverse intuition, so we won't get around discussing this with more technical arguments. The purpose of the type system we have been designing is not to replace or augment the JSON type system (which doesn't exist, I'd argue, but that is a different discussion). The purpose is to be able to statically check how functions fit together, and how arguments for functions can be derived from JSONPath filter expressions and how returns from functions can feed back into JSONPath filter expressions. It turns out we have three places where function expressions fit: In a test-expr, in a filter-path, and in a comparable. We wouldn't need to list filter-path here; for some reason we have made this one of the two things that can go into a function argument, along with comparable. This anomaly in the grammar can probably be fixed. The current definition of the function expression type system is clouded by the fact that there is both a subtype relationship and a compatibility relationship (more specifically, a type can be used in place of another type, which involves coercion of the value). We care about: nodelists, which can also stand in for a test (by checking whether the nodelist is empty) single nodes (which almost always need to be optional), which can stand in for a value values, which can also be used as comparable; these are often, but not always Optional. Functions like match or search return a special kind of optional value, which can be used in test expressions, and as a comparable, and as an argument for a function that expects just such a an instance and/or a JSONValue . You don't want them to be used in comparables or as function arguments where a JSONValue can be used. Why. (Note that you can't argue from the grammar, as that doesn't express the function argument/return value type system. Of course, you can ask why literal true and literal false cannot be used in a logical expression, and I'd say that this is a little wart, but it also isn't particularly useful.)
No, I've presented quite a lot of logical evidence that shows they MUST be different. What we have now is rife with contradiction. I agree with this stated purpose. We have failed to fulfil this purpose. What we have failed to do is identify when it's appropriate for a function to appear in each of these places. An individual function needs to be identified as valid as either or (XOR if you prefer) , but it can't be valid as both. My evidence is , which (as currently defined) is capable of returning JSON or . These are s. Everyone is saying that can be used as a , but JSON and cannot be used as . (That's the contradiction.) Because has the potential to return these JSON values, the function MUST be restricted to locations where those values are valid, namely . This would mean that MUST be invalid. If returns here, it reduces to , which we have declared invalid in in order to resolve an ambiguity. How are you not seeing this very obvious contradiction?
The solution to making valid is to make it return what calls a "TestBoolean," optional or not. But that means it's no longer valid as a and or even become invalid.
Note that the first half of this sentence is about the type system, and the second appears to be about the grammar (which just doesn't allow literals as test-expr). I tried to clean up the grammar (without changing it) over in so that may become more obvious.
The grammar and the type system need to align.
That is a desirable, but there are some other desirables in conflict with that. The grammar can very well disallow confusing expressions that do not violate the type system.
Like what? What desirable can be so important that it overrides a logical contradiction?
I'm saying the type system is allowing something that the grammar disallows.
What is obvious to you is far from obvious to me. My starting point is that the aspect of the spec being scrutinised by this issue is actually consistent and maybe just needs tweaking or explaining better. I've to point out what I believe to be the root cause of the confusion. (I am now essentially AFK for a while, so apologies for any delayed responses.)
The underlying requirement is that we want to convert Nodelists into Booleans when used in a test-expr (true if non-empty). We want to convert OptionalNodes into OptionalValues (by looking up the JSON value) when used in a comparable. We somehow need to map this into the function-expr type system as well. Since a function-argument says whether it is a nodelist or a value, this transfers one to one. Smearing up the type system by equating subtyping with coercion does not help, though. I would prefer to have the conversion from a Nodelist to an OptionalNode be explicit (possibly supported by a function that does just that). Then we don't have an unclear situation when a Nodelist is offered as an argument to a function that declares this as a Boolean.
NAME / NAME I think I have a good explanation of the problem. In this post, I will be using angle brackets to denote a nodelist to distinguish it from a JSON array and (code formatting) to indicate JSON literal values true and false (italics) to indicate the operands of a logical operation I would also like to use a couple definitions for expressions: A comparison context is any that involves a comparison operator, like and . A logical context is any that either can be used as the final result of an expression or involves a logical operator, , , or . These are the only two contexts which exists in our expressions. Further, it's important to recognize that the result of a comparison is used in a logical context. We decided that in a logical context was to be interpreted as an existence test. This means that if an item had an property, regardless of its value, the result of would be a logical true to select both items in To illustrate why, consider an implementation that interprets the value at as the result of the expression. It would consider the JSON as logical true and select the first item, but it would consider JSON as logical false and not select the second. With this behavior, there is be no way to select the node with , so we decided this was to always be interpreted as an existence test. This showed that JSON values, even and , only have meaning in a comparison context. Thus we made a distinction between the JSON literals and and the logical values true and false. We then codified this decision by making JSON literals (, , ) and other raw JSON values (strings, numbers) only valid in a comparison context using the ABNF. These functions are defined to return , which is defined by Table 13 to be the JSON literals and and the value . Because they return JSON values, and following from the decisions above, they can only be used in a comparison context. That is, they must be compared to another JSON value using a comparison operator to yield a logical result; they cannot be used directly in a logical context. But we also want these functions to be valid in both a comparison context and a logical context. This is the contradiction. They way they're defined, they cannot be used in both contexts. My original suggestion was to have the functions instead return a logical true or false instead of the JSON values. However, this only reverses the contraction rather than solving it. This weekend, I thought about how to define them so that they work in both contexts. I started by analyzing a construct that already does: a path. can exist in both a comparison context and a logical context, and it always returns a nodelist. How that nodelist is interpreted in each context determines the different behaviors it exhibits. To illustrate, let's take a look at what happens for in a comparison context with (1) and in a logical context (2) when evaluating Most importantly, always returns and , respectively, for these items, regardless of the context. In a comparison context, the single value is extracted from and -compared with , and the node is selected. is converted to and -compared with , and the node is not selected. In a logical context, contains a node, so it evaluates to logical true, and the node is selected. contains no nodes, so it evaluates to logical false, and the node is not selected. returns the same nodelist for both contexts because it's unaware of that context. It's just a path evaluating data. While always returns the same nodelist, how that nodelist is interpreted changes with context. Specifically, in a comparison context, is interpreted as JSON . in a logical context, is interpreted as logical true. In order to make these functions valid in both contexts, they need to return something that is valid in both contexts, and the only thing we have that works in both contexts is a nodelist, as demonstrated by . Let's perform our analysis again for the comparison context with (1) and the logical context with (2). Theoretically, these should select the same nodes, namely nodes that DO NOT match. Starting with the affirmative case where the function identifies a match, let's say that returns . In a comparison context, the single value is extracted from and -compared with , and the node is not selected. In a logical context, has nodes so is evaluated to logical true and negated by , and the node is not selected. Good so far. Both paths return the same thing for cases where a match is found. Now, let's look at the negative case where the function does not identify a match. For this, let's say that the function returns . In a comparison context, the single value is extracted from and -compared with , and the node is selected. In a logical context, has nodes so is evaluated to logical true and negated by , and the node is NOT selected. Uh, oh. We have differing results. This doesn't work. Let's try returning for the negative case instead. In a comparison context, is converted to and -compared with , and the node is NOT selected. In a logical context, has no nodes so is evaluated to logical false and negated by , and the node is selected. Hm... that doesn't work either. In fact, there is NO SINGLE VALUE that can return to make these paths select the same nodes for the non-matching case. The only recourse, then is to restrict to only one of: comparison context: logical context: and can't support both. Doing this would mean that when appears outside of its decided context, it will not be considered well-formed, requiring a parsing error (or compilation error, if you prefer). Note that doesn't preclude some other function from returning a nodelist in a manner that is consistent between contexts.
NAME thanks for the detailed analysis. The current spec defines a special case which produces different behaviour. Section 2.5.5 says (emphasis added): So, when a function returning an (or one of its subtypes) is used in a test expression, this is not treated as an existence test. Instead, the value returned by the function is compared against . So, for the three possible return values, yields true whereas and yield false. On the other hand, when a function returning an (or one of its subtypes) is used in a comparison, the value returned by the function is compared to the other side of the comparison. Thus and produce identical nodelists. The crucial thing to note, and which avoids the above contradiction, is that a function returning an (or one of its subtypes) is treated differently depending on where it is used. This difference is determined at parse/compile time. During execution, the return value of the function is treated accordingly.
I appreciate that the spec says this explicitly. However, I maintain that this special case needs to be removed. This contextual difference in behavior represents an inconsistency in the overall grammar of expressions. Why are functions treated specially? The underlying mechanics should operate the same, and they don't. The spec very explicitly says that a JSON / is not valid in a logical context... unless it comes from a function. Why?! Why special-case functions? A function (or any expression component) should only be valid where its return type is valid, that is, where a value of that return type is valid. I should be able to replace the function with a value of its return type and still have a valid expression. But if I do that for , I get which is explicitly invalid. Therefore we have an inconsistent grammar. That statement from the spec you quoted is merely a bandage trying to cover up a more serious problem. In order to have a consistent grammar, this sort of value substitution MUST work. Finally, as the only person who has even attempted to implement this, I find it confusing and difficult, which should be more than enough reason to change it.
We don’t agree. They are not treated specially. It is just a case that only applies to functions. No. It doesn’t provide a way to put JSON literals there, so the issue only occurs for function returns. (Because it is the only case.) (If talking about the type system:). Yes. Maybe we should say “well-typed” instead of “valid”... Where you can’t notate that value, this is moot. It is not invalid, it is not well-formed. I don’t follow. You could say that. The original problem is that NAME is not saying whether it talks about the node(s) or about the value(s) there. We have made sure this is well-defined in all cases: in a test-expr, it means (the existence) of nodes, and in a comparison-expr (where we restrict the paths to singular ones) it means the value (or Nothing if no node). Function expressions straddle this boundary, so something needed to be done. Hence the type system, which is not about values. I don’t follow. Maybe we can learn from this exercise that a mental model that conflates values and types doesn’t work too well here. Grüße, Carsten
THAT'S WHAT A SPECIAL CASE IS! A special case is precisely a case that only applies to one thing. Precisely! JSON literals can't go there because the values are invalid there. That's not a reason. You're saying functions are special-cased because they're special cases. They shouldn't be. They should be treated like everything else in the expression. Special casing yields an inconsistent grammar. It's inconsistent because the same grammar can behave differently. and have the same grammar: and both act in a logical capacity (i.e. ). They should behave the same: return a nodelist which is analyzed for contents. But they don't; they are inconsistent in their behavior. There's no reason for them to behave differently except that the spec says that they do. When evaluating an expression such as , I must first analyze the components, , , and . From those, I get values which can be operated on by and . Those values MUST be of the correct type for those operators. The operators, though, have no knowledge of what produced the values. But you're saying that needs to know if is a function or a path, and depending on which, it changes its behavior. That's inconsistent behavior. should do one thing, and it should do that thing without context. For the purpose of this discussion, "valid" means that it doesn't produce an error and can be evaluated. I'm not making a distinction between being ABNF-valid and well-formed. I don't understand what you mean by "notating" the value. The example I gave shows that what would be considered a valid expression () is actually invalid when you consider the return type . The problem is that, within a single context, you're defining one behavior for a path and another for a function. Functions should behave the same as paths in the same contexts. The spec disagrees with you: "A type is a set of instances." The type system is very much about values. (Note that the spec doesn't define "instance." This section is the only place the word is used. This is the example I gave. always finds a match. According to the spec, it should return JSON . Thus the expectation is that will return a nodelist containing all of the nodes of the subject data. Because the return value is always the same, I should be able to substitute the return value into the expression and yield another valid (and well-formed) expression that yields the same result. For example, in , I can substitue a logical true (not JSON ) for and get . Performing that substitution on yields (that's the JSON literal ). This is an invalid expression. It produces an error. Becuase and do not produce the same outcome, we have an inconsistency. It follows that if we are to have a consistent grammar and is invalid (explicitly via the ABNF), must also be invalid. This kind of substitution exists in every programming language and every mathematical convention I've ever seen. It is the very definition of logical consistency. It is foolish of us to define JSON Path expressions in a way that operates in a different manner. As mentioned, I'm not conflating values and types. The spec clearly defines a type as a set of instances/values. I understand the difference. To accuse me of misunderstanding this is rude, especially knowing that I come from .Net and C#, which are very strictly typed. You're very clearly holding onto keeping this for some reason despite the many logical arguments I've made against it. I don't understand why. Is it pride? Is it unwillingness to change? Is it laziness? For the benefit of the specification and its wide adoption, I urge you to actually consider my arguments and remove this inconsistency from the spec.
these are the two options I present. I don't see a way to have a consistent typing system and allow and in both logical and comparative contexts. The other option I can see is just doing away with the typing system altogether and just checking syntax (and maybe argument count). Then functions can return whatever they like and the expression evaluation system will just run. If it encounters something invalid, then it doesn't select that node, just like it would if returned a string in .
NAME you have refused to acknowledge that there is anything wrong with the current solution of special-casing function in test expressions, and yet you have also failed to explain why such a special case is acceptable. The only argument you have presented is to quote the spec and say, "The document provides a solution for this." As I'm arguing that the document (and therefore the solution) is wrong, the document cannot be used as evidence. There is a logical and syntactical inconsistency in the solution that the document presents ("in the document"). I have repeatedly offered multiple analyses, explanations, logical arguments, and examples detailing why the current solution is inconsistent and contradictory. Yet it seems you're not reading them. You have not in any way refuted my objections with why my analyses are incorrect or how the current solution is consistent and non-contradictory (I re-read the issue to be sure), yet you continue to oppose me. I ask you to provide such reasoning or stand down and accept one of my proposals.
Yes. Of course. I had problems responding to your critique because it doesn't seem to me to be aligned with the document. So the logical next step is to improve the document editorially, and then see whether your critique still applies. Unfortunately, I don't have much time this week, as I'm part of a small group that teaches an intensive course.. I can't really ask the others to do my work while I tend to this. So I ask for a little more patience.
Co-chair hat on. Clearly we are having a communication problem here. Greg is making two claims: The language/meaning of the spec contains unacceptable special-casing and/or inconsistency The current version of the spec is unreasonably hard to implement. On (1.) he has failed to convince the editors; Carsten has suggested that there is room for editorial improvements to clear the air. My notes: I'm inclined to give the editors a chance to see if they can reduce the discomfort and find us a path forward. I confess that after repeated readings of the spec and the discussion, I'm finding it hard to figure out what is what. In particular I find the language describing the type system to be really opaque. I see no particular reason why the input type repertoire and output type repertoire for functions need to be identical. See my other issue about the return values for and . In particular I suspect that strictly limiting the output type repertoire might simplify the lives of spec readers and implementors too. Options such as limiting return types to JSON primitive types (, , number, string, ) or alternately a strictly-constrained set of node-lists may be worth considering? I'm probably more concerned about Greg's concern (2.), that the current spec is hard to implement. I have painful personal experience with beautifully crafted specs that are unreasonably hard to code up. Greg, perhaps a little more on the difficulty would be useful? It does seem to me that any implementation is going to have some special-casing around the functions, since they are a different kind of beast than the rest of the spec. If we are unable to come to rough consensus on how to specify the function-extension framework, one plausible path forward is simply to remove it. I think that would be well within the scope of our charter.
NAME thank you for taking the time to review this issue. I've been asking for this and haven't received anthing except "the spec says you're wrong." (See below) I agree with your statement, but aligning function input with function output is not what this discussion is about. This is about how functions should only be valid where values of their return types would be valid. E.g. A function which returns a JSON or should only be valid where a JSON or literal would be valid. This makes for a consistent type system. For the current document, this is not the case. I've tried to explain it above by describing how typed systems should work, but maybe I can rewrite my arguments from an implementation point of view. I'll work on this and post back. I might be happy removing the type system (I still need to explore that), but I think we need to keep functions. And that means that we still need to address what happens when a function appears in a place it's not (shouldn't be) expected, like a function that returns a JSON or appearing as an operand for when JSON or themselves cannot appear there. As mentioned in your , we need functions to fill in the gaps for "traditional" syntaxes that don't fit within the spec's syntax. You're right, though: removing functions would make this issue moot, but we'd also lose some of the functionality that users are used to when working with JSON Path. To address the general response that I've received, I need to remove JSON Path from the argument and make the dispute absurd. Suppose we have the image below, and we want to write a document that describes how to reproduce this image. ! There are two scenarios I want to explore: The document says, "The sky is red." I see this and open an issue saying that the specification is wrong, that it should say that the sky is blue. In this scenario, I am correct in saying the spec is wrong. The document says, "The sky is blue," and I open an issue saying that the specification is wrong and suggest that it should say that the sky is red. In this scenario, I am incorrect. In both cases, I lay out an argument. And in both cases, the response should be, "Let's check the source image," not "No, the spec says that the sky is [color]. It's fine the way it is." We check the source image, and the issue is resolved. My point is that if I'm saying the spec is wrong, the spec can't be used as evidence to say that the spec is right. The evidence has to come from an external source.
The editors did not have much time in the last week, so that's why you didn't get a detailed explanation. Responding to each of these reminders doesn't increase the speed at which I can work. I'll submit a PR with an improved description of function expressions that should make the whole thing moot. But I can't do this in zero time. I do have empathy for your impatience. We can choose to do so, we can choose not to do so. The only reason you have given so far is that you want that to be the case. I believe that allowing JSON literals in the test-expr grammar would be very confusing. But we could. It just isn't needed. It is indeed hard to introduce a useful function extension without a type system. The requirement on the type system I'm trying to fulfill is that it meshes with the properties of the non-extended parts of JSONPath, in particular that we can examine well-formed expressions whether they also are well-typed, and that this can be done independently of the actual JSON data that will be fed to the expression. This works best with a type system that is entirely static. Yes. No. Again, that equivalence is a preference from you that I don't share. I'm not discussing your blue sky, because we do agree that you do have that preference. We just don't agree about that preference. If you have a little time left, can you please check ? I don't think Glyn is getting to this.
Understood. I await your detailed comments. There was no such reminder until recently. No, I have given several very detailed logically reasoned explanations backing up my case. Yes, it would. I'm not advocating for this. I'm advocating that because "allowing JSON literals in the test-expr grammar would be very confusing," we also shouldn't allow functions that return JSON literals in test-expr grammar. This works best with a type system that is entirely static. I 100% agree with this. This is not a preference. It is an aspect of typed systems in general. Functions may only appear where their return values may appear.
I'm AFK until Tuesday, but I wonder if Greg's concerns would be addressed if it was possible to support referential transparency where a function could be replaced, per call (tricky to implement in general), with (a concrete representation of) its return value without changing the result of the query.
(1) We would need literals for the whole type system. We can't notate Nothing or other node lists at the moment. (2) We would need to tag the literal with the function return type that is intended.
Not sure we would need all those literals. Suppose there was a node whose value was the same as the return value of a function. One referential transparency rule would be that replacing the function call with the path results in the same query result. But I take your point about possibly needing to tag values with types. This makes me wonder how we position the type system as a natural extension of the syntactic rules for non-function expressions.
I agree that we don't need representation of all values in the syntax. For example, JSON arrays and objects are values that are not allowed in the syntax. Here's how I see the type landscape. We have three types: values, boolean, and nodelists. Values include any and all JSON values: arrays, objects, strings, numbers, and the three literals , , and . We don't need any more precise typing. These can all be considered "values." Also included as a value is , which we designated to represent the absence of a value, e.g. the value of in the object . Strings, numbers, and the three JSON literals can be expressed in the syntax. JSON objects, JSON arrays, and have no representation in the syntax. Booleans are the result of any operation, e.g. , , or . Expressions in their entirety must also result in a boolean. There are two states (I'll use "state" to not confuse it with "value") for a boolean: true and false. The relation operators, and et al., take two values (from above) as arguments and return a boolean. The logical operators, , , and , take two (one for ) booleans as arguments and return a boolean. There is no relationship between the boolean states true and false and the JSON literals and . There is no representation for booleans in the syntax. Nodelists result from path evaluation. There is no representation for nodelists in the syntax. A nodelist is convertible to a Value if it has at most a single node. If a nodelist contains a node, the result is the value of that node. If a nodelist contains no nodes, the resulting value is . To enable syntax checking for paths in expressions, we have defined "Singular Path" as an identifiable syntax to ensure that a path can return at most one node and thus may be converted to a Value. A nodelist is always convertible to a Boolean. The resulting state is whether it contains nodes: true if it contains nodes false otherwise Setting up the type system this way covers the entire expression syntax. If we use V for a value, B for a boolean, and SNL and MNL for single nodelist and multiple nodelist respectively, the following are all valid expressions. V == V B && B B V != V B !B !B && V < V SNL == V MNL && B SNL && B MNL SNL The following are not valid expressions. V B && V B == V !V MNL == V Note that values and booleans each have their place in the syntax, and one cannot be substituted for the other because there is no mapping between them. This is all well-formed and consistent. Now, when we add functions, we need to add them in such a way that they fit into this system. The way to ensure that is to type the function based on its return. This way, we know where in the expression the function is valid. If a function returns a value, it may only be used where a value may be used, i.e. relations. If a function returns a boolean, it may only be used where a boolean may be used, i.e. logical operations and entire expressions. If a function returns a nodelist, it may only be used where a nodelist may be used, i.e. anywhere. While the first two are fairly straightforward, a function returning a nodelist presents an inconsistency: a function returning a nodelist can appear in an expression as a value only if it contains a single node, but we have no syntactic way to ensure that the nodelist it returns has at most a single value. To address this inconsistency, we need to update our conversion rules for nodelists. Instead of: We now use: We also keep the singular path requirement in the syntax because identifying paths that are guaranteed to result in zero-or-one -length nodelists is easy. This also means that MNL == V is now valid, but because of the singular path requirement it can only occur if the MNL is returned by a function. Finally, because of the conversion rules from nodelist to value and from nodelist to boolean, the same function return may behave differently (and perhaps unexpectedly) when a nodelist function appears as value vs as a boolean. I believe this is okay but I advise that we explicitly call it out as a note, possibly with an example as well.
(Note that this removes all of the "optional" types as well.)
NAME here's what I came up with regarding my experience trying to implement the current document. In implementing expressions, I developed a typed binary tree structure with three kinds of nodes: Value nodes - These represent values and can only be used in comparisons. Logic nodes - These represent logical outcomes and come in two flavors: Comparison nodes - These take two values and an operator. Logical operation nodes - These take one or two logical outcomes and an operator. Nodelist nodes - These represent nodelists and (prior to functions) are only returned by paths. (Note, this isn't actually my code, but it's similar enough to what I have.) Functions need to fit into this structure, too, so we create function types for each type of node. With this, I can now represent expressions like . As you can see, everything is typed nicely. Everything gets what it expects. Logic get other logic and comparisons get values. Now, according to the current document, and should return a boolean: which means that with my model, they can't appear in a comparison. But the document shows examples where it's being used both in a test expression and in a comparison: Something is off. I considered that my model was wrong for a while, but I've made many similar expression-parsing apps previously (it has been a special project of mine since high school), and I've never run into a case where a value makes sense where logic should be. So I started to explore the type system more deeply. The many comments above have resulted from this exploration. This is certainly the most loosely typed system I've ever parsed, but it's still typed, so it should behave accordingly. In JSON Path, we say that is valid and returns false because the string and the number are both typed as "values" and those values are not equal. (In C#, they're not even comparable because they're different types, and compilation will fail.) However, (which is perfectly legal in C#) is not valid for JSON Path because (disregarding the ABNF; I'm just discussing the theory here) the logical produced by is not a "value" and can't be used with the operator. We encoded this into the ABNF because of this reasoning. So this is the first problem (and what this issue was created for): if a is to be used as a "logical", it cannot be a subtype of . They cannot be relatable at all because we decided previously that JSON and literals were merely constant values, not "logicals." This also applies to functions. If a function is defined to return a , then it can only appear in an expression where a "logical" would appear. Conversely, if a function is defined to return a , then it can only appear in an expression where a "value" would appear. This means must be invalid. The same logic applies, though in reverse, if you take to mean "JSON or ," and is the invalid one in this case. So we can't have both. Traditionally, a "function" takes a number of arguments and deterministically and in a context-free manner produces the same result. The document, however says this for when the function exists in a test expression: and also this: This means that the function changes behavior depending on the context, meaning it's not really a "function." If you want to claim that there is an implicit conversion from the or that the function returns to the "logical" required by the existence test, then it follows that the literals themselves should also receive that implicit conversion, meaning should be valid, and we already decided that it's not for the reasons I mentioned before. It always comes down to an intentional separation between "logicals" and JSON /, and the type system as it exists tries to blur that line. If we want to re-hash that discussion, I'm sure we'll come to the same conclusion that they are not the same, nor can they be converted between. To do so would introduce ambiguities and remove functionality. (This was the reasoning behind the initial decision.)

optional step value that moves the position from the start to the end.  <ins> Filter expressions "?<logical-expr>" select certain children of an </ins> object or array, as in: 1.4.4. )-1].  <ins> Uses of function extensions must be , as described in fnex. </ins> A JSONPath implementation MUST raise an error for any query which is not well-formed and valid.  The well-formedness and the validity of 2.5.5.  <ins> Filter selectors are used to iterate over the elements or members of structured values, i.e., JSON arrays and objects.  The structured values are identified in the nodelist offered by the child or descendant segment using the filter selector. For each iteration (element/member), a logical expression, the , is evaluated which decides whether the node of the element/member is selected.  (While a logical expression evaluates to what mathematically is a Boolean value, this specification uses the term to maintain a distinction from the Boolean values that JSON can represent.) During the iteration process, the filter expression receives the node of each array element or object member value of the structured value being filtered; this element or member value is then known as the . The current node can be used as the start of one or more JSONPath queries in subexpressions of the filter expression, notated via the current-node-identifier "@".  Each JSONPath query can be used either for testing existence of a result of the query, for obtaining a specific JSON value resulting from that query that can then be used in a comparison, or as a . </ins>  <ins> Within the logical expression for a filter selector, function expressions can be used to operate on nodelists and values.  The set of available functions is extensible, with a number of functions predefined, see fnex, and the ability to register further functions provided by the Function Extensions sub-registry (iana-fnex).  When a function is defined, it is given a unique name, and its return value and each of its parameters is given a </ins>  <ins> .  The type system is limited in scope; its purpose is to express restrictions that, without functions, are implicit in the grammar of filter expressions.  The type system also guides conversions (type- conv) that mimic the way different kinds of expressions are handled in the grammar when function expressions are not in use. </ins>  <ins> 2.5.5.1. The filter selector has the form "?<logical-expr>". As the filter expression is composed of side-effect free constituents, the order of evaluation does not need to be (and is not) defined.  Similarly, for conjunction ("&&") and disjunction ("||") (defined later), both a short-circuiting and a fully evaluating implementation will lead to the same result; both implementation strategies are therefore valid. </ins> The current node is accessible via the current node identifier "@". This identifier addresses the current node of the filter-selector filter-selectors enclosing the filter-selector that is directly enclosing the identifier). <ins> Logical expressions offer the usual Boolean operators ("||" for OR, "&&" for AND, and "!" for NOT).  Parentheses MAY be used within "logical-expr" for grouping. </ins> A test expression either tests the existence of a node designated by an embedded query (see extest) or tests the result of a function  <ins> expression (see fnex).  In the latter case, if the function result type is declared as "LogicalType" (see typesys), it tests whether the result is "LogicalTrue"; if the function result type is declared as "NodesType", it tests whether the result is non-empty.  If the declared function result type is "ValueType", its use in a test expression is not well-typed. Comparison expressions are available for comparisons between primitive values (that is, numbers, strings, "true", "false", and "null").  These can be obtained via literal values; Singular Paths, each of which selects at most one node the value of which is then used; and function expressions (see fnex) of type "ValueType" or "NodesType" (see type-conv). Literals can be notated in the way that is usual for JSON (with the extension that strings can use single-quote delimiters).  Alphabetic characters in ABNF are case-insensitive, so within a floating point number the ABNF expression "e" can be either the value 'e' or 'E'. </ins> "true", "false", and "null" are lower-case only (case-sensitive). 2.5.5.2.1.  <ins> A path by itself in a Logical context is an existence test which </ins> yields true if the path selects at least one node and yields false if the path does not select any nodes. "filter-path" or a "comparable". Any function expressions in a query must be well-formed (by  <ins> conforming to the above ABNF) and well-typed, otherwise the JSONPath implementation MUST raise an error (see synsem-overview).  To define which function expressions are well-typed, a type system is first introduced. </ins> 2.6.1.  <ins> Each parameter and the result of a function extension must have a declared type. </ins>  <ins> A type is a set of instances.  Declared types enable checking a JSONPath query for well-typedness independent of any argument the JSONPath query is applied to. </ins> tbl-types defines the available types in terms of abstract instances, where "n" denotes a node, "v" denotes a value, and "nl" denotes a  <ins> nodelist. </ins> Notes:  <ins> "ValueType" is an abstraction of a JSON value or "Nothing". </ins>  <ins> "LogicalType" is an abstraction of the result of a "logical-expr". Its two instances, "LogicalTrue" and "LogicalFalse", are not related to the JSON literals "true" and "false" and have no syntactical representation in JSONPath. </ins>  <ins> "NodesType" is an abstraction of a "filter-path" (which appears in a test expression or as a function argument).  Members of "NodesType" have no syntactical representation in JSONPath. </ins>  <ins> The abstract instances above can be obtained from the concrete representations in tbl-typerep. </ins>  <ins> 2.6.2. </ins>  <ins> The following type conversions may occur: </ins>  <ins> Where an expression with a declared type of "NodesType" needs to be converted to a "ValueType", the conversion proceeds as follows: If the nodelist contains a single node, the conversion result is the value of the node. If the nodelist is empty or contains multiple nodes, the conversion result is "Nothing". </ins>  <ins> Where a member of "NodesType" needs to be converted to a "LogicalType", the conversion proceeds as follows: </ins>  <ins> If the nodelist contains one or more nodes, the conversion result is "LogicalTrue". </ins>  <ins> If the nodelist is empty, the conversion result is "LogicalFalse". </ins>  <ins> The well-typedness of function expressions can now be defined in </ins> terms of this type system.  <ins> 2.6.3. </ins>  <ins> A function expression is well-typed if all of the following are true: </ins> If it occurs directly in a test expression, the function is  <ins> declared to have a result type of "LogicalType", or (conversion applies) "NodesType". If it occurs directly as a "comparable" in a comparison, the function is declared to have a result type of "ValueType", or (conversion applies) "NodesType". </ins>  <ins> Otherwise, it is occurring as an argument to a further function expression, and the following rules for function arguments apply to its declared result type. </ins>  <ins> Each argument of the function can be used for the declared type of the corresponding declared parameter according to one of the following rules: </ins>  <ins> The argument is a function expression with declared result type that is the same as the declared type of the parameter. </ins> The argument is a literal primitive value and the defined type  <ins> of the parameter is "ValueType". </ins>  <ins> The argument is a Singular Path or "filter-path" (which includes Singular Paths), or a function expression with declared result type "NodesType".  Where the declared type of the parameter is not "NodesType", a conversion applies. </ins>  <ins> 2.6.3.1. </ins>  <ins> While functions returning "NodesType" can appear in both comparisons and test expressions, path authors must be aware of the conversions and the potential outcomes.  Specifically, nodelists convert to values and logicals according to tbl-typeconv: For example, given a function "myfunc()" of declared result type "NodesType", the following filter expressions are both valid but can have different and unexpected results: Note that a multiple-node nodelist result is deemed "LogicalTrue" in the test expression, but cannot be used for conversion to a single value and thus feeds a comparison with "Nothing". 2.6.4. </ins> The "length" function extension provides a way to compute the length of a value and make that available for further processing in the filter expression:  <ins> Its only argument is an instance of "ValueType" (possibly taken from a singular path as in the example above).  The result also is an instance of "ValueType": an unsigned integer or "Nothing". </ins> If the argument value is a string, the result is the number of Unicode scalar values in the string. For any other argument value, the result is "Nothing".  <ins> 2.6.5. </ins> The "count" function extension provides a way to obtain the number of nodes in a nodelist and make that available for further processing in integer, that gives the number of nodes in the nodelist.  Note that there is no deduplication of the nodelist.  <ins> 2.6.6. </ins> The "match" function extension provides a way to check whether (the entirety of, see search below) a given string matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form.  <ins> Its arguments are instances of "ValueType".  If the first argument is not a string or the second argument is not a string conforming to I- D.draft-ietf-jsonpath-iregexp, the result is "LogicalFalse". Otherwise, the string that is the first argument is matched against the iregexp contained in the string that is the second argument; the result is "LogicalTrue" if the string matches the iregexp and "LogicalFalse" otherwise. </ins>  <ins> 2.6.7. </ins> The "search" function extension provides a way to check whether a given string contains a substring that matches a given regular expression, which is in I-D.draft-ietf-jsonpath-iregexp form.  <ins> Its arguments are instances of "ValueType".  If the first argument is not a string or the second argument is not a string conforming to I- D.draft-ietf-jsonpath-iregexp, the result is "LogicalFalse". Otherwise, the string that is the first argument is searched for at </ins> least one substring that matches the iregexp contained in the string  <ins> that is the second argument; the result is "LogicalTrue" if such a substring exists and "LogicalFalse" otherwise. </ins>  <ins> 2.6.8. </ins> 2.7.


empty nodelist is a valid query result. A segment operates on each of the nodes in its input nodelist in <del> turn, and the resultant nodelists are concatenated to produce the result of the segment.  A node may be selected more than once and appears that number of times in the nodelist.  Duplicate nodes are not removed. </del>  A syntactically valid segment MUST NOT produce errors when executing the query.  This means that some operations that might be considered 2.7. <del> Segments apply one or more selectors to an input value and concatenate the results into a single nodelist. </del>  It turns out that the more segments there are in a query, the greater the depth in the input value of the nodes of the resultant nodelist: Selectors of different kinds may be combined within a single child segment. <del> The resulting nodelist of a child segment is the concatenation of the nodelists from each of its selectors in the order that the selectors appear in the list.  Note that any node matched by more than one selector is kept as many times in the nodelist. </del>  Where a selector can produce a nodelist in more than one possible order, each occurrence of the selector in the child segment may 2.7.2.2. <del> A descendant segment produces zero or more descendants of the input </del>  value. <del> A descendant selector visits the input value and each of its descendants such that: </del>  nodes of any array are visited in array order, and be a result of applying the child segment "[<selectors>]" to the node "Di". <del> The result of the descendant selector is the concatenation of "R1", ..., "Rn" (in that order). </del>  So a descendant segment drills down one or more levels into the <del> structure of the input value. </del>  2.7.2.3.

Add clarifications that the input nodelist provides the outer order, the segments provide a middle-level order, and selectors (if multiple) provide the inner order. This may have been obvious to some readers, but now should be obvious to more, including yours truly.
I don't think the example for in Table 17 is right. URL It says: It should be: Note that the descendants list puts out before and . The wildcard selector then has no choice but outputting , , , .
Nice catch! The example needs fixing as does the ordering note below Table 17.
I let the computer do the catching :-) Do you want to make these changes? I'd do a new issue then. (And wait a bit for Greg to catch up.)
NAME Yes, I'm happy to do that and please do create a new issue. I won't be able to fix it before the weekend at the earliest though.
URL
LGTM thanks. Semantics unchanged, but hopefully a lot clearer.

empty nodelist is a valid query result. A segment operates on each of the nodes in its input nodelist in  <ins> turn, and the resultant nodelists are concatenated in the order of the input nodelist they were derived from to produce the result of the segment.  A node may be selected more than once and appears that number of times in the nodelist.  Duplicate nodes are not removed. </ins> A syntactically valid segment MUST NOT produce errors when executing the query.  This means that some operations that might be considered 2.7.  <ins> For each node in an input nodelist, segments apply one or more selectors to the node and concatenate the results of each selector into per-input-node nodelists, which are then concatenated in the order of the input nodelist to form a single segment result nodelist. </ins> It turns out that the more segments there are in a query, the greater the depth in the input value of the nodes of the resultant nodelist: Selectors of different kinds may be combined within a single child segment.  <ins> For each node in the input nodelist, the resulting nodelist of a child segment is the concatenation of the nodelists from each of its selectors in the order that the selectors appear in the list.  Note that any node matched by more than one selector is kept as many times in the nodelist. </ins> Where a selector can produce a nodelist in more than one possible order, each occurrence of the selector in the child segment may 2.7.2.2.  <ins> A descendant segment produces zero or more descendants of an input </ins> value.  <ins> For each node in the input nodelist, a descendant selector visits the input node and each of its descendants such that: </ins> nodes of any array are visited in array order, and be a result of applying the child segment "[<selectors>]" to the node "Di".  <ins> For each node in the input nodelist, the result of the descendant segment is the concatenation of "R1", ..., "Rn" (in that order). These results are then concatenated in input nodelist order to form the result of the segment. </ins> So a descendant segment drills down one or more levels into the  <ins> structure of each input value. </ins> 2.7.2.3.


"[{"j": 4}, {"k": 6}]" must appear before "{"j": 4}" and "{"k": 6}", <del> "{"j": 4}" must appear before "4", and </del>  <del> "{"k": 6}" must appear before "6". </del>  The example above with the query "$.o..[*, *]" shows that a selector may produce nodelists in distinct orders each time it appears in the

URL
This change looks good, thank you. I have one more, though (looking more closely at the diagnostics): I don't think the "Alternative result" can be generated, as outside arrays come before inside arrays.
Yes, nodes are visited before their descendants. Fixed in 1b85e02c7a5e60ba87ab1c455502c66816327f9d.
URL

"[{"j": 4}, {"k": 6}]" must appear before "{"j": 4}" and "{"k": 6}",  <ins> "{"j": 4}" must appear before "{"k": 6}", </ins>  <ins> "{"k": 6}" must appear before "4", and "4" must appear before "6". </ins> The example above with the query "$.o..[*, *]" shows that a selector may produce nodelists in distinct orders each time it appears in the


This section defines the extension point as well as four function extensions that use this extension point.  While these mechanisms are designed to use the extension point, they are an integral part of the <del> JSONPath specification and are mandatory to implement. </del>  A function extension defines a registered name (see iana-fnex) that can be applied to a sequence of zero or more arguments, producing a

Fix a right margin incursion, as well.
I'm commenting on the editor's draft. Section 2.6 says, of len(), count(), match(), and search(), that “they are an integral part of the JSONPath specification and are mandatory to implement.” My intuition is that this is not consistent with the WG charter. I think we're supposed to be specifying the GCD of JSONPath implementations, not enriching the language. The actual words of the charter are “based on the common semantics and other aspects of existing implementations”. Do len(), count(), match(), and search() qualify?
Most implications support regexps in filters, so we need to support those somehow. A function extension seems like the neatest solution.
Most implementations also support length in some way, usually , which we've decided against.
Guilty as charged. In that sentence I tried to express that, although the four functions make use of an extension point, they are not extensions of the base standard, as are none of the other features defined in the document. We don't define compliance, but we do fan the expectation that a JSONPath implementation will not leave out a feature that seems inconvenient to implement (say, descendants segments). If you have better wording, we could take that.
In all implementations there is some degree or level of compliance with the specification. No matter what the wording is in the specification the implementations will stand as they are and users will evaluate them based on how well or how compliant it is with the spec but more importantly how useful they are to them. In my implementation for go I have not yet implemented functions but do support regex with a . As this specification solidifies I'll also support the changes but in the interest of not breaking existing code that uses the OjG package I'll keep the original implementation as well.
NAME that's expected from most implementers, including myself. However, also as one of the editors, I would hope that implementors default to spec compliant behavior, offering other functionality behind options that the user must explicitly set. This is fairly common for spec implementations and what I will be doing if I decide to support extra-spec functionality.
I may not put it as eloquently as you do but I agree 100%. I hope I can contribute at some point.

This section defines the extension point as well as four function extensions that use this extension point.  While these mechanisms are designed to use the extension point, they are an integral part of the  <ins> JSONPath specification and are expected to be implemented like any other integral part of this specification. </ins> A function extension defines a registered name (see iana-fnex) that can be applied to a sequence of zero or more arguments, producing a


"LogicalType" is an abstraction of the result of a "logical-expr". Its two instances, "LogicalTrue" and "LogicalFalse", are not <del> related to the JSON literals "true" and "false" and have no </del>  syntactical representation in JSONPath. "NodesType" is an abstraction of a "filter-path" (which appears in a test expression or as a function argument).  Members of <del> "NodesType" have no syntactical representation in JSONPath. </del>  The abstract instances above can be obtained from the concrete representations in tbl-typerep.

There are some syntactical representations, e.g.: LogicalTrue can be represented by (1==1). LogicalFalse can be represented by (1==0). Nothing can be represented by a query that always returns the empty nodelist, such as $[?1==0]. But these are somewhat circuitous rather than direct. See URL
NAME I also considered deleting the statements about there being no (direct) syntactical representation of the values of certain types because I'm not sure they add much. The text in this PR adds an air of mystery - what are "direct syntactical representations"? WDYT?
I'm not sure how "direct" clarifies "syntactical representation." Either something can be represented in the syntax or it can't. Merge if you like.
This is obviously an improvement. We also could say "literals", but then we would assume knowledge about this programming language construct. So I think this should simply be merged.

"LogicalType" is an abstraction of the result of a "logical-expr". Its two instances, "LogicalTrue" and "LogicalFalse", are not  <ins> related to the JSON literals "true" and "false" and have no direct </ins> syntactical representation in JSONPath. "NodesType" is an abstraction of a "filter-path" (which appears in a test expression or as a function argument).  Members of  <ins> "NodesType" have no direct syntactical representation in JSONPath. </ins> The abstract instances above can be obtained from the concrete representations in tbl-typerep.


Note: "double-quoted" strings follow the JSON string syntax (RFC8259); "single-quoted" strings follow an analogous pattern <del> (syntax-index).  No attempt was made to improve on this syntax, so characters with scalar values above 0x10000, such as U+0026 U+0023 U+0031 U+0032 U+0039 U+0033 U+0030 U+0030 U+003B ("🤔", AMPERSAND, NUMBER SIGN, DIGIT ONE, DIGIT TWO, DIGIT NINE, DIGIT THREE, DIGIT ZERO, DIGIT ZERO, SEMICOLON), need to be represented by a pair of surrogate escapes (""uD83EuDD14"" in this case). </del>  2.5.1.2. <del> A "name-selector" string MUST be converted to a member name M by </del>  removing the surrounding quotes and replacing each escape sequence with its equivalent Unicode character, as in the table below: Applying the "name-selector" to an object node selects a member value <del> whose name equals the member name M, or selects nothing if there is </del>  no such member value.  Nothing is selected from a value that is not an object. Note that processing the name selector requires comparing the member <del> name string M with member name strings in the JSON to which the </del>  selector is being applied.  Two strings MUST be considered equal if and only if they are identical sequences of Unicode scalar values. In other words, normalization operations MUST NOT be applied to <del> either the member name string M from the JSONPath or to the member </del>  name strings in the JSON prior to comparison. 2.5.1.3.

Wouldn't mind if we could get these editorials merged quickly so the bulk of the other PRs shrinks a bit.
LGTM, thanks.

Note: "double-quoted" strings follow the JSON string syntax (RFC8259); "single-quoted" strings follow an analogous pattern  <ins> (syntax-index).  No attempt was made to improve on this syntax, so if it is desired to escape characters with scalar values above 0x10000, such as U+0026 U+0023 U+0031 U+0032 U+0039 U+0033 U+0030 U+0030 U+003B ("🤔", AMPERSAND, NUMBER SIGN, DIGIT ONE, DIGIT TWO, DIGIT NINE, DIGIT THREE, DIGIT ZERO, DIGIT ZERO, SEMICOLON), they need to be represented by a pair of surrogate escapes (""uD83EuDD14"" in this case). </ins> 2.5.1.2.  <ins> A "name-selector" string MUST be converted to a member name "M" by </ins> removing the surrounding quotes and replacing each escape sequence with its equivalent Unicode character, as in the table below: Applying the "name-selector" to an object node selects a member value  <ins> whose name equals the member name "M", or selects nothing if there is </ins> no such member value.  Nothing is selected from a value that is not an object. Note that processing the name selector requires comparing the member  <ins> name string "M" with member name strings in the JSON to which the </ins> selector is being applied.  Two strings MUST be considered equal if and only if they are identical sequences of Unicode scalar values. In other words, normalization operations MUST NOT be applied to  <ins> either the member name string "M" from the JSONPath or to the member </ins> name strings in the JSON prior to comparison. 2.5.1.3.