example, the Unicode PLACE OF INTEREST SIGN (U+2318) would be defined in ABNF as "%x2318". <del> The terminology of RFC8259 applies except where clarified below. Definitions for "Object", "Array", and "Number" remain unchanged. Importantly "object" and "array" in particular do not take on a generic meaning, such as they would in a general programming context. </del>  Additional terms used in this specification are defined below. value is anonymous, JSONPath uses the abstract name "$" to refer to the root node of the argument. <del> JSONPath expressions can use the _dot-notation_ </del>  <del> or the _bracket-notation_ </del>  for paths input to a JSONPath processor.  [1] Where a JSONPath processor uses JSONPath expressions as output paths, these will always be converted to Output Paths which employ the more general <del> _bracket-notation_.  [2] Bracket notation is more general than dot </del>  notation and can serve as a canonical form when a JSONPath processor uses JSONPath expressions as output paths. Firstly, "$" selects the root node which is the argument.  So the result is a list consisting of just the root node. <del> Next, ".a" selects from any input node of type object and selects any value of the input node corresponding to the member name ""a"".  The result is again a list of one node: "[{"b":0},{"b":1},{"c":2}]". </del>  Next, "[*]" selects from any input node which is an array and selects all the elements of the input node.  The result is a list of three nodes: "{"b":0}", "{"b":1}", and "{"c":2}". Finally, ".b" selects from any input node of type object with a <del> member name "b" and selects the value of the input node corresponding to that name.  The result is a list containing "0", "1".  This is the concatenation of three lists, two of length one containing "0", "1", respectively, and one of length zero. </del>  As a consequence of this approach, if any of the selectors selects no nodes, then the whole query selects no nodes. 3.5. <del> 3.5.1. </del>  <del> 3.5.1.1. </del>  <del> A dot child selector has a member name known as a dot child name or a single asterisk ("*"). </del>  <del> A dot child name corresponds to a name in a object. </del>  <del> More general child names, such as the empty string, are supported by "Union Child" (unionchild). </del>  <del> Note that the "dot-child-name" rule follows the philosophy of JSON strings and is allowed to contain bit sequences that cannot encode Unicode characters (a single unpaired UTF-16 surrogate, for example). The behaviour of an implementation is undefined for child names which do not encode Unicode characters. </del>  <del> 3.5.1.2. </del>  <del> A dot child name which is not a single asterisk ("*") is considered to have a member name.  It selects the value corresponding to the name from any object node.  It selects no nodes from a node which is not a object. </del>  <del> The member name of a dot child name is the sequence of Unicode characters contained in that name. </del>  <del> A dot child name consisting of a single asterisk is a wild card.  It selects all the values of any object node.  It also selects all the elements of any array node.  It selects no nodes from number, string, or literal nodes. </del>  3.5.2. 3.5.2.1. <del> A union selector consists of one or more union elements. </del>  3.5.2.2. <del> A union selects any node which is selected by at least one of the union selectors and selects the concatenation of the lists (in the order of the selectors) of nodes selected by the union elements. </del>  <del> 3.5.2.3. </del>  <del> 3.5.2.3.1. </del>  <del> A child is a quoted string. </del>  <del> Notes: 1. double-quoted strings follow JSON in RFC8259.  Single- quoted strings follow an analogous pattern.  2.  "HEXDIG" includes A-F and a-f. </del>  <del> 3.5.2.3.2. </del>  <del> If the child is a quoted string, the string MUST be converted to a member name by removing the surrounding quotes and replacing each escape sequence with its equivalent Unicode character, as in the table below: </del>  <del> The child selects the value corresponding to the member name from any object node that has a member with that name.  It selects no nodes from a node which is not a object. </del>  <del> 3.5.2.4. </del>  <del> 3.5.2.4.1. </del>  <del> An array selector selects zero or more elements of an array node.  An array selector takes the form of an index, which selects at most one element, or a slice, which selects zero or more elements. </del>  <del> An array index is an integer (in base 10). </del>  <del> Note: the syntax does not allow integers with leading zeros such as "01" and "-01". </del>  <del> An array slice consists of three optional integers (in base 10) </del>  separated by colons. <del> Note: the array slices ":" and "::" are both syntactically valid, as are ":2:2", "2::2", and "2:4:". </del>  <del> 3.5.2.4.2. </del>  <del> 3.5.2.4.2.1. </del>  This section is non-normative. The following section specifies the behaviour fully, without depending on JavaScript or Python behaviour. <del> 3.5.2.4.2.2. </del>  An array selector is either an array slice or an array index, which is defined in terms of an array slice. against an array whose size doesn't fit in the implementation's representation of an integer, the implementation MUST raise an error.  4. <del> TBD: Define a media type for JSONPath expressions. </del>  5.  This section gives security considerations, as required by RFC3552. <del> 6.  References </del>  <del> 6.1.  URIs </del>  [1] https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath

This is a full rebase of branch pr-98 () on current main.
Does this incorporate any changes from the discussions from , or is it a simple rebase?
It is a simple rebase, followed by a number of edits coming from the discussions here. Most of the remaining conversations need some further discussion, so I hope to cover them with my updated slides in a couple of hours.
Looks like a good next step on our path.

example, the Unicode PLACE OF INTEREST SIGN (U+2318) would be defined in ABNF as "%x2318".  <ins> The terminology of RFC8259 applies except where clarified below.  The terms "Primitive" and "Structured" are used to group the types as in Section 1 of RFC8259.  Definitions for "Object", "Array", "Number", and "String" remain unchanged.  Importantly "object" and "array" in particular do not take on a generic meaning, such as they would in a general programming context. </ins> Additional terms used in this specification are defined below. value is anonymous, JSONPath uses the abstract name "$" to refer to the root node of the argument.  <ins> JSONPath expressions can use the _dot notation_ </ins>  <ins> or the _bracket notation_ </ins> for paths input to a JSONPath processor.  [1] Where a JSONPath processor uses JSONPath expressions as output paths, these will always be converted to Output Paths which employ the more general  <ins> _bracket notation_.  [2] Bracket notation is more general than dot </ins> notation and can serve as a canonical form when a JSONPath processor uses JSONPath expressions as output paths. Firstly, "$" selects the root node which is the argument.  So the result is a list consisting of just the root node.  <ins> Next, ".a" selects from any input node of type object and selects the node of any member value of the input node corresponding to the member name ""a"".  The result is again a list of one node: "[{"b":0},{"b":1},{"c":2}]". </ins> Next, "[*]" selects from any input node which is an array and selects all the elements of the input node.  The result is a list of three nodes: "{"b":0}", "{"b":1}", and "{"c":2}". Finally, ".b" selects from any input node of type object with a  <ins> member name "b" and selects the node of the member value of the input node corresponding to that name.  The result is a list containing "0", "1".  This is the concatenation of three lists, two of length one containing "0", "1", respectively, and one of length zero. </ins> As a consequence of this approach, if any of the selectors selects no nodes, then the whole query selects no nodes. 3.5.  <ins> A JSONPath query consists of a sequence of selectors.  Valid selectors are </ins>  <ins> Root selector "$" </ins>  <ins> Dot selector ".<name>", used with object member names exclusively. </ins>  <ins> Dot wild card selector ".*". </ins>  <ins> Index selector "[<index>]", where "<index>" is either an (possibly negative) array index or an object member name. </ins>  <ins> Index wild card selector "[*]". </ins>  <ins> Array slice selector "[<start>:<end>:<step>]", where "<start>", "<end>", "<step>" are integer literals. Nested descendants selector "..". Union selector "[<sel1>,<sel2>,...,<selN>]", holding a comma delimited list of index, index wild card, array slice, and filter selectors. </ins>  <ins> Filter selector "[?(<expr>)]" </ins>  <ins> Current item selector "@" 3.5.1. 3.5.1.1. </ins>  <ins> Every valid JSONPath query MUST begin with the root selector "$". 3.5.1.2. The Argument -- the root JSON value -- becomes the root node, which is addressed by the root selector "$". </ins> 3.5.2. 3.5.2.1.  <ins> A dot selector starts with a dot "." followed by an object's member name. Member names containing other characters than allowed by "dot- selector" -- such as space ` ` and minus "-" characters -- MUST NOT be used with the "dot-selector".  (Such member names can be addressed by the "index-selector" instead.) </ins> 3.5.2.2.  <ins> The "dot-selector" selects the node of the member value corresponding to the member name from any JSON object.  It selects no nodes from any other JSON value. Note that the "dot-selector" follows the philosophy of JSON strings and is allowed to contain bit sequences that cannot encode Unicode characters (a single unpaired UTF-16 surrogate, for example).  The behaviour of an implementation is undefined for member names which do not encode Unicode characters. </ins>  <ins> 3.5.3. </ins>  <ins> 3.5.3.1. </ins>  <ins> The dot wild card selector has the form ".*". </ins>  <ins> 3.5.3.2. </ins>  <ins> A "dot-wild-selector" acts as a wild card by selecting the nodes of all member values of an object as well as all element nodes of an array.  Applying the "dot-wild-selector" to a primitive JSON value (number, string, or true/false/null) selects no node. </ins>  <ins> 3.5.4. </ins>  <ins> 3.5.4.1. </ins>  <ins> An index selector "[<index>]" addresses at most one object member value or at most one array element value. </ins>  <ins> Applying the "index-selector" to an object value, a "quoted-member- name" string is required.  JSONPath allows it to be enclosed in _single_ or _double_ quotes. </ins>  <ins> Applying the "index-selector" to an array, a numerical "element- index" is required.  JSONPath allows it to be negative. </ins>  <ins> Notes: 1. "double-quoted" strings follow JSON in RFC8259; "single- quoted" strings follow an analogous pattern.  2.  An "element-index" is an integer (in base 10, as in JSON numbers).  3.  As in JSON numbers, the syntax does not allow octal-like integers with leading zeros such as "01" or "-01". </ins>  <ins> 3.5.4.2. </ins>  <ins> A "quoted-member-name" string MUST be converted to a member name by removing the surrounding quotes and replacing each escape sequence with its equivalent Unicode character, as in the table below: The "index-selector" applied with a "quoted-member-name" to an object selects the node of the corresponding member value from it, if and only if that object has a member with that name.  Nothing is selected from a value which is not a object. Array indexing via "element-index" is a way of selecting a particular array element using a zero-based index.  For example, selector "[0]" selects the first and selector "[4]" the fifth element of a sufficiently long array. A negative "element-index" counts from the array end.  For example, selector "[-1]" selects the last and selector "[-2]" selects the last but one element of an array with at least two elements. 3.5.5. 3.5.5.1. The index wild card selector has the form "[*]". 3.5.5.2. An "index-wild-selector" selects the nodes of all member values of an object as well as of all elements of an array.  Applying the "index- wild-selector" to a primitive JSON value (such as a number, string, or true/false/null) selects no node. The "index-wild-selector" behaves identically to the "dot-wild- selector". 3.5.6. 3.5.6.1. The array slice selector has the form "[<start>:<end>:<step>]".  It selects elements starting at index "<start>", ending at -- but not including -- "<end>", while incrementing by "step". The "slice-selector" consists of three optional decimal integers </ins> separated by colons.  <ins> 3.5.6.2. </ins>  <ins> The "slice-selector" was inspired by the slice operator of ECMAScript 4 (ES4), which was deprecated in 2014, and that of Python. </ins>  <ins> 3.5.6.2.1. </ins> This section is non-normative. The following section specifies the behaviour fully, without depending on JavaScript or Python behaviour.  <ins> 3.5.6.2.2. </ins> An array selector is either an array slice or an array index, which is defined in terms of an array slice. against an array whose size doesn't fit in the implementation's representation of an integer, the implementation MUST raise an error. <ins> 3.5.7. 3.5.7.1. The descendant selector starts with a double dot ".." and can be followed by an object member name (similar to the "dot-selector"), by an "index-selector" acting on objects or arrays, or by a wild card. 3.5.7.2. The "descendant-selector" is inspired by ECMAScript for XML (E4X). It selects the node and all its descendants. 3.5.8. 3.5.8.1. The union selector is syntactically related to the "index-selector". It contains multiple, comma separated entries. Task (T1): This, besides slice-index, is currently one of only two places in the document that mentions whitespace.  Whitespace needs to be handled throughout the ABNF syntax.  Room Consensus at the 2021-06-15 interim was that JSONPath generally is generous with allowing insignificant whitespace throughout.  Minimizing the impact of the many whitespace insertion points by choosing a rule name such as "S" was mentioned.  Some conventions will probably help with minimizing the number of places where S needs to be inserted. 3.5.8.2. A union selects any node which is selected by at least one of the union selectors and selects the concatenation of the lists (in the order of the selectors) of nodes selected by the union elements. Note that any node selected in more than one of the union selectors is kept as many times in the node list. 3.5.9. 3.5.9.1. The filter selector has the form "[?<expr>]".  It works via iterating over structured values, i.e. arrays and objects. During iteration process each array element or object member is visited and its value -- accessible via symbol "@" -- or one of its descendants -- uniquely defined by a relative path -- is tested against a boolean expression "boolean-expr". The current item is selected if and only if the result is "true". Notes: Parentheses can be used with "boolean-expr" for grouping.  So filter selection syntax in the original proposal "[?(<expr>)]" is naturally contained in the current lean syntax "[?<expr>]" as a special case. Comparisons are restricted to primitive values (such as number, string, "true", "false", "null").  Comparisons with complex values will fail, i.e. no selection occurs. Types are not implicitly converted in comparisons.  So ""13 == '13'"" selects no node. A member or element value by itself is _falsy_ only, if it does not exist.  Otherwise it is _truthy_, resulting in its value.  To be more specific explicit comparisons are necessary.  This existence test -- as an exception of the general rule -- also works with complex values. Regular expression tests can be applied to "string" values only. Containment tests work with arrays and objects. Explicit boolean type conversion is done by the not operator "neg- op". The behaviour of operators is consistent with the 'C'-family of programming languages. 3.5.9.2. The "filter-selector" works with arrays and objects exclusively.  Its result might be a list of _zero_, _one_, _multiple_ or _all_ of their element or member values then.  Applied to other value types, it will select nothing. Negation operator "neg-op" allows to test _falsiness_ of values. Applying negation operator twice "!!" gives us _truthiness_ of values. Some examples: </ins> 4.  <ins> Task (T2): Separate out expression language.  For now, this section is a repository for ABNF taken from RFC8259.  This needs to be deduplicated with definitions above. </ins> 5. <ins> TBD: Define a media type for JSONPath expressions. 6. </ins> This section gives security considerations, as required by RFC3552.  <ins> 7.  References </ins>  <ins> 7.1.  URIs </ins> [1] https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath


JSONPath query, e.g., because of resource depletion, but this is not modeled in the present specification.) <del> Discussion (D1): are we ready for the error model defined here, i.e. one that does not have runtime errors, but only compile time errors? </del> 3.2. In this specification, the semantics of a JSONPath query are defined

Here's a chance to actually have the discussion if anyone wants to.
Well, I'm certainly ready for this model.
I'm good with the approach described in the text. As long as a string can be parsed as a Path, it should evaluate. I can't see a case where processing a valid path against a value would produce an error; merely empty result sets. This means that the only errors should be syntax errors, correct?
I think so.
May I encourage reviewers who are in favour of this PR to approve it?
I was awaiting a bit more discussion.
Ok, I've requested a review from Stefan.
Yes absolutely ... I like that clean concept a lot.

JSONPath query, e.g., because of resource depletion, but this is not modeled in the present specification.)  3.2. In this specification, the semantics of a JSONPath query are defined


For each node in the list, the selector selects zero or more nodes, each of which is a descendant of the node or the node itself. <del> Discussion (D4): Is that a common requirement?  Or can a selector also go up, or to the query argument? </del> For instance, with the argument "{"a":[{"b":0},{"b":1},{"c":2}]}", the query "$.a[*].b" selects the following list of nodes: "0", "1" (denoted here by their value).  Let's walk through this in detail.

Here's a chance to actually have the discussion if anyone wants to.
One related question is whether the selector can use $ internally (still returning nodes within the current node).
I am in favour of supporting inside filter expressions as a way to access the root node of the argument, but I don't think that should block this PR as the context of the discussion seems to be which nodes are selected. As you point out, using in a filter expression doesn't select the root node - it selects nodes within the current node as usual.
I agree. Paths like should be allowed, but wouldn't select anything that is not a descendent of . Here the provides a basis for comparison used to select the current node in the iteration. I see no value (for an initial release anyway) in allowing selection of ancestor nodes. If a use case and syntax for it are later defined, I'm happy to release that as a follow-up version.
yes ... using in a logical expression is valid and does no harm.
NAME yes, but the issue at hand here is whether there is a mechanism that can select ancestor or cousin nodes. I think we're in agreement that that there is no such mechanism.
Indeed, we seem to be in agreement.

For each node in the list, the selector selects zero or more nodes, each of which is a descendant of the node or the node itself.  For instance, with the argument "{"a":[{"b":0},{"b":1},{"c":2}]}", the query "$.a[*].b" selects the following list of nodes: "0", "1" (denoted here by their value).  Let's walk through this in detail.


Containment tests work with arrays and objects. <del> Explicit boolean type conversion is done by the not operator "neg- op". </del> The behaviour of operators is consistent with the 'C'-family of programming languages.

Fixes URL
Thanks for the various comments. I've been busy with other things, but intend to come back before too long, absorb those comments, and push up some changes.
Yes. How is that "little response”? I’m still trying to parse what you mean by "This was asked earlier and received little response.”. What are you trying to say? Grüße, Carsten
Ok, so I pushed up some changes which attempt to address all the issues raised. Please see the individual commit logs for more detailed comments.
Maybe we want to add an [operator precedence table][1] into the text and try to make sure the grammar is somewhat congruence to that table. [1]: URL
Good idea, but I'd prefer to defer as I've got enough to deal with in this PR! It will also be even more valuable if we end up supporting arithmetic expressions...
Having an operator precedence table will help us understand what we are trying to do here. But I believe in incremental improvement, so completing this PR and then re-approaching it from the angle of the precedence table works for me.
NAME Please approve if your concerns are now addressed.
I've started the next batch of predicate improvements in draft PR URL which is based on this PR.
Currently, filter comparison require a path on the left hand side and a constant expression on the right hand side. This seems overly restrictive. We should generalise it to allow the opposite order as well as path to path comparisons and, for completeness, constant expression to constant expression comparisons. For example, the following comparisons should be valid:
Should be symmetric, too?
Did you mean (which is what we use in the current draft)? I think not because its operands have distinct types and it's probably easier to police restrictions and give good diagnostics if we keep asymmetric. For instance, with the current syntax: can fail with an error such as "NAME needs to be a regular expression literal", which is a bit easier to understand than something like "one side of ~= needs to be a regular expression literal". Having said that, other generalisations of may be worth it, such as allowing a string literal (rather than only a path) on the left hand side, for consistency with the above syntax for the other comparison operators, but even that could be give rise to confusion, e.g.:
Oops. It really should be . Perl introduced as (where regexp actually could be a substitute operator etc.). Ruby generalized this to and (actually, even more general...). Java, JavaScript, Python, PowerShell don't have an operator at all; this is all handled using method calls. So if we want to have syntax for regexp matching, we probably want to go with Perl's. We also need to discuss whether we need syntax for regexp literals. But this is maybe leading beyond this github issue.
When witing the current node represented by will be selected. So resulting in the same as . Is this really, what we want?
Anything else we might want? (I'm not sure we can write a syntax that only lets "meaningful" expressions through.)
Convinced ... you made the point.
The filter syntax currently requires predicates to be expressed in (something close to) disjunctive normal form (OR of ANDs), but this is likely to be inconvenient when the intention of the filter is expressed in a different way (e.g. as an AND or ORs or as a nested predicate with ANDs and ORs at various levels). The JSONPath comparison project doesn't yet test these more complex predicates, although some, possibly most, implementations allow them. The standard could allow arbitrarily structured predicates via syntax along these lines: For example, this would allow filters such as: and (where the semantics could, in line with mathematical convention and many programming languages, define to bind more tightly than ). Note: the comparison project's restricts the syntax so that brackets are required_ instead of making bind more tightly than . This is a clever approach but is probably not suitable for the standard because it is rather unconventional.
I don't have a problem with requiring parentheses instead of creating an elaborate precedence hierarchy (or at least partially instead of). I also don't have a problem with arbitrary nesting here.
For clarity, this is because is a multiplicative operation whereas is additive. Standard order of operations then apply. I don't know any programming language where this is reversed. I'm not opposed to requiring , however.
If the fact that binds more tightly than must be made explicite in the ABNF, it should be done. Using pairs of parentheses should be arbitrary.
Agreed. Is there a preferred way to specify operator precedence in ABNF? The examples I've seen use a textual ordered list which won't help parser generators...
The doesn't use ANBF, but it is pretty clear by using prose and a table. See section 12.4.2. There's also a bit more about associativity, but I figured this was enough.
Interesting. Does anyone know if ABNF parser generators infer operator precedence from the structure of the productions?
I would venture to say that is a near-universal property of BNF-based parser generators. Strictly speaking, BNF provides only a Boolean output: word is in language or it is not. But in reality, people use BNF for parsing, and build the parse tree along the structure of the BNF productions that was used for matching. That may be ambiguous even when the language created by the BNF isn’t, and that’s one reason why we have all these variants like LL(1) and LR(k) etc. The parse tree is then used to build an “abstract syntax tree”, ast, in many cases by attaching actions to the productions in the grammar. Grüße, Carsten

Containment tests work with arrays and objects.  The behaviour of operators is consistent with the 'C'-family of programming languages.


The behaviour of operators is consistent with the 'C'-family of programming languages.  3.5.9.2. The "filter-selector" works with arrays and objects exclusively.  Its

Generalise comparisons Make regex operator conform to Perl etc. Add operator precedence table Fixes URL Fixes URL Fixes URL
I think having the precedence table now, above remark is obsolete.
Thanks NAME I raised URL to remove that remark.
Perl introduced as (where regexp actually could be a substitute operator etc.). Ruby generalized this to and . (Java, JavaScript, Python, PowerShell don't have an operator at all; this is all handled using method calls.) So if we want to have syntax for regexp matching, we probably want to go with Perl's. Originally posted by NAME in URL
Introducing an operator precedence table will help us comunicate what we are trying to achieve precedence-wise in the ABNF syntax. Originally posted by NAME in URL
Here is a first version ... taken from
Thanks for that NAME I suggest we combine the third and fourth rows and give and the same precedence as the other comparison operators as I don't think there is a situation where their relative precedence is necessary. Also, having a separate row for and gives the impression we want to entertain expressions such as: and which I'd prefer to avoid
Yes ... so ...
Currently, filter comparison require a path on the left hand side and a constant expression on the right hand side. This seems overly restrictive. We should generalise it to allow the opposite order as well as path to path comparisons and, for completeness, constant expression to constant expression comparisons. For example, the following comparisons should be valid:
Should be symmetric, too?
Did you mean (which is what we use in the current draft)? I think not because its operands have distinct types and it's probably easier to police restrictions and give good diagnostics if we keep asymmetric. For instance, with the current syntax: can fail with an error such as "NAME needs to be a regular expression literal", which is a bit easier to understand than something like "one side of ~= needs to be a regular expression literal". Having said that, other generalisations of may be worth it, such as allowing a string literal (rather than only a path) on the left hand side, for consistency with the above syntax for the other comparison operators, but even that could be give rise to confusion, e.g.:
Oops. It really should be . Perl introduced as (where regexp actually could be a substitute operator etc.). Ruby generalized this to and (actually, even more general...). Java, JavaScript, Python, PowerShell don't have an operator at all; this is all handled using method calls. So if we want to have syntax for regexp matching, we probably want to go with Perl's. We also need to discuss whether we need syntax for regexp literals. But this is maybe leading beyond this github issue.
When witing the current node represented by will be selected. So resulting in the same as . Is this really, what we want?
Anything else we might want? (I'm not sure we can write a syntax that only lets "meaningful" expressions through.)
Convinced ... you made the point.
This looks good. Just some curiosity questions from me.

The behaviour of operators is consistent with the 'C'-family of programming languages. <ins> The following table lists filter expression operators in order of precedence from highest (binds most tightly) to lowest (binds least tightly). </ins> 3.5.9.2. The "filter-selector" works with arrays and objects exclusively.  Its


not exist.  Otherwise it is _truthy_, resulting in its value.  To be more specific explicit comparisons are necessary.  This existence test -- as an exception of the general rule -- also <del> works with complex values. </del>  Regular expression tests can be applied to "string" values only. <del> Containment tests work with arrays and objects. </del>  The following table lists filter expression operators in order of precedence from highest (binds most tightly) to lowest (binds least

URL
After thinking a while about comparing structured types, I now see no problem to support equality of them. I agree to handling them is straightforward: also see ... not exactly, what we want though. We need to be clear about order ... ! Containables may be defined on exactly that comparison rule. I believe, we need to add objects here again. Also we need to discuss subsets. Once having agreement about handling that edge cases (maybe there are more), we might add it to the spec. Btw.: JavaScript resolves all comparisons above to , since objects are equal only, if they are referencing the same object (no deep equals).
Comments inline. ordering. surprise and so is likely to cause confusion. Also I am not aware of any precedent for such conflation in mathematics or programming language semantics. [1,2] in [[1,2]] // structured value membership Also, we need to discuss ordering of arrays. For example: [1,2] in [[2,1]] should be false, if we agree that [1,2] != [2,1]. I don't think we should conflate membership and subset. Since "in" has a stronger connotation of membership than subset, I don't think we can easily define "in" with an object on the right hand side. Note also that there isn't yet a notation for a mathematical "maplet", but if we represented it as a pair, e.g. ('x',1), then we could test membership of objects thus: ('x',1) in {x:1} (However, I don't think this is worth the cost of increased complexity of the spec.)
I totally support a decision to: not dealing with subsets restrict the rhs to arrays only with contain-expr's.
We decided not to have " in ", so this is now closed/revisit-after-base-done.
[ ] define a production , replacing [ ] go through ABNF and make sure blank space is allowed where it needs to be (Do not use the term "whitespace" any more.) Task "T1" in the document URL
and
While this may be a bit premature as long as we don't know what the rhs of contain-expr should be, it is missing its lhs. What is in the container?
Oh, and the same with :
Well ... good point. Starting with these examples: lhs is restricted to a primitive value (possibly only number and string). lhs holding a complex value will fail, i.e. no selection occurs. rhs must be resolved to a complex value (array or object). rhs holding a simple value will fail, i.e. no selection occurs. Given this we have something like this ... ... as a starting proposal.
(From 2021-09-17 interim:) So we would have syntax for array literals composed of primitive values only (subset of JSON array literals); other JSON array literals cause a syntax error. (Empty array literals should be allowed to simplify code generation.) A LHS evaluating to a complex value would not match. The path on the RHS may also yield an array, we would match against every single element of that like we would match against a literal array. Example: tending towards consensus on: if $.colors is an array of structured values, NAME is a single structured value? only match primitive values from the LHS (counterintuitive!?) = [128, 255, 17] no selection even if = [[128, 255, 17], [1, 2, 3]] (we cannot avoid structured data to turn up in a query, but we don't match) Avoid deep equals! (We already avoid it in simple comparisons.)
Reminder: lhs of ... From to as a proposal.
Fixed in 9da13ab4 (Still needs resolution of to become active.)
Let's merge this so we have a better base for discussion.

not exist.  Otherwise it is _truthy_, resulting in its value.  To be more specific explicit comparisons are necessary.  This existence test -- as an exception of the general rule -- also  <ins> works with structured values. </ins> Regular expression tests can be applied to "string" values only.  <ins> The value of the first operand ("containable") of a "contain-expr" is compared to every single element of the RHS "container".  In case of a match a selection occurs.  Containment tests -- like comparisons -- are restricted to primitive values.  So even if a structured "containable" value is equal to a certain structured value in "container", no selection is done. The value of the second operand ("container") of a "contain-expr" needs to be resolved to an array.  Otherwise nothing is selected. </ins> The following table lists filter expression operators in order of precedence from highest (binds most tightly) to lowest (binds least


member name ""a"".  The result is again a list of one node: "[{"b":0},{"b":1},{"c":2}]". <del> Next, "[*]" selects from any input node which is an array and selects all the elements of the input node.  The result is a list of three </del>  nodes: "{"b":0}", "{"b":1}", and "{"c":2}". Finally, ".b" selects from any input node of type object with a

... and some minor edits ...
Indeed. I'll do a small PR with my points.
a little late to the party ... thanks
[ ] define a production , replacing [ ] go through ABNF and make sure blank space is allowed where it needs to be (Do not use the term "whitespace" any more.) Task "T1" in the document URL
and
From the document: (whitespace is now generally called blank space, but that doesn't change the work to be done.)
and
How much do we want to reference XPath within the spec? I understand that XPath is the inspiration (and in the tagline that NAME used in his blog post: "XPath for JSON"), but I wonder if much of this can be left out of the spec. The argument for this is that we're defining JSONPath. It shouldn't assume any understanding of XPath or any other technology. It's not the job of the specification to convince readers to use it. It's the job of the specification to define the technology. I'm perfectly happy moving this into an external "Understanding JSONPath" website at some point.
I seem to remember that last time we discussed this we agreed to move the introductory comparison with XPath to an appendix, with the possible exception of the overview table (but that could be duplicated).
The XPath inclusion was moved into an appendix.
Do we allow spaces (outside of string literals)? If so, where is whitespace allowed? I think whitespace in query expressions should be allowed. This would fall in to the context of the . Maybe inside the union/ selector: Are there any other places where whitespace could be considered useful?
It may be convenient to allow spaces between matchers - to allow well-readable multi-line queries for complex cases.
Looks like the allows whitespace internally, so that (with leading and trailing whitespace) is valid.
Closed in and
What was the resolution here? Where did we end up allowing whitespace?
This PR could be merged to make progress. Further changes as indicated by NAME are probably needed, either now or in a subsequent PR.

member name ""a"".  The result is again a list of one node: "[{"b":0},{"b":1},{"c":2}]".  <ins> Next, "[*]" selects from any input node -- either an array or an object -- all its elements or members.  The result is a list of three </ins> nodes: "{"b":0}", "{"b":1}", and "{"c":2}". Finally, ".b" selects from any input node of type object with a


The "descendant-selector" selects the node and all its descendants.  3.5.8. 3.5.8.1.

Fixes URL
At today's meeting, I agreed to submit a PR to float the idea of partial determinism for the descendants operator, even though the consensus seems to be against any determinism for that operator. Given the descendants operator PR URL isn't yet merged, I'll start with an issue to capture discussion and avoid rebasing effort. We probably don't want to force any determinism on implementations, so the idea is to provide a recommendation for implementations which want to provide partial determinism and use "SHOULD" language so the recommendation is optional. Then implementations which opt in will agree on the ordering recommentations. The benefits of partial determinism are: improved predictability and interoperation relative ease for JSONPath users writing tests, as there are fewer possible outputs. In PR URL there are no constraints on the non-determinism of the descendants operator: We agreed in issue URL to allow non-determinism in the order in which objects are iterated and the same should be true when the descendants operator encounters an object. But there are other details of the ordering which could be tied down: arrays SHOULD be visited in array order nodes SHOULD be visited strictly before their descendants (so-called pre-order traversal). (Post-order traversal, in which nodes are visited strictly after their descendants, would be an equally valid alternative.) One of the reasons given at today's meeting for allowing full non-determinism was for parallel processing, so it would be interesting to discuss what constraints the above rules produce on parallel processing. The order of collection of the results seems to be more significant than whether or not parallel processing is used.
My preference is pre-order traversal (node then descendants).
Except that I don't know what's "SHOULD" about that, I agree with these partial ordering guidelines. If we want to introduce different compliance levels, that is fine, but it SHOULD :-) not be coupled to using the term SHOULD vs. MUST. "Full compliance requires..." vs. "For conditional compliance, at least..."
I probably misunderstood , or applied it atypically. I am comfortable with using NAME compliance level language instead.
We need to be careful here that we don't optimize for tests. It seems to be a focus of this discussion that we want to make verification of output in the test suite easier. Rather, we need to focus on supportability and interoperability. If that aligns with running the test, that's good. But don't let making running the tests easier at the cost of making things significantly more difficult for implementors. It's non-trivial, but also not very complex, to verify unordered sets of nodes in output.
More significant than complications for the JSONPath test suite is the amount of non-determinism that tests of code which uses JSONPath will have to accommodate.
I recommend checking out the language around interoperability in 8259. Something like that might work here.
I'm having a hard time judging the consensus, or lack of it, here. Regardless of the detailed language, please could interested parties indicate which of these options they favour: A. Don't tie down the non-determinism of at all B. Require to visit arrays in array order and visit nodes before their descendants C. Some other option involving B with the option of A or A with the option of B, using some kind of language TBD. /cc NAME
I’m not sure you actually mean that, but for me that would be about the worst result we could arrive at. 8259 was worked on in a politicized context, which simply didn’t allow us to clean it up. There barely was the courage to finally say that interchanging JSON implies UTF-8, which was at least some progress. But there was no way to clarify the data model issues, and this is why JSON in practice essentially has been reduced to I-JSON, which is not bad at all, but really different from what JSON originally promised to be. I’d prefer if we don’t need an I-JSONPath document to clean up the interoperability issues left in JSONPath-base. Given that we have JSON and I-JSON, it may be useful to acknowledge (or violently reject! — we get to choose) the emergence of multiple levels/classes of interoperability with JSONPath, too. Grüße, Carsten
I would vote for this option, but in the same time we have slices with negative steps, which naturally do sort of "backwards" iteration in Python. - while there's no clear consensus, we see that all implementations with "python-compatible" slices return values in reverse order, no exceptions. If we determine order of array items visiting (thus determining the order of selected nodes in output), shouldn't we also take into consideration the effect of selectors, and if yes - then should the order of union items affect the order of visit? We can consider part of selectors as "order neutral" and others are "order affecting", with affecting selectors overriding natural order. At first glance I see slices and unions as order-affecting selectors.
NAME I don't follow. The ordering of is not affected by other selectors (e.g. involving negative array steps or unions).
This is not absolutely clear for me. Let us have the following structure: And we apply query. At first we apply selector to the root node, and it returns following nodes: : : : : : : : : : : : Then we apply selector to each these nodes. Applying slice to number will return empty node list, so I consider only arrays below: : : : : : : : : : : : : : The resulting node list is: : : : : : : : : : : I didn't notice any problems following both "natural order" while applying and "reversed order" while applying , because they never apply simultaneously. Please show me if I missed something in my reasoning.
The idea of determinism in arrays itself seems important to me. For example, let some service provides array of some data in some garanteed order that is important for end user, and and another service wants to use JSONPath to filter them on some fancy condition. If JSONPath provides determined order of results, then it's at least possible to solve the problem with a single query. If no - we'll have no easy way to do it. We will need to extract from JSON original order somehow (with alternative tool or manual parsing), and then use it to rebuild original order of filtered items. It will be much more convenient to receive results in a determined order.
The main aspect of JSON that leads to non-determinism is the fact that no specification makes any guarantee that the order of Members in an Object carries any meaning. RFC8259, last para of Section 4 , explicitly cautions against relying on this. On the other hand, order in arrays is explicitly significant and I would be shocked if application of array-based selectors did not produce identical results in all implementations. Or am I missing something obvious?
Yes, I think we agree that array order should be preserved and that object order is not guaranteed, but I'd like this issue to focus specifically on the ordering of .
Added text indicating node-first / children-second ordering.
... and preserving array order.
Sure. That too. But I don't think that was ever in question.
Does applying a JSONPath query to a JSON data item always generate the same result? Some selectors do not have a natural order of the nodelists they return, e.g., $. applied to a JSON object We could prescribe an order (e.g., sorting keys lexicographically or in JavaScript order) if determinism is important We could disallow . on JSON objects
hmm .. depends, how deterministic JSON parsers are when iterating over objects ... they should work in ascending chronological order ... URL JavaScript 'URL' also guaranties the same order URL So ... order should be always the same, as long the implementation is the same. How different implementations behave is uncertain at best. But why should we care and heal deficiencies of JSON parsers in that case? I can't see a benefit of always guaranteeing the same -whatever- order. And disallowing iterating over objects by '$.*' might break existing implementations.
A minor benefit would be that the Compliance Test Suite would be simpler if the output always had a defined implementation-independent ordering. Agreed.
(Determinism in interop testing is a major benefit for me. But that can be added by reprocessing after the JSONpath processing, so it is not completely a killer argument.)
Another selector which requires careful definition for ordering is , particularly when this recurses over an object or objects.
hmm ... compliance testing is a point, sure. Simple first tests of and regarding ordering should be possible with current test suite ... ?
Anything is "possible". ;-) Tests involving objects which have only one name/value pair are trivially easy to test because any non-determinism of object enumeration is irrelevant. But tests involving objects with two or more name/value pairs need to capture the ordering possibilities somehow. Note that the is a JSON document so there would need to be a declarative way of specifying the ordering possibilities.
We covered this topic over in the JSON Schema repo somewhere with quite a lengthy discussion. I'll see if I can dig it up. In short, parsers in some languages just can't be made to be deterministic. Reading the same object from a file (where the keys are certainly in a defined sequence) will yield differing orderings when printing or even debugging the value. This topic, therefore, falls into the "language agnosticity" category. I don't think it's something we can enforce.
is accepted (and correctly evalueted) by ~93% of the implementations on the . Each of them has a number of users, it is really important to not disrupt their existing expressions and their habits. I don't think determinism is that important, on a object evals to an array but semantically it is an unordered set (which is not a JSON type, therefore a different type is used ¹) which is still a meaningful result and developers should remember that. Hence we shouldn't discard this feature. Anyway what I found confusing when implementing was the lack of any official recommendation about on a object behaviour. I think we should clearly state that the result has to be treated as unordered but we might suggest using lexicographic order for validation purposes when possibile (I think a huge number of implementations are using ordered maps for JSON objects storage so it shouldn't be hard). Removing an implemented fature used from a number of users for testing purposes is something that I wish to not discuss. I think a number of JSONPath implementations are completely decoupled from the JSON parser, and they just accepted maps/lists/atomic-values as input ( is one of them, it doesn't depend on any JSON parser and it doesn't care about JSON parsing). I'm not even sure if JSONPath is only used for JSON inputs, it works so nicely with any JSON-like structure. ¹ I'm not 100% sure that all implementations are using JSON types, there might be implementations using unordered sets for real as containers for results.
A noble goal, to be sure. But I imagine that sudden conformance to a spec would natural engender breaking changes, especially in a landscape of Implementation as large as we're dealing with. I don't think it'll be avoided. As such, my feeling is that we get the spec right rather than make concessions just because the majority does it some way. Absolutely correct.
We have some of this in our DNA, but we also have a clear mandate not to disregard the installed base (this is JSONpath, not a new protocol based on JSONpath). So any argument on this spectrum will need to be carefully weighed, because both extremes are wrong. Grüße, Carsten
I think the summary so far is We recognize that there are operations in JSONPath whose result is ordered but that order needs to be created based on unordered input. We are not going to cut off these noses to spite our face. We have a choice between imposing a language-defined order (e.g., sorting) or leaving the result non-deterministic. I gather that there is a general direction toward non-determinism, as imposing an order would be too expensive, but I'm not sure we have a conclusion here. Also:
I agree with Carsten's general direction in support of nondeterminism, simply because 8259 explicitly discourages assigning semantics to the ordering of object members. If we want determinism, we probably have to specify RFC8785 (JCS) conformance, which doesn't feel reasonable. As for testing, anyone who's doing this seriously in the real world has to have an URL() method already, right?
I also agree with supporting non-determinism, but please note that URL() won't necessarily help because the non-deterministic portions of the output won't necessarily be part of a JSONObject. For example, applying to an input consisting of a JSON object will produce a node list in a non-deterministic order. I think we'd either need to decorate such tests with an indication of where non-determinism can occur so that each test framework can compare the output to the expected value and take into account non-determinism or we'd need to have a way of capturing a set of possible alternative outputs. The latter may be preferable as it would complicate the generation of tests, which is a one-off process, but would simplify implementing test frameworks, of which there are likely to be many. (There may be other solutions too - haven't spent very long on this.)
FWIW, in implementing JSON Schema, I've found that I need both contents-equality and sequence-equality on arrays depending on the circumstance. Given that JSON Path outputs an array, and the contents of that array are likely to be generated non-deterministically due to evaluating objects, I think it makes sense for implementations to verify evaluation results (e.g. when processing a test suite) in a "contents-equality" sense. It's not per the JSON definition of array equality (which mandates sequence-equality), but I think the circumstance fits. The fact that the result array is not guaranteed to be deterministic would need to be declared in the specification, probably associated with a remark about it deviating from JSON array equalilty and perhaps even an explanation why.
Nodelists are not arrays. (They can be represented in a JSONPath API as arrays. Or as lists. Or as tuples. ...)
I do agree with common non-determinism here. Naming results as nodelists is clever. Shouldn't we define that term 'nodelist' then ?
Yes, we should define nodelist. I made a first attempt in PR now.
We need to tread carefully here. Are we saying we're comfortable outputting something that's not JSON but acts really similar? All implementations (including NAME output an array (or optionally ) as JSON. If we suddenly call the output a "nodelist" are we breaking that?
No. I agree that many APIs will output platform arrays here (which may or may not feel similar to JSON arrays). Saying that the output is a nodelist is gives them freedom to define their API and how these present the nodelist. More importantly, we make processing for embedded queries less finicky.
Daniel, nobody can (or will try to) stop you thinking about a nodelist as a JSON array, if that suits your mental model. Giving a separate name to that concept does have its advantages. It, for instance, helps finding answers to the role of nested queries in filter expressions. It also helps reflect the fact that some implementations handle nodelists different from others. So I don't know why we are having this discussion. (And why is it under "Determinism?".)
This is irrelevant because of many of the points that NAME mentioned: essentially, implementations don't use nodelists from what we can tell, they use arrays. And isn't our charter to break existing implementations as little as possible? If all implementations use arrays, changing that is not something we should consider lightly. NAME We're discussing it here because of . Following some discussion about expressions, you spammed "nodelist" comments anywhere it might be related in order to promote the idea before anyone had a chance to discuss it. So now we're discussing it. All of that said, I think we should consider the output format ( & ), determinism (this issue), and paths embedded in expressions () together. There is some subtle interplay going on. If the output is an array of objects that specify location and value, then both that format and the sequence of the items within it need to be considered when returning from an embedded path so that the remainder of the expression can handle it properly. If embedded paths are to follow some different output that operators can then use, we need to specify what that is as well. Maybe that is the idea of a nodelist. But we need to discuss it first and come to some level of agreement.
It is a pretty common technique in model building to identify important concepts, and once one has identified them, to give them (tentative) names. That allows the participants to refer to the concept-in-formation, and to gradually increase certainty about it. Once there is enough certainty, it may be possible to identify some concepts with each other and turn them into one, but you don't do that before achieving that certainty. I think much of the confusion comes from a perception that whatever is in the draft at one time is cast in concrete. That is not a healthy way to run this, and maybe I was making an unwarranted assumption that this would be obvious. So, indeed, feedback is good, but not so much about reasons wanting to eliminate/equate concepts prematurely and end the discussion about them before understanding them, but more about the properties of the concept itself. The fact that XPath arrived at a specific form of nodelist after trying nodesets is certainly interesting feedback. We may not need that complexity, but it is much easier to eliminate complexity once we understand the concepts.
I'm not sure this is sufficient. Let me explain why with an example. Take the input argument: and the selector . I think the valid outputs should be: But I think the following output, for example, should be invalid: The spec should allow for this level of non-determinism, but not more. Similarly, if the Compliance Test Suite covers this example, it should allow precisely the valid outputs to pass. I don't see how a "contents-equality" comparison of arrays would help. This makes me favour encoding sets of possible outputs in the CTS as described in URL
Actually, [2, 3, 0, 1] etc. as well (the outer container also is a map).
Whoops! Yes. Edited the list so it is now complete.
I'm curious why you think this should be invalid. I understand how you got the result, and if you include (or replace with) the paths to those values, it's just as (or more) clear. Partial determinism is going to create a bunch of weird edge cases that will be impossible to completely define.
That of course depends on the semantics of the nodelist. If, as I'm assuming, it is sequence-preserving, you cannot arrive at [0, 2, 1, 3], which would be (JSON pointer syntax, I'm lazy) /a/c, /b/e, /a/d, /b/f.
This is quite a complicated discussion. I'm trying to sum things up: JSON objects as maps are unordered by nature. Due to that, JSONPath query results are non-deterministic regarding their order – which is common sense here. But determinism i.e. reproducability of JSONPath query results is required for compliance tests. Now implementations usually return query results as an array – pragmatically JSON in ... JSON out. JSON arrays are ordered by nature, which might falsy signal determinism of results. So calling the result a "node-list" might help here out, but introduces another level of complexity in terms and their usage. Having understood, that determinism is solely required for compliance testing, I am not sure, how to interprete Glyn's example above. Should future compliance tests really tell about every possible valid and invalid result ordering? I am sure someone of us would be able to implement something, exactly reproducing as result (noone would of course). Thus it's implementation dependent. But if we only need to insure, to always generate a single reproducable result, we can achieve this by something like ... Generate a list of output pathes. Sort it lexically or however. Generate the list of values from that path list – preserving order. ... which we can even communicate in the draft as a recipe. Regarding "node-list", there is indeed in JavaScript a concept of "array-like objects". Those objects only need to be able to tell the list length and provide an iterator to visit all entries. Defacto dom node-lists are exactly such things. This is what above discussion also reminds me on. Maybe I still miss another important point.
Neither determinism of the spec nor determinism for a given JSONPath implementation are necessary for compliance tests, but non-determinism does add some complexity to testing. That said, I am in favour of non-determinism in the spec because it gives the most freedom to implementations.
Because I would expect each "stage" of a selector to be evaluated in turn. In other words, the intermediate results which are output from one stage to the next can be in a non-deterministic order, but that order won't be modified by other stages as those stages are operating on the individual values from earlier stages. Sorry, but what is more clear? I don't think so. I think once we have addressed the non-determinism of specific selector "stages" applied to objects, that's pretty much it. What you describes as "partial determinism" is simply a result of how the spec is constructed from small pieces in a uniform way.
To be clear, non-determinism in the spec doesn't force implementations to be non-deterministic. But it allows implementations to trade off efficiency against non-determinism. It also allows for variations between (deterministic) implementations.
Also related to (maybe duplicates) .
A on the duplicates issue brought up the ordering of values in the result output. This does relate directly to whether we remove duplicates in that if we define that sequence matters and duplicates are removed, which copies of duplicated values are kept? Before we can answer that, I think we need to define whether the sequence of the results matter. This has implications for users of the Compliance Test Suite (CTS) as that it lists the expected results. If we decide that output sequence is important, then an implementation using the CTS would need to ensure that the actual result (from the implementation) and the expected result (from the CTS) must be sequence-equal. This is fairly straightforward and easy to process. However if we decide that output sequence does not matter, implementations would need to check for contents-equal. This is more complex to implement (but not impossible). Once we have defined this, we can go back to and address whether to remove duplicates and, if so, which duplicates to remove.
Each selector may each return multiple nodes, or even the same node multiple times. This raises the question of whether (and if so, how) the resulting collection of nodes needs to be distinct, i.e. without duplicates. An example: The raw result set of this selector is . If we say that we expect only the distinct set of nodes, then the results would be , removing the duplicate . So the question up for discussion is, "Should selectors return distinct sets of nodes or are duplicates preferred?" I had been considering a location-based distinction (based on the location of the value in the original JSON structure) as a possibility, but I don't think it makes much sense given the definition of a selector in Section 2.5, taking multiple nodes and outputing multiple nodes. A selector knows nothing about the original JSON structure that is being evaluated. However for posterity, consider the following (rather contrived) path evaluated against the above instance: In this case, the value which is the first element of the property would be selected twice, and the value in the second element of the property would be selected once, resulting in three values in the results. For location-based distinction, only the value from would be considered duplicate, leaving two instances of in the result set. Again, I don't think that this is a viable approach due to how a selector works and the information it's given. However, if we were to also require , then this would be a viable solution.
As for me I haven't see any implementations that tries to remove duplicate values from result, and I don't think it's a good idea to introduce this behavior as default one. But maybe it's useful to provide some way to filter off duplicated values explicitly, like . But your second case (about selecting the same node twice) is more interesting, because we have to provide some default behavior; I will investigate the implementations on this case.
I think it might be useful to throw a scenario in the consensus project to get actual results of what the various implementations do.
A scenario has been . Just waiting on the report to regenerate. It looks like the majority just don't support the example query. For those that do, it seems to be an even mix of all three approaches. No concensus implies no pressure to implement any particular scenario, but we need to discuss and agree on it.
An option is to allow some latitude in the spec and say that "duplicates MAY be removed, either by location and value". But I think in that case we'd also need to tie down when that applies. Should it just apply to unions? Could it also apply in other cases, e.g. when using against an array containing duplicate values (e.g. because there are duplicate values in the input document)? "MAY" will considerably complicate the CTS FWIW, especially if the CTS is purely declarative (e.g. URL) as that would put the onus on each CTS implementation to allow duplicates to be removed if appropriate. Also, does allowing duplicates to be removed imply anything about ordering? Can non-duplicates change order, for example?
I think it can apply to any case where the same value appears in the result more than once. I've opened a new issue for discussing result sequence. I think we need to answer that before we can discuss how it affects duplicates.
To expand on the reason for combining duplicates a little, I offer a more extensive example and a somewhat informal proof of why eliminating duplicates would be a good thing (when not including location in the output): The instance is similar to Goessner's example, but I've added color to the books and moved the other items under a "toy" category so that I can search for array items and use an expression. Now if we want the prices of all the things that are red, we can use . This results in without removal and with removal. Let's ignore the for now to see what how the duplicate values are produced. The path returns three objects: Notice that the results for and are the same. Any processing on further path segments will result in the same output. Therefore we should only need to perform further selection once on these values. Duplicate removal resolves this problem. Even when we include locations, this can be used as an internal optimization. An implementation could identify that these are the same object and keep track of all the locations which contain it.
I'm confused. Why do and in your last example lack the property?
Ugh... Oversight of editing while authoring. I'll fix it. Aaand now I have to rework it to get the duplicates again.
NAME updated.
I think this is trading off performance, in some fairly rare cases, against simplicity of the spec. I'd prefer to keep the spec simpler.
Allowing latitude defeats interoperability, which I had thought was the raison d'etre for the IETF. But I note there's no mention of interoperability as a design goal in the draft. From previous discussions, I think it is only necessary to consider removal of duplicates by location (not value.) I don't think so. I think it should apply however duplicates arise, whether unions, wildcards, recursive descent. This also simplifies removal. Or perhaps, either duplicates must be removed, or duplicates must not be removed, or implementations must default to one of these two, and provide an option for the other. I don't think it affects order at all. Given the data and selector The results with duplicates are: Values|Paths -------|------------------ "The Night Watch" | "The Night Watch" | "The Night Watch" | The results without duplicates are: Values | Paths ---------|------------------ "The Night Watch" | "The Night Watch" | Clearly the second set can easily be obtained from the first without rearranging the items, see Note that implementations have some latitude as to when to remove duplicates. As long as both values and normalized paths are collected, the implementation could delay removal until the last step, before returning results to the user.
Following up on 's motivation for wanting to remove duplicates, there's a related use case that's applicable when a user wants to modify the original data based on addressing information obtained from a query. Such addressing information could be in the form of normalized paths, or it could be provided by references or pointers into the original data, in languages that support that. In either case, the issue is the same. So suppose that a user queries for a collection of prices for all books in the store, and wishes to discount them all by twenty percent. If duplicates come back, if the same price gets marked down twice, that's going to be a problem. This is not entirely hypothetical, there are jsonpath libraries that are used in this way. More generally, it's hard to think of use cases where duplicates would be beneficial. It's possible to think of use cases where they would be detrimental. For an implementation that collects normalized paths along with values, removal is easy. Of course, removal comes with a cost, and some users may prefer not to pay that cost, particularly as most straightforward queries don't result in duplicates anyway.
I know the jury is still out on whether we remove duplicate nodes, but I've prepared draft PRs for the compliance test suite (URL) and the reference implementation (URL) to handle removal of duplicate nodes. Any comments gratefully received.
I am preparing an issue (URL) about unions, where removing duplicate nodes is also a point
I have a question about duplicates. Consider a root value and suppose that a query returned two values corresponding to paths Would the node identified by be considered a duplicate?
No, we only consider duplicates at the top level of the returned nodelist. So the values to compare are an array and a string. That the array also happens to contain the string makes no difference.
Consider: and this jsonpath: : The first part of the jsonpath selects nodes and applying on that result set, produces the node twice
Yes, and we need to agree whether: (1) the duplicate nodes are always delivered (which is what we decided today) (2) the duplicate nodes are never delivered (which is what we went into the discussion with) (3) it depends on the implementation (which IMHO would be a disaster)
IIRC today we were wondering whether: (5) the behavior of the descendent selector may be a special case
Maybe we can record the resolution that there is no removal, neither of duplicate values nor of duplicate nodes.
I'd be interested in how this resolution came about because it seems like the discussion here was leaning the other way: removing duplicates to some degree. To what degree was still under discussion.
Slides for 112 said this: We didn't have time to discuss this at 112, so let's confirm the resolution on the 2022-01-18 interim. I think the summary so far is We recognize that there are operations in JSONPath whose result is ordered but that order needs to be created based on unordered input. We are not going to cut off these noses to spite our face. We have a choice between imposing a language-defined order (e.g., sorting) or leaving the result non-deterministic. I gather that there is a general direction toward non-determinism, as imposing an order would be too expensive, but I'm not sure we have a conclusion here. Grüße, Carsten

The "descendant-selector" selects the node and all its descendants. <ins> In the resultant nodelist: * nodes occur before their children, and * nodes of an array occur in array order. Children of an object may occur in any order, since JSON objects are unordered. </ins> 3.5.8. 3.5.8.1.


dated 2007-02-21 JSONPath-orig and provides a normative definition for it. <del> inspired-by-xpath provides a brief synopsis with XML's XPath XPath, which was the inspiration that led to the definition of JSONPath. </del>  JSONPath was intended as a light-weight companion to JSON implementations on platforms such as PHP and JavaScript, so instead The examples in tbl-example use the expression mechanism to obtain the number of elements in an array, to test for the presence of a member in a object, and to perform numeric comparisons of member <del> values with a constant.  For further illustration, the table shows XPath expressions that would be used with a comparable XML document; see also xpath-overview. </del>  3.

XPath details are a distraction in the body of the spec. In the future, readers are increasingly likely to come to this spec with no exposure to XPath or, for that matter, XML.
Looks very good, with the one nit. Thanks!

dated 2007-02-21 JSONPath-orig and provides a normative definition for it.  <ins> inspired-by-xpath describes how JSONPath was inspired by XML's XPath XPath. </ins> JSONPath was intended as a light-weight companion to JSON implementations on platforms such as PHP and JavaScript, so instead The examples in tbl-example use the expression mechanism to obtain the number of elements in an array, to test for the presence of a member in a object, and to perform numeric comparisons of member  <ins> values with a constant. </ins> 3.


expressions between the ensuing platform-specific dialects. The present specification intends to remove platform dependencies and <del> server as a common JSONPath specification that can be used across </del>  platforms.  Obviously, this means that backwards compatibility could not always be achieved; a design principle of this specification is to go with a "consensus" between implementations even if it is rough, start position, an end position, and possibly a step value that moves the position from the start to the end (slice). <del> JSONPath employs an for computing values and making decisions.  The present specification defines a simple expression language that is independent from any scripting language in use on the platform.  JSONPath can use such expression language expressions, written in parentheses: "(<expr>)", as an alternative to explicit names or indices as in [no-dot-length]: The symbol "@" is used for the current node, i.e., the node in the context of which the expression is evaluated. </del> Filter expressions are supported via the syntax "?(<boolean expr>)" as in 4. <del> 5. </del> TBD: Define a media type for JSONPath expressions. <del> 6. </del>  This section gives security considerations, as required by RFC3552. isolating a single location within a JSON document.  Path is a query syntax that can also be used to pull multiple locations. <del> [no-dot-length] We probably want to use an expression that does not use </del>

There does not seem to be an appetite for doing the necessary spec work. This PR will test that assertion. Filter expressions are still supported, but not "container expressions", of the form [(...)].
I think this can be added later, similar to the consensus for . Personally I still see value in having it.

expressions between the ensuing platform-specific dialects. The present specification intends to remove platform dependencies and  <ins> serve as a common JSONPath specification that can be used across </ins> platforms.  Obviously, this means that backwards compatibility could not always be achieved; a design principle of this specification is to go with a "consensus" between implementations even if it is rough, start position, an end position, and possibly a step value that moves the position from the start to the end (slice).  Filter expressions are supported via the syntax "?(<boolean expr>)" as in 4.  TBD: Define a media type for JSONPath expressions.  <ins> 5. </ins> This section gives security considerations, as required by RFC3552. isolating a single location within a JSON document.  Path is a query syntax that can also be used to pull multiple locations.


Regular expression tests can be applied to "string" values only.  The following table lists filter expression operators in order of precedence from highest (binds most tightly) to lowest (binds least tightly).

Thanks, the rebase looks good.
NAME After merging some PRs I fixed a conflict here and then added a commit. PTAL and merge if you are happy.

Regular expression tests can be applied to "string" values only. <ins> Alphabetic characters in ABNF are case-insensitive, so "e" can be either "e" or "E". false, null, true are lower-case only (case-sensitive). </ins> The following table lists filter expression operators in order of precedence from highest (binds most tightly) to lowest (binds least tightly).


Nested descendants selector "..". <del> Union selector "[<sel1>,<sel2>,...,<selN>]", holding a comma delimited list of index, index wild card, array slice, and filter selectors. </del>  Filter selector "[?(<expr>)]" 3.5.8.  3.5.8.1. <del> The union selector is syntactically related to the "index-selector". It contains two or more entries, separated by commas. </del>  3.5.8.2. <del> A union selects any node which is selected by at least one of the union selectors and selects the concatenation of the lists (in the order of the selectors) of nodes selected by the union elements. Note that any node selected in more than one of the union selectors is kept as many times in the node list. </del>  To be valid, integer values in the "element-index" and "slice-index" components MUST be in the I-JSON range of exact values, see synsem-

Clarify related text; add explanation of change. Take slice-selector apart to define slice-index.
I'm going to merge this PR to make forward progress. Tweaks can be added via further PRs as usual.
URL URL
(To me it is inexplicable they aren't already. But changing that was not the remit of .)
I am fine with the status quo of excluding filters within list selectors. One is either building a list of explicite indices/indexes or a boolean expression for filtering. Two or more filter expressions in a list are equivalent to have a single one or'ed them together. Most cases in practises could be solved by a sequence list selector following filter selector filter selector following list selector Any striking use cases I am not aware of?
As mentioned in my comment, I see no reason why a filter can't be included in a list. Some people may prefer to have multiple simple expressions for readability. My implementation already supports this (I'll have to disable this functionality or hide it behind an option if we don't include it).
I don't have a strong opinion on this question. But not only does it seem natural to include filter selectors in list selectors: the comparison project also shows that most implementations that do provide this, also agree on what the result should be. So this would not be a risky addition.
I do have a strong opinion here, as ... index expressions (member names, index and slices) are strongly explicit and single level only. list-selector is merely a list of those index expressions. Example above is easily - and more consistently - reformulated as a pure filter-selector So I would like to see real use cases to be convinced here. If filter selectors are natural to be included in list selectors, then so are index-wild-selector descendant-selector and even list-selector
(The sequence of data items is different between the two.) Could be done. There is no natural syntax for that. That already works :-) If you have the list selector [1, 2, 3], you can include that in [4, 5, 6] by unpacking the brackets, so you have 1, 2, 3, and appending that to the comma-separated list, so you have [4, 5, 6, 1, 2, 3] (Note that all the embedded selectors have their brackets removed before including them in a list-selector.) Grüße, Carsten
??? ... even with filter selectors ?
(I just pushed a slight editorial fix that distinguishes between selectors in general and selector entries in a list selector — the latter don’t have the brackets.) Note the "(in the order of the selector entries)”. So applying $[?(@ % 2 == 0), ?(@ % 2 != 0)] to [1, 2, 3, 4] yields 2, 4, 1, 3 …while applying $[?(@ % 2 == 0 @ % 2 != 0)] to [1, 2, 3, 4] yields 1, 2, 3, 4 Yes, putting [?(@ % 2 == 0)] and [?(@ % 2 != 0)] into a list selector yields [?(@ % 2 == 0), ?(@ % 2 != 0)]. The brackets of the individual filter selectors are gone. Grüße, Carsten
My preference is to include filter selectors in list selectors simply because omitting them seems arbitrary and rather surprising. I don't have a use case to justify this, although I could probably fabricate one if pushed. ;-)
I think the key to the utility here is mixing selector types. If the user wanted to get the first item and also any item where , then they can do: To my knowledge, this can't be done in a single expression.
... this is true ...
I think you are aware of using arithmethic operators here. JSONPath is for selection exclusively. JSONPath is not for calculation.
Selection and calculation are not mutually exclusive, though. Consider Javascript Goessner returns
I do know this very well ... since I am pragmatically reusing the javascript engine for evaluation. People then would want to know/apply ... all products above/below the average product price. normalized values distances between locations trigonometric functions logarithmic scaling vector arithmetic Even when restricting to basic arithmetic operations we need to handle division by zero, NaN, equality of floats/doubles, truncation, rounding, ... Your example above can be solved with a three-liner ... or by smart post-processing of JSONPath selection results. JSONPath is for selection ... the right tool for the right task.
NAME I think it all comes down to taste. The taste of the language designer. With regards to prior experience, there are query languages that have arithmetic operations (SQL, XPath, JSONiq, all JSONPath implementations that use Javascript, Python, Lua) and others that do not (JMESPath, Jayway JSONPath). Either way, there is plenty of prior experience to draw on. There are answers to the technical questions. But taste is another matter. Best regards, Daniel
I think we're getting a little of topic here. We want to know if there are use cases for including the syntax as a viable member of a list selector. I think my example shows that there is a use case, even if somewhat constructed. Someone is going to try it.
That's my position too Daniel.
I have so far not contributed to this discussion, and don't have a strong position myself, but I wanted to explain the quoted argument in a bit more detail. In my (outdated) proposal for an aligned understanding, I identified two major operators: recursive decent and children selection. The latter is implemented here: URL If we were to exclude wildcard or filters, my code would grow. Happy to see (pseudo) code for the other side of the argument to compare.
The term "union" will likely be unfamiliar for readers (it is for me) and could use a bit more description than "A union ~matcher~ selector consists of one or more union elements." "Union" isn't a common term outside of set manipulation. I would call the an "indexer," which is a much more well-known term and is already associated with this construct in many programming languages. The values inside are indices (as opposed to "union children" or "union elements"), of which there are multiple types: Integer Slice String Container query expression Item query expression See for more details on the query-style indexers.
Good point. We have yet to decide whether unions remove or preserve duplicates and I think the outcome of that decision should influence the naming. "Union" as you point out has a set theory connotation and implies removing duplicates. It's tricky though because I think the result is ordered, whereas sets aren't ordered.
Regarding duplicates specifically, please see .
Not much actionable discussion here. Confirm no change needed at 2022-01-18 interim.

Nested descendants selector "..".  <ins> List selector "[<sel1>,<sel2>,...,<selN>]", holding a comma separated list of index and slice selectors. </ins> Filter selector "[?(<expr>)]" 3.5.8. <ins> The list selector allows combining member names, array indices, and slices in a single selector. Note: The list selector was called "union selector" in JSONPath-orig, as it was intended to solve use cases addressed by the union selector in XPath.  However, the term "union" has the connotation of a set operation that involves merging input sets while avoiding duplicates, so the concept was renamed into "list selector". </ins> 3.5.8.1.  <ins> The list selector is syntactically related to the "index-selector" and the "slice-selector".  It contains two or more entries, separated by commas. </ins> 3.5.8.2.  <ins> A list selector selects the nodes that are selected by at least one of the selector entries in the list and yields the concatenation of the lists (in the order of the selector entries) of nodes selected by the selector entries.  Note that any node selected in more than one of the selector entries is kept as many times in the node list. </ins> To be valid, integer values in the "element-index" and "slice-index" components MUST be in the I-JSON range of exact values, see synsem-


4. <del> TBD: Define a media type for JSONPath expressions. </del>  5.

I merged this so once we have published an I-D with this change in it we can initiate the necessary review on the media-types list.
Looks plausible.

4.  <ins> 4.1. IANA is requested to register the following media type RFC6838: Person & email address to contact for further information: iesg@ietf.org </ins> 5.