tool that shows all possible jq paths in a given json file · Issue #243 · jqlang/jq

link管理
链接快照平台
输入网页链接，自动生成快照
标签化管理网页链接
相关文章推荐
没人理的油条 · jq命令 :: AWS Workshop· 3 月前 ·
仗义的单车 · apache spark - ...· 2 年前 ·
强健的啄木鸟 · python 直接打开终端macos ...· 2 年前 ·
傻傻的滑板 · 修饰名 | Microsoft Learn· 2 年前 ·
纯真的水煮鱼 · json字符串大括号里的必须全部双引号 - ...· 2 年前 ·
酷酷的火锅 · python+opencv+skimage ...· 2 年前 ·
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
( Copy+pasting unanswered email: )
I have just prepared a tiny tool that shows all possible jq paths in a given json file. I guess it can help people that use jq. I'm not sure where in the docs it goes. Can you please add a link wherever you think is appropriate?
Please note (also, should be clear to users before following the link) that the script was not heavily tested and I guess is alpha quality in general, quick and dirty, naive implementation.... but still works for me.
https://github.com/ilyash/show-struct
Thanks!
FYI, I believe that this:
jq 'path(recurse(if type|. == "array" or . =="object" then .[] else
empty end))'
or, with master, this:
jq 'path(..)'
will list all the possible paths.
          +1 to @slapresta 's idea.
@ilyash This produces a list of paths in the input:
jq -c 'path(..)|[.[]|tostring]|join("/")'
It's a very simple, trivial program :)
  rmoff, ettorerizza, jacobsalmela, jcaesar, Kvrepo, libredot, AustinBGibbons, reardenlife, TryTryAgain, onlyjob, and 22 more reacted with thumbs up emoji
  subeshb1, mouchtaris, alahijani, orenli, javiertejero, ipleten, tomklino, sandeepkunkunuru, m-czernek, jakubleska, and 6 more reacted with heart emoji
      All reactions
          @nicowilliams You are totally ignoring the differences between outputs and the fact that show-struct is helpful for constructing the path part of jq command. Have you seen the sample output of the show-struct? Let's compare full (but censored) outputs for the same input.
show_struct:
.Records -- (Array of 3 elements)
.Records[]
.Records[].awsRegion -- us-east-1
.Records[].eventName -- DescribeInstances
.Records[].eventSource -- ec2.amazonaws.com
.Records[].eventTime -- 2013-12-06T10:34:34Z .. 2013-12-06T10:36:36Z (3 unique values)
.Records[].eventVersion -- 1.0
.Records[].requestParameters
.Records[].requestParameters.filterSet
.Records[].requestParameters.filterSet.items -- (Array of 3 elements)
.Records[].requestParameters.filterSet.items[]
.Records[].requestParameters.filterSet.items[].name -- group-name .. tag:role (3 unique values)
.Records[].requestParameters.filterSet.items[].valueSet
.Records[].requestParameters.filterSet.items[].valueSet.items -- (Array of 1 elements)
.Records[].requestParameters.filterSet.items[].valueSet.items[]
.Records[].requestParameters.filterSet.items[].valueSet.items[].value -- dsp_worker .. staging-dsp-processor (5 unique values)
.Records[].requestParameters.instancesSet -- (Empty hash)
.Records[].responseElements -- <responseOmitted>
.Records[].sourceIPAddress -- X.Y.Z.W .. X.Y.Z.WW (3 unique values)
.Records[].userAgent -- aws-sdk-java/1.5.0 Linux/3.2.0-4-amd64 OpenJDK_64-Bit_Server_VM/23.7-b01 .. aws-sdk-java/1.6.7 Linux/3.2.0-4-amd64 OpenJDK_64-Bit_Server_VM/23.7-b01 (2 unique values)
.Records[].userIdentity
.Records[].userIdentity.accessKeyId -- XXXXXXXXXXXXXXXXXXXX
.Records[].userIdentity.accountId -- NNNNNNNNNNNN
.Records[].userIdentity.arn -- arn:aws:iam::NNNNNNNNNNNN:user/stage-dsp-manager
.Records[].userIdentity.principalId -- XXXXXXXXXXXXXXXXXXXXX
.Records[].userIdentity.type -- IAMUser
.Records[].userIdentity.userName -- stage-dsp-manager
In contrast, the jq program above shows:
"Records"
"Records/0"
"Records/0/eventVersion"
"Records/0/userIdentity"
"Records/0/userIdentity/type"
"Records/0/userIdentity/principalId"
"Records/0/userIdentity/arn"
"Records/0/userIdentity/accountId"
"Records/0/userIdentity/accessKeyId"
"Records/0/userIdentity/userName"
"Records/0/eventTime"
"Records/0/eventSource"
"Records/0/eventName"
"Records/0/awsRegion"
"Records/0/sourceIPAddress"
"Records/0/userAgent"
"Records/0/requestParameters"
"Records/0/requestParameters/instancesSet"
"Records/0/requestParameters/filterSet"
"Records/0/requestParameters/filterSet/items"
"Records/0/requestParameters/filterSet/items/0"
"Records/0/requestParameters/filterSet/items/0/name"
"Records/0/requestParameters/filterSet/items/0/valueSet"
"Records/0/requestParameters/filterSet/items/0/valueSet/items"
"Records/0/requestParameters/filterSet/items/0/valueSet/items/0"
"Records/0/requestParameters/filterSet/items/0/valueSet/items/0/value"
"Records/0/requestParameters/filterSet/items/1"
"Records/0/requestParameters/filterSet/items/1/name"
"Records/0/requestParameters/filterSet/items/1/valueSet"
"Records/0/requestParameters/filterSet/items/1/valueSet/items"
"Records/0/requestParameters/filterSet/items/1/valueSet/items/0"
"Records/0/requestParameters/filterSet/items/1/valueSet/items/0/value"
"Records/0/requestParameters/filterSet/items/2"
"Records/0/requestParameters/filterSet/items/2/name"
"Records/0/requestParameters/filterSet/items/2/valueSet"
"Records/0/requestParameters/filterSet/items/2/valueSet/items"
"Records/0/requestParameters/filterSet/items/2/valueSet/items/0"
"Records/0/requestParameters/filterSet/items/2/valueSet/items/0/value"
"Records/0/responseElements"
"Records/1"
"Records/1/eventVersion"
"Records/1/userIdentity"
"Records/1/userIdentity/type"
"Records/1/userIdentity/principalId"
"Records/1/userIdentity/arn"
"Records/1/userIdentity/accountId"
"Records/1/userIdentity/accessKeyId"
"Records/1/userIdentity/userName"
"Records/1/eventTime"
"Records/1/eventSource"
"Records/1/eventName"
"Records/1/awsRegion"
"Records/1/sourceIPAddress"
"Records/1/userAgent"
"Records/1/requestParameters"
"Records/1/requestParameters/instancesSet"
"Records/1/requestParameters/filterSet"
"Records/1/requestParameters/filterSet/items"
"Records/1/requestParameters/filterSet/items/0"
"Records/1/requestParameters/filterSet/items/0/name"
"Records/1/requestParameters/filterSet/items/0/valueSet"
"Records/1/requestParameters/filterSet/items/0/valueSet/items"
"Records/1/requestParameters/filterSet/items/0/valueSet/items/0"
"Records/1/requestParameters/filterSet/items/0/valueSet/items/0/value"
"Records/1/requestParameters/filterSet/items/1"
"Records/1/requestParameters/filterSet/items/1/name"
"Records/1/requestParameters/filterSet/items/1/valueSet"
"Records/1/requestParameters/filterSet/items/1/valueSet/items"
"Records/1/requestParameters/filterSet/items/1/valueSet/items/0"
"Records/1/requestParameters/filterSet/items/1/valueSet/items/0/value"
"Records/1/requestParameters/filterSet/items/2"
"Records/1/requestParameters/filterSet/items/2/name"
"Records/1/requestParameters/filterSet/items/2/valueSet"
"Records/1/requestParameters/filterSet/items/2/valueSet/items"
"Records/1/requestParameters/filterSet/items/2/valueSet/items/0"
"Records/1/requestParameters/filterSet/items/2/valueSet/items/0/value"
"Records/1/responseElements"
"Records/2"
"Records/2/eventVersion"
"Records/2/userIdentity"
"Records/2/userIdentity/type"
"Records/2/userIdentity/principalId"
"Records/2/userIdentity/arn"
"Records/2/userIdentity/accountId"
"Records/2/userIdentity/accessKeyId"
"Records/2/userIdentity/userName"
"Records/2/eventTime"
"Records/2/eventSource"
"Records/2/eventName"
"Records/2/awsRegion"
"Records/2/sourceIPAddress"
"Records/2/userAgent"
"Records/2/requestParameters"
"Records/2/requestParameters/instancesSet"
"Records/2/requestParameters/filterSet"
"Records/2/requestParameters/filterSet/items"
"Records/2/requestParameters/filterSet/items/0"
"Records/2/requestParameters/filterSet/items/0/name"
"Records/2/requestParameters/filterSet/items/0/valueSet"
"Records/2/requestParameters/filterSet/items/0/valueSet/items"
"Records/2/requestParameters/filterSet/items/0/valueSet/items/0"
"Records/2/requestParameters/filterSet/items/0/valueSet/items/0/value"
"Records/2/requestParameters/filterSet/items/1"
"Records/2/requestParameters/filterSet/items/1/name"
"Records/2/requestParameters/filterSet/items/1/valueSet"
"Records/2/requestParameters/filterSet/items/1/valueSet/items"
"Records/2/requestParameters/filterSet/items/1/valueSet/items/0"
"Records/2/requestParameters/filterSet/items/1/valueSet/items/0/value"
"Records/2/requestParameters/filterSet/items/2"
"Records/2/requestParameters/filterSet/items/2/name"
"Records/2/requestParameters/filterSet/items/2/valueSet"
"Records/2/requestParameters/filterSet/items/2/valueSet/items"
"Records/2/requestParameters/filterSet/items/2/valueSet/items/0"
"Records/2/requestParameters/filterSet/items/2/valueSet/items/0/value"
"Records/2/responseElements"
          I like the idea of a compact structural overview, but am not as interested in the value or array lengths.
jq '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]'
    path(..)
    | map(
        if type == "number" then
            tostring
    | join(".")
    | split(".[]")
    | join("[]")
| unique
| map("." + .)
| .[]
See also structure.sh in jq-hopkok for a more fleshed out script, in particular for property names containing special characters.
  aleksandrserbin, rwiggins, PaulTaykalo, dentarg, trufae, wilkerlucio, ettorerizza, kam1kaze, jacobsalmela, liath, and 26 more reacted with thumbs up emoji
  jpmorin, subeshb1, albertogalan, and greenaussie reacted with hooray emoji
  aleksandrserbin, PaulTaykalo, trufae, jacobsalmela, vouill, danoyoung, rdoering, InsOpDe, FLS-BrycePlatt, jpmorin, and 4 more reacted with heart emoji
      All reactions
      See discussion regarding the `show-struct` tool by @ilyash.
https://github.com/ilyash/show-struct
jqlang/jq#243
          @joelpurra , I have a sample file (not sure where I got it from, it's just there):
  {"$jt:env": "HOME"},
  ["$jt:env", "HOME"]
Both outputs from your snippet and my show_struct are pretty confusing:
".[]"
".[].$jt:env"
[] -- (Array of 2 elements)
[].$jt:env -- HOME
[][] -- $jt:env .. HOME (2 unique values)
I guess we could both improve...
          @ilyash: Yes, it's a confusing example =)
I have an updated version that works slightly differently. Plan to move it from har-dulcify to jq-hopkok at some point.
<temp.json har-dulcify/src/util/structure.sh
.[]["$jt:env"]
.[][]

Because of the mixed data types, jq doesn't fully agree when using this as input on the same file.
          On Sat, Aug 16, 2014 at 2:51 PM, Ilya Sher [email protected] wrote:
@joelpurra https://github.com/joelpurra , I have a sample file (not

sure where I got it from, it's just there):
{"$jt:env": "HOME"},

["$jt:env", "HOME"]

Both outputs from your snippet and my show_struct are pretty confusing:
".[]"

".[].$jt:env"
[] -- (Array of 2 elements)

[].$jt:env -- HOME

[][] -- $jt:env .. HOME (2 unique values)
I guess we could both improve...
A path-based representation of the above text which papers over the

difference in (non-scalar) types of the elements of the top-level array...

must be lossy.
          If anyone is interested, the following code show all paths with their values:
paths(scalars) as $p
  | [ ( [ $p[] | tostring ] | join(".") )
    , ( getpath($p) | tojson )
  | join(" = ")
Example:
$ jq -r '
paths(scalars) as $p
  | [ ( [ $p[] | tostring ] | join(".") )
    , ( getpath($p) | tojson )
  | join(" = ")
' <<'INPUT'
  "a": 1,
  "b": [ "red", "green", "blue" ],
  "c": {
    "d": [
        "a": 100,
        "b": 200,
        "c": "x\ny\nz"
        "a": 101,
        "b": 201
INPUT
Output:
a = 1
b.0 = "red"
b.1 = "green"
b.2 = "blue"
c.d.0.a = 100
c.d.0.b = 200
c.d.0.c = "x\ny\nz"
c.d.1.a = 101
c.d.1.b = 201
  jpmorin, subeshb1, sirlatrom, ukos-git, timtofan, oparent-sharecare, and fefa4ka reacted with thumbs up emoji
  gertcuykens, JeffMv, subeshb1, timtofan, oparent-sharecare, and laq reacted with hooray emoji
  jacobsalmela, thn929, subeshb1, sirlatrom, timtofan, and oparent-sharecare reacted with heart emoji
      All reactions
          @gertcuykens - The following produces the output you've indicated you want:
def path2text($value):
  def tos: if type == "number" then . else "\"\(tojson)\"" end;
  reduce .[] as $segment ("";  .
    + ($segment
       | if type == "string" then "." + . else "[\(.)]" end))
  + " = \($value | tos)";
paths(scalars) as $p
  | getpath($p) as $v
  | $p | path2text($v)
          yep works awesome thx, wouldn't be able to find it myself in a million years :)
PS just made a slide modification else you get double quotes  = ""test""
def path2text($value):
  def tos: if type == "number" then . else tojson end;
  reduce .[] as $segment ("";  .
    + ($segment
       | if type == "string" then "." + . else "[\(.)]" end))
  + " = \($value | tos)";
paths(scalars) as $p
  | getpath($p) as $v
  | $p | path2text($v)
          I've been using this syntax
jq 'select(objects)|=[.] | map( paths(scalars) ) | map( map(select(numbers)="[]") | join(".")) | unique'
That last pipe to unique is also helpful for really large objects.