What is the format in jq for calling a custom module?

Question

I need to url decode a string in a json structure using jq. I have a custom module defined under ~/.jq/urldecode.jq but when calling it:

jq '.http.referrer | url_decode::url_decode' file.json

I get the error message:

jq: 1 compile error

The module source is:

def url_decode:
  # The helper function converts the input string written in the given
  # "base" to an integer
  def to_i(base):
    explode
    | reverse
    | map(if 65 <= . and . <= 90 then . + 32  else . end)   # downcase
    | map(if . > 96  then . - 87 else . - 48 end)  # "a" ~ 97 => 10 ~ 87
    | reduce .[] as $c
        # base: [power, ans]
        ([1,0]; (.[0] * base) as $b | [$b, .[1] + (.[0] * $c)]) | .[1];

  .  as $in
  | length as $length
  | [0, ""]  # i, answer
  | until ( .[0] >= $length;
      .[0] as $i
      |  if $in[$i:$i+1] == "%"
         then [ $i + 3, .[1] + ([$in[$i+1:$i+3] | to_i(16)] | implode) ]
         else [ $i + 1, .[1] + $in[$i:$i+1] ]
         end)
  | .[1];  # answer

What is the proper syntax?

peak · Answer 1 · 2020-04-25T03:13:09.047

In theory, with your setup, you should be able to invoke jq along the lines of

jq 'import "urldecode" as url_decode;
  .http.referrer | url_decode::url_decode' file.json

or more simply:

jq 'include "urldecode";
  .http.referrer | url_decode' file.json

However, there are some circumstances in which theory does not quite apply. In such cases, the following workarounds may be used with jq 1.5 and 1.6:

(1) specify -L $HOME as a command-line parameter, and give the relative path name in the module specification. Thus, in your case, the command line would look like:

jq -L $HOME 'import ".jq/urldecode" as url_decode; ...

or:

jq -L $HOME 'include ".jq/urldecode"; ...

(2) Use the {search: _} feature, e.g.

jq 'include "urldecode" {search: "~/.jq"} ; ...' ...

jq 'import "urldecode" as url_decode {search: "~/.jq"} ; ...' ...

What is the bug / what circumstances are you referring to? None of the permutations you posted here work for me on `jq` 1.5 — Huckle, Apr 24 '20 at 21:37
@Huckle - Since the Q specifies the module filename as urldecode.jq I've updated the answer accordingly. — peak, Apr 25 '20 at 03:00

score 0 · Answer 2 · answered Dec 26 '18 at 21:34

jq reads by default from a hidden folder in your root directory files marked with .jq file extension: ~/.jq (["~/.jq", "$ORIGIN/../lib/jq", "$ORIGIN/../lib"])

To reference the module you can use the import function, then follow your normal jq command after a semicolon. The "as lib" below allows you to change the name of the namespace as well:

 jq 'import "urldecode" as lib; .http.referrer | lib::url_decode' file.json

You can override the location where the .jq file are stored w/ the -L option.

Unfortunately there is a bug in jq 1.5 and 1.6, so that this theoretically valid approach won't always work. See the workaround elsewhere on this page. — peak, Dec 27 '18 at 01:14

score -2 · Answer 3 · answered Dec 26 '18 at 23:24

I'm not sure how to do custom modules in JQ, but If you are using bash I would suggest piping to PERL for this. So far this is the easiest way I have found to quickly url-encode/decode HTML entities and I typically use this in conjunction with JQ

echo 'http://domain.tld/?fields={fieldname_of_type_Tab&#125' | perl -MHTML::Entities -pe 'decode_entities($_)'

Decode URL Unix/Bash Command Line (without sed)

What is the format in jq for calling a custom module?

3 Answers3