How to use a token for a rex in Splunk?

Question

I have a token $token_rex$ set up as follows in the dashboard:

<set>mvjoin(mvmap('token_keywords_mv',"(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</set>

token_keyrowrds_mv is basically the following:

lorem
ipsum
situs

The resulting token looks like this:

(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)

The query I am trying to run is as follows:

index=* | rex field _raw "(?i)".$token_rex$

Ideally the above should be rendered like so:

index=* | rex field _raw "(?i)(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)"

I have tried using the token filters $token_rex|s$ and $token_rex|n$ but neither work. I have even tried to return the value from a subsearch:

index=* | rex field _raw [| makeresults | eval string_rex=$token_rex$ | return $token_rex]

Update: Found out that the token is being given as verbose to the rex command. My token is set as follows:

<set token="token_rex">mvjoin(mvmap('token_keywords_mv',"(?&gt;".'token_keywords_mv'."&lt;".'token_keywords_mv'."+?)"), "|")</set>

When I used it as ... | rex field=_raw '(?i)$token_rex$' it gives me the following error:

Error in 'rex' command: Encountered the following error while compiling the regex ''(?i)mvjoin(mvmap('token_keywords_mv'': Regex: missing closing parenthesis.

When I set it as the value directly, however, it works:

<set token="token_rex">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>

Update: Here's a sample dashboard.

<dashboard theme="dark">
  <init>
    <set token="token_keywords_mv">split("lorem,ipsum,situs", ",")</set>
    <set token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</set>
    <set token="token_rex">"(?i)".mvjoin(mvmap('token_keywords_mv', "(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</set>
    <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>
  </init>
  <label>Test Search</label>
  <description>Multivalue</description>
  <row>
    <panel>
      <table>
        <search>
          <query>index=* 
    [| makeresults 
    | eval string_search=$token_keywords_starred$ 
    | return $string_search] 
| rex field=_raw $token_raw|s$ 
| stats count(lorem) AS Lorem, count(ipsum) AS Ipsum, count(situs) AS Situs
| eval header="Count" 
| transpose column_name="String" header_field=header</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>ECHO</title>
        <search>
          <query>| localop 
| makeresults 
| eval token_keywords_starred=$token_keywords_starred$, token_keywords_mv=$token_keywords_mv$, token_rex=$token_rex$, token_raw=$token_raw|s$</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</dashboard>

If any other token replaces $token_raw|s$ , there will be an error. Using Splunk 8.0.3.

You say what rendering you expect from `rex field=_raw "(?i)".$token_rex|s$`, but not what you get. Have you tried assigning the token to a new field and using that in `rex`. Something like `| eval token_rex=$token_rex|s$ | rex "(?i)".token_rex`. This is all in a dashboard, right? — RichG, Apr 05 '22 at 14:31
@RichG Yeah it's on Dashboard. Your solution still gives me an error `error in 'eval' command: The expression is malformed. Expected: ")"`. Updating question with some new findings. — yaserso, Apr 05 '22 at 16:22
I suspect `rex` does not support the concatenation operator. Try moving the concatenation to the `eval`. — RichG, Apr 05 '22 at 16:50
Even if I remove it, it does not recognize the field as a value; `... | eval token_rex=$token_rex$ | rex token_rex`. Says "The token 'token_rex' does not extract anything..." — yaserso, Apr 05 '22 at 17:09

score 1 · Accepted Answer · edited Apr 06 '22 at 10:49

1

When I try to reproduce your results with this dashboard code:

<form>
  <label>test</label>
  <fieldset submitButton="false">
    <input type="text" token="token_rex">
      <label>field1</label>
    </input>
  </fieldset>
  <row>
    <panel>
      <event>
        <title>test</title>
        <search>
          <query>| makeresults | eval _raw="lorem ipsum dolor"
| rex "(?i)$token_rex$"
|  table lorem ipsum situs</query>
          <earliest>$earliest$</earliest>
          <latest>$latest$</latest>
        </search>
        <option name="refresh.display">progressbar</option>
      </event>
    </panel>
  </row>
</form>

I get this optimized search (as reported by Job Inspector):

| makeresults | rex field=_raw "(?i)(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)"

It looks like the concatenation is not needed.

Update: Change all the set tags to eval and use mvzip instead of mvmap. The set tag does not compute, while the eval tag does.

    <eval token="token_keywords_mv">split("lorem,ipsum,situs", ",")</eval>
    <eval token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</eval>
    <eval token="token_rex">"(&lt;".mvjoin(mvzip('token_keywords_mv','token_keywords_mv',"&gt;"),"+?)|(?&lt;")."+?)"</eval>
    <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>

edited Apr 06 '22 at 10:49

yaserso

2,638
5
41
73

answered Apr 05 '22 at 17:40

RichG

9,063
2
18
29

When it's a raw token, it works fine, but my token is evaluated; `mvjoin(mvmap('token_keywords_mv',"(?>".'token_keywords_mv'."<".'token_keywords_mv'."+?)"), "|")`. It works fine with a normal text, but not when a function is used. – yaserso Apr 05 '22 at 18:18
I've added a sample dashboard in my question to help experimentation. – yaserso Apr 05 '22 at 18:21
Thanks for the example code. It helped me realize the problem is the `set` keyword. Use `eval` instead if an expression must be evaluated to set the token value. – RichG Apr 05 '22 at 19:47
It doesn't seem to make much difference, however. – RichG Apr 05 '22 at 20:09
Had to use mvzip instead of mvmap and changed all the `` tags to `` tags like you said. – yaserso Apr 06 '22 at 10:47

How to use a token for a rex in Splunk?

1 Answers1