0

trying to use subprocess python module to apply some commands if a dbt macro logic is not met

macro:

{% macro check_select_arg(run_all=False, dbt_command='dbt run') %}
{% do log("ON START ✅", info=True) %}

{% set subprocess = imprt( 'subprocess' ) %}

{% if run_all %}
{{subprocess.run(dbt_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)}}

{% elif not invocation_args_dict.select
        and target.name not in ['prod']
        and invocation_args_dict.which in ['build', 'run', 'test', 'source', 'snapshot', 'seed'] %}
    {{ exceptions.raise_compiler_error("❌ Error: You must provide at least one select argument") }}

{% endif %}

{% endmacro %}

my python code:

import jinja2
import importlib
FOLDER_PATH = "check_select_arg.sql"


environment = jinja2.Environment()

template_file = open(FOLDER_PATH, "r")
template_string = template_file.read()
template_file.close()
template = environment.from_string(template_string)
template.render(imprt = importlib.import_module)

the problem is that when i run the command dbt run-operation check_select_arg --arg "{run_all: True, dbt_command: 'dbt run'}"

it returns the following error: Encountered an error while running operation: Compilation Error 'imprt' is undefined

i was expecting to be able to access subprocess and thus, run the dbt command.

thanks all for the help in advance

tangen14
  • 21
  • 3
  • so what heck is 'imprt' and where did you define it? – Serge Aug 09 '23 at 16:56
  • ```template.render(imprt = importlib.import_module)``` – tangen14 Aug 09 '23 at 17:05
  • 1
    On a *philosophical* point of view, what you are trying to do is heresia. From its foundation, Jinja is a template engine, not a general usage language. The expectation is that you provide you Jinja template with a set of high level routines and that the gory details (module import, subprocess management) are handled by those Python routines. A number of procedural concepts are of course directly available in Jinja, but they are intended to support the template, not to be used as a general program. TL/DR export the logic to a dedicated Python module. – Serge Ballesta Aug 09 '23 at 17:14
  • Correct, yet, again, on a philosophical note, I have seen people using a template engine to generate executable code rather than mere markup. Most often javascript, but sometimes staying in the same language. Which maybe not the best pattern, but does not require understanding of meta-programming and may work with some skill. – Serge Aug 09 '23 at 17:16
  • Or even double dipping by using templates that yield new templates. Which may work in this case (though might be not recommended) – Serge Aug 09 '23 at 17:31
  • 1
    In Jinja 2 you need special import so context is visible. Also you might need use context_functions https://stackoverflow.com/a/54774811/5874981 – Serge Aug 09 '23 at 18:33
  • 1
    see also https://stackoverflow.com/a/54774811/5874981 – Serge Aug 09 '23 at 19:41

1 Answers1

0

While I totally agree with Serge Ballesta recommendation to avoid to use templates for complex things such as subprocessing, there must be a technical Jinja2 bug in your code : whenever you use a context in a macro, macro should be imported along with the context, or, with Jinja 2.1 and up use include instead of import

See https://tedboy.github.io/jinja2/templ12.html

Serge
  • 3,387
  • 3
  • 16
  • 34