Implement f-string like magic to pimp Django's format_html()

Question

I would like to pimp format_html() of Django.

It already works quite nicely, but my IDE (PyCharm) thinks the variables are not used and paints them in light-gray color:

AFAIK f-strings use some magic rewriting.

Is there a way to implement this, so that the IDE knows that the variables get used?

Related: Implement f-string like syntax, with Django SafeString support

Here is my current implementation:

def h(html):
    """
    Django's format_html() on steroids
    """
    def replacer(match):
        call_frame = sys._getframe(3)
        return conditional_escape(
            eval(match.group(1), call_frame.f_globals, call_frame.f_locals))
    return mark_safe(re.sub(r'{(.*?)}', replacer, html))

Somebody raised security concerns: I don't plan to create CMS where a user can edit these templates. These template h-strings are only for developers to have a convenient way to create HTML.

Before writing an answer, be sure you know the magic of conditional_escape()

Those calls don't look right - aren't you supposed to do `format_html('{foo}', foo=foo)`? — user2357112, Mar 28 '21 at 09:22
Oh - `h` is a thing you defined, not just a funny `as` alias for `format_html`. — user2357112, Mar 28 '21 at 09:24
This isn't "magic" - f-strings are an actual language feature, so the devs at JetBrains implemented the logic for that in the IDE (see https://youtrack.jetbrains.com/issue/PY-18972 and a bunch of other issues linked to it). I'd guess it'd be considered outside their responsibility to go looking for any use of frame hacks that could possibly be referencing variables in a given scope for the purposes of marking them unused. — jonrsharpe, Mar 28 '21 at 14:18
@Melvyn the django template language lets some errors pass silently. That's ok in some cases, in some not. And for a django template I would need to pass in the Context somehow. I like the f-string way, it is easy, nice, DRY. I enjoy typing, but typing foo=foo too often does not feel productive. — guettli, Mar 28 '21 at 20:39
And yet, f strings replace whatever is available and allow code execution. Are you ready to deal with the security nightmare? — , Mar 28 '21 at 20:54
@Melvyn I updated the question to include the security perspective. — guettli, Mar 29 '21 at 16:23
I believe that your options are: 1. Disable unused local variables inspection. 2. Write a plugin for PyCharm that suppresses this inspection when you use your function. (probably doable by adding a special comment and using `com.intellij.lang.inspectionSuppressor` extension point) 3. Write a plugin that figures out what's used inside the template, and marks only those variables as used. 4. Pass `locals()` as second argument like `h('{a}', locals())` to make use of PyCharm exception for this case. Won't help with nested functions and `globals()` — Marian, Apr 05 '21 at 11:45
I tried this with VSCode and the highlighting works as you would expect. Not sure why PyCharm does it differently — Salvatore, Apr 06 '21 at 22:03
Tangentially, this ought to have used `string.Formatter` instead of a blunt `re.sub`. — user3840170, Apr 09 '21 at 11:23
@user3840170 could you please elaborate what you mean with `string.Formatter`? — guettli, Apr 09 '21 at 11:31
For example, `string.Formatter().parse` can parse the formatting template for you and correctly take care of escaping constructs like `{{`. — user3840170, Apr 09 '21 at 11:40
Does this answer your question? [How to postpone/defer the evaluation of f-strings?](https://stackoverflow.com/questions/42497625/how-to-postpone-defer-the-evaluation-of-f-strings) — user3840170, Apr 17 '21 at 18:34
@user3840170 the question is "Is there a way to implement this, so that the IDE knows that the variables get used?" I looked at your link ("how to postpone/defer ...") and I guess this solution has the same problem: The IDE thinks that the variables are unsed. — guettli, Apr 18 '21 at 09:21

user3840170 · Answer 1 · 2021-07-30T19:31:24.293

Since you don’t seem above using dirty hacks, here’s a hack even dirtier than the one in the question:

class _escaper(dict):
    def __init__(self, other):
        super().__init__(other)
        
    def __getitem__(self, name):
        return conditional_escape(super().__getitem__(name))

_C = lambda value: (lambda: value).__closure__[0]
_F = type(_C)
try:
    type(_C(None))(None)
except:
    pass
else:
    _C = type(_C(None))

def h(f):
    if not isinstance(f, _F):
        raise TypeError(f"expected a closure, a {type(f).__name__} was given")
    closure = None
    if f.__closure__:
        closure = tuple(
            _C(conditional_escape(cell.cell_contents))
            for cell in f.__closure__
        )
    fp = _F(
        f.__code__, _escaper(f.__globals__),
        f.__name__, f.__defaults__, closure
    )
    return mark_safe(fp())

The h function takes a closure, and for each variable closed over, it creates another, escaped copy of the variable, and modifies the closure to capture that copy instead. The globals dict is also wrapped to ensure references to globals are likewise escaped. The modified closure is then immediately executed, its return value is marked safe and returned. So you must pass h an ordinary function (no bound methods, for example) which accepts no arguments, preferably a lambda returning an f-string.

Your example then becomes:

foo = '&'
bar = h(lambda: f'<span>{foo}</span>')
assert h(lambda: f'<div>{bar}</div>') == '<div><span>&amp;</span></div>'

There’s one thing, though. Due to how this has been implemented, you can only ever refer directly to variables inside interpolation points; no attribute accesses, no item lookups, nothing else. (You shouldn’t put string literals at interpolation points either.) Otherwise though, it works in CPython 3.9.2 and even in PyPy 7.3.3. I make no claims about it ever working in any other environment, or being in any way future-proof.

Implement f-string like magic to pimp Django's format_html()

1 Answers1

Linked