How to make an analog of InString[]?

Question

I have discovered that InString[] does not work in MathLink mode when sending input with EnterExpressionPacket header. So I need to define my own function that returns previous input line. One way I have developed here does not work in some cases:

In[1]:= Unevaluated[2 + 2]
With[{line = $Line - 1}, HoldForm[In[line]]] /. (DownValues[In])
Out[1]= Unevaluated[2 + 2]
Out[2]= 2 + 2

This is because RuleDelayed has no HoldAllComplete attribute. Adding this attribute makes this OK:

In[1]:= Unprotect[RuleDelayed];
SetAttributes[RuleDelayed, HoldAllComplete];
Protect[RuleDelayed];
Unevaluated[2 + 2]
With[{line = $Line - 1}, HoldForm[In[line]]] /. DownValues[In]

Out[4]= Unevaluated[2 + 2]

Out[5]= Unevaluated[2 + 2]

But modifying built-in functions generally is not a good idea. Is there a better way to do this?

If you use simply In[$Line-1] the result is Unevaluated[2+2], as it should. :) (in this case, of course) — Dr. belisarius, Feb 18 '11 at 02:37
@Alexey yeap, was just a comment. You solved a problem to have it again in the simplest case. Sometimes life is like that! :*) — Dr. belisarius, Feb 18 '11 at 03:31
@Alexey You raised a very interesting point. I was in fact unaware of such behavior of `RuleDelayed`. But the culprit is likely `RuleDelayed` itself, not the absence of `HoldAllComplete` attribute. Rather, you can use the latter to get what you want, but that's not the only way. For example, `Block[{RuleDelayed = Hold}, DownValues[In]]` will show the original form as well. Normally, `Unevaluated` wrappers should be restored if no evaluation took place. Apparently, `RuleDelayed` violates this, e.g. `a :> Unevaluated[b]`. For other wrappers with just normal `HoldAll`, `Unevaluated` is restored. — Leonid Shifrin, Feb 18 '11 at 10:48
@Leonid I have [already exploited](http://stackoverflow.com/questions/5036907/how-to-make-an-analog-of-instring/5038576#5038576) the method with `Block`. Is there a way to make the code shorter? — Alexey Popkov, Feb 18 '11 at 14:00
@Alexei `Block` seems the only way to go. I wasn't suggesting a new method in my comment. I was just pointing out that the problem is not in the absence of `HoldAllComplete`, but in the peculiarities of the implementation of `RuleDelayed`, which go at odds with the standard evaluation sequence. The distinction may not play a role here, but IMO is important for anyone who wants a detailed and correct understanding of various aspects of evaluation (frankly, I was puzzled for a while. For me, this was important to understand, may be some other people will find it interesting too). — Leonid Shifrin, Feb 18 '11 at 14:27
@Alexei But if you want shorter code, here it is: `Clear[getLastInput]; getLastInput := #[[Position[#[[All, 1]], _?(! FreeQ[#, $Line - 1] &), 1, 1][[1, 1]], 2]] &@ Block[{RuleDelayed = Hold}, DownValues[In]]` — Leonid Shifrin, Feb 18 '11 at 14:44
@Alexey Sorry for the name spelling typo. I have a few friends with this other spelling - must have been a typing reflex — Leonid Shifrin, Feb 18 '11 at 14:58
@Leonid Why did you say above "For other wrappers with just normal HoldAll, `Unevaluated` is restored." Is `Unevaluated` restored or just leaved unchanged? And about you code above. Try to evaluate `getLastInput` twice after defining it. The first time it shows nothing, the second generates error. — Alexey Popkov, Feb 19 '11 at 23:21
@Alexey Ok, sorry, forgot about the `HoldForm`. Try any of these, should work: `getLastInput := First@Extract[ Extract[#, Position[#[[All, 1]], _?(! FreeQ[#, $Line - 1] &), 1, 1][[1, 1]]], {{2}}, HoldForm] &@ Block[{RuleDelayed = Hold}, DownValues[In]]` or `getLastInput := With[{line = $Line - 1, myHold = Unique["hold", {HoldAll}]}, HoldForm[ In[line]] /. (Block[{RuleDelayed = myHold}, Map[HoldForm, DownValues[In], {2}]] /. myHold -> RuleDelayed)]`. Both are a bit longer than my first suggestion, but neither uses `HoldComplete`, which was my main point — Leonid Shifrin, Feb 20 '11 at 13:11
@Alexey Regarding `Unevaluated` being stripped off/ restored, you won't find this in the documentation, but search for old WRI technical report by David Withoff named "Mathematica internals" - it is there, along with some other interesting details. It should be freely available on the web. — Leonid Shifrin, Feb 20 '11 at 13:15
@Alexey Here is a more elegant solution which best illustrates my point about `HoldComplete` and `RuleDelayed`: `getLastInput := With[{line = $Line - 1}, HoldForm[In[line]] /. Block[{RuleDelayed}, SetAttributes[RuleDelayed, HoldAll]; Map[HoldForm, DownValues[In], {2}]]]` — Leonid Shifrin, Feb 20 '11 at 13:44
@Leonid The last example is very interesting but I do not understand the underlying mechanism. Why this method does not work in the case `HoldForm[x] /. Block[{RuleDelayed}, SetAttributes[RuleDelayed, HoldAll]; x :> Unevaluated[1 + 1]]` (I mean `Unevaluated` is removed)? And I should point out that all your functions cannot work with inputs with `Evaluate` head. — Alexey Popkov, Feb 20 '11 at 16:35
@Alexey Your example does not work because you leave the scope of `Block` before the rule is applied, so `RuleDelayed` gets a chance to remove `Unevaluated` (just as if you'd use it directly). Regarding `Evaluate` - true, for *this* you need `HoldComplete` (change `HoldForm` to `HoldComplete` everywhere in my code, and also `HoldAll` to `HoldAllComplete` in `SetAttributes`). But here we use `HoldComplete` to solve a different problem, namely, deal with explicit `Evaluate` entered at the top-level. — Leonid Shifrin, Feb 20 '11 at 16:59
@Alexey But apparently, there is something even more interesting going on. Namely, we can not `Block` `RuleDelayed` completely, in the sense that it will still do replacements (perhaps, it is not about `RuleDelayed` but about replacement operators). Anyways, we can use this to our advantage, to produce even more elegant solution: `getLastInput := Block[{RuleDelayed}, SetAttributes[RuleDelayed, HoldAll]; With[{line = $Line - 1}, HoldForm[In[line]] /. DownValues[In]]]`. The same remark here - replace `HoldForm` with `HoldComplete` and `HoldAll` with `HoldAllComplete` to deal with `Evaluate`. — Leonid Shifrin, Feb 20 '11 at 17:02
@Alexey There is at least one more case where you need `HoldComplete` to get it right. Here is an illustration (try it with `HoldForm` - based `getLastInput`): `Clear[a]; a /: f_[l_, a] := f[l, 1]; a getLastInput` (again, all statements on separate lines. There should be 4 lines here in this example, with `a` occupying a separate line) — Leonid Shifrin, Feb 20 '11 at 17:18
The last `getLastInput` function is just brilliant! The only sad thing is that such things probably will never be officially documented because they breaks the official slogan of "deeply integrated system". — Alexey Popkov, Feb 20 '11 at 17:31
@Alexey I don't see all this stuff as evidence against the "slogan". The only surprising thing for me was the mentioned partial evaluation inside `RuleDelayed`. It indeed contradicts somewhat the phrase in the documentation that `RuleDelayed` "does not evaluate its r.h.s". But, so to say, the exceptions are "of the measure zero". It would be a documentation problem for any industrial system which exposes its internals to the end user (who then can do almost as much as a developer). I personally quite like the latter approach and consider such cases as a necessary evil - the price for power. — Leonid Shifrin, Feb 20 '11 at 17:54
@Alexey I agree that this is indeed a big problem. I think that some intermediate (perhaps, Python - style) language layer would be very beneficial to have within Mathematica, since lots of problems don't inherently require the full knowledge of say evaluation mechanics to be solved, but currently this deep layer is not "insulated" enough, and lots of users get confused. I'd go even further and say that a general set of language-building tools to build domain-specific little languages which would compile into Mathematica code, would be great, but that's not an easy task. — Leonid Shifrin, Feb 20 '11 at 18:44

Alexey Popkov · Accepted Answer · 2011-03-31T08:16:03.267

It seems that I have solved the problem. Here is the function:

In[1]:=
getLastInput := Module[{num, f},
    f = Function[{u, v},
        {u /. {In -> num, HoldPattern -> First}, HoldForm[v]}, HoldAllComplete];
    First@Cases[
        Block[{RuleDelayed = f}, DownValues[In]],
        {$Line - 1, x_} -> x, {1}, 1]]

In[2]:=
Unevaluated[2+2]
getLastInput

Out[2]=
Unevaluated[2+2]

Out[3]=
Unevaluated[2+2]

And I just have got the answer to the question on InString in MathLink mode from Todd Gayley (Wolfram Research):

InString is only assigned when using EnterTextPacket, not EnterExpressionPacket. There is no string form of the input when sending EnterExpressionPacket (whose content is, by definition, already an expression).

EDIT:

I just have found that my code does not work with input expressions with head Evaluate. The solution is to replace HoldForm by HoldComplete in my code:

getLastInput := Module[{num, f},
    f = Function[{u, v},
        {u /. {In -> num, HoldPattern -> First}, HoldComplete[v]}, HoldAllComplete];
    First@Cases[
        Block[{RuleDelayed = f}, DownValues[In]],
        {$Line - 1, x_} -> x, {1}, 1]]

This works well. Another approach would be to unprotect HoldForm and set up attribute HoldAllComplete on it. I'm wondering why HoldForm does not have this attribute by default?

EDIT 2:

In the comments for the main question Leonid Shifrin suggested much better solution:

getLastInput := 
 Block[{RuleDelayed},SetAttributes[RuleDelayed,HoldAllComplete];
  With[{line=$Line-1},HoldComplete[In[line]]/.DownValues[In]]]

See comments for details.

EDIT 3: The last code can be made even better for by replacing HoldComplete by double HoldForm:

getLastInput := 
 Block[{RuleDelayed},SetAttributes[RuleDelayed,HoldAllComplete];
  With[{line=$Line-1},HoldForm@HoldForm[In[line]]/.DownValues[In]]]

The idea is taken from presentation by Robby Villegas of Wolfram Research at the 1999 Developer Conference. See subsection "HoldCompleteForm: a non-printing variant of HoldComplete (just as HoldForm is to Hold)" in "Working With Unevaluated Expressions" notebook posted here.

score 1 · Answer 2 · answered Feb 18 '11 at 14:36

I would use $Pre and $Line for this; unlike $PreRead, it's applied to input expressions, not input strings or box forms. All you need is to assign it a function that has the HoldAllComplete attribute, like this one which I've adapted from the example in the documentation:

SetAttributes[saveinputs, HoldAllComplete];
saveinputs[new_] :=
 With[{line = $Line},
  inputs[line] = HoldComplete[new]; new]
$Pre = saveinputs;

I tested this with MathLink, and the behavior seems to be what you desired (I've elided some of the transcript to highlight the key point):

In[14]:= LinkWrite[link,
 Unevaluated[
  EnterExpressionPacket[
   SetAttributes[saveinputs, HoldAllComplete];
   saveinputs[new_] :=
    With[{line = $Line},
     inputs[line] = HoldComplete[new]; new];
   $Pre = saveinputs;]]]

In[15]:= LinkRead[link]
Out[15]= InputNamePacket["In[2]:= "]

In[20]:= LinkWrite[link,
 Unevaluated[EnterExpressionPacket[Evaluate[1 + 1]]]]

In[21]:= LinkRead[link]
Out[21]= OutputNamePacket["Out[2]= "]

In[21]:= LinkRead[link]
Out[21]= ReturnExpressionPacket[2]

In[24]:= LinkWrite[link, Unevaluated[EnterExpressionPacket[DownValues[inputs]]]]

In[26]:= LinkRead[link]
Out[26]= ReturnExpressionPacket[
  {HoldPattern[inputs[2]] :> HoldComplete[Evaluate[1 + 1]], 
   HoldPattern[inputs[3]] :> HoldComplete[DownValues[inputs]]}]

Good idea! But your code more correctly should be written as `SetAttributes[saveinputs, HoldAllComplete]; saveinputs[new_] := (inputs[$Line] = HoldComplete[new]; Unevaluated[new]); $Pre = saveinputs;`. — Alexey Popkov, Feb 20 '11 at 02:58

score 0 · Answer 3 · answered Feb 18 '11 at 11:15

0

I just have found simpler but dangerous way:

In[3]:= Unevaluated[2 + 2]
Trace[In[$Line - 1]] // Last
Trace[In[$Line - 1]] // Last

Out[3]= Unevaluated[2 + 2]

Out[4]= Unevaluated[2 + 2]

During evaluation of In[3]:= $RecursionLimit::reclim: Recursion depth of 256 exceeded. >>

During evaluation of In[3]:= $RecursionLimit::reclim: Recursion depth of 256 exceeded. >>

During evaluation of In[3]:= $IterationLimit::itlim: Iteration limit of 4096 exceeded. >>

Out[5]= Hold[In[$Line-1]]

Does anybody know a way to make it safe?

answered Feb 18 '11 at 11:15

Alexey Popkov

9,355
4
42
93

1

This probably can not be made safe, since `Trace` will re-evaluate the input. Therefore, if any stage of the evaluation of the input contained side effects, you will change the global state in a generally irreversible way - and there is no easy way for you to determine the presence or absence of side effects. As a result, in the following, for example, `i = 0; i++ Trace[In[$Line - 1]] // Last`, (the `i=0` and `i++` are assumed to occupy separate lines, the comment format does not allow me to do this), `i` becomes 2 after you call `Trace`, while it was 1 before that. – Leonid Shifrin Feb 18 '11 at 16:58
It is interesting how `On[]` works: it does not re-evaluate anything! – Alexey Popkov Feb 19 '11 at 23:29
But in some cases `On[]` can easily be broken. Try for example `On[General::newsym]; HoldComplete[a]; Names["```*"]`. It does not generate message that symbol `a` is created! On the other side it seems that `HoldComplete` should not create new symbols! – Alexey Popkov Feb 19 '11 at 23:38
1

You are probably using version 7, where the stuff related to event-handling of creation of new symbols was broken. It seems to have been fixed in v.8. Regarding general mechanics of new symbols creation, this happens at parse-time, not evaluation-time, so `HoldComplete` is irrelevant. If you want to delay the symbol's creation until run-time, use `Symbol["symbol name"]`. I gave an example of its usage here: http://stackoverflow.com/questions/4988427/accidental-shadowing-and-removedsymbol/4991503#4991503 – Leonid Shifrin Feb 20 '11 at 13:23

How to make an analog of InString[]?

3 Answers3

Linked