Mathematica Table function

Question

I'm running a Table function which will take too much time to complete.

I wanted to know if there's a way to retrieve the results computed so far.

Answer 1

The proposed solution

Here is a version of Table that is Abort-able and will keep the intermediate results collected so far. It is a modified version of the solution posted here.

ClearAll[abortableTable];
SetAttributes[abortableTable, HoldAll];
abortableTable[expr_, iter__List] :=
  Module[{indices, indexedRes, sowTag},
  SetDelayed @@ 
     Prepend[Thread[Map[Take[#, 1] &, List @@ Hold @@@ Hold[iter]], 
       Hold], indices];
  indexedRes =
     If[# === {}, #, First@#] &@Last@Reap[
        CheckAbort[Do[Sow[{expr, indices}, sowTag], iter], {}], sowTag];
  AbortProtect[
     Map[First,
       SplitBy[indexedRes,
          Table[
            With[{i = i}, Function[Slot[1][[2, i]]]], 
            {i, Length[Hold[iter]] - 1}]], 
       {-3}]]];

It should be able to take the same iterator specification as Table.

How it works

Here is how it works. The first statement (SetDelayed @@...) "parses" the iterators, assuming that they are each of the form {iteratorSymbol_,bounds__}, and assigns the list of iterator variables to the variable indices. The construction with Hold is needed to prevent possible evaluation of iterator variables. There are many ways to do this, I used just one of them. Here is how it works:

In[44]:= 
{i, j, k} = {1, 2, 3}; 
Prepend[Thread[Map[Take[#, 1] &, List @@ Hold @@@ 
   Hold[{i, 1, 10}, {j, 1, 5}, {k, 1, 3}]], Hold], indices]

Out[45]= Hold[indices, {i, j, k}] 

Using SetDelayed @@ the-above will then naturally produce the delayed definition of the form indices:={i,j,k}. I assigned the values to indices i,j,k to demonstrate that no unwanted evaluation of them happens when using this construct.

The next statement produces a list of collected results, where each result is grouped in a list with the list of indices used to produce it. Since indices variable is defined by delayed definition, it will evaluate every time afresh, for a new combination of indices. Another crucial feature used here is that the Do loop accepts the same iterator syntax as Table (and also dynamically localizes the iterator variables), while being a sequential (constant memory) construct. To collect the intermediate results, Reap and Sow were used. Since expr can be any piece of code, and can in particular also use Sow, a custom tag with a unique name is needed to only Reap those values that were Sown by our function, but not the code it executes. Since Module naturally produces (temporary) symbols with unique name, I simply used a Module - generated variable without a value, as a tag. This is a generally useful technique.

To be able to collect the results in the case of Abort[] issued by the user interactively or in the code, we wrap the Do loop in CheckAbort. The code that is executed on Abort[] ({} here) is largely arbitrary in this approach, since the collection of results is anyway done by Sow and Reap, although may be useful in a more elaborate version that would save the result into some variable provided by the user and then re-issue the Abort[] (the functionality not currently implemented).

As a result, we get into a variable indexedRes a flat list of the form

{{expr1, {ind11,ind21,...indn1}},...,{exprk, {ind1k,ind2k,...indnk}}

where the results are grouped with the corresponding index combination. We need these index combinations to reconstruct the multi-dimensional resulting list from a flat list. The way to do it is to repeatedly split the list according to the value of i-th index. The function SplitBy has this functionality, but we need to provide a list of functions to be used for splitting steps. Since the index of i-th iterator index in the sublist {expr,{ind1,...,indn}} is 2,i, the function to do the splitting at i-th step is #[[2, i]]&, and we need to construct the list of such functions dynamically to feed it to SplitBy. Here is an example:

In[46]:= Table[With[{i = i}, Function[Slot[1][[2, i]]]], {i, 5}]

Out[46]= {#1[[2, 1]] &, #1[[2, 2]] &, #1[[2, 3]] &, #1[[2, 4]] &, #1[[2, 5]] &}

The With[{i=i},body] construct was used to inject the specific values of i inside pure functions. The alternatives to inject the value of i into Function do exist, such as e.g.:

In[75]:= 
Function[Slot[1][[2, i]]] /. Map[List, Thread[HoldPattern[i] -> Range[5]]]

Out[75]= {#1[[2, 1]] &, #1[[2, 2]] &, #1[[2, 3]] &, #1[[2, 4]] &, #1[[2, 5]] &}

or

In[80]:= Block[{Part}, Function /@ Thread[Slot[1][[2, Range[5]]]]]

Out[80]= {#1[[2, 1]] &, #1[[2, 2]] &, #1[[2, 3]] &, #1[[2, 4]] &, #1[[ 2, 5]] &}

or

In[86]:= Replace[Table[{2, i}, {i, 5}], {inds__} :> (#[[inds]] &), 1]

Out[86]= {#1[[2, 1]] &, #1[[2, 2]] &, #1[[2, 3]] &, #1[[2, 4]] &, #1[[ 2, 5]] &}

but are probably even more obscure (perhaps except the last one).

The resulting nested list has a proper structure, with sublists {expr,{ind1,...,indn}} being at level -3 (third level from the bottom). By using Map[First,lst,{-3}], we remove the index combinations, since the nested list has been reconstructed already and they are no longer needed. What remains is our result - a nested list of resulting expressions, whose structure corresponds to the structure of a similar nested list produced by Table. The last statement is wrapped in AbortProtect - just in case, to make sure that the result is returned before the possible Abort[] fires.

Example of use

Here is an example where I pressed Alt+. (Abort[]) soon after evaluating the command:

In[133]:= abortableTable[N[(1+1/i)^i],{i,20000}]//Short
Out[133]//Short= {2.,2.25,2.37037,2.44141,<<6496>>,2.71807,2.71807,2.71807}

It is almost as fast as Table:

In[132]:= abortableTable[N[(1+1/i)^i,20],{i,10000}]//Short//Timing
Out[132]= {1.515,{2.0000000000000000000,2.2500000000000000000,<<9997>>,2.7181459268252248640}}

In[131]:= Table[N[(1+1/i)^i,20],{i,10000}]//Short//Timing
Out[131]= {1.5,{2.0000000000000000000,2.2500000000000000000,<<9997>>,2.7181459268252248640}}

But it does not auto-compile while Table does:

In[134]:= Table[N[(1+1/i)^i],{i,10000}]//Short//Timing
Out[134]= {0.,{2.,2.25,2.37037,2.44141,<<9993>>,2.71815,2.71815,2.71815}}

One can code the auto-compilation and add it to the above solution, I just did not do it since it will be a lot of work to do it right.

EDIT

I rewrote the function to make some parts both more concise and easier to understand. Also, it is about 25 % faster than the first version, on large lists.

ClearAll[abortableTableAlt];
SetAttributes[abortableTableAlt, HoldAll];
abortableTableAlt[expr_, iter : {_Symbol, __} ..] :=
  Module[{indices, indexedRes, sowTag, depth =  Length[Hold[iter]] - 1},
   Hold[iter] /. {sym_Symbol, __} :> sym /. Hold[syms__] :> (indices := {syms});
   indexedRes =  Replace[#, {x_} :> x] &@ Last@Reap[
      CheckAbort[Do[Sow[{expr, indices}, sowTag], iter], Null],sowTag];
   AbortProtect[
      SplitBy[indexedRes, Array[Function[x, #[[2, x]] &], {depth}]][[##,1]] & @@ 
      Table[All, {depth + 1}]
   ]];

Answer 2

Unfortunately no. If you want to do something like lst=Table[f[i],{i,1,10000}] so that if aborted you still have results, you could do

Clear[lst2];
lst2 = {};
(Do[lst2 = {lst2, f[i]}, {i, 1, 10000}];
lst2=Flatten[lst2];) // Timing

which, for undefined f, takes 0.173066s on my machine, while lst = Table[f[i], {i, 1, 100000}] takes roughly 0.06s (ie, Table it is 3 times faster at the expense of not being interruptible).

Note that the obvious "interruptible" solution, lst = {}; Do[AppendTo[lst, f[i]], {i, 1, 100000}] takes around 40s, so don't do that: use linked lists and flatten at the end, like in my first example (however, that will break if f[i] returns a list, and more care is then needed).

Answer 3

Another solution is to export results of intermediate computations to a running log file as described in this answer by WReach (see the "File-backed In-memory Approach" section). With this you will newer loose results of intermediate computations and will always be able to investigate what is computed so far.

P.S. I think usage of Monitor as suggested in this recent Mathematica tip on twitter is also useful in such cases:

Monitor[Table[Integrate[1/(x^n + 1), x], {n, 20}], 
 ProgressIndicator[n, {1, 20}]]