5

All the ..Values functions have undocumented option Sort:

In[1]:= Options /@ {OwnValues, DownValues, UpValues, SubValues, 
  DefaultValues, FormatValues, NValues}

Out[1]= {{Sort -> True}, {Sort -> True}, {Sort -> True}, {Sort -> 
   True}, {Sort -> True}, {Sort -> True}, {Sort -> True}}

What this option does and for which purposes it may be useful?

Alexey Popkov
  • 9,355
  • 4
  • 42
  • 93

1 Answers1

8

When you enter your definitions for a function, you enter them in a particular order. In case when your definitions do not contain patterns, they are stored internally in a hash table. When you request them, they will be sorted by default:

In[11]:= 
Clear[g];
g[5]=1;
g[4]=2;
g[3]=3;

In[15]:= DownValues[g]
Out[15]= {HoldPattern[g[3]]:>3,HoldPattern[g[4]]:>2,HoldPattern[g[5]]:>1}

However, often you may want to see the original order in which the values were assigned. Here is how:

In[16]:= DownValues[g,Sort->False]
Out[16]= {HoldPattern[g[5]]:>1,HoldPattern[g[4]]:>2,HoldPattern[g[3]]:>3}

An example of one instance where you may want to use it, is when you need to implement a cache (my attempt to do that based on this option can be found here). For a large number of definitions, however, it is probably not guaranteed that the definitions will follow in the original order, since the internal hash-tables will probably be expanded and rehashed several times. Generically, a hash table implementation does not guarantee the order in which the key-value pairs are stored. So, what you achieve by setting Sort->False is that the ...Values are not sorted by Mathematica before they are returned to you, so you get them pretty much in the order Mathematica currently stores them.

Another case when you may want this is when you want to build a dictionary-like structure using DownValues - in this case, key extraction will be much faster with Sort->False, since no sorting on the key set has to be performed (matters for large key sets):

In[31]:= 
 Clear[gg];
(gg[#]:=200000-#)&/@Range[200000];

In[33]:= DownValues[gg][[All,1,1,1]]//Short//Timing
Out[33]= {4.867,{1,2,3,<<199994>>,199998,199999,200000}}

In[34]:= DownValues[gg,Sort->False][[All,1,1,1]]//Short//Timing
Out[34]= {2.103,{95090,102286,<<199996>>,38082,26686}}

You can find an example of such use e.g. here.

As far as I know, the Sort option only affects the non-pattern-based DownValues (and other ...Values), because for pattern-based DownValues Mathematica anyway attempts to reorder them in the order of their generality, as it understands that. OTOH, for pattern-based DownValues, you may do manual rule reordering and Mathematica will keep your changes, while for definitions without patterns, attempts to reorder the definitions manually after they were originally given seem to have no effect on them (perhaps, again because they are hashed internally, and hash-tables don't generally care about order).

Community
  • 1
  • 1
Leonid Shifrin
  • 22,449
  • 4
  • 68
  • 100
  • 1
    could you expand a bit on "...since the internal hash-tables will probably be expanded and rehashed several times"? – acl Aug 15 '11 at 09:41
  • 2
    @acl A typical hash-table has a certain capacity - a number of elements it can effectively hash without the collision probability becoming too high. When we keep adding definitions, at some point the hash-table is likely to perform dynamic resizing, increasing its capacity. It may or may not need to re-hash existing entries, depending on implementation. These things are very nicely explained here: http://en.wikipedia.org/wiki/Hash_table. Most hash tables do not maintain the order of the keys, but there are some (e.g. `LinkedHashMap` in Java) which do. – Leonid Shifrin Aug 15 '11 at 09:50
  • thanks. I guess I was more asking whether you have observed mma doing this and, if yes, how. – acl Aug 15 '11 at 09:57
  • 1
    @acl Well, regarding the ordering of keys, my second example shows that keys are in no particular order. My experiments show that the original order is only maintained for very small key sets, about 15 elements or less. There may be a parameter in `SystemOptions` controlling this, but I did not find it. As for dynamic growing, look at these system options: `SystemOptions["HashTableOptions"]` (I have `{"HashTableOptions" -> {"GrowLoadFactor" -> 2., "ShrinkLoadFactor" -> 0.5}}`). Load factor is basically an average number of entries per bucket (the cited Wikipedia article has more info). – Leonid Shifrin Aug 15 '11 at 10:19
  • Indeed with a bit more effort I did manage to make them reorder themselves (I had made some superficial tests and they did not reorder). Thanks. – acl Aug 15 '11 at 10:34
  • 1
    @Leonid: As always, leaving the question to be answered by you has paid off! Thanks for the interesting read. +1 – Simon Aug 15 '11 at 11:02
  • @Simon Don't make it a habit :) Eventually, I may let you down :) – Leonid Shifrin Aug 15 '11 at 12:02
  • +1 for your elaboration. Once again you do more than merely answer the question. – Mr.Wizard Aug 17 '11 at 00:00