0

I have a large Map in matlab and want to traverse all the keys of it. But I can only store a cell for the keys first and traverse the cell, which calls for memory to store all the keys. While in python, the generator/iterator can sequentially give the indexes or entries of list back in loops, which does not cost much memory.

In matlab, I tried for loop:

% Let MAP is a containers.Map with 10000000 keys.
keys = MAP.keys;
for keyIdx = 1:size(keys,2)
    MAP(keys{1,keyIdx});
end

While in python, the for loop with iterator:

% Let MAP is a dictionary with 10000000 keys.
for key in MAP.iterkeys():
    MAP[key]

So, is there generator/iterator like that in python in matlab to save memory when I process the maps?

L. Li
  • 113
  • 2
  • 7
  • This doens't exist in pure MATLAB. But as stated in this [question](https://stackoverflow.com/questions/21099040/what-is-the-matlab-equivalent-of-the-yield-keyword-in-python) you can implement it yourself – v.tralala Dec 23 '18 at 18:29
  • Note that MATLAB uses lazy copying, which means that, as long as you don’t modify `keys`, it will point to the same memory that stores the keys in the map. Don’t worry about memory usage with MATLAB unless you run out of memory! :) – Cris Luengo Dec 23 '18 at 19:32
  • Thank you@CrisLuengo.`keys = MAP.keys` actually costs memory to store the `cell` _keys_, while I cannot traverse the keys by directly using `MAP.keys(idx)` or `MAP.keys{idx}`. – L. Li Dec 24 '18 at 01:44
  • I didn’t get a notification for your comment, make sure you have a space before the `@`! – Cris Luengo Dec 25 '18 at 06:11
  • 1
    What I said was that `keys = MAP.keys` doesn’t take up any additional memory, as the cell array `keys` will point to the same memory for the keys as they are stored inside the `MAP`. MATLAB doesn’t copy the data until you try to modify the copy. – Cris Luengo Dec 25 '18 at 06:11

2 Answers2

2

To answer your main question: No. There is no generator in MATLAB. Also, the Python code you wrote does not use a Python generator. The memory footprint of both snippets are similar. But I think I did not get the details about your question....

Last:

  • you can use numel(keys) instead of size(keys, 2) since keys is of one dimension
  • you can use MAP(keys{keyIdx}) instead of MAP(keys{1, keyIdx})
erkandem
  • 136
  • 1
  • 7
  • Thank you! `.iterkeys()` is the generator method for dictionary in python where a key is returned once in a loop, thus a generator in python can avoid costing the memory to store the whole key cell of MAP as in matlab. Am I wrong? I cannot understand your _'The memory footprint of both snippets are similar'_. – L. Li Dec 24 '18 at 01:52
  • is that a generator? I miss the ``yield`` syntax. I thought the "one element at a time" feature is used to save memory for **what 's stored in ``value``** – erkandem Dec 24 '18 at 21:58
  • Yes. In python, `.values()` return the list of all values of a dictionary while `.itervalues()` is an iterator (sorry I mixed `iterator` and `generator`, and learned from [Difference between Python's Generators and Iterators](https://stackoverflow.com/questions/2776829/difference-between-pythons-generators-and-iterators)), so do `.keys()` and `.iterkeys()`. – L. Li Dec 25 '18 at 02:53
  • So, I have no way to avoid storing the huge key cell of the MAP in matlab? Or is there any other ways to traverse all the key-values of maps in matlab? – L. Li Dec 25 '18 at 02:55
2

MATLAB uses the lazy copy or copy-on-write machinism. This means that when you do

B = A;

both A and B point to the same memory until you try to modify one of the arrays, at which point MATLAB will copy the data so that modifying one doesn’t affect the other copy.

This means also that, given a Containers.Map object map,

keys = map.keys

will not copy the keys, but instead keys will share data with the keys as they are stored inside the map.

Note that

keys = map.keys;
for keyIdx = 1:numel(keys)
   value = map(keys{keyIdx})
end

is the same as

for key = map.keys
   value = map(key{1})
end

and the same as

for value = map.values
   value = value{1}
end

In none of these cases is data copied!

(I am guessing at the syntax, as I’m typing on my phone right now, I think the {1} indexing is required here, but I might be wrong about it.)

Cris Luengo
  • 55,762
  • 10
  • 62
  • 120