2

I am using ecCodes library in my project, and I have encountered an issue that memory is not freed between reading the files.

The minimal example representing the problem is this (and is basically a combination of those two library API usage examples [1](https://confluence.ecmwf.int/display/ECC/grib_get_keys) [2]:

#include <string>
#include <vector>
#include <iostream>
#include "eccodes.h"

int main() {

    std::string filenames[] = {"../data/era5_model.grib", "../data/era5_model2.grib", "../data/era5_model3.grib",
                               "../data/era5_model4.grib"};

    std::vector<long> vec = {};

    for (auto & filename : filenames) {
        FILE* f = fopen(filename.c_str(), "r");
        int err = 0;
        codes_handle* h;

        while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
            long k1 = 0;
            err = codes_get_long(h, "level", &k1);
            vec.push_back(k1);
        }

        codes_handle_delete(h);
        fclose(f);
    }

    std::cout << vec[52];

    return 0;
}

In the example the program reads 4 identical ERA5 files, each of size 1.5GB. Before opening new file previous one is closed with codes_handle_delete() and fclose(). Therefore, the expected behaviour would be for the memory usage to stay at about 1.5GB. However, in reality the memory usage steadily increases to about 6.5GB and is freed when program closes (see screenshot below).

screenshot showing memory usage

This particular example has been run on CLion with CMake (Release configuration), but the issue occurs with every other configuration and also in my other Rust project which calls ecCodes with FFI.

The library seems well tested and supported so it seems unlikely that it is a library bug. Therefore, is that an expected behaviour or is my code wrong? If the latter, how can I correct it?

I am using Ubuntu 21.04 and ecCodes 2.20.0 installed with apt

J.Lewandowski
  • 71
  • 1
  • 11
  • 1
    The link I gave is maybe irrelevant, but look at this: https://stackoverflow.com/questions/47384177/memory-usage-doesnt-decrease-upon-calling-free/47384216. Since your RAM is at least about 16.7 GB, try to debug it by re-iterating thrice; if it fails, then I do not know the answer, if it is not, the link I gave is relevant. – FaranAiki Sep 29 '21 at 10:16
  • You are pushing some data to vector. What is the size of vector after you complete reading each file? – kiner_shah Sep 29 '21 at 10:49
  • BTW you are not checking if the files have been opened successfully or not. You can put a safety check just after `fopen()` like this: `if (f == nullptr) return 1;` – kiner_shah Sep 29 '21 at 10:52
  • 1
    @kiner_shah the size of the vector after all iterations is 6600 elements (50 KB) so it is negligible. In my actual code I have checks for sucessful file opening in place. – J.Lewandowski Sep 29 '21 at 12:11
  • @FaranAiki Thanks for your suggestion. I did the experiment (and turned off swap for it) and the memory usage rised until all hell broke loose. So this might actually be a bug in the library. – J.Lewandowski Sep 29 '21 at 12:35

1 Answers1

2

So I contacted the library authors and realized that I have not read this example carefully enough.

For the ecCodes to correctly free the memory codes_handle should be deleted every time it is created (analogically to how you should free the memory every time you alloc it). Therefore in my example codes_handle_delete() should be INSIDE the while loop:

while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
    long k1 = 0;
    err = codes_get_long(h, "level", &k1);
    vec.push_back(k1);
    codes_handle_delete(h);
}

After that change memory usage is almost unnoticeable.

J.Lewandowski
  • 71
  • 1
  • 11