0

I tried many times and an error Error: cannot allocate vector of size 893.9 Mb always appears. Is it not possible for R to allocate such size of data without using arrow?

I cannot use skimr::skim to conduct summary statistics... An error is showed:

Error in `dplyr::summarize()`:
ℹ In argument: `skimmed = purrr::map2(...)`.
Caused by error in `purrr::map2()`:
ℹ In index: 1.
ℹ With name: Date.
Caused by error in `dplyr::summarize()`:
ℹ In argument: `dplyr::across(tidyselect::any_of(variable_names), mangled_skimmers$funs)`.
Caused by error in `across()`:
! Can't compute column `Date`.
Caused by error in `order()`:
! Failed to realloc working memory 109972517*8bytes (xsub in dradix), radix=6
Run `rlang::last_error()` to see where the error occurred.
doraemon
  • 439
  • 3
  • 10
  • 6
    The error means you're out of memory; you might have to subset your data or do the merge with `arrow` before trying to import the data into memory for subsequent analysis. R can do the merge but only if there are enough computing resources. – rps1227 Mar 21 '23 at 09:00
  • 2
    If you post the code that's currently 'failing', you might get some potential solutions, e.g. perform the join using data.table, or filter the data prior to the join, or look at other languages (e.g. awk) – jared_mamrot Mar 21 '23 at 10:02
  • 2
    I'll pile on this conversation briefly: not only is making this more [reproducible](https://stackoverflow.com/q/5963269) (sample data, code used, etc) _in general_ a good thing, for larger-data operations it becomes very important to demonstrate the size, structure, and variability of the data so that we can attempt/benchmark larger samples of it. Often simple solutions on 10 lines scale geometrically and will not work for you. Since it's arrow, I suggest using a couple of small frames wrapped in `arrow_table` so that we can attempt arrow-only dplyr operations. – r2evans Mar 21 '23 at 13:10
  • Thank you for your advice. I will try to post the reproducible data and code tonight. Please give me some time... – doraemon Mar 22 '23 at 02:23

0 Answers0