Google openrefine don't load big csv-file

Question

When i try to create project, i load csv file with 3,5 millions rows(400mb) and refine doesn't upload it. it indicates 100% 1037 mb i opened refine.ini and fixed memory limit, but there is no result

 NOTE: This file is not read if you run the Refine executable directly
# It is only read of you use the refine shell script or refine.bat

no_proxy="localhost,127.0.0.1"
#REFINE_PORT=3334
#REFINE_HOST=127.0.0.1
#REFINE_WEBAPP=main\webapp

# Memory and max form size allocations
#REFINE_MAX_FORM_CONTENT_SIZE=104857600
REFINE_MEMORY=100000M

# Set initial java heap space (default: 256M) for better performance with large datasets
REFINE_MIN_MEMORY=100000M

# Some sample configurations. These have no defaults.
#ANT_HOME=C:\grefine\tools\apache-ant-1.8.1
#JAVA_HOME=C:\Program Files\Java\jdk1.6.0_25
#JAVA_OPTIONS=-XX:+UseParallelGC -verbose:gc -Drefine.headless=true
#JAVA_OPTIONS=-Drefine.data_dir=C:\Users\user\AppData\Roaming\OpenRefine

# Uncomment to increase autosave period to 60 mins (default: 5 minutes) for better performance of long-lasting transformations
#REFINE_AUTOSAVE_PERIOD=60

What i should do?

I'm pretty sure you do not have 100 000 megabytes of RAM on your computer ... But that kind of technical question is not very appropriate for StackOverflow. Could you post it on the [Open Refine Google Group](https://groups.google.com/forum/#!forum/openrefine)? — Ettore Rizza, Dec 21 '17 at 13:37
Wow. But in any case, such an amount of RAM exceeds the possibilities of OpenRefine. Since you seem to be on Linux, have you tried to run Open Refine on the command line, for example with `./refine -m 15G` ? — Ettore Rizza, Dec 21 '17 at 14:12
@EttoreRizza. i use ms2012 server:)) what coomand i should enter and where in windows, here the screen)) http://s020.radikal.ru/i701/1712/67/5f6ffa912b0b.jpg — psysky, Dec 21 '17 at 14:57
If you are on Windows, you had to edit the `refine.l4j.ini` file, and not refine.ini : https://github.com/OpenRefine/OpenRefine/wiki/FAQ:-Allocate-More-Memory — Ettore Rizza, Dec 21 '17 at 15:01
@EttoreRizza, yes i saw this hint, but when i edited ini-file, openrefine couldn't launch, that why i decided to create topic here, ok, i gonna write to google group — psysky, Dec 21 '17 at 15:05

score 1 · Answer 1 · answered Dec 21 '17 at 16:09

Based on the testing I did and published at https://groups.google.com/d/msg/openrefine/-loChQe4CNg/eroRAq9_BwAJ, to process 3.5 million rows you probably need to allocate around 8Gb RAM to have a reasonably responsive project.

As documented in OpenRefine changing the port and host when executable is run directly, when running OpenRefine on Windows where you set the options depends on whether you are starting OpenRefine via the exe file or the bat file.

To allocate over 4Gb of RAM, you definitely need to be using a 64-bit Java version - please check what version of Java OpenRefine is running in (it will use the Java specified in JAVA_HOME). However, you may find issues allocating 4Gb on 32-bit Java on a 64-Bit OS (see Maximum Java heap size of a 32-bit JVM on a 64-bit OS)

Google openrefine don't load big csv-file

1 Answers1