Stack Overflow
Stack Overflow

Questions tagged [bulk-load]

To bulk load means to import a data file in a user-specified format into a database table or view.

351 questions
69
votes
7 answers

Django bulk_create with ignore rows that cause IntegrityError?

I am using bulk_create to loads thousands or rows into a postgresql DB. Unfortunately some of the rows are causing IntegrityError and stoping the bulk_create process. I was wondering if there was a way to tell django to ignore such rows and save as…
Meitham
  • 9,178
  • 5
  • 34
  • 45
56
votes
3 answers

In PostgreSQL, how to insert data with COPY command?

I have problem when run 1 project NodeJs with PostgreSQL database. I have error when trying to insert data in pgAdmin using the COPY command. COPY beer (name, tags, alcohol, brewery, id, brewery_id, image) FROM stdin; Bons Voeux blonde 9.5…
user4646310
51
votes
7 answers

Cannot bulk load because the file could not be opened. Operating System Error Code 3

I'm trying to set up a Stored Procedure as a SQL Server Agent Job and it's giving me the following error, Cannot bulk load because the file "P:\file.csv" could not be opened. Operating system error code 3(failed to retrieve text for this error.…
user1345260
  • 2,151
  • 12
  • 33
  • 42
43
votes
2 answers

Bulk load data into sqlite?

Does anybody have any tips on utilities that can be used to bulk load data that is stored in delimited text files into an SQLite database? Ideally something that can be called as a stand-alone program from a script etc. A group I work with has an…
Mat Nadrofsky
  • 8,289
  • 8
  • 49
  • 73
26
votes
19 answers

Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

I recently upgraded to Elasticsearch version 6.1.1 and now I can't bulk index documents from a JSON file. When I do it inline, it works fine. Here are the contents of the document: {"index" : {}} {"name": "Carlson Barnes", "age":…
Judy T Raj
  • 1,755
  • 3
  • 27
  • 41
22
votes
4 answers

What does "bulk load" mean?

Jumping from article to article, I can see everywhere the expression "bulk loading". What does it really (technically) mean? What does it imply? Explanation based on use-cases is welcome.
Spredzy
  • 4,982
  • 13
  • 53
  • 69
16
votes
1 answer

sqlalchemy bulk update performance problems

I need to increment values in a column periodically with data I receive in a file. The table has > 400000 rows. So far, all my attempts result in very poor performance. I have written an experiment that reflects my requirements: #create…
devboell
  • 1,180
  • 2
  • 16
  • 34
15
votes
2 answers

How to import tables with missing values?

I use basketball data tables to get some understanding of Postgres 9.2 & phppgadmin. Therefore I would like to import csv tables into that database. However, I get: ERROR: missing data for column "year" CONTEXT: COPY coaches, line 1:…
user3833190
13
votes
3 answers

How to convert date strings to timestamp without knowing the date format

I am trying to write a query to insert a value into a timestamp with no timezone data type field. The value is coming from CSV file. The version I am working with is PostgreSQL 8.1.21. The CSV file upload is done by the client and it has a date…
Shiver
  • 225
  • 2
  • 3
  • 6
13
votes
4 answers

INSERT of 10 million queries under 10 minutes in Oracle?

I am working on a file loader program. The purpose of this program is to take an input file, do some conversions on its data and then upload the data into the database of Oracle. The problem that I am facing is that I need to optimize the insertion…
badola
  • 820
  • 1
  • 13
  • 26
11
votes
2 answers

improving performance of mysql load data infile

I'm trying to bulk load around 12m records into a InnoDB table in a (local) mysql using LOAD DATA INFILE (from CSV) and finding it's taking a very long time to complete. The primary key type is UUID and the keys are unsorted in the data files. I've…
Michael
  • 210
  • 2
  • 7
9
votes
1 answer

Changing time zone value of data

I have to import data without time zone information in it (however, I know the specific time zone of the data I want to import), but I need the timestamp with time zone format in the database. Once I import it and set the timestamp data type to…
harbun
  • 515
  • 6
  • 23
7
votes
3 answers

MS SQL Bulk Insert

I have a requirement to insert a large 2 GB CSV file into my MS SQL database. most of the rows in this is not required to insert. I did not find anything to filter rows while doing bulk insert.I am using MS SQL bulk insert command to do this. Is…
Robi
  • 83
  • 1
  • 7
5
votes
1 answer

When creating and loading HFile programmatically to HBase new entries are unavailable

I'm trying to create HFiles programmatically and loading them in a running HBase instance. I found a lot of info in HFileOutputFormat and in LoadIncrementalHFiles I managed to create the new HFile, send it to the cluster. In the cluster web…
Nicolas
  • 277
  • 5
  • 13
5
votes
3 answers

What mysql settings affect the speed of LOAD DATA INFILE?

Let me set up the situation. We are trying to insert a modestly high number of rows (roughly 10-20M a day) into a MyISAM table that is modestly wide: +--------------+--------------+------+-----+---------+-------+ | Field | Type |…
Jason
1
2 3
…
23 24