Stack Overflow
Stack Overflow

Questions tagged [staging-table]

Staging tables are used in data ETL (extraction transformation and loading) processes and data warehousing.

Staging tables are intermediate storage tables for storing data prior to loading into the main data warehouse. Staging tables schema have similar structure to the source table and are used reduce calls to the source table before loading into the main data warehouse.

18 questions
37
votes
3 answers

What is a staging table?

Are staging tables used only in Data warehouse project or in any SSIS Project? I would like to know what is a staging table? Can anyone give me some examples on how to use it and in what circumstances it is implemented? Also, may I please know the…
Suj
  • 416
  • 1
  • 5
  • 11
5
votes
5 answers

How to properly truncate a staging table in an ETL pipeline?

We have an ETL pipeline that runs for each CSV uploaded into an storage account (Azure). It runs some transformations on the CSV and writes the outputs to another location, also as CSV, and calls a stored procedure on the database (SQL Azure) which…
empz
  • 11,509
  • 16
  • 65
  • 106
2
votes
1 answer

Designing a staging table to import data

I'm using SQL Server 2012 Standard Edition After setting an SSIS package that imports data from a flat file, I wanted to know how would I proceed to insert data into staging tables and what the structure of staging tables is?
Ahmed Attia
  • 127
  • 1
  • 10
2
votes
2 answers

How to archive old data in Sybase database?

I am working on an application which is built on Struts as server side java technology and Sybase ASE 15 is used as database for storing the data for the application. Now the tables in Sybase have been filled with huge chunk of data, so higher…
azaveri7
  • 793
  • 3
  • 18
  • 48
2
votes
1 answer

Best strategy to perform ETL from staging to warehouse

I am building a C# console application that will be executed by SQL Server Agent in a scenario where we want to transfer a big amount of employee data: Between two employee tables with the same schema on two different networks from one SQL Server…
Jean-François Beaulieu
  • 4,305
  • 22
  • 74
  • 107
1
vote
0 answers

How to best stage large amounts of data with Hibernate/JPA?

How can I best stage large amounts of data for migration into our database using Hibernate efficiently? Performance when dealing with >25K records that are 100+ columns are not ideal. Let me explain: Background I'm working for a large company that…
ElderFuthark
  • 496
  • 2
  • 11
0
votes
0 answers

Flat File Staging For Format Conversion:

I have a code the converts data from one format to another by inserting the data manually. I need assistance on how to stage the flat/txt file and apply the same code to come up with same results. DBFiddle https://dbfiddle.uk/cwfvntlt -- DDL and…
Jimrosy P Madzokere
  • 23
  • 4
0
votes
0 answers

Business key combination

I have a Dimension in my DW in which 4 columns together forms the business key ,so there is no single BK. To insert into the fact table (Staging area) i would need a FK, so should the fact table have the combination of these 4 columns (4 FK) for…
rafamaniac
  • 57
  • 7
0
votes
0 answers

Loading local CSV into snowflake

After 4 days of trying everything to load data into snowflake, nothing seems to work at all. Now as my last option I want to load a local CSV file into snowflake in order to be able to follow the tutorial I am watching. Unfortunately even this step…
Siver R
  • 13
  • 7
0
votes
2 answers

Get all constraint errors when inserting data from another table

I have a staging table without any constraints in my Azure SQL database (Azure SQL database 12.0.2000.8). I want to insert the data from the Staging table into the "real" table on which multiple constraints are set. When inserting the data, I use a…
Lennart
  • 383
  • 4
  • 16
0
votes
2 answers

Getting duplicates in the Table when an ETL job Is ruined twice.ETL job fetch data from RDS to S3 bucket

When the ETL job is run it execute properly but as the table is not having Timestamp it duplicate the data when the same ETL job is run.How to perform staging and solve this problem using Upsert or if any other you are welcome to answer.How do I get…
RAHUL VISHWAKARMA
  • 99
  • 3
  • 14
0
votes
1 answer

T-SQL | Better alternative to WHERE NOT EXISTS

I've the following query: INSERT INTO [Table A] ([student_name], [class_id],[contact_detail], [birth_date],[note_average]) SELECT [student_name] = case when CHARINDEX('.', [student_name])>0 then LEFT([student_name],CHARINDEX('.',…
John_Rodgers
  • 181
  • 1
  • 11
0
votes
1 answer

Modeling DW staging from JSON

I am building a DW. The sources are comming from rest API that returns Json. I need to design a staging area. I think I have 2 approaches: 1. Transform Json into a relational model. 2. Store the Json into a relational table using a key value. The…
p.magalhaes
  • 7,595
  • 10
  • 53
  • 108
0
votes
2 answers

Where to create staging data table in BigData environment?

I am currently having Hadoop-2, PIG, HIVE and HBASE. I have an inputdata. I have loaded that data in HDFS. I want to create staging data in this environment. My query is - In which BigData component, I should create Staging Table(Pig/HIVE/HBASE) ;…
user3343543
  • 143
  • 2
  • 14
0
votes
1 answer

RoR: resetting the staging table after processing user-uploaded csv file?

I am pretty new to Ruby on Rails and have been studying it using the Ruby on Rails Tutorial by Michael Hartl. I am now working on my own project, which allows users to log in the website, provide personal biometric information and upload a csv file…
codeinstyle
  • 63
  • 1
  • 1
  • 4
1
2