JSON Flattening and table creation

Question

Could somebody help me to create an SQL statement to flatten JSON data in Snowflake Table1 table, in one JSON_DATA column that has an array?

JSON Data

{
  "id": "1234-567-890",
  "parent_id": "00-123-safsf-3345",
  "data": [
    {
      "id": "sfsfd-234-fgf-55-4545",
      "values": [
        {
          "name": "one",
          "value": "32"
        },
        {
          "name": "Two",
          "value": "MMAD"
        },
        {
          "name": "three",
          "value": ""
        },
        {
          "name": "four",
          "value": "Bacra-Dacra"
        },
        {
          "name": "five",
          "value": "33-5455-9"
        },
        {
          "name": "six",
          "value": ""
        },
        {
          "name": "seven",
          "value": "4056"
        },
        {
          "name": "eight",
          "value": "TUU-WWW"
        },
        {
          "name": "nine",
          "value": ""
        },
        {
          "name": "ten",
          "value": "234234"
        }
      ]
    },
    {
      "id": "asdfsdfsdf-23423-fsff-3445435",
      "values": [
        {
          "name": "One",
          "value": "32"
        },
        {
          "name": "Two",
          "value": "MMDI"
        },
        {
          "name": "Three",
          "value": ""
        },
        {
          "name": "four",
          "value": "THis is a Test"
        },
        {
          "name": "five",
          "value": "11-4543535-2"
        },
        {
          "name": "six",
          "value": ""
        },
        {
          "name": "seven",
          "value": "4056"
        },
        {
          "name": "eight",
          "value": "ert erte"
        },
        {
          "name": "nine",
          "value": ""
        },
        {
          "name": "ten",
          "value": "343534"
        }
      ]
    }
  ]
}

Table Format required :

id	one	two	three	four	five	six	seven	eight	nine	ten
sfsfd-234-fgf-55-4545	32	MMAD	:--:	Bacra-Dacra	33-5455-9	4056	TUU-WWW		234234
asdfsdfsdf-23423-fsff-3445435	32	MMDI	:--:	THis is a Test	11-4543535-2	4056	ert erte		343534

Simon D · Answer 1 · 2021-02-21T12:56:34.447

You can do this with a couple of flattens and a pivot. You need to know how many columns you want to pivot beforehand. In your example data, both records only have 10 columns but you will need to update the pivot statement if some records contain more.

I think you made a mistake with your results table. You've missed out key "six" from the json which has pushed your results back a by 1. For example, the last column "ten" should contain the numbers 234234 and 343534 but you have them in column "nine". Same deal for columns after 5.

Here is reproducible example setup:

-- create example source table
create or replace table source_table
(
    json_data variant
);

-- create example target table
create or replace table target_table
(
    id    number,
    one   varchar,
    two   varchar,
    three varchar,
    four  varchar,
    five  varchar,
    six   varchar,
    seven varchar,
    eight varchar,
    nine  varchar,
    ten   varchar
);

-- Insert json data into source table
insert overwrite into source_table (json_data)
select
    parse_json('{
  "id": "1234-567-890",
  "parent_id": "00-123-safsf-3345",
  "data": [
    {
      "id": "sfsfd-234-fgf-55-4545",
      "values": [
        {
          "name": "one",
          "value": "32"
        },
        {
          "name": "Two",
          "value": "MMAD"
        },
        {
          "name": "three",
          "value": ""
        },
        {
          "name": "four",
          "value": "Bacra-Dacra"
        },
        {
          "name": "five",
          "value": "33-5455-9"
        },
        {
          "name": "six",
          "value": ""
        },
        {
          "name": "seven",
          "value": "4056"
        },
        {
          "name": "eight",
          "value": "TUU-WWW"
        },
        {
          "name": "nine",
          "value": ""
        },
        {
          "name": "ten",
          "value": "234234"
        }
      ]
    },
    {
      "id": "asdfsdfsdf-23423-fsff-3445435",
      "values": [
        {
          "name": "One",
          "value": "32"
        },
        {
          "name": "Two",
          "value": "MMDI"
        },
        {
          "name": "Three",
          "value": ""
        },
        {
          "name": "four",
          "value": "THis is a Test"
        },
        {
          "name": "five",
          "value": "11-4543535-2"
        },
        {
          "name": "six",
          "value": ""
        },
        {
          "name": "seven",
          "value": "4056"
        },
        {
          "name": "eight",
          "value": "ert erte"
        },
        {
          "name": "nine",
          "value": ""
        },
        {
          "name": "ten",
          "value": "343534"
        }
      ]
    }
  ]
}');

select *
from (
 select
     st.json_data:id::varchar         as main_id,
     st.json_data:parent_id::varchar  as parent_id,
     data.value:id::varchar           as id,
     upper(vals.value: name::varchar) as col_name,
     vals.value: value::varchar       as col_value
 from source_table st,
      lateral flatten(input => json_data: data) data,
      lateral flatten(input => data.value: values) vals
 )
 pivot (max(col_value) for col_name in ('ONE', 'TWO', 'THREE', 'FOUR', 'FIVE', 'SIX', 'SEVEN', 'EIGHT', 'NINE', 'TEN'))

The above produces results that look like this:

MAIN_ID	PARENT_ID	ID	'ONE'	'TWO'	'THREE'	'FOUR'	'FIVE'	'SIX'	'SEVEN'	'EIGHT'	'NINE'	'TEN'
1234-567-890	00-123-safsf-3345	asdfsdfsdf-23423-fsff-3445435	32	MMDI		THis is a Test	11-4543535-2		4056	ert erte		343534
1234-567-890	00-123-safsf-3345	sfsfd-234-fgf-55-4545	32	MMAD		Bacra-Dacra	33-5455-9		4056	TUU-WWW		234234

score 0 · Answer 2 · answered Feb 23 '22 at 16:27

add your json to S3 or other storage layer or inline, create stage and try below method

    create or replace table DATABASE.STAGE.jsonSrc
(
    src variant
)
as 
select parse_json($1) as src
from DATABASE.STAGE.json_flatten f;

select src:id as ID, 
src:parent_id as P_ID,
src:data.id as DATA_ID,
src:data.values.name as DV_NAME
from DATABASE.STAGE.jsonSrc;

you can simply traverse through the json by '.' notation and passing attributes.

JSON Flattening and table creation

2 Answers2