How to flatten nested JSON arrays to single array object

Question

I have 50,000+ json objects buried in nested arrays. I want to pull them out so that they can be in single array. Also these nested arrays are quite random having no pattern. E.g

[ [ [ [ [ {"a":1 } ], [ {"b":2 } ] ], [[{"c":3 }]] ] ], [{"d":4 }] ]

has to be converted to

[{"a":1},{"b":2},{"c":3},{"d":4}]

using Dataweave 2.0

Used flatten but doesn't look like its the right function.

flatten works on the first level of an array, not in nested arrays. — aled, Nov 03 '22 at 19:04

score 5 · Accepted Answer · answered Nov 03 '22 at 18:55

5

Here's one way to do it with a recursive function:

%dw 2.0
output application/json
fun flattenAllLevels(arr: Array) = do {
    arr reduce ((item, acc = []) -> 
        item match {
            case x is Array -> acc ++ flattenAllLevels(x)
            else ->  acc << item
        }
    )
}
---
flattenAllLevels(payload)

answered Nov 03 '22 at 18:55

Shoki

1,508
8
13

Since the number of object is large and can come in any format you should keep [Dataweave recursion limit](https://stackoverflow.com/questions/58105923/dataweave-recursion-limit) in mind – Harshank Bansal Nov 04 '22 at 04:27

score 1 · Answer 2 · answered Nov 04 '22 at 12:59

This is another version of flattening the nested Arrays. It is most likely to be less efficient then the already accepted solution because it needs more iteration. However, this uses Tail Recursion so you do not have to worry about Dataweave recursion limit and so it can even handle deeper nesting level.

If you need to handle heavily nested Arrays (let's say > 50) you can use this. Otherwise, Shoki's solution is more effective

%dw 2.0
import some from dw::core::Arrays
output application/json

@TailRec()
fun flattenAllLevels(arr: Array) = do {
    var flattenedLevel1 = flatten(arr)
    ---
    if(flattenedLevel1 some ($ is Array)) flattenAllLevels(flattenedLevel1)
    else flattenedLevel1
}

// Helper function to create deeply nested array. 
// for example, calling getDeepArray(5) will create [5,[4,[3,[2,[1,[0]]]]]]
fun getDeepArray(level) = 
    1 to level
    reduce ((item, acc=[0]) -> ( [item] << acc ))
---
flattenAllLevels(getDeepArray(500))

How to flatten nested JSON arrays to single array object

2 Answers2