How convert array of struct into struct in spark?-CodePudding

There is an array field in dataset like:

my_array:
[
{id: 1, value: x},
{id: 2, value: y}
]

How make it like:

my_strcut: {
  1: {value: x},
  2: {value: y}
}

I have tried map_from_entries with transform but still have array of structs as output.

UPDATED

There is a dataset which read data from json. Data like that:

{"id":1, ... "arrayOfStructs" : [{"name": "x", "key":"value"}, {"name": "y", "key":"value2"}]}

The output should be something the like:

{"id":1, ... "structsOnly" : { "x": {"name": "x", "key":"value"}}, { "y": {"name": "y", "key":"value2"}}}

CodePudding user response：

I think you want to use MapType not StructType in this case, as struct requires you to know all the values for field id. Something like this using transform aggregate functions:

val df1 = df.withColumn(
    "structsOnly",
    expr("""aggregate(
              transform(arrayOfStructs, x -> map(x.name, x)), 
              cast(map() as map<string,struct<name:string,key:string>>), 
              (acc, x) -> map_concat(acc, x)
           )
    """)
  ).drop("arrayOfStructs")

df1.printSchema
//root
// |-- id: integer (nullable = false)
// |-- structsOnly: map (nullable = true)
// |    |-- key: string
// |    |-- value: struct (valueContainsNull = true)
// |    |    |-- name: string (nullable = true)
// |    |    |-- key: string (nullable = true)

df1.toJSON.show(false)
// --------------------------------------------------------------------------------------- 
//|value                                                                                  |
// --------------------------------------------------------------------------------------- 
//|{"id":1,"structsOnly":{"x":{"name":"x","key":"value"},"y":{"name":"y","key":"value2"}}}|
// ---------------------------------------------------------------------------------------

Now, if you really want to have struct type column then you'll need to collect all possible values of field key then construct the the column like this:

val keys = df1.select(map_keys($"structsOnly")).as[Seq[String]].collect.flatten.distinct

val df2 = df1.withColumn(
  "structsOnly",
  struct(keys.map(k => col("structsOnly").getField(k).as(k)): _*)
)