I'm trying to achieve something similar using spark and scala

Updating BigQuery data using Java https://cloud.google.com/bigquery/docs/updating-data

I want to update existing data and also insert new data into Bigquery table. Any ideas if we can using some sort of DML within spark to do an upsert operation against BigQuery ??

I found that BigQuery supports merge but I'm not sure if we can do something similar using spark and scala Google BQ - how to upsert existing data in tables?

CodePudding user response：

The spark API does not support upsert yet. The best workaround at this moment is to write the dataframe to a temporary table, run a MERGE job and then delete the temporary table.