I am posting this question after searching a lot on the web but couldn't find the answer. I have a JSONArray in below format

[ 
  {
    "firstName":"John",
    "lastName":"Doe",
    "deparment" : {
       "DeptCode":"10",
       "deptName" : "HR"
     }
  },
  {
    "firstName":"Mel",
    "lastName":"Gibson",
    "deparment" : {
       "DeptCode":"20",
       "deptName" : "IT"
 }
}
]

The JSONArray is from org.json.simple.JSONArray package. I am trying to convert this into Java Spark Dataframe. I was trying with the below code :

SparkConf conf = new SparkConf().setAppName("linecount").setMaster("local[*]");
SparkSession session = SparkSession.builder().config(conf).getOrCreate();       
Dataset<Row> dataset = session.read().json(array.toString());

But no luck. I am facing below error. Also I can see in scala we can convert it to Dataframe using DS method. has someone tried this before ?

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: [{"firstName":"John","lastName":"Doe"},{"firstName":"Mel","lastName":"Gibson"}]
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:172)
at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:615)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:349)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:333)
at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:279)
at com.vikas.rawat.AnotherMainClass.main(AnotherMainClass.java:34)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: [{"firstName":"John","lastName":"Doe"},{"firstName":"Mel","lastName":"Gibson"}]
at java.net.URI.checkPath(Unknown Source)
at java.net.URI.<init>(Unknown Source)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
... 14 more

CodePudding user response：

You should create a RDD from JSON string and pass that to spark.read.json method.

SparkSession spark = SparkSession.builder().master("local").getOrCreate();

String s = "{\"root\":[ \n"  
                "  {\n"  
                "    \"firstName\":\"John\",\n"  
                "    \"lastName\":\"Doe\",\n"  
                "    \"deparment\" : {\n"  
                "       \"DeptCode\":\"10\",\n"  
                "       \"deptName\" : \"HR\"\n"  
                "     }\n"  
                "  },\n"  
                "  {\n"  
                "    \"firstName\":\"Mel\",\n"  
                "    \"lastName\":\"Gibson\",\n"  
                "    \"deparment\" : {\n"  
                "       \"DeptCode\":\"20\",\n"  
                "       \"deptName\" : \"IT\"\n"  
                " }\n"  
                "}\n"  
                "]}";
JSONObject json = (JSONObject) JSONValue.parse(s);
JSONArray msgsArray = (JSONArray) json.get("root");

scala.collection.Seq<String> seq = scala.collection.JavaConverters.asScalaIteratorConverter
                       (Arrays.asList(msgsArray.toString()).iterator()).asScala().toSeq();

RDD<String> jsonRDD = spark.sparkContext().
parallelize(seq, 4, scala.reflect.ClassTag$.MODULE$.apply(String.class));

spark.read().json(jsonRDD).show();


 --------- --------- -------- 
|deparment|firstName|lastName|
 --------- --------- -------- 
| {10, HR}|     John|     Doe|
| {20, IT}|      Mel|  Gibson|
 --------- --------- -------- 

CodePudding user response：

You can import json from a string to a dataset, but there is a caveat that it has to be one object per string.

Spark Documentation:

// Alternatively, a DataFrame can be created for a JSON dataset represented by // a Dataset[String] storing one JSON object per string val otherPeopleDataset = spark.createDataset( """{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}""" :: Nil) val otherPeople = spark.read.json(otherPeopleDataset) otherPeople.show()