Why do the persist() and cache() methods shorten DataFrame plan in Spark?-CodePudding

The

Please help me figure it out. I need to get the full plan in json format.

UPD(1):

Now I'm trying to convert each node to json separately. Now it doesn't work perfectly, but I think we need to go in this direction.

val jsonPlan = s"[${getJson(result_df.queryExecution.optimizedPlan).mkString(",")}]" def getJson(lp: TreeNode[_]): Seq[String] = { val children = (lp.innerChildren lp.children.map(c => c.asInstanceOf[TreeNode[_]])).distinct JsonMethods.compact(JsonMethods.render(JsonMethods.parse(lp.toJSON)(0))) : getJson(t.asInstanceOf[TreeNode[_]]))) children.flatMap(t => getJson(t)) }

CodePudding user response：

Well, you already kinda have your answer.

TLDR: Upgrade.

You could, could, look into the github repo at the logical plan code, and look for difference in how logical plans are calculated between. 3.0.1 and 3.1.2. (Or look at the code for persist and see how it's changed.) You could then back port a patch to 3.0.1. But you'd still need to build a new version of spark and then deploy it so that the plan returned. But if you are doing all that work why not upgrade to 3.1.2 if you know it works? (Or some later version of Spark?)

(You must have some dependency on a sub-component that only compatible with 3.0.1?)

CodePudding user response：

You can cherry pick below 2 commits into spark 3.0.1 to fix this issue.

* 1603775934 - [SPARK-35411][SQL][FOLLOWUP] Handle Currying Product while serializing TreeNode to JSON (8 months ago) <Tengfei Huang> * 9804f07c17 - [SPARK-35411][SQL] Add essential information while serializing TreeNode to json (9 months ago) <Tengfei Huang>