In

And to efficiently transfer data between JVM and Python processes We can configure Apache arrow with spark here are some links for same

Official Apache arrow documentation

Apache arrow and spark configuration