Log4j2 displays log in foreach within executor log but not in map method in executor log-CodePudding

I am using Spark 2.3.0

When I use foreach within dataframe and logging I am able to see the logs in executor log. But why is it not printing the logs in map method?

val df = Seq((0)).toDF("a")

df.foreachPartition { iterator =>
  {
    iterator.map { row =>
      {
        val LOGGER = LogManager.getLogger(getClass.getName)
        Configurator.setRootLevel(Level.INFO);
        LOGGER.info("Testing logger in executor")
      }
    }
  }
} //Not printing in executor log

df.foreachPartition { iterator =>
  {
    iterator.foreach { row =>
      {
        val LOGGER = LogManager.getLogger(getClass.getName)
        Configurator.setRootLevel(Level.INFO);
        LOGGER.info("Testing logger in executor")
      }
    }
  }
} //Prints executor logs

What is the reason for this and is there a way to achieve this(I am using Log4j2 with log42.properties file)?

CodePudding user response：

foreach is eager while map is lazy.