BigDL need some Spark conf values to be set correctly to have better performance.
BigDL need some Spark conf values to be set correctly to have better performance.
This method will create a SparkConf, or use the existing one if you provide one. Populate it with correct values.
We recommend you use this method instead of setting spark conf values directly. This can make the Spark conf values changes transparent to you. However, if you use spark-shell or Jupyter notebook, as the Spark context is created before your code, you have to set them directly (through command line options or properties file)
This method should be call before any BigDL procedure and after the Spark context is created.
This method should be call before any BigDL procedure and after the Spark context is created.
BigDL needs some Spark conf values to be set correctly to have a better performance. There's also multi-thread engines so executor number and core number per executor need to be known to set the parameter of these engines correctly.
The method can set parameters of multi-thread engines, verify spark conf values of an existing spark context.