Hive metadata repository for Spark usage

Don't forget to add path to the Hive configuration folder in spark-defaults.conf:

spark.driver.extraClassPath /opt/hive/conf/

It must appears in the Spark application UI on Environment tab.

Without it Spark can not found you config and will be use defaults and create derby.log and metastore_db in current folder every time then you use hive context.