Hive metadata repository for Spark usage

Don't forget to add path to the Hive configuration folder in spark-defaults.conf:

spark.driver.extraClassPath /opt/hive/conf/

It must appears in the Spark application UI on Environment tab.

Without it Spark can not found you config and will be use defaults and create derby.log and metastore_db in current folder every time then you use hive context.

Simple way to update rows in DataFrame

Lifehack for oldschool peoples, who prefer RDBMS-like approach.

  DataFrame reslt = sqlContext.createDataFrame(  
   srcDatafr.javaRDD().map(row -> {  
   return RowFactory.create(row.get(0), row.get(1), row.get(2), "You value", row.get(4) );  
   }), srcDatafr.schema());  
   reslt.printSchema();  
   reslt.show();   
   }     

Very beautiful solution, get it here.