Monday, October 17, 2016

Convert dataframe/dataset to libsvm format row in Spark 2.0 in Java



Creating typed dataset. No difference, it's just a type that you pass into map function later


            SparkSession ss = SparkSession.builder().appName("JavaALSExample").master("local[*]").getOrCreate();  
            Dataset dsout = ss.read().format("parquet").load("hdfs://spark16/user/gollum/ads_par_clear").as(Encoders.bean(AdsFrame.class));  
            dsout.printSchema();  
            dsout.show();  

Create struct of required format

 StructType libsvmSchema = new StructType().add("label", "double").add("features", new VectorUDT());  

 Convert

            Dataset dsLibsvm = ss.createDataFrame(  
            dsout.javaRDD().map(new Function() {  
                 public Row call(AdsFrame a) {  
                      Double label = (double) 0;  
                      Vector currentRow = Vectors.dense(a.getAlti(), a.getLati(), a.getLongi(), a.getSpeed(), a.getTrck() );  
                      return RowFactory.create(label, currentRow);  
                 }  
                 }),  
            libsvmSchema);  

I have not tried it in [Structured] streaming, but can works too

No comments:

Post a Comment