apache spark - How to load CSVs with timestamps in custom format? -


i have timestamp field in tiny csv file reading dataframe using spark csv library. same piece of code works on local machine databricks spark 2.0 version throws error on azure hortownworks hdp 3.5 , 3.6. have checked , azure hd insight 3.5 using same spark version isn't problem spark version.

import org.apache.spark.sql.types._ val sourcefile = "c:\\2017\\datetest" val sourceschemastruct = new structtype()   .add("eventdate",datatypes.timestamptype)   .add("name",datatypes.stringtype) val df = spark.read   .format("com.databricks.spark.csv")   .option("header","true")   .option("delimiter","|")   .option("mode","failfast")   .option("inferschema","false")   .option("dateformat","yyyy/mm/dd hh:mm:ss.sss")   .schema(sourceschemastruct)   .load(sourcefile)

java.lang.illegalargumentexception: timestamp format must yyyy-mm-dd hh:mm:ss[.fffffffff]

the csv file has 1 row follows:

"eventdate"|"name" "2016/12/19 00:43:27.583"|"adam"

i've managed reproduce issue in latest spark version 2.2.0-snapshot (built master).

// os shell $ cat so-43259485.csv "eventdate"|"name" "2016/12/19 00:43:27.583"|"adam"  // spark-shell case class event(eventdate: java.sql.timestamp, name: string) import org.apache.spark.sql.encoders val schema = encoders.product[event].schema  scala> spark   .read   .format("csv")   .option("header", true)   .option("mode","failfast")   .option("delimiter","|")   .schema(schema)   .load("so-43259485.csv")   .show(false) 17/04/08 11:03:42 error executor: exception in task 0.0 in stage 7.0 (tid 7) java.lang.illegalargumentexception: timestamp format must yyyy-mm-dd hh:mm:ss[.fffffffff]     @ java.sql.timestamp.valueof(timestamp.java:237)     @ org.apache.spark.sql.catalyst.util.datetimeutils$.stringtotime(datetimeutils.scala:167)     @ org.apache.spark.sql.execution.datasources.csv.univocityparser$$anonfun$makeconverter$9$$anonfun$apply$17$$anonfun$apply$6.apply$mcj$sp(univocityparser.scala:146)     @ org.apache.spark.sql.execution.datasources.csv.univocityparser$$anonfun$makeconverter$9$$anonfun$apply$17$$anonfun$apply$6.apply(univocityparser.scala:146)     @ org.apache.spark.sql.execution.datasources.csv.univocityparser$$anonfun$makeconverter$9$$anonfun$apply$17$$anonfun$apply$6.apply(univocityparser.scala:146)     @ scala.util.try.getorelse(try.scala:79)

the corresponding line in spark sources following:

timestamp.valueof(s)

having read javadoc of timestamp.valueof, can learn argument should be:

timestamp in format yyyy-[m]m-[d]d hh:mm:ss[.f...]. fractional seconds may omitted. leading 0 mm , dd may omitted.

note "the fractional seconds may omitted" let's cut off first loading eventdate string , after removing unneeded fractional seconds convert timestamp.

val eventsasstring = spark.read.format("csv")   .option("header", true)   .option("mode","failfast")   .option("delimiter","|")   .load("so-43259485.csv")

spark 2.1.0

use schema inference in csv using inferschema option custom timestampformat.

it's important trigger schema inference using inferschema timestampformat take effect.

val events = spark.read   .format("csv")   .option("header", true)   .option("mode","failfast")   .option("delimiter","|")   .option("inferschema", true)   .option("timestampformat", "yyyy/mm/dd hh:mm:ss")   .load("so-43259485.csv")  scala> events.show(false) +-------------------+----+ |eventdate          |name| +-------------------+----+ |2016-12-19 00:43:27|adam| +-------------------+----+  scala> events.printschema root  |-- eventdate: timestamp (nullable = true)  |-- name: string (nullable = true)

"incorrect" initial version left learning purposes

val events = eventsasstring   .withcolumn("date", split($"eventdate", " ")(0))   .withcolumn("date", translate($"date", "/", "-"))   .withcolumn("time", split($"eventdate", " ")(1))   .withcolumn("time", split($"time", "[.]")(0))    // <-- remove millis part   .withcolumn("eventdate", concat($"date", lit(" "), $"time")) // <-- make eventdate right   .select($"eventdate" cast "timestamp", $"name")  scala> events.printschema root  |-- eventdate: timestamp (nullable = true)  |-- name: string (nullable = true)     events.show(false)  scala> events.show +-------------------+----+ |          eventdate|name| +-------------------+----+ |2016-12-19 00:43:27|adam| +-------------------+----+

spark 2.2.0 (not released yet)

as of spark 2.2 (which not available yet) can use to_timestamp function string timestamp conversion.

eventsasstring.select($"eventdate", to_timestamp($"eventdate", "yyyy/mm/dd hh:mm:ss.sss")).show(false)  scala> eventsasstring.select($"eventdate", to_timestamp($"eventdate", "yyyy/mm/dd hh:mm:ss.sss")).show(false) +-----------------------+----------------------------------------------------+ |eventdate              |to_timestamp(`eventdate`, 'yyyy/mm/dd hh:mm:ss.sss')| +-----------------------+----------------------------------------------------+ |2016/12/19 00:43:27.583|2016-12-19 00:43:27                                 | +-----------------------+----------------------------------------------------+

Comments

Post a Comment

Popular posts from this blog

inversion of control - Autofac named registration constructor injection -

does autofac support specifying registration name in components' constructors? example: ninject's namedattribute . you need use autofac.extras.attributed package on top achieve this so let's have 1 interface , 2 classes: public interface ihello { string sayhello(); } public class englishhello : ihello { public string sayhello() { return "hello"; } } public class frenchhello : ihello { public string sayhello() { return "bonjour"; } } then have consumer class, in want select instance injected: public class helloconsumer { private readonly ihello helloservice; public helloconsumer([withkey("en")] ihello helloservice) { if (helloservice == null) { throw new argumentnullexception("helloservice"); } this.helloservice = helloservice; } public string sayhello() { return this.helloservice.sayhello()...
Read more

verilog - Systemverilog dynamic casting issues -

i've code snippet following in testbench function void write_to_port( my_data_type_base data ); my_data_type_extended data_ext; if(!$cast(data_ext, data)); `uvm_error(get_type_name(), "failed cast"); `uvm_info(get_name(), $psprintf("data_ext :\n%s", data_ext.sprint()), uvm_medium) // write data_ext out port.... endfunction when run it, i'm getting uvm_error "failed cast." i'm not quire sure why $cast not returning 1. can see, i'm printing out extended class data item after casting uvm_info. can see it's being cast properly. if don't use $cast if condition, don't runtime error. isn't coding practice use if dynamic cast check if $cast returning 1 ? what might reason behind cast not returning 1 in above case? i think semicolon on line 'if' not belong? i think consumes if statement, , uvm_error executes regardless of how if evaluates: if(!$cast(data_ext, data)); <- no semico...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more