apache spark - Issue in creating Hive Context in Itellij -


while creating hive context finding below error. please me resolve . have attached code , pom file , exception thins post .

while creating hive context finding below error. please me resolve . have attached code , pom file , exception thins post .

while creating hive context finding below error. please me resolve . have attached code , pom file , exception thins post .

pom file:-

<?xml version="1.0" encoding="utf-8"?> <project xmlns="http://maven.apache.org/pom/4.0.0"          xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"          xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">     <modelversion>4.0.0</modelversion>      <groupid>scalatest</groupid>     <artifactid>test</artifactid>     <version>1.0-snapshot</version>      <dependencies>          <dependency>             <groupid>org.apache.spark</groupid>             <artifactid>spark-core_2.10</artifactid>             <version>1.6.0</version>             <type>test-jar</type>             <scope>test</scope>         </dependency>          <dependency>             <groupid>org.scalatest</groupid>             <artifactid>scalatest_2.10</artifactid>             <version>2.2.6</version>         </dependency>          <dependency>             <groupid>org.apache.spark</groupid>             <artifactid>spark-sql_2.10</artifactid>             <version>1.6.0</version>           </dependency>          <dependency>             <groupid>org.apache.spark</groupid>             <artifactid>spark-hive_2.10</artifactid>             <version>1.5.0</version>           </dependency>         <!-- scala -->          <dependency>             <groupid>org.scala-lang</groupid>             <artifactid>scala-library</artifactid>             <version>2.10.5</version>             <scope>compile</scope>             <optional>true</optional>         </dependency>          <dependency>                <groupid>log4j</groupid>                <artifactid>log4j</artifactid>                <version>1.2.17</version>         </dependency>          <dependency>             <groupid>org.scala-lang</groupid>             <artifactid>scala-compiler</artifactid>             <version>2.10.5</version>             <scope>compile</scope>             <optional>true</optional>         </dependency>         <!-- end scala -->     </dependencies>     <build>         <sourcedirectory>src/main/scala</sourcedirectory>         <plugins>             <plugin>                 <artifactid>maven-assembly-plugin</artifactid>                 <version>2.2.1</version>                 <configuration>                     <descriptorrefs>                         <descriptorref>jar-with-dependencies</descriptorref>                     </descriptorrefs>                 </configuration>                 <executions>                     <execution>                         <id>make-assembly</id>                         <phase>package</phase>                         <goals>                             <goal>single</goal>                         </goals>                     </execution>                 </executions>             </plugin>             <plugin>                 <groupid>net.alchim31.maven</groupid>                 <artifactid>scala-maven-plugin</artifactid>                 <version>3.2.0</version>                 <executions>                     <execution>                         <goals>                             <goal>compile</goal>                             <goal>testcompile</goal>                         </goals>                     </execution>                 </executions>                 <configuration>                     <sourcedir>src/main/scala</sourcedir>                      <jvmargs>                         <jvmarg>-xms64m</jvmarg>                         <jvmarg>-xmx1024m</jvmarg>                     </jvmargs>                 </configuration>             </plugin>         </plugins>     </build>  </project>

code:-

import org.apache.spark.sharedsparkcontext import org.scalatest.funsuite import org.apache.spark.{sparkcontext, sparkconf} import org.apache.spark.sql.hive.hivecontext  /**   * created pswain on 4/5/2017.   */ class test extends funsuite sharedsparkcontext  {     test("test initializing spark context") {     //val list = list(1, 2, 3, 4)     //val rdd = sc.parallelize(list)     val hc = new hivecontext(sc)     //assert(list.length === rdd.count())   } }

exception:-

a needed class not found. due error in runpath. missing class: org/apache/spark/sql/sqlcontext$sqlsession java.lang.noclassdeffounderror: org/apache/spark/sql/sqlcontext$sqlsession     @ test$$anonfun$1.apply$mcv$sp(test.scala:15)     @ test$$anonfun$1.apply(test.scala:12)     @ test$$anonfun$1.apply(test.scala:12)     @ org.scalatest.transformer$$anonfun$apply$1.apply$mcv$sp(transformer.scala:22)     @ org.scalatest.outcomeof$class.outcomeof(outcomeof.scala:85)     @ org.scalatest.outcomeof$.outcomeof(outcomeof.scala:104)     @ org.scalatest.transformer.apply(transformer.scala:22)     @ org.scalatest.transformer.apply(transformer.scala:20)     @ org.scalatest.funsuitelike$$anon$1.apply(funsuitelike.scala:166)     @ org.scalatest.suite$class.withfixture(suite.scala:1122)     @ org.scalatest.funsuite.withfixture(funsuite.scala:1555)     @ org.scalatest.funsuitelike$class.invokewithfixture$1(funsuitelike.scala:163)     @ org.scalatest.funsuitelike$$anonfun$runtest$1.apply(funsuitelike.scala:175)     @ org.scalatest.funsuitelike$$anonfun$runtest$1.apply(funsuitelike.scala:175)     @ org.scalatest.superengine.runtestimpl(engine.scala:306)     @ org.scalatest.funsuitelike$class.runtest(funsuitelike.scala:175)     @ org.scalatest.funsuite.runtest(funsuite.scala:1555)     @ org.scalatest.funsuitelike$$anonfun$runtests$1.apply(funsuitelike.scala:208)     @ org.scalatest.funsuitelike$$anonfun$runtests$1.apply(funsuitelike.scala:208)     @ org.scalatest.superengine$$anonfun$traversesubnodes$1$1.apply(engine.scala:413)     @ org.scalatest.superengine$$anonfun$traversesubnodes$1$1.apply(engine.scala:401)     @ scala.collection.immutable.list.foreach(list.scala:318)     @ org.scalatest.superengine.traversesubnodes$1(engine.scala:401)     @ org.scalatest.superengine.org$scalatest$superengine$$runtestsinbranch(engine.scala:396)     @ org.scalatest.superengine.runtestsimpl(engine.scala:483)     @ org.scalatest.funsuitelike$class.runtests(funsuitelike.scala:208)     @ org.scalatest.funsuite.runtests(funsuite.scala:1555)     @ org.scalatest.suite$class.run(suite.scala:1424)     @ org.scalatest.funsuite.org$scalatest$funsuitelike$$super$run(funsuite.scala:1555)     @ org.scalatest.funsuitelike$$anonfun$run$1.apply(funsuitelike.scala:212)     @ org.scalatest.funsuitelike$$anonfun$run$1.apply(funsuitelike.scala:212)     @ org.scalatest.superengine.runimpl(engine.scala:545)     @ org.scalatest.funsuitelike$class.run(funsuitelike.scala:212)     @ test.org$scalatest$beforeandafterall$$super$run(test.scala:9)     @ org.scalatest.beforeandafterall$class.liftedtree1$1(beforeandafterall.scala:257)     @ org.scalatest.beforeandafterall$class.run(beforeandafterall.scala:256)     @ test.run(test.scala:9)     @ org.scalatest.tools.suiterunner.run(suiterunner.scala:55)     @ org.scalatest.tools.runner$$anonfun$dorunrunrundadorunrun$3.apply(runner.scala:2563)     @ org.scalatest.tools.runner$$anonfun$dorunrunrundadorunrun$3.apply(runner.scala:2557)     @ scala.collection.immutable.list.foreach(list.scala:318)     @ org.scalatest.tools.runner$.dorunrunrundadorunrun(runner.scala:2557)     @ org.scalatest.tools.runner$$anonfun$runoptionallywithpassfailreporter$2.apply(runner.scala:1044)     @ org.scalatest.tools.runner$$anonfun$runoptionallywithpassfailreporter$2.apply(runner.scala:1043)     @ org.scalatest.tools.runner$.withclassloaderanddispatchreporter(runner.scala:2722)     @ org.scalatest.tools.runner$.runoptionallywithpassfailreporter(runner.scala:1043)     @ org.scalatest.tools.runner$.run(runner.scala:883)     @ org.scalatest.tools.runner.run(runner.scala)     @ org.jetbrains.plugins.scala.testingsupport.scalatest.scalatestrunner.runscalatest2(scalatestrunner.java:138)     @ org.jetbrains.plugins.scala.testingsupport.scalatest.scalatestrunner.main(scalatestrunner.java:28)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:498)     @ com.intellij.rt.execution.application.appmain.main(appmain.java:144) caused by: java.lang.classnotfoundexception: org.apache.spark.sql.sqlcontext$sqlsession     @ java.net.urlclassloader.findclass(urlclassloader.java:381)     @ java.lang.classloader.loadclass(classloader.java:424)     @ sun.misc.launcher$appclassloader.loadclass(launcher.java:331)     @ java.lang.classloader.loadclass(classloader.java:357)     ... 55 more

your version spark-hive import doesn't match other spark dependencies. should be:

<dependency>   <groupid>org.apache.spark</groupid>   <artifactid>spark-hive_2.10</artifactid>   <version>1.6.0</version> </dependency>

Comments

Post a Comment

Popular posts from this blog

inversion of control - Autofac named registration constructor injection -

does autofac support specifying registration name in components' constructors? example: ninject's namedattribute . you need use autofac.extras.attributed package on top achieve this so let's have 1 interface , 2 classes: public interface ihello { string sayhello(); } public class englishhello : ihello { public string sayhello() { return "hello"; } } public class frenchhello : ihello { public string sayhello() { return "bonjour"; } } then have consumer class, in want select instance injected: public class helloconsumer { private readonly ihello helloservice; public helloconsumer([withkey("en")] ihello helloservice) { if (helloservice == null) { throw new argumentnullexception("helloservice"); } this.helloservice = helloservice; } public string sayhello() { return this.helloservice.sayhello()...
Read more

verilog - Systemverilog dynamic casting issues -

i've code snippet following in testbench function void write_to_port( my_data_type_base data ); my_data_type_extended data_ext; if(!$cast(data_ext, data)); `uvm_error(get_type_name(), "failed cast"); `uvm_info(get_name(), $psprintf("data_ext :\n%s", data_ext.sprint()), uvm_medium) // write data_ext out port.... endfunction when run it, i'm getting uvm_error "failed cast." i'm not quire sure why $cast not returning 1. can see, i'm printing out extended class data item after casting uvm_info. can see it's being cast properly. if don't use $cast if condition, don't runtime error. isn't coding practice use if dynamic cast check if $cast returning 1 ? what might reason behind cast not returning 1 in above case? i think semicolon on line 'if' not belong? i think consumes if statement, , uvm_error executes regardless of how if evaluates: if(!$cast(data_ext, data)); <- no semico...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more