Cassandra as input to Hadoop job exception -
for reason i'm getting following exception while trying use cassandra input hadoop
exception in thread "main" java.lang.noclassdeffounderror: com/datastax/driver/core/policies/loadbalancingpolicy
here code
public class cdriver extends configured implements tool{ public static void main(string[] args) throws ioexception, interruptedexception, classnotfoundexception, exception { configuration conf = new configuration(); toolrunner.run(new cdriver(), args); } @override public int run(string[] args) throws exception { string output = args[0]; configuration conf = super.getconf(); job job = new job(conf); job.setjarbyclass(cdriver.class); job.setjobname("cassandra input"); confighelper.setinputinitialaddress(conf, "127.0.0.1"); confighelper.setinputcolumnfamily(conf, "basketball", "nba"); confighelper.setinputpartitioner(conf, "murmur3partitioner"); cqlconfighelper.setinputcqlpagerowsize(conf, "3"); job.setinputformatclass(cqlinputformat.class); fileoutputformat.setoutputpath(job, new path(output)); job.setmapperclass(cmapper.class); job.setreducerclass(creducer.class); job.setmapoutputkeyclass(text.class); job.setmapoutputvalueclass(intwritable.class); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); job.waitforcompletion(true); return 0; }
}
it goes off on following line
cqlconfighelper.setinputcqlpagerowsize(conf, "3");
and here maven dependencies:
<?xml version="1.0" encoding="utf-8"?> <project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>com.nissatech</groupid> <artifactid>testingcassandra</artifactid> <version>1.0-snapshot</version> <packaging>jar</packaging> <properties> <project.build.sourceencoding>utf-8</project.build.sourceencoding> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> </properties> <dependencies> <dependency> <groupid>org.apache.hadoop</groupid> <artifactid>hadoop-core</artifactid> <version>1.2.1</version> </dependency> <dependency> <groupid>org.apache.cassandra</groupid> <artifactid>cassandra-all</artifactid> <version>2.1.5</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactid>maven-assembly-plugin</artifactid> <configuration> <archive> <manifest> <mainclass>com.nissatech.testingcassandra.cdriver</mainclass> </manifest> </archive> <descriptorrefs> <descriptorref> jar-with-dependencies </descriptorref> </descriptorrefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>
can explain problem? have cassandra running on localhost.
you need include datastax java driver third party dependency in pom.xml
, too. see https://github.com/datastax/java-driver#maven. if have transitive dependency, make sure using right version of it.
Comments
Post a Comment