Expected timestamp in the Flume event headers, but it was null -


i using below configuration details push twitter feeds hdfs using flume, getting expected timestamp in flume event headers, null

twitter.conf 

twitteragent.sources = twitter twitteragent.channels = memchannel twitteragent.sinks = hdfs  twitteragent.sources.twitter.type = org.apache.flume.source.twitter.twittersource twitteragent.sources.twitter.channels = memchannel twitteragent.sources.twitter.consumerkey = xxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.consumersecret = xxxxxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.accesstoken =  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.accesstokensecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.keywords = bigdata, hadoop, hive, hbase twitteragent.sinks.hdfs.channel = memchannel twitteragent.sinks.hdfs.type = hdfs twitteragent.sinks.hdfs.hdfs.path = /user/farooque/bigdata/tweets/%y/%m/%d/%h/ twitteragent.sinks.hdfs.hdfs.filetype = datastream twitteragent.sinks.hdfs.hdfs.writeformat = text twitteragent.sinks.hdfs.hdfs.batchsize = 10 twitteragent.sinks.hdfs.hdfs.rollsize = 0 twitteragent.sinks.hdfs.hdfs.rollcount = 10000  twitteragent.channels.memchannel.type = memory twitteragent.channels.memchannel.capacity = 10000 twitteragent.channels.memchannel.transactioncapacity = 100

running command 

$ flume-ng agent --conf-file twitter.conf --name twitteragent

where twitter.conf config file name

but getting error as:

java.lang.nullpointerexception: expected timestamp in flume event headers, null         @ com.google.common.base.preconditions.checknotnull(preconditions.java:204)         @ org.apache.flume.formatter.output.bucketpath.replaceshorthand(bucketpath.java:200)         @ org.apache.flume.formatter.output.bucketpath.escapestring(bucketpath.java:396)         @ org.apache.flume.sink.hdfs.hdfseventsink.process(hdfseventsink.java:388)         @ org.apache.flume.sink.defaultsinkprocessor.process(defaultsinkprocessor.java:68)         @ org.apache.flume.sinkrunner$pollingrunner.run(sinkrunner.java:147)         @ java.lang.thread.run(thread.java:745) 15/06/04 18:26:01 error flume.sinkrunner: unable deliver event. exception follows.

looking further help??

in twitter.conf added 1 more config property as

twitteragent.sinks.hdfs.hdfs.uselocaltimestamp = true

and issue got resolved.

for more details refer hadoop tutorial.info


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more