Expected timestamp in the Flume event headers, but it was null -


i using below configuration details push twitter feeds hdfs using flume, getting expected timestamp in flume event headers, null

twitter.conf 

twitteragent.sources = twitter twitteragent.channels = memchannel twitteragent.sinks = hdfs  twitteragent.sources.twitter.type = org.apache.flume.source.twitter.twittersource twitteragent.sources.twitter.channels = memchannel twitteragent.sources.twitter.consumerkey = xxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.consumersecret = xxxxxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.accesstoken =  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.accesstokensecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx twitteragent.sources.twitter.keywords = bigdata, hadoop, hive, hbase twitteragent.sinks.hdfs.channel = memchannel twitteragent.sinks.hdfs.type = hdfs twitteragent.sinks.hdfs.hdfs.path = /user/farooque/bigdata/tweets/%y/%m/%d/%h/ twitteragent.sinks.hdfs.hdfs.filetype = datastream twitteragent.sinks.hdfs.hdfs.writeformat = text twitteragent.sinks.hdfs.hdfs.batchsize = 10 twitteragent.sinks.hdfs.hdfs.rollsize = 0 twitteragent.sinks.hdfs.hdfs.rollcount = 10000  twitteragent.channels.memchannel.type = memory twitteragent.channels.memchannel.capacity = 10000 twitteragent.channels.memchannel.transactioncapacity = 100

running command 

$ flume-ng agent --conf-file twitter.conf --name twitteragent

where twitter.conf config file name

but getting error as:

java.lang.nullpointerexception: expected timestamp in flume event headers, null         @ com.google.common.base.preconditions.checknotnull(preconditions.java:204)         @ org.apache.flume.formatter.output.bucketpath.replaceshorthand(bucketpath.java:200)         @ org.apache.flume.formatter.output.bucketpath.escapestring(bucketpath.java:396)         @ org.apache.flume.sink.hdfs.hdfseventsink.process(hdfseventsink.java:388)         @ org.apache.flume.sink.defaultsinkprocessor.process(defaultsinkprocessor.java:68)         @ org.apache.flume.sinkrunner$pollingrunner.run(sinkrunner.java:147)         @ java.lang.thread.run(thread.java:745) 15/06/04 18:26:01 error flume.sinkrunner: unable deliver event. exception follows.

looking further help??

in twitter.conf added 1 more config property as

twitteragent.sinks.hdfs.hdfs.uselocaltimestamp = true

and issue got resolved.

for more details refer hadoop tutorial.info


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get...
Read more

php - Bypass Geo Redirect for specific directories -

i'm trying bypass redirect specific url within site. wondering if might able help. the code below code use, original urls replaces example.com i'd able access http://example.com/admin without redirect occuring. is there way this, , there way using own ip? function country_geo_redirect() { $country = getenv('http_geoip_country_code'); if ($country == "us" ) { wp_redirect("http://usa.example.com/"); exit; } else if ($country == "gb" ) { wp_redirect("http://uk.example.com/"); exit; } } add_action('init', 'country_geo_redirect'); you can bypass redirection code if server found ip address. if ip address == <my ip address> ' nothing here else country_georedirect() end if
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more