hadoop + Flume + HDFS IO error + ConnectException -


i'm working cloudera manager cdh 5.4.2, installed flume, can not save information twitter

when run flume agent, starts okay ends in error when attempts writing new event data hdfs. got following error:

 info org.apache.flume.sink.hdfs.bucketwriter: creating hdfs://192.168.109.6:8020/user/flume/tweets/2015/06/03/06//flumedata.1433311217583.tmp  warn org.apache.flume.sink.hdfs.hdfseventsink: hdfs io error java.net.connectexception: call cluster-05.xxxx.com/192.168.109.6 cluster-05.xxxx.com:8020 failed on connection exception: java.net.connectexception: connection refused; more details see: http://wiki.apache.org/hadoop/connectionrefused

the configuration did :

flume-conf.property:

twitteragent.sinks.hdfs.channel = memchannel twitteragent.sinks.hdfs.type = hdfs twitteragent.sinks.hdfs.hdfs.path = hdfs://192.168.109.6:8020/user/flume/tweets/%y/%m/%d/%h/ twitteragent.sinks.hdfs.hdfs.filetype = datastream twitteragent.sinks.hdfs.hdfs.writeformat = text twitteragent.sinks.hdfs.hdfs.batchsize = 1000 twitteragent.sinks.hdfs.hdfs.rollsize = 0 twitteragent.sinks.hdfs.hdfs.rollcount = 10000

i using follown pluggins:

flume-sources-1.0-snapshot.jar twitter4j-core-2.2.6.jar twitter4j-media-support-2.2.6.jar twitter4j-stream-2.2.6.jar

(i replace version of twitter4j--3.0.3.jar twitter4j--2.2.6.jar)

also directory using hdfs user

hadoop fs -ls /user/flume :  drwxrwxrwx - flume flume  /user/flume/tweets

core-site.xml ( @ /hadoop/conf ) added:

<property> <name>fs.default.name</name> <value>hdfs://localhost:8020</value> </property>

i run hadoop dfsadmin -safemode leave on host left flume agent hdfs user

i appreciate help, on issue.


Comments

Post a Comment

Popular posts from this blog

javascript - Bootstrap Popover: iOS Safari strange behaviour -

i looking through twitter bootstrap documentation, , i'm interested in implementing dismissible popover when button clicked. here example bootstrap's docs: http://getbootstrap.com/javascript/#dismiss-on-next-click when press "dismissible popover" button os x safari browser works fine. when access same link above ios safari, button not work. is, popover not dismissed on click. why button not working ios safari? the example linked using data-trigger="focus" attribute. try changing data-trigger="click" . <a tabindex="0" class="btn btn-lg btn-danger" role="button" data-toggle="popover" data-trigger="click" title="dismissible popover" data-content="and here's amazing content. it's engaging. right?">dismissible popover</a>
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more

Can the constants be defined inside a model file of a framework in PHP? -

i'm using phpfox framework. i've define 2 constants used 2 functions present within model class file. can define constants @ beginning of model class file or cause issue or against coding standards? please me in regard. following 1 method model class file. i want write following code : <?php /** * [phpfox_header] */ /*header('content-type: text/event-stream'); header('cache-control: no-cache');*/ defined('phpfox') or exit('no dice!'); /** * * * @copyright [phpfox_copyright] * @author raymond benc * @package phpfox_service * @version $id: service.class.php 67 2009-01-20 11:32:45z raymond_benc $ */ class notification_service_process extends phpfox_service { /** * class constructor */ public function __construct() { $this->_stable = phpfox::gett('notification'); } public function add($stype, $iitemid, $iowneruserid, $isenderuserid = null) ...
Read more