mapreduce - How Hadoop calculate physical memory and virutal memory during a job execution -


i have few queries related counters used in hadoop display memory usage. map reduce job executed on cluster gives me below menitoned counter values. input file used in kbs, these counter shows 35gb , 420 gb usage. physical_memory_bytes=35110662144 virtual_memory_bytes=420121841664

for different job on same input file shows 309 mb (physical) , 3g(vitual) usage

physical_memory_bytes=309526528 virtual_memory_bytes=3435827200

first job more cpu intensive other , creates more objects other 1 still usage shown seems high. wanted know how memory usage calculated. tried going through posts , gave on view on below link seems requirement task describing these variables (https://issues.apache.org/jira/i#browse/mapreduce-1218 ) couldnt find how these calculated. gives me idea on how these values passed job tracker,but no information on how these determined. if 1 give insight on helpfull.

you can find few references here , here. second link in particular map , reducer job , how slots decided based on memory allocations. happy learning


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more