amazon s3 - Predetermining number of partitions of RDD -


1)how pre-determine number of rdd partitions created?
2)what factors partitioning of data depend on? size of data , way stored(compressed, sequence etc..)

for simplicity, assume have 6gb file in hdfs stored plain text file.

my cluster ec2 cluster below config,

1 master node - m3.xlarge(4 cores, 15gb ram)

4 core nodes - m3.xlarge(4 cores , 15gb ram each)

update: happens if same stored in s3, hbase or nosql?

the partitions dependent on file type. in case, since hdfs file, default number of partitions number of input splits , depend on hadoop setup. if want way of understanding how works.

from hadooprdd.getpartitions:

val inputsplits = inputformat.getsplits(jobconf, minpartitions) val array = new array[partition](inputsplits.size)

Comments

Post a Comment

Popular posts from this blog

javascript - Bootstrap Popover: iOS Safari strange behaviour -

i looking through twitter bootstrap documentation, , i'm interested in implementing dismissible popover when button clicked. here example bootstrap's docs: http://getbootstrap.com/javascript/#dismiss-on-next-click when press "dismissible popover" button os x safari browser works fine. when access same link above ios safari, button not work. is, popover not dismissed on click. why button not working ios safari? the example linked using data-trigger="focus" attribute. try changing data-trigger="click" . <a tabindex="0" class="btn btn-lg btn-danger" role="button" data-toggle="popover" data-trigger="click" title="dismissible popover" data-content="and here's amazing content. it's engaging. right?">dismissible popover</a>
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more

Can the constants be defined inside a model file of a framework in PHP? -

i'm using phpfox framework. i've define 2 constants used 2 functions present within model class file. can define constants @ beginning of model class file or cause issue or against coding standards? please me in regard. following 1 method model class file. i want write following code : <?php /** * [phpfox_header] */ /*header('content-type: text/event-stream'); header('cache-control: no-cache');*/ defined('phpfox') or exit('no dice!'); /** * * * @copyright [phpfox_copyright] * @author raymond benc * @package phpfox_service * @version $id: service.class.php 67 2009-01-20 11:32:45z raymond_benc $ */ class notification_service_process extends phpfox_service { /** * class constructor */ public function __construct() { $this->_stable = phpfox::gett('notification'); } public function add($stype, $iitemid, $iowneruserid, $isenderuserid = null) ...
Read more