How to automate Amazon aws EC2 for scraping -


hi i'd set amazon ec2 instances (multiple) scrape data arbitrary sites. way imagine being set 1 amazon instance that's master programatically set other instances scrape. right have php scripts can scrape way want to, how can set master server to...

1) make other ec2 instances

2) communicate between master server , slave servers

you could build having master launch worker instances when needed, send them scrape requests, terminate them when needed , code orchestration , try make highly available. that's not way this. instead, should take advantage of aws features.

you use combination of sqs , auto scaling groups. master instance add scrape requests sqs queue , have auto scaling group triggered on sqs queue depth launches new worker instances - helps automate launching of workers (scrapers) when workload high , terminate workers when workload low. worker instances pull scrape request sqs queue, scraping work, , repeat.

another way use aws lambda. can trigger lambda functions sqs or sns. have master add scrape requests sqs queue or have master publish requests sns topic, , drive web-scraper lambda function (written in javascript) sqs queue or sns topic.

personally, investigate lambda route first.


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more