ruby - Scraping successive pages until the last page using Nokogiri and Mechanize -


i trying scrape multiple pages website. want scrape page, click on next, page, , repeat until hit end. wrote far:

page = agent.submit(form, form.buttons.first) #submitting form while lien = page.link_with(:text=>'next')   # while have next link on page, keep scraping   html_body = nokogiri::html(body)   links = html_body.css('.list').xpath("//table/tbody/tr/td[2]/a[1]")   links.each |link|     purelink = link['href']     puts purelink[/codeclub=([^&]*)/].gsub('codeclub=', '')     lien.click   end end

unfortunately, script keep on scraping same page in infinite loop... how can achieve want ?

i try this, replace lien.click page = lien.click.


Comments

Post a Comment

Popular posts from this blog

javascript - Bootstrap Popover: iOS Safari strange behaviour -

i looking through twitter bootstrap documentation, , i'm interested in implementing dismissible popover when button clicked. here example bootstrap's docs: http://getbootstrap.com/javascript/#dismiss-on-next-click when press "dismissible popover" button os x safari browser works fine. when access same link above ios safari, button not work. is, popover not dismissed on click. why button not working ios safari? the example linked using data-trigger="focus" attribute. try changing data-trigger="click" . <a tabindex="0" class="btn btn-lg btn-danger" role="button" data-toggle="popover" data-trigger="click" title="dismissible popover" data-content="and here's amazing content. it's engaging. right?">dismissible popover</a>
Read more

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get...
Read more

session - Logging Out Using PHP -

i have seen other programs regarding can't seem make work. doing wrong? want achieve after user logged out, going previous pages become restricted. i have 3 php files: 1. sample.php - main page username , password filled in 2. successful.php - user's page after logging in 3. logout.php - destorys session hinder user access pages after logging out sample.php </html> <form action="" method="post"> <table> <tr> <td>username:</td> <td>password:</td> </tr> <tr> <td><input type="text" name="username"></td> <td><input type="password" name="password"></td> </tr> <tr> <td><input type="submit" value="submit" name="submit"></td> </tr> </table> </form> </html> <?php $servername = "localhost"; $username = ...
Read more