python - Mechanize br.submit() limitations? -


my intention submit search query website using mechanize , analyse results using beautifulsoup. used same website , form names etc. can hardcoded. having issues initial query, shown below:

 import mechanize import urllib2 #from bs4 import beautifulsoup   def inspect_page(url):     br = mechanize.browser(factory=mechanize.robustfactory())     br.set_handle_robots(false)     br.addheaders = [('user-agent',                       'mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.8.1.6) gecko/20070725 firefox/2.0.0.6')]     br.set_handle_redirect(mechanize.httpredirecthandler)      try:         br.open(url)     except mechanize.httperror, e:         print "http error", e.code,     except urllib2.urlerror e:         print "url error", e.reason,         return      form in br.forms():         print form      br.select_form(name="dataform")     br.form['pcode'] = 'wv14 8ew'     br.form['premise'] = '66'     response = br.submit()     print response.read()      #soup = beautifulsoup(response.read())  inspect_page('http://www.fensa.co.uk/asp/certificate.asp')

this did not redirect results page , print response.read() displayed html of page submitted query on, assumed had made error in code. when tested site (inspect_page('https://publicaccess.glasgow.gov.uk/online-applications/search.do?action=simple')) , changed forms match on site:

`br.select_form(name="searchcriteriaform") br.form['searchcriteria.simplesearchstring'] = 'queen elizabeth gardens' response = br.submit() print response.read()`

i redirected expected. there stop page being redirected when br.submit() called? i've checked site not gzipped.

one limitation mechanize doesn't know javascript. submitting search form on site in script triggers javascript function validates input , changes action attribute of <form> before submitting form values.

here html part of form:

<a onclick="return validate_required()" name="submit" href="#">   <input class="button" type="button" value="search" name="submit2"> </a>

and validate_required() function defined near beginning of html document:

function validate_required() {      error = "";     if (document.getelementbyid("pcode").value == '') { error = error + "postcode\n"; }     if (document.getelementbyid("premise").value == '') { error = error + "premise\n"; }      if (error != '') {         alert("please enter:\n\n" + error);         return false;     }     else {         document.dataform.action = "certificate_results.asp";         document.dataform.submit();      } }

Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get...
Read more

php - Bypass Geo Redirect for specific directories -

i'm trying bypass redirect specific url within site. wondering if might able help. the code below code use, original urls replaces example.com i'd able access http://example.com/admin without redirect occuring. is there way this, , there way using own ip? function country_geo_redirect() { $country = getenv('http_geoip_country_code'); if ($country == "us" ) { wp_redirect("http://usa.example.com/"); exit; } else if ($country == "gb" ) { wp_redirect("http://uk.example.com/"); exit; } } add_action('init', 'country_geo_redirect'); you can bypass redirection code if server found ip address. if ip address == <my ip address> ' nothing here else country_georedirect() end if
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more