database - DB bulk validation and upload -


i designing application involve bulk upload of records postgres db (lets call schema db-1). uploads done every week. size range few million billion records. data going uploaded needs validated/cleansed first need conform constaints , format of db-1. thinking of adopting following approach:

  1. everytime new upload needs done, new schema created (lets call db-2 - staging place) same db-1 but lenient constraints. make sure data gets loaded in db-2 start with.
  2. run validation process on data. thinking middleware process when realized amount of data processed, kind of started thinking coding validation+cleansing layer in db - set of stored procs run on db-2, check data , generate report records not conform rules (ie constaints present in db-1, data format etc).
  3. after this, data needs changed again @ source, step 1 repeated , if looks ok, select db-1 db-2 shift valid data final desitnation.

what opinion on above process? obvious/hidden issues see here? suggestions make better welcome.

thanks

j


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more