arrays - Creating Really Big Python lists -


i trying create image database compatible cpickle. list empty. data of each image in directory added new row list. images 224x224. size of images on average 8kb. after loading around 10000 images pc hangs. there no mouse movement. nothing happens. needs restart. below code snippet this.. 

cr=csv.reader(open(csv_file,"rb"))     row in cr:         print row[0], row[1]         try:              image=image.open(row[0]+'.jpg').convert('la')             pixels=[]             pixels=[f[0] f in list(image.getdata())]             #pix=np.array(image)             dataset.append(pixels)             #dataset.append(pix)             labels.append(row[1])              del image          except:              print("image not found")

i tried reducing size of images 28x28 , works. don't want reduce size of images. using python 64 bit executable. ram 4gb. ubuntu 14.04. suspect happening due limited stack space, , list taking more available stack space. if so, how create huge list? there workaround issue? end goal create numpy array pixel data rows. converting list numpy array.. there solution problem??

if data numpy array, maybe try using numpy.memmap. works "normal" numpy arrays, difference data stored on disk in binary. requested chunks of array put in ram, may rid of problem.

if size of data array determined, need set correct dimension when creating memmap object. if not, check out numpy.memmap.resize, , should able create anyways.

oh, , there other solutions such pytables.
luck!


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more