Dataflow streaming - does it fit our use case? -


we've been using dataflow in batch mode while now. however, can't seem find info on streaming mode.

we have following use case:

  • data/events being streamed real-time bigquery
  • we need transform/clean/denormalize data before analysis business

now, of course use dataflow in batch mode, , take chucks of data bigquery (based on timestamps), , transform/clean/denormalize way.

but that's bit of messy approach, because data being streamed real-time , real gnarly working out data needs worked on. sounds brittle too.

it great if transform/clean/denormalize in dataflow, , write bigquery as it's streaming in.

is dataflow streaming intended for? if so, data source can dataflow read in streaming mode?

yes, reasonable use case streaming mode. support reading cloud pub/sub via pubsubio source. additional sources in works. output can written bigquery via bigqueryio sink. pcollection docs cover distinction between bounded , unbounded sources/sinks, available concrete implementations.

as apparent lack of streaming-specific documentation, majority of unified model applicable in batch , streaming, there no streaming-specific section. said, i'd recommend looking on windowing , triggers sections of pcollection docs, particularly applicable when dealing unbounded pcollections.


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more

Website Login Issue developed in magento -

i have website developed using magento . able login. right now, login failed. infact operations interacts user table failing. need know issue. how figure out? first clear try clear log file second make sure have not upgraded magento and 3rd make sure db accessible in magento -harry p
Read more