Dataflow streaming - does it fit our use case? -


we've been using dataflow in batch mode while now. however, can't seem find info on streaming mode.

we have following use case:

  • data/events being streamed real-time bigquery
  • we need transform/clean/denormalize data before analysis business

now, of course use dataflow in batch mode, , take chucks of data bigquery (based on timestamps), , transform/clean/denormalize way.

but that's bit of messy approach, because data being streamed real-time , real gnarly working out data needs worked on. sounds brittle too.

it great if transform/clean/denormalize in dataflow, , write bigquery as it's streaming in.

is dataflow streaming intended for? if so, data source can dataflow read in streaming mode?

yes, reasonable use case streaming mode. support reading cloud pub/sub via pubsubio source. additional sources in works. output can written bigquery via bigqueryio sink. pcollection docs cover distinction between bounded , unbounded sources/sinks, available concrete implementations.

as apparent lack of streaming-specific documentation, majority of unified model applicable in batch , streaming, there no streaming-specific section. said, i'd recommend looking on windowing , triggers sections of pcollection docs, particularly applicable when dealing unbounded pcollections.


Comments

Post a Comment

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

i working on custom magento extension. working around adminhtml part , i've created custom form there. here form code: <?php class vivasindustries_smsnotification_block_adminhtml_sms_sendmass_edit_form extends mage_adminhtml_block_widget_form { public function _preparelayout() { $extensionpath = mage::getmoduledir('js', 'vivasindustries_smsnotification'); $head = $this->getlayout()->getblock('head'); $head->addjs('jquery.js'); $head->addjs('vivas.js'); return parent::_preparelayout(); } protected function _prepareform() { $form = new varien_data_form(array( 'id' => 'edit_form', 'action' => $this->geturl('*/*/save', array('id' => $this->getrequest()->get...
Read more

php - Bypass Geo Redirect for specific directories -

i'm trying bypass redirect specific url within site. wondering if might able help. the code below code use, original urls replaces example.com i'd able access http://example.com/admin without redirect occuring. is there way this, , there way using own ip? function country_geo_redirect() { $country = getenv('http_geoip_country_code'); if ($country == "us" ) { wp_redirect("http://usa.example.com/"); exit; } else if ($country == "gb" ) { wp_redirect("http://uk.example.com/"); exit; } } add_action('init', 'country_geo_redirect'); you can bypass redirection code if server found ip address. if ip address == <my ip address> ' nothing here else country_georedirect() end if
Read more

php - .htaccess mod_rewrite for dynamic url which has domain names -

i want convert url ex:( example.com/link.php?url=facebook.com ) ( example.com/facebook.com.php ) how can using .htaccess. or there other process make job easier ? i tried adding script .htaccess file options +followsymlinks rewriteengine on rewriterule url-(.*)\.htm$ link.php?url=$1 but converting url adding "url-" ( example.com/url-facebook.com.php ) i want remove "url-" , want make ( example.com/facebook.com.php ) rewriteengine on rewritecond %{the_request} ^get.*link\.php\?url\= [nc] rewritecond %{request_uri} !/system/.* [nc] rewriterule (.*?)link\.php\?url\=/*(.*) /$1$2 [r=301,ne,l] not sure how above work edit of used use remove index.php url
Read more