mysql - Could it make sense to schedule an export of SQL database to NoSQL for graphical data mining? -


would make sense me schedule export sql database graph database (such neo4j) in order generate interactive graphics of relationships such here?

update: or extension, should looking move on graph database altogeher?

my graphical database not need live reflection of relational database - extract every few days more sufficient.

in case, have relational database (mysql) i’m recording stock items pass between individuals/depots. concept follows:

items:

stockid dispatchdate 0001    2014-01-01 0002    2015-06-03 

individuals:

userid firstname 0001   tom 0002   jones 

depots:

depotid zipcode 0001    50421 0002    71028 

owners:

stock_id user_id received   dispatched 0001     0001    2015-05-01 2015-05-10 0001     0002    2015-05-11 2015-05-20 

from nosql database able visually see things such as:

  • the flow of people item has passed through (and dates of each relationship)
  • which items @ each individual/depot (on given date)
  • which individuals @ depots (on given date)

as n.b. says in comments, if tool useful use - worst case find tool isn't useful after , stop using (having wasted time in setting up, such life).

in general, there 3 ways sync database:

  1. two phase commit: modify mysql in 1 transaction, modify neo4j in transaction, if either transaction fails roll both transactions; transactions don't commit until both signal can committed. provides highest data integrity expensive.
  2. loosely synchronized transactions: modify mysql in 1 transaction, modify neo4j in transaction, if 1 succeeds , other fails retry failed transaction few times, , if still fails decide (e.g. undo successful transaction, complicated fact transaction has committed , values may have been used; or log error , ask database administrator manually sync databases; or third option). offers decent data integrity , cheaper 2 phase commit, more difficult recover if goes horribly wrong.
  3. batch synchronization: modify mysql, , after time interval (five minutes, hour, whatever's appropriate) sync changes neo4j based on row version number or timestamp (note it's not of problem if sync bit data since you'll overwriting value same value, err on side of syncing per batch). solution easy program, , appropriate if neo4j doesn't need latest , greatest data.

i worked on similar project syncing mysql key-value nosql database (caching expensive queries), using loosely synchronized transactions. wrote customized transaction wrapper contained concurrent queue of side-effects (i.e. changes made key-value database); if mysql transaction succeeded committed of side-effects in queue key-value database (with 3 retries in case of transient network failure, after logged error, invalidated key-value database entry result in fallback mysql, , notified database admin - happened 1 time when key-value database crashed extended period, , solved running batch synchronization), else discarded them.


Comments

Popular posts from this blog

Magento/PHP - Get phones on all members in a customer group -

php - .htaccess mod_rewrite for dynamic url which has domain names -

Website Login Issue developed in magento -