postgresql - Java.net.socketexception Connection reset -
i trying download large set of data remote amazon redshift server (a postgresql database). data user visiting log. since data large. extract ids of users visit website specified time period, , recursively extract logs.
the code below.
static connection getuserlogconn() throws sqlexception, classnotfoundexception { system.out.println("-------- postgresql " + "jdbc connection testing ------------"); class.forname("org.postgresql.driver"); connection connection = null; connection = drivermanager.getconnection("<address>", "<username>", "<password>"); return connection; } static linkedlist<string> extractalluidsfromremote( connection connection ) throws sqlexception, unknownhostexception { linkedlist<string> alluids = new linkedlist<string>(); string query = "select distinct uid " + fromstr + " ts >= " + starttime; if(!endtime.equals("")) query += " , ts < " + endtime; system.out.println("sent sql redshift: " + query); // ***below statement exception occurs *** resultset rs_uid = connection.createstatement().executequery( query ); system.out.println( "received uids successfully" ); int n = 0; while( rs_uid.next() ) { // cursor points row in result n++; string uid = rs_uid.getstring( "uid" ); alluids.add(uid); } system.out.println( n + " docs retrieved." ); return alluids; } static void queryindividualuserlog( connection connection, linkedlist<string> uids ) throws sqlexception, unknownhostexception { mongodbmanager db = new mongodbmanager( database, "frequserlog" ); db.createindex("uid"); db.createindex("url"); stringbuffer sb = new stringbuffer(); int = 0; for( string uid : uids ) { sb.append( "uid='" + uid + "'" ); // compose sql query every 10000 users if( ( != 0 && % 10000 == 0 ) || == uids.size() - 1 ){ system.out.println("processing user " + i); string query = "select * " + fromstr + " " + sb.tostring() + " , ts >= " + starttime; if(!endtime.equals("")) query += " , ts < " + endtime; system.out.println("sent sql redshift retrieving individual users' logs"); **resultset rs_log = connection.createstatement().executequery( query );** // step takes time wait response redshift system.out.println( "received individual users' logs succesfully" ); while( rs_log.next() ) { db.insertonelog( rs_log ); // 1 log = 1 doc, i.e. 1 row } system.out.println( "have written db." ); sb = new stringbuffer(); } else { sb.append( " or " ); } i++; } system.out.println(uids.size() + " user's log stored db"); } public static void main(string[] args) throws classnotfoundexception, sqlexception, unknownhostexception { connection connection = getuserlogconn(); if(connection != null) { system.out.println( "connect succesfully" ); /** extract users' uids, , store them in frequserids collection */ linkedlist<string> alluids = extractalluidsfromremote( connection ); /** query records of freq users redshift, , store them in frequserlog collection */ queryindividualuserlog( connection, alluids ); connection.close(); } however, problem exception thrown out. statement under "***" comment in code problem happens.
org.postgresql.util.psqlexception: i/o error occured while sending backend. @ org.postgresql.core.v3.queryexecutorimpl.execute(queryexecutorimpl.java:218) @ org.postgresql.jdbc2.abstractabstractjdbcabstractedly5statement.excute(abstractjdbcabstractedly5statement.java:561) ... caused java.net.socketexception: connection reset @ java.net.socketinputstream.read(unknown source) @ java.net.socketinputstream.read(unknown source) @ org.postgresql.core.visiblebufferedinputstream.readmore(visiblebufferedinputstream.java:143) org.postgresql.core.visiblebufferedinputstream.ensurebytes(visiblebufferedinputstream.java:112) org.postgresql.core.visiblebufferedinputstream.read(visiblebufferedinputstream.java:194) org.postgresql.core.pgstream.receive(pgstream.read) since can not access remote postgresql server, not have database log. googled problem. many of related issues "connection reset peer", not "connection reset" here. says "connect reset" means connection closed @ side, i.e. side. not know why happens , how fix it. thanks.
update: guess query process takes long time because data large. problem keeps waiting response redshift. in case, program closes connection due timeout. not know if truth... if so, there better solution? (i mean, better decreasing number of users inquiry each time. right number 10000).
Comments
Post a Comment