java - How to interpret G1 GC logs leading up to an OutOfMemoryError? -

i wondering if able explain me how interpret g1 gc logs lead outofmemoryerror?

i know heap dump best bet finding out using heap can't since contains protected information cannot leave client site. have application logs (which include stack oome) , g1 gc logs.

the full g1 gc logs have lot of detail won't put them here unless needs see them.

the specific java version these came was:

> java -version java version "1.7.0_21" java(tm) se runtime environment (build 1.7.0_21-b11) java hotspot(tm) 64-bit server vm (build 23.21-b01, mixed mode) 

and gc options using create gc log are:

-xx:+printgctimestamps -xx:+printgcdetails -xloggc:log/gc.log 

below memory stats each young , full gc during last 30 minutes leading oome:

info   | jvm 1    | 2015/05/28 04:29:34 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:33:21 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:37:09 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:40:58 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:44:44 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:48:30 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:52:17 | [eden: 1290m(1290m)->0b(1290m) survivors: 20m->20m heap: 2445m(3932m)->1155m(3932m)] info   | jvm 1    | 2015/05/28 04:52:58 | [eden: 639m(1290m)->0b(1295m) survivors: 20m->15m heap: 2278m(3932m)->1635m(3932m)] info   | jvm 1    | 2015/05/28 04:52:59 | [eden: 51m(1295m)->0b(1300m) survivors: 15m->10m heap: 2561m(3932m)->2505m(3932m)] info   | jvm 1    | 2015/05/28 04:52:59 | [full gc 2505m->1170m(3901m), 1.9469560 secs] info   | jvm 1    | 2015/05/28 04:53:01 | [eden: 44m(1300m)->0b(1299m) survivors: 0b->1024k heap: 1653m(3901m)->1610m(3901m)] info   | jvm 1    | 2015/05/28 04:53:01 | [eden: 1024k(1299m)->0b(1299m) survivors: 1024k->1024k heap: 1610m(3901m)->1610m(3901m)] info   | jvm 1    | 2015/05/28 04:53:02 | [full gc 1610m->1158m(3891m), 1.4317370 secs] info   | jvm 1    | 2015/05/28 04:53:03 | [eden: 112m(1299m)->0b(1296m) survivors: 0b->1024k heap: 1758m(3891m)->1647m(3891m)] info   | jvm 1    | 2015/05/28 04:53:06 | [eden: 49m(1296m)->0b(1360m) survivors: 1024k->1024k heap: 2776m(4084m)->2728m(4084m)] info   | jvm 1    | 2015/05/28 04:53:06 | [eden: 0b(1360m)->0b(1360m) survivors: 1024k->1024k heap: 2837m(4084m)->2836m(4084m)] info   | jvm 1    | 2015/05/28 04:53:06 | [full gc 2836m->1158m(3891m), 1.4847750 secs] info   | jvm 1    | 2015/05/28 04:53:08 | [full gc 1158m->1158m(3891m), 1.5313770 secs] 

* different format raw logs , removed timing details make shorter , easier read.

i graphed raw gc logs these in gcviewer:

it seems going ok far:

• the used tenured constant (the dark magenta line @ bottom of graph).
• young gc pauses occurred every couple of minutes cleaning young objects (the grey line @ top of graph).
• allocated sizes each generation constant.
• heap usage after young gcs around 1155m.

then @ 2015/05/28 04:52:59 things went pear shaped:

• the tenured of sudden started growing.
• when young gc ran there 51m in eden space.
• full gcs started occurring.
• the first 3 full gcs seemed ok, reduced heap usage 1158m-1170m (which close normal 1155m).
• the final full gc started 1158m used , still 1158m after.

the memory tab in screenshot shows:

tenured heap (usage / alloc. max)    2,836 (104.1%) / 2,723m total promotion                                       2,048k 

now explain briefly occurred @ 2015/05/28 04:52:59. @ point whole bunch of configuration objects serialised custom format using stringbuilder. resulted in bunch of array copies resulted in following exception @ 2015/05/28 04:53:09:

java.lang.outofmemoryerror: java heap space     @ java.util.arrays.copyof(arrays.java:2367)     @ java.lang.abstractstringbuilder.expandcapacity(abstractstringbuilder.java:130)     @ java.lang.abstractstringbuilder.ensurecapacityinternal(abstractstringbuilder.java:114)     @ java.lang.abstractstringbuilder.append(abstractstringbuilder.java:587)     @ java.lang.stringbuilder.append(stringbuilder.java:214)     ... 

there few things can't explain:

1. where in gc logs find used tenured memory?
2. what cause such significant spike in used tenured memory cause gc? there 20m of survivors wouldn't in worst case these go tenured , no more?
3. could perhaps explained humongous object allocation?
4. why last full gc triggered when there (apparently) little used heap , @ afterwards cleaned nothing?
5. if there 3891m heap allocated , 1158m used why there oome?

your out of memory happens during stringbuilder.append - bear in mind every time append string , buffer inside stringbuilder small such expands capacity, try allocate dobule buffer double current length of string in builder plus 2 or new length if bigger . (see source code abstractstringbuilder.java)

for example, if string builder has 100 characters , full, append 10 characters it, expand by:

100*2+2 = 202, more 10.

so if have long string (10mb), attempt create 20mb buffer , on.

check code , make sure not creating huge strings in builder.