+------------------------------------------------------------------------------+
| |
| Dhananjaya D R @/logs @/software @/resume @/contact |
| |
+------------------------------------------------------------------------------+
JVM Dump Hacks to Reduce Resource Spikes
________________________________________________________________________________
Taking heap or thread dumps in a containerised environment can cause CPU and
I/O spikes that degrade application performance or trigger container restarts.
Here are few hacks to minimise those
[1] Instead of writing a large dump file to disk and compressing it afterward,
stream the output from 'jmap' directly to 'gzip'
+-----------------------------------------------------------------------------+
| Bash |
+-----------------------------------------------------------------------------+
| 1 docker exec -i $ID /bin/bash -c \ |
| 2 "/jdk-11.0.17+8/bin/jmap -dump:format=b,file=/dev/stdout \ |
| 3 ${ProID} 2>/dev/null | gzip -c" \ |
| 4 > "$DumpDir/${Heapdump}.gz" |
+-----------------------------------------------------------------------------+
[2] Use a FIFO to control memory usage. A named pipe (FIFO) allows the data to
be streamed in small chunks without consuming excess memory.
+-----------------------------------------------------------------------------+
| Bash |
+-----------------------------------------------------------------------------+
| 1 FIFO="/tmp/heap_dump_fifo" |
| 2 mkfifo "$FIFO" |
| 3 |
| 4 gzip -c < "$FIFO" > "$DumpDir/${Heapdump}.gz" & |
| 5 |
| 6 docker exec -i $ID /bin/bash -c \ |
| 7 "nice -n 19 /jdk-11.0.17+8/bin/jmap \ |
| 8 -dump:format=b,file=/dev/null,printtostream=true ${ProID}" \ |
| 9 > "$FIFO" |
+-----------------------------------------------------------------------------+
[3] To avoid blocking your main process, run the dump operation as a detached
process.
+-----------------------------------------------------------------------------+
| Bash |
+-----------------------------------------------------------------------------+
| 1 cat > "$DumpDir/external_heap_dump.sh" <<EOF |
| 2 #!/bin/bash |
| 3 # Dump logic here |
| 4 EOF |
| 5 |
| 6 nohup "$DumpDir/external_heap_dump.sh" \ |
| 7 > "$DumpDir/dump_progress.log" 2>&1 & |
+-----------------------------------------------------------------------------+
[4] Reduce CPU impact by running the dump with low priority.
+-----------------------------------------------------------------------------+
| Bash |
+-----------------------------------------------------------------------------+
| 1 docker exec -i $ID /bin/bash -c \ |
| 2 "nice -n 19 /jdk-11.0.17+8/bin/jmap ${ProID}" |
+-----------------------------------------------------------------------------+
Optional - Add I/O priority with ionice
+-----------------------------------------------------------------------------+
| Bash |
+-----------------------------------------------------------------------------+
| 1 ionice -c2 -n7 nice -n19 /jdk-11.0.17+8/bin/jmap ${ProID} |
+-----------------------------------------------------------------------------+
_______________________________________________________________________________