Producing Artifacts for Troubleshooting
There are several types of files that are critical for troubleshooting.
Geode logs and statistics are the two most important artifacts used in troubleshooting. In addition, they are required for Geode system health verification and performance analysis. For these reasons, logging and statistics should always be enabled, especially in production. Save the following files for troubleshooting purposes:
- Log files. Even at the default logging level, the log contains data that may be important. Save the whole log, not just the stack. For comparison, save log files from before, during, and after the problem occurred.
- Statistics archive files.
- Core files or stack traces.
- For Linux, you can use gdb to extract a stack from a core file.
- Crash dumps.
- For Windows, save the user mode dump files. Some locations to check for these files:
When a problem arises that involves more than one process, a network problem is the most likely cause. When you diagnose a problem, create a log file for each member of all the distributed systems involved. If you are running a client/server architecture, create log files for the clients.
Note: You must run a time synchronization service on all hosts for troubleshooting. Synchronized time stamps ensure that log messages on different hosts can be merged to accurately reproduce a chronological history of a distributed run.
For each process, complete these steps:
- Make sure the host’s clock is synchronized with the other hosts. Use a time synchronization tool such as Network Time Protocol (NTP).
Enable logging to a file instead of standard output by editing
gemfire.propertiesto include this line:
Keep the log level at
configto avoid filling up the disk while including configuration information. Add this line to
Note: Running with the log level at
finecan impact system performance and fill up your disk.
Enable statistics gathering for the distributed system either by modifying
or by using the
gfsh alter rutimecommand:
alter runtime --group=myMemberGroup --enable-statistics=true --statistic-archive-file=StatisticsArchiveFile.gfs
Note: Collecting statistics at the default sample rate frequency of 1000 milliseconds does not incur performance overhead.
Run the application again.
Examine the log files. To get the clearest picture, merge the files. To find all the errors in the log file, search for lines that begin with these strings:
For details on merging log files, see the
--merge-logargument for the export logscommand.
Export and analyze the stack traces on the member or member group where the application is running. Use the
gfsh export stack-traces command. For example:
gfsh> export stack-traces --file=ApplicationStackTrace.txt --member=member1