Understanding your graphs part 6 - System services
In part 5 of our ‘Understanding your graphs’ mini-series, we talked about GitHub Enterprise Network and Storage graphs. In part 6, we’re going to talk about GitHub Enterprise system service graphs.
System services graphs contain data related to the major databases on GitHub Enterprise. These are MySQL and Elasticseach persistent databases, as well as Redis and Memcached which contain ephemeral data.
Memcached provides a layer of in-memory caching for web and API operations. Memcached helps to provide quicker response times for users and integrations interacting with the system.
- Displays the remaining free memory that Memcached can consume if required.
- On systems with over 64GB RAM, it is possible for this to reach 0, indicating that the daemon has cached the maximum amount of data for the instance.
- Abnormal spikes or plateaus in operations per second could indicate polling or other busy activity periods.
- API and Web UI requests will often have portions of the response cached in memcache to speed up future requests for the same data.
MySQL is the primary database in GitHub Enterprise. User, issue, and other non-git or search related metadata is stored within MySQL.
- Reflects memory InnoDB buffer pool memory used.
- Memory used by the buffer pool is typically very constant after the system has been warmed up and running with a few hours of normal usage.
- Running and cached threads may fluctuate based on activity.
- Connected will reflect a constant value in most cases.
- Select activity is most commonly the highest value on this graph, and reflects user and API integration activity most closely as data is read from the database.
- Insert and update trends will be influenced by API POST activity, or other operations which write to the database, such as issue comments.
- The rows read from the database is most commonly the highest value on this graph.
The Redis database mainly contains background job queue, as well as session state information.
- Memory used by Redis is typically constant after the system has warmed up and is running with normal usage patterns.
- Large spikes in Redis operations may indicate an issue with background job processing.
Elasticsearch powers the built in search features in GitHub Enterprise.
- The number of documents and bytes may increase over time, as data is indexed through normal operation of the GitHub Enterprise appliance.
- Fetch and query operation trends will fluctuate based on usage of GitHub Enterprise Search, and the Search API.
- Non-heap typically remains constant, while the heap memory fluctuates as background garbage collection and reindexing occurs.
Elasticsearch garbage collection
- Elasticsearch is a Java based service, and Java is a garbage-collected language. It is normal for these operations to regularly occur on a GitHub Enterprise system.
- Abnormal trends of garbage collection could indicate a performance problem with the Elasticsearch service.
Continue the conversation
There’s more to come in the “Understanding your graphs” mini-series. If you’d like to follow along, just subscribe to the “Understanding your graphs” label (link below). Please let us know if you have any questions in the comments.