Help
cancel
Showing results for 
Search instead for 
Did you mean: 

Understanding your graphs part 6 - System services

GitHub Staff

Understanding your graphs part 6 - System services

 

In part 5 of our 'Understanding your graphs' mini-series, we talked about GitHub Enterprise Network and Storage graphs. In part 6, we're going to talk about GitHub Enterprise system service graphs.

System services graphs contain data related to the major databases on GitHub Enterprise. These are MySQL and Elasticseach persistent databases, as well as Redis and Memcached which contain ephemeral data.

 

Memcached

 

Memcached provides a layer of in-memory caching for web and API operations. Memcached helps to provide quicker response times for users and integrations interacting with the system.

 

Memcached usage

Memcached usage graph

  • Displays the remaining free memory that Memcached can consume if required.
  • On systems with over 64GB RAM, it is possible for this to reach 0, indicating that the daemon has cached the maximum amount of data for the instance.

 

Memcached operations

Memcached operations graph

  • Abnormal spikes or plateaus in operations per second could indicate polling or other busy activity periods.
  • API and Web UI requests will often have portions of the response cached in memcache to speed up future requests for the same data.

 

MySQL

 

MySQL is the primary database in GitHub Enterprise. User, issue, and other non-git or search related metadata is stored within MySQL.

 

MySQL usage

MySQL usage graph

  • Reflects memory InnoDB buffer pool memory used.
  • Memory used by the buffer pool is typically very constant after the system has been warmed up and running with a few hours of normal usage.

 

MySQL threads

MySQL threads graph

  • Running and cached threads may fluctuate based on activity.
  • Connected will reflect a constant value in most cases.

 

MySQL operations

MySQL operations graph

  • Select activity is most commonly the highest value on this graph, and reflects user and API integration activity most closely as data is read from the database.
  • Insert and update trends will be influenced by API POST activity, or other operations which write to the database, such as issue comments.

 

MySQL rows

MySQL rows graph

  • The rows read from the database is most commonly the highest value on this graph.

 

Redis

 

The Redis database mainly contains background job queue, as well as session state information.

 

Redis usage

Redis usage graph

  • Memory used by Redis is typically constant after the system has warmed up and is running with normal usage patterns.

 

Redis operations

Redis operations graph

  • Large spikes in Redis operations may indicate an issue with background job processing.

 

Elasticsearch

 

Elasticsearch powers the built in search features in GitHub Enterprise.

 

Elasticsearch index

Elasticsearch index graph

  • The number of documents and bytes may increase over time, as data is indexed through normal operation of the GitHub Enterprise appliance.

 

Elasticsearch operations

Elasticsearch operations graph

  • Fetch and query operation trends will fluctuate based on usage of GitHub Enterprise Search, and the Search API.

 

Elasticsearch memory

Elasticsearch memory graph

  • Non-heap typically remains constant, while the heap memory fluctuates as background garbage collection and reindexing occurs.

 

Elasticsearch garbage collection

Elasticsearch garbage collection graph

  • Elasticsearch is a Java based service, and Java is a garbage-collected language. It is normal for these operations to regularly occur on a GitHub Enterprise system.
  • Abnormal trends of garbage collection could indicate a performance problem with the Elasticsearch service.

 

Continue the conversation

 

There's more to come in the "Understanding your graphs" mini-series. If you'd like to follow along, just subscribe to the "Understanding your graphs" label (link below). Please let us know if you have any questions in the comments.

1 Comment
Ground Controller Lvl 1

Look at the system sir child