Guides on Using Cryostat
This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.
- Start/Stop a Recording
- Snapshot a Recording
- Archive a Recording
- Download an Active or Archived Recording
- Download, Edit, and Upload a Customized Event Template
- Edit Template With JMC
- Re-Upload a Recording to Archives
- View a Recording in Grafana
- View and Download Automated Analysis for a Recording
- Add and Edit Recording Metadata Labels
- Store JMX Credentials
- Create an Automated Rule
- Configure Graphical Notifications
Start/Stop a Recording
The first and most basic workflow is to start a Flight Recording in a target JVM.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Configure the new recording
Enter a name for the new recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.
Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.
-
(Optional) Adjust any Advanced Options for the recording
In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:
To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.
Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.
Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the recording before being discarded. This parameter is valid only when the disk parameter is set to true. By default, this parameter is set to 0, which means there is no limit set.
-
Click the Create button -
Stop the recording
When you no longer desire for the Flight Recording to be active and collecting data in your target application, select the recording from the list by clicking the checkbox to the left of the recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.
If the recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the recording was created with a continuous duration then it will collect data until explicitly stopped.
After stopping a recording it remains in the Active section of the Recordings view. This signifies that the recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.
Snapshot a Recording
Snapshotting an active recording produces a new recording in the target JVM
named snapshot-n
, where n
is a natural number. This snapshot contains all of
the JFR data that was present in the target JVM at the time that the snapshot
was taken and is in the STOPPED
state as soon as it is created (that is, the
snapshot will never record any further information and is fixed in size).
This is useful when you have an ongoing, continuous recording active in a
target application and want to preserve the specific information at a given
point in time. If the continuous source recording is configured with a maximum
event age or maximum recording size then the need for snapshots is more
apparent, since without snapshotting some older data will eventually be dropped
from the source recording and lost.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.
Archive a Recording
In contrast to Active Recordings, which reside within the container
of the target application, Archived Recordings reside within persistent
storage attached to a Cryostat instance. In OpenShift, for example, the
archives are stored in a PersistentVolumeClaim
.
Archived recordings are created by performing archival upon active recordings.
When this is requested, Cryostat connects to the target application and copies
the Flight Recorder data from the selected active recording into an archived
recording file in storage. The active source recording may be continuous or
fixed-duration, using any event template or event specifier string, may be in
any state (RUNNING
, STOPPED
, etc.), and may even be a snapshot.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Select an active recording
If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.
-
Click the Archive button -
Navigate to the Archived Recordings tab
Once the recording has been archived, a new entry will appear in the Archived Recordings table. All recordings in the archives are displayed here, regardless of the selected target application. The name of the archived recording reflects the address of the target application and the original name of the active recording that it was retrieved from, and includes a timestamp indicating when the archived recording was created.
Download an Active or Archived Recording
Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use JDK Mission Control for the heavy lifting.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
Download, Edit, and Upload a Customized Event Template
Most JVMs will come with at least two JFR event template definitions included: the Continuous and Profiling templates.
The Continuous template:
- has very low runtime overhead
- collects basic data
- is intended to be left on at all times, even in production
The Profiling template:
- may have some detectable runtime overhead
- collects more in-depth data
- should be used for a fixed duration when a specific problem is discovered at runtime
These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.
Of special note, JFR allows for the definition of application-specific custom events, which would not be captured in either of the default templates above.
Cryostat also provides the ALL
meta-template, which enables all
event types in the selected target application, with default values for each
event option. This is not a true event template and does not have an XML
definition to download.
-
Navigate to Events
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Edit the template
The template definition is an XML file, so you may open and edit it with your favourite text editor or XML document editor. The JMC Template Manager can also be used for this purpose, as described in this section . <configuration>
element at the top of the file and setting thelabel=
attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. Thedescription
andprovider
attributes can also be used to help identify this template later and to explain its goals and purpose.
Edit Template With JMC
JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc event template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.
-
Acquire an Event Template
Ensure that you have an event template definition .jfc file somewhere on your local workstation, with read/write permissions. $JAVA_HOME/lib/jfr
. -
Export the Template
Click Export File... to save the event template to a .jfc file on your local workstation.
Re-Upload a Recording to Archives
After downloading an archived recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage lost, and you then want to use Cryostat’s analysis tools with a previously retrieved recording in a new Cryostat instance.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
View a Recording in Grafana
Cryostat provides integration with Grafana to plot curated time series metrics from a recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator, you must retrieve the credentials generated by the operator in order to log into Grafana. These credentials are stored in a Kubernetes Secret. Once logged in, navigate to the dashboard titled Dashboard. You will see a variety of metrics plotted from your recording.
-
Retrieve the Grafana credentials
CRYOSTAT_NAME=$(kubectl get cryostat -o jsonpath='{$.items[0].metadata.name}') # Username kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_USER}' | base64 -d # Password kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d
If you installed Cryostat into Kubernetes or OpenShift using the Cryostat Operator, use kubectl or oc to get the generated username and password for Grafana and save them for later. -
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Choose an active or archived recording
If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.
View and Download Automated Analysis for a Recording
Cryostat integrates the same automated analysis reports that you would find in JDK Mission Control. The JMC rules engine analyzes your recording and looks for common problems and assigns a severity score from 0 (no problem) to 100 (potentially severe problem). Results with score 0 are hidden from the report unless you select the Show OK Results checkbox. More details on each result can be found by clicking on the + symbol to the right of the rule name. These details often include suggestions on how to fix the problem. Cryostat also allows you to download the report HTML file for offline use.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Choose an active or archived recording
If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.
Add and Edit Recording Metadata Labels
Users can attach metadata or custom labels to JDK flight recordings managed by Cryostat. Recording labels can be used to identify recordings during queries or perform actions on multiple recordings containing the same labels. Here’s how to add and edit metadata labels on your JDK flight recordings.
-
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Edit an existing label
-
(Optional) Archive your recording to view labels copied onto the archived recording
Store JMX Credentials
If you have Java Management Extensions (JMX) authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your JMX credentials before it can access the JDK flight recordings on your target JVMs. When monitoring multiple target JVMs with Cryostat features such as Automatic Rules, you may want Cryostat to remember and reuse your JMX credentials for each target connection. JMX credentials are automatically stored when you navigate to the Recordings tab and select a target JVM with JMX authentication enabled.
Cryostat stores JMX credentials according to each target’s unique JMX service URL, also known as a connection URL. Even if the underlying JVM instance changes, the target alias changes, or the target application restarts, Cryostat will apply the stored JMX credentials to the connection URL that the credentials are associated with.
Here’s how to manage stored JMX credentials with the Cryostat web UI.
-
(Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM
-
View your stored JMX credentials
Create an Automated Rule
Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.
Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your flight recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.
We’ll walk through two use cases: continuous monitoring in a containerized JVM, and custom monitoring with Kubernetes labels or annotations.
Continuous monitoring in a containerized JVM
Previously, if we wanted to enable always-on continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s automated rules, we can enable JDK Flight Recorder at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.
-
Click the Create button -
Configure the new automated rule
Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.
Description: Enter an optional description for your rule.
Match Expression: We will fill this field in the next step.
Template: Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.
-
Create your match expression
The match expression in a rule definition is a Java-like snippet of code that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match expressions should thus evaluate to a boolean value. The simplest match expressions would be the booleans true or false; if we use true, the rule will apply to every target. The expression has a target object in global scope, with the following form in JSON notation:
{ “alias”: “myAppAlias”, “connectUrl”: “service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi”, “labels”: { “com.example/service”: “customer-login”, }, “annotations”: { “platform”: { “io.kubernetes/annotation”: “annotated” }, “cryostat”: { “PORT”: 9091, “HOST”: “cryostat”, “NAMESPACE”: “myproject” } } }
The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The labels and platform annotations may be empty—in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Here are some examples of match expressions:
target.alias == ’com.example.MainClass’ target.alias == ’myAlias’ target.labels[‘com.example/service’] == ’customer-login’ target.labels[‘com.example/service’] != ’customer-login’ target.annotations.cryostat.PORT > 3000 target.annotations.cryostat.PORT > 3000 && target.annotations.platform[‘io.kubernetes/annotation’] == ‘enabled’ !!target.annotations.platform[‘io.kubernetes/annotation’] /^customer-login[0-9]\*$/.test(target.alias)
-
Evaluate your match expression
When you enter a match expression in the Match Expression field, the Match Expression Evaluator will test if your match expression matches the target selected in the dropdown. You can check that your match expression matches all of your desired targets by selecting different targets from the dropdown.
-
(Optional) Adjust recording options
Optionally set the recording options. Maximum Size: The maximum size of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.
Maximum Age: The maximum age of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.
Archival Period: Time between copies of active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.
Preserved Archives: The number of recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.
In the example image, the maximum recording age was set to 300 seconds and the archival period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your flight recorder data is preserved in Cryostat's archives.
-
Click the Create button -
Navigate to Recordings
Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials. If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
-
Observe the automatically generated recording in the Active Recordings table
Switch to the Recordings tab and view the new recording in the Active Recordings table.
Custom monitoring with Kubernetes labels or annotations
We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:
{
"name": "k8sMonitoring",
"description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
"matchExpression": "target.annotations.platform[‘jfrMonitoring’]==’enabled’",
"eventSpecifier": "template=Demo,type=CUSTOM",
"archivalPeriodSeconds": 300,
"preservedArchives": 12
}
Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a recording started automatically using the custom Demo template from our first use case. It will take an archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.
With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.
As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these archived recordings in Cryostat’s Grafana dashboard.
Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.
Configure Graphical Notifications
Actions such as starting/stopping/archiving recordings and state changes such as recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active then the notification stream within the web-client graphical console can become quite noisy.
-
Navigate to Settings
Click the cog or gear icon in the application masthead to access the Settings view. The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.
-
Locate the Notifications card