Guides on Using Cryostat

This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.

• Start/Stop a Recording
• Snapshot a Recording
• Archive a Recording
• Download an Active or Archived Recording
• Download, Edit, and Upload a Customized Event Template
• Edit Template With JMC
• Re-Upload a Recording to Archives
• View a Recording in Grafana
• View and Download Automated Analysis for a Recording
• Add and Edit Recording Metadata Labels
• Store JMX Credentials
• Add a Trusted Certificate
• Create an Automated Rule
  • Continuous monitoring in a containerized JVM
  • Custom monitoring with Kubernetes labels or annotations
• Upload an Automated Rule
• Download an Automated Rule
• Configure Graphical Notifications

Start/Stop a Recording

The first and most basic workflow is to start a Flight Recording in a target JVM.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Click Create

   

4. Configure the new recording

   

   Enter a name for the new recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

   Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

5. (Optional) Adjust any Advanced Options for the recording

   

   In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:

   To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.

   Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.

   Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the recording before being discarded. This parameter is valid only when the disk parameter is set to true. By default, this parameter is set to 0, which means there is no limit set.

6. Click the Create button
7. Stop the recording

   

   When you no longer desire for the Flight Recording to be active and collecting data in your target application, select the recording from the list by clicking the checkbox to the left of the recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.

   If the recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the recording was created with a continuous duration then it will collect data until explicitly stopped.

   After stopping a recording it remains in the Active section of the Recordings view. This signifies that the recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.

Snapshot a Recording

Snapshotting an active recording produces a new recording in the target JVM named snapshot-n, where n is a natural number. This snapshot contains all of the JFR data that was present in the target JVM at the time that the snapshot was taken and is in the STOPPED state as soon as it is created (that is, the snapshot will never record any further information and is fixed in size). This is useful when you have an ongoing, continuous recording active in a target application and want to preserve the specific information at a given point in time. If the continuous source recording is configured with a maximum event age or maximum recording size then the need for snapshots is more apparent, since without snapshotting some older data will eventually be dropped from the source recording and lost.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

   If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

3. Click Create

   

4. Switch to the Snapshot Recording tab

   

5. Observe the new snapshot

   

Archive a Recording

In contrast to Active Recordings, which reside within the container of the target application, Archived Recordings reside within persistent storage attached to a Cryostat instance. In OpenShift, for example, the archives are stored in a PersistentVolumeClaim.

Archived recordings are created by performing archival upon active recordings. When this is requested, Cryostat connects to the target application and copies the Flight Recorder data from the selected active recording into an archived recording file in storage. The active source recording may be continuous or fixed-duration, using any event template or event specifier string, may be in any state (RUNNING, STOPPED, etc.), and may even be a snapshot.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Select an active recording

   

4. Click the Archive button
5. Navigate to the Archived Recordings tab

   

Download an Active or Archived Recording

Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use JDK Mission Control for the heavy lifting.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Select a recording

   

4. Download the recording

   

5. Choose what to do with the recording file

   

Download, Edit, and Upload a Customized Event Template

Most JVMs will come with at least two JFR event template definitions included: the Continuous and Profiling templates.

The Continuous template:

• has very low runtime overhead
• collects basic data
• is intended to be left on at all times, even in production

The Profiling template:

• may have some detectable runtime overhead
• collects more in-depth data
• should be used for a fixed duration when a specific problem is discovered at runtime

These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.

Of special note, JFR allows for the definition of application-specific custom events, which would not be captured in either of the default templates above.

Cryostat also provides the ALL meta-template, which enables all event types in the selected target application, with default values for each event option. This is not a true event template and does not have an XML definition to download.

1. Select the target application

   

   The target selected will provide the base Continuous and Profiling template definitions to start from. Most target applications will be interchangeable here.
2. Navigate to Events

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Download the template

   

4. Edit the template

   

   Edit the template definition to suit your requirements. When you are satisfied with the new configuration, give your new template a meaningful and recognizable tag by editing the <configuration> element at the top of the file and setting the label= attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. The description and provider attributes can also be used to help identify this template later and to explain its goals and purpose.
5. Open the template upload dialog

   

6. Upload the template

   

7. Observe the new template

   

Edit Template With JMC

JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc event template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.

1. Acquire an Event Template

   

   If you do not have an event template definition then you can visit Download, Edit, and Upload a Customized Event Template and follow the first few steps to retrieve one from a remote target application accessible by Cryostat. Otherwise, if you have the JDK installed locally, you can find templates at $JAVA_HOME/lib/jfr.
2. Launch JDK Mission Control

   

3. Open the Template Manager

   

4. Import the Base Template

   

5. Edit the Template

   

6. Export the Template

   

   Once your edited template is saved to to a file on your local workstation it can be used when starting new Flight Recordings. To do this using Cryostat, see Download, Edit, and Upload a Customized Event Template for instructions on how to upload the new template to Cryostat and Start/Stop a Recording for instructions on how to create a new recording using this template.

Re-Upload a Recording to Archives

After downloading an archived recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage lost, and you then want to use Cryostat’s analysis tools with a previously retrieved recording in a new Cryostat instance.

1. Select an application

   

   Although the archived storage is global to the Cryostat instance and is not target application specific, the view for Archived Recordings is only visible when a target application is selected.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Select the Archived Recordings tab

   

4. Select the Recording to Upload

   

5. Upload the Recording

   

View a Recording in Grafana

Cryostat provides integration with Grafana to plot curated time series metrics from a recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator, you must retrieve the credentials generated by the operator in order to log into Grafana. These credentials are stored in a Kubernetes Secret. Once logged in, navigate to the dashboard titled Dashboard. You will see a variety of metrics plotted from your recording.

1. Retrieve the Grafana credentials

   CRYOSTAT_NAME=$(kubectl get cryostat -o jsonpath='{$.items[0].metadata.name}')
   # Username
   kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_USER}' | base64 -d
   # Password
   kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d
           

2. Select the target application

   

3. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

4. Choose an active or archived recording

   

5. Select View in Grafana...

   

6. Log into Grafana

   

7. Select Cryostat's dashboard

   

8. View and interact with data from your recording

   

View and Download Automated Analysis for a Recording

Cryostat integrates the same automated analysis reports that you would find in JDK Mission Control. The JMC rules engine analyzes your recording and looks for common problems and assigns a severity score from 0 (no problem) to 100 (potentially severe problem). Results with score 0 are hidden from the report unless you select the Show OK Results checkbox. More details on each result can be found by clicking on the + symbol to the right of the rule name. These details often include suggestions on how to fix the problem. Cryostat also allows you to download the report HTML file for offline use.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Choose an active or archived recording

   

4. Expand the recording

   

5. View details and suggestions for results

   

6. Select Download Report

   

7. Open the downloaded report in your browser

   

Add and Edit Recording Metadata Labels

Users can attach metadata or custom labels to JDK flight recordings managed by Cryostat. Recording labels can be used to identify recordings during queries or perform actions on multiple recordings containing the same labels. Here’s how to add and edit metadata labels on your JDK flight recordings.

1. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

2. Click Create

   

3. Add a label on the Custom Flight Recording form

   

   When creating a custom flight recording with Cryostat, expand the form section Show metadata options. Click Add Label to add a key-value label pair to the recording.
4. View your labels on the Active Recordings Table

   

   The new recording will appear in the Recordings tab with your custom label as well as default labels containing information about the selected recording template.
5. Edit an existing label

   

   Recording labels can also be edited after recordings have been created or re-uploaded to archives. It looks like the custom label in our example contains a typo - we can fix the typo by editing the label. Click the ellipsis menu beside the recording table entry, then click Edit Metadata. The labels section will appear as a form where you can add, edit, or delete existing labels. Click Save to save your edited labels.
6. View your edited labels

   

   The recording labels should be updated in the Active Recordings table.
7. (Optional) Archive your recording to view labels copied onto the archived recording

   

   On the Active Recordings table, click the checkbox next to the recording, then click Archive to archive your recording. Notice that the archived recording also copies the labels from the active recording. You can also add labels to any recording uploaded to Cryostat’s archives.

Store JMX Credentials

If you have Java Management Extensions (JMX) authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your JMX credentials before it can access the JDK flight recordings on your target JVMs. When monitoring multiple target JVMs with Cryostat features such as Automatic Rules, you may want Cryostat to remember and reuse your JMX credentials for each target connection. JMX credentials are automatically stored when you navigate to the Recordings tab and select a target JVM with JMX authentication enabled.

Cryostat stores JMX credentials according to each target’s unique JMX service URL, also known as a connection URL. Even if the underlying JVM instance changes, the target alias changes, or the target application restarts, Cryostat will apply the stored JMX credentials to the connection URL that the credentials are associated with.

Here’s how to manage stored JMX credentials with the Cryostat web UI.

1. Navigate to the Security tab

   

   First, navigate to the Security tab. The Stored Credentials table lists all targets for which Cryostat has stored JMX credentials. Click Add to enter JMX credentials for your desired target JVM.
2. Enter your JMX credentials

   

   A modal will appear, prompting you to select a target JVM and enter your JMX credentials.
3. (Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM

   

   Alternatively, JMX credentials will also be stored if you navigate to either the Recordings tab or the Events tab and select a target JVM with JMX authentication enabled. The authentication form will appear, prompting you to enter your JMX credentials. The credentials will be automatically stored and will appear in the Stored JMX Credentials table.
4. View your stored JMX credentials

   

   The target name will appear on the Stored Credentials table to indicate that Cryostat has stored JMX credentials for that target. As a security precaution, you will not be able to view the actual credentials after you have submitted them. To remove any stored credentials, select the checkbox next to the target and click Delete.

Add a Trusted Certificate

If you have Java Management Extensions (JMX) over SSL enabled on your containerized JVMs, you must configure Cryostat to trust the SSL certificate presented by the containerized JVM when Cryostat attempts to open a JMX connection. If you do not complete this configuration, Cryostat cannot open a JMX connection for the purposes of performing JFR management tasks.

Here’s how to add a trusted SSL certificate with the Cryostat web UI.

1. Navigate to the Security tab

   

   Click the Security tab.
2. Upload the Certificate

   

   Click the Upload button on the Import SSL Certificates card. This action opens a file-upload dialog, where you can choose the certificate that you want to upload to Cryostat. You can repeat this process multiple times to add multiple trusted certificates.
3. Restart Cryostat to apply the changes. If you do not restart your Cryostat instance, the added certificates are not reloaded. This causes connections to fail because the Cryostat JMX client cannot trust the certificates. Depending on your deployment platform and configuration, restarting Cryostat might require any of the following:
   • On OpenShift/Kubernetes with Cryostat Operator:

     oc delete cryostat/<my-cryostat-cr-name>
     oc create -f <my-cryostat-cr-name>.yaml

   • On OpenShift/Kubernetes without Cryostat Operator:

     oc rollout retry dc/<my-cryostat-dc-name>

   • On Podman:

     podman restart <my-cryostat-container-id>

Create an Automated Rule

Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your flight recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

We’ll walk through two use cases: continuous monitoring in a containerized JVM, and custom monitoring with Kubernetes labels or annotations.

Continuous monitoring in a containerized JVM

Previously, if we wanted to enable always-on continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s automated rules, we can enable JDK Flight Recorder at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

1. Navigate to the Automated Rules tab

   

2. Click the Create button
3. Configure the new automated rule

   

   Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

   Description: Enter an optional description for your rule.

   Match Expression: We will fill this field in the next step.

   Template: Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

4. Create your match expression

   The match expression in a rule definition is a Java-like snippet of code that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match expressions should thus evaluate to a boolean value. The simplest match expressions would be the booleans true or false; if we use true, the rule will apply to every target. The expression has a target object in global scope, with the following form in JSON notation:

   {
     "alias": "myAppAlias",
     "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
     "labels": {
       "com.example/service": "customer-login",
     },
     "annotations": {
       "platform": {
         "io.kubernetes/annotation": "annotated"
       },
       "cryostat": {
         "PORT": 9091,
         "HOST": "cryostat",
         "NAMESPACE": "myproject"
       }
     }
   }

   The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The labels and platform annotations may be empty—in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Here are some examples of match expressions:

   target.alias == ’com.example.MainClass’
   
   target.alias == ’myAlias’
   
   target.labels[‘com.example/service’] == ’customer-login’
   
   target.labels[‘com.example/service’] != ’customer-login’
   
   target.annotations.cryostat.PORT > 3000
   
   target.annotations.cryostat.PORT > 3000 && target.annotations.platform[‘io.kubernetes/annotation’] == ‘enabled’
   
   !!target.annotations.platform[‘io.kubernetes/annotation’]
   
   /^customer-login[0-9]\*$/.test(target.alias)

5. Evaluate your match expression

   

   When you enter a match expression in the Match Expression field, the Match Expression Evaluator will test if your match expression matches the target selected in the dropdown. You can check that your match expression matches all of your desired targets by selecting different targets from the dropdown.

6. (Optional) Adjust recording options

   

   Maximum Size: The maximum size of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

   Maximum Age: The maximum age of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

   Archival Period: Time between copies of active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

   Preserved Archives: The number of recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

   In the example image, the maximum recording age was set to 300 seconds and the archival period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your flight recorder data is preserved in Cryostat's archives.

7. Click the Create button
8. Observe the new rule in the Automated Rules table

   

9. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

10. Observe the automatically generated recording in the Active Recordings table

    

    Once you've set up your automated rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications.

Custom monitoring with Kubernetes labels or annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "target.annotations.platform[‘jfrMonitoring’]==’enabled’",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}

Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a recording started automatically using the custom Demo template from our first use case. It will take an archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these archived recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

Upload an Automated Rule

Instead of stepping through the form every time, you can also edit your Automated Rule definition files (JSON format) locally and quickly upload them.

Below is an example of a rule file content:

{
  "name":"My Rule",
  "description":"",
  "matchExpression":"target.alias == 'io.cryostat.Cryostat' || target.annotations.cryostat['PORT'] == 9091","eventSpecifier":"template=Profiling,type=TARGET",
  "archivalPeriodSeconds":300,
  "preservedArchives":12,
  "maxAgeSeconds":1000,
  "maxSizeBytes":1000
}

1. Open Automated Rules upload prompt

   

2. Attach a rule file by dragging & dropping or clicking Browse....

   

3. Click Submit button to upload.
4. Observe the new rule in the Automated Rules table

   

Download an Automated Rule

Automated Rules can easily be downloaded for viewing and editting.

1. Open action menu for a selected rule

   

2. Click Download button to download.

Configure Graphical Notifications

Actions such as starting/stopping/archiving recordings and state changes such as recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active then the notification stream within the web-client graphical console can become quite noisy.

1. Navigate to Settings

   

   The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.

2. Locate the Notifications card

   

   The Notifications card within this view is used to control the graphical display of notifications that correspond to WebSocket messages sent by the Cryostat backend when actions and state changes occur. The card contains a toggle switch for each category of notification, as well as a global switch to enable/disable all categories at once. Disabling a notification category only prevents it from appearing as a pop-up notification toast in the current web-client instance. The notification will still appear in the notification drawer, accessed by clicking the bell icon in the application masthead, and other web-client instances or API clients will still receive the notification to process as they will.