Guides on Using Cryostat

This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.

Start/Stop a Recording

The first and most basic workflow is to start a Flight Recording in a target JVM.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Click Create
    Click <i>Create</i>
  4. Configure the new recording
    Configure the new recording

    Enter a name for the new recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

    Select the duration of the recording. The duration can be specified in seconds, minutes, or hours. If you want to record indefinitely, select the Continuous option. You may also want to save the recording to Cryostat archive storage when the recording stops with the Archive on Stop option.

    Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the Continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

    To learn about Metadata Options, see Add and Edit Recording Metadata Labels.

  5. (Optional) Adjust any Advanced Options for the recording
    <i>(Optional)</i> Adjust any <i>Advanced Options</i> for the recording

    In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:

    To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.

    Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.

    Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the recording before being discarded. This parameter is valid only when the disk parameter is set to true, and the recording duration is not Continuous. By default, this parameter is set to 0, which means there is no limit set.

  6. Click the Create button
  7. Stop the recording
    Stop the recording

    When you no longer desire for the Flight Recording to be active and collecting data in your target application, select the recording from the list by clicking the checkbox to the left of the recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.

    If the recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the recording was created with a continuous duration then it will collect data until explicitly stopped.

    After stopping a recording it remains in the Active section of the Recordings view. This signifies that the recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.

Snapshot a Recording

Snapshotting an active recording produces a new recording in the target JVM named snapshot-n, where n is a natural number. This snapshot contains all of the JFR data that was present in the target JVM at the time that the snapshot was taken and is in the STOPPED state as soon as it is created (that is, the snapshot will never record any further information and is fixed in size). This is useful when you have an ongoing, continuous recording active in a target application and want to preserve the specific information at a given point in time. If the continuous source recording is configured with a maximum event age or maximum recording size then the need for snapshots is more apparent, since without snapshotting some older data will eventually be dropped from the source recording and lost.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  3. Click Create
    Click <i>Create</i>
  4. Switch to the Snapshot Recording tab
    Switch to the <i>Snapshot Recording</i> tab
    Switch to the Snapshot Recording tab and click the Create button.
  5. Observe the new snapshot
    Observe the new snapshot
    After clicking the Create button you will be returned to the Active Recordings view and the new snapshot recording will be present in the table. The snapshot recording can then be treated as any other active recording in the target application's JVM memory.

Archive a Recording

In contrast to Active Recordings, which reside within the container of the target application, Archived Recordings reside within persistent storage attached to a Cryostat instance. In OpenShift, for example, the archives are stored in a PersistentVolumeClaim.

Archived recordings are created by performing archival upon active recordings. When this is requested, Cryostat connects to the target application and copies the Flight Recorder data from the selected active recording into an archived recording file in storage. The active source recording may be continuous or fixed-duration, may be using any event template, may be in any state (RUNNING, STOPPED, etc.), and may even be a snapshot.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Select an Active Recording
    Select an Active Recording

    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  4. Click the Archive button
  5. Navigate to the Archived Recordings tab
    Navigate to the <i>Archived Recordings</i> tab
    Once the recording has been archived, a new entry will appear in the target JVM's Archived Recordings table. All recordings that were saved from the current target will be listed here in their own table. Switching to a different target from the dropdown will list only the recordings archived from that source target. The name of the archived recording reflects the address of the target application, the original name of the active recording that it was retrieved from, and includes a timestamp indicating when the archived recording was created.

Viewing Archived Recordings

There are several ways to view Archived Recordings. The first method is to navigate to the Archived Recordings tab. See Archive a Recording.

All-Targets Archived Recordings View

The second method is to navigate to the new All-Targets archived recording view within the Archives tab on Cryostat console sidebar.

The All-Targets view gathers all of Cryostat’s discovered target JVM applications into one section for ease of access. Here, we are able to interact with any archived recordings that have been saved from a source target by opening a target’s nested recordings table.

  1. Navigate to the All-Targets Archived Recordings view
    Navigate to the All-Targets Archived Recordings view
    Click on the Archives tab on the sidebar, and the first tab should automatically be selected as the All Targets Archives view.

    The option to automatically hide all targets with zero archived recordings is on by default and can be toggled. Targets can also be filtered in the search bar.

  2. Select a source target application
    Select a source target application
    Clicking the dropdown arrow next to a target name will list any archived recordings originating from that source target.

All-Archives Archived Recordings View

The third method is to navigate to the new All-Archives archived recording view within the Archives tab on Cryostat console sidebar.

The All-Archives view is a view which queries Cryostat’s internal storage for any created archived recordings and the directories that contain them. Here, we are able to interact with any archived recordings that have been saved into storage by opening a nested recordings table within each directory.

This view is used to save any lost archived recordings in case any target JVM restarts or exits. It will be empty if no recordings are currently saved to storage.

  1. Navigate to the All-Archives Archived Recordings view
    Navigate to the All-Archives Archived Recordings view
    Click on the Archives tab on the sidebar, and select the second tab titled All Archives.

    Directories can be filtered in the search bar.

  2. Select an archives directory
    Select an archives directory
    Clicking the dropdown arrow next to a directory name will list any archived recordings within that directory in the Cryostat storage. Again, we can interact with any archived recordings in a similar fashion as before.

    A directory name is related to its corresponding source target's serviceUrl. Mousing over the tooltip, we can also see a Cryostat generated hash ID for that target.

Download an Active or Archived Recording

Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use JDK Mission Control for the heavy lifting.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Select a recording
    Select a recording
    Determine the recording you wish to download to your local workstation. This may be either an Active or Archived recording. Select the appropriate tab in the Recordings view. No actual action needs to be taken at this step.
  4. Download the recording
    Download the recording
    Click the action overflow three-dot menu on the right side of the recording entry in the table, then click Download Recording.
  5. Choose what to do with the recording file
    Choose what to do with the recording file
    Your browser will present you with its standard file-save dialog for both the Archived Recording, and a .json file containing any metadata labels that were attached. It may take some time for these dialogs to appear. Once it does, you can choose to open the recording file directly in JDK Mission Control, or (recommended) to save the recording to disk.

Re-Upload a Recording to Archives

After downloading an archived recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage lost, and you then want to use Cryostat’s analysis tools with a previously retrieved recording in a new Cryostat instance.

  1. Navigate to the Archives tab
    Navigate to the <i>Archives</i> tab
    Select the Uploads tab.
    The entire Archives view contains several tabs related to performing operations on archived recordings. See Viewing Archived Recordings.
  2. Select the Recording to Upload
    Select the Recording to Upload
    Click the '+' button to bring up the upload prompt. Then click Browse... and select the .jfr file to upload. Note that the file must follow the Cryostat archive recording naming convention. Mouse over the [?] tooltip on the prompt for more information.
  3. (Optional) Add metadata labels
    <i>(Optional)</i> Add metadata labels
    Click Show metadata options to add optional metadata labels to recording. You can either add labels by clicking the Add Label button, or by uploading a custom .json labels file from your local storage. A custom labels file can be downloaded alongside downloading a recording. To learn how to download a recording and any associated labels, see Download and Active or Archived Recording. and for more on Cryostat metadata labels, see Add and Edit Recording Metadata Labels.
  4. Upload the Recording
    Upload the Recording
    Click Upload and observe that the recording is now present in the Uploads tab of the Archives view.

Download, Edit, and Upload a Customized Event Template

Most JVMs will come with at least two JFR event template definitions included: the Continuous and Profiling templates.

The Continuous template:

  • has very low runtime overhead
  • collects basic data
  • is intended to be left on at all times, even in production

The Profiling template:

  • may have some detectable runtime overhead
  • collects more in-depth data
  • should be used for a fixed duration when a specific problem is discovered at runtime

These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.

Of special note, JFR allows for the definition of application-specific custom events, which would not be captured in either of the default templates above.

Cryostat also provides the ALL meta-template, which enables all event types in the selected target application, with default values for each event option. This is not a true event template and does not have an XML definition to download.

  1. Select the target application
    Select the target application
    The target selected will provide the base Continuous and Profiling template definitions to start from. Most target applications will be interchangeable here.
  2. Navigate to Events
    Navigate to Events
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Download the Template
    Download the Template
    Click the action overflow three-dot menu on the right side of the template entry in the table, then click Download.
  4. Edit the Template
    Edit the Template
    The template definition is an XML file, so you may open and edit it with your favourite text editor or XML document editor. The JMC Template Manager can also be used for this purpose, as described in this section .
    Edit the template definition to suit your requirements. When you are satisfied with the new configuration, give your new template a meaningful and recognizable tag by editing the <configuration> element at the top of the file and setting the label= attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. The description and provider attributes can also be used to help identify this template later and to explain its goals and purpose.
  5. Open the Template upload dialog
    Open the Template upload dialog
    Back on the Cryostat Events view, click the plus (+) icon in the table toolbar. A dialog will appear.
  6. Upload the Template
    Upload the Template
    Click the Browse button. Your browser will present its native file chooser dialog to select the file to upload. Browse and select the template file, then click OK/Select/Continue/Open. Click Submit on the Cryostat dialog.
  7. Observe the new Template
    Observe the new Template
    Once the template has been uploaded you will be returned to the Events view, and your template will be present in the table.

Edit Template With JMC

JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc event template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.

  1. Acquire an Event Template
    Acquire an Event Template
    Ensure that you have an event template definition .jfc file somewhere on your local workstation, with read/write permissions.
    If you do not have an event template definition then you can visit Download, Edit, and Upload a Customized Event Template and follow the first few steps to retrieve one from a remote target application accessible by Cryostat. Otherwise, if you have the JDK installed locally, you can find templates at $JAVA_HOME/lib/jfr.
  2. Launch JDK Mission Control
    Launch JDK Mission Control
    The main JMC window after launch
  3. Open the Template Manager
    Open the Template Manager
    Select Window > Flight Recording Template Manager
  4. Import the Base Template
    Import the Base Template
    Click Import Files..., then browse to and select the template from Step 1.
  5. Edit the Template
    Edit the Template
    Select the imported template, then click Edit and make your desired changes. Before making changes you may also click Duplicate to create an identical copy of the template as a base to iterate upon.
  6. Export the Template
    Export the Template
    Click Export File... to save the event template to a .jfc file on your local workstation.
    Once your edited template is saved to to a file on your local workstation it can be used when starting new Flight Recordings. To do this using Cryostat, see Download, Edit, and Upload a Customized Event Template for instructions on how to upload the new template to Cryostat and Start/Stop a Recording for instructions on how to create a new recording using this template.

View a Recording in Grafana

Cryostat provides integration with Grafana to plot curated time series metrics from a recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator, you must retrieve the credentials generated by the operator in order to log into Grafana. These credentials are stored in a Kubernetes Secret. Once logged in, navigate to the dashboard titled Dashboard. You will see a variety of metrics plotted from your recording.

  1. Retrieve the Grafana credentials
    CRYOSTAT_NAME=$(kubectl get cryostat -o jsonpath='{$.items[0].metadata.name}')
    # Username
    kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_USER}' | base64 -d
    # Password
    kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d
            
    If you installed Cryostat into Kubernetes or OpenShift using the Cryostat Operator, use kubectl or oc to get the generated username and password for Grafana and save them for later.
  2. Select the target application
    Select the target application
  3. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  4. Select a recording
    Select a recording
    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application. You may also select an archived recording to view in Grafana.
  5. Select View in Grafana...
    Select <i>View in Grafana...</i>
    Select View in Grafana... from the recording's overflow menu.
  6. Log into Grafana
    Log into Grafana
    Use the credentials retrieved in step 1 to log into the Grafana web client.
  7. View and interact with data from your recording
    View and interact with data from your recording
    Observe the plotted time series data from curated metrics in your recording. Select time ranges to zoom into the data.

View and Download Automated Analysis for a Recording

Cryostat integrates the same automated analysis reports that you would find in JDK Mission Control. The JMC rules engine analyzes your recording and looks for common problems and assigns a severity score from 0 (no problem) to 100 (potentially severe problem). Results with score 0 are hidden from the report unless you select the Show OK Results checkbox. More details on each result can be found by clicking on the + symbol to the right of the rule name. These details often include suggestions on how to fix the problem. Cryostat also allows you to download the report HTML file for offline use.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Select a recording
    Select a recording
    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application. You may also select an archived recording for automated analysis.
  4. Expand the recording
    Expand the recording
    Expand the recording with the chevron to the left of the recording name. The automated analysis report will appear below the recording.
  5. View details and suggestions for results
    View details and suggestions for results
    Click the + button on the right side of each result to view specifics on what the result means and possible suggestions to fix the potential problem.
  6. Download the report
    Download the report
    To download the automated analysis report for offline viewing, select View Report ... from the recording's overflow menu.

    View the report on its own without connecting to Cryostat. Check Show OK Results to include results where the rules engine found no issues in the recording.
    To download the HTML Automated Analysis report to local disk, right click the page and select Save Page As.... Alternatively, press Ctrl+S on Windows/Linux, or +S on macOS.

Add and Edit Recording Metadata Labels

Users can attach metadata or custom labels to JDK flight recordings managed by Cryostat. Recording labels can be used to identify recordings during queries or perform actions on multiple recordings containing the same labels. Here’s how to add and edit metadata labels on your JDK flight recordings.

  1. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  2. Click Create
    Click <i>Create</i>
  3. Add metadata labels to the Create Recording form
    Add metadata labels to the Create Recording form
    When creating a custom flight recording with Cryostat, expand the form section Show metadata options. Click Add Label and add key-value label pairs to the form. Finally, click Create to attach the labels and create the recording.
  4. View your labels on the Active Recordings Table
    View your labels on the Active Recordings Table
    The new recording will appear in the Recordings tab with your custom label as well as default labels containing information about the selected recording template.
  5. Edit an existing label
    Edit an existing label

    Recording labels can also be edited after recordings have been created or re-uploaded to archives. It looks like the custom label in our example contains a typo - we can fix the typo by editing the label. First select a recording from the table with the 🗹 checkbox. Then, click the Edit Labels button to bring up the label drawer. Finally, click the Edit button that appears from the drawer.



    The labels section will appear as a form where you can add, edit, or delete existing labels, just like before when we created the recording. Fix the typo, and click Save to save your edited labels.

  6. View your edited labels
    View your edited labels
    The recording labels should be updated in the Active Recordings table.
  7. (Optional) Archive your recording to view labels copied onto the archived recording
    <i>(Optional)</i> Archive your recording to view labels copied onto the archived recording
    On the Active Recordings table, click the checkbox next to the recording, then click Archive to archive your recording. Notice that the archived recording also copies the labels from the active recording. You can also add labels to any recording uploaded to Cryostat’s archives.
  8. (Optional) Bulk-edit recording labels across multiple recordings
    <i>(Optional)</i> Bulk-edit recording labels across multiple recordings

    Create another recording on the same target. Then select both recordings on the Recordings Table and click Edit Labels and Edit.This time, only labels that are present on both recordings will be shown in the form. Let's delete the two common template-related labels, and add a new label to both recordings. Then finally, click Save.



    Congratulations, you have successfully bulk-edited labels across multiple recordings!

Store JMX Credentials

If you have Java Management Extensions (JMX) authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your JMX credentials before it can access the JDK flight recordings on your target JVMs. You can Configure JMX Credentials Storage and choose whether these credentials are held in browser memory for the current session only, or if they are uploaded to Cryostat’s JMX credentials keyring.

If you intend to use Cryostat Automated Rules, then you should store target application JMX credentials in Cryostat’s JMX credentials keyring, which is outlined below. Even if you do not use Automated Rules, you may find it more convenient to store credentials in the keyring. This way, the Cryostat web UI does not need to prompt you for credentials when you manually access JFR information about target applications.

  1. Navigate to the Security tab
    Navigate to the Security tab
    First, navigate to the Security tab. The Store JMX Credentials table lists all credentials in Cryostat's keyring. Click Add to define a new set of credentials.
  2. Enter your JMX credentials
    Enter your JMX credentials
    A modal will appear. You can select a target JVM to view its properties. These properties can be used to define a matchExpression that tests whether this set of credentials should apply to the selected target JVM. This is the same kind of matchExpression as those used by Automated Rules, so you may find it useful to reuse the same matchExpression from your rule definitions for your credentials definitions. The matchExpression, username, and password fields are all required.
  3. (Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM
    (Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM
    Alternatively, JMX credentials may also be stored if you navigate to either the Recordings tab or the Events tab and select a target JVM with JMX authentication enabled. The authentication form will appear, prompting you to enter your JMX credentials. If you have configured Backend JMX Credentials Storage, then a new credentials definition with the provided username and password will be stored for this specific target application in the Cryostat keyring. If you have configured JMX credentials with the Session option then these entered credentials will not be stored in the Cryostat keyring.
  4. View your stored JMX credentials
    View your stored JMX credentials
    The Store JMX Credentials table will update with rows for each of the credentials you have stored in the keyring. The matchExpression is visible along with a count of the number of known target applications that these credentials will apply to. You can click the reverse chevron icon at the left of the table row to expand the row, which will display a list of those matched target applications. For security reasons it is not possible to view the stored username and password associated with a credentials definition in the keyring.

Configure JMX Credentials Storage

Target JVM applications may require Cryostat to pass an authentication challenge before being able to communicate over JMX and manage JFR. Cryostat has two supported mechanisms for these JMX credentials:

  1. JMX Credentials Keyring: see Store JMX Credentials for more detail. This mechanism entails uploading a JMX Credentials definition to the Cryostat backend keyring storage. Cryostat automatically checks the keyring for credentials matching a target application when a request to that application is opened. If no credentials are found, Cryostat responds to the requesting client (e.g. the Cryostat Web UI) with a response indicating the JMX authentication failure.
  2. X-JMX-Authorization header passthrough: when the Cryostat Web UI receives a JMX authentication failure response from the Cryostat server, it displays a prompt asking for JMX credentials. Depending on the configuration set in this section, those credentials may either be held locally in your browser session, or they may be uploaded to the Cryostat credentials keyring. If the credentials are held locally in your browser session, then the Cryostat Web UI will automatically attach a custom header with API requests containing the JMX credentials you specify. The Cryostat server sees this header and uses the credentials when opening the JMX connection to the target, but does not persist these credentials. When the JMX credentials passthrough header is present, they supersede credentials stored in the Cryostat server's keyring.

Now that you understand the difference, let’s continue to see how you can configure the Cryostat Web UI to use one or the other when you complete a JMX Authentication prompt.

  1. Navigate to Settings
    Navigate to Settings
    Click the cog or gear icon in the application masthead to access the Settings view.

    The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.

  2. Locate the JMX Credentials Storage card
    Locate the JMX Credentials Storage card
    This card contains a brief explanation of its purpose and a simple dropdown menu with selections for where any JMX Credentials entered into a JMX Authentication Required challenge modal will be stored. Choose Session (Browser Memory) to use the header passthrough mechanism described above, or choose Backend to automatically store the credentials in the Cryostat server keyring.

Add a Trusted Certificate

If you have Java Management Extensions (JMX) over SSL enabled on your containerized JVMs, you must configure Cryostat to trust the SSL certificate presented by the containerized JVM when Cryostat attempts to open a JMX connection. If you do not complete this configuration, Cryostat cannot open a JMX connection for the purposes of performing JFR management tasks.

Here’s how to add a trusted SSL certificate with the Cryostat web UI.

  1. Navigate to the Security tab
    Navigate to the Security tab
    Click the Security tab.
  2. Upload the Certificate
    Upload the Certificate
    Click the Upload button on the Import SSL Certificates card. This action opens a file-upload dialog, where you can choose the certificate that you want to upload to Cryostat. You can repeat this process multiple times to add multiple trusted certificates.
  3. Restart Cryostat to apply the changes. If you do not restart your Cryostat instance, the added certificates are not reloaded. This causes connections to fail because the Cryostat JMX client cannot trust the certificates. Depending on your deployment platform and configuration, restarting Cryostat might require any of the following:
    • On OpenShift/Kubernetes with Cryostat Operator:
      oc delete cryostat/<my-cryostat-cr-name>
      oc create -f <my-cryostat-cr-name>.yaml
    • On OpenShift/Kubernetes without Cryostat Operator:
      oc rollout retry dc/<my-cryostat-dc-name>
    • On Podman:
      podman restart <my-cryostat-container-id>

Create an Automated Rule

Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your flight recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

We’ll walk through two use cases: continuous monitoring in a containerized JVM, and custom monitoring with Kubernetes labels or annotations.

Continuous monitoring in a containerized JVM

Previously, if we wanted to enable always-on continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s automated rules, we can enable JDK Flight Recorder at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

  1. Navigate to the Automated Rules tab
    Navigate to the <i>Automated Rules</i> tab
    Switch to the Automated Rules tab.
  2. Click the Create button
  3. Configure the new automated rule
    Configure the new automated rule

    Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

    Description: Enter an optional description for your rule.

    Match Expression: We will fill this field in the next step.

    Enabled: Enable or disable the rule. If disabled, the rule will not be applied to any targets.

    Template: Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

  4. Create your match expression

    The match expression in a rule definition is a Java-like snippet of code that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match expressions should thus evaluate to a boolean value. The simplest match expressions would be the booleans true or false; if we use true, the rule will apply to every target. The expression has a target object in global scope, with the following form in JSON notation:

    {
      "alias": "myAppAlias",
      "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
      "labels": {
        "com.example/service": "customer-login",
      },
      "annotations": {
        "platform": {
          "io.kubernetes/annotation": "annotated"
        },
        "cryostat": {
          "PORT": 9091,
          "HOST": "cryostat",
          "NAMESPACE": "myproject"
        }
      }
    }

    The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The labels and platform annotations may be empty—in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Here are some examples of match expressions:

    target.alias == ’com.example.MainClass’
    
    target.alias == ’myAlias’
    
    target.labels[‘com.example/service’] == ’customer-login’
    
    target.labels[‘com.example/service’] != ’customer-login’
    
    target.annotations.cryostat.PORT > 3000
    
    target.annotations.cryostat.PORT > 3000 && target.annotations.platform[‘io.kubernetes/annotation’] == ‘enabled’
    
    !!target.annotations.platform[‘io.kubernetes/annotation’]
    
    /^customer-login[0-9]\*$/.test(target.alias)
  5. Evaluate your match expression
    Evaluate your match expression

    When you enter a match expression in the Match Expression field, the Match Expression Evaluator will test if your match expression matches the target selected in the dropdown. You can check that your match expression matches all of your desired targets by selecting different targets from the dropdown.

  6. (Optional) Adjust rule parameters
    <i>(Optional)</i> Adjust rule parameters
    Optionally set the recording options and rule parameters.

    Maximum Size: The maximum size of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

    Maximum Age: The maximum age of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

    Archival Period: Time between copies of active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

    Initial Delay: Time between rule creation and when the first archived copy should be transferred. Values less than 1 are treated as being equal to the Archival Period above.

    Preserved Archives: The number of recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

    In the example image, the maximum recording age was set to 300 seconds and the archival period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your flight recorder data is preserved in Cryostat's archives. The initial delay is set to 60 seconds however, so the first archive copy will be made 1 minute after the rule is created. The next copy will be made 5 minutes after that, the next another 5 minute later, etc.

  7. Click the Create button
  8. Observe the new rule in the Automated Rules table
    Observe the new rule in the <i>Automated Rules</i> table
    The new rule will appear in the Automated Rules table.
  9. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  10. Observe the automatically generated recording in the Active Recordings table
    Observe the automatically generated recording in the <i>Active Recordings</i> table
    Switch to the Recordings tab and view the new recording in the Active Recordings table.
    Once you've set up your automated rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications.

Custom monitoring with Kubernetes labels or annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "target.annotations.platform[‘jfrMonitoring’]==’enabled’",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}
To create this rule with the Cryostat UI, follow the guide in Continuous monitoring in a containerized JVM.

Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a recording started automatically using the custom Demo template from our first use case. It will take an archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these archived recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

Upload an Automated Rule

Instead of stepping through the form every time, you can also edit your Automated Rule definition files (JSON format) locally and quickly upload them.

Below is an example of a rule file content:

{
  "name":"my_automated_rule",
  "description":"This is an example automated rule.",
  "matchExpression":"target.alias == 'io.cryostat.Cryostat'",
  "eventSpecifier":"template=Continuous,type=TARGET",
  "archivalPeriodSeconds":285,
  "initialDelaySeconds":60,
  "preservedArchives":10,
  "maxAgeSeconds":300,
  "maxSizeBytes":0,
  "enabled":true
}
  1. Open Automated Rules upload prompt
    Open <i>Automated Rules</i> upload prompt
    Click on upload icon button to open upload prompt.
  2. Attach a rule file by dragging & dropping or clicking Browse....
    Attach a rule file by dragging & dropping or clicking <i>Browse...</i>.
    Select a rule file to upload.
  3. Click Submit button to upload.
  4. Observe the new rule in the Automated Rules table
    Observe the new rule in the <i>Automated Rules</i> table
    The new rule will appear in the Automated Rules table.

Download an Automated Rule

Automated Rules can easily be downloaded for viewing and editing.

  1. Open action menu for a selected rule
    Open action menu for a selected rule
    Click the action overflow three-dot menu on the right side of selected rule entry in the table.
  2. Click Download button to download.

Configure Graphical Notifications

Actions such as starting/stopping/archiving recordings and state changes such as recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active then the notification stream within the web-client graphical console can become quite noisy.

  1. Navigate to Settings
    Navigate to Settings
    Click the cog or gear icon in the application masthead to access the Settings view.

    The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.

  2. Locate the Notifications card
    Locate the Notifications card
    The Notifications card within this view is used to control the graphical display of notifications that correspond to WebSocket messages sent by the Cryostat backend when actions and state changes occur. The card contains a toggle switch for each category of notification, as well as a global switch to enable/disable all categories at once. Disabling a notification category only prevents it from appearing as a pop-up notification toast in the current web-client instance. The notification will still appear in the notification drawer, accessed by clicking the bell icon in the application masthead, and other web-client instances or API clients will still receive the notification to process as they will.