Guides on Using Cryostat

This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.

Start/Stop a Recording

The first and most basic workflow is to start a Flight Recording in a target JVM.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Click Create
    Click <i>Create</i>
  4. Configure the new recording
    Configure the new recording

    Enter a name for the new recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

    Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

  5. (Optional) Adjust any Advanced Options for the recording
    <i>(Optional)</i> Adjust any <i>Advanced Options</i> for the recording

    In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:

    To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.

    Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.

    Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the recording before being discarded. This parameter is valid only when the disk parameter is set to true. By default, this parameter is set to 0, which means there is no limit set.

  6. Click the Create button
  7. Stop the recording
    Stop the recording

    When you no longer desire for the Flight Recording to be active and collecting data in your target application, select the recording from the list by clicking the checkbox to the left of the recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.

    If the recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the recording was created with a continuous duration then it will collect data until explicitly stopped.

    After stopping a recording it remains in the Active section of the Recordings view. This signifies that the recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.

Snapshot a Recording

Snapshotting an active recording produces a new recording in the target JVM named snapshot-n, where n is a natural number. This snapshot contains all of the JFR data that was present in the target JVM at the time that the snapshot was taken and is in the STOPPED state as soon as it is created (that is, the snapshot will never record any further information and is fixed in size). This is useful when you have an ongoing, continuous recording active in a target application and want to preserve the specific information at a given point in time. If the continuous source recording is configured with a maximum event age or maximum recording size then the need for snapshots is more apparent, since without snapshotting some older data will eventually be dropped from the source recording and lost.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  3. Click Create
    Click <i>Create</i>
  4. Switch to the Snapshot Recording tab
    Switch to the <i>Snapshot Recording</i> tab
    Switch to the Snapshot Recording tab and click the Create button.
  5. Observe the new snapshot
    Observe the new snapshot
    After clicking the Create button you will be returned to the Active Recordings view and the new snapshot recording will be present in the table. The snapshot recording can then be treated as any other active recording in the target application's JVM memory.

Archive a Recording

In contrast to Active Recordings, which reside within the container of the target application, Archived Recordings reside within persistent storage attached to a Cryostat instance. In OpenShift, for example, the archives are stored in a PersistentVolumeClaim.

Archived recordings are created by performing archival upon active recordings. When this is requested, Cryostat connects to the target application and copies the Flight Recorder data from the selected active recording into an archived recording file in storage. The active source recording may be continuous or fixed-duration, using any event template or event specifier string, may be in any state (RUNNING, STOPPED, etc.), and may even be a snapshot.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Select an active recording
    Select an active recording

    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  4. Click the Archive button
  5. Navigate to the Archived Recordings tab
    Navigate to the <i>Archived Recordings</i> tab
    Once the recording has been archived, a new entry will appear in the Archived Recordings table. All recordings in the archives are displayed here, regardless of the selected target application. The name of the archived recording reflects the address of the target application and the original name of the active recording that it was retrieved from, and includes a timestamp indicating when the archived recording was created.

Download an Active or Archived Recording

Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use JDK Mission Control for the heavy lifting.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Select a recording
    Select a recording
    Determine the recording you wish to download to your local workstation. This may be either an Active or Archived recording. Select the appropriate tab in the Recordings view. No actual action needs to be taken at this step.
  4. Download the recording
    Download the recording
    Click the action overflow three-dot menu on the right side of the recording entry in the table, then click Download Recording.
  5. Choose what to do with the recording file
    Choose what to do with the recording file
    Your browser will present you with its standard file-save dialog. It may take some time for this to appear. Once it does, you can choose to open the file directly in JDK Mission Control, or (recommended) to save the file to disk.

Download, Edit, and Upload a Customized Event Template

Most JVMs will come with at least two JFR event template definitions included: the Continuous and Profiling templates.

The Continuous template:

  • has very low runtime overhead
  • collects basic data
  • is intended to be left on at all times, even in production

The Profiling template:

  • may have some detectable runtime overhead
  • collects more in-depth data
  • should be used for a fixed duration when a specific problem is discovered at runtime

These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.

Of special note, JFR allows for the definition of application-specific custom events, which would not be captured in either of the default templates above.

Cryostat also provides the ALL meta-template, which enables all event types in the selected target application, with default values for each event option. This is not a true event template and does not have an XML definition to download.

  1. Select the target application
    Select the target application
    The target selected will provide the base Continuous and Profiling template definitions to start from. Most target applications will be interchangeable here.
  2. Navigate to Events
    Navigate to Events
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Download the template
    Download the template
    Click the action overflow three-dot menu on the right side of the template entry in the table, then click Download.
  4. Edit the template
    Edit the template
    The template definition is an XML file, so you may open and edit it with your favourite text editor or XML document editor. The JMC Template Manager can also be used for this purpose, as described in this section .
    Edit the template definition to suit your requirements. When you are satisfied with the new configuration, give your new template a meaningful and recognizable tag by editing the <configuration> element at the top of the file and setting the label= attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. The description and provider attributes can also be used to help identify this template later and to explain its goals and purpose.
  5. Open the template upload dialog
    Open the template upload dialog
    Back on the Cryostat Events view, click the plus (+) icon in the table toolbar. A dialog will appear.
  6. Upload the template
    Upload the template
    Click the Browse button. Your browser will present its native file chooser dialog to select the file to upload. Browse and select the template file, then click OK/Select/Continue/Open. Click Submit on the Cryostat dialog.
  7. Observe the new template
    Observe the new template
    Once the template has been uploaded you will be returned to the Events view, and your template will be present in the table.

Edit Template With JMC

JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc event template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.

  1. Acquire an Event Template
    Acquire an Event Template
    Ensure that you have an event template definition .jfc file somewhere on your local workstation, with read/write permissions.
    If you do not have an event template definition then you can visit Download, Edit, and Upload a Customized Event Template and follow the first few steps to retrieve one from a remote target application accessible by Cryostat. Otherwise, if you have the JDK installed locally, you can find templates at $JAVA_HOME/lib/jfr.
  2. Launch JDK Mission Control
    Launch JDK Mission Control
    The main JMC window after launch
  3. Open the Template Manager
    Open the Template Manager
    Select Window > Flight Recording Template Manager
  4. Import the Base Template
    Import the Base Template
    Click Import Files..., then browse to and select the template from Step 1.
  5. Edit the Template
    Edit the Template
    Select the imported template, then click Edit and make your desired changes. Before making changes you may also click Duplicate to create an identical copy of the template as a base to iterate upon.
  6. Export the Template
    Export the Template
    Click Export File... to save the event template to a .jfc file on your local workstation.
    Once your edited template is saved to to a file on your local workstation it can be used when starting new Flight Recordings. To do this using Cryostat, see Download, Edit, and Upload a Customized Event Template for instructions on how to upload the new template to Cryostat and Start/Stop a Recording for instructions on how to create a new recording using this template.

Re-Upload a Recording to Archives

After downloading an archived recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage lost, and you then want to use Cryostat’s analysis tools with a previously retrieved recording in a new Cryostat instance.

  1. Select an application
    Select an application
    Although the archived storage is global to the Cryostat instance and is not target application specific, the view for Archived Recordings is only visible when a target application is selected.
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Select the Archived Recordings tab
    Select the Archived Recordings tab
  4. Select the Recording to Upload
    Select the Recording to Upload
    Click the '+' button, then click Browse... and select the .jfr file to upload. Then click 'Submit'.
  5. Upload the Recording
    Upload the Recording
    Click 'Upload' and observe that the recording is now present in the Archived Recordings tab.

View a Recording in Grafana

Cryostat provides integration with Grafana to plot curated time series metrics from a recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator, you must retrieve the credentials generated by the operator in order to log into Grafana. These credentials are stored in a Kubernetes Secret. Once logged in, navigate to the dashboard titled Dashboard. You will see a variety of metrics plotted from your recording.

  1. Retrieve the Grafana credentials
    CRYOSTAT_NAME=$(kubectl get cryostat -o jsonpath='{$.items[0].metadata.name}')
    # Username
    kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_USER}' | base64 -d
    # Password
    kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d
            
    If you installed Cryostat into Kubernetes or OpenShift using the Cryostat Operator, use kubectl or oc to get the generated username and password for Grafana and save them for later.
  2. Select the target application
    Select the target application
  3. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  4. Choose an active or archived recording
    Choose an active or archived recording

    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  5. Select View in Grafana...
    Select <i>View in Grafana...</i>
    Select View in Grafana... from the recording's overflow menu.
  6. Log into Grafana
    Log into Grafana
    Use the credentials retrieved in step 1 to log into the Grafana web client.
  7. Select Cryostat's dashboard
    Select Cryostat's dashboard
    Navigate to the Dashboards tab in Grafana and select the dashboard named Dashboard.
  8. View and interact with data from your recording
    View and interact with data from your recording
    Observe the plotted time series data from curated metrics in your recording. Select time ranges to zoom into the data.

View and Download Automated Analysis for a Recording

Cryostat integrates the same automated analysis reports that you would find in JDK Mission Control. The JMC rules engine analyzes your recording and looks for common problems and assigns a severity score from 0 (no problem) to 100 (potentially severe problem). Results with score 0 are hidden from the report unless you select the Show OK Results checkbox. More details on each result can be found by clicking on the + symbol to the right of the rule name. These details often include suggestions on how to fix the problem. Cryostat also allows you to download the report HTML file for offline use.

  1. Select the target application
    Select the target application
  2. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Choose an active or archived recording
    Choose an active or archived recording

    If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  4. Expand the recording
    Expand the recording
    Expand the recording with the chevron to the left of the recording name. The automated analysis report will appear below the recording.
  5. View details and suggestions for results
    View details and suggestions for results
    Click the + button on the right side of each result to view specifics on what the result means and possible suggestions to fix the potential problem.
  6. Select Download Report
    Select <i>Download Report</i>
    To download the automated analysis report for offline viewing, select Download Report from the recording's overflow menu.
  7. Open the downloaded report in your browser
    Open the downloaded report in your browser
    View the report on its own without connecting to Cryostat. Check Show OK Results to include results where the rules engine found no issues in the recording.

Add and Edit Recording Metadata Labels

Users can attach metadata or custom labels to JDK flight recordings managed by Cryostat. Recording labels can be used to identify recordings during queries or perform actions on multiple recordings containing the same labels. Here’s how to add and edit metadata labels on your JDK flight recordings.

  1. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  2. Click Create
    Click <i>Create</i>
  3. Add a label on the Custom Flight Recording form
    Add a label on the Custom Flight Recording form
    When creating a custom flight recording with Cryostat, expand the form section Show metadata options. Click Add Label to add a key-value label pair to the recording.
  4. View your labels on the Active Recordings Table
    View your labels on the Active Recordings Table
    The new recording will appear in the Recordings tab with your custom label as well as default labels containing information about the selected recording template.
  5. Edit an existing label
    Edit an existing label
    Recording labels can also be edited after recordings have been created or re-uploaded to archives. It looks like the custom label in our example contains a typo - we can fix the typo by editing the label. Click the ellipsis menu beside the recording table entry, then click Edit Metadata. The labels section will appear as a form where you can add, edit, or delete existing labels. Click Save to save your edited labels.
  6. View your edited labels
    View your edited labels
    The recording labels should be updated in the Active Recordings table.
  7. (Optional) Archive your recording to view labels copied onto the archived recording
    (Optional) Archive your recording to view labels copied onto the archived recording
    On the Active Recordings table, click the checkbox next to the recording, then click Archive to archive your recording. Notice that the archived recording also copies the labels from the active recording. You can also add labels to any recording uploaded to Cryostat’s archives.

Store JMX Credentials

If you have Java Management Extensions (JMX) authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your JMX credentials before it can access the JDK flight recordings on your target JVMs. When monitoring multiple target JVMs with Cryostat features such as Automatic Rules, you may want Cryostat to remember and reuse your JMX credentials for each target connection. JMX credentials are automatically stored when you navigate to the Recordings tab and select a target JVM with JMX authentication enabled.

Cryostat stores JMX credentials according to each target’s unique JMX service URL, also known as a connection URL. Even if the underlying JVM instance changes, the target alias changes, or the target application restarts, Cryostat will apply the stored JMX credentials to the connection URL that the credentials are associated with.

Here’s how to manage stored JMX credentials with the Cryostat web UI.

  1. Navigate to the Security tab
    Navigate to the Security tab
    First, navigate to the Security tab. The Stored Credentials table lists all targets for which Cryostat has stored JMX credentials. Click Add to enter JMX credentials for your desired target JVM.
  2. Enter your JMX credentials
    Enter your JMX credentials
    A modal will appear, prompting you to select a target JVM and enter your JMX credentials.
  3. (Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM
    (Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM
    Alternatively, JMX credentials will also be stored if you navigate to either the Recordings tab or the Events tab and select a target JVM with JMX authentication enabled. The authentication form will appear, prompting you to enter your JMX credentials. The credentials will be automatically stored and will appear in the Stored JMX Credentials table.
  4. View your stored JMX credentials
    View your stored JMX credentials
    The target name will appear on the Stored Credentials table to indicate that Cryostat has stored JMX credentials for that target. As a security precaution, you will not be able to view the actual credentials after you have submitted them. To remove any stored credentials, select the checkbox next to the target and click Delete.

Add a Trusted Certificate

If you have Java Management Extensions (JMX) over SSL enabled on your containerized JVMs, you must configure Cryostat to trust the SSL certificate presented by the containerized JVM when Cryostat attempts to open a JMX connection. If you do not complete this configuration, Cryostat cannot open a JMX connection for the purposes of performing JFR management tasks.

Here’s how to add a trusted SSL certificate with the Cryostat web UI.

  1. Navigate to the Security tab
    Navigate to the Security tab
    Click the Security tab.
  2. Upload the Certificate
    Upload the Certificate
    Click the Upload button on the Import SSL Certificates card. This action opens a file-upload dialog, where you can choose the certificate that you want to upload to Cryostat. You can repeat this process multiple times to add multiple trusted certificates.
  3. Restart Cryostat to apply the changes. If you do not restart your Cryostat instance, the added certificates are not reloaded. This causes connections to fail because the Cryostat JMX client cannot trust the certificates. Depending on your deployment platform and configuration, restarting Cryostat might require any of the following:
    • On OpenShift/Kubernetes with Cryostat Operator:
      oc delete cryostat/<my-cryostat-cr-name>
      oc create -f <my-cryostat-cr-name>.yaml
    • On OpenShift/Kubernetes without Cryostat Operator:
      oc rollout retry dc/<my-cryostat-dc-name>
    • On Podman:
      podman restart <my-cryostat-container-id>

Create an Automated Rule

Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your flight recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

We’ll walk through two use cases: continuous monitoring in a containerized JVM, and custom monitoring with Kubernetes labels or annotations.

Continuous monitoring in a containerized JVM

Previously, if we wanted to enable always-on continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s automated rules, we can enable JDK Flight Recorder at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

  1. Navigate to the Automated Rules tab
    Navigate to the <i>Automated Rules</i> tab
    Switch to the Automated Rules tab.
  2. Click the Create button
  3. Configure the new automated rule
    Configure the new automated rule

    Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

    Description: Enter an optional description for your rule.

    Match Expression: We will fill this field in the next step.

    Template: Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

  4. Create your match expression

    The match expression in a rule definition is a Java-like snippet of code that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match expressions should thus evaluate to a boolean value. The simplest match expressions would be the booleans true or false; if we use true, the rule will apply to every target. The expression has a target object in global scope, with the following form in JSON notation:

    {
      "alias": "myAppAlias",
      "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
      "labels": {
        "com.example/service": "customer-login",
      },
      "annotations": {
        "platform": {
          "io.kubernetes/annotation": "annotated"
        },
        "cryostat": {
          "PORT": 9091,
          "HOST": "cryostat",
          "NAMESPACE": "myproject"
        }
      }
    }

    The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The labels and platform annotations may be empty—in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Here are some examples of match expressions:

    target.alias == ’com.example.MainClass’
    
    target.alias == ’myAlias’
    
    target.labels[‘com.example/service’] == ’customer-login’
    
    target.labels[‘com.example/service’] != ’customer-login’
    
    target.annotations.cryostat.PORT > 3000
    
    target.annotations.cryostat.PORT > 3000 && target.annotations.platform[‘io.kubernetes/annotation’] == ‘enabled’
    
    !!target.annotations.platform[‘io.kubernetes/annotation’]
    
    /^customer-login[0-9]\*$/.test(target.alias)
  5. Evaluate your match expression
    Evaluate your match expression

    When you enter a match expression in the Match Expression field, the Match Expression Evaluator will test if your match expression matches the target selected in the dropdown. You can check that your match expression matches all of your desired targets by selecting different targets from the dropdown.

  6. (Optional) Adjust recording options
    <i>(Optional)</i> Adjust recording options
    Optionally set the recording options.

    Maximum Size: The maximum size of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

    Maximum Age: The maximum age of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

    Archival Period: Time between copies of active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

    Preserved Archives: The number of recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

    In the example image, the maximum recording age was set to 300 seconds and the archival period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your flight recorder data is preserved in Cryostat's archives.

  7. Click the Create button
  8. Observe the new rule in the Automated Rules table
    Observe the new rule in the <i>Automated Rules</i> table
    The new rule will appear in the Automated Rules table.
  9. Navigate to Recordings
    Navigate to Recordings
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  10. Observe the automatically generated recording in the Active Recordings table
    Observe the automatically generated recording in the <i>Active Recordings</i> table
    Switch to the Recordings tab and view the new recording in the Active Recordings table.
    Once you've set up your automated rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications.

Custom monitoring with Kubernetes labels or annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "target.annotations.platform[‘jfrMonitoring’]==’enabled’",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}
To create this rule with the Cryostat UI, follow the guide in Continuous monitoring in a containerized JVM.

Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a recording started automatically using the custom Demo template from our first use case. It will take an archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these archived recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

Upload an Automated Rule

Instead of stepping through the form every time, you can also edit your Automated Rule definition files (JSON format) locally and quickly upload them.

Below is an example of a rule file content:

{
  "name":"My Rule",
  "description":"",
  "matchExpression":"target.alias == 'io.cryostat.Cryostat' || target.annotations.cryostat['PORT'] == 9091","eventSpecifier":"template=Profiling,type=TARGET",
  "archivalPeriodSeconds":300,
  "preservedArchives":12,
  "maxAgeSeconds":1000,
  "maxSizeBytes":1000
}
  1. Open Automated Rules upload prompt
    Open <i>Automated Rules</i> upload prompt
    Click on upload icon button to open upload prompt.
  2. Attach a rule file by dragging & dropping or clicking Browse....
    Attach a rule file by dragging & dropping or clicking <i>Browse...</i>.
    Select a rule file to upload.
  3. Click Submit button to upload.
  4. Observe the new rule in the Automated Rules table
    Observe the new rule in the <i>Automated Rules</i> table
    The new rule will appear in the Automated Rules table.

Download an Automated Rule

Automated Rules can easily be downloaded for viewing and editting.

  1. Open action menu for a selected rule
    Open action menu for a selected rule
    Click the action overflow three-dot menu on the right side of selected rule entry in the table.
  2. Click Download button to download.

Configure Graphical Notifications

Actions such as starting/stopping/archiving recordings and state changes such as recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active then the notification stream within the web-client graphical console can become quite noisy.

  1. Navigate to Settings
    Navigate to Settings
    Click the cog or gear icon in the application masthead to access the Settings view.

    The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.

  2. Locate the Notifications card
    Locate the Notifications card
    The Notifications card within this view is used to control the graphical display of notifications that correspond to WebSocket messages sent by the Cryostat backend when actions and state changes occur. The card contains a toggle switch for each category of notification, as well as a global switch to enable/disable all categories at once. Disabling a notification category only prevents it from appearing as a pop-up notification toast in the current web-client instance. The notification will still appear in the notification drawer, accessed by clicking the bell icon in the application masthead, and other web-client instances or API clients will still receive the notification to process as they will.