Guides on Using Cryostat

This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.

• Start/Stop a Recording
• Snapshot a Recording
• Archive a Recording
• Viewing Archived Recordings
  • All-Targets Archived Recordings View
  • All-Archives Archived Recordings View
• Download an Active or Archived Recording
• Re-Upload a Recording to Archives
• Download, Edit, and Upload a Customized Event Template
• Edit Template With JMC
• View a Recording in Grafana
• View and Download Automated Analysis for a Recording
• Add and Edit Recording Metadata Labels
• Store JMX Credentials
• Configure JMX Credentials Storage
• Add a Trusted Certificate
• Create an Automated Rule
  • Continuous monitoring in a containerized JVM
  • Custom monitoring with Kubernetes labels or annotations
• Upload an Automated Rule
• Download an Automated Rule
• Configure Graphical Notifications

Start/Stop a Recording

The first and most basic workflow is to start a Flight Recording in a target JVM.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Click Create

   

4. Configure the new recording

   

   Enter a name for the new recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

   Select the duration of the recording. The duration can be specified in seconds, minutes, or hours. If you want to record indefinitely, select the Continuous option. You may also want to save the recording to Cryostat archive storage when the recording stops with the Archive on Stop option.

   Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the Continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

   To learn about Metadata Options, see Add and Edit Recording Metadata Labels.

5. (Optional) Adjust any Advanced Options for the recording

   

   In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:

   To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.

   Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.

   Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the recording before being discarded. This parameter is valid only when the disk parameter is set to true, and the recording duration is not Continuous. By default, this parameter is set to 0, which means there is no limit set.

6. Click the Create button
7. Stop the recording

   

   When you no longer desire for the Flight Recording to be active and collecting data in your target application, select the recording from the list by clicking the checkbox to the left of the recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.

   If the recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the recording was created with a continuous duration then it will collect data until explicitly stopped.

   After stopping a recording it remains in the Active section of the Recordings view. This signifies that the recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.

Snapshot a Recording

Snapshotting an active recording produces a new recording in the target JVM named snapshot-n, where n is a natural number. This snapshot contains all of the JFR data that was present in the target JVM at the time that the snapshot was taken and is in the STOPPED state as soon as it is created (that is, the snapshot will never record any further information and is fixed in size). This is useful when you have an ongoing, continuous recording active in a target application and want to preserve the specific information at a given point in time. If the continuous source recording is configured with a maximum event age or maximum recording size then the need for snapshots is more apparent, since without snapshotting some older data will eventually be dropped from the source recording and lost.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

   If you do not have any recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

3. Click Create

   

4. Switch to the Snapshot Recording tab

   

5. Observe the new snapshot

   

Archive a Recording

In contrast to Active Recordings, which reside within the container of the target application, Archived Recordings reside within persistent storage attached to a Cryostat instance. In OpenShift, for example, the archives are stored in a PersistentVolumeClaim.

Archived recordings are created by performing archival upon active recordings. When this is requested, Cryostat connects to the target application and copies the Flight Recorder data from the selected active recording into an archived recording file in storage. The active source recording may be continuous or fixed-duration, may be using any event template, may be in any state (RUNNING, STOPPED, etc.), and may even be a snapshot.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Select an Active Recording

   

4. Click the Archive button
5. Navigate to the Archived Recordings tab

   

Viewing Archived Recordings

There are several ways to view Archived Recordings. The first method is to navigate to the Archived Recordings tab. See Archive a Recording.

All-Targets Archived Recordings View

The second method is to navigate to the new All-Targets archived recording view within the Archives tab on Cryostat console sidebar.

The All-Targets view gathers all of Cryostat’s discovered target JVM applications into one section for ease of access. Here, we are able to interact with any archived recordings that have been saved from a source target by opening a target’s nested recordings table.

1. Navigate to the All-Targets Archived Recordings view

   

   The option to automatically hide all targets with zero archived recordings is on by default and can be toggled. Targets can also be filtered in the search bar.

2. Select a source target application

   

All-Archives Archived Recordings View

The third method is to navigate to the new All-Archives archived recording view within the Archives tab on Cryostat console sidebar.

The All-Archives view is a view which queries Cryostat’s internal storage for any created archived recordings and the directories that contain them. Here, we are able to interact with any archived recordings that have been saved into storage by opening a nested recordings table within each directory.

This view is used to save any lost archived recordings in case any target JVM restarts or exits. It will be empty if no recordings are currently saved to storage.

1. Navigate to the All-Archives Archived Recordings view

   

   Directories can be filtered in the search bar.

2. Select an archives directory

   

   A directory name is related to its corresponding source target's serviceUrl. Mousing over the tooltip, we can also see a Cryostat generated hash ID for that target.

Download an Active or Archived Recording

Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use JDK Mission Control for the heavy lifting.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Select a recording

   

4. Download the recording

   

5. Choose what to do with the recording file

   

Re-Upload a Recording to Archives

After downloading an archived recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage lost, and you then want to use Cryostat’s analysis tools with a previously retrieved recording in a new Cryostat instance.

1. Navigate to the Archives tab

   

   The entire Archives view contains several tabs related to performing operations on archived recordings. See Viewing Archived Recordings.
2. Select the Recording to Upload

   

3. (Optional) Add metadata labels

   

4. Upload the Recording

   

Download, Edit, and Upload a Customized Event Template

Most JVMs will come with at least two JFR event template definitions included: the Continuous and Profiling templates.

The Continuous template:

• has very low runtime overhead
• collects basic data
• is intended to be left on at all times, even in production

The Profiling template:

• may have some detectable runtime overhead
• collects more in-depth data
• should be used for a fixed duration when a specific problem is discovered at runtime

These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.

Of special note, JFR allows for the definition of application-specific custom events, which would not be captured in either of the default templates above.

Cryostat also provides the ALL meta-template, which enables all event types in the selected target application, with default values for each event option. This is not a true event template and does not have an XML definition to download.

1. Select the target application

   

   The target selected will provide the base Continuous and Profiling template definitions to start from. Most target applications will be interchangeable here.
2. Navigate to Events

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Download the Template

   

4. Edit the Template

   

   Edit the template definition to suit your requirements. When you are satisfied with the new configuration, give your new template a meaningful and recognizable tag by editing the <configuration> element at the top of the file and setting the label= attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. The description and provider attributes can also be used to help identify this template later and to explain its goals and purpose.
5. Open the Template upload dialog

   

6. Upload the Template

   

7. Observe the new Template

   

Edit Template With JMC

JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc event template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.

1. Acquire an Event Template

   

   If you do not have an event template definition then you can visit Download, Edit, and Upload a Customized Event Template and follow the first few steps to retrieve one from a remote target application accessible by Cryostat. Otherwise, if you have the JDK installed locally, you can find templates at $JAVA_HOME/lib/jfr.
2. Launch JDK Mission Control

   

3. Open the Template Manager

   

4. Import the Base Template

   

5. Edit the Template

   

6. Export the Template

   

   Once your edited template is saved to to a file on your local workstation it can be used when starting new Flight Recordings. To do this using Cryostat, see Download, Edit, and Upload a Customized Event Template for instructions on how to upload the new template to Cryostat and Start/Stop a Recording for instructions on how to create a new recording using this template.

View a Recording in Grafana

Cryostat provides integration with Grafana to plot curated time series metrics from a recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator, you must retrieve the credentials generated by the operator in order to log into Grafana. These credentials are stored in a Kubernetes Secret. Once logged in, navigate to the dashboard titled Dashboard. You will see a variety of metrics plotted from your recording.

1. Retrieve the Grafana credentials

   CRYOSTAT_NAME=$(kubectl get cryostat -o jsonpath='{$.items[0].metadata.name}')
   # Username
   kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_USER}' | base64 -d
   # Password
   kubectl get secret ${CRYOSTAT_NAME}-grafana-basic -o jsonpath='{$.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d
           

2. Select the target application

   

3. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

4. Select a recording

   

5. Select View in Grafana...

   

6. Log into Grafana

   

7. View and interact with data from your recording

   

View and Download Automated Analysis for a Recording

Cryostat integrates the same automated analysis reports that you would find in JDK Mission Control. The JMC rules engine analyzes your recording and looks for common problems and assigns a severity score from 0 (no problem) to 100 (potentially severe problem). Results with score 0 are hidden from the report unless you select the Show OK Results checkbox. More details on each result can be found by clicking on the + symbol to the right of the rule name. These details often include suggestions on how to fix the problem. Cryostat also allows you to download the report HTML file for offline use.

1. Select the target application

   

2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Select a recording

   

4. Expand the recording

   

5. View details and suggestions for results

   

6. Download the report

   

   

   

Add and Edit Recording Metadata Labels

Users can attach metadata or custom labels to JDK flight recordings managed by Cryostat. Recording labels can be used to identify recordings during queries or perform actions on multiple recordings containing the same labels. Here’s how to add and edit metadata labels on your JDK flight recordings.

1. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

2. Click Create

   

3. Add metadata labels to the Create Recording form

   

   When creating a custom flight recording with Cryostat, expand the form section Show metadata options. Click Add Label and add key-value label pairs to the form. Finally, click Create to attach the labels and create the recording.
4. View your labels on the Active Recordings Table

   

   The new recording will appear in the Recordings tab with your custom label as well as default labels containing information about the selected recording template.
5. Edit an existing label

   

   Recording labels can also be edited after recordings have been created or re-uploaded to archives. It looks like the custom label in our example contains a typo - we can fix the typo by editing the label. First select a recording from the table with the 🗹 checkbox. Then, click the Edit Labels button to bring up the label drawer. Finally, click the Edit button that appears from the drawer.
   
   
   
   The labels section will appear as a form where you can add, edit, or delete existing labels, just like before when we created the recording. Fix the typo, and click Save to save your edited labels.

6. View your edited labels

   

   The recording labels should be updated in the Active Recordings table.
7. (Optional) Archive your recording to view labels copied onto the archived recording

   

   On the Active Recordings table, click the checkbox next to the recording, then click Archive to archive your recording. Notice that the archived recording also copies the labels from the active recording. You can also add labels to any recording uploaded to Cryostat’s archives.
8. (Optional) Bulk-edit recording labels across multiple recordings

   

   Create another recording on the same target. Then select both recordings on the Recordings Table and click Edit Labels and Edit.This time, only labels that are present on both recordings will be shown in the form. Let's delete the two common template-related labels, and add a new label to both recordings. Then finally, click Save.
   
   
   
   Congratulations, you have successfully bulk-edited labels across multiple recordings!

Store JMX Credentials

If you have Java Management Extensions (JMX) authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your JMX credentials before it can access the JDK flight recordings on your target JVMs. You can Configure JMX Credentials Storage and choose whether these credentials are held in browser memory for the current session only, or if they are uploaded to Cryostat’s JMX credentials keyring.

If you intend to use Cryostat Automated Rules, then you should store target application JMX credentials in Cryostat’s JMX credentials keyring, which is outlined below. Even if you do not use Automated Rules, you may find it more convenient to store credentials in the keyring. This way, the Cryostat web UI does not need to prompt you for credentials when you manually access JFR information about target applications.

1. Navigate to the Security tab

   

   First, navigate to the Security tab. The Store JMX Credentials table lists all credentials in Cryostat's keyring. Click Add to define a new set of credentials.
2. Enter your JMX credentials

   

   A modal will appear. You can select a target JVM to view its properties. These properties can be used to define a matchExpression that tests whether this set of credentials should apply to the selected target JVM. This is the same kind of matchExpression as those used by Automated Rules, so you may find it useful to reuse the same matchExpression from your rule definitions for your credentials definitions. The matchExpression, username, and password fields are all required.
3. (Alternative to Steps 1 and 2) Store JMX Credentials when connecting to a target JVM

   

   Alternatively, JMX credentials may also be stored if you navigate to either the Recordings tab or the Events tab and select a target JVM with JMX authentication enabled. The authentication form will appear, prompting you to enter your JMX credentials. If you have configured Backend JMX Credentials Storage, then a new credentials definition with the provided username and password will be stored for this specific target application in the Cryostat keyring. If you have configured JMX credentials with the Session option then these entered credentials will not be stored in the Cryostat keyring.
4. View your stored JMX credentials

   

   The Store JMX Credentials table will update with rows for each of the credentials you have stored in the keyring. The matchExpression is visible along with a count of the number of known target applications that these credentials will apply to. You can click the reverse chevron icon at the left of the table row to expand the row, which will display a list of those matched target applications. For security reasons it is not possible to view the stored username and password associated with a credentials definition in the keyring.

Configure JMX Credentials Storage

Target JVM applications may require Cryostat to pass an authentication challenge before being able to communicate over JMX and manage JFR. Cryostat has two supported mechanisms for these JMX credentials:

1. JMX Credentials Keyring: see Store JMX Credentials for more detail. This mechanism entails uploading a JMX Credentials definition to the Cryostat backend keyring storage. Cryostat automatically checks the keyring for credentials matching a target application when a request to that application is opened. If no credentials are found, Cryostat responds to the requesting client (e.g. the Cryostat Web UI) with a response indicating the JMX authentication failure.
2. X-JMX-Authorization header passthrough: when the Cryostat Web UI receives a JMX authentication failure response from the Cryostat server, it displays a prompt asking for JMX credentials. Depending on the configuration set in this section, those credentials may either be held locally in your browser session, or they may be uploaded to the Cryostat credentials keyring. If the credentials are held locally in your browser session, then the Cryostat Web UI will automatically attach a custom header with API requests containing the JMX credentials you specify. The Cryostat server sees this header and uses the credentials when opening the JMX connection to the target, but does not persist these credentials. When the JMX credentials passthrough header is present, they supersede credentials stored in the Cryostat server's keyring.

Now that you understand the difference, let’s continue to see how you can configure the Cryostat Web UI to use one or the other when you complete a JMX Authentication prompt.

1. Navigate to Settings

   

   The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.

2. Locate the JMX Credentials Storage card

   

   This card contains a brief explanation of its purpose and a simple dropdown menu with selections for where any JMX Credentials entered into a JMX Authentication Required challenge modal will be stored. Choose Session (Browser Memory) to use the header passthrough mechanism described above, or choose Backend to automatically store the credentials in the Cryostat server keyring.

Add a Trusted Certificate

If you have Java Management Extensions (JMX) over SSL enabled on your containerized JVMs, you must configure Cryostat to trust the SSL certificate presented by the containerized JVM when Cryostat attempts to open a JMX connection. If you do not complete this configuration, Cryostat cannot open a JMX connection for the purposes of performing JFR management tasks.

Here’s how to add a trusted SSL certificate with the Cryostat web UI.

1. Navigate to the Security tab

   

   Click the Security tab.
2. Upload the Certificate

   

   Click the Upload button on the Import SSL Certificates card. This action opens a file-upload dialog, where you can choose the certificate that you want to upload to Cryostat. You can repeat this process multiple times to add multiple trusted certificates.
3. Restart Cryostat to apply the changes. If you do not restart your Cryostat instance, the added certificates are not reloaded. This causes connections to fail because the Cryostat JMX client cannot trust the certificates. Depending on your deployment platform and configuration, restarting Cryostat might require any of the following:
   • On OpenShift/Kubernetes with Cryostat Operator:

     oc delete cryostat/<my-cryostat-cr-name>
     oc create -f <my-cryostat-cr-name>.yaml

   • On OpenShift/Kubernetes without Cryostat Operator:

     oc rollout retry dc/<my-cryostat-dc-name>

   • On Podman:

     podman restart <my-cryostat-container-id>

Create an Automated Rule

Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your flight recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

We’ll walk through two use cases: continuous monitoring in a containerized JVM, and custom monitoring with Kubernetes labels or annotations.

Continuous monitoring in a containerized JVM

Previously, if we wanted to enable always-on continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s automated rules, we can enable JDK Flight Recorder at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

1. Navigate to the Automated Rules tab

   

2. Click the Create button
3. Configure the new automated rule

   

   Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

   Description: Enter an optional description for your rule.

   Match Expression: We will fill this field in the next step.

   Enabled: Enable or disable the rule. If disabled, the rule will not be applied to any targets.

   Template: Select an event template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

4. Create your match expression

   The match expression in a rule definition is a Java-like snippet of code that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match expressions should thus evaluate to a boolean value. The simplest match expressions would be the booleans true or false; if we use true, the rule will apply to every target. The expression has a target object in global scope, with the following form in JSON notation:

   {
     "alias": "myAppAlias",
     "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
     "labels": {
       "com.example/service": "customer-login",
     },
     "annotations": {
       "platform": {
         "io.kubernetes/annotation": "annotated"
       },
       "cryostat": {
         "PORT": 9091,
         "HOST": "cryostat",
         "NAMESPACE": "myproject"
       }
     }
   }

   The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The labels and platform annotations may be empty—in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Here are some examples of match expressions:

   target.alias == ’com.example.MainClass’
   
   target.alias == ’myAlias’
   
   target.labels[‘com.example/service’] == ’customer-login’
   
   target.labels[‘com.example/service’] != ’customer-login’
   
   target.annotations.cryostat.PORT > 3000
   
   target.annotations.cryostat.PORT > 3000 && target.annotations.platform[‘io.kubernetes/annotation’] == ‘enabled’
   
   !!target.annotations.platform[‘io.kubernetes/annotation’]
   
   /^customer-login[0-9]\*$/.test(target.alias)

5. Evaluate your match expression

   

   When you enter a match expression in the Match Expression field, the Match Expression Evaluator will test if your match expression matches the target selected in the dropdown. You can check that your match expression matches all of your desired targets by selecting different targets from the dropdown.

6. (Optional) Adjust rule parameters

   

   Maximum Size: The maximum size of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

   Maximum Age: The maximum age of recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

   Archival Period: Time between copies of active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

   Initial Delay: Time between rule creation and when the first archived copy should be transferred. Values less than 1 are treated as being equal to the Archival Period above.

   Preserved Archives: The number of recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - recordings will be started and remain only in target JVM memory.

   In the example image, the maximum recording age was set to 300 seconds and the archival period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your flight recorder data is preserved in Cryostat's archives. The initial delay is set to 60 seconds however, so the first archive copy will be made 1 minute after the rule is created. The next copy will be made 5 minutes after that, the next another 5 minute later, etc.

7. Click the Create button
8. Observe the new rule in the Automated Rules table

   

9. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

10. Observe the automatically generated recording in the Active Recordings table

    

    Once you've set up your automated rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications.

Custom monitoring with Kubernetes labels or annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "target.annotations.platform[‘jfrMonitoring’]==’enabled’",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}

Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a recording started automatically using the custom Demo template from our first use case. It will take an archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these archived recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

Upload an Automated Rule

Instead of stepping through the form every time, you can also edit your Automated Rule definition files (JSON format) locally and quickly upload them.

Below is an example of a rule file content:

{
  "name":"my_automated_rule",
  "description":"This is an example automated rule.",
  "matchExpression":"target.alias == 'io.cryostat.Cryostat'",
  "eventSpecifier":"template=Continuous,type=TARGET",
  "archivalPeriodSeconds":285,
  "initialDelaySeconds":60,
  "preservedArchives":10,
  "maxAgeSeconds":300,
  "maxSizeBytes":0,
  "enabled":true
}

1. Open Automated Rules upload prompt

   

2. Attach a rule file by dragging & dropping or clicking Browse....

   

3. Click Submit button to upload.
4. Observe the new rule in the Automated Rules table

   

Download an Automated Rule

Automated Rules can easily be downloaded for viewing and editing.

1. Open action menu for a selected rule

   

2. Click Download button to download.

Configure Graphical Notifications

Actions such as starting/stopping/archiving recordings and state changes such as recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active then the notification stream within the web-client graphical console can become quite noisy.

1. Navigate to Settings

   

   The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.

2. Locate the Notifications card

   

   The Notifications card within this view is used to control the graphical display of notifications that correspond to WebSocket messages sent by the Cryostat backend when actions and state changes occur. The card contains a toggle switch for each category of notification, as well as a global switch to enable/disable all categories at once. Disabling a notification category only prevents it from appearing as a pop-up notification toast in the current web-client instance. The notification will still appear in the notification drawer, accessed by clicking the bell icon in the application masthead, and other web-client instances or API clients will still receive the notification to process as they will.