Guides on Using Cryostat

This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.

Getting Help

The Cryostat web application contains online help resources that you can use to learn more about how to use Cryostat directly within its interface.

Welcome Tour

The Welcome Tour is a first-run introduction to Cryostat that you will see the first time you log in. If you clear your browser’s local storage, use a private browsing window, or use a different browser, you will see the Tour again.

Welcome Tour
Welcome Tour
Click through the Welcome Tour to learn how to use Cryostat.
The Welcome Tour provides an introduction to Cryostat's layout and indicates where further tutorials and help can be found.

If you have completed the tour previously, feel free to click Skip Tour and dive right in to Cryostat. If you have not completed the tour previously and are unsure what Cryostat can do or how to use it, click Get Started to begin the guided walkthrough of the Cryostat Web application.

Tour Begins
Tour Begins
Click Next through the Welcome Tour to learn how to use **Cryostat**.
Continue through the tour to learn how the UI is organized and where you can find additional help.

Return to the Tour

You can return to the tour and revisit its contents at any time.

Whenever you feel the need to return to the tour and revisit its contents, simply click the ? Help icon in the application masthead, then click Guided tour in the dropdown menu that appears.

Open Help Menu
Open Help Menu
The ? Help menu on the application masthead contains additional resources.
Click Guided tour to relaunch the same tour.

Quick Starts

Cryostat also includes some Quick Start online tutorials to guide you through some common tasks. These are more in-depth and will directly guide you through the UI as you complete them by operating on your own real applications.

Open Help Menu
Open Help Menu
The ? Help menu on the application masthead contains additional resources.
Click Quick Starts to view the tutorial catalog.
View the Quick Start Catalog
View the Quick Start Catalog
The Quick Start catalog contains a series of tutorials you can follow to learn how to accomplish specific tasks.
Select and complete the tutorials which interest you. You can revisit this catalog at any time and re-do tutorials as many times as you like.

The Dashboard is the first view you will see when you log into Cryostat. It provides a high-level overview of the state of your Cryostat instance and the target JVM applications it is monitoring.

Navigate to Dashboard
Navigate to <i>Dashboard</i>
Add dashboard cards, switch between dashboard layouts, and modify the layout configuration to suit your needs.
The Cryostat Dashboard provides a high-level overview of the state of your Cryostat instance and the target JVM applications it is monitoring.

Dashboard Cards

Dashboard cards are widgets that display information about your Cryostat instance and the target JVM applications it is monitoring. Let’s walk through the available cards and how to add them to your Dashboard.

Open the Card Catalog
Open the <i>Card Catalog</i>
Open the Card Catalog by clicking the Catalog icon on the Dashboard toolbar.
The Dashboard Card Catalog contains a list of available cards that can be added to the Dashboard. Clicking on a card will open a panel containing a preview.

Target JVM Details Card

  1. Add the Target JVM Details Card
    Add the <i>Target JVM Details</i> <code>Card</code>
    Click on the Target JVM Details card for a preview.
    The Target JVM Details card displays information about the target JVM application that is currently selected. There are two tabs that display different information:
    • Details Tab - Displays information about the target JVM application, including:
      1. Connection URL - The JMX connection URL of the target JVM application.
      2. Alias - The alias of the target JVM application.
      3. JVM ID - The JVM ID of the target JVM application.
      4. Labels - The labels of the target JVM application.
      5. Annotations - The annotations of the target JVM application.
      6. and more...
    • Resources Tab - Displays the resources associated with the target JVM application. There are two tables:
  2. Finish Card Creation There are no extra steps in the creation wizard for this card. Click Finish to add the card to your dashboard.
    Details tab
    Details Tab
    Resources tab
    Resources Tab

Automated Analysis Card

  1. Add the Automated Analysis Card
    Add the <i>Automated Analysis</i> <code>Card</code>
    Click on the Automated Analysis card for a preview.

    The Automated Analysis card allows users to view JMC Automated Analysis reports in a nicely formatted dashboard card. The card allows the user to create a special Recording, and then automatically generates an Automated Analysis report. The report displays potential problems with your JVM, and provides suggestions on how to improve the performance and security of your selected JVM application. The card also contains a toolbar that allows you to refresh the report, delete the report, filter results, and change the view.

    Gallery view

    Gallery view

    The Gallery view of the Automated Analysis Card displays a Result, a report summary, for each Rule that was triggered in the selected Recording. In this view, each Rule is listed in categories based on the event type. For example, the Thrown Errors and the Thrown Exceptions Rules are part of the exceptions category, as seen in the figure above. By clicking on each Rule, you can view more details about the Rule and the Result that was generated.

    Automated Analysis Result
    The Discouraged Management Agent Settings Rule result with a severity score of 25.0. A Summary, Explanation, and Solution can be seen in the Result.

    A Result has a severity score from 0 (no problem) to 100 (potentially severe problem).

    The Result will also show three text details, if applicable:

    • Summary: A short summary of the problem.
    • Explanation: A detailed explanation of the problem.
    • Solution: A suggested solution to the problem.

    List View

    List view

    The List view of the Automated Analysis Card displays each Result in a listed table. The table can be sorted by clicking on the column headers. The Result will also show the Summary, Explanation, and Solution in the Description column.

    Toolbar

    The Toolbar allows you to filter results, change the view, refresh the report, and delete the report. You are able to filter:
    1. By severity: You can filter by severity score by dragging the score slider or typing a score in the score input. The Result table will only show results with the score greater or equal to the selected filter score. Additionally, if there are Critical or Warning results, click the corresponding labels in the card header to only show those Results. Reset the filter by clicking on the buttons next to the Reset text (i.e the `0`).
    2. By category: You can filter by Rule Name or Topic by clicking on the Name filter dropdown. Then select a filtered item by clicking the Dropdown next to it. You may also type in this Dropdown to search for a specific item. The Result table will only show results that match the selected filter.
  2. Configure the Automated Analysis Card
    Configure the <i>Automated Analysis</i> <code>Card</code>
    Click Next to optionally provide Advanced Configuration.

    In the next steps of the card creation, you can optionally provide Advanced Configuration. You can configure the settings of the special Recording that is used to generate the report. The Current Configuration will be shown and can be edited by clicking the Pencil icon. By default, the Recording uses a Continuous template, a Maximum size of 10MB, and a 0 second Maximum age (meaning an unlimited recording duration).

    Note: It is possible that setting both an infinite maximum size and age will result in an Out Of Memory error during report generation.

  3. Finish Card Creation
    Finish <code>Card</code> Creation
    The card will be added to the dashboard with an error view.
    After clicking Finish, the card will be added to the dashboard with an error view. This is because the card has not yet detected a special Automated Analysis Recording to source reports from.
  4. Click Create Recording
    Click <i>Create Recording</i>
    The Automated Analysis card displayed with a successful report.
    After clicking Create Recording, the card should be populated with report data containing the Results of the Automated Analysis report.

MBean Metrics Chart Card

  1. Add the MBean Metrics Chart Card
    Add the <i>MBean Metrics Chart</i> <code>Card</code>
    Click on the MBean Metrics Chart card for a preview.

    The MBean Metrics Chart card displays performance metrics about the target JVM through remote access to supported Java MXBeans interfaces of the JVM, including Thread, Runtime, OperatingSystem, and Memory MXBeans.

    Cryostat gathers this data and displays them in various charts. You can customize each card by going through the card creation wizard. The wizard will guide you through the process of selecting the metrics you want to display, how you want to display them, and other various configuration options. Some examples of Performance Metrics that can be displayed are:

    • Process CPU Load
    • System Load Average
    • Heap Memory Usage
    • ...
  2. Configure the MBean Metrics Chart Card
    Configure the <i>MBean Metrics Chart</i> <code>Card</code>
    Click Next to provide card configuration.

    In the next steps of the card creation, you can configure the details of the chart card.

    Configure the metric data by clicking the Performance Metric dropdown and selecting a metric. You can also configure the Data Window to display a specific time range of data, the Refresh Period to control how often the chart is updated, and the Color Theme to change the chart metric color.

  3. Finish Card Creation
    Finish <code>Card</code> Creation
    The MBean Metrics Chart card displayed with the Process CPU Load metric.

    After clicking Finish, the card will be added to the dashboard. You can click the refresh button "↻" on the top right of the card at any time to reload the metrics.

Configuring the Dashboard

The Dashboard is highly customizable and can be configured to display the cards you want to see. You can customize the layout of the cards on the dashboard by Moving, Resizing, and Removing cards.

Moving, Resizing, and Removing cards

  1. Add a Card to the Dashboard
    Add a <code>Card</code> to the <i>Dashboard</i>
    Open the card catalog by clicking the Catalog icon on the Dashboard toolbar.
    Let's add the Target JVM Details card.
  2. Resize the Card
    Resize the <code>Card</code>
    Click and drag the right edge of the card to resize it.
  3. Add another Card to the Dashboard
    Add another <code>Card</code> to the <i>Dashboard</i>
    Open the card catalog by clicking the Catalog icon on the Dashboard toolbar.
    Let's add the MBean Metrics Chart card this time.
  4. Rearrange Cards
    Rearrange <code>Cards</code>
    Click and drag the Target JVM Details card's header on top or to the right of the MBeans Metrics Chart card to swap their positions.
  5. Remove Cards
    Remove <code>Cards</code>
    Click the Kebab icon on the card header to open the card actions menu.

    Each card header contains a kebab icon that opens a menu of card actions. The actions menu contains the following options:

    • View - Opens the card in fullscreen mode.
    • Remove - Removes the card from the dashboard.
    • Reset Size - Resets the card to its default size.

    Click Remove to remove the card.

Dashboard Layouts and Templates

Dashboard Layouts are a way to organize your dashboard cards into different views. You can create multiple layouts and switch between them to view different cards. Favorite, Rename, and Delete layouts to customize your dashboard.

By default, the Default layout is created for you. This layout contains the cards three MBean Metrics Chart cards. You can modify this layout’s card configuration, but you cannot Rename or Delete it.

Layout Templates save your layouts for later use. You can Create a template from a layout, and then use that template to Create a new layout with the same cards. You can also Import and Export templates to share them with other Cryostat users.

Create a new Dashboard Layout

  1. Open the Layout Selector Dropdown
    Open the <i>Layout Selector Dropdown</i>
    Click the layout selector dropdown to view the available layouts.
    The Layout Selector contains a list of all available layouts. The currently selected layout is indicated with a check mark "✓".
  2. Click New Layout
    Click <i>New Layout</i>
    Clicking New Layout will create a new blank layout and switch the dashboard view to the new layout. The layout should be called something like Custom1.
  3. (Optional) Rename Layouts
    <i>(Optional)</i> Rename Layouts
    Click the Pencil button to rename the currently selected layout.
    You are able to rename layouts by clicking the Pencil button next to the layout selector. This will rename the currently selected layout. You can also rename layouts within the layout selector dropdown itself.
  4. (Optional) Delete Layouts
    <i>(Optional)</i> Delete Layouts
    Click 🗑️ Delete to delete the currently selected layout.
    Deletion is similar to renaming. Click the trash can icon with the text Delete next to the layout selector to delete the currently selected layout. You can also delete layouts within the layout selector dropdown itself.

Set a Layout as a Template

  1. Save a layout as a template
    Save a layout as a template
    Click more options "⋮" on the dashboard toolbar, then click Set as template to set the desired layout as a template.
    The layout template will be saved as a User-submitted template in the template picker.

Create a new Dashboard Layout from a Template

  1. Open the Layout Selector Dropdown
    Open the <i>Layout Selector Dropdown</i>
    Click the layout selector dropdown to view the available layouts.
    The Layout Selector contains a list of all available layouts. The currently selected layout is indicated with a check mark "✓".
  2. Select Choose Template
    Select <i>Choose Template</i>
    Click the expandable menu on New Layout button and select Choose Template. This will open the template picker.
  3. Choose a Template
    Choose a Template
    Clicking a template will open a preview where you can view the template's cards.

    The template picker displays all the available templates. Templates are categorized into 3 groups.

    1. Suggested: Templates that are suggested for you based on recent activity.
    2. Cryostat: Templates that come with Cryostat.
    3. User-submitted: Templates that you have created or imported.

    Additionally, you can search for templates by typing in the Search bar or Filter templates by category. You can also upload templates directly here by clicking Upload.

    Select a template by clicking on it.

  4. Enter a Name for the New Layout
    Enter a Name for the New Layout
    A layout name must be entered before the Create button is enabled. The name must be alphanumeric, can only contain underscores, dashes, and periods, and must be 20 characters or less.
  5. Click Create
    Click <i>Create</i>
    The new layout will be created and the dashboard view will switch to the new layout with the template applied.

Use Topology View

The Topology View provides a visual presentation of all the discovered JVM applications, and all their associated resources. It also allows users to perform actions on one or multiple targets.

Navigate to Topology
Navigate to <i>Topology</i>
Use the bottom Control Bar to adjust the Graph View as needed.
When you need to view or perform actions on multiple target JVMs at the same time, Topology View is the way.

View all Target JVMs

    View JVM Applications with Graph View
    View <b>JVM</b> Applications with <i>Graph View</i>
    View JVM applications with Topology Graph view.
    By default, an interactive Graph View of target JVMs (nodes) are shown nested within their associated groups (surrounding lines), for example, Pods, or Realms (i.e. discovery mechanisms to discover Java Applications, such as Kubernetes API, JDP or Cryostat Agent).

    You can drag target nodes/groups or use the bottom control bar to adjust the graph. For example, zoom out or fit all nodes into view. A toolbar is also available to allow further customization:
    1. Display Options: adjust how the nodes and groups are displayed, for example, whether to show connection URL.
    2. Filters: determine which targets or groups to show.
    3. Search bar: find a target using Match Expression. The matched targets will be highlighted.
    View JVM Applications with List View
    View <b>JVM</b> Applications with <i>List View</i>
    View JVM applications with Topology List view.
    Topology View also supports List View mode, where your JVM targets and their groups are shown as expandable rows.

    Click the List icon on the toolbar to switch to List View. Expand each row to see nested groups or targets. All the above features of the toolbar can also be used to customize your view.

View JVM Details and Resources

    View JVM Details
    View <b>JVM</b> Details
    View target JVM’s details with drawer panel.
    In Graph View, select a target JVM node to open the drawer panel that shows its details, for example, Connection URL, Labels and Annotations.

    In List View, expand each row to open nested groups until you find the target. Expand the target to see its details and associated resources.
    View JVM's Associated Resources
    View <b>JVM's</b> Associated Resources
    View JVM's associated resources with drawer panel.
    Navigate to the Resources tab to see the target’s associated resources. There are 2 tables:
    1. Owned Resources: Resources that the JVM owns (i.e. Active Recordings, Archived Recordings, Event Templates and Event Types).
    2. Related Resources: Resources that are tied to the JVM by Match Expression (i.e. Automated Rules and Credentials).
    In the Graph View, each target node also has an indicator that tells whether the target has any running Active Recordings. The same information can be seen within the Owned Resources table by expanding the Active Recordings row.

Perform actions on JVMs

    Perform Actions for a Single JVM
    Perform Actions for a Single <b>JVM</b>
    Perform actions on an individual JVM.
    The details panel for each target JVM supports performing simple actions on the JVM. Click the Actions menu to show available options.

    For example, select View Recordings to be redirected to the Recordings View for the target JVM, where you can view and manage Active Recordings.
    Perform Actions for a Group of JVMs
    Perform Actions for a Group of <b>JVMs</b>
    Perform actions on a group of JVMs.
    The Details panel also supports performing actions on multiple target JVMs. Select a group of targets, for example, a Pod. A drawer panel will appear to show the group details. Select Actions menu to show available options.

    For example, select Start recording to start a Recording on all targets JVMs under this group. If this action fails (for example, due to intermittent network issues) then it is safe to repeat this action again: the action will only start new Recordings or replace STOPPED Recordings on targets under this group, it will not affect RUNNING Recordings.

Create a Custom Target

Cryostat automatically discovers target JVMs using various mechanisms (e.g. Kubernetes API, JDP, Cryostat Agent plugin). However, in some cases it may not be feasible or desirable to configure your application to suit Cryostat’s discovery requirements. In these scenarios you can tell Cryostat about them by filling out the Custom Target form to specify Custom Targets. This can also be used to have Cryostat register itself as a discovered target by using the special value localhost:0, which informs Cryostat’s JVM to use a special JMX connection to itself without going through the network stack.

  1. Navigate to Topology
    Navigate to <i>Topology</i>
    Use the bottom Control Bar to adjust the Graph View as needed.
    When you need to view or perform actions on multiple target JVMs at the same time, Topology View is the way.
  2. Open the Custom Target Form
    Open the <i>Custom Target</i> Form
    Click on Catalog Icon to open the Topology Entity Catalog.

    Search for Custom Target tab and click Create to open the Custom Target form.

    Use the Custom Target form to tell Cryostat about JVM applications that are not automatically discovered.

  3. Enter Custom Target Definition
    Enter <i>Custom Target</i> Definition
    Use the form to enter the Custom Target definition.
    The Connection URL is required for Cryostat to attempt to find the target. You can optionally provide an Alias or JMX credentials if the target has JMX authentication enabled.
  4. Test the Custom Target Definition
    Test the <i>Custom Target</i> Definition
    A ✅ checkmark will show if Cryostat can connect to the sample app.

    Once you enter a valid Connection URL, click on the sample node to test the target connection.

    An exclamation mark and an alert banner will show if an error occurs while connecting to the target. This may occur if the URL is incorrect due to incorrect hostname or port number, or if the port is blocked by a firewall or network policy, or if the JVM listening on the specified port requires JMX credentials that Cryostat does not have in its keyring, or if the JVM listening on the specified port presents an SSL/TLS certificate which Cryostat does not trust.

  5. Click Create Button to Generate a New Custom Target definition once the test is successful.
  6. View the Custom Targets
    View the <i>Custom Targets</i>
    The Custom Target will be available under Custom Targets realm.
    Once the Custom Target form is successfully submitted, you will be redirected to the Topology View. Here, you can view your target under Custom Targets realm.
  7. (Optional) Delete Custom Targets
    <i>(Optional)</i> Delete <i>Custom Targets</i>
    Custom Targets can be cleaned up with Actions menu.
    If the Custom Targets is no longer needed, in the target detail panel, click the Actions menu and select Delete Target.

Using the Cryostat Agent

The Cryostat Agent is an optional component of Cryostat, implemented as a Java Instrumentation Agent, which acts as a plugin for applications running on the JVM. Prior to the Agent, Cryostat always extracted data from the JVM by initiating a connection over JMX. It then fetched the JFR data from an MBean and pulled it over the network back toward the Cryostat server to make it accessible to end users.

The Agent works differently. It is responsible for fetching data from the JVM and sending it back to Cryostat over HTTP. The Agent works by looking for MBean and JFR data within itself and the application it is plugged into. It is also able to communicate back to Cryostat about the application instance the Cryostat Agent is attached to and how to reach it. The Cryostat Agent also pushes its own Java Flight Recorder (JFR) data back to Cryostat by initiating network connections with Cryostat, which may then analyze and save the data to make it accessible to end users.

The Agent may also be configured, using the property cryostat.agent.api.writes-enabled or the corresponding environment variable CRYOSTAT_AGENT_API_WRITES_ENABLED, to allow bi-directional read-write access over HTTP. This enables dynamic Start/Stop/Delete of Flight Recordings as well as on-demand JFR pulls much like what Cryostat does over JMX.

The programming interfaces for Cryostat and its Agent are designed to implement Cryostat’s specific feature set, rather than being generalized and flexible like JMX. The benefit of this is that the security considerations are easier to understand and model, but choosing to use the Cryostat Agent over JMX may also forego the ability to interoperate with other JMX tooling such as JDK Mission Control, visualvm, jconsole, hawtio, etc.

  1. The Cryostat Agent retrieves a wide range of information from those Cryostat applications such as memory usage, CPU utilization, etc.
  2. The Cryostat analyzes these collected data to identify problems that might be affecting the application’s performance.
  3. The Agent is a third-party Java Instrumentation Agent for developers which can be installed on the target JVM program through the command-line arguments or directly attaching to the running JVM instance.
  4. The Agent is foreign code for developers to audit and inspect before including it in their application builds. It is a small amount of code to inspect and likely easier to trust than JMX.
  5. Unlike JMX, the JVM doesn’t come with the Agent included, so developers are required to add the Cryostat Agent to their application builds, then rebuild and deploy the application.
  6. Once the Agent has been installed or attached to the running JVM instance, it can begin collecting data and sending it to Cryostat for analysis. If enabled, the Cryostat server that the Cryostat Agent is registered with may also begin to send remote management requests to dynamically Start, Stop, or Delete Flight Recordings as well as to retrieve JFR and MBean data.

For instructions on how to install the Cryostat Agent into your applications, check the Setup section in Getting Started.

Start/Stop a Recording

The first and most basic workflow is to start a Flight Recording in a target JVM.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  3. Click Create
    Click <i>Create</i>
  4. Configure the new Recording
    Configure the new <code>Recording</code>

    Enter a Name for the new Recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again, or check the Restart if recording already exists checkbox to allow Cryostat to reuse the same recording name for a new recording with new settings.

    Select the Duration of the Recording. The duration can be specified in seconds, minutes, or hours. If you want to record indefinitely, select the Continuous option. You may also want to save the Recording to Cryostat archive storage when the Recording stops with the Archive on Stop option.

    Select an event Template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the Continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed Recording duration. You can also upload a custom Event Template and select it here.

    To learn about Metadata Options, see Add and Edit Recording Metadata Labels.

  5. (Optional) Adjust any Advanced Options for the Recording
    <i>(Optional)</i> Adjust any <i>Advanced Options</i> for the <code>Recording</code>

    In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:

    To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.

    Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the Recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.

    Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the Recording before being discarded. This parameter is valid only when the disk parameter is set to true, and the Recording duration is not Continuous. By default, this parameter is set to 0, which means there is no limit set.

  6. Click the Create button
  7. Stop the Recording
    <i>Stop</i> the <code>Recording</code>

    When you no longer desire for the Flight Recording to be Active and collecting data in your target application, select the Recording from the list by clicking the checkbox to the left of the Recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.

    If the Recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the Recording was created with a Continuous Duration then it will collect data until explicitly stopped.

    After Stopping a Recording it remains in the Active section of the Recordings view. This signifies that the Recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the Recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.

Snapshot a Recording

Snapshotting an application produces a new Flight Recording named snapshot-n, where n is a natural number. This snapshot contains all of the JFR data that was present in the target JVM at the time that the snapshot was taken and is in the STOPPED state as soon as it is created (that is, the snapshot will never record any further information and is fixed in size). This is useful when you have an ongoing, Continuous Recording active in a target application and want to preserve the specific information at a given point in time. If the Continuous source Recording is configured with a maximum event age or maximum recording size then the need for snapshots is more apparent, since without snapshotting, some older data will eventually be dropped and lost from the source Recording. It is also useful when you have multiple concurrent source Recordings running and want an easy way to capture the sum of all of their data at a given point in time.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

    If you do not have any Recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  3. Click Create
    Click <i>Create</i>
  4. Switch to the Snapshot Recording tab
    Switch to the <i>Snapshot Recording</i> tab
    Switch to the Snapshot Recording Tab and Click the Create button.
  5. Observe the New snapshot
    Observe the New <code>snapshot</code>
    After clicking the Create button you will be returned to the Active Recordings view and the new snapshot recording will be present in the table. The snapshot recording can then be treated as any other Active Recording in the target application's JVM memory.

Archive a Recording

In contrast to Active Recordings, which reside within the container of the target application, Archived Recordings reside within persistent storage attached to a Cryostat instance. In OpenShift, for example, the archives are stored in a PersistentVolumeClaim.

Archived Recordings are created by performing archival upon Active Recordings. When this is requested, Cryostat connects to the target application and copies the Flight Recorder data from the selected Active Recording into an Archived Recording file in storage. The Active Source Recording may be Continuous or Fixed-duration, using any Event Template, in any state (RUNNING, STOPPED, etc.), and may even be a snapshot.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  3. Select an Active Recording
    Select an <i>Active Recording</i>

    If you do not have any Recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

  4. Click the Archive button
  5. Navigate to the Archived Recordings tab
    Navigate to the <i>Archived Recordings</i> tab
    Once the Recording has been archived, a new entry will appear in the target JVM's Archived Recordings table. All Recordings that were saved from the current target will be listed here in their own table. Switching to a different target from the dropdown will list only the Recordings archived from that source target. The name of the Archived Recording reflects the address of the target application, the original name of the Active Recording that it was retrieved from, and includes a timestamp indicating when the Archived Recording was created.

Viewing Archived Recordings

There are several ways to view Archived Recordings. The first method is to navigate to the Archived Recordings tab. See Archive a Recording.

All-Targets Archived Recordings View

The second method is to navigate to the All-Targets archived recording view within the Archives tab on Cryostat console sidebar.

The All-Targets view gathers all of Cryostat’s discovered target JVM applications into one section for ease of access. Here, we are able to interact with any Archived Recordings that have been saved from a source target by opening a target’s nested recordings table.

  1. Navigate to the All-Targets Archived Recordings view
    Navigate to the <i>All-Targets</i> Archived Recordings view
    Click on the Archives tab on the sidebar, and the first tab should automatically be selected as the All Targets Archives view.

    The option to automatically hide all targets with zero Archived Recordings is on by default and can be toggled. Targets can also be filtered in the search bar.

  2. Select a Source Target Application
    Select a Source Target Application
    Clicking the dropdown arrow next to a target name will list any Archived Recordings originating from that source target.

All-Archives Archived Recordings View

The third method is to navigate to the All-Archives Archived Recording view within the Archives tab on Cryostat console sidebar.

The All-Archives view is a view which queries Cryostat’s internal storage for any created Archived Recordings and the directories that contain them. Here, we are able to interact with any Archived Recordings that have been saved into storage by opening a nested recordings table within each directory.

This view is used to save any lost Archived Recordings in case any target JVM restarts or exits. It will be empty if no Recordings are currently saved to storage.

  1. Navigate to the All-Archives Archived Recordings View
    Navigate to the <i>All-Archives</i> <code>Archived Recordings</code> View
    Click on the Archives tab on the sidebar, and select the second tab titled All Archives.

    Directories can be filtered in the search bar.

  2. Select an Archives directory
    Select an <i>Archives</i> directory
    Clicking the dropdown arrow next to a directory name will list any Archived Recordings within that directory in the Cryostat storage. Again, we can interact with any Archived Recordings in a similar fashion as before.

    A directory name is related to its corresponding source target's serviceUrl. Mousing over the tooltip, we can also see a Cryostat generated hash ID for that target.

Download an Active or Archived Recording

Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use tools such as JDK Mission Control, Visual VM, binjr, or jfr for the heavy lifting on your workstation.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  3. Select a Recording
    Select a <code>Recording</code>
    Determine the Recording you wish to download to your local workstation. This may be either an Active or Archived Recording. Select the appropriate tab in the Recordings view. No actual action needs to be taken at this step.
  4. Download the Recording
    Download the <code>Recording</code>
    Click the action overflow "︙" three-dot menu on the right side of the recording entry in the table, then click Download Recording.
  5. Choose what to do with the Recording File
    Choose what to do with the <code>Recording</code> File
    Your browser will present you with its standard file-save dialog for both the Archived Recording, and a .metadata.json file containing any Metadata Labels that were attached. It may take some time for these dialogs to appear. Once it does, you can choose to open the Recording file directly in an analysis tool, or to save the Recording to disk for later analysis.

Re-Upload a Recording to Archives

After downloading an Archived Recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage is lost, and you then want to use Cryostat’s analysis tools with a previously retrieved Recording in a new Cryostat instance.

  1. Navigate to the Archives Tab
    Navigate to the <i>Archives</i> Tab
    Select the Uploads tab.
    The entire Archives view contains several tabs related to performing operations on Archived Recordings. See Viewing Archived Recordings.
  2. Select the Recording to Upload
    Select the Recording to Upload
    Click the Upload Icon to bring up the upload prompt. Then click Upload and select the .jfr file to upload. Note that the file must follow the Cryostat Archive Recording naming convention. Mouse over the [?] tooltip on the prompt for more information.
  3. (Optional) Add Metadata Labels
    (Optional) Add <code>Metadata Labels</code>
    Click Show metadata options to add optional Metadata Labels to a Recording. You can either add Labels by clicking the Add Label button, or by uploading a custom .json Labels file from your local storage. A custom Labels file can be downloaded alongside downloading a Recording. To learn how to download a Recording and any associated Labels, see Download an Active or Archived Recording. and for more on Cryostat Metadata Labels, see Add and Edit Recording Metadata Labels.
  4. Upload the Recording
    Upload the <code>Recording</code>
    Click Upload and observe that the Recording is now present in the Uploads tab of the Archives view.

Download, Edit, and Upload a Customized Event Template

Most JVMs will come with at least two JFR Event Templates definitions included: the Continuous and Profiling templates.

The Continuous template:

  • has very low runtime overhead
  • collects basic data
  • is intended to be left on at all times, even in production

The Profiling template:

  • may have some detectable runtime overhead
  • collects more in-depth data
  • should be used for a fixed duration when a specific problem is discovered at runtime

These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.

Of special note, JFR allows for the definition of application-specific custom events, which would not be captured in either of the default templates above.

Cryostat also provides the ALL meta-template, which enables all event types in the selected target application, with default values for each event option. This is not a true Event Template and does not have an XML definition to download.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Events
    Navigate to Events
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.

    If the Target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the Target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

  3. Download the Template
    Download the <i>Template</i>
    Click the action overflow "︙" three-dot menu on the right side of the template entry in the table, then click Download.
  4. Edit the Template
    Edit the <i>Template</i>
    The Template definition is an XML file, so you may open and edit it with your favourite text editor or XML document editor. The JMC Template Manager can also be used for this purpose, as described in Edit Template With JMC .
    Edit the Template definition to suit your requirements. When you are satisfied with the new configuration, give your new Template a meaningful and recognizable tag by editing the <configuration> element at the top of the file and setting the label= attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. The description and provider attributes can also be used to help identify this Template later and to explain its goals and purpose.
  5. Open the Template Upload Dialog
    Open the <i>Template</i> Upload Dialog
    Back on the Cryostat Events View, click the Upload Icon in the table toolbar. A dialog will appear.
  6. Upload the Template
    Upload the <i>Template</i>
    Click the Browse button. Your browser will present its native file chooser dialog to select the file to upload. Browse and select the Template file, then click OK/Select/Continue/Open. Click Submit on the Cryostat dialog.
  7. Observe the new Template
    Observe the new <i>Template</i>
    Once the Template has been uploaded you will be returned to the Events View, and your template will be present in the table.

Edit Template With JMC

JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc Event Template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.

  1. Acquire an Event Template
    Acquire an <i>Event Template</i>
    Ensure that you have an event template definition .jfc file somewhere on your local workstation, with read/write permissions.
    If you do not have an event template definition then you can visit Download, Edit, and Upload a Customized Event Template and follow the first few steps to retrieve one from a remote target application accessible by Cryostat. Otherwise, if you have the JDK installed locally, you can find templates at $JAVA_HOME/lib/jfr.
  2. Launch JDK Mission Control
    Launch <code>JDK Mission Control</code>
    The main JMC window after launch
  3. Open the Template Manager
    Open the <i>Template Manager</i>
    Select Window > Flight Recording Template Manager
  4. Import the Base Template
    Import the Base <i>Template</i>
    Click Import Files..., then browse to and select the Template from Step 1.
  5. Edit the Template
    Edit the <i>Template</i>
    Select the imported Template, then click Edit and make your desired changes. Before making changes you may also click Duplicate to create an identical copy of the Template as a base to iterate upon.
  6. Export the Template
    Export the <i>Template</i>
    Click Export File... to save the event template to a .jfc file on your local workstation.
    Once your edited Template is saved to a file on your local workstation it can be used when starting new Flight Recordings. To do this using Cryostat, see Download, Edit, and Upload a Customized Event Template for instructions on how to upload the new template to Cryostat and Start/Stop a Recording for instructions on how to create a new Recording using this Template.

View a Recording in Grafana

Cryostat provides integration with Grafana to plot curated time series metrics from a Recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your Recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator or Cryostat Helm Chart, your browser should reuse the same login credentials (if any) as for the Cryostat UI itself. If you are greeted by a login page then simply log in again using your same account credentials which you use to log in to Cryostat.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  3. Select a Recording
    Select a <code>Recording</code>
    If you do not have any Recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different Target application. You may also select an Archived Recording to view in Grafana.
  4. Select View in Grafana...
    Select <i>View in Grafana...</i>
    Select View in Grafana... from the Recording's overflow menu.
  5. View and Interact With Data From Your Recording
    View and Interact With Data From Your <code>Recording</code>
    Observe the plotted time series data from curated metrics in your Recording. Select time ranges to zoom into the data.

View Automated Analysis for a Recording

Cryostat integrates the same Automated Analysis reports that you would find in JDK Mission Control (“JMC”). The JMC rules engine analyzes your Recording and looks for common problems, assigning a severity score from 0 (no problem) to 100 (potentially severe problem) to each potential problem type.

Cryostat also provides an Automated Analysis Card that is able to display the same information in a more flexible format, with more tools and control over the data you see and the ability to resize the view. The Card is available for use in the Dashboard. Read the section on the Automated Analysis Card for more information.

  1. Select the Target Application
    Select the <i>Target Application</i>
    Clik the Dropdown arrow on the right side of Target prompt to select or create a target.
    If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
  2. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  3. Select a Recording
    Select a <code>Recording</code>
    If you do not have any Recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application. You may also select an Archived Recording for Automated Analysis.
  4. Expand the Recording
    Expand the <code>Recording</code>
    Expand the Recording with the Chevron to the left of the Recording name. The Automated Analysis report will appear below the Recording.
  5. View Details and Suggestions for Results
    View Details and Suggestions for Results
    Click on each result to view specifics on what the result means and possible suggestions to fix the potential problem.

Add and Edit Recording Metadata Labels

Users can attach Metadata or Custom Labels to JDK Flight Recordings managed by Cryostat. Recording Labels can be used to identify Recordings during queries or perform actions on multiple Recordings containing the same Labels. Here’s how to Add and Edit Metadata Labels on your JDK Flight Recordings.

  1. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  2. Click Create
    Click <i>Create</i>
  3. Add Metadata Labels to the Create Recording form
    Add <code>Metadata Labels</code> to the <i>Create Recording</i> form
    When creating a custom Flight Recording with Cryostat, expand the form section Show metadata options. Click Add Label and add key-Value Label pairs to the form. Finally, click Create to attach the Labels and create the Recording.
  4. View your Labels on the Active Recordings Table
    View your <code>Labels</code> on the <i>Active Recordings</i> Table
    The new Recording will appear in the Recordings Tab with your Custom Label as well as default Labels containing information about the selected Recording template.
  5. Edit an Existing Label
    Edit an Existing <code>Label</code>

    Recording Labels can also be edited after Recordings have been created or re-uploaded to archives. It looks like the Custom Label in our example contains a typo - we can fix the typo by editing the Label. First select a Recording from the table with the 🗹 checkbox. Then, click the Edit Labels button to bring up the label drawer. Finally, click the Edit button that appears from the drawer.



    The Labels section will appear as a form where you can Add, Edit, or Delete existing Labels, just like before when we created the Recording. Fix the typo, and click Save to save your edited Labels.

  6. View your edited Labels
    View your edited <code>Labels</code>
    The Recording Labels should be updated in the Active Recordings table.
  7. (Optional) Archive your Recording to view Labels copied onto the Archived Recording
    <i>(Optional)</i> Archive your <code>Recording</code> to view <code>Labels</code> copied onto the <code>Archived Recording</code>
    On the Active Recordings table, click the Checkbox next to the Recording, then click Archive to archive your Recording. Notice that the Archived Recording also copies the Labels from the Active Recording. You can also add Labels to any Recording uploaded to Cryostat’s archives.
  8. (Optional) Bulk-edit Recording Labels across multiple Recordings
    <i>(Optional)</i> Bulk-edit <code>Recording Labels</code> across multiple <code>Recordings</code>

    Create another Recording on the same target. Then select both Recordings on the Recordings Table and click Edit Labels and start editing. This time, only Labels that are present on both Recordings will be shown in the form. Let's Delete the two common template-related Labels, and Add a new Label to both Recordings. Then finally, click Save.



    Congratulations, you have successfully bulk-edited Labels across multiple Recordings!

Store Credentials

If you have Java Management Extensions (JMX) or HTTP authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your credentials before it can access the JDK Flight Recordings on your target JVMs.

  1. Navigate to the Security tab
    Navigate to the <i>Security</i> tab
    First, navigate to the Security tab. The Store Credentials table lists all credentials in Cryostat's Keyring. Click Add to define a new credential.
  2. Enter your Credentials
    Enter your Credentials
    A modal will appear. You can select a target JVM to view its properties. These properties can be used to define a matchExpression that tests whether this credential should apply to the selected target JVM. This is the same kind of matchExpression as those used by Automated Rules, so you may find it useful to reuse the same matchExpressions from your rule definitions for your credential definitions. The matchExpression, username, and password fields are all required.
  3. Test your Credentials
    Test your Credentials
    Navigate to the Test tab to use the Credential Test table to check if the entered credential is valid for the matched targets.

    You can individually test each target or bulk test all targets at the same time. The Status column will report the following test status:
    1. Valid: valid credential for the target JVM.
    2. Invalid: invalid credential for the target JVM.
    3. Not applicable: JMX authentication is not enabled on the target JVM.
  4. (Alternative to Steps 1 and 2) Store Credentials when Connecting to a Target JVM
    
          (Alternative to Steps 1 and 2) Store Credentials when Connecting to a
          <code>Target</code> <b>JVM</b>
    Alternatively, credentials may also be stored if you navigate to either the Recordings tab or the Events tab and select a target JVM with authentication enabled. The authentication form will appear, prompting you to enter your credentials. A new credential definition with the provided username and password will be stored for this specific target application in the Cryostat Keyring.
  5. View your Stored Credentials
    View your Stored Credentials
    The Store Credentials table will update with a row for each of the credentials you have stored in the Keyring. The matchExpression is visible along with a count of the number of known target applications that these credentials will apply to. You can expand the row to display a list of those matched target applications. For security reasons it is not possible to view the stored username and password associated with a credential definition in the Keyring.

Add a Trusted Certificate

If you have Java Management Extensions (JMX) over SSL enabled on your containerized JVMs, you must configure Cryostat to trust the SSL/TLS certificate presented by the containerized JVM when Cryostat attempts to open a JMX connection. If you do not complete this configuration, Cryostat cannot open a JMX connection for the purposes of performing JFR management tasks.

  1. Navigate to the Security Tab
    Navigate to the <i>Security</i> Tab
    Click the Security tab. This should initially be in an empty state if you have not yet defined any additional trusted certificates.

In order to add a trusted certificate to Cryostat’s custom truststore you must first determine the directory that Cryostat loads certificates from. This is controlled by the configuration property ssl.truststore.dir and defaults to /truststore. If you are deploying Cryostat manually in an environment like Podman or Docker Compose, you should create a volume containing the certificates and mount it to this location, or bind-mount a host directory to this location. If you are using the Cryostat Helm Chart then you should create Secrets containing each trusted certificate and mount each within this location. If you are using the Cryostat Operator, you should use the .Spec.TrustedCertSecrets CR property.

Once you have loaded your additional certificates to the truststore you must restart the Cryostat container, since it can only load certificates into the JVM truststore at startup time. The container may be restarted automatically when you modify the configuration, depending on your deployment platform.

After you have loaded the certificates and verified that the Cryostat container has restarted, you can verify that Cryostat correctly found the certificate(s) within the truststore directory.

  1. Navigate to the Security Tab
    Navigate to the <i>Security</i> Tab
    Click the Security tab. The file paths of any additional trusted certificates you have added should appear in the list.

Create an Automated Rule

Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your Flight Recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

We’ll walk through two use cases: Continuous monitoring in a containerized JVM, and Custom monitoring with Kubernetes labels or annotations.

Continuous Monitoring in a Containerized JVM

Previously, if we wanted to enable always-on Continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s Automated Rules, we can enable JDK Flight Recorder (JFR) at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

  1. Navigate to the Automated Rules Tab
    Navigate to the <i>Automated Rules</i> Tab
    Switch to the Automated Rules tab.
  2. Click the Create Button
  3. Configure the new Automated Rule
    Configure the new <code>Automated Rule</code>

    Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

    Description: Enter an optional description for your rule.

    Match Expression: We will fill this field in the next step.

    Enabled: Enable or disable the rule. If disabled, the rule will not be applied to any targets.

    Template: Select an Event Template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the Continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

  4. Create your Match Expression

    The Match Expression in a rule definition is a Common Expression Language expression that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match Expressions should thus evaluate to a boolean value. The simplest Match Expressions would be the booleans true or false; if we use true, the rule will apply to every target. The Expression has a target object in global scope, with the following form in JSON notation:

    {
      "jvmId": "abcd1234",
      "alias": "myAppAlias",
      "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
      "labels": {
        "com.example/service": "customer-login",
      },
      "annotations": {
        "platform": {
          "io.kubernetes/annotation": "annotated"
        },
        "cryostat": {
          "PORT": 9091,
          "HOST": "cryostat",
          "NAMESPACE": "myproject"
        }
      }
    }

    The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The jvmId is a hash string computed by Cryostat after it successfully connects to a target JVM and is used to uniquely identify that JVM instance - it will be empty if Cryostat has not yet connected to that target (for example, if its SSL/TLS certificate is not trusted or if Cryostat is missing the required credentials) The labels and platform annotations may be empty — in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Take care to use the `has` or `in` operators when dealing with the labels and annotations map structures where specific keys may not exist. Here are some examples of Match Expressions:

    target.alias == 'com.example.MainClass'
    
    target.alias == 'myAlias'
    
    'com.example/service' in target.labels && target.labels[‘com.example/service’] == 'customer-login'
    
    'com.example/service' in target.labels && target.labels[‘com.example/service’] != 'customer-login'
    
    has(target.annotations.cryostat.PORT) && target.annotations.cryostat.PORT > 3000
    
    has(target.annotations.cryostat.PORT) && 'io.kubernetes/annotation' in target.annotations.platform && target.annotations.cryostat.PORT > 3000 && target.annotations.platform['io.kubernetes/annotation'] == 'enabled'
    
    !('io.kubernetes/annotation' in target.annotations.platform)
    
    target.alias.matches("^customer-login[0-9]\*$")
  5. Check your Match Expression
    Check your <code>Match Expression</code>
    You can select a target JVM to view its properties and use them to build your Match Expression.

    When you enter a Match Expression in the Match Expression field, the Match Expression Visualizer will highlight which targets are matched.

  6. (Optional) Adjust Rule Parameters
    <i>(Optional)</i> Adjust <i>Rule Parameters</i>
    Optionally set the Recording Options and Rule Parameters.

    Maximum Size: The maximum size of Recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

    Maximum Age: The maximum age of Recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

    Archival Period: Time between copies of Active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - Recordings will be started and remain only in target JVM memory.

    Initial Delay: Time between rule creation and when the first archived copy should be transferred. Values less than 1 are treated as being equal to the Archival Period above.

    Preserved Archives: The number of Recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - Recordings will be started and remain only in target JVM memory.

    In the example image, the Maximum Recording age was set to 300 seconds and the Archival Period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your Flight Recorder data is preserved in Cryostat's archives. The initial delay is set to 60 seconds however, so the first archive copy will be made 1 minute after the rule is created. The next copy will be made 5 minutes after that, the next another 5 minute later, etc.

  7. Click the Create Button
  8. Observe the new Rule in the Automated Rules Table
    Observe the new Rule in the <i>Automated Rules</i> Table
    The new rule will appear in the Automated Rules table.
  9. Navigate to Recordings
    Navigate to <i>Recordings</i>
    Supply JMX credentials to authenticate to the target, if necessary. If the target is not configured with JMX authentication then the connection attempt will continue without prompting for credentials.
    If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
  10. Observe the automatically generated Recording in the Active Recordings Table
    Observe the automatically generated <code>Recording</code> in the <i>Active Recordings</i> Table
    Switch to the Recordings tab and view the new Recording in the Active Recordings table.
    Once you've set up your Automated Rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications.

Custom Monitoring with Kubernetes Labels or Annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "'jfrMonitoring' in target.annotations.platform && target.annotations.platform['jfrMonitoring']=='enabled'",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}
To create this rule with the Cryostat UI, follow the guide in Continuous Monitoring in a Containerized JVM.

Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a Recording started automatically using the Custom Demo template from our first use case. It will take an Archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these Archived Recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

Upload an Automated Rule

Instead of stepping through the form every time, you can also edit your Automated Rule definition files (JSON format) locally and quickly upload them.

Below is an example of a rule file content:

{
  "name":"my_automated_rule",
  "description":"This is an example automated rule.",
  "matchExpression":"target.alias == 'io.cryostat.Cryostat'",
  "eventSpecifier":"template=Continuous,type=TARGET",
  "archivalPeriodSeconds":285,
  "initialDelaySeconds":60,
  "preservedArchives":10,
  "maxAgeSeconds":300,
  "maxSizeBytes":0,
  "enabled":true
}
  1. Open Automated Rules upload Prompt
    Open <i>Automated Rules</i> upload Prompt
    Click on Upload Icon button to open upload prompt.
  2. Attach a Rule File by Dragging & Dropping or clicking Browse....
    Attach a Rule File by Dragging & Dropping or clicking <i>Browse...</i>.
    Select a rule file to upload.
  3. Click Submit Button to Upload.
  4. Observe the new rule in the Automated Rules table
    Observe the new rule in the <i>Automated Rules</i> table
    The new rule will appear in the Automated Rules table.

Download an Automated Rule

Automated Rules can easily be downloaded for viewing and editing.

  1. Open Action Menu for a selected Rule
    Open Action Menu for a selected Rule
    Click the action overflow "︙" three-dot menu on the right side of selected rule entry in the table.
  2. Click Download Button to Download.

Configure Graphical Notifications

Actions such as starting/stopping/archiving Recordings and state changes such as Recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active, the notification stream within the web-client graphical console can become quite noisy.

  1. Navigate to Settings
    Navigate to <i>Settings</i>
    Click the cog or gear icon in the application masthead to access the Settings view.
    The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.
  2. Locate the Notifications Setting
    Locate the <i>Notifications</i> Setting
    The Notifications setting within this view is used to control the graphical display of notifications that correspond to WebSocket messages sent by the Cryostat backend when actions and state changes occur. The setting contains a toggle switch for each category of notification, as well as a global switch All Notifications , to enable/disable all categories at once. Disabling a notification category only prevents it from appearing as a pop-up notification toast in the current web-client instance. The notification will still appear in the notification drawer, accessed by clicking the bell icon in the application masthead, and other web-client instances or API clients will still receive the notification to process as they will.

Using Smart Triggers

Cryostat Agent supports custom triggers that are based on MBean metric values. You can configure Cryostat Agent to start JFR Recordings dynamically when these custom trigger conditions are met.

You can set up a custom trigger condition in Cryostat to initiate Java Flight Recorder (JFR) Recordings dynamically. This trigger condition is based on MBean counters, covering various runtime, memory, thread, and OS metrics. The condition includes MBean counter types and allows specifying a duration for the trigger to activate only if the specified values persist for that duration. Cryostat’s Agent supports smart triggers, which continuously monitor MBean counter values. When the current values match the configured conditions for the specified duration, the trigger initiates a JFR Recording dynamically through the Cryostat Agent.

Note: A JFR Recording will not start dynamically if the custom trigger condition associated with this Recording is not met.

Configure Custom Trigger for Dynamic Recording

When you configure your target application to load the Cryostat Agent, you can define one or more custom triggers that are then passed as arguments to the Agent. For more information, learn how to use the Cryostat Agent.

Options for Defining a Custom Trigger

You can define a custom trigger in any of the following ways:

  1. Appending a Custom Trigger to the Cryostat Agent’s JAR File Path

    The following example shows how to append a simple custom trigger to the Cryostat Agent’s JAR file path:

      JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent-shaded.jar=[ProcessCpuLoad > 0.2 ; TargetDuration > duration("30s")]~profile"
          

    The preceding example trigger instructs the Agent to start a JFR Recording if the ProcessCpuLoad metric has a value greater than 0.2 for a duration of more than 30 seconds: This example also instructs the Agent to use the Profile event template for the JFR Recording.

  2. Using a JVM System Property Flag

    The following example shows how to specify a simple custom trigger by using a JVM system property flag:

      -Dcryostat.agent.smart-trigger.definitions="[ProcessCpuLoad > 0.2 ; TargetDuration > duration(\"30s\")]~profile"
          

    This example uses the same custom trigger criteria as the preceding example.

  3. Using an Environment Variable

    The following example shows how to specify a simple custom trigger by using an environment variable:

      - name: CRYOSTAT_AGENT_SMART_TRIGGER_DEFINITIONS
        value: "[ProcessCpuLoad > 0.2 ; TargetDuration > duration(\"30s\")]~profile"
          

    This example uses the same custom trigger criteria as the preceding examples.

General Syntax Rules for Custom Triggers

Consider the following syntax guidelines for defining custom triggers:

  1. A custom trigger definition must consist of both an expression that defines the overall trigger condition and the name of an event template that is used for the JFR Recording.
  2. The entire trigger expression must be enclosed in square brackets. For example:
    [ProcessCpuLoad > 0.2 ; TargetDuration < duration("30s")]
  3. The name of the event template must be defined after the trigger expression and preceded by a tilde (~) character. For example:
    ~profile
  4. A trigger expression can consist of one or more constraints and a target duration. The set of constraints and target duration must be separated by a semicolon (;) character.
  5. Each constraint must include: the name of an MBean counter; a relational operator such as > (greater than), = (equal to), <(less than), and so on; and a specified value. The type of relational operator and value that you can specify depends on the associated MBean counter type. For example:
    ProcessCpuLoad > 0.2
  6. Constraints can be grouped together by using logical operators such as && (AND), || (OR), or ! (NOT) logic. For readability and clarity around the order of operations and operator precedence, grouped constraints may be enclosed in round brackets, but this is not a requirement. For example:
      [(MetricA > value1 && MetricB < value2) || MetricC == 'stringvalue' ; TargetDuration > duration("30s")]
        
  7. The name of each MBean counter that is specified as part of a custom trigger must follow precise syntax rules in terms of spelling and capitalization.
  8. Only one target duration can be defined for a custom trigger. The target duration is applied to the entire trigger expression that is enclosed within the square brackets
  9. A target duration can be expressed in terms of seconds, minutes, or hours. For example, 30s means 30 seconds, 5m means five minutes, 2h means two hours, and so on.
  10. A target duration is optional. If a target duration is not specified, triggering will occur immediately once the trigger conditions are met.
  11. Multiple custom trigger definitions can be specified together, each of which relates to a separate JFR Recording. Different custom trigger definitions must be separated by a comma (,) character. For example:
      [ProcessCpuLoad>0.2]~profile,[ThreadCount>30]~Continuous