Guides on Using Cryostat

This section explains some common Cryostat features by example, illustrating common actions and workflows of interest and why they are useful.

• Getting Help
  • Welcome Tour
    • Return to the Tour
  • Quick Starts
  • Configure Feature Level
• Navigate the Dashboard
  • Dashboard Cards
    • Target JVM Details Card
    • Automated Analysis Card
    • MBean Metrics Chart Card
    • JFR Metrics Chart Card
    • Diagnostic Actions Card
  • Configuring the Dashboard
    • Moving, Resizing, and Removing cards
  • Dashboard Layouts and Templates
    • Create a new Dashboard Layout
    • Set a Layout as a Template
    • Create a new Dashboard Layout from a Template
• Perform Garbage Collection
• Use Topology View
  • View all Target JVMs
  • View JVM Details and Resources
  • Perform actions on JVMs
• Create a Custom Target
• Using the Cryostat Agent
• Configure the Agent Harvester
  • Mbean Trigger Integration
• Using Smart Triggers
  • Configure Custom Trigger for Dynamic Recording
  • Options for Defining a Custom Trigger
  • General Syntax Rules for Custom Triggers
• Start/Stop a Recording
• Snapshot a Recording
• Archive a Recording
• Viewing Archived Recordings
  • All-Targets Archived Recordings View
  • All-Archives Archived Recordings View
• Download an Active or Archived Recording
• Re-Upload a Recording to Archives
• View JFR Event Types
• Download, Edit, and Upload a Customized Event Template
• Edit Template With JMC
• View a Recording in Grafana
• View Automated Analysis for a Recording
• Add and Edit Recording Metadata Labels
• Store Credentials
• Add a Trusted Certificate
• Create an Automated Rule
  • Continuous Monitoring in a Containerized JVM
  • Custom Monitoring with Kubernetes Labels or Annotations
• Upload an Automated Rule
• Download an Automated Rule
• Configure Graphical Notifications

Getting Help

The Cryostat web application contains online help resources that you can use to learn more about how to use Cryostat directly within its interface.

Welcome Tour

The Welcome Tour is a first-run introduction to Cryostat that you will see the first time you log in. If you clear your browser’s local storage, use a private browsing window, or use a different browser, you will see the Tour again.

Welcome Tour

The Welcome Tour provides an introduction to Cryostat's layout and indicates where further tutorials and help can be found.

If you have completed the tour previously, feel free to click Skip Tour and dive right in to Cryostat. If you have not completed the tour previously and are unsure what Cryostat can do or how to use it, click Get Started to begin the guided walkthrough of the Cryostat Web application.

Tour Begins

Continue through the tour to learn how the UI is organized and where you can find additional help.

Return to the Tour

You can return to the tour and revisit its contents at any time.

Whenever you feel the need to return to the tour and revisit its contents, simply click the ? Help icon in the application masthead, then click Guided tour in the dropdown menu that appears.

Open Help Menu

Click Guided tour to relaunch the same tour.

Quick Starts

Cryostat also includes some Quick Start online tutorials to guide you through some common tasks. These are more in-depth and will directly guide you through the UI as you complete them by operating on your own real applications.

Open Help Menu

Click Quick Starts to view the tutorial catalog.

View the Quick Start Catalog

Select and complete the tutorials which interest you. You can revisit this catalog at any time and re-do tutorials as many times as you like.

Configure Feature Level

Some features in the web-client UI are Beta-level features. This indicates that they are still underdoing design and development, and may have significant limitations, or be redesigned or even removed in the future.

For those reasons, Beta features are hidden by default in the Cryostat UI. They can be enabled by following the steps below.

1. Navigate to Settings

   

   The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.
2. Locate the Advanced settings

   

   The Advanced tab within this view contains a control to set the Feature Level of the UI. This is set to Production by default. You can enable additional features by setting this to Beta, with the aforementioned caveats in mind. Once this is set, a Beta badge will appear on the Cryostat application titlebar. Additional features enabled by this setting, such as Dashboard cards, will be labelled with a similar badge to indicate the feature level. If you set the feature level back to Production then any Beta-level features will be hidden from the UI again.

Navigate the Dashboard

The Dashboard is the first view you will see when you log into Cryostat. It provides a high-level overview of the state of your Cryostat instance and the target JVM applications it is monitoring.

Navigate to Dashboard

The Cryostat Dashboard provides a high-level overview of the state of your Cryostat instance and the target JVM applications it is monitoring.

Dashboard Cards

Dashboard cards are widgets that display information about your Cryostat instance and the target JVM applications it is monitoring, or allow you to perform diagnostic actions against the targets. Let’s walk through the available cards and how to add them to your Dashboard.

Open the Card Catalog

The Dashboard Card Catalog contains a list of available cards that can be added to the Dashboard. Clicking on a card will open a panel containing a preview.

Target JVM Details Card

1. Add the Target JVM Details Card

   

   The Target JVM Details card displays information about the target JVM application that is currently selected. There are two tabs that display different information:

   • Details Tab - Displays information about the target JVM application, including:
     1. Connection URL - The JMX connection URL of the target JVM application.
     2. Alias - The alias of the target JVM application.
     3. JVM ID - The JVM ID of the target JVM application.
     4. Labels - The labels of the target JVM application.
     5. Annotations - The annotations of the target JVM application.
     and more...
   • Resources Tab - Displays the resources associated with the target JVM application. There are two tables:
     • Owned Resources: Resources that the JVM owns (i.e. Active Recordings, Archived Recordings, Event Templates and Event Types).
     • Related Resources: Resources that are tied to the JVM by Match Expression (i.e. Automated Rules and Credentials).
2. Finish Card Creation

   There are no extra steps in the creation wizard for this card. Click Finish to add the card to your dashboard.

   

   

Automated Analysis Card

1. Add the Automated Analysis Card

   

   The Automated Analysis card allows users to view JMC Automated Analysis reports in a nicely formatted dashboard card. The card allows the user to create a special Recording, and then automatically generates an Automated Analysis report. The report displays potential problems with your JVM, and provides suggestions on how to improve the performance and security of your selected JVM application. The card also contains a toolbar that allows you to refresh the report, delete the report, filter results, and change the view.

   Gallery view

   

   The Gallery view of the Automated Analysis Card displays a Result, a report summary, for each Rule that was triggered in the selected Recording. In this view, each Rule is listed in categories based on the event type. For example, the Thrown Errors and the Thrown Exceptions Rules are part of the exceptions category, as seen in the figure above. By clicking on each Rule, you can view more details about the Rule and the Result that was generated.

   

   A Result has a severity score from 0 (no problem) to 100 (potentially severe problem).

   The Result will also show three text details, if applicable:

   • Summary: A short summary of the problem.
   • Explanation: A detailed explanation of the problem.
   • Solution: A suggested solution to the problem.

   List View

   

   The List view of the Automated Analysis Card displays each Result in a listed table. The table can be sorted by clicking on the column headers. The Result will also show the Summary, Explanation, and Solution in the Description column.

   Toolbar

   The Toolbar allows you to filter results, change the view, refresh the report, and delete the report. You are able to filter:

   1. By severity: You can filter by severity score by dragging the score slider or typing a score in the score input. The Result table will only show results with the score greater or equal to the selected filter score. Additionally, if there are Critical or Warning results, click the corresponding labels in the card header to only show those Results. Reset the filter by clicking on the buttons next to the Reset text (i.e the `0`).
   2. By category: You can filter by Rule Name or Topic by clicking on the Name filter dropdown. Then select a filtered item by clicking the Dropdown next to it. You may also type in this Dropdown to search for a specific item. The Result table will only show results that match the selected filter.
2. Configure the Automated Analysis Card

   

   In the next steps of the card creation, you can optionally provide Advanced Configuration. You can configure the settings of the special Recording that is used to generate the report. The Current Configuration will be shown and can be edited by clicking the Pencil icon. By default, the Recording uses a Continuous template, a Maximum size of 10MB, and a 0 second Maximum age (meaning an unlimited recording duration).

   Note: It is possible that setting both an infinite maximum size and age will result in an Out Of Memory error during report generation.

3. Finish Card Creation

   

   After clicking Finish, the card will be added to the dashboard with an error view. This is because the card has not yet detected a special Automated Analysis Recording to source reports from.
4. Click Create Recording

   

   After clicking Create Recording, the card should be populated with report data containing the Results of the Automated Analysis report.

MBean Metrics Chart Card

1. Add the MBean Metrics Chart Card

   

   The MBean Metrics Chart card displays performance metrics about the target JVM through remote access to supported Java MXBeans interfaces of the JVM, including Thread, Runtime, OperatingSystem, and Memory MXBeans.

   Cryostat gathers this data and displays them in various charts. You can customize each card by going through the card creation wizard. The wizard will guide you through the process of selecting the metrics you want to display, how you want to display them, and other various configuration options. Some examples of Performance Metrics that can be displayed are:

   • Process CPU Load
   • System Load Average
   • Heap Memory Usage
   • ...
2. Configure the MBean Metrics Chart Card

   

   In the next steps of the card creation, you can configure the details of the chart card.

   Configure the metric data by clicking the Performance Metric dropdown and selecting a metric. You can also configure the Data Window to display a specific time range of data, the Refresh Period to control how often the chart is updated, and the Color Theme to change the chart metric color.

3. Finish Card Creation

   

   After clicking Finish, the card will be added to the dashboard. You can click the refresh button "↻" on the top right of the card at any time to reload the metrics.

JFR Metrics Chart Card

1. Add the JFR Metrics Chart Card

   

   The JFR Metrics Chart card displays performance metrics about the target JVM by visualizing JFR data snapshots via embedded Grafana visualization panels as Dashboard cards. This Beta-level feature. A significant limitation of this card is that it depends upon the stateful jfr-datasource backend component, which only converts one Flight Recording file at a time to Grafana data. Therefore, this card does not behave well if multiple web-client instances are open at the same time, whether used by one user or multiple human users.

   Cryostat gathers typical JFR data from the selected Targetand periodically updates the Grafana visualizations. You can customize each card by going through the card creation wizard. The wizard will guide you through the process of selecting the metrics you want to display, how you want to display them, and other various configuration options. Some examples of Performance Metrics that can be displayed are:

   • CPU Load
   • Memory Usage
   • Heap Usage
   • Network Utilization
   • File I/O
   • Exception Statistics
   • ...
2. Configure the JFR Metrics Chart Card

   

   In the next steps of the card creation, you can configure the details of the chart card.

   Configure the metric data by clicking the Performance Metric dropdown and selecting a metric. You can also configure the Data Window to display a specific time range of data and the Refresh Period to control how often the chart is updated.

3. Finish Card Creation

   

   After clicking Finish, the card will be added to the dashboard. Initially the card will have no source Recording and display no data.

4. Start source Flight Recording

   

   Click Create on the card to create a source Recording. Cryostat will walk you through creating the Recording. You can simply click through the form and accept the suggested default settings. This will begin a Flight Recording on the selected Target and send you to the Recordings view. Once you return to the Dashboard and the recording is available then Cryostat will begin to process and update the Recording to update the card visualization.

Diagnostic Actions Card

1. Add the Diagnostics Card

   

   The Diagnostics card allows you to perform non-metrics diagnostic actions against the selected Target. This Beta-level feature

   The following diagnostic actions are available:

   • Request JVM to perform Garbage Collection

2. Finish Card Creation

   

   After clicking Finish, the card will be added to the dashboard.

Configuring the Dashboard

The Dashboard is highly customizable and can be configured to display the cards you want to see. You can customize the layout of the cards on the dashboard by Moving, Resizing, and Removing cards.

Moving, Resizing, and Removing cards

1. Add a Card to the Dashboard

   

   Let's add the Target JVM Details card.
2. Resize the Card

   

3. Add another Card to the Dashboard

   

   Let's add the MBean Metrics Chart card this time.
4. Rearrange Cards

   

5. Remove Cards

   

   Each card header contains a kebab icon that opens a menu of card actions. The actions menu contains the following options:

   • View - Opens the card in fullscreen mode.
   • Remove - Removes the card from the dashboard.
   • Reset Size - Resets the card to its default size.

   

Dashboard Layouts and Templates

Dashboard Layouts are a way to organize your dashboard cards into different views. You can create multiple layouts and switch between them to view different cards. Favorite, Rename, and Delete layouts to customize your dashboard.

By default, the Default layout is created for you. This layout contains the cards three MBean Metrics Chart cards. You can modify this layout’s card configuration, but you cannot Rename or Delete it.

Layout Templates save your layouts for later use. You can Create a template from a layout, and then use that template to Create a new layout with the same cards. You can also Import and Export templates to share them with other Cryostat users.

Create a new Dashboard Layout

1. Open the Layout Selector Dropdown

   

   The Layout Selector contains a list of all available layouts. The currently selected layout is indicated with a check mark "✓".
2. Click New Layout

   

   Clicking New Layout will create a new blank layout and switch the dashboard view to the new layout. The layout should be called something like Custom1.
3. (Optional) Rename Layouts

   

   You are able to rename layouts by clicking the Pencil button next to the layout selector. This will rename the currently selected layout. You can also rename layouts within the layout selector dropdown itself.
4. (Optional) Delete Layouts

   

   Deletion is similar to renaming. Click the trash can icon with the text Delete next to the layout selector to delete the currently selected layout. You can also delete layouts within the layout selector dropdown itself.

Set a Layout as a Template

1. Save a layout as a template

   

   The layout template will be saved as a User-submitted template in the template picker.

Create a new Dashboard Layout from a Template

1. Open the Layout Selector Dropdown

   

   The Layout Selector contains a list of all available layouts. The currently selected layout is indicated with a check mark "✓".
2. Select Choose Template

   

   Click the expandable menu on New Layout button and select Choose Template. This will open the template picker.
3. Choose a Template

   

   The template picker displays all the available templates. Templates are categorized into 3 groups.

   1. Suggested: Templates that are suggested for you based on recent activity.
   2. Cryostat: Templates that come with Cryostat.
   3. User-submitted: Templates that you have created or imported.

   Additionally, you can search for templates by typing in the Search bar or Filter templates by category. You can also upload templates directly here by clicking Upload.

   Select a template by clicking on it.

4. Enter a Name for the New Layout

   

   A layout name must be entered before the Create button is enabled. The name must be alphanumeric, can only contain underscores, dashes, and periods, and must be 20 characters or less.
5. Click Create

   

Perform Garbage Collection

1. Add the Diagnostics Card

   

   The Diagnostics card allows you to perform diagnostic operations on a target JVM through remote access to supported Java MXBeans.

   Currently Cryostat supports invoking garbage collection on target JVMs.

2. Finish Card Creation

   

   After clicking Finish, the card will be added to the dashboard. You can click the Start Garbage Collection button in the middle of the card at any time to trigger a garbage collection cycle.

Use Topology View

The Topology View provides a visual presentation of all the discovered JVM applications, and all their associated resources. It also allows users to perform actions on one or multiple targets.

Navigate to Topology

When you need to view or perform actions on multiple target JVMs at the same time, Topology View is the way.

View all Target JVMs

View JVM Applications with Graph View

By default, an interactive Graph View of target JVMs (nodes) are shown nested within their associated groups (surrounding lines), for example, Pods, or Realms (i.e. discovery mechanisms to discover Java Applications, such as Kubernetes API, JDP or Cryostat Agent).

You can drag target nodes/groups or use the bottom control bar to adjust the graph. For example, zoom out or fit all nodes into view. A toolbar is also available to allow further customization:
1. Display Options: adjust how the nodes and groups are displayed, for example, whether to show connection URL.
2. Filters: determine which targets or groups to show.
3. Search bar: find a target using Match Expression. The matched targets will be highlighted.
View JVM Applications with List View

Topology View also supports List View mode, where your JVM targets and their groups are shown as expandable rows.

Click the List icon on the toolbar to switch to List View. Expand each row to see nested groups or targets. All the above features of the toolbar can also be used to customize your view.

View JVM Details and Resources

View JVM Details

In Graph View, select a target JVM node to open the drawer panel that shows its details, for example, Connection URL, Labels and Annotations.

In List View, expand each row to open nested groups until you find the target. Expand the target to see its details and associated resources.
View JVM's Associated Resources

Navigate to the Resources tab to see the target’s associated resources. There are 2 tables:
1. Owned Resources: Resources that the JVM owns (i.e. Active Recordings, Archived Recordings, Event Templates and Event Types).
2. Related Resources: Resources that are tied to the JVM by Match Expression (i.e. Automated Rules and Credentials).
In the Graph View, each target node also has an indicator that tells whether the target has any running Active Recordings. The same information can be seen within the Owned Resources table by expanding the Active Recordings row.

Perform actions on JVMs

Perform Actions for a Single JVM

The details panel for each target JVM supports performing simple actions on the JVM. Click the Actions menu to show available options.

For example, select View Recordings to be redirected to the Recordings View for the target JVM, where you can view and manage Active Recordings.
Perform Actions for a Group of JVMs

The Details panel also supports performing actions on multiple target JVMs. Select a group of targets, for example, a Pod. A drawer panel will appear to show the group details. Select Actions menu to show available options.

For example, select Start recording to start a Recording on all targets JVMs under this group. If this action fails (for example, due to intermittent network issues) then it is safe to repeat this action again: the action will only start new Recordings or replace STOPPED Recordings on targets under this group, it will not affect RUNNING Recordings.

Create a Custom Target

Cryostat automatically discovers target JVMs using various mechanisms (e.g. Kubernetes API, JDP, Cryostat Agent plugin). However, in some cases it may not be feasible or desirable to configure your application to suit Cryostat’s discovery requirements. In these scenarios you can tell Cryostat about them by filling out the Custom Target form to specify Custom Targets. This can also be used to have Cryostat register itself as a discovered target by using the special value localhost:0, which informs Cryostat’s JVM to use a special JMX connection to itself without going through the network stack.

1. Navigate to Topology

   

   When you need to view or perform actions on multiple target JVMs at the same time, Topology View is the way.
2. Open the Custom Target Form

   

   Search for Custom Target tab and click Create to open the Custom Target form.

   

3. Enter Custom Target Definition

   

   The Connection URL is required for Cryostat to attempt to find the target. You can optionally provide an Alias or JMX credentials if the target has JMX authentication enabled.
4. Test the Custom Target Definition

   

   Once you enter a valid Connection URL, click on the sample node to test the target connection.

   

5. Click Create Button to Generate a New Custom Target definition once the test is successful.
6. View the Custom Targets

   

   Once the Custom Target form is successfully submitted, you will be redirected to the Topology View. Here, you can view your target under Custom Targets realm.
7. (Optional) Delete Custom Targets

   

   If the Custom Targets is no longer needed, in the target detail panel, click the Actions menu and select Delete Target.

Using the Cryostat Agent

The Cryostat Agent is an optional component of Cryostat, implemented as a Java Instrumentation Agent, which acts as a plugin for applications running on the JVM. Prior to the Agent, Cryostat always extracted data from the JVM by initiating a connection over JMX. It then fetched the JFR data from an MBean and pulled it over the network back toward the Cryostat server to make it accessible to end users.

The Agent works differently. It is responsible for fetching data from the JVM and sending it back to Cryostat over HTTP. The Agent works by looking for MBean and JFR data within itself and the application it is plugged into. It is also able to communicate back to Cryostat about the application instance the Cryostat Agent is attached to and how to reach it. The Cryostat Agent also pushes its own Java Flight Recorder (JFR) data back to Cryostat by initiating network connections with Cryostat, which may then analyze and save the data to make it accessible to end users.

The Agent may also be configured, using the property cryostat.agent.api.writes-enabled or the corresponding environment variable CRYOSTAT_AGENT_API_WRITES_ENABLED, to allow bi-directional read-write access over HTTP. This enables dynamic Start/Stop/Delete of Flight Recordings as well as on-demand JFR pulls much like what Cryostat does over JMX.

The programming interfaces for Cryostat and its Agent are designed to implement Cryostat’s specific feature set, rather than being generalized and flexible like JMX. The benefit of this is that the security considerations are easier to understand and model, but choosing to use the Cryostat Agent over JMX may also forego the ability to interoperate with other JMX tooling such as JDK Mission Control, visualvm, jconsole, hawtio, etc.

1. The Cryostat Agent retrieves a wide range of information from those Cryostat applications such as memory usage, CPU utilization, etc.
2. The Cryostat analyzes these collected data to identify problems that might be affecting the application’s performance.
3. The Agent is a third-party Java Instrumentation Agent for developers which can be installed on the target JVM program through the command-line arguments or directly attaching to the running JVM instance.
4. The Agent is foreign code for developers to audit and inspect before including it in their application builds. It is a small amount of code to inspect and likely easier to trust than JMX.
5. Unlike JMX, the JVM doesn’t come with the Agent included, so developers are required to add the Cryostat Agent to their application builds, then rebuild and deploy the application.
6. Once the Agent has been installed or attached to the running JVM instance, it can begin collecting data and sending it to Cryostat for analysis. If enabled, the Cryostat server that the Cryostat Agent is registered with may also begin to send remote management requests to dynamically Start, Stop, or Delete Flight Recordings as well as to retrieve JFR and MBean data.

For instructions on how to install the Cryostat Agent into your applications, check the Setup section in Getting Started.

Configure the Agent Harvester

The Cryostat Agent contains a Harvester feature that allows you to start a new recording with a given event template on agent initialization, and periodically push the collected JFR data to the associated Cryostat server. The Agent will also attempt to push the tail end of this recording on JVM shutdown so that the cause of an unexpected JVM shutdown can be captured for later analysis.

The Harvester supports a number of configuration options that can be used to determine how often it pushes the collected JFR data, the template to be used and limitations on how much data to collect, as well as how long the upload may take. The following configuration options are available:

• cryostat.agent.harvester.period-ms [long]: the length of time between JFR collections and pushes by the harvester. This also controls the maximum age of data stored in the buffer for the harvester’s managed Flight Recording. Every period-ms the harvester will upload a JFR binary file to the cryostat.agent.baseuri archives. Default -1, which indicates no scheduled harvest uploading will be performed.
• cryostat.agent.harvester.template [String]: the name of the .jfc event template configuration to use for the harvester’s managed Flight Recording. For example, if the application image contains /usr/lib/jvm/java-17-openjdk/lib/jfr/default.jfc then this value would be simply be default. This can also be the value of the label attribute of the root element within that file, for example Continuous. Defaults to the empty string, so that no recording is started.
• cryostat.agent.harvester.max-files [String]: the maximum number of pushed files that Cryostat will keep from the agent. This is included with the harvester’s push requests and instructs Cryostat to prune, in a FIFO manner, the oldest JFR files within the attached JVM target’s storage, while the number of stored recordings is greater than this configuration’s maximum file limit. Default 2147483647 (Integer.MAX_VALUE).
• cryostat.agent.harvester.upload.timeout-ms [long]: the duration in milliseconds to wait for HTTP upload requests to the Cryostat server to complete and respond. Default 30000.
• cryostat.agent.harvester.exit.max-age-ms [long]: the JFR maxage setting, specified in milliseconds, to apply to recording data uploaded to the Cryostat server when the JVM this Agent instance is attached to exits. This ensures that tail-end data is captured between the last periodic push and the application exit. Exit uploads only occur when the application receives SIGINT/SIGTERM from the operating system or container platform.
• cryostat.agent.harvester.exit.max-size-b [long]: the JFR maxsize setting, specified in bytes, to apply to exit uploads as described above.
• cryostat.agent.harvester.max-age-ms [long]: the JFR maxage setting, specified in milliseconds, to apply to periodic uploads during the application lifecycle. Defaults to 0, which is interpreted as 1.5x the harvester period (cryostat.agent.harvester.period-ms).
• cryostat.agent.harvester.max-size-b [long]: the JFR maxsize setting, specified in bytes, to apply to periodic uploads during the application lifecycle. Defaults to 0, which means unlimited.

Note that the Harvester Period and Template options must be set for the Harvester to regularly push JFR data. If only the Template is set the Harvester will only attempt to push on shutdown. If neither are set the Harvester will not do anything unless configured alongside Smart Triggers as described below.

These configuration options may be set either as JVM system properties, for example:

-Dcryostat.agent.harvester.period-ms=1000
-Dcryostat.agent.harvester.template=Profiling

or by setting them as environment variables, for example:

- name: CRYOSTAT_AGENT_HARVESTER_PERIOD_MS
  value: 1000
- name: CRYOSTAT_AGENT_HARVESTER_TEMPLATE  
  value: Profiling

Mbean Trigger Integration

When the Cryostat Agent is configured to start dynamic recordings based on custom MBean triggers, you can also integrate them with the Harvester to automatically push the collected JFR data to the Cryostat Server.

By defining MBean custom triggers and an agent harvester period without a harvester template, you can achieve a setup where the agent does both of the following:

• Agent dynamically starts JFR recordings based on MBean custom triggers.
• Agent uses configured harvester periods to periodically capture snapshots of the recording data and upload this data to the Cryostat server.

In this situation, the agent will continue to capture recording data until you manually stop the dynamic JFR recording or the host JVM shuts down.

For instructions on how to install the Cryostat Agent into your applications, check the Setup section in Getting Started.

Using Smart Triggers

Cryostat Agent supports custom triggers that are based on MBean metric values. You can configure Cryostat Agent to start JFR Recordings dynamically when these custom trigger conditions are met.

You can set up a custom trigger condition in Cryostat to initiate Java Flight Recorder (JFR) Recordings dynamically. This trigger condition is based on MBean counters, covering various runtime, memory, thread, and OS metrics. The condition includes MBean counter types and allows specifying a duration for the trigger to activate only if the specified values persist for that duration. Cryostat’s Agent supports smart triggers, which continuously monitor MBean counter values. When the current values match the configured conditions for the specified duration, the trigger initiates a JFR Recording dynamically through the Cryostat Agent.

Note: A JFR Recording will not start dynamically if the custom trigger condition associated with this Recording is not met.

Configure Custom Trigger for Dynamic Recording

When you configure your target application to load the Cryostat Agent, you can define one or more custom triggers that are then passed as arguments to the Agent. For more information, learn how to use the Cryostat Agent.

Options for Defining a Custom Trigger

You can define a custom trigger in any of the following ways:

1. Appending a Custom Trigger to the Cryostat Agent’s JAR File Path

   The following example shows how to append a simple custom trigger to the Cryostat Agent’s JAR file path:

     JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent-shaded.jar=[ProcessCpuLoad > 0.2 ; TargetDuration > duration("30s")]~profile"
         

   The preceding example trigger instructs the Agent to start a JFR Recording if the ProcessCpuLoad metric has a value greater than 0.2 for a duration of more than 30 seconds: This example also instructs the Agent to use the Profile event template for the JFR Recording.

2. Using a JVM System Property Flag

   The following example shows how to specify a simple custom trigger by using a JVM system property flag:

     -Dcryostat.agent.smart-trigger.definitions="[ProcessCpuLoad > 0.2 ; TargetDuration > duration(\"30s\")]~profile"
         

   This example uses the same custom trigger criteria as the preceding example.

3. Using an Environment Variable

   The following example shows how to specify a simple custom trigger by using an environment variable:

     - name: CRYOSTAT_AGENT_SMART_TRIGGER_DEFINITIONS
       value: "[ProcessCpuLoad > 0.2 ; TargetDuration > duration(\"30s\")]~profile"
         

   This example uses the same custom trigger criteria as the preceding examples.

General Syntax Rules for Custom Triggers

Consider the following syntax guidelines for defining custom triggers:

1. A custom trigger definition must consist of both an expression that defines the overall trigger condition and the name of an event template that is used for the JFR Recording.
2. The entire trigger expression must be enclosed in square brackets. For example:

   [ProcessCpuLoad > 0.2 ; TargetDuration < duration("30s")]

3. The name of the event template must be defined after the trigger expression and preceded by a tilde (~) character. For example:

   ~profile

4. A trigger expression can consist of one or more constraints and a target duration. The set of constraints and target duration must be separated by a semicolon (;) character.
5. Each constraint must include: the name of an MBean counter; a relational operator such as > (greater than), = (equal to), <(less than), and so on; and a specified value. The type of relational operator and value that you can specify depends on the associated MBean counter type. For example:

   ProcessCpuLoad > 0.2

6. Constraints can be grouped together by using logical operators such as && (AND), || (OR), or ! (NOT) logic. For readability and clarity around the order of operations and operator precedence, grouped constraints may be enclosed in round brackets, but this is not a requirement. For example:

     [(MetricA > value1 && MetricB < value2) || MetricC == 'stringvalue' ; TargetDuration > duration("30s")]
       

7. The name of each MBean counter that is specified as part of a custom trigger must follow precise syntax rules in terms of spelling and capitalization.
8. Only one target duration can be defined for a custom trigger. The target duration is applied to the entire trigger expression that is enclosed within the square brackets
9. A target duration can be expressed in terms of seconds, minutes, or hours. For example, 30s means 30 seconds, 5m means five minutes, 2h means two hours, and so on.
10. A target duration is optional. If a target duration is not specified, triggering will occur immediately once the trigger conditions are met.
11. Multiple custom trigger definitions can be specified together, each of which relates to a separate JFR Recording. Different custom trigger definitions must be separated by a comma (,) character. For example:

      [ProcessCpuLoad>0.2]~profile,[ThreadCount>30]~Continuous
        

Start/Stop a Recording

The first and most basic workflow is to start a Flight Recording in a target JVM.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
3. Click Create

   

4. Configure the new Recording

   

   Enter a Name for the new Recording. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again, or check the Restart if recording already exists checkbox to allow Cryostat to reuse the same recording name for a new recording with new settings.

   Select the Duration of the Recording. The duration can be specified in seconds, minutes, or hours. If you want to record indefinitely, select the Continuous option. You may also want to save the Recording to Cryostat archive storage when the Recording stops with the Archive on Stop option.

   Select an event Template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the Continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed Recording duration. You can also upload a custom Event Template and select it here.

   To learn about Metadata Options, see Add and Edit Recording Metadata Labels.

5. (Optional) Adjust any Advanced Options for the Recording

   

   In many cases, the Advanced Options can be left with their default values. To view and change their values, select Show advanced options above the Create button. The following options are configurable:

   To Disk: Use this parameter to specify whether to write data to your JVM container's disk while recording. By default, this parameter is true.

   Maximum Size: Use this parameter to specify the maximum size of disk data to keep for the Recording. This parameter is valid only when the disk parameter is set to true. By default, the maximum size of disk data isn’t limited, and this parameter is set to 0.

   Maximum Age: Use this parameter to specify the maximum number of seconds, minutes, or hours that events should be kept in the Recording before being discarded. This parameter is valid only when the disk parameter is set to true, and the Recording duration is not Continuous. By default, this parameter is set to 0, which means there is no limit set.

6. Click the Create button
7. Stop the Recording

   

   When you no longer desire for the Flight Recording to be Active and collecting data in your target application, select the Recording from the list by clicking the checkbox to the left of the Recording name. This will enable the Stop button in the toolbar. Click the Stop button to end the data collection.

   If the Recording has a fixed duration then it will automatically stop after the target JVM measures that the duration has elapsed. If the Recording was created with a Continuous Duration then it will collect data until explicitly stopped.

   After Stopping a Recording it remains in the Active section of the Recordings view. This signifies that the Recording data is still present in the target JVM, and not within Cryostat's storage. If the target JVM crashes, is killed, or the process otherwise restarts, then the Recording data will be lost. To learn how to persist the Flight Recording data, continue on to Archive a Recording.

Snapshot a Recording

Snapshotting an application produces a new Flight Recording named snapshot-n, where n is a natural number. This snapshot contains all of the JFR data that was present in the target JVM at the time that the snapshot was taken and is in the STOPPED state as soon as it is created (that is, the snapshot will never record any further information and is fixed in size). This is useful when you have an ongoing, Continuous Recording active in a target application and want to preserve the specific information at a given point in time. If the Continuous source Recording is configured with a maximum event age or maximum recording size then the need for snapshots is more apparent, since without snapshotting, some older data will eventually be dropped and lost from the source Recording. It is also useful when you have multiple concurrent source Recordings running and want an easy way to capture the sum of all of their data at a given point in time.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

   If you do not have any Recordings present in the Active Recordings view, follow Start/Stop a Recording to create one, or select a different target application.

3. Click Create

   

4. Switch to the Snapshot Recording tab

   

5. Observe the New snapshot

   

Archive a Recording

In contrast to Active Recordings, which reside within the container of the target application, Archived Recordings reside within persistent storage attached to a Cryostat instance. In OpenShift, for example, the archives are stored in a PersistentVolumeClaim.

Archived Recordings are created by performing archival upon Active Recordings. When this is requested, Cryostat connects to the target application and copies the Flight Recorder data from the selected Active Recording into an Archived Recording file in storage. The Active Source Recording may be Continuous or Fixed-duration, using any Event Template, in any state (RUNNING, STOPPED, etc.), and may even be a snapshot.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
3. Select an Active Recording

   

4. Click the Archive button
5. Navigate to the Archived Recordings tab

   

Viewing Archived Recordings

There are several ways to view Archived Recordings. The first method is to navigate to the Archived Recordings tab. See Archive a Recording.

All-Targets Archived Recordings View

The second method is to navigate to the All-Targets archived recording view within the Archives tab on Cryostat console sidebar.

The All-Targets view gathers all of Cryostat’s discovered target JVM applications into one section for ease of access. Here, we are able to interact with any Archived Recordings that have been saved from a source target by opening a target’s nested recordings table.

1. Navigate to the All-Targets Archived Recordings view

   

   The option to automatically hide all targets with zero Archived Recordings is on by default and can be toggled. Targets can also be filtered in the search bar.

2. Select a Source Target Application

   

All-Archives Archived Recordings View

The third method is to navigate to the All-Archives Archived Recording view within the Archives tab on Cryostat console sidebar.

The All-Archives view is a view which queries Cryostat’s internal storage for any created Archived Recordings and the directories that contain them. Here, we are able to interact with any Archived Recordings that have been saved into storage by opening a nested recordings table within each directory.

This view is used to save any lost Archived Recordings in case any target JVM restarts or exits. It will be empty if no Recordings are currently saved to storage.

1. Navigate to the All-Archives Archived Recordings View

   

   Directories can be filtered in the search bar.

2. Select an Archives directory

   

   A directory name is related to its corresponding source target's serviceUrl. Mousing over the tooltip, we can also see a Cryostat generated hash ID for that target.

Download an Active or Archived Recording

Cryostat provides some basic capabilities for analysis of Flight Recording data in-cluster. However, the core analysis workflow is to collect JFR files from target applications and copy them to local developer or admin workstations, then use tools such as JDK Mission Control, Visual VM, binjr, or jfr for the heavy lifting on your workstation.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
3. Select a Recording

   

4. Download the Recording

   

5. Choose what to do with the Recording File

   

Re-Upload a Recording to Archives

After downloading an Archived Recording from Cryostat, it can be re-uploaded into Cryostat’s archives later. This is useful if, for example, your Cryostat instance has been scaled down or undeployed for some time and the attached storage is lost, and you then want to use Cryostat’s analysis tools with a previously retrieved Recording in a new Cryostat instance.

1. Navigate to the Archives Tab

   

   The entire Archives view contains several tabs related to performing operations on Archived Recordings. See Viewing Archived Recordings.
2. Select the Recording to Upload

   

3. (Optional) Add Metadata Labels

   

4. Upload the Recording

   

View JFR Event Types

The JVM comes with many built-in JDK Flight Recorder Event Types out-of-the-box. You can also register new event types at the framework- or application-level using the jdk.jfr API. Cryostat can list out all of a Target JVM’s registered event types so you can see what kind of data may be captured by Flight Recordings.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Events

   

   If the Target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the Target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Navigate to Event Types

   

Download, Edit, and Upload a Customized Event Template

Most JVMs will come with at least two JFR Event Templates definitions included: the Continuous and Profiling templates.

The Continuous template:

• has very low runtime overhead
• collects basic data
• is intended to be left on at all times, even in production

The Profiling template:

• may have some detectable runtime overhead
• collects more in-depth data
• should be used for a fixed duration when a specific problem is discovered at runtime

These two definitions will fit many monitoring and profiling workflows, but not all. It may be useful to use either of these as a starting point and tailor it to meet your specific monitoring/profiling needs by including/excluding events, increasing/decreasing sample rates, raising/lowering thresholds, etc.

Of special note, JFR allows for the definition of application-specific or framework-level custom events, which would not be captured in either of the default templates above. You can view all of the JFR Event Types for a Target application.

Cryostat also provides the ALL meta-template, which enables all event types in the selected target application, with default values for each event option. This is not a true Event Template and does not have an XML definition to download.

Finally, Cryostat also includes some Preset Event Templates. These behave like Custom Event Templates in that the event definition is stored by Cryostat, rather than the target application, but are handled separately can cannot be deleted like Custom Event Templates. Preset Event Templates can only be added to a Cryostat instance by adding files to a specific configuration directory and restarting Cryostat.

The steps below assume that you have at least one Target discovered. If you select a discovered Target then you can use its templates as starting points for your customizations. If you do not first select a Target then you may still upload and delete Custom Event Templates.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Events

   

   If the Target JVM has SSL/TLS enabled on JMX connections then it may be necessary to add the Target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.

3. Download the Template

   

4. Edit the Template

   

   Edit the Template definition to suit your requirements. When you are satisfied with the new configuration, give your new Template a meaningful and recognizable tag by editing the <configuration> element at the top of the file and setting the label= attribute to the desired name. You should also save the file according to this new name, but this is not strictly necessary. The description and provider attributes can also be used to help identify this Template later and to explain its goals and purpose.
5. Open the Template Upload Dialog

   

6. Upload the Template

   

7. Observe the new Template

   

Edit Template With JMC

JDK Mission Control has a Template Manager functionality which also contains a graphical wizard for editing .jfc Event Template files. This may be preferred to editing the templates directly as XML plaintext or using XML document editors, which do not have .jfc-specific XSD validation or any hinting for event types and options.

1. Acquire an Event Template

   

   If you do not have an event template definition then you can visit Download, Edit, and Upload a Customized Event Template and follow the first few steps to retrieve one from a remote target application accessible by Cryostat. Otherwise, if you have the JDK installed locally, you can find templates at $JAVA_HOME/lib/jfr.
2. Launch JDK Mission Control

   

3. Open the Template Manager

   

4. Import the Base Template

   

5. Edit the Template

   

6. Export the Template

   

   Once your edited Template is saved to a file on your local workstation it can be used when starting new Flight Recordings. To do this using Cryostat, see Download, Edit, and Upload a Customized Event Template for instructions on how to upload the new template to Cryostat and Start/Stop a Recording for instructions on how to create a new Recording using this Template.

View a Recording in Grafana

Cryostat provides integration with Grafana to plot curated time series metrics from a Recording. By selecting View in Grafana on either an Active or Archived Recording, Cryostat uploads your Recording to a custom Grafana Data Source, and launches Grafana in a new browser tab. If Cryostat was installed to a Kubernetes/OpenShift cluster using the Cryostat Operator or Cryostat Helm Chart, your browser should reuse the same login credentials (if any) as for the Cryostat UI itself. If you are greeted by a login page then simply log in again using your same account credentials which you use to log in to Cryostat.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
3. Select a Recording

   

4. Select View in Grafana...

   

5. View and Interact With Data From Your Recording

   

View Automated Analysis for a Recording

Cryostat integrates the same Automated Analysis reports that you would find in JDK Mission Control (“JMC”). The JMC rules engine analyzes your Recording and looks for common problems, assigning a severity score from 0 (no problem) to 100 (potentially severe problem) to each potential problem type.

Cryostat also provides an Automated Analysis Card that is able to display the same information in a more flexible format, with more tools and control over the data you see and the ability to resize the view. The Card is available for use in the Dashboard. Read the section on the Automated Analysis Card for more information.

1. Select the Target Application

   

   If you wish to create a Custom Target from here, click Create Target displayed at the bottom. Go to Create a Custom Target and proceed from step 3. Also, see Using the Cryostat Agent section to help discover other targets.
2. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
3. Select a Recording

   

4. Expand the Recording

   

5. View Details and Suggestions for Results

   

Add and Edit Recording Metadata Labels

Users can attach Metadata or Custom Labels to JDK Flight Recordings managed by Cryostat. Recording Labels can be used to identify Recordings during queries or perform actions on multiple Recordings containing the same Labels. Here’s how to Add and Edit Metadata Labels on your JDK Flight Recordings.

1. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
2. Click Create

   

3. Add Metadata Labels to the Create Recording form

   

   When creating a custom Flight Recording with Cryostat, expand the form section Show metadata options. Click Add Label and add key-Value Label pairs to the form. Finally, click Create to attach the Labels and create the Recording.
4. View your Labels on the Active Recordings Table

   

   The new Recording will appear in the Recordings Tab with your Custom Label as well as default Labels containing information about the selected Recording template.
5. Edit an Existing Label

   

   Recording Labels can also be edited after Recordings have been created or re-uploaded to archives. It looks like the Custom Label in our example contains a typo - we can fix the typo by editing the Label. First select a Recording from the table with the 🗹 checkbox. Then, click the Edit Labels button to bring up the label drawer. Finally, click the Edit button that appears from the drawer.
   
   
   
   The Labels section will appear as a form where you can Add, Edit, or Delete existing Labels, just like before when we created the Recording. Fix the typo, and click Save to save your edited Labels.

6. View your edited Labels

   

   The Recording Labels should be updated in the Active Recordings table.
7. (Optional) Archive your Recording to view Labels copied onto the Archived Recording

   

   On the Active Recordings table, click the Checkbox next to the Recording, then click Archive to archive your Recording. Notice that the Archived Recording also copies the Labels from the Active Recording. You can also add Labels to any Recording uploaded to Cryostat’s archives.
8. (Optional) Bulk-edit Recording Labels across multiple Recordings

   

   Create another Recording on the same target. Then select both Recordings on the Recordings Table and click Edit Labels and start editing. This time, only Labels that are present on both Recordings will be shown in the form. Let's Delete the two common template-related Labels, and Add a new Label to both Recordings. Then finally, click Save.
   
   
   
   Congratulations, you have successfully bulk-edited Labels across multiple Recordings!

Store Credentials

If you have Java Management Extensions (JMX) or HTTP authentication enabled on your containerized JVMs, Cryostat will prompt you to enter your credentials before it can access the JDK Flight Recordings on your target JVMs.

1. Navigate to the Security tab

   

   First, navigate to the Security tab. The Store Credentials table lists all credentials in Cryostat's Keyring. Click Add to define a new credential.
2. Enter your Credentials

   

   A modal will appear. You can select a target JVM to view its properties. These properties can be used to define a matchExpression that tests whether this credential should apply to the selected target JVM. This is the same kind of matchExpression as those used by Automated Rules, so you may find it useful to reuse the same matchExpressions from your rule definitions for your credential definitions. The matchExpression, username, and password fields are all required.
3. Test your Credentials

   

   Navigate to the Test tab to use the Credential Test table to check if the entered credential is valid for the matched targets.
   
   You can individually test each target or bulk test all targets at the same time. The Status column will report the following test status:

   1. Valid: valid credential for the target JVM.
   2. Invalid: invalid credential for the target JVM.
   3. Not applicable: JMX authentication is not enabled on the target JVM.
4. (Alternative to Steps 1 and 2) Store Credentials when Connecting to a Target JVM

   

   Alternatively, credentials may also be stored if you navigate to either the Recordings tab or the Events tab and select a target JVM with authentication enabled. The authentication form will appear, prompting you to enter your credentials. A new credential definition with the provided username and password will be stored for this specific target application in the Cryostat Keyring.
5. View your Stored Credentials

   

   The Store Credentials table will update with a row for each of the credentials you have stored in the Keyring. The matchExpression is visible along with a count of the number of known target applications that these credentials will apply to. You can expand the row to display a list of those matched target applications. For security reasons it is not possible to view the stored username and password associated with a credential definition in the Keyring.

Add a Trusted Certificate

If you have Java Management Extensions (JMX) over SSL enabled on your containerized JVMs, you must configure Cryostat to trust the SSL/TLS certificate presented by the containerized JVM when Cryostat attempts to open a JMX connection. If you do not complete this configuration, Cryostat cannot open a JMX connection for the purposes of performing JFR management tasks.

1. Navigate to the Security Tab

   

   Click the Security tab. This should initially be in an empty state if you have not yet defined any additional trusted certificates.

In order to add a trusted certificate to Cryostat’s custom truststore you must first determine the directory that Cryostat loads certificates from. This is controlled by the configuration property ssl.truststore.dir and defaults to /truststore. If you are deploying Cryostat manually in an environment like Podman or Docker Compose, you should create a volume containing the certificates and mount it to this location, or bind-mount a host directory to this location. If you are using the Cryostat Helm Chart then you should create Secrets containing each trusted certificate and mount each within this location. If you are using the Cryostat Operator, you should use the .Spec.TrustedCertSecrets CR property.

Once you have loaded your additional certificates to the truststore you must restart the Cryostat container, since it can only load certificates into the JVM truststore at startup time. The container may be restarted automatically when you modify the configuration, depending on your deployment platform.

After you have loaded the certificates and verified that the Cryostat container has restarted, you can verify that Cryostat correctly found the certificate(s) within the truststore directory.

1. Navigate to the Security Tab

   

   Click the Security tab. The file paths of any additional trusted certificates you have added should appear in the list.

Create an Automated Rule

Automated Rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each Automated Rule specifies parameters for which Event Template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

Once you’ve created a rule, Cryostat immediately matches it against all existing discovered targets and starts your Flight Recording. Cryostat will also apply the rule to newly discovered targets that match its definition. You can create multiple rules to match different subsets of targets or to layer different recording options for your needs.

We’ll walk through two use cases: Continuous monitoring in a containerized JVM, and Custom monitoring with Kubernetes labels or annotations.

Continuous Monitoring in a Containerized JVM

Previously, if we wanted to enable always-on Continuous monitoring using JDK Flight Recorder (JFR) in a containerized Java virtual machine (JVM), we would set JVM flags on the target application, then restart the application to start monitoring. With Cryostat’s Automated Rules, we can enable JDK Flight Recorder (JFR) at runtime to continuously monitor an already-running target application, with no restart, no redeploy, and no downtime.

1. Navigate to the Automated Rules Tab

   

2. Click the Create Button
3. Configure the new Automated Rule

   

   Name: Enter a name for the new rule. The form will alert you if the name entered has an invalid format. If the name is already in use then the creation will fail and you will need to try again.

   Description: Enter an optional description for your rule.

   Match Expression: We will fill this field in the next step.

   Enabled: Enable or disable the rule. If disabled, the rule will not be applied to any targets.

   Template: Select an Event Template or enter a custom event definition. If you are unsure which to choose, the Continuous template is useful for always-on production monitoring with the Continuous recording duration setting, and the Profiling template is useful for collecting extra information for troubleshooting a specifically identified problem with a fixed recording duration.

4. Create your Match Expression

   The Match Expression in a rule definition is a Common Expression Language expression that Cryostat interprets and uses to determine if a rule should be applied to any given target. Match Expressions should thus evaluate to a boolean value. The simplest Match Expressions would be the booleans true or false; if we use true, the rule will apply to every target. The Expression has a target object in global scope, with the following form in JSON notation:

   {
     "jvmId": "abcd1234",
     "alias": "myAppAlias",
     "connectUrl": "service:jmx:rmi:///jndi/rmi://cryostat:9091/jmxrmi",
     "labels": {
       "com.example/service": "customer-login",
     },
     "annotations": {
       "platform": {
         "io.kubernetes/annotation": "annotated"
       },
       "cryostat": {
         "PORT": 9091,
         "HOST": "cryostat",
         "NAMESPACE": "myproject"
       }
     }
   }

   The following functions are also available in the Expression execution context:

   /**
   * @param target the target context object
   * @returns a list of JFR Event Type IDs
   */
   jfrEventTypeIds(target: Target): string[]

   See also:

   • JFR Event Types

   The alias, connectUrl, labels, annotations.platform, and annotations.cryostat properties are all guaranteed to be present on the target object. alias and connectUrl will be non-empty strings. The jvmId is a hash string computed by Cryostat after it successfully connects to a target JVM and is used to uniquely identify that JVM instance - it will be empty if Cryostat has not yet connected to that target (for example, if its SSL/TLS certificate is not trusted or if Cryostat is missing the required credentials) The labels and platform annotations may be empty — in OpenShift or Kubernetes, these are populated from the labels and annotations applied to the target’s pod, if any. The Cryostat annotations map will vary per platform, but on OpenShift or Kubernetes you can expect the HOST, PORT, NAMESPACE, and POD_NAME keys to be present and non-empty. Take care to use the has or in operators when dealing with the labels and annotations map structures where specific keys may not exist.

   The Expression also has a jfrEventTypeIds function in global scope, which takes the target object as a parameter and returns a list of strings corresponding to the Flight Recorder Event Types registered in the target JVM.

   jfrEventTypeIds(target: Target): string[];

   Here are some examples of Match Expressions:

   target.alias == 'com.example.MainClass'
   
   target.alias == 'myAlias'
   
   'com.example/service' in target.labels && target.labels[‘com.example/service’] == 'customer-login'
   
   'com.example/service' in target.labels && target.labels[‘com.example/service’] != 'customer-login'
   
   has(target.annotations.cryostat.PORT) && target.annotations.cryostat.PORT > 3000
   
   has(target.annotations.cryostat.PORT) && 'io.kubernetes/annotation' in target.annotations.platform && target.annotations.cryostat.PORT > 3000 && target.annotations.platform['io.kubernetes/annotation'] == 'enabled'
   
   !('io.kubernetes/annotation' in target.annotations.platform)
   
   target.alias.matches("^customer-login[0-9]\*$")
   
   jfrEventTypeIds(target).exists(t, t.startsWith('myorg.myapp.'))

5. Check your Match Expression

   

   When you enter a Match Expression in the Match Expression field, the Match Expression Visualizer will highlight which targets are matched.

6. (Optional) Adjust Rule Parameters

   

   Maximum Size: The maximum size of Recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

   Maximum Age: The maximum age of Recording data retained in the target application's recording buffer. Values less than 1 indicate no limit.

   Archival Period: Time between copies of Active recording data being pulled into Cryostat archive storage. Values less than 1 prevent data from being copied into archives - Recordings will be started and remain only in target JVM memory.

   Initial Delay: Time between rule creation and when the first archived copy should be transferred. Values less than 1 are treated as being equal to the Archival Period above.

   Preserved Archives: The number of Recording copies to preserve in archives for each target application affected by this rule. Values less than 1 prevent data from being copied into archives - Recordings will be started and remain only in target JVM memory.

   In the example image, the Maximum Recording age was set to 300 seconds and the Archival Period was set to a slightly shorter time period of 285 seconds. This overlap ensures that all of your Flight Recorder data is preserved in Cryostat's archives. The initial delay is set to 60 seconds however, so the first archive copy will be made 1 minute after the rule is created. The next copy will be made 5 minutes after that, the next another 5 minute later, etc.

7. Click the Create Button
8. Observe the new Rule in the Automated Rules Table

   

9. Navigate to Recordings

   

   If the target JVM has SSL/TLS enabled on JMX connections, it may be necessary to add the target's certificate to Cryostat's trust store. Go to Add a Trusted Certificate and return to this section after completing that guide.
10. Observe the automatically generated Recording in the Active Recordings Table

    

    Once you've set up your Automated Rules, Cryostat will continuously monitor applications that meet the criteria defined in those rules, with no need to restart or redeploy those applications.

Custom Monitoring with Kubernetes Labels or Annotations

We can define a rule that applies to any target application that has platform-specific attributes, such as Kubernetes labels or annotations. Here’s an example in JSON notation:

{
  "name": "k8sMonitoring",
  "description": "Enable the Demo template on any target with the jfrMonitoring=true annotation",
  "matchExpression": "'jfrMonitoring' in target.annotations.platform && target.annotations.platform['jfrMonitoring']=='enabled'",
  "eventSpecifier": "template=Demo,type=CUSTOM",
  "archivalPeriodSeconds": 300,
  "preservedArchives": 12
}

Once we’ve created this rule definition, Cryostat will check all of the existing target applications and watch for new targets that appear with the jfrMonitoring=enabled annotation. Any matching targets found will have a Recording started automatically using the Custom Demo template from our first use case. It will take an Archived snapshot every five minutes and maintain an hour’s worth of these archives in storage.

With this rule definition in place, Kubernetes or Red Hat OpenShift users can use familiar tools like kubectl/oc or the OpenShift console to mark target applications for monitoring, without needing to interact directly with the Cryostat API or UI. This opens the door to further automating your workflow.

As an example, you might use or implement an Operator that monitors traffic flow or pod restarts and enables monitoring on pods after some criterion threshold is met, then disables it again if the target application’s behavior returns to normal. As a Kubernetes administrator, you could receive a notification when this occurs and check the Cryostat archives to retrieve JDK Flight Recorder data from the target application recorded during the problematic period, or you could view these Archived Recordings in Cryostat’s Grafana dashboard.

Note: An important caveat is that Cryostat does not watch for changes in the Kubernetes annotations or labels; it only watches to see if target applications appear or disappear. To apply the annotation to a target application, we must apply the annotation or label to the application pod (which will cause Kubernetes to roll out a new replica), and not to the deployment.

Upload an Automated Rule

Instead of stepping through the form every time, you can also edit your Automated Rule definition files (JSON format) locally and quickly upload them.

Below is an example of a rule file content:

{
  "name":"my_automated_rule",
  "description":"This is an example automated rule.",
  "matchExpression":"target.alias == 'io.cryostat.Cryostat'",
  "eventSpecifier":"template=Continuous,type=TARGET",
  "archivalPeriodSeconds":285,
  "initialDelaySeconds":60,
  "preservedArchives":10,
  "maxAgeSeconds":300,
  "maxSizeBytes":0,
  "enabled":true
}

1. Open Automated Rules upload Prompt

   

2. Attach a Rule File by Dragging & Dropping or clicking Browse....

   

3. Click Submit Button to Upload.
4. Observe the new rule in the Automated Rules table

   

Download an Automated Rule

Automated Rules can easily be downloaded for viewing and editing.

1. Open Action Menu for a selected Rule

   

2. Click Download Button to Download.

Configure Graphical Notifications

Actions such as starting/stopping/archiving Recordings and state changes such as Recordings stopping after a fixed duration produce graphical notifications. These notifications appear both when an interactive user performs the action via the web-client, as well as when an Automated Rule performs the action, or when any other API client performs the action. If many Automated Rules or other API clients are active, the notification stream within the web-client graphical console can become quite noisy.

1. Navigate to Settings

   

   The Settings view contains web-client instance configuration settings. These affect the appearance or behaviour of the web-client frontend only, not the behaviour of the Cryostat backend server. These frontend settings are persisted to web browser local storage only.
2. Locate the Notifications Setting

   

   The Notifications setting within this view is used to control the graphical display of notifications that correspond to WebSocket messages sent by the Cryostat backend when actions and state changes occur. The setting contains a toggle switch for each category of notification, as well as a global switch All Notifications , to enable/disable all categories at once. Disabling a notification category only prevents it from appearing as a pop-up notification toast in the current web-client instance. The notification will still appear in the notification drawer, accessed by clicking the bell icon in the application masthead, and other web-client instances or API clients will still receive the notification to process as they will.