Cryostat 2.4.0
Installing Cryostat Operator
Follow the steps below to install the Cryostat Operator via OperatorHub.
Install cert-manager
The Cryostat Operator requires cert-manager v1.11.5+ to run. If not already installed in your cluster, please install it using your preferred method. Once installed, proceed with the Operator installation steps below.
Warning: Although it is possible to disable cert-manager integration, it is NOT recommended to do so unless cert-manager is unavailable AND one of the following applies to you:
- You have another solution for encrypting traffic
- You trust everything running in the same cluster where the Cryostat Operator is deployed
Install via OperatorHub
See below for a summary of the installation steps from the Cryostat Operator page on OperatorHub. For more details, visit Installing the Cryostat Operator from OperatorHub.
- Install the Operator Lifecycle Manager (OLM):
Check if OLM is already installed:
$ operator-sdk olm status $ # or without the operator-sdk binary, $ POD="$(kubectl get -l app=olm-operator -n olm pod -o 'jsonpath={.items[0].metadata.name}')" $ kubectl exec -n olm "pod/${POD}" -- olm --version
If Operator Lifecycle Manager (OLM) and OperatorHub are already installed and available on your cluster, skip to Step 3:
Install OLM:
$ operator-sdk olm install $ # or without the operator-sdk binary, $ curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.25.0/install.sh | bash -s v0.25.0
- Verify the installation was successful by confirming all pods are
Running
:$ kubectl get pods -n olm NAME READY STATUS RESTARTS AGE catalog-operator-77b8589cd8-8bntf 1/1 Running 0 3m9s olm-operator-5ccf676d8b-7rgss 1/1 Running 0 3m9s operatorhubio-catalog-pb896 1/1 Running 0 3m3s packageserver-8cccc99dd-dv8rp 1/1 Running 0 3m3s packageserver-8cccc99dd-g7lkh 1/1 Running 0 3m3s
- Install Cryostat from OperatorHub:
Cryostat on OperatorHub
Use the search bar to find the Red Hat build of Cryostat catalog item.
Select the Cryostat Operator and click the Install button
Choose your Operator installation mode:
- In
All Namespaces
installation mode, the Cryostat Operator instance will watch for Cryostat or ClusterCryostat Custom Resources (CRs) created in anyNamespace
and create corresponding Cryostat instances. - In the
A specific namespace
installation mode, you must also select an installationNamespace
, and the Cryostat Operator instance will only watch for Cryostat or ClusterCryostat instances created in that sameNamespace
.
Install the Operator
Click “Install” and wait for the installation to complete. In this example we will proceed with All Namespaces.
Continue to Setup.
Setup
Note: An alternative setup using the multi-namespace ClusterCryostat CR is described in Alternate Setup. For simplicity we will continue with the single-namespace Cryostat CR.
Deploying Cryostat
Create a Cryostat instance
Once the installation is complete, click Create Cryostat to create a Cryostat CR instance in an OpenShift Project (Kubernetes Namespace) of your choice. This provides configuration information for the Operator to know the specifics of how to deploy your Cryostat instance. For full details on how to configure the Cryostat deployment, see Configuring Cryostat.
If running Cryostat on Kubernetes and not OpenShift, you will also need to add Ingress configurations to your Cryostat custom resource (CR). See the Network Options section of Configuring Cryostat for examples.
You can create the CR graphically in the OperatorHub UI after following Install via OperatorHub:
Installed Operators View
Cryostat Resources Before
Cryostat Resource Creation
Cryostat Resources After
You can also create the CR manually using a YAML definition like the following:
apiVersion: operator.cryostat.io/v1beta1
kind: Cryostat
metadata:
name: cryostat-sample
spec:
minimal: false
enableCertManager: true
trustedCertSecrets: []
eventTemplates: []
storageOptions:
pvc:
labels: {}
annotations: {}
spec: {}
reportOptions:
replicas: 0
Then apply the resource:
$ oc apply -f cryostat.yaml
$ # or alternatively
$ kubectl apply -f cryostat.yaml
Open the Cryostat Web UI
Let’s visit the Cryostat web dashboard UI.
We can get there from the Cryostat CR’s status fields:
Cryostat Resource Status
Or, we can open the application link from the OpenShift Console Topology view:
Topology View
We can also find the URL using oc
:
$ oc get cryostat -o jsonpath='{$.items[0].status.applicationUrl}'
Authenticate through Cryostat
OpenShift Authentication
When deployed in OpenShift, Cryostat will use the existing internal cluster authentication system to ensure all requests come from users with correct access to the Cryostat instance and the namespace that it is deployed within.
OpenShift SSO Login
OpenShift Service Account Permissions
Once you have authenticated through the cluster’s SSO login you will be
redirected back to the Cryostat web application. The redirect URL contains
an access token for Cryostat’s service account with the permissions you have
granted to it. The Cryostat web application passes this OpenShift token back
to the Cryostat server on each request using Bearer
authorization headers.
The Cryostat server forwards this token back to the OpenShift auth server on
each client request to check the token authorization for the current request.
This access token will eventually expire and you will be required to log back
in on the cluster SSO login page.
For direct access to the Cryostat HTTP API you may follow the same pattern.
Using a client such as curl
, an OpenShift auth token can be passed with
requests using the Authorization: Bearer
header. The token must be base64
encoded. For example,
$ curl -v -H "Authorization: Bearer $(oc whoami -t | base64)" https://cryostat.example.com:8181/api/v1/targets
Other Platforms Authentication
In non-OpenShift environments, Cryostat will default to no authentication.
Access to the web application and the HTTP API will be unsecured. You should
either configure Cryostat’s built-in Basic
authentication system, or better,
place an authenticating reverse proxy server in front of Cryostat so that
accesses to the Cryostat application must first pass through the reverse
proxy. The configuration of a reverse proxy is out of scope of this guide.
Basic Auth
Cryostat includes a very rudimentary HTTP Basic
authentication implementation.
This can be configured by creating a cryostat-users.properties
file in the
Cryostat server conf
directory, defined by the environment variable
CRYOSTAT_CONFIG_PATH
and defaulting to /opt/cryostat.d/conf.d
.
The credentials stored in the Java properties file are the user name and a
SHA-256 sum hex of the user’s password. The property file contents should look
like:
user1=abc123
user2=def987
Where abc123
and def987
are substituted for the SHA-256 sum hexes of the
desired user passwords. These can be obtained by ex.
echo -n PASS | sha256sum | cut -d' ' -f1
. The Basic
user credentials user:pass
would therefore be entered as
user:d74ff0ee8da3b9806b18c877dbf29bbde50b5bd8e4dad7a3a725000feb82e8f1
.
This mechanism only supports fully-privileged user definitions, authorized to perform any action within the Cryostat API.
Once the cryostat-users.properties
file defining the user credentials is
created, the environment variable CRYOSTAT_AUTH_MANAGER
should be set
to the value io.cryostat.net.BasicAuthManager
to enable the corresponding
auth implementation.
Deploy an Application
For demo purposes, let’s go ahead and deploy a sample application to our
OpenShift cluster in the same namespace
as our Cryostat instance. If you have
deployed Cryostat into a namespace
where you are already running other
applications, feel free to continue to the next step.
$ oc new-app --docker-image=quay.io/andrewazores/quarkus-test:0.0.10
$ oc patch svc/quarkus-test -p '{"spec":{"$setElementOrder/ports":[{"port":9097},{"port":8080}],"ports":[{"name":"jfr-jmx","port":9097}]}}'
This is a Quarkus container in JVM mode with JMX enabled and pre-configured to
listen on port 9097
. After deploying the container we patch its service to
name the 9097
service port jfr-jmx
. Cryostat will detect and use this port
to determine that this is a compatible Java application that it should monitor.
Configuring Applications
There are two methods of configuring your Java applications so that Cryostat is able to discover and monitor them:
- using the Cryostat Agent for discovery and connectivity
- using platform mechanisms for discovery and Java Management Extensions (JMX) for connectivity
The following sections will briefly explain how to accomplish each of these approaches by example. For simplicity the examples will assume your application
is built with Maven, packaged into an image with a Dockerfile
, and running in Kubernetes, but the procedure will be similar for other toolchains and platforms as well.
Using the Cryostat Agent
The Cryostat Agent is compatible with Cryostat versions 2.3.0 and newer, and application JDKs 11 and newer. If you are using an older version of Cryostat, we recommend upgrading to ensure compatibility. If your application uses a recent version of JDK8 with JFR support, please either upgrade to JDK11+ or continue to the next section to learn how to configure your application without the Cryostat Agent.
The Cryostat Agent JAR must be available to your application JVM. The JAR asset can be downloaded directly from upstream,
or you may use the following snippet in your pom.xml
to streamline this.
<project>
...
<repositories>
<repository>
<id>github</id>
<url>https://maven.pkg.github.com/cryostatio/cryostat-agent</url>
</repository>
</repositories>
...
<build>
<plugins>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>io.cryostat</groupId>
<artifactId>cryostat-agent</artifactId>
<version>0.3.0</version>
<classifier>shaded</classifier>
</artifactItem>
</artifactItems>
<stripVersion>true</stripVersion>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
...
</build>
...
</project>
Note: You may be required to authenticate to the GitHub Maven Packages registry in order to pull this JAR.
The next time we build our application, the Cryostat Agent JAR will be located at target/dependency/cryostat-agent-shaded.jar
. Then we can update our Dockerfile:
...
COPY target/dependency/cryostat-agent-shaded.jar /deployments/app/
...
# Assume we are using an application framework where the JAVA_OPTS environment variable can be used to pass JVM flags
ENV JAVA_OPTS="-javaagent:/deployments/app/cryostat-agent-shaded.jar"
Next we must rebuild our container image. This is specific to your application but will likely look something like docker build -t docker.io/myorg/myapp:latest -f src/main/docker/Dockerfile .
.
Push that updated image or otherwise get it updated in your Kubernetes registry, then modify your application Deployment
to supply JVM system properties or environment variables configuring
the Cryostat Agent:
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
containers:
- name: sample-app
image: docker.io/myorg/myapp:latest
env:
- name: CRYOSTAT_AGENT_APP_NAME
# Replace this with any value you like to use to identify your application.
value: "myapp"
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CRYOSTAT_AGENT_BASEURI
# Update this to correspond to the name of your Cryostat instance
# if it is not 'cryostat'. This assumes that the target application
# and the Cryostat instance are in the same Namespace, but you may
# choose to configure the Agent to communicate with a Cryostat in
# a different Namespace, too.
# (https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/)
value: https://cryostat.$(NAMESPACE).svc.cluster.local:8181
- name: CRYOSTAT_AGENT_API_WRITES_ENABLED
# Set this to 'true' to turn on the "write" or "mutation" capabilities of the
# Agent's HTTP API. This defaults to 'false', so the Agent HTTP API only exposes
# readonly access to certain low-sensitivity calls. If this is 'true' then the
# Agent will allow Cryostat to dynamically request JFR recordings to be started,
# stopped, deleted, etc. as well as listed and retrieved.
value: true
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CRYOSTAT_AGENT_CALLBACK
# This infers the Agent Callback directly from the Pod's IP address using the
# Kubernetes Downward API. Use this value directly as provided. The port number
# 9977 can be changed but must match the containerPort below.
value: "http://$(POD_IP):9977"
# Replace "abcd1234" with a base64-encoded authentication token. For example,
# in your terminal, do 'oc whoami -t | base64' to use your user account's
# token as the token that the Agent will pass to authorize itself with
# the Cryostat server.
- name: CRYOSTAT_AGENT_AUTHORIZATION
value: "Bearer abcd1234"
ports:
- containerPort: 9977
protocol: TCP
resources: {}
restartPolicy: Always
status: {}
Port number 9977
is the default HTTP port that the Agent exposes for its internal webserver that services Cryostat requests. The CRYOSTAT_AGENT_AUTHORIZATION
value is particularly
noteworthy: these are the credentials that the Agent will include in API requests it makes to Cryostat to advertise its own presence. You should create a Kubernetes Service Account
for
this purpose and replace abcd1234
with the base64-encoded
authentication token associated with the service account. For testing purposes you may use your own user account’s
authentication token, for example with oc whoami --show-token
.
Finally, create a Service
to enable Cryostat to make requests to this Agent:
apiVersion: v1
kind: Service
...
spec:
ports:
- name: "cryostat-agent"
port: 9977
targetPort: 9977
...
More details about the configuration options for the Cryostat Agent are available here.
You may also be interested in using the Cryostat Agent for application discovery, but using JMX for remote management and data
access rather than the Cryostat Agent HTTP API. In that case, simply set CRYOSTAT_AGENT_API_WRITES_ENABLED=false
to turn off as much
of the Cryostat Agent HTTP API as possible, then continue to the next section to additionally configure your application
to enable and expose JMX for remote management and data access. If the Cryostat Agent detects that the application it is attached
to has JMX enabled then it will publish itself to the Cryostat server with both an Agent HTTP URL and a JMX URL. If JMX
is not detected then it will only publish the HTTP URL.
Using JMX
Cryostat is also able to use Java Management Extensions (JMX) to communicate with target applications. This is a standard JDK feature that can be enabled by passing JVM flags to your application at startup. A basic and insecure setup suitable for testing requires only the following three flags:
-Dcom.sun.management.jmxremote.port=9091
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
It is recommended that you enable both SSL
and authentication on your application. You can then trust the certificate
and store the credentials.
Depending on your application or its framework, you may set these flags directly in a Dockerfile
entrypoint, an environment variable, or similar. This may or
may not require a container image rebuild, and it will require the container to be restarted. Once this is done the application container will be listening for
incoming JMX connections on port 9091
. Let’s assume it can be done by setting an environment variable, so we only need to modify our Deployment
:
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
containers:
- name: sample-app
image: docker.io/myorg/myapp:latest
env:
- name: JAVA_OPTS
value: >-
-Dcom.sun.management.jmxremote.port=9091
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
...
Next, we need to configure a Kubernetes Service
to expose this port for cluster-internal traffic, so that Cryostat can see
and connect to this application JMX port.
apiVersion: v1
kind: Service
...
spec:
ports:
- name: "jfr-jmx"
port: 9091
targetPort: 9091
...
Cryostat queries the Kubernetes API server and looks for Service
s with a port either named jfr-jmx
or with the number 9091
.
One or both of these conditions must be met or else Cryostat will not automatically detect your
application. In this case you may wish to use the Cryostat Agent to enable discovery, while keeping
communications over JMX rather than HTTP. If you do use the Cryostat Agent for discovery and JMX for remote management,
you may combine both of the Service
definitions into a single Service
with two exposed ports
.
Alternate Setup
Using ClusterCryostats
In Deploying Cryostat, you created a single-namespace Cryostat Custom Resource (CR) instance.
Single-namespace Cryostat CRs instruct the Operator to deploy restricted Cryostat instances which are only able
to see target applications deployed in the same namespace as the Cryostat instance, which is the same Namespace
that
the CR is created within.
If you chose to install the Operator in All Namespaces mode as assumed in this guide, you may also be interested in
creating CluterCryostat CRs. In this configuration, the Operator is able to see Cryostat and ClusterCryostat
CRs in any project (Namespace
) and create Cryostat deployments corresponding to either CR kind in each of their
respective Namespaces. Both of these CRs are Namespace-specific, and the Namespace is used to determine which
OpenShift users are able to access the Cryostat instance. For more information, please see the following documents:
ClusterCryostat CRs instruct the Operator to deploy cross-namespace Cryostat instances. A ClusterCryostat has
an installNamespace
, which is the namespace where the Cryostat Deployment will reside, and a list of
targetNamespaces
, which are all of the namespaces that the Cryostat server will watch for target applications.
The targetNamespaces
list does not necessarily need to contain the installNamespace
, if you do not want Cryostat
to see itself in the target applications that it watches.
apiVersion: operator.cryostat.io/v1beta1
kind: ClusterCryostat
metadata:
name: clustercryostat-sample
spec:
enableCertManager: true
installNamespace: cryostat-testing
minimal: false
reportOptions:
resources: {}
storageOptions:
pvc:
spec:
resources: {}
targetNamespaces:
- cryostat-testing
- my-apps-a
- my-apps-b
Next Steps
Now that you have installed and deployed Cryostat and know how to access its web client, continue on to Guides for guides through various common actions and workflows.
Uninstalling Cryostat Operator
Reference OLM guide on uninstalling Operators. Please be sure to delete all Cryostat and ClusterCryostat Custom Resources before uninstalling the Cryostat Operator.
- If your Cryostat Operator was installed in All Namespaces mode, then its ClusterServiceVersion and
Subscription
can be found in the Namespace openshift-operators. - If your Cryostat Operator was installed in A specific Namespace, then the ClusterServiceVersion and
Subscription
will be in that same Namespace.