1. Documentation
    2. history bug_report picture_as_pdf
×

Upgrading ROSA (classic architecture) clusters

edit

Use one of the following methods to upgrade ROSA (classic architecture) clusters:

  • Manually through the ROSA CLI (rosa) - Start a one-time immediate upgrade or schedule a one-time upgrade for a future date or time.

  • Manually through the OpenShift Cluster Manager UI - Start a one-time immediate upgrade or schedule a one-time upgrade for a future date or time; or schedule an upgrade window for automatic recurring upgrades whenever a new z-version is available.

edit

Life cycle policies and planning

To plan an upgrade, review the Red Hat OpenShift Service on AWS update life cycle. The life cycle page includes release definitions, support and upgrade requirements, installation policy information and life cycle dates.

You can use update channels to decide which Red Hat OpenShift Container Platform minor version to update your clusters to. Red Hat OpenShift Service on AWS supports updates only through the stable channel. To learn more about OpenShift update channels and releases, see Understanding update channels and releases.

Upgrading a ROSA Classic cluster

You must upgrade ROSA (classic architecture) clusters using either the ROSA CLI (rosa) or the OpenShift Cluster Manager console.

The actual start time of the cluster upgrade will be within one hour of the upgrade schedule time. Additionally, the duration of the upgrade might vary based on your workload configuration.

When a ROSA (classic architecture) cluster that uses AWS Security Token Services (STS) is upgraded, the ROSA CLI verifies the account and Operator role policies for the chosen cluster are compatible with the target version of the upgrade. If the policies are compatible, the CLI automatically upgrades the cluster. If the policies are not compatible with the chosen upgrade version, the CLI automatically upgrades IAM policies before upgrading the cluster. When scheduling the upgrade, you give administrative acknowledgment to confirm you have reviewed the changes involved with the upgrade, if required.

How ROSA (classic architecture) cluster upgrades work

Upgrades are manually initiated (one-time) or automatically scheduled (recurring). Red Hat Site Reliability Engineers (SREs) monitor upgrade progress and either proactively notify you to take corrective actions or remedy issues encountered.

The Cluster Version Operator (CVO) is the primary component that orchestrates and facilitates the OpenShift Container Platform update process.

The Managed Upgrade Operator (MUO) handles the scheduling, monitoring, and notifications of ROSA (classic architecture) cluster upgrades. The MUO orchestrates the automated in-place cluster upgrades by ensuring the operating conditions are met before and after the upgrade of a managed cluster.

edit

Cluster upgrade scheduled time

You can schedule cluster upgrades by setting the scheduled time. This is when the preparation for the cluster upgrade begins with pre-upgrade health checks and additional compute capacity creation. The actual cluster upgrade starts within one hour from the scheduled time. You receive an email notification when the cluster upgrade starts.

The Pre-Health Check (PHC) provides extra protection to ensure the scheduled update proceeds as expected and runs in the following two scenarios:

  • If an upgrade is scheduled for more than 2 hours from the current time, the PHC runs and the user is notified if there are any failures. This PHC is in the New phase of the upgrade.

  • When the upgrade is immediate or within 2 hours, the PHC runs just before the upgrade begins. This PHC is in the Upgrading phase of the upgrade. This means that PHC is always run at least one time during the upgrading phase but can also be run additionally in advance if the upgrade is scheduled for more than 2 hours from the current time.

You can observe the status of the cluster upgrade by running the rosa describe upgrade --cluster=<cluster name|cluster_id> command in the ROSA CLI (rosa).

ROSA (classic architecture) upgrade overview

The following are the high-level steps that occur during the ROSA (classic architecture) cluster update:

  1. Scheduling the upgrade in advance triggers the PreHealthCheck and notifies users of any failures that they can then address before the upgrade begins.

  2. Before the cluster upgrade begins, a cluster health check is performed by the MUO. If the MUO identifies an issue that requires corrective action, you will be notified. Some examples of the cluster health checks that MUO performs include:

    • Identifying any Pod Disruption Budgets (PDBs) that can potentially block or delay the update of the nodes.

    • Ensuring cluster Operators are available and healthy.

    • Ensuring cluster critical alerts are not firing.

  3. A temporary compute node is created in the cluster to allow for the scheduling of drained pods during the update.

    The temporary compute node creation does not always happen. For example, if there is no worker machine pool, the temporary compute node will not be created. This might happen when a cluster admin deletes the existing worker machine pool and creates another worker machine pool with a different name or instance type.

  4. The cluster version is set to the target version.

    In certain situations, an upgrade path can become unavailable since the time the cluster update was requested but before it was completed. In such cases, the upgrade is automatically canceled and a notification is sent. You must pick another target version to request the upgrade.

  5. During the upgrade, the control plane components are updated to the new version.

  6. Next, individual cluster Operators perform update tasks on their domain of the cluster.

  7. Finally, the MCO updates the system configuration and operating system of every node. During this step, each node is rebooted after successfully draining the workloads running on the node.

    1. During the update of each node, workloads are drained, honoring the PDBs. Workloads with PDBs that do not allow disruptions essentially block the draining of the node, increasing the elapsed time for the cluster update.

    2. During the update of every node in the cluster, the cluster update waits until the time specified by the node drain grace period to allow for safely draining the workloads. Upon reaching the node drain grace period, the node is forcibly drained to allow for cluster upgrade to progress. You can only configure the node drain grace period before initiating the upgrade and you cannot change it after the cluster upgrade begins.

    3. When cluster nodes are updated, the MCO selects one node at a time per machine config pool according to their age, starting with the oldest.

edit

Upgrading with the ROSA CLI

You can use the ROSA CLI (rosa) to upgrade a Red Hat OpenShift Service on AWS (ROSA) cluster either immediately within one hour or at a future time.

Prerequisites

  • You have installed and configured the latest ROSA CLI on your installation host.

  • Your Red Hat OpenShift Service on AWS cluster is in a Ready state.

Procedure

  1. To verify the current version of your cluster, enter the following command:

    $ rosa describe cluster --cluster=<cluster_name|cluster_id> (1)
    1 Replace <cluster_name|cluster_id> with the cluster name or the ID of the cluster.

  2. To verify that an upgrade is available, enter the following command:

    $ rosa list upgrade --cluster=<cluster_name|cluster_id>

    The command returns a list of versions to which the cluster can be upgraded, including a recommended version. The recommendation is based on the conditional update risks. Each known risk might apply to all clusters or only clusters matching certain conditions. Refer to the OpenShift release notes to evaluate, validate and determine the appropriate version to upgrade to.

  3. To upgrade the cluster to a specified version immediately within the next hour, enter the following command:

    $ rosa upgrade cluster --cluster=<cluster_name|cluster_id> --version <version-id>

    If you are upgrading an AWS Security Token Service (STS) cluster, this command starts an interactive IAM Roles/policies upgrade mode process that verifies the account and operator role policies for the chosen cluster are compatible with the target version of the upgrade. If the policies are not compatible with the chosen upgrade version, the CLI automatically upgrades them in auto mode.

    The cluster is scheduled for an immediate upgrade as denoted by the Scheduled Time. The upgrade will begin within one hour from the scheduled time.

  4. Alternatively, to upgrade the cluster at a future time in UTC, enter the following command:

    $ rosa upgrade cluster --cluster=<cluster_name|cluster_id>   \
          --version <version-id>   \
          --schedule-date yyyy-mm-dd \
          --schedule-time HH:mm

  5. To customize the grace period for every node to be drained during the cluster upgrade, enter the following command:

    $ rosa upgrade cluster --cluster=<cluster_name|cluster_id>   \
          --version <version-id>   \
          --node-drain-grace-period 15 minutes

  6. You can view the status of the upgrade by entering the following command, which shows both the status (scheduled or started) and the scheduled time.

    $ rosa list upgrade --cluster=<cluster_name|cluster_id>
    Example output
    VERSION  NOTES
4.15.14  recommended - scheduled for 2024-06-02 15:00 UTC
4.15.13

You will receive email notifications confirming the scheduling, beginning, and completion of the cluster upgrade.

Troubleshooting
edit

Deleting a ROSA cluster upgrade with the ROSA CLI

You can use either the ROSA CLI (rosa) or OpenShift Cluster Manager console to delete a scheduled upgrade. This procedure uses the ROSA CLI.

Procedure

  1. Verify the cluster update has not started using the following command:

    $ rosa list upgrades --cluster=<cluster_name|cluster_id>
    Example output
    VERSION  NOTES
4.15.14  recommended - scheduled for 2024-06-02 15:00 UTC
4.15.13

  2. Delete a scheduled update by running the following command:

    $ rosa delete upgrade --cluster=<cluster_name|cluster_id>

  3. Confirm the deletion by entering Yes at the confirmation prompt.

    Example output
    I: Successfully canceled scheduled upgrade on cluster 'my-cluster'

You will receive an email notification confirming that the scheduled upgrade has been canceled.

edit

Upgrading with the OpenShift Cluster Manager console

You can schedule upgrades for a ROSA (classic architecture) cluster manually either one time or on a recurring schedule by using OpenShift Cluster Manager console.

Procedure

  1. Log in to OpenShift Cluster Manager.

  2. Select a cluster to upgrade.

  3. Click the Settings tab.

  4. In the Update strategy pane, select which type of update you want:

    • For individual updates, you can request the upgrade either immediately (to start within an hour) or at a future time.

    • For recurring updates, choose a recurring date and time to start the upgrade automatically to the latest x.y.Z (z-stream) version available.

      Recurring updates are applicable only for z-stream updates. Minor version or y-stream updates need to be done manually. You will be notified when a new y-stream update is available.

  5. Optional: In the Node draining pane, select a grace period interval from the list. The grace period enables the nodes to gracefully drain before forcing the pod eviction. The default is 1 hour.

    You cannot change the node drain grace period after you start the upgrade process.

  6. In the Update strategy pane, click Save to apply your update strategy.

  7. In the Update status pane, review the Update available information and click Update.

    The Update button is enabled only when an upgrade is available.

  8. The Update cluster dialog opens. Recommended cluster upgrades appear in the Select version pane. Select the version you want to upgrade your cluster to, and click Next.

  9. Optional: For ROSA clusters that use AWS Security Token Service (STS), the account-level and cluster-specific Operator roles might need to be updated, depending on the selected target version.

    1. In the ROSA CLI, run the rosa list account-roles command to list and verify that the account roles are compatible with the target minor version chosen for the upgrade. If the roles are not compatible, run the rosa upgrade account-roles command to upgrade the account roles to the latest OpenShift version.

    2. In the ROSA CLI, run the rosa list operator-roles command to list and verify that Operator roles associated with the cluster are compatible with the target minor version chosen for the upgrade. If not, run the rosa upgrade operators-roles command to upgrade the cluster’s Operator roles to the latest OpenShift version.

    3. If you select an update version that requires approval, provide an administrator’s acknowledgment by typing Acknowledge into the field provided, and click Next.

  10. In the Schedule update dialog, schedule your cluster upgrade.

    • To upgrade within an hour, select Update now and click Next.

    • To upgrade at a later time, select Schedule a different time and set a time and date for your upgrade. Click Next to proceed to the confirmation dialog.

  11. After reviewing the version and schedule summary, select Confirm update.

  12. Click Close to exit out of the Update cluster dialog.

The cluster is scheduled for an upgrade to the target version. This action can take up to an hour, depending on the selected upgrade schedule and your workload configuration, such as pod disruption budgets.

The status is displayed in the Update status pane.

Troubleshooting
edit

Deleting a ROSA cluster upgrade with the OpenShift Cluster Manager console

You can use the OpenShift Cluster Manager console to delete a scheduled upgrade.

Procedure

  1. Log in to OpenShift Cluster Manager.

  2. Select the cluster with the scheduled upgrade.

  3. Click the Settings tab.

  4. In the Update status pane, click Cancel this update.

  5. Review the update details in the Cancel update dialog and click Cancel this update.

You will receive an email notification confirming that the scheduled upgrade has been canceled.