OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Managing hosted control planes on AWS

Prerequisites to manage AWS infrastructure and IAM permissions

To configure hosted control planes for OpenShift Container Platform on Amazon Web Services (AWS), you must meet the following the infrastructure requirements:

  • You configured hosted control planes before you can create hosted clusters.

  • You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials.

Infrastructure requirements for AWS

When you use hosted control planes on Amazon Web Services (AWS), the infrastructure requirements fit in the following categories:

  • Prerequired and unmanaged infrastructure for the HyperShift Operator in an arbitrary AWS account

  • Prerequired and unmanaged infrastructure in a hosted cluster AWS account

  • Hosted control planes-managed infrastructure in a management AWS account

  • Hosted control planes-managed infrastructure in a hosted cluster AWS account

  • Kubernetes-managed infrastructure in a hosted cluster AWS account

Prerequired means that hosted control planes requires AWS infrastructure to properly work. Unmanaged means that no Operator or controller creates the infrastructure for you.

Unmanaged infrastructure for the HyperShift Operator in an AWS account

An arbitrary Amazon Web Services (AWS) account depends on the provider of the hosted control planes service.

In self-managed hosted control planes, the cluster service provider controls the AWS account. The cluster service provider is the administrator who hosts cluster control planes and is responsible for uptime. In managed hosted control planes, the AWS account belongs to Red Hat.

In a prerequired and unmanaged infrastructure for the HyperShift Operator, the following infrastructure requirements apply for a management cluster AWS account:

  • One S3 Bucket

    • OpenID Connect (OIDC)

  • Route 53 hosted zones

    • A domain to host private and public entries for hosted clusters

Unmanaged infrastructure requirements for a management AWS account

When your infrastructure is prerequired and unmanaged in a hosted cluster Amazon Web Services (AWS) account, the infrastructure requirements for all access modes are as follows:

  • One VPC

  • One DHCP Option

  • Two subnets

    • A private subnet that is an internal data plane subnet

    • A public subnet that enables access to the internet from the data plane

  • One internet gateway

  • One elastic IP

  • One NAT gateway

  • One security group (worker nodes)

  • Two route tables (one private and one public)

  • Two Route 53 hosted zones

  • Enough quota for the following items:

    • One Ingress service load balancer for public hosted clusters

    • One private link endpoint for private hosted clusters

For private link networking to work, the endpoint zone in the hosted cluster AWS account must match the zone of the instance that is resolved by the service endpoint in the management cluster AWS account. In AWS, the zone names are aliases, such as us-east-2b, which do not necessarily map to the same zone in different accounts. As a result, for private link to work, the management cluster must have subnets or workers in all zones of its region.

Infrastructure requirements for a management AWS account

When your infrastructure is managed by hosted control planes in a management AWS account, the infrastructure requirements differ depending on whether your clusters are public, private, or a combination.

For accounts with public clusters, the infrastructure requirements are as follows:

  • Network load balancer: a load balancer Kube API server

    • Kubernetes creates a security group

  • Volumes

    • For etcd (one or three depending on high availability)

    • For OVN-Kube

For accounts with private clusters, the infrastructure requirements are as follows:

  • Network load balancer: a load balancer private router

  • Endpoint service (private link)

For accounts with public and private clusters, the infrastructure requirements are as follows:

  • Network load balancer: a load balancer public router

  • Network load balancer: a load balancer private router

  • Endpoint service (private link)

  • Volumes

    • For etcd (one or three depending on high availability)

    • For OVN-Kube

Infrastructure requirements for an AWS account in a hosted cluster

When your infrastructure is managed by hosted control planes in a hosted cluster Amazon Web Services (AWS) account, the infrastructure requirements differ depending on whether your clusters are public, private, or a combination.

For accounts with public clusters, the infrastructure requirements are as follows:

  • Node pools must have EC2 instances that have Role and RolePolicy defined.

For accounts with private clusters, the infrastructure requirements are as follows:

  • One private link endpoint for each availability zone

  • EC2 instances for node pools

For accounts with public and private clusters, the infrastructure requirements are as follows:

  • One private link endpoint for each availability zone

  • EC2 instances for node pools

Kubernetes-managed infrastructure in a hosted cluster AWS account

When Kubernetes manages your infrastructure in a hosted cluster Amazon Web Services (AWS) account, the infrastructure requirements are as follows:

  • A network load balancer for default Ingress

  • An S3 bucket for registry

Identity and Access Management (IAM) permissions

In the context of hosted control planes, the consumer is responsible to create the Amazon Resource Name (ARN) roles. The consumer is an automated process to generate the permissions files. The consumer might be the CLI or OpenShift Cluster Manager. Hosted control planes can enable granularity to honor the principle of least-privilege components, which means that every component uses its own role to operate or create Amazon Web Services (AWS) objects, and the roles are limited to what is required for the product to function normally.

The hosted cluster receives the ARN roles as input and the consumer creates an AWS permission configuration for each component. As a result, the component can authenticate through STS and preconfigured OIDC IDP.

The following roles are consumed by some of the components from hosted control planes that run on the control plane and operate on the data plane:

  • controlPlaneOperatorARN

  • imageRegistryARN

  • ingressARN

  • kubeCloudControllerARN

  • nodePoolManagementARN

  • storageARN

  • networkARN

The following example shows a reference to the IAM roles from the hosted cluster:

...
endpointAccess: Public
  region: us-east-2
  resourceTags:
  - key: kubernetes.io/cluster/example-cluster-bz4j5
    value: owned
rolesRef:
    controlPlaneOperatorARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-control-plane-operator
    imageRegistryARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-openshift-image-registry
    ingressARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-openshift-ingress
    kubeCloudControllerARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-cloud-controller
    networkARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-cloud-network-config-controller
    nodePoolManagementARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-node-pool
    storageARN: arn:aws:iam::820196288204:role/example-cluster-bz4j5-aws-ebs-csi-driver-controller
type: AWS
...

The roles that hosted control planes uses are shown in the following examples:

  • ingressARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:DescribeLoadBalancers",
                "tag:GetResources",
                "route53:ListHostedZones"
            ],
            "Resource": "\*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::PUBLIC_ZONE_ID",
                "arn:aws:route53:::PRIVATE_ZONE_ID"
            ]
        }
    ]
}

  • imageRegistryARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket",
                "s3:DeleteBucket",
                "s3:PutBucketTagging",
                "s3:GetBucketTagging",
                "s3:PutBucketPublicAccessBlock",
                "s3:GetBucketPublicAccessBlock",
                "s3:PutEncryptionConfiguration",
                "s3:GetEncryptionConfiguration",
                "s3:PutLifecycleConfiguration",
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "\*"
        }
    ]
}

  • storageARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AttachVolume",
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteSnapshot",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DescribeInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:DetachVolume",
                "ec2:ModifyVolume"
            ],
            "Resource": "\*"
        }
    ]
}

  • networkARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstanceTypes",
                "ec2:UnassignPrivateIpAddresses",
                "ec2:AssignPrivateIpAddresses",
                "ec2:UnassignIpv6Addresses",
                "ec2:AssignIpv6Addresses",
                "ec2:DescribeSubnets",
                "ec2:DescribeNetworkInterfaces"
            ],
            "Resource": "\*"
        }
    ]
}

  • kubeCloudControllerARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeImages",
                "ec2:DescribeRegions",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifyVolume",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateRoute",
                "ec2:DeleteRoute",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteVolume",
                "ec2:DetachVolume",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DescribeVpcs",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancerPolicy",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DeleteLoadBalancerListeners",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DetachLoadBalancerFromSubnets",
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
                "iam:CreateServiceLinkedRole",
                "kms:DescribeKey"
            ],
            "Resource": [
                "\*"
            ],
            "Effect": "Allow"
        }
    ]
}

  • nodePoolManagementARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:AllocateAddress",
                "ec2:AssociateRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateInternetGateway",
                "ec2:CreateNatGateway",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteNatGateway",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeImages",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeNatGateways",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeVolumes",
                "ec2:DetachInternetGateway",
                "ec2:DisassociateRouteTable",
                "ec2:DisassociateAddress",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:ModifySubnetAttribute",
                "ec2:ReleaseAddress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "tag:GetResources",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateLaunchTemplateVersion",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DeleteLaunchTemplate",
                "ec2:DeleteLaunchTemplateVersions"
            ],
            "Resource": [
                "\*"
            ],
            "Effect": "Allow"
        },
        {
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            },
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": [
                "arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                "arn:*:iam::*:role/*-worker-role"
            ],
            "Effect": "Allow"
        }
    ]
}

  • controlPlaneOperatorARN

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateVpcEndpoint",
                "ec2:DescribeVpcEndpoints",
                "ec2:ModifyVpcEndpoint",
                "ec2:DeleteVpcEndpoints",
                "ec2:CreateTags",
                "route53:ListHostedZones"
            ],
            "Resource": "\*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets"
            ],
            "Resource": "arn:aws:route53:::%s"
        }
    ]
}

Creating AWS infrastructure and IAM resources separate

By default, the hcp create cluster aws command creates cloud infrastructure with the hosted cluster and applies it. You can create the cloud infrastructure portion separately so that you can use the hcp create cluster aws command only to create the cluster, or render it to modify it before you apply it.

To create the cloud infrastructure portion separately, you need to create the Amazon Web Services (AWS) infrastructure, create the AWS Identity and Access (IAM) resources, and create the cluster.

Creating the AWS infrastructure separately

To create the Amazon Web Services (AWS) infrastructure, you need to create a Virtual Private Cloud (VPC) and other resources for your cluster. You can use the AWS console or an infrastructure automation and provisioning tool. For instructions to use the AWS console, see Create a VPC plus other VPC resources in the AWS Documentation.

The VPC must include private and public subnets and resources for external access, such as a network address translation (NAT) gateway and an internet gateway. In addition to the VPC, you need a private hosted zone for the ingress of your cluster. If you are creating clusters that use PrivateLink (Private or PublicAndPrivate access modes), you need an additional hosted zone for PrivateLink.

Create the AWS infrastructure for your hosted cluster by using the following example configuration:

---
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: clusters
spec: {}
status: {}
---
apiVersion: v1
data:
  .dockerconfigjson: xxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    hypershift.openshift.io/safe-to-delete-with-cluster: "true"
  name: <pull_secret_name> (1)
  namespace: clusters
---
apiVersion: v1
data:
  key: xxxxxxxxxxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    hypershift.openshift.io/safe-to-delete-with-cluster: "true"
  name: <etcd_encryption_key_name> (2)
  namespace: clusters
type: Opaque
---
apiVersion: v1
data:
  id_rsa: xxxxxxxxx
  id_rsa.pub: xxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  labels:
    hypershift.openshift.io/safe-to-delete-with-cluster: "true"
  name: <ssh-key-name> (3)
  namespace: clusters
---
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  creationTimestamp: null
  name: <hosted_cluster_name> (4)
  namespace: clusters
spec:
  autoscaling: {}
  configuration: {}
  controllerAvailabilityPolicy: SingleReplica
  dns:
    baseDomain: <dns_domain> (5)
    privateZoneID: xxxxxxxx
    publicZoneID: xxxxxxxx
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
          storageClassName: gp3-csi
        type: PersistentVolume
    managementType: Managed
  fips: false
  infraID: <infra_id> (6)
  issuerURL: <issuer_url> (7)
  networking:
    clusterNetwork:
    - cidr: 10.132.0.0/14
    machineNetwork:
    - cidr: 10.0.0.0/16
    networkType: OVNKubernetes
    serviceNetwork:
    - cidr: 172.31.0.0/16
  olmCatalogPlacement: management
  platform:
    aws:
      cloudProviderConfig:
        subnet:
          id: <subnet_xxx> (8)
        vpc: <vpc_xxx> (9)
        zone: us-west-1b
      endpointAccess: Public
      multiArch: false
      region: us-west-1
      rolesRef:
        controlPlaneOperatorARN: arn:aws:iam::820196288204:role/<infra_id>-control-plane-operator
        imageRegistryARN: arn:aws:iam::820196288204:role/<infra_id>-openshift-image-registry
        ingressARN: arn:aws:iam::820196288204:role/<infra_id>-openshift-ingress
        kubeCloudControllerARN: arn:aws:iam::820196288204:role/<infra_id>-cloud-controller
        networkARN: arn:aws:iam::820196288204:role/<infra_id>-cloud-network-config-controller
        nodePoolManagementARN: arn:aws:iam::820196288204:role/<infra_id>-node-pool
        storageARN: arn:aws:iam::820196288204:role/<infra_id>-aws-ebs-csi-driver-controller
    type: AWS
  pullSecret:
    name: <pull_secret_name>
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.16-x86_64
  secretEncryption:
    aescbc:
      activeKey:
        name: <etcd_encryption_key_name>
    type: aescbc
  services:
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: OVNSbDb
    servicePublishingStrategy:
      type: Route
  sshKey:
    name: <ssh_key_name>
status:
  controlPlaneEndpoint:
    host: ""
    port: 0
---
apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  creationTimestamp: null
  name: <node_pool_name> (10)
  namespace: clusters
spec:
  arch: amd64
  clusterName: <hosted_cluster_name>
  management:
    autoRepair: true
    upgradeType: Replace
  nodeDrainTimeout: 0s
  platform:
    aws:
      instanceProfile: <instance_profile_name> (11)
      instanceType: m6i.xlarge
      rootVolume:
        size: 120
        type: gp3
      subnet:
        id: <subnet_xxx>
    type: AWS
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.16-x86_64
  replicas: 2
status:
  replicas: 0
1 Replace <pull_secret_name> with the name of your pull secret.
2 Replace <etcd_encryption_key_name> with the name of your etcd encryption key.
3 Replace <ssh_key_name> with the name of your SSH key.
4 Replace <hosted_cluster_name> with the name of your hosted cluster.
5 Replace <dns_domain> with your base DNS domain, such as example.com.
6 Replace <infra_id> with the value that identifies the IAM resources that are associated with the hosted cluster.
7 Replace <issuer_url> with your issuer URL, which ends with your infra_id value. For example, https://example-hosted-us-west-1.s3.us-west-1.amazonaws.com/example-hosted-infra-id.
8 Replace <subnet_xxx> with your subnet ID. Both private and public subnets need to be tagged. For public subnets, use kubernetes.io/role/elb=1. For private subnets, use kubernetes.io/role/internal-elb=1.
9 Replace <vpc_xxx> with your VPC ID.
10 Replace <node_pool_name> with the name of your NodePool resource.
11 Replace <instance_profile_name> with the name of your AWS instance.

Creating the AWS IAM resources

In Amazon Web Services (AWS), you must create the following IAM resources:

Creating a hosted cluster separately

You can create a hosted cluster separately on Amazon Web Services (AWS).

To create a hosted cluster separately, enter the following command:

$ hcp create cluster aws \
    --infra-id <infra_id> \(1)
    --name <hosted_cluster_name> \(2)
    --sts-creds <path_to_sts_credential_file> \(3)
    --pull-secret <path_to_pull_secret> \(4)
    --generate-ssh \(5)
    --node-pool-replicas 3
    --role-arn <role_name> (6)
1 Replace <infra_id> with the same ID that you specified in the create infra aws command. This value identifies the IAM resources that are associated with the hosted cluster.
2 Replace <hosted_cluster_name> with the name of your hosted cluster.
3 Replace <path_to_sts_credential_file> with the same name that you specified in the create infra aws command.
4 Replace <path_to_pull_secret> with the name of the file that contains a valid OpenShift Container Platform pull secret.
5 The --generate-ssh flag is optional, but is good to include in case you need to SSH to your workers. An SSH key is generated for you and is stored as a secret in the same namespace as the hosted cluster.
6 Replace <role_name> with the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole. Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole. For more information about ARN roles, see "Identity and Access Management (IAM) permissions".

You can also add the --render flag to the command and redirect output to a file where you can edit the resources before you apply them to the cluster.

After you run the command, the following resources are applied to your cluster:

  • A namespace

  • A secret with your pull secret

  • A HostedCluster

  • A NodePool

  • Three AWS STS secrets for control plane components

  • One SSH key secret if you specified the --generate-ssh flag.

Transitioning a hosted cluster from single-architecture to multi-architecture

You can transition your single-architecture 64-bit AMD hosted cluster to a multi-architecture hosted cluster on Amazon Web Services (AWS), to reduce the cost of running workloads on your cluster. For example, you can run existing workloads on 64-bit AMD while transitioning to 64-bit ARM and you can manage these workloads from a central Kubernetes cluster.

A single-architecture hosted cluster can manage node pools of only one particular CPU architecture. However, a multi-architecture hosted cluster can manage node pools with different CPU architectures. On AWS, a multi-architecture hosted cluster can manage both 64-bit AMD and 64-bit ARM node pools.

Prerequisites

  • You have installed an OpenShift Container Platform management cluster for AWS on Red Hat Advanced Cluster Management (RHACM) with the multicluster engine for Kubernetes Operator.

  • You have an existing single-architecture hosted cluster that uses 64-bit AMD variant of the OpenShift Container Platform release payload.

  • An existing node pool that uses the same 64-bit AMD variant of the OpenShift Container Platform release payload and is managed by an existing hosted cluster.

  • Ensure that you installed the following command-line tools:

    • oc

    • kubectl

    • hcp

    • skopeo

Procedure

  1. Review an existing OpenShift Container Platform release image of the single-architecture hosted cluster by running the following command:

    $ oc get hostedcluster/<hosted_cluster_name> -o jsonpath='{.spec.release.image}' (1)
    1 Replace <hosted_cluster_name> with your hosted cluster name.
    Example output
    quay.io/openshift-release-dev/ocp-release:<4.y.z>-x86_64 (1)
    1 Replace <4.y.z> with the supported OpenShift Container Platform version that you use.

  2. In your OpenShift Container Platform release image, if you use the digest instead of a tag, find the multi-architecture tag version of your release image:

    1. Set the OCP_VERSION environment variable for the OpenShift Container Platform version by running the following command:

      $ OCP_VERSION=$(oc image info quay.io/openshift-release-dev/ocp-release@sha256:ac78ebf77f95ab8ff52847ecd22592b545415e1ff6c7ff7f66bf81f158ae4f5e -o jsonpath='{.config.config.Labels["io.openshift.release"]}')

    2. Set the MULTI_ARCH_TAG environment variable for the multi-architecture tag version of your release image by running the following command:

      $ MULTI_ARCH_TAG=$(skopeo inspect docker://quay.io/openshift-release-dev/ocp-release@sha256:ac78ebf77f95ab8ff52847ecd22592b545415e1ff6c7ff7f66bf81f158ae4f5e | jq -r '.RepoTags' | sed 's/"//g' | sed 's/,//g' | grep -w "$OCP_VERSION-multi$" | xargs)

    3. Set the IMAGE environment variable for the multi-architecture release image name by running the following command:

      $ IMAGE=quay.io/openshift-release-dev/ocp-release:$MULTI_ARCH_TAG

    4. To see the list of multi-architecture image digests, run the following command:

      $ oc image info $IMAGE
      Example output
      OS            DIGEST
linux/amd64   sha256:b4c7a91802c09a5a748fe19ddd99a8ffab52d8a31db3a081a956a87f22a22ff8
linux/ppc64le sha256:66fda2ff6bd7704f1ba72be8bfe3e399c323de92262f594f8e482d110ec37388
linux/s390x   sha256:b1c1072dc639aaa2b50ec99b530012e3ceac19ddc28adcbcdc9643f2dfd14f34
linux/arm64   sha256:7b046404572ac96202d82b6cb029b421dddd40e88c73bbf35f602ffc13017f21

  3. Transition the hosted cluster from single-architecture to multi-architecture:

    1. Set the multiArch flag to true on your hosted cluster so that the hosted cluster can create both 64-bit AMD and 64-bit ARM node pools. Run the following command:

      $ oc patch -n clusters hostedclusters/<hosted_cluster_name> -p '{"spec":{"platform":{"aws":{"multiArch":true}}}}' --type=merge

    2. Confirm that the multiArch flag is set to true on your hosted cluster by running the following command:

      $ oc get hostedcluster/<hosted_cluster_name> -o jsonpath='{.spec.platform.aws.multiArch}'

    3. Set the multi-architecture OpenShift Container Platform release image for the hosted cluster by ensuring that you use the same OpenShift Container Platform version as the hosted cluster. Run the following command:

      $ oc patch -n clusters hostedclusters/<hosted_cluster_name> -p '{"spec":{"release":{"image":"quay.io/openshift-release-dev/ocp-release:<4.x.y>-multi"}}}' --type=merge (1)
      1 Replace <4.y.z> with the supported OpenShift Container Platform version that you use.

    4. Confirm that the multi-architecture image is set in your hosted cluster by running the following command:

      $ oc get hostedcluster/<hosted_cluster_name> -o jsonpath='{.spec.release.image}'

  4. Check that the status of the HostedControlPlane resource is Progressing by running the following command:

    $ oc get hostedcontrolplane -n <hosted_control_plane_namespace> -oyaml
    Example output
    #...
  - lastTransitionTime: "2024-07-28T13:07:18Z"
    message: HostedCluster is deploying, upgrading, or reconfiguring
    observedGeneration: 5
    reason: Progressing
    status: "True"
    type: Progressing
#...

  5. Check that the status of the HostedCluster resource is Progressing by running the following command:

    $ oc get hostedcluster <hosted_cluster_name> -n <hosted_cluster_namespace> -oyaml
Verification

  • Verify that a node pool is using the multi-architecture release image in your HostedControlPlane resource by running the following command:

    $ oc get hostedcontrolplane -n clusters-example -oyaml
    Example output
    #...
version:
    availableUpdates: null
    desired:
      image: quay.io/openshift-release-dev/ocp-release:<4.x.y>-multi (1)
      url: https://access.redhat.com/errata/RHBA-2024:4855
      version: 4.16.5
    history:
    - completionTime: "2024-07-28T13:10:58Z"
      image: quay.io/openshift-release-dev/ocp-release:<4.x.y>-multi
      startedTime: "2024-07-28T13:10:27Z"
      state: Completed
      verified: false
      version: <4.x.y>
    1 Replace <4.y.z> with the supported OpenShift Container Platform version that you use.

    The multi-architecture OpenShift Container Platform release image is updated in your HostedCluster, HostedControlPlane resources, and hosted control plane pods. However, your existing node pools do not transition with the multi-architecture image automatically, because the release image transition is decoupled between the hosted cluster and node pools. You must create new node pools on your new multi-architecture hosted cluster.
Next steps

  • Creating node pools on the multi-architecture hosted cluster

Creating node pools on the multi-architecture hosted cluster

After transitioning your hosted cluster from single-architecture to multi-architecture, create node pools on compute machines based on 64-bit AMD and 64-bit ARM architectures.

Procedure

  1. Create node pools based on 64-bit ARM architecture by entering the following command:

    $ hcp create nodepool aws \
--cluster-name <hosted_cluster_name> \(1)
--name <nodepool_name> \(2)
--node-count=<node_count> \(3)
--arch arm64
    1 Replace <hosted_cluster_name> with your hosted cluster name.
    2 Replace <nodepool_name> with your node pool name.
    3 Replace <node_count> with integer for your node count, for example, 2.

  2. Create node pools based on 64-bit AMD architecture by entering the following command:

    $ hcp create nodepool aws \
--cluster-name <hosted_cluster_name> \(1)
--name <nodepool_name> \(2)
--node-count=<node_count> \(3)
--arch amd64
    1 Replace <hosted_cluster_name> with your hosted cluster name.
    2 Replace <nodepool_name> with your node pool name.
    3 Replace <node_count> with integer for your node count, for example, 2.
Verification

  • Verify that a node pool is using the multi-architecture release image by entering the following command:

    $ oc get nodepool/<nodepool_name> -oyaml
    Example output for 64-bit AMD node pools
    #...
spec:
  arch: amd64
#...
  release:
    image: quay.io/openshift-release-dev/ocp-release:<4.x.y>-multi (1)
    1 Replace <4.y.z> with the supported OpenShift Container Platform version that you use.
    Example output for 64-bit ARM node pools
    #...
spec:
  arch: arm64
#...
  release:
    image: quay.io/openshift-release-dev/ocp-release:<4.x.y>-multi