OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Scaling a user-provisioned cluster with the Bare Metal Operator

About scaling a user-provisioned cluster with the Bare Metal Operator

You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal3 components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal nodes in a cluster. However, it is possible to use the BMO and other metal3 components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.

Prerequisites for scaling a user-provisioned cluster

  • You installed a user-provisioned infrastructure cluster on bare metal.

  • You have baseboard management controller (BMC) access to the hosts.

Limitations for scaling a user-provisioned cluster

  • You cannot use a provisioning network to scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO).

    • Consequentially, you can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.

  • You cannot scale MachineSet objects in user-provisioned infrastructure clusters by using the BMO.

Configuring a provisioning resource to scale user-provisioned clusters

Create a Provisioning custom resource (CR) to enable Metal platform components on a user-provisioned infrastructure cluster.

Prerequisites

  • You installed a user-provisioned infrastructure cluster on bare metal.

Procedure

  1. Create a Provisioning CR.

    1. Save the following YAML in the provisioning.yaml file:

      apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
  name: provisioning-configuration
spec:
  provisioningNetwork: "Disabled"
  watchAllNamespaces: false

      OpenShift Container Platform 4.17 does not support enabling a provisioning network when you scale a user-provisioned cluster by using the Bare Metal Operator.

  2. Create the Provisioning CR by running the following command:

    $ oc create -f provisioning.yaml
    Example output
    provisioning.metal3.io/provisioning-configuration created
Verification

  • Verify that the provisioning service is running by running the following command:

    $ oc get pods -n openshift-machine-api
    Example output
    NAME                                                  READY   STATUS    RESTARTS        AGE
cluster-autoscaler-operator-678c476f4c-jjdn5          2/2     Running   0               5d21h
cluster-baremetal-operator-6866f7b976-gmvgh           2/2     Running   0               5d21h
control-plane-machine-set-operator-7d8566696c-bh4jz   1/1     Running   0               5d21h
ironic-proxy-64bdw                                    1/1     Running   0               5d21h
ironic-proxy-rbggf                                    1/1     Running   0               5d21h
ironic-proxy-vj54c                                    1/1     Running   0               5d21h
machine-api-controllers-544d6849d5-tgj9l              7/7     Running   1 (5d21h ago)   5d21h
machine-api-operator-5c4ff4b86d-6fjmq                 2/2     Running   0               5d21h
metal3-6d98f84cc8-zn2mx                               5/5     Running   0               5d21h
metal3-image-customization-59d745768d-bhrp7           1/1     Running   0               5d21h

Provisioning new hosts in a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to provision bare-metal hosts in a user-provisioned cluster by creating a BareMetalHost custom resource (CR).

Provisioning bare-metal hosts to the cluster by using the BMO sets the spec.externallyProvisioned specification in the BareMetalHost custom resource to false by default. Do not set the spec.externallyProvisioned specification to true, because this setting results in unexpected behavior.
Prerequisites

  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Create a configuration file for the bare-metal node. Depending if you use either a static configuration or a DHCP server, choose one of the following example bmh.yaml files and configure it to your needs by replacing values in the YAML to match your environment:

    • To deploy with a static configuration, create the following bmh.yaml file:

      ---
apiVersion: v1
kind: Secret
metadata:
  name: openshift-worker-<num>-network-config-secret (1)
  namespace: openshift-machine-api
type: Opaque
stringData:
  nmstate: | (2)
    interfaces: (3)
    - name: <nic1_name> (4)
      type: ethernet
      state: up
      ipv4:
        address:
        - ip: <ip_address> (5)
          prefix-length: 24
        enabled: true
    dns-resolver:
      config:
        server:
        - <dns_ip_address> (6)
    routes:
      config:
      - destination: 0.0.0.0/0
        next-hop-address: <next_hop_ip_address> (7)
        next-hop-interface: <next_hop_nic1_name> (8)
---
apiVersion: v1
kind: Secret
metadata:
  name: openshift-worker-<num>-bmc-secret
  namespace: openshift-machine-api
type: Opaque
data:
  username: <base64_of_uid> (9)
  password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: openshift-worker-<num>
  namespace: openshift-machine-api
spec:
  online: true
  bootMACAddress: <nic1_mac_address> (10)
  bmc:
    address: <protocol>://<bmc_url> (11)
    credentialsName: openshift-worker-<num>-bmc-secret
    disableCertificateVerification: false
  customDeploy:
    method: install_coreos
  userData:
    name: worker-user-data-managed
    namespace: openshift-machine-api
  rootDeviceHints:
    deviceName: <root_device_hint> (12)
  preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret
      1 Replace all instances of <num> with a unique compute node number for the bare-metal nodes in the name, credentialsName, and preprovisioningNetworkDataName fields.
      2 Add the NMState YAML syntax to configure the host interfaces. To configure the network interface for a newly created node, specify the name of the secret that has the network configuration. Follow the nmstate syntax to define the network configuration for your node. See "Preparing the bare-metal node" for details on configuring NMState syntax.
      3 Optional: If you have configured the network interface with nmstate, and you want to disable an interface, set state: up with the IP addresses set to enabled: false.
      4 Replace <nic1_name> with the name of the bare-metal node’s first network interface controller (NIC).
      5 Replace <ip_address> with the IP address of the bare-metal node’s NIC.
      6 Replace <dns_ip_address> with the IP address of the bare-metal node’s DNS resolver.
      7 Replace <next_hop_ip_address> with the IP address of the bare-metal node’s external gateway.
      8 Replace <next_hop_nic1_name> with the name of the bare-metal node’s external gateway.
      9 Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.
      10 Replace <nic1_mac_address> with the MAC address of the bare-metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.
      11 Replace <protocol> with the BMC protocol, such as IPMI, Redfish, or others. Replace <bmc_url> with the URL of the bare-metal node’s baseboard management controller.
      12 Optional: Replace <root_device_hint> with a device path when specifying a root device hint. See "Root device hints" for additional details.

    • When configuring the network interface with a static configuration by using nmstate, set state: up with the IP addresses set to enabled: false:

      ---
apiVersion: v1
kind: Secret
metadata:
  name: openshift-worker-<num>-network-config-secret
  namespace: openshift-machine-api
 # ...
interfaces:
  - name: <nic_name>
    type: ethernet
    state: up
    ipv4:
      enabled: false
    ipv6:
      enabled: false
# ...

    • To deploy with a DHCP configuration, create the following bmh.yaml file:

      ---
apiVersion: v1
kind: Secret
metadata:
  name: openshift-worker-<num>-bmc-secret (1)
  namespace: openshift-machine-api
type: Opaque
data:
  username: <base64_of_uid> (2)
  password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: openshift-worker-<num>
  namespace: openshift-machine-api
spec:
  online: true
  bootMACAddress: <nic1_mac_address> (3)
  bmc:
    address: <protocol>://<bmc_url> (4)
    credentialsName: openshift-worker-<num>-bmc
    disableCertificateVerification: false
  customDeploy:
    method: install_coreos
  userData:
    name: worker-user-data-managed
    namespace: openshift-machine-api
  rootDeviceHints:
    deviceName: <root_device_hint> (5)
      1 Replace <num> with a unique compute node number for the bare-metal nodes in the name and credentialsName fields.
      2 Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.
      3 Replace <nic1_mac_address> with the MAC address of the bare-metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.
      4 Replace <protocol> with the BMC protocol, such as IPMI, Redfish, or others. Replace <bmc_url> with the URL of the bare-metal node’s baseboard management controller.
      5 Optional: Replace <root_device_hint> with a device path when specifying a root device hint. See "Root device hints" for additional details.

      If the MAC address of an existing bare-metal node matches the MAC address of the bare-metal host that you are attempting to provision, then the installation will fail. If the host enrollment, inspection, cleaning, or other steps fail, the Bare Metal Operator retries the installation continuously. See "Diagnosing a duplicate MAC address when provisioning a new host in the cluster" for additional details.

  2. Create the bare-metal node by running the following command:

    $ oc create -f bmh.yaml
    Example output
    secret/openshift-worker-<num>-network-config-secret created
secret/openshift-worker-<num>-bmc-secret created
baremetalhost.metal3.io/openshift-worker-<num> created

  3. Inspect the bare-metal node by running the following command:

    $ oc -n openshift-machine-api get bmh openshift-worker-<num>

    where:

    <num>

    Specifies the compute node number.

    Example output
    NAME                    STATE       CONSUMER   ONLINE   ERROR
openshift-worker-<num>  provisioned true

  4. Approve all certificate signing requests (CSRs).

    1. Get the list of pending CSRs by running the following command:

      $ oc get csr
      Example output
      NAME        AGE   SIGNERNAME                                    REQUESTOR                                         REQUESTEDDURATION CONDITION
csr-gfm9f   33s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-o
perator:node-bootstrapper   <none>              Pending

    2. Approve the CSR by running the following command:

      $ oc adm certificate approve <csr_name>
      Example output
      certificatesigningrequest.certificates.k8s.io/<csr_name> approved
Verification

  • Verify that the node is ready by running the following command:

    $ oc get nodes
    Example output
    NAME        STATUS   ROLES           AGE     VERSION
app1        Ready    worker          47s     v1.24.0+dc5a2fd
controller1 Ready    master,worker   2d22h   v1.24.0+dc5a2fd
Additional resources

Optional: Managing existing hosts in a user-provisioned cluster by using the BMO

Optionally, you can use the Bare Metal Operator (BMO) to manage existing bare-metal controller hosts in a user-provisioned cluster by creating a BareMetalHost object for the existing host. It is not a requirement to manage existing user-provisioned hosts; however, you can enroll them as externally-provisioned hosts for inventory purposes.

To manage existing hosts by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to true to prevent the BMO from re-provisioning the host.
Prerequisites

  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the controller.yaml file:

      ---
apiVersion: v1
kind: Secret
metadata:
  name: controller1-bmc
  namespace: openshift-machine-api
type: Opaque
data:
  username: <base64_of_uid>
  password: <base64_of_pwd>
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: controller1
  namespace: openshift-machine-api
spec:
  bmc:
    address: <protocol>://<bmc_url> (1)
    credentialsName: "controller1-bmc"
  bootMACAddress: <nic1_mac_address>
  customDeploy:
    method: install_coreos
  externallyProvisioned: true (2)
  online: true
  userData:
    name: controller-user-data-managed
    namespace: openshift-machine-api
      1 You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2 You must set the value to true to prevent the BMO from re-provisioning the bare-metal controller host.

  2. Create the bare-metal host object by running the following command:

    $ oc create -f controller.yaml
    Example output
    secret/controller1-bmc created
baremetalhost.metal3.io/controller1 created
Verification

  • Verify that the BMO created the bare-metal host object by running the following command:

    $ oc get bmh -A
    Example output
    NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
openshift-machine-api   controller1   externally provisioned              true             13s

Removing hosts from a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to remove bare-metal hosts from a user-provisioned cluster.

Prerequisites

  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure

  1. Cordon and drain the node by running the following command:

    $ oc adm drain app1 --force --ignore-daemonsets=true
    Example output
    node/app1 cordoned
WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-tvthg, openshift-dns/dns-
default-9q6rz, openshift-dns/node-resolver-zvt42, openshift-image-registry/node-ca-mzxth, openshift-ingress-cana
ry/ingress-canary-qq5lf, openshift-machine-config-operator/machine-config-daemon-v79dm, openshift-monitoring/nod
e-exporter-2vn59, openshift-multus/multus-additional-cni-plugins-wssvj, openshift-multus/multus-fn8tg, openshift
-multus/network-metrics-daemon-5qv55, openshift-network-diagnostics/network-check-target-jqxn2, openshift-ovn-ku
bernetes/ovnkube-node-rsvqg
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766965-258vp
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766950-kg5mk
evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766935-stf4s
pod/collect-profiles-27766965-258vp evicted
pod/collect-profiles-27766950-kg5mk evicted
pod/collect-profiles-27766935-stf4s evicted
node/app1 drained

  2. Delete the customDeploy specification from the BareMetalHost CR.

    1. Edit the BareMetalHost CR for the host by running the following command:

      $ oc edit bmh -n openshift-machine-api <host_name>

    2. Delete the lines spec.customDeploy and spec.customDeploy.method:

      ...
  customDeploy:
    method: install_coreos

    3. Verify that the provisioning state of the host changes to deprovisioning by running the following command:

      $ oc get bmh -A
      Example output
      NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
openshift-machine-api   controller1   externally provisioned              true             58m
openshift-machine-api   worker1       deprovisioning                      true             57m

  3. Delete the host by running the following command when the BareMetalHost state changes to available:

    $ oc delete bmh -n openshift-machine-api <bmh_name>

    You can run this step without having to edit the BareMetalHost CR. It might take some time for the BareMetalHost state to change from deprovisioning to available.

  4. Delete the node by running the following command:

    $ oc delete node <node_name>
Verification

  • Verify that you deleted the node by running the following command:

    $ oc get nodes
    Example output
    NAME          STATUS   ROLES           AGE     VERSION
controller1   Ready    master,worker   2d23h   v1.24.0+dc5a2fd