HyperShift API Reference

Spec is the desired behavior of the HostedCluster.

Status is the latest observed status of the HostedCluster.

Spec is the desired behavior of the NodePool.

Status is the latest observed status of the NodePool.

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

AdvertiseAddress is the address that nodes will use to talk to the API server. This is an address associated with the loopback adapter of each node. If not specified, the controller will take default values. The default values will be set as 172.20.0.1 or fd00::1.

Port is the port at which the APIServer is exposed inside a node. Other pods using host networking cannot listen on this port. If unset 6443 is used. This is useful to choose a port other than the default one which might interfere with customer environments e.g. https://github.com/openshift/hypershift/pull/356. Setting this to 443 is possible only for backward compatibility reasons and it’s discouraged. Doing so, it would result in the controller overriding the KAS endpoint in the guest cluster having a discrepancy with the KAS Pod and potentially causing temporarily network failures.

AllowedCIDRBlocks is an allow list of CIDR blocks that can access the APIServer If not specified, traffic is allowed from all addresses. This depends on underlying support by the cloud provider for Service LoadBalancerSourceRanges

Subnet is the subnet to use for control plane cloud resources.

Zone is the availability zone where control plane cloud resources are created.

VPC is the VPC to use for control plane cloud resources.

"Private"

Private endpoint access allows only private API server access and private node communication with the control plane.

"Public"

Public endpoint access allows public API server access and public node communication with the control plane.

"PublicAndPrivate"

PublicAndPrivate endpoint access allows public API server access and private node communication with the control plane.

The referenced role must have a trust relationship that allows it to be assumed via web identity. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html. Example: { “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “{{ .ProviderARN }}” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringEquals”: { “{{ .ProviderName }}:sub”: {{ .ServiceAccounts }} } } } ] }

AWSKMSARN is an ARN value referencing a role appropriate for managing the auth via the AWS KMS key.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “kms:Encrypt”, “kms:Decrypt”, “kms:ReEncrypt”, “kms:GenerateDataKey”, “kms:DescribeKey” ], “Resource”: %q } ] }

ARN is the Amazon Resource Name for the encryption key

Region contains the AWS region

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

Auth defines metadata about the management of credentials used to interact with AWS KMS

InstanceType is an ec2 instance type for node instances (e.g. m5.large).

InstanceProfile is the AWS EC2 instance profile, which is a container for an IAM role that the EC2 instance uses.

Subnet is the subnet to use for node instances.

AMI is the image id to use for node instances. If unspecified, the default is chosen based on the NodePool release payload image.

SecurityGroups is an optional set of security groups to associate with node instances.

RootVolume specifies configuration for the root volume of node instances.

ResourceTags is an optional list of additional tags to apply to AWS node instances.

These will be merged with HostedCluster scoped tags, and HostedCluster tags take precedence in case of conflicts.

See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html for information on tagging AWS resources. AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags available for the user.

Region is the AWS region in which the cluster resides. This configures the OCP control plane cloud integrations, and is used by NodePool to resolve the correct boot AMI for a given release.

CloudProviderConfig specifies AWS networking configuration for the control plane. This is mainly used for cloud provider controller config: https://github.com/kubernetes/kubernetes/blob/f5be5052e3d0808abb904aebd3218fe4a5c2dd82/staging/src/k8s.io/legacy-cloud-providers/aws/aws.go#L1347-L1364 TODO(dan): should this be named AWSNetworkConfig?

ServiceEndpoints specifies optional custom endpoints which will override the default service endpoint of specific AWS Services.

There must be only one ServiceEndpoint for a given service name.

RolesRef contains references to various AWS IAM roles required to enable integrations such as OIDC.

ResourceTags is a list of additional tags to apply to AWS resources created for the cluster. See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html for information on tagging AWS resources. AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags available for the user.

EndpointAccess specifies the publishing scope of cluster endpoints. The default is Public.

AdditionalAllowedPrincipals specifies a list of additional allowed principal ARNs to be added to the hosted control plane’s VPC Endpoint Service to enable additional VPC Endpoint connection requests to be automatically accepted. See https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html for more details around VPC Endpoint Service allowed principals.

MultiArch specifies whether the Hosted Cluster will be expected to support NodePools with different CPU architectures, i.e., supporting arm64 NodePools and supporting amd64 NodePools on the same Hosted Cluster.

DefaultWorkerSecurityGroupID is the ID of a security group created by the control plane operator. It is always added to worker machines in addition to any security groups specified in the NodePool.

ID of resource

Filters is a set of key/value pairs used to identify a resource They are applied according to the rules defined by the AWS API: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html

Key is the key of the tag.

Value is the value of the tag.

Some AWS service do not support empty values. Since tags are added to resources in many services, the length of the tag value must meet the requirements of all services.

The referenced role must have a trust relationship that allows it to be assumed via web identity. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html. Example: { “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “{{ .ProviderARN }}” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringEquals”: { “{{ .ProviderName }}:sub”: {{ .ServiceAccounts }} } } } ] }

IngressARN is an ARN value referencing a role appropriate for the Ingress Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “elasticloadbalancing:DescribeLoadBalancers”, “tag:GetResources”, “route53:ListHostedZones” ], “Resource”: “*” }, { “Effect”: “Allow”, “Action”: [ “route53:ChangeResourceRecordSets” ], “Resource”: [ “arn:aws:route53:::PUBLIC_ZONE_ID”, “arn:aws:route53:::PRIVATE_ZONE_ID” ] } ] }

ImageRegistryARN is an ARN value referencing a role appropriate for the Image Registry Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “s3:CreateBucket”, “s3:DeleteBucket”, “s3:PutBucketTagging”, “s3:GetBucketTagging”, “s3:PutBucketPublicAccessBlock”, “s3:GetBucketPublicAccessBlock”, “s3:PutEncryptionConfiguration”, “s3:GetEncryptionConfiguration”, “s3:PutLifecycleConfiguration”, “s3:GetLifecycleConfiguration”, “s3:GetBucketLocation”, “s3:ListBucket”, “s3:GetObject”, “s3:PutObject”, “s3:DeleteObject”, “s3:ListBucketMultipartUploads”, “s3:AbortMultipartUpload”, “s3:ListMultipartUploadParts” ], “Resource”: “*” } ] }

StorageARN is an ARN value referencing a role appropriate for the Storage Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:AttachVolume”, “ec2:CreateSnapshot”, “ec2:CreateTags”, “ec2:CreateVolume”, “ec2:DeleteSnapshot”, “ec2:DeleteTags”, “ec2:DeleteVolume”, “ec2:DescribeInstances”, “ec2:DescribeSnapshots”, “ec2:DescribeTags”, “ec2:DescribeVolumes”, “ec2:DescribeVolumesModifications”, “ec2:DetachVolume”, “ec2:ModifyVolume” ], “Resource”: “*” } ] }

NetworkARN is an ARN value referencing a role appropriate for the Network Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:DescribeInstances”, “ec2:DescribeInstanceStatus”, “ec2:DescribeInstanceTypes”, “ec2:UnassignPrivateIpAddresses”, “ec2:AssignPrivateIpAddresses”, “ec2:UnassignIpv6Addresses”, “ec2:AssignIpv6Addresses”, “ec2:DescribeSubnets”, “ec2:DescribeNetworkInterfaces” ], “Resource”: “*” } ] }

KubeCloudControllerARN is an ARN value referencing a role appropriate for the KCM/KCC. Source: https://cloud-provider-aws.sigs.k8s.io/prerequisites/#iam-policies

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Action”: [ “autoscaling:DescribeAutoScalingGroups”, “autoscaling:DescribeLaunchConfigurations”, “autoscaling:DescribeTags”, “ec2:DescribeAvailabilityZones”, “ec2:DescribeInstances”, “ec2:DescribeImages”, “ec2:DescribeRegions”, “ec2:DescribeRouteTables”, “ec2:DescribeSecurityGroups”, “ec2:DescribeSubnets”, “ec2:DescribeVolumes”, “ec2:CreateSecurityGroup”, “ec2:CreateTags”, “ec2:CreateVolume”, “ec2:ModifyInstanceAttribute”, “ec2:ModifyVolume”, “ec2:AttachVolume”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:CreateRoute”, “ec2:DeleteRoute”, “ec2:DeleteSecurityGroup”, “ec2:DeleteVolume”, “ec2:DetachVolume”, “ec2:RevokeSecurityGroupIngress”, “ec2:DescribeVpcs”, “elasticloadbalancing:AddTags”, “elasticloadbalancing:AttachLoadBalancerToSubnets”, “elasticloadbalancing:ApplySecurityGroupsToLoadBalancer”, “elasticloadbalancing:CreateLoadBalancer”, “elasticloadbalancing:CreateLoadBalancerPolicy”, “elasticloadbalancing:CreateLoadBalancerListeners”, “elasticloadbalancing:ConfigureHealthCheck”, “elasticloadbalancing:DeleteLoadBalancer”, “elasticloadbalancing:DeleteLoadBalancerListeners”, “elasticloadbalancing:DescribeLoadBalancers”, “elasticloadbalancing:DescribeLoadBalancerAttributes”, “elasticloadbalancing:DetachLoadBalancerFromSubnets”, “elasticloadbalancing:DeregisterInstancesFromLoadBalancer”, “elasticloadbalancing:ModifyLoadBalancerAttributes”, “elasticloadbalancing:RegisterInstancesWithLoadBalancer”, “elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer”, “elasticloadbalancing:AddTags”, “elasticloadbalancing:CreateListener”, “elasticloadbalancing:CreateTargetGroup”, “elasticloadbalancing:DeleteListener”, “elasticloadbalancing:DeleteTargetGroup”, “elasticloadbalancing:DeregisterTargets”, “elasticloadbalancing:DescribeListeners”, “elasticloadbalancing:DescribeLoadBalancerPolicies”, “elasticloadbalancing:DescribeTargetGroups”, “elasticloadbalancing:DescribeTargetHealth”, “elasticloadbalancing:ModifyListener”, “elasticloadbalancing:ModifyTargetGroup”, “elasticloadbalancing:RegisterTargets”, “elasticloadbalancing:SetLoadBalancerPoliciesOfListener”, “iam:CreateServiceLinkedRole”, “kms:DescribeKey” ], “Resource”: [ “*” ], “Effect”: “Allow” } ] }

NodePoolManagementARN is an ARN value referencing a role appropriate for the CAPI Controller.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Action”: [ “ec2:AssociateRouteTable”, “ec2:AttachInternetGateway”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:CreateInternetGateway”, “ec2:CreateNatGateway”, “ec2:CreateRoute”, “ec2:CreateRouteTable”, “ec2:CreateSecurityGroup”, “ec2:CreateSubnet”, “ec2:CreateTags”, “ec2:DeleteInternetGateway”, “ec2:DeleteNatGateway”, “ec2:DeleteRouteTable”, “ec2:DeleteSecurityGroup”, “ec2:DeleteSubnet”, “ec2:DeleteTags”, “ec2:DescribeAccountAttributes”, “ec2:DescribeAddresses”, “ec2:DescribeAvailabilityZones”, “ec2:DescribeImages”, “ec2:DescribeInstances”, “ec2:DescribeInternetGateways”, “ec2:DescribeNatGateways”, “ec2:DescribeNetworkInterfaces”, “ec2:DescribeNetworkInterfaceAttribute”, “ec2:DescribeRouteTables”, “ec2:DescribeSecurityGroups”, “ec2:DescribeSubnets”, “ec2:DescribeVpcs”, “ec2:DescribeVpcAttribute”, “ec2:DescribeVolumes”, “ec2:DetachInternetGateway”, “ec2:DisassociateRouteTable”, “ec2:DisassociateAddress”, “ec2:ModifyInstanceAttribute”, “ec2:ModifyNetworkInterfaceAttribute”, “ec2:ModifySubnetAttribute”, “ec2:RevokeSecurityGroupIngress”, “ec2:RunInstances”, “ec2:TerminateInstances”, “tag:GetResources”, “ec2:CreateLaunchTemplate”, “ec2:CreateLaunchTemplateVersion”, “ec2:DescribeLaunchTemplates”, “ec2:DescribeLaunchTemplateVersions”, “ec2:DeleteLaunchTemplate”, “ec2:DeleteLaunchTemplateVersions” ], “Resource”: [ “” ], “Effect”: “Allow” }, { “Condition”: { “StringLike”: { “iam:AWSServiceName”: “elasticloadbalancing.amazonaws.com” } }, “Action”: [ “iam:CreateServiceLinkedRole” ], “Resource”: [ “arn::iam:::role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing” ], “Effect”: “Allow” }, { “Action”: [ “iam:PassRole” ], “Resource”: [ “arn::iam:::role/-worker-role” ], “Effect”: “Allow” }, { “Effect”: “Allow”, “Action”: [ “kms:Decrypt”, “kms:ReEncrypt”, “kms:GenerateDataKeyWithoutPlainText”, “kms:DescribeKey” ], “Resource”: “” }, { “Effect”: “Allow”, “Action”: [ “kms:CreateGrant” ], “Resource”: “”, “Condition”: { “Bool”: { “kms:GrantIsForAWSResource”: true } } } ] }

ControlPlaneOperatorARN is an ARN value referencing a role appropriate for the Control Plane Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:CreateVpcEndpoint”, “ec2:DescribeVpcEndpoints”, “ec2:ModifyVpcEndpoint”, “ec2:DeleteVpcEndpoints”, “ec2:CreateTags”, “route53:ListHostedZones”, “ec2:CreateSecurityGroup”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:AuthorizeSecurityGroupEgress”, “ec2:DeleteSecurityGroup”, “ec2:RevokeSecurityGroupIngress”, “ec2:RevokeSecurityGroupEgress”, “ec2:DescribeSecurityGroups”, “ec2:DescribeVpcs”, ], “Resource”: “*” }, { “Effect”: “Allow”, “Action”: [ “route53:ChangeResourceRecordSets”, “route53:ListResourceRecordSets” ], “Resource”: “arn:aws:route53:::%s” } ] }

Name is the name of the AWS service. This must be provided and cannot be empty.

URL is fully qualified URI with scheme https, that overrides the default generated endpoint for a client. This must be provided and cannot be empty.

AgentLabelSelector contains labels that must be set on an Agent in order to be selected for a Machine.

AgentNamespace is the namespace where to search for Agents for this cluster

"HighlyAvailable"

HighlyAvailable means components should be resilient to problems across fault boundaries as defined by the component to which the policy is attached. This usually means running critical workloads with 3 replicas and with little or no toleration of disruption of the component.

"SingleReplica"

SingleReplica means components are not expected to be resilient to problems across most fault boundaries associated with high availability. This usually means running critical workloads with just 1 replica and with toleration of full disruption of the component.

Location contains the Azure region

KeyVaultName is the name of the keyvault. Must match criteria specified at https://docs.microsoft.com/en-us/azure/key-vault/general/about-keys-secrets-certificates#vault-name-and-object-name Your Microsoft Entra application used to create the cluster must be authorized to access this keyvault, e.g using the AzureCLI: az keyvault set-policy -n $KEYVAULT_NAME --key-permissions decrypt encrypt --spn <YOUR APPLICATION CLIENT ID>

KeyName is the name of the keyvault key used for encrypt/decrypt

KeyVersion contains the version of the key to use

ImageID is the id of the image to boot from. If unset, the default image at the location below will be used: subscription/$subscriptionID/resourceGroups/$resourceGroupName/providers/Microsoft.Compute/images/rhcos.x86_64.vhd

DiskSizeGB is the size in GB to assign to the OS disk CAPZ default is 30GB, https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/b3708019a67ff19407b87d63c402af94ca4246f6/api/v1beta1/types.go#L599

DiskStorageAccountType is the disk storage account type to use. Valid values are: * Standard_LRS: HDD * StandardSSD_LRS: Standard SSD * Premium_LRS: Premium SDD * UltraSSD_LRS: Ultra SDD

Defaults to Premium_LRS. For more details, visit the Azure documentation: https://docs.microsoft.com/en-us/azure/virtual-machines/disks-types#disk-type-comparison

AvailabilityZone of the nodepool. Must not be specified for clusters in a location that does not support AvailabilityZone.

DiskEncryptionSetID is the ID of the DiskEncryptionSet resource to use to encrypt the OS disks for the VMs.

EnableEphemeralOSDisk enables ephemeral OS disk

SubnetName is the name of the subnet to place the Nodes into

The cloud environment identifier, valid values could be found here: https://github.com/Azure/go-autorest/blob/4c0e21ca2bbb3251fe7853e6f9df6397f53dd419/autorest/azure/environments.go#L33

MaxNodesTotal is the maximum allowable number of nodes across all NodePools for a HostedCluster. The autoscaler will not grow the cluster beyond this number.

MaxPodGracePeriod is the maximum seconds to wait for graceful pod termination before scaling down a NodePool. The default is 600 seconds.

MaxNodeProvisionTime is the maximum time to wait for node provisioning before considering the provisioning to be unsuccessful, expressed as a Go duration string. The default is 15 minutes.

PodPriorityThreshold enables users to schedule “best-effort” pods, which shouldn’t trigger autoscaler actions, but only run when there are spare resources available. The default is -10.

See the following for more details: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-does-cluster-autoscaler-work-with-pod-priority-and-preemption

APIServer holds configuration (like serving certificates, client CA and CORS domains) shared by all API servers in the system, among them especially kube-apiserver and openshift-apiserver.

Authentication specifies cluster-wide settings for authentication (like OAuth and webhook token authenticators).

FeatureGate holds cluster-wide information about feature gates.

Image governs policies related to imagestream imports and runtime configuration for external registries. It allows cluster admins to configure which registries OpenShift is allowed to import images from, extra CA trust bundles for external registries, and policies to block or allow registry hostnames. When exposing OpenShift’s image registry to the public, this also lets cluster admins specify the external hostname.

Ingress holds cluster-wide information about ingress, including the default ingress domain used for routes.

Network holds cluster-wide information about the network. It is used to configure the desired network configuration, such as: IP address pools for services/pod IPs, network plugin, etc. Please view network.spec for an explanation on what applies when configuring this resource. TODO (csrwng): Add validation here to exclude changes that conflict with networking settings in the HostedCluster.Spec.Networking field.

OAuth holds cluster-wide information about OAuth. It is used to configure the integrated OAuth server. This configuration is only honored when the top level Authentication config has type set to IntegratedOAuth.

OperatorHub specifies the configuration for the Operator Lifecycle Manager in the HostedCluster. This is only configured at deployment time but the controller are not reconcilling over it. The OperatorHub configuration will be constantly reconciled if catalog placement is management, but only on cluster creation otherwise.

Scheduler holds cluster-wide config information to run the Kubernetes Scheduler and influence its placement decisions. The canonical name for this config is cluster.

Proxy holds cluster-wide information on how to configure default proxies for the cluster.

CIDR is the IP block address pool.

HostPrefix is the prefix size to allocate to each node from the CIDR. For example, 24 would allocate 2^8=256 adresses to each node. If this field is not used by the plugin, it can be left unset.

MachineNetwork is the list of IP address pools for machines.

ClusterNetwork is the list of IP address pools for pods.

ServiceNetwork is the list of IP address pools for services. NOTE: currently only one entry is supported.

NetworkType specifies the SDN provider used for cluster networking.

APIServer contains advanced network settings for the API server that affect how the APIServer is exposed inside a cluster node.

desired is the version that the cluster is reconciling towards. If the cluster is not yet fully initialized desired will be set with the information available, which may be an image or a tag.

history contains a list of the most recent versions applied to the cluster. This value may be empty during cluster startup, and then will be updated when a new update is being applied. The newest update is first in the list and it is ordered by recency. Updates in the history have state Completed if the rollout completed - if an update was failing or halfway applied the state will be Partial. Only a limited amount of update history is preserved.

observedGeneration reports which version of the spec is being synced. If this value is not equal to metadata.generation, then the desired and conditions fields may represent a previous version.

availableUpdates contains updates recommended for this cluster. Updates which appear in conditionalUpdates but not in availableUpdates may expose this cluster to known issues. This list may be empty if no updates are recommended, if the update service is unavailable, or if an invalid channel has been specified.

conditionalUpdates contains the list of updates that may be recommended for this cluster if it meets specific required conditions. Consumers interested in the set of updates that are actually recommended for this cluster should use availableUpdates. This list may be empty if no updates are recommended, if the update service is unavailable, or if an empty or invalid channel has been specified.

"AWSDefaultSecurityGroupCreated"

AWSDefaultSecurityGroupCreated indicates whether the default security group for AWS workers has been created. A failure here indicates that NodePools without a security group will be blocked from creating machines.

"AWSDefaultSecurityGroupDeleted"

AWSDefaultSecurityGroupDeleted indicates whether the default security group for AWS workers has been deleted. A failure here indicates that the Security Group has some dependencies that there are still pending cloud resources to be deleted that are using that SG.

"AWSEndpointAvailable"

AWSEndpointServiceAvailable indicates whether the AWS Endpoint has been created in the guest VPC

"AWSEndpointServiceAvailable"

AWSEndpointServiceAvailable indicates whether the AWS Endpoint Service has been created for the specified NLB in the management VPC

"CVOScaledDown"

"CloudResourcesDestroyed"

CloudResourcesDestroyed bubbles up the same condition from HCP. It signals if the cloud provider infrastructure created by Kubernetes in the consumer cloud provider account was destroyed. A failure here may require external user intervention to resolve. E.g. cloud provider perms were corrupted. E.g. the guest cluster was broken and kube resource deletion that affects cloud infra like service type load balancer can’t succeed.

"ClusterVersionAvailable"

ClusterVersionAvailable bubbles up Failing configv1.OperatorAvailable from the CVO.

"ClusterVersionFailing"

ClusterVersionFailing bubbles up Failing from the CVO.

"ClusterVersionProgressing"

ClusterVersionProgressing bubbles up configv1.OperatorProgressing from the CVO.

"ClusterVersionReleaseAccepted"

ClusterVersionReleaseAccepted bubbles up Failing ReleaseAccepted from the CVO.

"ClusterVersionSucceeding"

ClusterVersionSucceeding indicates the current status of the desired release version of the HostedCluster as indicated by the Failing condition in the underlying cluster’s ClusterVersion.

"ClusterVersionUpgradeable"

ClusterVersionUpgradeable indicates the Upgradeable condition in the underlying cluster’s ClusterVersion.

"EtcdAvailable"

EtcdAvailable bubbles up the same condition from HCP. It signals if etcd is available. A failure here often means a software bug or a non-stable cluster.

"EtcdSnapshotRestored"

"ExternalDNSReachable"

ExternalDNSReachable bubbles up the same condition from HCP. It signals if the configured external DNS is reachable. A failure here requires external user intervention to resolve. E.g. changing the external DNS domain or making sure the domain is created and registered correctly.

"Available"

HostedClusterAvailable indicates whether the HostedCluster has a healthy control plane. When this is false for too long and there’s no clear indication in the “Reason”, please check the remaining more granular conditions.

"Degraded"

HostedClusterDegraded indicates whether the HostedCluster is encountering an error that may require user intervention to resolve.

"HostedClusterDestroyed"

HostedClusterDestroyed indicates that a hosted has finished destroying and that it is waiting for a destroy grace period to go away. The grace period is determined by the hypershift.openshift.io/destroy-grace-period annotation in the HostedCluster if present.

"Progressing"

HostedClusterProgressing indicates whether the HostedCluster is attempting an initial deployment or upgrade. When this is false for too long and there’s no clear indication in the “Reason”, please check the remaining more granular conditions.

"Available"

"Degraded"

"IgnitionEndpointAvailable"

IgnitionEndpointAvailable indicates whether the ignition server for the HostedCluster is available to handle ignition requests. A failure here often means a software bug or a non-stable cluster.

"IgnitionServerValidReleaseInfo"

IgnitionServerValidReleaseInfo indicates if the release contains all the images used by the local ignition provider and reports missing images if any.

"InfrastructureReady"

InfrastructureReady bubbles up the same condition from HCP. It signals if the infrastructure for a control plane to be operational, e.g. load balancers were created successfully. A failure here may require external user intervention to resolve. E.g. hitting quotas on the cloud provider.

"KubeAPIServerAvailable"

KubeAPIServerAvailable bubbles up the same condition from HCP. It signals if the kube API server is available. A failure here often means a software bug or a non-stable cluster.

"PlatformCredentialsFound"

PlatformCredentialsFound indicates that credentials required for the desired platform are valid. A failure here is unlikely to resolve without the changing user input.

"ReconciliationActive"

ReconciliationActive indicates if reconciliation of the HostedCluster is active or paused hostedCluster.spec.pausedUntil.

"ReconciliationSucceeded"

ReconciliationSucceeded indicates if the HostedCluster reconciliation succeeded. A failure here often means a software bug or a non-stable cluster.

"SupportedHostedCluster"

SupportedHostedCluster indicates whether a HostedCluster is supported by the current configuration of the hypershift-operator. e.g. If HostedCluster requests endpointAcess Private but the hypershift-operator is running on a management cluster outside AWS or is not configured with AWS credentials, the HostedCluster is not supported. A failure here is unlikely to resolve without the changing user input.

"UnmanagedEtcdAvailable"

UnmanagedEtcdAvailable indicates whether a user-managed etcd cluster is healthy.

"ValidAWSIdentityProvider"

ValidAWSIdentityProvider indicates if the Identity Provider referenced in the cloud credentials is healthy. E.g. for AWS the idp ARN is referenced in the iam roles. “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “{{ .ProviderARN }}” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringEquals”: { “{{ .ProviderName }}:sub”: {{ .ServiceAccounts }} } } } ]

A failure here may require external user intervention to resolve.

"ValidAWSKMSConfig"

ValidAWSKMSConfig indicates whether the AWS KMS role and encryption key are valid and operational A failure here indicates that the role or the key are invalid, or the role doesn’t have access to use the key.

"ValidAzureKMSConfig"

ValidAzureKMSConfig indicates whether the given KMS input for the Azure platform is valid and operational A failure here indicates that the input is invalid, or permissions are missing to use the encryption key.

"ValidConfiguration"

ValidHostedClusterConfiguration signals if the hostedCluster input is valid and supported by the underlying management cluster. A failure here is unlikely to resolve without the changing user input.

"ValidHostedControlPlaneConfiguration"

ValidHostedControlPlaneConfiguration bubbles up the same condition from HCP. It signals if the hostedControlPlane input is valid and supported by the underlying management cluster. A failure here is unlikely to resolve without the changing user input.

"ValidKubeVirtInfraNetworkMTU"

ValidKubeVirtInfraNetworkMTU indicates if the MTU configured on an infra cluster hosting a guest cluster utilizing kubevirt platform is a sufficient value that will avoid performance degradation due to fragmentation of the double encapsulation in ovn-kubernetes

"ValidOIDCConfiguration"

ValidOIDCConfiguration indicates if an AWS cluster’s OIDC condition is detected as invalid. A failure here may require external user intervention to resolve. E.g. oidc was deleted out of band.

"ValidReleaseImage"

ValidReleaseImage indicates if the release image set in the spec is valid for the HostedCluster. For example, this can be set false if the HostedCluster itself attempts an unsupported version before 4.9 or an unsupported upgrade e.g y-stream upgrade before 4.11. A failure here is unlikely to resolve without the changing user input.

"ValidReleaseInfo"

ValidReleaseInfo bubbles up the same condition from HCP. It indicates if the release contains all the images used by hypershift and reports missing images if any.

BaseDomain is the base domain of the cluster.

BaseDomainPrefix is the base domain prefix of the cluster. defaults to clusterName if not set. Set it to “” if you don’t want a prefix to be prepended to BaseDomain.

PublicZoneID is the Hosted Zone ID where all the DNS records that are publicly accessible to the internet exist.

PrivateZoneID is the Hosted Zone ID where all the DNS records that are only available internally to the cluster exist.

"Managed"

Managed means HyperShift should provision and operator the etcd cluster automatically.

"Unmanaged"

Unmanaged means HyperShift will not provision or manage the etcd cluster, and the user is responsible for doing so.

ManagementType defines how the etcd cluster is managed.

Managed specifies the behavior of an etcd cluster managed by HyperShift.

Unmanaged specifies configuration which enables the control plane to integrate with an eternally managed etcd cluster.

ClientSecret refers to a secret for client mTLS authentication with the etcd cluster. It may have the following key/value pairs:

etcd-client-ca.crt: Certificate Authority value
etcd-client.crt: Client certificate value
etcd-client.key: Client certificate key value

Name of the filter. Filter names are case-sensitive.

Values includes one or more filter values. Filter values are case-sensitive.

Release specifies the desired OCP release payload for the hosted cluster.

Updating this field will trigger a rollout of the control plane. The behavior of the rollout will be driven by the ControllerAvailabilityPolicy and InfrastructureAvailabilityPolicy.

ControlPlaneRelease specifies the desired OCP release payload for control plane components running on the management cluster. Updating this field will trigger a rollout of the control plane. The behavior of the rollout will be driven by the ControllerAvailabilityPolicy and InfrastructureAvailabilityPolicy. If not defined, Release is used

ClusterID uniquely identifies this cluster. This is expected to be an RFC4122 UUID value (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx in hexadecimal values). As with a Kubernetes metadata.uid, this ID uniquely identifies this cluster in space and time. This value identifies the cluster in metrics pushed to telemetry and metrics produced by the control plane operators. If a value is not specified, an ID is generated. After initial creation, the value is immutable.

updateService may be used to specify the preferred upstream update service. By default it will use the appropriate update service for the cluster and region.

channel is an identifier for explicitly requesting that a non-default set of updates be applied to this cluster. The default channel will be contain stable updates that are appropriate for production clusters.

InfraID is a globally unique identifier for the cluster. This identifier will be used to associate various cloud resources with the HostedCluster and its associated NodePools.

Platform specifies the underlying infrastructure provider for the cluster and is used to configure platform specific behavior.

ControllerAvailabilityPolicy specifies the availability policy applied to critical control plane components. The default value is HighlyAvailable.

InfrastructureAvailabilityPolicy specifies the availability policy applied to infrastructure services which run on cluster nodes. The default value is SingleReplica.

DNS specifies DNS configuration for the cluster.

Networking specifies network configuration for the cluster.

Autoscaling specifies auto-scaling behavior that applies to all NodePools associated with the control plane.

Etcd specifies configuration for the control plane etcd cluster. The default ManagementType is Managed. Once set, the ManagementType cannot be changed.

Services specifies how individual control plane services are published from the hosting cluster of the control plane.

If a given service is not present in this list, it will be exposed publicly by default.

PullSecret references a pull secret to be injected into the container runtime of all cluster nodes. The secret must have a key named “.dockerconfigjson” whose value is the pull secret JSON.

SSHKey references an SSH key to be injected into all cluster node sshd servers. The secret must have a single key “id_rsa.pub” whose value is the public part of an SSH key.

IssuerURL is an OIDC issuer URL which is used as the issuer in all ServiceAccount tokens generated by the control plane API server. The default value is kubernetes.default.svc, which only works for in-cluster validation.

ServiceAccountSigningKey is a reference to a secret containing the private key used by the service account token issuer. The secret is expected to contain a single key named “key”. If not specified, a service account signing key will be generated automatically for the cluster. When specifying a service account signing key, a IssuerURL must also be specified.

Configuration specifies configuration for individual OCP components in the cluster, represented as embedded resources that correspond to the openshift configuration API.

AuditWebhook contains metadata for configuring an audit webhook endpoint for a cluster to process cluster audit events. It references a secret that contains the webhook information for the audit webhook endpoint. It is a secret because if the endpoint has mTLS the kubeconfig will contain client keys. The kubeconfig needs to be stored in the secret with a secret key name that corresponds to the constant AuditWebhookKubeconfigKey.

ImageContentSources specifies image mirrors that can be used by cluster nodes to pull content.

AdditionalTrustBundle is a reference to a ConfigMap containing a PEM-encoded X.509 certificate bundle that will be added to the hosted controlplane and nodes

SecretEncryption specifies a Kubernetes secret encryption strategy for the control plane.

FIPS indicates whether this cluster’s nodes will be running in FIPS mode. If set to true, the control plane’s ignition server will be configured to expect that nodes joining the cluster will be FIPS-enabled.

PausedUntil is a field that can be used to pause reconciliation on a resource. Either a date can be provided in RFC3339 format or a boolean. If a date is provided: reconciliation is paused on the resource until that date. If the boolean true is provided: reconciliation is paused on the resource until the field is removed.

OLMCatalogPlacement specifies the placement of OLM catalog components. By default, this is set to management and OLM catalog components are deployed onto the management cluster. If set to guest, the OLM catalog components will be deployed onto the guest cluster.

NodeSelector when specified, must be true for the pods managed by the HostedCluster to be scheduled.

Version is the status of the release version applied to the HostedCluster.

KubeConfig is a reference to the secret containing the default kubeconfig for the cluster.

KubeadminPassword is a reference to the secret that contains the initial kubeadmin user password for the guest cluster.

IgnitionEndpoint is the endpoint injected in the ign config userdata. It exposes the config for instances to become kubernetes nodes.

ControlPlaneEndpoint contains the endpoint information by which external clients can access the control plane. This is populated after the infrastructure is ready.

OAuthCallbackURLTemplate contains a template for the URL to use as a callback for identity providers. The [identity-provider-name] placeholder must be replaced with the name of an identity provider defined on the HostedCluster. This is populated after the infrastructure is ready.

Conditions represents the latest available observations of a control plane’s current state.

Platform contains platform-specific status of the HostedCluster

ReleaseImage is the release image applied to the hosted control plane.

ControlPlaneReleaseImage specifies the desired OCP release payload for control plane components running on the management cluster. If not defined, ReleaseImage is used

updateService may be used to specify the preferred upstream update service. By default it will use the appropriate update service for the cluster and region.

channel is an identifier for explicitly requesting that a non-default set of updates be applied to this cluster. The default channel will be contain stable updates that are appropriate for production clusters.

IssuerURL is an OIDC issuer URL which is used as the issuer in all ServiceAccount tokens generated by the control plane API server. The default value is kubernetes.default.svc, which only works for in-cluster validation.

Networking specifies network configuration for the cluster. Temporarily optional for backward compatibility, required in future releases.

ClusterID is the unique id that identifies the cluster externally. Making it optional here allows us to keep compatibility with previous versions of the control-plane-operator that have no knowledge of this field.

ServiceAccountSigningKey is a reference to a secret containing the private key used by the service account token issuer. The secret is expected to contain a single key named “key”. If not specified, a service account signing key will be generated automatically for the cluster.

ControllerAvailabilityPolicy specifies the availability policy applied to critical control plane components. The default value is SingleReplica.

InfrastructureAvailabilityPolicy specifies the availability policy applied to infrastructure services which run on cluster nodes. The default value is SingleReplica.

FIPS specifies if the nodes for the cluster will be running in FIPS mode

KubeConfig specifies the name and key for the kubeconfig secret

Services defines metadata about how control plane services are published in the management cluster.

AuditWebhook contains metadata for configuring an audit webhook endpoint for a cluster to process cluster audit events. It references a secret that contains the webhook information for the audit webhook endpoint. It is a secret because if the endpoint has MTLS the kubeconfig will contain client keys. This is currently only supported in IBM Cloud. The kubeconfig needs to be stored in the secret with a secret key name that corresponds to the constant AuditWebhookKubeconfigKey.

Etcd contains metadata about the etcd cluster the hypershift managed Openshift control plane components use to store data.

Configuration embeds resources that correspond to the openshift configuration API: https://docs.openshift.com/container-platform/4.7/rest_api/config_apis/config-apis-index.html

ImageContentSources lists sources/repositories for the release-image content.

AdditionalTrustBundle references a ConfigMap containing a PEM-encoded X.509 certificate bundle

SecretEncryption contains metadata about the kubernetes secret encryption strategy being used for the cluster when applicable.

PausedUntil is a field that can be used to pause reconciliation on a resource. Either a date can be provided in RFC3339 format or a boolean. If a date is provided: reconciliation is paused on the resource until that date. If the boolean true is provided: reconciliation is paused on the resource until the field is removed.

OLMCatalogPlacement specifies the placement of OLM catalog components. By default, this is set to management and OLM catalog components are deployed onto the management cluster. If set to guest, the OLM catalog components will be deployed onto the guest cluster.

Autoscaling specifies auto-scaling behavior that applies to all NodePools associated with the control plane.

NodeSelector when specified, must be true for the pods managed by the HostedCluster to be scheduled.

Ready denotes that the HostedControlPlane API Server is ready to receive requests This satisfies CAPI contract https://github.com/kubernetes-sigs/cluster-api/blob/cd3a694deac89d5ebeb888307deaa61487207aa0/controllers/cluster_controller_phases.go#L226-L230

Initialized denotes whether or not the control plane has provided a kubeadm-config. Once this condition is marked true, its value is never changed. See the Ready condition for an indication of the current readiness of the cluster’s control plane. This satisfies CAPI contract https://github.com/kubernetes-sigs/cluster-api/blob/cd3a694deac89d5ebeb888307deaa61487207aa0/controllers/cluster_controller_phases.go#L238-L252

ExternalManagedControlPlane indicates to cluster-api that the control plane is managed by an external service. https://github.com/kubernetes-sigs/cluster-api/blob/65e5385bffd71bf4aad3cf34a537f11b217c7fab/controllers/machine_controller.go#L468

ControlPlaneEndpoint contains the endpoint information by which external clients can access the control plane. This is populated after the infrastructure is ready.

OAuthCallbackURLTemplate contains a template for the URL to use as a callback for identity providers. The [identity-provider-name] placeholder must be replaced with the name of an identity provider defined on the HostedCluster. This is populated after the infrastructure is ready.

versionStatus is the status of the release version applied by the hosted control plane operator.

Version is the semantic version of the release applied by the hosted control plane operator

Deprecated: Use versionStatus.desired.version instead.

ReleaseImage is the release image applied to the hosted control plane.

Deprecated: Use versionStatus.desired.image instead.

lastReleaseImageTransitionTime is the time of the last update to the current releaseImage property.

Deprecated: Use versionStatus.history[0].startedTime instead.

KubeConfig is a reference to the secret containing the default kubeconfig for this control plane.

KubeadminPassword is a reference to the secret containing the initial kubeadmin password for the guest cluster.

Condition contains details for one aspect of the current state of the HostedControlPlane. Current condition types are: “Available”

Platform contains platform-specific status of the HostedCluster

NodeCount tracks the number of nodes in the HostedControlPlane.

Type defines the IBM Cloud KMS authentication strategy

Unmanaged defines the auth metadata the customer provides to interact with IBM Cloud KMS

Managed defines metadata around the service to service authentication strategy for the IBM Cloud KMS system (all provider managed).

"Managed"

IBMCloudKMSManagedAuth defines the KMS authentication strategy where the IKS/ROKS platform uses service to service auth to call IBM Cloud KMS APIs (no customer credentials requried)

"Unmanaged"

IBMCloudKMSUnmanagedAuth defines the KMS authentication strategy where a customer supplies IBM Cloud authentication to interact with IBM Cloud KMS APIs

CRKID is the customer rook key id

InstanceID is the id for the key protect instance

CorrelationID is an identifier used to track all api call usage from hypershift

URL is the url to call key protect apis over

KeyVersion is a unique number associated with the key. The number increments whenever a new key is enabled for data encryption.

Region is the IBM Cloud region

Auth defines metadata for how authentication is done with IBM Cloud KMS

KeyList defines the list of keys used for data encryption

Credentials should reference a secret with a key field of IBMCloudIAMAPIKeySecretKey that contains a apikey to call IBM Cloud KMS APIs

ProviderType is a specific supported infrastructure provider within IBM Cloud.

Source is the repository that users refer to, e.g. in image pull specifications.

Mirrors are one or more repositories that may also contain the same images.

MaxUnavailable is the maximum number of nodes that can be unavailable during the update.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding down.

Defaults to 1.

Example: when this is set to 30%, a max of 30% of the nodes can be made unschedulable/unavailable immediately when the update starts. Once a set of nodes is updated, more nodes can be made unschedulable for update, ensuring that the total number of nodes schedulable at all times during the update is at least 70% of desired nodes.

"AWS"

"Azure"

"IBMCloud"

Provider defines the KMS provider

IBMCloud defines metadata for the IBM Cloud KMS encryption strategy

AWS defines metadata about the configuration of the AWS KMS Secret Encryption provider

Azure defines metadata about the configuration of the Azure KMS Secret Encryption provider using Azure key vault

CacheName holds the name of the cache DataVolume, if exists

Credentials shows the client credentials used when creating KubeVirt virtual machines. This filed is only exists when the KubeVirt virtual machines are being placed on a cluster separate from the one hosting the Hosted Control Plane components.

The default behavior when Credentials is not defined is for the KubeVirt VMs to be placed on the same cluster and namespace as the Hosted Control Plane.

Type is the type of the caching strategy

"None"

KubevirtCachingStrategyNone means that hypershift will not cache the boot image

"PVC"

KubevirtCachingStrategyPVC means that hypershift will cache the boot image into a PVC; only relevant when using a QCOW boot image, and is ignored when using a container image

Memory represents how much guest memory the VM should have

Cores represents how many cores the guest VM should have

QosClass If set to “Guaranteed”, requests the scheduler to place the VirtualMachineInstance on a node with limit memory and CPU, equal to be the requested values, to set the VMI as a Guaranteed QoS Class; See here for more details: https://kubevirt.io/user-guide/operations/node_overcommit/#requesting-the-right-qos-class-for-virtualmachineinstances

ContainerDiskImage is a string representing the container image that holds the root disk

StorageClassMapping maps StorageClasses on the infra cluster hosting the KubeVirt VMs to StorageClasses that are made available within the Guest Cluster.

NOTE: It is possible that not all capablities of an infra cluster’s storageclass will be present for the corresponding guest clusters storageclass.

Name specify the network attached to the nodes it is a value with the format “[namespace]/[name]” to reference the multus network attachment definition

RootVolume represents values associated with the VM volume that will host rhcos

Compute contains values representing the virtual hardware requested for the VM

NetworkInterfaceMultiQueue If set to “Enable”, virtual network interfaces configured with a virtio bus will also enable the vhost multiqueue feature for network devices. The number of queues created depends on additional factors of the VirtualMachineInstance, like the number of guest CPUs.

AdditionalNetworks specify the extra networks attached to the nodes

AttachDefaultNetwork specify if the default pod network should be attached to the nodes this can only be set to false if AdditionalNetworks are configured

NodeSelector is a selector which must be true for the kubevirt VirtualMachine to fit on a node. Selector which must match a node’s labels for the VM to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

Size is the size of the persistent storage volume

StorageClass is the storageClass used for the underlying PVC that hosts the volume

AccessModes is an array that contains the desired Access Modes the root volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

VolumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.

InfraKubeConfigSecret is a reference to a secret that contains the kubeconfig for the external infra cluster that will be used to host the KubeVirt virtual machines for this cluster.

InfraNamespace defines the namespace on the external infra cluster that is used to host the KubeVirt virtual machines. This namespace must already exist before creating the HostedCluster and the kubeconfig referenced in the InfraKubeConfigSecret must have access to manage the required resources within this namespace.

BaseDomainPassthrough toggles whether or not an automatically generated base domain for the guest cluster should be used that is a subdomain of the management cluster’s *.apps DNS.

For the KubeVirt platform, the basedomain can be autogenerated using the *.apps domain of the management/infra hosting cluster This makes the guest cluster’s base domain a subdomain of the hypershift infra/mgmt cluster’s base domain.

Example: Infra/Mgmt cluster’s DNS Base: example.com Cluster: mgmt-cluster.example.com Apps: *.apps.mgmt-cluster.example.com KubeVirt Guest cluster’s DNS Base: apps.mgmt-cluster.example.com Cluster: guest.apps.mgmt-cluster.example.com Apps: *.apps.guest.apps.mgmt-cluster.example.com

This is possible using OCP wildcard routes

GenerateID is used to uniquely apply a name suffix to resources associated with kubevirt infrastructure resources

Credentials defines the client credentials used when creating KubeVirt virtual machines. Defining credentials is only necessary when the KubeVirt virtual machines are being placed on a cluster separate from the one hosting the Hosted Control Plane components.

The default behavior when Credentials is not defined is for the KubeVirt VMs to be placed on the same cluster and namespace as the Hosted Control Plane.

StorageDriver defines how the KubeVirt CSI driver exposes StorageClasses on the infra cluster (hosting the VMs) to the guest cluster.

Image represents what rhcos image to use for the node pool

(Members of KubevirtVolume are embedded into this type.)

KubevirtVolume represents of type of storage to run the image on

CacheStrategy defines the boot image caching strategy. Default - no caching

Group contains which group this mapping belongs to.

InfraStorageClassName is the name of the infra cluster storage class that will be exposed to the guest.

GuestStorageClassName is the name that the corresponding storageclass will be called within the guest cluster

"Default"

DefaultKubevirtStorageDriverConfigType means the kubevirt storage driver maps to the underlying infra cluster’s default storageclass

"Manual"

ManualKubevirtStorageDriverConfigType means the kubevirt storage driver mapping is explicitly defined.

"None"

NoneKubevirtStorageDriverConfigType means no kubevirt storage driver is used

Type represents the type of kubevirt csi driver configuration to use

Manual is used to explicilty define how the infra storageclasses are mapped to guest storageclasses

Type represents the type of storage to associate with the kubevirt VMs.

Persistent volume type means the VM’s storage is backed by a PVC VMs that use persistent volumes can survive disruption events like restart and eviction This is the default type used when no storage type is defined.

Group contains which group this mapping belongs to.

InfraStorageClassName is the name of the infra cluster volume snapshot class that will be exposed to the guest.

GuestVolumeSnapshotClassName is the name that the corresponding volumeSnapshotClass will be called within the guest cluster

"Persistent"

KubevirtVolumeTypePersistent represents persistent volume for kubevirt VMs

Hostname is the name of the DNS record that will be created pointing to the LoadBalancer.

CIDR is the IP block address pool for machines within the cluster.

Storage specifies how etcd data is persisted.

Type is the kind of persistent storage implementation to use for etcd.

PersistentVolume is the configuration for PersistentVolume etcd storage. With this implementation, a PersistentVolume will be allocated for every etcd member (either 1 or 3 depending on the HostedCluster control plane availability configuration).

RestoreSnapshotURL allows an optional URL to be provided where an etcd snapshot can be downloaded, for example a pre-signed URL referencing a storage service. This snapshot will be restored on initial startup, only when the etcd PV is empty.

"PersistentVolume"

PersistentVolumeEtcdStorage uses PersistentVolumes for etcd storage.

"Disable"

"Enable"

"Calico"

Calico specifies Calico as the SDN provider

"OVNKubernetes"

OVNKubernetes specifies OVN as the SDN provider

"OpenShiftSDN"

OpenShiftSDN specifies OpenShiftSDN as the SDN provider

"Other"

Other specifies an undefined SDN provider

Min is the minimum number of nodes to maintain in the pool. Must be >= 1.

Max is the maximum number of nodes allowed in the pool. Must be >= 1.

Type of condition in CamelCase or in foo.example.com/CamelCase. Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be useful (see .node.status.conditions), the ability to deconflict is important.

Status of the condition, one of True, False, Unknown.

Severity provides an explicit classification of Reason code, so the users or machines can immediately understand the current situation and act accordingly. The Severity field MUST be set only when Status=False.

Last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.

The reason for the condition’s last transition in CamelCase. The specific API may choose whether or not this field is considered a guaranteed API. This field may not be empty.

A human readable message indicating details about the transition. This field may be empty.

UpgradeType specifies the type of strategy for handling upgrades.

Replace is the configuration for rolling upgrades.

InPlace is the configuration for in-place upgrades.

AutoRepair specifies whether health checks should be enabled for machines in the NodePool. The default is false.

Type specifies the platform name.

AWS specifies the configuration used when operating on AWS.

IBMCloud defines IBMCloud specific settings for components

Kubevirt specifies the configuration used when operating on KubeVirt platform.

Agent specifies the configuration used when using Agent platform.

PowerVS specifies the configuration used when using IBMCloud PowerVS platform.

KubeVirt contains the KubeVirt platform statuses

ClusterName is the name of the HostedCluster this NodePool belongs to.

TODO(dan): Should this be a LocalObjectReference?

Release specifies the OCP release used for the NodePool. This informs the ignition configuration for machines, as well as other platform specific machine properties (e.g. an AMI on the AWS platform).

Platform specifies the underlying infrastructure provider for the NodePool and is used to configure platform specific behavior.

Replicas is the desired number of nodes the pool should maintain. If unset, the default value is 0.

Management specifies behavior for managing nodes in the pool, such as upgrade strategies and auto-repair behaviors.

Autoscaling specifies auto-scaling behavior for the NodePool.

Config is a list of references to ConfigMaps containing serialized MachineConfig resources to be injected into the ignition configurations of nodes in the NodePool. The MachineConfig API schema is defined here:

https://github.com/openshift/machine-config-operator/blob/18963e4f8fe66e8c513ca4b131620760a414997f/pkg/apis/machineconfiguration.openshift.io/v1/types.go#L185

Each ConfigMap must have a single key named “config” whose value is the JSON or YAML of a serialized Resource for machineconfiguration.openshift.io: KubeletConfig ContainerRuntimeConfig MachineConfig ClusterImagePolicy ImageContentSourcePolicy or ImageDigestMirrorSet

NodeDrainTimeout is the total amount of time that the controller will spend on draining a node. The default value is 0, meaning that the node can be drained without any time limitations. NOTE: NodeDrainTimeout is different from kubectl drain --timeout TODO (alberto): Today changing this field will trigger a recreate rolling update, which kind of defeats the purpose of the change. In future we plan to propagate this field in-place. https://github.com/kubernetes-sigs/cluster-api/issues/5880

NodeLabels propagates a list of labels to Nodes, only once on creation. Valid values are those in https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

Taints if specified, propagates a list of taints to Nodes, only once on creation.

PausedUntil is a field that can be used to pause reconciliation on a resource. Either a date can be provided in RFC3339 format or a boolean. If a date is provided: reconciliation is paused on the resource until that date. If the boolean true is provided: reconciliation is paused on the resource until the field is removed.

TuningConfig is a list of references to ConfigMaps containing serialized Tuned or PerformanceProfile resources to define the tuning configuration to be applied to nodes in the NodePool. The Tuned API is defined here:

https://github.com/openshift/cluster-node-tuning-operator/blob/2c76314fb3cc8f12aef4a0dcd67ddc3677d5b54f/pkg/apis/tuned/v1/tuned_types.go

The PerformanceProfile API is defined here: https://github.com/openshift/cluster-node-tuning-operator/tree/b41042d42d4ba5bb2e99960248cf1d6ae4935018/pkg/apis/performanceprofile/v2

Each ConfigMap must have a single key named “tuning” whose value is the JSON or YAML of a serialized Tuned or PerformanceProfile.

Arch is the preferred processor architecture for the NodePool (currently only supported on AWS) NOTE: This is set as optional to prevent validation from failing due to a limitation on client side validation with open API machinery: https://github.com/kubernetes/kubernetes/issues/108768#issuecomment-1253912215 TODO Add ppc64le and s390x to enum validation once the architectures are supported

Replicas is the latest observed number of nodes in the pool.

Version is the semantic version of the latest applied release specified by the NodePool.

Platform hols the specific statuses

Conditions represents the latest available observations of the node pool’s current state.

Address is the host/ip that the NodePort service is exposed over.

Port is the port of the NodePort service. If <=0, the port is dynamically assigned when the service is created.

"guest"

GuestOLMCatalogPlacement indicates OLM catalog components will be placed in the guest cluster.

"management"

ManagementOLMCatalogPlacement indicates OLM catalog components will be placed in the management cluster.

StorageClassName is the StorageClass of the data volume for each etcd member.

See https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1.

Size is the minimum size of the data volume for each etcd member.

Type is the type of infrastructure provider for the cluster.

AWS specifies configuration for clusters running on Amazon Web Services.

Agent specifies configuration for agent-based installations.

IBMCloud defines IBMCloud specific settings for components

Azure defines azure specific settings

PowerVS specifies configuration for clusters running on IBMCloud Power VS Service. This field is immutable. Once set, It can’t be changed.

KubeVirt defines KubeVirt specific settings for cluster components.

"AWS"

AWSPlatform represents Amazon Web Services infrastructure.

"Agent"

AgentPlatform represents user supplied insfrastructure booted with agents.

"Azure"

AzurePlatform represents Azure infrastructure.

"IBMCloud"

IBMCloudPlatform represents IBM Cloud infrastructure.

"KubeVirt"

KubevirtPlatform represents Kubevirt infrastructure.

"None"

NonePlatform represents user supplied (e.g. bare metal) infrastructure.

"PowerVS"

PowerVSPlatform represents PowerVS infrastructure.

SystemType is the System type used to host the instance. systemType determines the number of cores and memory that is available. Few of the supported SystemTypes are s922,e880,e980. e880 systemType available only in Dallas Datacenters. e980 systemType available in Datacenters except Dallas and Washington. When omitted, this means that the user has no opinion and the platform is left to choose a reasonable default. The current default is s922 which is generally available.

ProcessorType is the VM instance processor type. It must be set to one of the following values: Dedicated, Capped or Shared.

Dedicated: resources are allocated for a specific client, The hypervisor makes a 1:1 binding of a partition’s processor to a physical processor core. Shared: Shared among other clients. Capped: Shared, but resources do not expand beyond those that are requested, the amount of CPU time is Capped to the value specified for the entitlement.

if the processorType is selected as Dedicated, then Processors value cannot be fractional. When omitted, this means that the user has no opinion and the platform is left to choose a reasonable default. The current default is shared.

Processors is the number of virtual processors in a virtual machine. when the processorType is selected as Dedicated the processors value cannot be fractional. maximum value for the Processors depends on the selected SystemType. when SystemType is set to e880 or e980 maximum Processors value is 143. when SystemType is set to s922 maximum Processors value is 15. minimum value for Processors depends on the selected ProcessorType. when ProcessorType is set as Shared or Capped, The minimum processors is 0.5. when ProcessorType is set as Dedicated, The minimum processors is 1. When omitted, this means that the user has no opinion and the platform is left to choose a reasonable default. The default is set based on the selected ProcessorType. when ProcessorType selected as Dedicated, the default is set to 1. when ProcessorType selected as Shared or Capped, the default is set to 0.5.

MemoryGiB is the size of a virtual machine’s memory, in GiB. maximum value for the MemoryGiB depends on the selected SystemType. when SystemType is set to e880 maximum MemoryGiB value is 7463 GiB. when SystemType is set to e980 maximum MemoryGiB value is 15307 GiB. when SystemType is set to s922 maximum MemoryGiB value is 942 GiB. The minimum memory is 32 GiB.

When omitted, this means the user has no opinion and the platform is left to choose a reasonable default. The current default is 32.

Image used for deploying the nodes. If unspecified, the default is chosen based on the NodePool release payload image.

StorageType for the image and nodes, this will be ignored if Image is specified. The storage tiers in PowerVS are based on I/O operations per second (IOPS). It means that the performance of your storage volumes is limited to the maximum number of IOPS based on volume size and storage tier. Although, the exact numbers might change over time, the Tier 3 storage is currently set to 3 IOPS/GB, and the Tier 1 storage is currently set to 10 IOPS/GB.

The default is tier1

ImageDeletePolicy is policy for the image deletion.

delete: delete the image from the infrastructure. retain: delete the image from the openshift but retain in the infrastructure.

The default is delete

"capped"

PowerVSNodePoolCappedProcType defines capped processor type

"dedicated"

PowerVSNodePoolDedicatedProcType defines dedicated processor type

"shared"

PowerVSNodePoolSharedProcType defines shared processor type

AccountID is the IBMCloud account id. This field is immutable. Once set, It can’t be changed.

CISInstanceCRN is the IBMCloud CIS Service Instance’s Cloud Resource Name This field is immutable. Once set, It can’t be changed.

ResourceGroup is the IBMCloud Resource Group in which the cluster resides. This field is immutable. Once set, It can’t be changed.

Region is the IBMCloud region in which the cluster resides. This configures the OCP control plane cloud integrations, and is used by NodePool to resolve the correct boot image for a given release. This field is immutable. Once set, It can’t be changed.

Zone is the availability zone where control plane cloud resources are created. This field is immutable. Once set, It can’t be changed.

Subnet is the subnet to use for control plane cloud resources. This field is immutable. Once set, It can’t be changed.

ServiceInstance is the reference to the Power VS service on which the server instance(VM) will be created. Power VS service is a container for all Power VS instances at a specific geographic region. serviceInstance can be created via IBM Cloud catalog or CLI. ServiceInstanceID is the unique identifier that can be obtained from IBM Cloud UI or IBM Cloud cli.

More detail about Power VS service instance. https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-creating-power-virtual-server

This field is immutable. Once set, It can’t be changed.

VPC specifies IBM Cloud PowerVS Load Balancing configuration for the control plane. This field is immutable. Once set, It can’t be changed.

KubeCloudControllerCreds is a reference to a secret containing cloud credentials with permissions matching the cloud controller policy. This field is immutable. Once set, It can’t be changed.

TODO(dan): document the “cloud controller policy”

NodePoolManagementCreds is a reference to a secret containing cloud credentials with permissions matching the node pool management policy. This field is immutable. Once set, It can’t be changed.

TODO(dan): document the “node pool management policy”

IngressOperatorCloudCreds is a reference to a secret containing ibm cloud credentials for ingress operator to get authenticated with ibm cloud.

StorageOperatorCloudCreds is a reference to a secret containing ibm cloud credentials for storage operator to get authenticated with ibm cloud.

ImageRegistryOperatorCloudCreds is a reference to a secret containing ibm cloud credentials for image registry operator to get authenticated with ibm cloud.

ID of resource

Name of resource

Name for VPC to used for all the service load balancer. This field is immutable. Once set, It can’t be changed.

Region is the IBMCloud region in which VPC gets created, this VPC used for all the ingress traffic into the OCP cluster. This field is immutable. Once set, It can’t be changed.

Zone is the availability zone where load balancer cloud resources are created. This field is immutable. Once set, It can’t be changed.

Subnet is the subnet to use for load balancer. This field is immutable. Once set, It can’t be changed.

"Burstable"

"Guaranteed"

Image is the image pullspec of an OCP release payload image.

Strategy is the node replacement strategy for nodes in the pool.

RollingUpdate specifies a rolling update strategy which upgrades nodes by creating new nodes and deleting the old ones.

MaxUnavailable is the maximum number of nodes that can be unavailable during the update.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding down.

This can not be 0 if MaxSurge is 0.

Defaults to 0.

Example: when this is set to 30%, old nodes can be deleted down to 70% of desired nodes immediately when the rolling update starts. Once new nodes are ready, more old nodes be deleted, followed by provisioning new nodes, ensuring that the total number of nodes available at all times during the update is at least 70% of desired nodes.

MaxSurge is the maximum number of nodes that can be provisioned above the desired number of nodes.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding up.

This can not be 0 if MaxUnavailable is 0.

Defaults to 1.

Example: when this is set to 30%, new nodes can be provisioned immediately when the rolling update starts, such that the total number of old and new nodes do not exceed 130% of desired nodes. Once old nodes have been deleted, new nodes can be provisioned, ensuring that total number of nodes running at any time during the update is at most 130% of desired nodes.

Hostname is the name of the DNS record that will be created pointing to the Route.

Type defines the type of kube secret encryption being used

KMS defines metadata about the kms secret encryption strategy

AESCBC defines metadata about the AESCBC secret encryption strategy

"aescbc"

AESCBC uses AES-CBC with PKCS#7 padding to do secret encryption

"kms"

KMS integrates with a cloud provider’s key management service to do secret encryption

CIDR is the IP block address pool for services within the cluster.

Type is the publishing strategy used for the service.

NodePort configures exposing a service using a NodePort.

LoadBalancer configures exposing a service using a LoadBalancer.

Route configures exposing a service using a Route.

Service identifies the type of service being published.

ServicePublishingStrategy specifies how to publish Service.

Required. The taint key to be applied to a node.

The taint value corresponding to the taint key.

Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.

Endpoint is the full etcd cluster client endpoint URL. For example:

https://etcd-client:2379

If the URL uses an HTTPS scheme, the TLS field is required.

TLS specifies TLS configuration for HTTPS etcd client endpoints.

"OnDelete"

UpgradeStrategyOnDelete replaces old nodes when the deletion of the associated node instances are completed.

"RollingUpdate"

UpgradeStrategyRollingUpdate means use a rolling update for nodes.

"InPlace"

UpgradeTypeInPlace is a strategy which replaces nodes in-place with no additional node capacity requirements.

"Replace"

UpgradeTypeReplace is a strategy which replaces nodes using surge node capacity.

Size specifies size (in Gi) of the storage device.

Must be greater than the image snapshot size or 8 (whichever is greater).

Type is the type of the volume.

IOPS is the number of IOPS requested for the disk. This is only valid for type io1.

Encrypted is whether the volume should be encrypted or not.

EncryptionKey is the KMS key to use to encrypt the volume. Can be either a KMS key ID or ARN. If Encrypted is set and this is omitted, the default AWS key will be used. The key must already exist and be accessible by the controller.