HyperShift API Reference

Spec is the desired behavior of the HostedCluster.

Status is the latest observed status of the HostedCluster.

Spec is the desired behavior of the NodePool.

Status is the latest observed status of the NodePool.

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

advertiseAddress is the address that pods within the nodes will use to talk to the API server. This is an address associated with the loopback adapter of each node. If not specified, the controller will take default values. The default values will be set as 172.20.0.1 or fd00::1. This value is immutable.

port is the port at which the APIServer is exposed inside a node. Other pods using host networking cannot listen on this port. If ommited 6443 is used. This is useful to choose a port other than the default one which might interfere with customer environments e.g. https://github.com/openshift/hypershift/pull/356. Setting this to 443 is possible only for backward compatibility reasons and it’s discouraged. Doing so, it would result in the controller overriding the KAS endpoint in the guest cluster having a discrepancy with the KAS Pod and potentially causing temporarily network failures. This value is immutable.

allowedCIDRBlocks is an allow list of CIDR blocks that can access the APIServer If not specified, traffic is allowed from all addresses. This depends on underlying support by the cloud provider for Service LoadBalancerSourceRanges

Subnet is the subnet to use for control plane cloud resources.

Zone is the availability zone where control plane cloud resources are created.

VPC is the VPC to use for control plane cloud resources.

"Private"

Private endpoint access allows only private API server access and private node communication with the control plane.

"Public"

Public endpoint access allows public API server access and public node communication with the control plane.

"PublicAndPrivate"

PublicAndPrivate endpoint access allows public API server access and private node communication with the control plane.

The referenced role must have a trust relationship that allows it to be assumed via web identity. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html. Example: { “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “{{ .ProviderARN }}” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringEquals”: { “{{ .ProviderName }}:sub”: {{ .ServiceAccounts }} } } } ] }

AWSKMSARN is an ARN value referencing a role appropriate for managing the auth via the AWS KMS key.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “kms:Encrypt”, “kms:Decrypt”, “kms:ReEncrypt”, “kms:GenerateDataKey”, “kms:DescribeKey” ], “Resource”: %q } ] }

ARN is the Amazon Resource Name for the encryption key

Region contains the AWS region

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

Auth defines metadata about the management of credentials used to interact with AWS KMS

InstanceType is an ec2 instance type for node instances (e.g. m5.large).

InstanceProfile is the AWS EC2 instance profile, which is a container for an IAM role that the EC2 instance uses.

Subnet is the subnet to use for node instances.

AMI is the image id to use for node instances. If unspecified, the default is chosen based on the NodePool release payload image.

SecurityGroups is an optional set of security groups to associate with node instances.

RootVolume specifies configuration for the root volume of node instances.

ResourceTags is an optional list of additional tags to apply to AWS node instances.

These will be merged with HostedCluster scoped tags, and HostedCluster tags take precedence in case of conflicts.

See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html for information on tagging AWS resources. AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags available for the user.

placement specifies the placement options for the EC2 instances.

Region is the AWS region in which the cluster resides. This configures the OCP control plane cloud integrations, and is used by NodePool to resolve the correct boot AMI for a given release.

CloudProviderConfig specifies AWS networking configuration for the control plane. This is mainly used for cloud provider controller config: https://github.com/kubernetes/kubernetes/blob/f5be5052e3d0808abb904aebd3218fe4a5c2dd82/staging/src/k8s.io/legacy-cloud-providers/aws/aws.go#L1347-L1364 TODO(dan): should this be named AWSNetworkConfig?

ServiceEndpoints specifies optional custom endpoints which will override the default service endpoint of specific AWS Services.

There must be only one ServiceEndpoint for a given service name.

RolesRef contains references to various AWS IAM roles required to enable integrations such as OIDC.

ResourceTags is a list of additional tags to apply to AWS resources created for the cluster. See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html for information on tagging AWS resources. AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags available for the user.

EndpointAccess specifies the publishing scope of cluster endpoints. The default is Public.

AdditionalAllowedPrincipals specifies a list of additional allowed principal ARNs to be added to the hosted control plane’s VPC Endpoint Service to enable additional VPC Endpoint connection requests to be automatically accepted. See https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html for more details around VPC Endpoint Service allowed principals.

MultiArch specifies whether the Hosted Cluster will be expected to support NodePools with different CPU architectures, i.e., supporting arm64 NodePools and supporting amd64 NodePools on the same Hosted Cluster. Deprecated: This field is no longer used. The HyperShift Operator now performs multi-arch validations automatically despite the platform type. The HyperShift Operator will set HostedCluster.Status.PayloadArch based on the HostedCluster release image. This field is used by the NodePool controller to validate the NodePool.Spec.Arch is supported.

SharedVPC contains fields that must be specified if the HostedCluster must use a VPC that is created in a different AWS account and is shared with the AWS account where the HostedCluster will be created.

DefaultWorkerSecurityGroupID is the ID of a security group created by the control plane operator. It is always added to worker machines in addition to any security groups specified in the NodePool.

ID of resource

Filters is a set of key/value pairs used to identify a resource They are applied according to the rules defined by the AWS API: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html

Key is the key of the tag.

Value is the value of the tag.

Some AWS service do not support empty values. Since tags are added to resources in many services, the length of the tag value must meet the requirements of all services.

The referenced role must have a trust relationship that allows it to be assumed via web identity. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc.html. Example: { “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “{{ .ProviderARN }}” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringEquals”: { “{{ .ProviderName }}:sub”: {{ .ServiceAccounts }} } } } ] }

IngressARN is an ARN value referencing a role appropriate for the Ingress Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “elasticloadbalancing:DescribeLoadBalancers”, “tag:GetResources”, “route53:ListHostedZones” ], “Resource”: “*” }, { “Effect”: “Allow”, “Action”: [ “route53:ChangeResourceRecordSets” ], “Resource”: [ “arn:aws:route53:::PUBLIC_ZONE_ID”, “arn:aws:route53:::PRIVATE_ZONE_ID” ] } ] }

ImageRegistryARN is an ARN value referencing a role appropriate for the Image Registry Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “s3:CreateBucket”, “s3:DeleteBucket”, “s3:PutBucketTagging”, “s3:GetBucketTagging”, “s3:PutBucketPublicAccessBlock”, “s3:GetBucketPublicAccessBlock”, “s3:PutEncryptionConfiguration”, “s3:GetEncryptionConfiguration”, “s3:PutLifecycleConfiguration”, “s3:GetLifecycleConfiguration”, “s3:GetBucketLocation”, “s3:ListBucket”, “s3:GetObject”, “s3:PutObject”, “s3:DeleteObject”, “s3:ListBucketMultipartUploads”, “s3:AbortMultipartUpload”, “s3:ListMultipartUploadParts” ], “Resource”: “*” } ] }

StorageARN is an ARN value referencing a role appropriate for the Storage Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:AttachVolume”, “ec2:CreateSnapshot”, “ec2:CreateTags”, “ec2:CreateVolume”, “ec2:DeleteSnapshot”, “ec2:DeleteTags”, “ec2:DeleteVolume”, “ec2:DescribeInstances”, “ec2:DescribeSnapshots”, “ec2:DescribeTags”, “ec2:DescribeVolumes”, “ec2:DescribeVolumesModifications”, “ec2:DetachVolume”, “ec2:ModifyVolume” ], “Resource”: “*” } ] }

NetworkARN is an ARN value referencing a role appropriate for the Network Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:DescribeInstances”, “ec2:DescribeInstanceStatus”, “ec2:DescribeInstanceTypes”, “ec2:UnassignPrivateIpAddresses”, “ec2:AssignPrivateIpAddresses”, “ec2:UnassignIpv6Addresses”, “ec2:AssignIpv6Addresses”, “ec2:DescribeSubnets”, “ec2:DescribeNetworkInterfaces” ], “Resource”: “*” } ] }

KubeCloudControllerARN is an ARN value referencing a role appropriate for the KCM/KCC. Source: https://cloud-provider-aws.sigs.k8s.io/prerequisites/#iam-policies

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Action”: [ “autoscaling:DescribeAutoScalingGroups”, “autoscaling:DescribeLaunchConfigurations”, “autoscaling:DescribeTags”, “ec2:DescribeAvailabilityZones”, “ec2:DescribeInstances”, “ec2:DescribeImages”, “ec2:DescribeRegions”, “ec2:DescribeRouteTables”, “ec2:DescribeSecurityGroups”, “ec2:DescribeSubnets”, “ec2:DescribeVolumes”, “ec2:CreateSecurityGroup”, “ec2:CreateTags”, “ec2:CreateVolume”, “ec2:ModifyInstanceAttribute”, “ec2:ModifyVolume”, “ec2:AttachVolume”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:CreateRoute”, “ec2:DeleteRoute”, “ec2:DeleteSecurityGroup”, “ec2:DeleteVolume”, “ec2:DetachVolume”, “ec2:RevokeSecurityGroupIngress”, “ec2:DescribeVpcs”, “elasticloadbalancing:AddTags”, “elasticloadbalancing:AttachLoadBalancerToSubnets”, “elasticloadbalancing:ApplySecurityGroupsToLoadBalancer”, “elasticloadbalancing:CreateLoadBalancer”, “elasticloadbalancing:CreateLoadBalancerPolicy”, “elasticloadbalancing:CreateLoadBalancerListeners”, “elasticloadbalancing:ConfigureHealthCheck”, “elasticloadbalancing:DeleteLoadBalancer”, “elasticloadbalancing:DeleteLoadBalancerListeners”, “elasticloadbalancing:DescribeLoadBalancers”, “elasticloadbalancing:DescribeLoadBalancerAttributes”, “elasticloadbalancing:DetachLoadBalancerFromSubnets”, “elasticloadbalancing:DeregisterInstancesFromLoadBalancer”, “elasticloadbalancing:ModifyLoadBalancerAttributes”, “elasticloadbalancing:RegisterInstancesWithLoadBalancer”, “elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer”, “elasticloadbalancing:AddTags”, “elasticloadbalancing:CreateListener”, “elasticloadbalancing:CreateTargetGroup”, “elasticloadbalancing:DeleteListener”, “elasticloadbalancing:DeleteTargetGroup”, “elasticloadbalancing:DeregisterTargets”, “elasticloadbalancing:DescribeListeners”, “elasticloadbalancing:DescribeLoadBalancerPolicies”, “elasticloadbalancing:DescribeTargetGroups”, “elasticloadbalancing:DescribeTargetHealth”, “elasticloadbalancing:ModifyListener”, “elasticloadbalancing:ModifyTargetGroup”, “elasticloadbalancing:RegisterTargets”, “elasticloadbalancing:SetLoadBalancerPoliciesOfListener”, “iam:CreateServiceLinkedRole”, “kms:DescribeKey” ], “Resource”: [ “*” ], “Effect”: “Allow” } ] }

NodePoolManagementARN is an ARN value referencing a role appropriate for the CAPI Controller.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Action”: [ “ec2:AssociateRouteTable”, “ec2:AttachInternetGateway”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:CreateInternetGateway”, “ec2:CreateNatGateway”, “ec2:CreateRoute”, “ec2:CreateRouteTable”, “ec2:CreateSecurityGroup”, “ec2:CreateSubnet”, “ec2:CreateTags”, “ec2:DeleteInternetGateway”, “ec2:DeleteNatGateway”, “ec2:DeleteRouteTable”, “ec2:DeleteSecurityGroup”, “ec2:DeleteSubnet”, “ec2:DeleteTags”, “ec2:DescribeAccountAttributes”, “ec2:DescribeAddresses”, “ec2:DescribeAvailabilityZones”, “ec2:DescribeImages”, “ec2:DescribeInstances”, “ec2:DescribeInternetGateways”, “ec2:DescribeNatGateways”, “ec2:DescribeNetworkInterfaces”, “ec2:DescribeNetworkInterfaceAttribute”, “ec2:DescribeRouteTables”, “ec2:DescribeSecurityGroups”, “ec2:DescribeSubnets”, “ec2:DescribeVpcs”, “ec2:DescribeVpcAttribute”, “ec2:DescribeVolumes”, “ec2:DetachInternetGateway”, “ec2:DisassociateRouteTable”, “ec2:DisassociateAddress”, “ec2:ModifyInstanceAttribute”, “ec2:ModifyNetworkInterfaceAttribute”, “ec2:ModifySubnetAttribute”, “ec2:RevokeSecurityGroupIngress”, “ec2:RunInstances”, “ec2:TerminateInstances”, “tag:GetResources”, “ec2:CreateLaunchTemplate”, “ec2:CreateLaunchTemplateVersion”, “ec2:DescribeLaunchTemplates”, “ec2:DescribeLaunchTemplateVersions”, “ec2:DeleteLaunchTemplate”, “ec2:DeleteLaunchTemplateVersions” ], “Resource”: [ “” ], “Effect”: “Allow” }, { “Condition”: { “StringLike”: { “iam:AWSServiceName”: “elasticloadbalancing.amazonaws.com” } }, “Action”: [ “iam:CreateServiceLinkedRole” ], “Resource”: [ “arn::iam:::role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing” ], “Effect”: “Allow” }, { “Action”: [ “iam:PassRole” ], “Resource”: [ “arn::iam:::role/-worker-role” ], “Effect”: “Allow” }, { “Effect”: “Allow”, “Action”: [ “kms:Decrypt”, “kms:ReEncrypt”, “kms:GenerateDataKeyWithoutPlainText”, “kms:DescribeKey” ], “Resource”: “” }, { “Effect”: “Allow”, “Action”: [ “kms:CreateGrant” ], “Resource”: “”, “Condition”: { “Bool”: { “kms:GrantIsForAWSResource”: true } } } ] }

ControlPlaneOperatorARN is an ARN value referencing a role appropriate for the Control Plane Operator.

The following is an example of a valid policy document:

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:CreateVpcEndpoint”, “ec2:DescribeVpcEndpoints”, “ec2:ModifyVpcEndpoint”, “ec2:DeleteVpcEndpoints”, “ec2:CreateTags”, “route53:ListHostedZones”, “ec2:CreateSecurityGroup”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:AuthorizeSecurityGroupEgress”, “ec2:DeleteSecurityGroup”, “ec2:RevokeSecurityGroupIngress”, “ec2:RevokeSecurityGroupEgress”, “ec2:DescribeSecurityGroups”, “ec2:DescribeVpcs”, ], “Resource”: “*” }, { “Effect”: “Allow”, “Action”: [ “route53:ChangeResourceRecordSets”, “route53:ListResourceRecordSets” ], “Resource”: “arn:aws:route53:::%s” } ] }

Name is the name of the AWS service. This must be provided and cannot be empty.

URL is fully qualified URI with scheme https, that overrides the default generated endpoint for a client. This must be provided and cannot be empty.

RolesRef contains references to roles in the VPC owner account that enable a HostedCluster on a shared VPC.

LocalZoneID is the ID of the route53 hosted zone for [cluster-name].hypershift.local that is associated with the HostedCluster’s VPC and exists in the VPC owner account.

IngressARN is an ARN value referencing the role in the VPC owner account that allows the ingress operator in the cluster account to create and manage records in the private DNS hosted zone.

The referenced role must have a trust relationship that allows it to be assumed by the ingress operator role in the VPC creator account. Example: { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “Statement1”, “Effect”: “Allow”, “Principal”: { “AWS”: “arn:aws:iam::[cluster-creator-account-id]:role/[infra-id]-openshift-ingress” }, “Action”: “sts:AssumeRole” } ] }

The following is an example of the policy document for this role. (Based on https://docs.openshift.com/rosa/rosa_install_access_delete_clusters/rosa-shared-vpc-config.html#rosa-sharing-vpc-dns-and-roles_rosa-shared-vpc-config)

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “route53:ListHostedZones”, “route53:ListHostedZonesByName”, “route53:ChangeTagsForResource”, “route53:GetAccountLimit”, “route53:GetChange”, “route53:GetHostedZone”, “route53:ListTagsForResource”, “route53:UpdateHostedZoneComment”, “tag:GetResources”, “tag:UntagResources” “route53:ChangeResourceRecordSets”, “route53:ListResourceRecordSets” ], “Resource”: “*” }, ] }

ControlPlaneARN is an ARN value referencing the role in the VPC owner account that allows the control plane operator in the cluster account to create and manage a VPC endpoint, its corresponding Security Group, and DNS records in the hypershift local hosted zone.

The referenced role must have a trust relationship that allows it to be assumed by the control plane operator role in the VPC creator account. Example: { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “Statement1”, “Effect”: “Allow”, “Principal”: { “AWS”: “arn:aws:iam::[cluster-creator-account-id]:role/[infra-id]-control-plane-operator” }, “Action”: “sts:AssumeRole” } ] }

The following is an example of the policy document for this role.

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “ec2:CreateVpcEndpoint”, “ec2:DescribeVpcEndpoints”, “ec2:ModifyVpcEndpoint”, “ec2:DeleteVpcEndpoints”, “ec2:CreateTags”, “route53:ListHostedZones”, “ec2:CreateSecurityGroup”, “ec2:AuthorizeSecurityGroupIngress”, “ec2:AuthorizeSecurityGroupEgress”, “ec2:DeleteSecurityGroup”, “ec2:RevokeSecurityGroupIngress”, “ec2:RevokeSecurityGroupEgress”, “ec2:DescribeSecurityGroups”, “ec2:DescribeVpcs”, “route53:ChangeResourceRecordSets”, “route53:ListResourceRecordSets” ], “Resource”: “*” } ] }

IPAddress is the IP address of the allowed address pair. Depending on the configuration of Neutron, it may be supported to specify a CIDR instead of a specific IP address.

AgentLabelSelector contains labels that must be set on an Agent in order to be selected for a Machine.

AgentNamespace is the namespace where to search for Agents for this cluster

Start represents the start of the AllocationPool, that is the lowest IP of the pool.

End represents the end of the AlloctionPool, that is the highest IP of the pool.

"HighlyAvailable"

HighlyAvailable means components should be resilient to problems across fault boundaries as defined by the component to which the policy is attached. This usually means running critical workloads with 3 replicas and with little or no toleration of disruption of the component.

"SingleReplica"

SingleReplica means components are not expected to be resilient to problems across most fault boundaries associated with high availability. This usually means running critical workloads with just 1 replica and with toleration of full disruption of the component.

"Disabled"

"Managed"

"UserManaged"

"Ephemeral"

EphemeralDiskPersistence is the ephemeral disk type.

"Persistent"

PersistentDiskPersistence is the persistent disk type.

"Premium_LRS"

DiskStorageAccountTypesPremiumLRS - Premium SSD locally redundant storage. Best for production and performance sensitive workloads.

"PremiumV2_LRS"

DiskStorageAccountTypesPremiumV2LRS - Premium SSD v2 locally redundant storage. Best for production and performance-sensitive workloads that consistently require low latency and high IOPS and throughput.

"Standard_LRS"

DiskStorageAccountTypesStandardLRS - Standard HDD locally redundant storage. Best for backup, non-critical, and infrequent access.

"StandardSSD_LRS"

DiskStorageAccountTypesStandardSSDLRS - Standard SSD locally redundant storage. Best for web servers, lightly used enterprise applications and dev/test.

"UltraSSD_LRS"

DiskStorageAccountTypesUltraSSDLRS - Ultra SSD locally redundant storage. Best for IO-intensive workloads such as SAP HANA, top tier databases (for example, SQL, Oracle), and other transaction-heavy workloads.

KeyVaultName is the name of the keyvault. Must match criteria specified at https://docs.microsoft.com/en-us/azure/key-vault/general/about-keys-secrets-certificates#vault-name-and-object-name Your Microsoft Entra application used to create the cluster must be authorized to access this keyvault, e.g using the AzureCLI: az keyvault set-policy -n $KEYVAULT_NAME --key-permissions decrypt encrypt --spn <YOUR APPLICATION CLIENT ID>

KeyName is the name of the keyvault key used for encrypt/decrypt

KeyVersion contains the version of the key to use

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

kms is a pre-existing managed identity used to authenticate with Azure KMS.

publisher is the name of the organization that created the image. It must be between 3 and 50 characters in length, and consist of only lowercase letters, numbers, and hyphens (-) and underscores (_). It must start with a lowercase letter or a number. TODO: Can we explain where a user might find this value, or provide an example of one they might want to use

offer specifies the name of a group of related images created by the publisher. TODO: What is the valid character set for this field? What about minimum and maximum lengths?

sku specifies an instance of an offer, such as a major release of a distribution. For example, 2204-lts-gen2, 8-lvm-gen2. The value must consist only of lowercase letters, numbers, and hyphens (-) and underscores (). TODO: What about length limits?

version specifies the version of an image sku. The allowed formats are Major.Minor.Build or ‘latest’. Major, Minor, and Build are decimal numbers, e.g. ‘1.2.0’. Specify ‘latest’ to use the latest version of an image available at deployment time. Even if you use ‘latest’, the VM image will not automatically update after deploy time even if a new version becomes available.

SizeGiB is the size in GiB (1024^3 bytes) to assign to the OS disk. This should be between 16 and 65,536 when using the UltraSSD_LRS storage account type and between 16 and 32,767 when using any other storage account type. When not set, this means no opinion and the platform is left to choose a reasonable default, which is subject to change over time. The current default is 30.

diskStorageAccountType is the disk storage account type to use. Valid values are Premium_LRS, PremiumV2_LRS, Standard_LRS, StandardSSD_LRS, UltraSSD_LRS. Note that Standard means a HDD. The disk performance is tied to the disk type, please refer to the Azure documentation for further details https://docs.microsoft.com/en-us/azure/virtual-machines/disks-types#disk-type-comparison. When omitted this means no opinion and the platform is left to choose a reasonable default, which is subject to change over time. The current default is Premium SSD LRS.

encryptionSetID is the ID of the DiskEncryptionSet resource to use to encrypt the OS disks for the VMs. Configuring a DiskEncyptionSet allows greater control over the encryption of the VM OS disk at rest. Can be used with either platform (Azure) managed, or customer managed encryption keys. This needs to exist in the same subscription id listed in the Hosted Cluster, HostedCluster.Spec.Platform.Azure.SubscriptionID. DiskEncryptionSetID should also exist in a resource group under the same subscription id and the same location listed in the Hosted Cluster, HostedCluster.Spec.Platform.Azure.Location. The encryptionSetID should be in the format /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Copmute/diskEncryptionSets/{resourceName}. The subscriptionId in the encryptionSetID must be a valid UUID. It should be 5 groups of hyphen separated hexadecimal characters in the form 8-4-4-4-12. The resourceGroupName should be between 1 and 90 characters, consisting only of alphanumeric characters, hyphens, underscores, periods and paranthesis and must not end with a period (.) character. The resourceName should be between 1 and 80 characters, consisting only of alphanumeric characters, hyphens and underscores. TODO: Are there other encryption related options we may want to expose, should this be in a struct as well?

persistence determines whether the OS disk should be persisted beyond the life of the VM. Valid values are Persistent and Ephemeral. When set to Ephmeral, the OS disk will not be persisted to Azure storage and implies restrictions to the VM size and caching type. Full details can be found in the Azure documentation https://learn.microsoft.com/en-us/azure/virtual-machines/ephemeral-os-disks. Ephmeral disks are primarily used for stateless applications, provide lower latency than Persistent disks and also incur no storage costs. When not set, this means no opinion and the platform is left to choose a reasonable default, which is subject to change over time.

vmSize is the Azure VM instance type to use for the nodes being created in the nodepool. The size naming convention is documented here https://learn.microsoft.com/en-us/azure/virtual-machines/vm-naming-conventions. Size names should start with a Family name, which is represented by one of more capital letters, and then be followed by the CPU count. This is followed by 0 or more additional features, represented by a, b, d, i, l, m, p, t, s, C, and NP, refer to the Azure documentation for an explanation of these features. Optionally an accelerator such as a GPU can be added, prefixed by an underscore, for example A100, H100 or MI300X. The size may also be versioned, in which case it should be suffixed with _v where the version is a number. For example, “D32ads_v5” would be a suitable general purpose VM size, or “ND96_MI300X_v5” would represent a GPU accelerated VM.

image is used to configure the VM boot image. If unset, the default image at the location below will be used and is expected to exist: subscription//resourceGroups//providers/Microsoft.Compute/images/rhcos.x86_64.vhd. The and the are expected to be the same resource group documented in the Hosted Cluster specification respectively, HostedCluster.Spec.Platform.Azure.SubscriptionID and HostedCluster.Spec.Platform.Azure.ResourceGroupName.

osDisk provides configuration for the OS disk for the nodepool. This can be used to configure the size, storage account type, encryption options and whether the disk is persistent or ephemeral. When not provided, the platform will choose reasonable defaults which are subject to change over time. Review the fields within the osDisk for more details.

availabilityZone is the failure domain identifier where the VM should be attached to. This must not be specified for clusters in a location that does not support AvailabilityZone because it would cause a failure from Azure API.

encryptionAtHost enables encryption at host on virtual machines. According to Microsoft documentation, this means data stored on the VM host is encrypted at rest and flows encrypted to the Storage service. See https://learn.microsoft.com/en-us/azure/virtual-machines/disks-enable-host-based-encryption-portal?tabs=azure-powershell for more information.

subnetID is the subnet ID of an existing subnet where the nodes in the nodepool will be created. This can be a different subnet than the one listed in the HostedCluster, HostedCluster.Spec.Platform.Azure.SubnetID, but must exist in the same network, HostedCluster.Spec.Platform.Azure.VnetID, and must exist under the same subscription ID, HostedCluster.Spec.Platform.Azure.SubscriptionID. subnetID is immutable once set. The subnetID should be in the format /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName}. The subscriptionId in the encryptionSetID must be a valid UUID. It should be 5 groups of hyphen separated hexadecimal characters in the form 8-4-4-4-12. The resourceGroupName should be between 1 and 90 characters, consisting only of alphanumeric characters, hyphens, underscores, periods and paranthesis and must not end with a period (.) character. The vnetName should be between 2 and 64 characters, consisting only of alphanumeric characters, hyphens, underscores and periods and must not end with either a period (.) or hyphen (-) character. The subnetName should be between 1 and 80 characters, consisting only of alphanumeric characters, hyphens and underscores and must start with an alphanumeric character and must not end with a period (.) or hyphen (-) character.

diagnostics specifies the diagnostics settings for a virtual machine. If not specified, then Boot diagnostics will be disabled.

machineIdentityID is a user-assigned identity assigned to the VMs used to authenticate with Azure services. The identify is expected to exist under the same resource group as HostedCluster.Spec.Platform.Azure.ResourceGroupName. This user assigned identity is expected to have the Contributor role assigned to it and scoped to the resource group under HostedCluster.Spec.Platform.Azure.ResourceGroupName.

If this field is not supplied, the Service Principal credentials will be written to a file on the disk of each VM in order to be accessible by the cloud provider; the aforementioned credentials provided are the same ones as HostedCluster.Spec.Platform.Azure.Credentials. However, this is less secure than using a managed identity.

TODO: What is the valid character set for this field? What about minimum and maximum lengths?

Credentials is the object containing existing Azure credentials needed for creating and managing cloud infrastructure resources.

Cloud is the cloud environment identifier, valid values could be found here: https://github.com/Azure/go-autorest/blob/4c0e21ca2bbb3251fe7853e6f9df6397f53dd419/autorest/azure/environments.go#L33

Location is the Azure region in where all the cloud infrastructure resources will be created.

Example: eastus

ResourceGroupName is the name of an existing resource group where all cloud resources created by the Hosted Cluster are to be placed. The resource group is expected to exist under the same subscription as SubscriptionID.

In ARO HCP, this will be the managed resource group where customer cloud resources will be created.

Resource group naming requirements can be found here: https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.ResourceGroup.Name/.

Example: if your resource group ID is /subscriptions//resourceGroups/, your ResourceGroupName is .

VnetID is the ID of an existing VNET to use in creating VMs. The VNET can exist in a different resource group other than the one specified in ResourceGroupName, but it must exist under the same subscription as SubscriptionID.

In ARO HCP, this will be the ID of the customer provided VNET.

Example: /subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks/

subnetID is the subnet ID of an existing subnet where the nodes in the nodepool will be created. This can be a different subnet than the one listed in the HostedCluster, HostedCluster.Spec.Platform.Azure.SubnetID, but must exist in the same network, HostedCluster.Spec.Platform.Azure.VnetID, and must exist under the same subscription ID, HostedCluster.Spec.Platform.Azure.SubscriptionID. subnetID is immutable once set. The subnetID should be in the format /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName}. The subscriptionId in the encryptionSetID must be a valid UUID. It should be 5 groups of hyphen separated hexadecimal characters in the form 8-4-4-4-12. The resourceGroupName should be between 1 and 90 characters, consisting only of alphanumeric characters, hyphens, underscores, periods and paranthesis and must not end with a period (.) character. The vnetName should be between 2 and 64 characters, consisting only of alphanumeric characters, hyphens, underscores and periods and must not end with either a period (.) or hyphen (-) character. The subnetName should be between 1 and 80 characters, consisting only of alphanumeric characters, hyphens and underscores and must start with an alphanumeric character and must not end with a period (.) or hyphen (-) character.

SubscriptionID is a unique identifier for an Azure subscription used to manage resources.

SecurityGroupID is the ID of an existing security group on the SubnetID. This field is provided as part of the configuration for the Azure cloud provider, aka Azure cloud controller manager (CCM). This security group is expected to exist under the same subscription as SubscriptionID.

managedIdentities contains the managed identities needed for HCP control plane and data plane components that authenticate with Azure’s API.

controlPlane contains the client IDs of all the managed identities on the HCP control plane needing to authenticate with Azure’s API.

type is the type of image data that will be provided to the Azure VM. Valid values are “ImageID” and “AzureMarketplace”. ImageID means is used for legacy managed VM images. This is where the user uploads a VM image directly to their resource group. AzureMarketplace means the VM will boot from an Azure Marketplace image. Marketplace images are preconfigured and published by the OS vendors and may include preconfigured software for the VM.

imageID is the Azure resource ID of a VHD image to use to boot the Azure VMs from. TODO: What is the valid character set for this field? What about minimum and maximum lengths?

azureMarketplace contains the Azure Marketplace image info to use to boot the Azure VMs from.

"AzureMarketplace"

AzureMarketplace is used to specify the Azure Marketplace image info to use to boot the Azure VMs from.

"ImageID"

ImageID is the used to specify that an Azure resource ID of a VHD image is used to boot the Azure VMs from.

maxNodesTotal is the maximum allowable number of nodes for the Autoscaler scale out to be operational. The autoscaler will not grow the cluster beyond this number. If omitted, the autoscaler will not have a maximum limit. number.

maxPodGracePeriod is the maximum seconds to wait for graceful pod termination before scaling down a NodePool. The default is 600 seconds.

maxNodeProvisionTime is the maximum time to wait for node provisioning before considering the provisioning to be unsuccessful, expressed as a Go duration string. The default is 15 minutes.

podPriorityThreshold enables users to schedule “best-effort” pods, which shouldn’t trigger autoscaler actions, but only run when there are spare resources available. The default is -10.

See the following for more details: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-does-cluster-autoscaler-work-with-pod-priority-and-preemption

APIServer holds configuration (like serving certificates, client CA and CORS domains) shared by all API servers in the system, among them especially kube-apiserver and openshift-apiserver.

Authentication specifies cluster-wide settings for authentication (like OAuth and webhook token authenticators).

FeatureGate holds cluster-wide information about feature gates.

Image governs policies related to imagestream imports and runtime configuration for external registries. It allows cluster admins to configure which registries OpenShift is allowed to import images from, extra CA trust bundles for external registries, and policies to block or allow registry hostnames. When exposing OpenShift’s image registry to the public, this also lets cluster admins specify the external hostname. Changing this value will trigger a rollout for all existing NodePools in the cluster. TODO(alberto): elaborate why.

Ingress holds cluster-wide information about ingress, including the default ingress domain used for routes.

Network holds cluster-wide information about the network. It is used to configure the desired network configuration, such as: IP address pools for services/pod IPs, network plugin, etc. Please view network.spec for an explanation on what applies when configuring this resource. TODO (csrwng): Add validation here to exclude changes that conflict with networking settings in the HostedCluster.Spec.Networking field.

OAuth holds cluster-wide information about OAuth. It is used to configure the integrated OAuth server. This configuration is only honored when the top level Authentication config has type set to IntegratedOAuth.

OperatorHub specifies the configuration for the Operator Lifecycle Manager in the HostedCluster. This is only configured at deployment time but the controller are not reconcilling over it. The OperatorHub configuration will be constantly reconciled if catalog placement is management, but only on cluster creation otherwise.

Scheduler holds cluster-wide config information to run the Kubernetes Scheduler and influence its placement decisions. The canonical name for this config is cluster.

cidr is the IP block address pool.

hostPrefix is the prefix size to allocate to each node from the CIDR. For example, 24 would allocate 2^(32-24)=2^8=256 adresses to each node. If this field is not used by the plugin, it can be left unset.

machineNetwork is the list of IP address pools for machines. This might be used among other things to generate appropriate networking security groups in some clouds providers. Currently only one entry or two for dual stack is supported. This field is immutable.

clusterNetwork is the list of IP address pools for pods. Defaults to cidr: “10.132.0.0/14”. Currently only one entry is supported. This field is immutable.

serviceNetwork is the list of IP address pools for services. Defaults to cidr: “172.31.0.0/16”. Currently only one entry is supported. This field is immutable.

networkType specifies the SDN provider used for cluster networking. Defaults to OVNKubernetes. This field is required and immutable. kubebuilder:validation:XValidation:rule=“self == oldSelf”, message=“networkType is immutable”

apiServer contains advanced network settings for the API server that affect how the APIServer is exposed inside a hosted cluster node.

desired is the version that the cluster is reconciling towards. If the cluster is not yet fully initialized desired will be set with the information available, which may be an image or a tag.

history contains a list of the most recent versions applied to the cluster. This value may be empty during cluster startup, and then will be updated when a new update is being applied. The newest update is first in the list and it is ordered by recency. Updates in the history have state Completed if the rollout completed - if an update was failing or halfway applied the state will be Partial. Only a limited amount of update history is preserved.

observedGeneration reports which version of the spec is being synced. If this value is not equal to metadata.generation, then the desired and conditions fields may represent a previous version.

availableUpdates contains updates recommended for this cluster. Updates which appear in conditionalUpdates but not in availableUpdates may expose this cluster to known issues. This list may be empty if no updates are recommended, if the update service is unavailable, or if an invalid channel has been specified.

conditionalUpdates contains the list of updates that may be recommended for this cluster if it meets specific required conditions. Consumers interested in the set of updates that are actually recommended for this cluster should use availableUpdates. This list may be empty if no updates are recommended, if the update service is unavailable, or if an empty or invalid channel has been specified.

kind is the name of the resource schema.

group is the API group for this resource type.

name is the name of this resource.

"AWSDefaultSecurityGroupCreated"

AWSDefaultSecurityGroupCreated indicates whether the default security group for AWS workers has been created. A failure here indicates that NodePools without a security group will be blocked from creating machines.

"AWSDefaultSecurityGroupDeleted"

AWSDefaultSecurityGroupDeleted indicates whether the default security group for AWS workers has been deleted. A failure here indicates that the Security Group has some dependencies that there are still pending cloud resources to be deleted that are using that SG.

"AWSEndpointAvailable"

AWSEndpointServiceAvailable indicates whether the AWS Endpoint has been created in the guest VPC

"AWSEndpointServiceAvailable"

AWSEndpointServiceAvailable indicates whether the AWS Endpoint Service has been created for the specified NLB in the management VPC

"CVOScaledDown"

"CloudResourcesDestroyed"

CloudResourcesDestroyed bubbles up the same condition from HCP. It signals if the cloud provider infrastructure created by Kubernetes in the consumer cloud provider account was destroyed. A failure here may require external user intervention to resolve. E.g. cloud provider perms were corrupted. E.g. the guest cluster was broken and kube resource deletion that affects cloud infra like service type load balancer can’t succeed.

"ClusterVersionAvailable"

ClusterVersionAvailable bubbles up Failing configv1.OperatorAvailable from the CVO.

"ClusterVersionFailing"

ClusterVersionFailing bubbles up Failing from the CVO.

"ClusterVersionProgressing"

ClusterVersionProgressing bubbles up configv1.OperatorProgressing from the CVO.

"ClusterVersionReleaseAccepted"

ClusterVersionReleaseAccepted bubbles up Failing ReleaseAccepted from the CVO.

"ClusterVersionRetrievedUpdates"

ClusterVersionRetrievedUpdates bubbles up RetrievedUpdates from the CVO.

"ClusterVersionSucceeding"

ClusterVersionSucceeding indicates the current status of the desired release version of the HostedCluster as indicated by the Failing condition in the underlying cluster’s ClusterVersion.

"ClusterVersionUpgradeable"

ClusterVersionUpgradeable indicates the Upgradeable condition in the underlying cluster’s ClusterVersion.

"Available"

ControlPlaneComponentAvailable indicates whether the ControlPlaneComponent is available.

"Progressing"

ControlPlaneComponentProgressing indicates whether the ControlPlaneComponent is progressing.

"EtcdAvailable"

EtcdAvailable bubbles up the same condition from HCP. It signals if etcd is available. A failure here often means a software bug or a non-stable cluster.

"EtcdRecoveryActive"

EtcdRecoveryActive indicates that the Etcd cluster is failing and the recovery job was triggered.

"EtcdSnapshotRestored"

"ExternalDNSReachable"

ExternalDNSReachable bubbles up the same condition from HCP. It signals if the configured external DNS is reachable. A failure here requires external user intervention to resolve. E.g. changing the external DNS domain or making sure the domain is created and registered correctly.

"Available"

HostedClusterAvailable indicates whether the HostedCluster has a healthy control plane. When this is false for too long and there’s no clear indication in the “Reason”, please check the remaining more granular conditions.

"Degraded"

HostedClusterDegraded indicates whether the HostedCluster is encountering an error that may require user intervention to resolve.

"HostedClusterDestroyed"

HostedClusterDestroyed indicates that a hosted has finished destroying and that it is waiting for a destroy grace period to go away. The grace period is determined by the hypershift.openshift.io/destroy-grace-period annotation in the HostedCluster if present.

"Progressing"

HostedClusterProgressing indicates whether the HostedCluster is attempting an initial deployment or upgrade. When this is false for too long and there’s no clear indication in the “Reason”, please check the remaining more granular conditions.

"Available"

"Degraded"

"IgnitionEndpointAvailable"

IgnitionEndpointAvailable indicates whether the ignition server for the HostedCluster is available to handle ignition requests. A failure here often means a software bug or a non-stable cluster.

"IgnitionServerValidReleaseInfo"

IgnitionServerValidReleaseInfo indicates if the release contains all the images used by the local ignition provider and reports missing images if any.

"InfrastructureReady"

InfrastructureReady bubbles up the same condition from HCP. It signals if the infrastructure for a control plane to be operational, e.g. load balancers were created successfully. A failure here may require external user intervention to resolve. E.g. hitting quotas on the cloud provider.

"KubeAPIServerAvailable"

KubeAPIServerAvailable bubbles up the same condition from HCP. It signals if the kube API server is available. A failure here often means a software bug or a non-stable cluster.

"KubeVirtNodesLiveMigratable"

KubeVirtNodesLiveMigratable indicates if all nodes (VirtualMachines) of the kubevirt hosted cluster can be live migrated without experiencing a node restart

"PlatformCredentialsFound"

PlatformCredentialsFound indicates that credentials required for the desired platform are valid. A failure here is unlikely to resolve without the changing user input.

"ReconciliationActive"

ReconciliationActive indicates if reconciliation of the HostedCluster is active or paused hostedCluster.spec.pausedUntil.

"ReconciliationSucceeded"

ReconciliationSucceeded indicates if the HostedCluster reconciliation succeeded. A failure here often means a software bug or a non-stable cluster.

"SupportedHostedCluster"

SupportedHostedCluster indicates whether a HostedCluster is supported by the current configuration of the hypershift-operator. e.g. If HostedCluster requests endpointAcess Private but the hypershift-operator is running on a management cluster outside AWS or is not configured with AWS credentials, the HostedCluster is not supported. A failure here is unlikely to resolve without the changing user input.

"UnmanagedEtcdAvailable"

UnmanagedEtcdAvailable indicates whether a user-managed etcd cluster is healthy.

"ValidAWSIdentityProvider"

ValidAWSIdentityProvider indicates if the Identity Provider referenced in the cloud credentials is healthy. E.g. for AWS the idp ARN is referenced in the iam roles. “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “{{ .ProviderARN }}” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringEquals”: { “{{ .ProviderName }}:sub”: {{ .ServiceAccounts }} } } } ]

A failure here may require external user intervention to resolve.

"ValidAWSKMSConfig"

ValidAWSKMSConfig indicates whether the AWS KMS role and encryption key are valid and operational A failure here indicates that the role or the key are invalid, or the role doesn’t have access to use the key.

"ValidAzureKMSConfig"

ValidAzureKMSConfig indicates whether the given KMS input for the Azure platform is valid and operational A failure here indicates that the input is invalid, or permissions are missing to use the encryption key.

"ValidConfiguration"

ValidHostedClusterConfiguration signals if the hostedCluster input is valid and supported by the underlying management cluster. A failure here is unlikely to resolve without the changing user input.

"ValidHostedControlPlaneConfiguration"

ValidHostedControlPlaneConfiguration bubbles up the same condition from HCP. It signals if the hostedControlPlane input is valid and supported by the underlying management cluster. A failure here is unlikely to resolve without the changing user input.

"ValidIDPConfiguration"

ValidIDPConfiguration indicates if the Identity Provider configuration is valid. A failure here may require external user intervention to resolve e.g. the user-provided IDP configuration provided is invalid or the IDP is not reachable.

"ValidKubeVirtInfraNetworkMTU"

ValidKubeVirtInfraNetworkMTU indicates if the MTU configured on an infra cluster hosting a guest cluster utilizing kubevirt platform is a sufficient value that will avoid performance degradation due to fragmentation of the double encapsulation in ovn-kubernetes

"ValidOIDCConfiguration"

ValidOIDCConfiguration indicates if an AWS cluster’s OIDC condition is detected as invalid. A failure here may require external user intervention to resolve. E.g. oidc was deleted out of band.

"ValidReleaseImage"

ValidReleaseImage indicates if the release image set in the spec is valid for the HostedCluster. For example, this can be set false if the HostedCluster itself attempts an unsupported version before 4.9 or an unsupported upgrade e.g y-stream upgrade before 4.11. A failure here is unlikely to resolve without the changing user input.

"ValidReleaseInfo"

ValidReleaseInfo bubbles up the same condition from HCP. It indicates if the release contains all the images used by hypershift and reports missing images if any.

version reports the current version of this component.

resources is a list of the resources reconciled by this component.

Conditions contains details for the current state of the ControlPlane Component. If there is an error, then the Available condition will be false.

Current condition types are: “Available”

managedIdentitiesKeyVault contains information on the management cluster’s managed identities Azure Key Vault. This Key Vault is where the managed identities certificates are stored. These certificates are pulled out of the Key Vault by the Secrets Store CSI driver and mounted into a volume on control plane pods requiring authentication with Azure API.

More information on how the Secrets Store CSI driver works to do this can be found here: https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver.

cloudProvider is a pre-existing managed identity associated with the azure cloud provider, aka cloud controller manager.

nodePoolManagement is a pre-existing managed identity associated with the operator managing the NodePools.

controlPlaneOperator is a pre-existing managed identity associated with the control plane operator.

imageRegistry is a pre-existing managed identity associated with the cluster-image-registry-operator.

ingress is a pre-existing managed identity associated with the cluster-ingress-operator.

network is a pre-existing managed identity associated with the cluster-network-operator.

diskClientID is a pre-existing managed identity associated with the azure-disk-controller.

fileClientID is a pre-existing managed identity associated with the azure-disk-controller.

baseDomain is the base domain of the hosted cluster. It will be used to confgure ingress in the hosted cluster through the subdomain baseDomainPrefix.baseDomain. If baseDomainPrefix is ommitted, the hostedCluster.name will be used as the subdomain. Once set, this field is immutable. When the value is the empty string “”, the controller might default to a value depending on the platform.

baseDomainPrefix is the base domain prefix for the hosted cluster ingress. It will be used to confgure ingress in the hosted cluster through the subdomain baseDomainPrefix.baseDomain. If baseDomainPrefix is ommitted, the hostedCluster.name will be used as the subdomain. Set baseDomainPrefix to an empty string “”, if you don’t want a prefix at all (not even hostedCluster.name) to be prepended to baseDomain. This field is immutable.

publicZoneID is the Hosted Zone ID where all the DNS records that are publicly accessible to the internet exist. This field is optional and mainly leveraged in cloud environments where the DNS records for the .baseDomain are created by controllers in this zone. Once set, this value is immutable.

privateZoneID is the Hosted Zone ID where all the DNS records that are only available internally to the cluster exist. This field is optional and mainly leveraged in cloud environments where the DNS records for the .baseDomain are created by controllers in this zone. Once set, this value is immutable.

storageAccountType determines if the storage account for storing the diagnostics data should be disabled (Disabled), provisioned by Azure (Managed) or by the user (UserManaged).

userManaged specifies the diagnostics settings for a virtual machine when the storage account is managed by the user.

"Managed"

Managed means HyperShift should provision and operator the etcd cluster automatically.

"Unmanaged"

Unmanaged means HyperShift will not provision or manage the etcd cluster, and the user is responsible for doing so.

managementType defines how the etcd cluster is managed. This can be either Managed or Unmanaged. This field is immutable.

managed specifies the behavior of an etcd cluster managed by HyperShift.

unmanaged specifies configuration which enables the control plane to integrate with an externally managed etcd cluster.

ClientSecret refers to a secret for client mTLS authentication with the etcd cluster. It may have the following key/value pairs:

etcd-client-ca.crt: Certificate Authority value
etcd-client.crt: Client certificate value
etcd-client.key: Client certificate key value

Name of the filter. Filter names are case-sensitive.

Values includes one or more filter values. Filter values are case-sensitive.

Tags is a list of tags to filter by. If specified, the resource must have all of the tags specified to be included in the result.

TagsAny is a list of tags to filter by. If specified, the resource must have at least one of the tags specified to be included in the result.

NotTags is a list of tags to filter by. If specified, resources which contain all of the given tags will be excluded from the result.

NotTagsAny is a list of tags to filter by. If specified, resources which contain any of the given tags will be excluded from the result.

release specifies the desired OCP release payload for all the hosted cluster components. This includes those components running management side like the Kube API Server and the CVO but also the operands which land in the hosted cluster data plane like the ingress controller, ovn agents, etc. The maximum and minimum supported release versions are determined by the running Hypersfhit Operator. Attempting to use an unsupported version will result in the HostedCluster being degraded and the validateReleaseImage condition being false. Attempting to use a release with a skew against a NodePool release bigger than N-2 for the y-stream will result in leaving the NodePool in an unsupported state. Changing this field will trigger a rollout of the control plane components. The behavior of the rollout will be driven by the ControllerAvailabilityPolicy and InfrastructureAvailabilityPolicy for PDBs and maxUnavailable and surce policies.

controlPlaneRelease is like spec.release but only for the components running on the management cluster. This excludes any operand which will land in the hosted cluster data plane. It is useful when you need to apply patch management side like a CVE, transparently for the hosted cluster. Version input for this field is free, no validation is performed against spec.release or maximum and minimum is performed. If defined, it will dicate the version of the components running management side, while spec.release will dictate the version of the components landing in the hosted cluster data plane. If not defined, spec.release is used for both. Changing this field will trigger a rollout of the control plane. The behavior of the rollout will be driven by the ControllerAvailabilityPolicy and InfrastructureAvailabilityPolicy for PDBs and maxUnavailable and surce policies.

clusterID uniquely identifies this cluster. This is expected to be an RFC4122 UUID value (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx in hexadecimal digits). As with a Kubernetes metadata.uid, this ID uniquely identifies this cluster in space and time. This value identifies the cluster in metrics pushed to telemetry and metrics produced by the control plane operators. If a value is not specified, a random clusterID will be generated and set by the controller. Once set, this value is immutable.

infraID is a globally unique identifier for the cluster. It must consist of lowercase alphanumeric characters and hyphens (‘-’) only, and start and end with an alphanumeric character. It must be no more than 253 characters in length. This identifier will be used to associate various cloud resources with the HostedCluster and its associated NodePools. infraID is used to compute and tag created resources with “kubernetes.io/cluster/”+hcluster.Spec.InfraID which has contractual meaning for the cloud provider implementations. If a value is not specified, a random infraID will be generated and set by the controller. Once set, this value is immutable.

updateService may be used to specify the preferred upstream update service. If ommitted we will use the appropriate update service for the cluster and region. This is used by the control plane operator to determine and signal the appropriate available upgrades in the hostedCluster.status.

channel is an identifier for explicitly requesting that a non-default set of updates be applied to this cluster. If ommited no particular upgrades are suggested. TODO(alberto): Consider the backend to use the default channel by default. Default channel will contain stable updates that are appropriate for production clusters.

platform specifies the underlying infrastructure provider for the cluster and is used to configure platform specific behavior.

controllerAvailabilityPolicy specifies the availability policy applied to critical control plane components like the Kube API Server. Possible values are HighlyAvailable and SingleReplica. The default value is HighlyAvailable.

infrastructureAvailabilityPolicy specifies the availability policy applied to infrastructure services which run on the hosted cluster data plane like the ingress controller and image registry controller. Possible values are HighlyAvailable and SingleReplica. The default value is SingleReplica.

dns specifies the DNS configuration for the hosted cluster ingress.

networking specifies network configuration for the hosted cluster. Defaults to OVNKubernetes with a cluster network of cidr: “10.132.0.0/14” and a service network of cidr: “172.31.0.0/16”.

autoscaling specifies auto-scaling behavior that applies to all NodePools associated with this HostedCluster.

etcd specifies configuration for the control plane etcd cluster. The default managementType is Managed. Once set, the managementType cannot be changed.

services specifies how individual control plane services endpoints are published for consumption. This requires APIServer;OAuthServer;Konnectivity;Ignition. This field is immutable for all platforms but IBMCloud. Max is 6 to account for OIDC;OVNSbDb for backward compability though they are no-op.

-kubebuilder:validation:XValidation:rule=“self.all(s, !(s.service == ‘APIServer’ && s.servicePublishingStrategy.type == ‘Route’) || has(s.servicePublishingStrategy.route.hostname))”,message=“If serviceType is ‘APIServer’ and publishing strategy is ‘Route’, then hostname must be set” -kubebuilder:validation:XValidation:rule=“[‘APIServer’, ‘OAuthServer’, ‘Konnectivity’, ‘Ignition’].all(requiredType, self.exists(s, s.service == requiredType))”,message=“Services list must contain at least ‘APIServer’, ‘OAuthServer’, ‘Konnectivity’, and ‘Ignition’ service types” -kubebuilder:validation:XValidation:rule=“self.filter(s, s.servicePublishingStrategy.type == ‘Route’ && has(s.servicePublishingStrategy.route) && has(s.servicePublishingStrategy.route.hostname)).all(x, self.filter(y, y.servicePublishingStrategy.type == ‘Route’ && (has(y.servicePublishingStrategy.route) && has(y.servicePublishingStrategy.route.hostname) && y.servicePublishingStrategy.route.hostname == x.servicePublishingStrategy.route.hostname)).size() <= 1)”,message=“Each route publishingStrategy ‘hostname’ must be unique within the Services list.” -kubebuilder:validation:XValidation:rule=“self.filter(s, s.servicePublishingStrategy.type == ‘NodePort’ && has(s.servicePublishingStrategy.nodePort) && has(s.servicePublishingStrategy.nodePort.address) && has(s.servicePublishingStrategy.nodePort.port)).all(x, self.filter(y, y.servicePublishingStrategy.type == ‘NodePort’ && (has(y.servicePublishingStrategy.nodePort) && has(y.servicePublishingStrategy.nodePort.address) && y.servicePublishingStrategy.nodePort.address == x.servicePublishingStrategy.nodePort.address && has(y.servicePublishingStrategy.nodePort.port) && y.servicePublishingStrategy.nodePort.port == x.servicePublishingStrategy.nodePort.port )).size() <= 1)”,message=“Each nodePort publishingStrategy ‘nodePort’ and ‘hostname’ must be unique within the Services list.” TODO(alberto): this breaks the cost budget for < 4.17. We should figure why and enable it back. And If not fixable, consider imposing a minimum version on the management cluster.

pullSecret is a local reference to a Secret that must have a “.dockerconfigjson” key whose content must be a valid Openshift pull secret JSON. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. This pull secret will be part of every payload generated by the controllers for any NodePool of the HostedCluster and it will be injected into the container runtime of all NodePools. Changing this value will trigger a rollout for all existing NodePools in the cluster. Changing the content of the secret inplace will not trigger a rollout and might result in unpredicatble behaviour. TODO(alberto): have our own local reference type to include our opinions and avoid transparent changes.

sshKey is a local reference to a Secret that must have a “id_rsa.pub” key whose content must be the public part of 1..N SSH keys. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. When sshKey is set, the controllers will generate a machineConfig with the sshAuthorizedKeys https://coreos.github.io/ignition/configuration-v3_2/ populated with this value. This MachineConfig will be part of every payload generated by the controllers for any NodePool of the HostedCluster. Changing this value will trigger a rollout for all existing NodePools in the cluster.

issuerURL is an OIDC issuer URL which will be used as the issuer in all ServiceAccount tokens generated by the control plane API server via –service-account-issuer kube api server flag. https://k8s-docs.netlify.app/en/docs/reference/command-line-tools-reference/kube-apiserver/ https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection The default value is kubernetes.default.svc, which only works for in-cluster validation. If the platform is AWS and this value is set, the controller will update an s3 object with the appropriate OIDC documents (using the serviceAccountSigningKey info) into that issuerURL. The expectation is for this s3 url to be backed by an OIDC provider in the AWS IAM.

serviceAccountSigningKey is a local reference to a secret that must have a “key” key whose content must be the private key used by the service account token issuer. If not specified, a service account signing key will be generated automatically for the cluster. When specifying a service account signing key, an IssuerURL must also be specified. If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition.

Configuration specifies configuration for individual OCP components in the cluster, represented as embedded resources that correspond to the openshift configuration API.

AuditWebhook contains metadata for configuring an audit webhook endpoint for a cluster to process cluster audit events. It references a secret that contains the webhook information for the audit webhook endpoint. It is a secret because if the endpoint has mTLS the kubeconfig will contain client keys. The kubeconfig needs to be stored in the secret with a secret key name that corresponds to the constant AuditWebhookKubeconfigKey.

imageContentSources specifies image mirrors that can be used by cluster nodes to pull content. When imageContentSources is set, the controllers will generate a machineConfig. This MachineConfig will be part of every payload generated by the controllers for any NodePool of the HostedCluster. Changing this value will trigger a rollout for all existing NodePools in the cluster.

additionalTrustBundle is a local reference to a ConfigMap that must have a “ca-bundle.crt” key whose content must be a PEM-encoded X.509 certificate bundle that will be added to the hosted controlplane and nodes If the reference is set but none of the above requirements are met, the HostedCluster will enter a degraded state. TODO(alberto): Signal this in a condition. This will be part of every payload generated by the controllers for any NodePool of the HostedCluster. Changing this value will trigger a rollout for all existing NodePools in the cluster.

secretEncryption specifies a Kubernetes secret encryption strategy for the control plane.

fips indicates whether this cluster’s nodes will be running in FIPS mode. If set to true, the control plane’s ignition server will be configured to expect that nodes joining the cluster will be FIPS-enabled.

pausedUntil is a field that can be used to pause reconciliation on the HostedCluster controller, resulting in any change to the HostedCluster being ignored. Either a date can be provided in RFC3339 format or a boolean as in ‘true’, ‘false’, ‘True’, ‘False’. If a date is provided: reconciliation is paused on the resource until that date. If the boolean true is provided: reconciliation is paused on the resource until the field is removed.

OLMCatalogPlacement specifies the placement of OLM catalog components. By default, this is set to management and OLM catalog components are deployed onto the management cluster. If set to guest, the OLM catalog components will be deployed onto the guest cluster.

NodeSelector when specified, is propagated to all control plane Deployments and Stateful sets running management side. It must be satisfied by the management Nodes for the pods to be scheduled. Otherwise the HostedCluster will enter a degraded state. Changes to this field will propagate to existing Deployments and StatefulSets. TODO(alberto): add additional validation for the map key/values.

Tolerations when specified, define what custome tolerations are added to the hcp pods.

Version is the status of the release version applied to the HostedCluster.

KubeConfig is a reference to the secret containing the default kubeconfig for the cluster.

KubeadminPassword is a reference to the secret that contains the initial kubeadmin user password for the guest cluster.

IgnitionEndpoint is the endpoint injected in the ign config userdata. It exposes the config for instances to become kubernetes nodes.

ControlPlaneEndpoint contains the endpoint information by which external clients can access the control plane. This is populated after the infrastructure is ready.

OAuthCallbackURLTemplate contains a template for the URL to use as a callback for identity providers. The [identity-provider-name] placeholder must be replaced with the name of an identity provider defined on the HostedCluster. This is populated after the infrastructure is ready.

Conditions represents the latest available observations of a control plane’s current state.

payloadArch represents the CPU architecture type of the HostedCluster.Spec.Release.Image. The valid values are: Multi, ARM64, AMD64, S390X, or PPC64LE.

Platform contains platform-specific status of the HostedCluster

ReleaseImage is the release image applied to the hosted control plane.

ControlPlaneReleaseImage specifies the desired OCP release payload for control plane components running on the management cluster. If not defined, ReleaseImage is used

updateService may be used to specify the preferred upstream update service. By default it will use the appropriate update service for the cluster and region.

channel is an identifier for explicitly requesting that a non-default set of updates be applied to this cluster. The default channel will be contain stable updates that are appropriate for production clusters.

IssuerURL is an OIDC issuer URL which is used as the issuer in all ServiceAccount tokens generated by the control plane API server. The default value is kubernetes.default.svc, which only works for in-cluster validation.

Networking specifies network configuration for the cluster. Temporarily optional for backward compatibility, required in future releases.

ClusterID is the unique id that identifies the cluster externally. Making it optional here allows us to keep compatibility with previous versions of the control-plane-operator that have no knowledge of this field.

ServiceAccountSigningKey is a reference to a secret containing the private key used by the service account token issuer. The secret is expected to contain a single key named “key”. If not specified, a service account signing key will be generated automatically for the cluster.

ControllerAvailabilityPolicy specifies the availability policy applied to critical control plane components. The default value is SingleReplica.

InfrastructureAvailabilityPolicy specifies the availability policy applied to infrastructure services which run on cluster nodes. The default value is SingleReplica.

FIPS specifies if the nodes for the cluster will be running in FIPS mode

KubeConfig specifies the name and key for the kubeconfig secret

Services defines metadata about how control plane services are published in the management cluster.

AuditWebhook contains metadata for configuring an audit webhook endpoint for a cluster to process cluster audit events. It references a secret that contains the webhook information for the audit webhook endpoint. It is a secret because if the endpoint has MTLS the kubeconfig will contain client keys. This is currently only supported in IBM Cloud. The kubeconfig needs to be stored in the secret with a secret key name that corresponds to the constant AuditWebhookKubeconfigKey.

Etcd contains metadata about the etcd cluster the hypershift managed Openshift control plane components use to store data.

Configuration embeds resources that correspond to the openshift configuration API: https://docs.openshift.com/container-platform/4.7/rest_api/config_apis/config-apis-index.html

ImageContentSources lists sources/repositories for the release-image content.

AdditionalTrustBundle references a ConfigMap containing a PEM-encoded X.509 certificate bundle

SecretEncryption contains metadata about the kubernetes secret encryption strategy being used for the cluster when applicable.

PausedUntil is a field that can be used to pause reconciliation on a resource. Either a date can be provided in RFC3339 format or a boolean. If a date is provided: reconciliation is paused on the resource until that date. If the boolean true is provided: reconciliation is paused on the resource until the field is removed.

OLMCatalogPlacement specifies the placement of OLM catalog components. By default, this is set to management and OLM catalog components are deployed onto the management cluster. If set to guest, the OLM catalog components will be deployed onto the guest cluster.

Autoscaling specifies auto-scaling behavior that applies to all NodePools associated with the control plane.

NodeSelector when specified, must be true for the pods managed by the HostedCluster to be scheduled.

Tolerations when specified, define what custome tolerations are added to the hcp pods.

Ready denotes that the HostedControlPlane API Server is ready to receive requests This satisfies CAPI contract https://github.com/kubernetes-sigs/cluster-api/blob/cd3a694deac89d5ebeb888307deaa61487207aa0/controllers/cluster_controller_phases.go#L226-L230

Initialized denotes whether or not the control plane has provided a kubeadm-config. Once this condition is marked true, its value is never changed. See the Ready condition for an indication of the current readiness of the cluster’s control plane. This satisfies CAPI contract https://github.com/kubernetes-sigs/cluster-api/blob/cd3a694deac89d5ebeb888307deaa61487207aa0/controllers/cluster_controller_phases.go#L238-L252

ExternalManagedControlPlane indicates to cluster-api that the control plane is managed by an external service. https://github.com/kubernetes-sigs/cluster-api/blob/65e5385bffd71bf4aad3cf34a537f11b217c7fab/controllers/machine_controller.go#L468

ControlPlaneEndpoint contains the endpoint information by which external clients can access the control plane. This is populated after the infrastructure is ready.

OAuthCallbackURLTemplate contains a template for the URL to use as a callback for identity providers. The [identity-provider-name] placeholder must be replaced with the name of an identity provider defined on the HostedCluster. This is populated after the infrastructure is ready.

versionStatus is the status of the release version applied by the hosted control plane operator.

Version is the semantic version of the release applied by the hosted control plane operator

Deprecated: Use versionStatus.desired.version instead.

ReleaseImage is the release image applied to the hosted control plane.

Deprecated: Use versionStatus.desired.image instead.

lastReleaseImageTransitionTime is the time of the last update to the current releaseImage property.

Deprecated: Use versionStatus.history[0].startedTime instead.

KubeConfig is a reference to the secret containing the default kubeconfig for this control plane.

KubeadminPassword is a reference to the secret containing the initial kubeadmin password for the guest cluster.

Condition contains details for one aspect of the current state of the HostedControlPlane. Current condition types are: “Available”

Platform contains platform-specific status of the HostedCluster

NodeCount tracks the number of nodes in the HostedControlPlane.

Type defines the IBM Cloud KMS authentication strategy

Unmanaged defines the auth metadata the customer provides to interact with IBM Cloud KMS

Managed defines metadata around the service to service authentication strategy for the IBM Cloud KMS system (all provider managed).

"Managed"

IBMCloudKMSManagedAuth defines the KMS authentication strategy where the IKS/ROKS platform uses service to service auth to call IBM Cloud KMS APIs (no customer credentials requried)

"Unmanaged"

IBMCloudKMSUnmanagedAuth defines the KMS authentication strategy where a customer supplies IBM Cloud authentication to interact with IBM Cloud KMS APIs

CRKID is the customer rook key id

InstanceID is the id for the key protect instance

CorrelationID is an identifier used to track all api call usage from hypershift

URL is the url to call key protect apis over

KeyVersion is a unique number associated with the key. The number increments whenever a new key is enabled for data encryption.

Region is the IBM Cloud region

Auth defines metadata for how authentication is done with IBM Cloud KMS

KeyList defines the list of keys used for data encryption

Credentials should reference a secret with a key field of IBMCloudIAMAPIKeySecretKey that contains a apikey to call IBM Cloud KMS APIs

ProviderType is a specific supported infrastructure provider within IBM Cloud.

Source is the repository that users refer to, e.g. in image pull specifications.

Mirrors are one or more repositories that may also contain the same images.

maxUnavailable is the maximum number of nodes that can be unavailable during the update.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding down.

Defaults to 1.

Example: when this is set to 30%, a max of 30% of the nodes can be made unschedulable/unavailable immediately when the update starts. Once a set of nodes is updated, more nodes can be made unschedulable for update, ensuring that the total number of nodes schedulable at all times during the update is at least 70% of desired nodes.

"AWS"

"Azure"

"IBMCloud"

Provider defines the KMS provider

IBMCloud defines metadata for the IBM Cloud KMS encryption strategy

AWS defines metadata about the configuration of the AWS KMS Secret Encryption provider

Azure defines metadata about the configuration of the Azure KMS Secret Encryption provider using Azure key vault

CacheName holds the name of the cache DataVolume, if exists

Credentials shows the client credentials used when creating KubeVirt virtual machines. This filed is only exists when the KubeVirt virtual machines are being placed on a cluster separate from the one hosting the Hosted Control Plane components.

The default behavior when Credentials is not defined is for the KubeVirt VMs to be placed on the same cluster and namespace as the Hosted Control Plane.

Type is the type of the caching strategy

"None"

KubevirtCachingStrategyNone means that hypershift will not cache the boot image

"PVC"

KubevirtCachingStrategyPVC means that hypershift will cache the boot image into a PVC; only relevant when using a QCOW boot image, and is ignored when using a container image

Memory represents how much guest memory the VM should have

Cores represents how many cores the guest VM should have

QosClass If set to “Guaranteed”, requests the scheduler to place the VirtualMachineInstance on a node with limit memory and CPU, equal to be the requested values, to set the VMI as a Guaranteed QoS Class; See here for more details: https://kubevirt.io/user-guide/operations/node_overcommit/#requesting-the-right-qos-class-for-virtualmachineinstances

ContainerDiskImage is a string representing the container image that holds the root disk

DeviceName is the name of the host device that is desired to be utilized in the HostedCluster’s NodePool The device can be any supported PCI device, including GPU, either as a passthrough or a vGPU slice.

Count is the number of instances the specified host device will be attached to each of the NodePool’s nodes. Default is 1.

StorageClassMapping maps StorageClasses on the infra cluster hosting the KubeVirt VMs to StorageClasses that are made available within the Guest Cluster.

NOTE: It is possible that not all capablities of an infra cluster’s storageclass will be present for the corresponding guest clusters storageclass.

Name specify the network attached to the nodes it is a value with the format “[namespace]/[name]” to reference the multus network attachment definition

RootVolume represents values associated with the VM volume that will host rhcos

Compute contains values representing the virtual hardware requested for the VM

NetworkInterfaceMultiQueue If set to “Enable”, virtual network interfaces configured with a virtio bus will also enable the vhost multiqueue feature for network devices. The number of queues created depends on additional factors of the VirtualMachineInstance, like the number of guest CPUs.

AdditionalNetworks specify the extra networks attached to the nodes

AttachDefaultNetwork specify if the default pod network should be attached to the nodes this can only be set to false if AdditionalNetworks are configured

NodeSelector is a selector which must be true for the kubevirt VirtualMachine to fit on a node. Selector which must match a node’s labels for the VM to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

KubevirtHostDevices specifies the host devices (e.g. GPU devices) to be passed from the management cluster, to the nodepool nodes

Size is the size of the persistent storage volume

StorageClass is the storageClass used for the underlying PVC that hosts the volume

AccessModes is an array that contains the desired Access Modes the root volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

VolumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.

InfraKubeConfigSecret is a reference to a secret that contains the kubeconfig for the external infra cluster that will be used to host the KubeVirt virtual machines for this cluster.

InfraNamespace defines the namespace on the external infra cluster that is used to host the KubeVirt virtual machines. This namespace must already exist before creating the HostedCluster and the kubeconfig referenced in the InfraKubeConfigSecret must have access to manage the required resources within this namespace.

BaseDomainPassthrough toggles whether or not an automatically generated base domain for the guest cluster should be used that is a subdomain of the management cluster’s *.apps DNS.

For the KubeVirt platform, the basedomain can be autogenerated using the *.apps domain of the management/infra hosting cluster This makes the guest cluster’s base domain a subdomain of the hypershift infra/mgmt cluster’s base domain.

Example: Infra/Mgmt cluster’s DNS Base: example.com Cluster: mgmt-cluster.example.com Apps: *.apps.mgmt-cluster.example.com KubeVirt Guest cluster’s DNS Base: apps.mgmt-cluster.example.com Cluster: guest.apps.mgmt-cluster.example.com Apps: *.apps.guest.apps.mgmt-cluster.example.com

This is possible using OCP wildcard routes

GenerateID is used to uniquely apply a name suffix to resources associated with kubevirt infrastructure resources

Credentials defines the client credentials used when creating KubeVirt virtual machines. Defining credentials is only necessary when the KubeVirt virtual machines are being placed on a cluster separate from the one hosting the Hosted Control Plane components.

The default behavior when Credentials is not defined is for the KubeVirt VMs to be placed on the same cluster and namespace as the Hosted Control Plane.

StorageDriver defines how the KubeVirt CSI driver exposes StorageClasses on the infra cluster (hosting the VMs) to the guest cluster.

Image represents what rhcos image to use for the node pool

(Members of KubevirtVolume are embedded into this type.)

KubevirtVolume represents of type of storage to run the image on

CacheStrategy defines the boot image caching strategy. Default - no caching

Group contains which group this mapping belongs to.

InfraStorageClassName is the name of the infra cluster storage class that will be exposed to the guest.

GuestStorageClassName is the name that the corresponding storageclass will be called within the guest cluster

"Default"

DefaultKubevirtStorageDriverConfigType means the kubevirt storage driver maps to the underlying infra cluster’s default storageclass

"Manual"

ManualKubevirtStorageDriverConfigType means the kubevirt storage driver mapping is explicitly defined.

"None"

NoneKubevirtStorageDriverConfigType means no kubevirt storage driver is used

Type represents the type of kubevirt csi driver configuration to use

Manual is used to explicilty define how the infra storageclasses are mapped to guest storageclasses

Type represents the type of storage to associate with the kubevirt VMs.

Persistent volume type means the VM’s storage is backed by a PVC VMs that use persistent volumes can survive disruption events like restart and eviction This is the default type used when no storage type is defined.

Group contains which group this mapping belongs to.

InfraStorageClassName is the name of the infra cluster volume snapshot class that will be exposed to the guest.

GuestVolumeSnapshotClassName is the name that the corresponding volumeSnapshotClass will be called within the guest cluster

"Persistent"

KubevirtVolumeTypePersistent represents persistent volume for kubevirt VMs

hostname is the name of the DNS record that will be created pointing to the LoadBalancer and passed through to consumers of the service. If ommited, the value will be infered from the corev1.Service Load balancer type .status.

CIDR is the IP block address pool for machines within the cluster.

name is the name of the Azure Key Vault on the management cluster.

tenantID is the tenant ID of the Azure Key Vault on the management cluster.

storage specifies how etcd data is persisted.

type is the kind of persistent storage implementation to use for etcd. Only PersistentVolume is supported at the moment.

persistentVolume is the configuration for PersistentVolume etcd storage. With this implementation, a PersistentVolume will be allocated for every etcd member (either 1 or 3 depending on the HostedCluster control plane availability configuration).

restoreSnapshotURL allows an optional URL to be provided where an etcd snapshot can be downloaded, for example a pre-signed URL referencing a storage service. This snapshot will be restored on initial startup, only when the etcd PV is empty.

"PersistentVolume"

PersistentVolumeEtcdStorage uses PersistentVolumes for etcd storage.

clientID is the client ID of a managed identity.

certificateName is the name of the certificate backing the managed identity. This certificate is expected to reside in an Azure Key Vault on the management cluster.

"Disable"

"Enable"

Name is the name of the network to filter by.

Description is the description of the network to filter by.

ProjectID is the project ID of the network to filter by.

(Members of FilterByNeutronTags are embedded into this type.)

FilterByNeutronTags specifies tags to filter by.

ID is the ID of the network to use. If ID is provided, the other filters cannot be provided. Must be in UUID format.

Filter specifies a filter to select an OpenStack network. If provided, cannot be empty.

"Calico"

Calico specifies Calico as the SDN provider

"OVNKubernetes"

OVNKubernetes specifies OVN as the SDN provider

"OpenShiftSDN"

OpenShiftSDN specifies OpenShiftSDN as the SDN provider

"Other"

Other specifies an undefined SDN provider

Min is the minimum number of nodes to maintain in the pool. Must be >= 1 and <= .Max.

Max is the maximum number of nodes allowed in the pool. Must be >= 1 and >= Min.

Type of condition in CamelCase or in foo.example.com/CamelCase. Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be useful (see .node.status.conditions), the ability to deconflict is important.

Status of the condition, one of True, False, Unknown.

Severity provides an explicit classification of Reason code, so the users or machines can immediately understand the current situation and act accordingly. The Severity field MUST be set only when Status=False.

Last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.

The reason for the condition’s last transition in CamelCase. The specific API may choose whether or not this field is considered a guaranteed API. This field may not be empty.

A human readable message indicating details about the transition. This field may be empty.

upgradeType specifies the type of strategy for handling upgrades. This can be either “Replace” or “InPlace”. “Replace” will update Nodes by recreating the underlying instances. “InPlace” will update Nodes by applying changes to the existing instances. This might or might not result in a reboot.

replace is the configuration for rolling upgrades. It defaults to a RollingUpdate strategy with maxSurge of 1 and maxUnavailable of 0.

inPlace is the configuration for in-place upgrades.

autoRepair specifies whether health checks should be enabled for machines in the NodePool. The default is false. Enabling this feature will cause the controller to automatically delete unhealthy machines. The unhealthy criteria is reserved for the controller implementation and subject to change. But generally it’s determined by checking the Node ready condition is true and a timeout that might vary depending on the platform provider. AutoRepair will no-op when more than 2 Nodes are unhealthy at the same time. Giving time for the cluster to stabilize or to the user to manually intervene.

Type specifies the platform name.

AWS specifies the configuration used when operating on AWS.

IBMCloud defines IBMCloud specific settings for components

Kubevirt specifies the configuration used when operating on KubeVirt platform.

Agent specifies the configuration used when using Agent platform.

PowerVS specifies the configuration used when using IBMCloud PowerVS platform.

OpenStack specifies the configuration used when using OpenStack platform.

KubeVirt contains the KubeVirt platform statuses

clusterName is the name of the HostedCluster this NodePool belongs to. If a HostedCluster with this name doesn’t exist, the controller will no-op until it exists.

release specifies the OCP release used for the NodePool. This informs the ignition configuration for machines which includes the kubelet version, as well as other platform specific machine properties (e.g. an AMI on the AWS platform). It’s not supported to use a release in a NodePool which minor version skew against the Control Plane release is bigger than N-2. Although there’s no enforcement that prevents this from happening. Attempting to use a release with a bigger skew might result in unpredictable behaviour. Attempting to use a release higher than the HosterCluster one will result in the NodePool being degraded and the ValidReleaseImage condition being false. Attempting to use a release lower than the current NodePool y-stream will result in the NodePool being degraded and the ValidReleaseImage condition being false. Changing this field will trigger a NodePool rollout.

platform specifies the underlying infrastructure provider for the NodePool and is used to configure platform specific behavior.

replicas is the desired number of nodes the pool should maintain. If unset, the controller default value is 0. replicas is mutually exclusive with autoscaling. If autoscaling is configured, replicas must be omitted and autoscaling will control the NodePool size internally.

management specifies behavior for managing nodes in the pool, such as upgrade strategies and auto-repair behaviors.

autoscaling specifies auto-scaling behavior for the NodePool. autoscaling is mutually exclusive with replicas. If replicas is set, this field must be ommited.

config is a list of references to ConfigMaps containing serialized MachineConfig resources to be injected into the ignition configurations of nodes in the NodePool. The MachineConfig API schema is defined here:

https://github.com/openshift/machine-config-operator/blob/18963e4f8fe66e8c513ca4b131620760a414997f/pkg/apis/machineconfiguration.openshift.io/v1/types.go#L185

Each ConfigMap must have a single key named “config” whose value is the YML with one or more serialized machineconfiguration.openshift.io resources:

• KubeletConfig
• ContainerRuntimeConfig
• MachineConfig
• ClusterImagePolicy
• ImageContentSourcePolicy
• ImageDigestMirrorSet

This is validated in the backend and signaled back via validMachineConfig condition. Changing this field will trigger a NodePool rollout.

nodeDrainTimeout is the maximum amount of time that the controller will spend on retrying to drain a node until it succeeds. The default value is 0, meaning that the node can retry drain without any time limitations. Changing this field propagate inplace into existing Nodes.

nodeVolumeDetachTimeout is the maximum amount of time that the controller will spend on detaching volumes from a node. The default value is 0, meaning that the volumes will be detached from the node without any time limitations. After the timeout, any remaining attached volumes will be ignored and the removal of the machine will continue. Changing this field propagate inplace into existing Nodes.

nodeLabels propagates a list of labels to Nodes, only once on creation. Valid values are those in https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

taints if specified, propagates a list of taints to Nodes, only once on creation. These taints are additive to the ones applied by other controllers

pausedUntil is a field that can be used to pause reconciliation on the NodePool controller. Resulting in any change to the NodePool being ignored. Either a date can be provided in RFC3339 format or a boolean as in ‘true’, ‘false’, ‘True’, ‘False’. If a date is provided: reconciliation is paused on the resource until that date. If the boolean true is provided: reconciliation is paused on the resource until the field is removed.

tuningConfig is a list of references to ConfigMaps containing serialized Tuned or PerformanceProfile resources to define the tuning configuration to be applied to nodes in the NodePool. The Tuned API is defined here:

https://github.com/openshift/cluster-node-tuning-operator/blob/2c76314fb3cc8f12aef4a0dcd67ddc3677d5b54f/pkg/apis/tuned/v1/tuned_types.go

The PerformanceProfile API is defined here: https://github.com/openshift/cluster-node-tuning-operator/tree/b41042d42d4ba5bb2e99960248cf1d6ae4935018/pkg/apis/performanceprofile/v2

Each ConfigMap must have a single key named “tuning” whose value is the JSON or YAML of a serialized Tuned or PerformanceProfile. Changing this field will trigger a NodePool rollout.

arch is the preferred processor architecture for the NodePool. Different platforms might have different supported architectures. TODO: This is set as optional to prevent validation from failing due to a limitation on client side validation with open API machinery: https://github.com/kubernetes/kubernetes/issues/108768#issuecomment-1253912215 TODO Add s390x to enum validation once the architecture is supported

Replicas is the latest observed number of nodes in the pool.

Version is the semantic version of the latest applied release specified by the NodePool.

Platform hols the specific statuses

Conditions represents the latest available observations of the node pool’s current state.

address is the host/ip that the NodePort service is exposed over.

port is the port of the NodePort service. If <=0, the port is dynamically assigned when the service is created.

"guest"

GuestOLMCatalogPlacement indicates OLM catalog components will be placed in the guest cluster.

"management"

ManagementOLMCatalogPlacement indicates OLM catalog components will be placed in the management cluster.

Name is the name of a secret in the same namespace as the resource being provisioned. The secret must contain a key named clouds.yaml which contains an OpenStack clouds.yaml file. The secret may optionally contain a key named cacert containing a PEM-encoded CA certificate.

CloudName specifies the name of the entry in the clouds.yaml file to use.

Flavor is the OpenStack flavor to use for the node instances.

ImageName is the OpenStack Glance image name to use for node instances. If unspecified, the default is chosen based on the NodePool release payload image.

availabilityZone is the nova availability zone in which the provider will create the VM. If not specified, the VM will be created in the default availability zone specified in the nova configuration. Availability zone names must NOT contain : since it is used by admin users to specify hosts where instances are launched in server creation. Also, it must not contain spaces otherwise it will lead to node that belongs to this availability zone register failure, see kubernetes/cloud-provider-openstack#1379 for further information. The maximum length of availability zone name is 63 as per labels limits.

AdditionalPorts is a list of additional ports to create on the node instances.

IdentityRef is a reference to a secret holding OpenStack credentials to be used when reconciling the hosted cluster.

ManagedSubnets describe the OpenStack Subnet to be created. Cluster actuator will create a network, and a subnet with the defined DNSNameservers, AllocationPools and the CIDR defined in the HostedCluster MachineNetwork, and a router connected to the subnet. Currently only one IPv4 subnet is supported.

Router specifies an existing router to be used if ManagedSubnets are specified. If specified, no new router will be created.

Network specifies an existing network to use if no ManagedSubnets are specified.

Subnets specifies existing subnets to use if not ManagedSubnets are specified. All subnets must be in the network specified by Network. There can be zero, one, or two subnets. If no subnets are specified, all subnets in Network will be used. If 2 subnets are specified, one must be IPv4 and the other IPv6.

NetworkMTU sets the maximum transmission unit (MTU) value to address fragmentation for the private network ID. This value will be used only if the Cluster actuator creates the network. If left empty, the network will have the default MTU defined in Openstack network service. To use this field, the Openstack installation requires the net-mtu neutron API extension.

ExternalNetwork is the OpenStack Network to be used to get public internet to the VMs. This option is ignored if DisableExternalNetwork is set to true.

If ExternalNetwork is defined it must refer to exactly one external network.

If ExternalNetwork is not defined or is empty the controller will use any existing external network as long as there is only one. It is an error if ExternalNetwork is not defined and there are multiple external networks unless DisableExternalNetwork is also set.

If ExternalNetwork is not defined and there are no external networks the controller will proceed as though DisableExternalNetwork was set.

DisableExternalNetwork specifies whether or not to attempt to connect the cluster to an external network. This allows for the creation of clusters when connecting to an external network is not possible or desirable, e.g. if using a provider network.

Tags to set on all resources in cluster which support tags

"AMD64"

"ARM64"

"Multi"

"PPC64LE"

"S390X"

storageClassName is the StorageClass of the data volume for each etcd member. See https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1. TODO(alberto): This shouldn’t really be a pointer. There’s no real different semantic for nil and empty string. Revisit all pointer vs non-pointer choices.

size is the minimum size of the data volume for each etcd member. Default is 8Gi. This field is immutable

Tenancy indicates if instance should run on shared or single-tenant hardware.

Possible values: default: NodePool instances run on shared hardware. dedicated: Each NodePool instance runs on single-tenant hardware. host: NodePool instances run on user’s pre-allocated dedicated hosts.

Type is the type of infrastructure provider for the cluster.

AWS specifies configuration for clusters running on Amazon Web Services.

Agent specifies configuration for agent-based installations.

IBMCloud defines IBMCloud specific settings for components

Azure defines azure specific settings

PowerVS specifies configuration for clusters running on IBMCloud Power VS Service. This field is immutable. Once set, It can’t be changed.

KubeVirt defines KubeVirt specific settings for cluster components.

OpenStack specifies configuration for clusters running on OpenStack.

"AWS"

AWSPlatform represents Amazon Web Services infrastructure.

"Agent"

AgentPlatform represents user supplied insfrastructure booted with agents.

"Azure"

AzurePlatform represents Azure infrastructure.

"IBMCloud"

IBMCloudPlatform represents IBM Cloud infrastructure.

"KubeVirt"

KubevirtPlatform represents Kubevirt infrastructure.

"None"

NonePlatform represents user supplied (e.g. bare metal) infrastructure.

"OpenStack"

OpenStackPlatform represents OpenStack infrastructure.

"PowerVS"

PowerVSPlatform represents PowerVS infrastructure.

""

PortSecurityDefault uses the default port security policy.

"Disabled"

PortSecurityDisabled disables port security on a port.

"Enabled"

PortSecurityEnabled enables port security on a port.

Network is a query for an openstack network that the port will be created or discovered on. This will fail if the query returns more than one network.

Description is a human-readable description for the port.

AllowedAddressPairs is a list of address pairs which Neutron will allow the port to send traffic from in addition to the port’s addresses. If not specified, the MAC Address will be the MAC Address of the port. Depending on the configuration of Neutron, it may be supported to specify a CIDR instead of a specific IP address.

VNICType specifies the type of vNIC which this port should be attached to. This is used to determine which mechanism driver(s) to be used to bind the port. The valid values are normal, macvtap, direct, baremetal, direct-physical, virtio-forwarder, smart-nic and remote-managed, although these values will not be validated in this API to ensure compatibility with future neutron changes or custom implementations. What type of vNIC is actually available depends on deployments. If not specified, the Neutron default value is used.

PortSecurityPolicy specifies whether or not to enable port security on the port. Allowed values are “Enabled”, “Disabled” and omitted. When not set, it takes the value of the corresponding field at the network level.

SystemType is the System type used to host the instance. systemType determines the number of cores and memory that is available. Few of the supported SystemTypes are s922,e880,e980. e880 systemType available only in Dallas Datacenters. e980 systemType available in Datacenters except Dallas and Washington. When omitted, this means that the user has no opinion and the platform is left to choose a reasonable default. The current default is s922 which is generally available.

ProcessorType is the VM instance processor type. It must be set to one of the following values: Dedicated, Capped or Shared.

Dedicated: resources are allocated for a specific client, The hypervisor makes a 1:1 binding of a partition’s processor to a physical processor core. Shared: Shared among other clients. Capped: Shared, but resources do not expand beyond those that are requested, the amount of CPU time is Capped to the value specified for the entitlement.

if the processorType is selected as Dedicated, then Processors value cannot be fractional. When omitted, this means that the user has no opinion and the platform is left to choose a reasonable default. The current default is shared.

Processors is the number of virtual processors in a virtual machine. when the processorType is selected as Dedicated the processors value cannot be fractional. maximum value for the Processors depends on the selected SystemType. when SystemType is set to e880 or e980 maximum Processors value is 143. when SystemType is set to s922 maximum Processors value is 15. minimum value for Processors depends on the selected ProcessorType. when ProcessorType is set as Shared or Capped, The minimum processors is 0.5. when ProcessorType is set as Dedicated, The minimum processors is 1. When omitted, this means that the user has no opinion and the platform is left to choose a reasonable default. The default is set based on the selected ProcessorType. when ProcessorType selected as Dedicated, the default is set to 1. when ProcessorType selected as Shared or Capped, the default is set to 0.5.

MemoryGiB is the size of a virtual machine’s memory, in GiB. maximum value for the MemoryGiB depends on the selected SystemType. when SystemType is set to e880 maximum MemoryGiB value is 7463 GiB. when SystemType is set to e980 maximum MemoryGiB value is 15307 GiB. when SystemType is set to s922 maximum MemoryGiB value is 942 GiB. The minimum memory is 32 GiB.

When omitted, this means the user has no opinion and the platform is left to choose a reasonable default. The current default is 32.

Image used for deploying the nodes. If unspecified, the default is chosen based on the NodePool release payload image.

StorageType for the image and nodes, this will be ignored if Image is specified. The storage tiers in PowerVS are based on I/O operations per second (IOPS). It means that the performance of your storage volumes is limited to the maximum number of IOPS based on volume size and storage tier. Although, the exact numbers might change over time, the Tier 3 storage is currently set to 3 IOPS/GB, and the Tier 1 storage is currently set to 10 IOPS/GB.

The default is tier1

ImageDeletePolicy is policy for the image deletion.

delete: delete the image from the infrastructure. retain: delete the image from the openshift but retain in the infrastructure.

The default is delete

"capped"

PowerVSNodePoolCappedProcType defines capped processor type

"dedicated"

PowerVSNodePoolDedicatedProcType defines dedicated processor type

"shared"

PowerVSNodePoolSharedProcType defines shared processor type

AccountID is the IBMCloud account id. This field is immutable. Once set, It can’t be changed.

CISInstanceCRN is the IBMCloud CIS Service Instance’s Cloud Resource Name This field is immutable. Once set, It can’t be changed.

ResourceGroup is the IBMCloud Resource Group in which the cluster resides. This field is immutable. Once set, It can’t be changed.

Region is the IBMCloud region in which the cluster resides. This configures the OCP control plane cloud integrations, and is used by NodePool to resolve the correct boot image for a given release. This field is immutable. Once set, It can’t be changed.

Zone is the availability zone where control plane cloud resources are created. This field is immutable. Once set, It can’t be changed.

Subnet is the subnet to use for control plane cloud resources. This field is immutable. Once set, It can’t be changed.

ServiceInstance is the reference to the Power VS service on which the server instance(VM) will be created. Power VS service is a container for all Power VS instances at a specific geographic region. serviceInstance can be created via IBM Cloud catalog or CLI. ServiceInstanceID is the unique identifier that can be obtained from IBM Cloud UI or IBM Cloud cli.

More detail about Power VS service instance. https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-creating-power-virtual-server

This field is immutable. Once set, It can’t be changed.

VPC specifies IBM Cloud PowerVS Load Balancing configuration for the control plane. This field is immutable. Once set, It can’t be changed.

KubeCloudControllerCreds is a reference to a secret containing cloud credentials with permissions matching the cloud controller policy. This field is immutable. Once set, It can’t be changed.

TODO(dan): document the “cloud controller policy”

NodePoolManagementCreds is a reference to a secret containing cloud credentials with permissions matching the node pool management policy. This field is immutable. Once set, It can’t be changed.

TODO(dan): document the “node pool management policy”

IngressOperatorCloudCreds is a reference to a secret containing ibm cloud credentials for ingress operator to get authenticated with ibm cloud.

StorageOperatorCloudCreds is a reference to a secret containing ibm cloud credentials for storage operator to get authenticated with ibm cloud.

ImageRegistryOperatorCloudCreds is a reference to a secret containing ibm cloud credentials for image registry operator to get authenticated with ibm cloud.

ID of resource

Name of resource

Name for VPC to used for all the service load balancer. This field is immutable. Once set, It can’t be changed.

Region is the IBMCloud region in which VPC gets created, this VPC used for all the ingress traffic into the OCP cluster. This field is immutable. Once set, It can’t be changed.

Zone is the availability zone where load balancer cloud resources are created. This field is immutable. Once set, It can’t be changed.

Subnet is the subnet to use for load balancer. This field is immutable. Once set, It can’t be changed.

"Burstable"

"Guaranteed"

Image is the image pullspec of an OCP release payload image. See https://quay.io/repository/openshift-release-dev/ocp-release?tab=tags for a list of available images.

strategy is the node replacement strategy for nodes in the pool. In can be either “RollingUpdate” or “OnDelete”. RollingUpdate will rollout Nodes honoring maxSurge and maxUnavailable. OnDelete provide more granular control and will replace nodes as the old ones are manually deleted.

rollingUpdate specifies a rolling update strategy which upgrades nodes by creating new nodes and deleting the old ones.

maxUnavailable is the maximum number of nodes that can be unavailable during the update.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding down.

This can not be 0 if MaxSurge is 0.

Defaults to 0.

Example: when this is set to 30%, old nodes can be deleted down to 70% of desired nodes immediately when the rolling update starts. Once new nodes are ready, more old nodes be deleted, followed by provisioning new nodes, ensuring that the total number of nodes available at all times during the update is at least 70% of desired nodes.

maxSurge is the maximum number of nodes that can be provisioned above the desired number of nodes.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding up.

This can not be 0 if MaxUnavailable is 0.

Defaults to 1.

Example: when this is set to 30%, new nodes can be provisioned immediately when the rolling update starts, such that the total number of old and new nodes do not exceed 130% of desired nodes. Once old nodes have been deleted, new nodes can be provisioned, ensuring that total number of nodes running at any time during the update is at most 130% of desired nodes.

Hostname is the name of the DNS record that will be created pointing to the Route and passed through to consumers of the service. If ommited, the value will be infered from management ingress.Spec.Domain.

Name is the name of the router to filter by.

Description is the description of the router to filter by.

ProjectID is the project ID of the router to filter by.

(Members of FilterByNeutronTags are embedded into this type.)

FilterByNeutronTags specifies tags to filter by.

ID is the ID of the router to use. If ID is provided, the other filters cannot be provided. Must be in UUID format.

Filter specifies a filter to select an OpenStack router. If provided, cannot be empty.

Type defines the type of kube secret encryption being used

KMS defines metadata about the kms secret encryption strategy

AESCBC defines metadata about the AESCBC secret encryption strategy

"aescbc"

AESCBC uses AES-CBC with PKCS#7 padding to do secret encryption

"kms"

KMS integrates with a cloud provider’s key management service to do secret encryption

cidr is the IP block address pool for services within the cluster in CIDR format (e.g., 192.168.1.0/24 or 2001:0db8::/64)

type is the publishing strategy used for the service. It can be LoadBalancer;NodePort;Route;None;S3

nodePort configures exposing a service using a NodePort.

loadBalancer configures exposing a service using a dedicated LoadBalancer.

route configures exposing a service using a Route through and an ingress controller behind a cloud Load Balancer. The specifics of the setup are platform dependent.

service identifies the type of service being published. It can be APIServer;OAuthServer;Konnectivity;Ignition OVNSbDb;OIDC are no-op and kept for backward compatibility. This field is immutable.

servicePublishingStrategy specifies how to publish a service endpoint.

Name is the name of the subnet to filter by.

Description is the description of the subnet to filter by.

ProjectID is the project ID of the subnet to filter by.

IPVersion is the IP version of the subnet to filter by.

GatewayIP is the gateway IP of the subnet to filter by.

CIDR is the CIDR of the subnet to filter by.

IPv6AddressMode is the IPv6 address mode of the subnet to filter by.

IPv6RAMode is the IPv6 RA mode of the subnet to filter by.

(Members of FilterByNeutronTags are embedded into this type.)

FilterByNeutronTags specifies tags to filter by.

ID is the uuid of the subnet. It will not be validated.

Filter specifies a filter to select the subnet. It must match exactly one subnet.

DNSNameservers holds a list of DNS server addresses that will be provided when creating the subnet. These addresses need to have the same IP version as CIDR.

AllocationPools is an array of AllocationPool objects that will be applied to OpenStack Subnet being created. If set, OpenStack will only allocate these IPs for Machines. It will still be possible to create ports from outside of these ranges manually.

key is the taint key to be applied to a node.

value is the taint value corresponding to the taint key.

effect is the effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.

endpoint is the full etcd cluster client endpoint URL. For example:

https://etcd-client:2379

If the URL uses an HTTPS scheme, the TLS field is required.

tls specifies TLS configuration for HTTPS etcd client endpoints.

"OnDelete"

UpgradeStrategyOnDelete replaces old nodes when the deletion of the associated node instances are completed.

"RollingUpdate"

UpgradeStrategyRollingUpdate means use a rolling update for nodes.

"InPlace"

UpgradeTypeInPlace is a strategy which replaces nodes in-place with no additional node capacity requirements.

"Replace"

UpgradeTypeReplace is a strategy which replaces nodes using surge node capacity.

storageAccountURI is the URI of the user-managed storage account. The URI typically will be https://<mystorageaccountname>.blob.core.windows.net/ but may differ if you are using Azure DNS zone endpoints. You can find the correct endpoint by looking for the Blob Primary Endpoint in the endpoints tab in the Azure console or with the CLI by issuing az storage account list --query='[].{name: name, "resource group": resourceGroup, "blob endpoint": primaryEndpoints.blob}'.

Size specifies size (in Gi) of the storage device.

Must be greater than the image snapshot size or 8 (whichever is greater).

Type is the type of the volume.

IOPS is the number of IOPS requested for the disk. This is only valid for type io1.

Encrypted is whether the volume should be encrypted or not.

EncryptionKey is the KMS key to use to encrypt the volume. Can be either a KMS key ID or ARN. If Encrypted is set and this is omitted, the default AWS key will be used. The key must already exist and be accessible by the controller.