Production-Grade Container Orchestration on Kubernetes

APIService

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apiregistration.k8s.io/v1

import "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1"

APIService

APIService represents a server for a particular GroupVersion. Name must be "version.group".

Babylon Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

A large number of Babylon's products leverage machine learning and artificial intelligence, and in 2019, there wasn't enough computing power in-house to run a particular experiment. The company was also growing (from 100 to 1,600 in three years) and planning expansion into other countries.

Solution

Babylon had migrated its user-facing applications to a Kubernetes platform in 2018, so the infrastructure team turned to Kubeflow, a toolkit for machine learning on Kubernetes. "We tried to create a Kubernetes core server, we deployed Kubeflow, and we orchestrated the whole experiment, which ended up being a really good success," says AI Infrastructure Lead Jérémie Vallée. The team began building a self-service AI training platform on top of Kubernetes.

ConfigMap

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

ConfigMap

ConfigMap holds configuration data for pods to consume.

apiVersion: v1
kind: ConfigMap

CustomResourceDefinition

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apiextensions.k8s.io/v1

import "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"

CustomResourceDefinition

CustomResourceDefinition represents a resource that should be exposed on the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.

DeleteOptions

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/apis/meta/v1"

DeleteOptions may be provided when deleting an API object.

apiVersion (string)

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

FlowSchema

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: flowcontrol.apiserver.k8s.io/v1

import "k8s.io/api/flowcontrol/v1"

FlowSchema

FlowSchema defines the schema of a group of flows. Note that a flow is made up of a set of inbound API requests with similar attributes and is identified by a pair of strings: the name of the FlowSchema and a "flow distinguisher".

Introduction to kubectl

Mon, 01 Jan 0001 00:00:00 +0000

kubectl is the Kubernetes cli version of a swiss army knife, and can do many things.

While this Book is focused on using kubectl to declaratively manage applications in Kubernetes, it also covers other kubectl functions.

Command Families

Most kubectl commands typically fall into one of a few categories:

Type	Used For	Description
Declarative Resource Management	Deployment and operations (e.g. GitOps)	Declaratively manage Kubernetes workloads using resource configuration
Imperative Resource Management	Development Only	Run commands to manage Kubernetes workloads using Command Line arguments and flags
Printing Workload State	Debugging	Print information about workloads
Interacting with Containers	Debugging	Exec, attach, cp, logs
Cluster Management	Cluster operations	Drain and cordon Nodes

Declarative Application Management

The preferred approach for managing resources is through declarative files called resource configuration used with the kubectl Apply command. This command reads a local (or remote) file structure and modifies cluster state to reflect the declared intent.

LocalSubjectAccessReview

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authorization.k8s.io/v1

import "k8s.io/api/authorization/v1"

LocalSubjectAccessReview

LocalSubjectAccessReview checks whether or not a user or group can perform an action in a given namespace. Having a namespace scoped resource makes it much easier to grant namespace scoped policy that includes permissions checking.

MutatingAdmissionPolicyBindingList v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1beta1

import "k8s.io/api/admissionregistration/v1beta1"

Pod

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

Pod

Pod is a collection of containers that can run on a host. This resource is created by clients and scheduled onto hosts.

Service

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

Service

Service is a named abstraction of software service (for example, mysql) consisting of local port (for example 3306) that the proxy listens on, and the selector that determines which pods will answer requests sent through the proxy.

ServiceAccount

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

ComponentStatus (and ComponentStatusList) holds the cluster validation info. Deprecated: This API is deprecated in v1.19+

DeviceClass

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: resource.k8s.io/v1

import "k8s.io/api/resource/v1"

DeviceClass

DeviceClass is a vendor- or admin-provided resource that contains device configuration and selectors. It can be referenced in the device requests of a claim to apply these presets. Cluster scoped.

Endpoints

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

Endpoints

Endpoints is a collection of endpoints that implement the actual service. Example:

LabelSelector

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/apis/meta/v1"

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.

LimitRange

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

LimitRange

LimitRange sets resource usage limits for each kind of resource in a Namespace.

Secret

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

Secret

Secret holds secret data of a certain type. The total bytes of the values in the Data field must be less than MaxSecretSize bytes.

SelfSubjectAccessReview

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authorization.k8s.io/v1

import "k8s.io/api/authorization/v1"

SelfSubjectAccessReview

SelfSubjectAccessReview checks whether or the current user can perform an action. Not filling in a spec.namespace means "in all namespaces". Self is a special case, because users should always be able to check whether they can perform an action

TokenRequest

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authentication.k8s.io/v1

import "k8s.io/api/authentication/v1"

TokenRequest

TokenRequest requests a token for a given service account.

apiVersion: authentication.k8s.io/v1

Booking.com Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

In 2016, Booking.com migrated to an OpenShift platform, which gave product developers faster access to infrastructure. But because Kubernetes was abstracted away from the developers, the infrastructure team became a "knowledge bottleneck" when challenges arose. Trying to scale that support wasn't sustainable.

Solution

After a year operating OpenShift, the platform team decided to build its own vanilla Kubernetes platform—and ask developers to learn some Kubernetes in order to use it. "This is not a magical platform," says Ben Tyler, Principal Developer, B Platform Track. "We're not claiming that you can just use it with your eyes closed. Developers need to do some learning, and we're going to do everything we can to make sure they have access to that knowledge."

CSIDriver

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storage.k8s.io/v1

import "k8s.io/api/storage/v1"

CSIDriver

CSIDriver captures information about a Container Storage Interface (CSI) volume driver deployed on the cluster. Kubernetes attach detach controller uses this object to determine whether attach is required. Kubelet uses this object to determine whether pod information needs to be passed on mount. CSIDriver objects are non-namespaced.

EndpointSlice

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: discovery.k8s.io/v1

import "k8s.io/api/discovery/v1"

EndpointSlice

EndpointSlice represents a set of service endpoints. Most EndpointSlices are created by the EndpointSlice controller to represent the Pods selected by Service objects. For a given service there may be multiple EndpointSlice objects which must be joined to produce the full set of endpoints; you can find all of the slices for a given service by listing EndpointSlices in the service's namespace whose kubernetes.io/service-name label contains the service's name.

Event

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: events.k8s.io/v1

import "k8s.io/api/events/v1"

Event

Event is a report of an event somewhere in the cluster. It generally denotes some state change in the system. Events have a limited retention time and triggers and messages may evolve with time. Event consumers should not rely on the timing of an event with a given Reason reflecting a consistent underlying trigger, or the continued existence of events with that Reason. Events should be treated as informative, best-effort, supplemental data.

ListMeta

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/apis/meta/v1"

ListMeta describes metadata that synthetic resources must have, including lists and various status objects. A resource may have only one of {ObjectMeta, ListMeta}.

MutatingWebhookConfiguration

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1

import "k8s.io/api/admissionregistration/v1"

MutatingWebhookConfiguration

MutatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and may change the object.

PodTemplate

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

PodTemplate

PodTemplate describes a template for creating copies of a predefined pod.

ResourceQuota

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

ResourceQuota

ResourceQuota sets aggregate quota restrictions enforced per namespace

apiVersion: v1
kind: ResourceQuota

SelfSubjectRulesReview

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authorization.k8s.io/v1

import "k8s.io/api/authorization/v1"

SelfSubjectRulesReview

SelfSubjectRulesReview enumerates the set of actions the current user can perform within a namespace. The returned list of actions may be incomplete depending on the server's authorization mode, and any errors experienced during the evaluation. SelfSubjectRulesReview should be used by UIs to show/hide actions, or to quickly let an end user reason about their permissions. It should NOT Be used by external systems to drive authorization decisions as this raises confused deputy, cache lifetime/revocation, and correctness concerns. SubjectAccessReview, and LocalAccessReview are the correct way to defer authorization decisions to the API server.

TokenReview

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authentication.k8s.io/v1

import "k8s.io/api/authentication/v1"

TokenReview

TokenReview attempts to authenticate a token to a known user. Note: TokenReview requests may be cached by the webhook token authenticator plugin in the kube-apiserver.

AppDirect Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

AppDirect provides an end-to-end commerce platform for cloud-based products and services. When Director of Software Development Pierre-Alexandre Lacerte began working there in 2014, the company had a monolith application deployed on a "tomcat infrastructure, and the whole release process was complex for what it should be," he says. "There were a lot of manual steps involved, with one engineer building a feature, then another team picking up the change. So you had bottlenecks in the pipeline to ship a feature to production." At the same time, the engineering team was growing, and the company realized it needed a better infrastructure to both support that growth and increase velocity.

CertificateSigningRequest

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: certificates.k8s.io/v1

import "k8s.io/api/certificates/v1"

CertificateSigningRequest

CertificateSigningRequest objects provide a mechanism to obtain x509 certificates by submitting a certificate signing request, and having it asynchronously approved and issued.

CSINode

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storage.k8s.io/v1

import "k8s.io/api/storage/v1"

CSINode

CSINode holds information about all CSI drivers installed on a node. CSI drivers do not need to create the CSINode object directly. As long as they use the node-driver-registrar sidecar container, the kubelet will automatically populate the CSINode object for the CSI driver as part of kubelet plugin registration. CSINode has the same name as a node. If the object is missing, it means either there are no CSI Drivers available on the node, or the Kubelet version is low enough that it doesn't create this object. CSINode has an OwnerReference that points to the corresponding node object.

Denso Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

DENSO Corporation is one of the biggest automotive components suppliers in the world. With the advent of connected cars, the company launched a Digital Innovation Department to expand into software, working on vehicle edge and vehicle cloud products. But there were several technical challenges to creating an integrated vehicle edge/cloud platform: "the amount of computing resources, the occasional lack of mobile signal, and an enormous number of distributed vehicles," says R&D Product Manager Seiichi Koizumi.

Ingress

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: networking.k8s.io/v1

import "k8s.io/api/networking/v1"

Ingress

Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend. An Ingress can be configured to give services externally-reachable urls, load balance traffic, terminate SSL, offer name based virtual hosting etc.

IPAddress

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: networking.k8s.io/v1

import "k8s.io/api/networking/v1"

IPAddress

IPAddress represents a single IP of a single IP Family. The object is designed to be used by APIs that operate on IP addresses. The object is used by the Service core API for allocation of IP addresses. An IP address can be represented in different formats, to guarantee the uniqueness of the IP, the name of the object is the IP address in canonical format, four decimal digits separated by dots suppressing leading zeros for IPv4 and the representation defined by RFC 5952 for IPv6. Valid: 192.168.1.5 or 2001:db8::1 or 2001:db8:aaaa:bbbb:cccc:dddd:eeee:1 Invalid: 10.01.2.3 or 2001:db8:0:0:0::1

LocalObjectReference

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.

NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: networking.k8s.io/v1

import "k8s.io/api/networking/v1"

NetworkPolicy

NetworkPolicy describes what network traffic is allowed for a set of Pods

Ocado Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

The world's largest online-only grocery retailer, Ocado developed the Ocado Smart Platform to manage its own operations, from websites to warehouses, and is now licensing the technology to other retailers such as Kroger. To set up the first warehouses for the platform, Ocado shifted from virtual machines and Puppet infrastructure to Docker containers, using CoreOS's fleet scheduler to provision all the services on its OpenStack-based private cloud on bare metal. As the Smart Platform grew and "fleet was going end-of-life," says Platform Engineer Mike Bryant, "we started looking for a more complete platform, with all of these disparate infrastructure services being brought together in one unified API."

ReplicationController

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

ReplicationController

ReplicationController represents the configuration of a replication controller.

apiVersion: v1
kind: ReplicationController

SubjectAccessReview

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authorization.k8s.io/v1

import "k8s.io/api/authorization/v1"

SubjectAccessReview

SubjectAccessReview checks whether or not a user or group can perform an action.

ValidatingWebhookConfiguration

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1

import "k8s.io/api/admissionregistration/v1"

ValidatingWebhookConfiguration

ValidatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and object without changing it.

Building a Basic DaemonSet

Mon, 01 Jan 0001 00:00:00 +0000

This page demonstrates how to build a basic DaemonSet that runs a Pod on every node in a Kubernetes cluster. It covers a simple use case of mounting a file from the host, logging its contents using an init container, and utilizing a pause container.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

ClusterRole

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: rbac.authorization.k8s.io/v1

import "k8s.io/api/rbac/v1"

ClusterRole

ClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding.

ClusterTrustBundle v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: certificates.k8s.io/v1beta1

import "k8s.io/api/certificates/v1beta1"

ClusterTrustBundle

ClusterTrustBundle is a cluster-scoped container for X.509 trust anchors (root certificates).

CSIStorageCapacity

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storage.k8s.io/v1

import "k8s.io/api/storage/v1"

CSIStorageCapacity

CSIStorageCapacity stores the result of one CSI GetCapacity call. For a given StorageClass, this describes the available capacity in a particular topology segment. This can be used when considering where to instantiate new PersistentVolumes.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

Hello Minikube

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to run a sample app on Kubernetes using minikube. The tutorial provides a container image that uses NGINX to echo back all the requests.

Objectives

Deploy a sample application to minikube.
Run the app.
View application logs.

Before you begin

This tutorial assumes that you have already set up minikube. See Step 1 in minikube start for installation instructions.

Note:

Only execute the instructions in Step 1, Installation. The rest is covered on this page.

IngressClass

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: networking.k8s.io/v1

import "k8s.io/api/networking/v1"

IngressClass

IngressClass represents the class of the Ingress, referenced by the Ingress Spec. The ingressclass.kubernetes.io/is-default-class annotation can be used to indicate that an IngressClass should be considered default. When a single IngressClass resource has this annotation set to true, new Ingress resources without a class specified will be assigned this default class.

Lease

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: coordination.k8s.io/v1

import "k8s.io/api/coordination/v1"

Lease

Lease defines a lease concept.

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata (ObjectMeta)

NodeSelectorRequirement

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

PodDisruptionBudget

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: policy/v1

import "k8s.io/api/policy/v1"

PodDisruptionBudget

PodDisruptionBudget is an object to define the max disruption that can be caused to a collection of pods

ReplicaSet

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apps/v1

import "k8s.io/api/apps/v1"

ReplicaSet

ReplicaSet ensures that a specified number of pod replicas are running at any given time.

ClusterRoleBinding

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: rbac.authorization.k8s.io/v1

import "k8s.io/api/rbac/v1"

ClusterRoleBinding

ClusterRoleBinding references a ClusterRole, but not contain it. It can reference a ClusterRole in the global namespace, and adds who information via Subject.

Deployment

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apps/v1

import "k8s.io/api/apps/v1"

Deployment

Deployment enables declarative updates for Pods and ReplicaSets.

apiVersion: apps/v1
kind: Deployment

LeaseCandidate v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: coordination.k8s.io/v1beta1

import "k8s.io/api/coordination/v1beta1"

LeaseCandidate

LeaseCandidate defines a candidate for a Lease object. Candidates are created such that coordinated leader election will pick the best leader from the list of candidates.

ObjectFieldSelector

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

ObjectFieldSelector selects an APIVersioned field of an object.

fieldPath (string), required

Path of the field to select in the specified API version.

PersistentVolumeClaim

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

PersistentVolumeClaim

PersistentVolumeClaim is a user's request for and claim to a persistent volume

PriorityLevelConfiguration

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: flowcontrol.apiserver.k8s.io/v1

import "k8s.io/api/flowcontrol/v1"

PriorityLevelConfiguration

PriorityLevelConfiguration represents the configuration of a priority level.

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration

SelfSubjectReview

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: authentication.k8s.io/v1

import "k8s.io/api/authentication/v1"

SelfSubjectReview

SelfSubjectReview contains the user information that the kube-apiserver has about the user making this request. When using impersonation, users will receive the user info of the user being impersonated. If impersonation or request header authentication is used, any extra keys will have their case ignored and returned as lowercase.

Namespace

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

Namespace

Namespace provides a scope for Names. Use of multiple namespaces is optional.

ObjectMeta

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/apis/meta/v1"

ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create.

PersistentVolume

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

PersistentVolume

PersistentVolume (PV) is a storage resource provisioned by an administrator. It is analogous to a node. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes

PodCertificateRequest v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: certificates.k8s.io/v1beta1

import "k8s.io/api/certificates/v1beta1"

PodCertificateRequest

PodCertificateRequest encodes a pod requesting a certificate from a given signer.

Role

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: rbac.authorization.k8s.io/v1

import "k8s.io/api/rbac/v1"

Role

Role is a namespaced, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding.

StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apps/v1

import "k8s.io/api/apps/v1"

StatefulSet

StatefulSet represents a set of pods with consistent identities. Identities are defined as:

ValidatingAdmissionPolicy

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1

import "k8s.io/api/admissionregistration/v1"

ValidatingAdmissionPolicy

ValidatingAdmissionPolicy describes the definition of an admission validation policy that accepts or rejects an object without changing it.

ControllerRevision

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apps/v1

import "k8s.io/api/apps/v1"

ControllerRevision

ControllerRevision implements an immutable snapshot of state data. Clients are responsible for serializing and deserializing the objects that contain their internal state. Once a ControllerRevision has been successfully created, it can not be updated. The API Server will fail validation of all requests that attempt to mutate the Data field. ControllerRevisions may, however, be deleted. Note that, due to its use by both the DaemonSet and StatefulSet controllers for update and rollback, this object is beta. However, it may be subject to name and representation changes in future releases, and clients should not depend on its stability. It is primarily for internal use by controllers.

Node

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: v1

import "k8s.io/api/core/v1"

Node

Node is a worker node in Kubernetes. Each node will have a unique identifier in the cache (i.e. in etcd).

ObjectReference

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

ObjectReference contains enough information to let you inspect or modify the referred object.

apiVersion (string)

RoleBinding

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: rbac.authorization.k8s.io/v1

import "k8s.io/api/rbac/v1"

RoleBinding

RoleBinding references a role, but does not contain it. It can reference a Role in the same namespace or a ClusterRole in the global namespace. It adds who information via Subjects and namespace information by which namespace it exists in. RoleBindings in a given namespace only have effect in that namespace.

StorageClass

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storage.k8s.io/v1

import "k8s.io/api/storage/v1"

StorageClass

StorageClass describes the parameters for a class of storage for which PersistentVolumes can be dynamically provisioned.

ValidatingAdmissionPolicyBinding

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1

import "k8s.io/api/admissionregistration/v1"

ValidatingAdmissionPolicyBinding

ValidatingAdmissionPolicyBinding binds the ValidatingAdmissionPolicy with paramerized resources. ValidatingAdmissionPolicyBinding and parameter CRDs together define how cluster administrators configure policies for clusters.

DaemonSet

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: apps/v1

import "k8s.io/api/apps/v1"

DaemonSet

DaemonSet represents the configuration of a daemon set.

apiVersion: apps/v1
kind: DaemonSet

MutatingAdmissionPolicy v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1beta1

import "k8s.io/api/admissionregistration/v1beta1"

MutatingAdmissionPolicy

MutatingAdmissionPolicy describes the definition of an admission mutation policy that mutates the object coming into admission chain.

Patch

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/apis/meta/v1"

Patch is provided to give a concrete name and type to the Kubernetes PATCH request body.

RuntimeClass

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: node.k8s.io/v1

import "k8s.io/api/node/v1"

RuntimeClass

RuntimeClass defines a class of container runtime supported in the cluster. The RuntimeClass is used to determine which container runtime is used to run all containers in a pod. RuntimeClasses are manually defined by a user or cluster provisioner, and referenced in the PodSpec. The Kubelet is responsible for resolving the RuntimeClassName reference before running the pod. For more details, see https://kubernetes.io/docs/concepts/containers/runtime-class/

StorageVersionMigration v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storagemigration.k8s.io/v1beta1

import "k8s.io/api/storagemigration/v1beta1"

StorageVersionMigration

StorageVersionMigration represents a migration of stored data to the latest storage version.

Adding Linux worker nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to add Linux worker nodes to a kubeadm cluster.

Before you begin

Each joining worker node has installed the required components from Installing kubeadm, such as, kubeadm, the kubelet and a container runtime.
A running kubeadm cluster created by kubeadm init and following the steps in the document Creating a cluster with kubeadm.
You need superuser access to the node.

Adding Linux worker nodes

To add new Linux worker nodes to your cluster do the following for each machine:

Apply Pod Security Standards at the Cluster Level

Mon, 01 Jan 0001 00:00:00 +0000

Note

This tutorial applies only for new clusters.

Pod Security is an admission controller that carries out checks against the Kubernetes Pod Security Standards when new pods are created. It is a feature GA'ed in v1.25. This tutorial shows you how to enforce the baseline Pod Security Standard at the cluster level which applies a standard configuration to all namespaces in a cluster.

To apply Pod Security Standards to specific namespaces, refer to Apply Pod Security Standards at the namespace level.

Assign Memory Resources to Containers and Pods

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to assign a memory request and a memory limit to a Container. A Container is guaranteed to have as much memory as it requests, but is not allowed to use more memory than its limit.

Before you begin

Audit Annotations

Mon, 01 Jan 0001 00:00:00 +0000

This page serves as a reference for the audit annotations of the kubernetes.io namespace. These annotations apply to Event object from API group audit.k8s.io.

Note:

The following annotations are not used within the Kubernetes API. When you enable auditing in your cluster, audit event data is written using Event from API group audit.k8s.io. The annotations apply to audit events. Audit events are different from objects in the Event API (API group events.k8s.io).

k8s.io/deprecated

Example: k8s.io/deprecated: "true"

Authenticating

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of authentication in Kubernetes, with a focus on authentication to the Kubernetes API.

Users in Kubernetes

All Kubernetes clusters have two categories of users: service accounts managed by Kubernetes, and normal users.

It is assumed that a cluster-independent service manages normal users in the following ways:

an administrator distributing private keys
a user store like Keystone or Google Accounts
a file with a list of usernames and passwords

In this regard, Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.

Available Documentation Versions

Mon, 01 Jan 0001 00:00:00 +0000

This website contains documentation for the current version of Kubernetes and the four previous versions of Kubernetes.

The availability of documentation for a Kubernetes version is separate from whether that release is currently supported. Read Support period to learn about which versions of Kubernetes are officially supported, and for how long.

Changing the Container Runtime on a Node from Docker Engine to containerd

Mon, 01 Jan 0001 00:00:00 +0000

This task outlines the steps needed to update your container runtime to containerd from Docker. It is applicable for cluster operators running Kubernetes 1.23 or earlier. This also covers an example scenario for migrating from dockershim to containerd. Alternative container runtimes can be picked from this page.

Before you begin

Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide before submitting a change. More information.

Install containerd. For more information see containerd's installation documentation and for specific prerequisite follow the containerd guide.

Cloud Native Security and Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes is based on a cloud native architecture and draws on advice from the CNCF about good practices for cloud native information security.

Read on for an overview of how Kubernetes is designed to help you deploy a secure cloud native platform.

Cloud native information security

The CNCF white paper on cloud native security defines security controls and practices that are appropriate to different lifecycle phases.

Configure Default Memory Requests and Limits for a Namespace

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure default memory requests and limits for a namespace.

A Kubernetes cluster can be divided into namespaces. Once you have a namespace that has a default memory limit, and you then try to create a Pod with a container that does not specify its own memory limit, then the control plane assigns the default memory limit to that container.

Configure the Aggregation Layer

Mon, 01 Jan 0001 00:00:00 +0000

Configuring the aggregation layer allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.

Before you begin

Considerations for large clusters

Mon, 01 Jan 0001 00:00:00 +0000

A cluster is a set of nodes (physical or virtual machines) running Kubernetes agents, managed by the control plane. Kubernetes v1.35 supports clusters with up to 5,000 nodes. More specifically, Kubernetes is designed to accommodate configurations that meet all of the following criteria:

Documentation Content Guide

Mon, 01 Jan 0001 00:00:00 +0000

This page contains guidelines for Kubernetes documentation.

If you have questions about what's allowed, join the #sig-docs channel in Kubernetes Slack and ask!

You can register for Kubernetes Slack at https://slack.k8s.io/.

For information on creating new content for the Kubernetes docs, follow the style guide.

Overview

Source for the Kubernetes website, including the docs, resides in the kubernetes/website repository.

Located in the kubernetes/website/content/<language_code>/docs folder, the majority of Kubernetes documentation is specific to the Kubernetes project.

Custom Resources

Mon, 01 Jan 0001 00:00:00 +0000

Custom resources are extensions of the Kubernetes API. This page discusses when to add a custom resource to your Kubernetes cluster and when to use a standalone service. It describes the two methods for adding custom resources and how to choose between them.

Custom resources

A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in pods resource contains a collection of Pod objects.

Debug Pods

Mon, 01 Jan 0001 00:00:00 +0000

This guide is to help users debug applications that are deployed into Kubernetes and not behaving correctly. This is not a guide for people who want to debug their cluster. For that you should check out this guide.

Diagnosing the problem

The first step in troubleshooting is triage. What is the problem? Is it your Pods, your Replication Controller or your Service?

Debugging Pods
Debugging Replication Controllers
Debugging Services

Debugging Pods

The first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events with the following command:

Declarative Management of Kubernetes Objects Using Configuration Files

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed. This method retains writes made to live objects without merging the changes back into the object configuration files. kubectl diff also gives you a preview of what changes apply will make.

Before you begin

Install kubectl.

Define a Command and Arguments for a Container

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to define commands and arguments when you run a container in a Pod.

Before you begin

Deploy and Access the Kubernetes Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Dashboard is deprecated and unmaintained.

The Kubernetes Dashboard project has been archived and is no longer actively maintained. For new installations, consider using Headlamp.

Note:

For in-cluster deployments similar to Kubernetes Dashboard, see the Headlamp in-cluster installation guide.

Dashboard is a web-based Kubernetes user interface. You can use Dashboard to deploy containerized applications to a Kubernetes cluster, troubleshoot your containerized application, and manage the cluster resources. You can use Dashboard to get an overview of applications running on your cluster, as well as for creating or modifying individual Kubernetes resources (such as Deployments, Jobs, DaemonSets, etc). For example, you can scale a Deployment, initiate a rolling update, restart a pod or deploy new applications using a deploy wizard.

Deployments

Mon, 01 Jan 0001 00:00:00 +0000

A Deployment provides declarative updates for Pods and ReplicaSets.

You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.

Exposing an External IP Address to Access an Application in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to create a Kubernetes Service object that exposes an external IP address.

Before you begin

Install kubectl.
Use a cloud provider like Google Kubernetes Engine or Amazon Web Services to create a Kubernetes cluster. This tutorial creates an external load balancer, which requires a cloud provider.
Configure kubectl to communicate with your Kubernetes API server. For instructions, see the documentation for your cloud provider.

Objectives

Run five instances of a Hello World application.
Create a Service object that exposes an external IP address.
Use the Service object to access the running application.

Creating a service for an application running in five pods

Run a Hello World application in your cluster:

Feature Gates

Mon, 01 Jan 0001 00:00:00 +0000

This page contains an overview of the various feature gates an administrator can specify on different Kubernetes components.

See feature stages for an explanation of the stages for a feature.

Overview

Feature gates are a set of key=value pairs that describe Kubernetes features. You can turn these features on or off using the --feature-gates command line flag on each Kubernetes component.

How to enable Feature Gates

To enable or disable a feature gate for a particular Kubernetes component, use the --feature-gates flag.

Images

Mon, 01 Jan 0001 00:00:00 +0000

A container image represents binary data that encapsulates an application and all its software dependencies. Container images are executable software bundles that can run standalone and that make very well-defined assumptions about their runtime environment.

You typically create a container image of your application and push it to a registry before referring to it in a Pod.

Install and Set Up kubectl on Linux

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

You must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.35 client can communicate with v1.34, v1.35, and v1.36 control planes. Using the latest compatible version of kubectl helps avoid unforeseen issues.

Install kubectl on Linux

The following methods exist for installing kubectl on Linux:

Install kubectl binary with curl on Linux
Install using native package management
Install using other package management

Install kubectl binary with curl on Linux

Download the latest release with the command:

Install and Set Up kubectl on macOS

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install kubectl on macOS

The following methods exist for installing kubectl on macOS:

Install kubectl on macOS
Verify kubectl configuration
Optional kubectl configurations and plugins
- Enable shell autocompletion
- Install kubectl convert plugin

Install kubectl binary with curl on macOS

Download the latest release:

Install and Set Up kubectl on Windows

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install kubectl on Windows

The following methods exist for installing kubectl on Windows:

Install kubectl binary on Windows (via direct download or curl)
Install on Windows using Chocolatey, Scoop, or winget

Install kubectl binary on Windows (via direct download or curl)

You have two options for installing kubectl on your Windows device

Installing kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to install the kubeadm toolbox. For information on how to create a cluster with kubeadm once you have performed this installation process, see the Creating a cluster with kubeadm page.

This installation guide is for Kubernetes v1.35. If you want to use a different Kubernetes version, please refer to the following pages instead:

Before you begin

A compatible Linux host. The Kubernetes project provides generic instructions for Linux distributions based on Debian and Red Hat, and those distributions without a package manager.
2 GB or more of RAM per machine (any less will leave little room for your apps).
2 CPUs or more for control plane machines.
Full network connectivity between all machines in the cluster (public or private network is fine).
Unique hostname, MAC address, and product_uuid for every node. See here for more details.
Certain ports are open on your machines. See here for more details.

Note:

The kubeadm installation is done via binaries that use dynamic linking and assumes that your target system provides glibc. This is a reasonable assumption on many Linux distributions (including Debian, Ubuntu, Fedora, CentOS, etc.) but it is not always the case with custom and lightweight distributions which don't include glibc by default, such as Alpine Linux. The expectation is that the distribution either includes glibc or a compatibility layer that provides the expected symbols.

The kubeadm project supports LTS kernels. See List of LTS kernels.
You can get the kernel version using the command uname -r

For more information, see Linux Kernel Requirements.

Job

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: batch/v1

import "k8s.io/api/batch/v1"

Job

Job represents the configuration of a single job.

apiVersion: batch/v1
kind: Job

kubectl Quick Reference

Mon, 01 Jan 0001 00:00:00 +0000

This page contains a list of commonly used kubectl commands and flags.

Note:

These instructions are for Kubernetes v1.35. To check the version, use the kubectl version command.

Kubectl autocomplete

BASH

source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.

You can also use a shorthand alias for kubectl that also works with completion:

Kubelet Checkpoint API

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.30 [beta](enabled by default)

Checkpointing a container is the functionality to create a stateful copy of a running container. Once you have a stateful copy of a container, you could move it to a different computer for debugging or similar purposes.

If you move the checkpointed container data to a computer that's able to restore it, that restored container continues to run at exactly the same point it was checkpointed. You can also inspect the saved data, provided that you have suitable tools for doing so.

Kubernetes Components

Mon, 01 Jan 0001 00:00:00 +0000

This page provides a high-level overview of the essential components that make up a Kubernetes cluster.

The components of a Kubernetes cluster

Core Components

A Kubernetes cluster consists of a control plane and one or more worker nodes. Here's a brief overview of the main components:

Control Plane Components

Manage the overall state of the cluster:

kube-apiserver: The core component server that exposes the Kubernetes HTTP API.
etcd: Consistent and highly-available key value store for all API server data.
kube-scheduler: Looks for Pods not yet bound to a node, and assigns each Pod to a suitable node.
kube-controller-manager: Runs controllers to implement Kubernetes API behavior.
cloud-controller-manager (optional): Integrates with underlying cloud provider(s).

Node Components

Run on every node, maintaining running pods and providing the Kubernetes runtime environment:

Kubernetes Issue Tracker

Mon, 01 Jan 0001 00:00:00 +0000

To report a security issue, please follow the Kubernetes security disclosure process.

Work on Kubernetes code and public issues are tracked using GitHub Issues.

Official list of known CVEs (security vulnerabilities) that have been announced by the Security Response Committee
CVE-related GitHub issues

Security-related announcements are sent to the kubernetes-security-announce@googlegroups.com mailing list.

Kubernetes Scheduler

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, scheduling refers to making sure that Pods are matched to Nodes so that Kubelet can run them.

Scheduling overview

A scheduler watches for newly created Pods that have no Node assigned. For every Pod that the scheduler discovers, the scheduler becomes responsible for finding the best Node for that Pod to run on. The scheduler reaches this placement decision taking into account the scheduling principles described below.

Limit Ranges

Mon, 01 Jan 0001 00:00:00 +0000

By default, containers run with unbounded compute resources on a Kubernetes cluster. Using Kubernetes resource quotas, administrators (also termed cluster operators) can restrict consumption and creation of cluster resources (such as CPU time, memory, and persistent storage) within a specified namespace. Within a namespace, a Pod can consume as much CPU and memory as is allowed by the ResourceQuotas that apply to that namespace. As a cluster operator, or as a namespace-level administrator, you might also be concerned about making sure that a single object cannot monopolize all available resources within a namespace.

Linux Kernel Version Requirements

Mon, 01 Jan 0001 00:00:00 +0000

Many features rely on specific kernel functionalities and have minimum kernel version requirements. However, relying solely on kernel version numbers may not be sufficient for certain operating system distributions, as maintainers for distributions such as RHEL, Ubuntu and SUSE often backport selected features to older kernel releases (retaining the older kernel version).

Managing Secrets using kubectl

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to create, edit, manage, and delete Kubernetes Secrets using the kubectl command-line tool.

Before you begin

MutatingAdmissionPolicyBinding v1alpha1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: admissionregistration.k8s.io/v1alpha1

import "k8s.io/api/admissionregistration/v1alpha1"

MutatingAdmissionPolicyBinding

MutatingAdmissionPolicyBinding binds the MutatingAdmissionPolicy with parametrized resources. MutatingAdmissionPolicyBinding and the optional parameter resource together define how cluster administrators configure policies for clusters.

Network Plugins

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes (version 1.3 through to the latest 1.35, and likely onwards) lets you use Container Network Interface (CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with your cluster and that suits your needs. Different plugins are available (both open- and closed- source) in the wider Kubernetes ecosystem.

A CNI plugin is required to implement the Kubernetes network model.

You must use a CNI plugin that is compatible with the v0.4.0 or later releases of the CNI specification. The Kubernetes project recommends using a plugin that is compatible with the v1.0.0 CNI specification (plugins can be compatible with multiple spec versions).

Node Shutdowns

Mon, 01 Jan 0001 00:00:00 +0000

In a Kubernetes cluster, a node can be shut down in a planned graceful way or unexpectedly because of reasons such as a power outage or something else external. A node shutdown could lead to workload failure if the node is not drained before the shutdown. A node shutdown can be either graceful or non-graceful.

Graceful node shutdown

The kubelet attempts to detect node system shutdown and terminates pods running on the node.

Nodes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes runs your workload by placing containers into Pods to run on Nodes. A node may be a virtual or physical machine, depending on the cluster. Each node is managed by the control plane and contains the services necessary to run Pods.

Opening a pull request

Mon, 01 Jan 0001 00:00:00 +0000

Note:

Code developers: If you are documenting a new feature for an upcoming Kubernetes release, see Document a new feature.

To contribute new content pages or improve existing content pages, open a pull request (PR). Make sure you follow all the requirements in the Before you begin section.

If your change is small, or you're unfamiliar with git, read Changes using GitHub to learn how to edit a page.

If your changes are large, read Work from a local fork to learn how to make changes locally on your computer.

Overprovision Node Capacity For A Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page guides you through configuring Node overprovisioning in your Kubernetes cluster. Node overprovisioning is a strategy that proactively reserves a portion of your cluster's compute resources. This reservation helps reduce the time required to schedule new pods during scaling events, enhancing your cluster's responsiveness to sudden spikes in traffic or workload demands.

By maintaining some unused capacity, you ensure that resources are immediately available when new pods are created, preventing them from entering a pending state while the cluster scales up.

Perform a Rolling Update on a DaemonSet

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to perform a rolling update on a DaemonSet.

Before you begin

Performing a Rolling Update

Mon, 01 Jan 0001 00:00:00 +0000

Objectives

Perform a rolling update using kubectl.

Updating an application

Rolling updates allow Deployments' update to take place with zero downtime by incrementally updating Pods instances with new ones.

Users expect applications to be available all the time, and developers are expected to deploy new versions of them several times a day. In Kubernetes this is done with rolling updates. A rolling update allows a Deployment update to take place with zero downtime. It does this by incrementally replacing the current Pods with new ones. The new Pods are scheduled on Nodes with available resources, and Kubernetes waits for those new Pods to start before removing the old Pods.

Pod Group Policies

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [alpha](disabled by default)

Every pod group defined in a Workload must declare a scheduling policy. This policy dictates how the scheduler treats the collection of Pods.

Policy types

The API currently supports two policy types: basic and gang. You must specify exactly one policy for each group.

Basic policy

The basic policy instructs the scheduler to treat all Pods in the group as independent entities, scheduling them using the standard Kubernetes behavior.

Protocols for Services

Mon, 01 Jan 0001 00:00:00 +0000

If you configure a Service, you can select from any network protocol that Kubernetes supports.

Kubernetes supports the following protocols with Services:

SCTP
TCP (the default)
UDP

When you define a Service, you can also specify the application protocol that it uses.

This document details some special cases, all of them typically using TCP as a transport protocol:

Quantity

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/api/resource"

Quantity is a fixed-point representation of a number. It provides convenient marshaling/unmarshaling in JSON and YAML, in addition to String() and AsInt64() accessors.

Reference Documentation Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use the update-imported-docs.py script to generate the Kubernetes reference documentation. The script automates the build setup and generates the reference documentation for a release.

Before you begin

Requirements:

You need a machine that is running Linux or macOS.
You need to have these tools installed:
- Python v3.7.x+
- Git
- Golang version 1.13+
- Pip used to install PyYAML
- PyYAML v5.1.2
- make
- gcc compiler/linker
- Docker (Required only for kubectl command reference)
Your PATH environment variable must include the required build tools, such as the Go binary and python.

Reviewing pull requests

Mon, 01 Jan 0001 00:00:00 +0000

Anyone can review a documentation pull request. Visit the pull requests section in the Kubernetes website repository to see open pull requests.

Reviewing documentation pull requests is a great way to introduce yourself to the Kubernetes community. It helps you learn the code base and build trust with other contributors.

Before reviewing, it's a good idea to:

Read the content guide and style guide so you can leave informed comments.
Understand the different roles and responsibilities in the Kubernetes documentation community.

Before you begin

Before you start a review:

Roles and responsibilities

Mon, 01 Jan 0001 00:00:00 +0000

Anyone can contribute to Kubernetes. As your contributions to SIG Docs grow, you can apply for different levels of membership in the community. These roles allow you to take on more responsibility within the community. Each role requires more time and commitment. The roles are:

Anyone: regular contributors to the Kubernetes documentation
Members: can assign and triage issues and provide non-binding review on pull requests
Reviewers: can lead reviews on documentation pull requests and can vouch for a change's quality
Approvers: can lead reviews on documentation and merge changes

Anyone

Anyone with a GitHub account can contribute to Kubernetes. SIG Docs welcomes all new contributors!

Run a Stateless Application Using a Deployment

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to run an application using a Kubernetes Deployment object.

Objectives

Create an nginx deployment.
Use kubectl to list information about the deployment.
Update the deployment.

Before you begin

Running Automated Tasks with a CronJob

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to run automated tasks using Kubernetes CronJob object.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Running Kubelet in Standalone Mode

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to run a standalone kubelet instance.

You may have different motivations for running a standalone kubelet. This tutorial is aimed at introducing you to Kubernetes, even if you don't have much experience with it. You can follow this tutorial and learn about node setup, basic (static) Pods, and how Kubernetes manages containers.

Once you have followed this tutorial, you could try using a cluster that has a control plane to manage pods and nodes, and other types of objects. For example, Hello, minikube.

Running Multiple Instances of Your App

Mon, 01 Jan 0001 00:00:00 +0000

Objectives

Scale an existing app manually using kubectl.

Scaling an application

You can create from the start a Deployment with multiple instances using the --replicas parameter for the kubectl create deployment command.

Previously we created a Deployment, and then exposed it publicly via a Service. The Deployment created only one Pod for running our application. When traffic increases, we will need to scale the application to keep up with user demand.

Service

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, a Service is a method for exposing a network application that is running as one or more Pods in your cluster.

A key aim of Services in Kubernetes is that you don't need to modify your existing application to use an unfamiliar service discovery mechanism. You can run code in Pods, whether this is a code designed for a cloud-native world, or an older app you've containerized. You use a Service to make that set of Pods available on the network so that clients can interact with it.

ServiceCIDR

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: networking.k8s.io/v1

import "k8s.io/api/networking/v1"

ServiceCIDR

ServiceCIDR defines a range of IP addresses using CIDR format (e.g. 192.168.0.0/24 or 2001:db2::/64). This range is used to allocate ClusterIPs to Service objects.

Set Up DRA in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [stable](enabled by default)

This page shows you how to configure dynamic resource allocation (DRA) in a Kubernetes cluster by enabling API groups and configuring classes of devices. These instructions are for cluster administrators.

About DRA

A Kubernetes feature that lets you request and share resources among Pods. These resources are often attached devices like hardware accelerators.

StatefulSet Basics

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial provides an introduction to managing applications with StatefulSets. It demonstrates how to create, delete, scale, and update the Pods of StatefulSets.

Before you begin

Before you begin this tutorial, you should familiarize yourself with the following Kubernetes concepts:

Swap memory management

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes can be configured to use swap memory on a node, allowing the kernel to free up physical memory by swapping out pages to backing storage. This is useful for multiple use-cases. For example, nodes running workloads that can benefit from using swap, such as those that have large memory footprints but only access a portion of that memory at any given time. It also helps prevent Pods from being terminated during memory pressure spikes, shields nodes from system-level memory spikes that might compromise its stability, allows for more flexible memory management on the node, and much more.

Troubleshooting kubectl

Mon, 01 Jan 0001 00:00:00 +0000

This documentation is about investigating and diagnosing kubectl related issues. If you encounter issues accessing kubectl or connecting to your cluster, this document outlines various common scenarios and potential solutions to help identify and address the likely cause.

Before you begin

You need to have a Kubernetes cluster.
You also need to have kubectl installed - see install tools

Verify kubectl setup

Make sure you have installed and configured kubectl correctly on your local machine. Check the kubectl version to ensure it is up-to-date and compatible with your cluster.

Use Antrea for NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to install and use Antrea CNI plugin on Kubernetes. For background on Project Antrea, read the Introduction to Antrea.

Before you begin

You need to have a Kubernetes cluster. Follow the kubeadm getting started guide to bootstrap one.

Deploying Antrea with kubeadm

Follow Getting Started guide to deploy Antrea for kubeadm.

What's next

Once your cluster is running, you can follow the Declare Network Policy to try out Kubernetes NetworkPolicy.

Using a Service to Expose Your App

Mon, 01 Jan 0001 00:00:00 +0000

Objectives

Learn about a Service in Kubernetes.
Understand how labels and selectors relate to a Service.
Expose an application outside a Kubernetes cluster.

Overview of Kubernetes Services

Kubernetes Pods are mortal. Pods have a lifecycle. When a worker node dies, the Pods running on the Node are also lost. A Replicaset might then dynamically drive the cluster back to the desired state via the creation of new Pods to keep your application running. As another example, consider an image-processing backend with 3 replicas. Those replicas are exchangeable; the front-end system should not care about backend replicas or even if a Pod is lost and recreated. That said, each Pod in a Kubernetes cluster has a unique IP address, even Pods on the same Node, so there needs to be a way of automatically reconciling changes among Pods so that your applications continue to function.

Using kubectl to Create a Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Objectives

Learn about application Deployments.
Deploy your first app on Kubernetes with kubectl.

Kubernetes Deployments

A Deployment is responsible for creating and updating instances of your application.

Note:

This tutorial uses a container that requires the AMD64 architecture. If you are using minikube on a computer with a different CPU architecture, you could try using minikube with a driver that can emulate AMD64. For example, the Docker Desktop driver can do this.

Once you have a running Kubernetes cluster, you can deploy your containerized applications on top of it. To do so, you create a Kubernetes Deployment. The Deployment instructs Kubernetes how to create and update instances of your application. Once you've created a Deployment, the Kubernetes control plane schedules the application instances included in that Deployment to run on individual Nodes in the cluster.

Using Minikube to Create a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Objectives

Learn what a Kubernetes cluster is.
Learn what Minikube is.
Start a Kubernetes cluster on your computer.

Kubernetes Clusters

Kubernetes is a production-grade, open-source platform that orchestrates the placement (scheduling) and execution of application containers within and across computer clusters.

Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit. The abstractions in Kubernetes allow you to deploy containerized applications to a cluster without tying them specifically to individual machines. To make use of this new model of deployment, applications need to be packaged in a way that decouples them from individual hosts: they need to be containerized. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way. Kubernetes is an open-source platform and is production-ready.

Viewing Pods and Nodes

Mon, 01 Jan 0001 00:00:00 +0000

Objectives

Learn about Kubernetes Pods.
Learn about Kubernetes Nodes.
Troubleshoot deployed applications.

Kubernetes Pods

A Pod is a group of one or more application containers (such as Docker) and includes shared storage (volumes), IP address and information about how to run them.

When you created a Deployment in Module 2, Kubernetes created a Pod to host your application instance. A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. Those resources include:

Volume

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

Volume

Volume represents a named volume in a pod that may be accessed by any container in the pod.

Volumes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes volumes provide a way for containers in a pod to access and share data via the filesystem. There are different kinds of volume that you can use for different purposes, such as:

populating a configuration file based on a ConfigMap or a Secret
providing some temporary scratch space for a pod
sharing a filesystem between two different containers in the same pod
sharing a filesystem between two different pods (even if those Pods run on different nodes)
durably storing data so that it stays available even if the Pod restarts or is replaced
passing configuration information to an app running in a container, based on details of the Pod the container is in (for example: telling a sidecar container what namespace the Pod is running in)
providing read-only access to data in a different container image

Data sharing can be between different local processes within a container, or between different containers, or between Pods.

Adding Windows worker nodes

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.18 [beta]

This page explains how to add Windows worker nodes to a kubeadm cluster.

Before you begin

A running Windows Server 2022 (or higher) instance with administrative access.
A running kubeadm cluster created by kubeadm init and following the steps in the document Creating a cluster with kubeadm.

Adding Windows worker nodes

Note:

To facilitate the addition of Windows worker nodes to a cluster, PowerShell scripts from the repository https://sigs.k8s.io/sig-windows-tools are used.

Do the following for each machine:

Common Parameters

Mon, 01 Jan 0001 00:00:00 +0000

allowWatchBookmarks

allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.

CronJob

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: batch/v1

import "k8s.io/api/batch/v1"

CronJob

CronJob represents the configuration of a single cron job.

apiVersion: batch/v1

Previewing locally

Mon, 01 Jan 0001 00:00:00 +0000

Before you're going to open a new PR, previewing your changes is recommended. A preview lets you catch build errors or markdown formatting problems.

Preview your changes locally

You can either build the website's container image or run Hugo locally. Building the container image is slower but displays Hugo shortcodes, which can be useful for debugging.

Hugo in a container
Hugo on the command line

Note:

The commands below use Docker as default container engine. Set the CONTAINER_ENGINE environment variable to override this behaviour.

Build the container image locally
You only need this step if you are testing a change to the Hugo tool itself

ResourceFieldSelector

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

ResourceFieldSelector represents container resources (cpu, memory) and their output format

resource (string), required

Required: resource to select

VolumeAttachment

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storage.k8s.io/v1

import "k8s.io/api/storage/v1"

VolumeAttachment

VolumeAttachment captures the intent to attach or detach the specified volume to/from the specified node.

HorizontalPodAutoscaler

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: autoscaling/v1

import "k8s.io/api/autoscaling/v1"

HorizontalPodAutoscaler

configuration of a horizontal pod autoscaler.

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata (ObjectMeta)

Status

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/apimachinery/pkg/apis/meta/v1"

Status is a return value for calls that don't return other objects.

apiVersion (string)

VolumeAttributesClass

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: storage.k8s.io/v1

import "k8s.io/api/storage/v1"

VolumeAttributesClass

VolumeAttributesClass represents a specification of mutable volume attributes defined by the CSI driver. The class can be specified during dynamic provisioning of PersistentVolumeClaims, and changed in the PersistentVolumeClaim spec after provisioning.

HorizontalPodAutoscaler

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: autoscaling/v2

import "k8s.io/api/autoscaling/v2"

HorizontalPodAutoscaler

HorizontalPodAutoscaler is the configuration for a horizontal pod autoscaler, which automatically manages the replica count of any resource implementing the scale subresource based on the metrics specified.

TypedLocalObjectReference

Mon, 01 Jan 0001 00:00:00 +0000

import "k8s.io/api/core/v1"

TypedLocalObjectReference contains enough information to let you locate the typed referenced object inside the same namespace.

PriorityClass

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: scheduling.k8s.io/v1

import "k8s.io/api/scheduling/v1"

PriorityClass

PriorityClass defines mapping from a priority class name to the priority integer value. The value can be any valid integer.

DeviceTaintRule v1alpha3

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: resource.k8s.io/v1alpha3

import "k8s.io/api/resource/v1alpha3"

DeviceTaintRule

DeviceTaintRule adds one taint to all devices which match the selector. This has the same effect as if the taint was specified directly in the ResourceSlice by the DRA driver.

Feature Gates (removed)

Mon, 01 Jan 0001 00:00:00 +0000

This page contains list of feature gates that have been removed. The information on this page is for reference. A removed feature gate is different from a GA'ed or deprecated one in that a removed one is no longer recognized as a valid feature gate. However, a GA'ed or a deprecated feature gate is still recognized by the corresponding Kubernetes components although they are unable to cause any behavior differences in a cluster.

Node Autoscaling

Mon, 01 Jan 0001 00:00:00 +0000

In order to run workloads in your cluster, you need Nodes. Nodes in your cluster can be autoscaled - dynamically provisioned, or consolidated to provide needed capacity while optimizing cost. Autoscaling is performed by Node autoscalers.

Node provisioning

If there are Pods in a cluster that can't be scheduled on existing Nodes, new Nodes can be automatically added to the cluster—provisioned—to accommodate the Pods. This is especially useful if the number of Pods changes over time, for example as a result of combining horizontal workload with Node autoscaling.

Pod Security Standards

Mon, 01 Jan 0001 00:00:00 +0000

The Pod Security Standards define three different policies to broadly cover the security spectrum. These policies are cumulative and range from highly-permissive to highly-restrictive. This guide outlines the requirements of each policy.

Profile	Description
Privileged	Unrestricted policy, providing the widest possible level of permissions. This policy allows for known privilege escalations.
Baseline	Minimally restrictive policy which prevents known privilege escalations. Allows the default (minimally specified) Pod configuration.
Restricted	Heavily restricted policy, following current Pod hardening best practices.

Profile Details

Privileged

The Privileged policy is purposely-open, and entirely unrestricted. This type of policy is typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users.

Resource metrics pipeline

Mon, 01 Jan 0001 00:00:00 +0000

For Kubernetes, the Metrics API offers a basic set of metrics to support automatic scaling and similar use cases. This API makes information available about resource usage for node and pod, including metrics for CPU and memory. If you deploy the Metrics API into your cluster, clients of the Kubernetes API can then query for this information, and you can use Kubernetes' access control mechanisms to manage permissions to do so.

Set up an Extension API Server

Mon, 01 Jan 0001 00:00:00 +0000

Setting up an extension API server to work with the aggregation layer allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.

Before you begin

Tools for Monitoring Resources

Mon, 01 Jan 0001 00:00:00 +0000

To scale an application and provide a reliable service, you need to understand how the application behaves when it is deployed. You can examine application performance in a Kubernetes cluster by examining the containers, pods, services, and the characteristics of the overall cluster. Kubernetes provides detailed information about an application's resource usage at each of these levels. This information allows you to evaluate your application's performance and where bottlenecks can be removed to improve overall performance.

ResourceClaim

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: resource.k8s.io/v1

import "k8s.io/api/resource/v1"

ResourceClaim

ResourceClaim describes a request for access to resources in the cluster, for use by workloads. For example, if a workload needs an accelerator device with specific properties, this is how that request is expressed. The status stanza tracks whether this claim has been satisfied and what specific resources have been allocated.

ResourceClaimTemplate

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: resource.k8s.io/v1

import "k8s.io/api/resource/v1"

ResourceClaimTemplate

ResourceClaimTemplate is used to produce ResourceClaim objects.

This is an alpha type and requires enabling the DynamicResourceAllocation feature gate.

ResourceSlice

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: resource.k8s.io/v1

import "k8s.io/api/resource/v1"

ResourceSlice

ResourceSlice represents one or more resources in a pool of similar resources, managed by a common driver. A pool may span more than one ResourceSlice, and exactly how many ResourceSlices comprise a pool is determined by the driver.

Workload v1alpha1

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: scheduling.k8s.io/v1alpha1

import "k8s.io/api/scheduling/v1alpha1"

Workload

Workload allows for expressing scheduling constraints that should be used when managing lifecycle of workloads from scheduling perspective, including scheduling, preemption, eviction and other phases.

Accessing Clusters

Mon, 01 Jan 0001 00:00:00 +0000

This topic discusses multiple ways to interact with clusters.

Accessing for the first time with kubectl

When accessing the Kubernetes API for the first time, we suggest using the Kubernetes CLI, kubectl.

To access a cluster, you need to know the location of the cluster and have credentials to access it. Typically, this is automatically set-up when you work through a Getting started guide, or someone else set up the cluster and provided you with credentials and a location.

Allocate Devices to Workloads with DRA

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [stable](enabled by default)

This page shows you how to allocate devices to your Pods by using dynamic resource allocation (DRA). These instructions are for workload operators. Before reading this page, familiarize yourself with how DRA works and with DRA terminology like ResourceClaims and ResourceClaimTemplates. For more information, see Dynamic Resource Allocation (DRA).

Apply Pod Security Standards at the Namespace Level

Mon, 01 Jan 0001 00:00:00 +0000

Note

This tutorial applies only for new clusters.

Pod Security Admission is an admission controller that applies Pod Security Standards when pods are created. It is a feature GA'ed in v1.25. In this tutorial, you will enforce the baseline Pod Security Standard, one namespace at a time.

You can also apply Pod Security Standards to multiple namespaces at once at the cluster level. For instructions, refer to Apply Pod Security Standards at the cluster level.

Articles on dockershim Removal and on Using CRI-compatible Runtimes

Mon, 01 Jan 0001 00:00:00 +0000

This is a list of articles and other pages that are either about the Kubernetes' deprecation and removal of dockershim, or about using CRI-compatible container runtimes, in connection with that removal.

Kubernetes project

Kubernetes blog: Dockershim Removal FAQ (originally published 2020/12/02)
Kubernetes blog: Updated: Dockershim Removal FAQ (updated published 2022/02/17)
Kubernetes blog: Kubernetes is Moving on From Dockershim: Commitments and Next Steps (published 2022/01/07)
Kubernetes blog: Dockershim removal is coming. Are you ready? (published 2021/11/12)

Assign CPU Resources to Containers and Pods

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to assign a CPU request and a CPU limit to a container. Containers cannot use more CPU than the configured limit. Provided the system has CPU time free, a container is guaranteed to be allocated as much CPU as it requests.

Before you begin

Assigning Pods to Nodes

Mon, 01 Jan 0001 00:00:00 +0000

You can constrain a Pod so that it is restricted to run on particular node(s), or to prefer to run on particular nodes. There are several ways to do this and the recommended approaches all use label selectors to facilitate the selection. Often, you do not need to set any such constraints; the scheduler will automatically do a reasonable placement (for example, spreading your Pods across nodes so as not place Pods on a node with insufficient free resources). However, there are some circumstances where you may want to control which node the Pod deploys to, for example, to ensure that a Pod ends up on a node with an SSD attached to it, or to co-locate Pods from two different services that communicate a lot into the same availability zone.

Authenticating with Bootstrap Tokens

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.18 [stable]

Bootstrap tokens are a simple bearer token that is meant to be used when creating new clusters or joining new nodes to an existing cluster. It was built to support kubeadm, but can be used in other contexts for users that wish to start clusters without kubeadm. It is also built to work, via RBAC policy, with the kubelet TLS Bootstrapping system.

Bootstrap Tokens Overview

Bootstrap Tokens are defined with a specific type (bootstrap.kubernetes.io/token) of secrets that lives in the kube-system namespace. These Secrets are then read by the Bootstrap Authenticator in the API Server. Expired tokens are removed with the TokenCleaner controller in the Controller Manager. The tokens are also used to create a signature for a specific ConfigMap used in a "discovery" process through a BootstrapSigner controller.

Certificates

Mon, 01 Jan 0001 00:00:00 +0000

To learn how to generate certificates for your cluster, see Certificates.

Coarse Parallel Processing Using a Work Queue

Mon, 01 Jan 0001 00:00:00 +0000

In this example, you will run a Kubernetes Job with multiple parallel worker processes.

In this example, as each pod is created, it picks up one unit of work from a task queue, completes it, deletes it from the queue, and exits.

Here is an overview of the steps in this example:

Start a message queue service. In this example, you use RabbitMQ, but you could use another one. In practice you would set up a message queue service once and reuse it for many jobs.
Create a queue, and fill it with messages. Each message represents one task to be done. In this example, a message is an integer that we will do a lengthy computation on.
Start a Job that works on tasks from the queue. The Job starts several pods. Each pod takes one task from the message queue, processes it, and exits.

Before you begin

You should already be familiar with the basic, non-parallel, use of Job.

Communication between Nodes and the Control Plane

Mon, 01 Jan 0001 00:00:00 +0000

This document catalogs the communication paths between the API server and the Kubernetes cluster. The intent is to allow users to customize their installation to harden the network configuration such that the cluster can be run on an untrusted network (or on fully public IPs on a cloud provider).

ConfigMaps

Mon, 01 Jan 0001 00:00:00 +0000

A ConfigMap is an API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume.

A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.

Configure Default CPU Requests and Limits for a Namespace

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure default CPU requests and limits for a namespace.

A Kubernetes cluster can be divided into namespaces. If you create a Pod within a namespace that has a default CPU limit, and any container in that Pod does not specify its own CPU limit, then the control plane assigns the default CPU limit to that container.

Configure Multiple Schedulers

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes ships with a default scheduler that is described here. If the default scheduler does not suit your needs you can implement your own scheduler. Moreover, you can even run multiple schedulers simultaneously alongside the default scheduler and instruct Kubernetes what scheduler to use for each of your pods. Let's learn how to run multiple schedulers in Kubernetes with an example.

A detailed description of how to implement a scheduler is outside the scope of this document. Please refer to the kube-scheduler implementation in pkg/scheduler in the Kubernetes source directory for a canonical example.

Connecting Applications with Services

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes model for connecting containers

Now that you have a continuously running, replicated application you can expose it on a network.

Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. Kubernetes gives every pod its own cluster-private IP address, so you do not need to explicitly create links between pods or map container ports to host ports. This means that containers within a Pod can all reach each other's ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document elaborates on how you can run reliable services on such a networking model.

Container Environment

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the resources available to Containers in the Container environment.

Container environment

The Kubernetes Container environment provides several important resources to Containers:

A filesystem, which is a combination of an image and one or more volumes.
Information about the Container itself.
Information about other objects in the cluster.

Container information

The hostname of a Container is the name of the Pod in which the Container is running. It is available through the hostname command or the gethostname function call in libc.

Container Runtimes

Mon, 01 Jan 0001 00:00:00 +0000

Note: Dockershim has been removed from the Kubernetes project as of release 1.24. Read the Dockershim Removal FAQ for further details.

You need to install a container runtime into each node in the cluster so that Pods can run there. This page outlines what is involved and describes related tasks for setting up nodes.

Contributing to the Upstream Kubernetes Code

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to contribute to the upstream kubernetes/kubernetes project. You can fix bugs found in the Kubernetes API documentation or the content of the Kubernetes components such as kubeadm, kube-apiserver, and kube-controller-manager.

If you instead want to regenerate the reference documentation for the Kubernetes API or the kube-* components from the upstream code, see the following instructions:

Before you begin

You need to have these tools installed:

Debug Services

Mon, 01 Jan 0001 00:00:00 +0000

An issue that comes up rather frequently for new installations of Kubernetes is that a Service is not working properly. You've run your Pods through a Deployment (or other workload controller) and created a Service, but you get no response when you try to access it. This document will hopefully help you to figure out what's going wrong.

Running commands in a Pod

For many steps here you will want to see what a Pod running in the cluster sees. The simplest way to do this is to run an interactive busybox Pod:

Declarative API Validation

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.33 [beta]

Kubernetes 1.35 includes optional declarative validation for APIs. When enabled, the Kubernetes API server can use this mechanism rather than the legacy approach that relies on hand-written Go code (validation.go files) to ensure that requests against the API are valid. Kubernetes developers, and people extending the Kubernetes API, can define validation rules directly alongside the API type definitions (types.go files). Code authors define special comment tags (e.g., +k8s:minimum=0). A code generator (validation-gen) then uses these tags to produce optimized Go code for API validation.

Declarative Management of Kubernetes Objects Using Kustomize

Mon, 01 Jan 0001 00:00:00 +0000

Kustomize is a standalone tool to customize Kubernetes objects through a kustomization file.

Since 1.14, kubectl also supports the management of Kubernetes objects using a kustomization file. To view resources found in a directory containing a kustomization file, run the following command:

kubectl kustomize <kustomization_directory>

To apply those resources, run kubectl apply with --kustomize or -k flag:

kubectl apply -k <kustomization_directory>

Before you begin

Install kubectl.

Define Dependent Environment Variables

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to define dependent environment variables for a container in a Kubernetes Pod.

Before you begin

Define Environment Variables for a Container

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to define environment variables for a container in a Kubernetes Pod.

Before you begin

Device Plugins

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [stable]

Kubernetes provides a device plugin framework that you can use to advertise system hardware resources to the Kubelet.

Instead of customizing the code for Kubernetes itself, vendors can implement a device plugin that you deploy either manually or as a DaemonSet. The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters, and other similar computing resources that may require vendor specific initialization and setup.

Documenting a feature for a release

Mon, 01 Jan 0001 00:00:00 +0000

Each major Kubernetes release introduces new features that require documentation. New releases also bring updates to existing features and documentation (such as upgrading a feature from alpha to beta).

Generally, the SIG responsible for a feature submits draft documentation of the feature as a pull request to the appropriate development branch of the kubernetes/website repository, and someone on the SIG Docs team provides editorial feedback or edits the draft directly. This section covers the branching conventions and process used during a release by both groups.

Example: Deploying PHP Guestbook application with Redis

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to build and deploy a simple (not production ready), multi-tier web application using Kubernetes and Docker. This example consists of the following components:

A single-instance Redis to store guestbook entries
Multiple web frontend instances

Objectives

Start up a Redis leader.
Start up two Redis followers.
Start up the guestbook frontend.
Expose and view the Frontend Service.
Clean up.

Before you begin

Example: Deploying WordPress and MySQL with Persistent Volumes

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to deploy a WordPress site and a MySQL database using Minikube. Both applications use PersistentVolumes and PersistentVolumeClaims to store data.

A PersistentVolume (PV) is a piece of storage in the cluster that has been manually provisioned by an administrator, or dynamically provisioned by Kubernetes using a StorageClass. A PersistentVolumeClaim (PVC) is a request for storage by a user that can be fulfilled by a PV. PersistentVolumes and PersistentVolumeClaims are independent from Pod lifecycles and preserve data through restarting, rescheduling, and even deleting Pods.

Extend the Kubernetes API with CustomResourceDefinitions

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to install a custom resource into the Kubernetes API by creating a CustomResourceDefinition.

Before you begin

Reviewing for approvers and reviewers

Mon, 01 Jan 0001 00:00:00 +0000

SIG Docs Reviewers and Approvers do a few extra things when reviewing a change.

Every week a specific docs approver volunteers to triage and review pull requests. This person is the "PR Wrangler" for the week. See the PR Wrangler scheduler for more information. To become a PR Wrangler, attend the weekly SIG Docs meeting and volunteer. Even if you are not on the schedule for the current week, you can still review pull requests (PRs) that are not already under active review.

Issue Wranglers

Mon, 01 Jan 0001 00:00:00 +0000

Alongside the PR Wrangler, formal approvers, reviewers and members of SIG Docs take week-long shifts triaging and categorising issues for the repository.

Duties

Each day in a week-long shift the Issue Wrangler will be responsible for:

Triaging and tagging incoming issues daily. See Triage and categorize issues for guidelines on how SIG Docs uses metadata.
Keeping an eye on stale & rotten issues within the kubernetes/website repository.
Maintenance of the Issues board.

Requirements

Must be an active member of the Kubernetes organization.
A minimum of 15 non-trivial contributions to Kubernetes (of which a certain amount should be directed towards kubernetes/website).
Performing the role in an informal capacity already.

Helpful Prow commands for wranglers

Below are some commonly used commands for Issue Wranglers:

kubeadm init

Mon, 01 Jan 0001 00:00:00 +0000

This command initializes a Kubernetes control plane node.

Synopsis

Run this command in order to set up the Kubernetes control plane

kubectl Commands

Mon, 01 Jan 0001 00:00:00 +0000

kubectl Command Reference

Kubernetes API Aggregation Layer

Mon, 01 Jan 0001 00:00:00 +0000

The aggregation layer allows Kubernetes to be extended with additional APIs, beyond what is offered by the core Kubernetes APIs. The additional APIs can either be ready-made solutions such as a metrics server, or APIs that you develop yourself.

The aggregation layer is different from Custom Resource Definitions, which are a way to make the kube-apiserver recognise new kinds of object.

Kubernetes API Concepts

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes API is a resource-based (RESTful) programmatic interface provided via HTTP. It supports retrieving, creating, updating, and deleting primary resources via the standard HTTP verbs (POST, PUT, PATCH, DELETE, GET).

For some resources, the API includes additional subresources that allow fine-grained authorization (such as separate views for Pod details and log retrievals), and can accept and serve those resources in different representations for convenience or efficiency.

Kubernetes Object Management

Mon, 01 Jan 0001 00:00:00 +0000

The kubectl command-line tool supports several different ways to create and manage Kubernetes objects. This document provides an overview of the different approaches. Read the Kubectl book for details of managing objects by Kubectl.

Management techniques

Warning:

A Kubernetes object should be managed using only one technique. Mixing and matching techniques for the same object results in undefined behavior.

Management technique	Operates on	Recommended environment	Supported writers	Learning curve
Imperative commands	Live objects	Development projects	1+	Lowest
Imperative object configuration	Individual files	Production projects	1	Moderate
Declarative object configuration	Directories of files	Production projects	1+	Highest

Imperative commands

When using imperative commands, a user operates directly on live objects in a cluster. The user provides operations to the kubectl command as arguments or flags.

Kubernetes Security and Disclosure Information

Mon, 01 Jan 0001 00:00:00 +0000

This page describes Kubernetes security and disclosure information.

Security Announcements

Join the kubernetes-security-announce group for emails about security and major API announcements.

Report a Vulnerability

We're extremely grateful for security researchers and users that report vulnerabilities to the Kubernetes Open Source Community. All reports are thoroughly investigated by a set of community volunteers.

To make a report, submit your vulnerability to the Kubernetes bug bounty program. This allows triage and handling of the vulnerability with standardized response times.

Managing Secrets using Configuration File

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Monitor Node Health

Mon, 01 Jan 0001 00:00:00 +0000

Node Problem Detector is a daemon for monitoring and reporting about a node's health. You can run Node Problem Detector as a DaemonSet or as a standalone daemon. Node Problem Detector collects information about node problems from various daemons and reports these conditions to the API server as Node Conditions or as Events.

To learn how to install and use Node Problem Detector, see Node Problem Detector project documentation.

Before you begin

Perform a Rollback on a DaemonSet

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to perform a rollback on a DaemonSet.

Before you begin

Persistent Volumes

Mon, 01 Jan 0001 00:00:00 +0000

This document describes persistent volumes in Kubernetes. Familiarity with volumes, StorageClasses and VolumeAttributesClasses is suggested.

Introduction

Managing storage is a distinct problem from managing compute instances. The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, we introduce two new API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

Pod Security Admission

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.25 [stable]

The Kubernetes Pod Security Standards define different isolation levels for Pods. These standards let you define how you want to restrict the behavior of pods in a clear, consistent fashion.

Kubernetes offers a built-in Pod Security admission controller to enforce the Pod Security Standards. Pod security restrictions are applied at the namespace level when pods are created.

PR wranglers

Mon, 01 Jan 0001 00:00:00 +0000

SIG Docs approvers take week-long shifts managing pull requests for the repository.

This section covers the duties of a PR wrangler. For more information on giving good reviews, see Reviewing changes.

Duties

Each day in a week-long shift as PR Wrangler:

Review open pull requests for quality and adherence to the Style and Content guides.
- Start with the smallest PRs (size/XS) first, and end with the largest (size/XXL). Review as many PRs as you can.
Make sure PR contributors sign the CLA.
- Use this script to remind contributors that haven't signed the CLA to do so.
Provide feedback on changes and ask for technical reviews from members of other SIGs.
- Provide inline suggestions on the PR for the proposed content changes.
- If you need to verify content, comment on the PR and request more details.
- Assign relevant sig/ label(s).
- If needed, assign reviewers from the reviewers: block in the file's front matter.
- You can also tag a SIG for a review by commenting @kubernetes/<sig>-pr-reviews on the PR.
Use the /approve comment to approve a PR for merging. Merge the PR when ready.
- PRs should have a /lgtm comment from another member before merging.
- Consider accepting technically accurate content that doesn't meet the style guidelines. As you approve the change, open a new issue to address the style concern. You can usually write these style fix issues as good first issues.
- Using style fixups as good first issues is a good way to ensure a supply of easier tasks to help onboard new contributors.
Also check for pull requests against the reference docs generator code, and review those (or bring in help).
Support the issue wrangler to triage and tag incoming issues daily. See Triage and categorize issues for guidelines on how SIG Docs uses metadata.

Note:

PR wrangler duties do not apply to localization PRs (non-English PRs). Localization teams have their own processes and teams for reviewing their language PRs. However, it's often helpful to ensure language PRs are labeled correctly, review small non-language dependent PRs (like a link update), or tag reviewers or contributors in long-running PRs (ones opened more than 6 months ago and have not been updated in a month or more).

Helpful GitHub queries for wranglers

The following queries are helpful when wrangling. After working through these queries, the remaining list of PRs to review is usually small. These queries exclude localization PRs. All queries are against the main branch except the last one.

ReplicaSet

Mon, 01 Jan 0001 00:00:00 +0000

A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

How a ReplicaSet works

A ReplicaSet is defined with fields, including a selector that specifies how to identify Pods it can acquire, a number of replicas indicating how many Pods it should be maintaining, and a pod template specifying the data of new Pods it should create to meet the number of replicas criteria. A ReplicaSet then fulfills its purpose by creating and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod template.

Resource Quotas

Mon, 01 Jan 0001 00:00:00 +0000

When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.

Resource quotas are a tool for administrators to address this concern.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. A ResourceQuota can also limit the quantity of objects that can be created in a namespace by API kind, as well as the total amount of infrastructure resources that may be consumed by API objects found in that namespace.

Run a Single-Instance Stateful Application

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to run a single-instance stateful application in Kubernetes using a PersistentVolume and a Deployment. The application is MySQL.

Objectives

Create a PersistentVolume referencing a disk in your environment.
Create a MySQL Deployment.
Expose MySQL to other pods in the cluster at a known DNS name.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Running in multiple zones

Mon, 01 Jan 0001 00:00:00 +0000

This page describes running Kubernetes across multiple zones.

Background

Kubernetes is designed so that a single Kubernetes cluster can run across multiple failure zones, typically where these zones fit within a logical grouping called a region. Major cloud providers define a region as a set of failure zones (also called availability zones) that provide a consistent set of features: within a region, each zone offers the same APIs and services.

Scheduler Configuration

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.25 [stable]

You can customize the behavior of the kube-scheduler by writing a configuration file and passing its path as a command line argument.

A scheduling Profile allows you to configure the different stages of scheduling in the kube-scheduler. Each stage is exposed in an extension point. Plugins provide scheduling behaviors by implementing one or more of these extension points.

Kubernetes Component SLI Metrics

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.32 [stable](enabled by default)

By default, Kubernetes 1.35 publishes Service Level Indicator (SLI) metrics for each Kubernetes component binary. This metric endpoint is exposed on the serving HTTPS port of each component, at the path /metrics/slis. The ComponentSLIs feature gate defaults to enabled for each Kubernetes component as of v1.27.

SLI Metrics

With SLI metrics enabled, each Kubernetes component exposes two metrics, labeled per healthcheck:

Suggesting content improvements

Mon, 01 Jan 0001 00:00:00 +0000

If you notice an issue with Kubernetes documentation or have an idea for new content, then open an issue. All you need is a GitHub account and a web browser.

In most cases, new work on Kubernetes documentation begins with an issue in GitHub. Kubernetes contributors then review, categorize and tag issues as needed. Next, you or another member of the Kubernetes community open a pull request with changes to resolve the issue.

Troubleshooting kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

As with any program, you might run into an error installing or running kubeadm. This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem.

If your problem is not listed below, please follow the following steps:

If you think your problem is a bug with kubeadm:
- Go to github.com/kubernetes/kubeadm and search for existing issues.
- If no issue exists, please open one and follow the issue template.
If you are unsure about how kubeadm works, you can ask on Slack in #kubeadm, or open a question on StackOverflow. Please include relevant tags like #kubernetes and #kubeadm so folks can help you.

Updating Configuration via a ConfigMap

Mon, 01 Jan 0001 00:00:00 +0000

This page provides a step-by-step example of updating configuration within a Pod via a ConfigMap and builds upon the Configure a Pod to Use a ConfigMap task.
At the end of this tutorial, you will understand how to change the configuration for a running application.
This tutorial uses the alpine and nginx images as examples.

Before you begin

Use Calico for NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

This page shows a couple of quick ways to create a Calico cluster on Kubernetes.

Before you begin

Decide whether you want to deploy a cloud or local cluster.

Creating a Calico cluster with Google Kubernetes Engine (GKE)

Prerequisite: gcloud.

To launch a GKE cluster with Calico, include the --enable-network-policy flag.

Syntax

gcloud container clusters create [CLUSTER_NAME] --enable-network-policy

Example

gcloud container clusters create my-calico-cluster --enable-network-policy

To verify the deployment, use the following command.

Projected Volumes

Mon, 01 Jan 0001 00:00:00 +0000

This document describes projected volumes in Kubernetes. Familiarity with volumes is suggested.

Introduction

A projected volume maps several existing volume sources into the same directory.

Currently, the following types of volume sources can be projected:

secret
downwardAPI
configMap
serviceAccountToken
clusterTrustBundle
podCertificate

All sources are required to be in the same namespace as the Pod. For more details, see the all-in-one volume design document.

Example configuration with a secret, a downwardAPI, and a configMap

pods/storage/projected-secret-downwardapi-configmap.yaml

apiVersion: v1
kind: Pod
metadata:
 name: volume-test
spec:
 containers:
 - name: container-test
 image: busybox:1.28
 command: ["sleep", "3600"]
 volumeMounts:
 - name: all-in-one
 mountPath: "/projected-volume"
 readOnly: true
 volumes:
 - name: all-in-one
 projected:
 sources:
 - secret:
 name: mysecret
 items:
 - key: username
 path: my-group/my-username
 - downwardAPI:
 items:
 - path: "labels"
 fieldRef:
 fieldPath: metadata.labels
 - path: "cpu_limit"
 resourceFieldRef:
 containerName: container-test
 resource: limits.cpu
 - configMap:
 name: myconfigmap
 items:
 - key: config
 path: my-group/my-config

Example configuration: secrets with a non-default permission mode set

pods/storage/projected-secrets-nondefault-permission-mode.yaml

apiVersion: v1
kind: Pod
metadata:
 name: volume-test
spec:
 containers:
 - name: container-test
 image: busybox:1.28
 command: ["sleep", "3600"]
 volumeMounts:
 - name: all-in-one
 mountPath: "/projected-volume"
 readOnly: true
 volumes:
 - name: all-in-one
 projected:
 sources:
 - secret:
 name: mysecret
 items:
 - key: username
 path: my-group/my-username
 - secret:
 name: mysecret2
 items:
 - key: password
 path: my-group/my-password
 mode: 511

Each projected volume source is listed in the spec under sources. The parameters are nearly the same with two exceptions:

Official CVE Feed

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.27 [beta]

This is a community maintained list of official CVEs announced by the Kubernetes Security Response Committee. See Kubernetes Security and Disclosure Information for more details.

The Kubernetes project publishes a programmatically accessible feed of published security issues in JSON feed and RSS feed formats. You can access it by executing the following commands:

JSON feed
RSS feed

Link to JSON format

Server-Side Apply

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.22 [stable](enabled by default)

Kubernetes supports multiple appliers collaborating to manage the fields of a single object.

Server-Side Apply provides an optional mechanism for your cluster's control plane to track changes to an object's fields. At the level of a specific resource, Server-Side Apply records and tracks information about control over the fields of that object.

Server-Side Apply helps users and controllers manage their resources through declarative configuration. Clients can create and modify objects declaratively by submitting their fully specified intent.

Service Accounts

Mon, 01 Jan 0001 00:00:00 +0000

This page introduces the ServiceAccount object in Kubernetes, providing information about how service accounts work, use cases, limitations, alternatives, and links to resources for additional guidance.

What are service accounts?

A service account is a type of non-human account that, in Kubernetes, provides a distinct identity in a Kubernetes cluster. Application Pods, system components, and entities inside and outside the cluster can use a specific ServiceAccount's credentials to identify as that ServiceAccount. This identity is useful in various situations, including authenticating to the API server or implementing identity-based security policies.

Assign Pod-level CPU and memory resources

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.34 [beta](enabled by default)

This page shows how to specify CPU and memory resources for a Pod at pod-level in addition to container-level resource specifications. A Kubernetes node allocates resources to a pod based on the pod's resource requests. These requests can be defined at the pod level or individually for containers within the pod. When both are present, the pod-level requests take precedence.

Authorization

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes authorization takes place following authentication. Usually, a client making a request must be authenticated (logged in) before its request can be allowed; however, Kubernetes also allows anonymous requests in some circumstances.

For an overview of how authorization fits into the wider context of API access control, read Controlling Access to the Kubernetes API.

Authorization verdicts

Kubernetes authorization of API requests takes place within the API server. The API server evaluates all of the request attributes against all policies, potentially also consulting external services, and then allows or denies the request.

Submitting case studies

Mon, 01 Jan 0001 00:00:00 +0000

Case studies highlight how organizations are using Kubernetes to solve real-world problems. The Kubernetes marketing team and members of the CNCF collaborate with you on all case studies.

Case studies require extensive review before they're approved.

Submit a case study

Have a look at the source for the existing case studies.

Refer to the case study guidelines and submit your request as outlined in the guidelines.

Client Libraries

Mon, 01 Jan 0001 00:00:00 +0000

This page contains an overview of the client libraries for using the Kubernetes API from various programming languages.

To write applications using the Kubernetes REST API, you do not need to implement the API calls and request/response types yourself. You can use a client library for the programming language you are using.

Client libraries often handle common tasks such as authentication for you. Most client libraries can discover and use the Kubernetes Service Account to authenticate if the API client is running inside the Kubernetes cluster, or can understand the kubeconfig file format to read the credentials and the API Server address.

Configure Access to Multiple Clusters

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure access to multiple clusters by using configuration files. After your clusters, users, and contexts are defined in one or more configuration files, you can quickly switch between clusters by using the kubectl config use-context command.

Note:

A file that is used to configure access to a cluster is sometimes called a kubeconfig file. This is a generic way of referring to configuration files. It does not mean that there is a file named kubeconfig.

Warning:

Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script.

Before you begin

Configure GMSA for Windows Pods and containers

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.18 [stable]

This page shows how to configure Group Managed Service Accounts (GMSA) for Pods and containers that will run on Windows nodes. Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.

In Kubernetes, GMSA credential specs are configured at a Kubernetes cluster-wide scope as Custom Resources. Windows Pods, as well as individual containers within a Pod, can be configured to use a GMSA for domain based functions (e.g. Kerberos authentication) when interacting with other Windows services.

Configure Minimum and Maximum Memory Constraints for a Namespace

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to set minimum and maximum values for memory used by containers running in a namespace. You specify minimum and maximum memory values in a LimitRange object. If a Pod does not meet the constraints imposed by the LimitRange, it cannot be created in the namespace.

Before you begin

Configuring Redis using a ConfigMap

Mon, 01 Jan 0001 00:00:00 +0000

This page provides a real world example of how to configure Redis using a ConfigMap and builds upon the Configure a Pod to Use a ConfigMap task.

Objectives

Create a ConfigMap with Redis configuration values
Create a Redis Pod that mounts and uses the created ConfigMap
Verify that the configuration was correctly applied.

Before you begin

Controllers

Mon, 01 Jan 0001 00:00:00 +0000

In robotics and automation, a control loop is a non-terminating loop that regulates the state of a system.

Here is one example of a control loop: a thermostat in a room.

When you set the temperature, that's telling the thermostat about your desired state. The actual room temperature is the current state. The thermostat acts to bring the current state closer to the desired state, by turning equipment on or off.

Creating a cluster with kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

Using kubeadm, you can create a minimum viable Kubernetes cluster that conforms to best practices. In fact, you can use kubeadm to set up a cluster that will pass the Kubernetes Conformance tests. kubeadm also supports other cluster lifecycle functions, such as bootstrap tokens and cluster upgrades.

The kubeadm tool is good if you need:

A simple way for you to try out Kubernetes, possibly for the first time.
A way for existing users to automate setting up a cluster and test their application.
A building block in other ecosystem and/or installer tools with a larger scope.

You can install and use kubeadm on various machines: your laptop, a set of cloud servers, a Raspberry Pi, and more. Whether you're deploying into the cloud or on-premises, you can integrate kubeadm into provisioning systems such as Ansible or Terraform.

Debug a StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

This task shows you how to debug a StatefulSet.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.
You should have a StatefulSet running that you want to investigate.

Debugging a StatefulSet

In order to list all the pods which belong to a StatefulSet, which have a label app.kubernetes.io/name=MyApp set on them, you can use the following:

Debugging Kubernetes nodes with crictl

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.11 [stable]

crictl is a command-line interface for CRI-compatible container runtimes. You can use it to inspect and debug container runtimes and applications on a Kubernetes node. crictl and its source are hosted in the cri-tools repository.

Before you begin

crictl requires a Linux operating system with a CRI runtime.

Installing crictl

You can download a compressed archive crictl from the cri-tools release page, for several different architectures. Download the version that corresponds to your version of Kubernetes. Extract it and move it to a location on your system path, such as /usr/local/bin/.

Define Environment Variable Values Using An Init Container

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [beta](enabled by default)

This page show how to configure environment variables for containers in a Pod via file.

Before you begin

Determine the Reason for Pod Failure

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to write and read a Container termination message.

Termination messages provide a way for containers to write information about fatal events to a location where it can be easily retrieved and surfaced by tools like dashboards and monitoring software. In most cases, information that you put in a termination message should also be written to the general Kubernetes logs.

Before you begin

Ephemeral Volumes

Mon, 01 Jan 0001 00:00:00 +0000

This document describes ephemeral volumes in Kubernetes. Familiarity with volumes is suggested, in particular PersistentVolumeClaim and PersistentVolume.

Some applications need additional storage but don't care whether that data is stored persistently across restarts. For example, caching services are often limited by memory size and can move infrequently used data into storage that is slower than memory with little impact on overall performance.

Other applications expect some read-only input data to be present in files, like configuration data or secret keys.

Example: Deploying Cassandra with a StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial shows you how to run Apache Cassandra on Kubernetes. Cassandra, a database, needs persistent storage to provide data durability (application state). In this example, a custom Cassandra seed provider lets the database discover new Cassandra instances as they join the Cassandra cluster.

StatefulSets make it easier to deploy stateful applications into your Kubernetes cluster. For more information on the features used in this tutorial, see StatefulSet.

Note:

Cassandra and Kubernetes both use the term node to mean a member of a cluster. In this tutorial, the Pods that belong to the StatefulSet are Cassandra nodes and are members of the Cassandra cluster (called a ring). When those Pods run in your Kubernetes cluster, the Kubernetes control plane schedules those Pods onto Kubernetes Nodes.

Expose Pod Information to Containers Through Environment Variables

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how a Pod can use environment variables to expose information about itself to containers running in the Pod, using the downward API. You can use environment variables to expose Pod fields, container fields, or both.

In Kubernetes, there are two ways to expose Pod and container fields to a running container:

Environment variables, as explained in this task
Volume files

Together, these two ways of exposing Pod and container fields are called the downward API.

Find Out What Container Runtime is Used on a Node

Mon, 01 Jan 0001 00:00:00 +0000

This page outlines steps to find out what container runtime the nodes in your cluster use.

Depending on the way you run your cluster, the container runtime for the nodes may have been pre-configured or you need to configure it. If you're using a managed Kubernetes service, there might be vendor-specific ways to check what container runtime is configured for the nodes. The method described on this page should work whenever the execution of kubectl is allowed.

Fine Parallel Processing Using a Work Queue

Mon, 01 Jan 0001 00:00:00 +0000

In this example, you will run a Kubernetes Job that runs multiple parallel tasks as worker processes, each running as a separate Pod.

In this example, as each pod is created, it picks up one unit of work from a task queue, processes it, and repeats until the end of the queue is reached.

Here is an overview of the steps in this example:

Start a storage service to hold the work queue. In this example, you will use Redis to store work items. In the previous example, you used RabbitMQ. In this example, you will use Redis and a custom work-queue client library; this is because AMQP does not provide a good way for clients to detect when a finite-length work queue is empty. In practice you would set up a store such as Redis once and reuse it for the work queues of many jobs, and other things.
Create a queue, and fill it with messages. Each message represents one task to be done. In this example, a message is an integer that we will do a lengthy computation on.
Start a Job that works on tasks from the queue. The Job starts several pods. Each pod takes one task from the message queue, processes it, and repeats until the end of the queue is reached.

Before you begin

Generate Certificates Manually

Mon, 01 Jan 0001 00:00:00 +0000

When using client certificate authentication, you can generate certificates manually through easyrsa, openssl or cfssl.

easyrsa

easyrsa can manually generate certificates for your cluster.

Download, unpack, and initialize the patched version of easyrsa3.

curl -LO https://dl.k8s.io/easy-rsa/easy-rsa.tar.gz
tar xzf easy-rsa.tar.gz
cd easy-rsa-master/easyrsa3
./easyrsa init-pki

Generate a new certificate authority (CA). --batch sets automatic mode; --req-cn specifies the Common Name (CN) for the CA's new root certificate.
```
./easyrsa --batch "--req-cn=${MASTER_IP}@`date +%s`" build-ca nopass
```
Generate server certificate and key.

Indexed Job for Parallel Processing with Static Work Assignment

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.24 [stable]

In this example, you will run a Kubernetes Job that uses multiple parallel worker processes. Each worker is a different container running in its own Pod. The Pods have an index number that the control plane sets automatically, which allows each Pod to identify which part of the overall task to work on.

The pod index is available in the annotation batch.kubernetes.io/job-completion-index as a string representing its decimal value. In order for the containerized task process to obtain this index, you can publish the value of the annotation using the downward API mechanism. For convenience, the control plane automatically sets the downward API to expose the index in the JOB_COMPLETION_INDEX environment variable.

Ingress

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.19 [stable]

An API object that manages external access to the services in a cluster, typically HTTP.

Ingress may provide load balancing, SSL termination and name-based virtual hosting.

Note:

The Kubernetes project recommends using Gateway instead of Ingress. The Ingress API has been frozen.

This means that:

The Ingress API is generally available, and is subject to the stability guarantees for generally available APIs. The Kubernetes project has no plans to remove Ingress from Kubernetes.
The Ingress API is no longer being developed, and will have no further changes or updates made to it.

This command initializes a new Kubernetes node and joins it to the existing cluster.

Run this on any machine you wish to join an existing cluster

kubectl

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

kubectl controls the Kubernetes cluster manager.

Find more information at: https://kubernetes.io/docs/reference/kubectl/

kubectl [flags]

Options

--as string

Username to impersonate for the operation. User could be a regular user or a service account in a namespace.

kubectl

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

kubectl controls the Kubernetes cluster manager.

Find more information in Command line tool (kubectl).

kubectl [flags]

Options

--add-dir-header
	If true, adds the file directory to the header of the log messages
--alsologtostderr
	log to standard error as well as files
--as string
	Username to impersonate for the operation
--as-group stringArray
	Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--azure-container-registry-config string
	Path to the file containing Azure container registry configuration information.
--cache-dir string Default: "$HOME/.kube/cache"
	Default cache directory
--certificate-authority string
	Path to a cert file for the certificate authority
--client-certificate string
	Path to a client certificate file for TLS
--client-key string
	Path to a client key file for TLS
--cloud-provider-gce-l7lb-src-cidrs cidrs Default: 130.211.0.0/22,35.191.0.0/16
	CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks
--cloud-provider-gce-lb-src-cidrs cidrs Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16
	CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks
--cluster string
	The name of the kubeconfig cluster to use
--context string
	The name of the kubeconfig context to use
--default-not-ready-toleration-seconds int Default: 300
	Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.
--default-unreachable-toleration-seconds int Default: 300
	Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.
-h, --help
	help for kubectl
--insecure-skip-tls-verify
	If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
--kubeconfig string
	Path to the kubeconfig file to use for CLI requests.
--log-backtrace-at traceLocation Default: :0
	when logging hits line file:N, emit a stack trace
--log-dir string
	If non-empty, write log files in this directory
--log-file string
	If non-empty, use this log file
--log-file-max-size uint Default: 1800
	Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.
--log-flush-frequency duration Default: 5s
	Maximum number of seconds between log flushes
--logtostderr Default: true
	log to standard error instead of files
--match-server-version
	Require server version to match client version
-n, --namespace string
	If present, the namespace scope for this CLI request
--one-output
	If true, only write logs to their native severity level (vs also writing to each lower severity level)
--password string
	Password for basic authentication to the API server
--profile string Default: "none"
	Name of profile to capture. One of (none\|cpu\|heap\|goroutine\|threadcreate\|block\|mutex)
--profile-output string Default: "profile.pprof"
	Name of the file to write the profile to
--request-timeout string Default: "0"
	The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.
-s, --server string
	The address and port of the Kubernetes API server
--skip-headers
	If true, avoid header prefixes in the log messages
--skip-log-headers
	If true, avoid headers when opening log files
--stderrthreshold severity Default: 2
	logs at or above this threshold go to stderr
--tls-server-name string
	Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
--token string
	Bearer token for authentication to the API server
--user string
	The name of the kubeconfig user to use
--username string
	Username for basic authentication to the API server
-v, --v Level
	number for the log level verbosity
--version version[=true]
	Print version information and quit
--vmodule moduleSpec
	comma-separated list of pattern=N settings for file-filtered logging
--warnings-as-errors
	Treat warnings received from the server as errors and exit with a non-zero exit code

Environment variables

KUBECONFIG
	Path to the kubectl configuration ("kubeconfig") file. Default: "$HOME/.kube/config"
KUBECTL_EXPLAIN_OPENAPIV3
	Toggles whether calls to `kubectl explain` use the new OpenAPIv3 data source available. OpenAPIV3 is enabled by default since Kubernetes 1.24.
KUBECTL_ENABLE_CMD_SHADOW
	When set to true, external plugins can be used as subcommands for builtin commands if subcommand does not exist. In alpha stage, this feature can only be used for create command(e.g. kubectl create networkpolicy).
KUBECTL_PORT_FORWARD_WEBSOCKETS
	When set to true, the kubectl port-forward command will attempt to stream using the websockets protocol. If the upgrade to websockets fails, the commands will fallback to use the current SPDY protocol.
KUBECTL_REMOTE_COMMAND_WEBSOCKETS
	When set to true, the kubectl exec, cp, and attach commands will attempt to stream using the websockets protocol. If the upgrade to websockets fails, the commands will fallback to use the current SPDY protocol.
KUBECTL_KUBERC
	When set to true, kuberc file is taken into account to define user specific preferences.
KUBECTL_KYAML
	When set to true, kubectl is capable of producing Kubernetes-specific dialect of YAML output format.

kubectl alpha kuberc

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Manage user preferences (kuberc) file.

The kuberc file allows you to customize your kubectl experience.

kubectl alpha kuberc set

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Set values in the kuberc configuration file.

Use --section to specify whether to set defaults or aliases.

kubectl alpha kuberc view

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Display the contents of the kuberc file in the specified output format.

kubectl alpha kuberc view

Examples

 # View kuberc configuration in YAML format (default)
 kubectl alpha kuberc view
 
 # View kuberc configuration in JSON format
 kubectl alpha kuberc view --output json
 
 # View a specific kuberc file
 kubectl alpha kuberc view --kuberc /path/to/kuberc

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl apply edit-last-applied

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Edit the latest last-applied-configuration annotations of resources from the default editor.

The edit-last-applied command allows you to directly edit any API resource you can retrieve via the command-line tools. It will open the editor defined by your KUBE_EDITOR, or EDITOR environment variables, or fall back to 'vi' for Linux or 'notepad' for Windows. You can edit multiple objects, although changes are applied one at a time. The command accepts file names as well as command-line arguments, although the files you point to must be previously saved versions of resources.

kubectl apply set-last-applied

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Set the latest last-applied-configuration annotations by setting it to match the contents of a file. This results in the last-applied-configuration being updated as though 'kubectl apply -f<file> ' was run, without updating any other parts of the object.

kubectl apply view-last-applied

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Experimental: Check who you are and your attributes (groups, extra).

 This command is helpful to get yourself aware of the current user attributes,
 especially when dynamic authentication, e.g., token webhook, auth proxy, or OIDC provider,
 is enabled in the Kubernetes cluster.

kubectl auth whoami

Examples

 # Get your subject attributes
 kubectl auth whoami
 
 # Get your subject attributes in JSON format
 kubectl auth whoami -o json

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl certificate approve

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Approve a certificate signing request.

kubectl certificate approve allows a cluster admin to approve a certificate signing request (CSR). This action tells a certificate signing controller to issue a certificate to the requester with the attributes requested in the CSR.

kubectl certificate deny

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Deny a certificate signing request.

kubectl certificate deny allows a cluster admin to deny a certificate signing request (CSR). This action tells a certificate signing controller to not to issue a certificate to the requester.

kubectl cluster-info dump

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Dump cluster information out suitable for debugging and diagnosing cluster problems. By default, dumps everything to stdout. You can optionally specify a directory with --output-directory. If you specify a directory, Kubernetes will build a set of files in that directory. By default, only dumps things in the current namespace and 'kube-system' namespace, but you can switch to a different namespace with the --namespaces flag, or specify --all-namespaces to dump all namespaces.

kubectl config current-context

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Display the current-context.

kubectl config current-context [flags]

Examples

 # Display the current-context
 kubectl config current-context

Options

-h, --help

help for current-context

kubectl config delete-cluster

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Delete the specified cluster from the kubeconfig.

kubectl config delete-cluster NAME

Examples

 # Delete the minikube cluster
 kubectl config delete-cluster minikube

Options

-h, --help

help for delete-cluster

kubectl config delete-context

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Delete the specified context from the kubeconfig.

kubectl config delete-context NAME

Examples

 # Delete the context for the minikube cluster
 kubectl config delete-context minikube

Options

-h, --help

help for delete-context

kubectl config delete-user

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Delete the specified user from the kubeconfig.

kubectl config delete-user NAME

Examples

 # Delete the minikube user
 kubectl config delete-user minikube

Options

-h, --help

help for delete-user

kubectl config get-clusters

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Display clusters defined in the kubeconfig.

kubectl config get-clusters [flags]

Examples

 # List the clusters that kubectl knows about
 kubectl config get-clusters

Options

-h, --help

help for get-clusters

kubectl config get-contexts

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Display one or many contexts from the kubeconfig file.

kubectl config get-contexts [(-o|--output=)name)]

Examples

 # List all the contexts in your kubeconfig file
 kubectl config get-contexts
 
 # Describe one context in your kubeconfig file
 kubectl config get-contexts my-context

Options

-h, --help

help for get-contexts

kubectl config get-users

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Display users defined in the kubeconfig.

kubectl config get-users [flags]

Examples

 # List the users that kubectl knows about
 kubectl config get-users

Synopsis

Unset an individual value in a kubeconfig file.

PROPERTY_NAME is a dot delimited name where each token represents either an attribute name or a map key. Map keys may not contain dots.

kubectl config use-context

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Set the current-context in a kubeconfig file.

kubectl config use-context CONTEXT_NAME

Examples

 # Use the context for the minikube cluster
 kubectl config use-context minikube

Options

-h, --help

help for use-context

kubectl config view

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Display merged kubeconfig settings or a specified kubeconfig file.

You can use --output jsonpath={...} to extract specific values using a jsonpath expression.

kubectl create clusterrole

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a cluster role.

kubectl create clusterrole NAME --verb=verb --resource=resource.group [--resource-name=resourcename] [--dry-run=server|client|none]

Examples

 # Create a cluster role named "pod-reader" that allows user to perform "get", "watch" and "list" on pods
 kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods
 
 # Create a cluster role named "pod-reader" with ResourceName specified
 kubectl create clusterrole pod-reader --verb=get --resource=pods --resource-name=readablepod --resource-name=anotherpod
 
 # Create a cluster role named "foo" with API Group specified
 kubectl create clusterrole foo --verb=get,list,watch --resource=rs.apps
 
 # Create a cluster role named "foo" with SubResource specified
 kubectl create clusterrole foo --verb=get,list,watch --resource=pods,pods/status
 
 # Create a cluster role name "foo" with NonResourceURL specified
 kubectl create clusterrole "foo" --verb=get --non-resource-url=https://andygol-k8s.netlify.app/logs/*
 
 # Create a cluster role name "monitoring" with AggregationRule specified
 kubectl create clusterrole monitoring --aggregation-rule="rbac.example.com/aggregate-to-monitoring=true"

Options

--aggregation-rule <comma-separated 'key=value' pairs>

An aggregation label selector for combining ClusterRoles.

kubectl create clusterrolebinding

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a cluster role binding for a particular cluster role.

kubectl create clusterrolebinding NAME --clusterrole=NAME [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none]

Examples

 # Create a cluster role binding for user1, user2, and group1 using the cluster-admin cluster role
 kubectl create clusterrolebinding cluster-admin --clusterrole=cluster-admin --user=user1 --user=user2 --group=group1

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create configmap

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a config map based on a file, directory, or specified literal value.

A single config map may package one or more key/value pairs.

kubectl create cronjob

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a cron job with the specified name.

kubectl create cronjob NAME --image=image --schedule='0/5 * * * ?' -- [COMMAND] [args...] [flags]

Examples

 # Create a cron job
 kubectl create cronjob my-job --image=busybox --schedule="*/1 * * * *"
 
 # Create a cron job with a command
 kubectl create cronjob my-job --image=busybox --schedule="*/1 * * * *" -- date

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create deployment

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a deployment with the specified name.

kubectl create deployment NAME --image=image -- [COMMAND] [args...]

Examples

 # Create a deployment named my-dep that runs the busybox image
 kubectl create deployment my-dep --image=busybox
 
 # Create a deployment with a command
 kubectl create deployment my-dep --image=busybox -- date
 
 # Create a deployment named my-dep that runs the nginx image with 3 replicas
 kubectl create deployment my-dep --image=nginx --replicas=3
 
 # Create a deployment named my-dep that runs the busybox image and expose port 5701
 kubectl create deployment my-dep --image=busybox --port=5701
 
 # Create a deployment named my-dep that runs multiple containers
 kubectl create deployment my-dep --image=busybox:latest --image=ubuntu:latest --image=nginx

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create ingress

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create an ingress with the specified name.

kubectl create ingress NAME --rule=host/path=service:port[,tls[=secret]]

Examples

 # Create a single ingress called 'simple' that directs requests to foo.com/bar to svc
 # svc1:8080 with a TLS secret "my-cert"
 kubectl create ingress simple --rule="foo.com/bar=svc1:8080,tls=my-cert"
 
 # Create a catch all ingress of "/path" pointing to service svc:port and Ingress Class as "otheringress"
 kubectl create ingress catch-all --class=otheringress --rule="/path=svc:port"
 
 # Create an ingress with two annotations: ingress.annotation1 and ingress.annotations2
 kubectl create ingress annotated --class=default --rule="foo.com/bar=svc:port" \
 --annotation ingress.annotation1=foo \
 --annotation ingress.annotation2=bla
 
 # Create an ingress with the same host and multiple paths
 kubectl create ingress multipath --class=default \
 --rule="foo.com/=svc:port" \
 --rule="foo.com/admin/=svcadmin:portadmin"
 
 # Create an ingress with multiple hosts and the pathType as Prefix
 kubectl create ingress ingress1 --class=default \
 --rule="foo.com/path*=svc:8080" \
 --rule="bar.com/admin*=svc2:http"
 
 # Create an ingress with TLS enabled using the default ingress certificate and different path types
 kubectl create ingress ingtls --class=default \
 --rule="foo.com/=svc:https,tls" \
 --rule="foo.com/path/subpath*=othersvc:8080"
 
 # Create an ingress with TLS enabled using a specific secret and pathType as Prefix
 kubectl create ingress ingsecret --class=default \
 --rule="foo.com/*=svc:8080,tls=secret1"
 
 # Create an ingress with a default backend
 kubectl create ingress ingdefault --class=default \
 --default-backend=defaultsvc:http \
 --rule="foo.com/*=svc:8080,tls=secret1"

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create job

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a job with the specified name.

kubectl create job NAME --image=image [--from=cronjob/name] -- [COMMAND] [args...]

Examples

 # Create a job
 kubectl create job my-job --image=busybox
 
 # Create a job with a command
 kubectl create job my-job --image=busybox -- date
 
 # Create a job from a cron job named "a-cronjob"
 kubectl create job test-job --from=cronjob/a-cronjob

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create namespace

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a namespace with the specified name.

kubectl create namespace NAME [--dry-run=server|client|none]

Examples

 # Create a new namespace named my-namespace
 kubectl create namespace my-namespace

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create poddisruptionbudget

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a pod disruption budget with the specified name, selector, and desired minimum available pods.

kubectl create priorityclass

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a priority class with the specified name, value, globalDefault and description.

kubectl create priorityclass NAME --value=VALUE --global-default=BOOL [--dry-run=server|client|none]

Examples

 # Create a priority class named high-priority
 kubectl create priorityclass high-priority --value=1000 --description="high priority"
 
 # Create a priority class named default-priority that is considered as the global default priority
 kubectl create priorityclass default-priority --value=1000 --global-default=true --description="default priority"
 
 # Create a priority class named high-priority that cannot preempt pods with lower priority
 kubectl create priorityclass high-priority --value=1000 --description="high priority" --preemption-policy="Never"

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create quota

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a resource quota with the specified name, hard limits, and optional scopes.

kubectl create quota NAME [--hard=key1=value1,key2=value2] [--scopes=Scope1,Scope2] [--dry-run=server|client|none]

Examples

 # Create a new resource quota named my-quota
 kubectl create quota my-quota --hard=cpu=1,memory=1G,pods=2,services=3,replicationcontrollers=2,resourcequotas=1,secrets=5,persistentvolumeclaims=10
 
 # Create a new resource quota named best-effort
 kubectl create quota best-effort --hard=pods=100 --scopes=BestEffort

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create role

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a role with single rule.

kubectl create role NAME --verb=verb --resource=resource.group/subresource [--resource-name=resourcename] [--dry-run=server|client|none]

Examples

 # Create a role named "pod-reader" that allows user to perform "get", "watch" and "list" on pods
 kubectl create role pod-reader --verb=get --verb=list --verb=watch --resource=pods
 
 # Create a role named "pod-reader" with ResourceName specified
 kubectl create role pod-reader --verb=get --resource=pods --resource-name=readablepod --resource-name=anotherpod
 
 # Create a role named "foo" with API Group specified
 kubectl create role foo --verb=get,list,watch --resource=rs.apps
 
 # Create a role named "foo" with SubResource specified
 kubectl create role foo --verb=get,list,watch --resource=pods,pods/status

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create rolebinding

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a role binding for a particular role or cluster role.

kubectl create rolebinding NAME --clusterrole=NAME|--role=NAME [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none]

Examples

 # Create a role binding for user1, user2, and group1 using the admin cluster role
 kubectl create rolebinding admin --clusterrole=admin --user=user1 --user=user2 --group=group1
 
 # Create a role binding for service account monitoring:sa-dev using the admin role
 kubectl create rolebinding admin-binding --role=admin --serviceaccount=monitoring:sa-dev

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create secret

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a secret with specified type.

A docker-registry type secret is for accessing a container registry.

kubectl create secret docker-registry

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a new secret for use with Docker registries.

 Dockercfg secrets are used to authenticate against Docker registries.
 
 When using the Docker command line to push images, you can authenticate to a given registry by running:
 '$ docker login DOCKER_REGISTRY_SERVER --username=DOCKER_USER --password=DOCKER_PASSWORD --email=DOCKER_EMAIL'.

kubectl create service [flags]

Options

-h, --help

help for service

kubectl create service clusterip

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a ClusterIP service with the specified name.

kubectl create service clusterip NAME [--tcp=<port>:<targetPort>] [--dry-run=server|client|none]

Examples

 # Create a new ClusterIP service named my-cs
 kubectl create service clusterip my-cs --tcp=5678:8080
 
 # Create a new ClusterIP service named my-cs (in headless mode)
 kubectl create service clusterip my-cs --clusterip="None"

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create service externalname

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create an ExternalName service with the specified name.

ExternalName service references to an external DNS address instead of only pods, which will allow application authors to reference services that exist off platform, on other clusters, or locally.

kubectl create service loadbalancer

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a LoadBalancer service with the specified name.

kubectl create service loadbalancer NAME [--tcp=port:targetPort] [--dry-run=server|client|none]

Examples

 # Create a new LoadBalancer service named my-lbs
 kubectl create service loadbalancer my-lbs --tcp=5678:8080

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create service nodeport

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a NodePort service with the specified name.

kubectl create service nodeport NAME [--tcp=port:targetPort] [--dry-run=server|client|none]

Examples

 # Create a new NodePort service named my-ns
 kubectl create service nodeport my-ns --tcp=5678:8080

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create serviceaccount

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Create a service account with the specified name.

kubectl create serviceaccount NAME [--dry-run=server|client|none]

Examples

 # Create a new service account named my-service-account
 kubectl create serviceaccount my-service-account

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl create token

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Request a service account token.

kubectl create token SERVICE_ACCOUNT_NAME

Examples

 # Request a token to authenticate to the kube-apiserver as the service account "myapp" in the current namespace
 kubectl create token myapp
 
 # Request a token for a service account in a custom namespace
 kubectl create token myapp --namespace myns
 
 # Request a token with a custom expiration
 kubectl create token myapp --duration 10m
 
 # Request a token with a custom audience
 kubectl create token myapp --audience https://example.com
 
 # Request a token bound to an instance of a Secret object
 kubectl create token myapp --bound-object-kind Secret --bound-object-name mysecret
 
 # Request a token bound to an instance of a Secret object with a specific UID
 kubectl create token myapp --bound-object-kind Secret --bound-object-name mysecret --bound-object-uid 0d4691ed-659b-4935-a832-355f77ee47cc

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl plugin list

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

List all available plugin files on a user's PATH. To see plugins binary names without the full path use --name-only flag.

kubectl rollout history

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

View previous rollout revisions and configurations.

kubectl rollout history (TYPE NAME | TYPE/NAME) [flags]

Examples

 # View the rollout history of a deployment
 kubectl rollout history deployment/abc
 
 # View the details of daemonset revision 3
 kubectl rollout history daemonset/abc --revision=3

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl rollout pause

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Mark the provided resource as paused.

Paused resources will not be reconciled by a controller. Use "kubectl rollout resume" to resume a paused resource. Currently only deployments support being paused.

kubectl rollout restart

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Restart a resource.

 Resource rollout will be restarted.

kubectl rollout restart RESOURCE

Examples

 # Restart all deployments in the test-namespace namespace
 kubectl rollout restart deployment -n test-namespace
 
 # Restart a deployment
 kubectl rollout restart deployment/nginx
 
 # Restart a daemon set
 kubectl rollout restart daemonset/abc
 
 # Restart deployments with the app=nginx label
 kubectl rollout restart deployment --selector=app=nginx

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl rollout resume

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Resume a paused resource.

Paused resources will not be reconciled by a controller. By resuming a resource, we allow it to be reconciled again. Currently only deployments support being resumed.

kubectl rollout status

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Show the status of the rollout.

By default 'rollout status' will watch the status of the latest rollout until it's done. If you don't want to wait for the rollout to finish then you can use --watch=false. Note that if a new rollout starts in-between, then 'rollout status' will continue watching the latest revision. If you want to pin to a specific revision and abort if it is rolled over by another revision, use --revision=N where N is the revision you need to watch for.

kubectl rollout undo

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Roll back to a previous rollout.

kubectl rollout undo (TYPE NAME | TYPE/NAME) [flags]

Examples

 # Roll back to the previous deployment
 kubectl rollout undo deployment/abc
 
 # Roll back to daemonset revision 3
 kubectl rollout undo daemonset/abc --to-revision=3
 
 # Roll back to the previous deployment with dry-run
 kubectl rollout undo --dry-run=server deployment/abc

Options

--allow-missing-template-keys Default: true

If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.

kubectl set env

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Update environment variables on a pod template.

List environment variable definitions in one or more pods, pod templates. Add, update, or remove container environment variable definitions in one or more pod templates (within replication controllers or deployment configurations). View or modify the environment variable definitions on all containers in the specified pods or pod templates, or just those that match a wildcard.

kubectl set image

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Update existing container image(s) of resources.

Possible resources include (case insensitive):

 pod (po), replicationcontroller (rc), deployment (deploy), daemonset (ds), statefulset (sts), cronjob (cj), replicaset (rs)

kubectl set image (-f FILENAME | TYPE NAME) CONTAINER_NAME_1=CONTAINER_IMAGE_1 ... CONTAINER_NAME_N=CONTAINER_IMAGE_N

Examples

 # Set a deployment's nginx container image to 'nginx:1.9.1', and its busybox container image to 'busybox'
 kubectl set image deployment/nginx busybox=busybox nginx=nginx:1.9.1
 
 # Update all deployments' and rc's nginx container's image to 'nginx:1.9.1'
 kubectl set image deployments,rc nginx=nginx:1.9.1 --all
 
 # Update image of all containers of daemonset abc to 'nginx:1.9.1'
 kubectl set image daemonset abc *=nginx:1.9.1
 
 # Print result (in yaml format) of updating nginx container image from local file, without hitting the server
 kubectl set image -f path/to/file.yaml nginx=nginx:1.9.1 --local -o yaml

Options

--all

The kubelet is the primary "node agent" that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.

Leases

Mon, 01 Jan 0001 00:00:00 +0000

Distributed systems often have a need for leases, which provide a mechanism to lock shared resources and coordinate activity between members of a set. In Kubernetes, the lease concept is represented by Lease objects in the coordination.k8s.io API Group, which are used for system-critical capabilities such as node heartbeats and component-level leader election.

Node heartbeats

Kubernetes uses the Lease API to communicate kubelet node heartbeats to the Kubernetes API server. For every Node , there is a Lease object with a matching name in the kube-node-lease namespace. Under the hood, every kubelet heartbeat is an update request to this Lease object, updating the spec.renewTime field for the Lease. The Kubernetes control plane uses the time stamp of this field to determine the availability of this Node.

Managing Kubernetes Objects Using Imperative Commands

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes objects can quickly be created, updated, and deleted directly using imperative commands built into the kubectl command-line tool. This document explains how those commands are organized and how to use them to manage live objects.

Before you begin

Install kubectl.

Managing Secrets using Kustomize

Mon, 01 Jan 0001 00:00:00 +0000

kubectl supports using the Kustomize object management tool to manage Secrets and ConfigMaps. You create a resource generator using Kustomize, which generates a Secret that you can apply to the API server using kubectl.

Before you begin

Object Names and IDs

Mon, 01 Jan 0001 00:00:00 +0000

Each object in your cluster has a Name that is unique for that type of resource. Every Kubernetes object also has a UID that is unique across your whole cluster.

For example, you can only have one Pod named myapp-1234 within the same namespace, but you can have one Pod and one Deployment that are each named myapp-1234.

Operator pattern

Mon, 01 Jan 0001 00:00:00 +0000

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Motivation

The operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.

Pinterest Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

After eight years in existence, Pinterest had grown into 1,000 microservices and multiple layers of infrastructure and diverse set-up tools and platforms. In 2016 the company launched a roadmap towards a new compute platform, led by the vision of creating the fastest path from an idea to production, without making engineers worry about the underlying infrastructure.

Solution

The first phase involved moving services to Docker containers. Once these services went into production in early 2017, the team began looking at orchestration to help create efficiencies and manage them in a decentralized way. After an evaluation of various solutions, Pinterest went with Kubernetes.

Pod Lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the lifecycle of a Pod. Pods follow a defined lifecycle, starting in the Pending phase, moving through Running if at least one of its primary containers starts OK, and then through either the Succeeded or Failed phases depending on whether any container in the Pod terminated in failure.

Like individual application containers, Pods are considered to be relatively ephemeral (rather than durable) entities. Pods are created, assigned a unique ID (UID), and scheduled to run on nodes where they remain until termination (according to restart policy) or deletion. If a Node dies, the Pods running on (or scheduled to run on) that node are marked for deletion. The control plane marks the Pods for removal after a timeout period.

Pod Overhead

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.24 [stable]

When you run a Pod on a Node, the Pod itself takes an amount of system resources. These resources are additional to the resources needed to run the container(s) inside the Pod. In Kubernetes, Pod Overhead is a way to account for the resources consumed by the Pod infrastructure on top of the container requests & limits.

In Kubernetes, the Pod's overhead is set at admission time according to the overhead associated with the Pod's RuntimeClass.

Pod Security Policies

Mon, 01 Jan 0001 00:00:00 +0000

Removed feature

PodSecurityPolicy was deprecated in Kubernetes v1.21, and removed from Kubernetes in v1.25.

Instead of using PodSecurityPolicy, you can enforce similar restrictions on Pods using either or both:

Pod Security Admission
a 3rd party admission plugin, that you deploy and configure yourself

For a migration guide, see Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller. For more information on the removal of this API, see PodSecurityPolicy Deprecation: Past, Present, and Future.

Resize CPU and Memory Resources assigned to Containers

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [stable](enabled by default)

This page explains how to change the CPU and memory resource requests and limits assigned to a container without recreating the Pod.

Traditionally, changing a Pod's resource requirements necessitated deleting the existing Pod and creating a replacement, often managed by a workload controller. In-place Pod Resize allows changing the CPU/memory allocation of container(s) within a running Pod while potentially avoiding application disruption. The process for resizing Pod resources is covered in Resize CPU and Memory Resources assigned to Pods.

Resize CPU and Memory Resources assigned to Pods

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [alpha](disabled by default)

This page explains how to change the CPU and memory resources set at the Pod level without recreating the Pod.

The In-place Pod Resize feature allows modifying resource allocations for a running Pod, avoiding application disruption. The process for resizing individual container resources is covered in Resize CPU and Memory Resources assigned to Containers.

This page highlights In-place Pod-level resources resize. Pod-level resources are defined in spec.resources and they act as the upper bound on the aggregate resources consumed by all containers in the Pod. The In-place Pod-level resources resize feature lets you change these aggregate CPU and memory allocations for a running Pod directly.

Restrict a Container's Access to Resources with AppArmor

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.31 [stable](enabled by default)

This page shows you how to load AppArmor profiles on your nodes and enforce those profiles in Pods. To learn more about how Kubernetes can confine Pods using AppArmor, see Linux kernel security constraints for Pods and containers.

Objectives

See an example of how to load a profile on a Node
Learn how to enforce the profile on a Pod
Learn how to check that the profile is loaded
See what happens when a profile is violated
See what happens when a profile cannot be loaded

Before you begin

AppArmor is an optional kernel module and Kubernetes feature, so verify it is supported on your Nodes before proceeding:

Run a Replicated Stateful Application

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to run a replicated stateful application using a StatefulSet. This application is a replicated MySQL database. The example topology has a single primary server and multiple replicas, using asynchronous row-based replication.

Note:

This is not a production configuration. MySQL settings remain on insecure defaults to keep the focus on general patterns for running stateful applications in Kubernetes.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Running Pods on Only Some Nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page demonstrates how can you run Pods on only some Nodes as part of a DaemonSet

Before you begin

Runtime Class

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.20 [stable]

This page describes the RuntimeClass resource and runtime selection mechanism.

RuntimeClass is a feature for selecting the container runtime configuration. The container runtime configuration is used to run a Pod's containers.

Motivation

You can set a different RuntimeClass between different Pods to provide a balance of performance versus security. For example, if part of your workload deserves a high level of information security assurance, you might choose to schedule those Pods so that they run in a container runtime that uses hardware virtualization. You'd then benefit from the extra isolation of the alternative runtime, at the expense of some additional overhead.

Scheduling Policies

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes versions before v1.23, a scheduling policy can be used to specify the predicates and priorities process. For example, you can set a scheduling policy by running kube-scheduler --policy-config-file <filename> or kube-scheduler --policy-configmap <ConfigMap>.

This scheduling policy is not supported since Kubernetes v1.23. Associated flags policy-config-file, policy-configmap, policy-configmap-namespace and use-legacy-policy-config are also not supported. Instead, use the Scheduler Configuration to achieve similar behavior.

What's next

Learn about scheduling
Learn about kube-scheduler Configuration
Read the kube-scheduler configuration reference (v1)

Secrets

Mon, 01 Jan 0001 00:00:00 +0000

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don't need to include confidential data in your application code.

StatefulSets

Mon, 01 Jan 0001 00:00:00 +0000

StatefulSet is the workload API object used to manage stateful applications.

Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods.

Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of its Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

Submitting articles to Kubernetes blogs

Mon, 01 Jan 0001 00:00:00 +0000

There are two official Kubernetes blogs, and the CNCF has its own blog where you can cover Kubernetes too. For the main Kubernetes blog, we (the Kubernetes project) like to publish articles with different perspectives and special focuses, that have a link to Kubernetes.

With only a few special case exceptions, we only publish content that hasn't been submitted or published anywhere else.

Writing for the Kubernetes blog(s)

As an author, you have three different routes towards publication.

Upgrading kubeadm clusters

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to upgrade a Kubernetes cluster created with kubeadm from version 1.34.x to version 1.35.x, and from version 1.35.x to 1.35.y (where y > x). Skipping MINOR versions when upgrading is unsupported. For more details, please visit Version Skew Policy.

To see information about upgrading clusters created using older versions of kubeadm, please refer to following pages instead:

The Kubernetes project recommends upgrading to the latest patch releases promptly, and to ensure that you are running a supported minor release of Kubernetes. Following this recommendation helps you to stay secure.

Use Cilium for NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use Cilium for NetworkPolicy.

For background on Cilium, read the Introduction to Cilium.

Before you begin

Validate node setup

Mon, 01 Jan 0001 00:00:00 +0000

Node Conformance Test

Node conformance test is a containerized test framework that provides a system verification and functionality test for a node. The test validates whether the node meets the minimum requirements for Kubernetes; a node that passes the test is qualified to join a Kubernetes cluster.

Node Prerequisite

To run node conformance test, a node must satisfy the same prerequisites as a standard Kubernetes node. At a minimum, the node should have the following daemons installed:

Versions in CustomResourceDefinitions

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to add versioning information to CustomResourceDefinitions, to indicate the stability level of your CustomResourceDefinitions or advance your API to a new version with conversion between API representations. It also describes how to upgrade an object from one version to another.

Before you begin

Using RBAC Authorization

Mon, 01 Jan 0001 00:00:00 +0000

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.

RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.

To enable RBAC, start the API server with the --authorization-config flag set to a file that includes the RBAC authorizer; for example:

Using Node Authorization

Mon, 01 Jan 0001 00:00:00 +0000

Node authorization is a special-purpose authorization mode that specifically authorizes API requests made by kubelets.

Overview

The Node authorizer allows a kubelet to perform API operations. This includes:

Read operations:

services
endpoints
nodes
pods
secrets, configmaps, persistent volume claims and persistent volumes related to pods bound to the kubelet's node

FEATURE STATE: Kubernetes v1.34 [stable](enabled by default)

Kubelets are limited to reading their own Node objects, and only reading pods bound to their node.

Common Expression Language in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

The Common Expression Language (CEL) is used in the Kubernetes API to declare validation rules, policy rules, and other constraints or conditions.

CEL expressions are evaluated directly in the API server, making CEL a convenient alternative to out-of-process mechanisms, such as webhooks, for many extensibility use cases. Your CEL expressions continue to execute so long as the control plane's API server component remains available.

Configuring swap memory on Kubernetes nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an example of how to provision and configure swap memory on a Kubernetes node using kubeadm.

Objectives

Provision swap memory on a Kubernetes node using kubeadm.
Learn to configure both encrypted and unencrypted swap.
Learn to enable swap on boot.

Before you begin

Webhook Mode

Mon, 01 Jan 0001 00:00:00 +0000

A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. A web application implementing WebHooks will POST a message to a URL when certain things happen.

When specified, mode Webhook causes Kubernetes to query an outside REST service when determining user privileges.

Configuration File Format

Mode Webhook requires a file for HTTP configuration, specify by the --authorization-webhook-config-file=SOME_FILENAME flag.

Using ABAC Authorization

Mon, 01 Jan 0001 00:00:00 +0000

Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together.

Policy File Format

To enable ABAC mode, specify --authorization-policy-file=SOME_FILENAME and --authorization-mode=ABAC on startup.

The file format is one JSON object per line. There should be no enclosing list or map, only one map per line.

Each line is a "policy object", where each such object is a map with the following properties:

Admission Control in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of admission controllers.

An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the resource, but after the request is authenticated and authorized.

Several important features of Kubernetes require an admission controller to be enabled in order to properly support the feature. As a result, a Kubernetes API server that is not properly configured with the right set of admission controllers is an incomplete server that will not support all the features you expect.

Adopting Sidecar Containers

Mon, 01 Jan 0001 00:00:00 +0000

This section is relevant for people adopting a new built-in sidecar containers feature for their workloads.

Sidecar container is not a new concept as posted in the blog post. Kubernetes allows running multiple containers in a Pod to implement this concept. However, running a sidecar container as a regular container has a lot of limitations being fixed with the new built-in sidecar containers support.

FEATURE STATE: Kubernetes v1.33 [stable](enabled by default)

Objectives

Understand the need for sidecar containers
Be able to troubleshoot issues with the sidecar containers
Understand options to universally "inject" sidecar containers to any workload

Before you begin

Blog guidelines

Mon, 01 Jan 0001 00:00:00 +0000

These guidelines cover the main Kubernetes blog and the Kubernetes contributor blog.

All blog content must also adhere to the overall policy in the content guide.

Before you begin

Make sure you are familiar with the introduction sections of contributing to Kubernetes blogs, not just to learn about the two official blogs and the differences between them, but also to get an overview of the process.

Original content

The Kubernetes project accepts original content only, in English.

Cloud Controller Manager

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.11 [beta]

Cloud infrastructure technologies let you run Kubernetes on public, private, and hybrid clouds. Kubernetes believes in automated, API-driven infrastructure without tight coupling between components.

The cloud-controller-manager is a Kubernetes control plane component that embeds cloud-specific control logic. The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster.

Configure Minimum and Maximum CPU Constraints for a Namespace

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to set minimum and maximum values for the CPU resources used by containers and Pods in a namespace. You specify minimum and maximum CPU values in a LimitRange object. If a Pod does not meet the constraints imposed by the LimitRange, it cannot be created in the namespace.

Configure RunAsUserName for Windows pods and containers

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.18 [stable]

This page shows how to use the runAsUserName setting for Pods and containers that will run on Windows nodes. This is roughly equivalent of the Linux-specific runAsUser setting, allowing you to run applications in a container as a different username than the default.

Before you begin

You need to have a Kubernetes cluster and the kubectl command-line tool must be configured to communicate with your cluster. The cluster is expected to have Windows worker nodes where pods with containers running Windows workloads will get scheduled.

Container Lifecycle Hooks

Mon, 01 Jan 0001 00:00:00 +0000

This page describes how kubelet managed Containers can use the Container lifecycle hook framework to run code triggered by events during their management lifecycle.

Overview

Analogous to many programming language frameworks that have component lifecycle hooks, such as Angular, Kubernetes provides Containers with lifecycle hooks. The hooks enable Containers to be aware of events in their management lifecycle and run code implemented in a handler when the corresponding lifecycle hook is executed.

Customizing components with the kubeadm API

Mon, 01 Jan 0001 00:00:00 +0000

This page covers how to customize the components that kubeadm deploys. For control plane components you can use flags in the ClusterConfiguration structure or patches per-node. For the kubelet and kube-proxy you can use KubeletConfiguration and KubeProxyConfiguration, accordingly.

All of these options are possible via the kubeadm configuration API. For more details on each field in the configuration you can navigate to our API reference pages.

Note:

To reconfigure a cluster that has already been created see Reconfiguring a kubeadm cluster.

Customizing the control plane with flags in `ClusterConfiguration`

The kubeadm ClusterConfiguration object exposes a way for users to override the default flags passed to control plane components such as the APIServer, ControllerManager, Scheduler and Etcd. The components are defined using the following structures:

DaemonSet

Mon, 01 Jan 0001 00:00:00 +0000

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

Some typical uses of a DaemonSet are:

running a cluster storage daemon on every node
running a logs collection daemon on every node
running a node monitoring daemon on every node

In a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon. A more complex setup might use multiple DaemonSets for a single type of daemon, but with different flags and/or different memory and cpu requests for different hardware types.

Debug Init Containers

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to investigate problems related to the execution of Init Containers. The example command lines below refer to the Pod as <pod-name> and the Init Containers as <init-container-1> and <init-container-2>.

Before you begin

Enforcing Pod Security Standards

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of best practices when it comes to enforcing Pod Security Standards.

Using the built-in Pod Security Admission Controller

FEATURE STATE: Kubernetes v1.25 [stable]

The Pod Security Admission Controller intends to replace the deprecated PodSecurityPolicies.

Configure all cluster namespaces

Namespaces that lack any configuration at all should be considered significant gaps in your cluster security model. We recommend taking the time to analyze the types of workloads occurring in each namespace, and by referencing the Pod Security Standards, decide on an appropriate level for each of them. Unlabeled namespaces should only indicate that they've yet to be evaluated.

Expose Pod Information to Containers Through Files

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how a Pod can use a downwardAPI volume, to expose information about itself to containers running in the Pod. A downwardAPI volume can expose Pod fields and container fields.

In Kubernetes, there are two ways to expose Pod and container fields to a running container:

Environment variables
Volume files, as explained in this task

Together, these two ways of exposing Pod and container fields are called the downward API.

Imperative Management of Kubernetes Objects Using Configuration Files

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes objects can be created, updated, and deleted by using the kubectl command-line tool along with an object configuration file written in YAML or JSON. This document explains how to define and manage objects using configuration files.

Before you begin

Install kubectl.

Init Containers

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of init containers: specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.

You can specify init containers in the Pod specification alongside the containers array (which describes app containers).

In Kubernetes, a sidecar container is a container that starts before the main application container and continues to run. This document is about init containers: containers that run to completion during Pod initialization.

JSONPath Support

Mon, 01 Jan 0001 00:00:00 +0000

The kubectl tool supports JSONPath templates as an output format.

A JSONPath template is composed of JSONPath expressions enclosed by curly braces: { and }. Kubectl uses JSONPath expressions to filter on specific fields in the JSON object and format the output. In addition to the original JSONPath template syntax, the following functions and syntax are valid:

kubeadm upgrade

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm upgrade is a user-friendly command that wraps complex upgrading logic behind one command, with support for both planning an upgrade and actually performing it.

kubeadm upgrade guidance

The steps for performing an upgrade using kubeadm are outlined in this document. For older versions of kubeadm, please refer to older documentation sets of the Kubernetes website.

You can use kubeadm upgrade diff to see the changes that would be applied to static pod manifests.

kubeadm upgrade phases

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm upgrade apply phase

Using the phases of kubeadm upgrade apply, you can choose to execute the separate steps of the initial upgrade of a control plane node.

Synopsis

Use this command to invoke single phase of the "apply" workflow

Kubernetes Deprecation Policy

Mon, 01 Jan 0001 00:00:00 +0000

This document details the deprecation policy for various facets of the system.

Kubernetes is a large system with many components and many contributors. As with any such software, the feature set naturally evolves over time, and sometimes a feature may need to be removed. This could include an API, a flag, or even an entire feature. To avoid breaking existing users, Kubernetes follows a deprecation policy for aspects of the system that are slated to be removed.

KYAML Reference

Mon, 01 Jan 0001 00:00:00 +0000

KYAML is a safer and less ambiguous subset of YAML, initially introduced in Kubernetes v1.34 (alpha) and enabled by default in v1.35 (beta). Designed specifically for Kubernetes, KYAML addresses common YAML pitfalls such as whitespace sensitivity and implicit type coercion while maintaining full compatibility with existing YAML parsers and tooling.

This reference describes KYAML syntax.

Getting started with KYAML

YAML’s reliance on indentation and implicit type coercion often leads to configuration errors, especially in CI/CD pipelines and templating systems like Helm. KYAML eliminates these issues by enforcing explicit syntax and structure, making configurations more reliable and easier to debug.

Labels and Selectors

Mon, 01 Jan 0001 00:00:00 +0000

Labels are key/value pairs that are attached to objects such as Pods. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time. Each object can have a set of key/value labels defined. Each Key must be unique for a given object.

Liveness, Readiness, and Startup Probes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes has various types of probes:

Liveness probe
Readiness probe
Startup probe

Liveness probe

Liveness probes determine when to restart a container. For example, liveness probes could catch a deadlock when an application is running but unable to make progress.

If a container fails its liveness probe repeatedly, the kubelet restarts the container.

Liveness probes do not wait for readiness probes to succeed. If you want to wait before executing a liveness probe, you can either define initialDelaySeconds or use a startup probe.

Managing Workloads

Mon, 01 Jan 0001 00:00:00 +0000

You've deployed your application and exposed it via a Service. Now what? Kubernetes provides a number of tools to help you manage your application deployment, including scaling and updating.

Organizing resource configurations

Many applications require multiple resources to be created, such as a Deployment along with a Service. Management of multiple resources can be simplified by grouping them together in the same file (separated by --- in YAML). For example:

Node Labels Populated By The Kubelet

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes nodes come pre-populated with a standard set of labels.

You can also set your own labels on nodes, either through the kubelet configuration or using the Kubernetes API.

Preset labels

The preset labels that Kubernetes sets on nodes are:

kubernetes.io/arch
kubernetes.io/hostname
kubernetes.io/os
node.kubernetes.io/instance-type (if known to the kubelet – Kubernetes may not have this information to set the label)
topology.kubernetes.io/region (if known to the kubelet – Kubernetes may not have this information to set the label)
topology.kubernetes.io/zone (if known to the kubelet – Kubernetes may not have this information to set the label)

Note:

The value of these labels is cloud provider specific and is not guaranteed to be reliable. For example, the value of kubernetes.io/hostname may be the same as the node name in some environments and a different value in other environments.

What's next

See Well-Known Labels, Annotations and Taints for a list of common labels.
Learn how to add a label to a node.

Pod Scheduling Readiness

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.30 [stable]

Pods were considered ready for scheduling once created. Kubernetes scheduler does its due diligence to find nodes to place all pending Pods. However, in a real-world case, some Pods may stay in a "miss-essential-resources" state for a long period. These Pods actually churn the scheduler (and downstream integrators like Cluster AutoScaler) in an unnecessary manner.

By specifying/removing a Pod's .spec.schedulingGates, you can control when a Pod is ready to be considered for scheduling.

Pod Topology Spread Constraints

Mon, 01 Jan 0001 00:00:00 +0000

You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization.

You can set cluster-level constraints as a default, or configure topology spread constraints for individual workloads.

Ports and Protocols

Mon, 01 Jan 0001 00:00:00 +0000

When running Kubernetes in an environment with strict network boundaries, such as on-premises datacenter with physical network firewalls or Virtual Networks in Public Cloud, it is useful to be aware of the ports and protocols used by Kubernetes components.

Control plane

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	6443	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10259	kube-scheduler	Self
TCP	Inbound	10257	kube-controller-manager	Self

Although etcd ports are included in control plane section, you can also host your own etcd cluster externally or on custom ports.

Process ID Limits And Reservations

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.20 [stable]

Kubernetes allow you to limit the number of process IDs (PIDs) that a Pod can use. You can also reserve a number of allocatable PIDs for each node for use by the operating system and daemons (rather than by Pods).

Process IDs (PIDs) are a fundamental resource on nodes. It is trivial to hit the task limit without hitting any other resource limits, which can then cause instability to a host machine.

Resource Management for Pods and Containers

Mon, 01 Jan 0001 00:00:00 +0000

When you specify a Pod, you can optionally specify how much of each resource a container needs. The most common resources to specify are CPU and memory (RAM); there are others.

When you specify the resource request for containers in a Pod, the kube-scheduler uses this information to decide which node to place the Pod on. When you specify a resource limit for a container, the kubelet enforces those limits so that the running container is not allowed to use more of that resource than the limit you set. The kubelet also reserves at least the request amount of that system resource specifically for that container to use.

Restrict a Container's Syscalls with seccomp

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.19 [stable]

Seccomp stands for secure computing mode and has been a feature of the Linux kernel since version 2.6.12. It can be used to sandbox the privileges of a process, restricting the calls it is able to make from userspace into the kernel. Kubernetes lets you automatically apply seccomp profiles loaded onto a node to your Pods and containers.

Running ZooKeeper, A Distributed System Coordinator

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial demonstrates running Apache Zookeeper on Kubernetes using StatefulSets, PodDisruptionBudgets, and PodAntiAffinity.

Before you begin

Before starting this tutorial, you should be familiar with the following Kubernetes concepts:

You must have a cluster with at least four nodes, and each node requires at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and drain the cluster's nodes. This means that the cluster will terminate and evict all Pods on its nodes, and the nodes will temporarily become unschedulable. You should use a dedicated cluster for this tutorial, or you should ensure that the disruption you cause will not interfere with other tenants.

Security For Linux Nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes security considerations and best practices specific to the Linux operating system.

Protection for Secret data on nodes

On Linux nodes, memory-backed volumes (such as secret volume mounts, or emptyDir with medium: Memory) are implemented with a tmpfs filesystem.

If you have swap configured and use an older Linux kernel (or a current kernel and an unsupported configuration of Kubernetes), memory backed volumes can have data written to persistent storage.

Security For Windows Nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes security considerations and best practices specific to the Windows operating system.

Protection for Secret data on nodes

On Windows, data from Secrets are written out in clear text onto the node's local storage (as compared to using tmpfs / in-memory filesystems on Linux). As a cluster operator, you should take both of the following additional measures:

Use file ACLs to secure the Secrets' file location.
Apply volume-level encryption using BitLocker.

Container users

RunAsUsername can be specified for Windows Pods or containers to execute the container processes as specific user. This is roughly equivalent to RunAsUser.

Storage Classes

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the concept of a StorageClass in Kubernetes. Familiarity with volumes and persistent volumes is suggested.

A StorageClass provides a way for administrators to describe the classes of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is unopinionated about what classes represent.

The Kubernetes concept of a storage class is similar to “profiles” in some other storage system designs.

Documentation Style Guide

Mon, 01 Jan 0001 00:00:00 +0000

This page gives writing style guidelines for the Kubernetes documentation. These are guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.

For additional information on creating new content for the Kubernetes documentation, read the Documentation Content Guide.

Changes to the style guide are made by SIG Docs as a group. To propose a change or addition, add it to the agenda for an upcoming SIG Docs meeting, and attend the meeting to participate in the discussion.

The Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

The core of Kubernetes' control plane is the API server. The API server exposes an HTTP API that lets end users, different parts of your cluster, and external components communicate with one another.

The Kubernetes API lets you query and manipulate the state of API objects in Kubernetes (for example: Pods, Namespaces, ConfigMaps, and Events).

Troubleshooting CNI plugin-related errors

Mon, 01 Jan 0001 00:00:00 +0000

To avoid CNI plugin-related errors, verify that you are using or upgrading to a container runtime that has been tested to work correctly with your version of Kubernetes.

About the "Incompatible CNI versions" and "Failed to destroy network for sandbox" errors

Service issues exist for pod CNI network setup and tear down in containerd v1.6.0-v1.6.3 when the CNI plugins have not been upgraded and/or the CNI config version is not declared in the CNI config files. The containerd team reports, "these issues are resolved in containerd v1.6.4."

Turnkey Cloud Solutions

Mon, 01 Jan 0001 00:00:00 +0000

This page provides a list of Kubernetes certified solution providers. From each provider page, you can learn how to install and setup production ready clusters.

Upgrading Linux nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to upgrade a Linux Worker Nodes created with kubeadm.

Before you begin

You need to have shell access to all the nodes, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts.

To check the version, enter kubectl version.

Use an HTTP Proxy to Access the Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use an HTTP proxy to access the Kubernetes API.

Before you begin

Use Kube-router for NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use Kube-router for NetworkPolicy.

Before you begin

You need to have a Kubernetes cluster running. If you do not already have a cluster, you can create one by using any of the cluster installers like Kops, Bootkube, Kubeadm etc.

Installing Kube-router addon

The Kube-router Addon comes with a Network Policy Controller that watches Kubernetes API server for any NetworkPolicy and pods updated and configures iptables rules and ipsets to allow or block traffic as directed by the policies. Please follow the trying Kube-router with cluster installers guide to install Kube-router addon.

Use Port Forwarding to Access Applications in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use kubectl port-forward to connect to a MongoDB server running in a Kubernetes cluster. This type of connection can be useful for database debugging.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Using Source IP

Mon, 01 Jan 0001 00:00:00 +0000

Applications running in a Kubernetes cluster find and communicate with each other, and the outside world, through the Service abstraction. This document explains what happens to the source IP of packets sent to different types of Services, and how you can toggle this behavior according to your needs.

Before you begin

Terminology

This document makes use of the following terms:

NAT: Network address translation
Source NAT: Replacing the source IP on a packet; in this page, that usually means replacing with the IP address of a node.
Destination NAT: Replacing the destination IP on a packet; in this page, that usually means replacing with the IP address of a Pod
VIP: A virtual IP address, such as the one assigned to every Service in Kubernetes
kube-proxy: A network daemon that orchestrates Service VIP management on every node

Prerequisites

Volume Attributes Classes

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.34 [stable](enabled by default)

This page assumes that you are familiar with StorageClasses, volumes and PersistentVolumes in Kubernetes.

A VolumeAttributesClass provides a way for administrators to describe the mutable "classes" of storage they offer. Different classes might map to different quality-of-service levels. Kubernetes itself is un-opinionated about what these classes represent.

This feature is generally available (GA) as of version 1.34, and users have the option to disable it.

Upgrading Windows nodes

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.18 [beta]

This page explains how to upgrade a Windows node created with kubeadm.

Before you begin

Your Kubernetes server must be at or later than version 1.17.

To check the version, enter kubectl version.

Local Files And Paths Used By The Kubelet

Mon, 01 Jan 0001 00:00:00 +0000

The kubelet is mostly a stateless process running on a Kubernetes node. This document outlines files that kubelet reads and writes.

Note:

This document is for informational purpose and not describing any guaranteed behaviors or APIs. It lists resources used by the kubelet, which is an implementation detail and a subject to change at any release.

The kubelet typically uses the control plane as the source of truth on what needs to run on the Node, and the container runtime to retrieve the current state of containers. So long as you provide a kubeconfig (API client configuration) to the kubelet, the kubelet does connect to your control plane; otherwise the node operates in standalone mode.

Use a SOCKS5 Proxy to Access the Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.24 [stable]

This page shows how to use a SOCKS5 proxy to access the API of a remote Kubernetes cluster. This is useful when the cluster you want to access does not expose its API directly on the public internet.

Before you begin

Deprecated API Migration Guide

Mon, 01 Jan 0001 00:00:00 +0000

As the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old API is deprecated and eventually removed. This page contains information you need to know when migrating from deprecated API versions to newer and more stable API versions.

Removed APIs by release

v1.32

The v1.32 release stopped serving the following deprecated API versions:

Flow control resources

The flowcontrol.apiserver.k8s.io/v1beta3 API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.32.

Dynamic Admission Control

Mon, 01 Jan 0001 00:00:00 +0000

In addition to compiled-in admission plugins, admission plugins can be developed as extensions and run as webhooks configured at runtime. This page describes how to build, configure, use, and monitor admission webhooks.

What are admission webhooks?

Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked and can reject requests to enforce custom policies.

Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects (e.g. Deployments, Services, etc.) and not for cluster-wide objects (e.g. StorageClass, Nodes, PersistentVolumes, etc.).

When to Use Multiple Namespaces

Namespaces are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide.

Adform Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

Adform's mission is to provide a secure and transparent full stack of advertising technology to enable digital ads across devices. The company has a large infrastructure: OpenStack-based private clouds running on 1,100 physical servers in 7 data centers around the world, 3 of which were opened in the past year. With the company's growth, the infrastructure team felt that "our private cloud was not really flexible enough," says IT System Engineer Edgaras Apšega. "The biggest pain point is that our developers need to maintain their virtual machines, so rolling out technology and new software takes time. We were really struggling with our releases, and we didn't have self-healing infrastructure."

Ygrene Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

A PACE (Property Assessed Clean Energy) financing company, Ygrene has funded more than $1 billion in loans since 2010. In order to approve and process those loans, "We have lots of data sources that are being aggregated, and we also have lots of systems that need to churn on that data," says Ygrene Development Manager Austin Adams. The company was utilizing massive servers, and "we just reached the limit of being able to scale them vertically. We had a really unstable system that became overwhelmed with requests just for doing background data processing in real time. The performance the users saw was very poor. We needed a solution that wouldn't require us to make huge refactors to the code base." As a finance company, Ygrene also needed to ensure that they were shipping their applications securely.

SlingTV Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

Launched by DISH Network in 2015, Sling TV experienced great customer growth from the beginning. After just a year, "we were going through some growing pains of some of the legacy systems and trying to find the right architecture to enable our future," says Brad Linder, Sling TV's Cloud Native & Big Data Evangelist. The company has particular challenges: "We take live TV and distribute it over the internet out to a user's device that we do not control," says Linder. "In a lot of ways, we are working in the Wild West: The internet is what it is going to be, and if a customer's service does not work for whatever reason, they do not care why. They just want things to work. Those are the variables of the equation that we have to try to solve. We really have to try to enable optionality and good customer experience at web scale."

About cgroup v2

Mon, 01 Jan 0001 00:00:00 +0000

On Linux, control groups constrain resources that are allocated to processes.

The kubelet and the underlying container runtime need to interface with cgroups to enforce resource management for pods and containers which includes cpu/memory requests and limits for containerized workloads.

Autoscaling Workloads

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, you can scale a workload depending on the current demand of resources. This allows your cluster to react to changes in resource demand more elastically and efficiently.

When you scale a workload, you can either increase or decrease the number of replicas managed by the workload, or adjust the resources available to the replicas in-place.

The first approach is referred to as horizontal scaling, while the second is referred to as vertical scaling.

Blog article mirroring

Mon, 01 Jan 0001 00:00:00 +0000

Some articles appear on both blogs: there is a primary version of the article, and a mirror article on the other blog.

This page describes the criteria for mirroring, the motivation for mirroring, and explains what you should do to ensure that an article publishes to both blogs.

Check whether dockershim removal affects you

Mon, 01 Jan 0001 00:00:00 +0000

The dockershim component of Kubernetes allows the use of Docker as a Kubernetes's container runtime. Kubernetes' built-in dockershim component was removed in release v1.24.

This page explains how your cluster could be using Docker as a container runtime, provides details on the role that dockershim plays when in use, and shows steps you can take to check whether any workloads could be affected by dockershim removal.

Cluster Networking

Mon, 01 Jan 0001 00:00:00 +0000

Networking is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work. There are 4 distinct networking problems to address:

Highly-coupled container-to-container communications: this is solved by Pods and localhost communications.
Pod-to-Pod communications: this is the primary focus of this document.
Pod-to-Service communications: this is covered by Services.
External-to-Service communications: this is also covered by Services.

Kubernetes is all about sharing machines among applications. Typically, sharing machines requires ensuring that two applications do not try to use the same ports. Coordinating ports across multiple developers is very difficult to do at scale and exposes users to cluster-level issues outside of their control.

Configure Memory and CPU Quotas for a Namespace

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to set quotas for the total amount memory and CPU that can be used by all Pods running in a namespace. You specify quotas in a ResourceQuota object.

Before you begin

Configuring a cgroup driver

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to configure the kubelet's cgroup driver to match the container runtime cgroup driver for kubeadm clusters.

Before you begin

You should be familiar with the Kubernetes container runtime requirements.

Configuring the container runtime cgroup driver

The Container runtimes page explains that the systemd driver is recommended for kubeadm based setups instead of the kubelet's default cgroupfs driver, because kubeadm manages the kubelet as a systemd service.

Controlling Access to the Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of controlling access to the Kubernetes API.

Users access the Kubernetes API using kubectl, client libraries, or by making REST requests. Both human users and Kubernetes service accounts can be authorized for API access. When a request reaches the API, it goes through several stages, illustrated in the following diagram:

Transport security

By default, the Kubernetes API server listens on port 6443 on the first non-localhost network interface, protected by TLS. In a typical production Kubernetes cluster, the API serves on port 443. The port can be changed with the --secure-port, and the listening IP address with the --bind-address flag.

Create a Windows HostProcess Pod

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [stable]

Windows HostProcess containers enable you to run containerized workloads on a Windows host. These containers operate as normal processes but have access to the host network namespace, storage, and devices when given the appropriate user privileges. HostProcess containers can be used to deploy network plugins, storage configurations, device plugins, kube-proxy, and other components to Windows nodes without the need for dedicated proxies or the direct installation of host services.

CRI Pod & Container Metrics

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.23 [alpha]

The kubelet collects pod and container metrics via cAdvisor. As an alpha feature, Kubernetes lets you configure the collection of pod and container metrics via the Container Runtime Interface (CRI). You must enable the PodAndContainerStatsFromCRI feature gate and use a compatible CRI implementation (containerd >= 1.6.0, CRI-O >= 1.23.0) to use the CRI based collection mechanism.

Distribute Credentials Securely Using Secrets

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to securely inject sensitive data, such as passwords and encryption keys, into Pods.

Before you begin

Dynamic Volume Provisioning

Mon, 01 Jan 0001 00:00:00 +0000

Dynamic volume provisioning allows storage volumes to be created on-demand. Without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, and then create PersistentVolume objects to represent them in Kubernetes. The dynamic provisioning feature eliminates the need for cluster administrators to pre-provision storage. Instead, it automatically provisions storage when users create PersistentVolumeClaim objects.

Background

The implementation of dynamic volume provisioning is based on the API object StorageClass from the API group storage.k8s.io. A cluster administrator can define as many StorageClass objects as needed, each specifying a volume plugin (aka provisioner) that provisions a volume and the set of parameters to pass to that provisioner when provisioning. A cluster administrator can define and expose multiple flavors of storage (from the same or different storage systems) within a cluster, each with a custom set of parameters. This design also ensures that end users don't have to worry about the complexity and nuances of how storage is provisioned, but still have the ability to select from multiple storage options.

Generating Reference Documentation for the Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to update the Kubernetes API reference documentation.

The Kubernetes API reference documentation is built from the Kubernetes OpenAPI spec using the kubernetes-sigs/reference-docs generation code.

If you find bugs in the generated documentation, you need to fix them upstream.

If you need only to regenerate the reference documentation from the OpenAPI spec, continue reading this page.

Before you begin

Requirements:

You need a machine that is running Linux or macOS.

ING Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

After undergoing an agile transformation, ING realized it needed a standardized platform to support the work their developers were doing. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with Docker, Docker Swarm, Kubernetes, Mesos. Well, it's not really useful for a company to have one hundred wheels, instead of one good wheel.

Ingress Controllers

Mon, 01 Jan 0001 00:00:00 +0000

Note:

The Kubernetes project recommends using Gateway instead of Ingress. The Ingress API has been frozen.

This means that:

The Ingress API is generally available, and is subject to the stability guarantees for generally available APIs. The Kubernetes project has no plans to remove Ingress from Kubernetes.
The Ingress API is no longer being developed, and will have no further changes or updates made to it.

Ingress controllers

Kubernetes as a project supports and maintains AWS, and GCE ingress controllers.

Jobs

Mon, 01 Jan 0001 00:00:00 +0000

A Job creates one or more Pods and will continue to retry execution of the Pods until a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up the Pods it created. Suspending a Job will delete its active Pods until the Job is resumed again.

kubeadm config

Mon, 01 Jan 0001 00:00:00 +0000

During kubeadm init, kubeadm uploads the ClusterConfiguration object to your cluster in a ConfigMap called kubeadm-config in the kube-system namespace. This configuration is then read during kubeadm join, kubeadm reset and kubeadm upgrade.

You can use kubeadm config print to print the default static configuration that kubeadm uses for kubeadm init and kubeadm join.

Note:

The output of the command is meant to serve as an example. You must manually edit the output of this command to adapt to your setup. Remove the fields that you are not certain about and kubeadm will try to default them on runtime by examining the host.

For more information on init and join navigate to Using kubeadm init with a configuration file or Using kubeadm join with a configuration file.

kubectl for Docker Users

Mon, 01 Jan 0001 00:00:00 +0000

You can use the Kubernetes command line tool kubectl to interact with the API Server. Using kubectl is straightforward if you are familiar with the Docker command line tool. However, there are a few differences between the Docker commands and the kubectl commands. The following sections show a Docker sub-command and describe the equivalent kubectl command.

docker run

To run an nginx Deployment and expose the Deployment, see kubectl create deployment.

Kubelet Configuration Directory Merging

Mon, 01 Jan 0001 00:00:00 +0000

When using the kubelet's --config-dir flag to specify a drop-in directory for configuration, there is some specific behavior on how different types are merged.

Here are some examples of how different data types behave during configuration merging:

Structure Fields

There are two types of structure fields in a YAML structure: singular (or a scalar type) and embedded (structures that contain scalar types). The configuration merging process handles the overriding of singular and embedded struct fields to create a resulting kubelet configuration.

Kubelet Device Manager API Versions

Mon, 01 Jan 0001 00:00:00 +0000

This page provides details of version compatibility between the Kubernetes device plugin API, and different versions of Kubernetes itself.

Compatibility matrix

	`v1alpha1`	`v1beta1`
Kubernetes 1.21	-	✓
Kubernetes 1.22	-	✓
Kubernetes 1.23	-	✓
Kubernetes 1.24	-	✓
Kubernetes 1.25	-	✓
Kubernetes 1.26	-	✓

Key:

✓ Exactly the same features / API objects in both device plugin API and the Kubernetes version.
+ The device plugin API has features or API objects that may not be present in the Kubernetes cluster, either because the device plugin API has added additional new API calls, or that the server has removed an old API call. However, everything they have in common (most other APIs) will work. Note that alpha APIs may vanish or change significantly between one minor release and the next.
- The Kubernetes cluster has features the device plugin API can't use, either because server has added additional API calls, or that device plugin API has removed an old API call. However, everything they share in common (most APIs) will work.

Kubernetes API health endpoints

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes API server provides API endpoints to indicate the current status of the API server. This page describes these API endpoints and explains how you can use them.

API endpoints for health

The Kubernetes API server provides 3 API endpoints (healthz, livez and readyz) to indicate the current status of the API server. The healthz endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific livez and readyz endpoints instead. The livez endpoint can be used with the --livez-grace-period flag to specify the startup duration. For a graceful shutdown you can specify the --shutdown-delay-duration flag with the /readyz endpoint. Machines that check the healthz/livez/readyz of the API server should rely on the HTTP status code. A status code 200 indicates the API server is healthy/live/ready, depending on the called endpoint.

Kubernetes Self-Healing

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes is designed with self-healing capabilities that help maintain the health and availability of workloads. It automatically replaces failed containers, reschedules workloads when nodes become unavailable, and ensures that the desired state of the system is maintained.

Self-Healing capabilities

Container-level restarts: If a container inside a Pod fails, Kubernetes restarts it based on the restartPolicy.
Replica replacement: If a Pod in a Deployment or StatefulSet fails, Kubernetes creates a replacement Pod to maintain the specified number of replicas. If a Pod that is part of a DaemonSet fails, the control plane creates a replacement Pod to run on the same node.

Localizing Kubernetes documentation

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to localize the docs for a different language.

Contribute to an existing localization

You can help add or improve the content of an existing localization. In Kubernetes Slack, you can find a channel for each localization. There is also a general SIG Docs Localizations Slack channel where you can say hello.

Note:

For extra details on how to contribute to a specific localization, look for a localized version of this page.

Find your two-letter language code

First, consult the ISO 639-1 standard to find your localization's two-letter language code. For example, the two-letter code for Korean is ko.

Managing Service Accounts

Mon, 01 Jan 0001 00:00:00 +0000

A ServiceAccount provides an identity for processes that run in a Pod.

A process inside a Pod can use the identity of its associated service account to authenticate to the cluster's API server.

For an introduction to service accounts, read configure service accounts.

This task guide explains some of the concepts behind ServiceAccounts. The guide also explains how to obtain or revoke tokens that represent ServiceAccounts, and how to (optionally) bind a ServiceAccount's validity to the lifetime of an API object.

Node metrics data

Mon, 01 Jan 0001 00:00:00 +0000

The kubelet gathers metric statistics at the node, volume, pod and container level, and emits this information in the Summary API.

You can send a proxied request to the stats summary API via the Kubernetes API server.

Here is an example of a Summary API request for a node named minikube:

kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary"

Here is the same API call using curl:

# You need to run "kubectl proxy" first
# Change 8080 to the port that "kubectl proxy" assigns
curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary

Note:

Beginning with metrics-server 0.6.x, metrics-server queries the /metrics/resource kubelet endpoint, and not /stats/summary.

Summary metrics API source

By default, Kubernetes fetches node summary metrics data using an embedded cAdvisor that runs within the kubelet. If you enable the PodAndContainerStatsFromCRI feature gate in your cluster, and you use a container runtime that supports statistics access via Container Runtime Interface (CRI), then the kubelet fetches Pod- and container-level metric data using CRI, and not via cAdvisor.

Node Resource Managers

Mon, 01 Jan 0001 00:00:00 +0000

In order to support latency-critical and high-throughput workloads, Kubernetes offers a suite of Resource Managers. The managers aim to co-ordinate and optimise the alignment of node's resources for pods configured with a specific requirement for CPUs, devices, and memory (hugepages) resources.

Hardware topology alignment policies

Topology Manager is a kubelet component that aims to coordinate the set of components that are responsible for these optimizations. The overall resource management process is governed using the policy you specify. To learn more, read Control Topology Management Policies on a Node.

Options for Highly Available Topology

Mon, 01 Jan 0001 00:00:00 +0000

This page explains the two options for configuring the topology of your highly available (HA) Kubernetes clusters.

You can set up an HA cluster:

With stacked control plane nodes, where etcd nodes are colocated with control plane nodes
With external etcd nodes, where etcd runs on separate nodes from the control plane

You should carefully consider the advantages and disadvantages of each topology before setting up an HA cluster.

Parallel Processing using Expansions

Mon, 01 Jan 0001 00:00:00 +0000

This task demonstrates running multiple Jobs based on a common template. You can use this approach to process batches of work in parallel.

For this example there are only three items: apple, banana, and cherry. The sample Jobs process each item by printing a string then pausing.

See using Jobs in real workloads to learn about how this pattern fits more realistic use cases.

PKI certificates and requirements

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes requires PKI certificates for authentication over TLS. If you install Kubernetes with kubeadm, the certificates that your cluster requires are automatically generated. You can also generate your own certificates -- for example, to keep your private keys more secure by not storing them on the API server. This page explains the certificates that your cluster requires.

How certificates are used by your cluster

Kubernetes requires PKI for the following operations:

Romana for NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use Romana for NetworkPolicy.

Before you begin

Complete steps 1, 2, and 3 of the kubeadm getting started guide.

Installing Romana with kubeadm

Follow the containerized installation guide for kubeadm.

Applying network policies

To apply network policies use one of the following:

Romana network policies.
- Example of Romana network policy.
The NetworkPolicy API.

What's next

Once you have installed Romana, you can follow the Declare Network Policy to try out Kubernetes NetworkPolicy.

Scale a StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

This task shows how to scale a StatefulSet. Scaling a StatefulSet refers to increasing or decreasing the number of replicas.

Before you begin

StatefulSets are only available in Kubernetes version 1.5 or later. To check your version of Kubernetes, run kubectl version.
Not all stateful applications scale nicely. If you are unsure about whether to scale your StatefulSets, see StatefulSet concepts or StatefulSet tutorial for further information.
You should perform scaling only when you are confident that your stateful application cluster is completely healthy.

Sidecar Containers

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.33 [stable](enabled by default)

Sidecar containers are the secondary containers that run along with the main application container within the same Pod. These containers are used to enhance or to extend the functionality of the primary app container by providing additional services, or functionality such as logging, monitoring, security, or data synchronization, without directly altering the primary application code.

Taints and Tolerations

Mon, 01 Jan 0001 00:00:00 +0000

Node affinity is a property of Pods that attracts them to a set of nodes (either as a preference or a hard requirement). Taints are the opposite -- they allow a node to repel a set of pods.

Tolerations are applied to pods. Tolerations allow the scheduler to schedule pods with matching taints. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also evaluates other parameters as part of its function.

Understand Pressure Stall Information (PSI) Metrics

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.34 [beta]

As a beta feature, Kubernetes lets you configure the kubelet to collect Linux kernel Pressure Stall Information (PSI) for CPU, memory, and I/O usage. The information is collected at node, pod and container level. This feature is enabled by default by setting the KubeletPSI feature gate.

PSI metrics are exposed through two different sources:

The kubelet's Summary API, which provides PSI data at the node, pod, and container level.
The /metrics/cadvisor endpoint on the kubelet, which exposes PSI metrics in the Prometheus format.

Requirements

Pressure Stall Information requires the following on your Linux nodes:

Update API Objects in Place Using kubectl patch

Mon, 01 Jan 0001 00:00:00 +0000

This task shows how to use kubectl patch to update an API object in place. The exercises in this task demonstrate a strategic merge patch and a JSON merge patch.

Before you begin

User Impersonation

Mon, 01 Jan 0001 00:00:00 +0000

User impersonation is a method of allowing authenticated users to act as another user, group, or service account through HTTP headers.

A user can act as another user through impersonation headers. These let requests manually override the user info a request authenticates as. For example, an admin could use this feature to debug an authorization policy by temporarily impersonating another user and seeing if a request was denied.

Virtual IPs and Service Proxies

Mon, 01 Jan 0001 00:00:00 +0000

Every node in a Kubernetes cluster runs a kube-proxy (unless you have deployed your own alternative component in place of kube-proxy).

The kube-proxy component is responsible for implementing a virtual IP mechanism for Services of type other than ExternalName. Each instance of kube-proxy watches the Kubernetes control plane for the addition and removal of Service and EndpointSlice objects. For each Service, kube-proxy calls appropriate APIs (depending on the kube-proxy mode) to configure the node to capture traffic to the Service's clusterIP and port, and redirect that traffic to one of the Service's endpoints (usually a Pod, but possibly an arbitrary user-provided IP address). A control loop ensures that the rules on each node are reliably synchronized with the Service and EndpointSlice state as indicated by the API server.

Gateway API

Mon, 01 Jan 0001 00:00:00 +0000

Make network services available by using an extensible, role-oriented, protocol-aware configuration mechanism. Gateway API is an add-on containing API kinds that provide dynamic infrastructure provisioning and advanced traffic routing.

Design principles

The following principles shaped the design and architecture of Gateway API:

Role-oriented: Gateway API kinds are modeled after organizational roles that are responsible for managing Kubernetes service networking:
- Infrastructure Provider: Manages infrastructure that allows multiple isolated clusters to serve multiple tenants, e.g. a cloud provider.
- Cluster Operator: Manages clusters and is typically concerned with policies, network access, application permissions, etc.
- Application Developer: Manages an application running in a cluster and is typically concerned with application-level configuration and Service composition.
Portable: Gateway API specifications are defined as custom resources and are supported by many implementations.
Expressive: Gateway API kinds support functionality for common traffic routing use cases such as header-based matching, traffic weighting, and others that were only possible in Ingress by using custom annotations.
Extensible: Gateway allows for custom resources to be linked at various layers of the API. This makes granular customization possible at the appropriate places within the API structure.

Resource model

Gateway API has four stable API kinds:

Observability

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, observability is the process of collecting and analyzing metrics, logs, and traces—often referred to as the three pillars of observability—in order to obtain a better understanding of the internal state, performance, and health of the cluster.

Kubernetes control plane components, as well as many add-ons, generate and emit these signals. By aggregating and correlating them, you can gain a unified picture of the control plane, add-ons, and applications across the cluster.

Access Clusters Using the Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to access clusters using the Kubernetes API.

Before you begin

Adding entries to Pod /etc/hosts with HostAliases

Mon, 01 Jan 0001 00:00:00 +0000

Adding entries to a Pod's /etc/hosts file provides Pod-level override of hostname resolution when DNS and other options are not applicable. You can add these custom entries with the HostAliases field in PodSpec.

The Kubernetes project recommends modifying DNS configuration using the hostAliases field (part of the .spec for a Pod), and not by using an init container or other means to edit /etc/hosts directly. Change made in other ways may be overwritten by the kubelet during Pod creation or restart.

Admission Webhook Good Practices

Mon, 01 Jan 0001 00:00:00 +0000

This page provides good practices and considerations when designing admission webhooks in Kubernetes. This information is intended for cluster operators who run admission webhook servers or third-party applications that modify or validate your API requests.

Before reading this page, ensure that you're familiar with the following concepts:

Importance of good webhook design

Admission control occurs when any create, update, or delete request is sent to the Kubernetes API. Admission controllers intercept requests that match specific criteria that you define. These requests are then sent to mutating admission webhooks or validating admission webhooks. These webhooks are often written to ensure that specific fields in object specifications exist or have specific allowed values.

Annotations

Mon, 01 Jan 0001 00:00:00 +0000

You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.

Attaching metadata to objects

You can use either labels or annotations to attach metadata to Kubernetes objects. Labels can be used to select objects and to find collections of objects that satisfy certain conditions. In contrast, annotations are not used to identify and select objects. The metadata in an annotation can be small or large, structured or unstructured, and can include characters not permitted by labels. It is possible to use labels as well as annotations in the metadata of the same object.

Certificates and Certificate Signing Requests

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes certificate and trust bundle APIs enable automation of X.509 credential provisioning by providing a programmatic interface for clients of the Kubernetes API to request and obtain X.509 certificates from a Certificate Authority (CA).

There is also experimental (alpha) support for distributing trust bundles.

Certificate signing requests

FEATURE STATE: Kubernetes v1.19 [stable]

A CertificateSigningRequest (CSR) resource is used to request that a certificate be signed by a denoted signer, after which the request may be approved or denied before finally being signed.

Configure a Pod Quota for a Namespace

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to set a quota for the total number of Pods that can run in a Namespace. You specify quotas in a ResourceQuota object.

Before you begin

Configure Quality of Service for Pods

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure Pods so that they will be assigned particular Quality of Service (QoS) classes. Kubernetes uses QoS classes to make decisions about evicting Pods when Node resources are exceeded.

When Kubernetes creates a Pod it assigns one of these QoS classes to the Pod:

Container Runtime Interface (CRI)

Mon, 01 Jan 0001 00:00:00 +0000

The CRI is a plugin interface which enables the kubelet to use a wide variety of container runtimes, without having a need to recompile the cluster components.

You need a working container runtime on each Node in your cluster, so that the kubelet can launch Pods and their containers.

Creating Highly Available Clusters with kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

This page explains two different approaches to setting up a highly available Kubernetes cluster using kubeadm:

With stacked control plane nodes. This approach requires less infrastructure. The etcd members and control plane nodes are co-located.
With an external etcd cluster. This approach requires more infrastructure. The control plane nodes and etcd members are separated.

Before proceeding, you should carefully consider which approach best meets the needs of your applications and environment. Options for Highly Available topology outlines the advantages and disadvantages of each.

Delete a StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

This task shows you how to delete a StatefulSet.

Before you begin

This task assumes you have an application running on your cluster represented by a StatefulSet.

Deleting a StatefulSet

You can delete a StatefulSet in the same way you delete other resources in Kubernetes: use the kubectl delete command, and specify the StatefulSet either by file or by name.

Diagram Guide

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to create, edit and share diagrams using the Mermaid JavaScript library. Mermaid.js allows you to generate diagrams using a simple markdown-like syntax inside Markdown files. You can also use Mermaid to generate .svg or .png image files that you can add to your documentation.

The target audience for this guide is anybody wishing to learn about Mermaid and/or how to create and add diagrams to Kubernetes documentation.

Enable Or Disable Feature Gates

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to enable or disable feature gates to control specific Kubernetes features in your cluster. Enabling feature gates allows you to test and use Alpha or Beta features before they become generally available.

Note:

For some stable (GA) gates, you can also disable them, usually for one minor release after GA; however if you do that, your cluster may not be conformant as Kubernetes.

Before you begin

EndpointSlices

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.21 [stable]

EndpointSlices track the IP addresses of backend endpoints. EndpointSlices are normally associated with a and the backend endpoints typically represent .

EndpointSlice API

In Kubernetes, an EndpointSlice contains references to a set of network endpoints. The control plane automatically creates EndpointSlices for any Kubernetes Service that has a selector specified. These EndpointSlices include references to all the Pods that match the Service selector. EndpointSlices group network endpoints together by unique combinations of IP family, protocol, port number, and Service name. The name of a EndpointSlice object must be a valid DNS subdomain name.

Ephemeral Containers

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.25 [stable]

This page provides an overview of ephemeral containers: a special type of container that runs temporarily in an existing Pod to accomplish user-initiated actions such as troubleshooting. You use ephemeral containers to inspect services rather than to build applications.

Understanding ephemeral containers

Pods are the fundamental building block of Kubernetes applications. Since Pods are intended to be disposable and replaceable, you cannot add a container to a Pod once it has been created. Instead, you usually delete and replace Pods in a controlled fashion using deployments.

Explore Termination Behavior for Pods And Their Endpoints

Mon, 01 Jan 0001 00:00:00 +0000

Once you connected your Application with Service following steps like those outlined in Connecting Applications with Services, you have a continuously running, replicated application, that is exposed on a network. This tutorial helps you look at the termination flow for Pods and to explore ways to implement graceful connection draining.

Termination process for Pods and their endpoints

There are often cases when you need to terminate a Pod - be it to upgrade or scale down. In order to improve application availability, it may be important to implement a proper active connections draining.

Good practices for Dynamic Resource Allocation as a Cluster Admin

Mon, 01 Jan 0001 00:00:00 +0000

This page describes good practices when configuring a Kubernetes cluster utilizing Dynamic Resource Allocation (DRA). These instructions are for cluster administrators.

DRA is orchestrated through a number of different APIs. Use authorization tools (like RBAC, or another solution) to control access to the right APIs depending on the persona of your user.

In general, DeviceClasses and ResourceSlices should be restricted to admins and the DRA drivers. Cluster operators that will be deploying Pods with claims will need access to ResourceClaim and ResourceClaimTemplate APIs; both of these APIs are namespace scoped.

Handling retriable and non-retriable pod failures with Pod failure policy

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.31 [stable](enabled by default)

This document shows you how to use the Pod failure policy, in combination with the default Pod backoff failure policy, to improve the control over the handling of container- or Pod-level failure within a Job.

The definition of Pod failure policy may help you to:

better utilize the computational resources by avoiding unnecessary Pod retries.
avoid Job failures due to Pod disruptions (such preemption, API-initiated eviction or taint-based eviction).

Before you begin

You should already be familiar with the basic use of Job.

Horizontal Pod Autoscaling

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, a HorizontalPodAutoscaler automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling capacity to match demand.

Horizontal scaling means that the response to increased load is to deploy more Pods. This is different from vertical scaling, which for Kubernetes would mean assigning more resources (for example: memory or CPU) to the Pods that are already running for the workload.

Install Drivers and Allocate Devices with DRA

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [stable](enabled by default)

This tutorial shows you how to install Dynamic Resource Allocation (DRA) drivers in your cluster and how to use them in conjunction with the DRA APIs to allocate devices to Pods. This page is intended for cluster administrators.

kubeadm reset

Mon, 01 Jan 0001 00:00:00 +0000

Performs a best effort revert of changes made by kubeadm init or kubeadm join.

Synopsis

Performs a best effort revert of changes made to this host by 'kubeadm init' or 'kubeadm join'

kubectl Usage Conventions

Mon, 01 Jan 0001 00:00:00 +0000

Recommended usage conventions for kubectl.

Using `kubectl` in Reusable Scripts

For a stable output in a script:

Request one of the machine-oriented output forms, such as -o name, -o json, -o yaml, -o go-template, or -o jsonpath.
Fully-qualify the version. For example, jobs.v1.batch/myjob. This will ensure that kubectl does not use its default version that can change over time.
Don't rely on context, preferences, or other implicit states.

Subresources

You can use the --subresource argument for kubectl subcommands such as get, patch, edit, apply and replace to fetch and update subresources for all resources that support them. In Kubernetes version 1.35, only the status, scale and resize subresources are supported.
- For kubectl edit, the scale subresource is not supported. If you use --subresource with kubectl edit and specify scale as the subresource, the command will error out.
The API contract against a subresource is identical to a full resource. While updating the status subresource to a new value, keep in mind that the subresource could be potentially reconciled by a controller to a different value.

Best Practices

`kubectl run`

For kubectl run to satisfy infrastructure as code:

Kubernetes z-pages

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.32 [alpha]

Kubernetes core components can expose a suite of z-endpoints to make it easier for users to debug their cluster and its components. These endpoints are strictly to be used for human inspection to gain real time debugging information of a component binary. Avoid automated scraping of data returned by these endpoints; in Kubernetes 1.35 these are an alpha feature and the response format may change in future releases.

Logging Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Application logs can help you understand what is happening inside your application. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams.

However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution.

Migrate Kubernetes Objects Using Storage Version Migration

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [beta](disabled by default)

Kubernetes relies on API data being actively re-written, to support some maintenance activities related to at rest storage. Two prominent examples are the versioned schema of stored resources (that is, the preferred storage schema changing from v1 to v2 for a given resource) and encryption at rest (that is, rewriting stale data based on a change in how the data should be encrypted).

Migrating telemetry and security agents from dockershim

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes' support for direct integration with Docker Engine is deprecated and has been removed. Most apps do not have a direct dependency on runtime hosting containers. However, there are still a lot of telemetry and monitoring agents that have a dependency on Docker to collect containers metadata, logs, and metrics. This document aggregates information on how to detect these dependencies as well as links on how to migrate these agents to use generic tools or alternative runtimes.

Organizing Cluster Access Using kubeconfig Files

Mon, 01 Jan 0001 00:00:00 +0000

Use kubeconfig files to organize information about clusters, users, namespaces, and authentication mechanisms. The kubectl command-line tool uses kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster.

Note:

A file that is used to configure access to clusters is called a kubeconfig file. This is a generic way of referring to configuration files. It does not mean that there is a file named kubeconfig.

Warning:

By default, kubectl looks for a file named config in the $HOME/.kube directory. You can specify other kubeconfig files by setting the KUBECONFIG environment variable or by setting the --kubeconfig flag.

Post-release communications

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes Release Comms team (part of SIG Release) looks after release announcements, which go onto the main project blog.

After each release, the Release Comms team take over the main blog for a period and publish a series of additional articles to explain or announce changes related to that release. These additional articles are termed post-release comms.

Opting in to post-release comms

During a release cycle, as a contributor, you can opt in to post-release comms about an upcoming change to Kubernetes.

Role Based Access Control Good Practices

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes RBAC is a key security control to ensure that cluster users and workloads have only the access to resources required to execute their roles. It is important to ensure that, when designing permissions for cluster users, the cluster administrator understands the areas where privilege escalation could occur, to reduce the risk of excessive access leading to security incidents.

Scheduling Framework

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.19 [stable]

The scheduling framework is a pluggable architecture for the Kubernetes scheduler. It consists of a set of "plugin" APIs that are compiled directly into the scheduler. These APIs allow most scheduling features to be implemented as plugins, while keeping the scheduling "core" lightweight and maintainable. Refer to the design proposal of the scheduling framework for more technical information on the design of the framework.

Troubleshooting Topology Management

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes keeps many aspects of how pods execute on nodes abstracted from the user. This is by design. However, some workloads require stronger guarantees in terms of latency and/or performance in order to operate acceptably. The kubelet provides methods to enable more complex workload placement policies while keeping the abstraction free from explicit placement directives.

You can manage topology within nodes. This means helping the kubelet to configure the host operating system so that Pods and containers are placed on the correct side of inner boundaries, such as NUMA domains. (NUMA is an abbreviation of non-uniform memory access, and refers to an idea that CPUs might be topologically closer to specific regions of memory, due to the physical layout of the hardware components and the way that these are connected).

Use a Service to Access an Application in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to create a Kubernetes Service object that external clients can use to access an application running in a cluster. The Service provides load balancing for an application that has two running instances.

Before you begin

Volume Snapshots

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, a VolumeSnapshot represents a snapshot of a volume on a storage system. This document assumes that you are already familiar with Kubernetes persistent volumes.

Introduction

Similar to how API resources PersistentVolume and PersistentVolumeClaim are used to provision volumes for users and administrators, VolumeSnapshotContent and VolumeSnapshot API resources are provided to create volume snapshots for users and administrators.

A VolumeSnapshotContent is a snapshot taken from a volume in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a PersistentVolume is a cluster resource.

Weave Net for NetworkPolicy

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use Weave Net for NetworkPolicy.

Before you begin

You need to have a Kubernetes cluster. Follow the kubeadm getting started guide to bootstrap one.

Install the Weave Net addon

Follow the Integrating Kubernetes via the Addon guide.

The Weave Net addon for Kubernetes comes with a Network Policy Controller that automatically monitors Kubernetes for any NetworkPolicy annotations on all namespaces and configures iptables rules to allow or block traffic as directed by the policies.

Volume Snapshot Classes

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the concept of VolumeSnapshotClass in Kubernetes. Familiarity with volume snapshots and storage classes is suggested.

Introduction

Just like StorageClass provides a way for administrators to describe the "classes" of storage they offer when provisioning a volume, VolumeSnapshotClass provides a way to describe the "classes" of storage when provisioning a volume snapshot.

The VolumeSnapshotClass Resource

Each VolumeSnapshotClass contains the fields driver, deletionPolicy, and parameters, which are used when a VolumeSnapshot belonging to the class needs to be dynamically provisioned.

Dynamic Resource Allocation

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [stable](enabled by default)

This page describes dynamic resource allocation (DRA) in Kubernetes.

About DRA

DRA is a Kubernetes feature that lets you request and share resources among Pods. These resources are often attached devices like hardware accelerators.

With DRA, device drivers and cluster admins define device classes that are available to claim in workloads. Kubernetes allocates matching devices to specific claims and places the corresponding Pods on nodes that can access the allocated devices.

Windows containers in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Windows applications constitute a large portion of the services and applications that run in many organizations. Windows containers provide a way to encapsulate processes and package dependencies, making it easier to use DevOps practices and follow cloud native patterns for Windows applications.

Organizations with investments in Windows-based applications and Linux-based applications don't have to look for separate orchestrators to manage their workloads, leading to increased operational efficiencies across their deployments, regardless of operating system.

Advertise Extended Resources for a Node

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to specify extended resources for a Node. Extended resources allow cluster administrators to advertise node-level resources that would otherwise be unknown to Kubernetes.

Before you begin

Assign Extended Resources to a Container

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [stable]

This page shows how to assign extended resources to a Container.

Before you begin

Automatic Cleanup for Finished Jobs

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.23 [stable]

When your Job has finished, it's useful to keep that Job in the API (and not immediately delete the Job) so that you can tell whether the Job succeeded or failed.

Kubernetes' TTL-after-finished controller provides a TTL (time to live) mechanism to limit the lifetime of Job objects that have finished execution.

Compatibility Version For Kubernetes Control Plane Components

Mon, 01 Jan 0001 00:00:00 +0000

Since release v1.32, we introduced configurable version compatibility and emulation options to Kubernetes control plane components to make upgrades safer by providing more control and increasing the granularity of steps available to cluster administrators.

Emulated Version

The emulation option is set by the --emulated-version flag of control plane components. It allows the component to emulate the behavior (APIs, features, ...) of an earlier version of Kubernetes.

Connect a Frontend to a Backend Using Services

Mon, 01 Jan 0001 00:00:00 +0000

This task shows how to create a frontend and a backend microservice. The backend microservice is a hello greeter. The frontend exposes the backend using nginx and a Kubernetes Service object.

Objectives

Create and run a sample hello backend microservice using a Deployment object.
Use a Service object to send traffic to the backend microservice's multiple replicas.
Create and run a nginx frontend microservice, also using a Deployment object.
Configure the frontend microservice to send traffic to the backend microservice.
Use a Service object of type=LoadBalancer to expose the frontend microservice outside the cluster.

Before you begin

CSI Volume Cloning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the concept of cloning existing CSI Volumes in Kubernetes. Familiarity with Volumes is suggested.

Introduction

The CSI Volume Cloning feature adds support for specifying existing PVCs in the dataSource field to indicate a user would like to clone a Volume.

Disruptions

Mon, 01 Jan 0001 00:00:00 +0000

This guide is for application owners who want to build highly available applications, and thus need to understand what types of disruptions can happen to Pods.

It is also for cluster administrators who want to perform automated cluster actions, like upgrading and autoscaling clusters.

Voluntary and involuntary disruptions

Pods do not disappear until someone (a person or a controller) destroys them, or there is an unavoidable hardware or system software error.

Field Selectors

Mon, 01 Jan 0001 00:00:00 +0000

Field selectors let you select Kubernetes objects based on the value of one or more resource fields. Here are some examples of field selector queries:

metadata.name=my-service
metadata.namespace!=default
status.phase=Pending

This kubectl command selects all Pods for which the value of the status.phase field is Running:

kubectl get pods --field-selector status.phase=Running

Note:

Field selectors are essentially resource filters. By default, no selectors/filters are applied, meaning that all resources of the specified type are selected. This makes the kubectl queries kubectl get pods and kubectl get pods --field-selector "" equivalent.

Supported fields

Supported field selectors vary by Kubernetes resource type. All resource types support the metadata.name and metadata.namespace fields. Using unsupported field selectors produces an error. For example:

Force Delete StatefulSet Pods

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to delete Pods which are part of a stateful set, and explains the considerations to keep in mind when doing so.

Before you begin

This is a fairly advanced task and has the potential to violate some of the properties inherent to StatefulSet.
Before proceeding, make yourself familiar with the considerations enumerated below.

StatefulSet considerations

In normal operation of a StatefulSet, there is never a need to force delete a StatefulSet Pod. The StatefulSet controller is responsible for creating, scaling and deleting members of the StatefulSet. It tries to ensure that the specified number of Pods from ordinal 0 through N-1 are alive and ready. StatefulSet ensures that, at any time, there is at most one Pod with a given identity running in a cluster. This is referred to as at most one semantics provided by a StatefulSet.

Gang Scheduling

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [alpha](disabled by default)

Gang scheduling ensures that a group of Pods are scheduled on an "all-or-nothing" basis. If the cluster cannot accommodate the entire group (or a defined minimum number of Pods), none of the Pods are bound to a node.

This feature depends on the Workload API. Ensure the GenericWorkload feature gate and the scheduling.k8s.io/v1alpha1 API group are enabled in the cluster.

Garbage Collection

Mon, 01 Jan 0001 00:00:00 +0000

Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up cluster resources. This allows the clean up of resources like the following:

Terminated pods
Completed Jobs
Objects without owner references
Unused containers and container images
Dynamically provisioned PersistentVolumes with a StorageClass reclaim policy of Delete
Stale or expired CertificateSigningRequests (CSRs)
Nodes deleted in the following scenarios:
- On a cloud when the cluster uses a cloud controller manager
- On-premises when the cluster uses an addon similar to a cloud controller manager
Node Lease objects

Owners and dependents

Many objects in Kubernetes link to each other through owner references. Owner references tell the control plane which objects are dependent on others. Kubernetes uses owner references to give the control plane, and other API clients, the opportunity to clean up related resources before deleting an object. In most cases, Kubernetes manages owner references automatically.

Good practices for Kubernetes Secrets

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, a Secret is an object that stores sensitive information, such as passwords, OAuth tokens, and SSH keys.

Secrets give you more control over how sensitive information is used and reduces the risk of accidental exposure. Secret values are encoded as base64 strings and are stored unencrypted by default, but can be configured to be encrypted at rest.

A Pod can reference the Secret in a variety of ways, such as in a volume mount or as an environment variable. Secrets are designed for confidential data and ConfigMaps are designed for non-confidential data.

Helping as a blog writing buddy

Mon, 01 Jan 0001 00:00:00 +0000

There are two official Kubernetes blogs, and the CNCF has its own blog where you can cover Kubernetes too. Read contributing to Kubernetes blogs to learn about these two blogs.

When people contribute to either blog as an author, the Kubernetes project pairs up authors as writing buddies. This page explains how to fulfil the buddy role.

You should make sure that you have at least read an outline of article submission before you read on within this page.

kubeadm token

Mon, 01 Jan 0001 00:00:00 +0000

Bootstrap tokens are used for establishing bidirectional trust between a node joining the cluster and a control-plane node, as described in authenticating with bootstrap tokens.

kubeadm init creates an initial token with a 24-hour TTL. The following commands allow you to manage such a token and also to create and manage new ones.

kubeadm token create

Create bootstrap tokens on the server

Kubectl user preferences (kuberc)

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes 1.34 [beta]

A Kubernetes kuberc configuration file allows you to define preferences for kubectl, such as default options and command aliases. Unlike the kubeconfig file, a kuberc configuration file does not contain cluster details, usernames or passwords.

On Linux / POSIX computers, the default location of this configuration file is $HOME/.kube/kuberc. The default path on Windows is similar: %USERPROFILE%\.kube\kuberc. To provide kubectl with a path to a custom kuberc file, use the --kuberc command line option, or set the KUBERC environment variable.

Metrics For Kubernetes System Components

Mon, 01 Jan 0001 00:00:00 +0000

System component metrics can give a better look into what is happening inside them. Metrics are particularly useful for building dashboards and alerts.

Kubernetes components emit metrics in Prometheus format. This format is structured plain text, designed so that people and machines can both read it.

Metrics in Kubernetes

In most cases metrics are available on /metrics endpoint of the HTTP server. For components that don't expose endpoint by default, it can be enabled using --bind-address flag.

Network Policies

Mon, 01 Jan 0001 00:00:00 +0000

If you want to control traffic flow at the IP address or port level for TCP, UDP, and SCTP protocols, then you might consider using Kubernetes NetworkPolicies for particular applications in your cluster. NetworkPolicies are an application-centric construct which allow you to specify how a pod is allowed to communicate with various network "entities" (we use the word "entity" here to avoid overloading the more common terms such as "endpoints" and "services", which have specific Kubernetes connotations) over the network. NetworkPolicies apply to a connection with a pod on one or both ends, and are not relevant to other connections.

Scheduler Performance Tuning

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.14 [beta]

kube-scheduler is the Kubernetes default scheduler. It is responsible for placement of Pods on Nodes in a cluster.

Nodes in a cluster that meet the scheduling requirements of a Pod are called feasible Nodes for the Pod. The scheduler finds feasible Nodes for a Pod and then runs a set of functions to score the feasible Nodes, picking a Node with the highest score among the feasible ones to run the Pod. The scheduler then notifies the API server about this decision in a process called Binding.

Set up a High Availability etcd Cluster with kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

By default, kubeadm runs a local etcd instance on each control plane node. It is also possible to treat the etcd cluster as external and provision etcd instances on separate hosts. The differences between the two approaches are covered in the Options for Highly Available topology page.

This task walks through the process of creating a high availability external etcd cluster of three members that can be used by kubeadm during cluster creation.

Set up Konnectivity service

Mon, 01 Jan 0001 00:00:00 +0000

The Konnectivity service provides a TCP level proxy for the control plane to cluster communication.

Before you begin

Vertical Pod Autoscaling

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, a VerticalPodAutoscaler automatically updates a workload management resource (such as a Deployment or StatefulSet), with the aim of automatically adjusting infrastructure resource requests and limits to match actual usage.

Writing a new topic

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to create a new topic for the Kubernetes docs.

Before you begin

Create a fork of the Kubernetes documentation repository as described in Open a PR.

Choosing a page type

As you prepare to write a new topic, think about the page type that would fit your content the best:

Guidelines for choosing a page type
Type	Description
Concept	A concept page explains some aspect of Kubernetes. For example, a concept page might describe the Kubernetes Deployment object and explain the role it plays as an application while it is deployed, scaled, and updated. Typically, concept pages don't include sequences of steps, but instead provide links to tasks or tutorials. For an example of a concept topic, see Nodes.
Task	A task page shows how to do a single thing. The idea is to give readers a sequence of steps that they can actually do as they read the page. A task page can be short or long, provided it stays focused on one area. In a task page, it is OK to blend brief explanations with the steps to be performed, but if you need to provide a lengthy explanation, you should do that in a concept topic. Related task and concept topics should link to each other. For an example of a short task page, see Configure a Pod to Use a Volume for Storage. For an example of a longer task page, see Configure Liveness and Readiness Probes
Tutorial	A tutorial page shows how to accomplish a goal that ties together several Kubernetes features. A tutorial might provide several sequences of steps that readers can actually do as they read the page. Or it might provide explanations of related pieces of code. For example, a tutorial could provide a walkthrough of a code sample. A tutorial can include brief explanations of the Kubernetes features that are being tied together, but should link to related concept topics for deep explanations of individual features.

Creating a new page

Use a content type for each new page that you write. The docs site provides templates or Hugo archetypes to create new content pages. To create a new type of page, run hugo new with the path to the file you want to create. For example:

Guide for Running Windows Containers in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

This page provides a walkthrough for some steps you can follow to run Windows containers using Kubernetes. The page also highlights some Windows specific functionality within Kubernetes.

It is important to note that creating and deploying services and workloads on Kubernetes behaves in much the same way for Linux and Windows containers. The kubectl commands to interface with the cluster are identical. The examples in this page are provided to jumpstart your experience with Windows containers.

Metrics for Kubernetes Object States

Mon, 01 Jan 0001 00:00:00 +0000

The state of Kubernetes objects in the Kubernetes API can be exposed as metrics. An add-on agent called kube-state-metrics can connect to the Kubernetes API server and expose a HTTP endpoint with metrics generated from the state of individual objects in the cluster. It exposes various information about the state of objects like labels and annotations, startup and termination times, status or the phase the object currently is in. For example, containers running in pods create a kube_pod_container_info metric. This includes the name of the container, the name of the pod it is part of, the namespace the pod is running in, the name of the container image, the ID of the image, the image name from the spec of the container, the ID of the running container and the ID of the pod as labels.

Resource Management for Windows nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page outlines the differences in how resources are managed between Linux and Windows.

On Linux nodes, cgroups are used as a pod boundary for resource control. Containers are created within that boundary for network, process and file system isolation. The Linux cgroup APIs can be used to gather CPU, I/O, and memory use statistics.

Autoscale the DNS Service in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to enable and configure autoscaling of the DNS service in your Kubernetes cluster.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Certificate Management with kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.15 [stable]

Client certificates generated by kubeadm expire after 1 year. This page explains how to manage certificate renewals with kubeadm. It also covers other tasks related to kubeadm certificate management.

Before you begin

You should be familiar with PKI certificates and requirements in Kubernetes.

Configure a Pod to Use a Volume for Storage

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure a Pod to use a Volume for storage.

A Container's file system lives only as long as the Container does. So when a Container terminates and restarts, filesystem changes are lost. For more consistent storage that is independent of the Container, you can use a Volume. This is especially important for stateful applications, such as key-value stores (such as Redis) and databases.

Configuring each kubelet in your cluster using kubeadm

Mon, 01 Jan 0001 00:00:00 +0000

Note: Dockershim has been removed from the Kubernetes project as of release 1.24. Read the Dockershim Removal FAQ for further details.

FEATURE STATE: Kubernetes v1.11 [stable]

The lifecycle of the kubeadm CLI tool is decoupled from the kubelet, which is a daemon that runs on each node within the Kubernetes cluster. The kubeadm CLI tool is executed by the user when Kubernetes is initialized or upgraded, whereas the kubelet is always running in the background.

Create an External Load Balancer

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to create an external load balancer.

When creating a Service, you have the option of automatically creating a cloud load balancer. This provides an externally-accessible IP address that sends traffic to the correct port on your cluster nodes, provided your cluster runs in a supported environment and is configured with the correct cloud load balancer provider package.

CronJob

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.21 [stable]

A CronJob creates Jobs on a repeating schedule.

CronJob is meant for performing regular scheduled actions such as backups, report generation, and so on. One CronJob object is like one line of a crontab (cron table) file on a Unix system. It runs a Job periodically on a given schedule, written in Cron format.

DNS for Services and Pods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes creates DNS records for Services and Pods. You can contact Services with consistent DNS names instead of IP addresses.

Kubernetes publishes information about Pods and Services which is used to program DNS. kubelet configures Pods' DNS so that running containers can look up Services by name rather than IP.

Services defined in the cluster are assigned DNS names. By default, a client Pod's DNS search list includes the Pod's own namespace and the cluster's default domain.

Finalizers

Mon, 01 Jan 0001 00:00:00 +0000

Finalizers are namespaced keys that tell Kubernetes to wait until specific conditions are met before it fully deletes resources that are marked for deletion. Finalizers alert controllers to clean up resources the deleted object owned.

Issue a Certificate for a Kubernetes API Client Using A CertificateSigningRequest

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes lets you use a public key infrastructure (PKI) to authenticate to your cluster as a client.

A few steps are required in order to get a normal user to be able to authenticate and invoke an API. First, this user must have an X.509 certificate issued by an authority that your Kubernetes cluster trusts. The client must then present that certificate to the Kubernetes API.

You use a CertificateSigningRequest as part of this process, and either you or some other principal must approve the request.

kubeadm version

Mon, 01 Jan 0001 00:00:00 +0000

This command prints the version of kubeadm.

Synopsis

Print the version of kubeadm

Kubelet Systemd Watchdog

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.32 [beta](enabled by default)

On Linux nodes, Kubernetes 1.35 supports integrating with systemd to allow the operating system supervisor to recover a failed kubelet. This integration is not enabled by default. It can be used as an alternative to periodically requesting the kubelet's /healthz endpoint for health checks. If the kubelet does not respond to the watchdog within the timeout period, the watchdog will kill the kubelet.

Multi-tenancy

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of available configuration options and best practices for cluster multi-tenancy.

Sharing clusters saves costs and simplifies administration. However, sharing clusters also presents challenges such as security, fairness, and managing noisy neighbors.

Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term multi-tenancy.

Node Status

Mon, 01 Jan 0001 00:00:00 +0000

The status of a node in Kubernetes is a critical aspect of managing a Kubernetes cluster. In this article, we'll cover the basics of monitoring and maintaining node status to ensure a healthy and stable cluster.

Node status fields

A Node's status contains the following information:

Addresses
Conditions
Capacity and Allocatable
Info
Declared Features

You can use kubectl to view a Node's status and other details:

kubectl describe node <insert-node-name-here>

Each section of the output is described below.

Page content types

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes documentation follows several types of page content:

Concept
Task
Tutorial
Reference

Content sections

Each page content type contains a number of sections defined by Markdown comments and HTML headings. You can add content headings to your page with the heading shortcode. The comments and headings help maintain the structure of the page content types.

Examples of Markdown comments defining page content sections:

Resource Bin Packing

Mon, 01 Jan 0001 00:00:00 +0000

In the scheduling-plugin NodeResourcesFit of kube-scheduler, there are two scoring strategies that support the bin packing of resources: MostAllocated and RequestedToCapacityRatio.

Enabling bin packing using MostAllocated strategy

The MostAllocated strategy scores the nodes based on the utilization of resources, favoring the ones with higher allocation. For each resource type, you can set a weight to modify its influence in the node score.

To set the MostAllocated strategy for the NodeResourcesFit plugin, use a scheduler configuration similar to the following:

Seccomp and Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Storage Capacity

Mon, 01 Jan 0001 00:00:00 +0000

Storage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with.

FEATURE STATE: Kubernetes v1.24 [stable]

This page describes how Kubernetes keeps track of storage capacity and how the scheduler uses that information to schedule Pods onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed.

System Logs

Mon, 01 Jan 0001 00:00:00 +0000

System component logs record events happening in cluster, which can be very useful for debugging. You can configure log verbosity to see more or less detail. Logs can be as coarse-grained as showing errors within a component, or as fine-grained as showing step-by-step traces of events (like HTTP access logs, pod state changes, controller actions, or scheduler decisions).

Warning:

In contrast to the command line flags described here, the log output itself does not fall under the Kubernetes API stability guarantees: individual log entries and their formatting may change from one release to the next!

Klog

klog is the Kubernetes logging library. klog generates log messages for the Kubernetes system components.

Pod Hostname

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to set a Pod's hostname, potential side effects after configuration, and the underlying mechanics.

Default Pod hostname

When a Pod is created, its hostname (as observed from within the Pod) is derived from the Pod's metadata.name value. Both the hostname and its corresponding fully qualified domain name (FQDN) are set to the metadata.name value (from the Pod's perspective)

apiVersion: v1
kind: Pod
metadata:
 name: busybox-1
spec:
 containers:
 - image: busybox:1.28
 command:
 - sleep
 - "3600"
 name: busybox

The Pod created by this manifest will have its hostname and fully qualified domain name (FQDN) set to busybox-1.

Pod Quality of Service Classes

Mon, 01 Jan 0001 00:00:00 +0000

This page introduces Quality of Service (QoS) classes in Kubernetes, and explains how Kubernetes assigns a QoS class to each Pod as a consequence of the resource constraints that you specify for the containers in that Pod. Kubernetes relies on this classification to make decisions about which Pods to evict when there are not enough available resources on a Node.

Quality of Service classes

Kubernetes classifies the Pods that you run and allocates each Pod into a specific quality of service (QoS) class. Kubernetes uses that classification to influence how different pods are handled. Kubernetes does this classification based on the resource requests of the Containers in that Pod, along with how those requests relate to resource limits. This is known as Quality of Service (QoS) class. Kubernetes assigns every Pod a QoS class based on the resource requests and limits of its component Containers. QoS classes are used by Kubernetes to decide which Pods to evict from a Node experiencing Node Pressure. The possible QoS classes are Guaranteed, Burstable, and BestEffort. When a Node runs out of resources, Kubernetes will first evict BestEffort Pods running on that Node, followed by Burstable and finally Guaranteed Pods. When this eviction is due to resource pressure, only Pods exceeding resource requests are candidates for eviction.

Change the Access Mode of a PersistentVolume to ReadWriteOncePod

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to change the access mode on an existing PersistentVolume to use ReadWriteOncePod.

Before you begin

Change the default StorageClass

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to change the default Storage Class that is used to provision volumes for PersistentVolumeClaims that have no special requirements.

Before you begin

Configure a Pod to Use a PersistentVolume for Storage

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to configure a Pod to use a PersistentVolumeClaim for storage. Here is a summary of the process:

You, as cluster administrator, create a PersistentVolume backed by physical storage. You do not associate the volume with any Pod.
You, now taking the role of a developer / cluster user, create a PersistentVolumeClaim that is automatically bound to a suitable PersistentVolume.

Content organization

Mon, 01 Jan 0001 00:00:00 +0000

This site uses Hugo. In Hugo, content organization is a core concept.

Note:

Hugo Tip: Start Hugo with hugo server --navigateToChanged for content edit-sessions.

Page Lists

Page Order

The documentation side menu, the documentation page browser etc. are listed using Hugo's default sort order, which sorts by weight (from 1), date (newest first), and finally by the link title.

Given that, if you want to move a page or a section up, set a weight in the page's front matter:

Generating Reference Documentation for kubectl Commands

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to generate the kubectl command reference.

Note:

This topic shows how to generate reference documentation for kubectl commands like kubectl apply and kubectl taint. This topic does not show how to generate the kubectl options reference page. For instructions on how to generate the kubectl options reference page, see Generating Reference Pages for Kubernetes Components and Tools.

Before you begin

Requirements:

You need a machine that is running Linux or macOS.

Hardening Guide - Authentication Mechanisms

Mon, 01 Jan 0001 00:00:00 +0000

Selecting the appropriate authentication mechanism(s) is a crucial aspect of securing your cluster. Kubernetes provides several built-in mechanisms, each with its own strengths and weaknesses that should be carefully considered when choosing the best authentication mechanism for your cluster.

In general, it is recommended to enable as few authentication mechanisms as possible to simplify user management and prevent cases where users retain access to a cluster that is no longer required.

Hardening Guide - Scheduler Configuration

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes scheduler is one of the critical components of the control plane.

This document covers how to improve the security posture of the Scheduler.

IPv4/IPv6 dual-stack

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.23 [stable]

IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to Pods and Services.

IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in 1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses.

kubeadm alpha

Mon, 01 Jan 0001 00:00:00 +0000

Caution:

kubeadm alpha provides a preview of a set of features made available for gathering feedback from the community. Please try it out and give us feedback!

Currently there are no experimental commands under kubeadm alpha.

What's next

kubeadm init to bootstrap a Kubernetes control-plane node
kubeadm join to connect a node to the cluster
kubeadm reset to revert any changes made to this host by kubeadm init or kubeadm join

kubeadm certs

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm certs provides utilities for managing certificates. For more details on how these commands can be used, see Certificate Management with kubeadm.

kubeadm certs

A collection of operations for operating Kubernetes certificates.

overview

Synopsis

Commands related to handling Kubernetes certificates

kubeadm init phase

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm init phase enables you to invoke atomic steps of the bootstrap process. Hence, you can let kubeadm do some of the work and you can fill in the gaps if you wish to apply customization.

kubeadm init phase is consistent with the kubeadm init workflow, and behind the scene both use the same code.

kubeadm init phase preflight

Using this command you can execute preflight checks on a control-plane node.

kubeadm join phase

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm join phase enables you to invoke atomic steps of the join process. Hence, you can let kubeadm do some of the work and you can fill in the gaps if you wish to apply customization.

kubeadm join phase is consistent with the kubeadm join workflow, and behind the scene both use the same code.

kubeadm join phase

phase

Synopsis

Use this command to invoke single phase of the "join" workflow

kubeadm kubeconfig

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm kubeconfig provides utilities for managing kubeconfig files.

For examples on how to use kubeadm kubeconfig user see Generating kubeconfig files for additional users.

kubeadm kubeconfig

overview

Synopsis

Kubeconfig file utilities.

kubeadm reset phase

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm reset phase enables you to invoke atomic steps of the node reset process. Hence, you can let kubeadm do some of the work and you can fill in the gaps if you wish to apply customization.

kubeadm reset phase is consistent with the kubeadm reset workflow, and behind the scene both use the same code.

kubeadm reset phase

phase

Synopsis

Use this command to invoke single phase of the "reset" workflow

Kubernetes API Server Bypass Risks

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes API server is the main point of entry to a cluster for external parties (users and services) interacting with it.

As part of this role, the API server has several key built-in security controls, such as audit logging and admission controllers. However, there are ways to modify the configuration or content of the cluster that bypass these controls.

Node-specific Volume Limits

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the maximum number of volumes that can be attached to a Node for various cloud providers.

Cloud providers like Google, Amazon, and Microsoft typically have a limit on how many volumes can be attached to a Node. It is important for Kubernetes to respect those limits. Otherwise, Pods scheduled on a Node could get stuck waiting for volumes to attach.

Kubernetes default limits

The Kubernetes scheduler has default limits on the number of volumes that can be attached to a Node:

Owners and Dependents

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, some objects are owners of other objects. For example, a ReplicaSet is the owner of a set of Pods. These owned objects are dependents of their owner.

Ownership is different from the labels and selectors mechanism that some resources also use. For example, consider a Service that creates EndpointSlice objects. The Service uses labels to allow the control plane to determine which EndpointSlice objects are used for that Service. In addition to the labels, each EndpointSlice that is managed on behalf of a Service has an owner reference. Owner references help different parts of Kubernetes avoid interfering with objects they don’t control.

Pod Priority and Preemption

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.14 [stable]

Pods can have priority. Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible.

Warning:

In a cluster where not all users are trusted, a malicious user could create Pods at the highest possible priorities, causing other Pods to be evicted/not get scheduled. An administrator can use ResourceQuota to prevent users from creating pods at high priorities.

Reconfiguring a kubeadm cluster

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm does not support automated ways of reconfiguring components that were deployed on managed nodes. One way of automating this would be by using a custom operator.

To modify the components configuration you must manually edit associated cluster objects and files on disk.

This guide shows the correct sequence of steps that need to be performed to achieve kubeadm cluster reconfiguration.

Before you begin

You need a cluster that was deployed using kubeadm
Have administrator credentials (/etc/kubernetes/admin.conf) and network connectivity to a running kube-apiserver in the cluster from a host that has kubectl installed
Have a text editor installed on all hosts

Reconfiguring the cluster

kubeadm writes a set of cluster wide component configuration options in ConfigMaps and other objects. These objects must be manually edited. The command kubectl edit can be used for that.

ReplicationController

Mon, 01 Jan 0001 00:00:00 +0000

Note:

A Deployment that configures a ReplicaSet is now the recommended way to set up replication.

A ReplicationController ensures that a specified number of pod replicas are running at any one time. In other words, a ReplicationController makes sure that a pod or a homogeneous set of pods is always up and available.

How a ReplicationController works

If there are too many pods, the ReplicationController terminates the extra pods. If there are too few, the ReplicationController starts more pods. Unlike manually created pods, the pods maintained by a ReplicationController are automatically replaced if they fail, are deleted, or are terminated. For example, your pods are re-created on a node after disruptive maintenance such as a kernel upgrade. For this reason, you should use a ReplicationController even if your application requires only a single pod. A ReplicationController is similar to a process supervisor, but instead of supervising individual processes on a single node, the ReplicationController supervises multiple pods across multiple nodes.

Switching from Polling to CRI Event-based Updates to Container Status

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [alpha](disabled by default)

This page shows how to migrate nodes to use event based updates for container status. The event-based implementation reduces node resource consumption by the kubelet, compared to the legacy approach that relies on polling. You may know this feature as evented Pod lifecycle event generator (PLEG). That's the name used internally within the Kubernetes project for a key implementation detail.

Traces For Kubernetes System Components

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.27 [beta]

System component traces record the latency of and relationships between operations in the cluster.

Kubernetes components emit traces using the OpenTelemetry Protocol with the gRPC exporter and can be collected and routed to tracing backends using an OpenTelemetry Collector.

Trace Collection

Kubernetes components have built-in gRPC exporters for OTLP to export traces, either with an OpenTelemetry Collector, or without an OpenTelemetry Collector.

Workload Reference

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [alpha](disabled by default)

You can link a Pod to a Workload object to indicate that the Pod belongs to a larger application or group. This enables the scheduler to make decisions based on the group's requirements rather than treating the Pod as an independent entity.

Specifying a Workload reference

When the GenericWorkload feature gate is enabled, you can use the spec.workloadRef field in your Pod manifest. This field establishes a link to a specific pod group defined within a Workload resource in the same namespace.

Local ephemeral storage

Mon, 01 Jan 0001 00:00:00 +0000

Nodes have local ephemeral storage, backed by locally-attached writeable devices or, sometimes, by RAM. "Ephemeral" means that there is no long-term guarantee about durability.

Pods use ephemeral local storage for scratch space, caching, and for logs. The kubelet can provide scratch space to Pods using local ephemeral storage to mount emptyDir volumes into containers.

The kubelet also uses this kind of storage to hold node-level container logs, container images, and the writable layers of running containers.

Mapping PodSecurityPolicies to Pod Security Standards

Mon, 01 Jan 0001 00:00:00 +0000

The tables below enumerate the configuration parameters on PodSecurityPolicy objects, whether the field mutates and/or validates pods, and how the configuration values map to the Pod Security Standards.

For each applicable parameter, the allowed values for the Baseline and Restricted profiles are listed. Anything outside the allowed values for those profiles would fall under the Privileged profile. "No opinion" means all values are allowed under all Pod Security Standards.

For a step-by-step migration guide, see Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller.

Advanced contributing

Mon, 01 Jan 0001 00:00:00 +0000

This page assumes that you understand how to contribute to new content and review others' work, and are ready to learn about more ways to contribute. You need to use the Git command line client and other tools for some of these tasks.

Requirements:

You need a machine that is running Linux or macOS.
You need to have these tools installed:
- Python v3.7.x+
- Git
- Golang version 1.13+
- Pip used to install PyYAML
- PyYAML v5.1.2
- make
- gcc compiler/linker
- Docker (Required only for kubectl command reference)
Your PATH environment variable must include the required build tools, such as the Go binary and python.

HorizontalPodAutoscaler Walkthrough

Mon, 01 Jan 0001 00:00:00 +0000

A HorizontalPodAutoscaler (HPA for short) automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand.

Implementation details

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.10 [stable]

kubeadm init and kubeadm join together provide a nice user experience for creating a bare Kubernetes cluster from scratch, that aligns with the best-practices. However, it might not be obvious how kubeadm does that.

This document provides additional details on what happens under the hood, with the aim of sharing knowledge on the best practices for a Kubernetes cluster.

Core design principles

The cluster that kubeadm init and kubeadm join set up should be:

Linux kernel security constraints for Pods and containers

Mon, 01 Jan 0001 00:00:00 +0000

This page describes some of the security features that are built into the Linux kernel that you can use in your Kubernetes workloads. To learn how to apply these features to your Pods and containers, refer to Configure a SecurityContext for a Pod or Container. You should already be familiar with Linux and with the basics of Kubernetes workloads.

Run workloads without root privileges

When you deploy a workload in Kubernetes, use the Pod specification to restrict that workload from running as the root user on the node. You can use the Pod securityContext to define the specific Linux user and group for the processes in the Pod, and explicitly restrict containers from running as root users. Setting these values in the Pod manifest takes precedence over similar values in the container image, which is especially useful if you're running images that you don't own.

List All Container Images Running in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use kubectl to list all of the Container images for Pods running in a cluster.

Before you begin

Node-pressure Eviction

Mon, 01 Jan 0001 00:00:00 +0000

Node-pressure eviction is the process by which the kubelet proactively terminates pods to reclaim resource on nodes.

The kubelet monitors resources like memory, disk space, and filesystem inodes on your cluster's nodes. When one or more of these resources reach specific consumption levels, the kubelet can proactively fail one or more pods on the node to reclaim resources and prevent starvation.

Proxies in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

This page explains proxies used with Kubernetes.

Proxies

There are several different proxies you may encounter when using Kubernetes:

The kubectl proxy:
- runs on a user's desktop or in a pod
- proxies from a localhost address to the Kubernetes apiserver
- client to proxy uses HTTP
- proxy to apiserver uses HTTPS
- locates apiserver
- adds authentication headers
The apiserver proxy:
- is a bastion built into the apiserver
- connects a user outside of the cluster to cluster IPs which otherwise might not be reachable
- runs in the apiserver processes
- client to proxy uses HTTPS (or http if apiserver so configured)
- proxy to target may use HTTP or HTTPS as chosen by proxy using available information
- can be used to reach a Node, Pod, or Service
- does load balancing when used to reach a Service
The kube proxy:

Recommended Labels

Mon, 01 Jan 0001 00:00:00 +0000

You can visualize and manage Kubernetes objects with more tools than kubectl and the dashboard. A common set of labels allows tools to work interoperably, describing objects in a common manner that all tools can understand.

In addition to supporting tooling, the recommended labels describe applications in a way that can be queried.

The metadata is organized around the concept of an application. Kubernetes is not a platform as a service (PaaS) and doesn't have or enforce a formal notion of an application. Instead, applications are informal and described with metadata. The definition of what an application contains is loose.

Security Checklist

Mon, 01 Jan 0001 00:00:00 +0000

This checklist aims at providing a basic list of guidance with links to more comprehensive documentation on each topic. It does not claim to be exhaustive and is meant to evolve.

On how to read and use this document:

The order of topics does not reflect an order of priority.
Some checklist items are detailed in the paragraph below the list of each section.

Caution:

Checklists are not sufficient for attaining a good security posture on their own. A good security posture requires constant attention and improvement, but a checklist can be the first step on the never-ending journey towards security preparedness. Some of the recommendations in this checklist may be too restrictive or too lax for your specific security needs. Since Kubernetes security is not "one size fits all", each category of checklist items should be evaluated on its merits.

Authentication & Authorization

system:masters group is not used for user or component authentication after bootstrapping.
The kube-controller-manager is running with --use-service-account-credentials enabled.
The root certificate is protected (either an offline CA, or a managed online CA with effective access controls).
Intermediate and leaf certificates have an expiry date no more than 3 years in the future.
A process exists for periodic access review, and reviews occur no more than 24 months apart.
The Role Based Access Control Good Practices are followed for guidance related to authentication and authorization.

After bootstrapping, neither users nor components should authenticate to the Kubernetes API as system:masters. Similarly, running all of kube-controller-manager as system:masters should be avoided. In fact, system:masters should only be used as a break-glass mechanism, as opposed to an admin user.

Topology Aware Routing

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.23 [beta]

Note:

Prior to Kubernetes 1.27, this feature was known as Topology Aware Hints.

Topology Aware Routing adjusts routing behavior to prefer keeping traffic in the zone it originated from. In some cases this can help reduce costs or improve network performance.

Motivation

Kubernetes clusters are increasingly deployed in multi-zone environments. Topology Aware Routing provides a mechanism to help keep traffic within the zone it originated from. When calculating the endpoints for a Service, the EndpointSlice controller considers the topology (region and zone) of each endpoint and populates the hints field to allocate it to a zone. Cluster components such as kube-proxy can then consume those hints, and use them to influence how the traffic is routed (favoring topologically closer endpoints).

Volume Health Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.21 [alpha]

CSI volume health monitoring allows CSI Drivers to detect abnormal volume conditions from the underlying storage systems and report them as events on PVCs or Pods.

API Priority and Fairness

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.29 [stable]

Controlling the behavior of the Kubernetes API server in an overload situation is a key task for cluster administrators. The kube-apiserver has some controls available (i.e. the --max-requests-inflight and --max-mutating-requests-inflight command-line flags) to limit the amount of outstanding work that will be accepted, preventing a flood of inbound requests from overloading and potentially crashing the API server, but these flags are not enough to ensure that the most important requests get through in a period of high traffic.

API-initiated Eviction

Mon, 01 Jan 0001 00:00:00 +0000

API-initiated eviction is the process by which you use the Eviction API to create an Eviction object that triggers graceful pod termination.

You can request eviction by calling the Eviction API directly, or programmatically using a client of the API server, like the kubectl drain command. This creates an Eviction object, which causes the API server to terminate the Pod.

API-initiated evictions respect your configured PodDisruptionBudgets and terminationGracePeriodSeconds.

Application Security Checklist

Mon, 01 Jan 0001 00:00:00 +0000

This checklist aims to provide basic guidelines on securing applications running in Kubernetes from a developer's perspective. This list is not meant to be exhaustive and is intended to evolve over time.

On how to read and use this document:

The order of topics does not reflect an order of priority.
Some checklist items are detailed in the paragraph below the list of each section.
This checklist assumes that a developer is a Kubernetes cluster user who interacts with namespaced scope objects.

Caution:

Checklists are not sufficient for attaining a good security posture on their own. A good security posture requires constant attention and improvement, but a checklist can be the first step on the never-ending journey towards security preparedness. Some recommendations in this checklist may be too restrictive or too lax for your specific security needs. Since Kubernetes security is not "one size fits all", each category of checklist items should be evaluated on its merits.

Base security hardening

The following checklist provides base security hardening recommendations that would apply to most applications deploying to Kubernetes.

Cloud Controller Manager Administration

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.11 [beta]

Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the cloud-controller-manager binary allows cloud vendors to evolve independently from the core Kubernetes code.

The cloud-controller-manager can be linked to any cloud provider that satisfies cloudprovider.Interface. For backwards compatibility, the cloud-controller-manager provided in the core Kubernetes project uses the same cloud libraries as kube-controller-manager. Cloud providers already supported in Kubernetes core are expected to use the in-tree cloud-controller-manager to transition out of Kubernetes core.

Configure a Security Context for a Pod or Container

Mon, 01 Jan 0001 00:00:00 +0000

A security context defines privilege and access control settings for a Pod or Container. Security context settings include, but are not limited to:

Discretionary Access Control: Permission to access an object, like a file, is based on user ID (UID) and group ID (GID).
Security Enhanced Linux (SELinux): Objects are assigned security labels.
Running as privileged or unprivileged.
Linux Capabilities: Give a process some privileges, but not all the privileges of the root user.

Kubelet authentication/authorization

Mon, 01 Jan 0001 00:00:00 +0000

Overview

A kubelet's HTTPS endpoint exposes APIs which give access to data of varying sensitivity, and allow you to perform operations with varying levels of power on the node and within containers.

This document describes how to authenticate and authorize access to the kubelet's HTTPS endpoint.

Kubelet authentication

By default, requests to the kubelet's HTTPS endpoint that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated.

Linux Node Swap Behaviors

Mon, 01 Jan 0001 00:00:00 +0000

To allow Kubernetes workloads to use swap, on a Linux node, you must disable the kubelet's default behavior of failing when swap is detected, and specify memory-swap behavior as LimitedSwap:

The available choices for swap behavior are:

NoSwap: (default) Workloads running as Pods on this node do not and cannot use swap. However, processes outside of Kubernetes' scope, such as system daemons (including the kubelet itself!) can utilize swap. This behavior is beneficial for protecting the node from system-level memory spikes, but it does not safeguard the workloads themselves from such spikes.
LimitedSwap: Kubernetes workloads can utilize swap memory. The amount of swap available to a Pod is determined automatically.

To learn more, read swap memory management.

Networking on Windows

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes supports running nodes on either Linux or Windows. You can mix both kinds of node within a single cluster. This page provides an overview to networking specific to the Windows operating system.

Container networking on Windows

Networking for Windows containers is exposed through CNI plugins. Windows containers function similarly to virtual machines in regards to networking. Each container has a virtual network adapter (vNIC) which is connected to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the Host Compute Service (HCS) work together to create containers and attach container vNICs to networks. HCS is responsible for the management of containers whereas HNS is responsible for the management of networking resources such as:

Specifying a Disruption Budget for your Application

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.21 [stable]

This page shows how to limit the number of concurrent disruptions that your application experiences, allowing for higher availability while permitting the cluster administrator to manage the clusters nodes.

Before you begin

Your Kubernetes server must be at or later than version v1.21.

To check the version, enter kubectl version.

You are the owner of an application running on a Kubernetes cluster that requires high availability.
You should know how to deploy Replicated Stateless Applications and/or Replicated Stateful Applications.
You should have read about Pod Disruptions.
You should confirm with your cluster owner or service provider that they respect Pod Disruption Budgets.

Protecting an Application with a PodDisruptionBudget

Identify what application you want to protect with a PodDisruptionBudget (PDB).
Think about how your application reacts to disruptions.
Create a PDB definition as a YAML file.
Create the PDB object from the YAML file.

Identify an Application to Protect

The most common use case when you want to protect an application specified by one of the built-in Kubernetes controllers:

Storage Versions

Mon, 01 Jan 0001 00:00:00 +0000

The Kubernetes API server stores objects, relying on an etcd-compatible backing store (often, the backing storage is etcd itself). Each object is serialized using a particular version of that API type; for example, the v1 representation of a ConfigMap. Kubernetes uses the term storage version to describe how an object is stored in your cluster.

The Kubernetes API also relies on automatic conversion; for example, if you have a HorizontalPodAutoscaler, then you can interact with that HorizontalPodAutoscaler using any mix of the v1 and v2 versions of the HorizontalPodAutoscaler API. Kubernetes is responsible for converting each API call so that clients do not see what version is actually serialized.

Windows Storage

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an storage overview specific to the Windows operating system.

Persistent storage

Windows has a layered filesystem driver to mount container layers and create a copy filesystem based on NTFS. All file paths in the container are resolved only within the context of that container.

With Docker, volume mounts can only target a directory in the container, and not an individual file. This limitation does not apply to containerd.
Volume mounts cannot project files or directories back to the host filesystem.
Read-only filesystems are not supported because write access is always required for the Windows registry and SAM database. However, read-only volumes are supported.
Volume user-masks and permissions are not available. Because the SAM is not shared between the host & container, there's no mapping between them. All permissions are resolved within the context of the container.

As a result, the following storage functionality is not supported on Windows nodes:

Accessing the Kubernetes API from a Pod

Mon, 01 Jan 0001 00:00:00 +0000

This guide demonstrates how to access the Kubernetes API from within a pod.

Before you begin

Communicate Between Containers in the Same Pod Using a Shared Volume

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use a Volume to communicate between two Containers running in the same Pod. See also how to allow processes to communicate by sharing process namespace between containers.

Before you begin

Configure a kubelet image credential provider

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [stable]

Starting from Kubernetes v1.20, the kubelet can dynamically retrieve credentials for a container image registry using exec plugins. The kubelet and the exec plugin communicate through stdio (stdin, stdout, and stderr) using Kubernetes versioned APIs. These plugins allow the kubelet to request credentials for a container registry dynamically as opposed to storing static credentials on disk. For example, the plugin may talk to a local metadata server to retrieve short-lived credentials for an image that is being pulled by the kubelet.

Configure Service Accounts for Pods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes offers two distinct ways for clients that run within your cluster, or that otherwise have a relationship to your cluster's control plane to authenticate to the API server.

A service account provides an identity for processes that run in a Pod, and maps to a ServiceAccount object. When you authenticate to the API server, you identify yourself as a particular user. Kubernetes recognises the concept of a user, however, Kubernetes itself does not have a User API.

Custom Hugo Shortcodes

Mon, 01 Jan 0001 00:00:00 +0000

This page explains the custom Hugo shortcodes that can be used in Kubernetes Markdown documentation.

Read more about shortcodes in the Hugo documentation.

Feature state

In a Markdown page (.md file) on this site, you can add a shortcode to display version and state of the documented feature.

Feature state demo

Below is a demo of the feature state snippet, which displays the feature as stable in the latest Kubernetes version.

Generating Reference Pages for Kubernetes Components and Tools

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to build the Kubernetes component and tool reference pages.

Before you begin

Start with the Prerequisites section in the Reference Documentation Quickstart guide.

Follow the Reference Documentation Quickstart to generate the Kubernetes component and tool reference pages.

What's next

Service ClusterIP allocation

Mon, 01 Jan 0001 00:00:00 +0000

In Kubernetes, Services are an abstract way to expose an application running on a set of Pods. Services can have a cluster-scoped virtual IP address (using a Service of type: ClusterIP). Clients can connect using that virtual IP address, and Kubernetes then load-balances traffic to that Service across the different backing Pods.

How Service ClusterIPs are allocated?

When Kubernetes needs to assign a virtual IP address for a Service, that assignment happens one of two ways:

Service Internal Traffic Policy

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [stable]

Service Internal Traffic Policy enables internal traffic restrictions to only route internal traffic to endpoints within the node the traffic originated from. The "internal" traffic here refers to traffic originated from Pods in the current cluster. This can help to reduce costs and improve performance.

Using Service Internal Traffic Policy

You can enable the internal-only traffic policy for a Service, by setting its .spec.internalTrafficPolicy to Local. This tells kube-proxy to only use node local endpoints for cluster internal traffic.

TLS bootstrapping

Mon, 01 Jan 0001 00:00:00 +0000

In a Kubernetes cluster, the components on the worker nodes - kubelet and kube-proxy - need to communicate with Kubernetes control plane components, specifically kube-apiserver. In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, we strongly recommend using client TLS certificates on nodes.

The normal process of bootstrapping these components, especially worker nodes that need certificates so they can communicate safely with kube-apiserver, can be a challenging process as it is often outside of the scope of Kubernetes and requires significant additional work. This in turn, can make it challenging to initialize or scale a cluster.

Viewing Site Analytics

Mon, 01 Jan 0001 00:00:00 +0000

This page contains information about the kubernetes.io analytics dashboard.

View the dashboard.

This dashboard is built using Google Looker Studio and shows information collected on kubernetes.io using Google Analytics 4 since August 2022.

Using the dashboard

By default, the dashboard shows all collected analytics for the past 30 days. Use the date selector to see data from a different date range. Other filtering options allow you to view data based on user location, the device used to access the site, the translation of the docs used, and more.

Configure DNS for a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes offers a DNS cluster addon, which most of the supported environments enable by default. In Kubernetes version 1.11 and later, CoreDNS is recommended and is installed by default with kubeadm.

For more information on how to configure CoreDNS for a Kubernetes cluster, see the Customizing DNS Service.

Configure Quotas for API Objects

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure quotas for API objects, including PersistentVolumeClaims and Services. A quota restricts the number of objects, of a particular type, that can be created in a namespace. You specify quotas in a ResourceQuota object.

Before you begin

Flow control

Mon, 01 Jan 0001 00:00:00 +0000

API Priority and Fairness controls the behavior of the Kubernetes API server in an overload situation. You can find more information about it in the API Priority and Fairness documentation.

Diagnostics

Every HTTP response from an API server with the priority and fairness feature enabled has two extra headers: X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID, noting the flow schema that matched the request and the priority level to which it was assigned, respectively. The API objects' names are not included in these headers (to avoid revealing details in case the requesting user does not have permission to view them). When debugging, you can use a command such as:

Pull an Image from a Private Registry

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to create a Pod that uses a Secret to pull an image from a private container image registry or repository. There are many private registries in use. This task uses Docker Hub as an example registry.

🛇 This item links to a third party project or product that is not part of Kubernetes itself. More information

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Access Services Running on Clusters

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to connect to services running on the Kubernetes cluster.

Before you begin

Configure Liveness, Readiness and Startup Probes

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure liveness, readiness and startup probes for containers.

For more information about probes, see Liveness, Readiness and Startup Probes

The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.

Control CPU Management Policies on the Node

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [stable]

Kubernetes keeps many aspects of how pods execute on nodes abstracted from the user. This is by design. However, some workloads require stronger guarantees in terms of latency and/or performance in order to operate acceptably. The kubelet provides methods to enable more complex workload placement policies while keeping the abstraction free from explicit placement directives.

For detailed information on resource management, please refer to the Resource Management for Pods and Containers documentation.

Control Memory Management Policies on a Node

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.32 [stable](enabled by default)

The Kubernetes Memory Manager enables the feature of guaranteed memory (and hugepages) allocation for pods in the Guaranteed QoS class.

The Memory Manager employs a hint generation protocol to yield the most suitable NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology Manager) with these affinity hints. Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.

Assign Pods to Nodes

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to assign a Kubernetes Pod to a particular node in a Kubernetes cluster.

Before you begin

Changing The Kubernetes Package Repository

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to enable a package repository for the desired Kubernetes minor release upon upgrading a cluster. This is only needed for users of the community-owned package repositories hosted at pkgs.k8s.io. Unlike the legacy package repositories, the community-owned package repositories are structured in a way that there's a dedicated package repository for each Kubernetes minor version.

Note:

This guide only covers a part of the Kubernetes upgrade process. Please see the upgrade guide for more information about upgrading Kubernetes clusters.

Note:

This step is only needed upon upgrading a cluster to another minor release. If you're upgrading to another patch release within the same minor release (e.g. v1.35.5 to v1.35.7), you don't need to follow this guide. However, if you're still using the legacy package repositories, you'll need to migrate to the new community-owned package repositories before upgrading (see the next section for more details on how to do this).

Before you begin

This document assumes that you're already using the community-owned package repositories (pkgs.k8s.io). If that's not the case, it's strongly recommended to migrate to the community-owned package repositories as described in the official announcement.

Control Topology Management Policies on a node

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.27 [stable]

An increasing number of systems leverage a combination of CPUs and hardware accelerators to support latency-critical execution and high-throughput parallel computation. These include workloads in fields such as telecommunications, scientific computing, machine learning, financial services and data analytics. Such hybrid systems comprise a high performance environment.

In order to extract the best performance, optimizations related to CPU isolation, memory and device locality are required. However, in Kubernetes, these optimizations are handled by a disjoint set of components.

Installing Addons

Mon, 01 Jan 0001 00:00:00 +0000

Add-ons extend the functionality of Kubernetes.

This page lists some of the available add-ons and links to their respective installation instructions. The list does not try to be exhaustive.

Assign Pods to Nodes using Node Affinity

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to assign a Kubernetes Pod to a particular node using Node Affinity in a Kubernetes cluster.

Before you begin

Customizing DNS Service

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to configure your DNS Pod(s) and customize the DNS resolution process in your cluster.

Before you begin

Node Declared Features

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [alpha](disabled by default)

Kubernetes nodes use declared features to report the availability of specific features that are new or feature-gated. Control plane components utilize this information to make better decisions. The kube-scheduler, via the NodeDeclaredFeatures plugin, ensures pods are only placed on nodes that explicitly support the features the pod requires. Additionally, the NodeDeclaredFeatureValidator admission controller validates pod updates against a node's declared features.

User Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.30 [beta]

This page explains how user namespaces are used in Kubernetes pods. A user namespace isolates the user running inside the container from the one in the host.

A process running as root in a container can run as a different (non-root) user in the host; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace.

Configure Pod Initialization

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to use an Init Container to initialize a Pod before an application Container runs.

Before you begin

Debugging DNS Resolution

Mon, 01 Jan 0001 00:00:00 +0000

This page provides hints on diagnosing DNS problems.

Before you begin

Downward API

Mon, 01 Jan 0001 00:00:00 +0000

It is sometimes useful for a container to have information about itself, without being overly coupled to Kubernetes. The downward API allows containers to consume information about themselves or the cluster without using the Kubernetes client or API server.

An example is an existing application that assumes a particular well-known environment variable holds a unique identifier. One possibility is to wrap the application, but that is tedious and error-prone, and it violates the goal of low coupling. A better option would be to use the Pod's name as an identifier, and inject the Pod's name into the well-known environment variable.

Advanced Pod Configuration

Mon, 01 Jan 0001 00:00:00 +0000

This page covers advanced Pod configuration topics including PriorityClasses, RuntimeClasses, security context within Pods, and introduces aspects of scheduling.

PriorityClasses

PriorityClasses allow you to set the importance of Pods relative to other Pods. If you assign a priority class to a Pod, Kubernetes sets the .spec.priority field for that Pod based on the PriorityClass you specified (you cannot set .spec.priority directly). If or when a Pod cannot be scheduled, and the problem is due to a lack of resources, the kube-scheduler tries to preempt lower priority Pods, in order to make scheduling of the higher priority Pod possible.

Attach Handlers to Container Lifecycle Events

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to attach handlers to Container lifecycle events. Kubernetes supports the postStart and preStop events. Kubernetes sends the postStart event immediately after a Container is started, and it sends the preStop event immediately before the Container is terminated. A Container may specify one handler per event.

Before you begin

Declare Network Policy

Mon, 01 Jan 0001 00:00:00 +0000

This document helps you get started using the Kubernetes NetworkPolicy API to declare network policies that govern how pods communicate with each other.

Before you begin

Configure a Pod to Use a ConfigMap

Mon, 01 Jan 0001 00:00:00 +0000

Many applications rely on configuration which is used during either application initialization or runtime. Most times, there is a requirement to adjust values assigned to configuration parameters. ConfigMaps are a Kubernetes mechanism that let you inject configuration data into application pods.

The ConfigMap concept allow you to decouple configuration artifacts from image content to keep containerized applications portable. For example, you can download and run the same container image to spin up containers for the purposes of local development, system test, or running a live end-user workload.

Developing Cloud Controller Manager

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.11 [beta]

Coordinated Leader Election

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.33 [beta](disabled by default)

Kubernetes 1.35 includes a beta feature that allows control plane components to deterministically select a leader via coordinated leader election. This is useful to satisfy Kubernetes version skew constraints during cluster upgrades. Currently, the only builtin selection strategy is OldestEmulationVersion, preferring the leader with the lowest emulation version, followed by binary version, followed by creation timestamp.

Enable Or Disable A Kubernetes API

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to enable or disable an API version from your cluster's control plane.

Specific API versions can be turned on or off by passing --runtime-config=api/<version> as a command line argument to the API server. The values for this argument are a comma-separated list of API versions. Later values override earlier values.

Share Process Namespace between Containers in a Pod

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure process namespace sharing for a pod. When process namespace sharing is enabled, processes in a container are visible to all other containers in the same pod.

You can use this feature to configure cooperating containers, such as a log handler sidecar container, or to troubleshoot container images that don't include debugging utilities like a shell.

Before you begin

Encrypting Confidential Data at Rest

Mon, 01 Jan 0001 00:00:00 +0000

All of the APIs in Kubernetes that let you write persistent API resource data support at-rest encryption. For example, you can enable at-rest encryption for Secrets. This at-rest encryption is additional to any system-level encryption for the etcd cluster or for the filesystem(s) on hosts where you are running the kube-apiserver.

This page shows how to enable and configure encryption of API data at rest.

Use a User Namespace With a Pod

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.33 [beta](enabled by default)

This page shows how to configure a user namespace for pods. This allows you to isolate the user running inside the container from the one in the host.

Use an Image Volume With a Pod

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.35 [beta](enabled by default)

This page shows how to configure a pod using image volumes. This allows you to mount content from OCI registries inside containers.

Before you begin

Decrypt Confidential Data that is Already Encrypted at Rest

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to switch from encryption of API data at rest, so that API data are stored unencrypted. You might want to do this to improve performance; usually, though, if it was a good idea to encrypt some data, it's also a good idea to leave them encrypted.

Create static Pods

Mon, 01 Jan 0001 00:00:00 +0000

Static Pods are managed directly by the kubelet daemon on a specific node, without the API server observing them. Unlike Pods that are managed by the control plane (for example, a Deployment); instead, the kubelet watches each static Pod (and restarts it if it fails).

Static Pods are always bound to one Kubelet on a specific node.

Guaranteed Scheduling For Critical Add-On Pods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes core components such as the API server, scheduler, and controller-manager run on a control plane node. However, add-ons must run on a regular cluster node. Some of these add-ons are critical to a fully functional cluster, such as metrics-server, DNS, and UI. A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade) and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason).

Mixed Version Proxy

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.28 [alpha](disabled by default)

Kubernetes 1.35 includes an alpha feature that lets an API Server proxy resource requests to other peer API servers. It also lets clients get a holistic view of resources served across the entire cluster through discovery. This is useful when there are multiple API servers running different versions of Kubernetes in one cluster (for example, during a long-lived rollout to a new release of Kubernetes).

IP Masquerade Agent User Guide

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure and enable the ip-masq-agent.

Before you begin

Translate a Docker Compose File to Kubernetes Resources

Mon, 01 Jan 0001 00:00:00 +0000

What's Kompose? It's a conversion tool for all things compose (namely Docker Compose) to container orchestrators (Kubernetes or OpenShift).

More information can be found on the Kompose website at https://kompose.io/.

Before you begin

Enforce Pod Security Standards by Configuring the Built-in Admission Controller

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes provides a built-in admission controller to enforce the Pod Security Standards. You can configure this admission controller to set cluster-wide defaults and exemptions.

Before you begin

Following an alpha release in Kubernetes v1.22, Pod Security Admission became available by default in Kubernetes v1.23, as a beta. From version 1.25 onwards, Pod Security Admission is generally available.

To check the version, enter kubectl version.

If you are not running Kubernetes 1.35, you can switch to viewing this page in the documentation for the Kubernetes version that you are running.

Limit Storage Consumption

Mon, 01 Jan 0001 00:00:00 +0000

This example demonstrates how to limit the amount of storage consumed in a namespace.

The following resources are used in the demonstration: ResourceQuota, LimitRange, and PersistentVolumeClaim.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Enforce Pod Security Standards with Namespace Labels

Mon, 01 Jan 0001 00:00:00 +0000

Namespaces can be labeled to enforce the Pod Security Standards. The three policies privileged, baseline and restricted broadly cover the security spectrum and are implemented by the Pod Security admission controller.

Before you begin

Pod Security Admission was available by default in Kubernetes v1.23, as a beta. From version 1.25 onwards, Pod Security Admission is generally available.

Migrate Replicated Control Plane To Use Cloud Controller Manager

Mon, 01 Jan 0001 00:00:00 +0000

Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller

Mon, 01 Jan 0001 00:00:00 +0000

This page describes the process of migrating from PodSecurityPolicies to the built-in PodSecurity admission controller. This can be done effectively using a combination of dry-run and audit and warn modes, although this becomes harder if mutating PSPs are used.

Before you begin

Your Kubernetes server must be at or later than version v1.22.

To check the version, enter kubectl version.

If you are currently running a version of Kubernetes other than 1.35, you may want to switch to viewing this page in the documentation for the version of Kubernetes that you are actually running.

Namespaces Walkthrough

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes namespaces help different projects, teams, or customers to share a Kubernetes cluster.

It does this by providing the following:

A scope for Names.
A mechanism to attach authorization and policy to a subsection of the cluster.

Use of multiple namespaces is optional.

This example demonstrates how to use Kubernetes namespaces to subdivide your cluster.

Operating etcd clusters for Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

etcd is a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data.

If your Kubernetes cluster uses etcd as its backing store, make sure you have a back up plan for the data.

You can find in-depth information about etcd in the official documentation.

Before you begin

Before you follow steps in this page to deploy, manage, back up or restore etcd, you need to understand the typical expectations for operating an etcd cluster. Refer to the etcd documentation for more context.

Reserve Compute Resources for System Daemons

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes nodes can be scheduled to Capacity. Pods can consume all the available capacity on a node by default. This is an issue because nodes typically run quite a few system daemons that power the OS and Kubernetes itself. Unless resources are set aside for these system daemons, pods and system daemons compete for resources and lead to resource starvation issues on the node.

The kubelet exposes a feature named 'Node Allocatable' that helps to reserve compute resources for system daemons. Kubernetes recommends cluster administrators to configure 'Node Allocatable' based on their workload density on each node.

Running Kubernetes Node Components as a Non-root User

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.22 [alpha]

This document describes how to run Kubernetes Node components such as kubelet, CRI, OCI, and CNI without root privileges, by using a user namespace.

This technique is also known as rootless mode.

Note:

This document describes how to run Kubernetes Node components (and hence pods) as a non-root user.

If you are just looking for how to run a pod as a non-root user, see SecurityContext.

Safely Drain a Node

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to safely drain a node, optionally respecting the PodDisruptionBudget you have defined.

Before you begin

This task assumes that you have met the following prerequisites:

You do not require your applications to be highly available during the node drain, or
You have read about the PodDisruptionBudget concept, and have configured PodDisruptionBudgets for applications that need them.

(Optional) Configure a disruption budget

To ensure that your workloads remain available during maintenance, you can configure a PodDisruptionBudget.

Securing a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This document covers topics related to protecting a cluster from accidental or malicious access and provides recommendations on overall security.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

Set Kubelet Parameters Via A Configuration File

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Some steps in this page use the jq tool. If you don't have jq, you can install it via your operating system's software sources, or fetch it from https://jqlang.github.io/jq/.

Some steps also involve installing curl, which can be installed via your operating system's software sources.

A subset of the kubelet's configuration parameters may be set via an on-disk config file, as a substitute for command-line flags.

Share a Cluster with Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to view, work in, and delete namespaces. The page also shows how to use Kubernetes namespaces to subdivide your cluster.

Before you begin

Have an existing Kubernetes cluster.
You have a basic understanding of Kubernetes Pods, Services, and Deployments.

Viewing namespaces

List the current namespaces in a cluster using:

Upgrade A Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page provides an overview of the steps you should follow to upgrade a Kubernetes cluster.

The way that you upgrade a cluster depends on how you initially deployed it and on any subsequent changes.

Use Cascading Deletion in a Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This page shows you how to specify the type of cascading deletion to use in your cluster during garbage collection.

Before you begin

Using a KMS provider for data encryption

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to configure a Key Management Service (KMS) provider and plugin to enable secret data encryption. In Kubernetes 1.35 there are two versions of KMS at-rest encryption. You should use KMS v2 if feasible because KMS v1 is deprecated (since Kubernetes v1.28) and disabled by default (since Kubernetes v1.29). KMS v2 offers significantly better performance characteristics than KMS v1.

Caution:

This documentation is for the generally available implementation of KMS v2 (and for the deprecated version 1 implementation). If you are using any control plane components older than Kubernetes v1.29, please check the equivalent page in the documentation for the version of Kubernetes that your cluster is running. Earlier releases of Kubernetes had different behavior that may be relevant for information security.

Starting from Kubernetes version 1.23, the kubelet supports the use of either / or . as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes. For example, you can represent the same sysctl name as kernel.shm_rmid_forced using a period as the separator, or as kernel/shm_rmid_forced using a slash as a separator. For more sysctl parameter conversion method details, please refer to the page sysctl.d(5) from the Linux man-pages project.

Before you begin

Note:

sysctl is a Linux-specific command-line tool used to configure various kernel parameters and it is not available on non-Linux operating systems.

Verify Signed Kubernetes Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.26 [beta]

Before you begin

You will need to have the following tools installed:

cosign (install guide)
curl (often provided by your operating system)
jq (download jq)

Verifying binary signatures

The Kubernetes release process signs all binary artifacts (tarballs, SPDX files, standalone binaries) by using cosign's keyless signing. To verify a particular binary, retrieve it together with its signature and certificate:

URL=https://dl.k8s.io/release/v1.35.0/bin/linux/amd64
BINARY=kubectl

FILES=(
 "$BINARY"
 "$BINARY.sig"
 "$BINARY.cert"
)

for FILE in "${FILES[@]}"; do
 curl -sSfL --retry 3 --retry-delay 3 "$URL/$FILE" -o "$FILE"
done

Then verify the blob by using cosign verify-blob:

Spotlight on SIG Architecture: API Governance

Thu, 12 Feb 2026 00:00:00 +0000

This is the fifth interview of a SIG Architecture Spotlight series that covers the different subprojects, and we will be covering SIG Architecture: API Governance.

In this SIG Architecture spotlight we talked with Jordan Liggitt, lead of the API Governance sub-project.

Introduction

FM: Hello Jordan, thank you for your availability. Tell us a bit about yourself, your role and how you got involved in Kubernetes.

JL: My name is Jordan Liggitt. I'm a Christian, husband, father of four, software engineer at Google by day, and amateur musician by stealth. I was born in Texas (and still like to claim it as my point of origin), but I've lived in North Carolina for most of my life.

Introducing Node Readiness Controller

Tue, 03 Feb 2026 10:00:00 +0800

In the standard Kubernetes model, a node’s suitability for workloads hinges on a single binary "Ready" condition. However, in modern Kubernetes environments, nodes require complex infrastructure dependencies—such as network agents, storage drivers, GPU firmware, or custom health checks—to be fully operational before they can reliably host pods.

Today, on behalf of the Kubernetes project, I am announcing the Node Readiness Controller. This project introduces a declarative system for managing node taints, extending the readiness guardrails during node bootstrapping beyond standard conditions. By dynamically managing taints based on custom health signals, the controller ensures that workloads are only placed on nodes that met all infrastructure-specific requirements.

New Conversion from cgroup v1 CPU Shares to v2 CPU Weight

Fri, 30 Jan 2026 08:00:00 -0800

I'm excited to announce the implementation of an improved conversion formula from cgroup v1 CPU shares to cgroup v2 CPU weight. This enhancement addresses critical issues with CPU priority allocation for Kubernetes workloads when running on systems with cgroup v2.

Background

Kubernetes was originally designed with cgroup v1 in mind, where CPU shares were defined simply by assigning the container's CPU requests in millicpu form.

For example, a container requesting 1 CPU (1024m) would get (cpu.shares = 1024).

Ingress NGINX: Statement from the Kubernetes Steering and Security Response Committees

Thu, 29 Jan 2026 00:00:00 +0000

In March 2026, Kubernetes will retire Ingress NGINX, a piece of critical infrastructure for about half of cloud native environments. The retirement of Ingress NGINX was announced for March 2026, after years of public warnings that the project was in dire need of contributors and maintainers. There will be no more releases for bug fixes, security patches, or any updates of any kind after the project is retired. This cannot be ignored, brushed off, or left until the last minute to address. We cannot overstate the severity of this situation or the importance of beginning migration to alternatives like Gateway API or one of the many third-party Ingress controllers immediately.

Experimenting with Gateway API using kind

Wed, 28 Jan 2026 00:00:00 +0000

This document will guide you through setting up a local experimental environment with Gateway API on kind. This setup is designed for learning and testing. It helps you understand Gateway API concepts without production complexity.

Caution:

This is an experimentation learning setup, and should not be used for production. The components used on this document are not suited for production usage. Once you're ready to deploy Gateway API in a production environment, select an implementation that suits your needs.

Overview

In this guide, you will:

Cluster API v1.12: Introducing In-place Updates and Chained Upgrades

Tue, 27 Jan 2026 08:00:00 -0800

Cluster API brings declarative management to Kubernetes cluster lifecycle, allowing users and platform teams to define the desired state of clusters and rely on controllers to continuously reconcile toward it.

Similar to how you can use StatefulSets or Deployments in Kubernetes to manage a group of Pods, in Cluster API you can use KubeadmControlPlane to manage a set of control plane Machines, or you can use MachineDeployments to manage a group of worker Nodes.

Headlamp in 2025: Project Highlights

Thu, 22 Jan 2026 10:00:00 +0800

This announcement is a recap from a post originally published on the Headlamp blog.

Headlamp has come a long way in 2025. The project has continued to grow – reaching more teams across platforms, powering new workflows and integrations through plugins, and seeing increased collaboration from the broader community.

We wanted to take a moment to share a few updates and highlight how Headlamp has evolved over the past year.

Updates

Joining Kubernetes SIG UI

This year marked a big milestone for the project: Headlamp is now officially part of Kubernetes SIG UI. This move brings roadmap and design discussions even closer to the core Kubernetes community and reinforces Headlamp’s role as a modern, extensible UI for the project.

Announcing the Checkpoint/Restore Working Group

Wed, 21 Jan 2026 10:00:00 -0800

The community around Kubernetes includes a number of Special Interest Groups (SIGs) and Working Groups (WGs) facilitating discussions on important topics between interested contributors. Today we would like to announce the new Kubernetes Checkpoint Restore WG focusing on the integration of Checkpoint/Restore functionality into Kubernetes.

Motivation and use cases

There are several high-level scenarios discussed in the working group:

Optimizing resource utilization for interactive workloads, such as Jupyter notebooks and AI chatbots
Accelerating startup of applications with long initialization times, including Java applications and LLM inference services
Using periodic checkpointing to enable fault-tolerance for long-running workloads, such as distributed model training
Providing interruption-aware scheduling with transparent checkpoint/restore, allowing lower-priority Pods to be preempted while preserving the runtime state of applications
Facilitating Pod migration across nodes for load balancing and maintenance, without disrupting workloads.
Enabling forensic checkpointing to investigate and analyze security incidents such as cyberattacks, data breaches, and unauthorized access.

Across these scenarios, the goal is to help facilitate discussions of ideas between the Kubernetes community and the growing Checkpoint/Restore in Userspace (CRIU) ecosystem. The CRIU community includes several projects that support these use cases, including:

Uniform API server access using clientcmd

Mon, 19 Jan 2026 10:00:00 -0800

If you've ever wanted to develop a command line client for a Kubernetes API, especially if you've considered making your client usable as a kubectl plugin, you might have wondered how to make your client feel familiar to users of kubectl. A quick glance at the output of kubectl options might put a damper on that: "Am I really supposed to implement all those options?"

Fear not, others have done a lot of the work involved for you. In fact, the Kubernetes project provides two libraries to help you handle kubectl-style command line arguments in Go programs: clientcmd and cli-runtime (which uses clientcmd). This article will show how to use the former.

Kubernetes v1.35: Restricting executables invoked by kubeconfigs via exec plugin allowList added to kuberc

Fri, 09 Jan 2026 10:30:00 -0800

Did you know that kubectl can run arbitrary executables, including shell scripts, with the full privileges of the invoking user, and without your knowledge? Whenever you download or auto-generate a kubeconfig, the users[n].exec.command field can specify an executable to fetch credentials on your behalf. Don't get me wrong, this is an incredible feature that allows you to authenticate to the cluster with external identity providers. Nevertheless, you probably see the problem: Do you know exactly what executables your kubeconfig is running on your system? Do you trust the pipeline that generated your kubeconfig? If there has been a supply-chain attack on the code that generates the kubeconfig, or if the generating pipeline has been compromised, an attacker might well be doing unsavory things to your machine by tricking your kubeconfig into running arbitrary code.

Kubernetes v1.35: Mutable PersistentVolume Node Affinity (alpha)

Thu, 08 Jan 2026 10:30:00 -0800

The PersistentVolume node affinity API dates back to Kubernetes v1.10. It is widely used to express that volumes may not be equally accessible by all nodes in the cluster. This field was previously immutable, and it is now mutable in Kubernetes v1.35 (alpha). This change opens a door to more flexible online volume management.

Why make node affinity mutable?

This raises an obvious question: why make node affinity mutable now? While stateless workloads like Deployments can be changed freely and the changes will be rolled out automatically by re-creating every Pod, PersistentVolumes (PVs) are stateful and cannot be re-created easily without losing data.

Kubernetes v1.35: A Better Way to Pass Service Account Tokens to CSI Drivers

Wed, 07 Jan 2026 10:30:00 -0800

If you maintain a CSI driver that uses service account tokens, Kubernetes v1.35 brings a refinement you'll want to know about. Since the introduction of the TokenRequests feature, service account tokens requested by CSI drivers have been passed to them through the volume_context field. While this has worked, it's not the ideal place for sensitive information, and we've seen instances where tokens were accidentally logged in CSI drivers.

Kubernetes v1.35 introduces a beta solution to address this: CSI Driver Opt-in for Service Account Tokens via Secrets Field. This allows CSI drivers to receive service account tokens through the secrets field in NodePublishVolumeRequest, which is the appropriate place for sensitive data in the CSI specification.

Kubernetes v1.35: Extended Toleration Operators to Support Numeric Comparisons (Alpha)

Mon, 05 Jan 2026 10:30:00 -0800

Many production Kubernetes clusters blend on-demand (higher-SLA) and spot/preemptible (lower-SLA) nodes to optimize costs while maintaining reliability for critical workloads. Platform teams need a safe default that keeps most workloads away from risky capacity, while allowing specific workloads to opt-in with explicit thresholds like "I can tolerate nodes with failure probability up to 5%".

Today, Kubernetes taints and tolerations can match exact values or check for existence, but they can't compare numeric thresholds. You'd need to create discrete taint categories, use external admission controllers, or accept less-than-optimal placement decisions.

Kubernetes v1.35: New level of efficiency with in-place Pod restart

Fri, 02 Jan 2026 10:30:00 -0800

The release of Kubernetes 1.35 introduces a powerful new feature that provides a much-requested capability: the ability to trigger a full, in-place restart of the Pod. This feature, Restart All Containers (alpha in 1.35), allows for an efficient way to reset a Pod's state compared to resource-intensive approach of deleting and recreating the entire Pod. This feature is especially useful for AI/ML workloads allowing application developers to concentrate on their core training logic while offloading complex failure-handling and recovery mechanisms to sidecars and declarative Kubernetes configuration. With RestartAllContainers and other planned enhancements, Kubernetes continues to add building blocks for creating the most flexible, robust, and efficient platforms for AI/ML workloads.

Kubernetes 1.35: Enhanced Debugging with Versioned z-pages APIs

Wed, 31 Dec 2025 10:30:00 -0800

Debugging Kubernetes control plane components can be challenging, especially when you need to quickly understand the runtime state of a component or verify its configuration. With Kubernetes 1.35, we're enhancing the z-pages debugging endpoints with structured, machine-parseable responses that make it easier to build tooling and automate troubleshooting workflows.

What are z-pages?

z-pages are special debugging endpoints exposed by Kubernetes control plane components. Introduced as an alpha feature in Kubernetes 1.32, these endpoints provide runtime diagnostics for components like kube-apiserver, kube-controller-manager, kube-scheduler, kubelet and kube-proxy. The name "z-pages" comes from the convention of using /*z paths for debugging endpoints.

Kubernetes v1.35: Watch Based Route Reconciliation in the Cloud Controller Manager

Tue, 30 Dec 2025 10:30:00 -0800

Up to and including Kubernetes v1.34, the route controller in Cloud Controller Manager (CCM) implementations built using the k8s.io/cloud-provider library reconciles routes at a fixed interval. This causes unnecessary API requests to the cloud provider when there are no changes to routes. Other controllers implemented through the same library already use watch-based mechanisms, leveraging informers to avoid unnecessary API calls. A new feature gate is being introduced in v1.35 to allow changing the behavior of the route controller to use watch-based informers.

Kubernetes v1.35: Introducing Workload Aware Scheduling

Mon, 29 Dec 2025 10:30:00 -0800

Scheduling large workloads is a much more complex and fragile operation than scheduling a single Pod, as it often requires considering all Pods together instead of scheduling each one independently. For example, when scheduling a machine learning batch job, you often need to place each worker strategically, such as on the same rack, to make the entire process as efficient as possible. At the same time, the Pods that are part of such a workload are very often identical from the scheduling perspective, which fundamentally changes how this process should look.

Kubernetes v1.35: Fine-grained Supplemental Groups Control Graduates to GA

Tue, 23 Dec 2025 10:30:00 -0800

On behalf of Kubernetes SIG Node, we are pleased to announce the graduation of fine-grained supplemental groups control to General Availability (GA) in Kubernetes v1.35!

The new Pod field, supplementalGroupsPolicy, was introduced as an opt-in alpha feature for Kubernetes v1.31, and then had graduated to beta in v1.33. Now, the feature is generally available. This feature allows you to implement more precise control over supplemental groups in Linux containers that can strengthen the security posture particularly in accessing volumes. Moreover, it also enhances the transparency of UID/GID details in containers, offering improved security oversight.

Kubernetes v1.35: Kubelet Configuration Drop-in Directory Graduates to GA

Mon, 22 Dec 2025 10:30:00 -0800

With the recent v1.35 release of Kubernetes, support for a kubelet configuration drop-in directory is generally available. The newly stable feature simplifies the management of kubelet configuration across large, heterogeneous clusters.

With v1.35, the kubelet command line argument --config-dir is production-ready and fully supported, allowing you to specify a directory containing kubelet configuration drop-in files. All files in that directory will be automatically merged with your main kubelet configuration. This allows cluster administrators to maintain a cohesive base configuration for kubelets while enabling targeted customizations for different node groups or use cases, and without complex tooling or manual configuration management.

Avoiding Zombie Cluster Members When Upgrading to etcd v3.6

Sun, 21 Dec 2025 00:00:00 +0000

The key takeaway? Always upgrade to etcd v3.5.26 or later before moving to v3.6. This ensures your cluster is automatically repaired, and avoids zombie members.

Kubernetes 1.35: In-Place Pod Resize Graduates to Stable

Fri, 19 Dec 2025 10:30:00 -0800

This release marks a major step: more than 6 years after its initial conception, the In-Place Pod Resize feature (also known as In-Place Pod Vertical Scaling), first introduced as alpha in Kubernetes v1.27, and graduated to beta in Kubernetes v1.33, is now stable (GA) in Kubernetes 1.35!

This graduation is a major milestone for improving resource efficiency and flexibility for workloads running on Kubernetes.

What is in-place Pod Resize?

In the past, the CPU and memory resources allocated to a container in a Pod were immutable. This meant changing them required deleting and recreating the entire Pod. For stateful services, batch jobs, or latency-sensitive workloads, this was an incredibly disruptive operation.

Kubernetes v1.35: Job Managed By Goes GA

Thu, 18 Dec 2025 10:30:00 -0800

In Kubernetes v1.35, the ability to specify an external Job controller (through .spec.managedBy) graduates to General Availability.

This feature allows external controllers to take full responsibility for Job reconciliation, unlocking powerful scheduling patterns like multi-cluster dispatching with MultiKueue.

Why delegate Job reconciliation?

The primary motivation for this feature is to support multi-cluster batch scheduling architectures, such as MultiKueue.

The MultiKueue architecture distinguishes between a Management Cluster and a pool of Worker Clusters:

Kubernetes v1.35: Timbernetes (The World Tree Release)

Wed, 17 Dec 2025 10:30:00 -0800

Editors: Aakanksha Bhende, Arujjwal Negi, Chad M. Crowell, Graziano Casto, Swathi Rao

Similar to previous releases, the release of Kubernetes v1.35 introduces new stable, beta, and alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 60 enhancements, including 17 stable, 19 beta, and 22 alpha features.

There are also some deprecations and removals in this release; make sure to read about those.

Kubernetes v1.35 Sneak Peek

Wed, 26 Nov 2025 00:00:00 +0000

As the release of Kubernetes v1.35 approaches, the Kubernetes project continues to evolve. Features may be deprecated, removed, or replaced to improve the project's overall health. This blog post outlines planned changes for the v1.35 release that the release team believes you should be aware of to ensure the continued smooth operation of your Kubernetes cluster(s), and to keep you up to date with the latest developments. The information below is based on the current status of the v1.35 release and is subject to change before the final release date.

Kubernetes Configuration Good Practices

Tue, 25 Nov 2025 00:00:00 +0000

Configuration is one of those things in Kubernetes that seems small until it's not. Configuration is at the heart of every Kubernetes workload. A missing quote, a wrong API version or a misplaced YAML indent can ruin your entire deploy.

This blog brings together tried-and-tested configuration best practices. The small habits that make your Kubernetes setup clean, consistent and easier to manage. Whether you are just starting out or already deploying apps daily, these are the little things that keep your cluster stable and your future self sane.

Ingress NGINX Retirement: What You Need to Know

Tue, 11 Nov 2025 10:30:00 -0800

To prioritize the safety and security of the ecosystem, Kubernetes SIG Network and the Security Response Committee are announcing the upcoming retirement of Ingress NGINX. Best-effort maintenance will continue until March 2026. Afterward, there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered. Existing deployments of Ingress NGINX will continue to function and installation artifacts will remain available.

We recommend migrating to one of the many alternatives. Consider migrating to Gateway API, the modern replacement for Ingress. If you must continue using Ingress, many alternative Ingress controllers are listed in the Kubernetes documentation. Continue reading for further information about the history and current state of Ingress NGINX, as well as next steps.

Announcing the 2025 Steering Committee Election Results

Sun, 09 Nov 2025 15:10:00 -0500

The 2025 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2025. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

The Steering Committee oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their charter.

Gateway API 1.4: New Features

Thu, 06 Nov 2025 09:00:00 -0800

Ready to rock your Kubernetes networking? The Kubernetes SIG Network community presented the General Availability (GA) release of Gateway API (v1.4.0)! Released on October 6, 2025, version 1.4.0 reinforces the path for modern, expressive, and extensible service networking in Kubernetes.

Gateway API v1.4.0 brings three new features to the Standard channel (Gateway API's GA release channel):

BackendTLSPolicy for TLS between gateways and backends
supportedFeatures in GatewayClass status
Named rules for Routes

and introduces three new experimental features:

7 Common Kubernetes Pitfalls (and How I Learned to Avoid Them)

Mon, 20 Oct 2025 08:30:00 -0700

It’s no secret that Kubernetes can be both powerful and frustrating at times. When I first started dabbling with container orchestration, I made more than my fair share of mistakes enough to compile a whole list of pitfalls. In this post, I want to walk through seven big gotchas I’ve encountered (or seen others run into) and share some tips on how to avoid them. Whether you’re just kicking the tires on Kubernetes or already managing production clusters, I hope these insights help you steer clear of a little extra stress.

Spotlight on Policy Working Group

Sat, 18 Oct 2025 00:00:00 +0000

(Note: The Policy Working Group has completed its mission and is no longer active. This article reflects its work, accomplishments, and insights into how a working group operates.)

In the complex world of Kubernetes, policies play a crucial role in managing and securing clusters. But have you ever wondered how these policies are developed, implemented, and standardized across the Kubernetes ecosystem? To answer that, let's take a look back at the work of the Policy Working Group.

Introducing Headlamp Plugin for Karpenter - Scaling and Visibility

Mon, 06 Oct 2025 00:00:00 +0000

Headlamp is an open‑source, extensible Kubernetes SIG UI project designed to let you explore, manage, and debug cluster resources.

Karpenter is a Kubernetes Autoscaling SIG node provisioning project that helps clusters scale quickly and efficiently. It launches new nodes in seconds, selects appropriate instance types for workloads, and manages the full node lifecycle, including scale-down.

The new Headlamp Karpenter Plugin adds real-time visibility into Karpenter’s activity directly from the Headlamp UI. It shows how Karpenter resources relate to Kubernetes objects, displays live metrics, and surfaces scaling events as they happen. You can inspect pending pods during provisioning, review scaling decisions, and edit Karpenter-managed resources with built-in validation. The Karpenter plugin was made as part of a LFX mentor project.

Announcing Changed Block Tracking API support (alpha)

Thu, 25 Sep 2025 05:00:00 -0800

We're excited to announce the alpha support for a changed block tracking mechanism. This enhances the Kubernetes storage ecosystem by providing an efficient way for CSI storage drivers to identify changed blocks in PersistentVolume snapshots. With a driver that can use the feature, you could benefit from faster and more resource-efficient backup operations.

If you're eager to try this feature, you can skip to the Getting Started section.

What is changed block tracking?

Changed block tracking enables storage systems to identify and track modifications at the block level between snapshots, eliminating the need to scan entire volumes during backup operations. The improvement is a change to the Container Storage Interface (CSI), and also to the storage support in Kubernetes itself. With the alpha feature enabled, your cluster can:

Kubernetes v1.34: Pod Level Resources Graduated to Beta

Mon, 22 Sep 2025 10:30:00 -0800

On behalf of the Kubernetes community, I am thrilled to announce that the Pod Level Resources feature has graduated to Beta in the Kubernetes v1.34 release and is enabled by default! This significant milestone introduces a new layer of flexibility for defining and managing resource allocation for your Pods. This flexibility stems from the ability to specify CPU and memory resources for the Pod as a whole. Pod level resources can be combined with the container-level specifications to express the exact resource requirements and limits your application needs.

Kubernetes v1.34: Recovery From Volume Expansion Failure (GA)

Fri, 19 Sep 2025 10:30:00 -0800

Have you ever made a typo when expanding your persistent volumes in Kubernetes? Meant to specify 2TB but specified 20TiB? This seemingly innocuous problem was kinda hard to fix - and took the project almost 5 years to fix. Automated recovery from storage expansion has been around for a while in beta; however, with the v1.34 release, we have graduated this to general availability.

While it was always possible to recover from failing volume expansions manually, it usually required cluster-admin access and was tedious to do (See aformentioned link for more information).

Kubernetes v1.34: DRA Consumable Capacity

Thu, 18 Sep 2025 10:30:00 -0800

Dynamic Resource Allocation (DRA) is a Kubernetes API for managing scarce resources across Pods and containers. It enables flexible resource requests, going beyond simply allocating N number of devices to support more granular usage scenarios. With DRA, users can request specific types of devices based on their attributes, define custom configurations tailored to their workloads, and even share the same resource among multiple containers or Pods.

In this blog, we focus on the device sharing feature and dive into a new capability introduced in Kubernetes 1.34: DRA consumable capacity, which extends DRA to support finer-grained device sharing.

Kubernetes v1.34: Pods Report DRA Resource Health

Wed, 17 Sep 2025 10:30:00 -0800

The rise of AI/ML and other high-performance workloads has made specialized hardware like GPUs, TPUs, and FPGAs a critical component of many Kubernetes clusters. However, as discussed in a previous blog post about navigating failures in Pods with devices, when this hardware fails, it can be difficult to diagnose, leading to significant downtime. With the release of Kubernetes v1.34, we are excited to announce a new alpha feature that brings much-needed visibility into the health of these devices.

Kubernetes v1.34: Moving Volume Group Snapshots to v1beta2

Tue, 16 Sep 2025 10:30:00 -0800

Volume group snapshots were introduced as an Alpha feature with the Kubernetes 1.27 release and moved to Beta in the Kubernetes 1.32 release. The recent release of Kubernetes v1.34 moved that support to a second beta. The support for volume group snapshots relies on a set of extension APIs for group snapshots. These APIs allow users to take crash consistent snapshots for a set of volumes. Behind the scenes, Kubernetes uses a label selector to group multiple PersistentVolumeClaims for snapshotting. A key aim is to allow you restore that set of snapshots to new volumes and recover your workload based on a crash consistent recovery point.

Kubernetes v1.34: Decoupled Taint Manager Is Now Stable

Mon, 15 Sep 2025 10:30:00 -0800

This enhancement separates the responsibility of managing node lifecycle and pod eviction into two distinct components. Previously, the node lifecycle controller handled both marking nodes as unhealthy with NoExecute taints and evicting pods from them. Now, a dedicated taint eviction controller manages the eviction process, while the node lifecycle controller focuses solely on applying taints. This separation not only improves code organization but also makes it easier to improve taint eviction controller or build custom implementations of the taint based eviction.

Kubernetes v1.34: Autoconfiguration for Node Cgroup Driver Goes GA

Fri, 12 Sep 2025 10:30:00 -0800

Historically, configuring the correct cgroup driver has been a pain point for users running new Kubernetes clusters. On Linux systems, there are two different cgroup drivers: cgroupfs and systemd. In the past, both the kubelet and CRI implementation (like CRI-O or containerd) needed to be configured to use the same cgroup driver, or else the kubelet would misbehave without any explicit error message. This was a source of headaches for many cluster admins. Now, we've (almost) arrived at the end of that headache.

Kubernetes v1.34: Mutable CSI Node Allocatable Graduates to Beta

Thu, 11 Sep 2025 10:30:00 -0800

The functionality for CSI drivers to update information about attachable volume count on the nodes, first introduced as Alpha in Kubernetes v1.33, has graduated to Beta in the Kubernetes v1.34 release! This marks a significant milestone in enhancing the accuracy of stateful pod scheduling by reducing failures due to outdated attachable volume capacity information.

Background

Traditionally, Kubernetes CSI drivers report a static maximum volume attachment limit when initializing. However, actual attachment capacities can change during a node's lifecycle for various reasons, such as:

Kubernetes v1.34: Use An Init Container To Define App Environment Variables

Wed, 10 Sep 2025 10:30:00 -0800

Kubernetes typically uses ConfigMaps and Secrets to set environment variables, which introduces additional API calls and complexity, For example, you need to separately manage the Pods of your workloads and their configurations, while ensuring orderly updates for both the configurations and the workload Pods.

Alternatively, you might be using a vendor-supplied container that requires environment variables (such as a license key or a one-time token), but you don’t want to hard-code them or mount volumes just to get the job done.

Kubernetes v1.34: Snapshottable API server cache

Tue, 09 Sep 2025 10:30:00 -0800

For years, the Kubernetes community has been on a mission to improve the stability and performance predictability of the API server. A major focus of this effort has been taming list requests, which have historically been a primary source of high memory usage and heavy load on the etcd datastore. With each release, we've chipped away at the problem, and today, we're thrilled to announce the final major piece of this puzzle.

Kubernetes v1.34: VolumeAttributesClass for Volume Modification GA

Mon, 08 Sep 2025 10:30:00 -0800

The VolumeAttributesClass API, which empowers users to dynamically modify volume attributes, has officially graduated to General Availability (GA) in Kubernetes v1.34. This marks a significant milestone, providing a robust and stable way to tune your persistent storage directly within Kubernetes.

What is VolumeAttributesClass?

At its core, VolumeAttributesClass is a cluster-scoped resource that defines a set of mutable parameters for a volume. Think of it as a "profile" for your storage, allowing cluster administrators to expose different quality-of-service (QoS) levels or performance tiers.

Kubernetes v1.34: Pod Replacement Policy for Jobs Goes GA

Fri, 05 Sep 2025 10:30:00 -0800

In Kubernetes v1.34, the Pod replacement policy feature has reached general availability (GA). This blog post describes the Pod replacement policy feature and how to use it in your Jobs.

About Pod Replacement Policy

By default, the Job controller immediately recreates Pods as soon as they fail or begin terminating (when they have a deletion timestamp).

As a result, while some Pods are terminating, the total number of running Pods for a Job can temporarily exceed the specified parallelism. For Indexed Jobs, this can even mean multiple Pods running for the same index at the same time.

Kubernetes v1.34: PSI Metrics for Kubernetes Graduates to Beta

Thu, 04 Sep 2025 10:30:00 -0800

As Kubernetes clusters grow in size and complexity, understanding the health and performance of individual nodes becomes increasingly critical. We are excited to announce that as of Kubernetes v1.34, Pressure Stall Information (PSI) Metrics has graduated to Beta.

What is Pressure Stall Information (PSI)?

Pressure Stall Information (PSI) is a feature of the Linux kernel (version 4.20 and later) that provides a canonical way to quantify pressure on infrastructure resources, in terms of whether demand for a resource exceeds current supply. It moves beyond simple resource utilization metrics and instead measures the amount of time that tasks are stalled due to resource contention. This is a powerful way to identify and diagnose resource bottlenecks that can impact application performance.

Kubernetes v1.34: Service Account Token Integration for Image Pulls Graduates to Beta

Wed, 03 Sep 2025 10:30:00 -0800

The Kubernetes community continues to advance security best practices by reducing reliance on long-lived credentials. Following the successful alpha release in Kubernetes v1.33, Service Account Token Integration for Kubelet Credential Providers has now graduated to beta in Kubernetes v1.34, bringing us closer to eliminating long-lived image pull secrets from Kubernetes clusters.

This enhancement allows credential providers to use workload-specific service account tokens to obtain registry credentials, providing a secure, ephemeral alternative to traditional image pull secrets.

Kubernetes v1.34: Introducing CPU Manager Static Policy Option for Uncore Cache Alignment

Tue, 02 Sep 2025 10:30:00 -0800

A new CPU Manager Static Policy Option called prefer-align-cpus-by-uncorecache was introduced in Kubernetes v1.32 as an alpha feature, and has graduated to beta in Kubernetes v1.34. This CPU Manager Policy Option is designed to optimize performance for specific workloads running on processors with a split uncore cache architecture. In this article, I'll explain what that means and why it's useful.

Understanding the feature

What is uncore cache?

Until relatively recently, nearly all mainstream computer processors had a monolithic last-level-cache cache that was shared across every core in a multiple CPU package. This monolithic cache is also referred to as uncore cache (because it is not linked to a specific core), or as Level 3 cache. As well as the Level 3 cache, there is other cache, commonly called Level 1 and Level 2 cache, that is associated with a specific CPU core.

Kubernetes v1.34: DRA has graduated to GA

Mon, 01 Sep 2025 10:30:00 -0800

Kubernetes 1.34 is here, and it has brought a huge wave of enhancements for Dynamic Resource Allocation (DRA)! This release marks a major milestone with many APIs in the resource.k8s.io group graduating to General Availability (GA), unlocking the full potential of how you manage devices on Kubernetes. On top of that, several key features have moved to beta, and a fresh batch of new alpha features promise even more expressiveness and flexibility.

Kubernetes v1.34: Finer-Grained Control Over Container Restarts

Fri, 29 Aug 2025 10:30:00 -0800

With the release of Kubernetes 1.34, a new alpha feature is introduced that gives you more granular control over container restarts within a Pod. This feature, named Container Restart Policy and Rules, allows you to specify a restart policy for each container individually, overriding the Pod's global restart policy. In addition, it also allows you to conditionally restart individual containers based on their exit codes. This feature is available behind the alpha feature gate ContainerRestartRules.

Kubernetes v1.34: User preferences (kuberc) are available for testing in kubectl 1.34

Thu, 28 Aug 2025 10:30:00 -0800

Have you ever wished you could enable interactive delete, by default, in kubectl? Or maybe, you'd like to have custom aliases defined, but not necessarily generate hundreds of them manually? Look no further. SIG-CLI has been working hard to add user preferences to kubectl, and we are happy to announce that this functionality is reaching beta as part of the Kubernetes v1.34 release.

How it works

A full description of this functionality is available in our official documentation, but this blog post will answer both of the questions from the beginning of this article.

Kubernetes v1.34: Of Wind & Will (O' WaW)

Wed, 27 Aug 2025 10:30:00 -0800

Editors: Agustina Barbetta, Alejandro Josue Leon Bellido, Graziano Casto, Melony Qin, Dipesh Rawat

Similar to previous releases, the release of Kubernetes v1.34 introduces new stable, beta, and alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 58 enhancements. Of those enhancements, 23 have graduated to Stable, 22 have entered Beta, and 13 have entered Alpha.

Tuning Linux Swap for Kubernetes: A Deep Dive

Tue, 19 Aug 2025 10:30:00 -0800

The Kubernetes NodeSwap feature, likely to graduate to stable in the upcoming Kubernetes v1.34 release, allows swap usage: a significant shift from the conventional practice of disabling swap for performance predictability. This article focuses exclusively on tuning swap on Linux nodes, where this feature is available. By allowing Linux nodes to use secondary storage for additional virtual memory when physical RAM is exhausted, node swap support aims to improve resource utilization and reduce out-of-memory (OOM) kills.

Introducing Headlamp AI Assistant

Thu, 07 Aug 2025 20:00:00 +0100

This announcement originally appeared on the Headlamp blog.

To simplify Kubernetes management and troubleshooting, we're thrilled to introduce Headlamp AI Assistant: a powerful new plugin for Headlamp that helps you understand and operate your Kubernetes clusters and applications with greater clarity and ease.

Whether you're a seasoned engineer or just getting started, the AI Assistant offers:

Fast time to value: Ask questions like "Is my application healthy?" or "How can I fix this?" without needing deep Kubernetes knowledge.
Deep insights: Start with high-level queries and dig deeper with prompts like "List all the problematic pods" or "How can I fix this pod?"
Focused & relevant: Ask questions in the context of what you're viewing in the UI, such as "What's wrong here?"
Action-oriented: Let the AI take action for you, like "Restart that deployment", with your permission.

Here is a demo of the AI Assistant in action as it helps troubleshoot an application running with issues in a Kubernetes cluster:

Kubernetes v1.34 Sneak Peek

Mon, 28 Jul 2025 00:00:00 +0000

Kubernetes v1.34 is coming at the end of August 2025. This release will not include any removal or deprecation, but it is packed with an impressive number of enhancements. Here are some of the features we are most excited about in this cycle!

Please note that this information reflects the current state of v1.34 development and may change before release.

Featured enhancements of Kubernetes v1.34

The following list highlights some of the notable enhancements likely to be included in the v1.34 release, but is not an exhaustive list of all planned changes. This is not a commitment and the release content is subject to change.

Post-Quantum Cryptography in Kubernetes

Fri, 18 Jul 2025 00:00:00 +0000

The world of cryptography is on the cusp of a major shift with the advent of quantum computing. While powerful quantum computers are still largely theoretical for many applications, their potential to break current cryptographic standards is a serious concern, especially for long-lived systems. This is where Post-Quantum Cryptography (PQC) comes in. In this article, I'll dive into what PQC means for TLS and, more specifically, for the Kubernetes ecosystem. I'll explain what the (suprising) state of PQC in Kubernetes is and what the implications are for current and future clusters.

Navigating Failures in Pods With Devices

Thu, 03 Jul 2025 00:00:00 +0000

Kubernetes is the de facto standard for container orchestration, but when it comes to handling specialized hardware like GPUs and other accelerators, things get a bit complicated. This blog post dives into the challenges of managing failure modes when operating pods with devices in Kubernetes, based on insights from Sergey Kanzhelev and Mrunal Patel's talk at KubeCon NA 2024. You can follow the links to slides and recording.

The AI/ML boom and its impact on Kubernetes

The rise of AI/ML workloads has brought new challenges to Kubernetes. These workloads often rely heavily on specialized hardware, and any device failure can significantly impact performance and lead to frustrating interruptions. As highlighted in the 2024 Llama paper, hardware issues, particularly GPU failures, are a major cause of disruption in AI/ML training. You can also learn how much effort NVIDIA spends on handling devices failures and maintenance in the KubeCon talk by Ryan Hallisey and Piotr Prokop All-Your-GPUs-Are-Belong-to-Us: An Inside Look at NVIDIA's Self-Healing GeForce NOW Infrastructure (recording) as they see 19 remediation requests per 1000 nodes a day! We also see data centers offering spot consumption models and overcommit on power, making device failures commonplace and a part of the business model.

Image Compatibility In Cloud Native Environments

Wed, 25 Jun 2025 00:00:00 +0000

In industries where systems must run very reliably and meet strict performance criteria such as telecommunication, high-performance or AI computing, containerized applications often need specific operating system configuration or hardware presence. It is common practice to require the use of specific versions of the kernel, its configuration, device drivers, or system components. Despite the existence of the Open Container Initiative (OCI), a governing community to define standards and specifications for container images, there has been a gap in expression of such compatibility requirements. The need to address this issue has led to different proposals and, ultimately, an implementation in Kubernetes' Node Feature Discovery (NFD).

Changes to Kubernetes Slack

Mon, 16 Jun 2025 00:00:00 +0000

UPDATE: We’ve received notice from Salesforce that our Slack workspace WILL NOT BE DOWNGRADED on June 20th. Stand by for more details, but for now, there is no urgency to back up private channels or direct messages.

~~Kubernetes Slack will lose its special status and will be changing into a standard free Slack on June 20, 2025~~. Sometime later this year, our community may move to a new platform. If you are responsible for a channel or private channel, or a member of a User Group, you will need to take some actions as soon as you can.

Enhancing Kubernetes Event Management with Custom Aggregation

Tue, 10 Jun 2025 00:00:00 +0000

Kubernetes Events provide crucial insights into cluster operations, but as clusters grow, managing and analyzing these events becomes increasingly challenging. This blog post explores how to build custom event aggregation systems that help engineering teams better understand cluster behavior and troubleshoot issues more effectively.

The challenge with Kubernetes events

In a Kubernetes cluster, events are generated for various operations - from pod scheduling and container starts to volume mounts and network configurations. While these events are invaluable for debugging and monitoring, several challenges emerge in production environments:

Introducing Gateway API Inference Extension

Thu, 05 Jun 2025 00:00:00 +0000

Modern generative AI and large language model (LLM) services create unique traffic-routing challenges on Kubernetes. Unlike typical short-lived, stateless web requests, LLM inference sessions are often long-running, resource-intensive, and partially stateful. For example, a single GPU-backed model server may keep multiple inference sessions active and maintain in-memory token caches.

Traditional load balancers focused on HTTP path or round-robin lack the specialized capabilities needed for these workloads. They also don’t account for model identity or request criticality (e.g., interactive chat vs. batch jobs). Organizations often patch together ad-hoc solutions, but a standardized approach is missing.

Start Sidecar First: How To Avoid Snags

Tue, 03 Jun 2025 00:00:00 +0000

From the Kubernetes Multicontainer Pods: An Overview blog post you know what their job is, what are the main architectural patterns, and how they are implemented in Kubernetes. The main thing I’ll cover in this article is how to ensure that your sidecar containers start before the main app. It’s more complicated than you might think!

A gentle refresher

I'd just like to remind readers that the v1.29.0 release of Kubernetes added native support for sidecar containers, which can now be defined within the .spec.initContainers field, but with restartPolicy: Always. You can see that illustrated in the following example Pod manifest snippet:

Gateway API v1.3.0: Advancements in Request Mirroring, CORS, Gateway Merging, and Retry Budgets

Mon, 02 Jun 2025 09:00:00 -0800

Join us in the Kubernetes SIG Network community in celebrating the general availability of Gateway API v1.3.0! We are also pleased to announce that there are already a number of conformant implementations to try, made possible by postponing this blog announcement. Version 1.3.0 of the API was released about a month ago on April 24, 2025.

Gateway API v1.3.0 brings a new feature to the Standard channel (Gateway API's GA release channel): percentage-based request mirroring, and introduces three new experimental features: cross-origin resource sharing (CORS) filters, a standardized mechanism for listener and gateway merging, and retry budgets.

Kubernetes v1.33: In-Place Pod Resize Graduated to Beta

Fri, 16 May 2025 10:30:00 -0800

On behalf of the Kubernetes project, I am excited to announce that the in-place Pod resize feature (also known as In-Place Pod Vertical Scaling), first introduced as alpha in Kubernetes v1.27, has graduated to Beta and will be enabled by default in the Kubernetes v1.33 release! This marks a significant milestone in making resource management for Kubernetes workloads more flexible and less disruptive.

What is in-place Pod resize?

Traditionally, changing the CPU or memory resources allocated to a container required restarting the Pod. While acceptable for many stateless applications, this could be disruptive for stateful services, batch jobs, or any workloads sensitive to restarts.

Announcing etcd v3.6.0

Thu, 15 May 2025 16:00:00 -0800

This announcement originally appeared on the etcd blog.

Today, we are releasing etcd v3.6.0, the first minor release since etcd v3.5.0 on June 15, 2021. This release introduces several new features, makes significant progress on long-standing efforts like downgrade support and migration to v3store, and addresses numerous critical & major issues. It also includes major optimizations in memory usage, improving efficiency and performance.

In addition to the features of v3.6.0, etcd has joined Kubernetes as a SIG (sig-etcd), enabling us to improve project sustainability. We've introduced systematic robustness testing to ensure correctness and reliability. Through the etcd-operator Working Group, we plan to improve usability as well.

Kubernetes 1.33: Job's SuccessPolicy Goes GA

Thu, 15 May 2025 10:30:00 -0800

On behalf of the Kubernetes project, I'm pleased to announce that Job success policy has graduated to General Availability (GA) as part of the v1.33 release.

About Job's Success Policy

In batch workloads, you might want to use leader-follower patterns like MPI, in which the leader controls the execution, including the followers' lifecycle.

In this case, you might want to mark it as succeeded even if some of the indexes failed. Unfortunately, a leader-follower Kubernetes Job that didn't use a success policy, in most cases, would have to require all Pods to finish successfully for that Job to reach an overall succeeded state.

Kubernetes v1.33: Updates to Container Lifecycle

Wed, 14 May 2025 10:30:00 -0800

Kubernetes v1.33 introduces a few updates to the lifecycle of containers. The Sleep action for container lifecycle hooks now supports a zero sleep duration (feature enabled by default). There is also alpha support for customizing the stop signal sent to containers when they are being terminated.

This blog post goes into the details of these new aspects of the container lifecycle, and how you can use them.

Zero value for Sleep action

Kubernetes v1.29 introduced the Sleep action for container PreStop and PostStart Lifecycle hooks. The Sleep action lets your containers pause for a specified duration after the container is started or before it is terminated. This was needed to provide a straightforward way to manage graceful shutdowns. Before the Sleep action, folks used to run the sleep command using the exec action in their container lifecycle hooks. If you wanted to do this you'd need to have the binary for the sleep command in your container image. This is difficult if you're using third party images.

Kubernetes v1.33: Job's Backoff Limit Per Index Goes GA

Tue, 13 May 2025 10:30:00 -0800

In Kubernetes v1.33, the Backoff Limit Per Index feature reaches general availability (GA). This blog describes the Backoff Limit Per Index feature and its benefits.

About backoff limit per index

When you run workloads on Kubernetes, you must consider scenarios where Pod failures can affect the completion of your workloads. Ideally, your workload should tolerate transient failures and continue running.

To achieve failure tolerance in a Kubernetes Job, you can set the spec.backoffLimit field. This field specifies the total number of tolerated failures.

Kubernetes v1.33: Image Pull Policy the way you always thought it worked!

Mon, 12 May 2025 10:30:00 -0800

Image Pull Policy the way you always thought it worked!

Some things in Kubernetes are surprising, and the way imagePullPolicy behaves might be one of them. Given Kubernetes is all about running pods, it may be peculiar to learn that there has been a caveat to restricting pod access to authenticated images for over 10 years in the form of issue 18787! It is an exciting release when you can resolve a ten-year-old issue.

Kubernetes v1.33: Streaming List responses

Fri, 09 May 2025 10:30:00 -0800

Managing Kubernetes cluster stability becomes increasingly critical as your infrastructure grows. One of the most challenging aspects of operating large-scale clusters has been handling List requests that fetch substantial datasets - a common operation that could unexpectedly impact your cluster's stability.

Today, the Kubernetes community is excited to announce a significant architectural improvement: streaming encoding for List responses.

The problem: unnecessary memory consumption with large resources

Current API response encoders just serialize an entire response into a single contiguous memory and perform one ResponseWriter.Write call to transmit data to the client. Despite HTTP/2's capability to split responses into smaller frames for transmission, the underlying HTTP server continues to hold the complete response data as a single buffer. Even as individual frames are transmitted to the client, the memory associated with these frames cannot be freed incrementally.

Kubernetes 1.33: Volume Populators Graduate to GA

Thu, 08 May 2025 10:30:00 -0800

Kubernetes volume populators are now generally available (GA)! The AnyVolumeDataSource feature gate is treated as always enabled for Kubernetes v1.33, which means that users can specify any appropriate custom resource as the data source of a PersistentVolumeClaim (PVC).

An example of how to use dataSourceRef in PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: pvc1
spec:
 ...
 dataSourceRef:
 apiGroup: provider.example.com
 kind: Provider
 name: provider1

What is new

There are four major enhancements from beta.

Kubernetes v1.33: From Secrets to Service Accounts: Kubernetes Image Pulls Evolved

Wed, 07 May 2025 10:30:00 -0800

Kubernetes has steadily evolved to reduce reliance on long-lived credentials stored in the API. A prime example of this shift is the transition of Kubernetes Service Account (KSA) tokens from long-lived, static tokens to ephemeral, automatically rotated tokens with OpenID Connect (OIDC)-compliant semantics. This advancement enables workloads to securely authenticate with external services without needing persistent secrets.

However, one major gap remains: image pull authentication. Today, Kubernetes clusters rely on image pull secrets stored in the API, which are long-lived and difficult to rotate, or on node-level kubelet credential providers, which allow any pod running on a node to access the same credentials. This presents security and operational challenges.

Kubernetes v1.33: Fine-grained SupplementalGroups Control Graduates to Beta

Tue, 06 May 2025 10:30:00 -0800

The new field, supplementalGroupsPolicy, was introduced as an opt-in alpha feature for Kubernetes v1.31 and has graduated to beta in v1.33; the corresponding feature gate (SupplementalGroupsPolicy) is now enabled by default. This feature enables to implement more precise control over supplemental groups in containers that can strengthen the security posture, particularly in accessing volumes. Moreover, it also enhances the transparency of UID/GID details in containers, offering improved security oversight.

Please be aware that this beta release contains some behavioral breaking change. See The Behavioral Changes Introduced In Beta and Upgrade Considerations sections for details.

Kubernetes v1.33: Prevent PersistentVolume Leaks When Deleting out of Order graduates to GA

Mon, 05 May 2025 10:30:00 -0800

I am thrilled to announce that the feature to prevent PersistentVolume (or PVs for short) leaks when deleting out of order has graduated to General Availability (GA) in Kubernetes v1.33! This improvement, initially introduced as a beta feature in Kubernetes v1.31, ensures that your storage resources are properly reclaimed, preventing unwanted leaks.

How did reclaim work in previous Kubernetes releases?

PersistentVolumeClaim (or PVC for short) is a user's request for storage. A PV and PVC are considered Bound if a newly created PV or a matching PV is found. The PVs themselves are backed by volumes allocated by the storage backend.

Kubernetes v1.33: Mutable CSI Node Allocatable Count

Fri, 02 May 2025 10:30:00 -0800

Scheduling stateful applications reliably depends heavily on accurate information about resource availability on nodes. Kubernetes v1.33 introduces an alpha feature called mutable CSI node allocatable count, allowing Container Storage Interface (CSI) drivers to dynamically update the reported maximum number of volumes that a node can handle. This capability significantly enhances the accuracy of pod scheduling decisions and reduces scheduling failures caused by outdated volume capacity information.

Background

Kubernetes v1.33: New features in DRA

Thu, 01 May 2025 10:30:00 -0800

Kubernetes Dynamic Resource Allocation (DRA) was originally introduced as an alpha feature in the v1.26 release, and then went through a significant redesign for Kubernetes v1.31. The main DRA feature went to beta in v1.32, and the project hopes it will be generally available in Kubernetes v1.34.

The basic feature set of DRA provides a far more powerful and flexible API for requesting devices than Device Plugin. And while DRA remains a beta feature for v1.33, the DRA team has been hard at work implementing a number of new features and UX improvements. One feature has been promoted to beta, while a number of new features have been added in alpha. The team has also made progress towards getting DRA ready for GA.

Kubernetes v1.33: Storage Capacity Scoring of Nodes for Dynamic Provisioning (alpha)

Wed, 30 Apr 2025 10:30:00 -0800

Kubernetes v1.33 introduces a new alpha feature called StorageCapacityScoring. This feature adds a scoring method for pod scheduling with the topology-aware volume provisioning. This feature eases to schedule pods on nodes with either the most or least available storage capacity.

About this feature

This feature extends the kube-scheduler's VolumeBinding plugin to perform scoring using node storage capacity information obtained from Storage Capacity. Currently, you can only filter out nodes with insufficient storage capacity. So, you have to use a scheduler extender to achieve storage-capacity-based pod scheduling.

Kubernetes v1.33: Image Volumes graduate to beta!

Tue, 29 Apr 2025 10:30:00 -0800

Image Volumes were introduced as an Alpha feature with the Kubernetes v1.31 release as part of KEP-4639. In Kubernetes v1.33, this feature graduates to beta.

Please note that the feature is still disabled by default, because not all container runtimes have full support for it. CRI-O supports the initial feature since version v1.31 and will add support for Image Volumes as beta in v1.33. containerd merged support for the alpha feature which will be part of the v2.1.0 release and is working on beta support as part of PR #11578.

Kubernetes v1.33: HorizontalPodAutoscaler Configurable Tolerance

Mon, 28 Apr 2025 10:30:00 -0800

This post describes configurable tolerance for horizontal Pod autoscaling, a new alpha feature first available in Kubernetes 1.33.

What is it?

Horizontal Pod Autoscaling is a well-known Kubernetes feature that allows your workload to automatically resize by adding or removing replicas based on resource utilization.

Let's say you have a web application running in a Kubernetes cluster with 50 replicas. You configure the HorizontalPodAutoscaler (HPA) to scale based on CPU utilization, with a target of 75% utilization. Now, imagine that the current CPU utilization across all replicas is 90%, which is higher than the desired 75%. The HPA will calculate the required number of replicas using the formula:

Kubernetes v1.33: User Namespaces enabled by default!

Fri, 25 Apr 2025 10:30:00 -0800

In Kubernetes v1.33 support for user namespaces is enabled by default. This means that, when the stack requirements are met, pods can opt-in to use user namespaces. To use the feature there is no need to enable any Kubernetes feature flag anymore!

In this blog post we answer some common questions about user namespaces. But, before we dive into that, let's recap what user namespaces are and why they are important.

Kubernetes v1.33: Continuing the transition from Endpoints to EndpointSlices

Thu, 24 Apr 2025 10:30:00 -0800

Since the addition of EndpointSlices (KEP-752) as alpha in v1.15 and later GA in v1.21, the Endpoints API in Kubernetes has been gathering dust. New Service features like dual-stack networking and traffic distribution are only supported via the EndpointSlice API, so all service proxies, Gateway API implementations, and similar controllers have had to be ported from using Endpoints to using EndpointSlices. At this point, the Endpoints API is really only there to avoid breaking end user workloads and scripts that still make use of it.

Kubernetes v1.33: Octarine

Wed, 23 Apr 2025 10:30:00 -0800

Editors: Agustina Barbetta, Aakanksha Bhende, Udi Hofesh, Ryota Sawada, Sneha Yadav

Similar to previous releases, the release of Kubernetes v1.33 introduces new stable, beta, and alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 64 enhancements. Of those enhancements, 18 have graduated to Stable, 20 are entering Beta, 24 have entered Alpha, and 2 are deprecated or withdrawn.

Kubernetes Multicontainer Pods: An Overview

Tue, 22 Apr 2025 00:00:00 +0000

As cloud-native architectures continue to evolve, Kubernetes has become the go-to platform for deploying complex, distributed systems. One of the most powerful yet nuanced design patterns in this ecosystem is the sidecar pattern—a technique that allows developers to extend application functionality without diving deep into source code.

The origins of the sidecar pattern

Think of a sidecar like a trusty companion motorcycle attachment. Historically, IT infrastructures have always used auxiliary services to handle critical tasks. Before containers, we relied on background processes and helper daemons to manage logging, monitoring, and networking. The microservices revolution transformed this approach, making sidecars a structured and intentional architectural choice. With the rise of microservices, the sidecar pattern became more clearly defined, allowing developers to offload specific responsibilities from the main service without altering its code. Service meshes like Istio and Linkerd have popularized sidecar proxies, demonstrating how these companion containers can elegantly handle observability, security, and traffic management in distributed systems.

Introducing kube-scheduler-simulator

Mon, 07 Apr 2025 00:00:00 +0000

The Kubernetes Scheduler is a crucial control plane component that determines which node a Pod will run on. Thus, anyone utilizing Kubernetes relies on a scheduler.

kube-scheduler-simulator is a simulator for the Kubernetes scheduler, that started as a Google Summer of Code 2021 project developed by me (Kensei Nakada) and later received a lot of contributions. This tool allows users to closely examine the scheduler’s behavior and decisions.

It is useful for casual users who employ scheduling constraints (for example, inter-Pod affinity) and experts who extend the scheduler with custom plugins.

Kubernetes v1.33 sneak peek

Wed, 26 Mar 2025 10:30:00 -0800

As the release of Kubernetes v1.33 approaches, the Kubernetes project continues to evolve. Features may be deprecated, removed, or replaced to improve the overall health of the project. This blog post outlines some planned changes for the v1.33 release, which the release team believes you should be aware of to ensure the continued smooth operation of your Kubernetes environment and to keep you up-to-date with the latest developments. The information below is based on the current status of the v1.33 release and is subject to change before the final release date.

Fresh Swap Features for Linux Users in Kubernetes 1.32

Tue, 25 Mar 2025 10:00:00 -0800

Swap is a fundamental and an invaluable Linux feature. It offers numerous benefits, such as effectively increasing a node’s memory by swapping out unused data, shielding nodes from system-level memory spikes, preventing Pods from crashing when they hit their memory limits, and much more. As a result, the node special interest group within the Kubernetes project has invested significant effort into supporting swap on Linux nodes.

The 1.22 release introduced Alpha support for configuring swap memory usage for Kubernetes workloads running on Linux on a per-node basis. Later, in release 1.28, support for swap on Linux nodes has graduated to Beta, along with many new improvements. In the following Kubernetes releases more improvements were made, paving the way to GA in the near future.

Ingress-nginx CVE-2025-1974: What You Need to Know

Mon, 24 Mar 2025 12:00:00 -0800

Today, the ingress-nginx maintainers have released patches for a batch of critical vulnerabilities that could make it easy for attackers to take over your Kubernetes cluster: ingress-nginx v1.12.1 and ingress-nginx v1.11.5. If you are among the over 40% of Kubernetes administrators using ingress-nginx, you should take action immediately to protect your users and data.

Background

Ingress is the traditional Kubernetes feature for exposing your workload Pods to the world so that they can be useful. In an implementation-agnostic way, Kubernetes users can define how their applications should be made available on the network. Then, an ingress controller uses that definition to set up local or cloud resources as required for the user’s particular situation and needs.

Introducing JobSet

Sun, 23 Mar 2025 00:00:00 +0000

Authors: Daniel Vega-Myhre (Google), Abdullah Gharaibeh (Google), Kevin Hannon (Red Hat)

In this article, we introduce JobSet, an open source API for representing distributed jobs. The goal of JobSet is to provide a unified API for distributed ML training and HPC workloads on Kubernetes.

Why JobSet?

The Kubernetes community’s recent enhancements to the batch ecosystem on Kubernetes has attracted ML engineers who have found it to be a natural fit for the requirements of running distributed training workloads.

Spotlight on SIG Apps

Wed, 12 Mar 2025 00:00:00 +0000

In our ongoing SIG Spotlight series, we dive into the heart of the Kubernetes project by talking to the leaders of its various Special Interest Groups (SIGs). This time, we focus on SIG Apps, the group responsible for everything related to developing, deploying, and operating applications on Kubernetes. Sandipan Panda (DevZero) had the opportunity to interview Maciej Szulik (Defense Unicorns) and Janet Kuo (Google), the chairs and tech leads of SIG Apps. They shared their experiences, challenges, and visions for the future of application management within the Kubernetes ecosystem.

Spotlight on SIG etcd

Tue, 04 Mar 2025 00:00:00 +0000

In this SIG etcd spotlight we talked with James Blair, Marek Siarkowicz, Wenjia Zhang, and Benjamin Wang to learn a bit more about this Kubernetes Special Interest Group.

Introducing SIG etcd

Frederico: Hello, thank you for the time! Let’s start with some introductions, could you tell us a bit about yourself, your role and how you got involved in Kubernetes.

Benjamin: Hello, I am Benjamin. I am a SIG etcd Tech Lead and one of the etcd maintainers. I work for VMware, which is part of the Broadcom group. I got involved in Kubernetes & etcd & CSI (Container Storage Interface) because of work and also a big passion for open source. I have been working on Kubernetes & etcd (and also CSI) since 2020.

NFTables mode for kube-proxy

Fri, 28 Feb 2025 00:00:00 +0000

A new nftables mode for kube-proxy was introduced as an alpha feature in Kubernetes 1.29. Currently in beta, it is expected to be GA as of 1.33. The new mode fixes long-standing performance problems with the iptables mode and all users running on systems with reasonably-recent kernels are encouraged to try it out. (For compatibility reasons, even once nftables becomes GA, iptables will still be the default.)

Why nftables? Part 1: data plane latency

The iptables API was designed for implementing simple firewalls, and has problems scaling up to support Service proxying in a large Kubernetes cluster with tens of thousands of Services.

The Cloud Controller Manager Chicken and Egg Problem

Fri, 14 Feb 2025 00:00:00 +0000

Kubernetes 1.31 completed the largest migration in Kubernetes history, removing the in-tree cloud provider. While the component migration is now done, this leaves some additional complexity for users and installer projects (for example, kOps or Cluster API) . We will go over those additional steps and failure points and make recommendations for cluster owners. This migration was complex and some logic had to be extracted from the core components, building four new subsystems.

Spotlight on SIG Architecture: Enhancements

Tue, 21 Jan 2025 00:00:00 +0000

This is the fourth interview of a SIG Architecture Spotlight series that will cover the different subprojects, and we will be covering SIG Architecture: Enhancements.

In this SIG Architecture spotlight we talked with Kirsten Garrison, lead of the Enhancements subproject.

The Enhancements subproject

Frederico (FSM): Hi Kirsten, very happy to have the opportunity to talk about the Enhancements subproject. Let's start with some quick information about yourself and your role.

Kirsten Garrison (KG): I’m a lead of the Enhancements subproject of SIG-Architecture and currently work at Google. I first got involved by contributing to the service-catalog project with the help of Carolyn Van Slyck. With time, I joined the Release team, eventually becoming the Enhancements Lead and a Release Lead shadow. While on the release team, I worked on some ideas to make the process better for the SIGs and Enhancements team (the opt-in process) based on my team’s experiences. Eventually, I started attending Subproject meetings and contributing to the Subproject’s work.

Kubernetes 1.32: Moving Volume Group Snapshots to Beta

Wed, 18 Dec 2024 00:00:00 +0000

Volume group snapshots were introduced as an Alpha feature with the Kubernetes 1.27 release. The recent release of Kubernetes v1.32 moved that support to beta. The support for volume group snapshots relies on a set of extension APIs for group snapshots. These APIs allow users to take crash consistent snapshots for a set of volumes. Behind the scenes, Kubernetes uses a label selector to group multiple PersistentVolumeClaims for snapshotting. A key aim is to allow you restore that set of snapshots to new volumes and recover your workload based on a crash consistent recovery point.

Enhancing Kubernetes API Server Efficiency with API Streaming

Tue, 17 Dec 2024 00:00:00 +0000

Managing Kubernetes clusters efficiently is critical, especially as their size is growing. A significant challenge with large clusters is the memory overhead caused by list requests.

In the existing implementation, the kube-apiserver processes list requests by assembling the entire response in-memory before transmitting any data to the client. But what if the response body is substantial, say hundreds of megabytes? Additionally, imagine a scenario where multiple list requests flood in simultaneously, perhaps after a brief network outage. While API Priority and Fairness has proven to reasonably protect kube-apiserver from CPU overload, its impact is visibly smaller for memory protection. This can be explained by the differing nature of resource consumption by a single API request - the CPU usage at any given time is capped by a constant, whereas memory, being uncompressible, can grow proportionally with the number of processed objects and is unbounded. This situation poses a genuine risk, potentially overwhelming and crashing any kube-apiserver within seconds due to out-of-memory (OOM) conditions. To better visualize the issue, let's consider the below graph.

Kubernetes v1.32 Adds A New CPU Manager Static Policy Option For Strict CPU Reservation

Mon, 16 Dec 2024 00:00:00 +0000

In Kubernetes v1.32, after years of community discussion, we are excited to introduce a strict-cpu-reservation option for the CPU Manager static policy. This feature is currently in alpha, with the associated policy hidden by default. You can only use the policy if you explicitly enable the alpha behavior in your cluster.

Understanding the feature

The CPU Manager static policy is used to reduce latency or improve performance. The reservedSystemCPUs defines an explicit CPU set for OS system daemons and kubernetes system daemons. This option is designed for Telco/NFV type use cases where uncontrolled interrupts/timers may impact the workload performance. you can use this option to define the explicit cpuset for the system/kubernetes daemons as well as the interrupts/timers, so the rest CPUs on the system can be used exclusively for workloads, with less impact from uncontrolled interrupts/timers. More details of this parameter can be found on the Explicitly Reserved CPU List page.

Kubernetes v1.32: Memory Manager Goes GA

Fri, 13 Dec 2024 00:00:00 +0000

With Kubernetes 1.32, the memory manager has officially graduated to General Availability (GA), marking a significant milestone in the journey toward efficient and predictable memory allocation for containerized applications. Since Kubernetes v1.22, where it graduated to beta, the memory manager has proved itself reliable, stable and a good complementary feature for the CPU Manager.

As part of kubelet's workload admission process, the memory manager provides topology hints to optimize memory allocation and alignment. This enables users to allocate exclusive memory for Pods in the Guaranteed QoS class. More details about the process can be found in the memory manager goes to beta blog.

Kubernetes v1.32: QueueingHint Brings a New Possibility to Optimize Pod Scheduling

Thu, 12 Dec 2024 00:00:00 +0000

The Kubernetes scheduler is the core component that selects the nodes on which new Pods run. The scheduler processes these new Pods one by one. Therefore, the larger your clusters, the more important the throughput of the scheduler becomes.

Over the years, Kubernetes SIG Scheduling has improved the throughput of the scheduler in multiple enhancements. This blog post describes a major improvement to the scheduler in Kubernetes v1.32: a scheduling context element named QueueingHint. This page provides background knowledge of the scheduler and explains how QueueingHint improves scheduling throughput.

Kubernetes v1.32: Penelope

Wed, 11 Dec 2024 00:00:00 +0000

Editors: Matteo Bianchi, Edith Puclla, William Rizzo, Ryota Sawada, Rashan Smith

Announcing the release of Kubernetes v1.32: Penelope!

In line with previous releases, the release of Kubernetes v1.32 introduces new stable, beta, and alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community. This release consists of 44 enhancements in total. Of those enhancements, 13 have graduated to Stable, 12 are entering Beta, and 19 have entered in Alpha.

Gateway API v1.2: WebSockets, Timeouts, Retries, and More

Thu, 21 Nov 2024 09:00:00 -0800

Kubernetes SIG Network is delighted to announce the general availability of Gateway API v1.2! This version of the API was released on October 3, and we're delighted to report that we now have a number of conformant implementations of it for you to try out.

Gateway API v1.2 brings a number of new features to the Standard channel (Gateway API's GA release channel), introduces some new experimental features, and inaugurates our new release process — but it also brings two breaking changes that you'll want to be careful of.

How we built a dynamic Kubernetes API Server for the API Aggregation Layer in Cozystack

Thu, 21 Nov 2024 00:00:00 +0000

Hi there! I'm Andrei Kvapil, but you might know me as @kvaps in communities dedicated to Kubernetes and cloud-native tools. In this article, I want to share how we implemented our own extension api-server in the open-source PaaS platform, Cozystack.

Kubernetes truly amazes me with its powerful extensibility features. You're probably already familiar with the controller concept and frameworks like kubebuilder and operator-sdk that help you implement it. In a nutshell, they allow you to extend your Kubernetes cluster by defining custom resources (CRDs) and writing additional controllers that handle your business logic for reconciling and managing these kinds of resources. This approach is well-documented, with a wealth of information available online on how to develop your own operators.

Kubernetes v1.32 sneak peek

Fri, 08 Nov 2024 00:00:00 +0000

As we get closer to the release date for Kubernetes v1.32, the project develops and matures. Features may be deprecated, removed, or replaced with better ones for the project's overall health.

This blog outlines some of the planned changes for the Kubernetes v1.32 release, that the release team feels you should be aware of, for the continued maintenance of your Kubernetes environment and keeping up to date with the latest changes. Information listed below is based on the current status of the v1.32 release and may change before the actual release date.

Spotlight on Kubernetes Upstream Training in Japan

Mon, 28 Oct 2024 00:00:00 +0000

We are organizers of Kubernetes Upstream Training in Japan. Our team is composed of members who actively contribute to Kubernetes, including individuals who hold roles such as member, reviewer, approver, and chair.

Our goal is to increase the number of Kubernetes contributors and foster the growth of the community. While Kubernetes community is friendly and collaborative, newcomers may find the first step of contributing to be a bit challenging. Our training program aims to lower that barrier and create an environment where even beginners can participate smoothly.

Announcing the 2024 Steering Committee Election Results

Wed, 02 Oct 2024 15:10:00 -0500

The 2024 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 3 of which were up for election in 2024. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

This community body is significant since it oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their charter.

Spotlight on CNCF Deaf and Hard-of-hearing Working Group (DHHWG)

Mon, 30 Sep 2024 00:00:00 +0000

In recognition of Deaf Awareness Month and the importance of inclusivity in the tech community, we are spotlighting Catherine Paganini, facilitator and one of the founding members of CNCF Deaf and Hard-of-Hearing Working Group (DHHWG). In this interview, Sandeep Kanabar, a deaf member of the DHHWG and part of the Kubernetes SIG ContribEx Communications team, sits down with Catherine to explore the impact of the DHHWG on cloud native projects like Kubernetes.

Sandeep’s journey is a testament to the power of inclusion. Through his involvement in the DHHWG, he connected with members of the Kubernetes community who encouraged him to join SIG ContribEx - the group responsible for sustaining the Kubernetes contributor experience. In an ecosystem where open-source projects are actively seeking contributors and maintainers, this story highlights how important it is to create pathways for underrepresented groups, including those with disabilities, to contribute their unique perspectives and skills.

Spotlight on SIG Scheduling

Tue, 24 Sep 2024 00:00:00 +0000

In this SIG Scheduling spotlight we talked with Kensei Nakada, an approver in SIG Scheduling.

Introductions

Arvind: Hello, thank you for the opportunity to learn more about SIG Scheduling! Would you like to introduce yourself and tell us a bit about your role, and how you got involved with Kubernetes?

Kensei: Hi, thanks for the opportunity! I’m Kensei Nakada (@sanposhiho), a software engineer at Tetrate.io. I have been contributing to Kubernetes in my free time for more than 3 years, and now I’m an approver of SIG Scheduling in Kubernetes. Also, I’m a founder/owner of two SIG subprojects, kube-scheduler-simulator and kube-scheduler-wasm-extension.

Kubernetes v1.31: kubeadm v1beta4

Fri, 23 Aug 2024 00:00:00 +0000

As part of the Kubernetes v1.31 release, kubeadm is adopting a new (v1beta4) version of its configuration file format. Configuration in the previous v1beta3 format is now formally deprecated, which means it's supported but you should migrate to v1beta4 and stop using the deprecated format. Support for v1beta3 configuration will be removed after a minimum of 3 Kubernetes minor releases.

In this article, I'll walk you through key changes; I'll explain about the kubeadm v1beta4 configuration format, and how to migrate from v1beta3 to v1beta4.

Kubernetes 1.31: Custom Profiling in Kubectl Debug Graduates to Beta

Thu, 22 Aug 2024 00:00:00 +0000

There are many ways of troubleshooting the pods and nodes in the cluster. However, kubectl debug is one of the easiest, highly used and most prominent ones. It provides a set of static profiles and each profile serves for a different kind of role. For instance, from the network administrator's point of view, debugging the node should be as easy as this:

$ kubectl debug node/mynode -it --image=busybox --profile=netadmin

On the other hand, static profiles also bring about inherent rigidity, which has some implications for some pods contrary to their ease of use. Because there are various kinds of pods (or nodes) that all have their specific necessities, and unfortunately, some can't be debugged by only using the static profiles.

Kubernetes 1.31: Fine-grained SupplementalGroups control

Thu, 22 Aug 2024 00:00:00 +0000

This blog discusses a new feature in Kubernetes 1.31 to improve the handling of supplementary groups in containers within Pods.

Motivation: Implicit group memberships defined in `/etc/group` in the container image

Although this behavior may not be popular with many Kubernetes cluster users/admins, kubernetes, by default, merges group information from the Pod with information defined in /etc/group in the container image.

Let's see an example, below Pod specifies runAsUser=1000, runAsGroup=3000 and supplementalGroups=4000 in the Pod's security context.

Kubernetes v1.31: New Kubernetes CPUManager Static Policy: Distribute CPUs Across Cores

Thu, 22 Aug 2024 00:00:00 +0000

In Kubernetes v1.31, we are excited to introduce a significant enhancement to CPU management capabilities: the distribute-cpus-across-cores option for the CPUManager static policy. This feature is currently in alpha and hidden by default, marking a strategic shift aimed at optimizing CPU utilization and improving system performance across multi-core processors.

Understanding the feature

Traditionally, Kubernetes' CPUManager tends to allocate CPUs as compactly as possible, typically packing them onto the fewest number of physical cores. However, allocation strategy matters, CPUs on the same physical host still share some resources of the physical core, such as the cache and execution units, etc.

Kubernetes 1.31: Autoconfiguration For Node Cgroup Driver (beta)

Wed, 21 Aug 2024 00:00:00 +0000

Historically, configuring the correct cgroup driver has been a pain point for users running new Kubernetes clusters. On Linux systems, there are two different cgroup drivers: cgroupfs and systemd. In the past, both the kubelet and CRI implementation (like CRI-O or containerd) needed to be configured to use the same cgroup driver, or else the kubelet would exit with an error. This was a source of headaches for many cluster admins. However, there is light at the end of the tunnel!

Kubernetes 1.31: Streaming Transitions from SPDY to WebSockets

Tue, 20 Aug 2024 00:00:00 +0000

In Kubernetes 1.31, by default kubectl now uses the WebSocket protocol instead of SPDY for streaming.

This post describes what these changes mean for you and why these streaming APIs matter.

Streaming APIs in Kubernetes

In Kubernetes, specific endpoints that are exposed as an HTTP or RESTful interface are upgraded to streaming connections, which require a streaming protocol. Unlike HTTP, which is a request-response protocol, a streaming protocol provides a persistent connection that's bi-directional, low-latency, and lets you interact in real-time. Streaming protocols support reading and writing data between your client and the server, in both directions, over the same connection. This type of connection is useful, for example, when you create a shell in a running container from your local workstation and run commands in the container.

Kubernetes 1.31: Pod Failure Policy for Jobs Goes GA

Mon, 19 Aug 2024 00:00:00 +0000

This post describes Pod failure policy, which graduates to stable in Kubernetes 1.31, and how to use it in your Jobs.

About Pod failure policy

When you run workloads on Kubernetes, Pods might fail for a variety of reasons. Ideally, workloads like Jobs should be able to ignore transient, retriable failures and continue running to completion.

To allow for these transient failures, Kubernetes Jobs include the backoffLimit field, which lets you specify a number of Pod failures that you're willing to tolerate during Job execution. However, if you set a large value for the backoffLimit field and rely solely on this field, you might notice unnecessary increases in operating costs as Pods restart excessively until the backoffLimit is met.

Kubernetes 1.31: MatchLabelKeys in PodAffinity graduates to beta

Fri, 16 Aug 2024 00:00:00 +0000

Kubernetes 1.29 introduced new fields matchLabelKeys and mismatchLabelKeys in podAffinity and podAntiAffinity.

In Kubernetes 1.31, this feature moves to beta and the corresponding feature gate (MatchLabelKeysInPodAffinity) gets enabled by default.

`matchLabelKeys` - Enhanced scheduling for versatile rolling updates

During a workload's (e.g., Deployment) rolling update, a cluster may have Pods from multiple versions at the same time. However, the scheduler cannot distinguish between old and new versions based on the labelSelector specified in podAffinity or podAntiAffinity. As a result, it will co-locate or disperse Pods regardless of their versions.

Kubernetes 1.31: Prevent PersistentVolume Leaks When Deleting out of Order

Fri, 16 Aug 2024 00:00:00 +0000

PersistentVolume (or PVs for short) are associated with Reclaim Policy. The reclaim policy is used to determine the actions that need to be taken by the storage backend on deletion of the PVC Bound to a PV. When the reclaim policy is Delete, the expectation is that the storage backend releases the storage resource allocated for the PV. In essence, the reclaim policy needs to be honored on PV deletion.

With the recent Kubernetes v1.31 release, a beta feature lets you configure your cluster to behave that way and honor the configured reclaim policy.

Kubernetes 1.31: Read Only Volumes Based On OCI Artifacts (alpha)

Fri, 16 Aug 2024 00:00:00 +0000

The Kubernetes community is moving towards fulfilling more Artificial Intelligence (AI) and Machine Learning (ML) use cases in the future. While the project has been designed to fulfill microservice architectures in the past, it’s now time to listen to the end users and introduce features which have a stronger focus on AI/ML.

One of these requirements is to support Open Container Initiative (OCI) compatible images and artifacts (referred as OCI objects) directly as a native volume source. This allows users to focus on OCI standards as well as enables them to store and distribute any content using OCI registries. A feature like this gives the Kubernetes project a chance to grow into use cases which go beyond running particular images.

Kubernetes 1.31: VolumeAttributesClass for Volume Modification Beta

Thu, 15 Aug 2024 00:00:00 +0000

Volumes in Kubernetes have been described by two attributes: their storage class, and their capacity. The storage class is an immutable property of the volume, while the capacity can be changed dynamically with volume resize.

This complicates vertical scaling of workloads with volumes. While cloud providers and storage vendors often offer volumes which allow specifying IO quality of service (Performance) parameters like IOPS or throughput and tuning them as workloads operate, Kubernetes has no API which allows changing them.

Kubernetes v1.31: Accelerating Cluster Performance with Consistent Reads from Cache

Thu, 15 Aug 2024 00:00:00 +0000

Kubernetes is renowned for its robust orchestration of containerized applications, but as clusters grow, the demands on the control plane can become a bottleneck. A key challenge has been ensuring strongly consistent reads from the etcd datastore, requiring resource-intensive quorum reads.

Today, the Kubernetes community is excited to announce a major improvement: consistent reads from cache, graduating to Beta in Kubernetes v1.31.

Why consistent reads matter

Consistent reads are essential for ensuring that Kubernetes components have an accurate view of the latest cluster state. Guaranteeing consistent reads is crucial for maintaining the accuracy and reliability of Kubernetes operations, enabling components to make informed decisions based on up-to-date information. In large-scale clusters, fetching and processing this data can be a performance bottleneck, especially for requests that involve filtering results. While Kubernetes can filter data by namespace directly within etcd, any other filtering by labels or field selectors requires the entire dataset to be fetched from etcd and then filtered in-memory by the Kubernetes API server. This is particularly impactful for components like the kubelet, which only needs to list pods scheduled to its node - but previously required the API Server and etcd to process all pods in the cluster.

Kubernetes 1.31: Moving cgroup v1 Support into Maintenance Mode

Wed, 14 Aug 2024 00:00:00 +0000

As Kubernetes continues to evolve and adapt to the changing landscape of container orchestration, the community has decided to move cgroup v1 support into maintenance mode in v1.31. This shift aligns with the broader industry's move towards cgroup v2, offering improved functionalities: including scalability and a more consistent interface. Before we dive into the consequences for Kubernetes, let's take a step back to understand what cgroups are and their significance in Linux.

Kubernetes v1.31: PersistentVolume Last Phase Transition Time Moves to GA

Wed, 14 Aug 2024 00:00:00 +0000

Announcing the graduation to General Availability (GA) of the PersistentVolume lastTransitionTime status field, in Kubernetes v1.31!

The Kubernetes SIG Storage team is excited to announce that the "PersistentVolumeLastPhaseTransitionTime" feature, introduced as an alpha in Kubernetes v1.28, has now reached GA status and is officially part of the Kubernetes v1.31 release. This enhancement helps Kubernetes users understand when a PersistentVolume transitions between different phases, allowing for more efficient and informed resource management.

Kubernetes v1.31: Elli

Tue, 13 Aug 2024 00:00:00 +0000

Editors: Matteo Bianchi, Yigit Demirbas, Abigail McCarthy, Edith Puclla, Rashan Smith

Announcing the release of Kubernetes v1.31: Elli!

Similar to previous releases, the release of Kubernetes v1.31 introduces new stable, beta, and alpha features. The consistent delivery of high-quality releases underscores the strength of our development cycle and the vibrant support from our community. This release consists of 45 enhancements. Of those enhancements, 11 have graduated to Stable, 22 are entering Beta, and 12 have graduated to Alpha.

Introducing Feature Gates to Client-Go: Enhancing Flexibility and Control

Mon, 12 Aug 2024 00:00:00 +0000

Kubernetes components use on-off switches called feature gates to manage the risk of adding a new feature. The feature gate mechanism is what enables incremental graduation of a feature through the stages Alpha, Beta, and GA.

Kubernetes components, such as kube-controller-manager and kube-scheduler, use the client-go library to interact with the API. The same library is used across the Kubernetes ecosystem to build controllers, tools, webhooks, and more. client-go now includes its own feature gating mechanism, giving developers and cluster administrators more control over how they adopt client features.

Spotlight on SIG API Machinery

Wed, 07 Aug 2024 00:00:00 +0000

We recently talked with Federico Bongiovanni (Google) and David Eads (Red Hat), Chairs of SIG API Machinery, to know a bit more about this Kubernetes Special Interest Group.

Introductions

Frederico (FSM): Hello, and thank your for your time. To start with, could you tell us about yourselves and how you got involved in Kubernetes?

David: I started working on OpenShift (the Red Hat distribution of Kubernetes) in the fall of 2014 and got involved pretty quickly in API Machinery. My first PRs were fixing kube-apiserver error messages and from there I branched out to kubectl (kubeconfigs are my fault!), auth (RBAC and *Review APIs are ports from OpenShift), apps (workqueues and sharedinformers for example). Don’t tell the others, but API Machinery is still my favorite :)

Kubernetes Removals and Major Changes In v1.31

Fri, 19 Jul 2024 00:00:00 +0000

As Kubernetes develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. This article outlines some planned changes for the Kubernetes v1.31 release that the release team feels you should be aware of for the continued maintenance of your Kubernetes environment. The information listed below is based on the current status of the v1.31 release. It may change before the actual release date.

Spotlight on SIG Node

Thu, 20 Jun 2024 00:00:00 +0000

In the world of container orchestration, Kubernetes reigns supreme, powering some of the most complex and dynamic applications across the globe. Behind the scenes, a network of Special Interest Groups (SIGs) drives Kubernetes' innovation and stability.

Today, I have the privilege of speaking with Matthias Bertschy, Gunju Kim, and Sergey Kanzhelev, members of SIG Node, who will shed some light on their roles, challenges, and the exciting developments within SIG Node.

Answers given collectively by all interviewees will be marked by their initials.

10 Years of Kubernetes

Thu, 06 Jun 2024 00:00:00 +0000

Ten (10) years ago, on June 6th, 2014, the first commit of Kubernetes was pushed to GitHub. That first commit with 250 files and 47,501 lines of go, bash and markdown kicked off the project we have today. Who could have predicted that 10 years later, Kubernetes would grow to become one of the largest Open Source projects to date with over 88,000 contributors from more than 8,000 companies, across 44 countries.

Completing the largest migration in Kubernetes history

Mon, 20 May 2024 00:00:00 +0000

Since as early as Kubernetes v1.7, the Kubernetes project has pursued the ambitious goal of removing built-in cloud provider integrations (KEP-2395). While these integrations were instrumental in Kubernetes' early development and growth, their removal was driven by two key factors: the growing complexity of maintaining native support for every cloud provider across millions of lines of Go code, and the desire to establish Kubernetes as a truly vendor-neutral platform.

After many releases, we're thrilled to announce that all cloud provider integrations have been successfully migrated from the core Kubernetes repository to external plugins. In addition to achieving our initial objectives, we've also significantly streamlined Kubernetes by removing roughly 1.5 million lines of code and reducing the binary sizes of core components by approximately 40%.

Gateway API v1.1: Service mesh, GRPCRoute, and a whole lot more

Thu, 09 May 2024 09:00:00 -0800

Following the GA release of Gateway API last October, Kubernetes SIG Network is pleased to announce the v1.1 release of Gateway API. In this release, several features are graduating to Standard Channel (GA), notably including support for service mesh and GRPCRoute. We're also introducing some new experimental features, including session persistence and client certificate verification.

What's new

Graduation to Standard

This release includes the graduation to Standard of four eagerly awaited features. This means they are no longer experimental concepts; inclusion in the Standard release channel denotes a high level of confidence in the API surface and provides guarantees of backward compatibility. Of course, as with any other Kubernetes API, Standard Channel features can continue to evolve with backward-compatible additions over time, and we certainly expect further refinements and improvements to these new features in the future. For more information on how all of this works, refer to the Gateway API Versioning Policy.

Container Runtime Interface streaming explained

Wed, 01 May 2024 00:00:00 +0000

The Kubernetes Container Runtime Interface (CRI) acts as the main connection between the kubelet and the Container Runtime. Those runtimes have to provide a gRPC server which has to fulfill a Kubernetes defined Protocol Buffer interface. This API definition evolves over time, for example when contributors add new features or fields are going to become deprecated.

In this blog post, I'd like to dive into the functionality and history of three extraordinary Remote Procedure Calls (RPCs), which are truly outstanding in terms of how they work: Exec, Attach and PortForward.

Kubernetes 1.30: Preventing unauthorized volume mode conversion moves to GA

Tue, 30 Apr 2024 00:00:00 +0000

With the release of Kubernetes 1.30, the feature to prevent the modification of the volume mode of a PersistentVolumeClaim that was created from an existing VolumeSnapshot in a Kubernetes cluster, has moved to GA!

The problem

The Volume Mode of a PersistentVolumeClaim refers to whether the underlying volume on the storage device is formatted into a filesystem or presented as a raw block device to the Pod that uses it.

Users can leverage the VolumeSnapshot feature, which has been stable since Kubernetes v1.20, to create a PersistentVolumeClaim (shortened as PVC) from an existing VolumeSnapshot in the Kubernetes cluster. The PVC spec includes a dataSource field, which can point to an existing VolumeSnapshot instance. Visit Create a PersistentVolumeClaim from a Volume Snapshot for more details on how to create a PVC from an existing VolumeSnapshot in a Kubernetes cluster.

Kubernetes 1.30: Multi-Webhook and Modular Authorization Made Much Easier

Fri, 26 Apr 2024 00:00:00 +0000

With Kubernetes 1.30, we (SIG Auth) are moving Structured Authorization Configuration to beta.

Today's article is about authorization: deciding what someone can and cannot access. Check a previous article from yesterday to find about what's new in Kubernetes v1.30 around authentication (finding out who's performing a task, and checking that they are who they say they are).

Introduction

Kubernetes continues to evolve to meet the intricate requirements of system administrators and developers alike. A critical aspect of Kubernetes that ensures the security and integrity of the cluster is the API server authorization. Until recently, the configuration of the authorization chain in kube-apiserver was somewhat rigid, limited to a set of command-line flags and allowing only a single webhook in the authorization chain. This approach, while functional, restricted the flexibility needed by cluster administrators to define complex, fine-grained authorization policies. The latest Structured Authorization Configuration feature (KEP-3221) aims to revolutionize this aspect by introducing a more structured and versatile way to configure the authorization chain, focusing on enabling multiple webhooks and providing explicit control mechanisms.

Kubernetes 1.30: Structured Authentication Configuration Moves to Beta

Thu, 25 Apr 2024 00:00:00 +0000

With Kubernetes 1.30, we (SIG Auth) are moving Structured Authentication Configuration to beta.

Today's article is about authentication: finding out who's performing a task, and checking that they are who they say they are. Check back in tomorrow to find about what's new in Kubernetes v1.30 around authorization (deciding what someone can and can't access).

Motivation

Kubernetes has had a long-standing need for a more flexible and extensible authentication system. The current system, while powerful, has some limitations that make it difficult to use in certain scenarios. For example, it is not possible to use multiple authenticators of the same type (e.g., multiple JWT authenticators) or to change the configuration without restarting the API server. The Structured Authentication Configuration feature is the first step towards addressing these limitations and providing a more flexible and extensible way to configure authentication in Kubernetes.

Kubernetes 1.30: Validating Admission Policy Is Generally Available

Wed, 24 Apr 2024 00:00:00 +0000

On behalf of the Kubernetes project, I am excited to announce that ValidatingAdmissionPolicy has reached general availability as part of Kubernetes 1.30 release. If you have not yet read about this new declarative alternative to validating admission webhooks, it may be interesting to read our previous post about the new feature. If you have already heard about ValidatingAdmissionPolicies and you are eager to try them out, there is no better time to do it than now.

Kubernetes 1.30: Read-only volume mounts can be finally literally read-only

Tue, 23 Apr 2024 00:00:00 +0000

Read-only volume mounts have been a feature of Kubernetes since the beginning. Surprisingly, read-only mounts are not completely read-only under certain conditions on Linux. As of the v1.30 release, they can be made completely read-only, with alpha support for recursive read-only mounts.

Read-only volume mounts are not really read-only by default

Volume mounts can be deceptively complicated.

You might expect that the following manifest makes everything under /mnt in the containers read-only:

Kubernetes 1.30: Beta Support For Pods With User Namespaces

Mon, 22 Apr 2024 00:00:00 +0000

Linux provides different namespaces to isolate processes from each other. For example, a typical Kubernetes pod runs within a network namespace to isolate the network identity and a PID namespace to isolate the processes.

One Linux namespace that was left behind is the user namespace. This namespace allows us to isolate the user and group identifiers (UIDs and GIDs) we use inside the container from the ones on the host.

This is a powerful abstraction that allows us to run containers as "root": we are root inside the container and can do everything root can inside the pod, but our interactions with the host are limited to what a non-privileged user can do. This is great for limiting the impact of a container breakout.

Kubernetes v1.30: Uwubernetes

Wed, 17 Apr 2024 00:00:00 +0000

Editors: Amit Dsouza, Frederick Kautz, Kristin Martin, Abigail McCarthy, Natali Vlatko

Announcing the release of Kubernetes v1.30: Uwubernetes, the cutest release!

Similar to previous releases, the release of Kubernetes v1.30 introduces new stable, beta, and alpha features. The consistent delivery of top-notch releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 45 enhancements. Of those enhancements, 17 have graduated to Stable, 18 are entering Beta, and 10 have graduated to Alpha.

Spotlight on SIG Architecture: Code Organization

Thu, 11 Apr 2024 00:00:00 +0000

This is the third interview of a SIG Architecture Spotlight series that will cover the different subprojects. We will cover SIG Architecture: Code Organization.

In this SIG Architecture spotlight I talked with Madhav Jivrajani (VMware), a member of the Code Organization subproject.

Introducing the Code Organization subproject

Frederico (FSM): Hello Madhav, thank you for your availability. Could you start by telling us a bit about yourself, your role and how you got involved in Kubernetes?

DIY: Create Your Own Cloud with Kubernetes (Part 3)

Fri, 05 Apr 2024 07:40:00 +0000

Approaching the most interesting phase, this article delves into running Kubernetes within Kubernetes. Technologies such as Kamaji and Cluster API are highlighted, along with their integration with KubeVirt.

Previous discussions have covered preparing Kubernetes on bare metal and how to turn Kubernetes into virtual machines management system. This article concludes the series by explaining how, using all of the above, you can build a full-fledged managed Kubernetes and run virtual Kubernetes clusters with just a click.

DIY: Create Your Own Cloud with Kubernetes (Part 2)

Fri, 05 Apr 2024 07:35:00 +0000

Continuing our series of posts on how to build your own cloud using just the Kubernetes ecosystem. In the previous article, we explained how we prepare a basic Kubernetes distribution based on Talos Linux and Flux CD. In this article, we'll show you a few various virtualization technologies in Kubernetes and prepare everything need to run virtual machines in Kubernetes, primarily storage and networking.

We will talk about technologies such as KubeVirt, LINSTOR, and Kube-OVN.

DIY: Create Your Own Cloud with Kubernetes (Part 1)

Fri, 05 Apr 2024 07:30:00 +0000

At Ænix, we have a deep affection for Kubernetes and dream that all modern technologies will soon start utilizing its remarkable patterns.

Have you ever thought about building your own cloud? I bet you have. But is it possible to do this using only modern technologies and approaches, without leaving the cozy Kubernetes ecosystem? Our experience in developing Cozystack required us to delve deeply into it.

You might argue that Kubernetes is not intended for this purpose and why not simply use OpenStack for bare metal servers and run Kubernetes inside it as intended. But by doing so, you would simply shift the responsibility from your hands to the hands of OpenStack administrators. This would add at least one more huge and complex system to your ecosystem.

Introducing the Windows Operational Readiness Specification

Wed, 03 Apr 2024 00:00:00 +0000

Since Windows support graduated to stable with Kubernetes 1.14 in 2019, the capability to run Windows workloads has been much appreciated by the end user community. The level of and availability of Windows workload support has consistently been a major differentiator for Kubernetes distributions used by large enterprises. However, with more Windows workloads being migrated to Kubernetes and new Windows features being continuously released, it became challenging to test Windows worker nodes in an effective and standardized way.

A Peek at Kubernetes v1.30

Tue, 12 Mar 2024 00:00:00 +0000

A quick look: exciting changes in Kubernetes v1.30

It's a new year and a new Kubernetes release. We're halfway through the release cycle and have quite a few interesting and exciting enhancements coming in v1.30. From brand new features in alpha, to established features graduating to stable, to long-awaited improvements, this release has something for everyone to pay attention to!

To tide you over until the official release, here's a sneak peek of the enhancements we're most excited about in this cycle!

CRI-O: Applying seccomp profiles from OCI registries

Thu, 07 Mar 2024 00:00:00 +0000

But distributing those seccomp profiles is a major challenge in Kubernetes, because the JSON files have to be available on all nodes where a workload can possibly run. Projects like the Security Profiles Operator solve that problem by running as a daemon within the cluster, which makes me wonder which part of that distribution could be done by the container runtime.

Spotlight on SIG Cloud Provider

Fri, 01 Mar 2024 00:00:00 +0000

One of the most popular ways developers use Kubernetes-related services is via cloud providers, but have you ever wondered how cloud providers can do that? How does this whole process of integration of Kubernetes to various cloud providers happen? To answer that, let's put the spotlight on SIG Cloud Provider.

SIG Cloud Provider works to create seamless integrations between Kubernetes and various cloud providers. Their mission? Keeping the Kubernetes ecosystem fair and open for all. By setting clear standards and requirements, they ensure every cloud provider plays nicely with Kubernetes. It is their responsibility to configure cluster components to enable cloud provider integrations.

A look into the Kubernetes Book Club

Thu, 22 Feb 2024 00:00:00 +0000

Learning Kubernetes and the entire ecosystem of technologies around it is not without its challenges. In this interview, we will talk with Carlos Santana (AWS) to learn a bit more about how he created the Kubernetes Book Club, how it works, and how anyone can join in to take advantage of a community-based learning experience.

Frederico Muñoz (FSM): Hello Carlos, thank you so much for your availability. To start with, could you tell us a bit about yourself?

Image Filesystem: Configuring Kubernetes to store containers on a separate filesystem

Tue, 23 Jan 2024 00:00:00 +0000

A common issue in running/operating Kubernetes clusters is running out of disk space. When the node is provisioned, you should aim to have a good amount of storage space for your container images and running containers. The container runtime usually writes to /var. This can be located as a separate partition or on the root filesystem. CRI-O, by default, writes its containers and images to /var/lib/containers, while containerd writes its containers and images to /var/lib/containerd.

Spotlight on SIG Release (Release Team Subproject)

Mon, 15 Jan 2024 00:00:00 +0000

The Release Special Interest Group (SIG Release), where Kubernetes sharpens its blade with cutting-edge features and bug fixes every 4 months. Have you ever considered how such a big project like Kubernetes manages its timeline so efficiently to release its new version, or how the internal workings of the Release Team look like? If you're curious about these questions or want to know more and get involved with the work SIG Release does, read on!

Contextual logging in Kubernetes 1.29: Better troubleshooting and enhanced logging

Wed, 20 Dec 2023 09:30:00 -0800

On behalf of the Structured Logging Working Group and SIG Instrumentation, we are pleased to announce that the contextual logging feature introduced in Kubernetes v1.24 has now been successfully migrated to two components (kube-scheduler and kube-controller-manager) as well as some directories. This feature aims to provide more useful logs for better troubleshooting of Kubernetes and to empower developers to enhance Kubernetes.

What is contextual logging?

Contextual logging is based on the go-logr API. The key idea is that libraries are passed a logger instance by their caller and use that for logging instead of accessing a global logger. The binary decides the logging implementation, not the libraries. The go-logr API is designed around structured logging and supports attaching additional information to a logger.

Kubernetes 1.29: Decoupling taint manager from node lifecycle controller

Tue, 19 Dec 2023 00:00:00 +0000

This blog discusses a new feature in Kubernetes 1.29 to improve the handling of taint-based pod eviction.

Background

In Kubernetes 1.29, an improvement has been introduced to enhance the taint-based pod eviction handling on nodes. This blog discusses the changes made to node-lifecycle-controller to separate its responsibilities and improve overall code maintainability.

Summary of changes

node-lifecycle-controller previously combined two independent functions:

Adding a pre-defined set of NoExecute taints to Node based on Node's condition.
Performing pod eviction on NoExecute taint.

With the Kubernetes 1.29 release, the taint-based eviction implementation has been moved out of node-lifecycle-controller into a separate and independent component called taint-eviction-controller. This separation aims to disentangle code, enhance code maintainability, and facilitate future extensions to either component.

Kubernetes 1.29: PodReadyToStartContainers Condition Moves to Beta

Tue, 19 Dec 2023 00:00:00 +0000

With the recent release of Kubernetes 1.29, the PodReadyToStartContainers condition is available by default. The kubelet manages the value for that condition throughout a Pod's lifecycle, in the status field of a Pod. The kubelet will use the PodReadyToStartContainers condition to accurately surface the initialization state of a Pod, from the perspective of Pod sandbox creation and network configuration by a container runtime.

What's the motivation for this feature?

Cluster administrators did not have a clear and easily accessible way to view the completion of Pod's sandbox creation and initialization. As of 1.28, the Initialized condition in Pods tracks the execution of init containers. However, it has limitations in accurately reflecting the completion of sandbox creation and readiness to start containers for all Pods in a cluster. This distinction is particularly important in multi-tenant clusters where tenants own the Pod specifications, including the set of init containers, while cluster administrators manage storage plugins, networking plugins, and container runtime handlers. Therefore, there is a need for an improved mechanism to provide cluster administrators with a clear and comprehensive view of Pod sandbox creation completion and container readiness.

Kubernetes 1.29: New (alpha) Feature, Load Balancer IP Mode for Services

Mon, 18 Dec 2023 00:00:00 +0000

This blog introduces a new alpha feature in Kubernetes 1.29. It provides a configurable approach to define how Service implementations, exemplified in this blog by kube-proxy, handle traffic from pods to the Service, within the cluster.

Background

In older Kubernetes releases, the kube-proxy would intercept traffic that was destined for the IP address associated with a Service of type: LoadBalancer. This happened whatever mode you used for kube-proxy. The interception implemented the expected behavior (traffic eventually reaching the expected endpoints behind the Service). The mechanism to make that work depended on the mode for kube-proxy; on Linux, kube-proxy in iptables mode would redirecting packets directly to the endpoint; in ipvs mode, kube-proxy would configure the load balancer's IP address to one interface on the node. The motivation for implementing that interception was for two reasons:

Kubernetes 1.29: Single Pod Access Mode for PersistentVolumes Graduates to Stable

Mon, 18 Dec 2023 00:00:00 +0000

With the release of Kubernetes v1.29, the ReadWriteOncePod volume access mode has graduated to general availability: it's part of Kubernetes' stable API. In this blog post, I'll take a closer look at this access mode and what it does.

What is `ReadWriteOncePod`?

ReadWriteOncePod is an access mode for PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) introduced in Kubernetes v1.22. This access mode enables you to restrict volume access to a single pod in the cluster, ensuring that only one pod can write to the volume at a time. This can be particularly useful for stateful workloads that require single-writer access to storage.

Kubernetes 1.29: CSI Storage Resizing Authenticated and Generally Available in v1.29

Fri, 15 Dec 2023 00:00:00 +0000

Kubernetes version v1.29 brings generally available support for authentication during CSI (Container Storage Interface) storage resize operations.

Let's embark on the evolution of this feature, initially introduced in alpha in Kubernetes v1.25, and unravel the changes accompanying its transition to GA.

Authenticated CSI storage resizing unveiled

Kubernetes harnesses the capabilities of CSI to seamlessly integrate with third-party storage systems, empowering your cluster to seamlessly expand storage volumes managed by the CSI driver. The recent elevation of authentication secret support for resizes from Beta to GA ushers in new horizons, enabling volume expansion in scenarios where the underlying storage operation demands credentials for backend cluster operations – such as accessing a SAN/NAS fabric. This enhancement addresses a critical limitation for CSI drivers, allowing volume expansion at the node level, especially in cases necessitating authentication for resize operations.

Kubernetes 1.29: VolumeAttributesClass for Volume Modification

Fri, 15 Dec 2023 00:00:00 +0000

The v1.29 release of Kubernetes introduced an alpha feature to support modifying a volume by changing the volumeAttributesClassName that was specified for a PersistentVolumeClaim (PVC). With the feature enabled, Kubernetes can handle updates of volume attributes other than capacity. Allowing volume attributes to be changed without managing it through different provider's APIs directly simplifies the current flow.

You can read about VolumeAttributesClass usage details in the Kubernetes documentation or you can read on to learn about why the Kubernetes project is supporting this feature.

Kubernetes 1.29: Cloud Provider Integrations Are Now Separate Components

Thu, 14 Dec 2023 09:30:00 -0800

For Kubernetes v1.29, you need to use additional components to integrate your Kubernetes cluster with a cloud infrastructure provider. By default, Kubernetes v1.29 components abort if you try to specify integration with any cloud provider using one of the legacy compiled-in cloud provider integrations. If you want to use a legacy integration, you have to opt back in - and a future release will remove even that option.

In 2018, the Kubernetes community agreed to form the Cloud Provider Special Interest Group (SIG), with a mission to externalize all cloud provider integrations and remove all the existing in-tree cloud provider integrations. In January 2019, the Kubernetes community approved the initial draft of KEP-2395: Removing In-Tree Cloud Provider Code. This KEP defines a process by which we can remove cloud provider specific code from the core Kubernetes source tree. From the KEP:

Kubernetes v1.29: Mandala

Wed, 13 Dec 2023 00:00:00 +0000

Editors: Carol Valencia, Kristin Martin, Abigail McCarthy, James Quigley

Announcing the release of Kubernetes v1.29: Mandala (The Universe), the last release of 2023!

Similar to previous releases, the release of Kubernetes v1.29 introduces new stable, beta, and alpha features. The consistent delivery of top-notch releases underscores the strength of our development cycle and the vibrant support from our community.

This release consists of 49 enhancements. Of those enhancements, 11 have graduated to Stable, 19 are entering Beta and 19 have graduated to Alpha.

New Experimental Features in Gateway API v1.0

Tue, 28 Nov 2023 10:00:00 -0800

Recently, the Gateway API announced its v1.0 GA release, marking a huge milestone for the project.

Along with stabilizing some of the core functionality in the API, a number of exciting new experimental features have been added.

Backend TLS Policy

BackendTLSPolicy is a new Gateway API type used for specifying the TLS configuration of the connection from the Gateway to backend Pods via the Service API object. It is specified as a Direct PolicyAttachment without defaults or overrides, applied to a Service that accesses a backend, where the BackendTLSPolicy resides in the same namespace as the Service to which it is applied. All Gateway API Routes that point to a referenced Service should respect a configured BackendTLSPolicy.

Spotlight on SIG Testing

Fri, 24 Nov 2023 00:00:00 +0000

Welcome to another edition of the SIG spotlight blog series, where we highlight the incredible work being done by various Special Interest Groups (SIGs) within the Kubernetes project. In this edition, we turn our attention to SIG Testing, a group interested in effective testing of Kubernetes and automating away project toil. SIG Testing focus on creating and running tools and infrastructure that make it easier for the community to write and run tests, and to contribute, analyze and act upon test results.

Kubernetes Removals, Deprecations, and Major Changes in Kubernetes 1.29

Thu, 16 Nov 2023 00:00:00 +0000

As with every release, Kubernetes v1.29 will introduce feature deprecations and removals. Our continued ability to produce high-quality releases is a testament to our robust development cycle and healthy community. The following are some of the deprecations and removals coming in the Kubernetes 1.29 release.

The Kubernetes API removal and deprecation process

The Kubernetes project has a well-documented deprecation policy for features. This policy states that stable APIs may only be deprecated when a newer, stable version of that same API is available and that APIs have a minimum lifetime for each stability level. A deprecated API is one that has been marked for removal in a future Kubernetes release; it will continue to function until removal (at least one year from the deprecation), but usage will result in a warning being displayed. Removed APIs are no longer available in the current version, at which point you must migrate to using the replacement.

The Case for Kubernetes Resource Limits: Predictability vs. Efficiency

Thu, 16 Nov 2023 00:00:00 +0000

There’s been quite a lot of posts suggesting that not using Kubernetes resource limits might be a fairly useful thing (for example, For the Love of God, Stop Using CPU Limits on Kubernetes or Kubernetes: Make your services faster by removing CPU limits ). The points made there are totally valid – it doesn’t make much sense to pay for compute power that will not be used due to limits, nor to artificially increase latency. This post strives to argue that limits have their legitimate use as well.

Introducing SIG etcd

Tue, 07 Nov 2023 00:00:00 +0000

Special Interest Groups (SIGs) are a fundamental part of the Kubernetes project, with a substantial share of the community activity happening within them. When the need arises, new SIGs can be created, and that was precisely what happened recently.

SIG etcd is the most recent addition to the list of Kubernetes SIGs. In this article we will get to know it a bit better, understand its origins, scope, and plans.

The critical role of etcd

If we look inside the control plane of a Kubernetes cluster, we will find etcd, a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data -- this description alone highlights the critical role that etcd plays, and the importance of it within the Kubernetes ecosystem.

Kubernetes Contributor Summit: Behind-the-scenes

Fri, 03 Nov 2023 00:00:00 +0000

Every year, just before the official start of KubeCon+CloudNativeCon, there's a special event that has a very special place in the hearts of those organizing and participating in it: the Kubernetes Contributor Summit. To find out why, and to provide a behind-the-scenes perspective, we interview Noah Abrahams, whom amongst other roles was the co-lead for the Kubernetes Contributor Summit in 2023.

Frederico Muñoz (FSM): Hello Noah, and welcome. Could you start by introducing yourself and telling us how you got involved in Kubernetes?

Spotlight on SIG Architecture: Production Readiness

Thu, 02 Nov 2023 00:00:00 +0000

This is the second interview of a SIG Architecture Spotlight series that will cover the different subprojects. In this blog, we will cover the SIG Architecture: Production Readiness subproject.

In this SIG Architecture spotlight, we talked with Wojciech Tyczynski (Google), lead of the Production Readiness subproject.

About SIG Architecture and the Production Readiness subproject

Frederico (FSM): Hello Wojciech, could you tell us a bit about yourself, your role and how you got involved in Kubernetes?

Gateway API v1.0: GA Release

Tue, 31 Oct 2023 10:00:00 -0800

On behalf of Kubernetes SIG Network, we are pleased to announce the v1.0 release of Gateway API! This release marks a huge milestone for this project. Several key APIs are graduating to GA (generally available), while other significant features have been added to the Experimental channel.

What's new

Graduation to v1

This release includes the graduation of Gateway, GatewayClass, and HTTPRoute to v1, which means they are now generally available (GA). This API version denotes a high level of confidence in the API surface and provides guarantees of backwards compatibility. Note that although, the version of these APIs included in the Standard channel are now considered stable, that does not mean that they are complete. These APIs will continue to receive new features via the Experimental channel as they meet graduation criteria. For more information on how all of this works, refer to the Gateway API Versioning Policy.

Introducing ingress2gateway; Simplifying Upgrades to Gateway API

Wed, 25 Oct 2023 10:00:00 -0800

Today we are releasing ingress2gateway, a tool that can help you migrate from Ingress to Gateway API. Gateway API is just weeks away from graduating to GA, if you haven't upgraded yet, now's the time to think about it!

Background

In the ever-evolving world of Kubernetes, networking plays a pivotal role. As more applications are deployed in Kubernetes clusters, effective exposure of these services to clients becomes a critical concern. If you've been working with Kubernetes, you're likely familiar with the Ingress API, which has been the go-to solution for managing external access to services.

Plants, process and parties: the Kubernetes 1.28 release interview

Tue, 24 Oct 2023 00:00:00 +0000

Since 2018, one of my favourite contributions to the Kubernetes community has been to share the story of each release. Many of these stories were told on behalf of a past employer; by popular demand, I've brought them back, now under my own name. If you were a fan of the old show, I would be delighted if you would subscribe.

Back in August, we welcomed the release of Kubernetes 1.28. That release was led by Grace Nguyen, a CS student at the University of Waterloo. Grace joined me for the traditional release interview, and while you can read her story below, I encourage you to listen to it if you can.

PersistentVolume Last Phase Transition Time in Kubernetes

Mon, 23 Oct 2023 00:00:00 +0000

In the recent Kubernetes v1.28 release, we (SIG Storage) introduced a new alpha feature that aims to improve PersistentVolume (PV) storage management and help cluster administrators gain better insights into the lifecycle of PVs. With the addition of the lastPhaseTransitionTime field into the status of a PV, cluster administrators are now able to track the last time a PV transitioned to a different phase, allowing for more efficient and informed resource management.

A Quick Recap of 2023 China Kubernetes Contributor Summit

Fri, 20 Oct 2023 00:00:00 +0000

On September 26, 2023, the first day of KubeCon + CloudNativeCon + Open Source Summit China 2023, nearly 50 contributors gathered in Shanghai for the Kubernetes Contributor Summit.

All participants in the 2023 Kubernetes Contributor Summit

This marked the first in-person offline gathering held in China after three years of the pandemic.

A joyful meetup

The event began with welcome speeches from Kevin Wang from Huawei Cloud, one of the co-chairs of KubeCon, and Puja from Giant Swarm.

Bootstrap an Air Gapped Cluster With Kubeadm

Thu, 12 Oct 2023 00:00:00 +0000

Ever wonder how software gets deployed onto a system that is deliberately disconnected from the Internet and other networks? These systems are typically disconnected due to their sensitive nature. Sensitive as in utilities (power/water), banking, healthcare, weapons systems, other government use cases, etc. Sometimes it's technically a water gap, if you're running Kubernetes on an underwater vessel. Still, these environments need software to operate. This concept of deployment in a disconnected state is what it means to deploy to the other side of an air gap.

CRI-O is moving towards pkgs.k8s.io

Tue, 10 Oct 2023 00:00:00 +0000

The Kubernetes community recently announced that their legacy package repositories are frozen, and now they moved to introduced community-owned package repositories powered by the OpenBuildService (OBS). CRI-O has a long history of utilizing OBS for their package builds, but all of the packaging efforts have been done manually so far.

The CRI-O community absolutely loves Kubernetes, which means that they're delighted to announce that:

All future CRI-O packages will be shipped as part of the officially supported Kubernetes infrastructure hosted on pkgs.k8s.io!

Spotlight on SIG Architecture: Conformance

Thu, 05 Oct 2023 00:00:00 +0000

This is the first interview of a SIG Architecture Spotlight series that will cover the different subprojects. We start with the SIG Architecture: Conformance subproject

In this SIG Architecture spotlight, we talked with Riaan Kleinhans (ii.nz), Lead for the Conformance sub-project.

About SIG Architecture and the Conformance subproject

Frederico (FSM): Hello Riaan, and welcome! For starters, tell us a bit about yourself, your role and how you got involved in Kubernetes.

Riaan Kleinhans (RK): Hi! My name is Riaan Kleinhans and I live in South Africa. I am the Project manager for the ii.nz team in New Zealand. When I joined ii the plan was to move to New Zealand in April 2020 and then Covid happened. Fortunately, being a flexible and dynamic team we were able to make it work remotely and in very different time zones.

Announcing the 2023 Steering Committee Election Results

Mon, 02 Oct 2023 00:00:00 +0000

The 2023 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2023. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

Happy 7th Birthday kubeadm!

Tue, 26 Sep 2023 00:00:00 +0000

What a journey so far!

Starting from the initial blog post “How we made Kubernetes insanely easy to install” in September 2016, followed by an exciting growth that lead to general availability / “Production-Ready Kubernetes Cluster Creation with kubeadm” two years later.

And later on a continuous, steady and reliable flow of small improvements that is still going on as of today.

What is kubeadm? (quick refresher)

kubeadm is focused on bootstrapping Kubernetes clusters on existing infrastructure and performing an essential set of maintenance tasks. The core of the kubeadm interface is quite simple: new control plane nodes are created by running kubeadm init and worker nodes are joined to the control plane by running kubeadm join. Also included are utilities for managing already bootstrapped clusters, such as control plane upgrades and token and certificate renewal.

kubeadm: Use etcd Learner to Join a Control Plane Node Safely

Mon, 25 Sep 2023 00:00:00 +0000

The kubeadm tool now supports etcd learner mode, which allows you to enhance the resilience and stability of your Kubernetes clusters by leveraging the learner mode feature introduced in etcd version 3.4. This guide will walk you through using etcd learner mode with kubeadm. By default, kubeadm runs a local etcd instance on each control plane node.

In v1.27, kubeadm introduced a new feature gate EtcdLearnerMode. With this feature gate enabled, when joining a new control plane node, a new etcd member will be created as a learner and promoted to a voting member only after the etcd data are fully aligned.

User Namespaces: Now Supports Running Stateful Pods in Alpha!

Wed, 13 Sep 2023 00:00:00 +0000

Kubernetes v1.25 introduced support for user namespaces for only stateless pods. Kubernetes 1.28 lifted that restriction, after some design changes were done in 1.27.

The beauty of this feature is that:

it is trivial to adopt (you just need to set a bool in the pod spec)
doesn't need any changes for most applications
improves security by drastically enhancing the isolation of containers and mitigating CVEs rated HIGH and CRITICAL.

This post explains the basics of user namespaces and also shows:

Comparing Local Kubernetes Development Tools: Telepresence, Gefyra, and mirrord

Tue, 12 Sep 2023 00:00:00 +0000

The Kubernetes development cycle is an evolving landscape with a myriad of tools seeking to streamline the process. Each tool has its unique approach, and the choice often comes down to individual project requirements, the team's expertise, and the preferred workflow.

Among the various solutions, a category we dubbed “Local K8S Development tools” has emerged, which seeks to enhance the Kubernetes development experience by connecting locally running components to the Kubernetes cluster. This facilitates rapid testing of new code in cloud conditions, circumventing the traditional cycle of Dockerization, CI, and deployment.

Kubernetes Legacy Package Repositories Will Be Frozen On September 13, 2023

Thu, 31 Aug 2023 15:30:00 -0700

On August 15, 2023, the Kubernetes project announced the general availability of the community-owned package repositories for Debian and RPM packages available at pkgs.k8s.io. The new package repositories are replacement for the legacy Google-hosted package repositories: apt.kubernetes.io and yum.kubernetes.io. The announcement blog post for pkgs.k8s.io highlighted that we will stop publishing packages to the legacy repositories in the future.

Today, we're formally deprecating the legacy package repositories (apt.kubernetes.io and yum.kubernetes.io), and we're announcing our plans to freeze the contents of the repositories as of September 13, 2023.

Gateway API v0.8.0: Introducing Service Mesh Support

Tue, 29 Aug 2023 10:00:00 -0800

We are thrilled to announce the v0.8.0 release of Gateway API! With this release, Gateway API support for service mesh has reached Experimental status. We look forward to your feedback!

We're especially delighted to announce that Kuma 2.3+, Linkerd 2.14+, and Istio 1.16+ are all fully-conformant implementations of Gateway API service mesh support.

Service mesh support in Gateway API

While the initial focus of Gateway API was always ingress (north-south) traffic, it was clear almost from the beginning that the same basic routing concepts should also be applicable to service mesh (east-west) traffic. In 2022, the Gateway API subproject started the GAMMA initiative, a dedicated vendor-neutral workstream, specifically to examine how best to fit service mesh support into the framework of the Gateway API resources, without requiring users of Gateway API to relearn everything they understand about the API.

Kubernetes 1.28: A New (alpha) Mechanism For Safer Cluster Upgrades

Mon, 28 Aug 2023 00:00:00 +0000

This blog describes the mixed version proxy, a new alpha feature in Kubernetes 1.28. The mixed version proxy enables an HTTP request for a resource to be served by the correct API server in cases where there are multiple API servers at varied versions in a cluster. For example, this is useful during a cluster upgrade, or when you're rolling out the runtime configuration of the cluster's control plane.

What problem does this solve?

When a cluster undergoes an upgrade, the kube-apiservers existing at different versions in that scenario can serve different sets (groups, versions, resources) of built-in resources. A resource request made in this scenario may be served by any of the available apiservers, potentially resulting in the request ending up at an apiserver that may not be aware of the requested resource; consequently it being served a 404 not found error which is incorrect. Furthermore, incorrect serving of the 404 errors can lead to serious consequences such as namespace deletion being blocked incorrectly or objects being garbage collected mistakenly.

Kubernetes v1.28: Introducing native sidecar containers

Fri, 25 Aug 2023 00:00:00 +0000

This post explains how to use the new sidecar feature, which enables restartable init containers and is available in alpha in Kubernetes 1.28. We want your feedback so that we can graduate this feature as soon as possible.

The concept of a “sidecar” has been part of Kubernetes since nearly the very beginning. In 2015, sidecars were described in a blog post about composite containers as additional containers that “extend and enhance the ‘main’ container”. Sidecar containers have become a common Kubernetes deployment pattern and are often used for network proxies or as part of a logging system. Until now, sidecars were a concept that Kubernetes users applied without native support. The lack of native support has caused some usage friction, which this enhancement aims to resolve.

Kubernetes 1.28: Beta support for using swap on Linux

Thu, 24 Aug 2023 10:00:00 -0800

The 1.22 release introduced Alpha support for configuring swap memory usage for Kubernetes workloads running on Linux on a per-node basis. Now, in release 1.28, support for swap on Linux nodes has graduated to Beta, along with many new improvements.

Prior to version 1.22, Kubernetes did not provide support for swap memory on Linux systems. This was due to the inherent difficulty in guaranteeing and accounting for pod memory utilization when swap memory was involved. As a result, swap support was deemed out of scope in the initial design of Kubernetes, and the default behavior of a kubelet was to fail to start if swap memory was detected on a node.

Kubernetes 1.28: Node podresources API Graduates to GA

Wed, 23 Aug 2023 00:00:00 +0000

The podresources API is an API served by the kubelet locally on the node, which exposes the compute resources exclusively allocated to containers. With the release of Kubernetes 1.28, that API is now Generally Available.

What problem does it solve?

The kubelet can allocate exclusive resources to containers, like CPUs, granting exclusive access to full cores or memory, either regions or hugepages. Workloads which require high performance, or low latency (or both) leverage these features. The kubelet also can assign devices to containers. Collectively, these features which enable exclusive assignments are known as "resource managers".

Kubernetes 1.28: Improved failure handling for Jobs

Mon, 21 Aug 2023 00:00:00 +0000

This blog discusses two new features in Kubernetes 1.28 to improve Jobs for batch users: Pod replacement policy and Backoff limit per index.

These features continue the effort started by the Pod failure policy to improve the handling of Pod failures in a Job.

Pod replacement policy

By default, when a pod enters a terminating state (e.g. due to preemption or eviction), Kubernetes immediately creates a replacement Pod. Therefore, both Pods are running at the same time. In API terms, a pod is considered terminating when it has a deletionTimestamp and it has a phase Pending or Running.

Kubernetes v1.28: Retroactive Default StorageClass move to GA

Fri, 18 Aug 2023 00:00:00 +0000

Announcing graduation to General Availability (GA) - Retroactive Default StorageClass Assignment in Kubernetes v1.28!

Kubernetes SIG Storage team is thrilled to announce that the "Retroactive Default StorageClass Assignment" feature, introduced as an alpha in Kubernetes v1.25, has now graduated to GA and is officially part of the Kubernetes v1.28 release. This enhancement brings a significant improvement to how default StorageClasses are assigned to PersistentVolumeClaims (PVCs).

With this feature enabled, you no longer need to create a default StorageClass first and then a PVC to assign the class. Instead, any PVCs without a StorageClass assigned will now be retroactively updated to include the default StorageClass. This enhancement ensures that PVCs no longer get stuck in an unbound state, and storage provisioning works seamlessly, even when a default StorageClass is not defined at the time of PVC creation.

Kubernetes 1.28: Non-Graceful Node Shutdown Moves to GA

Wed, 16 Aug 2023 10:00:00 -0800

The Kubernetes Non-Graceful Node Shutdown feature is now GA in Kubernetes v1.28. It was introduced as alpha in Kubernetes v1.24, and promoted to beta in Kubernetes v1.26. This feature allows stateful workloads to restart on a different node if the original node is shutdown unexpectedly or ends up in a non-recoverable state such as the hardware failure or unresponsive OS.

What is a Non-Graceful Node Shutdown

In a Kubernetes cluster, a node can be shutdown in a planned graceful way or unexpectedly because of reasons such as power outage or something else external. A node shutdown could lead to workload failure if the node is not drained before the shutdown. A node shutdown can be either graceful or non-graceful.

pkgs.k8s.io: Introducing Kubernetes Community-Owned Package Repositories

Tue, 15 Aug 2023 20:00:00 +0000

On behalf of Kubernetes SIG Release, I am very excited to introduce the Kubernetes community-owned software repositories for Debian and RPM packages: pkgs.k8s.io! The new package repositories are replacement for the Google-hosted package repositories (apt.kubernetes.io and yum.kubernetes.io) that we've been using since Kubernetes v1.5.

This blog post contains information about these new package repositories, what does it mean to you as an end user, and how to migrate to the new repositories.

Kubernetes v1.28: Planternetes

Tue, 15 Aug 2023 12:00:00 +0000

Announcing the release of Kubernetes v1.28 Planternetes, the second release of 2023!

This release consists of 45 enhancements. Of those enhancements, 19 are entering Alpha, 14 have graduated to Beta, and 12 have graduated to Stable.

Release Theme And Logo

Kubernetes v1.28: Planternetes

The theme for Kubernetes v1.28 is Planternetes.

Each Kubernetes release is the culmination of the hard work of thousands of individuals from our community. The people behind this release come from a wide range of backgrounds, some of us industry veterans, parents, others students and newcomers to open-source. We combine our unique experience to create a collective artifact with global impact.

Spotlight on SIG ContribEx

Mon, 14 Aug 2023 00:00:00 +0000

Welcome to the world of Kubernetes and its vibrant contributor community! In this blog post, we'll be shining a spotlight on the Special Interest Group for Contributor Experience (SIG ContribEx), an essential component of the Kubernetes project.

SIG ContribEx in Kubernetes is responsible for developing and maintaining a healthy and productive community of contributors to the project. This involves identifying and addressing bottlenecks that may hinder the project's growth and feature velocity, such as pull request latency and the number of open pull requests and issues.

Spotlight on SIG CLI

Thu, 20 Jul 2023 00:00:00 +0000

In the world of Kubernetes, managing containerized applications at scale requires powerful and efficient tools. The command-line interface (CLI) is an integral part of any developer or operator’s toolkit, offering a convenient and flexible way to interact with a Kubernetes cluster.

SIG CLI plays a crucial role in improving the Kubernetes CLI experience by focusing on the development and enhancement of kubectl, the primary command-line tool for Kubernetes.

In this SIG CLI Spotlight, Arpit Agrawal, SIG ContribEx-Comms team member, talked with Katrina Verey, Tech Lead & Chair of SIG CLI,and Maciej Szulik, SIG CLI Batch Lead, about SIG CLI, current projects, challenges and how anyone can get involved.

Confidential Kubernetes: Use Confidential Virtual Machines and Enclaves to improve your cluster security

Thu, 06 Jul 2023 00:00:00 +0000

In this blog post, we will introduce the concept of Confidential Computing (CC) to improve any computing environment's security and privacy properties. Further, we will show how the Cloud-Native ecosystem, particularly Kubernetes, can benefit from the new compute paradigm.

Confidential Computing is a concept that has been introduced previously in the cloud-native world. The Confidential Computing Consortium (CCC) is a project community in the Linux Foundation that already worked on Defining and Enabling Confidential Computing. In the Whitepaper, they provide a great motivation for the use of Confidential Computing:

Verifying Container Image Signatures Within CRI Runtimes

Thu, 29 Jun 2023 00:00:00 +0000

The Kubernetes community has been signing their container image-based artifacts since release v1.24. While the graduation of the corresponding enhancement from alpha to beta in v1.26 introduced signatures for the binary artifacts, other projects followed the approach by providing image signatures for their releases, too. This means that they either create the signatures within their own CI/CD pipelines, for example by using GitHub actions, or rely on the Kubernetes image promotion process to automatically sign the images by proposing pull requests to the k/k8s.io repository. A requirement for using this process is that the project is part of the kubernetes or kubernetes-sigs GitHub organization, so that they can utilize the community infrastructure for pushing images into staging buckets.

dl.k8s.io to adopt a Content Delivery Network

Fri, 09 Jun 2023 00:00:00 +0000

We're happy to announce that dl.k8s.io, home of the official Kubernetes binaries, will soon be powered by Fastly.

Fastly is known for its high-performance content delivery network (CDN) designed to deliver content quickly and reliably around the world. With its powerful network, Fastly will help us deliver official Kubernetes binaries to users faster and more reliably than ever before.

The decision to use Fastly was made after an extensive evaluation process in which we carefully evaluated several potential content delivery network providers. Ultimately, we chose Fastly because of their commitment to the open internet and proven track record of delivering fast and secure digital experiences to some of the most known open source projects (through their Fast Forward program).

Using OCI artifacts to distribute security profiles for seccomp, SELinux and AppArmor

Wed, 24 May 2023 00:00:00 +0000

The Security Profiles Operator (SPO) makes managing seccomp, SELinux and AppArmor profiles within Kubernetes easier than ever. It allows cluster administrators to define the profiles in a predefined custom resource YAML, which then gets distributed by the SPO into the whole cluster. Modification and removal of the security profiles are managed by the operator in the same way, but that’s a small subset of its capabilities.

Another core feature of the SPO is being able to stack seccomp profiles. This means that users can define a baseProfileName in the YAML specification, which then gets automatically resolved by the operator and combines the syscall rules. If a base profile has another baseProfileName, then the operator will recursively resolve the profiles up to a certain depth. A common use case is to define base profiles for low level container runtimes (like runc or crun) which then contain syscalls which are required in any case to run the container. Alternatively, application developers can define seccomp base profiles for their standard distribution containers and stack dedicated profiles for the application logic on top. This way developers can focus on maintaining seccomp profiles which are way simpler and scoped to the application logic, without having a need to take the whole infrastructure setup into account.

Having fun with seccomp profiles on the edge

Thu, 18 May 2023 00:00:00 +0000

The Security Profiles Operator (SPO) is a feature-rich operator for Kubernetes to make managing seccomp, SELinux and AppArmor profiles easier than ever. Recording those profiles from scratch is one of the key features of this operator, which usually involves the integration into large CI/CD systems. Being able to test the recording capabilities of the operator in edge cases is one of the recent development efforts of the SPO and makes it excitingly easy to play around with seccomp profiles.

Kubernetes 1.27: KMS V2 Moves to Beta

Tue, 16 May 2023 00:00:00 +0000

With Kubernetes 1.27, we (SIG Auth) are moving Key Management Service (KMS) v2 API to beta.

What is KMS?

One of the first things to consider when securing a Kubernetes cluster is encrypting etcd data at rest. KMS provides an interface for a provider to utilize a key stored in an external key service to perform this encryption.

KMS v1 has been a feature of Kubernetes since version 1.10, and is currently in beta as of version v1.12. KMS v2 was introduced as alpha in v1.25.

Kubernetes 1.27: updates on speeding up Pod startup

Mon, 15 May 2023 00:00:00 +0000

How can Pod start-up be accelerated on nodes in large clusters? This is a common issue that cluster administrators may face.

This blog post focuses on methods to speed up pod start-up from the kubelet side. It does not involve the creation time of pods by controller-manager through kube-apiserver, nor does it include scheduling time for pods or webhooks executed on it.

We have mentioned some important factors here to consider from the kubelet's perspective, but this is not an exhaustive list. As Kubernetes v1.27 is released, this blog highlights significant changes in v1.27 that aid in speeding up pod start-up.

Kubernetes 1.27: In-place Resource Resize for Kubernetes Pods (alpha)

Fri, 12 May 2023 00:00:00 +0000

If you have deployed Kubernetes pods with CPU and/or memory resources specified, you may have noticed that changing the resource values involves restarting the pod. This has been a disruptive operation for running workloads... until now.

In Kubernetes v1.27, we have added a new alpha feature that allows users to resize CPU/memory resources allocated to pods without restarting the containers. To facilitate this, the resources field in a pod's containers now allow mutation for cpu and memory resources. They can be changed simply by patching the running pod spec.

Kubernetes 1.27: Avoid Collisions Assigning Ports to NodePort Services

Thu, 11 May 2023 00:00:00 +0000

In Kubernetes, a Service can be used to provide a unified traffic endpoint for applications running on a set of Pods. Clients can use the virtual IP address (or VIP) provided by the Service for access, and Kubernetes provides load balancing for traffic accessing different back-end Pods, but a ClusterIP type of Service is limited to providing access to nodes within the cluster, while traffic from outside the cluster cannot be routed. One way to solve this problem is to use a type: NodePort Service, which sets up a mapping to a specific port of all nodes in the cluster, thus redirecting traffic from the outside to the inside of the cluster.

Kubernetes 1.27: Safer, More Performant Pruning in kubectl apply

Tue, 09 May 2023 00:00:00 +0000

Declarative configuration management with the kubectl apply command is the gold standard approach to creating or modifying Kubernetes resources. However, one challenge it presents is the deletion of resources that are no longer needed. In Kubernetes version 1.5, the --prune flag was introduced to address this issue, allowing kubectl apply to automatically clean up previously applied resources removed from the current configuration.

Unfortunately, that existing implementation of --prune has design flaws that diminish its performance and can result in unexpected behaviors. The main issue stems from the lack of explicit encoding of the previously applied set by the preceding apply operation, necessitating error-prone dynamic discovery. Object leakage, inadvertent over-selection of resources, and limited compatibility with custom resources are a few notable drawbacks of this implementation. Moreover, its coupling to client-side apply hinders user upgrades to the superior server-side apply mechanism.

Kubernetes 1.27: Introducing An API For Volume Group Snapshots

Mon, 08 May 2023 00:00:00 +0000

Volume group snapshot is introduced as an Alpha feature in Kubernetes v1.27. This feature introduces a Kubernetes API that allows users to take crash consistent snapshots for multiple volumes together. It uses a label selector to group multiple PersistentVolumeClaims for snapshotting. This new feature is only supported for CSI volume drivers.

An overview of volume group snapshots

Some storage systems provide the ability to create a crash consistent snapshot of multiple volumes. A group snapshot represents “copies” from multiple volumes that are taken at the same point-in-time. A group snapshot can be used either to rehydrate new volumes (pre-populated with the snapshot data) or to restore existing volumes to a previous state (represented by the snapshots).

Kubernetes 1.27: Quality-of-Service for Memory Resources (alpha)

Fri, 05 May 2023 00:00:00 +0000

Kubernetes v1.27, released in April 2023, introduced changes to Memory QoS (alpha) to improve memory management capabilites in Linux nodes.

Support for Memory QoS was initially added in Kubernetes v1.22, and later some limitations around the formula for calculating memory.high were identified. These limitations are addressed in Kubernetes v1.27.

Background

Kubernetes allows you to optionally specify how much of each resources a container needs in the Pod specification. The most common resources to specify are CPU and Memory.

Kubernetes 1.27: StatefulSet PVC Auto-Deletion (beta)

Thu, 04 May 2023 00:00:00 +0000

Kubernetes v1.27 graduated to beta a new policy mechanism for StatefulSets that controls the lifetime of their PersistentVolumeClaims (PVCs). The new PVC retention policy lets users specify if the PVCs generated from the StatefulSet spec template should be automatically deleted or retrained when the StatefulSet is deleted or replicas in the StatefulSet are scaled down.

What problem does this solve?

A StatefulSet spec can include Pod and PVC templates. When a replica is first created, the Kubernetes control plane creates a PVC for that replica if one does not already exist. The behavior before the PVC retention policy was that the control plane never cleaned up the PVCs created for StatefulSets - this was left up to the cluster administrator, or to some add-on automation that you’d have to find, check suitability, and deploy. The common pattern for managing PVCs, either manually or through tools such as Helm, is that the PVCs are tracked by the tool that manages them, with explicit lifecycle. Workflows that use StatefulSets must determine on their own what PVCs are created by a StatefulSet and what their lifecycle should be.

Kubernetes 1.27: HorizontalPodAutoscaler ContainerResource type metric moves to beta

Tue, 02 May 2023 12:00:00 +0800

Kubernetes 1.20 introduced the ContainerResource type metric in HorizontalPodAutoscaler (HPA).

In Kubernetes 1.27, this feature moves to beta and the corresponding feature gate (HPAContainerMetrics) gets enabled by default.

What is the ContainerResource type metric

The ContainerResource type metric allows us to configure the autoscaling based on resource usage of individual containers.

In the following example, the HPA controller scales the target so that the average utilization of the cpu in the application container of all the pods is around 60%. (See the algorithm details to know how the desired replica number is calculated exactly)

Kubernetes 1.27: StatefulSet Start Ordinal Simplifies Migration

Fri, 28 Apr 2023 00:00:00 +0000

Kubernetes v1.26 introduced a new, alpha-level feature for StatefulSets that controls the ordinal numbering of Pod replicas. As of Kubernetes v1.27, this feature is now beta. Ordinals can start from arbitrary non-negative numbers. This blog post will discuss how this feature can be used.

Background

StatefulSets ordinals provide sequential identities for pod replicas. When using OrderedReady Pod management Pods are created from ordinal index 0 up to N-1.

With Kubernetes today, orchestrating a StatefulSet migration across clusters is challenging. Backup and restore solutions exist, but these require the application to be scaled down to zero replicas prior to migration. In today's fully connected world, even planned application downtime may not allow you to meet your business goals. You could use Cascading Delete or On Delete to migrate individual pods, however this is error prone and tedious to manage. You lose the self-healing benefit of the StatefulSet controller when your Pods fail or are evicted.

Updates to the Auto-refreshing Official CVE Feed

Tue, 25 Apr 2023 00:00:00 +0000

Since launching the Auto-refreshing Official CVE feed as an alpha feature in the 1.25 release, we have made significant improvements and updates. We are excited to announce the release of the beta version of the feed. This blog post will outline the feedback received, the changes made, and talk about how you can help as we prepare to make this a stable feature in a future Kubernetes Release.

Feedback from end-users

SIG Security received some feedback from end-users:

Kubernetes 1.27: Server Side Field Validation and OpenAPI V3 move to GA

Mon, 24 Apr 2023 00:00:00 +0000

Before Kubernetes v1.8 (!), typos, mis-indentations or minor errors in YAMLs could have catastrophic consequences (e.g. a typo like forgetting the trailing s in replica: 1000 could cause an outage, because the value would be ignored and missing, forcing a reset of replicas back to 1). This was solved back then by fetching the OpenAPI v2 in kubectl and using it to verify that fields were correct and present before applying. Unfortunately, at that time, Custom Resource Definitions didn’t exist, and the code was written under that assumption. When CRDs were later introduced, the lack of flexibility in the validation code forced some hard decisions in the way CRDs exposed their schema, leaving us in a cycle of bad validation causing bad OpenAPI and vice-versa. With the new OpenAPI v3 and Server Field Validation being GA in 1.27, we’ve now solved both of these problems.

Kubernetes 1.27: Query Node Logs Using The Kubelet API

Fri, 21 Apr 2023 00:00:00 +0000

Kubernetes 1.27 introduced a new feature called Node log query that allows viewing logs of services running on the node.

What problem does it solve?

Cluster administrators face issues when debugging malfunctioning services running on the node. They usually have to SSH or RDP into the node to view the logs of the service to debug the issue. The Node log query feature helps with this scenario by allowing the cluster administrator to view the logs using kubectl. This is especially useful with Windows nodes where you run into the issue of the node going to the ready state but containers not coming up due to CNI misconfigurations and other issues that are not easily identifiable by looking at the Pod status.

Kubernetes 1.27: Single Pod Access Mode for PersistentVolumes Graduates to Beta

Thu, 20 Apr 2023 00:00:00 +0000

With the release of Kubernetes v1.27 the ReadWriteOncePod feature has graduated to beta. In this blog post, we'll take a closer look at this feature, what it does, and how it has evolved in the beta release.

What is ReadWriteOncePod?

ReadWriteOncePod is a new access mode for PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) introduced in Kubernetes v1.22. This access mode enables you to restrict volume access to a single pod in the cluster, ensuring that only one pod can write to the volume at a time. This can be particularly useful for stateful workloads that require single-writer access to storage.

Kubernetes 1.27: Efficient SELinux volume relabeling (Beta)

Tue, 18 Apr 2023 10:00:00 -0800

The problem

On Linux with Security-Enhanced Linux (SELinux) enabled, it's traditionally the container runtime that applies SELinux labels to a Pod and all its volumes. Kubernetes only passes the SELinux label from a Pod's securityContext fields to the container runtime.

The container runtime then recursively changes SELinux label on all files that are visible to the Pod's containers. This can be time-consuming if there are many files on the volume, especially when the volume is on a remote filesystem.

Kubernetes 1.27: More fine-grained pod topology spread policies reached beta

Mon, 17 Apr 2023 00:00:00 +0000

In Kubernetes v1.19, Pod topology spread constraints went to general availability (GA).

As time passed, we - SIG Scheduling - received feedback from users, and, as a result, we're actively working on improving the Topology Spread feature via three KEPs. All of these features have reached beta in Kubernetes v1.27 and are enabled by default.

This blog post introduces each feature and the use case behind each of them.

KEP-3022: min domains in Pod Topology Spread

Pod Topology Spread has the maxSkew parameter to define the degree to which Pods may be unevenly distributed.

Kubernetes v1.27: Chill Vibes

Tue, 11 Apr 2023 00:00:00 +0000

Announcing the release of Kubernetes v1.27, the first release of 2023!

This release consist of 60 enhancements. 18 of those enhancements are entering Alpha, 29 are graduating to Beta, and 13 are graduating to Stable.

Release theme and logo

Kubernetes v1.27: Chill Vibes

The theme for Kubernetes v1.27 is Chill Vibes.

It's a little silly, but there were some important shifts in this release that helped inspire the theme. Throughout a typical Kubernetes release cycle, there are several deadlines that features need to meet to remain included. If a feature misses any of these deadlines, there is an exception process they can go through. Handling these exceptions is a very normal part of the release. But v1.27 is the first release that anyone can remember where we didn't receive a single exception request after the enhancements freeze. Even as the release progressed, things remained much calmer than any of us are used to.

Keeping Kubernetes Secure with Updated Go Versions

Thu, 06 Apr 2023 00:00:00 +0000

The problem

Since v1.19 (released in 2020), the Kubernetes project provides 12-14 months of patch releases for each minor version. This enables users to qualify and adopt Kubernetes versions in an annual upgrade cycle and receive security fixes for a year.

The Go project releases new minor versions twice a year, and provides security fixes for the last two minor versions, resulting in about a year of support for each Go version. Even though each new Kubernetes minor version is built with a supported Go version when it is first released, that Go version falls out of support before the Kubernetes minor version does, and the lengthened Kubernetes patch support since v1.19 only widened that gap.

Kubernetes Validating Admission Policies: A Practical Example

Thu, 30 Mar 2023 00:00:00 +0000

Admission control is an important part of the Kubernetes control plane, with several internal features depending on the ability to approve or change an API object as it is submitted to the server. It is also useful for an administrator to be able to define business logic, or policies, regarding what objects can be admitted into a cluster. To better support that use case, Kubernetes introduced external admission control in v1.7.

Kubernetes Removals and Major Changes In v1.27

Fri, 17 Mar 2023 14:00:00 +0000

As Kubernetes develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. Based on the information available at this point in the v1.27 release process, which is still ongoing and can introduce additional changes, this article identifies and describes some of the planned changes for the Kubernetes v1.27 release.

A note about the k8s.gcr.io redirect to registry.k8s.io

To host its container images, the Kubernetes project uses a community-owned image registry called registry.k8s.io. On March 20th, all traffic from the out-of-date k8s.gcr.io registry will be redirected to registry.k8s.io. The deprecated k8s.gcr.io registry will eventually be phased out.

k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know

Fri, 10 Mar 2023 17:00:00 +0000

On Monday, March 20th, the k8s.gcr.io registry will be redirected to the community owned registry, registry.k8s.io .

TL;DR: What you need to know about this change

On Monday, March 20th, traffic from the older k8s.gcr.io registry will be redirected to registry.k8s.io with the eventual goal of sunsetting k8s.gcr.io.
If you run in a restricted environment, and apply strict domain name or IP address access policies limited to k8s.gcr.io, the image pulls will not function after k8s.gcr.io starts redirecting to the new registry.
A small subset of non-standard clients do not handle HTTP redirects by image registries, and will need to be pointed directly at registry.k8s.io.
The redirect is a stopgap to assist users in making the switch. The deprecated k8s.gcr.io registry will be phased out at some point. Please update your manifests as soon as possible to point to registry.k8s.io.
If you host your own image registry, you can copy images you need there as well to reduce traffic to community owned registries.

If you think you may be impacted, or would like to know more about this change, please keep reading.

Forensic container analysis

Fri, 10 Mar 2023 00:00:00 +0000

In my previous article, Forensic container checkpointing in Kubernetes, I introduced checkpointing in Kubernetes and how it has to be setup and how it can be used. The name of the feature is Forensic container checkpointing, but I did not go into any details how to do the actual analysis of the checkpoint created by Kubernetes. In this article I want to provide details how the checkpoint can be analyzed.

Checkpointing is still an alpha feature in Kubernetes and this article wants to provide a preview how the feature might work in the future.

Introducing KWOK: Kubernetes WithOut Kubelet

Wed, 01 Mar 2023 00:00:00 +0000

Have you ever wondered how to set up a cluster of thousands of nodes just in seconds, how to simulate real nodes with a low resource footprint, and how to test your Kubernetes controller at scale without spending much on infrastructure?

If you answered "yes" to any of these questions, then you might be interested in KWOK, a toolkit that enables you to create a cluster of thousands of nodes in seconds.

Free Katacoda Kubernetes Tutorials Are Shutting Down

Tue, 14 Feb 2023 00:00:00 +0000

Katacoda, the popular learning platform from O’Reilly that has been helping people learn all about Java, Docker, Kubernetes, Python, Go, C++, and more, shut down for public use in June 2022. However, tutorials specifically for Kubernetes, linked from the Kubernetes website for our project’s users and contributors, remained available and active after this change. Unfortunately, this will no longer be the case, and Katacoda tutorials for learning Kubernetes will cease working after March 31st, 2023.

k8s.gcr.io Image Registry Will Be Frozen From the 3rd of April 2023

Mon, 06 Feb 2023 00:00:00 +0000

The Kubernetes project runs a community-owned image registry called registry.k8s.io to host its container images. On the 3rd of April 2023, the old registry k8s.gcr.io will be frozen and no further images for Kubernetes and related subprojects will be pushed to the old registry.

This registry registry.k8s.io replaced the old one and has been generally available for several months. We have published a blog post about its benefits to the community and the Kubernetes project. This post also announced that future versions of Kubernetes will not be available in the old registry. Now that time has come.

Spotlight on SIG Instrumentation

Fri, 03 Feb 2023 00:00:00 +0000

Observability requires the right data at the right time for the right consumer (human or piece of software) to make the right decision. In the context of Kubernetes, having best practices for cluster observability across all Kubernetes components is crucial.

SIG Instrumentation helps to address this issue by providing best practices and tools that all other SIGs use to instrument Kubernetes components-like the API server, scheduler, kubelet and kube-controller-manager.

In this SIG Instrumentation spotlight, Imran Noor Mohamed, SIG ContribEx-Comms tech lead talked with Elana Hashman, and Han Kang, chairs of SIG Instrumentation, on how the SIG is organized, what are the current challenges and how anyone can get involved and contribute.

Consider All Microservices Vulnerable — And Monitor Their Behavior

Fri, 20 Jan 2023 00:00:00 +0000

This post warns Devops from a false sense of security. Following security best practices when developing and configuring microservices do not result in non-vulnerable microservices. The post shows that although all deployed microservices are vulnerable, there is much that can be done to ensure microservices are not exploited. It explains how analyzing the behavior of clients and services from a security standpoint, named here "Security-Behavior Analytics", can protect the deployed vulnerable microservices. It points to Guard, an open source project offering security-behavior monitoring and control of Kubernetes microservices presumed vulnerable.

Protect Your Mission-Critical Pods From Eviction With PriorityClass

Thu, 12 Jan 2023 00:00:00 +0000

Kubernetes has been widely adopted, and many organizations use it as their de-facto orchestration engine for running workloads that need to be created and deleted frequently.

Therefore, proper scheduling of the pods is key to ensuring that application pods are up and running within the Kubernetes cluster without any issues. This article delves into the use cases around resource management by leveraging the PriorityClass object to protect mission-critical or high-priority pods from getting evicted and making sure that the application pods are up, running, and serving traffic.

Kubernetes 1.26: Eviction policy for unhealthy pods guarded by PodDisruptionBudgets

Fri, 06 Jan 2023 00:00:00 +0000

Ensuring the disruptions to your applications do not affect its availability isn't a simple task. Last month's release of Kubernetes v1.26 lets you specify an unhealthy pod eviction policy for PodDisruptionBudgets (PDBs) to help you maintain that availability during node management operations. In this article, we will dive deeper into what modifications were introduced for PDBs to give application owners greater flexibility in managing disruptions.

What problems does this solve?

API-initiated eviction of pods respects PodDisruptionBudgets (PDBs). This means that a requested voluntary disruption via an eviction to a Pod, should not disrupt a guarded application and .status.currentHealthy of a PDB should not fall below .status.desiredHealthy. Running pods that are Unhealthy do not count towards the PDB status, but eviction of these is only possible in case the application is not disrupted. This helps disrupted or not yet started application to achieve availability as soon as possible without additional downtime that would be caused by evictions.

Kubernetes v1.26: Retroactive Default StorageClass

Thu, 05 Jan 2023 00:00:00 +0000

The v1.25 release of Kubernetes introduced an alpha feature to change how a default StorageClass was assigned to a PersistentVolumeClaim (PVC). With the feature enabled, you no longer need to create a default StorageClass first and PVC second to assign the class. Additionally, any PVCs without a StorageClass assigned can be updated later. This feature was graduated to beta in Kubernetes v1.26.

You can read retroactive default StorageClass assignment in the Kubernetes documentation for more details about how to use that, or you can read on to learn about why the Kubernetes project is making this change.

Kubernetes v1.26: Alpha support for cross-namespace storage data sources

Mon, 02 Jan 2023 00:00:00 +0000

Kubernetes v1.26, released last month, introduced an alpha feature that lets you specify a data source for a PersistentVolumeClaim, even where the source data belong to a different namespace. With the new feature enabled, you specify a namespace in the dataSourceRef field of a new PersistentVolumeClaim. Once Kubernetes checks that access is OK, the new PersistentVolume can populate its data from the storage source specified in that other namespace. Before Kubernetes v1.26, provided your cluster had the AnyVolumeDataSource feature enabled, you could already provision new volumes from a data source in the same namespace. However, that only worked for the data source in the same namespace, therefore users couldn't provision a PersistentVolume with a claim in one namespace from a data source in other namespace. To solve this problem, Kubernetes v1.26 added a new alpha namespace field to dataSourceRef field in PersistentVolumeClaim the API.

Kubernetes v1.26: Advancements in Kubernetes Traffic Engineering

Fri, 30 Dec 2022 00:00:00 +0000

Kubernetes v1.26 includes significant advancements in network traffic engineering with the graduation of two features (Service internal traffic policy support, and EndpointSlice terminating conditions) to GA, and a third feature (Proxy terminating endpoints) to beta. The combination of these enhancements aims to address short-comings in traffic engineering that people face today, and unlock new capabilities for the future.

Traffic Loss from Load Balancers During Rolling Updates

Prior to Kubernetes v1.26, clusters could experience loss of traffic from Service load balancers during rolling updates when setting the externalTrafficPolicy field to Local. There are a lot of moving parts at play here so a quick overview of how Kubernetes manages load balancers might help!

Kubernetes 1.26: Job Tracking, to Support Massively Parallel Batch Workloads, Is Generally Available

Thu, 29 Dec 2022 00:00:00 +0000

The Kubernetes 1.26 release includes a stable implementation of the Job controller that can reliably track a large amount of Jobs with high levels of parallelism. SIG Apps and WG Batch have worked on this foundational improvement since Kubernetes 1.22. After multiple iterations and scale verifications, this is now the default implementation of the Job controller.

Paired with the Indexed completion mode, the Job controller can handle massively parallel batch Jobs, supporting up to 100k concurrent Pods.

Kubernetes v1.26: CPUManager goes GA

Tue, 27 Dec 2022 00:00:00 +0000

The CPU Manager is a part of the kubelet, the Kubernetes node agent, which enables the user to allocate exclusive CPUs to containers. Since Kubernetes v1.10, where it graduated to Beta, the CPU Manager proved itself reliable and fulfilled its role of allocating exclusive CPUs to containers, so adoption has steadily grown making it a staple component of performance-critical and low-latency setups. Over time, most changes were about bugfixes or internal refactoring, with the following noteworthy user-visible changes:

Kubernetes 1.26: Pod Scheduling Readiness

Mon, 26 Dec 2022 00:00:00 +0000

Kubernetes 1.26 introduced a new Pod feature: scheduling gates. In Kubernetes, scheduling gates are keys that tell the scheduler when a Pod is ready to be considered for scheduling.

What problem does it solve?

When a Pod is created, the scheduler will continuously attempt to find a node that fits it. This infinite loop continues until the scheduler either finds a node for the Pod, or the Pod gets deleted.

Kubernetes 1.26: Support for Passing Pod fsGroup to CSI Drivers At Mount Time

Fri, 23 Dec 2022 00:00:00 +0000

Delegation of fsGroup to CSI drivers was first introduced as alpha in Kubernetes 1.22, and graduated to beta in Kubernetes 1.25. For Kubernetes 1.26, we are happy to announce that this feature has graduated to General Availability (GA).

In this release, if you specify a fsGroup in the security context, for a (Linux) Pod, all processes in the pod's containers are part of the additional group that you specified.

In previous Kubernetes releases, the kubelet would always apply the fsGroup ownership and permission changes to files in the volume according to the policy you specified in the Pod's .spec.securityContext.fsGroupChangePolicy field.

Kubernetes v1.26: GA Support for Kubelet Credential Providers

Thu, 22 Dec 2022 00:00:00 +0000

Kubernetes v1.26 introduced generally available (GA) support for kubelet credential provider plugins, offering an extensible plugin framework to dynamically fetch credentials for any container image registry.

Background

Kubernetes supports the ability to dynamically fetch credentials for a container registry service. Prior to Kubernetes v1.20, this capability was compiled into the kubelet and only available for Amazon Elastic Container Registry, Azure Container Registry, and Google Cloud Container Registry.

Figure 1: Kubelet built-in credential provider support for Amazon Elastic Container Registry, Azure Container Registry, and Google Cloud Container Registry.

Kubernetes 1.26: Introducing Validating Admission Policies

Tue, 20 Dec 2022 00:00:00 +0000

In Kubernetes 1.26, the 1st alpha release of validating admission policies is available!

Validating admission policies use the Common Expression Language (CEL) to offer a declarative, in-process alternative to validating admission webhooks.

CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions. This enhancement expands the use of CEL in Kubernetes to support a far wider range of admission use cases.

Admission webhooks can be burdensome to develop and operate. Webhook developers must implement and maintain a webhook binary to handle admission requests. Also, admission webhooks are complex to operate. Each webhook must be deployed, monitored and have a well defined upgrade and rollback plan. To make matters worse, if a webhook times out or becomes unavailable, the Kubernetes control plane can become unavailable. This enhancement avoids much of this complexity of admission webhooks by embedding CEL expressions into Kubernetes resources instead of calling out to a remote webhook binary.

Kubernetes 1.26: Device Manager graduates to GA

Mon, 19 Dec 2022 00:00:00 +0000

The Device Plugin framework was introduced in the Kubernetes v1.8 release as a vendor independent framework to enable discovery, advertisement and allocation of external devices without modifying core Kubernetes. The feature graduated to Beta in v1.10. With the recent release of Kubernetes v1.26, Device Manager is now generally available (GA).

Within the kubelet, the Device Manager facilitates communication with device plugins using gRPC through Unix sockets. Device Manager and Device plugins both act as gRPC servers and clients by serving and connecting to the exposed gRPC services respectively. Device plugins serve a gRPC service that kubelet connects to for device discovery, advertisement (as extended resources) and allocation. Device Manager connects to the Registration gRPC service served by kubelet to register itself with kubelet.

Kubernetes 1.26: Non-Graceful Node Shutdown Moves to Beta

Fri, 16 Dec 2022 10:00:00 -0800

Kubernetes v1.24 introduced an alpha quality implementation of improvements for handling a non-graceful node shutdown. In Kubernetes v1.26, this feature moves to beta. This feature allows stateful workloads to failover to a different node after the original node is shut down or in a non-recoverable state, such as the hardware failure or broken OS.

What is a node shutdown in Kubernetes?

In a Kubernetes cluster, it is possible for a node to shut down. This could happen either in a planned way or it could happen unexpectedly. You may plan for a security patch, or a kernel upgrade and need to reboot the node, or it may shut down due to preemption of VM instances. A node may also shut down due to a hardware failure or a software problem.

Kubernetes 1.26: Alpha API For Dynamic Resource Allocation

Thu, 15 Dec 2022 00:00:00 +0000

Dynamic resource allocation is a new API for requesting resources. It is a generalization of the persistent volumes API for generic resources, making it possible to:

access the same resource instance in different pods and containers,
attach arbitrary constraints to a resource request to get the exact resource you are looking for,
initialize a resource according to parameters provided by the user.

Third-party resource drivers are responsible for interpreting these parameters as well as tracking and allocating resources as requests come in.

Kubernetes 1.26: Windows HostProcess Containers Are Generally Available

Tue, 13 Dec 2022 00:00:00 +0000

The long-awaited day has arrived: HostProcess containers, the Windows equivalent to Linux privileged containers, has finally made it to GA in Kubernetes 1.26!

What are HostProcess containers and why are they useful?

Cluster operators are often faced with the need to configure their nodes upon provisioning such as installing Windows services, configuring registry keys, managing TLS certificates, making network configuration changes, or even deploying monitoring tools such as a Prometheus's node-exporter. Previously, performing these actions on Windows nodes was usually done by running PowerShell scripts over SSH or WinRM sessions and/or working with your cloud provider's virtual machine management tooling. HostProcess containers now enable you to do all of this and more with minimal effort using Kubernetes native APIs.

Kubernetes 1.26: We're now signing our binary release artifacts!

Mon, 12 Dec 2022 00:00:00 +0000

The Kubernetes Special Interest Group (SIG) Release is proud to announce that we are digitally signing all release artifacts, and that this aspect of Kubernetes has now reached beta.

Signing artifacts provides end users a chance to verify the integrity of the downloaded resource. It allows to mitigate man-in-the-middle attacks directly on the client side and therefore ensures the trustfulness of the remote serving the artifacts. The overall goal of out past work was to define the used tooling for signing all Kubernetes related artifacts as well as providing a standard signing process for related projects (for example for those in kubernetes-sigs).

Kubernetes v1.26: Electrifying

Fri, 09 Dec 2022 00:00:00 +0000

It's with immense joy that we announce the release of Kubernetes v1.26!

This release includes a total of 37 enhancements: eleven of them are graduating to Stable, ten are graduating to Beta, and sixteen of them are entering Alpha. We also have twelve features being deprecated or removed, three of which we better detail in this announcement.

Release theme and logo

Kubernetes 1.26: Electrifying

The theme for Kubernetes v1.26 is Electrifying.

Forensic container checkpointing in Kubernetes

Mon, 05 Dec 2022 00:00:00 +0000

Forensic container checkpointing is based on Checkpoint/Restore In Userspace (CRIU) and allows the creation of stateful copies of a running container without the container knowing that it is being checkpointed. The copy of the container can be analyzed and restored in a sandbox environment multiple times without the original container being aware of it. Forensic container checkpointing was introduced as an alpha feature in Kubernetes v1.25.

How does it work?

With the help of CRIU it is possible to checkpoint and restore containers. CRIU is integrated in runc, crun, CRI-O and containerd and forensic container checkpointing as implemented in Kubernetes uses these existing CRIU integrations.

Finding suspicious syscalls with the seccomp notifier

Fri, 02 Dec 2022 00:00:00 +0000

Debugging software in production is one of the biggest challenges we have to face in our containerized environments. Being able to understand the impact of the available security options, especially when it comes to configuring our deployments, is one of the key aspects to make the default security in Kubernetes stronger. We have all those logging, tracing and metrics data already at hand, but how do we assemble the information they provide into something human readable and actionable?

Boosting Kubernetes container runtime observability with OpenTelemetry

Thu, 01 Dec 2022 00:00:00 +0000

When speaking about observability in the cloud native space, then probably everyone will mention OpenTelemetry (OTEL) at some point in the conversation. That's great, because the community needs standards to rely on for developing all cluster components into the same direction. OpenTelemetry enables us to combine logs, metrics, traces and other contextual information (called baggage) into a single resource. Cluster administrators or software engineers can use this resource to get a viewport about what is going on in the cluster over a defined period of time. But how can Kubernetes itself make use of this technology stack?

registry.k8s.io: faster, cheaper and Generally Available (GA)

Mon, 28 Nov 2022 00:00:00 +0000

Starting with Kubernetes 1.25, our container image registry has changed from k8s.gcr.io to registry.k8s.io. This new registry spreads the load across multiple Cloud Providers & Regions, functioning as a sort of content delivery network (CDN) for Kubernetes container images. This change reduces the project’s reliance on a single entity and provides a faster download experience for a large number of users.

TL;DR: What you need to know about this change

Container images for Kubernetes releases from ~~1.25~~ 1.27 onward are not published to k8s.gcr.io, only to registry.k8s.io.
In the upcoming December patch releases, the new registry domain default will be backported to all branches still in support (1.22, 1.23, 1.24).
If you run in a restricted environment and apply strict domain/IP address access policies limited to k8s.gcr.io, the image pulls will not function after the migration to this new registry. For these users, the recommended method is to mirror the release images to a private registry.

If you’d like to know more about why we made this change, or some potential issues you might run into, keep reading.

Kubernetes Removals, Deprecations, and Major Changes in 1.26

Fri, 18 Nov 2022 00:00:00 +0000

Change is an integral part of the Kubernetes life-cycle: as Kubernetes grows and matures, features may be deprecated, removed, or replaced with improvements for the health of the project. For Kubernetes v1.26 there are several planned: this article identifies and describes some of them, based on the information available at this mid-cycle point in the v1.26 release process, which is still ongoing and can introduce additional changes.

The Kubernetes API Removal and Deprecation process

Live and let live with Kluctl and Server Side Apply

Fri, 04 Nov 2022 00:00:00 +0000

This blog post was inspired by a previous Kubernetes blog post about Advanced Server Side Apply. The author of said blog post listed multiple benefits for applications and controllers when switching to server-side apply (from now on abbreviated with SSA). Especially the chapter about CI/CD systems motivated me to respond and write down my thoughts and experiences.

These thoughts and experiences are the results of me working on Kluctl for the past 2 years. I describe Kluctl as "The missing glue to put together large Kubernetes deployments, composed of multiple smaller parts (Helm/Kustomize/...) in a manageable and unified way."

Server Side Apply Is Great And You Should Be Using It

Thu, 20 Oct 2022 00:00:00 +0000

Server-side apply (SSA) has now been GA for a few releases, and I have found myself in a number of conversations, recommending that people / teams in various situations use it. So I’d like to write down some of those reasons.

Obvious (and not-so-obvious) benefits of SSA

A list of improvements / niceties you get from switching from various things to Server-side apply!

Versus client-side-apply (that is, plain kubectl apply):
- The system gives you conflicts when you accidentally fight with another actor over the value of a field!
- When combined with --dry-run, there’s no chance of accidentally running a client-side dry run instead of a server side dry run.
Versus hand-rolling patches:
- The SSA patch format is extremely natural to write, with no weird syntax. It’s just a regular object, but you can (and should) omit any field you don’t care about.
- The old patch format (“strategic merge patch”) was ad-hoc and still has some bugs; JSON-patch and JSON merge-patch fail to handle some cases that are common in the Kubernetes API, namely lists with items that should be recursively merged based on a “name” or other identifying field.
- There’s also now great go-language library support for building apply calls programmatically!
- You can use SSA to explicitly delete fields you don’t “own” by setting them to null, which makes it a feature-complete replacement for all of the old patch formats.
Versus shelling out to kubectl:
- You can use the apply API call from any language without shelling out to kubectl!
- As stated above, the Go library has dedicated mechanisms to make this easy now.
Versus GET-modify-PUT:
- (This one is more complicated and you can skip it if you've never written a controller!)
- To use GET-modify-PUT correctly, you have to handle and retry a write failure in the case that someone else has modified the object in any way between your GET and PUT. This is an “optimistic concurrency failure” when it happens.
- SSA offloads this task to the server– you only have to retry if there’s a conflict, and the conflicts you can get are all meaningful, like when you’re actually trying to take a field away from another actor in the system.
- To put it another way, if 10 actors do a GET-modify-PUT cycle at the same time, 9 will get an optimistic concurrency failure and have to retry, then 8, etc, for up to 50 total GET-PUT attempts in the worst case (that’s .5N^2 GET and PUT calls for N actors making simultaneous changes). If the actors are using SSA instead, and the changes don’t actually conflict over specific fields, then all the changes can go in in any order. Additionally, SSA changes can often be done without a GET call at all. That’s only N apply requests for N actors, which is a drastic improvement!

How can I use SSA?

Users

Use kubectl apply --server-side! Soon we (SIG API Machinery) hope to make this the default and remove the “client side” apply completely!

Current State: 2019 Third Party Security Audit of Kubernetes

Wed, 05 Oct 2022 00:00:00 +0000

We expect the brand new Third Party Security Audit of Kubernetes will be published later this month (Oct 2022).

In preparation for that, let's look at the state of findings that were made public as part of the last third party security audit of 2019 that was based on Kubernetes v1.13.4.

Motivation

Craig Ingram has graciously attempted over the years to keep track of the status of the findings reported in the last audit in this issue: kubernetes/kubernetes#81146. This blog post will attempt to dive deeper into this, address any gaps in tracking and become a point in time summary of the state of the findings reported from 2019.

Introducing Kueue

Tue, 04 Oct 2022 00:00:00 +0000

Whether on-premises or in the cloud, clusters face real constraints for resource usage, quota, and cost management reasons. Regardless of the autoscalling capabilities, clusters have finite capacity. As a result, users want an easy way to fairly and efficiently share resources.

In this article, we introduce Kueue, an open source job queueing controller designed to manage batch jobs as a single unit. Kueue leaves pod-level orchestration to existing stable components of Kubernetes. Kueue natively supports the Kubernetes Job API and offers hooks for integrating other custom-built APIs for batch jobs.

Kubernetes 1.25: alpha support for running Pods with user namespaces

Mon, 03 Oct 2022 00:00:00 +0000

Kubernetes v1.25 introduces the support for user namespaces.

This is a major improvement for running secure workloads in Kubernetes. Each pod will have access only to a limited subset of the available UIDs and GIDs on the system, thus adding a new security layer to protect from other pods running on the same system.

How does it work?

A process running on Linux can use up to 4294967296 different UIDs and GIDs.

Enforce CRD Immutability with CEL Transition Rules

Thu, 29 Sep 2022 00:00:00 +0000

Immutable fields can be found in a few places in the built-in Kubernetes types. For example, you can't change the .metadata.name of an object. Specific objects have fields where changes to existing objects are constrained; for example, the .spec.selector of a Deployment.

Aside from simple immutability, there are other common design patterns such as lists which are append-only, or a map with mutable values and immutable keys.

Until recently the best way to restrict field mutability for CustomResourceDefinitions has been to create a validating admission webhook: this means a lot of complexity for the common case of making a field immutable.

Kubernetes 1.25: Kubernetes In-Tree to CSI Volume Migration Status Update

Mon, 26 Sep 2022 00:00:00 +0000

The Kubernetes in-tree storage plugin to Container Storage Interface (CSI) migration infrastructure has already been beta since v1.17. CSI migration was introduced as alpha in Kubernetes v1.14. Since then, SIG Storage and other Kubernetes special interest groups are working to ensure feature stability and compatibility in preparation for CSI Migration feature to go GA.

SIG Storage is excited to announce that the core CSI Migration feature is generally available in Kubernetes v1.25 release!

Kubernetes 1.25: CustomResourceDefinition Validation Rules Graduate to Beta

Fri, 23 Sep 2022 00:00:00 +0000

In Kubernetes 1.25, Validation rules for CustomResourceDefinitions (CRDs) have graduated to Beta!

Validation rules make it possible to declare how custom resources are validated using the Common Expression Language (CEL). For example:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
 ...
 openAPIV3Schema:
 type: object
 properties:
 spec:
 type: object
 x-kubernetes-validations:
 - rule: "self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas"
 message: "replicas should be in the range minReplicas..maxReplicas."
 properties:
 replicas:
 type: integer
 ...

Validation rules support a wide range of use cases. To get a sense of some of the capabilities, let's look at a few examples:

Kubernetes 1.25: Use Secrets for Node-Driven Expansion of CSI Volumes

Wed, 21 Sep 2022 00:00:00 +0000

Kubernetes v1.25, released earlier this month, introduced a new feature that lets your cluster expand storage volumes, even when access to those volumes requires a secret (for example: a credential for accessing a SAN fabric) to perform node expand operation. This new behavior is in alpha and you must enable a feature gate (CSINodeExpandSecret) to make use of it. You must also be using CSI storage; this change isn't relevant to storage drivers that are built in to Kubernetes.

Kubernetes 1.25: Local Storage Capacity Isolation Reaches GA

Mon, 19 Sep 2022 00:00:00 +0000

Local ephemeral storage capacity isolation was introduced as a alpha feature in Kubernetes 1.7 and it went beta in 1.9. With Kubernetes 1.25 we are excited to announce general availability(GA) of this feature.

Pods use ephemeral local storage for scratch space, caching, and logs. The lifetime of local ephemeral storage does not extend beyond the life of the individual pod. It is exposed to pods using the container’s writable layer, logs directory, and EmptyDir volumes. Before this feature was introduced, there were issues related to the lack of local storage accounting and isolation, such as Pods not knowing how much local storage is available and being unable to request guaranteed local storage. Local storage is a best-effort resource and pods can be evicted due to other pods filling the local storage.

Kubernetes 1.25: Two Features for Apps Rollouts Graduate to Stable

Thu, 15 Sep 2022 00:00:00 +0000

This blog describes the two features namely minReadySeconds for StatefulSets and maxSurge for DaemonSets that SIG Apps is happy to graduate to stable in Kubernetes 1.25.

Specifying minReadySeconds slows down a rollout of a StatefulSet, when using a RollingUpdate value in .spec.updateStrategy field, by waiting for each pod for a desired time. This time can be used for initializing the pod (e.g. warming up the cache) or as a delay before acknowledging the pod.

Kubernetes 1.25: PodHasNetwork Condition for Pods

Wed, 14 Sep 2022 00:00:00 +0000

Kubernetes 1.25 introduces Alpha support for a new kubelet-managed pod condition in the status field of a pod: PodHasNetwork. The kubelet, for a worker node, will use the PodHasNetwork condition to accurately surface the initialization state of a pod from the perspective of pod sandbox creation and network configuration by a container runtime (typically in coordination with CNI plugins). The kubelet starts to pull container images and start individual containers (including init containers) after the status of the PodHasNetwork condition is set to "True". Metrics collection services that report latency of pod initialization from a cluster infrastructural perspective (i.e. agnostic of per container characteristics like image size or payload) can utilize the PodHasNetwork condition to accurately generate Service Level Indicators (SLIs). Certain operators or controllers that manage underlying pods may utilize the PodHasNetwork condition to optimize the set of actions performed when pods repeatedly fail to come up.

Announcing the Auto-refreshing Official Kubernetes CVE Feed

Mon, 12 Sep 2022 00:00:00 +0000

A long-standing request from the Kubernetes community has been to have a programmatic way for end users to keep track of Kubernetes security issues (also called "CVEs", after the database that tracks public security issues across different products and vendors). Accompanying the release of Kubernetes v1.25, we are excited to announce availability of such a feed as an alpha feature. This blog will cover the background and scope of this new service.

Kubernetes 1.25: KMS V2 Improvements

Fri, 09 Sep 2022 00:00:00 +0000

With Kubernetes v1.25, SIG Auth is introducing a new v2alpha1 version of the Key Management Service (KMS) API. There are a lot of improvements in the works, and we're excited to be able to start down the path of a new and improved KMS!

What is KMS?

One of the first things to consider when securing a Kubernetes cluster is encrypting persisted API data at rest. KMS provides an interface for a provider to utilize a key stored in an external key service to perform this encryption.

Kubernetes’s IPTables Chains Are Not API

Wed, 07 Sep 2022 00:00:00 +0000

Some Kubernetes components (such as kubelet and kube-proxy) create iptables chains and rules as part of their operation. These chains were never intended to be part of any Kubernetes API/ABI guarantees, but some external components nonetheless make use of some of them (in particular, using KUBE-MARK-MASQ to mark packets as needing to be masqueraded).

As a part of the v1.25 release, SIG Network made this declaration explicit: that (with one exception), the iptables chains that Kubernetes creates are intended only for Kubernetes’s own internal use, and third-party components should not assume that Kubernetes will create any specific iptables chains, or that those chains will contain any specific rules if they do exist.

Introducing COSI: Object Storage Management using Kubernetes APIs

Fri, 02 Sep 2022 00:00:00 +0000

This article introduces the Container Object Storage Interface (COSI), a standard for provisioning and consuming object storage in Kubernetes. It is an alpha feature in Kubernetes v1.25.

File and block storage are treated as first class citizens in the Kubernetes ecosystem via Container Storage Interface (CSI). Workloads using CSI volumes enjoy the benefits of portability across vendors and across Kubernetes clusters without the need to change application manifests. An equivalent standard does not exist for Object storage.

Kubernetes 1.25: cgroup v2 graduates to GA

Wed, 31 Aug 2022 00:00:00 +0000

Kubernetes 1.25 brings cgroup v2 to GA (general availability), letting the kubelet use the latest container resource management capabilities.

What are cgroups?

Effective resource management is a critical aspect of Kubernetes. This involves managing the finite resources in your nodes, such as CPU, memory, and storage.

cgroups are a Linux kernel capability that establish resource management functionality like limiting CPU usage or setting memory limits for running processes.

When you use the resource management capabilities in Kubernetes, such as configuring requests and limits for Pods and containers, Kubernetes uses cgroups to enforce your resource requests and limits.

Kubernetes 1.25: CSI Inline Volumes have graduated to GA

Mon, 29 Aug 2022 00:00:00 +0000

CSI Inline Volumes were introduced as an alpha feature in Kubernetes 1.15 and have been beta since 1.16. We are happy to announce that this feature has graduated to General Availability (GA) status in Kubernetes 1.25.

CSI Inline Volumes are similar to other ephemeral volume types, such as configMap, downwardAPI and secret. The important difference is that the storage is provided by a CSI driver, which allows the use of ephemeral storage provided by third-party vendors. The volume is defined as part of the pod spec and follows the lifecycle of the pod, meaning the volume is created once the pod is scheduled and destroyed when the pod is destroyed.

Kubernetes v1.25: Pod Security Admission Controller in Stable

Thu, 25 Aug 2022 00:00:00 +0000

The release of Kubernetes v1.25 marks a major milestone for Kubernetes out-of-the-box pod security controls: Pod Security admission (PSA) graduated to stable, and Pod Security Policy (PSP) has been removed. PSP was deprecated in Kubernetes v1.21, and no longer functions in Kubernetes v1.25 and later.

The Pod Security admission controller replaces PodSecurityPolicy, making it easier to enforce predefined Pod Security Standards by simply adding a label to a namespace. The Pod Security Standards are maintained by the K8s community, which means you automatically get updated security policies whenever new security-impacting Kubernetes features are introduced.

PodSecurityPolicy: The Historical Context

Tue, 23 Aug 2022 15:00:00 -0800

The PodSecurityPolicy (PSP) admission controller has been removed, as of Kubernetes v1.25. Its deprecation was announced and detailed in the blog post PodSecurityPolicy Deprecation: Past, Present, and Future, published for the Kubernetes v1.21 release.

This article aims to provide historical context on the birth and evolution of PSP, explain why the feature never made it to stable, and show why it was removed and replaced by Pod Security admission control.

PodSecurityPolicy, like other specialized admission control plugins, provided fine-grained permissions on specific fields concerning the pod security settings as a built-in policy API. It acknowledged that cluster administrators and cluster users are usually not the same people, and that creating workloads in the form of a Pod or any resource that will create a Pod should not equal being "root on the cluster". It could also encourage best practices by configuring more secure defaults through mutation and decoupling low-level Linux security decisions from the deployment process.

Kubernetes v1.25: Combiner

Tue, 23 Aug 2022 00:00:00 +0000

Announcing the release of Kubernetes v1.25!

This release includes a total of 40 enhancements. Fifteen of those enhancements are entering Alpha, ten are graduating to Beta, and thirteen are graduating to Stable. We also have two features being deprecated or removed.

Release theme and logo

Kubernetes 1.25: Combiner

The theme for Kubernetes v1.25 is Combiner.

The Kubernetes project itself is made up of many, many individual components that, when combined, take the form of the project you see today. It is also built and maintained by many individuals, all of them with different skills, experiences, histories, and interests, who join forces not just as the release team but as the many SIGs that support the project and the community year-round.

Spotlight on SIG Storage

Mon, 22 Aug 2022 00:00:00 +0000

Since the very beginning of Kubernetes, the topic of persistent data and how to address the requirement of stateful applications has been an important topic. Support for stateless deployments was natural, present from the start, and garnered attention, becoming very well-known. Work on better support for stateful applications was also present from early on, with each release increasing the scope of what could be run on Kubernetes.

Message queues, databases, clustered filesystems: these are some examples of the solutions that have different storage requirements and that are, today, increasingly deployed in Kubernetes. Dealing with ephemeral and persistent storage, local or remote, file or block, from many different vendors, while considering how to provide the needed resiliency and data consistency that users expect, all of this is under SIG Storage's umbrella.

Meet Our Contributors - APAC (China region)

Mon, 15 Aug 2022 00:00:00 +0000

Authors & Interviewers: Avinesh Tripathi, Debabrata Panigrahi, Jayesh Srivastava, Priyanka Saggu, Purneswar Prasad, Vedant Kakde

Hello, everyone 👋

Welcome back to the third edition of the "Meet Our Contributors" blog post series for APAC.

This post features four outstanding contributors from China, who have played diverse leadership and community roles in the upstream Kubernetes project.

So, without further ado, let's get straight to the article.

Andy Zhang

Andy Zhang currently works for Microsoft China at the Shanghai site. His main focus is on Kubernetes storage drivers. Andy started contributing to Kubernetes about 5 years ago.

Enhancing Kubernetes one KEP at a Time

Thu, 11 Aug 2022 00:00:00 +0000

Did you know that Kubernetes v1.24 has 46 enhancements? That's a lot of new functionality packed into a 4-month release cycle. The Kubernetes release team coordinates the logistics of the release, from remediating test flakes to publishing updated docs. It's a ton of work, but they always deliver.

The release team comprises around 30 people across six subteams - Bug Triage, CI Signal, Enhancements, Release Notes, Communications, and Docs. Each of these subteams manages a component of the release. This post will focus on the role of the enhancements subteam and how you can get involved.

Kubernetes Removals and Major Changes In 1.25

Thu, 04 Aug 2022 00:00:00 +0000

As Kubernetes grows and matures, features may be deprecated, removed, or replaced with improvements for the health of the project. Kubernetes v1.25 includes several major changes and one major removal.

The Kubernetes API Removal and Deprecation process

Spotlight on SIG Docs

Tue, 02 Aug 2022 00:00:00 +0000

Introduction

The official documentation is the go-to source for any open source project. For Kubernetes, it's an ever-evolving Special Interest Group (SIG) with people constantly putting in their efforts to make details about the project easier to consume for new contributors and users. SIG Docs publishes the official documentation on kubernetes.io which includes, but is not limited to, documentation of the core APIs, core architectural details, and CLI tools shipped with the Kubernetes release.

Kubernetes Gateway API Graduates to Beta

Wed, 13 Jul 2022 00:00:00 +0000

We are excited to announce the v0.5.0 release of Gateway API. For the first time, several of our most important Gateway API resources are graduating to beta. Additionally, we are starting a new initiative to explore how Gateway API can be used for mesh and introducing new experimental concepts such as URL rewrites. We'll cover all of this and more below.

What is Gateway API?

Gateway API is a collection of resources centered around Gateway resources (which represent the underlying network gateways / proxy servers) to enable robust Kubernetes service networking through expressive, extensible and role-oriented interfaces that are implemented by many vendors and have broad industry support.

Annual Report Summary 2021

Wed, 01 Jun 2022 00:00:00 +0000

Last year, we published our first Annual Report Summary for 2020 and it's already time for our second edition!

2021 Annual Report Summary

This summary reflects the work that has been done in 2021 and the initiatives on deck for the rest of 2022. Please forward to organizations and indidviduals participating in upstream activities, planning cloud native strategies, and/or those looking to help out. To find a specific community group's complete report, go to the kubernetes/community repo under the groups folder. Example: sig-api-machinery/annual-report-2021.md

Kubernetes 1.24: Maximum Unavailable Replicas for StatefulSet

Fri, 27 May 2022 00:00:00 +0000

Kubernetes StatefulSets, since their introduction in 1.5 and becoming stable in 1.9, have been widely used to run stateful applications. They provide stable pod identity, persistent per pod storage and ordered graceful deployment, scaling and rolling updates. You can think of StatefulSet as the atomic building block for running complex stateful applications. As the use of Kubernetes has grown, so has the number of scenarios requiring StatefulSets. Many of these scenarios, require faster rolling updates than the currently supported one-pod-at-a-time updates, in the case where you're using the OrderedReady Pod management policy for a StatefulSet.

Contextual Logging in Kubernetes 1.24

Wed, 25 May 2022 00:00:00 +0000

The Structured Logging Working Group has added new capabilities to the logging infrastructure in Kubernetes 1.24. This blog post explains how developers can take advantage of those to make log output more useful and how they can get involved with improving Kubernetes.

Structured logging

The goal of structured logging is to replace C-style formatting and the resulting opaque log strings with log entries that have a well-defined syntax for storing message and parameters separately, for example as a JSON struct.

Kubernetes 1.24: Avoid Collisions Assigning IP Addresses to Services

Mon, 23 May 2022 00:00:00 +0000

How Service ClusterIPs are allocated?

A Service ClusterIP can be assigned:

dynamically: the cluster's control plane automatically picks a free IP address from within the configured IP range for type: ClusterIP Services.
statically: you specify an IP address of your choice, from within the configured IP range for Services.

Across your whole cluster, every Service ClusterIP must be unique. Trying to create a Service with a specific ClusterIP that has already been allocated will return an error.

Kubernetes 1.24: Introducing Non-Graceful Node Shutdown Alpha

Fri, 20 May 2022 00:00:00 +0000

Kubernetes v1.24 introduces alpha support for Non-Graceful Node Shutdown. This feature allows stateful workloads to failover to a different node after the original node is shutdown or in a non-recoverable state such as hardware failure or broken OS.

How is this different from Graceful Node Shutdown

You might have heard about the Graceful Node Shutdown capability of Kubernetes, and are wondering how the Non-Graceful Node Shutdown feature is different from that. Graceful Node Shutdown allows Kubernetes to detect when a node is shutting down cleanly, and handles that situation appropriately. A Node Shutdown can be "graceful" only if the node shutdown action can be detected by the kubelet ahead of the actual shutdown. However, there are cases where a node shutdown action may not be detected by the kubelet. This could happen either because the shutdown command does not trigger the systemd inhibitor locks mechanism that kubelet relies upon, or because of a configuration error (the ShutdownGracePeriod and ShutdownGracePeriodCriticalPods are not configured properly).

Kubernetes 1.24: Prevent unauthorised volume mode conversion

Wed, 18 May 2022 00:00:00 +0000

Kubernetes v1.24 introduces a new alpha-level feature that prevents unauthorised users from modifying the volume mode of a PersistentVolumeClaim created from an existing VolumeSnapshot in the Kubernetes cluster.

The problem

The Volume Mode determines whether a volume is formatted into a filesystem or presented as a raw block device.

Users can leverage the VolumeSnapshot feature, which has been stable since Kubernetes v1.20, to create a PersistentVolumeClaim (shortened as PVC) from an existing VolumeSnapshot in the Kubernetes cluster. The PVC spec includes a dataSource field, which can point to an existing VolumeSnapshot instance. Visit Create a PersistentVolumeClaim from a Volume Snapshot for more details.

Kubernetes 1.24: Volume Populators Graduate to Beta

Mon, 16 May 2022 00:00:00 +0000

The volume populators feature is now two releases old and entering beta! The AnyVolumeDataSource feature gate defaults to enabled in Kubernetes v1.24, which means that users can specify any custom resource as the data source of a PVC.

An earlier blog article detailed how the volume populators feature works. In short, a cluster administrator can install a CRD and associated populator controller in the cluster, and any user who can create instances of the CR can create pre-populated volumes by taking advantage of the populator.

Kubernetes 1.24: gRPC container probes in beta

Fri, 13 May 2022 00:00:00 +0000

_Update: Since this article was posted, the feature was graduated to GA in v1.27 and doesn't require any feature gates to be enabled.

With Kubernetes 1.24 the gRPC probes functionality entered beta and is available by default. Now you can configure startup, liveness, and readiness probes for your gRPC app without exposing any HTTP endpoint, nor do you need an executable. Kubernetes can natively connect to your workload via gRPC and query its status.

Kubernetes 1.24: Storage Capacity Tracking Now Generally Available

Fri, 06 May 2022 00:00:00 +0000

The v1.24 release of Kubernetes brings storage capacity tracking as a generally available feature.

Problems we have solved

As explained in more detail in the previous blog post about this feature, storage capacity tracking allows a CSI driver to publish information about remaining capacity. The kube-scheduler then uses that information to pick suitable nodes for a Pod when that Pod has volumes that still need to be provisioned.

Without this information, a Pod may get stuck without ever being scheduled onto a suitable node because kube-scheduler has to choose blindly and always ends up picking a node for which the volume cannot be provisioned because the underlying storage system managed by the CSI driver does not have sufficient capacity left.

Kubernetes 1.24: Volume Expansion Now A Stable Feature

Thu, 05 May 2022 00:00:00 +0000

Volume expansion was introduced as a alpha feature in Kubernetes 1.8 and it went beta in 1.11 and with Kubernetes 1.24 we are excited to announce general availability(GA) of volume expansion.

This feature allows Kubernetes users to simply edit their PersistentVolumeClaim objects and specify new size in PVC Spec and Kubernetes will automatically expand the volume using storage backend and also expand the underlying file system in-use by the Pod without requiring any downtime at all if possible.

Dockershim: The Historical Context

Tue, 03 May 2022 00:00:00 +0000

Dockershim has been removed as of Kubernetes v1.24, and this is a positive move for the project. However, context is important for fully understanding something, be it socially or in software development, and this deserves a more in-depth review. Alongside the dockershim removal in Kubernetes v1.24, we’ve seen some confusion (sometimes at a panic level) and dissatisfaction with this decision in the community, largely due to a lack of context around this removal. The decision to deprecate and eventually remove dockershim from Kubernetes was not made quickly or lightly. Still, it’s been in the works for so long that many of today’s users are newer than that decision, and certainly newer than the choices that led to the dockershim being necessary in the first place.

Kubernetes 1.24: Stargazer

Tue, 03 May 2022 00:00:00 +0000

We are excited to announce the release of Kubernetes 1.24, the first release of 2022!

This release consists of 46 enhancements: fourteen enhancements have graduated to stable, fifteen enhancements are moving to beta, and thirteen enhancements are entering alpha. Also, two features have been deprecated, and two features have been removed.

Major Themes

Dockershim Removed from kubelet

After its deprecation in v1.20, the dockershim component has been removed from the kubelet in Kubernetes v1.24. From v1.24 onwards, you will need to either use one of the other supported runtimes (such as containerd or CRI-O) or use cri-dockerd if you are relying on Docker Engine as your container runtime. For more information about ensuring your cluster is ready for this removal, please see this guide.

Increasing the security bar in Ingress-NGINX v1.2.0

Thu, 28 Apr 2022 00:00:00 +0000

The Ingress may be one of the most targeted components of Kubernetes. An Ingress typically defines an HTTP reverse proxy, exposed to the Internet, containing multiple websites, and with some privileged access to Kubernetes API (such as to read Secrets relating to TLS certificates and their private keys).

While it is a risky component in your architecture, it is still the most popular way to properly expose your services.

Ingress-NGINX has been part of security assessments that figured out we have a big problem: we don't do all proper sanitization before turning the configuration into an nginx.conf file, which may lead to information disclosure risks.

Kubernetes Removals and Deprecations In 1.24

Thu, 07 Apr 2022 00:00:00 +0000

As Kubernetes evolves, features and APIs are regularly revisited and removed. New features may offer an alternative or improved approach to solving existing problems, motivating the team to remove the old approach.

We want to make sure you are aware of the changes coming in the Kubernetes 1.24 release. The release will deprecate several (beta) APIs in favor of stable versions of the same APIs. The major change coming in the Kubernetes 1.24 release is the removal of Dockershim. This is discussed below and will be explored in more depth at release time. For an early look at the changes coming in Kubernetes 1.24, take a look at the in-progress CHANGELOG.

Is Your Cluster Ready for v1.24?

Thu, 31 Mar 2022 00:00:00 +0000

Way back in December of 2020, Kubernetes announced the deprecation of Dockershim. In Kubernetes, dockershim is a software shim that allows you to use the entire Docker engine as your container runtime within Kubernetes. In the upcoming v1.24 release, we are removing Dockershim - the delay between deprecation and removal in line with the project’s policy of supporting features for at least one year after deprecation. If you are a cluster operator, this guide includes the practical realities of what you need to know going into this release. Also, what do you need to do to ensure your cluster doesn’t fall over!

Meet Our Contributors - APAC (Aus-NZ region)

Wed, 16 Mar 2022 00:00:00 +0000

Authors & Interviewers: Anubhav Vardhan, Atharva Shinde, Avinesh Tripathi, Brad McCoy, Debabrata Panigrahi, Jayesh Srivastava, Kunal Verma, Pranshu Srivastava, Priyanka Saggu, Purneswar Prasad, Vedant Kakde

Good day, everyone 👋

Welcome back to the second episode of the "Meet Our Contributors" blog post series for APAC.

This post will feature four outstanding contributors from the Australia and New Zealand regions, who have played diverse leadership and community roles in the Upstream Kubernetes project.

So, without further ado, let's get straight to the blog.

Updated: Dockershim Removal FAQ

Thu, 17 Feb 2022 00:00:00 +0000

This supersedes the original Dockershim Deprecation FAQ article, published in late 2020. The article includes updates from the v1.24 release of Kubernetes.

This document goes over some frequently asked questions regarding the removal of dockershim from Kubernetes. The removal was originally announced as a part of the Kubernetes v1.20 release. The Kubernetes v1.24 release actually removed the dockershim from Kubernetes.

For more on what that means, check out the blog post Don't Panic: Kubernetes and Docker.

SIG Node CI Subproject Celebrates Two Years of Test Improvements

Wed, 16 Feb 2022 00:00:00 +0000

Ensuring the reliability of SIG Node upstream code is a continuous effort that takes a lot of behind-the-scenes effort from many contributors. There are frequent releases of Kubernetes, base operating systems, container runtimes, and test infrastructure that result in a complex matrix that requires attention and steady investment to "keep the lights on." In May 2020, the Kubernetes node special interest group ("SIG Node") organized a new subproject for continuous integration (CI) for node-related code and tests. Since its inauguration, the SIG Node CI subproject has run a weekly meeting, and even the full hour is often not enough to complete triage of all bugs, test-related PRs and issues, and discuss all related ongoing work within the subgroup.

Spotlight on SIG Multicluster

Mon, 07 Feb 2022 00:00:00 +0000

Introduction

SIG Multicluster is the SIG focused on how Kubernetes concepts are expanded and used beyond the cluster boundary. Historically, Kubernetes resources only interacted within that boundary - KRU or Kubernetes Resource Universe (not an actual Kubernetes concept). Kubernetes clusters, even now, don't really know anything about themselves or, about other clusters. Absence of cluster identifiers is a case in point. With the growing adoption of multicloud and multicluster deployments, the work SIG Multicluster doing is gaining a lot of attention. In this blog, Jeremy Olmsted-Thompson, Google and Chris Short, AWS discuss the interesting problems SIG Multicluster is solving and how you can get involved. Their initials JOT and CS will be used for brevity.

Securing Admission Controllers

Wed, 19 Jan 2022 00:00:00 +0000

Admission control is a key part of Kubernetes security, alongside authentication and authorization. Webhook admission controllers are extensively used to help improve the security of Kubernetes clusters in a variety of ways including restricting the privileges of workloads and ensuring that images deployed to the cluster meet organization’s security requirements.

However, as with any additional component added to a cluster, security risks can present themselves. A security risk example is if the deployment and management of the admission controller are not handled correctly. To help admission controller users and designers manage these risks appropriately, the security documentation subgroup of SIG Security has spent some time developing a threat model for admission controllers. This threat model looks at likely risks which may arise from the incorrect use of admission controllers, which could allow security policies to be bypassed, or even allow an attacker to get unauthorised access to the cluster.

Meet Our Contributors - APAC (India region)

Mon, 10 Jan 2022 00:00:00 +0000

Authors & Interviewers: Anubhav Vardhan, Atharva Shinde, Avinesh Tripathi, Debabrata Panigrahi, Kunal Verma, Pranshu Srivastava, Pritish Samal, Purneswar Prasad, Vedant Kakde

Editor: Priyanka Saggu

Good day, everyone 👋

Welcome to the first episode of the APAC edition of the "Meet Our Contributors" blog post series.

In this post, we'll introduce you to five amazing folks from the India region who have been actively contributing to the upstream Kubernetes projects in a variety of ways, as well as being the leaders or maintainers of numerous community initiatives.

Kubernetes is Moving on From Dockershim: Commitments and Next Steps

Fri, 07 Jan 2022 00:00:00 +0000

Kubernetes is removing dockershim in the upcoming v1.24 release. We're excited to reaffirm our community values by supporting open source container runtimes, enabling a smaller kubelet, and increasing engineering velocity for teams using Kubernetes. If you use Docker Engine as a container runtime for your Kubernetes cluster, get ready to migrate in 1.24! To check if you're affected, refer to Check whether dockershim removal affects you.

Why we’re moving away from dockershim

Docker was the first container runtime used by Kubernetes. This is one of the reasons why Docker is so familiar to many Kubernetes users and enthusiasts. Docker support was hardcoded into Kubernetes – a component the project refers to as dockershim. As containerization became an industry standard, the Kubernetes project added support for additional runtimes. This culminated in the implementation of the container runtime interface (CRI), letting system components (like the kubelet) talk to container runtimes in a standardized way. As a result, dockershim became an anomaly in the Kubernetes project. Dependencies on Docker and dockershim have crept into various tools and projects in the CNCF ecosystem ecosystem, resulting in fragile code.

Kubernetes-in-Kubernetes and the WEDOS PXE bootable server farm

Wed, 22 Dec 2021 00:00:00 +0000

When you own two data centers, thousands of physical servers, virtual machines and hosting for hundreds of thousands sites, Kubernetes can actually simplify the management of all these things. As practice has shown, by using Kubernetes, you can declaratively describe and manage not only applications, but also the infrastructure itself. I work for the largest Czech hosting provider WEDOS Internet a.s and today I'll show you two of my projects — Kubernetes-in-Kubernetes and Kubefarm.

Using Admission Controllers to Detect Container Drift at Runtime

Tue, 21 Dec 2021 00:00:00 +0000

Illustration by Munire Aireti

At Box, we use Kubernetes (K8s) to manage hundreds of micro-services that enable Box to stream data at a petabyte scale. When it comes to the deployment process, we run kube-applier as part of the GitOps workflows with declarative configuration and automated deployment. Developers declare their K8s apps manifest into a Git repository that requires code reviews and automatic checks to pass, before any changes can get merged and applied inside our K8s clusters. With kubectl exec and other similar commands, however, developers are able to directly interact with running containers and alter them from their deployed state. This interaction could then subvert the change control and code review processes that are enforced in our CI/CD pipelines. Further, it allows such impacted containers to continue receiving traffic long-term in production.

What's new in Security Profiles Operator v0.4.0

Fri, 17 Dec 2021 00:00:00 +0000

The Security Profiles Operator (SPO) is an out-of-tree Kubernetes enhancement to make the management of seccomp, SELinux and AppArmor profiles easier and more convenient. We're happy to announce that we recently released v0.4.0 of the operator, which contains a ton of new features, fixes and usability improvements.

What's new

It has been a while since the last v0.3.0 release of the operator. We added new features, fine-tuned existing ones and reworked our documentation in 290 commits over the past half year.

Kubernetes 1.23: StatefulSet PVC Auto-Deletion (alpha)

Thu, 16 Dec 2021 00:00:00 +0000

Kubernetes v1.23 introduced a new, alpha-level policy for StatefulSets that controls the lifetime of PersistentVolumeClaims (PVCs) generated from the StatefulSet spec template for cases when they should be deleted automatically when the StatefulSet is deleted or pods in the StatefulSet are scaled down.

What problem does this solve?

A StatefulSet spec can include Pod and PVC templates. When a replica is first created, the Kubernetes control plane creates a PVC for that replica if one does not already exist. The behavior before Kubernetes v1.23 was that the control plane never cleaned up the PVCs created for StatefulSets - this was left up to the cluster administrator, or to some add-on automation that you’d have to find, check suitability, and deploy. The common pattern for managing PVCs, either manually or through tools such as Helm, is that the PVCs are tracked by the tool that manages them, with explicit lifecycle. Workflows that use StatefulSets must determine on their own what PVCs are created by a StatefulSet and what their lifecycle should be.

Kubernetes 1.23: Prevent PersistentVolume leaks when deleting out of order

Wed, 15 Dec 2021 10:00:00 -0800

PersistentVolume (or PVs for short) are associated with Reclaim Policy. The Reclaim Policy is used to determine the actions that need to be taken by the storage backend on deletion of the PV. Where the reclaim policy is Delete, the expectation is that the storage backend releases the storage resource that was allocated for the PV. In essence, the reclaim policy needs to honored on PV deletion.

With the recent Kubernetes v1.23 release, an alpha feature lets you configure your cluster to behave that way and honor the configured reclaim policy.

Kubernetes 1.23: Kubernetes In-Tree to CSI Volume Migration Status Update

Fri, 10 Dec 2021 00:00:00 +0000

The Kubernetes in-tree storage plugin to Container Storage Interface (CSI) migration infrastructure has already been beta since v1.17. CSI migration was introduced as alpha in Kubernetes v1.14.

Since then, SIG Storage and other Kubernetes special interest groups are working to ensure feature stability and compatibility in preparation for GA. This article is intended to give a status update to the feature as well as changes between Kubernetes 1.17 and 1.23. In addition, I will also cover the future roadmap for the CSI migration feature GA for each storage plugin.

Kubernetes 1.23: Pod Security Graduates to Beta

Thu, 09 Dec 2021 00:00:00 +0000

With the release of Kubernetes v1.23, Pod Security admission has now entered beta. Pod Security is a built-in admission controller that evaluates pod specifications against a predefined set of Pod Security Standards and determines whether to admit or deny the pod from running.

Pod Security is the successor to PodSecurityPolicy which was deprecated in the v1.21 release, and will be removed in Kubernetes v1.25. In this article, we cover the key concepts of Pod Security along with how to use it. We hope that cluster administrators and developers alike will use this new mechanism to enforce secure defaults for their workloads.

Kubernetes 1.23: Dual-stack IPv4/IPv6 Networking Reaches GA

Wed, 08 Dec 2021 00:00:00 +0000

"When will Kubernetes have IPv6?" This question has been asked with increasing frequency ever since alpha support for IPv6 was first added in k8s v1.9. While Kubernetes has supported IPv6-only clusters since v1.18, migration from IPv4 to IPv6 was not yet possible at that point. At long last, dual-stack IPv4/IPv6 networking has reached general availability (GA) in Kubernetes v1.23.

What does dual-stack networking mean for you? Let’s take a look…

Service API updates

Services were single-stack before 1.20, so using both IP families meant creating one Service per IP family. The user experience was simplified in 1.20, when Services were re-implemented to allow both IP families, meaning a single Service can handle both IPv4 and IPv6 workloads. Dual-stack load balancing is possible between services running any combination of IPv4 and IPv6.

Kubernetes 1.23: The Next Frontier

Tue, 07 Dec 2021 00:00:00 +0000

We’re pleased to announce the release of Kubernetes 1.23, the last release of 2021!

This release consists of 47 enhancements: 11 enhancements have graduated to stable, 17 enhancements are moving to beta, and 19 enhancements are entering alpha. Also, 1 feature has been deprecated.

Major Themes

Deprecation of FlexVolume

FlexVolume is deprecated. The out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See this doc for more information. Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI. Users of FlexVolume should move their workloads to the CSI driver.

Contribution, containers and cricket: the Kubernetes 1.22 release interview

Wed, 01 Dec 2021 00:00:00 +0000

The Kubernetes release train rolls on, and we look ahead to the release of 1.23 next week. As is our tradition, I'm pleased to bring you a look back at the process that brought us the previous version.

The release team for 1.22 was led by Savitha Raghunathan, who was, at the time, a Senior Platform Engineer at MathWorks. I spoke to Savitha on the Kubernetes Podcast from Google, the weekly* show covering the Kubernetes and Cloud Native ecosystem.

Quality-of-Service for Memory Resources

Fri, 26 Nov 2021 00:00:00 +0000

Kubernetes v1.22, released in August 2021, introduced a new alpha feature that improves how Linux nodes implement memory resource requests and limits.

In prior releases, Kubernetes did not support memory quality guarantees. For example, if you set container resources as follows:

apiVersion: v1
kind: Pod
metadata:
 name: example
spec:
 containers:
 - name: nginx
 resources:
 requests:
 memory: "64Mi"
 cpu: "250m"
 limits:
 memory: "64Mi"
 cpu: "500m"

spec.containers[].resources.requests(e.g. cpu, memory) is designed for scheduling. When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled Containers is less than the capacity of the node.

Dockershim removal is coming. Are you ready?

Fri, 12 Nov 2021 00:00:00 +0000

Reviewers: Davanum Srinivas, Elana Hashman, Noah Kantrowitz, Rey Lejano.

Poll closed

This poll closed on January 7, 2022.

Last year we announced that Kubernetes' dockershim component (which provides a built-in integration for Docker Engine) is deprecated.

Update: There's a Dockershim Deprecation FAQ with more information, and you can also discuss the deprecation via a dedicated GitHub issue.

Our current plan is to remove dockershim from the Kubernetes codebase soon. We are looking for feedback from you whether you are ready for dockershim removal and to ensure that you are ready when the time comes.

Non-root Containers And Devices

Tue, 09 Nov 2021 00:00:00 +0000

The user/group ID related security settings in Pod's securityContext trigger a problem when users want to deploy containers that use accelerator devices (via Kubernetes Device Plugins) on Linux. In this blog post I talk about the problem and describe the work done so far to address it. It's not meant to be a long story about getting the k/k issue fixed.

Instead, this post aims to raise awareness of the issue and to highlight important device use-cases too. This is needed as Kubernetes works on new related features such as support for user namespaces.

Announcing the 2021 Steering Committee Election Results

Mon, 08 Nov 2021 00:00:00 +0000

The 2021 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2021. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

Use KPNG to Write Specialized kube-proxiers

Mon, 18 Oct 2021 00:00:00 +0000

The post will show you how to create a specialized service kube-proxy style network proxier using Kubernetes Proxy NG kpng without interfering with the existing kube-proxy. The kpng project aims at renewing the the default Kubernetes Service implementation, the "kube-proxy". An important feature of kpng is that it can be used as a library to create proxiers outside K8s. While this is useful for CNI-plugins that replaces the kube-proxy it also opens the possibility for anyone to create a proxier for a special purpose.

Introducing ClusterClass and Managed Topologies in Cluster API

Fri, 08 Oct 2021 00:00:00 +0000

The Cluster API community is happy to announce the implementation of ClusterClass and Managed Topologies, a new feature that will greatly simplify how you can provision, upgrade, and operate multiple Kubernetes clusters in a declarative way.

A little bit of context…

Before getting into the details, let's take a step back and look at the history of Cluster API.

The Cluster API project started three years ago, and the first releases focused on extensibility and implementing a declarative API that allows a seamless experience across infrastructure providers. This was a success with many cloud providers: AWS, Azure, Digital Ocean, GCP, Metal3, vSphere and still counting.

A Closer Look at NSA/CISA Kubernetes Hardening Guidance

Tue, 05 Oct 2021 00:00:00 +0000

Disclaimer

The open source tools listed in this article are to serve as examples only and are in no way a direct recommendation from the Kubernetes community or authors.

Background

USA's National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA) released Kubernetes Hardening Guidance on August 3rd, 2021. The guidance details threats to Kubernetes environments and provides secure configuration guidance to minimize risk.

The following sections of this blog correlate to the sections in the NSA/CISA guidance. Any missing sections are skipped because of limited opportunities to add anything new to the existing content.

How to Handle Data Duplication in Data-Heavy Kubernetes Environments

Wed, 29 Sep 2021 00:00:00 +0000

Why Duplicate Data?

It’s convenient to create a copy of your application with a copy of its state for each team. For example, you might want a separate database copy to test some significant schema changes or develop other disruptive operations like bulk insert/delete/update...

Duplicating data takes a lot of time. That’s because you need first to download all the data from a source block storage provider to compute and then send it back to a storage provider again. There’s a lot of network traffic and CPU/RAM used in this process. Hardware acceleration by offloading certain expensive operations to dedicated hardware is always a huge performance boost. It reduces the time required to complete an operation by orders of magnitude.

Spotlight on SIG Node

Mon, 27 Sep 2021 00:00:00 +0000

Introduction

In Kubernetes, a Node is a representation of a single machine in your cluster. SIG Node owns that very important Node component and supports various subprojects such as Kubelet, Container Runtime Interface (CRI) and more to support how the pods and host resources interact. In this blog, we have summarized our conversation with Elana Hashman (EH) & Sergey Kanzhelev (SK), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved.

Introducing Single Pod Access Mode for PersistentVolumes

Mon, 13 Sep 2021 00:00:00 +0000

Last month's release of Kubernetes v1.22 introduced a new ReadWriteOncePod access mode for PersistentVolumes and PersistentVolumeClaims. With this alpha feature, Kubernetes allows you to restrict volume access to a single pod in the cluster.

What are access modes and why are they important?

When using storage, there are different ways to model how that storage is consumed.

For example, a storage system like a network file share can have many users all reading and writing data simultaneously. In other cases maybe everyone is allowed to read data but not write it. For highly sensitive data, maybe only one user is allowed to read and write data but nobody else.

Alpha in Kubernetes v1.22: API Server Tracing

Fri, 03 Sep 2021 00:00:00 +0000

In distributed systems, it can be hard to figure out where problems are. You grep through one component's logs just to discover that the source of your problem is in another component. You search there only to discover that you need to enable debug logs to figure out what really went wrong... And it goes on. The more complex the path your request takes, the harder it is to answer questions about where it went. I've personally spent many hours doing this dance with a variety of Kubernetes components. Distributed tracing is a tool which is designed to help in these situations, and the Kubernetes API Server is, perhaps, the most important Kubernetes component to be able to debug. At Kubernetes' Sig Instrumentation, our mission is to make it easier to understand what's going on in your cluster, and we are happy to announce that distributed tracing in the Kubernetes API Server reached alpha in 1.22.

Kubernetes 1.22: A New Design for Volume Populators

Mon, 30 Aug 2021 00:00:00 +0000

Kubernetes v1.22, released earlier this month, introduced a redesigned approach for volume populators. Originally implemented in v1.18, the API suffered from backwards compatibility issues. Kubernetes v1.22 includes a new API field called dataSourceRef that fixes these problems.

Data sources

Earlier Kubernetes releases already added a dataSource field into the PersistentVolumeClaim API, used for cloning volumes and creating volumes from snapshots. You could use the dataSource field when creating a new PVC, referencing either an existing PVC or a VolumeSnapshot in the same namespace. That also modified the normal provisioning process so that instead of yielding an empty volume, the new PVC contained the same data as either the cloned PVC or the cloned VolumeSnapshot.

Minimum Ready Seconds for StatefulSets

Fri, 27 Aug 2021 00:00:00 +0000

This blog describes the notion of Availability for StatefulSet workloads, and a new alpha feature in Kubernetes 1.22 which adds minReadySeconds configuration for StatefulSets.

What problems does this solve?

Prior to Kubernetes 1.22 release, once a StatefulSet Pod is in the Ready state it is considered Available to receive traffic. For some of the StatefulSet workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered Available only when Alertmanager's state transfer is complete, not when the Pod is in Ready state. Since minReadySeconds adds buffer, the state transfer may be complete before the Pod becomes Available. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the Pod is considered Available and it is ready to serve requests.

Enable seccomp for all workloads with a new v1.22 alpha feature

Wed, 25 Aug 2021 00:00:00 +0000

This blog post is about a new Kubernetes feature introduced in v1.22, which adds an additional security layer on top of the existing seccomp support. Seccomp is a security mechanism for Linux processes to filter system calls (syscalls) based on a set of defined rules. Applying seccomp profiles to containerized workloads is one of the key tasks when it comes to enhancing the security of the application deployment. Developers, site reliability engineers and infrastructure administrators have to work hand in hand to create, distribute and maintain the profiles over the applications life-cycle.

Alpha in v1.22: Windows HostProcess Containers

Mon, 16 Aug 2021 00:00:00 +0000

Kubernetes v1.22 introduced a new alpha feature for clusters that include Windows nodes: HostProcess containers.

HostProcess containers aim to extend the Windows container model to enable a wider range of Kubernetes cluster management scenarios. HostProcess containers run directly on the host and maintain behavior and access similar to that of a regular process. With HostProcess containers, users can package and distribute management operations and functionalities that require host access while retaining versioning and deployment methods provided by containers. This allows Windows containers to be used for a variety of device plugin, storage, and networking management scenarios in Kubernetes. With this comes the enablement of host network mode—allowing HostProcess containers to be created within the host's network namespace instead of their own. HostProcess containers can also be built on top of existing Windows server 2019 (or later) base images, managed through the Windows container runtime, and run as any user that is available on or in the domain of the host machine.

Kubernetes Memory Manager moves to beta

Wed, 11 Aug 2021 00:00:00 +0000

The blog post explains some of the internals of the Memory manager, a beta feature of Kubernetes 1.22. In Kubernetes, the Memory Manager is a kubelet subcomponent. The memory manage provides guaranteed memory (and hugepages) allocation for pods in the Guaranteed QoS class.

This blog post covers:

Why do you need it?
The internal details of how the MemoryManager works
Current limitations of the MemoryManager
Future work for the MemoryManager

Why do you need it?

Some Kubernetes workloads run on nodes with non-uniform memory access (NUMA). Suppose you have NUMA nodes in your cluster. In that case, you'll know about the potential for extra latency when compute resources need to access memory on the different NUMA locality.

Kubernetes 1.22: CSI Windows Support (with CSI Proxy) reaches GA

Mon, 09 Aug 2021 00:00:00 +0000

The stable version of CSI Proxy for Windows has been released alongside Kubernetes 1.22. CSI Proxy enables CSI Drivers running on Windows nodes to perform privileged storage operations.

Background

Container Storage Interface (CSI) for Kubernetes went GA in the Kubernetes 1.13 release. CSI has become the standard for exposing block and file storage to containerized workloads on Container Orchestration systems (COs) like Kubernetes. It enables third-party storage providers to write and deploy plugins without the need to alter the core Kubernetes codebase. Legacy in-tree drivers are deprecated and new storage features are introduced in CSI, therefore it is important to get CSI Drivers to work on Windows.

New in Kubernetes v1.22: alpha support for using swap memory

Mon, 09 Aug 2021 00:00:00 +0000

The 1.22 release introduced alpha support for configuring swap memory usage for Kubernetes workloads on a per-node basis.

In prior releases, Kubernetes did not support the use of swap memory on Linux, as it is difficult to provide guarantees and account for pod memory utilization when swap is involved. As part of Kubernetes' earlier design, swap support was considered out of scope, and a kubelet would by default fail to start if swap was detected on a node.

Kubernetes 1.22: Server Side Apply moves to GA

Fri, 06 Aug 2021 00:00:00 +0000

Server-side Apply (SSA) has been promoted to GA in the Kubernetes v1.22 release. The GA milestone means you can depend on the feature and its API, without fear of future backwards-incompatible changes. GA features are protected by the Kubernetes deprecation policy.

What is Server-side Apply?

Server-side Apply helps users and controllers manage their resources through declarative configurations. Server-side Apply replaces the client side apply feature implemented by “kubectl apply” with a server-side implementation, permitting use by tools/clients other than kubectl. Server-side Apply is a new merging algorithm, as well as tracking of field ownership, running on the Kubernetes api-server. Server-side Apply enables new features like conflict detection, so the system knows when two actors are trying to edit the same field. Refer to the Server-side Apply Documentation and Beta 2 release announcement for more information.

Kubernetes 1.22: Reaching New Peaks

Wed, 04 Aug 2021 00:00:00 +0000

We’re pleased to announce the release of Kubernetes 1.22, the second release of 2021!

This release consists of 53 enhancements: 13 enhancements have graduated to stable, 24 enhancements are moving to beta, and 16 enhancements are entering alpha. Also, three features have been deprecated.

In April of this year, the Kubernetes release cadence was officially changed from four to three releases yearly. This is the first longer-cycle release related to that change. As the Kubernetes project matures, the number of enhancements per cycle grows. This means more work, from version to version, for the contributor community and Release Engineering team, and it can put pressure on the end-user community to stay up-to-date with releases containing increasingly more features.

Roorkee robots, releases and racing: the Kubernetes 1.21 release interview

Thu, 29 Jul 2021 00:00:00 +0000

With Kubernetes 1.22 due out next week, now is a great time to look back on 1.21. The release team for that version was led by Nabarun Pal from VMware.

Back in April I interviewed Nabarun on the weekly Kubernetes Podcast from Google; the latest in a series of release lead conversations that started back with 1.11, not long after the show started back in 2018.

In these interviews we learn a little about the release, but also about the process behind it, and the story behind the person chosen to lead it. Getting to know a community member is my favourite part of the show each week, and so I encourage you to subscribe wherever you get your podcasts. With a release coming next week, you can probably guess what our next topic will be!

Updating NGINX-Ingress to use the stable Ingress API

Mon, 26 Jul 2021 00:00:00 +0000

With all Kubernetes APIs, there is a process to creating, maintaining, and ultimately deprecating them once they become GA. The networking.k8s.io API group is no different. The upcoming Kubernetes 1.22 release will remove several deprecated APIs that are relevant to networking:

the networking.k8s.io/v1beta1 API version of IngressClass
all beta versions of Ingress: extensions/v1beta1 and networking.k8s.io/v1beta1

On a v1.22 Kubernetes cluster, you'll be able to access Ingress and IngressClass objects through the stable (v1) APIs, but access via their beta APIs won't be possible. This change has been in in discussion since 2017, 2019 with 1.16 Kubernetes API deprecations, and most recently in KEP-1453: Graduate Ingress API to GA.

Kubernetes Release Cadence Change: Here’s What You Need To Know

Tue, 20 Jul 2021 00:00:00 +0000

On April 23, 2021, the Release Team merged a Kubernetes Enhancement Proposal (KEP) changing the Kubernetes release cycle from four releases a year (once a quarter) to three releases a year.

This blog post provides a high level overview about what this means for the Kubernetes community's contributors and maintainers.

What's changing and when

Starting with the Kubernetes 1.22 release, a lightweight policy will drive the creation of each release schedule. This policy states:

Spotlight on SIG Usability

Thu, 15 Jul 2021 00:00:00 +0000

Note:

SIG Usability, which is featured in this Spotlight blog, has been deprecated and is no longer active. As a result, the links and information provided in this blog post may no longer be valid or relevant. Should there be renewed interest and increased participation in the future, the SIG may be revived. However, as of August 2023 the SIG is inactive per the Kubernetes community policy. The Kubernetes project encourages you to explore other SIGs and resources available on the Kubernetes website to stay up-to-date with the latest developments and enhancements in Kubernetes.

Introduction

Are you interested in learning about what SIG Usability does and how you can get involved? Well, you're at the right place. SIG Usability is all about making Kubernetes more accessible to new folks, and its main activity is conducting user research for the community. In this blog, we have summarized our conversation with Gaby Moreno, who walks us through the various aspects of being a part of the SIG and shares some insights about how others can get involved.

Kubernetes API and Feature Removals In 1.22: Here’s What You Need To Know

Wed, 14 Jul 2021 00:00:00 +0000

As the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old APIs they replace are deprecated, and eventually removed. See Kubernetes API removals to read more about Kubernetes' policy on removing APIs.

We want to make sure you're aware of some upcoming removals. These are beta APIs that you can use in current, supported Kubernetes versions, and they are already deprecated. The reason for all of these removals is that they have been superseded by a newer, stable (“GA”) API.

Announcing Kubernetes Community Group Annual Reports

Mon, 28 Jun 2021 10:00:00 -0800

Given the growth and scale of the Kubernetes project, the existing reporting mechanisms were proving to be inadequate and challenging. Kubernetes is a large open source project. With over 100000 commits just to the main k/kubernetes repository, hundreds of other code repositories in the project, and thousands of contributors, there's a lot going on. In fact, there are 37 contributor groups at the time of writing. We also value all forms of contribution and not just code changes.

Writing a Controller for Pod Labels

Mon, 21 Jun 2021 00:00:00 +0000

Operators are proving to be an excellent solution to running stateful distributed applications in Kubernetes. Open source tools like the Operator SDK provide ways to build reliable and maintainable operators, making it easier to extend Kubernetes and implement custom scheduling.

Kubernetes operators run complex software inside your cluster. The open source community has already built many operators for distributed applications like Prometheus, Elasticsearch, or Argo CD. Even outside of open source, operators can help to bring new functionality to your Kubernetes cluster.

Using Finalizers to Control Deletion

Fri, 14 May 2021 00:00:00 +0000

Deleting objects in Kubernetes can be challenging. You may think you’ve deleted something, only to find it still persists. While issuing a kubectl delete command and hoping for the best might work for day-to-day operations, understanding how Kubernetes delete commands operate will help you understand why some objects linger after deletion.

In this post, I’ll look at:

What properties of a resource govern deletion
How finalizers and owner references impact object deletion
How the propagation policy can be used to change the order of deletions
How deletion works, with examples

For simplicity, all examples will use ConfigMaps and basic shell commands to demonstrate the process. We’ll explore how the commands work and discuss repercussions and results from using them in practice.

Kubernetes 1.21: Metrics Stability hits GA

Fri, 23 Apr 2021 00:00:00 +0000

Kubernetes 1.21 marks the graduation of the metrics stability framework and along with it, the first officially supported stable metrics. Not only do stable metrics come with supportability guarantees, the metrics stability framework brings escape hatches that you can use if you encounter problematic metrics.

See the list of stable Kubernetes metrics here

What are stable metrics and why do we need them?

A stable metric is one which, from a consumption point of view, can be reliably consumed across a number of Kubernetes versions without risk of ingestion failure.

Evolving Kubernetes networking with the Gateway API

Thu, 22 Apr 2021 00:00:00 +0000

The Ingress resource is one of the many Kubernetes success stories. It created a diverse ecosystem of Ingress controllers which were used across hundreds of thousands of clusters in a standardized and consistent way. This standardization helped users adopt Kubernetes. However, five years after the creation of Ingress, there are signs of fragmentation into different but strikingly similar CRDs and overloaded annotations. The same portability that made Ingress pervasive also limited its future.

Graceful Node Shutdown Goes Beta

Wed, 21 Apr 2021 00:00:00 +0000

Graceful node shutdown, beta in 1.21, enables kubelet to gracefully evict pods during a node shutdown.

Kubernetes is a distributed system and as such we need to be prepared for inevitable failures — nodes will fail, containers might crash or be restarted, and - ideally - your workloads will be able to withstand these catastrophic events.

One of the common classes of issues are workload failures on node shutdown or restart. The best practice prior to bringing your node down is to safely drain and cordon your node. This will ensure that all pods running on this node can safely be evicted. An eviction will ensure your pods can follow the expected pod termination lifecycle meaning receiving a SIGTERM in your container and/or running preStopHooks.

Annotating Kubernetes Services for Humans

Tue, 20 Apr 2021 00:00:00 +0000

Have you ever been asked to troubleshoot a failing Kubernetes service and struggled to find basic information about the service such as the source repository and owner?

One of the problems as Kubernetes applications grow is the proliferation of services. As the number of services grows, developers start to specialize working with specific services. When it comes to troubleshooting, however, developers need to be able to find the source, understand the service and dependencies, and chat with the owning team for any service.

Defining Network Policy Conformance for Container Network Interface (CNI) providers

Tue, 20 Apr 2021 00:00:00 +0000

Special thanks to Tim Hockin and Bowie Du (Google), Dan Winship and Antonio Ojea (Red Hat), Casey Davenport and Shaun Crampton (Tigera), and Abhishek Raut and Antonin Bas (VMware) for being supportive of this work, and working with us to resolve issues in different Container Network Interfaces (CNIs) over time.

A brief conversation around "node local" Network Policies in April of 2020 inspired the creation of a NetworkPolicy subproject from SIG Network. It became clear that as a community, we need a rock-solid story around how to do pod network security on Kubernetes, and this story needed a community around it, so as to grow the cultural adoption of enterprise security patterns in K8s.

Introducing Indexed Jobs

Mon, 19 Apr 2021 00:00:00 +0000

Once you have containerized a non-parallel Job, it is quite easy to get it up and running on Kubernetes without modifications to the binary. In most cases, when running parallel distributed Jobs, you had to set a separate system to partition the work among the workers. For example, you could set up a task queue to assign one work item to each Pod or multiple items to each Pod until the queue is emptied.

Volume Health Monitoring Alpha Update

Fri, 16 Apr 2021 00:00:00 +0000

The CSI Volume Health Monitoring feature, originally introduced in 1.19 has undergone a large update for the 1.21 release.

Why add Volume Health Monitoring to Kubernetes?

Without Volume Health Monitoring, Kubernetes has no knowledge of the state of the underlying volumes of a storage system after a PVC is provisioned and used by a Pod. Many things could happen to the underlying storage system after a volume is provisioned in Kubernetes. For example, the volume could be deleted by accident outside of Kubernetes, the disk that the volume resides on could fail, it could be out of capacity, the disk may be degraded which affects its performance, and so on. Even when the volume is mounted on a pod and used by an application, there could be problems later on such as read/write I/O errors, file system corruption, accidental unmounting of the volume outside of Kubernetes, etc. It is very hard to debug and detect root causes when something happened like this.

Three Tenancy Models For Kubernetes

Thu, 15 Apr 2021 00:00:00 +0000

Kubernetes clusters are typically used by several teams in an organization. In other cases, Kubernetes may be used to deliver applications to end users requiring segmentation and isolation of resources across users from different organizations. Secure sharing of Kubernetes control plane and worker node resources allows maximizing productivity and saving costs in both cases.

The Kubernetes Multi-Tenancy Working Group is chartered with defining tenancy models for Kubernetes and making it easier to operationalize tenancy related use cases. This blog post, from the working group members, describes three common tenancy models and introduces related working group projects.

Local Storage: Storage Capacity Tracking, Distributed Provisioning and Generic Ephemeral Volumes hit Beta

Wed, 14 Apr 2021 00:00:00 +0000

The "generic ephemeral volumes" and "storage capacity tracking" features in Kubernetes are getting promoted to beta in Kubernetes 1.21. Together with the distributed provisioning support in the CSI external-provisioner, development and deployment of Container Storage Interface (CSI) drivers which manage storage locally on a node become a lot easier.

This blog post explains how such drivers worked before and how these features can be used to make drivers simpler.

Problems we are solving

There are drivers for local storage, like TopoLVM for traditional disks and PMEM-CSI for persistent memory. They work and are ready for usage today also on older Kubernetes releases, but making that possible was not trivial.

kube-state-metrics goes v2.0

Tue, 13 Apr 2021 00:00:00 +0000

What?

kube-state-metrics, a project under the Kubernetes organization, generates Prometheus format metrics based on the current state of the Kubernetes native resources. It does this by listening to the Kubernetes API and gathering information about resources and objects, e.g. Deployments, Pods, Services, and StatefulSets. A full list of resources is available in the documentation of kube-state-metrics.

Why?

There are numerous useful metrics and insights provided by kube-state-metrics right out of the box! These metrics can be used to serve as an insight into your cluster: Either through metrics alone, in the form of dashboards, or through an alerting pipeline. To provide a few examples:

Introducing Suspended Jobs

Mon, 12 Apr 2021 00:00:00 +0000

Jobs are a crucial part of Kubernetes' API. While other kinds of workloads such as Deployments, ReplicaSets, StatefulSets, and DaemonSets solve use-cases that require Pods to run forever, Jobs are useful when Pods need to run to completion. Commonly used in parallel batch processing, Jobs can be used in a variety of applications ranging from video rendering and database maintenance to sending bulk emails and scientific computing.

While the amount of parallelism and the conditions for Job completion are configurable, the Kubernetes API lacked the ability to suspend and resume Jobs. This is often desired when cluster resources are limited and a higher priority Job needs to execute in the place of another Job. Deleting the lower priority Job is a poor workaround as Pod completion history and other metrics associated with the Job will be lost.

Kubernetes 1.21: CronJob Reaches GA

Fri, 09 Apr 2021 00:00:00 +0000

In Kubernetes v1.21, the CronJob resource reached general availability (GA). We've also substantially improved the performance of CronJobs since Kubernetes v1.19, by implementing a new controller.

In Kubernetes v1.20 we launched a revised v2 controller for CronJobs, initially as an alpha feature. Kubernetes 1.21 uses the newer controller by default, and the CronJob resource itself is now GA (group version: batch/v1).

In this article, we'll take you through the driving forces behind this new development, give you a brief description of controller design for core Kubernetes, and we'll outline what you will gain from this improved controller.

Kubernetes 1.21: Power to the Community

Thu, 08 Apr 2021 00:00:00 +0000

We’re pleased to announce the release of Kubernetes 1.21, our first release of 2021! This release consists of 51 enhancements: 13 enhancements have graduated to stable, 16 enhancements are moving to beta, 20 enhancements are entering alpha, and 2 features have been deprecated.

This release cycle, we saw a major shift in ownership of processes around the release team. We moved from a synchronous mode of communication, where we periodically asked the community for inputs, to a mode where the community opts-in to contribute features and/or blogs to the release. These changes have resulted in an increase in collaboration and teamwork across the community. The result of all that is reflected in Kubernetes 1.21 having the most number of features in the recent times.

PodSecurityPolicy Deprecation: Past, Present, and Future

Tue, 06 Apr 2021 00:00:00 +0000

Update: With the release of Kubernetes v1.25, PodSecurityPolicy has been removed. You can read more information about the removal of PodSecurityPolicy in the Kubernetes 1.25 release notes.

PodSecurityPolicy (PSP) is being deprecated in Kubernetes 1.21, to be released later this week. This starts the countdown to its removal, but doesn’t change anything else. PodSecurityPolicy will continue to be fully functional for several more releases before being removed completely. In the meantime, we are developing a replacement for PSP that covers key use cases more easily and sustainably.

The Evolution of Kubernetes Dashboard

Tue, 09 Mar 2021 00:00:00 +0000

In October 2020, the Kubernetes Dashboard officially turned five. As main project maintainers, we can barely believe that so much time has passed since our very first commits to the project. However, looking back with a bit of nostalgia, we realize that quite a lot has happened since then. Now it’s due time to celebrate “our baby” with a short recap.

How It All Began

The initial idea behind the Kubernetes Dashboard project was to provide a web interface for Kubernetes. We wanted to reflect the kubectl functionality through an intuitive web UI. The main benefit from using the UI is to be able to quickly see things that do not work as expected (monitoring and troubleshooting). Also, the Kubernetes Dashboard is a great starting point for users that are new to the Kubernetes ecosystem.

A Custom Kubernetes Scheduler to Orchestrate Highly Available Applications

Mon, 21 Dec 2020 00:00:00 +0000

As long as you're willing to follow the rules, deploying on Kubernetes and air travel can be quite pleasant. More often than not, things will "just work". However, if one is interested in travelling with an alligator that must remain alive or scaling a database that must remain available, the situation is likely to become a bit more complicated. It may even be easier to build one's own plane or database for that matter. Travelling with reptiles aside, scaling a highly available stateful system is no trivial task.

Kubernetes 1.20: Pod Impersonation and Short-lived Volumes in CSI Drivers

Fri, 18 Dec 2020 00:00:00 +0000

Typically when a CSI driver mounts credentials such as secrets and certificates, it has to authenticate against storage providers to access the credentials. However, the access to those credentials are controlled on the basis of the pods' identities rather than the CSI driver's identity. CSI drivers, therefore, need some way to retrieve pod's service account token.

Currently there are two suboptimal approaches to achieve this, either by granting CSI drivers the permission to use TokenRequest API or by reading tokens directly from the host filesystem.

Third Party Device Metrics Reaches GA

Wed, 16 Dec 2020 00:00:00 +0000

With Kubernetes 1.20, infrastructure teams who manage large scale Kubernetes clusters, are seeing the graduation of two exciting and long awaited features:

The Pod Resources API (introduced in 1.13) is finally graduating to GA. This allows Kubernetes plugins to obtain information about the node’s resource usage and assignment; for example: which pod/container consumes which device.
The DisableAcceleratorMetrics feature (introduced in 1.19) is graduating to beta and will be enabled by default. This removes device metrics reported by the kubelet in favor of the new plugin architecture.

Many of the features related to fundamental device support (device discovery, plugin, and monitoring) are reaching a strong level of stability. Kubernetes users should see these features as stepping stones to enable more complex use cases (networking, scheduling, storage, etc.)!

Kubernetes 1.20: Granular Control of Volume Permission Changes

Mon, 14 Dec 2020 00:00:00 +0000

Kubernetes 1.20 brings two important beta features, allowing Kubernetes admins and users alike to have more adequate control over how volume permissions are applied when a volume is mounted inside a Pod.

Allow users to skip recursive permission changes on mount

Traditionally if your pod is running as a non-root user (which you should), you must specify a fsGroup inside the pod’s security context so that the volume can be readable and writable by the Pod. This requirement is covered in more detail in here.

Kubernetes 1.20: Kubernetes Volume Snapshot Moves to GA

Thu, 10 Dec 2020 00:00:00 +0000

The Kubernetes Volume Snapshot feature is now GA in Kubernetes v1.20. It was introduced as alpha in Kubernetes v1.12, followed by a second alpha with breaking changes in Kubernetes v1.13, and promotion to beta in Kubernetes 1.17. This blog post summarizes the changes releasing the feature from beta to GA.

What is a volume snapshot?

Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage, and many on-premise storage systems) provide the ability to create a “snapshot” of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to rehydrate a new volume (pre-populated with the snapshot data) or to restore an existing volume to a previous state (represented by the snapshot).

Kubernetes 1.20: The Raddest Release

Tue, 08 Dec 2020 00:00:00 +0000

We’re pleased to announce the release of Kubernetes 1.20, our third and final release of 2020! This release consists of 42 enhancements: 11 enhancements have graduated to stable, 15 enhancements are moving to beta, and 16 enhancements are entering alpha.

The 1.20 release cycle returned to its normal cadence of 11 weeks following the previous extended release cycle. This is one of the most feature dense releases in a while: the Kubernetes innovation cycle is still trending upward. This release has more alpha than stable enhancements, showing that there is still much to explore in the cloud native ecosystem.

GSoD 2020: Improving the API Reference Experience

Fri, 04 Dec 2020 00:00:00 +0000

Editor's note: Better API references have been my goal since I joined Kubernetes docs three and a half years ago. Philippe has succeeded fantastically. More than a better API reference, though, Philippe embodied the best of the Kubernetes community in this project: excellence through collaboration, and a process that made the community itself better. Thanks, Google Season of Docs, for making Philippe's work possible. —Zach Corleissen

Introduction

The Google Season of Docs project brings open source organizations and technical writers together to work closely on a specific documentation project.

Dockershim Deprecation FAQ

Wed, 02 Dec 2020 00:00:00 +0000

Update: There is a newer version of this article available.

This document goes over some frequently asked questions regarding the Dockershim deprecation announced as a part of the Kubernetes v1.20 release. For more detail on the deprecation of Docker as a container runtime for Kubernetes kubelets, and what that means, check out the blog post Don't Panic: Kubernetes and Docker.

Also, you can read check whether Dockershim removal affects you to check whether it does.

Don't Panic: Kubernetes and Docker

Wed, 02 Dec 2020 00:00:00 +0000

Update: Kubernetes support for Docker via dockershim is now removed. For more information, read the removal FAQ. You can also discuss the deprecation via a dedicated GitHub issue.

Kubernetes is deprecating Docker as a container runtime after v1.20.

You do not need to panic. It’s not as dramatic as it sounds.

TL;DR Docker as an underlying runtime is being deprecated in favor of runtimes that use the Container Runtime Interface (CRI) created for Kubernetes. Docker-produced images will continue to work in your cluster with all runtimes, as they always have.

Cloud native security for your clusters

Wed, 18 Nov 2020 00:00:00 +0000

Over the last few years a small, security focused community has been working diligently to deepen our understanding of security, given the evolving cloud native infrastructure and corresponding iterative deployment practices. To enable sharing of this knowledge with the rest of the community, members of CNCF SIG Security (a group which reports into CNCF TOC and who are friends with Kubernetes SIG Security) led by Emily Fox, collaborated on a whitepaper outlining holistic cloud native security concerns and best practices. After over 1200 comments, changes, and discussions from 35 members across the world, we are proud to share cloud native security whitepaper v1.0 that serves as essential reading for security leadership in enterprises, financial and healthcare industries, academia, government, and non-profit organizations.

Remembering Dan Kohn

Mon, 02 Nov 2020 00:00:00 +0000

Dan Kohn was instrumental in getting Kubernetes and CNCF community to where it is today. He shared our values, motivations, enthusiasm, community spirit, and helped the Kubernetes community to become the best that it could be. Dan loved getting people together to solve problems big and small. He enabled people to grow their individual scope in the community which often helped launch their career in open source software.

Dan built a coalition around the nascent Kubernetes project and turned that into a cornerstone to build the larger cloud native space. He loved challenges, especially ones where the payoff was great like building worldwide communities, spreading the love of open source, and helping diverse, underprivileged communities and students to get a head start in technology.

Announcing the 2020 Steering Committee Election Results

Mon, 12 Oct 2020 00:00:00 +0000

The 2020 Steering Committee Election is now complete. In 2019, the committee arrived at its final allocation of 7 seats, 3 of which were up for election in 2020. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

Contributing to the Development Guide

Thu, 01 Oct 2020 00:00:00 +0000

When most people think of contributing to an open source project, I suspect they probably think of contributing code changes, new features, and bug fixes. As a software engineer and a long-time open source user and contributor, that's certainly what I thought. Although I have written a good quantity of documentation in different workflows, the massive size of the Kubernetes community was a new kind of "client." I just didn't know what to expect when Google asked my compatriots and me at Lion's Way to make much-needed updates to the Kubernetes Development Guide.

GSoC 2020 - Building operators for cluster addons

Wed, 16 Sep 2020 00:00:00 +0000

Introduction

Google Summer of Code is a global program that is geared towards introducing students to open source. Students are matched with open-source organizations to work with them for three months during the summer.

My name is Somtochi Onyekwere from the Federal University of Technology, Owerri (Nigeria) and this year, I was given the opportunity to work with Kubernetes (under the CNCF organization) and this led to an amazing summer spent learning, contributing and interacting with the community.

Introducing Structured Logs

Fri, 04 Sep 2020 00:00:00 +0000

Logs are an essential aspect of observability and a critical tool for debugging. But Kubernetes logs have traditionally been unstructured strings, making any automated parsing difficult and any downstream processing, analysis, or querying challenging to do reliably.

In Kubernetes 1.19, we are adding support for structured logs, which natively support (key, value) pairs and object references. We have also updated many logging calls such that over 99% of logging volume in a typical deployment are now migrated to the structured format.

Warning: Helpful Warnings Ahead

Thu, 03 Sep 2020 00:00:00 +0000

As Kubernetes maintainers, we're always looking for ways to improve usability while preserving compatibility. As we develop features, triage bugs, and answer support questions, we accumulate information that would be helpful for Kubernetes users to know. In the past, sharing that information was limited to out-of-band methods like release notes, announcement emails, documentation, and blog posts. Unless someone knew to seek out that information and managed to find it, they would not benefit from it.

Scaling Kubernetes Networking With EndpointSlices

Wed, 02 Sep 2020 00:00:00 +0000

EndpointSlices are an exciting new API that provides a scalable and extensible alternative to the Endpoints API. EndpointSlices track IP addresses, ports, readiness, and topology information for Pods backing a Service.

In Kubernetes 1.19 this feature is enabled by default with kube-proxy reading from EndpointSlices instead of Endpoints. Although this will mostly be an invisible change, it should result in noticeable scalability improvements in large clusters. It also enables significant new features in future Kubernetes releases like Topology Aware Routing.

Ephemeral volumes with storage capacity tracking: EmptyDir on steroids

Tue, 01 Sep 2020 00:00:00 +0000

Kubernetes already supports several kinds of such ephemeral volumes, but the functionality of those is limited to what is implemented inside Kubernetes.

Increasing the Kubernetes Support Window to One Year

Mon, 31 Aug 2020 00:00:00 +0000

Starting with Kubernetes 1.19, the support window for Kubernetes versions will increase from 9 months to one year. The longer support window is intended to allow organizations to perform major upgrades at a time of the year that works the best for them.

This is a big change. For many years, the Kubernetes project has delivered a new minor release (e.g.: 1.13 or 1.14) every 3 months. The project provides bugfix support via patch releases (e.g.: 1.13.Y) for three parallel branches of the codebase. Combined, this led to each minor release (e.g.: 1.13) having a patch release stream of support for approximately 9 months. In the end, a cluster operator had to upgrade at least every 9 months to remain supported.

Kubernetes 1.19: Accentuate the Paw-sitive

Wed, 26 Aug 2020 00:00:00 +0000

Finally, we have arrived with Kubernetes 1.19, the second release for 2020, and by far the longest release cycle lasting 20 weeks in total. It consists of 34 enhancements: 10 enhancements are moving to stable, 15 enhancements in beta, and 9 enhancements in alpha.

The 1.19 release was quite different from a regular release due to COVID-19, the George Floyd protests, and several other global events that we experienced as a release team. Due to these events, we made the decision to adjust our timeline and allow the SIGs, Working Groups, and contributors more time to get things done. The extra time also allowed for people to take time to focus on their lives outside of the Kubernetes project, and ensure their mental wellbeing was in a good place.

Moving Forward From Beta

Fri, 21 Aug 2020 00:00:00 +0000

In Kubernetes, features follow a defined lifecycle. First, as the twinkle of an eye in an interested developer. Maybe, then, sketched in online discussions, drawn on the online equivalent of a cafe napkin. This rough work typically becomes a Kubernetes Enhancement Proposal (KEP), and from there it usually turns into code.

For Kubernetes v1.20 and onwards, we're focusing on helping that code graduate into stable features.

That lifecycle I mentioned runs as follows:

Introducing Hierarchical Namespaces

Fri, 14 Aug 2020 00:00:00 +0000

Safely hosting large numbers of users on a single Kubernetes cluster has always been a troublesome task. One key reason for this is that different organizations use Kubernetes in different ways, and so no one tenancy model is likely to suit everyone. Instead, Kubernetes offers you building blocks to create your own tenancy solution, such as Role Based Access Control (RBAC) and NetworkPolicies; the better these building blocks, the easier it is to safely build a multitenant cluster.

Physics, politics and Pull Requests: the Kubernetes 1.18 release interview

Mon, 03 Aug 2020 00:00:00 +0000

The start of the COVID-19 pandemic couldn't delay the release of Kubernetes 1.18, but unfortunately a small bug could — thankfully only by a day. This was the last cat that needed to be herded by 1.18 release lead Jorge Alarcón before the release on March 25.

One of the best parts about co-hosting the weekly Kubernetes Podcast from Google is the conversations we have with the people who help bring Kubernetes releases together. Jorge was our guest on episode 96 back in March, and just like last week we are delighted to bring you the transcript of this interview.

Music and math: the Kubernetes 1.17 release interview

Mon, 27 Jul 2020 00:00:00 +0000

Every time the Kubernetes release train stops at the station, we like to ask the release lead to take a moment to reflect on their experience. That takes the form of an interview on the weekly Kubernetes Podcast from Google that I co-host with Craig Box. If you're not familiar with the show, every week we summarise the new in the Cloud Native ecosystem, and have an insightful discussion with an interesting guest from the broader Kubernetes community.

SIG-Windows Spotlight

Tue, 30 Jun 2020 00:00:00 +0000

This post tells the story of how Kubernetes contributors work together to provide a container orchestrator that works for both Linux and Windows.

Most people who are familiar with Kubernetes are probably used to associating it with Linux. The connection makes sense, since Kubernetes ran on Linux from its very beginning. However, many teams and organizations working on adopting Kubernetes need the ability to orchestrate containers on Windows. Since the release of Docker and rise to popularity of containers, there have been efforts both from the community and from Microsoft itself to make container technology as accessible in Windows systems as it is in Linux systems.

Working with Terraform and Kubernetes

Mon, 29 Jun 2020 00:00:00 +0000

Maintaining Kubestack, an open-source Terraform GitOps Framework for Kubernetes, I unsurprisingly spend a lot of time working with Terraform and Kubernetes. Kubestack provisions managed Kubernetes services like AKS, EKS and GKE using Terraform but also integrates cluster services from Kustomize bases into the GitOps workflow. Think of cluster services as everything that's required on your Kubernetes cluster, before you can deploy application workloads.

Hashicorp recently announced better integration between Terraform and Kubernetes. I took this as an opportunity to give an overview of how Terraform can be used with Kubernetes today and what to be aware of.

A Better Docs UX With Docsy

Mon, 15 Jun 2020 00:00:00 +0000

Editor's note: Zach is one of the chairs for the Kubernetes documentation special interest group (SIG Docs).

I'm pleased to announce that the Kubernetes website now features the Docsy Hugo theme.

The Docsy theme improves the site's organization and navigability, and opens a path to improved API references. After over 4 years with few meaningful UX improvements, Docsy implements some best practices for technical content. The theme makes the Kubernetes site easier to read and makes individual pages easier to navigate. It gives the site a much-needed facelift.

Supporting the Evolving Ingress Specification in Kubernetes 1.18

Fri, 05 Jun 2020 00:00:00 +0000

Earlier this year, the Kubernetes team released Kubernetes 1.18, which extended Ingress. In this blog post, we’ll walk through what’s new in the new Ingress specification, what it means for your applications, and how to upgrade to an ingress controller that supports this new specification.

What is Kubernetes Ingress

When deploying your applications in Kubernetes, one of the first challenges many people encounter is how to get traffic into their cluster. Kubernetes ingress is a collection of routing rules that govern how external users access services running in a Kubernetes cluster. There are three general approaches for exposing your application:

K8s KPIs with Kuberhealthy

Fri, 29 May 2020 00:00:00 +0000

Building Onward from Kuberhealthy v2.0.0

Last November at KubeCon San Diego 2019, we announced the release of Kuberhealthy 2.0.0 - transforming Kuberhealthy into a Kubernetes operator for synthetic monitoring. This new ability granted developers the means to create their own Kuberhealthy check containers to synthetically monitor their applications and clusters. The community was quick to adopt this new feature and we're grateful for everyone who implemented and tested Kuberhealthy 2.0.0 in their clusters. Thanks to all of you who reported issues and contributed to discussions on the #kuberhealthy Slack channel. We quickly set to work to address all your feedback with a newer version of Kuberhealthy. Additionally, we created a guide on how to easily install and use Kuberhealthy in order to capture some helpful synthetic KPIs.

My exciting journey into Kubernetes’ history

Thu, 28 May 2020 00:00:00 +0000

Editor's note: Sascha is part of SIG Release and is working on many other different container runtime related topics. Feel free to reach him out on Twitter @saschagrunert.

A story of data science-ing 90,000 GitHub issues and pull requests by using Kubeflow, TensorFlow, Prow and a fully automated CI/CD pipeline.

Introduction
Getting the Data
Exploring the Data
- Labels, Labels, Labels
Building the Machine Learning Model
Automate Everything
Automatic Labeling of new PRs
Summary

Introduction

Choosing the right steps when working in the field of data science is truly no silver bullet. Most data scientists might have their custom workflow, which could be more or less automated, depending on their area of work. Using Kubernetes can be a tremendous enhancement when trying to automate workflows on a large scale. In this blog post, I would like to take you on my journey of doing data science while integrating the overall workflow into Kubernetes.

An Introduction to the K8s-Infrastructure Working Group

Wed, 27 May 2020 00:00:00 +0000

Author: Kiran "Rin" Oliver Storyteller, Kubernetes Upstream Marketing Team

An Introduction to the K8s-Infrastructure Working Group

Welcome to part one of a new series introducing the K8s-Infrastructure working group!

When Kubernetes was formed in 2014, Google undertook the task of building and maintaining the infrastructure necessary for keeping the project running smoothly. The tools itself were open source, but the Google Cloud Platform project used to run the infrastructure was internal-only, preventing contributors from being able to help out. In August 2018, Google granted the Cloud Native Computing Foundation $9M in credits for the operation of Kubernetes. The sentiment behind this was that a project such as Kubernetes should be both maintained and operated by the community itself rather than by a single vendor.

WSL+Docker: Kubernetes on the Windows Desktop

Thu, 21 May 2020 00:00:00 +0000

Introduction

New to Windows 10 and WSL2, or new to Docker and Kubernetes? Welcome to this blog post where we will install from scratch Kubernetes in Docker KinD and Minikube.

Why Kubernetes on Windows?

For the last few years, Kubernetes became a de-facto standard platform for running containerized services and applications in distributed environments. While a wide variety of distributions and installers exist to deploy Kubernetes in the cloud environments (public, private or hybrid), or within the bare metal environments, there is still a need to deploy and run Kubernetes locally, for example, on the developer's workstation.

How Docs Handle Third Party and Dual Sourced Content

Wed, 06 May 2020 00:00:00 +0000

Editor's note: Zach is one of the chairs for the Kubernetes documentation special interest group (SIG Docs).

Late last summer, SIG Docs started a community conversation about third party content in Kubernetes docs. This conversation became a Kubernetes Enhancement Proposal (KEP) and, after five months for review and comment, SIG Architecture approved the KEP as a content guide for Kubernetes docs.

Here's how Kubernetes docs handle third party content now:

Links to active content in the Kubernetes project (projects in the kubernetes and kubernetes-sigs GitHub orgs) are always allowed.

Introducing PodTopologySpread

Tue, 05 May 2020 00:00:00 +0000

Managing Pods distribution across a cluster is hard. The well-known Kubernetes features for Pod affinity and anti-affinity, allow some control of Pod placement in different topologies. However, these features only resolve part of Pods distribution use cases: either place unlimited Pods to a single topology, or disallow two Pods to co-locate in the same topology. In between these two extreme cases, there is a common need to distribute the Pods evenly across the topologies, so as to achieve better cluster utilization and high availability of applications.

Two-phased Canary Rollout with Open Source Gloo

Wed, 22 Apr 2020 00:00:00 +0000

Author: Rick Ducott | GitHub | Twitter

Every day, my colleagues and I are talking to platform owners, architects, and engineers who are using Gloo as an API gateway to expose their applications to end users. These applications may span legacy monoliths, microservices, managed cloud services, and Kubernetes clusters. Fortunately, Gloo makes it easy to set up routes to manage, secure, and observe application traffic while supporting a flexible deployment architecture to meet the varying production needs of our users.

Cluster API v1alpha3 Delivers New Features and an Improved User Experience

Tue, 21 Apr 2020 00:00:00 +0000

The Cluster API is a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management. It provides optional, additive functionality on top of core Kubernetes to manage the lifecycle of a Kubernetes cluster.

Following the v1alpha2 release in October 2019, many members of the Cluster API community met in San Francisco, California, to plan the next release. The project had just gone through a major transformation, delivering a new architecture that promised to make the project easier for users to adopt, and faster for the community to build. Over the course of those two days, we found our common goals: To implement the features critical to managing production clusters, to make its user experience more intuitive, and to make it a joy to develop.

How Kubernetes contributors are building a better communication process

Tue, 21 Apr 2020 00:00:00 +0000

"Perhaps we just need to use a different word. We may need to use community development or project advocacy as a word in the open source realm as opposed to marketing, and perhaps then people will realize that they need to do it." ~ Nithya Ruff (from TODO Group)

A common way to participate in the Kubernetes contributor community is to be everywhere.

We have an active Slack, many mailing lists, Twitter account(s), and dozens of community-driven podcasts and newsletters that highlight all end-user, contributor, and ecosystem topics. And to add on to that, we also have repositories of amazing documentation, tons of meetings that drive the project forward, and recorded code deep dives. All of this information is incredibly valuable, but it can be too much.

API Priority and Fairness Alpha

Mon, 06 Apr 2020 00:00:00 +0000

This blog describes “API Priority And Fairness”, a new alpha feature in Kubernetes 1.18. API Priority And Fairness permits cluster administrators to divide the concurrency of the control plane into different weighted priority levels. Every request arriving at a kube-apiserver will be categorized into one of the priority levels and get its fair share of the control plane’s throughput.

What problem does this solve?

Today the apiserver has a simple mechanism for protecting itself against CPU and memory overloads: max-in-flight limits for mutating and for readonly requests. Apart from the distinction between mutating and readonly, no other distinctions are made among requests; consequently, there can be undesirable scenarios where one subset of the requests crowds out other requests.

Introducing Windows CSI support alpha for Kubernetes

Fri, 03 Apr 2020 00:00:00 +0000

The alpha version of CSI Proxy for Windows is being released with Kubernetes 1.18. CSI proxy enables CSI Drivers on Windows by allowing containers in Windows to perform privileged storage operations.

Background

Container Storage Interface (CSI) for Kubernetes went GA in the Kubernetes 1.13 release. CSI has become the standard for exposing block and file storage to containerized workloads on Container Orchestration systems (COs) like Kubernetes. It enables third-party storage providers to write and deploy plugins without the need to alter the core Kubernetes codebase. All new storage features will utilize CSI, therefore it is important to get CSI drivers to work on Windows.

Improvements to the Ingress API in Kubernetes 1.18

Thu, 02 Apr 2020 00:00:00 +0000

The Ingress API in Kubernetes has enabled a large number of controllers to provide simple and powerful ways to manage inbound network traffic to Kubernetes workloads. In Kubernetes 1.18, we've made 3 significant additions to this API:

A new pathType field that can specify how Ingress paths should be matched.
A new IngressClass resource that can specify how Ingresses should be implemented by controllers.
Support for wildcards in hostnames.

Better Path Matching With Path Types

The new concept of a path type allows you to specify how a path should be matched. There are three supported types:

Kubernetes 1.18 Feature Server-side Apply Beta 2

Wed, 01 Apr 2020 00:00:00 +0000

What is Server-side Apply?

Server-side Apply is an important effort to migrate “kubectl apply” to the apiserver. It was started in 2018 by the Apply working group.

The use of kubectl to declaratively apply resources has exposed the following challenges:

One needs to use the kubectl go code, or they have to shell out to kubectl.
Strategic merge-patch, the patch format used by kubectl, grew organically and was challenging to fix while maintaining compatibility with various api-server versions.

Kubernetes Topology Manager Moves to Beta - Align Up!

Wed, 01 Apr 2020 00:00:00 +0000

This blog post describes the TopologyManager, a beta feature of Kubernetes in release 1.18. The TopologyManager feature enables NUMA alignment of CPUs and peripheral devices (such as SR-IOV VFs and GPUs), allowing your workload to run in an environment optimized for low-latency.

Prior to the introduction of the TopologyManager, the CPU and Device Manager would make resource allocation decisions independent of each other. This could result in undesirable allocations on multi-socket systems, causing degraded performance on latency critical applications. With the introduction of the TopologyManager, we now have a way to avoid this.

Kubernetes 1.18: Fit & Finish

Wed, 25 Mar 2020 00:00:00 +0000

We're pleased to announce the delivery of Kubernetes 1.18, our first release of 2020! Kubernetes 1.18 consists of 38 enhancements: 15 enhancements are moving to stable, 11 enhancements in beta, and 12 enhancements in alpha.

Kubernetes 1.18 is a "fit and finish" release. Significant work has gone into improving beta and stable features to ensure users have a better experience. An equal effort has gone into adding new developments and exciting new features that promise to enhance the user experience even more. Having almost as many enhancements in alpha, beta, and stable is a great achievement. It shows the tremendous effort made by the community on improving the reliability of Kubernetes as well as continuing to expand its existing functionality.

Join SIG Scalability and Learn Kubernetes the Hard Way

Thu, 19 Mar 2020 00:00:00 +0000

Contributing to SIG Scalability is a great way to learn Kubernetes in all its depth and breadth, and the team would love to have you join as a contributor. I took a look at the value of learning the hard way and interviewed the current SIG chairs to give you an idea of what contribution feels like.

The value of Learning The Hard Way

There is a belief in the software development community that pushes for the most challenging and rigorous possible method of learning a new language or system. These tend to go by the moniker of "Learn __ the Hard Way." Examples abound: Learn Code the Hard Way, Learn Python the Hard Way, and many others originating with Zed Shaw's courses in the topic.

Kong Ingress Controller and Service Mesh: Setting up Ingress to Istio on Kubernetes

Wed, 18 Mar 2020 00:00:00 +0000

Kubernetes has become the de facto way to orchestrate containers and the services within services. But how do we give services outside our cluster access to what is within? Kubernetes comes with the Ingress API object that manages external access to services within a cluster.

Ingress is a group of rules that will proxy inbound connections to endpoints defined by a backend. However, Kubernetes does not know what to do with Ingress resources without an Ingress controller, which is where an open source controller can come into play. In this post, we are going to use one option for this: the Kong Ingress Controller. The Kong Ingress Controller was open-sourced a year ago and recently reached one million downloads. In the recent 0.7 release, service mesh support was also added. Other features of this release include:

Contributor Summit Amsterdam Postponed

Wed, 04 Mar 2020 00:00:00 +0000

The CNCF has announced that KubeCon + CloudNativeCon EU has been delayed until July/August of 2020. As a result the Contributor Summit planning team is weighing options for how to proceed. Here’s the current plan:

There will be an in-person Contributor Summit as planned when KubeCon + CloudNativeCon is rescheduled.
We are looking at options for having additional virtual contributor activities in the meantime.

We will communicate via this blog and the usual communications channels on the final plan. Please bear with us as we adapt when we get more information. Thank you for being patient as the team pivots to bring you a great Contributor Summit!

Bring your ideas to the world with kubectl plugins

Fri, 28 Feb 2020 00:00:00 +0000

kubectl is the most critical tool to interact with Kubernetes and has to address multiple user personas, each with their own needs and opinions. One way to make kubectl do what you need is to build new functionality into kubectl.

Challenges with building commands into `kubectl`

However, that's easier said than done. Being such an important cornerstone of Kubernetes, any meaningful change to kubectl needs to undergo a Kubernetes Enhancement Proposal (KEP) where the intended change is discussed beforehand.

Contributor Summit Amsterdam Schedule Announced

Tue, 18 Feb 2020 00:00:00 +0000

Hello everyone and Happy 2020! It’s hard to believe that KubeCon EU 2020 is less than six weeks away, and with that another contributor summit! This year we have the pleasure of being in Amsterdam in early spring, so be sure to pack some warmer clothing. This summit looks to be exciting with a lot of fantastic community-driven content. We received 26 submissions from the CFP. From that, the events team selected 12 sessions. Each of the sessions falls into one of four categories:

Deploying External OpenStack Cloud Provider with Kubeadm

Fri, 07 Feb 2020 00:00:00 +0000

This document describes how to install a single control-plane Kubernetes cluster v1.15 with kubeadm on CentOS, and then deploy an external OpenStack cloud provider and Cinder CSI plugin to use Cinder volumes as persistent volumes in Kubernetes.

Preparation in OpenStack

This cluster runs on OpenStack VMs, so let's create a few things in OpenStack first.

A project/tenant for this Kubernetes cluster
A user in this project for Kubernetes, to query node information and attach volumes etc
A private network and subnet
A router for this private network and connect it to a public network for floating IPs
A security group for all Kubernetes VMs
A VM as a control-plane node and a few VMs as worker nodes

The security group will have the following rules to open ports for Kubernetes.

KubeInvaders - Gamified Chaos Engineering Tool for Kubernetes

Wed, 22 Jan 2020 00:00:00 +0000

Some months ago, I released my latest project called KubeInvaders. The first time I shared it with the community was during an Openshift Commons Briefing session. Kubenvaders is a Gamified Chaos Engineering tool for Kubernetes and Openshift and helps test how resilient your Kubernetes cluster is, in a fun way.

It is like Space Invaders, but the aliens are pods.

During my presentation at Codemotion Milan 2019, I started saying "of course you can do it with few lines of Bash, but it is boring."

CSI Ephemeral Inline Volumes

Tue, 21 Jan 2020 00:00:00 +0000

Typically, volumes provided by an external storage driver in Kubernetes are persistent, with a lifecycle that is completely independent of pods or (as a special case) loosely coupled to the first pod which uses a volume (late binding mode). The mechanism for requesting and defining such volumes in Kubernetes are Persistent Volume Claim (PVC) and Persistent Volume (PV) objects. Originally, volumes that are backed by a Container Storage Interface (CSI) driver could only be used via this PVC/PV mechanism.

Reviewing 2019 in Docs

Tue, 21 Jan 2020 00:00:00 +0000

Hi, folks! I'm one of the co-chairs for the Kubernetes documentation special interest group (SIG Docs). This blog post is a review of SIG Docs in 2019. Our contributors did amazing work last year, and I want to highlight their successes.

Although I review 2019 in this post, my goal is to point forward to 2020. I observe some trends in SIG Docs–some good, others troubling. I want to raise visibility before those challenges increase in severity.

Kubernetes on MIPS

Wed, 15 Jan 2020 00:00:00 +0000

Background

MIPS (Microprocessor without Interlocked Pipelined Stages) is a reduced instruction set computer (RISC) instruction set architecture (ISA), appeared in 1981 and developed by MIPS Technologies. Now MIPS architecture is widely used in many electronic products.

Kubernetes has officially supported a variety of CPU architectures such as x86, arm/arm64, ppc64le, s390x. However, it's a pity that Kubernetes doesn't support MIPS. With the widespread use of cloud native technology, users under MIPS architecture also have an urgent demand for Kubernetes on MIPS.

Announcing the Kubernetes bug bounty program

Tue, 14 Jan 2020 09:00:00 -0800

Authors: Maya Kaczorowski and Tim Allclair, Google, on behalf of the Kubernetes Product Security Committee

Today, the Kubernetes Product Security Committee is launching a new bug bounty program, funded by the CNCF, to reward researchers finding security vulnerabilities in Kubernetes.

Setting up a new bug bounty program

We aimed to set up this bug bounty program as transparently as possible, with an initial proposal, evaluation of vendors, and working draft of the components in scope. Once we onboarded the selected bug bounty program vendor, HackerOne, these documents were further refined based on the feedback from HackerOne, as well as what was learned in the recent Kubernetes security audit. The bug bounty program has been in a private release for several months now, with invited researchers able to submit bugs and help us test the triage process. After almost two years since the initial proposal, the program is now ready for all security researchers to contribute!

Remembering Brad Childs

Fri, 10 Jan 2020 10:00:00 -0800

Last year, the Kubernetes family lost one of its own. Brad Childs was a SIG Storage chair and long time contributor to the project. Brad worked on a number of features in storage and was known as much for his friendliness and sense of humor as for his technical contributions and leadership.

We recently spent time remembering Brad at Kubecon NA:

Our hearts go out to Brad’s friends and family and others whose lives he touched inside and outside the Kubernetes community.

Testing of CSI drivers

Wed, 08 Jan 2020 00:00:00 +0000

When developing a Container Storage Interface (CSI) driver, it is useful to leverage as much prior work as possible. This includes source code (like the sample CSI hostpath driver) but also existing tests. Besides saving time, using tests written by someone else has the advantage that it can point out aspects of the specification that might have been overlooked otherwise.

An earlier blog post about end-to-end testing already showed how to use the Kubernetes storage tests for testing of a third-party CSI driver. That approach makes sense when the goal to also add custom E2E tests, but depends on quite a bit of effort for setting up and maintaining a test suite.

Kubernetes 1.17: Stability

Mon, 09 Dec 2019 13:00:00 -0800

We’re pleased to announce the delivery of Kubernetes 1.17, our fourth and final release of 2019! Kubernetes v1.17 consists of 22 enhancements: 14 enhancements have graduated to stable, 4 enhancements are moving to beta, and 4 enhancements are entering alpha.

Major Themes

Cloud Provider Labels reach General Availability

Added as a beta feature way back in v1.2, v1.17 sees the general availability of cloud provider labels.

Volume Snapshot Moves to Beta

The Kubernetes Volume Snapshot feature is now beta in Kubernetes v1.17. It was introduced as alpha in Kubernetes v1.12, with a second alpha with breaking changes in Kubernetes v1.13.

Kubernetes 1.17 Feature: Kubernetes Volume Snapshot Moves to Beta

Mon, 09 Dec 2019 10:00:00 -0800

The Kubernetes Volume Snapshot feature is now beta in Kubernetes v1.17. It was introduced as alpha in Kubernetes v1.12, with a second alpha with breaking changes in Kubernetes v1.13. This post summarizes the changes in the beta release.

What is a Volume Snapshot?

Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage, and many on-premise storage systems) provide the ability to create a “snapshot” of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore an existing volume to a previous state (represented by the snapshot).

Kubernetes 1.17 Feature: Kubernetes In-Tree to CSI Volume Migration Moves to Beta

Mon, 09 Dec 2019 09:00:00 +0800

The Kubernetes in-tree storage plugin to Container Storage Interface (CSI) migration infrastructure is now beta in Kubernetes v1.17. CSI migration was introduced as alpha in Kubernetes v1.14.

Kubernetes features are generally introduced as alpha and moved to beta (and eventually to stable/GA) over subsequent Kubernetes releases. This process allows Kubernetes developers to get feedback, discover and fix issues, iterate on the designs, and deliver high quality, production grade features.

Why are we migrating in-tree plugins to CSI?

Prior to CSI, Kubernetes provided a powerful volume plugin system. These volume plugins were “in-tree” meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries. However, adding support for new volume plugins to Kubernetes was challenging. Vendors that wanted to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) were forced to align with the Kubernetes release process. In addition, third-party storage code caused reliability and security issues in core Kubernetes binaries and the code was often difficult (and in some cases impossible) for Kubernetes maintainers to test and maintain. Using the Container Storage Interface in Kubernetes resolves these major issues.

When you're in the release team, you're family: the Kubernetes 1.16 release interview

Fri, 06 Dec 2019 00:00:00 +0000

It is a pleasure to co-host the weekly Kubernetes Podcast from Google with Adam Glick. We get to talk to friends old and new from the community, as well as give people a download on the Cloud Native news every week.

It was also a pleasure to see Lachlan Evenson, the release team lead for Kubernetes 1.16, win the CNCF "Top Ambassador" award at KubeCon. We talked with Lachie when 1.16 was released, and as is becoming a tradition, we are delighted to share an abridged version of that interview with the readers of the Kubernetes Blog.

Gardener Project Update

Mon, 02 Dec 2019 00:00:00 +0000

Last year, we introduced Gardener in the Kubernetes Community Meeting and in a post on the Kubernetes Blog. At SAP, we have been running Gardener for more than two years, and are successfully managing thousands of conformant clusters in various versions on all major hyperscalers as well as in numerous infrastructures and private clouds that typically join an enterprise via acquisitions.

We are often asked why a handful of dynamically scalable clusters would not suffice. We also started our journey into Kubernetes with a similar mindset. But we realized that applying the architecture and principles of Kubernetes to productive scenarios, our internal and external customers very quickly required the rational separation of concerns and ownership, which in most circumstances led to the use of multiple clusters. Therefore, a scalable and managed Kubernetes as a service solution is often also the basis for adoption. Particularly, when a larger organization runs multiple products on different providers and in different regions, the number of clusters will quickly rise to the hundreds or even thousands.

Develop a Kubernetes controller in Java

Tue, 26 Nov 2019 00:00:00 +0000

The official Kubernetes Java SDK project recently released their latest work on providing the Java Kubernetes developers a handy Kubernetes controller-builder SDK which is helpful for easily developing advanced workloads or systems.

Overall

Java is no doubt one of the most popular programming languages in the world but it's been difficult for a period time for those non-Golang developers to build up their customized controller/operator due to the lack of library resources in the community. In the world of Golang, there're already some excellent controller frameworks, for example, controller runtime, operator SDK. These existing Golang frameworks are relying on the various utilities from the Kubernetes Golang SDK proven to be stable over years. Driven by the emerging need of further integration into the platform of Kubernetes, we not only ported many essential toolings from the Golang SDK into the kubernetes Java SDK including informers, work-queues, leader-elections, etc. but also developed a controller-builder SDK which wires up everything into a runnable controller without hiccups.

Running Kubernetes locally on Linux with Microk8s

Tue, 26 Nov 2019 00:00:00 +0000

This article, the second in a series about local deployment options on Linux, and covers MicroK8s. Microk8s is the click-and-run solution for deploying a Kubernetes cluster locally, originally developed by Canonical, the publisher of Ubuntu.

While Minikube usually spins up a local virtual machine (VM) for the Kubernetes cluster, MicroK8s doesn’t require a VM. It uses snap packages, an application packaging and isolation technology.

This difference has its pros and cons. Here we’ll discuss a few of the interesting differences, and comparing the benefits of a VM based approach with the benefits of a non-VM approach. One of the first factors is cross-platform portability. While a Minikube VM is portable across operating systems - it supports not only Linux, but Windows, macOS, and even FreeBSD - Microk8s requires Linux, and only on those distributions that support snaps. Most popular Linux distributions are supported.

Grokkin' the Docs

Tue, 05 Nov 2019 00:00:00 +0000

Definition courtesy of Merriam Webster online dictionary

Intro - Observations of a new SIG Docs contributor

I began contributing to the SIG Docs community in August 2019. Sometimes I feel like I am a stranger in a strange land adapting to a new community: investigating community organization, understanding contributor society, learning new lessons, and incorporating new jargon. I'm an observer as well as a contributor.

Kubernetes Documentation Survey

Tue, 29 Oct 2019 00:00:00 +0000

In September, SIG Docs conducted its first survey about the Kubernetes documentation. We'd like to thank the CNCF's Kim McMahon for helping us create the survey and access the results.

Key takeaways

Respondents would like more example code, more detailed content, and more diagrams in the Concepts, Tasks, and Reference sections.

74% of respondents would like the Tutorials section to contain advanced content.

69.70% said the Kubernetes documentation is the first place they look for information about Kubernetes.

Contributor Summit San Diego Schedule Announced!

Thu, 10 Oct 2019 00:00:00 +0000

There are many great sessions planned for the Contributor Summit, spread across five rooms of current contributor content in addition to the new contributor workshops. Since this is an upstream contributor summit and we don't often meet, being a globally distributed team, most of these sessions are discussions or hands-on labs, not just presentations. We want folks to learn and have a good time meeting their OSS teammates.

Unconference tracks are returning from last year with sessions to be chosen Monday morning. These are ideal for the latest hot topics and specific discussions that contributors want to have. In previous years, we've covered flaky tests, cluster lifecycle, KEPs (Kubernetes Enhancement Proposals), mentoring, security, and more.

2019 Steering Committee Election Results

Thu, 03 Oct 2019 00:00:00 +0000

The 2019 Steering Committee Election is a landmark milestone for the Kubernetes project. The initial bootstrap committee is graduating to emeritus and the committee has now shrunk to its final allocation of seven seats. All members of the Steering Committee are now fully elected by the Kubernetes Community.

Moving forward elections will elect either 3 or 4 people to the committee for two-year terms.

Results

The Kubernetes Steering Committee Election is now complete and the following candidates came ahead to secure two-year terms that start immediately (in alphabetical order by GitHub handle):

Contributor Summit San Diego Registration Open!

Tue, 24 Sep 2019 00:00:00 +0000

Contributor Summit San Diego 2019 Event Page
In record time, we’ve hit capacity for the new contributor workshop session of the event!

Sunday, November 17
Evening Contributor Celebration:
QuartYard†
Address: 1301 Market Street, San Diego, CA 92101
Time: 6:00PM - 9:00PM

Monday, November 18
All Day Contributor Summit:
Marriott Marquis San Diego Marina
Address: 333 W Harbor Dr, San Diego, CA 92101
Time: 9:00AM - 5:00PM

While the Kubernetes project is only five years old, we’re already going into our 9th Contributor Summit this November in San Diego before KubeCon + CloudNativeCon. The rapid increase is thanks to adding European and Asian Contributor Summits to the North American events we’ve done previously. We will continue to run Contributor Summits across the globe, as it is important that our contributor base grows in all forms of diversity.

Kubernetes 1.16: Custom Resources, Overhauled Metrics, and Volume Extensions

Wed, 18 Sep 2019 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.16, our third release of 2019! Kubernetes 1.16 consists of 31 enhancements: 8 enhancements moving to stable, 8 enhancements in beta, and 15 enhancements in alpha.

Major Themes

Custom resources

CRDs are in widespread use as a Kubernetes extensibility mechanism and have been available in beta since the 1.7 release. The 1.16 release marks the graduation of CRDs to general availability (GA).

Announcing etcd 3.4

Fri, 30 Aug 2019 00:00:00 +0000

etcd 3.4 focuses on stability, performance and ease of operation, with features like pre-vote and non-voting member and improvements to storage backend and client balancer.

Please see CHANGELOG for full lists of changes.

Better Storage Backend

etcd v3.4 includes a number of performance improvements for large scale Kubernetes workloads.

In particular, etcd experienced performance issues with a large number of concurrent read transactions even when there is no write (e.g. “read-only range request ... took too long to execute”). Previously, the storage backend commit operation on pending writes blocks incoming read transactions, even when there was no pending write. Now, the commit does not block reads which improve long-running read transaction performance.

OPA Gatekeeper: Policy and Governance for Kubernetes

Tue, 06 Aug 2019 00:00:00 +0000

The Open Policy Agent Gatekeeper project can be leveraged to help enforce policies and strengthen governance in your Kubernetes environment. In this post, we will walk through the goals, history, and current state of the project.

The following recordings from the Kubecon EU 2019 sessions are a great starting place in working with Gatekeeper:

Motivations

If your organization has been operating Kubernetes, you probably have been looking for ways to control what end-users can do on the cluster and ways to ensure that clusters are in compliance with company policies. These policies may be there to meet governance and legal requirements or to enforce best practices and organizational conventions. With Kubernetes, how do you ensure compliance without sacrificing development agility and operational independence?

Get started with Kubernetes (using Python)

Tue, 23 Jul 2019 00:00:00 +0000

So, you know you want to run your application in Kubernetes but don’t know where to start. Or maybe you’re getting started but still don’t know what you don’t know. In this blog you’ll walk through how to containerize an application and get it running in Kubernetes.

This walk-through assumes you are a developer or at least comfortable with the command line (preferably bash shell).

What we’ll do

Get the code and run the application locally
Create an image and run the application in Docker
Create a deployment and run the application in Kubernetes

Prerequisites

A Kubernetes service - I'm using Docker Desktop with Kubernetes in this walkthrough, but you can use one of the others. See Getting Started for a full listing.
Python 3.7 installed
Git installed

Containerizing an application

In this section you’ll take some source code, verify it runs locally, and then create a Docker image of the application. The sample application used is a very simple Flask web application; if you want to test it locally, you’ll need Python installed. Otherwise, you can skip to the "Create a Dockerfile" section.

Deprecated APIs Removed In 1.16: Here’s What You Need To Know

Thu, 18 Jul 2019 00:00:00 +0000

As the Kubernetes API evolves, APIs are periodically reorganized or upgraded. When APIs evolve, the old API is deprecated and eventually removed.

The v1.16 release will stop serving the following deprecated API versions in favor of newer and more stable API versions:

NetworkPolicy in the extensions/v1beta1 API version is no longer served
- Migrate to use the networking.k8s.io/v1 API version, available since v1.8. Existing persisted data can be retrieved/updated via the new version.
PodSecurityPolicy in the extensions/v1beta1 API version
- Migrate to use the policy/v1beta1 API, available since v1.10. Existing persisted data can be retrieved/updated via the new version.
DaemonSet in the extensions/v1beta1 and apps/v1beta2 API versions is no longer served
- Migrate to use the apps/v1 API version, available since v1.9. Existing persisted data can be retrieved/updated via the new version.
- Notable changes:
  - spec.templateGeneration is removed
  - spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  - spec.updateStrategy.type now defaults to RollingUpdate (the default in extensions/v1beta1 was OnDelete)
Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served
- Migrate to use the apps/v1 API version, available since v1.9. Existing persisted data can be retrieved/updated via the new version.
- Notable changes:
  - spec.rollbackTo is removed
  - spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  - spec.progressDeadlineSeconds now defaults to 600 seconds (the default in extensions/v1beta1 was no deadline)
  - spec.revisionHistoryLimit now defaults to 10 (the default in apps/v1beta1 was 2, the default in extensions/v1beta1 was to retain all)
  - maxSurge and maxUnavailable now default to 25% (the default in extensions/v1beta1 was 1)
StatefulSet in the apps/v1beta1 and apps/v1beta2 API versions is no longer served
- Migrate to use the apps/v1 API version, available since v1.9. Existing persisted data can be retrieved/updated via the new version.
- Notable changes:
  - spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
  - spec.updateStrategy.type now defaults to RollingUpdate (the default in apps/v1beta1 was OnDelete)
ReplicaSet in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served
- Migrate to use the apps/v1 API version, available since v1.9. Existing persisted data can be retrieved/updated via the new version.
- Notable changes:
  - spec.selector is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades

The v1.22 release will stop serving the following deprecated API versions in favor of newer and more stable API versions:

Recap of Kubernetes Contributor Summit Barcelona 2019

Tue, 25 Jun 2019 00:00:00 +0000

First of all, THANK YOU to everyone who made the Kubernetes Contributor Summit in Barcelona possible. We had an amazing team of volunteers tasked with planning and executing the event, and it was so much fun meeting and talking to all new and current contributors during the main event and the pre-event celebration.

Contributor Summit in Barcelona kicked off KubeCon + CloudNativeCon in a big way as it was the largest contributor summit to date with 331 people signed up, and only 9 didn't pick up their badges!

Automated High Availability in kubeadm v1.15: Batteries Included But Swappable

Mon, 24 Jun 2019 00:00:00 +0000

At the time of publication, Lucas Käldström was writing as SIG Cluster Lifecycle co-chair and as a subproject owner for kubeadm; Fabrizio Pandini was writing as a subproject owner for kubeadm.

kubeadm is a tool that enables Kubernetes administrators to quickly and easily bootstrap minimum viable clusters that are fully compliant with Certified Kubernetes guidelines. It’s been under active development by SIG Cluster Lifecycle since 2016 and graduated it from beta to generally available (GA) at the end of 2018.

Introducing Volume Cloning Alpha for Kubernetes

Fri, 21 Jun 2019 00:00:00 +0000

Kubernetes v1.15 introduces alpha support for volume cloning. This feature allows you to create new volumes using the contents of existing volumes in the user's namespace using the Kubernetes API.

What is a Clone?

Many storage systems provide the ability to create a "clone" of a volume. A clone is a duplicate of an existing volume that is its own unique volume on the system, but the data on the source is duplicated to the destination (clone). A clone is similar to a snapshot in that it's a point in time copy of a volume, however rather than creating a new snapshot object from a volume, we're instead creating a new independent volume, sometimes thought of as pre-populating the newly created volume.

Future of CRDs: Structural Schemas

Thu, 20 Jun 2019 00:00:00 +0000

CustomResourceDefinitions were introduced roughly two years ago as the primary way to extend the Kubernetes API with custom resources. From the beginning they stored arbitrary JSON data, with the exception that kind, apiVersion and metadata had to follow the Kubernetes API conventions. In Kubernetes 1.8 CRDs gained the ability to define an optional OpenAPI v3 based validation schema.

By the nature of OpenAPI specifications though—only describing what must be there, not what shouldn’t, and by being potentially incomplete specifications—the Kubernetes API server never knew the complete structure of CustomResource instances. As a consequence, kube-apiserver—until today—stores all JSON data received in an API request (if it validates against the OpenAPI spec). This especially includes anything that is not specified in the OpenAPI schema.

Kubernetes 1.15: Extensibility and Continuous Improvement

Wed, 19 Jun 2019 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.15, our second release of 2019! Kubernetes 1.15 consists of 25 enhancements: 2 moving to stable, 13 in beta, and 10 in alpha. The main themes of this release are:

Continuous Improvement
- Project sustainability is not just about features. Many SIGs have been working on improving test coverage, ensuring the basics stay reliable, and stability of the core feature set and working on maturing existing features and cleaning up the backlog.
Extensibility
- The community has been asking for continuing support of extensibility, so this cycle features more work around CRDs and API Machinery. Most of the enhancements in this cycle were from SIG API Machinery and related areas.

Let’s dive into the key features of this release:

Join us at the Contributor Summit in Shanghai

Wed, 12 Jun 2019 00:00:00 +0000

![Picture of contributor panel at 2018 Shanghai contributor summit. Photo by Josh Berkus, licensed CC-BY 4.0](/images/blog/2019- 06-11-contributor-summit-shanghai/panel.png)

For the second year, we will have a Contributor Summit event the day before KubeCon China in Shanghai. If you already contribute to Kubernetes or would like to contribute, please consider attending and register. The Summit will be held June 24th, at the Shanghai Expo Center (the same location where KubeCon will take place), and will include a Current Contributor Day as well as the New Contributor Workshop and the Documentation Sprints.

Kyma - extend and build on Kubernetes with ease

Thu, 23 May 2019 00:00:00 +0000

According to this recently completed CNCF Survey, the adoption rate of Cloud Native technologies in production is growing rapidly. Kubernetes is at the heart of this technological revolution. Naturally, the growth of cloud native technologies has been accompanied by the growth of the ecosystem that surrounds it. Of course, the complexity of cloud native technologies have increased as well. Just google for the phrase “Kubernetes is hard”, and you’ll get plenty of articles that explain this complexity problem. The best thing about the CNCF community is that problems like this can be solved by smart people building new tools to enable Kubernetes users: Projects like Knative and its Build resource extension, for example, serve to reduce complexity across a range of scenarios. Even though increasing complexity might seem like the most important issue to tackle, it is not the only challenge you face when transitioning to Cloud Native.

Kubernetes, Cloud Native, and the Future of Software

Fri, 17 May 2019 00:00:00 +0000

Kubernetes, Cloud Native, and the Future of Software

Five years ago this June, Google Cloud announced a new application management technology called Kubernetes. It began with a simple open source commit, followed the next day by a one-paragraph blog mention around container support. Later in the week, Eric Brewer talked about Kubernetes for the first time at DockerCon. And soon the world was watching.

We’re delighted to see Kubernetes become core to the creation and operation of modern software, and thereby a key part of the global economy. To us, the success of Kubernetes represents even more: A business transition with truly worldwide implications, thanks to the unprecedented cooperation afforded by the open source software movement.

Expanding our Contributor Workshops

Tue, 14 May 2019 00:00:00 +0000

tl;dr - learn about the contributor community with us and land your first PR! We have spots available in Barcelona (registration closes on Wednesday May 15, so grab your spot!) and the upcoming Shanghai Summit. The Barcelona event is poised to be our biggest one yet, with more registered attendees than ever before!

Have you always wanted to contribute to Kubernetes, but not sure where to begin? Have you seen our community’s many code bases and seen places to improve? We have a workshop for you!

Cat shirts and Groundhog Day: the Kubernetes 1.14 release interview

Mon, 13 May 2019 00:00:00 +0000

Last week we celebrated one year of the Kubernetes Podcast from Google. In this weekly show, my co-host Adam Glick and I focus on all the great things that are happening in the world of Kubernetes and Cloud Native. From the news of the week, to interviews with people in the community, we help you stay up to date on everything Kubernetes.

Every few cycles we check in on the release process for Kubernetes itself. Last year we interviewed the release managers for Kubernetes 1.11, and shared that transcript on the Kubernetes blog. We got such great feedback that we wanted to share the transcript of our recent conversation with Aaron Crickenberger, the release manager for Kubernetes 1.14.

Join us for the 2019 KubeCon Diversity Lunch & Hack

Thu, 02 May 2019 00:00:00 +0000

Join us for the 2019 KubeCon Diversity Lunch & Hack: Building Tech Skills & An Inclusive Community - Sponsored by Google Cloud and VMware

Registration for the Diversity Lunch opens today, May 2nd, 2019. To register, go to the main KubeCon + CloudNativeCon EU schedule, then log in to your Sched account, and confirm your attendance to the Diversity Lunch. Please sign up ASAP once the link is live, as spaces will fill quickly. We filled the event in just a few days last year, and anticipate doing so again this year.

How You Can Help Localize Kubernetes Docs

Fri, 26 Apr 2019 00:00:00 +0000

Last year we optimized the Kubernetes website for hosting multilingual content. Contributors responded by adding multiple new localizations: as of April 2019, Kubernetes docs are partially available in nine different languages, with six added in 2019 alone. You can see a list of available languages in the language selector at the top of each page.

By partially available, I mean that localizations are ongoing projects. They range from mostly complete (Chinese docs for 1.12) to brand new (1.14 docs in Portuguese). If you're interested in helping an existing localization, read on!

Hardware Accelerated SSL/TLS Termination in Ingress Controllers using Kubernetes Device Plugins and RuntimeClass

Wed, 24 Apr 2019 00:00:00 +0000

Abstract

A Kubernetes Ingress is a way to connect cluster services to the world outside the cluster. In order to correctly route the traffic to service backends, the cluster needs an Ingress controller. The Ingress controller is responsible for setting the right destinations to backends based on the Ingress API objects’ information. The actual traffic is routed through a proxy server that is responsible for tasks such as load balancing and SSL/TLS (later “SSL” refers to both SSL or TLS ) termination. The SSL termination is a CPU heavy operation due to the crypto operations involved. To offload some of the CPU intensive work away from the CPU, OpenSSL based proxy servers can take the benefit of OpenSSL Engine API and dedicated crypto hardware. This frees CPU cycles for other things and improves the overall throughput of the proxy server.

Introducing kube-iptables-tailer: Better Networking Visibility in Kubernetes Clusters

Fri, 19 Apr 2019 00:00:00 +0000

At Box, we use Kubernetes to empower our engineers to own the whole lifecycle of their microservices. When it comes to networking, our engineers use Tigera’s Project Calico to declaratively manage network policies for their apps running in our Kubernetes clusters. App owners define a Calico policy in order to enable their Pods to send/receive network traffic, which is instantiated as iptables rules.

There may be times, however, when such network policy is missing or declared incorrectly by app owners. In this situation, the iptables rules will cause network packet drops between the affected Pods, which get logged in a file that is inaccessible to app owners. We needed a mechanism to seamlessly deliver alerts about those iptables packet drops based on their network policies to help app owners quickly diagnose the corresponding issues. To solve this, we developed a service called kube-iptables-tailer to detect packet drops from iptables logs and report them as Kubernetes events. We are proud to open-source kube-iptables-tailer for you to utilize in your own cluster, regardless of whether you use Calico or other network policy tools.

The Future of Cloud Providers in Kubernetes

Wed, 17 Apr 2019 00:00:00 +0000

Approximately 9 months ago, the Kubernetes community agreed to form the Cloud Provider Special Interest Group (SIG). The justification was to have a single governing SIG to own and shape the integration points between Kubernetes and the many cloud providers it supported. A lot has been in motion since then and we’re here to share with you what has been accomplished so far and what we hope to see in the future.

Pod Priority and Preemption in Kubernetes

Tue, 16 Apr 2019 00:00:00 +0000

Kubernetes is well-known for running scalable workloads. It scales your workloads based on their resource usage. When a workload is scaled up, more instances of the application get created. When the application is critical for your product, you want to make sure that these new instances are scheduled even when your cluster is under resource pressure. One obvious solution to this problem is to over-provision your cluster resources to have some amount of slack resources available for scale-up situations. This approach often works, but costs more as you would have to pay for the resources that are idle most of the time.

Process ID Limiting for Stability Improvements in Kubernetes 1.14

Mon, 15 Apr 2019 00:00:00 +0000

Have you ever seen someone take more than their fair share of the cookies? The one person who reaches in and grabs a half dozen fresh baked chocolate chip chunk morsels and skitters off like Cookie Monster exclaiming “Om nom nom nom.”

In some rare workloads, a similar occurrence was taking place inside Kubernetes clusters. With each Pod and Node, there comes a finite number of possible process IDs (PIDs) for all applications to share. While it is rare for any one process or pod to reach in and grab all the PIDs, some users were experiencing resource starvation due to this type of behavior. So in Kubernetes 1.14, we introduced an enhancement to mitigate the risk of a single pod monopolizing all of the PIDs available.

Kubernetes 1.14: Local Persistent Volumes GA

Thu, 04 Apr 2019 00:00:00 +0000

The Local Persistent Volumes feature has been promoted to GA in Kubernetes 1.14. It was first introduced as alpha in Kubernetes 1.7, and then beta in Kubernetes 1.10. The GA milestone indicates that Kubernetes users may depend on the feature and its API for production use. GA features are protected by the Kubernetes deprecation policy.

What is a Local Persistent Volume?

A local persistent volume represents a local disk directly-attached to a single Kubernetes Node.

Kubernetes v1.14 delivers production-level support for Windows nodes and Windows containers

Mon, 01 Apr 2019 00:00:00 +0000

The first release of Kubernetes in 2019 brings a highly anticipated feature - production-level support for Windows workloads. Up until now Windows node support in Kubernetes has been in beta, allowing many users to experiment and see the value of Kubernetes for Windows containers. While in beta, developers in the Kubernetes community and Windows Server team worked together to improve the container runtime, build a continuous testing process, and complete features needed for a good user experience. Kubernetes now officially supports adding Windows nodes as worker nodes and scheduling Windows containers, enabling a vast ecosystem of Windows applications to leverage the power of our platform.

kube-proxy Subtleties: Debugging an Intermittent Connection Reset

Fri, 29 Mar 2019 00:00:00 +0000

I recently came across a bug that causes intermittent connection resets. After some digging, I found it was caused by a subtle combination of several different network subsystems. It helped me understand Kubernetes networking better, and I think it’s worthwhile to share with a wider audience who are interested in the same topic.

The symptom

We received a user report claiming they were getting connection resets while using a Kubernetes service of type ClusterIP to serve large files to pods running in the same cluster. Initial debugging of the cluster did not yield anything interesting: network connectivity was fine and downloading the files did not hit any issues. However, when we ran the workload in parallel across many clients, we were able to reproduce the problem. Adding to the mystery was the fact that the problem could not be reproduced when the workload was run using VMs without Kubernetes. The problem, which could be easily reproduced by a simple app, clearly has something to do with Kubernetes networking, but what?

Running Kubernetes locally on Linux with Minikube - now with Kubernetes 1.14 support

Thu, 28 Mar 2019 00:00:00 +0000

A few days ago, the Kubernetes community announced Kubernetes 1.14, the most recent version of Kubernetes. Alongside it, Minikube, a part of the Kubernetes project, recently hit the 1.0 milestone, which supports Kubernetes 1.14 by default.

Kubernetes is a real winner (and a de facto standard) in the world of distributed Cloud Native computing. While it can handle up to 5000 nodes in a single cluster, local deployment on a single machine (e.g. a laptop, a developer workstation, etc.) is an increasingly common scenario for using Kubernetes.

Kubernetes 1.14: Production-level support for Windows Nodes, Kubectl Updates, Persistent Local Volumes GA

Mon, 25 Mar 2019 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.14, our first release of 2019!

Kubernetes 1.14 consists of 31 enhancements: 10 moving to stable, 12 in beta, and 7 net new. The main themes of this release are extensibility and supporting more workloads on Kubernetes with three major features moving to general availability, and an important security feature moving to beta.

More enhancements graduated to stable in this release than any prior Kubernetes release. This represents an important milestone for users and operators in terms of setting support expectations. In addition, there are notable Pod and RBAC enhancements in this release, which are discussed in the “additional notable features” section below.

Kubernetes End-to-end Testing for Everyone

Fri, 22 Mar 2019 00:00:00 +0000

More and more components that used to be part of Kubernetes are now being developed outside of Kubernetes. For example, storage drivers used to be compiled into Kubernetes binaries, then were moved into stand-alone FlexVolume binaries on the host, and now are delivered as Container Storage Interface (CSI) drivers that get deployed in pods inside the Kubernetes cluster itself.

This poses a challenge for developers who work on such components: how can end-to-end (E2E) testing on a Kubernetes cluster be done for such external components? The E2E framework that is used for testing Kubernetes itself has all the necessary functionality. However, trying to use it outside of Kubernetes was difficult and only possible by carefully selecting the right versions of a large number of dependencies. E2E testing has become a lot simpler in Kubernetes 1.13.

A Guide to Kubernetes Admission Controllers

Thu, 21 Mar 2019 00:00:00 +0000

Kubernetes has greatly improved the speed and manageability of backend clusters in production today. Kubernetes has emerged as the de facto standard in container orchestrators thanks to its flexibility, scalability, and ease of use. Kubernetes also provides a range of features that secure production workloads. A more recent introduction in security features is a set of plugins called “admission controllers.” Admission controllers must be enabled to use some of the more advanced security features of Kubernetes, such as pod security policies that enforce a security configuration baseline across an entire namespace. The following must-know tips and tricks will help you leverage admission controllers to make the most of these security capabilities in Kubernetes.

A Look Back and What's in Store for Kubernetes Contributor Summits

Wed, 20 Mar 2019 00:00:00 +0000

Seattle Contributor Summit

As our contributing community grows in great numbers, with more than 16,000 contributors this year across 150+ GitHub repositories, it’s important to provide face to face connections for our large distributed teams to have opportunities for collaboration and learning. In Contributor Experience, our methodology with planning events is a lot like our documentation; we build from personas -- interests, skills, and motivators to name a few. This way we ensure there is valuable content and learning for everyone.

KubeEdge, a Kubernetes Native Edge Computing Framework

Tue, 19 Mar 2019 00:00:00 +0000

KubeEdge becomes the first Kubernetes Native Edge Computing Platform with both Edge and Cloud components open sourced!

Open source edge computing is going through its most dynamic phase of development in the industry. So many open source platforms, so many consolidations and so many initiatives for standardization! This shows the strong drive to build better platforms to bring cloud computing to the edges to meet ever increasing demand. KubeEdge, which was announced last year, now brings great news for cloud native computing! It provides a complete edge computing solution based on Kubernetes with separate cloud and edge core modules. Currently, both the cloud and edge modules are open sourced.

Kubernetes Setup Using Ansible and Vagrant

Fri, 15 Mar 2019 00:00:00 +0000

Objective

This blog post describes the steps required to setup a multi node Kubernetes cluster for development purposes. This setup provides a production-like cluster that can be setup on your local machine.

Why do we require multi node cluster setup?

Multi node Kubernetes clusters offer a production-like environment which has various advantages. Even though Minikube provides an excellent platform for getting started, it doesn't provide the opportunity to work with multi node clusters which can help solve problems or bugs that are related to application design and architecture. For instance, Ops can reproduce an issue in a multi node cluster environment, Testers can deploy multiple versions of an application for executing test cases and verifying changes. These benefits enable teams to resolve issues faster which make the more agile.

Raw Block Volume support to Beta

Thu, 07 Mar 2019 00:00:00 +0000

Kubernetes v1.13 moves raw block volume support to beta. This feature allows persistent volumes to be exposed inside containers as a block device instead of as a mounted file system.

What are block devices?

Block devices enable random access to data in fixed-size blocks. Hard drives, SSDs, and CD-ROMs drives are all examples of block devices.

Typically persistent storage is implemented in a layered maner with a file system (like ext4) on top of a block device (like a spinning disk or SSD). Applications then read and write files instead of operating on blocks. The operating systems take care of reading and writing files, using the specified filesystem, to the underlying device as blocks.

Automate Operations on your Cluster with OperatorHub.io

Thu, 28 Feb 2019 00:00:00 +0000

One of the important challenges facing developers and Kubernetes administrators has been a lack of ability to quickly find common services that are operationally ready for Kubernetes. Typically, the presence of an Operator for a specific service - a pattern that was introduced in 2016 and has gained momentum - is a good signal for the operational readiness of the service on Kubernetes. However, there has to date not existed a registry of Operators to simplify the discovery of such services.

Building a Kubernetes Edge (Ingress) Control Plane for Envoy v2

Tue, 12 Feb 2019 00:00:00 +0000

Kubernetes has become the de facto runtime for container-based microservice applications, but this orchestration framework alone does not provide all of the infrastructure necessary for running a distributed system. Microservices typically communicate through Layer 7 protocols such as HTTP, gRPC, or WebSockets, and therefore having the ability to make routing decisions, manipulate protocol metadata, and observe at this layer is vital. However, traditional load balancers and edge proxies have predominantly focused on L3/4 traffic. This is where the Envoy Proxy comes into play.

Runc and CVE-2019-5736

Mon, 11 Feb 2019 00:00:00 +0000

This morning a container escape vulnerability in runc was announced. We wanted to provide some guidance to Kubernetes users to ensure everyone is safe and secure.

What is runc?

Very briefly, runc is the low-level tool which does the heavy lifting of spawning a Linux container. Other tools like Docker, Containerd, and CRI-O sit on top of runc to deal with things like data formatting and serialization, but runc is at the heart of all of these systems.

Poseidon-Firmament Scheduler – Flow Network Graph Based Scheduler

Wed, 06 Feb 2019 00:00:00 +0000

Introduction

Cluster Management systems such as Mesos, Google Borg, Kubernetes etc. in a cloud scale datacenter environment (also termed as Datacenter-as-a-Computer or Warehouse-Scale Computing - WSC) typically manage application workloads by performing tasks such as tracking machine live-ness, starting, monitoring, terminating workloads and more importantly using a Cluster Scheduler to decide on workload placements.

A Cluster Scheduler essentially performs the scheduling of workloads to compute resources – combining the global placement of work across the WSC environment makes the “warehouse-scale computer” more efficient, increases utilization, and saves energy. Cluster Scheduler examples are Google Borg, Kubernetes, Firmament, Mesos, Tarcil, Quasar, Quincy, Swarm, YARN, Nomad, Sparrow, Apollo etc.

Update on Volume Snapshot Alpha for Kubernetes

Thu, 17 Jan 2019 00:00:00 +0000

Volume snapshotting support was introduced in Kubernetes v1.12 as an alpha feature. In Kubernetes v1.13, it remains an alpha feature, but a few enhancements were added and some breaking changes were made. This post summarizes the changes.

Breaking Changes

CSI spec v1.0 introduced a few breaking changes to the volume snapshot feature. CSI driver maintainers should be aware of these changes as they upgrade their drivers to support v1.0.

SnapshotStatus replaced with Boolean ReadyToUse

CSI v0.3.0, defined a SnapshotStatus enum in CreateSnapshotResponse which indicates whether the snapshot is READY, UPLOADING, or ERROR_UPLOADING. In CSI v1.0, SnapshotStatus has been removed from CreateSnapshotResponse and replaced with a boolean ReadyToUse. A ReadyToUse value of true indicates that post snapshot processing (such as uploading) is complete and the snapshot is ready to be used as a source to create a volume.

Container Storage Interface (CSI) for Kubernetes GA

Tue, 15 Jan 2019 00:00:00 +0000

The Kubernetes implementation of the Container Storage Interface (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was introduced as alpha in Kubernetes v1.9 release, and promoted to beta in the Kubernetes v1.10 release.

The GA milestone indicates that Kubernetes users may depend on the feature and its API without fear of backwards incompatible changes in future causing regressions. GA features are protected by the Kubernetes deprecation policy.

APIServer dry-run and kubectl diff

Mon, 14 Jan 2019 00:00:00 +0000

Declarative configuration management, also known as configuration-as-code, is one of the key strengths of Kubernetes. It allows users to commit the desired state of the cluster, and to keep track of the different versions, improve auditing and automation through CI/CD pipelines. The Apply working-group is working on fixing some of the gaps, and is happy to announce that Kubernetes 1.13 promoted server-side dry-run and kubectl diff to beta. These two features are big improvements for the Kubernetes declarative model.

Kubernetes Federation Evolution

Wed, 12 Dec 2018 00:00:00 +0000

Kubernetes provides great primitives for deploying applications to a cluster: it can be as simple as kubectl create -f app.yaml. Deploy apps across multiple clusters has never been that simple. How should app workloads be distributed? Should the app resources be replicated into all clusters, replicated into selected clusters, or partitioned into clusters? How is access to the clusters managed? What happens if some of the resources that a user wants to distribute pre-exist, in some or all of the clusters, in some form?

etcd: Current status and future roadmap

Tue, 11 Dec 2018 00:00:00 +0000

etcd is a distributed key value store that provides a reliable way to manage the coordination state of distributed systems. etcd was first announced in June 2013 by CoreOS (part of Red Hat as of 2018). Since its adoption in Kubernetes in 2014, etcd has become a fundamental part of the Kubernetes cluster management software design, and the etcd community has grown exponentially. etcd is now being used in production environments of multiple companies, including large cloud provider environments such as AWS, Google Cloud Platform, Azure, and other on-premises Kubernetes implementations. CNCF currently has 32 conformant Kubernetes platforms and distributions, all of which use etcd as the datastore.

New Contributor Workshop Shanghai

Wed, 05 Dec 2018 00:00:00 +0000

Kubecon Shanghai New Contributor Summit attendees. Photo by Jerry Zhang

We recently completed our first New Contributor Summit in China, at the first KubeCon in China. It was very exciting to see all of the Chinese and Asian developers (plus a few folks from around the world) interested in becoming contributors. Over the course of a long day, they learned how, why, and where to contribute to Kubernetes, created pull requests, attended a panel of current contributors, and got their CLAs signed.

Production-Ready Kubernetes Cluster Creation with kubeadm

Tue, 04 Dec 2018 00:00:00 +0000

kubeadm is a tool that enables Kubernetes administrators to quickly and easily bootstrap minimum viable clusters that are fully compliant with Certified Kubernetes guidelines. It's been under active development by SIG Cluster Lifecycle since 2016 and we're excited to announce that it has now graduated from beta to stable and generally available (GA)!

This GA release of kubeadm is an important event in the progression of the Kubernetes ecosystem, bringing stability to an area where stability is paramount.

Kubernetes 1.13: Simplified Cluster Management with Kubeadm, Container Storage Interface (CSI), and CoreDNS as Default DNS are Now Generally Available

Mon, 03 Dec 2018 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.13, our fourth and final release of 2018!

Kubernetes 1.13 has been one of the shortest releases to date at 10 weeks. This release continues to focus on stability and extensibility of Kubernetes with three major features graduating to general availability this cycle in the areas of Storage and Cluster Lifecycle. Notable features graduating in this release include: simplified cluster management with kubeadm, Container Storage Interface (CSI), and CoreDNS as the default DNS.

Kubernetes Docs Updates, International Edition

Thu, 08 Nov 2018 00:00:00 +0000

As a co-chair of SIG Docs, I'm excited to share that Kubernetes docs have a fully mature workflow for localization (l10n).

Abbreviations galore

L10n is an abbreviation for localization.

I18n is an abbreviation for internationalization.

I18n is what you do to make l10n easier. L10n is a fuller, more comprehensive process than translation (t9n).

Why localization matters

The goal of SIG Docs is to make Kubernetes easier to use for as many people as possible.

gRPC Load Balancing on Kubernetes without Tears

Wed, 07 Nov 2018 00:00:00 +0000

Many new gRPC users are surprised to find that Kubernetes's default load balancing often doesn't work out of the box with gRPC. For example, here's what happens when you take a simple gRPC Node.js microservices app and deploy it on Kubernetes:

While the voting service displayed here has several pods, it's clear from Kubernetes's CPU graphs that only one of the pods is actually doing any work—because only one of the pods is receiving any traffic. Why?

Tips for Your First Kubecon Presentation - Part 2

Fri, 26 Oct 2018 00:00:00 +0000

Hello and welcome back to the second and final part about tips for KubeCon first-time speakers. If you missed the last post, please give it a read here.

The Day before the Show

Tip #13 - Get enough sleep. I don't know about you, but when I don't get enough sleep (especially when beer is in the game), the next day my brain power is around 80% at best. It's very easy to get distracted at KubeCon (in a positive sense). "Let's have dinner tonight and chat about XYZ". Get some food, beer or wine because you're so excited and all the good resolutions you had set for the day before your presentation are forgotten :)

Tips for Your First Kubecon Presentation - Part 1

Thu, 18 Oct 2018 00:00:00 +0000

First of all, let me congratulate you to this outstanding achievement. Speaking at KubeCon, especially if it's your first time, is a tremendous honor and experience. Well done!

Congrats to everyone who got Kubecon talks accepted! 👏👏👏

To everyone who got a rejection don't feel bad. Only 13% could be accepted. Keep trying. There will be other opportunities.
— Justin Garrison (@rothgar) September 24, 2018

When I was informed that my KubeCon talk about Kubernetes Resource Management was accepted for KubeCon EU in Denmark (2018), I really could not believe it. By then, the chances to get your talk accepted were around 10% (or less, don't really remember the exact number). There were over a 1,000 submissions just for that KubeCon (recall that we now have three KubeCon events during the year - US, EU and Asia region). The popularity of Kubernetes is ever increasing and so is the number of people trying to get a talk accepted. Once again, outstanding achievement to get your talk in!

Kubernetes 2018 North American Contributor Summit

Tue, 16 Oct 2018 00:00:00 +0000

The 2018 North American Kubernetes Contributor Summit to be hosted right before KubeCon + CloudNativeCon Seattle is shaping up to be the largest yet. It is an event that brings together new and current contributors alike to connect and share face-to-face; and serves as an opportunity for existing contributors to help shape the future of community development. For new community members, it offers a welcoming space to learn, explore and put the contributor workflow to practice.

2018 Steering Committee Election Results

Mon, 15 Oct 2018 00:00:00 +0000

Results

The Kubernetes Steering Committee Election is now complete and the following candidates came ahead to secure two year terms that start immediately:

Aaron Crickenberger, Google, @spiffxp
Davanum Srinivas, Huawei, @dims
Tim St. Clair, Heptio, @timothysc

Big Thanks!

Steering Committee Member Emeritus Quinton Hoole for his service to the community over the past year. We look forward to
The candidates that came forward to run for election. May we always have a strong set of people who want to push community forward like yours in every election.
All 307 voters who cast a ballot.
And last but not least...Cornell University for hosting CIVS!

Get Involved with the Steering Committee

You can follow along to Steering Committee backlog items and weigh in by filing an issue or creating a PR against their repo. They meet bi-weekly on Wednesdays at 8pm UTC and regularly attend Meet Our Contributors.

Topology-Aware Volume Provisioning in Kubernetes

Thu, 11 Oct 2018 00:00:00 +0000

The multi-zone cluster experience with persistent volumes is improving in Kubernetes 1.12 with the topology-aware dynamic provisioning beta feature. This feature allows Kubernetes to make intelligent decisions when dynamically provisioning volumes by getting scheduler input on the best place to provision a volume for a pod. In multi-zone clusters, this means that volumes will get provisioned in an appropriate zone that can run your pod, allowing you to easily deploy and scale your stateful workloads across failure domains to provide high availability and fault tolerance.

Kubernetes v1.12: Introducing RuntimeClass

Wed, 10 Oct 2018 00:00:00 +0000

Kubernetes originally launched with support for Docker containers running native applications on a Linux host. Starting with rkt in Kubernetes 1.3 more runtimes were coming, which lead to the development of the Container Runtime Interface (CRI). Since then, the set of alternative runtimes has only expanded: projects like Kata Containers and gVisor were announced for stronger workload isolation, and Kubernetes' Windows support has been steadily progressing.

With runtimes targeting so many different use cases, a clear need for mixed runtimes in a cluster arose. But all these different ways of running containers have brought a new set of problems to deal with:

Introducing Volume Snapshot Alpha for Kubernetes

Tue, 09 Oct 2018 00:00:00 +0000

Kubernetes v1.12 introduces alpha support for volume snapshotting. This feature allows creating/deleting volume snapshots, and the ability to create new volumes from a snapshot natively using the Kubernetes API.

What is a Snapshot?

Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage, and many on-premise storage systems) provide the ability to create a "snapshot" of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore the existing volume to a previous state (represented by the snapshot).

Support for Azure VMSS, Cluster-Autoscaler and User Assigned Identity

Mon, 08 Oct 2018 00:00:00 +0000

Introduction

With Kubernetes v1.12, Azure virtual machine scale sets (VMSS) and cluster-autoscaler have reached their General Availability (GA) and User Assigned Identity is available as a preview feature.

Azure VMSS allow you to create and manage identical, load balanced VMs that automatically increase or decrease based on demand or a set schedule. This enables you to easily manage and scale multiple VMs to provide high availability and application resiliency, ideal for large-scale applications like container workloads [1].

Introducing the Non-Code Contributor’s Guide

Thu, 04 Oct 2018 00:00:00 +0000

It was May 2018 in Copenhagen, and the Kubernetes community was enjoying the contributor summit at KubeCon/CloudNativeCon, complete with the first run of the New Contributor Workshop. As a time of tremendous collaboration between contributors, the topics covered ranged from signing the CLA to deep technical conversations. Along with the vast exchange of information and ideas, however, came continued scrutiny of the topics at hand to ensure that the community was being as inclusive and accommodating as possible. Over that spring week, some of the pieces under the microscope included the many themes being covered, and how they were being presented, but also the overarching characteristics of the people contributing and the skill sets involved. From the discussions and analysis that followed grew the idea that the community was not benefiting as much as it could from the many people who wanted to contribute, but whose strengths were in areas other than writing code.

KubeDirector: The easy way to run complex stateful applications on Kubernetes

Wed, 03 Oct 2018 00:00:00 +0000

KubeDirector is an open source project designed to make it easy to run complex stateful scale-out application clusters on Kubernetes. KubeDirector is built using the custom resource definition (CRD) framework and leverages the native Kubernetes API extensions and design philosophy. This enables transparent integration with Kubernetes user/resource management as well as existing clients and tools.

We recently introduced the KubeDirector project, as part of a broader open source Kubernetes initiative we call BlueK8s. I’m happy to announce that the pre-alpha code for KubeDirector is now available. And in this blog post, I’ll show how it works.

Building a Network Bootable Server Farm for Kubernetes with LTSP

Tue, 02 Oct 2018 00:00:00 +0000

In this post, I'm going to introduce you to a cool technology for Kubernetes, LTSP. It is useful for large baremetal Kubernetes deployments.

You don't need to think about installing an OS and binaries on each node anymore. Why? You can do that automatically through Dockerfile!

You can buy and put 100 new servers into a production environment and get them working immediately - it's really amazing!

Intrigued? Let me walk you through how it works.

Health checking gRPC servers on Kubernetes

Mon, 01 Oct 2018 00:00:00 +0000

Update (December 2021): Kubernetes now has built-in gRPC health probes starting in v1.23. To learn more, see Configure Liveness, Readiness and Startup Probes. This article was originally written about an external tool to achieve the same task.

gRPC is on its way to becoming the lingua franca for communication between cloud-native microservices. If you are deploying gRPC applications to Kubernetes today, you may be wondering about the best way to configure health checks. In this article, we will talk about grpc-health-probe, a Kubernetes-native way to health check gRPC apps.

Kubernetes 1.12: Kubelet TLS Bootstrap and Azure Virtual Machine Scale Sets (VMSS) Move to General Availability

Thu, 27 Sep 2018 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.12, our third release of 2018!

Today’s release continues to focus on internal improvements and graduating features to stable in Kubernetes. This newest version graduates key features such as security and Azure. Notable additions in this release include two highly-anticipated features graduating to general availability: Kubelet TLS Bootstrap and Support for Azure Virtual Machine Scale Sets (VMSS).

These new features mean increased security, availability, resiliency, and ease of use to get production applications to market faster. The release also signifies the increasing maturation and sophistication of Kubernetes on the developer side.

Hands On With Linkerd 2.0

Tue, 18 Sep 2018 00:00:00 +0000

Linkerd 2.0 was recently announced as generally available (GA), signaling its readiness for production use. In this tutorial, we’ll walk you through how to get Linkerd 2.0 up and running on your Kubernetes cluster in a matter seconds.

But first, what is Linkerd and why should you care? Linkerd is a service sidecar that augments a Kubernetes service, providing zero-config dashboards and UNIX-style CLI tools for runtime debugging, diagnostics, and reliability. Linkerd is also a service mesh, applied to multiple (or all) services in a cluster to provide a uniform layer of telemetry, security, and control across them.

2018 Steering Committee Election Cycle Kicks Off

Thu, 06 Sep 2018 00:00:00 +0000

Having a clear, definable governance model is crucial for the health of open source projects. For one of the highest velocity projects in the open source world, governance is critical especially for one as large and active as Kubernetes, which is one of the most high-velocity projects in the open source world. A clear structure helps users trust that the project will be nurtured and progress forward. Initially, this structure was laid by the former 7 member bootstrap committee composed of founders and senior contributors with a goal to create the foundational governance building blocks.

The Machines Can Do the Work, a Story of Kubernetes Testing, CI, and Automating the Contributor Experience

Wed, 29 Aug 2018 00:00:00 +0000

“Large projects have a lot of less exciting, yet, hard work. We value time spent automating repetitive work more highly than toil. Where that work cannot be automated, it is our culture to recognize and reward all types of contributions. However, heroism is not sustainable.” - Kubernetes Community Values

Like many open source projects, Kubernetes is hosted on GitHub. We felt the barrier to participation would be lowest if the project lived where developers already worked, using tools and processes developers already knew. Thus the project embraced the service fully: it was the basis of our workflow, our issue tracker, our documentation, our blog platform, our team structure, and more.

Introducing Kubebuilder: an SDK for building Kubernetes APIs using CRDs

Fri, 10 Aug 2018 00:00:00 +0000

How can we enable applications such as MySQL, Spark and Cassandra to manage themselves just like Kubernetes Deployments and Pods do? How do we configure these applications as their own first class APIs instead of a collection of StatefulSets, Services, and ConfigMaps?

We have been working on a solution and are happy to introduce kubebuilder, a comprehensive development kit for rapidly building and publishing Kubernetes APIs and Controllers using CRDs. Kubebuilder scaffolds projects and API definitions and is built on top of the controller-runtime libraries.

Out of the Clouds onto the Ground: How to Make Kubernetes Production Grade Anywhere

Fri, 03 Aug 2018 00:00:00 +0000

This blog offers some guidelines for running a production grade Kubernetes cluster in an environment like an on-premise data center or edge location.

What does it mean to be “production grade”?

The installation is secure
The deployment is managed with a repeatable and recorded process
Performance is predictable and consistent
Updates and configuration changes can be safely applied
Logging and monitoring is in place to detect and diagnose failures and resource shortages
Service is “highly available enough” considering available resources, including constraints on money, physical space, power, etc.
A recovery process is available, documented, and tested for use in the event of failures

In short, production grade means anticipating accidents and preparing for recovery with minimal pain and delay.

Dynamically Expand Volume with CSI and Kubernetes

Thu, 02 Aug 2018 00:00:00 +0000

There is a very powerful storage subsystem within Kubernetes itself, covering a fairly broad spectrum of use cases. Whereas, when planning to build a product-grade relational database platform with Kubernetes, we face a big challenge: coming up with storage. This article describes how to extend latest Container Storage Interface 0.2.0 and integrate with Kubernetes, and demonstrates the essential facet of dynamically expanding volume capacity.

Introduction

As we focalize our customers, especially in financial space, there is a huge upswell in the adoption of container orchestration technology.

KubeVirt: Extending Kubernetes with CRDs for Virtualized Workloads

Fri, 27 Jul 2018 00:00:00 +0000

What is KubeVirt?

KubeVirt is a Kubernetes addon that provides users the ability to schedule traditional virtual machine workloads side by side with container workloads. Through the use of Custom Resource Definitions (CRDs) and other Kubernetes features, KubeVirt seamlessly extends existing Kubernetes clusters to provide a set of virtualization APIs that can be used to manage virtual machines.

Why Use CRDs Over an Aggregated API Server?

Back in the middle of 2017, those of us working on KubeVirt were at a crossroads. We had to make a decision whether or not to extend Kubernetes using an aggregated API server or to make use of the new Custom Resource Definitions (CRDs) feature.

Feature Highlight: CPU Manager

Tue, 24 Jul 2018 00:00:00 +0000

This blog post describes the CPU Manager, a beta feature in Kubernetes. The CPU manager feature enables better placement of workloads in the Kubelet, the Kubernetes node agent, by allocating exclusive CPUs to certain pod containers.

Sounds Good! But Does the CPU Manager Help Me?

It depends on your workload. A single compute node in a Kubernetes cluster can run many pods and some of these pods could be running CPU-intensive workloads. In such a scenario, the pods might contend for the CPU resources available in that compute node. When this contention intensifies, the workload can move to different CPUs depending on whether the pod is throttled and the availability of CPUs at scheduling time. There might also be cases where the workload could be sensitive to context switches. In all the above scenarios, the performance of the workload might be affected.

The History of Kubernetes & the Community Behind It

Fri, 20 Jul 2018 00:00:00 +0000

It is remarkable to me to return to Portland and OSCON to stand on stage with members of the Kubernetes community and accept this award for Most Impactful Open Source Project. It was scarcely three years ago, that on this very same stage we declared Kubernetes 1.0 and the project was added to the newly formed Cloud Native Computing Foundation.

To think about how far we have come in that short period of time and to see the ways in which this project has shaped the cloud computing landscape is nothing short of amazing. The success is a testament to the power and contributions of this amazing open source community. And the daily passion and quality contributions of our endlessly engaged, world-wide community is nothing short of humbling.

Kubernetes Wins the 2018 OSCON Most Impact Award

Thu, 19 Jul 2018 00:00:00 +0000

We are humbled to be recognized by the community with this award.

We had high hopes when we created Kubernetes. We wanted to change the way cloud applications were deployed and managed. Whether we’d succeed or not was very uncertain. And look how far we’ve come in such a short time.

The core technology behind Kubernetes was informed by lessons learned from Google’s internal infrastructure, but nobody can deny the enormous role of the Kubernetes community in the success of the project. The community, of which Google is a part, now drives every aspect of the project: the design, development, testing, documentation, releases, and more. That is what makes Kubernetes fly.

11 Ways (Not) to Get Hacked

Wed, 18 Jul 2018 00:00:00 +0000

Kubernetes security has come a long way since the project's inception, but still contains some gotchas. Starting with the control plane, building up through workload and network security, and finishing with a projection into the future of security, here is a list of handy tips to help harden your clusters and increase their resilience if compromised.

Part One: The Control Plane
Part Two: Workloads
Part Three: The Future
- 11. Run a Service Mesh
Conclusion

Part One: The Control Plane

The control plane is Kubernetes' brain. It has an overall view of every container and pod running on the cluster, can schedule new pods (which can include containers with root access to their parent node), and can read all the secrets stored in the cluster. This valuable cargo needs protecting from accidental leakage and malicious intent: when it's accessed, when it's at rest, and when it's being transported across the network.

How the sausage is made: the Kubernetes 1.11 release interview, from the Kubernetes Podcast

Mon, 16 Jul 2018 00:00:00 +0000

At KubeCon EU, my colleague Adam Glick and I were pleased to announce the Kubernetes Podcast from Google. In this weekly conversation, we focus on all the great things that are happening in the world of Kubernetes and Cloud Native. From the news of the week, to interviews with people in the community, we help you stay up to date on everything Kubernetes.

We recently had the pleasure of speaking to the release manager for Kubernetes 1.11, Josh Berkus from Red Hat, and the release manager for the upcoming 1.12, Tim Pepper from VMware.

Resizing Persistent Volumes using Kubernetes

Thu, 12 Jul 2018 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.11

In Kubernetes v1.11 the persistent volume expansion feature is being promoted to beta. This feature allows users to easily resize an existing volume by editing the PersistentVolumeClaim (PVC) object. Users no longer have to manually interact with the storage backend or delete and recreate PV and PVC objects to increase the size of a volume. Shrinking persistent volumes is not supported.

Dynamic Kubelet Configuration

Wed, 11 Jul 2018 00:00:00 +0000

Editor’s note: The feature has been removed in the version 1.24 after deprecation in 1.22.

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.11

Why Dynamic Kubelet Configuration?

Kubernetes provides API-centric tooling that significantly improves workflows for managing applications and infrastructure. Most Kubernetes installations, however, run the Kubelet as a native process on each host, outside the scope of standard Kubernetes APIs.

CoreDNS GA for Kubernetes Cluster DNS

Tue, 10 Jul 2018 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.11

Introduction

In Kubernetes 1.11, CoreDNS has reached General Availability (GA) for DNS-based service discovery, as an alternative to the kube-dns addon. This means that CoreDNS will be offered as an option in upcoming versions of the various installation tools. In fact, the kubeadm team chose to make it the default option starting with Kubernetes 1.11.

Meet Our Contributors - Monthly Streaming YouTube Mentoring Series

Tue, 10 Jul 2018 00:00:00 +0000

July 11th at 2:30pm and 8pm UTC kicks off our next installment of Meet Our Contributors YouTube series. This month is special: members of the steering committee will be on to answer any and all questions from the community on the first 30 minutes of the 8pm UTC session. More on submitting questions below.

Meet Our Contributors was created to give an opportunity to new and current contributors alike to get time in front of our upstream community to ask questions that you would typically ask a mentor. We have 3-6 contributors on each session (an AM and PM session depending on where you are in the world!) answer questions live on a YouTube stream. If you miss it, don’t stress, the recording is up after it’s over. Check out a past episode here.

IPVS-Based In-Cluster Load Balancing Deep Dive

Mon, 09 Jul 2018 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.11

Introduction

Per the Kubernetes 1.11 release blog post , we announced that IPVS-Based In-Cluster Service Load Balancing graduates to General Availability. In this blog, we will take you through a deep dive of the feature.

What Is IPVS?

IPVS (IP Virtual Server) is built on top of the Netfilter and implements transport-layer load balancing as part of the Linux kernel.

Airflow on Kubernetes (Part 1): A Different Kind of Operator

Thu, 28 Jun 2018 00:00:00 +0000

Introduction

As part of Bloomberg's continued commitment to developing the Kubernetes ecosystem, we are excited to announce the Kubernetes Airflow Operator; a mechanism for Apache Airflow, a popular workflow orchestration framework to natively launch arbitrary Kubernetes Pods using the Kubernetes API.

What Is Airflow?

Apache Airflow is one realization of the DevOps philosophy of "Configuration As Code." Airflow allows users to launch multi-step pipelines using a simple Python object DAG (Directed Acyclic Graph). You can define dependencies, programmatically construct complex workflows, and monitor scheduled jobs in an easy to read UI.

Kubernetes 1.11: In-Cluster Load Balancing and CoreDNS Plugin Graduate to General Availability

Wed, 27 Jun 2018 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.11, our second release of 2018!

Today’s release continues to advance maturity, scalability, and flexibility of Kubernetes, marking significant progress on features that the team has been hard at work on over the last year. This newest version graduates key features in networking, opens up two major features from SIG-API Machinery and SIG-Node for beta testing, and continues to enhance storage features that have been a focal point of the past two releases. The features in this release make it increasingly possible to plug any infrastructure, cloud or on-premise, into the Kubernetes system.

Dynamic Ingress in Kubernetes

Thu, 07 Jun 2018 00:00:00 +0000

Kubernetes makes it easy to deploy applications that consist of many microservices, but one of the key challenges with this type of architecture is dynamically routing ingress traffic to each of these services. One approach is Ambassador, a Kubernetes-native open source API Gateway built on the Envoy Proxy. Ambassador is designed for dynamic environment where services may come and go frequently.

Ambassador is configured using Kubernetes annotations. Annotations are used to configure specific mappings from a given Kubernetes service to a particular URL. A mapping can include a number of annotations for configuring a route. Examples include rate limiting, protocol, cross-origin request sharing, traffic shadowing, and routing rules.

4 Years of K8s

Wed, 06 Jun 2018 00:00:00 +0000

On June 6, 2014 I checked in the first commit of what would become the public repository for Kubernetes. Many would assume that is where the story starts. It is the beginning of history, right? But that really doesn’t tell the whole story.

The cast leading up to that commit was large and the success for Kubernetes since then is owed to an ever larger cast.

Kubernetes was built on ideas that had been proven out at Google over the previous ten years with Borg. And Borg, itself, owed its existence to even earlier efforts at Google and beyond.

Say Hello to Discuss Kubernetes

Wed, 30 May 2018 00:00:00 +0000

Communication is key when it comes to engaging a community of over 35,000 people in a global and remote environment. Keeping track of everything in the Kubernetes community can be an overwhelming task. On one hand we have our official resources, like Stack Overflow, GitHub, and the mailing lists, and on the other we have more ephemeral resources like Slack, where you can hop in, chat with someone, and then go on your merry way.

Introducing kustomize; Template-free Configuration Customization for Kubernetes

Tue, 29 May 2018 00:00:00 +0000

If you run a Kubernetes environment, chances are you’ve customized a Kubernetes configuration — you've copied some API object YAML files and edited them to suit your needs.

But there are drawbacks to this approach — it can be hard to go back to the source material and incorporate any improvements that were made to it. Today Google is announcing kustomize, a command-line tool contributed as a subproject of SIG-CLI. The tool provides a new, purely declarative approach to configuration customization that adheres to and leverages the familiar and carefully designed Kubernetes API.

Kubernetes Containerd Integration Goes GA

Thu, 24 May 2018 00:00:00 +0000

Kubernetes Containerd Integration Goes GA

In a previous blog - Containerd Brings More Container Runtime Options for Kubernetes, we introduced the alpha version of the Kubernetes containerd integration. With another 6 months of development, the integration with containerd is now generally available! You can now use containerd 1.1 as the container runtime for production Kubernetes clusters!

Containerd 1.1 works with Kubernetes 1.10 and above, and supports all Kubernetes features. The test coverage of containerd integration on Google Cloud Platform in Kubernetes test infrastructure is now equivalent to the Docker integration (See: test dashboard).

Getting to Know Kubevirt

Tue, 22 May 2018 00:00:00 +0000

Once you've become accustomed to running Linux container workloads on Kubernetes, you may find yourself wishing that you could run other sorts of workloads on your Kubernetes cluster. Maybe you need to run an application that isn't architected for containers, or that requires a different version of the Linux kernel -- or an all together different operating system -- than what's available on your container host.

These sorts of workloads are often well-suited to running in virtual machines (VMs), and KubeVirt, a virtual machine management add-on for Kubernetes, is aimed at allowing users to run VMs right alongside containers in the their Kubernetes or OpenShift clusters.

Gardener - The Kubernetes Botanist

Thu, 17 May 2018 00:00:00 +0000

Today, Kubernetes is the natural choice for running software in the Cloud. More and more developers and corporations are in the process of containerizing their applications, and many of them are adopting Kubernetes for automated deployments of their Cloud Native workloads.

There are many Open Source tools which help in creating and updating single Kubernetes clusters. However, the more clusters you need the harder it becomes to operate, monitor, manage, and keep all of them alive and up-to-date.

Docs are Migrating from Jekyll to Hugo

Sat, 05 May 2018 00:00:00 +0000

Changing the site framework

After nearly a year of investigating how to enable multilingual support for Kubernetes docs, we've decided to migrate the site's static generator from Jekyll to Hugo.

What does the Hugo migration mean for users and contributors?

Things will break

Hugo's Markdown parser is differently strict than Jekyll's. As a consequence, some Markdown formatting that rendered fine in Jekyll now produces some unexpected results: strange left nav ordering, vanishing tutorials, and broken links, among others.

Announcing Kubeflow 0.1

Fri, 04 May 2018 00:00:00 +0000

Since Last We Met

Since the initial announcement of Kubeflow at the last KubeCon+CloudNativeCon, we have been both surprised and delighted by the excitement for building great ML stacks for Kubernetes. In just over five months, the Kubeflow project now has:

70+ contributors
20+ contributing organizations
15 repositories
3100+ GitHub stars
700+ commits

and already is among the top 2% of GitHub projects ever.

People are excited to chat about Kubeflow as well! The Kubeflow community has also held meetups, talks and public sessions all around the world with thousands of attendees. With all this help, we’ve started to make substantial in every step of ML, from building your first model all the way to building a production-ready, high-scale deployments. At the end of the day, our mission remains the same: we want to let data scientists and software engineers focus on the things they do well by giving them an easy-to-use, portable and scalable ML stack.

Current State of Policy in Kubernetes

Wed, 02 May 2018 00:00:00 +0000

Kubernetes has grown dramatically in its impact to the industry; and with rapid growth, we are beginning to see variations across components in how they define and apply policies.

Currently, policy related components could be found in identity services, networking services, storage services, multi-cluster federation, RBAC and many other areas, with different degree of maturity and also different motivations for specific problems. Within each component, some policies are extensible while others are not. The languages used by each project to express intent vary based on the original authors and experience. Driving consistent views of policies across the entire domain is a daunting task.

Developing on Kubernetes

Tue, 01 May 2018 00:00:00 +0000

How do you develop a Kubernetes app? That is, how do you write and test an app that is supposed to run on Kubernetes? This article focuses on the challenges, tools and methods you might want to be aware of to successfully write Kubernetes apps alone or in a team setting.

We’re assuming you are a developer, you have a favorite programming language, editor/IDE, and a testing framework available. The overarching goal is to introduce minimal changes to your current workflow when developing the app for Kubernetes. For example, if you’re a Node.js developer and are used to a hot-reload setup—that is, on save in your editor the running app gets automagically updated—then dealing with containers and container images, with container registries, Kubernetes deployments, triggers, and more can not only be overwhelming but really take all the fun out if it.

Zero-downtime Deployment in Kubernetes with Jenkins

Mon, 30 Apr 2018 00:00:00 +0000

Ever since we added the Kubernetes Continuous Deploy and Azure Container Service plugins to the Jenkins update center, "How do I create zero-downtime deployments" is one of our most frequently-asked questions. We created a quickstart template on Azure to demonstrate what zero-downtime deployments can look like. Although our example uses Azure, the concept easily applies to all Kubernetes installations.

Rolling Update

Kubernetes supports the RollingUpdate strategy to replace old pods with new ones gradually, while continuing to serve clients without incurring downtime. To perform a RollingUpdate deployment:

Kubernetes Community - Top of the Open Source Charts in 2017

Wed, 25 Apr 2018 00:00:00 +0000

2017 was a huge year for Kubernetes, and GitHub’s latest Octoverse report illustrates just how much attention this project has been getting.

Kubernetes, an open source platform for running application containers, provides a consistent interface that enables developers and ops teams to automate the deployment, management, and scaling of a wide variety of applications on just about any infrastructure.

Solving these shared challenges by leveraging a wide community of expertise and industrial experience, as Kubernetes does, helps engineers focus on building their own products at the top of the stack, rather than needlessly duplicating work that now exists as a standard part of the “cloud native” toolkit.

Kubernetes Application Survey 2018 Results

Tue, 24 Apr 2018 00:00:00 +0000

Understanding how people use or want to use Kubernetes can help us shape everything from what we build to how we do it. To help us understand how application developers, application operators, and ecosystem tool developers are using and want to use Kubernetes, the Application Definition Working Group recently performed a survey. The survey focused in on these types of user roles and the features and sub-projects owned by the Kubernetes organization. That included kubectl, Dashboard, Minikube, Helm, the Workloads API, etc.

Local Persistent Volumes for Kubernetes Goes Beta

Fri, 13 Apr 2018 00:00:00 +0000

The Local Persistent Volumes beta feature in Kubernetes 1.10 makes it possible to leverage local disks in your StatefulSets. You can specify directly-attached local disks as PersistentVolumes, and use them in StatefulSets with the same PersistentVolumeClaim objects that previously only supported remote volume types.

Persistent storage is important for running stateful applications, and Kubernetes has supported these workloads with StatefulSets, PersistentVolumeClaims and PersistentVolumes. These primitives have supported remote volume types well, where the volumes can be accessed from any node in the cluster, but did not support local volumes, where the volumes can only be accessed from a specific node. The demand for using local, fast SSDs in replicated, stateful workloads has increased with demand to run more workloads in Kubernetes.

Migrating the Kubernetes Blog

Wed, 11 Apr 2018 00:00:00 +0000

We recently migrated the Kubernetes Blog from the Blogger platform to GitHub. With the change in platform comes a change in URL: formerly at http://blog.kubernetes.io, the blog now resides at https://kubernetes.io/blog.

All existing posts redirect from their former URLs with <rel=canonical> tags, preserving SEO values.

Why and how we migrated the blog

Our primary reasons for migrating were to streamline blog submissions and reviews, and to make the overall blog process faster and more transparent. Blogger's web interface made it difficult to provide drafts to multiple reviewers without also granting unnecessary access permissions and compromising security. GitHub's review process offered clear improvements.

Container Storage Interface (CSI) for Kubernetes Goes Beta

Tue, 10 Apr 2018 00:00:00 +0000

The Kubernetes implementation of the Container Storage Interface (CSI) is now beta in Kubernetes v1.10. CSI was introduced as alpha in Kubernetes v1.9.

Why introduce Container Storage Interface in Kubernetes?

Although Kubernetes already provides a powerful volume plugin system that makes it easy to consume different types of block and file storage, adding support for new volume plugins has been challenging. Because volume plugins are currently “in-tree”—volume plugins are part of the core Kubernetes code and shipped with the core Kubernetes binaries—vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) must align themselves with the Kubernetes release process.

Fixing the Subpath Volume Vulnerability in Kubernetes

Wed, 04 Apr 2018 00:00:00 +0000

On March 12, 2018, the Kubernetes Product Security team disclosed CVE-2017-1002101, which allowed containers using subpath volume mounts to access files outside of the volume. This means that a container could access any file available on the host, including volumes for other containers that it should not have access to.

The vulnerability has been fixed and released in the latest Kubernetes patch releases. We recommend that all users upgrade to get the fix. For more details on the impact and how to get the fix, please see the announcement. (Note, some functional regressions were found after the initial fix and are being tracked in issue #61563).

Kubernetes 1.10: Stabilizing Storage, Security, and Networking

Mon, 26 Mar 2018 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.10, our first release of 2018!

Today’s release continues to advance maturity, extensibility, and pluggability of Kubernetes. This newest version stabilizes features in 3 key areas, including storage, security, and networking. Notable additions in this release include the introduction of external kubectl credential providers (alpha), the ability to switch DNS service to CoreDNS at install time (beta), and the move of Container Storage Interface (CSI) and persistent local volumes to beta.

Principles of Container-based Application Design

Thu, 15 Mar 2018 00:00:00 +0000

It's possible nowadays to put almost any application in a container and run it. Creating cloud-native applications, however—containerized applications that are automated and orchestrated effectively by a cloud-native platform such as Kubernetes—requires additional effort. Cloud-native applications anticipate failure; they run and scale reliably even when their infrastructure experiences outages. To offer such capabilities, cloud-native platforms like Kubernetes impose a set of contracts and constraints on applications. These contracts ensure that applications they run conform to certain constraints and allow the platform to automate application management.

Expanding User Support with Office Hours

Wed, 14 Mar 2018 00:00:00 +0000

Today's post is on Kubernetes office hours.

Today's developer has an almost overwhelming amount of resources available for learning. Kubernetes development teams use StackOverflow, user documentation, Slack, and the mailing lists. Additionally, the community itself continues to amass an awesome list of resources.

One of the challenges of large projects is keeping user resources relevant and useful. While documentation can be useful, great learning also happens in Q&A sessions at conferences, or by learning with someone whose explanation matches your learning style. Consider that learning Kung Fu from Morpheus would be a lot more fun than reading a book about Kung Fu!

How to Integrate RollingUpdate Strategy for TPR in Kubernetes

Tue, 13 Mar 2018 00:00:00 +0000

With Kubernetes, it's easy to manage and scale stateless applications like web apps and API services right out of the box. To date, almost all of the talks about Kubernetes has been about microservices and stateless applications.

With the popularity of container-based microservice architectures, there is a strong need to deploy and manage RDBMS(Relational Database Management Systems). RDBMS requires experienced database-specific knowledge to correctly scale, upgrade, and re-configure while protecting against data loss or unavailability.

Apache Spark 2.3 with Native Kubernetes Support

Tue, 06 Mar 2018 00:00:00 +0000

Kubernetes and Big Data

The open source community has been working over the past year to enable first-class support for data processing, data analytics and machine learning workloads in Kubernetes. New extensibility features in Kubernetes, such as custom resources and custom controllers, can be used to create deep integrations with individual applications and frameworks.

Traditionally, data processing workloads have been run in dedicated setups like the YARN/Hadoop stack. However, unifying the control plane for all workloads on Kubernetes simplifies cluster management and can improve resource utilization.

Kubernetes: First Beta Version of Kubernetes 1.10 is Here

Fri, 02 Mar 2018 00:00:00 +0000

The Kubernetes community has released the first beta version of Kubernetes 1.10, which means you can now try out some of the new features and give your feedback to the release team ahead of the official release. The release, currently scheduled for March 21, 2018, is targeting the inclusion of more than a dozen brand new alpha features and more mature versions of more than two dozen more.

Specifically, Kubernetes 1.10 will include production-ready versions of Kubelet TLS Bootstrapping, API aggregation, and more detailed storage metrics.

Reporting Errors from Control Plane to Applications Using Kubernetes Events

Thu, 25 Jan 2018 00:00:00 +0000

At Box, we manage several large scale Kubernetes clusters that serve as an internal platform as a service (PaaS) for hundreds of deployed microservices. The majority of those microservices are applications that power box.com for over 80,000 customers. The PaaS team also deploys several services affiliated with the platform infrastructure as the control plane.

One use case of Box’s control plane is public key infrastructure (PKI) processing. In our infrastructure, applications needing a new SSL certificate also need to trigger some processing in the control plane. The majority of our applications are not allowed to generate new SSL certificates due to security reasons. The control plane has a different security boundary and network access, and is therefore allowed to generate certificates.

Core Workloads API GA

Mon, 15 Jan 2018 00:00:00 +0000

DaemonSet, Deployment, ReplicaSet, and StatefulSet are GA

Editor’s Note: We’re happy to announce that the Core Workloads API is GA in Kubernetes 1.9! This blog post from Kenneth Owens reviews how Core Workloads got to GA from its origins, reveals changes in 1.9, and talks about what you can expect going forward.

In the Beginning …

There were Pods, tightly coupled containers that share resource requirements, networking, storage, and a lifecycle. Pods were useful, but, as it turns out, users wanted to seamlessly, reproducibly, and automatically create many identical replicas of the same Pod, so we created ReplicationController.

Introducing client-go version 6

Fri, 12 Jan 2018 00:00:00 +0000

The Kubernetes API server exposes a REST interface consumable by any client. client-go is the official client library for the Go programming language. It is used both internally by Kubernetes itself (for example, inside kubectl) as well as by numerous external consumers:operators like the etcd-operator or prometheus-operator;higher level frameworks like KubeLess and OpenShift; and many more.

The version 6 update to client-go adds support for Kubernetes 1.9, allowing access to the latest Kubernetes features. While the changelog contains all the gory details, this blog post highlights the most prominent changes and intends to guide on how to upgrade from version 5.

Extensible Admission is Beta

Thu, 11 Jan 2018 00:00:00 +0000

In this post we review a feature, available in the Kubernetes API server, that allows you to implement arbitrary control decisions and which has matured considerably in Kubernetes 1.9.

The admission stage of API server processing is one of the most powerful tools for securing a Kubernetes cluster by restricting the objects that can be created, but it has always been limited to compiled code. In 1.9, we promoted webhooks for admission to beta, allowing you to leverage admission from outside the API server process.

Introducing Container Storage Interface (CSI) Alpha for Kubernetes

Wed, 10 Jan 2018 00:00:00 +0000

One of the key differentiators for Kubernetes has been a powerful volume plugin system that enables many different types of storage systems to:

Automatically create storage when required.
Make storage available to containers wherever they’re scheduled.
Automatically delete the storage when no longer needed. Adding support for new storage systems to Kubernetes, however, has been challenging.

Kubernetes 1.9 introduces an alpha implementation of the Container Storage Interface (CSI) which makes installing new volume plugins as easy as deploying a pod. It also enables third-party storage providers to develop solutions without the need to add to the core Kubernetes codebase.

Kubernetes v1.9 releases beta support for Windows Server Containers

Tue, 09 Jan 2018 00:00:00 +0000

At the time of publication, Michael Michael was writing as SIG-Windows Lead.

With the release of Kubernetes v1.9, our mission of ensuring Kubernetes works well everywhere and for everyone takes a great step forward. We’ve advanced support for Windows Server to beta along with continued feature and functional advancements on both the Kubernetes and Windows platforms. SIG-Windows has been working since March of 2016 to open the door for many Windows-specific applications and workloads to run on Kubernetes, significantly expanding the implementation scenarios and the enterprise reach of Kubernetes.

Five Days of Kubernetes 1.9

Mon, 08 Jan 2018 00:00:00 +0000

Kubernetes 1.9 is live, made possible by hundreds of contributors pushing thousands of commits in this latest releases.

The community has tallied around 32,300 commits in the main repo and continues rapid growth outside of the main repo, which signals growing maturity and stability for the project. The community has logged more than 90,700 commits across all repos and 7,800 commits across all repos for v1.8.0 to v1.9.0 alone.

With the help of our growing community of 1,400 plus contributors, we issued more than 4,490 PRs and pushed more than 7,800 commits to deliver Kubernetes 1.9 with many notable updates, including enhancements for the workloads and stateful application support areas. This all points to increased extensibility and standards-based Kubernetes ecosystem.

Introducing Kubeflow - A Composable, Portable, Scalable ML Stack Built for Kubernetes

Thu, 21 Dec 2017 00:00:00 +0000

Kubernetes and Machine Learning

Kubernetes has quickly become the hybrid solution for deploying complicated workloads anywhere. While it started with just stateless services, customers have begun to move complex workloads to the platform, taking advantage of rich APIs, reliability and performance provided by Kubernetes. One of the fastest growing use cases is to use Kubernetes as the deployment platform of choice for machine learning.

Building any production-ready machine learning system involves various components, often mixing vendors and hand-rolled solutions. Connecting and managing these services for even moderately sophisticated setups introduces huge barriers of complexity in adopting machine learning. Infrastructure engineers will often spend a significant amount of time manually tweaking deployments and hand rolling solutions before a single model can be tested.

Kubernetes 1.9: Apps Workloads GA and Expanded Ecosystem

Fri, 15 Dec 2017 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.9, our fourth and final release this year.

Today’s release continues the evolution of an increasingly rich feature set, more robust stability, and even greater community contributions. As the fourth release of the year, it gives us an opportunity to look back at the progress made in key areas. Particularly notable is the advancement of the Apps Workloads API to stable. This removes any reservations potential adopters might have had about the functional stability required to run mission-critical workloads. Another big milestone is the beta release of Windows support, which opens the door for many Windows-specific applications and workloads to run in Kubernetes, significantly expanding the implementation scenarios and enterprise readiness of Kubernetes.

Using eBPF in Kubernetes

Thu, 07 Dec 2017 00:00:00 +0000

Introduction

Kubernetes provides a high-level API and a set of components that hides almost all of the intricate and—to some of us—interesting details of what happens at the systems level. Application developers are not required to have knowledge of the machines' IP tables, cgroups, namespaces, seccomp, or, nowadays, even the container runtime that their application runs on top of. But underneath, Kubernetes and the technologies upon which it relies (for example, the container runtime) heavily leverage core Linux functionalities.

PaddlePaddle Fluid: Elastic Deep Learning on Kubernetes

Wed, 06 Dec 2017 00:00:00 +0000

Editor's note: Today's post is a joint post from the deep learning team at Baidu and the etcd team at CoreOS

PaddlePaddle Fluid: Elastic Deep Learning on Kubernetes

Two open source communities—PaddlePaddle, the deep learning framework originated in Baidu, and Kubernetes®, the most famous containerized application scheduler—are announcing the Elastic Deep Learning (EDL) feature in PaddlePaddle’s new release codenamed Fluid.

Fluid EDL includes a Kubernetes controller, PaddlePaddle auto-scaler, which changes the number of processes of distributed jobs according to the idle hardware resource in the cluster, and a new fault-tolerable architecture as described in the PaddlePaddle design doc.

Autoscaling in Kubernetes

Fri, 17 Nov 2017 00:00:00 +0000

Kubernetes allows developers to automatically adjust cluster sizes and the number of pod replicas based on current traffic and load. These adjustments reduce the amount of unused nodes, saving money and resources. In this talk, Marcin Wielgus of Google walks you through the current state of pod and node autoscaling in Kubernetes: how it works, and how to use it, including best practices for deployments in production applications.

Enjoyed this talk? Join us for more exciting sessions on scaling and automating your Kubernetes clusters at KubeCon in Austin on December 6-8. ~~Register now.~~

Certified Kubernetes Conformance Program: Launch Celebration Round Up

Thu, 16 Nov 2017 00:00:00 +0000

This week the CNCFⓇ certified the first group of KubernetesⓇ offerings under the Certified Kubernetes Conformance Program. These first certifications follow a beta phase during which we invited participants to submit conformance results. The community response was overwhelming: CNCF certified offerings from 32 vendors!

The new Certified Kubernetes Conformance Program gives enterprise organizations the confidence that workloads running on any Certified Kubernetes distribution or platform will work correctly on other Certified Kubernetes distributions or platforms. A Certified Kubernetes product guarantees that the complete Kubernetes API functions as specified, so users can rely on a seamless, stable experience.

Kubernetes is Still Hard (for Developers)

Wed, 15 Nov 2017 00:00:00 +0000

Kubernetes has made the Ops experience much easier, but how does the developer experience compare? Ops teams can deploy a Kubernetes cluster in a matter of minutes. But developers need to understand a host of new concepts before beginning to work with Kubernetes. This can be a tedious and manual process, but it doesn’t have to be. In this talk, Michelle Noorali, co-lead of SIG-Apps, reimagines the Kubernetes developer experience. She shares her top 3 tips for building a successful developer experience including:

Securing Software Supply Chain with Grafeas

Fri, 03 Nov 2017 00:00:00 +0000

Kubernetes has evolved to support increasingly complex classes of applications, enabling the development of two major industry trends: hybrid cloud and microservices. With increasing complexity in production environments, customers—especially enterprises—are demanding better ways to manage their software supply chain with more centralized visibility and control over production deployments.

On October 12th, Google and partners announced Grafeas, an open source initiative to define a best practice for auditing and governing the modern software supply chain. With Grafeas (“scribe” in Greek), developers can plug in components of the CI/CD pipeline into a central source of truth for tracking and enforcing policies. Google is also working on Kritis (“judge” in Greek), allowing devOps teams to enforce deploy-time image policy using metadata and attestations stored in Grafeas.

Containerd Brings More Container Runtime Options for Kubernetes

Thu, 02 Nov 2017 00:00:00 +0000

Update: Kubernetes support for Docker via dockershim is now deprecated. For more information, read the deprecation notice. You can also discuss the deprecation via a dedicated GitHub issue.

A container runtime is software that executes containers and manages container images on a node. Today, the most widely known container runtime is Docker, but there are other container runtimes in the ecosystem, such as rkt, containerd, and lxd. Docker is by far the most common container runtime used in production Kubernetes environments, but Docker’s smaller offspring, containerd, may prove to be a better option. This post describes using containerd with Kubernetes.

Kubernetes the Easy Way

Wed, 01 Nov 2017 00:00:00 +0000

Kelsey Hightower wrote an invaluable guide for Kubernetes called Kubernetes the Hard Way. It’s an awesome resource for those looking to understand the ins and outs of Kubernetes—but what if you want to put Kubernetes on easy mode? That’s something we’ve been working on together with Google Cloud. In this guide, we’ll show you how to get a cluster up and running, as well as how to actually deploy your code to that cluster and run it.

Enforcing Network Policies in Kubernetes

Mon, 30 Oct 2017 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.8.

Kubernetes now offers functionality to enforce rules about which pods can communicate with each other using network policies. This feature is has become stable Kubernetes 1.7 and is ready to use with supported networking plugins. The Kubernetes 1.8 release has added better capabilities to this feature.

Network policy: What does it mean?

In a Kubernetes cluster configured with default settings, all pods can discover and communicate with each other without any restrictions. The new Kubernetes object type NetworkPolicy lets you allow and block traffic to pods.

Using RBAC, Generally Available in Kubernetes v1.8

Sat, 28 Oct 2017 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.8.

Kubernetes 1.8 represents a significant milestone for the role-based access control (RBAC) authorizer, which was promoted to GA in this release. RBAC is a mechanism for controlling access to the Kubernetes API, and since its beta in 1.6, many Kubernetes clusters and provisioning strategies have enabled it by default.

Going forward, we expect to see RBAC become a fundamental building block for securing Kubernetes clusters. This post explores using RBAC to manage user and application access to the Kubernetes API.

It Takes a Village to Raise a Kubernetes

Thu, 26 Oct 2017 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.8, written by Jaice Singer DuMars from Microsoft.

Each time we release a new version of Kubernetes, it’s enthralling to see how the community responds to all of the hard work that went into it. Blogs on new or enhanced capabilities crop up all over the web like wildflowers in the spring. Talks, videos, webinars, and demos are not far behind. As soon as the community seems to take this all in, we turn around and add more to the mix. It’s a thrilling time to be a part of this project, and even more so, the movement. It’s not just software anymore.

kubeadm v1.8 Released: Introducing Easy Upgrades for Kubernetes Clusters

Wed, 25 Oct 2017 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.8.

Since its debut in September 2016, the Cluster Lifecycle Special Interest Group (SIG) has established kubeadm as the easiest Kubernetes bootstrap method. Now, we’re releasing kubeadm v1.8.0 in tandem with the release of Kubernetes v1.8.0. In this blog post, I’ll walk you through the changes we’ve made to kubeadm since the last update, the scope of kubeadm, and how you can contribute to this effort.

Five Days of Kubernetes 1.8

Tue, 24 Oct 2017 00:00:00 +0000

Kubernetes 1.8 is live, made possible by hundreds of contributors pushing thousands of commits in this latest releases.

The community has tallied more than 66,000 commits in the main repo and continues rapid growth outside of the main repo, which signals growing maturity and stability for the project. The community has logged more than 120,000 commits across all repos and 17,839 commits across all repos for v1.7.0 to v1.8.0 alone.

Introducing Software Certification for Kubernetes

Thu, 19 Oct 2017 00:00:00 +0000

Over the last three years, Kubernetes® has seen wide-scale adoption by a vibrant and diverse community of providers. In fact, there are now more than 60 known Kubernetes platforms and distributions. From the start, one goal of Kubernetes has been consistency and portability.

In order to better serve this goal, today the Kubernetes community and the Cloud Native Computing Foundation® (CNCF®) announce the availability of the beta Certified Kubernetes Conformance Program. The Kubernetes conformance certification program gives users the confidence that when they use a Certified Kubernetes™ product, they can rely on a high level of common functionality. Certification provides Independent Software Vendors (ISVs) confidence that if their customer is using a Certified Kubernetes product, their software will behave as expected.

Request Routing and Policy Management with the Istio Service Mesh

Tue, 10 Oct 2017 00:00:00 +0000

Editor's note: Today’s post is the second post in a three-part series on Istio.

In a previous article, we looked at a simple application (Bookinfo) that is composed of four separate microservices. The article showed how to deploy an application with Kubernetes and an Istio-enabled cluster without changing any application code. The article also outlined how to view Istio provided L7 metrics on the running services.

This article follows up by taking a deeper look at Istio using Bookinfo. Specifically, we’ll look at two more features of Istio: request routing and policy management.

Kubernetes Community Steering Committee Election Results

Thu, 05 Oct 2017 00:00:00 +0000

Beginning with the announcement of Kubernetes 1.0 at OSCON in 2015, there has been a concerted effort to share the power and burden of leadership across the Kubernetes community.

With the work of the Bootstrap Governance Committee, consisting of Brandon Philips, Brendan Burns, Brian Grant, Clayton Coleman, Joe Beda, Sarah Novotny and Tim Hockin - a cross section of long-time leaders representing 5 different companies with major investments of talent and effort in the Kubernetes Ecosystem - we wrote an initial Steering Committee Charter and launched a community wide election to seat a Kubernetes Steering Committee.

Kubernetes 1.8: Security, Workloads and Feature Depth

Fri, 29 Sep 2017 00:00:00 +0000

We’re pleased to announce the delivery of Kubernetes 1.8, our third release this year. Kubernetes 1.8 represents a snapshot of many exciting enhancements and refinements underway. In addition to functional improvements, we’re increasing project-wide focus on maturing process, formalizing architecture, and strengthening Kubernetes’ governance model. The evolution of mature processes clearly signals that sustainability is a driving concern, and helps to ensure that Kubernetes is a viable and thriving project far into the future.

Kubernetes StatefulSets & DaemonSets Updates

Wed, 27 Sep 2017 00:00:00 +0000

This post talks about recent updates to the DaemonSet and StatefulSet API objects for Kubernetes. We explore these features using Apache ZooKeeper and Apache Kafka StatefulSets and a Prometheus node exporter DaemonSet.

In Kubernetes 1.6, we added the RollingUpdate update strategy to the DaemonSet API Object. Configuring your DaemonSets with the RollingUpdate strategy causes the DaemonSet controller to perform automated rolling updates to the Pods in your DaemonSets when their spec.template are updated.

In Kubernetes 1.7, we enhanced the DaemonSet controller to track a history of revisions to the PodTemplateSpecs of DaemonSets. This allows the DaemonSet controller to roll back an update. We also added the RollingUpdate strategy to the StatefulSet API Object, and implemented revision history tracking for the StatefulSet controller. Additionally, we added the Parallel pod management policy to support stateful applications that require Pods with unique identities but not ordered Pod creation and termination.

Introducing the Resource Management Working Group

Thu, 21 Sep 2017 00:00:00 +0000

Why are we here?

Kubernetes has evolved to support diverse and increasingly complex classes of applications. We can onboard and scale out modern, cloud-native web applications based on microservices, batch jobs, and stateful applications with persistent storage requirements.

However, there are still opportunities to improve Kubernetes; for example, the ability to run workloads that require specialized hardware or those that perform measurably better when hardware topology is taken into account. These conflicts can make it difficult for application classes (particularly in established verticals) to adopt Kubernetes.

Windows Networking at Parity with Linux for Kubernetes

Fri, 08 Sep 2017 00:00:00 +0000

Since I last blogged about Kubernetes Networking for Windows four months ago, the Windows Core Networking team has made tremendous progress in both the platform and open source Kubernetes projects. With the updates, Windows is now on par with Linux in terms of networking. Customers can now deploy mixed-OS, Kubernetes clusters in any environment including Azure, on-premises, and on 3rd-party cloud stacks with the same network primitives and topologies supported on Linux without any workarounds, “hacks”, or 3rd-party switch extensions.

Kubernetes Meets High-Performance Computing

Tue, 22 Aug 2017 00:00:00 +0000

Anyone who has worked with Docker can appreciate the enormous gains in efficiency achievable with containers. While Kubernetes excels at orchestrating containers, high-performance computing (HPC) applications can be tricky to deploy on Kubernetes.

In this post, I discuss some of the challenges of running HPC workloads with Kubernetes, explain how organizations approach these challenges today, and suggest an approach for supporting mixed workloads on a shared Kubernetes cluster. We will also provide information and links to a case study on a customer, IHME, showing how Kubernetes is extended to service their HPC workloads seamlessly while retaining scalability and interfaces familiar to HPC users.

High Performance Networking with EC2 Virtual Private Clouds

Fri, 11 Aug 2017 00:00:00 +0000

One of the most popular platforms for running Kubernetes is Amazon Web Services’ Elastic Compute Cloud (AWS EC2). With more than a decade of experience delivering IaaS, and expanding over time to include a rich set of services with easy to consume APIs, EC2 has captured developer mindshare and loyalty worldwide.

When it comes to networking, however, EC2 has some limits that hinder performance and make deploying Kubernetes clusters to production unnecessarily complex. The preview release of Romana v2.0, a network and security automation solution for Cloud Native applications, includes features that address some well known network issues when running Kubernetes in EC2.

Kompose Helps Developers Move Docker Compose Files to Kubernetes

Thu, 10 Aug 2017 00:00:00 +0000

I'm pleased to announce that Kompose, a conversion tool for developers to transition Docker Compose applications to Kubernetes, has graduated from the Kubernetes Incubator to become an official part of the project.

Since our first commit on June 27, 2016, Kompose has achieved 13 releases over 851 commits, gaining 21 contributors since the inception of the project. Our work started at Skippbox (now part of Bitnami) and grew through contributions from Google and Red Hat.

Happy Second Birthday: A Kubernetes Retrospective

Fri, 28 Jul 2017 00:00:00 +0000

As we do every July, we’re excited to celebrate Kubernetes 2nd birthday! In the two years since GA 1.0 launched as an open source project, Kubernetes (abbreviated as K8s) has grown to become the highest velocity cloud-related project. With more than 2,611 diverse contributors, from independents to leading global companies, the project has had 50,685 commits in the last 12 months. Of the 54 million projects on GitHub, Kubernetes is in the top 5 for number of unique developers contributing code. It also has more pull requests and issue comments than any other project on GitHub.

How Watson Health Cloud Deploys Applications with Kubernetes

Fri, 14 Jul 2017 00:00:00 +0000

Today’s post is by Sandhya Kapoor, Senior Technologist, Watson Platform for Health, IBM

For more than a year, Watson Platform for Health at IBM deployed healthcare applications in virtual machines on our cloud platform. Because virtual machines had been a costly, heavyweight solution for us, we were interested to evaluate Kubernetes for our deployments.

Our design was to set up the application and data containers in the same namespace, along with the required agents using sidecars, to meet security and compliance requirements in the healthcare industry.

Kubernetes 1.7: Security Hardening, Stateful Application Updates and Extensibility

Fri, 30 Jun 2017 00:00:00 +0000

This article is by Aparna Sinha and Ihor Dvoretskyi, on behalf of the Kubernetes 1.7 release team.

Today we’re announcing Kubernetes 1.7, a milestone release that adds security, storage and extensibility features motivated by widespread production use of Kubernetes in the most demanding enterprise environments.

At-a-glance, security enhancements in this release include encrypted secrets, network policy for pod-to-pod communication, node authorizer to limit kubelet access and client / server TLS certificate rotation.

Managing microservices with the Istio service mesh

Wed, 31 May 2017 00:00:00 +0000

Today’s post is by the Istio team showing how you can get visibility, resiliency, security and control for your microservices in Kubernetes.

Services are at the core of modern software architecture. Deploying a series of modular, small (micro-)services rather than big monoliths gives developers the flexibility to work in different languages, technologies and release cadence across the system; resulting in higher productivity and velocity, especially for larger teams.

With the adoption of microservices, however, new problems emerge due to the sheer number of services that exist in a larger system. Problems that had to be solved once for a monolith, like security, load balancing, monitoring, and rate limiting need to be handled for each service.

Draft: Kubernetes container development made easy

Wed, 31 May 2017 00:00:00 +0000

About a month ago Microsoft announced the acquisition of Deis to expand our expertise in containers and Kubernetes. Today, I’m excited to announce a new open source project derived from this newly expanded Azure team: Draft.

While by now the strengths of Kubernetes for deploying and managing applications at scale are well understood. The process of developing a new application for Kubernetes is still too hard. It’s harder still if you are new to containers, Kubernetes, or developing cloud applications.

Kubernetes: a monitoring guide

Fri, 19 May 2017 00:00:00 +0000

Container technologies are taking the infrastructure world by storm. While containers solve or simplify infrastructure management processes, they also introduce significant complexity in terms of orchestration. That’s where Kubernetes comes to our rescue. Just like a conductor directs an orchestra, Kubernetes oversees our ensemble of containers—starting, stopping, creating, and destroying them automatically to keep our applications humming along.

Kubernetes makes managing a containerized infrastructure much easier by creating levels of abstractions such as pods and services. We no longer have to worry about where applications are running or if they have enough resources to work properly. But that doesn’t change the fact that, in order to ensure good performance, we need to monitor our applications, the containers running them, and Kubernetes itself.

Kubespray Ansible Playbooks foster Collaborative Kubernetes Ops

Fri, 19 May 2017 00:00:00 +0000

Why Kubespray?

Making Kubernetes operationally strong is a widely held priority and I track many deployment efforts around the project. The incubated Kubespray project is of particular interest for me because it uses the popular Ansible toolset to build robust, upgradable clusters on both cloud and physical targets. I believe using tools familiar to operators grows our community.

We’re excited to see the breadth of platforms enabled by Kubespray and how well it handles a wide range of options like integrating Ceph for StatefulSet persistence and Helm for easier application uploads. Those additions have allowed us to fully integrate the OpenStack Helm charts (demo video).

Dancing at the Lip of a Volcano: The Kubernetes Security Process - Explained

Thu, 18 May 2017 00:00:00 +0000

Software running on servers underpins ever growing amounts of the world's commerce, communications, and physical infrastructure. And nearly all of these systems are connected to the internet; which means vital security updates must be applied rapidly. As software developers and IT professionals, we often find ourselves dancing on the edge of a volcano: we may either fall into magma induced oblivion from a security vulnerability exploited before we can fix it, or we may slide off the side of the mountain because of an inadequate process to address security vulnerabilities.

How Bitmovin is Doing Multi-Stage Canary Deployments with Kubernetes in the Cloud and On-Prem

Fri, 21 Apr 2017 00:00:00 +0000

Running a large scale video encoding infrastructure on multiple public clouds is tough. At Bitmovin, we have been doing it successfully for the last few years, but from an engineering perspective, it’s neither been enjoyable nor particularly fun.

So obviously, one of the main things that really sold us on using Kubernetes, was it’s common abstraction from the different supported cloud providers and the well thought out programming interface it provides. More importantly, the Kubernetes project did not settle for the lowest common denominator approach. Instead, they added the necessary abstract concepts that are required and useful to run containerized workloads in a cloud and then did all the hard work to map these concepts to the different cloud providers and their offerings.

RBAC Support in Kubernetes

Thu, 06 Apr 2017 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.6

One of the highlights of the Kubernetes 1.6 release is the RBAC authorizer feature moving to beta. RBAC, Role-based access control, is an authorization mechanism for managing permissions around Kubernetes resources. RBAC allows configuration of flexible authorization policies that can be updated without cluster restarts.

The focus of this post is to highlight some of the interesting new capabilities and best practices.

Configuring Private DNS Zones and Upstream Nameservers in Kubernetes

Tue, 04 Apr 2017 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.6

Many users have existing domain name zones that they would like to integrate into their Kubernetes DNS namespace. For example, hybrid-cloud users may want to resolve their internal “.corp” domain addresses within the cluster. Other users may have a zone populated by a non-Kubernetes service discovery system (like Consul). We’re pleased to announce that, in Kubernetes 1.6, kube-dns adds support for configurable private DNS zones (often called “stub domains”) and external upstream DNS nameservers. In this blog post, we describe how to configure and use this feature.

Advanced Scheduling in Kubernetes

Fri, 31 Mar 2017 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.6

The Kubernetes scheduler’s default behavior works well for most cases -- for example, it ensures that pods are only placed on nodes that have sufficient free resources, it ties to spread pods from the same set (ReplicaSet, StatefulSet, etc.) across nodes, it tries to balance out the resource utilization of nodes, etc.

But sometimes you want to control how your pods are scheduled. For example, perhaps you want to ensure that certain pods only schedule on nodes with specialized hardware, or you want to co-locate services that communicate frequently, or you want to dedicate a set of nodes to a particular set of users. Ultimately, you know much more about how your applications should be scheduled and deployed than Kubernetes ever will. So Kubernetes 1.6 offers four advanced scheduling features: node affinity/anti-affinity, taints and tolerations, pod affinity/anti-affinity, and custom schedulers. Each of these features are now in beta in Kubernetes 1.6.

Scalability updates in Kubernetes 1.6: 5,000 node and 150,000 pod clusters

Thu, 30 Mar 2017 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.6

Last summer we shared updates on Kubernetes scalability, since then we’ve been working hard and are proud to announce that Kubernetes 1.6 can handle 5,000-node clusters with up to 150,000 pods. Moreover, those cluster have even better end-to-end pod startup time than the previous 2,000-node clusters in the 1.3 release; and latency of the API calls are within the one-second SLO.

Dynamic Provisioning and Storage Classes in Kubernetes

Wed, 29 Mar 2017 00:00:00 +0000

Editor’s note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.6

Storage is a critical part of running stateful containers, and Kubernetes offers powerful primitives for managing it. Dynamic volume provisioning, a feature unique to Kubernetes, allows storage volumes to be created on-demand. Before dynamic provisioning, cluster administrators had to manually make calls to their cloud or storage provider to provision new storage volumes, and then create PersistentVolume objects to represent them in Kubernetes. With dynamic provisioning, these two steps are automated, eliminating the need for cluster administrators to pre-provision storage. Instead, the storage resources can be dynamically provisioned using the provisioner specified by the StorageClass object (see user-guide). StorageClasses are essentially blueprints that abstract away the underlying storage provider, as well as other parameters, like disk-type (e.g.; solid-state vs standard disks).

Five Days of Kubernetes 1.6

Wed, 29 Mar 2017 00:00:00 +0000

With the help of our growing community of 1,110 plus contributors, we pushed around 5,000 commits to deliver Kubernetes 1.6, bringing focus on multi-user, multi-workloads at scale. While many improvements have been contributed, we selected few features to highlight in a series of in-depths posts listed below.

Follow along and read what’s new:

	Five Days of Kubernetes
Day 1	Dynamic Provisioning and Storage Classes in Kubernetes Stable in 1.6
Day 2	Scalability updates in Kubernetes 1.6
Day 3	Advanced Scheduling in Kubernetes 1.6
Day 4	Configuring Private DNS Zones and Upstream Nameservers in Kubernetes
Day 5	RBAC support in Kubernetes

Connect

Kubernetes 1.6: Multi-user, Multi-workloads at Scale

Tue, 28 Mar 2017 00:00:00 +0000

This article is by Aparna Sinha on behalf of the Kubernetes 1.6 release team.

Today we’re announcing the release of Kubernetes 1.6.

In this release the community’s focus is on scale and automation, to help you deploy multiple workloads to multiple users on a cluster. We are announcing that 5,000 node clusters are supported. We moved dynamic storage provisioning to stable. Role-based access control (RBAC), kubefed, kubeadm, and several scheduling features are moving to beta. We have also added intelligent defaults throughout to enable greater automation out of the box.

The K8sPort: Engaging Kubernetes Community One Activity at a Time

Fri, 24 Mar 2017 00:00:00 +0000

The K8sPort is a hub designed to help you, the Kubernetes community, earn credit for the hard work you’re putting forth in making this one of the most successful open source projects ever. Back at KubeCon Seattle in November, I presented a lightning talk of a preview of K8sPort.

This hub, and our intentions in helping to drive this initiative in the community, grew out of a desire to help cultivate an engaged community of Kubernetes advocates. This is done through gamification in a community hub full of different activities called “challenges,” which are activities meant to help direct members of the community to attend various events and meetings, share and provide feedback on important content, answer questions posed on sites like Stack Overflow, and more.

Deploying PostgreSQL Clusters using StatefulSets

Fri, 24 Feb 2017 00:00:00 +0000

Editor’s note: Today’s guest post is by Jeff McCormick, a developer at Crunchy Data, showing how to build a PostgreSQL cluster using the new Kubernetes StatefulSet feature.

In an earlier post, I described how to deploy a PostgreSQL cluster using Helm, a Kubernetes package manager. The following example provides the steps for building a PostgreSQL cluster using the new Kubernetes StatefulSets feature.

StatefulSets Example

Step 1 - Create Kubernetes Environment

StatefulSets is a new feature implemented in Kubernetes 1.5 (prior versions it was known as PetSets). As a result, running this example will require an environment based on Kubernetes 1.5.0 or above.

Containers as a Service, the foundation for next generation PaaS

Tue, 21 Feb 2017 00:00:00 +0000

Containers are revolutionizing the way that people build, package and deploy software. But what is often overlooked is how they are revolutionizing the way that people build the software that builds, packages and deploys software. (it’s ok if you have to read that sentence twice…) Today, and in a talk at Container World tomorrow, I’m taking a look at how container orchestrators like Kubernetes form the foundation for next generation platform as a service (PaaS). In particular, I’m interested in how cloud container as a service (CaaS) platforms like Azure Container Service, Google Container Engine and others are becoming the new infrastructure layer that PaaS is built upon.

Inside JD.com's Shift to Kubernetes from OpenStack

Fri, 10 Feb 2017 00:00:00 +0000

Editor's note: Today’s post is by the Infrastructure Platform Department team at JD.com about their transition from OpenStack to Kubernetes. JD.com is one of China’s largest companies and the first Chinese Internet company to make the Global Fortune 500 list.

History of cluster building

The era of physical machines (2004-2014)

Before 2014, our company's applications were all deployed on the physical machine. In the age of physical machines, we needed to wait an average of one week for the allocation to application coming on-line. Due to the lack of isolation, applications would affected each other, resulting in a lot of potential risks. At that time, the average number of tomcat instances on each physical machine was no more than nine. The resource of physical machine was seriously wasted and the scheduling was inflexible. The time of application migration took hours due to the breakdown of physical machines. And the auto-scaling cannot be achieved. To enhance the efficiency of application deployment, we developed compilation-packaging, automatic deployment, log collection, resource monitoring and some other systems.

Run Deep Learning with PaddlePaddle on Kubernetes

Wed, 08 Feb 2017 00:00:00 +0000

What is PaddlePaddle

PaddlePaddle is an easy-to-use, efficient, flexible and scalable deep learning platform originally developed at Baidu for applying deep learning to Baidu products since 2014.

There have been more than 50 innovations created using PaddlePaddle supporting 15 Baidu products ranging from the search engine, online advertising, to Q&A and system security.

In September 2016, Baidu open sourced PaddlePaddle, and it soon attracted many contributors from outside of Baidu.

Highly Available Kubernetes Clusters

Thu, 02 Feb 2017 00:00:00 +0000

Today’s post shows how to set-up a reliable, highly available distributed Kubernetes cluster. The support for running such clusters on Google Compute Engine (GCE) was added as an alpha feature in Kubernetes 1.5 release.

Motivation

We will create a Highly Available Kubernetes cluster, with master replicas and worker nodes distributed among three zones of a region. Such setup will ensure that the cluster will continue operating during a zone failure.

Setting Up HA cluster

Fission: Serverless Functions as a Service for Kubernetes

Mon, 30 Jan 2017 00:00:00 +0000

Fission is a Functions as a Service (FaaS) / Serverless function framework built on Kubernetes.

Fission allows you to easily create HTTP services on Kubernetes from functions. It works at the source level and abstracts away container images (in most cases). It also simplifies the Kubernetes learning curve, by enabling you to make useful services without knowing much about Kubernetes.

To use Fission, you simply create functions and add them with a CLI. You can associate functions with HTTP routes, Kubernetes events, or other triggers. Fission supports NodeJS and Python today.

Running MongoDB on Kubernetes with StatefulSets

Mon, 30 Jan 2017 00:00:00 +0000

Warning:

This post is several years old. The code examples need changes to work on a current Kubernetes cluster.

Conventional wisdom says you can’t run a database in a container. “Containers are stateless!” they say, and “databases are pointless without state!”

Of course, this is not true at all. At Google, everything runs in a container, including databases. You just need the right tools. Kubernetes 1.5 includes the new StatefulSet API object (in previous versions, StatefulSet was known as PetSet). With StatefulSets, Kubernetes makes it much easier to run stateful workloads such as databases.

How we run Kubernetes in Kubernetes aka Kubeception

Fri, 20 Jan 2017 00:00:00 +0000

Giant Swarm’s container infrastructure started out with the goal to be an easy way for developers to deploy containerized microservices. Our first generation was extensively using fleet as a base layer for our infrastructure components as well as for scheduling user containers.

In order to give our users a more powerful way to manage their containers we introduced Kubernetes into our stack in early 2016. However, as we needed a quick way to flexibly spin up and manage different users’ Kubernetes clusters resiliently we kept the underlying fleet layer.

Scaling Kubernetes deployments with Policy-Based Networking

Thu, 19 Jan 2017 00:00:00 +0000

Although it’s just been eighteen-months since Kubernetes 1.0 was released, we’ve seen Kubernetes emerge as the leading container orchestration platform for deploying distributed applications. One of the biggest reasons for this is the vibrant open source community that has developed around it. The large number of Kubernetes contributors come from diverse backgrounds means we, and the community of users, are assured that we are investing in an open platform. Companies like Google (Container Engine), Red Hat (OpenShift), and CoreOS (Tectonic) are developing their own commercial offerings based on Kubernetes. This is a good thing since it will lead to more standardization and offer choice to the users.

A Stronger Foundation for Creating and Managing Kubernetes Clusters

Thu, 12 Jan 2017 00:00:00 +0000

Last time you heard from us was in September, when we announced kubeadm. The work on making kubeadm a first-class citizen in the Kubernetes ecosystem has continued and evolved. Some of us also met before KubeCon and had a very productive meeting where we talked about what the scopes for our SIG, kubeadm, and kops are.

Continuing to Define SIG-Cluster-Lifecycle

What is the scope for kubeadm?
We want kubeadm to be a common set of building blocks for all Kubernetes deployments; the piece that provides secure and recommended ways to bootstrap Kubernetes. Since there is no one true way to setup Kubernetes, kubeadm will support more than one method for each phase. We want to identify the phases every deployment of Kubernetes has in common and make configurable and easy-to-use kubeadm commands for those phases. If your organization, for example, requires that you distribute the certificates in the cluster manually or in a custom way, skip using kubeadm just for that phase. We aim to keep kubeadm usable for all other phases in that case. We want you to be able to pick which things you want kubeadm to do and let you do the rest yourself.

Kubernetes UX Survey Infographic

Mon, 09 Jan 2017 00:00:00 +0000

The following infographic summarizes the findings of a survey that the team behind Dashboard, the official web UI for Kubernetes, sent during KubeCon in November 2016. Following the KubeCon launch of the survey, it was promoted on Twitter and various Slack channels over a two week period and generated over 100 responses. We’re delighted with the data it provides us to now make feature and roadmap decisions more in-line with the needs of you, our users.

Kubernetes supports OpenAPI

Fri, 23 Dec 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.5

OpenAPI allows API providers to define their operations and models, and enables developers to automate their tools and generate their favorite language’s client to talk to that API server. Kubernetes has supported swagger 1.2 (older version of OpenAPI spec) for a while, but the spec was incomplete and invalid, making it hard to generate tools/clients based on it.

Cluster Federation in Kubernetes 1.5

Thu, 22 Dec 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.5

In the latest Kubernetes 1.5 release, you’ll notice that support for Cluster Federation is maturing. That functionality was introduced in Kubernetes 1.3, and the 1.5 release includes a number of new features, including an easier setup experience and a step closer to supporting all Kubernetes API objects.

A new command line tool called ‘kubefed’ was introduced to make getting started with Cluster Federation much simpler. Also, alpha level support was added for Federated DaemonSets, Deployments and ConfigMaps. In summary:

Windows Server Support Comes to Kubernetes

Wed, 21 Dec 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.5

Extending on the theme of giving users choice, Kubernetes 1.5 release includes the support for Windows Servers. WIth more than 80% of enterprise apps running Java on Linux or .Net on Windows, Kubernetes is previewing capabilities that extends its reach to the mass majority of enterprise workloads.

The new Kubernetes Windows Server 2016 and Windows Container support includes public preview with the following features:

StatefulSet: Run and Scale Stateful Applications Easily in Kubernetes

Tue, 20 Dec 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.5

In the latest release, Kubernetes 1.5, we’ve moved the feature formerly known as PetSet into beta as StatefulSet. There were no major changes to the API Object, other than the community selected name, but we added the semantics of “at most one pod per index” for deployment of the Pods in the set. Along with ordered deployment, ordered termination, unique network names, and persistent stable storage, we think we have the right primitives to support many containerized stateful workloads. We don’t claim that the feature is 100% complete (it is software after all), but we believe that it is useful in its current form, and that we can extend the API in a backwards-compatible way as we progress toward an eventual GA release.

Five Days of Kubernetes 1.5

Mon, 19 Dec 2016 00:00:00 +0000

With the help of our growing community of 1,000 contributors, we pushed some 5,000 commits to extend support for production workloads and deliver Kubernetes 1.5. While many improvements and new features have been added, we selected few to highlight in a series of in-depths posts listed below.

This progress is our commitment in continuing to make Kubernetes best way to manage your production workloads at scale.

	Five Days of Kubernetes 1.5
Day 1	Introducing Container Runtime Interface (CRI) in Kubernetes
Day 2	StatefulSet: Run and Scale Stateful Applications Easily in Kubernetes
Day 3	Windows Server Support Comes to Kubernetes
Day 4	Cluster Federation in Kubernetes 1.5
Day 5	Kubernetes supports OpenAPI

Connect

Introducing Container Runtime Interface (CRI) in Kubernetes

Mon, 19 Dec 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.5

At the lowest layers of a Kubernetes node is the software that, among other things, starts and stops containers. We call this the “Container Runtime”. The most widely known container runtime is Docker, but it is not alone in this space. In fact, the container runtime space has been rapidly evolving. As part of the effort to make Kubernetes more extensible, we've been working on a new plugin API for container runtimes in Kubernetes, called "CRI".

Kubernetes 1.5: Supporting Production Workloads

Tue, 13 Dec 2016 00:00:00 +0000

Today we’re announcing the release of Kubernetes 1.5. This release follows close on the heels of KubeCon/CloundNativeCon, where users gathered to share how they’re running their applications on Kubernetes. Many of you expressed interest in running stateful applications in containers with the eventual goal of running all applications on Kubernetes. If you have been waiting to try running a distributed database on Kubernetes, or for ways to guarantee application disruption SLOs for stateful and stateless apps, this release has solutions for you.

From Network Policies to Security Policies

Thu, 08 Dec 2016 00:00:00 +0000

Kubernetes Network Policies

Kubernetes supports a new API for network policies that provides a sophisticated model for isolating applications and reducing their attack surface. This feature, which came out of the SIG-Network group, makes it very easy and elegant to define network policies by using the built-in labels and selectors Kubernetes constructs.

Kubernetes has left it up to third parties to implement these network policies and does not provide a default implementation.

Kompose: a tool to go from Docker-compose to Kubernetes

Tue, 22 Nov 2016 00:00:00 +0000

At Skippbox, we developed kompose a tool to automatically transform your Docker Compose application into Kubernetes manifests. Allowing you to start a Compose application on a Kubernetes cluster with a single kompose up command. We’re extremely happy to have donated kompose to the Kubernetes Incubator. So here’s a quick introduction about it and some motivating factors that got us to develop it.

Docker is terrific for developers. It allows everyone to get started quickly with an application that has been packaged in a Docker image and is available on a Docker registry. To build a multi-container application, Docker has developed Docker-compose (aka Compose). Compose takes in a yaml based manifest of your multi-container application and starts all the required containers with a single command docker-compose up. However Compose only works locally or with a Docker Swarm cluster.

Kubernetes Containers Logging and Monitoring with Sematext

Fri, 18 Nov 2016 00:00:00 +0000

Managing microservices in containers is typically done with Cluster Managers and Orchestration tools. Each container platform has a slightly different set of options to deploy containers or schedule tasks on each cluster node. Because we do container monitoring and logging at Sematext, part of our job is to share our knowledge of these tools, especially as it pertains to container observability and devops. Today we’ll show a tutorial for Container Monitoring and Log Collection on Kubernetes.

Visualize Kubelet Performance with Node Dashboard

Thu, 17 Nov 2016 00:00:00 +0000

Since this article was published, the Node Performance Dashboard was retired and is no longer available.

This retirement happened in early 2019, as part of the kubernetes/contrib repository deprecation.

In Kubernetes 1.4, we introduced a new node performance analysis tool, called the node performance dashboard, to visualize and explore the behavior of the Kubelet in much richer details. This new feature will make it easy to understand and improve code performance for Kubelet developers, and lets cluster maintainer set configuration according to provided Service Level Objectives (SLOs).

CNCF Partners With The Linux Foundation To Launch New Kubernetes Certification, Training and Managed Service Provider Program

Tue, 08 Nov 2016 00:00:00 +0000

Today the CNCF is pleased to launch a new training, certification and Kubernetes Managed Service Provider (KMSP) program.

The goal of the program is to ensure enterprises get the support they’re looking for to get up to speed and roll out new applications more quickly and more efficiently. The Linux Foundation, in partnership with CNCF, will develop and operate the Kubernetes training and certification.

Interested in this course? Sign up here to pre-register. The course, expected to be available in early 2017, is open now at the discounted price of $99 (regularly $199) for a limited time, and the certification program is expected to be available in the second quarter of 2017.

Bringing Kubernetes Support to Azure Container Service

Mon, 07 Nov 2016 00:00:00 +0000

With more than a thousand people coming to KubeCon in my hometown of Seattle, nearly three years after I helped start the Kubernetes project, it’s amazing and humbling to see what a small group of people and a radical idea have become after three years of hard work from a large and growing community. In July of 2014, scarcely a month after Kubernetes became publicly available, Microsoft announced its initial support for Azure. The release of Kubernetes 1.4, brought support for native Microsoft networking, load-balancer and disk integration.

Modernizing the Skytap Cloud Micro-Service Architecture with Kubernetes

Mon, 07 Nov 2016 00:00:00 +0000

Skytap is a global public cloud that provides our customers the ability to save and clone complex virtualized environments in any given state. Our customers include enterprise organizations running applications in a hybrid cloud, educational organizations providing virtual training labs, users who need easy-to-maintain development and test labs, and a variety of organizations with diverse DevOps workflows.

Some time ago, we started growing our business at an accelerated pace — our user base and our engineering organization continue to grow simultaneously. These are exciting, rewarding challenges! However, it's difficult to scale applications and organizations smoothly, and we’re approaching the task carefully. When we first began looking at improvements to scale our toolset, it was very clear that traditional OS virtualization was not going to be an effective way to achieve our scaling goals. We found that the persistent nature of VMs encouraged engineers to build and maintain bespoke ‘pet’ VMs; this did not align well with our desire to build reusable runtime environments with a stable, predictable state. Fortuitously, growth in the Docker and Kubernetes communities has aligned with our growth, and the concurrent explosion in community engagement has (from our perspective) helped these tools mature.

Introducing Kubernetes Service Partners program and a redesigned Partners page

Mon, 31 Oct 2016 00:00:00 +0000

Kubernetes has become a leading container orchestration system by being a powerful and flexible way to run distributed systems at scale. Through our very active open source community, equating to hundreds of person years of work, Kubernetes achieved four major releases in just one year to become a critical part of thousands of companies infrastructures. However, even with all that momentum, adopting cloud native computing is a significant transition for many organizations. It can be challenging to adopt a new methodology, and many teams are looking for advice and support through that journey.

Tail Kubernetes with Stern

Mon, 31 Oct 2016 00:00:00 +0000

We love Kubernetes here at Wercker and build all our infrastructure on top of it. When deploying anything you need to have good visibility to what's going on and logs are a first view into the inner workings of your application. Good old tail -f has been around for a long time and Kubernetes has this too, built right into kubectl.

I should say that tail is by no means the tool to use for debugging issues but instead you should feed the logs into a more persistent place, such as Elasticsearch. However, there's still a place for tail where you need to quickly debug something or perhaps you don't have persistent logging set up yet (such as when developing an app in Minikube).

How We Architected and Run Kubernetes on OpenStack at Scale at Yahoo! JAPAN

Mon, 24 Oct 2016 00:00:00 +0000

Editor’s note: today’s post is by the Infrastructure Engineering team at Yahoo! JAPAN, talking about how they run OpenStack on Kubernetes. This post has been translated and edited for context with permission -- originally published on the Yahoo! JAPAN engineering blog.

Intro
This post outlines how Yahoo! JAPAN, with help from Google and Solinea, built an automation tool chain for “one-click” code deployment to Kubernetes running on OpenStack.

We’ll also cover the basic security, networking, storage, and performance needs to ensure production readiness.

Building Globally Distributed Services using Kubernetes Cluster Federation

Fri, 14 Oct 2016 00:00:00 +0000

In Kubernetes 1.3, we announced Kubernetes Cluster Federation and introduced the concept of Cross Cluster Service Discovery, enabling developers to deploy a service that was sharded across a federation of clusters spanning different zones, regions or cloud providers. This enables developers to achieve higher availability for their applications, without sacrificing quality of service, as detailed in our previous blog post.

In the latest release, Kubernetes 1.4, we've extended Cluster Federation to support Replica Sets, Secrets, Namespaces and Ingress objects. This means that you no longer need to deploy and manage these objects individually in each of your federated clusters. Just create them once in the federation, and have its built-in controllers automatically handle that for you.

Helm Charts: making it simple to package and deploy common applications on Kubernetes

Mon, 10 Oct 2016 00:00:00 +0000

There are thousands of people and companies packaging their applications for deployment on Kubernetes. This usually involves crafting a few different Kubernetes resource definitions that configure the application runtime, as well as defining the mechanism that users and other apps leverage to communicate with the application. There are some very common applications that users regularly look for guidance on deploying, such as databases, CI tools, and content management systems. These types of applications are usually not ones that are developed and iterated on by end users, but rather their configuration is customized to fit a specific use case. Once that application is deployed users can link it to their existing systems or leverage their functionality to solve their pain points.

Dynamic Provisioning and Storage Classes in Kubernetes

Fri, 07 Oct 2016 00:00:00 +0000

Storage is a critical part of running containers, and Kubernetes offers some powerful primitives for managing it. Dynamic volume provisioning, a feature unique to Kubernetes, allows storage volumes to be created on-demand. Without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, and then create PersistentVolume objects to represent them in Kubernetes. The dynamic provisioning feature eliminates the need for cluster administrators to pre-provision storage. Instead, it automatically provisions storage when it is requested by users. This feature was introduced as alpha in Kubernetes 1.2, and has been improved and promoted to beta in the latest release, 1.4. This release makes dynamic provisioning far more flexible and useful.

How we improved Kubernetes Dashboard UI in 1.4 for your production needs

Mon, 03 Oct 2016 00:00:00 +0000

With the release of Kubernetes 1.4 last week, Dashboard – the official web UI for Kubernetes – has a number of exciting updates and improvements of its own. The past three months have been busy ones for the Dashboard team, and we’re excited to share the resulting features of that effort here. If you’re not familiar with Dashboard, the GitHub repo is a great place to get started.

A quick recap before unwrapping our shiny new features: Dashboard was initially released March 2016. One of the focuses for Dashboard throughout its lifetime has been the onboarding experience; it’s a less intimidating way for Kubernetes newcomers to get started, and by showing multiple resources at once, it provides contextualization lacking in kubectl (the CLI). After that initial release though, the product team realized that fine-tuning for a beginner audience was getting ahead of ourselves: there were still fundamental product requirements that Dashboard needed to satisfy in order to have a productive UX to onboard new users too. That became our mission for this release: closing the gap between Dashboard and kubectl by showing more resources, leveraging a web UI’s strengths in monitoring and troubleshooting, and architecting this all in a user friendly way.

How we made Kubernetes insanely easy to install

Wed, 28 Sep 2016 00:00:00 +0000

Over at SIG-cluster-lifecycle, we've been hard at work the last few months on kubeadm, a tool that makes Kubernetes dramatically easier to install. We've heard from users that installing Kubernetes is harder than it should be, and we want folks to be focused on writing great distributed apps not wrangling with infrastructure!

There are three stages in setting up a Kubernetes cluster, and we decided to focus on the second two (to begin with):

How Qbox Saved 50% per Month on AWS Bills Using Kubernetes and Supergiant

Tue, 27 Sep 2016 00:00:00 +0000

Editor’s Note: Today’s post is by the team at Qbox, a hosted Elasticsearch provider sharing their experience with Kubernetes and how it helped save them fifty-percent off their cloud bill.

A little over a year ago, we at Qbox faced an existential problem. Just about all of the major IaaS providers either launched or acquired services that competed directly with our Hosted Elasticsearch service, and many of them started offering it for free. The race to zero was afoot unless we could re-engineer our infrastructure to be more performant, more stable, and less expensive than the VM approach we had had before, and the one that is in use by our IaaS brethren. With the help of Kubernetes, Docker, and Supergiant (our own hand-rolled layer for managing distributed and stateful data), we were able to deliver 50% savings, a mid-five figure sum. At the same time, support tickets plummeted. We were so pleased with the results that we decided to open source Supergiant as its own standalone product. This post will demonstrate how we accomplished it.

Kubernetes 1.4: Making it easy to run on Kubernetes anywhere

Mon, 26 Sep 2016 00:00:00 +0000

Today we’re happy to announce the release of Kubernetes 1.4.

Since the release to general availability just over 15 months ago, Kubernetes has continued to grow and achieve broad adoption across the industry. From brand new startups to large-scale businesses, users have described how big a difference Kubernetes has made in building, deploying and managing distributed applications. However, one of our top user requests has been making Kubernetes itself easier to install and use. We’ve taken that feedback to heart, and 1.4 has several major improvements.

High performance network policies in Kubernetes clusters

Wed, 21 Sep 2016 00:00:00 +0000

Network Policies

Since the release of Kubernetes 1.3 back in July, users have been able to define and enforce network policies in their clusters. These policies are firewall rules that specify permissible types of traffic to, from and between pods. If requested, Kubernetes blocks all traffic that is not explicitly allowed. Policies are applied to groups of pods identified by common labels. Labels can then be used to mimic traditional segmented networks often used to isolate layers in a multi-tier application: You might identify your front-end and back-end pods by a specific “segment” label, for example. Policies control traffic between those segments and even traffic to or from external sources.

Creating a PostgreSQL Cluster using Helm

Fri, 09 Sep 2016 00:00:00 +0000

Crunchy Data supplies a set of open source PostgreSQL and PostgreSQL related containers. The Crunchy PostgreSQL Container Suite includes containers that deploy, monitor, and administer the open source PostgreSQL database, for more details view this GitHub repository.

In this post we’ll show you how to deploy a PostgreSQL cluster using Helm, a Kubernetes package manager. For reference, the Crunchy Helm Chart examples used within this post are located here, and the pre-built containers can be found on DockerHub at this location.

Deploying to Multiple Kubernetes Clusters with kit

Tue, 06 Sep 2016 00:00:00 +0000

Our Docker journey at InVision may sound familiar. We started with Docker in our development environments, trying to get consistency there first. We wrangled our legacy monolith application into Docker images and streamlined our Dockerfiles to minimize size and amp the efficiency. Things were looking good. Did we learn a lot along the way? For sure. But at the end of it all, we had our entire engineering team working with Docker locally for their development environments. Mission accomplished! Well, not quite. Development was one thing, but moving to production was a whole other ballgame.

Cloud Native Application Interfaces

Thu, 01 Sep 2016 00:00:00 +0000

Standard Interfaces (or, the Thirteenth Factor)

When you say we need ‘software standards’ in erudite company, you get some interesting looks. Most concede that software standards have been central to the success of the boldest and most successful projects out there (like the Internet). Most are also skeptical about how they apply to the innovative world we live in today. Our projects are executed in week increments, not years. Getting bogged down behind mega-software-corporation-driven standards practices would be the death knell in this fluid, highly competitive world.

Security Best Practices for Kubernetes Deployment

Wed, 31 Aug 2016 00:00:00 +0000

Note: some of the recommendations in this post are no longer current. Current cluster hardening options are described in this documentation.

Editor’s note: today’s post is by Amir Jerbi and Michael Cherny of Aqua Security, describing security best practices for Kubernetes deployments, based on data they’ve collected from various use-cases seen in both on-premises and cloud deployments.

Kubernetes provides many controls that can greatly improve your application security. Configuring them requires intimate knowledge with Kubernetes and the deployment’s security requirements. The best practices we highlight here are aligned to the container lifecycle: build, ship and run, and are specifically tailored to Kubernetes deployments. We adopted these best practices in our own SaaS deployment that runs Kubernetes on Google Cloud Platform.

Scaling Stateful Applications using Kubernetes Pet Sets and FlexVolumes with Datera Elastic Data Fabric

Mon, 29 Aug 2016 00:00:00 +0000

Introduction

Persistent volumes in Kubernetes are foundational as customers move beyond stateless workloads to run stateful applications. While Kubernetes has supported stateful applications such as MySQL, Kafka, Cassandra, and Couchbase for a while, the introduction of Pet Sets has significantly improved this support. In particular, the procedure to sequence the provisioning and startup, the ability to scale and associate durably by Pet Sets has provided the ability to automate to scale the “Pets” (applications that require consistent handling and durable placement).

Kubernetes Namespaces: use cases and insights

Tue, 16 Aug 2016 00:00:00 +0000

“Who's on first, What's on second, I Don't Know's on third”

Who's on First? by Abbott and Costello

Introduction

Kubernetes is a system with several concepts. Many of these concepts get manifested as “objects” in the RESTful API (often called “resources” or “kinds”). One of these concepts is Namespaces. In Kubernetes, Namespaces are the way to partition a single Kubernetes cluster into multiple virtual clusters. In this post we’ll highlight examples of how our customers are using Namespaces.

SIG Apps: build apps for and operate them in Kubernetes

Tue, 16 Aug 2016 00:00:00 +0000

Editor's note: This post is by the Kubernetes SIG-Apps team sharing how they focus on the developer and devops experience of running applications in Kubernetes.

Kubernetes is an incredible manager for containerized applications. Because of this, numerous companies have started to run their applications in Kubernetes.

Kubernetes Special Interest Groups (SIGs) have been around to support the community of developers and operators since around the 1.0 release. People organized around networking, storage, scaling and other operational areas.

Create a Couchbase cluster using Kubernetes

Mon, 15 Aug 2016 00:00:00 +0000

Couchbase Server is an open source, distributed NoSQL document-oriented database. It exposes a fast key-value store with managed cache for submillisecond data operations, purpose-built indexers for fast queries and a query engine for executing SQL queries. For mobile and Internet of Things (IoT) environments, Couchbase Lite runs native on-device and manages sync to Couchbase Server.

Couchbase Server 4.5 was recently announced, bringing many new features, including production certified support for Docker. Couchbase is supported on a wide variety of orchestration frameworks for Docker containers, such as Kubernetes, Docker Swarm and Mesos, for full details visit this page.

Challenges of a Remotely Managed, On-Premises, Bare-Metal Kubernetes Cluster

Tue, 02 Aug 2016 00:00:00 +0000

Introduction

The recently announced Platform9 Managed Kubernetes (PMK) is an on-premises enterprise Kubernetes solution with an unusual twist: while clusters run on a user’s internal hardware, their provisioning, monitoring, troubleshooting and overall life cycle is managed remotely from the Platform9 SaaS application. While users love the intuitive experience and ease of use of this deployment model, this approach poses interesting technical challenges. In this article, we will first describe the motivation and deployment architecture of PMK, and then present an overview of the technical challenges we faced and how our engineering team addressed them.

Why OpenStack's embrace of Kubernetes is great for both communities

Tue, 26 Jul 2016 00:00:00 +0000

Today, Mirantis, the leading contributor to OpenStack, announced that it will re-write its private cloud platform to use Kubernetes as its underlying orchestration engine. We think this is a great step forward for both the OpenStack and Kubernetes communities. With Kubernetes under the hood, OpenStack users will benefit from the tremendous efficiency, manageability and resiliency that Kubernetes brings to the table, while positioning their applications to use more cloud-native patterns. The Kubernetes community, meanwhile, can feel confident in their choice of orchestration framework, while gaining the ability to manage both container- and VM-based applications from a single platform.

A Very Happy Birthday Kubernetes

Thu, 21 Jul 2016 00:00:00 +0000

Last year at OSCON, I got to reconnect with a bunch of friends and see what they have been working on. That turned out to be the Kubernetes 1.0 launch event. Even that day, it was clear the project was supported by a broad community -- a group that showed an ambitious vision for distributed computing.

Today, on the first anniversary of the Kubernetes 1.0 launch, it’s amazing to see what a community of dedicated individuals can do. Kubernauts have collectively put in 237 person years of coding effort since launch to bring forward our most recent release 1.3. However the community is much more than simply coding effort. It is made up of people -- individuals that have given their expertise and energy to make this project flourish. With more than 830 diverse contributors, from independents to the largest companies in the world, it’s their work that makes Kubernetes stand out. Here are stories from a couple early contributors reflecting back on the project:

Happy Birthday Kubernetes. Oh, the places you’ll go!

Thu, 21 Jul 2016 00:00:00 +0000

Dear K8s,

It’s hard to believe you’re only one - you’ve grown up so fast. On the occasion of your first birthday, I thought I would write a little note about why I was so excited when you were born, why I feel fortunate to be part of the group that is raising you, and why I’m eager to watch you continue to grow up!

--Justin

You started with an excellent foundation - good declarative functionality, built around a solid API with a well defined schema and the machinery so that we could evolve going forwards. And sure enough, over your first year you grew so fast: autoscaling, HTTP load-balancing support (Ingress), support for persistent workloads including clustered databases (PetSets). You’ve made friends with more clouds (welcome Azure & OpenStack to the family), and even started to span zones and clusters (Federation). And these are just some of the most visible changes - there’s so much happening inside that brain of yours!

The Bet on Kubernetes, a Red Hat Perspective

Thu, 21 Jul 2016 00:00:00 +0000

Two years ago, Red Hat made a big bet on Kubernetes. We bet on a simple idea: that an open source community is the best place to build the future of application orchestration, and that only an open source community could successfully integrate the diverse range of capabilities necessary to succeed. As a Red Hatter, that idea is not far-fetched - we’ve seen it successfully applied in many communities, but we’ve also seen it fail, especially when a broad reach is not supported by solid foundations. On the one year anniversary of Kubernetes 1.0, two years after the first open-source commit to the Kubernetes project, it’s worth asking the question:

Bringing End-to-End Kubernetes Testing to Azure (Part 2)

Mon, 18 Jul 2016 00:00:00 +0000

Historically, Kubernetes testing has been hosted by Google, running e2e tests on Google Compute Engine (GCE) and Google Container Engine (GKE). In fact, the gating checks for the submit-queue are a subset of tests executed on these test platforms. Federated testing aims to expand test coverage by enabling organizations to host test jobs for a variety of platforms and contribute test results to benefit the Kubernetes project. Members of the Kubernetes test team at Google and SIG-Testing have created a Kubernetes test history dashboard that publishes the results from all federated test jobs (including those hosted by Google).

Dashboard - Full Featured Web Interface for Kubernetes

Fri, 15 Jul 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.3

Kubernetes Dashboard is a project that aims to bring a general purpose monitoring and operational web interface to the Kubernetes world. Three months ago we released the first production ready version, and since then the dashboard has made massive improvements. In a single UI, you’re able to perform majority of possible interactions with your Kubernetes clusters without ever leaving your browser. This blog post breaks down new features introduced in the latest release and outlines the roadmap for the future.

Steering an Automation Platform at Wercker with Kubernetes

Fri, 15 Jul 2016 00:00:00 +0000

At Wercker we run millions of containers that execute our users’ CI/CD jobs. The vast majority of them are ephemeral and only last as long as builds, tests and deploys take to run, the rest are ephemeral, too -- aren't we all --, but tend to last a bit longer and run our infrastructure. As we are running many containers across many nodes, we were in need of a highly scalable scheduler that would make our lives easier, and as such, decided to implement Kubernetes.

Citrix + Kubernetes = A Home Run

Thu, 14 Jul 2016 00:00:00 +0000

Technical collaboration is like sports. If you work together as a team, you can go down the homestretch and pull through for a win. That’s our experience with the Google Cloud Platform team.

Recently, we approached Google Cloud Platform (GCP) to collaborate on behalf of Citrix customers and the broader enterprise market looking to migrate workloads. This migration required including the NetScaler Docker load balancer, CPX, into Kubernetes nodes and resolving any issues with getting traffic into the CPX proxies.

Cross Cluster Services - Achieving Higher Availability for your Kubernetes Applications

Thu, 14 Jul 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.3

As Kubernetes users scale their production deployments we’ve heard a clear desire to deploy services across zone, region, cluster and cloud boundaries. Services that span clusters provide geographic distribution, enable hybrid and multi-cloud scenarios and improve the level of high availability beyond single cluster multi-zone deployments. Customers who want their services to span one or more (possibly remote) clusters, need them to be reachable in a consistent manner from both within and outside their clusters.

Stateful Applications in Containers!? Kubernetes 1.3 Says “Yes!”

Wed, 13 Jul 2016 00:00:00 +0000

Congratulations to the Kubernetes community on another value-packed release. A focus on stateful applications and federated clusters are two reasons why I’m so excited about 1.3. Kubernetes support for stateful apps such as Cassandra, Kafka, and MongoDB is critical. Important services rely on databases, key value stores, message queues, and more. Additionally, relying on one data center or container cluster simply won’t work as apps grow to serve millions of users around the world. Cluster federation allows users to deploy apps across multiple clusters and data centers for scale and resiliency.

Thousand Instances of Cassandra using Kubernetes Pet Set

Wed, 13 Jul 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.3

Running The Greek Pet Monster Races

For the Kubernetes 1.3 launch, we wanted to put the new Pet Set through its paces. By testing a thousand instances of Cassandra, we could make sure that Kubernetes 1.3 was production ready. Read on for how we adapted Cassandra to Kubernetes, and had our largest deployment ever.

It’s fairly straightforward to use containers with basic stateful applications today. Using a persistent volume, you can mount a disk in a pod, and ensure that your data lasts beyond the life of your pod. However, with deployments of distributed stateful applications, things can become more tricky. With Kubernetes 1.3, the new Pet Set component makes everything much easier. To test this new feature out at scale, we decided to host the Greek Pet Monster Races! We raced Centaurs and other Ancient Greek Monsters over hundreds of thousands of races across multiple availability zones.

Autoscaling in Kubernetes

Tue, 12 Jul 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.3

Customers using Kubernetes respond to end user requests quickly and ship software faster than ever before. But what happens when you build a service that is even more popular than you planned for, and run out of compute? In Kubernetes 1.3, we are proud to announce that we have a solution: autoscaling. On Google Compute Engine (GCE) and Google Container Engine (GKE) (and coming soon on AWS), Kubernetes will automatically scale up your cluster as soon as you need it, and scale it back down to save you money when you don’t.

Kubernetes in Rancher: the further evolution

Tue, 12 Jul 2016 00:00:00 +0000

Kubernetes was the first external orchestration platform supported by Rancher, and since its release, it has become one of the most widely used among our users, and continues to grow rapidly in adoption. As Kubernetes has evolved, so has Rancher in terms of adapting new Kubernetes features. We’ve started with supporting Kubernetes version 1.1, then switched to 1.2 as soon as it was released, and now we’re working on supporting the exciting new features in 1.3. I’d like to walk you through the features that we’ve been adding support for during each of these stages.

Five Days of Kubernetes 1.3

Mon, 11 Jul 2016 00:00:00 +0000

Last week we released Kubernetes 1.3, two years from the day when the first Kubernetes commit was pushed to GitHub. Now 30,000+ commits later from over 800 contributors, this 1.3 releases is jam packed with updates driven by feedback from users.

While many new improvements and features have been added in the latest release, we’ll be highlighting several that stand-out. Follow along and read these in-depth posts on what’s new and how we continue to make Kubernetes the best way to manage containers at scale.

Minikube: easily run Kubernetes locally

Mon, 11 Jul 2016 00:00:00 +0000

Editor's note: This is the first post in a series of in-depth articles on what's new in Kubernetes 1.3

While Kubernetes is one of the best tools for managing containerized applications available today, and has been production-ready for over a year, Kubernetes has been missing a great local development platform.

For the past several months, several of us from the Kubernetes community have been working to fix this in the Minikube repository on GitHub. Our goal is to build an easy-to-use, high-fidelity Kubernetes distribution that can be run locally on Mac, Linux and Windows workstations and laptops with a single command.

rktnetes brings rkt container engine to Kubernetes

Mon, 11 Jul 2016 00:00:00 +0000

Editor's note: this post is part of a series of in-depth articles on what's new in Kubernetes 1.3

As part of Kubernetes 1.3, we’re happy to report that our work to bring interchangeable container engines to Kubernetes is bearing early fruit. What we affectionately call “rktnetes” is included in the version 1.3 Kubernetes release, and is ready for development use. rktnetes integrates support for CoreOS rkt into Kubernetes as the container runtime on cluster nodes, and is now part of the mainline Kubernetes source code. Today it’s easier than ever for developers and ops pros with container portability in mind to try out running Kubernetes with a different container engine.

Updates to Performance and Scalability in Kubernetes 1.3 -- 2,000 node 60,000 pod clusters

Thu, 07 Jul 2016 00:00:00 +0000

We are proud to announce that with the release of version 1.3, Kubernetes now supports 2000-node clusters with even better end-to-end pod startup time. The latency of our API calls are within our one-second Service Level Objective (SLO) and most of them are even an order of magnitude better than that. It is possible to run larger deployments than a 2,000 node cluster, but performance may be degraded and it may not meet our strict SLO.

Kubernetes 1.3: Bridging Cloud Native and Enterprise Workloads

Wed, 06 Jul 2016 00:00:00 +0000

Nearly two years ago, when we officially kicked off the Kubernetes project, we wanted to simplify distributed systems management and provide the core technology required to everyone. The community’s response to this effort has blown us away. Today, thousands of customers, partners and developers are running clusters in production using Kubernetes and have joined the cloud native revolution.

Thanks to the help of over 800 contributors, we are pleased to announce today the availability of Kubernetes 1.3, our most robust and feature-rich release to date.

Container Design Patterns

Tue, 21 Jun 2016 00:00:00 +0000

Kubernetes automates deployment, operations, and scaling of applications, but our goals in the Kubernetes project extend beyond system management -- we want Kubernetes to help developers, too. Kubernetes should make it easy for them to write the distributed applications and services that run in cloud and datacenter environments. To enable this, Kubernetes defines not only an API for administrators to perform management actions, but also an API for containerized applications to interact with the management platform.

The Illustrated Children's Guide to Kubernetes

Thu, 09 Jun 2016 00:00:00 +0000

Kubernetes is an open source project with a growing community. We love seeing the ways that our community innovates inside and on top of Kubernetes. Deis is an excellent example of company who understands the strategic impact of strong container orchestration. They contribute directly to the project; in associated subprojects; and, delightfully, with a creative endeavor to help our user community understand more about what Kubernetes is. Want to contribute to Kubernetes? One way is to get involved here and help us with code. But, please don’t consider that the only way to contribute. This little adventure that Deis takes us is an example of how open source isn’t only code.

Bringing End-to-End Kubernetes Testing to Azure (Part 1)

Mon, 06 Jun 2016 00:00:00 +0000

At AppFormix, continuous integration testing is part of our culture. We see many benefits to running end-to-end tests regularly, including minimizing regressions and ensuring our software works together as a whole. To ensure a high quality experience for our customers, we require the ability to run end-to-end testing not just for our application, but for the entire orchestration stack. Our customers are adopting Kubernetes as their container orchestration technology of choice, and they demand choice when it comes to where their containers execute, from private infrastructure to public providers, including Azure. After several weeks of work, we are pleased to announce we are contributing a nightly, continuous integration job that executes e2e tests on the Azure platform. After running the e2e tests each night for only a few weeks, we have already found and fixed two issues in Kubernetes. We hope our contribution of an e2e job will help the community maintain support for the Azure platform as Kubernetes evolves.

Hypernetes: Bringing Security and Multi-tenancy to Kubernetes

Tue, 24 May 2016 00:00:00 +0000

While many developers and security professionals are comfortable with Linux containers as an effective boundary, many users need a stronger degree of isolation, particularly for those running in a multi-tenant environment. Sadly, today, those users are forced to run their containers inside virtual machines, even one VM per container.

Unfortunately, this results in the loss of many of the benefits of a cloud-native deployment: slow startup time of VMs; a memory tax for every container; low utilization resulting in wasting resources.

CoreOS Fest 2016: CoreOS and Kubernetes Community meet in Berlin (& San Francisco)

Tue, 03 May 2016 00:00:00 +0000

CoreOS Fest 2016 will bring together the container and open source distributed systems community, including many thought leaders in the Kubernetes space. It is the second annual CoreOS community conference, held for the first time in Berlin on May 9th and 10th. CoreOS believes Kubernetes is the container orchestration component to deliver GIFEE (Google’s Infrastructure for Everyone Else).

At this year’s CoreOS Fest, there are tracks dedicated to Kubernetes where you’ll hear about various topics ranging from Kubernetes performance and scalability, continuous delivery and Kubernetes, rktnetes, stackanetes and more. In addition, there will be a variety of talks, from introductory workshops to deep-dives into all things containers and related software.

Introducing the Kubernetes OpenStack Special Interest Group

Fri, 22 Apr 2016 00:00:00 +0000

Editor's note: This week we’re featuring Kubernetes Special Interest Groups; Today’s post is by the SIG-OpenStack team about their mission to facilitate ideas between the OpenStack and Kubernetes communities.

The community around the Kubernetes project includes a number of Special Interest Groups (SIGs) for the purposes of facilitating focused discussions relating to important subtopics between interested contributors. Today we would like to highlight the Kubernetes OpenStack SIG focused on the interaction between Kubernetes and OpenStack, the Open Source cloud computing platform.

SIG-UI: the place for building awesome user interfaces for Kubernetes

Wed, 20 Apr 2016 00:00:00 +0000

Editor's note: This week we’re featuring Kubernetes Special Interest Groups; Today’s post is by the SIG-UI team describing their mission and showing the cool projects they work on.

Kubernetes has been handling production workloads for a long time now (see case studies). It runs on public, private and hybrid clouds as well as bare metal. It can handle all types of workloads (web serving, batch and mixed) and enable zero-downtime rolling updates. It abstracts service discovery, load balancing and storage so that applications running on Kubernetes aren’t restricted to a specific cloud provider or environment.

SIG-ClusterOps: Promote operability and interoperability of Kubernetes clusters

Tue, 19 Apr 2016 00:00:00 +0000

Editor's note: This week we’re featuring Kubernetes Special Interest Groups; Today’s post is by the SIG-ClusterOps team whose mission is to promote operability and interoperability of Kubernetes clusters -- to listen, help & escalate.

We think Kubernetes is an awesome way to run applications at scale! Unfortunately, there's a bootstrapping problem: we need good ways to build secure & reliable scale environments around Kubernetes. While some parts of the platform administration leverage the platform (cool!), there are fundamental operational topics that need to be addressed and questions (like upgrade and conformance) that need to be answered.

SIG-Networking: Kubernetes Network Policy APIs Coming in 1.3

Mon, 18 Apr 2016 00:00:00 +0000

Editor's note: This week we’re featuring Kubernetes Special Interest Groups; Today’s post is by the Network-SIG team describing network policy APIs coming in 1.3 - policies for security, isolation and multi-tenancy.

The Kubernetes network SIG has been meeting regularly since late last year to work on bringing network policy to Kubernetes and we’re starting to see the results of this effort.

One problem many users have is that the open access network policy of Kubernetes is not suitable for applications that need more precise control over the traffic that accesses a pod or service. Today, this could be a multi-tier application where traffic is only allowed from a tier’s neighbor. But as new Cloud Native applications are built by composing microservices, the ability to control traffic as it flows among these services becomes even more critical.

How to deploy secure, auditable, and reproducible Kubernetes clusters on AWS

Fri, 15 Apr 2016 00:00:00 +0000

At CoreOS, we're all about deploying Kubernetes in production at scale. Today we are excited to share a tool that makes deploying Kubernetes on Amazon Web Services (AWS) a breeze. Kube-aws is a tool for deploying auditable and reproducible Kubernetes clusters to AWS, currently used by CoreOS to spin up production clusters.

Today you might be putting the Kubernetes components together in a more manual way. With this helpful tool, Kubernetes is delivered in a streamlined package to save time, minimize interdependencies and quickly create production-ready deployments.

Adding Support for Kubernetes in Rancher

Fri, 08 Apr 2016 00:00:00 +0000

Over the last year, we’ve seen a tremendous increase in the number of companies looking to leverage containers in their software development and IT organizations. To achieve this, organizations have been looking at how to build a centralized container management capability that will make it simple for users to get access to containers, while centralizing visibility and control with the IT organization. In 2014 we started the open-source Rancher project to address this by building a management platform for containers.

Container survey results - March 2016

Fri, 08 Apr 2016 00:00:00 +0000

Last month, we had our third installment of our container survey and today we look at the results. (raw data is available here)

Looking at the headline number, “how many people are using containers” we see a decrease in the number of people currently using containers from 89% to 80%. Obviously, we can’t be certain for the cause of this decrease, but it’s my believe that the previous number was artificially high due to sampling biases and we did a better job getting a broader reach of participants in the March survey and so the March numbers more accurately represent what is going on in the world.

Configuration management with Containers

Mon, 04 Apr 2016 00:00:00 +0000

Editor's note: this is our seventh post in a series of in-depth posts on what's new in Kubernetes 1.2

A good practice when writing applications is to separate application code from configuration. We want to enable application authors to easily employ this pattern within Kubernetes. While the Secrets API allows separating information like credentials and keys from an application, no object existed in the past for ordinary, non-secret configuration. In Kubernetes 1.2, we've added a new API resource called ConfigMap to handle this type of configuration data.

Using Deployment objects with Kubernetes 1.2

Fri, 01 Apr 2016 00:00:00 +0000

Editor's note: this is the seventh post in a series of in-depth posts on what's new in Kubernetes 1.2

Kubernetes has made deploying and managing applications very straightforward, with most actions a single API or command line away, including rolling out new applications, canary testing and upgrading. So why would we need Deployments?

Deployment objects automate deploying and rolling updating applications. Compared with kubectl rolling-update, Deployment API is much faster, is declarative, is implemented server-side and has more features (for example, you can rollback to any previous revision even after the rolling update is done).

Kubernetes 1.2 and simplifying advanced networking with Ingress

Thu, 31 Mar 2016 00:00:00 +0000

Editor's note: This is the sixth post in a series of in-depth posts on what's new in Kubernetes 1.2
Ingress is currently in beta and under active development.

In Kubernetes, Services and Pods have IPs only routable by the cluster network, by default. All traffic that ends up at an edge router is either dropped or forwarded elsewhere. In Kubernetes 1.2, we’ve made improvements to the Ingress object, to simplify allowing inbound connections to reach the cluster services. It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting and lots more.

Using Spark and Zeppelin to process big data on Kubernetes 1.2

Wed, 30 Mar 2016 00:00:00 +0000

Editor's note: this is the fifth post in a series of in-depth posts on what's new in Kubernetes 1.2

With big data usage growing exponentially, many Kubernetes customers have expressed interest in running Apache Spark on their Kubernetes clusters to take advantage of the portability and flexibility of containers. Fortunately, with Kubernetes 1.2, you can now have a platform that runs Spark and Zeppelin, and your other applications side-by-side.

Why Zeppelin?

Apache Zeppelin is a web-based notebook that enables interactive data analytics. As one of its backends, Zeppelin connects to Spark. Zeppelin allows the user to interact with the Spark cluster in a simple way, without having to deal with a command-line interpreter or a Scala compiler.

AppFormix: Helping Enterprises Operationalize Kubernetes

Tue, 29 Mar 2016 00:00:00 +0000

If you run clouds for a living, you’re well aware that the tools we've used since the client/server era for monitoring, analytics and optimization just don’t cut it when applied to the agile, dynamic and rapidly changing world of modern cloud infrastructure.

And, if you’re an operator of enterprise clouds, you know that implementing containers and container cluster management is all about giving your application developers a more agile, responsive and efficient cloud infrastructure. Applications are being rewritten and new ones developed – not for legacy environments where relatively static workloads are the norm, but for dynamic, scalable cloud environments. The dynamic nature of cloud native applications coupled with the shift to continuous deployment means that the demands placed by the applications on the infrastructure are constantly changing.

Building highly available applications using Kubernetes new multi-zone clusters (a.k.a. 'Ubernetes Lite')

Tue, 29 Mar 2016 00:00:00 +0000

Editor's note: this is the third post in a series of in-depth posts on what's new in Kubernetes 1.2

Introduction

One of the most frequently-requested features for Kubernetes is the ability to run applications across multiple zones. And with good reason — developers need to deploy applications across multiple domains, to improve availability in thxe advent of a single zone outage.

Kubernetes 1.2, released two weeks ago, adds support for running a single cluster across multiple failure zones (GCP calls them simply "zones," Amazon calls them "availability zones," here we'll refer to them as "zones"). This is the first step in a broader effort to allow federating multiple Kubernetes clusters together (sometimes referred to by the affectionate nickname "Ubernetes"). This initial version (referred to as "Ubernetes Lite") offers improved application availability by spreading applications across multiple zones within a single cloud provider.

1000 nodes and beyond: updates to Kubernetes performance and scalability in 1.2

Mon, 28 Mar 2016 00:00:00 +0000

Editor's note: this is the first in a series of in-depth posts on what's new in Kubernetes 1.2

We're proud to announce that with the release of 1.2, Kubernetes now supports 1000-node clusters, with a reduction of 80% in 99th percentile tail latency for most API operations. This means in just six months, we've increased our overall scale by 10 times while maintaining a great user experience — the 99th percentile pod startup times are less than 3 seconds, and 99th percentile latency of most API operations is tens of milliseconds (the exception being LIST operations, which take hundreds of milliseconds in very large clusters).

Five Days of Kubernetes 1.2

Mon, 28 Mar 2016 00:00:00 +0000

The Kubernetes project has had some huge milestones over the past few weeks. We released Kubernetes 1.2, had our first conference in Europe, and were accepted into the Cloud Native Computing Foundation. While we catch our breath, we would like to take a moment to highlight some of the great work contributed by the community since our last milestone, just four months ago.

Our mission is to make building distributed systems easy and accessible for all. While Kubernetes 1.2 has LOTS of new features, there are a few that really highlight the strides we’re making towards that goal. Over the course of the next week, we’ll be publishing a series of in-depth posts covering what’s new, so come back daily this week to read about the new features that continue to make Kubernetes the easiest way to run containers at scale. Thanks, and stay tuned!

How container metadata changes your point of view

Mon, 28 Mar 2016 00:00:00 +0000

Sure, metadata is a fancy word. It actually means “data that describes other data.” While that definition isn’t all that helpful, it turns out metadata itself is especially helpful in container environments. When you have any complex system, the availability of metadata helps you sort and process the variety of data coming out of that system, so that you can get to the heart of an issue with less headache.

Scaling neural network image classification using Kubernetes with TensorFlow Serving

Wed, 23 Mar 2016 00:00:00 +0000

In 2011, Google developed an internal deep learning infrastructure called DistBelief, which allowed Googlers to build ever larger neural networks and scale training to thousands of cores. Late last year, Google introduced TensorFlow, its second-generation machine learning system. TensorFlow is general, flexible, portable, easy-to-use and, most importantly, developed with the open source community.

The process of introducing machine learning into your product involves creating and training a model on your dataset, and then pushing the model to production to serve requests. In this blog post, we’ll show you how you can use Kubernetes with TensorFlow Serving, a high performance, open source serving system for machine learning models, to meet the scaling demands of your application.

Kubernetes 1.2: Even more performance upgrades, plus easier application deployment and management

Thu, 17 Mar 2016 00:00:00 +0000

Today the Kubernetes project released Kubernetes 1.2. This release represents significant improvements for large organizations building distributed systems. Now with over 680 unique contributors to the project, this release represents our largest yet.

From the beginning, our mission has been to make building distributed systems easy and accessible for all. With the Kubernetes 1.2 release we’ve made strides towards our goal by increasing scale, decreasing latency and overall simplifying the way applications are deployed and managed. Now, developers at organizations of all sizes can build production scale apps more easily than ever before.

ElasticBox introduces ElasticKube to help manage Kubernetes within the enterprise

Fri, 11 Mar 2016 00:00:00 +0000

Today’s guest post is brought to you by Brannan Matherson, from ElasticBox, who’ll discuss a new open source project to help standardize container deployment and management in enterprise environments. This highlights the advantages of authentication and user management for containerized applications

I’m delighted to share some exciting work that we’re doing at ElasticBox to contribute to the open source community regarding the rapidly changing advancements in container technologies. Our team is kicking off a new initiative called ElasticKube to help solve the problem of challenging container management scenarios within the enterprise. This project is a native container management experience that is specific to Kubernetes and leverages automation to provision clusters for containerized applications based on the latest release of Kubernetes 1.2.

Kubernetes in the Enterprise with Fujitsu’s Cloud Load Control

Fri, 11 Mar 2016 00:00:00 +0000

Earlier this year, Fujitsu released its Kubernetes-based offering Fujitsu ServerViewCloud Load Control (CLC) to the public. Some might be surprised since Fujitsu’s reputation is not necessarily related to software development, but rather to hardware manufacturing and IT services. As a long-time member of the Linux foundation and founding member of the Open Container Initiative and the Cloud Native Computing Foundation, Fujitsu does not only build software, but is committed to open source software, and contributes to several projects, including Kubernetes. But we not only believe in Kubernetes as an open source project, we also chose it as the core of our offering, because it provides the best balance of feature set, resource requirements and complexity to run distributed applications at scale.

Kubernetes Community Meeting Notes - 20160225

Tue, 01 Mar 2016 00:00:00 +0000

February 25th - Redspread demo, 1.2 update and planning 1.3, newbie introductions, SIG-networking and a shout out to CoreOS blog post.

The Kubernetes contributing community meets most Thursdays at 10:00PT to discuss the project's status via videoconference. Here are the notes from the latest meeting.

Note taker: [Ilan Rabinovich]

Quick call out for sharing presentations/slides [JBeda]
Demo (10 min): Redspread [Mackenzie Burnett, Dan Gillespie]
1.2 Release Watch [T.J. Goltermann]
- currently about 80 issues in the queue that need to be addressed before branching.
  - currently looks like March 7th may slip to later in the week, but up in the air until flakey tests are resolved.
  - non-1.2 changes may be delayed in review/merging until 1.2 stabilization work completes.
- 1.3 release planning
Newbie Introductions
SIG Reports -
- Networking [Tim Hockin]
- Scale [Bob Wise]
- meeting last Friday went very well. Discussed charter AND a working deployment
  - moved meeting to Thursdays @ 1 (so in 3 hours!)
  - Rob is posting a Cluster Ops announce on TheNewStack to recruit more members
GSoC participation -- no application submitted. [Sarah Novotny]
Brian Grant has offered to review PRs that need attention for 1.2
Dynamic Provisioning
- Currently overlaps a bit with the ubernetes work
- PR in progress.
- Should work in 1.2, but being targeted more in 1.3
Next meeting is March 3rd.
- Demo from Weave on Kubernetes Anywhere
- Another Kubernetes 1.2 update
- Update from CNCF update
- 1.3 commitments from google
No meeting on March 10th.

To get involved in the Kubernetes community consider joining our Slack channel, taking a look at the Kubernetes project on GitHub, or join the Kubernetes-dev Google group. If you're really excited, you can do all of the above and join us for the next community conversation — March 3rd, 2016. Please add yourself or a topic you want to know about to the agenda and get a calendar invitation by joining this group.

State of the Container World, February 2016

Tue, 01 Mar 2016 00:00:00 +0000

Hello, and welcome to the second installment of the Kubernetes state of the container world survey. At the beginning of February we sent out a survey about people’s usage of containers, and wrote about the results from the January survey. Here we are again, as before, while we try to reach a large and representative set of respondents, this survey was publicized across the social media account of myself and others on the Kubernetes team, so I expect some pro-container and Kubernetes bias in the data.We continue to try to get as large an audience as possible, and in that vein, please go and take the March survey and share it with your friends and followers everywhere! Without further ado, the numbers...

KubeCon EU 2016: Kubernetes Community in London

Wed, 24 Feb 2016 00:00:00 +0000

KubeCon EU 2016 is the inaugural European Kubernetes community conference that follows on the American launch in November 2015. KubeCon is fully dedicated to education and community engagement for Kubernetes enthusiasts, production users and the surrounding ecosystem.

Come join us in London and hang out with hundreds from the Kubernetes community and experience a wide variety of deep technical expert talks and use cases.

Don’t miss these great speaker sessions at the conference:

Kubernetes Community Meeting Notes - 20160218

Tue, 23 Feb 2016 00:00:00 +0000

February 18th - kmachine demo, clusterops SIG formed, new k8s.io website preview, 1.2 update and planning 1.3

The Kubernetes contributing community meets most Thursdays at 10:00PT to discuss the project's status via videoconference. Here are the notes from the latest meeting.

Note taker: Rob Hirschfeld
Demo (10 min): kmachine [Sebastien Goasguen]
- started :01 intro video
- looking to create mirror of Docker tools for Kubernetes (similar to machine, compose, etc)
- kmachine (forked from Docker Machine, so has the same endpoints)
Use Case (10 min): started at :15
SIG Report starter
- Cluster Ops launch meeting Friday (doc). [Rob Hirschfeld]
Time Zone Discussion [:22]
- This timezone does not work for Asia.
- Considering rotation - once per month
- Likely 5 or 6 PT
- Rob suggested moving the regular meeting up a little
k8s.io website preview [John Mulhausen] [:27]
- using github for docs. you can fork and do a pull request against the site
- will be its own kubernetes organization BUT not in the code repo
- Google will offer a "doc bounty" where you can get GCP credits for working on docs
- Uses Jekyll to generate the site (e.g. the ToC)
- Principle will be to 100% GitHub Pages; no script trickery or plugins, just fork/clone, edit, and push
- Hope to launch at Kubecon EU
- Home Page Only Preview: http://kub.unitedcreations.xyz
1.2 Release Watch [T.J. Goltermann] [:38]
1.3 Planning update [T.J. Goltermann]
GSoC participation -- deadline 2/19 [Sarah Novotny]
March 10th meeting? [Sarah Novotny]

To get involved in the Kubernetes community consider joining our Slack channel, taking a look at the Kubernetes project on GitHub, or join the Kubernetes-dev Google group. If you're really excited, you can do all of the above and join us for the next community conversation — February 25th, 2016. Please add yourself or a topic you want to know about to the agenda and get a calendar invitation by joining this group.

Kubernetes Community Meeting Notes - 20160211

Tue, 16 Feb 2016 00:00:00 +0000

February 11th - Pangaea Demo, #AWS SIG formed, release automation and documentation team introductions. 1.2 update and planning 1.3.

The Kubernetes contributing community meets most Thursdays at 10:00PT to discuss the project's status via videoconference. Here are the notes from the latest meeting.

Note taker: Rob Hirschfeld

Demo: Pangaea [Shahidh K Muhammed, Tanmai Gopal, and Akshaya Acharya]
- Microservices packages
- Focused on Application developers
- Demo at recording +4 minutes
- Single node kubernetes cluster — runs locally using Vagrant CoreOS image
- Single user/system cluster allows use of DNS integration (unlike Compose)
- Can run locally or in cloud
- SIG Report:
  - Release Automation and an introduction to David McMahon
  - Docs and k8s website redesign proposal and an introduction to John Mulhausen
  - This will allow the system to build docs correctly from GitHub w/ minimal effort
  - Will be check-in triggered
  - Getting website style updates
  - Want to keep authoring really light
  - There will be some automated checks
  - Next week: preview of the new website during the community meeting
[@goltermann] 1.2 Release Watch (time +34 minutes)

ShareThis: Kubernetes In Production

Thu, 11 Feb 2016 00:00:00 +0000

ShareThis has grown tremendously since its first days as a tiny widget that allowed you to share to your favorite social services. It now serves over 4.5 million domains per month, helping publishers create a more authentic digital experience.

Fast growth came with a price. We leveraged technical debt to scale fast and to grow our products, particularly when it came to infrastructure. As our company expanded, the infrastructure costs mounted as well - both in terms of inefficient utilization and in terms of people costs. About 1 year ago, it became clear something needed to change.

Kubernetes Community Meeting Notes - 20160204

Tue, 09 Feb 2016 00:00:00 +0000

February 4th - rkt demo (congratulations on the 1.0, CoreOS!), eBay puts k8s on Openstack and considers Openstack on k8s, SIGs, and flaky test surge makes progress.

The Kubernetes contributing community meets most Thursdays at 10:00PT to discuss the project's status via a videoconference. Here are the notes from the latest meeting.

Note taker: Rob Hirschfeld
Demo (20 min): CoreOS rkt + Kubernetes [Shaya Potter]
- expect to see integrations w/ rkt & k8s in the coming months ("rkt-netes"). not integrated into the v1.2 release.
- Shaya gave a demo (8 minutes into meeting for video reference)
  - CLI of rkt shown spinning up containers

Kubernetes Community Meeting Notes - 20160128

Tue, 02 Feb 2016 00:00:00 +0000

January 28 - 1.2 release update, Deis demo, flaky test surge and SIGs

The Kubernetes contributing community meets once a week to discuss the project's status via a videoconference. Here are the notes from the latest meeting.

Note taker: Erin Boyd

Discuss process around code freeze/code slush (TJ Goltermann)
- Code wind down was happening during holiday (for 1.1)
- Releasing ~ every 3 months
- Build stability is still missing
- Issue on Transparency (Bob Wise)
  - Email from Sarah for call to contribute (Monday, January 25)
    - Concern over publishing dates / understanding release schedule /etc…
- Release targeted for early March
  - Where does one find information on the release schedule with the committed features?
    - For 1.2 - Send email / Slack to TJ
    - For 1.3 - Working on better process to communicate to the community
      - Twitter
      - Wiki
      - GitHub Milestones
- How to better communicate issues discovered in the SIG
  - AI: People need to email the kubernetes-dev@ mailing list with summary of findings
  - AI: Each SIG needs a note taker
Release planning vs Release testing
- Testing SIG lead Ike McCreery
  - Also part of the testing infrastructure team at Google
  - Community being able to integrate into the testing framework
    - Federated testing
- Release Manager = David McMahon
  - Request to introduce him to the community meeting
Demo: Jason Hansen Deis
- Implemented simple REST API to interact with the platform
- Deis managed application (deployed via)
  - Source -> image
  - Rolling upgrades -> Rollbacks
  - AI: Jason will provide the slides & notes
    - Slides: https://speakerdeck.com/slack/kubernetes-community-meeting-demo-january-28th-2016
    - Alpha information: https://groups.google.com/forum/#!topic/deis-users/Qhia4DD2pv4
  - Adding an administrative component (dashboard)
  - Helm wraps kubectl
Testing
- Called for community interaction
- Need to understand friction points from community
  - Better documentation
  - Better communication on how things “should work"
- Internally, Google is having daily calls to resolve test flakes
- Started up SIG testing meetings (Tuesday at 10:30 am PT)
- Everyone wants it, but no one want to pony up the time to make it happen
  - Google is dedicating headcount to it (3-4 people, possibly more)
- https://groups.google.com/forum/?hl=en#!forum/kubernetes-sig-testing
Best practices for labeling
- Are there tools built on top of these to leverage
- AI: Generate artifact for labels and what they do (Create doc)
  - Help Wanted Label - good for new community members
  - Classify labels for team and area
    - User experience, test infrastructure, etc..
SIG Config (not about deployment)
- Any interest in ansible, etc.. type
SIG Scale meeting (Bob Wise & Tim StClair)
- Tests related to performance SLA get relaxed in order to get the tests to pass
  - exposed process issues
  - AI: outline of a proposal for a notice policy if things are being changed that are critical to the system (Bob Wise/Samsung)
    - Create a Best Practices of set of constants into well documented place

To get involved in the Kubernetes community consider joining our Slack channel, taking a look at the Kubernetes project on GitHub, or join the Kubernetes-dev Google group. If you’re really excited, you can do all of the above and join us for the next community conversation — February 4th, 2016. Please add yourself or a topic you want to know about to the agenda and get a calendar invitation by joining this group.

State of the Container World, January 2016

Mon, 01 Feb 2016 00:00:00 +0000

At the start of the new year, we sent out a survey to gauge the state of the container world. We’re ready to send the February edition, but before we do, let’s take a look at the January data from the 119 responses (thank you for participating!).

A note about these numbers: First, you may notice that the numbers don’t add up to 100%, the choices were not exclusive in most cases and so percentages given are the percentage of all respondents who selected a particular choice. Second, while we attempted to reach a broad cross-section of the cloud community, the survey was initially sent out via Twitter to followers of @brendandburns, @kelseyhightower, @sarahnovotny, @juliaferraioli, @thagomizer_rb, so the audience is likely not a perfect cross-section. We’re working to broaden our sample size (have I mentioned our February survey? Come take it now).

Kubernetes Community Meeting Notes - 20160114

Thu, 28 Jan 2016 00:00:00 +0000

January 14 - RackN demo, testing woes, and KubeCon EU CFP.

Note taker: Joe Beda

Demonstration: Automated Deploy on Metal, AWS and others w/ Digital Rebar, Rob Hirschfeld and Greg Althaus from RackN
- Greg Althaus. CTO. Digital Rebar is the product. Bare metal provisioning tool.
- Detect hardware, bring it up, configure raid, OS and get workload deployed.
- Been working on Kubernetes workload.
- Seeing trend to start in cloud and then move back to bare metal.

Kubernetes Community Meeting Notes - 20160121

Thu, 28 Jan 2016 00:00:00 +0000

January 21 - Configuration, Federation and Testing, oh my.

Note taker: Rob Hirshfeld

Use Case (10 min): SFDC Paul Brown
SIG Report - SIG-config and the story of #18215.
- Application config IN K8s not deployment of K8s
- Topic has been reuse of configuration,specifically parameterization(aka templates). Needs:
  - include scoping(cluster namespace)
  - slight customization (naming changes, but not major config)
  - multiple positions on how todo this including allowing external or simple extensions
- PetSet creates instances w/stable namespace
Workflow proposal

Why Kubernetes doesn’t use libnetwork

Thu, 14 Jan 2016 00:00:00 +0000

Kubernetes has had a very basic form of network plugins since before version 1.0 was released — around the same time as Docker's libnetwork and Container Network Model (CNM) was introduced. Unlike libnetwork, the Kubernetes plugin system still retains its "alpha" designation. Now that Docker's network plugin support is released and supported, an obvious question we get is why Kubernetes has not adopted it yet. After all, vendors will almost certainly be writing plugins for Docker — we would all be better off using the same drivers, right?

Simple leader election with Kubernetes and Docker

Mon, 11 Jan 2016 00:00:00 +0000

Overview

Kubernetes simplifies the deployment and operational management of services running on clusters. However, it also simplifies the development of these services. In this post we'll see how you can use Kubernetes to easily perform leader election in your distributed application. Distributed applications usually replicate the tasks of a service for reliability and scalability, but often it is necessary to designate one of the replicas as the leader who is responsible for coordination among all of the replicas.

Creating a Raspberry Pi cluster running Kubernetes, the installation (Part 2)

Tue, 22 Dec 2015 00:00:00 +0000

At Devoxx Belgium and Devoxx Morocco, Ray Tsang and I (Arjen Wassink) showed a Raspberry Pi cluster we built at Quintor running HypriotOS, Docker and Kubernetes. While we received many compliments on the talk, the most common question was about how to build a Pi cluster themselves! We’ll be doing just that, in two parts. The first part covered the shopping list for the cluster, and this second one will show you how to get kubernetes up and running . . .

Managing Kubernetes Pods, Services and Replication Controllers with Puppet

Thu, 17 Dec 2015 00:00:00 +0000

People familiar with Puppet might have used it for managing files, packages and users on host computers. But Puppet is first and foremost a configuration management tool, and config management is a much broader discipline than just managing host-level resources. A good definition of configuration management is that it aims to solve four related problems: identification, control, status accounting and verification and audit. These problems exist in the operation of any complex system, and with the new Puppet Kubernetes module we’re starting to look at how we can solve those problems for Kubernetes.

How Weave built a multi-deployment solution for Scope using Kubernetes

Sat, 12 Dec 2015 00:00:00 +0000

Earlier this year at Weaveworks we launched Weave Scope, an open source solution for visualization and monitoring of containerised apps and services. Recently we released a hosted Scope service into an Early Access Program. Today, we want to walk you through how we initially prototyped that service, and how we ultimately chose and deployed Kubernetes as our platform.

A cloud-native architecture

Scope already had a clean internal line of demarcation between data collection and user interaction, so it was straightforward to split the application on that line, distribute probes to customers, and host frontends in the cloud. We built out a small set of microservices in the 12-factor model, which includes:

Creating a Raspberry Pi cluster running Kubernetes, the shopping list (Part 1)

Wed, 25 Nov 2015 00:00:00 +0000

At Devoxx Belgium and Devoxx Morocco, Ray Tsang and I showed a Raspberry Pi cluster we built at Quintor running HypriotOS, Docker and Kubernetes. For those who did not see the talks, you can check out an abbreviated version of the demo or the full talk by Ray on developing and deploying Java-based microservices in Kubernetes. While we received many compliments on the talk, the most common question was about how to build a Pi cluster themselves! We’ll be doing just that, in two parts. This first post will cover the shopping list for the cluster, and the second will show you how to get it up and running . . .

Monitoring Kubernetes with Sysdig

Thu, 19 Nov 2015 00:00:00 +0000

Today we’re sharing a guest post by Chris Crane from Sysdig about their monitoring integration into Kubernetes.

Kubernetes offers a full environment to write scalable and service-based applications. It takes care of things like container grouping, discovery, load balancing and healing so you don’t have to worry about them. The design is elegant, scalable and the APIs are a pleasure to use.

And like any new infrastructure platform, if you want to run Kubernetes in production, you’re going to want to be able to monitor and troubleshoot it. We’re big fans of Kubernetes here at Sysdig, and, well: we’re here to help.

One million requests per second: Dependable and dynamic distributed systems at scale

Wed, 11 Nov 2015 00:00:00 +0000

Recently, I’ve gotten in the habit of telling people that building a reliable service isn’t that hard. If you give me two Compute Engine virtual machines, a Cloud Load balancer, supervisord and nginx, I can create you a static web service that will serve a static web page, effectively forever.

The real challenge is building agile AND reliable services. In the new world of software development it's trivial to spin up enormous numbers of machines and push software to them. Developing a successful product must also include the ability to respond to changes in a predictable way, to handle upgrades elegantly and to minimize downtime for users. Missing on any one of these elements results in an unsuccessful product that's flaky and unreliable. I remember a time, not that long ago, when it was common for websites to be unavailable for an hour around midnight each day as a safety window for software upgrades. My bank still does this. It’s really not cool.

Kubernetes 1.1 Performance upgrades, improved tooling and a growing community

Mon, 09 Nov 2015 00:00:00 +0000

Since the Kubernetes 1.0 release in July, we’ve seen tremendous adoption by companies building distributed systems to manage their container clusters. We’re also been humbled by the rapid growth of the community who help make Kubernetes better everyday. We have seen commercial offerings such as Tectonic by CoreOS and RedHat Atomic Host emerge to deliver deployment and support of Kubernetes. And a growing ecosystem has added Kubernetes support including tool vendors such as Sysdig and Project Calico.

Kubernetes as Foundation for Cloud Native PaaS

Tue, 03 Nov 2015 00:00:00 +0000

With Kubernetes continuing to gain momentum as a critical tool for building and scaling container based applications, we’ve been thrilled to see a growing number of platform as a service (PaaS) offerings adopt it as a foundation. PaaS developers have been drawn to Kubernetes by its rapid rate of maturation, the soundness of its core architectural concepts, and the strength of its contributor community. The Kubernetes ecosystem continues to grow, and these PaaS projects are great additions to it.

Some things you didn’t know about kubectl

Wed, 28 Oct 2015 00:00:00 +0000

kubectl is the command line tool for interacting with Kubernetes clusters. Many people use it every day to deploy their container workloads into production clusters. But there’s more to kubectl than just kubectl create -f or kubectl rolling-update. kubectl is a veritable multi-tool of container orchestration and management. Below we describe some of the features of kubectl that you may not have seen.

Run interactive commands

kubectl run has been in kubectl since the 1.0 release, but recently we added the ability to run interactive containers in your cluster. That means that an interactive shell in your Kubernetes cluster is as close as:

Kubernetes Performance Measurements and Roadmap

Thu, 10 Sep 2015 00:00:00 +0000

No matter how flexible and reliable your container orchestration system is, ultimately, you have some work to be done, and you want it completed quickly. For big problems, a common answer is to just throw more machines at the problem. After all, more compute = faster, right?

Interestingly, adding more nodes is a little like the tyranny of the rocket equation - in some systems, adding more machines can actually make your processing slower. However, unlike the rocket equation, we can do better. Kubernetes in v1.0 version supports clusters with up to 100 nodes. However, we have a goal to 10x the number of nodes we will support by the end of 2015. This blog post will cover where we are and how we intend to achieve the next level of performance.

Using Kubernetes Namespaces to Manage Environments

Fri, 28 Aug 2015 00:00:00 +0000

One of the advantages that Kubernetes provides is the ability to manage various environments easier and better than traditional deployment strategies. For most nontrivial applications, you have test, staging, and production environments. You can spin up a separate cluster of resources, such as VMs, with the same configuration in staging and production, but that can be costly and managing the differences between the environments can be difficult.

Kubernetes includes a cool feature called [namespaces][4], which enable you to manage different environments within the same cluster. For example, you can have different test and staging environments in the same cluster of machines, potentially saving resources. You can also run different types of server, batch, or other jobs in the same cluster without worrying about them affecting each other.

The Default Namespace

Specifying the namespace is optional in Kubernetes because by default Kubernetes uses the "default" namespace. If you've just created a cluster, you can check that the default namespace exists using this command:

Weekly Kubernetes Community Hangout Notes - July 31 2015

Tue, 04 Aug 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Here are the notes from today's meeting:

Private Registry Demo - Muhammed
- Run docker-registry as an RC/Pod/Service
- Run a proxy on every node
- Access as localhost:5000
- Discussion:
  - Should we back it by GCS or S3 when possible?
  - Run real registry backed by $object_store on each node
  - DNS instead of localhost?

The Growing Kubernetes Ecosystem

Fri, 24 Jul 2015 00:00:00 +0000

Over the past year, we’ve seen fantastic momentum in the Kubernetes project, culminating with the release of Kubernetes v1 earlier this week. We’ve also witnessed the ecosystem around Kubernetes blossom, and wanted to draw attention to some of the cooler offerings we’ve seen.

| ----- | |

CloudBees and the Jenkins community have created a Kubernetes plugin, allowing Jenkins slaves to be built as Docker images and run in Docker hosts managed by Kubernetes, either on the Google Cloud Platform or on a more local Kubernetes instance. These elastic slaves are then brought online as Jenkins schedules jobs for them and destroyed after their builds are complete, ensuring masters have steady access to clean workspaces and minimizing builds’ resource footprint.

Weekly Kubernetes Community Hangout Notes - July 17 2015

Thu, 23 Jul 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Here are the notes from today's meeting:

Eric Paris: replacing salt with ansible (if we want)
- In contrib, there is a provisioning tool written in ansible
- The goal in the rewrite was to eliminate as much of the cloud provider stuff as possible
- The salt setup does a bunch of setup in scripts and then the environment is setup with salt

Strong, Simple SSL for Kubernetes Services

Tue, 14 Jul 2015 00:00:00 +0000

Hi, I’m Evan Brown (@evandbrown) and I work on the solutions architecture team for Google Cloud Platform. I recently wrote an article and tutorial about using Jenkins on Kubernetes to automate the Docker and GCE image build process. Today I’m going to discuss how I used Kubernetes services and secrets to add SSL to the Jenkins web UI. After reading this, you’ll be able to add SSL termination (and HTTP->HTTPS redirects + basic auth) to your public HTTP Kubernetes services.

Weekly Kubernetes Community Hangout Notes - July 10 2015

Mon, 13 Jul 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Here are the notes from today's meeting:

Eric Paris: replacing salt with ansible (if we want)
- In contrib, there is a provisioning tool written in ansible
- The goal in the rewrite was to eliminate as much of the cloud provider stuff as possible
- The salt setup does a bunch of setup in scripts and then the environment is setup with salt
  - This means that things like generating certs is done differently on GCE/AWS/Vagrant
- For ansible, everything must be done within ansible
- Background on ansible
  - Does not have clients
  - Provisioner ssh into the machine and runs scripts on the machine
  - You define what you want your cluster to look like, run the script, and it sets up everything at once
  - If you make one change in a config file, ansible re-runs everything (which isn’t always desirable)
  - Uses a jinja2 template
- Create machines with minimal software, then use ansible to get that machine into a runnable state
  - Sets up all of the add-ons
- Eliminates the provisioner shell scripts
- Full cluster setup currently takes about 6 minutes
  - CentOS with some packages
  - Redeploy to the cluster takes 25 seconds
- Questions for Eric
  - Where does the provider-specific configuration go?
    - The only network setup that the ansible config does is flannel; you can turn it off
  - What about init vs. systemd?
    - Should be able to support in the code w/o any trouble (not yet implemented)
- Discussion
  - Why not push the setup work into containers or kubernetes config?
    - To bootstrap a cluster drop a kubelet and a manifest
  - Running a kubelet and configuring the network should be the only things required. We can cut a machine image that is preconfigured minus the data package (certs, etc)
    - The ansible scripts install kubelet & docker if they aren’t already installed
  - Each OS (RedHat, Debian, Ubuntu) could have a different image. We could view this as part of the build process instead of the install process.
  - There needs to be solution for bare metal as well.
  - In favor of the overall goal -- reducing the special configuration in the salt configuration
  - Everything except the kubelet should run inside a container (eventually the kubelet should as well)
    - Running in a container doesn’t cut down on the complexity that we currently have
    - But it does more clearly define the interface about what the code expects
  - These tools (Chef, Puppet, Ansible) conflate binary distribution with configuration
    - Containers more clearly separate these problems
  - The mesos deployment is not completely automated yet, but the mesos deployment is completely different: kubelets get put on top on an existing mesos cluster
    - The bash scripts allow the mesos devs to see what each cloud provider is doing and re-use the relevant bits
    - There was a large reverse engineering curve, but the bash is at least readable as opposed to the salt
  - Openstack uses a different deployment as well
  - We need a well documented list of steps (e.g. create certs) that are necessary to stand up a cluster
    - This would allow us to compare across cloud providers
    - We should reduce the number of steps as much as possible
    - Ansible has 241 steps to launch a cluster
1.0 Code freeze
- How are we getting out of code freeze?
- This is a topic for next week, but the preview is that we will move slowly rather than totally opening the firehose
  - We want to clear the backlog as fast as possible while maintaining stability both on HEAD and on the 1.0 branch
  - The backlog of almost 300 PRs but there are also various parallel feature branches that have been developed during the freeze
- Cutting a cherry pick release today (1.0.1) that fixes a few issues
- Next week we will discuss the cadence for patch releases

Announcing the First Kubernetes Enterprise Training Course

Wed, 08 Jul 2015 00:00:00 +0000

At Google we rely on Linux application containers to run our core infrastructure. Everything from Search to Gmail runs in containers. In fact, we like containers so much that even our Google Compute Engine VMs run in containers! Because containers are critical to our business, we have been working with the community on many of the basic container technologies (from cgroups to Docker’s LibContainer) and even decided to build the next generation of Google’s container scheduling technology, Kubernetes, in the open.

How did the Quake demo from DockerCon Work?

Thu, 02 Jul 2015 00:00:00 +0000

Shortly after its release in 2013, Docker became a very popular open source container management tool for Linux. Docker has a rich set of commands to control the execution of a container. Commands such as start, stop, restart, kill, pause, and unpause. However, what is still missing is the ability to Checkpoint and Restore (C/R) a container natively via Docker itself.

We’ve been actively working with upstream and community developers to add support in Docker for native C/R and hope that checkpoint and restore commands will be introduced in Docker 1.8. As of this writing, it’s possible to C/R a container externally because this functionality was recently merged in libcontainer.

Kubernetes 1.0 Launch Event at OSCON

Thu, 02 Jul 2015 00:00:00 +0000

In case you haven't heard, the Kubernetes project team & community have some awesome stuff lined up for our release event at OSCON in a few weeks.

If you haven't already registered for in person or live stream, please do it now! check out kuberneteslaunch.com for all the details. You can also find out there how to get a free expo pass for OSCON which you'll need to attend in person.

The Distributed System ToolKit: Patterns for Composite Containers

Mon, 29 Jun 2015 00:00:00 +0000

Having had the privilege of presenting some ideas from Kubernetes at DockerCon 2015, I thought I would make a blog post to share some of these ideas for those of you who couldn’t be there.

Over the past two years containers have become an increasingly popular way to package and deploy code. Container images solve many real-world problems with existing packaging and deployment tools, but in addition to these significant benefits, containers offer us an opportunity to fundamentally re-think the way we build distributed applications. Just as service oriented architectures (SOA) encouraged the decomposition of applications into modular, focused services, containers should encourage the further decomposition of these services into closely cooperating modular containers. By virtue of establishing a boundary, containers enable users to build their services using modular, reusable components, and this in turn leads to services that are more reliable, more scalable and faster to build than applications built from monolithic containers.

Slides: Cluster Management with Kubernetes, talk given at the University of Edinburgh

Fri, 26 Jun 2015 00:00:00 +0000

On Friday 5 June 2015 I gave a talk called Cluster Management with Kubernetes to a general audience at the University of Edinburgh. The talk includes an example of a music store system with a Kibana front end UI and an Elasticsearch based back end which helps to make concrete concepts like pods, replication controllers and services.

Cluster Management with Kubernetes.

Cluster Level Logging with Kubernetes

Thu, 11 Jun 2015 00:00:00 +0000

A Kubernetes cluster will typically be humming along running many system and application pods. How does the system administrator collect, manage and query the logs of the system pods? How does a user query the logs of their application which is composed of many pods which may be restarted or automatically generated by the Kubernetes system? These questions are addressed by the Kubernetes cluster level logging services.

Cluster level logging for Kubernetes allows us to collect logs which persist beyond the lifetime of the pod’s container images or the lifetime of the pod or even cluster. In this article we assume that a Kubernetes cluster has been created with cluster level logging support for sending logs to Google Cloud Logging. This is an option when creating a Google Container Engine (GKE) cluster, and is enabled by default for the open source Google Compute Engine (GCE) Kubernetes distribution. After a cluster has been created you will have a collection of system pods running that support monitoring, logging and DNS resolution for names of Kubernetes services:

Weekly Kubernetes Community Hangout Notes - May 22 2015

Tue, 02 Jun 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Discussion / Topics

Code Freeze
Upgrades of cluster
E2E test issues

Code Freeze process starts EOD 22-May, including

Code Slush -- draining PRs that are active. If there are issues for v1 to raise, please do so today.
Community PRs -- plan is to reopen in ~6 weeks.
Key areas for fixes in v1 -- docs, the experience.

E2E issues and LGTM process

Kubernetes on OpenStack

Tue, 19 May 2015 00:00:00 +0000

Today, the OpenStack foundation made it even easier for you deploy and manage clusters of Docker containers on OpenStack clouds by including Kubernetes in its Community App Catalog. At a keynote today at the OpenStack Summit in Vancouver, Mark Collier, COO of the OpenStack Foundation, and Craig Peters, Mirantis product line manager, demonstrated the Community App Catalog workflow by launching a Kubernetes cluster in a matter of seconds by leveraging the compute, storage, networking and identity systems already present in an OpenStack cloud.

Docker and Kubernetes and AppC

Mon, 18 May 2015 00:00:00 +0000

Recently we announced the intent in Kubernetes, our open source cluster manager, to support AppC and RKT, an alternative container format that has been driven by CoreOS with input from many companies (including Google). This announcement has generated a surprising amount of buzz and has been construed as a move from Google to support Appc over Docker. Many have taken it as signal that Google is moving away from supporting Docker. I would like to take a moment to clarify Google’s position in this.

Weekly Kubernetes Community Hangout Notes - May 15 2015

Mon, 18 May 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

v1 API - what's in, what's out
- We're trying to fix critical issues we discover with v1beta3
- Would like to make a number of minor cleanups that will be expensive to do later
  - defaulting replication controller spec default to 1
  - deduplicating security context
  - change id field to name
  - rename host
  - inconsistent times
  - typo in container states terminated (termination vs. terminated)
  - flatten structure (requested by heavy API user)
  - pod templates - could be added after V1, field is not implemented, remove template ref field
  - in general remove any fields not implemented (can be added later)
  - if we want to change any of the identifier validation rules, should do it now
  - recently changed label validation rules to be more precise
- Bigger changes
  - generalized label selectors
  - service - change the fields in a way that we can add features in a forward compatible manner if possible
  - public IPs - what to do from a security perspective
  - Support aci format - there is an image field - add properties to signify the image, or include it in a string
  - inconsistent on object use / cross reference - needs design discussion
- Things to do later
  - volume source cleanup
  - multiple API prefixes
  - watch changes - watch client is not notified of progress
A few other proposals
- swagger spec fixes - ongoing
- additional field selectors - additive, backward compatible
- additional status - additive, backward compatible
- elimination of phase - won't make it for v1
Service discussion - Public IPs
- with public IPs as it exists we can't go to v1
- Tim has been developing a mitigation if we can't get Justin's overhaul in (but hopefully we will)
- Justin's fix will describe public IPs in a much better way
- The general problem is it's too flexible and you can do things that are scary, the mitigation is to restrict public ip usage to specific use cases -- validated public IPs would be copied to status, which is what kube-proxy would use
- public IPs used for -
  - binding to nodes / node
  - request a specific load balancer IP (GCE only)
  - emulate multi-port services -- now we support multi-port services, so no longer necessary
- This is a large change, 70% code complete, Tim & Justin working together, parallel code review and updates, need to reconcile and test
- Do we want to allow people to request host ports - is there any value in letting people ask for a public port? or should we assign you one?
  - Tim: we should assign one
- discussion of what to do with status - if users set to empty then probably their intention
- general answer to the pattern is binding
- post v1: if we can make portal ip a non-user settable field, then we need to figure out the transition plan. need to have a fixed ip for dns.
- we should be able to just randomly assign services a new port and everything should adjust, but this is not feasible for v1
- next iteration of the proposal: PR is being iterated on, testing over the weekend, so PR hopefully ready early next week - gonna be a doozie!
API transition
- actively removing all dependencies on v1beta1 and v1beta2, announced their going away
- working on a script that will touch everything in the system and will force everything to flip to v1beta3
- a release with both APIs supported and with this script can make sure clusters are moved over and we can move the API
- Should be gone by 0.19
- Help is welcome, especially for trivial things and will try to get as much done as possible in next few weeks
- Release candidate targeting mid june
- The new kubectl will not work for old APIs, will be a problem for GKE for clusters pinned to old version. Will be a problem for k8s users as well if they update kubectl
- Since there's no way to upgrade a GKE cluster, users are going to have to tear down and upgrade their cluster
- we're going to stop testing v1beta1 very soon, trying to streamline the testing paths in our CI pipelines
Did we decide we are not going to do namespace autoprovisioning?
- Brian would like to turn it off - no objections
- Documentation should include creating namepspaces
- Would like to impose a default CPU for the default namespace
- would cap the number of pods, would reduce the resource exhaustion issue
- would eliminate need to explicitly cap the number of pods on a node due to IP exhaustion
- could add resources as arguments to the porcelain commands
- kubectl run is a simplified command, but it could include some common things (image, command, ports). but could add resources
Kubernetes 1.0 Launch Event
- Save the * Blog posts, whitepapers, etc. welcome to be published
- Event will be live streamed, mostly demos & customer talks, keynote
- Big launch party in the evening
- Kit to send more info in next couple weeks

Kubernetes Release: 0.17.0

Fri, 15 May 2015 00:00:00 +0000

Release Notes:

Cleanups
- Remove old salt configs #8065 (roberthbailey)
- Kubelet: minor cleanups #8069 (yujuhong)
v1beta3
- update example/walkthrough to v1beta3 #7940 (caesarxuchao)
- update example/rethinkdb to v1beta3 #7946 (caesarxuchao)
- verify the v1beta3 yaml files all work; merge the yaml files #7917 (caesarxuchao)
- update examples/cassandra to api v1beta3 #7258 (caesarxuchao)
- update service.json in persistent-volume example to v1beta3 #7899 (caesarxuchao)
- update mysql-wordpress example to use v1beta3 API #7864 (caesarxuchao)
- Update examples/meteor to use API v1beta3 #7848 (caesarxuchao)
- update node-selector example to API v1beta3 #7872 (caesarxuchao)
- update logging-demo to use API v1beta3; modify the way to access Elasticsearch and Kibana services #7824 (caesarxuchao)
- Convert the skydns rc to use v1beta3 and add a health check to it #7619 (a-robinson)
- update the hazelcast example to API version v1beta3 #7728 (caesarxuchao)
- Fix YAML parsing for v1beta3 objects in the kubelet for file/http #7515 (brendandburns)
- Updated kubectl cluster-info to show v1beta3 addresses #7502 (piosz)
Kubelet

Resource Usage Monitoring in Kubernetes

Tue, 12 May 2015 00:00:00 +0000

Understanding how an application behaves when deployed is crucial to scaling the application and providing a reliable service. In a Kubernetes cluster, application performance can be examined at many different levels: containers, pods, services, and whole clusters. As part of Kubernetes we want to provide users with detailed resource usage information about their running applications at all these levels. This will give users deep insights into how their applications are performing and where possible application bottlenecks may be found. In comes Heapster, a project meant to provide a base monitoring platform on Kubernetes.

Weekly Kubernetes Community Hangout Notes - May 1 2015

Mon, 11 May 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Simple rolling update - Brendan
- Rolling update = nice example of why RCs and Pods are good.
- ...pause… (Brendan needs demo recovery tips from Kelsey)
- Rolling update has recovery: Cancel update and restart, update continues from where it stopped.
- New controller gets name of old controller, so appearance is pure update.

Kubernetes Release: 0.16.0

Mon, 11 May 2015 00:00:00 +0000

Release Notes:

Bring up a kuberenetes cluster using coreos image as worker nodes #7445 (dchen1107)
Cloning v1beta3 as v1 and exposing it in the apiserver #7454 (nikhiljindal)
API Conventions for Late-initializers #7366 (erictune)
Upgrade Elasticsearch to 1.5.2 for cluster logging #7455 (satnam6502)
Make delete actually stop resources by default. #7210 (brendandburns)
Change kube2sky to use token-system-dns secret, point at https endpoint ... #7154(cjcullen)
Updated CoreOS bare metal docs for 0.15.0 #7364 (hvolkmer)
Print named ports in 'describe service' #7424 (thockin)
AWS
Return public & private addresses in GetNodeAddresses #7040 (justinsb)
Improving getting existing VPC and subnet #6606 (gust1n)
Set hostname_override for minions, back to fully-qualified name #7182 (justinsb)
Conversion to v1beta3
Convert node level logging agents to v1beta3 #7274 (satnam6502)
Removing more references to v1beta1 from pkg/ #7128 (nikhiljindal)
update examples/cassandra to api v1beta3 #7258 (caesarxuchao)
Convert Elasticsearch logging to v1beta3 and de-salt #7246 (satnam6502)
Update examples/storm for v1beta3 #7231 (bcbroussard)
Update examples/spark for v1beta3 #7230 (bcbroussard)
Update Kibana RC and service to v1beta3 #7240 (satnam6502)
Updating the guestbook example to v1beta3 #7194 (nikhiljindal)
Update Phabricator to v1beta3 example #7232 (bcbroussard)
Update Kibana pod to speak to Elasticsearch using v1beta3 #7206 (satnam6502)
Validate Node IPs; clean up validation code #7180 (ddysher)
Add PortForward to runtime API. #7391 (vmarmol)
kube-proxy uses token to access port 443 of apiserver #7303 (erictune)
Move the logging-related directories to where I think they belong #7014 (a-robinson)
Make client service requests use the default timeout now that external load balancers are created asynchronously #6870 (a-robinson)
Fix bug in kube-proxy of not updating iptables rules if a service's public IPs change #6123(a-robinson)
PersistentVolumeClaimBinder #6105 (markturansky)
Fixed validation message when trying to submit incorrect secret #7356 (soltysh)
First step to supporting multiple k8s clusters #6006 (justinsb)
Parity for namespace handling in secrets E2E #7361 (pmorie)
Add cleanup policy to RollingUpdater #6996 (ironcladlou)
Use narrowly scoped interfaces for client access #6871 (ironcladlou)
Warning about Critical bug in the GlusterFS Volume Plugin #7319 (wattsteve)
Rolling update
First part of improved rolling update, allow dynamic next replication controller generation. #7268 (brendandburns)
Further implementation of rolling-update, add rename #7279 (brendandburns)
Added basic apiserver authz tests. #7293 (ashcrow)
Retry pod update on version conflict error in e2e test. #7297 (quinton-hoole)
Add "kubectl validate" command to do a cluster health check. #6597 (fabioy)
coreos/azure: Weave version bump, various other enhancements #7224 (errordeveloper)
Azure: Wait for salt completion on cluster initialization #6576 (jeffmendoza)
Tighten label parsing #6674 (kargakis)
fix watch of single object #7263 (lavalamp)
Upgrade go-dockerclient dependency to support CgroupParent #7247 (guenter)
Make secret volume plugin idempotent #7166 (pmorie)
Salt reconfiguration to get rid of nginx on GCE #6618 (roberthbailey)
Revert "Change kube2sky to use token-system-dns secret, point at https e... #7207 (fabioy)
Pod templates as their own type #5012 (smarterclayton)
iscsi Test: Add explicit check for attach and detach calls. #7110 (swagiaal)
Added field selector for listing pods #7067 (ravigadde)
Record an event on node schedulable changes #7138 (pravisankar)
Resolve #6812, limit length of load balancer names #7145 (caesarxuchao)
Convert error strings to proper validation errors. #7131 (rjnagal)
ResourceQuota add object count support for secret and volume claims #6593(derekwaynecarr)
Use Pod.Spec.Host instead of Pod.Status.HostIP for pod subresources #6985 (csrwng)
Prioritize deleting the non-running pods when reducing replicas #6992 (yujuhong)
Kubernetes UI with Dashboard component #7056 (preillyme)

To download, please visit https://github.com/GoogleCloudPlatform/kubernetes/releases/tag/v0.16.0

AppC Support for Kubernetes through RKT

Mon, 04 May 2015 00:00:00 +0000

We very recently accepted a pull request to the Kubernetes project to add appc support for the Kubernetes community. Appc is a new open container specification that was initiated by CoreOS, and is supported through CoreOS rkt container runtime.

This is an important step forward for the Kubernetes project and for the broader containers community. It adds flexibility and choice to the container-verse and brings the promise of compelling new security and performance capabilities to the Kubernetes developer.

Weekly Kubernetes Community Hangout Notes - April 24 2015

Thu, 30 Apr 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Agenda:

Flocker and Kubernetes integration demo

Notes:

flocker and kubernetes integration demo
- Flocker Q/A
  - Does the file still exists on node1 after migration?
  - Brendan: Any plan this to make it a volume? So we don't need powerstrip?
    - Luke: Need to figure out interest to decide if we want to make it a first-class persistent disk provider in kube.

Borg: The Predecessor to Kubernetes

Thu, 23 Apr 2015 00:00:00 +0000

Google has been running containerized workloads in production for more than a decade. Whether it's service jobs like web front-ends and stateful servers, infrastructure systems like Bigtable and Spanner, or batch frameworks like MapReduce and Millwheel, virtually everything at Google runs as a container. Today, we took the wraps off of Borg, Google’s long-rumored internal container-oriented cluster-management system, publishing details at the academic computer systems conference Eurosys. You can find the paper here.

Kubernetes traces its lineage directly from Borg. Many of the developers at Google working on Kubernetes were formerly developers on the Borg project. We've incorporated the best ideas from Borg in Kubernetes, and have tried to address some pain points that users identified with Borg over the years.

Kubernetes and the Mesosphere DCOS

Wed, 22 Apr 2015 00:00:00 +0000

Kubernetes and the Mesosphere DCOS

Today Mesosphere announced the addition of Kubernetes as a standard part of their DCOS offering. This is a great step forwards in bringing cloud native application management to the world, and should lay to rest many questions we hear about 'Kubernetes or Mesos, which one should I use?'. Now you can have your cake and eat it too: use both. Today's announcement extends the reach of Kubernetes to a new class of users, and add some exciting new capabilities for everyone.

Weekly Kubernetes Community Hangout Notes - April 17 2015

Fri, 17 Apr 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Agenda

Mesos Integration
High Availability (HA)
Adding performance and profiling details to e2e to track regressions
Versioned clients

Notes

Mesos integration
- Mesos integration proposal:
- No blockers to integration.
- Documentation needs to be updated.
HA
- Proposal should land today.
- Etcd cluster.
- Load-balance apiserver.
- Cold standby for controller manager and other master components.

Introducing Kubernetes API Version v1beta3

Thu, 16 Apr 2015 00:00:00 +0000

We've been hard at work on cleaning up the API over the past several months (see https://github.com/GoogleCloudPlatform/kubernetes/issues/1519 for details). The result is v1beta3, which is considered to be the release candidate for the v1 API.

We would like you to move to this new API version as soon as possible. v1beta1 and v1beta2 are deprecated, and will be removed by the end of June, shortly after we introduce the v1 API.

Kubernetes Release: 0.15.0

Thu, 16 Apr 2015 00:00:00 +0000

Release Notes:

Enables v1beta3 API and sets it to the default API version (#6098)
Added multi-port Services (#6182)
- New Getting Started Guides
- Multi-node local startup guide (#6505)
- Mesos on Google Cloud Platform (#5442)
- Ansible Setup instructions (#6237)
Added a controller framework (#5270, #5473)
The Kubelet now listens on a secure HTTPS port (#6380)
Made kubectl errors more user-friendly (#6338)
The apiserver now supports client cert authentication (#6190)
The apiserver now limits the number of concurrent requests it processes (#6207)
Added rate limiting to pod deleting (#6355)
Implement Balanced Resource Allocation algorithm as a PriorityFunction in scheduler package (#6150)
Enabled log collection from master (#6396)
Added an api endpoint to pull logs from Pods (#6497)
Added latency metrics to scheduler (#6368)
Added latency metrics to REST client (#6409)
etcd now runs in a pod on the master (#6221)
nginx now runs in a container on the master (#6334)
Began creating Docker images for master components (#6326)
Updated GCE provider to work with gcloud 0.9.54 (#6270)
Updated AWS provider to fix Region vs Zone semantics (#6011)
Record event when image GC fails (#6091)
Add a QPS limiter to the kubernetes client (#6203)
Decrease the time it takes to run make release (#6196)
New volume support
- Added iscsi volume plugin (#5506)
- Added glusterfs volume plugin (#6174)
- AWS EBS volume support (#5138)
Updated to heapster version to v0.10.0 (#6331)
Updated to etcd 2.0.9 (#6544)
Updated to Kibana to v1.2 (#6426)
Bug Fixes
- Kube-proxy now updates iptables rules if a service's public IPs change (#6123)
- Retry kube-addons creation if the initial creation fails (#6200)
- Make kube-proxy more resiliant to running out of file descriptors (#6727)

To download, please visit https://github.com/GoogleCloudPlatform/kubernetes/releases/tag/v0.15.0

Weekly Kubernetes Community Hangout Notes - April 10 2015

Sat, 11 Apr 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Agenda:

kubectl tooling, rolling update, deployments, imperative commands.
Downward API / env. substitution, and maybe preconditions/dependencies.

Notes from meeting:

1. kubectl improvements

make it simpler to use, finish rolling update, higher-level deployment concepts.
rolling update
- today
  - can replace one rc by another rc specified by a file.

Faster than a speeding Latte

Mon, 06 Apr 2015 00:00:00 +0000

Check out Brendan Burns racing Kubernetes.

Weekly Kubernetes Community Hangout Notes - April 3 2015

Sat, 04 Apr 2015 00:00:00 +0000

Kubernetes: Weekly Kubernetes Community Hangout Notes

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Agenda:

Quinton - Cluster federation
Satnam - Performance benchmarking update

Notes from meeting:

Quinton - Cluster federation

Ideas floating around after meetup in SF
- Please read and comment
Not 1.0, but put a doc together to show roadmap
Can be built outside of Kubernetes
API to control things across multiple clusters, include some logic

Auth(n)(z)

Participate in a Kubernetes User Experience Study

Tue, 31 Mar 2015 00:00:00 +0000

We need your help in shaping the future of Kubernetes and Google Container Engine, and we'd love to have you participate in a remote UX research study to help us learn about your experiences! If you're interested in participating, we invite you to take this brief survey to see if you qualify. If you’re selected to participate, we’ll follow up with you directly.

Length: 60 minute interview
Date: April 7th-15th
Location: Remote
Your gift: $100 Perks gift code*
Study format: Interview with our researcher

Interested in participating? Take this brief survey.

Weekly Kubernetes Community Hangout Notes - March 27 2015

Sat, 28 Mar 2015 00:00:00 +0000

Every week the Kubernetes contributing community meet virtually over Google Hangouts. We want anyone who's interested to know what's discussed in this forum.

Agenda:

- Andy - demo remote execution and port forwarding

- Quinton - Cluster federation - Postponed

- Clayton - UI code sharing and collaboration around Kubernetes

Notes from meeting:

1. Andy from RedHat:

Demo remote execution
- kubectl exec -p $POD -- $CMD
- Makes a connection to the master as proxy, figures out which node the pod is on, proxies connection to kubelet, which does the interesting bit. via nsenter.

Kubernetes Gathering Videos

Mon, 23 Mar 2015 00:00:00 +0000

If you missed the Kubernetes Gathering in SF last month, fear not! Here are the videos from the evening presentations organized into a playlist on YouTube

Welcome to the Kubernetes Blog!

Fri, 20 Mar 2015 00:00:00 +0000

Welcome to the new Kubernetes Blog. Follow this blog to learn about the Kubernetes Open Source project. We plan to post release notes, how-to articles, events, and maybe even some off topic fun here from time to time.

If you are using Kubernetes or contributing to the project and would like to do a guest post, please let me know.

To start things off, here's a roundup of recent Kubernetes posts from other sites:

Mon, 01 Jan 0001 00:00:00 +0000

CNCF Community Code of Conduct v1.3

Community Code of Conduct

As contributors, maintainers, and participants in the CNCF community, and in the interest of fostering an open and welcoming community, we pledge to respect all people who participate or contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, attending conferences or events, or engaging in other community or project activities.

Mon, 01 Jan 0001 00:00:00 +0000

The files in this directory have been imported from other sources. Do not edit them directly, except by replacing them with new versions.

Localization note: you do not need to create localized versions of any of the files in this directory.

Mon, 01 Jan 0001 00:00:00 +0000

Requirements:

You need a machine that is running Linux or macOS.
You need to have these tools installed:
- Python v3.7.x+
- Git
- Golang version 1.13+
- Pip used to install PyYAML
- PyYAML v5.1.2
- make
- gcc compiler/linker
- Docker (Required only for kubectl command reference)
Your PATH environment variable must include the required build tools, such as the Go binary and python.
You need to know how to create a pull request to a GitHub repository. This involves creating your own fork of the repository. For more information, see Work from a local clone.

Mon, 01 Jan 0001 00:00:00 +0000

kubeadm: easily bootstrap a secure Kubernetes cluster

Synopsis

┌──────────────────────────────────────────────────────────┐
│ KUBEADM │
│ Easily bootstrap a secure Kubernetes cluster │
│ │
│ Please give us feedback at: │
│ https://github.com/kubernetes/kubeadm/issues │
└──────────────────────────────────────────────────────────┘

Example usage:

Mon, 01 Jan 0001 00:00:00 +0000

Generate certificate keys

Synopsis

This command will print out a secure randomly-generated certificate key that can be used with the "init" command.

Mon, 01 Jan 0001 00:00:00 +0000

Check certificates expiration for a Kubernetes cluster

Synopsis

Checks expiration for the certificates in the local PKI managed by kubeadm.

Mon, 01 Jan 0001 00:00:00 +0000

Generate keys and certificate signing requests

Synopsis

Generates keys and certificate signing requests (CSRs) for all the certificates required to run the control plane. This command also generates partial kubeconfig files with private key data in the "users > user > client-key-data" field, and for each kubeconfig file an accompanying ".csr" file is created.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Renew certificates for a Kubernetes cluster

kubeadm certs renew [flags]

Options

-h, --help

help for renew

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself.

Mon, 01 Jan 0001 00:00:00 +0000

Renew all available certificates

Interact with container images used by kubeadm

kubeadm config images [flags]

Options

-h, --help

help for images

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Print a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customized

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Pull images used by kubeadm

kubeadm config images pull [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Read an older version of the kubeadm configuration API types from a file, and output the similar config object for the newer version

Mon, 01 Jan 0001 00:00:00 +0000

Print configuration

Synopsis

This command prints configurations for subcommands provided. For details, see: https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#section-directories

kubeadm config print [flags]

Options

-h, --help

help for print

Mon, 01 Jan 0001 00:00:00 +0000

Print default init configuration, that can be used for 'kubeadm init'

Synopsis

This command prints objects such as the default init configuration that is used for 'kubeadm init'.

Mon, 01 Jan 0001 00:00:00 +0000

Print default join configuration, that can be used for 'kubeadm join'

Synopsis

This command prints objects such as the default join configuration that is used for 'kubeadm join'.

Mon, 01 Jan 0001 00:00:00 +0000

Print default reset configuration, that can be used for 'kubeadm reset'

Synopsis

This command prints objects such as the default reset configuration that is used for 'kubeadm reset'.

Mon, 01 Jan 0001 00:00:00 +0000

Print default upgrade configuration, that can be used for 'kubeadm upgrade'

Synopsis

This command prints objects such as the default upgrade configuration that is used for 'kubeadm upgrade'.

Mon, 01 Jan 0001 00:00:00 +0000

Read a file containing the kubeadm configuration API and report any validation problems

Synopsis

This command lets you validate a kubeadm configuration API file and report any warnings and errors. If there are no errors the exit status will be zero, otherwise it will be non-zero. Any unmarshaling problems such as unknown API fields will trigger errors. Unknown API versions and fields with invalid values will also trigger errors. Any other errors or warnings may be reported depending on contents of the input file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Use this command to invoke single phase of the "init" workflow

kubeadm init phase [flags]

Options

-h, --help

help for phase

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Install required addons for passing conformance tests

kubeadm init phase addon [flags]

Options

-h, --help

help for addon

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Install all the addons

kubeadm init phase addon all [flags]

Options

--apiserver-advertise-address string

The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Install the CoreDNS addon to a Kubernetes cluster

Synopsis

Install the CoreDNS addon components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed.

Mon, 01 Jan 0001 00:00:00 +0000

Install the kube-proxy addon to a Kubernetes cluster

Synopsis

Install the kube-proxy addon components via the API server.

Mon, 01 Jan 0001 00:00:00 +0000

Generates bootstrap tokens used to join a node to a cluster

Synopsis

Bootstrap tokens are used for establishing bidirectional trust between a node joining the cluster and a control-plane node.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Certificate generation

kubeadm init phase certs [flags]

Options

-h, --help

help for certs

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate all certificates

kubeadm init phase certs all [flags]

Options

--apiserver-advertise-address string

Generate the certificate for serving etcd, and save them into etcd/server.crt and etcd/server.key files.

Default SANs are localhost, 127.0.0.1, 127.0.0.1, ::1

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate the self-signed CA to provision identities for front proxy, and save them into front-proxy-ca.crt and front-proxy-ca.key files.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate the certificate for the front proxy client, and save them into front-proxy-client.crt and front-proxy-client.key files.

Mon, 01 Jan 0001 00:00:00 +0000

Generate a private key for signing service account tokens along with its public key

Synopsis

Generate the private key for signing service account tokens along with its public key, and save them into sa.key and sa.pub files.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate all static Pod manifest files necessary to establish the control plane

kubeadm init phase control-plane [flags]

Options

-h, --help

help for control-plane

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate all static Pod manifest files

kubeadm init phase control-plane all [flags]

Examples

 # Generates all static Pod manifest files for control plane components,
 # functionally equivalent to what is generated by kubeadm init.
 kubeadm init phase control-plane all
 
 # Generates all static Pod manifest files using options read from a configuration file.
 kubeadm init phase control-plane all --config config.yaml

Options

--apiserver-advertise-address string

The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generates the kube-apiserver static Pod manifest

kubeadm init phase control-plane apiserver [flags]

Options

--apiserver-advertise-address string

The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generates the kube-controller-manager static Pod manifest

kubeadm init phase control-plane controller-manager [flags]

Options

--cert-dir string Default: "/etc/kubernetes/pki"

The path where to save and store the certificates.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generates the kube-scheduler static Pod manifest

kubeadm init phase control-plane scheduler [flags]

Options

--cert-dir string Default: "/etc/kubernetes/pki"

The path where to save and store the certificates.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate static Pod manifest file for local etcd

kubeadm init phase etcd [flags]

Options

-h, --help

help for etcd

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate the static Pod manifest file for a local, single-node local etcd instance

kubeadm init phase etcd local [flags]

Examples

 # Generates the static Pod manifest file for etcd, functionally
 # equivalent to what is generated by kubeadm init.
 kubeadm init phase etcd local
 
 # Generates the static Pod manifest file for etcd using options
 # read from a configuration file.
 kubeadm init phase etcd local --config config.yaml

Options

--cert-dir string Default: "/etc/kubernetes/pki"

The path where to save and store the certificates.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file

Mon, 01 Jan 0001 00:00:00 +0000

Generate a kubeconfig file for the admin to use and for kubeadm itself

Synopsis

Generate the kubeconfig file for the admin and for kubeadm itself, and save it to admin.conf file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate all kubeconfig files

kubeadm init phase kubeconfig all [flags]

Options

--apiserver-advertise-address string

The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Generate a kubeconfig file for the controller manager to use

Synopsis

Generate the kubeconfig file for the controller manager to use and save it to controller-manager.conf file

Mon, 01 Jan 0001 00:00:00 +0000

Generate a kubeconfig file for the kubelet to use only for cluster bootstrapping purposes

Synopsis

Generate the kubeconfig file for the kubelet to use and save it to kubelet.conf file.

Mon, 01 Jan 0001 00:00:00 +0000

Generate a kubeconfig file for the scheduler to use

Synopsis

Generate the kubeconfig file for the scheduler to use and save it to scheduler.conf file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate a kubeconfig file for the super-admin, and save it to super-admin.conf file.

kubeadm init phase kubeconfig super-admin [flags]

Options

--apiserver-advertise-address string

The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Updates settings relevant to the kubelet after TLS bootstrap

kubeadm init phase kubelet-finalize [flags]

Examples

 # Updates settings relevant to the kubelet after TLS bootstrap"
 kubeadm init phase kubelet-finalize all --config

Options

-h, --help

help for kubelet-finalize

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Run all kubelet-finalize phases

kubeadm init phase kubelet-finalize all [flags]

Examples

 # Updates settings relevant to the kubelet after TLS bootstrap"
 kubeadm init phase kubelet-finalize all --config

Options

--cert-dir string Default: "/etc/kubernetes/pki"

The path where to save and store the certificates.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Enable kubelet client certificate rotation

kubeadm init phase kubelet-finalize enable-client-cert-rotation [flags]

Options

--cert-dir string Default: "/etc/kubernetes/pki"

The path where to save and store the certificates.

Mon, 01 Jan 0001 00:00:00 +0000

Write kubelet settings and (re)start the kubelet

Synopsis

Write a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)start kubelet.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Mark a node as a control-plane

kubeadm init phase mark-control-plane [flags]

Examples

 # Applies control-plane label and taint to the current node, functionally equivalent to what executed by kubeadm init.
 kubeadm init phase mark-control-plane --config config.yaml
 
 # Applies control-plane label and taint to a specific node
 kubeadm init phase mark-control-plane --node-name myNode

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Run pre-flight checks for kubeadm init.

kubeadm init phase preflight [flags]

Examples

 # Run pre-flight checks for kubeadm init using a config file.
 kubeadm init phase preflight --config kubeadm-config.yaml

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Show the join command for control-plane and worker node

kubeadm init phase show-join-command [flags]

Options

-h, --help

help for show-join-command

Mon, 01 Jan 0001 00:00:00 +0000

Upload certificates to kubeadm-certs

Synopsis

Upload control plane certificates to the kubeadm-certs Secret

kubeadm init phase upload-certs [flags]

Options

--certificate-key string

Key used to encrypt the control-plane certificates in the kubeadm-certs Secret. The certificate key is a hex encoded string that is an AES key of size 32 bytes.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload the kubeadm and kubelet configuration to a ConfigMap

kubeadm init phase upload-config [flags]

Options

-h, --help

help for upload-config

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload all configuration to a config map

kubeadm init phase upload-config all [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload the kubeadm ClusterConfiguration to a ConfigMap called kubeadm-config in the kube-system namespace. This enables correct configuration of system components and a seamless user experience when upgrading.

Mon, 01 Jan 0001 00:00:00 +0000

Upload the kubelet component config to a ConfigMap

Synopsis

Upload the kubelet configuration extracted from the kubeadm InitConfiguration object to a kubelet-config ConfigMap in the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Wait for the control plane to start

kubeadm init phase wait-control-plane [flags]

Options

-h, --help

help for wait-control-plane

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Use this command to invoke single phase of the "join" workflow

kubeadm join phase [flags]

Options

-h, --help

help for phase

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Join a machine as a control plane instance

kubeadm join phase control-plane-join [flags]

Examples

 # Joins a machine as a control plane instance
 kubeadm join phase control-plane-join all

Options

-h, --help

help for control-plane-join

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Join a machine as a control plane instance

kubeadm join phase control-plane-join all [flags]

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Add a new local etcd member

kubeadm join phase control-plane-join etcd [flags]

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Mark a node as a control-plane

kubeadm join phase control-plane-join mark-control-plane [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Prepare the machine for serving a control plane

kubeadm join phase control-plane-prepare [flags]

Examples

 # Prepares the machine for serving a control plane
 kubeadm join phase control-plane-prepare all

Options

-h, --help

help for control-plane-prepare

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Prepare the machine for serving a control plane

kubeadm join phase control-plane-prepare all [api-server-endpoint] [flags]

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate the certificates for the new control plane components

kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags]

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate the manifests for the new control plane components

kubeadm join phase control-plane-prepare control-plane [flags]

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Download certificates shared among control-plane nodes from the kubeadm-certs Secret

kubeadm join phase control-plane-prepare download-certs [api-server-endpoint] [flags]

Options

--certificate-key string

Use this key to decrypt the certificate secrets uploaded by init. The certificate key is a hex encoded string that is an AES key of size 32 bytes.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Generate the kubeconfig for the new control plane components

kubeadm join phase control-plane-prepare kubeconfig [api-server-endpoint] [flags]

Options

--certificate-key string

Use this key to decrypt the certificate secrets uploaded by init. The certificate key is a hex encoded string that is an AES key of size 32 bytes.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Join etcd for control plane nodes

kubeadm join phase etcd-join [flags]

Examples

 # Joins etcd for a control plane instance
 kubeadm join phase control-plane-join-etcd all

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Write kubelet settings, certificates and (re)start the kubelet

Synopsis

Write a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)start kubelet.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Wait for the kubelet to bootstrap itself

kubeadm join phase kubelet-wait-bootstrap [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Run join pre-flight checks

Synopsis

Run pre-flight checks for kubeadm join.

kubeadm join phase preflight [api-server-endpoint] [flags]

Examples

 # Run join pre-flight checks using a config file.
 kubeadm join phase preflight --config kubeadm-config.yaml

Options

--apiserver-advertise-address string

If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Wait for the control plane to start

kubeadm join phase wait-control-plane [flags]

Options

-h, --help

help for wait-control-plane

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Output a kubeconfig file for an additional user.

kubeadm kubeconfig user [flags]

Examples

 # Output a kubeconfig file for an additional user named foo
 kubeadm kubeconfig user --client-name=foo
 
 # Output a kubeconfig file for an additional user named foo using a kubeadm config file bar
 kubeadm kubeconfig user --client-name=foo --config=bar

Options

--client-name string

The name of user. It will be used as the CN if client certificates are created

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Use this command to invoke single phase of the "reset" workflow

kubeadm reset phase [flags]

Options

-h, --help

help for phase

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Run cleanup node.

kubeadm reset phase cleanup-node [flags]

Options

--cert-dir string Default: "/etc/kubernetes/pki"

The path to the directory where the certificates are stored. If specified, clean this directory.

Mon, 01 Jan 0001 00:00:00 +0000

Run reset pre-flight checks

Synopsis

Run pre-flight checks for kubeadm reset.

kubeadm reset phase preflight [flags]

Options

--dry-run

Don't apply any changes; just output what would be done.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Remove a local etcd member for a control plane node.

kubeadm reset phase remove-etcd-member [flags]

Options

--dry-run

Don't apply any changes; just output what would be done.

Mon, 01 Jan 0001 00:00:00 +0000

Create bootstrap tokens on the server

Synopsis

This command will create a bootstrap token for you. You can specify the usages for this token, the "time to live" and an optional human friendly description.

Mon, 01 Jan 0001 00:00:00 +0000

Delete bootstrap tokens on the server

Synopsis

This command will delete a list of bootstrap tokens for you.

Upgrade the kubelet configuration for this node by downloading it from the kubelet-config ConfigMap stored in the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Run post upgrade tasks

kubeadm upgrade apply phase post-upgrade [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Run preflight checks before upgrade

kubeadm upgrade apply phase preflight [flags]

Options

--allow-experimental-upgrades

Show unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload the kubeadm and kubelet configurations to ConfigMaps

kubeadm upgrade apply phase upload-config [flags]

Options

-h, --help

help for upload-config

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload all the configurations to ConfigMaps

kubeadm upgrade apply phase upload-config all [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload the kubeadm ClusterConfiguration to a ConfigMap

kubeadm upgrade apply phase upload-config kubeadm [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Upload the kubelet configuration to a ConfigMap

kubeadm upgrade apply phase upload-config kubelet [flags]

Synopsis

Run post upgrade tasks

kubeadm upgrade node phase post-upgrade [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Run upgrade node pre-flight checks

Synopsis

Run pre-flight checks for kubeadm upgrade node.

kubeadm upgrade node phase preflight [flags]

Options

--config string

Path to a kubeadm configuration file.

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. This command can only run on the control plane nodes where the kubeconfig file "admin.conf" exists. To skip the internet check, pass in the optional [version] parameter.

Mon, 01 Jan 0001 00:00:00 +0000

All files in this directory are auto-generated from other repos. Do not edit them manually. You must edit them in their upstream repo.

Mon, 01 Jan 0001 00:00:00 +0000

To run the tests for a localization, use the following command:

go test k8s.io/website/content/<lang>/examples

where <lang> is the two character representation of a language. For example:

go test k8s.io/website/content/en/examples

Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

Auditing allows cluster administrators to answer the following questions:

what happened?
when did it happen?
who initiated it?
on what did it happen?
where was it observed?
from where was it initiated?
to where was it going?

ExecCredential is used by exec-based plugins to communicate credentials to HTTP transports.

Field	Description
`apiVersion` string	`client.authentication.k8s.io/v1`
`kind` string	`ExecCredential`
`spec` [Required] `ExecCredentialSpec`	Spec holds information passed to the plugin by the transport.
`status` `ExecCredentialStatus`	Status is filled in by the plugin and holds the credentials that the transport should use to contact the API.

`Cluster`

Appears in:

ExecCredentialSpec

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to.

Client Authentication (v1beta1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

ExecCredential

`ExecCredential`

ExecCredential is used by exec-based plugins to communicate credentials to HTTP transports.

Field	Description
`apiVersion` string	`client.authentication.k8s.io/v1beta1`
`kind` string	`ExecCredential`
`spec` [Required] `ExecCredentialSpec`	Spec holds information passed to the plugin by the transport.
`status` `ExecCredentialStatus`	Status is filled in by the plugin and holds the credentials that the transport should use to contact the API.

`Cluster`

Appears in:

ExecCredentialSpec

Cluster contains information to allow an exec plugin to communicate with the kubernetes cluster being authenticated to.

Configure Certificate Rotation for the Kubelet

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to enable and configure certificate rotation for the kubelet.

FEATURE STATE: Kubernetes v1.19 [stable]

Before you begin

Kubernetes version 1.8.0 or later is required

Overview

The kubelet uses certificates for authenticating to the Kubernetes API. By default, these certificates are issued with one year expiration so that they do not need to be renewed too frequently.

Kubernetes contains kubelet certificate rotation, that will automatically generate a new key and request a new certificate from the Kubernetes API as the current certificate approaches expiration. Once the new certificate is available, it will be used for authenticating connections to the Kubernetes API.

Contribute to Kubernetes Documentation

Mon, 01 Jan 0001 00:00:00 +0000

This website is maintained by Kubernetes SIG Docs. The Kubernetes project welcomes help from all contributors, new or experienced!

Kubernetes documentation contributors:

Improve existing content
Create new content
Translate the documentation
Manage and publish the documentation parts of the Kubernetes release cycle

The blog team, part of SIG Docs, helps manage the official blogs. Read contributing to Kubernetes blogs to learn more.

Note:

To learn more about contributing to Kubernetes in general, see the general contributor documentation site.

Getting started

Anyone can open an issue about documentation, or contribute a change with a pull request (PR) to the kubernetes/website GitHub repository. You need to be comfortable with git and GitHub to work effectively in the Kubernetes community.

Crowdfire Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

Crowdfire helps content creators create their content anywhere on the Internet and publish it everywhere else in the right format. Since its launch in 2010, it has grown to 16 million users. The product began as a monolith app running on Google App Engine, and in 2015, the company began a transformation to microservices running on Amazon Web Services Elastic Beanstalk. "It was okay for our use cases initially, but as the number of services, development teams and scale increased, the deploy times, self-healing capabilities and resource utilization started to become problems for us," says Software Engineer Amanpreet Singh, who leads the infrastructure team for Crowdfire.

DaoCloud Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenges

DaoCloud, founded in 2014, is an innovation leader in the field of cloud native. It boasts independent intellectual property rights of core technologies for crafting an open cloud platform to empower the digital transformation of enterprises.

DaoCloud has been engaged in cloud native since its inception. As containerization is crucial for cloud native business, a cloud platform that does not have containers as infrastructure is unlikely to attract its potential users. Therefore, the first challenge confronting DaoCloud is how to efficiently manage and schedule numerous containers while maintaining stable connectivity between them.

Debug Running Pods

Mon, 01 Jan 0001 00:00:00 +0000

This page explains how to debug Pods running (or crashing) on a Node.

Before you begin

Your Pod should already be scheduled and running. If your Pod is not yet running, start with Debugging Pods.
For some of the advanced debugging steps you need to know on which Node the Pod is running and have shell access to run commands on that Node. You don't need that access to run the standard debug steps that use kubectl.

Using `kubectl describe pod` to fetch details about pods

For this example we'll use a Deployment to create two pods, similar to the earlier example.

Debugging Kubernetes Nodes With Kubectl

Mon, 01 Jan 0001 00:00:00 +0000

This page shows how to debug a node running on the Kubernetes cluster using kubectl debug command.

Before you begin

Developing and debugging services locally using telepresence

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to get a shell on a running container in order to run debugging tools.

Docs smoke test page

Mon, 01 Jan 0001 00:00:00 +0000

This page serves two purposes:

Demonstrate how the Kubernetes documentation uses Markdown
Provide a "smoke test" document we can use to test HTML, CSS, and template changes that affect the overall documentation.

Heading levels

The above heading is an H2. The page title renders as an H1. The following sections show H3 - H6.

### H3

This is in an H3 section.

#### H4

This is in an H4 section.

##### H5

This is in an H5 section.

###### H6

This is in an H6 section.

Produces:

Download Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes ships binaries for each component as well as a standard set of client applications to bootstrap or interact with a cluster. Components like the API server are capable of running within container images inside of a cluster. Those components are also shipped in container images as part of the official release process. All binaries as well as container images are available for multiple operating systems as well as hardware architectures.

Event Rate Limit Configuration (v1alpha1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Configuration

`Configuration`

Configuration provides configuration for the EventRateLimit admission controller.

Field	Description
`apiVersion` string	`eventratelimit.admission.k8s.io/v1alpha1`
`kind` string	`Configuration`
`limits` [Required] `[]Limit`	limits are the limits to place on event queries received. Limits can be placed on events received server-wide, per namespace, per user, and per source+object. At least one limit is required.

`Limit`

Appears in:

Configuration

Limit is the configuration for a particular limit type

Field Description

type [Required]
LimitType

Resource Types

ImageReview

`ImageReview`

ImageReview checks if the set of images in a pod are allowed.

Field	Description
`apiVersion` string	`imagepolicy.k8s.io/v1alpha1`
`kind` string	`ImageReview`
`metadata` `meta/v1.ObjectMeta`	Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` [Required] `ImageReviewSpec`	Spec holds information about the pod being evaluated
`status` `ImageReviewStatus`	Status is filled in by the backend and indicates whether the pod should be allowed.

`ImageReviewContainerSpec`

Appears in:

ImageReviewSpec

ImageReviewContainerSpec is a description of a container within the pod creation request.

JD.com Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

With more than 300 million active users and total 2017 revenue of more than $55 billion, JD.com is China's largest retailer, and its operations are the epitome of hyperscale. For example, there are more than a trillion images in JD.com's product databases—with 100 million being added daily—and this enormous amount of data needs to be instantly accessible. In 2014, JD.com moved its applications to containers running on bare metal machines using OpenStack and Docker to "speed up the delivery of our computing resources and make the operations much simpler," says Haifeng Liu, JD.com's Chief Architect. But by the end of 2015, with tens of thousands of nodes running in multiple data centers, "we encountered a lot of problems because our platform was not strong enough, and we suffered from bottlenecks and scalability issues," says Liu. "We needed infrastructure for the next five years of development, now."

kube-apiserver Admission (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

AdmissionReview

`AdmissionReview`

AdmissionReview describes an admission review request/response.

Field	Description
`apiVersion` string	`admission.k8s.io/v1`
`kind` string	`AdmissionReview`
`request` `AdmissionRequest`	request describes the attributes for the admission request.
`response` `AdmissionResponse`	response describes the attributes for the admission response.

`AdmissionRequest`

Appears in:

AdmissionReview

AdmissionRequest describes the admission.Attributes for the admission request.

Field Description

uid [Required]
k8s.io/apimachinery/pkg/types.UID

uid is an identifier for the individual request/response. It allows us to distinguish instances of requests which are otherwise identical (parallel requests, requests when earlier requests did not modify etc) The UID is meant to track the round trip (request/response) between the KAS and the WebHook, not the user request. It is suitable for correlating log entries between the webhook and apiserver, for either auditing or debugging.

kube-apiserver Audit Configuration (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Event
EventList
Policy
PolicyList

`Event`

Appears in:

EventList

Event captures all the information that can be included in an API audit log.

Field Description

apiVersion
string audit.k8s.io/v1

kind
string Event

level [Required]
Level

AuditLevel at which event was generated

auditID [Required]
k8s.io/apimachinery/pkg/types.UID

Unique audit ID, generated for each request.

stage [Required]
Stage

Stage of the request handling when this event instance was generated.

requestURI [Required]
string

RequestURI is the request URI as sent by the client to a server.

kube-apiserver Configuration (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1 is the v1 version of the API.

Resource Types

AdmissionConfiguration
AuthenticationConfiguration
AuthorizationConfiguration
EncryptionConfiguration
TracingConfiguration

`TracingConfiguration`

Appears in:

KubeletConfiguration
TracingConfiguration
TracingConfiguration
TracingConfiguration

TracingConfiguration provides versioned configuration for OpenTelemetry tracing clients.

Field Description

endpoint
string

Endpoint of the collector this component will report traces to. The connection is insecure, and does not currently support TLS. Recommended is unset, and endpoint is the otlp grpc default, localhost:4317.

samplingRatePerMillion
int32

SamplingRatePerMillion is the number of samples to collect per million spans. Recommended is unset. If unset, sampler respects its parent span's sampling rate, but otherwise never samples.

kube-apiserver Configuration (v1alpha1)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1alpha1 is the v1alpha1 version of the API.

Resource Types

AdmissionConfiguration
AuthenticationConfiguration
AuthorizationConfiguration
EgressSelectorConfiguration
TracingConfiguration

`TracingConfiguration`

Appears in:

KubeletConfiguration
TracingConfiguration

TracingConfiguration provides versioned configuration for OpenTelemetry tracing clients.

Field Description

endpoint
string

samplingRatePerMillion
int32

SamplingRatePerMillion is the number of samples to collect per million spans. Recommended is unset. If unset, sampler respects its parent span's sampling rate, but otherwise never samples.

kube-apiserver Configuration (v1beta1)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1beta1 is the v1beta1 version of the API.

Resource Types

AuthenticationConfiguration
AuthorizationConfiguration
EgressSelectorConfiguration
TracingConfiguration

`TracingConfiguration`

Appears in:

KubeletConfiguration
TracingConfiguration
TracingConfiguration

TracingConfiguration provides versioned configuration for OpenTelemetry tracing clients.

Field Description

endpoint
string

samplingRatePerMillion
int32

SamplingRatePerMillion is the number of samples to collect per million spans. Recommended is unset. If unset, sampler respects its parent span's sampling rate, but otherwise never samples.

kube-controller-manager Configuration (v1alpha1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

CloudControllerManagerConfiguration
LeaderMigrationConfiguration
KubeControllerManagerConfiguration

`ClientConnectionConfiguration`

Appears in:

KubeSchedulerConfiguration
GenericControllerManagerConfiguration

ClientConnectionConfiguration contains details for constructing a client.

Field Description

kubeconfig [Required]
string

kubeconfig is the path to a KubeConfig file.

acceptContentTypes [Required]
string

acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the default value of 'application/json'. This field will control all connections to the server used by a particular client.

contentType [Required]
string

contentType is the content type used when sending data to the server from this client.

kube-proxy Configuration (v1alpha1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

KubeProxyConfiguration

`FormatOptions`

Appears in:

LoggingConfiguration

FormatOptions contains options for the different logging formats.

Field	Description
`text` [Required] `TextOptions`	[Alpha] Text contains options for logging format "text". Only available when the LoggingAlphaOptions feature gate is enabled.
`json` [Required] `JSONOptions`	[Alpha] JSON contains options for logging format "json". Only available when the LoggingAlphaOptions feature gate is enabled.

`JSONOptions`

Appears in:

FormatOptions

JSONOptions contains options for logging format "json".

Field	Description
`OutputRoutingOptions` [Required] `OutputRoutingOptions`	(Members of `OutputRoutingOptions` are embedded into this type.) No description provided.

`LogFormatFactory`

LogFormatFactory provides support for a certain additional, non-default log format.

kube-scheduler Configuration (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

DefaultPreemptionArgs
DynamicResourcesArgs
InterPodAffinityArgs
KubeSchedulerConfiguration
NodeAffinityArgs
NodeResourcesBalancedAllocationArgs
NodeResourcesFitArgs
PodTopologySpreadArgs
VolumeBindingArgs

`ClientConnectionConfiguration`

Appears in:

KubeSchedulerConfiguration

ClientConnectionConfiguration contains details for constructing a client.

Field Description

kubeconfig [Required]
string

kubeconfig is the path to a KubeConfig file.

acceptContentTypes [Required]
string

kubeadm Configuration (v1beta3)

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Package v1beta3 defines the v1beta3 version of the kubeadm configuration file format. This version improves on the v1beta2 format by fixing some minor issues and adding a few new fields.

A list of changes since v1beta2:

The deprecated "ClusterConfiguration.useHyperKubeImage" field has been removed. Kubeadm no longer supports the hyperkube image.
The "ClusterConfiguration.dns.type" field has been removed since CoreDNS is the only supported DNS server type by kubeadm.
Include "datapolicy" tags on the fields that hold secrets. This would result in the field values to be omitted when API structures are printed with klog.
Add "InitConfiguration.skipPhases", "JoinConfiguration.skipPhases" to allow skipping a list of phases during kubeadm init/join command execution.
Add "InitConfiguration.nodeRegistration.imagePullPolicy" and "JoinConfiguration.nodeRegistration.imagePullPolicy" to allow specifying the images pull policy during kubeadm "init" and "join". The value must be one of "Always", "Never" or "IfNotPresent". "IfNotPresent" is the default, which has been the existing behavior prior to this addition.
Add "InitConfiguration.patches.directory", "JoinConfiguration.patches.directory" to allow the user to configure a directory from which to take patches for components deployed by kubeadm.
Move the BootstrapToken* API and related utilities out of the "kubeadm" API group to a new group "bootstraptoken". The kubeadm API version v1beta3 no longer contains the BootstrapToken* structures.

Migration from old kubeadm config versions

kubeadm Configuration (v1beta4)

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Package v1beta4 defines the v1beta4 version of the kubeadm configuration file format. This version improves on the v1beta3 format by fixing some minor issues and adding a few new fields.

A list of changes since v1beta3:

v1.35:

Add httpEndpoints field to ClusterConfiguration.etcd.externalEtcd that can be used to configure the HTTP endpoints for etcd communication in v1beta4. This field is used to separate the HTTP traffic (such as /metrics and /health endpoints) from the gRPC traffic handled by endpoints. This separation allows for better access control, as HTTP endpoints can be exposed without exposing the primary gRPC interface. Corresponds to etcd's --listen-client-http-urls configuration. If not provided, endpoints will be used for both gRPC and HTTP traffic.

v1.34:

kubeconfig (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Config

`Config`

Config holds the information needed to build connect to remote kubernetes clusters as a given user

Field Description

apiVersion
string /v1

kind
string Config

kind
string

Legacy field from pkg/api/types.go TypeMeta. TODO(jlowdermilk): remove this after eliminating downstream dependencies.

apiVersion
string

Legacy field from pkg/api/types.go TypeMeta. TODO(jlowdermilk): remove this after eliminating downstream dependencies.

preferences,omitzero [Required]
Preferences

Preferences holds general information to be use for cli interactions Deprecated: this field is deprecated in v1.34. It is not used by any of the Kubernetes components.

Kubelet Configuration (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

CredentialProviderConfig

`CredentialProviderConfig`

CredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.

Field Description

apiVersion
string kubelet.config.k8s.io/v1

kind
string CredentialProviderConfig

providers [Required]
[]CredentialProvider

providers is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is attempted first.

Kubelet Configuration (v1alpha1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

CredentialProviderConfig
ImagePullIntent
ImagePulledRecord

`CredentialProviderConfig`

Field Description

apiVersion
string kubelet.config.k8s.io/v1alpha1

kind
string CredentialProviderConfig

providers [Required]
[]CredentialProvider

Kubelet Configuration (v1beta1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

CredentialProviderConfig
ImagePullIntent
ImagePulledRecord
KubeletConfiguration
SerializedNodeConfigSource

`FormatOptions`

Appears in:

LoggingConfiguration

FormatOptions contains options for the different logging formats.

Field	Description
`text` [Required] `TextOptions`	[Alpha] Text contains options for logging format "text". Only available when the LoggingAlphaOptions feature gate is enabled.
`json` [Required] `JSONOptions`	[Alpha] JSON contains options for logging format "json". Only available when the LoggingAlphaOptions feature gate is enabled.

`JSONOptions`

Appears in:

FormatOptions

JSONOptions contains options for logging format "json".

Field	Description
`OutputRoutingOptions` [Required] `OutputRoutingOptions`	(Members of `OutputRoutingOptions` are embedded into this type.) No description provided.

`LogFormatFactory`

LogFormatFactory provides support for a certain additional, non-default log format.

Kubelet CredentialProvider (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

CredentialProviderRequest
CredentialProviderResponse

`CredentialProviderRequest`

CredentialProviderRequest includes the image that the kubelet requires authentication for. Kubelet will pass this request object to the plugin via stdin. In general, plugins should prefer responding with the same apiVersion they were sent.

Field Description

apiVersion
string credentialprovider.kubelet.k8s.io/v1

kind
string CredentialProviderRequest

image [Required]
string

image is the container image that is being pulled as part of the credential provider plugin request. Plugins may optionally parse the image to extract any information required to fetch credentials.

kuberc (v1alpha1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Preference

`Preference`

Preference stores elements of KubeRC configuration file

Field Description

apiVersion
string kubectl.config.k8s.io/v1alpha1

kind
string Preference

overrides [Required]
[]CommandDefaults

overrides allows changing default flag values of commands. This is especially useful, when user doesn't want to explicitly set flags each time.

aliases [Required]
[]AliasOverride

aliases allow defining command aliases for existing kubectl commands, with optional default flag values. If the alias name collides with a built-in command, built-in command always takes precedence. Flag overrides defined in the overrides section do NOT apply to aliases for the same command. kubectl [ALIAS NAME] [USER_FLAGS] [USER_EXPLICIT_ARGS] expands to kubectl [COMMAND] # built-in command alias points to [KUBERC_PREPEND_ARGS] [USER_FLAGS] [KUBERC_FLAGS] # rest of the flags that are not passed by user in [USER_FLAGS] [USER_EXPLICIT_ARGS] [KUBERC_APPEND_ARGS] e.g.

kuberc (v1beta1)

Mon, 01 Jan 0001 00:00:00 +0000

Resource Types

Preference

`Preference`

Preference stores elements of KubeRC configuration file

Field Description

apiVersion
string kubectl.config.k8s.io/v1beta1

kind
string Preference

defaults [Required]
[]CommandDefaults

defaults allow changing default option values of commands. This is especially useful, when user doesn't want to explicitly set options each time.

aliases [Required]
[]AliasOverride

aliases allow defining command aliases for existing kubectl commands, with optional default option values. If the alias name collides with a built-in command, built-in command always takes precedence. Option overrides defined in the defaults section do NOT apply to aliases for the same command. kubectl [ALIAS NAME] [USER_OPTIONS] [USER_EXPLICIT_ARGS] expands to kubectl [COMMAND] # built-in command alias points to [KUBERC_PREPEND_ARGS] [USER_OPTIONS] [KUBERC_OPTIONS] # rest of the options that are not passed by user in [USER_OPTIONS] [USER_EXPLICIT_ARGS] [KUBERC_APPEND_ARGS] e.g.

Kubernetes Community Code of Conduct

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes follows the CNCF Code of Conduct. The text of the CNCF CoC is replicated below, as of commit 71412bb02.

CNCF Community Code of Conduct v1.3

Community Code of Conduct

Kubernetes Custom Metrics (v1beta2)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1beta2 is the v1beta2 version of the custom_metrics API.

Resource Types

MetricListOptions
MetricValue
MetricValueList

`MetricListOptions`

MetricListOptions is used to select metrics by their label selectors

Field	Description
`apiVersion` string	`custom.metrics.k8s.io/v1beta2`
`kind` string	`MetricListOptions`
`labelSelector` `string`	A selector to restrict the list of returned objects by their labels. Defaults to everything.
`metricLabelSelector` `string`	A selector to restrict the list of returned metrics by their labels

`MetricValue`

Appears in:

MetricValueList

MetricValue is the metric value for some object

Kubernetes Default ServiceCIDR Reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.33 [stable](enabled by default)

This document shares how to reconfigure the default Service IP range(s) assigned to a cluster.

Before you begin

Kubernetes External Metrics (v1beta1)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1beta1 is the v1beta1 version of the external metrics API.

Resource Types

ExternalMetricValue
ExternalMetricValueList

`ExternalMetricValue`

Appears in:

ExternalMetricValueList

ExternalMetricValue is a metric value for external metric A single metric value is identified by metric name and a set of string labels. For one metric there can be multiple values with different sets of labels.

Field Description

apiVersion
string external.metrics.k8s.io/v1beta1

kind
string ExternalMetricValue

metricName [Required]
string

the name of the metric

metricLabels [Required]
map[string]string

a set of labels that identify a single time series for the metric

Kubernetes Metrics (v1beta1)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1beta1 is the v1beta1 version of the metrics API.

Resource Types

NodeMetrics
NodeMetricsList
PodMetrics
PodMetricsList

`NodeMetrics`

Appears in:

NodeMetricsList

NodeMetrics sets resource usage metrics of a node.

Field Description

apiVersion
string metrics.k8s.io/v1beta1

kind
string NodeMetrics

metadata
meta/v1.ObjectMeta

Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

Refer to the Kubernetes API documentation for the fields of the metadata field.

timestamp [Required]
meta/v1.Time

The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp].

window [Required]
meta/v1.Duration No description provided.

usage [Required]
core/v1.ResourceList

The memory usage is the memory working set.

Kubernetes Metrics Reference

Mon, 01 Jan 0001 00:00:00 +0000

Metrics (v1.35)

This page details the metrics that different Kubernetes components export. You can query the metrics endpoint for these components using an HTTP scrape, and fetch the current metrics data in Prometheus format.

List of Stable Kubernetes Metrics

Stable metrics observe strict API contracts and no labels can be added or removed from stable metrics during their lifetime.

apiserver_admission_controller_admission_duration_seconds

Admission controller latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).

Stability Level:STABLE
Type: Histogram
Labels:nameoperationrejectedtype

apiserver_admission_step_admission_duration_seconds

Admission sub-step latency histogram in seconds, broken out for each operation and API resource and step type (validate or admit).

Stability Level:STABLE
Type: Histogram
Labels:operationrejectedtype

apiserver_admission_webhook_admission_duration_seconds

Admission webhook latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).

Stability Level:STABLE
Type: Histogram
Labels:nameoperationrejectedtype

apiserver_current_inflight_requests

Maximal number of currently used inflight request limit of this apiserver per request kind in last second.

Stability Level:STABLE
Type: Gauge
Labels:request_kind

apiserver_longrunning_requests

Gauge of all active long-running apiserver requests broken out by verb, group, version, resource, scope and component. Not all requests are tracked this way.

Stability Level:STABLE
Type: Gauge
Labels:componentgroupresourcescopesubresourceverbversion

apiserver_request_duration_seconds

Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.

Stability Level:STABLE
Type: Histogram
Labels:componentdry_rungroupresourcescopesubresourceverbversion

apiserver_request_total

Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.

Stability Level:STABLE
Type: Counter
Labels:codecomponentdry_rungroupresourcescopesubresourceverbversion

apiserver_requested_deprecated_apis

Gauge of deprecated APIs that have been requested, broken out by API group, version, resource, subresource, and removed_release.

Stability Level:STABLE
Type: Gauge
Labels:groupremoved_releaseresourcesubresourceversion

apiserver_response_sizes

Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.

Stability Level:STABLE
Type: Histogram
Labels:componentgroupresourcescopesubresourceverbversion

apiserver_storage_objects

[DEPRECATED, consider using apiserver_resource_objects instead] Number of stored objects at the time of last check split by kind. In case of a fetching error, the value will be -1.

Stability Level:STABLE
Type: Gauge
Labels:resource
Deprecated Versions:1.34.0

apiserver_storage_size_bytes

Size of the storage database file physically allocated in bytes.

Stability Level:STABLE
Type: Custom
Labels:storage_cluster_id

container_cpu_usage_seconds_total

Cumulative cpu time consumed by the container in core-seconds

Stability Level:STABLE
Type: Custom
Labels:containerpodnamespace

container_memory_working_set_bytes

Current working set of the container in bytes

Stability Level:STABLE
Type: Custom
Labels:containerpodnamespace

container_start_time_seconds

Start time of the container since unix epoch in seconds

Stability Level:STABLE
Type: Custom
Labels:containerpodnamespace

cronjob_controller_job_creation_skew_duration_seconds

Time between when a cronjob is scheduled to be run, and when the corresponding job is created

Stability Level:STABLE
Type: Histogram

job_controller_job_pods_finished_total

The number of finished Pods that are fully tracked

Stability Level:STABLE
Type: Counter
Labels:completion_moderesult

job_controller_job_sync_duration_seconds

The time it took to sync a job

Stability Level:STABLE
Type: Histogram
Labels:actioncompletion_moderesult

job_controller_job_syncs_total

The number of job syncs

Stability Level:STABLE
Type: Counter
Labels:actioncompletion_moderesult

job_controller_jobs_finished_total

The number of finished jobs

Stability Level:STABLE
Type: Counter
Labels:completion_modereasonresult

kube_pod_resource_limit

Resources limit for workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.

Stability Level:STABLE
Type: Custom
Labels:namespacepodnodeschedulerpriorityresourceunit

kube_pod_resource_request

Resources requested by workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.

Stability Level:STABLE
Type: Custom
Labels:namespacepodnodeschedulerpriorityresourceunit

kubernetes_healthcheck

This metric records the result of a single healthcheck.

Stability Level:STABLE
Type: Gauge
Labels:nametype

kubernetes_healthchecks_total

This metric records the results of all healthcheck.

Stability Level:STABLE
Type: Counter
Labels:namestatustype

node_collector_evictions_total

Number of Node evictions that happened since current instance of NodeController started.

Stability Level:STABLE
Type: Counter
Labels:zone

node_cpu_usage_seconds_total

Cumulative cpu time consumed by the node in core-seconds

Stability Level:STABLE
Type: Custom

node_memory_working_set_bytes

Current working set of the node in bytes

Stability Level:STABLE
Type: Custom

pod_cpu_usage_seconds_total

Cumulative cpu time consumed by the pod in core-seconds

Stability Level:STABLE
Type: Custom
Labels:podnamespace

pod_memory_working_set_bytes

Current working set of the pod in bytes

Stability Level:STABLE
Type: Custom
Labels:podnamespace

resource_scrape_error

1 if there was an error while getting container metrics, 0 otherwise

Stability Level:STABLE
Type: Custom

scheduler_framework_extension_point_duration_seconds

Latency for running all plugins of a specific extension point.

Stability Level:STABLE
Type: Histogram
Labels:extension_pointprofilestatus

scheduler_pending_pods

Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulablePods that the scheduler attempted to schedule and failed; 'gated' is the number of unschedulable pods that the scheduler never attempted to schedule because they are gated.

Stability Level:STABLE
Type: Gauge
Labels:queue

scheduler_pod_scheduling_attempts

Number of attempts to successfully schedule a pod.

Stability Level:STABLE
Type: Histogram

scheduler_preemption_attempts_total

Total preemption attempts in the cluster till now

Stability Level:STABLE
Type: Counter

scheduler_preemption_victims

Number of selected preemption victims

Stability Level:STABLE
Type: Histogram

scheduler_queue_incoming_pods_total

Number of pods added to scheduling queues by event and queue type.

Stability Level:STABLE
Type: Counter
Labels:eventqueue

scheduler_schedule_attempts_total

Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.

Stability Level:STABLE
Type: Counter
Labels:profileresult

scheduler_scheduling_attempt_duration_seconds

Scheduling attempt latency in seconds (scheduling algorithm + binding)

Stability Level:STABLE
Type: Histogram
Labels:profileresult

List of Beta Kubernetes Metrics

Beta metrics observe a looser API contract than its stable counterparts. No labels can be removed from beta metrics during their lifetime, however, labels can be added while the metric is in the beta stage. This offers the assurance that beta metrics will honor existing dashboards and alerts, while allowing for amendments in the future.

Kubernetes Release Cycle

Mon, 01 Jan 0001 00:00:00 +0000

This content is auto-generated and links may not function. The source of the document is located here.

Targeting enhancements, Issues and PRs to Release Milestones

This document is focused on Kubernetes developers and contributors who need to create an enhancement, issue, or pull request which targets a specific release milestone.

TL;DR
Definitions
The Release Cycle
Removal Of Items From The Milestone
Adding An Item To The Milestone
Other Required Labels

The process for shepherding enhancements, issues, and pull requests into a Kubernetes release spans multiple stakeholders:

Release Managers

Mon, 01 Jan 0001 00:00:00 +0000

"Release Managers" is an umbrella term that encompasses the set of Kubernetes contributors responsible for maintaining release branches and creating releases by using the tools SIG Release provides.

The responsibilities of each role are described below.

Contact
- Security Embargo Policy
Handbooks
Release Managers
- Becoming a Release Manager
Release Manager Associates
- Becoming a Release Manager Associate
SIG Release Leads
- Chairs
- Technical Leads

Contact

Mailing List	Slack	Visibility	Usage	Membership
release-managers@kubernetes.io	#release-management (channel) / @release-managers (user group)	Public	Public discussion for Release Managers	All Release Managers (including Associates, and SIG Chairs)
release-managers-private@kubernetes.io	N/A	Private	Private discussion for privileged Release Managers	Release Managers, SIG Release leadership
security-release-team@kubernetes.io	#security-release-team (channel) / @security-rel-team (user group)	Private	Security release coordination with the Security Response Committee	security-discuss-private@kubernetes.io, release-managers-private@kubernetes.io

Security Embargo Policy

Some information about releases is subject to embargo and we have defined policy about how those embargoes are set. Please refer to the Security Embargo Policy for more information.

Notes

Mon, 01 Jan 0001 00:00:00 +0000

Release notes can be found by reading the Changelog that matches your Kubernetes version. View the changelog for 1.35 on GitHub.

Alternately, release notes can be searched and filtered online at: relnotes.k8s.io. View filtered release notes for 1.35 on relnotes.k8s.io.

ricardo.ch Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

A Swiss online marketplace, ricardo.ch was experiencing problems with velocity, as well as a "classic gap" between Development and Operations, with the two sides unable to work well together. "They wanted to, but they didn't have common ground," says Cedric Meury, Head of Platform Engineering. "This was one of the root causes that slowed us down." The company began breaking down the legacy monolith into microservices, and needed orchestration to support the new architecture in its own data centers—as well as bring together Dev and Ops.

Schedule GPUs

Mon, 01 Jan 0001 00:00:00 +0000

Mon, 01 Jan 0001 00:00:00 +0000

This document shares how to validate IPv4/IPv6 dual-stack enabled Kubernetes clusters.

Before you begin

Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide Kubernetes nodes with routable IPv4/IPv6 network interfaces)
A network plugin that supports dual-stack networking.
Dual-stack enabled cluster

Your Kubernetes server must be at or later than version v1.23.

To check the version, enter kubectl version.

Note:

While you can validate with an earlier version, the feature is only GA and officially supported since v1.23.

Validate addressing

Validate node addressing

Each dual-stack Node should have a single IPv4 block and a single IPv6 block allocated. Validate that IPv4/IPv6 Pod address ranges are configured by running the following command. Replace the sample node name with a valid dual-stack Node from your cluster. In this example, the Node's name is k8s-linuxpool1-34450317-0:

Validating Admission Policy

Mon, 01 Jan 0001 00:00:00 +0000

FEATURE STATE: Kubernetes v1.30 [stable]

This page provides an overview of Validating Admission Policy.

What is Validating Admission Policy?

Validating admission policies offer a declarative, in-process alternative to validating admission webhooks.

Validating admission policies use the Common Expression Language (CEL) to declare the validation rules of a policy. Validation admission policies are highly configurable, enabling policy authors to define policies that can be parameterized and scoped to resources as needed by cluster administrators.

Version Skew Policy

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the maximum version skew supported between various Kubernetes components. Specific cluster deployment tools may place additional restrictions on version skew.

Supported versions

Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology. For more information, see Kubernetes Release Versioning.

The Kubernetes project maintains release branches for the most recent three minor releases (1.35, 1.34, 1.33). Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.

vsco Case Study

Mon, 01 Jan 0001 00:00:00 +0000

Challenge

After moving from Rackspace to AWS in 2015, VSCO began building Node.js and Go microservices in addition to running its PHP monolith. The team containerized the microservices using Docker, but "they were all in separate groups of EC2 instances that were dedicated per service," says Melinda Lu, Engineering Manager for the Machine Learning Team. Adds Naveen Gattu, Senior Software Engineer on the Community Team: "That yielded a lot of wasted resources. We started looking for a way to consolidate and be more efficient in the AWS EC2 instances."

WebhookAdmission Configuration (v1)

Mon, 01 Jan 0001 00:00:00 +0000

Package v1 is the v1 version of the API.

Resource Types

WebhookAdmission

`WebhookAdmission`

WebhookAdmission provides configuration for the webhook admission controller.

Field	Description
`apiVersion` string	`apiserver.config.k8s.io/v1`
`kind` string	`WebhookAdmission`
`kubeConfigFile` [Required] `string`	KubeConfigFile is the path to the kubeconfig file.

Wikimedia Case Study

Mon, 01 Jan 0001 00:00:00 +0000

The non-profit Wikimedia Foundation operates some of the largest collaboratively edited reference projects in the world, including Wikipedia. To help users maintain and use wikis, it runs Wikimedia Tool Labs, a hosting environment for community developers working on tools and bots to help editors and other volunteers do their work, including reducing vandalism. The community around Wikimedia Tool Labs began forming nearly 10 years ago.

"Wikimedia Tool Labs is vital for making sure wikis all around the world work as well as they possibly can. Because it's grown organically for almost 10 years, it has become an extremely challenging environment and difficult to maintain. It's like a big ball of mud — you really can't see through it. With Kubernetes, we're simplifying the environment and making it easier for developers to build the tools that make wikis run better." — Yuvi Panda, operations engineer at Wikimedia Foundation and Wikimedia Tool Labs

Challenges

Simplify a complex, difficult-to-manage infrastructure
Allow developers to continue writing tools and bots using existing techniques

Why Kubernetes

Wikimedia Tool Labs chose Kubernetes because it can mimic existing workflows, while reducing complexity

Approach

Migrate old systems and a complex infrastructure to Kubernetes

Results

20 percent of web tools that account for more than 40 percent of web traffic now run on Kubernetes
A 25-node cluster that keeps up with each new Kubernetes release
Thousands of lines of old code have been deleted, thanks to Kubernetes

Using Kubernetes to provide tools for maintaining wikis

Wikimedia Tool Labs is run by a staff of four-and-a-half paid employees and two volunteers. The infrastructure didn't make it easy or intuitive for developers to build bots and other tools to make wikis work more easily. Yuvi says, "It's incredibly chaotic. We have lots of Perl and Bash duct tape on top of it. Everything is super fragile."

Windows debugging tips

Mon, 01 Jan 0001 00:00:00 +0000

Node-level troubleshooting

My Pods are stuck at "Container Creating" or restarting over and over

Ensure that your pause image is compatible with your Windows OS version. See Pause container to see the latest / recommended pause image and/or get more information.

Note:
If using containerd as your container runtime the pause image is specified in the plugins.plugins.cri.sandbox_image field of the of config.toml configuration file.
My pods show status as ErrImgPull or ImagePullBackOff