Image Pull Policy

The Always policy instructs the kubelet to pull the image from the container registry every time a pod is started or restarted. This is the default policy when:

• Using the :latest tag.
• The image tag is omitted.

Key Use Cases:

• Development & CI/CD: Ensures the absolute latest build is always used.
• Security: Guarantees the node always fetches an image, which can be combined with registry authentication and vulnerability scanning on pull.

Considerations:

• Increases deployment latency due to network I/O.
• Can cause version indeterminism if the :latest tag is mutable.
• Consumes more network bandwidth.

The IfNotPresent policy tells the kubelet to pull the image only if it is not already present on the node. This is the default policy for all images with a specific tag other than :latest.

Key Use Cases:

• Production Stability: Favors speed and reduces external dependencies by using cached images.
• Air-Gapped/Offline Environments: Essential for clusters with limited or no external registry access.
• Bandwidth Conservation: Minimizes network traffic to the registry.

Considerations:

• Nodes may run stale images if the cache is not manually cleared.
• Requires a separate image update process (e.g., node recycling) for cluster-wide rollouts of a new image with the same tag.

The Never policy instructs the kubelet to never pull an image. It will only start a container if the specified image is already present locally on the node.

Key Use Cases:

• Disconnected/High-Security Clusters: Where all container images are pre-loaded onto nodes via secure, offline processes.
• Testing with Custom Local Builds: Using images built directly on the node during development.
• Deterministic Binary Artifacts: Treating container images as immutable, pre-vetted binaries deployed via cluster management tooling.

Considerations:

• Pod scheduling is constrained to nodes that already have the image.
• Image updates require a deliberate, node-level orchestration process outside of standard Kubernetes deployments.

The image tag directly influences the default pull policy, creating a crucial link between image versioning and deployment behavior.

Default Behavior Matrix:

• Image with tag :latest or no tag: Defaults to Always.
• Image with any other tag (e.g., v1.2.3, sha256:abc123): Defaults to IfNotPresent.

Best Practices:

• Use Immutable Tags: Prefer cryptographically secure, immutable identifiers like SHA256 digests (e.g., myapp@sha256:abc123...). This combines the safety of a unique image with the speed of IfNotPresent.
• Avoid :latest in Production: The mutable :latest tag can lead to unpredictable rollouts and makes rollbacks difficult.
• Explicit Overrides: Always explicitly set the imagePullPolicy in production manifests to avoid reliance on default behaviors.

For Agent Deployment Observability, the pull policy is a key lever controlling rollout determinism and telemetry consistency.

Observability Implications:

• Rollout Speed: IfNotPresent leads to faster pod startups on nodes with cached images, improving perceived agent availability.
• Version Consistency: Using Always with mutable tags can cause a mixed-version deployment during a rollout, complicating performance benchmarking and anomaly detection.
• Failure Diagnosis: Pods stuck in ImagePullBackOff status indicate a failed pull due to policy Always or IfNotPresent with a missing image. This is a critical signal for deployment health.

Recommended Pattern for Agents:

1. Build agent images with immutable tags (Git SHA).
2. Set imagePullPolicy: IfNotPresent.
3. Use a DaemonSet with a RollingUpdate strategy.
4. Force a pull of the new image by deleting pods or using a tool that updates the pod template, causing new pods to be created with the new image reference.

The image pull policy interacts with several other core Kubernetes features for robust deployments.

Image Pull Secrets: The imagePullPolicy governs when to pull, while imagePullSecrets provide the credentials for pulling from private registries. Both must be correctly configured.

Node Selection & Affinity: The Never policy makes node image caching a critical scheduling constraint. You may need to use nodeName or nodeSelector to target pre-loaded nodes.

Pod Lifecycle: The pull attempt occurs during the container creation phase. Failures here are reflected in pod events and statuses (ErrImagePull, ImagePullBackOff).

Runtime Class: Advanced use cases may use different container runtimes (e.g., gVisor, Kata Containers) which all respect the standard imagePullPolicy.

Init Containers: Each init container can have its own imagePullPolicy, allowing flexible staging of prerequisite images.

A container image is a static, immutable file containing an application's code, runtime, system tools, libraries, and settings. It is the executable package used to instantiate a container. In Kubernetes, the spec.containers[].image field in a pod specification references this image, which the Image Pull Policy governs when to fetch.

• Key Layers: Base OS layer, dependency layers, application layer.
• Registries: Stored in repositories like Docker Hub, Google Container Registry (GCR), or Amazon ECR.
• Tags: Identifiers like myapp:v1.2.3 or myapp:latest; the tag influences pull policy behavior.

An image tag is an alphanumeric identifier appended to a container image name, specifying a particular version or variant (e.g., nginx:1.23-alpine). It is critical for Image Pull Policy logic:

• Mutable Tags (e.g., :latest): The same tag can point to different image digests over time. A policy of Always is often used to ensure the latest digest is pulled.
• Immutable Tags (e.g., :v1.2.3-abc123): The tag is permanently associated with a specific image digest. A policy of IfNotPresent is safe and efficient.
• Digest: The unique, immutable SHA256 hash of an image (e.g., nginx@sha256:abc123...). Referencing by digest bypasses tag semantics and is the most deterministic practice.

An image registry is a centralized service for storing, managing, and distributing container images. Kubernetes interacts with a registry to pull images based on the Image Pull Policy. Common registries include Docker Hub, Google Artifact Registry, and Azure Container Registry.

• Authentication: Pods often need imagePullSecrets to authenticate with private registries.
• Availability: Registry downtime can cause pod startup failures if the policy requires a pull.
• Mirrors & Caches: Organizations use local registry mirrors or caches (e.g., Harbor, Nexus) to improve pull performance, reduce external dependencies, and enforce security scanning. The pull policy interacts with these local endpoints.

The node image cache is a local store on each Kubernetes worker node where pulled container images are retained. The Image Pull Policy (IfNotPresent or Never) directly leverages this cache to avoid redundant network transfers.

• Cache Eviction: Nodes automatically garbage collect unused images based on disk pressure. This can cause subsequent pod startups to re-pull an image even with IfNotPresent.
• Performance: Using IfNotPresent for large, stable images significantly speeds up pod scheduling and scaling events.
• Security Consideration: A cached image might contain vulnerabilities that are patched in a newer version in the registry. Policies like Always or periodic forced rollouts are used to ensure updates.

An Image Pull Secret is a Kubernetes Secret of type kubernetes.io/dockerconfigjson that stores credentials for authenticating to a private container image registry. It is mounted into pods (via spec.imagePullSecrets) to allow the kubelet to pull images when the Image Pull Policy is invoked.

• Scope: Typically defined at the Pod or ServiceAccount level.
• Failure Mode: If a secret is missing or has invalid credentials, an image pull will fail, preventing pod startup.
• Best Practice: Use a dedicated service account with attached image pull secrets rather than defining them in individual pod specs.

An immutable image reference is a method of specifying a container image using its cryptographically secure digest (SHA256 hash) instead of a tag (e.g., nginx@sha256:abc123...). This practice guarantees the exact same binary artifact is deployed every time, making the Image Pull Policy largely irrelevant for version consistency.

• Determinism: Eliminates the risk of a mutable tag (:latest) pointing to a different, potentially broken, image.
• Policy Implication: With an immutable digest, IfNotPresent is sufficient and optimal, as the content can never change. The pull will only occur if the image is not already cached on the node.
• Deployment Integration: CI/CD pipelines must resolve the latest build's digest and update the deployment manifest, enabling full traceability from git commit to running container.