How to set FLUX auto image update using ECR as a registry for helm packages and docker images, Flux will use HelmRelease for detecting changes.

Necessary Tools

  • AWS CLI version 2.9.2 or newer
  • Flux version 0.38.2 or newer

Requirements

  • Repository on ECR for docker images (https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
  • Repository on ECR for helm packages (https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html)

Install Flux

Flux bootstrap command (using a github token with all repo access permissions):

flux bootstrap github \
  --owner=$GITHUB-ACCOUNT-NAME \  
  --repository=$REPOSITORY-NAME \
  --path=clusters/$CLUSTER-NAME \
  --components-extra=image-reflector-controller,image-automation-controller \
  --personal

FLUX configuration

  • Use the CRD ImageRepository to track your image:
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: $NAME-IMAGE-REPOSITORY
  namespace: flux-system
spec:
  image: $AWS-ACCOUNT-ID.dkr.ecr.$AWS-REGION.amazonaws.com/$REPOSITORY-NAME
  interval: 1m0s
  • Also, you need to make the cluster able to do this, if not you’ll get a permissions error. In your GitOps repository add a patch to kustomization.yaml belong to flux-system (more details about it here):
patches:
- target:
    version: v1
    group: apps
    kind: Deployment
    name: image-reflector-controller
    namespace: flux-system
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/args/-
      value: --aws-autologin-for-ecr
  • Use the CRD ImagePolicy to specify which version you want to track, there are plenty ways to filter tags here. I’m tagging images using this prefix dev-1675960453 the number on the prefix it’s the current date, so each new image it’s going to have a greater number in the tag, I made this using bash dev-$(date +%s). The following policy is detecting this pattern and extracting the timestamp, which is sorted by numerical ascendant order to identify the last one.
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: $
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: $NAME-IMAGE-POLICY
  filterTags:
    pattern: '^dev-(?P<ts>.*)'
    extract: '$ts'
  policy:
    numerical:
      order: asc
  • You can check that the tracking works as expected by running the following commands, READY should be True for both, if it’s False, in MESSAGE you can see the reason

flux get image policy $NAME-IMAGE-POLICY

flux get image repository $NAME-IMAGE-REPOSITORY

  • Preparing your HelmRelease file/configuration for Flux image update, this part is very important to get this done.

HelmRepository file example:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: $NAME-HELM-REPOSITORY
  namespace: $NAMESPACE
spec:
  type: oci
  interval: 1m
  url: oci://$AWS-ACCOUNT-ID.dkr.ecr.$AWS-REGION.amazonaws.com
  provider: aws

HelmRelease file example:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: $NAME-HELM-RELEASE
  namespace: $NAMESPACE
spec:
  interval: 5m
  chart:
    spec:
      chart: $CHART-NAME
      version: "0.1.0"
      sourceRef:
        kind: HelmRepository
        name: $NAME-IMAGE-REPOSITORY
        namespace: $NAMESPACE
      interval: 1m
  valuesFrom:
    - kind: ConfigMap
      name: $NAME-CONFIGMAP

HelmRelease only upgrades resources if helm chart version changes. In order to make image update works we have to generate a dynamic name for our ConfigMap so any time the repository receives a push then this name change and will trigger a new release.

In the same folder where is your helmrelease and kustomization, create/edit the following files:

kustomizeconfig.yaml

nameReference:
- kind: ConfigMap
  version: v1
  fieldSpecs:
  - path: spec/valuesFrom/name
    kind: HelmRelease

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $NAMESPACE
resources:
  - namespace.yaml
  - repository.yaml
  - release.yaml
configMapGenerator:
  - name: $NAME-CONFIGMAP
    files:
      - values.yaml=NAME-FILE-WITH-VALUES.yaml
configurations:
  - kustomizeconfig.yaml

file with custom values:

Deployment:
  imagePullPolicy: Always
  image: $AWS-ACCOUNT-ID.dkr.ecr.$AWS-REGION.amazonaws.com/$REPOSITORY-NAME:$TAG # {"$imagepolicy": "POLICY_NAMESPACE:POLICY_NAME"}

Notice that in line image we are telling Flux which image would need to be updated, to do so, you need to add a marker to the line with image, this marker can be added to any part of the GitOps repository in any file.

  • Check if deploy key has write access in GitHub, go to github repository and then to settings/Deploy Keys the Deploy key should say Read/Write, if not, Delete the current one, run $ kubectl -n flux-system get secret flux-system -o json | jq '.data."identity.pub"' -r | base64 -d the output of this command is the key that you will add as a deploy key, make sure mark the option “Allow write access”. Add a new resource called ImageUpdateAutomation, this will commit the new tag where you wrote the marker {"$imagepolicy": "POLICY_NAMESPACE:POLICY_NAME"}.
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
    push:
      branch: main
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  update:
    strategy: Setters
  • Push a new image with the tag mentioned before and then you will see a new release on k8s cluster.

References

  • https://fluxcd.io/flux/guides/image-update
  • https://fluxcd.io/flux/guides/helmreleases/#refer-to-values-in-configmaps-generated-with-kustomize