Fixing Argo CD "Too long must have at most 262144 bytes" error

In this post, I'll explain how to fix the "Too long: must have at most 262144 bytes" error in Argo CD. It often appears when trying to sync a Helm chart that includes large CRDs such as kube-prometheus-stack.

Problem

Kubernetes objects have a fixed size limit of 256kb for annotations. When using kubectl apply to update resources (which Argo CD does) it tries to set a last-applied-configuration annotation that contains the JSON representation of the last applied object configuration.

For example, if we have a Deployment object that we update with kubectl apply, the last-applied-configuration annotation will contain the JSON serialized format of the whole Deployment object:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment", "metadata":{}, ...}

This is not a problem for most resources, but there are objects which go over the 256kb limit, such as the Prometheus CRD from the kube-prometheus-stack Helm chart, which is 500kb in size.

Syncing the Prometheus CRD in Argo CD will run kubectl apply and try to add the 500kb JSON representation of it as an annotation. This will then result in the "Too long: must have at most 262144 bytes" error as it goes over the Kubernetes annotations size limit of 256kb (or, 262144 bytes).

Solution

The solution is to stop using Client Side Apply (the current default when running kubectl apply) and instead use Server Side Apply which will not add the last-applied-configuration annotation to objects.

Server Side Apply is planned to be the default apply method in future Kubernetes and Argo CD versions, but for now we have to explicitly enable it.

Support for Server Side Apply was added in Argo CD v2.5 and can be enabled by setting it in the sync options for an Application resource:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

Argo CD versions lower than v2.5 don't have support for Server Side Apply. In these cases, we can fall back to using kubectl replace to update the object. We can do this by setting the Replace=true sync option. But be aware that replace can lead to unexpected results when you have multiple clients modifying an object.