Looking for help on using the EKSPodOperator. My set up is as follows: Airflow Version: 2.6.2 deployed with the official helm chart v1.15.0 Kubernetes Cluster: EKS 1.30 Executor: LocalExecutor Postgres Database is accessed through AWS secrets backend connection.
My intention is to authenticate to the cluster through the scheduler's service account which has been annotated with the appropriate IAM role and policies.
Issue When I triggered the DAGs, I got a permission error relating to kubernetes_default and aws_default secrets which I didn't even create in the first place. To get past this, I granted the permission to the Scheduler's IAM role, and also created both secrets with the following content to facilitate the connection:
kubernetes_default: kubernetes://?extra__kubernetes__namespace=airflow&extra__kubernetes__in_cluster=True
aws_default: aws://?region_name=eu-west-1
Result:
"ERROR - Invalid connection configuration. Options kube_config_path, kube_config, in_cluster are mutually exclusive. You can only use one option at a time.
I do not have kube_config_path and kube_config set anywhere.
If I set in_cluster to false, I get the error - 'NoneType' object has no attribute 'metadata'.
I get the same errors when I delete the secrets just in case they are causing some sort of conflict.
My preference is to use the in_cluster configuration since the tasks will be executed within the cluster and I'd like to use a service account for authentication.
Has anyone successfully used EKSPodOperator with in-cluster auth on EKS? What steps did you follow? Any help or guide will be much appreciated. Thank you.
