50 questions
0
votes
0
answers
50
views
KubernetesPodOperator - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
I’m running Apache Airflow inside a Docker container and trying to use the KubernetesPodOperator to run a simple “hello world” pod in an external Kubernetes cluster (not the same one where Airflow ...
0
votes
0
answers
42
views
Mount Google Composer's data folder in a custom Kubernetes pod
To execute compute-intensive tasks (such as unzipping files), I decided to use the KubernetesPodOperator, which runs Bash commands on a pod with a custom Docker image inside the composer-user-workload ...
- 1
0
votes
1
answer
180
views
How do I use the EKSPodOperator with in_cluster set to True with a ServiceAccount without getting an Error
Looking for help on using the EKSPodOperator.
My set up is as follows:
Airflow Version: 2.6.2 deployed with the official helm chart v1.15.0
Kubernetes Cluster: EKS 1.30
Executor: LocalExecutor
...
- 1
0
votes
0
answers
69
views
How to Set Dynamic execution_timeout for a Task in Airflow Using GKEPodOperator?
I have an Airflow DAG with six tasks, all using GKEPodOperator. My DAG has an execution_timeout of 1 hour, but sometimes Task 5 takes longer to execute, causing failures.
I want to set a dynamic ...
- 131
0
votes
1
answer
77
views
Is it possible to wrap @task.kubernetes so it assigns a pod name based on the decorated function name?
I tried creating a decorator that would automatically assign a pod name based on the wrapped function, something like:
def default_kubernetes_task(
**task_kwargs: Any,
) -> Callable[[Callable[.....
- 3,252
0
votes
1
answer
238
views
How to create an airflow sensor using k8s pod in (similar to KubernetesPodOperator)
I have recently started to use the KubernetesPodOperator that provides a lot of scale to airflow tasks (on 2.10) and has been working very well.
However, I would also like to kubernetes pods as sensor ...
- 485
1
vote
1
answer
445
views
GKEStarPodOperator/Kubectl API: Error 500 "No agent available"
try to launch operator GKEStartPodOperator, to launch a simple test again GKE cluster in GCP.
Using service account with these roles:
Kubernetes Engine Admin
Kubernetes Engine Developer
Kubernetes ...
- 127
0
votes
1
answer
229
views
How CeleryExecutor executes KubernetesPodOperator?
I have an Airlfow instance spinning in Azure Kubernetes Service. Deployed with official Helm chart. And I noticed CeleryExecutor is used there. I have KubernetesPodOperator tasks, and wondering how ...
2
votes
0
answers
589
views
How to use dynamic task mapping in Airflow with KubernetesPodOperator
It seems like the only way to pull xcom values in a KubernetesPodOperator task is by setting do_xcom_push=True and using a templated xcom_pull, ie "{{ ti.xcom_pull(task_ids='foo', key='bar') }}&...
- 21
0
votes
0
answers
88
views
Airflow - K8s- Unable to mount HostPath using KubenetesPodOperator
I have a ADF's Airflow managed instance provisioned. And, I'm trying to schedule a DAG.
In this DAG, I'm trying to run a shell script which is present in the HOSTPATH "/opt/airflow/dags" ...
- 913
0
votes
1
answer
2k
views
KubernetesPodOperator shows completed in Airflow UI but keep on running in backend
I am working on KubernetesPodOperator for one of the development.
In that, I have used this to generate the files in GCP.
I tried the below approach and it was working fine too.
The ...
- 1
0
votes
1
answer
214
views
not able to access xcom value in env_var of KubernetesPodOperator
We have dags running in astronomer 9.4.0 using airflow version 2.7.2+astro.3
I have a dag which have KubernetesPodOperator as a task , which runs the golang application.
Inside the Golang application ...
- 21
1
vote
0
answers
43
views
Airflow "ghost" running task - task remains in running state even after failure
I'm working on an Airflow project where I have a KubernetesPodOperator task that sometimes fails but remains in a running state after passing number of defined retries. Status in UI is Running but no ...
1
vote
0
answers
203
views
Airflow custom Kube Operator template fields not working for image_pull_Secrets
In my KubernetesPodOperator I want to assign dynamic values to namespace, servic_account_name and image_pull secrets. I am using jijna template. Below is the code for my custom KubernetesPodOperator. ...
- 11
1
vote
0
answers
664
views
Airflow container OOM killed even though memory usage was below requested
I am running Airflow 2.5.3 and I am tearing my hair out over this one:
A task in my DAG (KubernestPodOperator) was configured to request 3Gi memory and it runs a loop. At the bottom of the loop, I log ...
- 127
0
votes
1
answer
239
views
dynamically generate cmds in KubernetesPodOperator with Jinja2 template
I have a conf json looks like this
{ "customer": "customer1",
"region": "south"
}
I want generate a
cmds: ["aws", "domain_name", "...
- 774
-1
votes
1
answer
2k
views
Airflow: How to trigger a dag multiple times and queuing the DagRuns for running one at a time
I have a job that i want airflow to do, but this job cannot run in parralel.
This job is based on a docker image that airflow will trigger using KubernetesPodOperator with the specific command.
I want ...
0
votes
1
answer
1k
views
Airflow Kubernetes Pod Operator XCom return value is "None" when accessed in pythonOperator
I am running a kubernetes pod using airflow KubernetesPodOperator. Then executing a jar file in the pod and writing the output to the /airflow/xcom/return.json. When checking the task's XCom value it ...
- 27
0
votes
2
answers
891
views
json.decoder.JSONDecodeError when getting string from XCOM
In a task, I serialise a dict (converting a dict to string) and I pushed it to XCOM
result[data] = json.dumps({"agents": ["[email protected]"], "houses": ["jane....
0
votes
1
answer
1k
views
Variable.get() not working in KubernetesPodOperator?
I have a task defined using KubernetesPodOperator and I am passing value of the templated variable run_id as an environment variable.
What baffles me is that when I use the Jinja template syntax, it ...
- 127
0
votes
2
answers
5k
views
How to parameterize KubernetesPodOperator's resources in Airflow 2.4.3 with params?
I have an Airflow DAG that uses a KubernetesPodOperator to run a containerized task. I would like to be able to parameterize the resources (memory and CPU) allocated to the container, so that I can ...
- 66
1
vote
1
answer
4k
views
How to use pod_template_file parameter with KubernetesPodOperator in airflow 2?
I am running airflow via MWAA on aws and the worker nodes are running k8s. The pods are getting scheduled just fine but I am trying to use pod_template_file with KubernetesPodOperator, it's giving me ...
- 2,264
1
vote
1
answer
2k
views
I am relatively new to Airflow and spark and want to use the Kubernetes Operator in airflow Dag to run a Spark Submit command
I am using the kubernetes version 1.25 client and server, I have deployed Airflow using the official helm charts on the environment. I want the Airflow dags kubernetes pod operator that has code to ...
- 51
1
vote
1
answer
1k
views
Passing Airflow Context to init_container in KubernetesPodOperator
Airflow's KubernetesPodOperator provides an init_containers parameter, with which you can specify kubernetes init_containers. However init_containers expects a list of kubernetes.models.V1Container, ...
- 6,157
1
vote
2
answers
3k
views
How to Get Xcom Values inside Kubernetes Pod Operator
Im Trying to pass a Xcom value as arguments inside a kubernetes pod operator But values are not populating.
Checked Xcom's List in Airflow UI I am able to see the key,value.
This is my syntax:
...
- 61
0
votes
1
answer
1k
views
Unable to run KubernetesPodOperator
I am using airflow 2.4.3 and running KubernetesPodOperator
Below is the code and error:-
Please help me with creating a KubernetesPosOperator in python. I have tried on both GCP and Azure.
Also adding ...
2
votes
0
answers
243
views
Airflow 2.1.4 Composer V2 GKE kubernetes in custom VPC Subnet returning 404
So I have two V2 Composers running in the same project, the only difference in these two is that in one of them I'm using the default subnet and default values/autogenerated values for cluster-ipv4-...
- 601
1
vote
1
answer
1k
views
KubernetesPodOperator on_failure_callback doesnt work on Airflow 2.4.2
When using the newest Airflow version (2.4.2), the on_failure_callback doesnt get triggered on KubernetesPodOperator tasks. The on_success_callback works just fine.
For each KubernetesPodOperator task ...
- 11
0
votes
1
answer
540
views
Cloud composer running Docker image inside Kubernetes pod using KubernetesPodOperator
I want to run a docker inside Kubernetes pod via Cloud composer, so to do so I followed all the mentioned steps here and also verified the binding, I have used default service account along with ...
0
votes
1
answer
1k
views
Airflow KubernetesOperator Error: `Invalid connection configuration. Options kube_config_path, kube_config, in_cluster are mutually exclusive`
I am pulling my dags from github using git-sync. The only changes I made to my values.yaml file was using KubernetesExecutor, configuring git-sync and logs.
Is there something else missing because ...
- 179
1
vote
0
answers
496
views
Accessing minikube via the KubernetesPodOperator in airflow
I am trying to make the airflow KubernetesPodOperator work with minikube. But unfortunately, the operator does not find the kubernetes cluster.
The dag returned to me the following error:
ERROR - ...
- 83
1
vote
1
answer
2k
views
KubernetesPodOperator is not able to access the Airflow Connections stored in Google Secrets Manager
I am using the Composer version 2.0.0 Airflow 2.1.4 and I have created a KubernetesPodOperator that is trying to access the Airflow connection stored in the Google Secrets Manager. But it isn't able ...
- 525
1
vote
0
answers
754
views
Accessing airflow KubernetesPodOperator environment variables
I am trying to pass environment variables to my container using env_vars parameters of the KubernetesPodOperator.
task_fetch = KubernetesPodOperator(
task_id=FETCH,
name=FETCH,
...
- 83
0
votes
1
answer
1k
views
bash command while loop only run once within Airflow kubernetespodoperator
I am trying to run one Kubernetes pod job 6 times. Each time it will print a number and sleep for 5 seconds. However, it only runs once, and then it stops. Here is the full code for the dag file:
from ...
- 31
0
votes
1
answer
163
views
How to configure Elyra to use another Jinja template when utilizing custom components?
I am trying to make Elyra to use a custom Jinja template for my custom component. I have tried modifying Elyra's configuration file for this two items:
c.ElyraApp.template_paths = ['/home/templates']
...
- 312
1
vote
1
answer
276
views
How to add custom component to Elyra's list of available airflow operators?
Trying to make my own component based on KubernetesPodOperator. I am able to define and add the component to the list of components but when trying to run it, I get:
Operator 'KubernetesPodOperator' ...
- 312
1
vote
0
answers
1k
views
Airflow doesn't "see" that the underlying Kubernetes pod completed
We are using a hosted Airflow 1.10.2 in Google Composer 1.7.5 to launch jobs via the KubernetesPodOperator (tasks that will be run in a Kubernetes pod inside a worker cluster)
There has been several ...
- 18.6k
3
votes
1
answer
1k
views
Airflow - Unable to use jinja template for resources in Kubernetes Pod Operator task
Airflow - Unable to use jinja template for resources in Kubernetes Pod Operator task.
Able to use jinja template for environment variables, image but not able to use for resources to specify CPU and ...
- 31
0
votes
1
answer
580
views
airflow2.0 with KubernetesPodOperator: TemplateNotFound
I am using Airflow2.0 with KubernetesPodOperator want to run a command that use as a parameter a file from inside the image ran by the Operator. This is what I used:
KubernetesPodOperator(
...
- 1,380
0
votes
1
answer
2k
views
Mount volumes with Secrets using Python Kubernetes API
I'm writing an Airflow DAG using the KubernetesPodOperator. A Python process running in the container must open a file with sensitive data:
with open('credentials/jira_credentials.json', 'r') as f:
...
- 1,467
3
votes
1
answer
3k
views
Airflow - KubernetesPodOperator - Role binding a service account
I am currently using the KubernetesPodOperator to run a Pod on a Kubernetes cluster. I am getting the below error:
kubernetes.client.rest.ApiException: (403) Reason: Forbidden
HTTP response headers: ...
- 807
0
votes
0
answers
745
views
Testing airflow dags with KubernetesPodOperator
We have a sort of self-serve Airflow cluster that mandates that all of the tasks are wrapped as KubernetesPodOperator tasks. With this setup what are the possible options to test the dags in CI/CD ...
- 5,594
0
votes
1
answer
1k
views
Airflow Rendered Template changes when task starts running?
I'm having a very weird airflow bug.
Problem
I have a dag that has a bash operator as step 1 and a KubernetesPodOperator as step 2. The issue is regarding the KubernetesPodOperator. Basically, I was ...
- 105
1
vote
2
answers
752
views
How to run docker image with --privileged in KubernetesPodOperator on airflow
How do I docker run private image from Container Registry in GCP using --privileged
Running this locally works fine:
docker run -it --privileged --entrypoint /bin/bash ${GKE_APP}
I followed this but ...
- 761
-1
votes
1
answer
303
views
Bypass resources provided by pod mutation hook
We have airflow running on kubernetes. Below is my airflowlocalsettings:
airflowLocalSettings: |
from airflow.contrib.kubernetes.pod import Pod, Resources
from airflow.configuration import ...
- 331
0
votes
1
answer
945
views
How can I set dns config in airflow kuber pod operator?
I am scheduling some tasks on airflow using KubernetesPodOperator; I want to deploy my pod with custom dns configuration:
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- 10.10....
3
votes
2
answers
9k
views
passing --serviceaccount in airflow kubernetes pod operator
I am trying to create and run a pod using Airflow kubernetes pod operator.
The command below is tried and confirmed to be working and I am trying to replicate the same using the kubernetes pod ...
- 51
8
votes
5
answers
9k
views
How to install dependency modules for airflow DAG task(or python code)? , "Failed to import module" in airflow DAG when using kuberentesExecutor
I have a airflow DAG "example_ml.py" which has a task "train_ml_model" and this task is calling/running a python script "training.py".
-Dags/example_ml.py
-Dags/training....
- 1,506
4
votes
0
answers
5k
views
Airflow KubernetesPodOperator : how to access secret passed to Pod?
I am trying to pass secret variables to my KubernetesPodOperator in airflow
Here is what I have done :
Create a secret.yaml file that looks like the following
apiVersion: v1
kind: Secret
metadata:
...
- 637
6
votes
0
answers
1k
views
Airflow KubernetesPodOperator - using a ConfigMap value as an environmental variable
I have a Kubernetes ConfigMap called test that contains a key foobar with some value. I would like to use that key's value in an environmental variable.
import datetime
import os
from airflow import ...
- 4,054