Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.
Questions tagged [apache-airflow]
12 questions
2
votes
1 answer
Airflow: Can't connect to ('0.0.0.0', 8080)
I am trying to do some stuff with Airflow.I have installed it with pip(latter in a virtual) but problem is when I want to start webserver.
airflow webserver -p 8080
[2017-07-29 12:20:45,913] [4585] {models.py:167} INFO - Filling up the DagBag from…
MishaVacic
- 123
1
vote
0 answers
How can I get Airflow on Linux run a VBS script that is automatically downloading SAP data?
I have an Airflow installed on CentOS and I am able to run scripts/commando's on my Windows 10 machine.
However, I have a VBS script that starts Excel and SAP Logon and then automatically logins, fills in some fields and download a file. When I run…
Billy
- 11
0
votes
1 answer
Google Cloud Composer "Error: Server Error"
My Google Cloud Composer is not working since yesterday.
I can't access the airflow interface, returning:
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.
A warning is…
0
votes
1 answer
ETL to bigquery using airflow without have permission cloud storage/ cloud sql
i have done ETL from MySql to bigQuery with python, but because i haven't permission to connect google cloud storage/ cloud sql, i must dump data and partition that by last date, this way easy but didn't worth it because take a much time, i want to…
0
votes
0 answers
Apache Airflow for IoT device simulators
I want to use Apache Airflow for IoT device simulator written in python. Let's say I want to run that script for 25000 devices and the 25000 invocations should run concurrently. Is apache airflow is right tool to be used or there are other…
0
votes
0 answers
Clould composer airflow dag dataflow job error
I am running dag for dataflow java operator in composer airflow,
default_args = {
'retries': 0,
'start_date': airflow.utils.dates.days_ago(0),
'owner': 'Airflow_TEST',
'dataflow_default_options': {
'project': …
Olive
- 25
0
votes
1 answer
Google Cloud Composer 2 Environment won't become healthy
I am trying to create a Google Cloud Composer 2 environment in my project, but it fails to become healthy.
I am creating it with the default settings and a service account that has the Cloud Composer v2 API Service Agent Extension, Composer Worker…
0
votes
1 answer
How to check current host os in Docker compose?
For linux and airflow containers, we have to provide host user id to the container:
"On Linux, the quick-start needs to know your host user id and needs
to have group id set to 0. Otherwise the files created in dags, logs
and plugins will be…
Павел Иванов
- 113
0
votes
0 answers
Which server specs to look for for a Scheduler in a data stack?
I'm in the process of building a data stack for a small company : the choice has been made to have a UNIX server running as a "Scheduler". The goal of this scheduler is just to extract data from different applications and send it back to a in-cloud…
Aradyan
- 1
0
votes
1 answer
How should "CPU usage per node" be interpreted in Google Cloud monitoring?
In the monitoring tab for Composer (Airflow) on Google Cloud there is a graph showing "CPU usage per node". How should the values in this graph be interpreted? What value would indicate that the load is reaching the limit and that there might be…
Niemi
- 101
0
votes
1 answer
Running a task in parallel in multiple machines in GCP and orchestrating it
I have a python program for scraping that needs a lot of time to run. To parallelize it, I have modified the code so that the program can run in parallel in different machines. I also created a docker image and pushed it to Dockerhub.
I tried to use…
vettipayyan
- 101
-1
votes
1 answer
How to set smtp_password in GCP Cloud Composer for configuring third-party SMTP services
I can use only G Suit SMTP mail server due to company security policy.
I want to configuring 3rd-party SMTP service for using G Suit mail service but I can't set "smtp smtp_password" config in AIRFLOW CONFIGURATION OVERRIDES tap of Cloud…
