i have done ETL from MySql to bigQuery with python, but because i haven't permission to connect google cloud storage/ cloud sql, i must dump data and partition that by last date, this way easy but didn't worth it because take a much time, i want to ETL using airflow from MySql/mongo to bigQuery without google cloud storage/ cloud sql it is possible ?
Asked
Active
Viewed 709 times
1 Answers
0
Apache Airflow (which is used under Cloud Composer in Google Cloud) is generally used to schedule a migration. The actual migration along the complete ETL procedure can be performed with Cloud Dataflow (which uses Apache Beam) and is the recommended tool for this use case. Here is a complete tutorial for performing a ETL procedure from a relational database (such as MySQL) into BigQuery using Cloud Dataflow.
PYB
- 101
- 1