| 100% Remote - Airflow Data Engineer - Any visa is fine at Fine, New York, USA |
| Email: [email protected] |
| 100% Remote - Airflow Data Engineer - Any visa is fine Hi, This is Mahendra from Empower Professionals. We have an Airflow Data Engineer position for one of my clients, so please share your updated resume if you are interested in the below role. Role: Airflow Data Engineer Location: 100% Remote Duration: 12+ Months Required Qualifications: Proven experience with Astronomer Airflow and data workflow orchestration. Strong understanding of CI/CD pipelines and version control systems (e.g., GitHub). Experience with cloud platforms (e.g., Azure) and integrating external services. Proficiency in Python and SQL. Excellent problem-solving skills and attention to detail. Required Skills: MUST have experience standing up an Apache Airflow environment from the ground up. REQUIRED Tech Stack ----Apache Airflow----Astonomer---Azure----Databricks Job Summary: We are seeking a skilled Data Engineer join our data engineering team. The successful candidate will be responsible for enhancing and automating our data workflows using Astronomer Airflow. This role involves improving data processing capabilities, optimizing task scheduling, and reducing our dependency on Databricks for orchestration. The specialist will also manage and monitor our data pipelines to ensure efficient and reliable operations. Key Responsibilities: Astronomer Airflow Setup: Install and configure Astronomer Airflow. Establish necessary connections and ensure the Airflow web server and scheduler are operational. Integrate CI/CD pipelines and GitHub for version control and automated deployment of workflows. Workflow Migration: Migrate existing workflows to the Astronomer Airflow platform. Ensure workflows run error-free and meet performance benchmarks. Team Training: Conduct training sessions for the data engineering team. Provide supporting materials and documentation. Ensure team proficiency in creating and managing workflows and DAGs. Monitoring and Alerting: Set up monitoring tools to track workflow performance. Configure alerts for critical issues. Establish regular reporting procedures for ongoing monitoring. Optimization and Documentation: Review and optimize workflows for performance. Create detailed documentation covering workflows, configurations, and best practices. Establish a feedback loop for continuous improvement. Expand Orchestration Capabilities: Integrate Airflow with Azure functions, REST endpoints, and event-driven architectures. Implement support for external services to extend orchestration beyond Databricks. Create modular workflows for integrating ML/AI batch workloads. Job and Data Dependency Management: Develop a system to handle job dependencies. Implement checks to ensure jobs run only if the underlying data has changed. Preferred Qualifications: Experience with Databricks and Snowflake. Familiarity with machine learning and artificial intelligence workflows. Strong communication and training skills. Ability to work collaboratively in a team environment. Mahendra Bisht Empower Professionals 100 Franklin Square Drive Suite 104, Somerset, NJ 08873 [email protected]| www.empowerprofessionals.com Certified NJ and NY Minority Business Enterprise (NMSDC) Mahendra [email protected], Somerset, NJ 08873, NJ 08873 Keywords: continuous integration continuous deployment artificial intelligence machine learning New Jersey New York 100% Remote - Airflow Data Engineer - Any visa is fine [email protected] |
| [email protected] View all |
| Sat Oct 05 01:29:00 UTC 2024 |