| Azure Data Engineer with Databricks and Unity Catalog Expertise at Unity, Wisconsin, USA |
| Email: [email protected] |
| Location: Remote Duration: 12+ Months Job Title: Azure Data Engineer with Databricks and Unity Catalog Expertise Job Summary: As an Azure Data Engineer with expertise in Databricks and Unity Catalog, you will play a critical role in building, managing, and optimizing our data infrastructure. You will work closely with data scientists, analysts, and other engineers to ensure that data is accessible, well-governed, and efficiently processed across our cloud platforms. The ideal candidate will have a strong background in data engineering, cloud computing, and data governance. Key Responsibilities: Data Pipeline Development: Design, develop, and migrate robust ETL/ELT pipelines in Azure Data Factory and Databricks to support data ingestion, transformation, and storage. Unity Catalog Implementation & Migration: Implement and manage Unity Catalog within the Databricks environment to organize, secure, and govern data assets. Migrate legacy pipelines, notebooks, and tables to a Unity Catalog compliant format. Data Modeling: Develop and maintain scalable data models and architectures that support business intelligence and analytics needs. Data Governance: Collaborate with data governance teams to ensure data is properly classified, secured, and audited within Unity Catalog. Performance Optimization: Monitor and optimize the performance of data pipelines and Databricks clusters to ensure cost-effective and efficient data processing. Data Security: Implement and manage role-based access controls (RBAC) and data encryption practices to safeguard sensitive data. Collaboration: Work with cross-functional teams to understand data requirements and deliver high-quality data solutions that meet business needs. Automation: Develop automation scripts and tools to streamline data engineering processes and improve productivity. Troubleshooting: Identify and resolve issues related to data pipelines, data quality, and performance bottlenecks. Qualifications: Educational Background: Bachelors or masters degree in computer science, Data Engineering, Information Technology, or a related field. Experience: 10+ years of experience in data engineering, with a focus on cloud-based data platforms. Proven experience with Databricks, including hands-on work with Unity Catalog. Strong experience in building and optimizing data pipelines using Apache Spark, Databricks, and Azure Data Factory Experience with cloud platforms such as AWS, Azure, or GCP -- Keywords: information technology Azure Data Engineer with Databricks and Unity Catalog Expertise [email protected] |
| [email protected] View all |
| Fri Aug 30 01:52:00 UTC 2024 |