| Unity Catalog Engineer at Unity, Wisconsin, USA |
| Email: [email protected] |
| From: Ravina, Vyze Inc [email protected] Reply to: [email protected] Job Title: Unity Catalog Engineer Location: 100 % Remote Duration: 12+ months Moi: Skypec2c Responsibilities: Collaborate closely with clients to design and customize the Unity Catalog structure according to their specific requirements and workflows. Develop mappings of Metastore artifacts to Unity Catalog Catalogs and Schemas, ensuring seamless integration and data consistency. Lead the upgrade process for tables, clusters, and workflows, migrating 293 tables and 100 jobs to the upgraded Databricks Unity Catalog. Prepare continuous integration and continuous deployment (CI/CD) pipelines for managing code changes in jobs. Ensure data quality and consistency throughout the migration process, optimizing data structures and configurations for enhanced performance and scalability. Establish robust rollback procedures to address any migration issues effectively. Synchronize data to mapped schemas to maintain data integrity and coherence. QA, Enablement, and Cutover: Conduct comprehensive testing of upgraded Databricks Unity Catalog objects to validate functionality and reliability. Conduct training sessions for end-users to familiarize them with Unity Catalog usage and best practices for metadata management. Provide hands-on training for the internal IT team on catalog management processes and procedures. Share expertise and insights on metadata management best practices to optimize system performance and user experience. Skills and Qualifications: Bachelor's degree in Computer Science, Engineering, or a related field. 7+ years of experience in data engineering or related roles, with a focus on metadata management and data catalog solutions. Proficiency in designing and implementing complex data structures and workflows using Databricks Unity Catalog or similar platforms. Strong expertise in data migration processes, including mapping, transformation, and validation. Experience with continuous integration and continuous deployment (CI/CD) practices and tools. Excellent problem-solving skills and attention to detail, with a focus on ensuring data quality and integrity. Strong communication and interpersonal skills, with the ability to collaborate effectively with clients and internal teams. Proven track record of delivering high-quality solutions on time and within budget. Keywords: continuous integration continuous deployment quality analyst information technology Unity Catalog Engineer [email protected] |
| [email protected] View all |
| Tue May 21 08:17:00 UTC 2024 |