| urgent need Databricks Senior Data Engineer at Remote, Remote, USA |
| Email: [email protected] |
| From: Bhavya, Source infotec inc [email protected] Reply to: [email protected] Databricks Senior Data Engineer New York, NY 6+months Visa: GC/USC Only local, must be really good in ETL, Databricks, Azure. Any certification related to Data Engineering OR Databricks would a huge plus! Responsibilities: Strategize, design, develop and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions Design and develop ETL code to integrate various data sources Create and maintain efficient data pipelines on the Databricks platform Develop, maintain, and optimize ETL pipelines for both streaming and batch data processing Ensure data integrity and consistency throughout the ETL lifecycle Provide comprehensive support for ETL processes, from data ingestion to final output Write and execute unit tests and integration test cases for ETL code to ensure high-quality outcomes Implement monitoring solutions for ETL jobs to ensure timely and successful data processing Proactively identify, troubleshoot, and resolve issues in ETL workflows Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting Mentor other data engineers in the team, cross-train and provides guidance . Minimum Qualifications : Minimum of bachelors degree in computer science, Information systems, Engineering, or Data Science Minimum 10 years of experience in designing, developing, and optimizing ETL processes 5+ years of experience in developing/supporting a data platform in Azure Databricks Proficiency in creating and maintaining efficient data pipelines on the Databricks platform Well-rounded in working in a DEV Ops environment supporting processes in data platform in supporting business units and their use cases Strong verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders Ability to troubleshoot and resolve issues in a timely manner Experience working in an Agile/Scrum environment Team player with the ability to work collaboratively in a cross-functional team Strong analytical and problem-solving skills Ability to work independently, handle multiple tasks simultaneously and adapt quickly to change with a variety of people and work styles Willingness to learn new technologies and continuously improve skills . Preferred Qualifications: In-depth knowledge of Databricks platform and technologies, including Delta Lake, Databricks SQL, and Databricks Workflows Experience with Azure cloud platforms and Azure Data Lake cloud storage Knowledge of data warehousing, data modeling, and best practices Proficiency in programming languages such as Python, SQL, Scala, or R Experience with big data technologies such as Apache Spark, Hadoop, or Kafka Knowledge of working with Hadoop for migration projects from Hadoop to Databricks Familiarity with DevOps practices and tools such as CI/CD, Git, etc. Knowledge of infrastructure as Code (IaC) tools like Terraform Experience with implementing data governance and security measures in a cloud environment. THANKS & REGARDS Bhavya Chaturvedi Source Infotech Inc P.O Box 577, EDISON, NJ 08818-0577 EMAIL: [email protected] Phone: 6097335170 | | Hangout:bhavyachaturvedi389 Keywords: continuous integration continuous deployment rlang green card New Jersey New York urgent need Databricks Senior Data Engineer [email protected] |
| [email protected] View all |
| Wed Sep 11 22:22:00 UTC 2024 |