| Devops lead Automation with python & SCALA || Philadelphia PA onsite at Philadelphia, Pennsylvania, USA |
| Email: [email protected] |
| Hi Employer, Job Description: We are seeking a highly skilled Automation Lead with expertise in data engineering, development, and automation. In this role, you will be responsible for designing and implementing automated solutions using big data technologies such as Spark, Scala, cloud platforms like AWS S3, AWS Athena, SQL technologies including Hive, Teradata, Spark SQL, Databricks, and Python (with a focus on Data Quality). As an Automation Lead, you will play a key role in driving efficiency, scalability, and quality in our data engineering processes. Responsibilities: Lead the design, development, and implementation of automated data engineering solutions leveraging big data technologies. Utilize Spark and Scala to build scalable and efficient data pipelines for data ingestion, processing, and analysis. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. Design and implement automated workflows using cloud technologies like AWS S3 and AWS Athena for data management and storage. Apply SQL skills, including Hive, Teradata, and Spark SQL, to perform data transformations and analysis. Leverage the Databricks platform to develop and optimize data processing, analytics, and machine learning tasks. Develop Python scripts and libraries to support data quality initiatives and automate data validation processes. Establish data governance and data quality standards to ensure accuracy, reliability, and consistency of data. Monitor and optimize data pipelines for performance, scalability, and reliability. Stay up-to-date with industry trends and emerging technologies in the big data, cloud computing, and automation domains. Provide technical leadership and mentorship to junior team members. Qualifications: Bachelor's degree in Computer Science, Engineering, or a related field. A master's degree is a plus. Extensive experience in data engineering, development, and automation. Strong proficiency in big data technologies, including Spark and Scala. Solid understanding of cloud technologies, particularly AWS S3 and AWS Athena. Expertise in SQL-based technologies like Hive, Teradata, and Spark SQL. Hands-on experience with the Databricks platform and its various components. Strong programming skills in Python, with a focus on data quality and automation. Experience in designing and implementing data pipelines, ETL processes, and workflows. Knowledge of data governance principles and best practices. Docker VM, Containers, Kubernetes Database - Cassandra, Mongo, MySQL, Redis Strong problem-solving abilities and the ability to lead complex projects. Excellent communication and interpersonal skills to effectively collaborate with team members and stakeholders. Mohd Faisal -- Keywords: sthree information technology |
| [email protected] View all |
| Tue Nov 28 21:16:00 UTC 2023 |