| Need Profiles :: Lead Data Engineer in Chicago, IL :: Hybrid - Locals preferred at Chicago, Illinois, USA |
| Email: [email protected] |
| From: Ashish Kumar, THEMESOFT INC [email protected] Reply to: [email protected] Job Title: Lead Data Engineer Location: Chicago, IL - 3 days onsite. Local preferred Job Description: We are seeking a Lead Data Engineer with a robust background in data engineering, cloud technologies, and ETL processes. The ideal candidate will possess strong hands-on experience in Python/PySpark scripting, cloud services like AWS Glue/Lambda, Snowflake/SQL, and other data integration tools. As a lead, you will drive the design, development, and deployment of data pipelines, leveraging advanced cloud technologies to support scalable, high-performance analytics platforms. This role also demands proficiency in workflow orchestration tools such as Airflow or similar, alongside AWS CloudFormation for infrastructure management. Strong experience with ETL tools like Informatica is mandatory.Key Responsibilities: Lead the design, development, and deployment of data pipelines in cloud environments. Collaborate with cross-functional teams to architect scalable, resilient data solutions. Develop, test, and maintain ETL processes using Informatica or similar tools. Implement and manage orchestration workflows using Airflow or equivalent. Automate infrastructure setup and management using AWS CloudFormation. Build and manage data pipelines in Snowflake with complex SQL queries for data transformation. Leverage AWS Glue/Lambda for data integration and serverless computation. Write efficient and optimized Python/PySpark scripts for data processing. Ensure best practices in coding, security, and compliance within the data engineering team.Required Skills: Python/PySpark Scripting: Proficiency in data manipulation and processing using Python and PySpark. Glue/Lambda: Strong hands-on experience with AWS services like Glue for ETL and Lambda for serverless applications. Snowflake/SQL: Expertise in building scalable data models, writing complex SQL queries, and working with Snowflake. AWS CloudFormation: Ability to automate and manage cloud infrastructure using CloudFormation templates. Airflow (or equivalent): Experience with orchestration tools such as Airflow to manage workflow automation. Informatica (or similar ETL tools): Proven experience in ETL development using Informatica or any comparable tool. Strong knowledge of AWS ecosystem and services. Excellent problem-solving skills with a focus on scalable solutions. Strong communication skills and the ability to lead a team.Preferred Qualifications: Bachelors or Masters degree in Computer Science, Data Engineering, or a related field. 7+ years of experience in data engineering with a focus on cloud-based platforms. AWS certification is a plus. Experience in Agile development practices. Keywords: Illinois Need Profiles :: Lead Data Engineer in Chicago, IL :: Hybrid - Locals preferred [email protected] |
| [email protected] View all |
| Wed Oct 09 02:49:00 UTC 2024 |