| AWS Data Engineer - Cincinnati, OH - REMOTE at Cincinnati, Ohio, USA |
| Email: [email protected] |
| From: Ashwini Lokhande, ChabezTech LLC [email protected] Reply to: [email protected] Hello, Please review the JD below and let me know your interest. Role - AWS Data Engineer Location: Cincinnati, OH - REMOTE Experience : 12+ years Data Engineer with Data Stage and Snowflake MUST JOB DESCRIPTION: Key Responsibilities: Purpose of a Stream: Understand the concept of data streams and design real-time data processing pipelines to enable near real-time analytics and decision-making. Language Choice: Utilize Python as the primary programming language for data engineering tasks, leveraging its rich ecosystem of libraries and frameworks for data manipulation, transformation, and integration. Loading Data into Snowflake Tables: Implement data loading strategies and techniques that ensure data integrity and eliminate duplicates when loading data into Snowflake tables. Airflow-Snowflake Integration: Configure Airflow to seamlessly invoke and interact with Snowflake, ensuring smooth data pipeline execution and monitoring. Big Data and Hadoop: Demonstrate experience and knowledge in working with Big Data technologies and Hadoop ecosystem, including data ingestion, processing, and storage. Big Data File Processing: Experience in handling and processing large-scale Big Data files, such as Parquet, Avro, or ORC, optimizing performance and data retrieval. Excel File in Snowflake: Proficiently load and query Excel files in Snowflake, leveraging appropriate data connectors or integration tools. SQS and SNS Experience: Utilize Amazon Simple Queue Service (SQS) and Simple Notification Service (SNS) to build scalable and reliable event-driven data processing architectures. Event Tables: Implement event-based data architectures using event tables, capturing and processing real-time events for timely data updates and analytics. Dataframes: Work with various data manipulation frameworks and libraries such as Pandas, PySpark, or Dask, leveraging their capabilities to process and transform data efficiently. Qualifications: Bachelor's degree in Computer Science, Data Engineering, or a related field. Proven experience as a Data Engineer or similar role, with a strong focus on data pipeline design, implementation, and optimization. Proficient in Python and its data manipulation libraries (e.g., Pandas, NumPy). Familiarity with Apache Airflow or similar workflow orchestration tools. Experience with Snowflake data warehouse and associated functionalities. Strong SQL skills and knowledge of database concepts. Familiarity with Big Data technologies and Hadoop ecosystem. Excellent problem-solving and analytical skills. Strong attention to detail and ability to deliver high-quality work. Strong communication and collaboration skills to work effectively with cross-functional teams. Join our team and contribute to building a scalable and efficient data infrastructure that empowers data-driven decision-making across our organization. Keywords: Ohio |
| [email protected] View all |
| Wed Jan 31 23:58:00 UTC 2024 |