| Lead Data Engineer at Atlanta, GA [Hybrid Only locals] at Atlanta, Georgia, USA |
| Email: [email protected] |
| From: Devyani Kumari, Absolute IT [email protected] Reply to: [email protected] Lead Data Engineer Atlanta, GA [Hybrid Only locals] Duration 6+ months Min years of exp 11 years We are currently seeking a Lead Data Engineer experienced with cloud conversion/migration, with a preference for Teradata to AWS migration. The Engineer must have Knowledge of Databricks. Must have demonstrable hands-on experience in creating/designing/implementing end-to-end data pipeline solutions. (Data ingestion, Data extraction, Data transformation, Data enrichment, Data loading, Data validation, Data processing, Data monitoring and maintenance, Data governance and security) In-depth understanding of ETL and data ingestion processes used in large-scale data warehouses Position Overview: The Data Engineer is a skilled professional responsible for creating end-to-end data pipelines, managing data ingestion, transformation, enrichment, loading, validation, monitoring, and maintenance processes. This role plays a crucial part in building and optimizing data architecture, ensuring seamless data flow between various systems, and supporting data-driven decision-making. The Data Engineer should possess expertise in cloud migration, Databricks, ETL processes, and data governance, particularly for large-scale data warehouses. Key Responsibilities: End-to-End Data Pipelines : Design, implement, and maintain robust data pipelines that cover the entire data lifecycle, from data ingestion to data loading, and ensure the efficient flow of data through the entire process. Data Ingestion: Develop and manage data ingestion processes to collect data from different sources, such as databases, APIs, logs, and other data repositories, ensuring data is collected in a reliable and scalable manner. Data Transformation: Perform data transformations and data enrichment to convert raw data into a structured and usable format, preparing it for storage and analysis. Data Loading: Implement data loading procedures to efficiently store processed data into data warehouses, data lakes, or other storage systems, ensuring data quality and integrity. Data Validation : Design and implement data validation mechanisms to verify data accuracy, consistency, and completeness, ensuring the data is reliable for analytical purposes. Data Processing: Utilize various data processing technologies and frameworks to handle large-scale data volumes efficiently and effectively. Data Monitoring and Maintenance: Establish monitoring processes to ensure data pipelines run smoothly, and proactively address any issues that arise. Regularly maintain and optimize data pipelines for better performance and scalability. Cloud Migration : Plan and execute cloud migration strategies, transferring data and applications to cloud-based platforms while ensuring security, scalability, and cost-efficiency. Databricks Expertise : Leverage Databricks and Apache Spark to build scalable and high-performance data processing workflows, providing insights and analytics on large datasets. ETL and Data Ingestion Processes : Understand and implement ETL (Extract, Transform, Load) processes used in large-scale data warehouses, ensuring efficient data movement and integration. Data Governance and Security : Implement data governance best practices and security measures to protect sensitive data and ensure compliance with data privacy regulations. Qualifications and Skills: Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. Proven experience as a Data Engineer or similar role with a strong focus on data pipeline development. Expertise in creating end-to-end data pipelines, including data ingestion, extraction, transformation, loading, and validation. Proficiency in cloud migration, particularly with cloud platforms like AWS, Azure, or Google Cloud. Experience with Databricks and Apache Spark for large-scale data processing and analytics. In-depth knowledge of ETL processes and best practices used in large-scale data warehouses. Knowledge of big data platforms such as Hadoop, Spark, HBase, and Teradata SQL Databases: Oracle DB, DB2 - On-Prem and AWS (preferred) Strong programming skills in languages like Python, SQL, or Java for data manipulation and scripting. Knowledge of distributed data storage systems, data lakes, and data warehousing concepts. Excellent problem-solving and analytical skills, with an ability to handle complex data-related challenges. Strong communication and collaboration skills to work effectively with cross-functional teams. Nice to have Precisely Connect Fivetran Transformations Prophecy As a Data Engineer, the individual plays a pivotal role in designing and implementing robust data pipelines and facilitating smooth data integration, empowering organizations to harness the power of data for informed decision-making and business growth. Keywords: database information technology Georgia |
| [email protected] View all |
| Fri Dec 01 00:58:00 UTC 2023 |