| Data Engineer , Richmond, VA or MCLean, VA or Plano, TX at Mclean, Virginia, USA |
| Email: [email protected] |
| From: Asha, Northite [email protected] Reply to: [email protected] Job Title: Data Engineer Location: Richmond, VA or MCLean, VA or Plano, TX Duration: 12 Months Preferred EX Capital One Candidates only ( please share only EX Capital one candidates) Key Responsibilities: Create, maintain, and optimize ETL/ELT pipelines to ingest, process, and manage data from various sources using Python, Apache Spark, and AWS services. Design data models, build data structures, and implement data storage solutions that ensure data integrity, consistency, and security. Tune data processing workflows for performance, scalability, and cost efficiency on distributed systems using Spark and AWS. Work with cross-functional teams (e.g., data science, product, analytics) to understand data requirements and support business needs. Document data workflows, processes, and solutions for transparency and reproducibility Implement data quality checks, error handling, and recovery processes. Ensure compliance with data governance and security protocols. Key Qualifications: Proficient in Python for data processing, scripting, and automation. Experience with Spark for data transformation, distributed processing, and ETL workflows. Hands-on experience with core AWS services like S3, Lambda, Glue, EMR, Redshift, and RDS. Knowledge of IAM, CloudFormation, and/or Terraform for infrastructure management is a plus. Strong understanding of SQL, data warehousing, and database design principles. Familiarity with data modeling, schema design, and query optimization. Other Skills: Experience with version control (Git) and CI/CD practices. Strong problem-solving skills and ability to work in an Agile environment. Excellent communication skills and ability to work with non-technical stakeholders. Preferred Qualifications: Familiarity with additional tools like Airflow for workflow orchestration. Experience with data streaming technologies (e.g., Kafka, Kinesis). [email protected] Keywords: continuous integration continuous deployment sthree Texas Virginia Data Engineer , Richmond, VA or MCLean, VA or Plano, TX [email protected] |
| [email protected] View all |
| Fri Nov 15 02:33:00 UTC 2024 |