| Data Engineer - EX Capital One Candidates at Mclean, Virginia, USA |
| Email: [email protected] |
| From: Devendra, Northite [email protected] Reply to: [email protected] Job Title: Data Engineer Location: Richmond, VA or MCLean, VA or Plano, TX (Onsite) Preferred EX Capital One Candidates only Key Responsibilities: Create, maintain, and optimize ETL/ELT pipelines to ingest, process, and manage data from various sources using Python, Apache Spark, and AWS services. Design data models, build data structures, and implement data storage solutions that ensure data integrity, consistency, and security. Tune data processing workflows for performance, scalability, and cost efficiency on distributed systems using Spark and AWS. Work with cross-functional teams (e.g., data science, product, analytics) to understand data requirements and support business needs. Document data workflows, processes, and solutions for transparency and reproducibility Implement data quality checks, error handling, and recovery processes. Ensure compliance with data governance and security protocols. Key Qualifications: Proficient in Python for data processing, scripting, and automation. Experience with Spark for data transformation, distributed processing, and ETL workflows. Hands-on experience with core AWS services like S3, Lambda, Glue, EMR, Redshift, and RDS. Knowledge of IAM, CloudFormation, and/or Terraform for infrastructure management is a plus. Strong understanding of SQL, data warehousing, and database design principles. Familiarity with data modeling, schema design, and query optimization. Other Skills: Experience with version control (Git) and CI/CD practices. Strong problem-solving skills and ability to work in an Agile environment. Excellent communication skills and ability to work with non-technical stakeholders. Preferred Qualifications: Familiarity with additional tools like Airflow for workflow orchestration. Experience with data streaming technologies (e.g., Kafka, Kinesis). Keywords: continuous integration continuous deployment sthree Texas Virginia Data Engineer - EX Capital One Candidates [email protected] |
| [email protected] View all |
| Tue Nov 12 01:43:00 UTC 2024 |