| Hybrid Work -- Lead Data Engineer in Atlanta, GA (on-site as needed, up to 2 days per week) at Atlanta, Georgia, USA |
| Email: [email protected] |
| From: Vivek Yadav, SoniTalent corp [email protected] Reply to: [email protected] Hi, Please find the below job description and let me know your thoughts. Job Title Lead Data Engineer Job Location Atlanta, GA (on-site as needed, up to 2 days per week). Need to be local or very nearby Duration 6 Months+ Visa -Any Visa except H1B,CPT,OPT Only Mode Of Interview - Phone/Skype Job Description Seeking an experienced hands-on enterprise Data Engineer lead. The successful candidate must have Big Data engineering experience and must demonstrate an affinity for working with others to create successful solutions. They must be a great communicator, both written and verbal, and have some experience working with business areas to translate their business data needs and data questions into project requirements. The candidate will participate in all phases of the Data Engineering life cycle and will work independently and collaboratively write project requirements, architect solutions, and perform data ingestion development and support duties. This lead Data Engineer will be responsible for creating new data flows into AWS including coordinating with business and functional areas to establish and communicate our data fabric best practices, framework, and tools. Together we will establish a new data fabric for Norfolk Southern that will help create a common view of data and provide a centralized mechanism for its aggregation, cleansing, transformation, augmentation, validation, and syndication. Responsibilities: Sharing project solutions and outcomes with colleagues to improve delivery on future projects Analyzing and translating business needs into long-term solution data pipelines. Evaluating existing data systems. Working with the development team to create conceptual data flows. Developing best practices for data coding to ensure consistency within the system. Reviewing modifications of existing systems for cross-compatibility. Implementing data strategies and developing data integration points . Evaluating implemented data systems for variances, discrepancies, and efficiency. Troubleshooting and optimizing data systems. Interpreting and delivering impactful strategic plans improving data integration, data quality, and data delivery in support of business initiatives and roadmaps Formulating and articulating architectural trade-offs across solution options before recommending an optimal solution ensuring technical requirements are met Motivating and developing staff through teaching, empowering, and influencing technical and consulting soft skills Driving innovative technology solutions through thought leadership on emerging trends Skills: Required: 6+ years of experience with high-velocity high-volume stream processing: Apache Kafka and Spark Streaming Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka Deep knowledge of troubleshooting and tuning Spark applications 3+ years of experience with data ingestion from Message Queues (Tibco, IBM, etc.) and different file formats across different platforms like JSON, XML, CSV 3+ years of experience with Big Data tools/technologies like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, or HDFS 3+ years of experience building, testing, and optimizing Big Data data ingestion pipelines, architectures, and data sets 2+ years of experience with Python (and/or Scala) and PySpark/Scala-Spark 3+ years of experience with AWS cloud platform 3+ years of experience with database solutions like Databricks or Snowflake 2+ years of experience with NoSQL databases, including HBASE and/or Cassandra Knowledge of Unix/Linux platform and shell scripting is a must Strong analytical and problem-solving skills Preferred (Not Required): Experience with Cloudera/Hortonworks CDP, HDP and HDF platforms Strong SQL skills with ability to write intermediate complexity queries Strong understanding of Relational & Dimensional modeling Experience with GIT code versioning software Experience with REST API and Web Services Good business analyst and requirements gathering/writing skills Keywords: sthree Georgia |
| [email protected] View all |
| Tue Aug 01 02:32:00 UTC 2023 |