| (Locals only) Data Engineer || Minneapolis, MN (Hybrid) at Minneapolis, Minnesota, USA |
| Email: [email protected] |
| From: Shubham, USC Group [email protected] Reply to: [email protected] Role:- Data Engineer Location:- Minneapolis, MN (Hybrid) Only USC, TN Visa, H4-EAD, L2-EAD (Genuine visa only for this role) REQUIRED SKILLS Senior Software Engineer (Big Data - Spark) Primary Responsibilities: Design and build highly available, scalable, resilient and fault tolerant distributed real-time data streaming system. Automate test coverage (90+%) for data pipelines, best practices and frameworks for unit, functional and integration tests. Automate CI and deployment processes and best practices for the production data pipelines. Build AI/ML model based alert mechanism and anomaly detection system for the product. The goal is have a self-annealing product. Basic Qualifications Azure, Spark, Kafka experience required Bachelors degree or equivalent work experience. Preferred Skills/Experience 10+ years of overall experience in software development with 5-6 years of relevant experience in designing, developing, deploying and operating data streaming pipelines at scale. 3-4 years experience with Apache Kafka and Apache Spark Streaming Hands on experience with Spark Structured Streaming Experience in tuning Spark Data pipeline to achieve high throughput. Drive efforts to improve the data quality across data pipelines and implement system controls for managing data quality Programming proficiency in Scala or Java. Open Source Committer (Apache Spark or related Big Data open source technologies) Experience with Containers, Kubernetes and scaling elastically Strong background in algorithms and data structures and continuously develop and acquire new technical skills Lead and mentor junior engineers to ensure systems are built with highest quality and leveraging best practices. Experience in automating Spark pipeline deployment/testing (DevOps, CI/CD) Passion for data engineering and for enabling others by making the product easier to use. Excellent communication in sharing context to effectively collaborate with analytical partners, domain experts and other consumers of your work, preferably in supporting an engineering or product function Required Skills Summary: Apache Spark, Apache Kafka, Scala/Java, NoSQL Databases, Elasticsearch & Kibana, Kubernetes, Docker Containers Keywords: continuous integration continuous deployment artificial intelligence machine learning trade national Minnesota Tennessee (Locals only) Data Engineer || Minneapolis, MN (Hybrid) [email protected] |
| [email protected] View all |
| Mon Aug 05 20:43:00 UTC 2024 |