| Sr. Data Engineer, Des Moines, IA (CAN Automotive Exp Required) at Remote, Remote, USA |
| Email: [email protected] |
| From: Abhishek, StellentIT [email protected] Reply to: [email protected] Sr. Data Engineer, Des Moines, IA 12+ Months Phone + Skype Masters in Data Science 7+ years as a Data Engineer creating data dashboards, summarizing data parameters, and supporting more simplistic access to logs 7+ years experience working with python for scripting and data manipulation Strong background in the automotive industry from ford, and exposure to CAN messaging and protocols Expertise in databricks (3+ years) and AWS for cloud applications Project finishing in a week, can interview right away and start with a 1-2 weeks notice Brother works at John Deere in Moline and has for several years. He has heard great things and its a company he really is hoping to work for! Role summary: The Data Engineering role for FurrowVision is intended to make it easier for team members and experts to access data, fetch and assemble data sets, and generate insights. The Data Engineer will be responsible for wrangling data from multiple streaming sources (AWS S3 buckets, Enterprise data lake, TEL, over the air, etc.) into data sets contained in the EDL in the form of Delta Tables in the Databricks platform. From those data sets, the Data Engineer should be able to query results that are of interest given requests from the broader team. Typical responsibilities: Data dashboards for incoming data, summary of data set parameters, support easier access to specific logs Support one-off data analysis at smaller scale. Ex: What was the uptime for customer X, what are the metrics around the frequency and duration of down time Can you create the data set to investigate what happened Required competencies: Professional Python proficiency Need databricks Working knowledge of CAN protocols and CAN message parsing. Comfortable dealing with large amounts of data from multiple sources, cleaning data, and polishing datasets Knowledge of ETL (Extract, Transform, Load) processes and best practices, data warehousing, and database management. Experience with tools such as MongoDB, AWS Athena, or Databricks Delta Lake Delta Tables, HiveMeta Store, Enterprise Datalake, Databricks Notebooks Familiar ingesting data ingest data from various internal and external sources, including data warehouses, data lakes, APIs, third-party platforms, and cloud services. Experience with AWS tools such as S3, ECR, and Lambda Familiar with SQL and NoSQL databases, and/or object-relational models. Keywords: sthree information technology Iowa Sr. Data Engineer, Des Moines, IA (CAN Automotive Exp Required) [email protected] |
| [email protected] View all |
| Wed Apr 10 21:58:00 UTC 2024 |