| Sr. Data Engineer, Des Moines, IA (Onsite) CAN Protocol Exp required at Remote, Remote, USA |
| Email: [email protected] |
| From: abhishek, StellentIT [email protected] Reply to: [email protected] Sr. Data Engineer, Des Moines, IA 12+ Months Phone + Skype Role summary: The Data Engineering role for FurrowVision is intended to make it easier for team members and experts to access data, fetch and assemble data sets, and generate insights. The Data Engineer will be responsible for wrangling data from multiple streaming sources (AWS S3 buckets, Enterprise data lake, TEL, over the air, etc.) into data sets contained in the EDL in the form of Delta Tables in the Databricks platform. From those data sets, the Data Engineer should be able to query results that are of interest given requests from the broader team. Typical responsibilities: Data dashboards for incoming data, summary of data set parameters, support easier access to specific logs Support one-off data analysis at smaller scale. Ex: What was the uptime for customer X, what are the metrics around the frequency and duration of down time Can you create the data set to investigate what happened Required competencies: Professional Python proficiency Working knowledge of CAN protocols and CAN message parsing. Comfortable dealing with large amounts of data from multiple sources, cleaning data, and polishing datasets Knowledge of ETL (Extract, Transform, Load) processes and best practices, data warehousing, and database management. Experience with tools such as MongoDB, AWS Athena, or Databricks Delta Lake Delta Tables, HiveMeta Store, Enterprise Datalake, Databricks Notebooks Familiar ingesting data ingest data from various internal and external sources, including data warehouses, data lakes, APIs, third-party platforms, and cloud services. Experience with AWS tools such as S3, ECR, and Lambda Familiar with SQL and NoSQL databases, and/or object-relational models. Keywords: sthree information technology Iowa |
| [email protected] View all |
| Tue Oct 03 22:18:00 UTC 2023 |