| Data Engineering- Des moines IA(Onsite) at Remote, Remote, USA |
| Email: [email protected] |
| From: pankaj, Stellent IT [email protected] Reply to: [email protected] Data Engineering Des moines IA(Onsite) Phone and Skype Long Term Job Description: The Data Engineer will be responsible for wrangling data from multiple streaming sources (AWS S3 buckets, Enterprise data lake, TEL, over the air, etc.) into data sets contained in the EDL in the form of Delta Tables in the Databricks platform. From those data sets, the Data Engineer should be able to query results that are of interest given requests from the broader team. Typical responsibilities: Data dashboards for incoming data, summary of data set parameters, support easier access to specific logs Support one-off data analysis at smaller scale. Ex: What was the uptime for customer X, what are the metrics around the frequency and duration of down time Can you create the data set to investigate what happened Required competencies: Professional Python proficiency Working knowledge of CAN protocols and CAN message parsing. Comfortable dealing with large amounts of data from multiple sources, cleaning data, and polishing datasets Knowledge of ETL (Extract, Transform, Load) processes and best practices, data warehousing, and database management. Experience with tools such as MongoDB, AWS Athena, or Databricks Delta Lake Delta Tables, Hive MetaStore, Enterprise Datalake, Databricks Notebooks Familiar ingesting data ingest data from various internal and external sources, including data warehouses, data lakes, APIs, third-party platforms, and cloud services. Experience with AWS tools such as S3, ECR, and Lambda Familiar with SQL and NoSQL databases, and/or object-relational models. Keywords: sthree information technology Iowa |
| [email protected] View all |
| Tue Oct 03 23:52:00 UTC 2023 |