| (10+ years only)Immediate job opening for Senior Data Engineer and it's a 100% Remote position at Remote, Remote, USA |
| Email: [email protected] |
| From: Chandra, RHG [email protected] Reply to: [email protected] Hi Professional, Hope you're doing great! Looking for 10+ year candidates only Title: Senior Data Engineer Location: 100% Remote Duration: Long term contract Job Description: Mandatory Skills: Spark expertise, S3 storage, data management, Rapid prototyping & Agile Regarding years of experience and qualifications +10 years of relevant experience Senior Data Engineer / Spark / S3 / Rapid prototyping expert Establish fully functional object store compatible with S3 access APIs which will: Provide ability to define fine grain access control lists on HCP objects Support multiple storage tiers in the view of cost optimization and allows moving objects between storage tiers Support event notifications over objects, allowing a subscription model to build data pipelines Record all the object level operations, and expose access logs for audit Support multi-part uploads to move handle large objects into the store Support replication of objects between different object stores Provide comprehensive monitoring and observability metrics, generate alerts on configured thresholds Create appropriate documentation for the activities above Integrate Iceberg with HCP, Spark, and Trino which will: Provide a central metastore or catalog pluggable with multiple compute engines Support time travel, automated maintenance like purging, compression Support native reads on Iceberg, which can be leveraged by compute engines Create appropriate documentation for the activities above Spark S3, Apache Iceberg (Nice to have) or similar Can search and investigate on tools PoC approach implementation experience Delivery Ownership Autonomous Best Regards, Chandra Keywords: sthree |
| [email protected] View all |
| Sat Jan 14 00:29:00 UTC 2023 |