| Data Engineer with Python, Kube, Big Data, AlgoCD and AirFlow : onsite Austin or Sunnyvale : 55 perhr max at Sunnyvale, California, USA |
| Email: [email protected] |
| From: Jay, Brillius [email protected] Reply to: [email protected] Data Engineer with Python, Kube, Big Data, AlgoCD and AirFlow Austin/ Sunnyvale * Experience in monitoring and troubleshooting large scale data platforms, data pipelines, complex backend services * Able to effectively communicate incidents, coordinate incident responses, document/manage runbooks, set up processes to support other software engineering teams and data analysts * Experience in managing large datasets * Experience in creating Kubernetes configurations, managing services in Kubernetes, building docker images * Demonstrate good understanding & troubleshooting Spark jobs * Experience in developing/maintaining automation tools in programming/scripting languages like Python * Demonstrate good understanding in Big data eco-systems such as HDFS, Kafka, SQL etc. * Experience in using IT automation tools such as AlgoCD, and job orchestrator systems such as AirFlow, Jenkins * Experience in cloud based environment (AWS/GCP etc.) is a big plus. * Experience in monitoring tools such as Splunk, Prometheus/Grafana is a big plus. * Experience with Full-Stack web development (React, Django etc.) is a big plus. Responsibilities * Support large scale data pipelines and backend services by monitoring and troubleshooting/recovering incidents * Manage large volume datasets in Spark and Hadoop environments using scripts and automation * Build and operate data management infra services * Automate build, deployment, and monitoring * Participate at an onCall & release turn * Taking an oncall turn including weekend coverage Keywords: information technology |
| [email protected] View all |
| Thu Aug 24 21:19:00 UTC 2023 |