| IT Engineer (Data Engineer) at Merrifield, Virginia, USA |
| Email: [email protected] |
| From: Jyothi, NCS [email protected] Reply to: [email protected] Hi , Hope youre doing great today, Role: IT Engineer (Data Engineer) Location: Merrifield, VA (Remote) 6+ Months Long Term Contract Need GC, USC, H4EAD visa only Description: Open to Hybrid or Remote Mid to Senior level with 5+ years of min. Experience needed on Data Engineering skills on Azure Cloud. Responsibilities Design and implement highly performant data ingestion & transformation pipelines from multiple sources using Azure Databricks and PySpark. SPARK Performance\\tuning\\optimization. Identify and optimize long running Databricks SPARK jobs. Monitor resource utilization and costs associated with Databricks usage, identifying opportunities for optimization and cost reduction. Analyze cluster performance metrics and implement tuning strategies to ensure optimal query execution times and resource utilization. Develop scalable and re-usable frameworks for ingestion and transformation of large data sets. Hand-On Experience in creating ADF Pipelines and Orchestrate various notebooks in Azure Data Factory. Work closely with data engineers to ensure that our Databricks platform is optimized for performance, scalability, and reliability. Ensure the quality of deliverables by developing automated controls and performing unit, integration, and user acceptance testing. Perform other duties as assigned. Skills Needed: Significant experience building Data Warehouses (Azure Synapse Analytics or similar), Data Lakes (Azure Data Lake or similar), ETL/ELT pipelines (Databricks or similar), and nice to have skills on data streaming (Azure Event Hub, Kafka, Cosmos, MongoDB, or similar). Expert knowledge of SQL. Experience in data engineering programming languages. (Azure Databricks , PySpark, Python,). Expertise on Azure Cloud platform. Keywords: information technology green card Virginia IT Engineer (Data Engineer) [email protected] |
| [email protected] View all |
| Wed May 15 21:31:00 UTC 2024 |