| Azure Datanricks Engineer - Rate 40$ (need OPT Candidates) at Remote, Remote, USA |
| Email: rohitk@insourcetechsolutions.com |
|
https://jobs.nvoids.com/job_details.jsp?id=2303310&uid= Job Title: Azure Datanricks Engineer - Rate 40$ Location: Remote Job Description: We are seeking an experienced Databricks Data Engineer to build and maintain robust, scalable data pipelines and workflows. The ideal candidate will have strong knowledge of Databricks, cloud platforms (Azure), and big data processing frameworks. You will be responsible for working on data processing from raw layers to production-ready datasets, utilizing both Databricks and Azure tools for efficient and reliable data engineering practices. Key Responsibilities: Process data from the Raw (Bronze) layer to the Silver layer in Databricks, transforming raw data into structured, clean datasets for analysis. Pull data from Azure Blob Storage into Databricks and configure the required connections, ensuring data is ingested seamlessly into the platform. Understand and utilize SQL Pools in Azure Synapse Analytics, applying them to improve the performance of large-scale data queries and transformations. Implement performance optimization techniques in Databricks, such as caching, partitioning, and tuning Spark configurations. Utilize Auto Loader in Databricks to automatically ingest data from cloud storage in near real-time. Integrate and leverage Synapse Serverless SQL Pools for serverless querying and scaling in your data workflows. Process multiple Parquet files from a control table and append them to Delta tables with schema enforcement, ensuring data integrity. Integrate and configure connections from various data sources using Databricks, including both structured and semi-structured data. Work with Oracle databases within data engineering pipelines for data extraction and transformation. Handle a variety of file formats (e.g., Parquet, CSV, JSON, Delta) to support data workflows and transformation needs. Collaborate with data scientists, analysts, and other engineers to ensure optimal performance and integration across data systems. Ensure best practices in data security, governance, and compliance across the data pipeline. Required Skills and Experience: Strong experience working with Databricks for building and managing data pipelines, from ingestion to transformation. Proficiency in using Apache Spark (PySpark/Scala) and Delta Lake for big data processing and management. Experience working with Azure Blob Storage, configuring data ingestion from Azure to Databricks. Solid understanding of SQL Pools in Azure Synapse Analytics and their application in large-scale data transformations. Proven ability to optimize Spark jobs in Databricks for performance (e.g., through partitioning, caching, optimizing Spark configurations). Hands-on experience with Auto Loader in Databricks for automating real-time data ingestion. Proficient in integrating Databricks with Synapse Analytics to enable seamless workflows across platforms. Expertise in working with Parquet and Delta formats, including schema enforcement and schema evolution. Experience with integrating multiple data sources within Databricks, including databases like Oracle and other third-party data systems. In-depth knowledge of Azure Data Factory, including its Integration Runtime and data movement capabilities. Familiarity with various file formats in data pipelines, including CSV, JSON, Parquet, and Delta. Strong programming skills in Python, Scala, or SQL. Excellent communication and problem-solving skills, with the ability to collaborate across teams. Preferred Qualifications: Experience working with serverless SQL pools in Synapse Analytics. Knowledge of machine learning integration in Databricks and its use in data pipelines. Familiarity with version control systems like Git and CI/CD processes. Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field. -- Thanks & Regards, Rohit Technical Recruiter rohitk@insourcetechsolutions.com www.insourcetechsolutions.com Marion rd, Westport, CT 06880, -- Keywords: continuous integration continuous deployment information technology Connecticut Azure Datanricks Engineer - Rate 40$ (need OPT Candidates) rohitk@insourcetechsolutions.com https://jobs.nvoids.com/job_details.jsp?id=2303310&uid= |
| rohitk@insourcetechsolutions.com View All |
| 06:50 PM 01-Apr-25 |