| Azure Databricks developer || Auburn Hills, MI - Onsite || Contract at Auburn Hills, Michigan, USA |
| Email: [email protected] |
| From: Sandeep Sharma, Siri Info [email protected] Reply to: [email protected] Hello, Hope you are doing good.!! Please let me know your interest for below position. Position- Azure Databricks developer Location- Auburn Hills, MI - Onsite Job Type-Contract Job Description : Develop deep understanding of the data sources, implement data standards, and maintain data quality and master data management. Expert in building Databricks notebooks in extracting the data from various source systems like DB2, Teradata and perform data cleansing, data wrangling, data ETL processing and loading to AZURE SQL DB. Expert in building Ephemeral Notebooks in Databricks like wrapper, driver and config for processing the data, back feeding the data to DB2 using multiprocessing thread pool. Expert in developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data. Expert in using Databricks with Azure Data Factory (ADF) to compute large volumes of data. Performed ETL operations in Azure Databricks by connecting to different relational database source systems using jdbc connectors. Developed Python scripts to do file validations in Databricks and automated the process using ADF. Analyzed the SQL scripts and designed it by using Pyspark SQL for faster performance. Worked on reading and writing multiple data formats like JSON, Parquet, and delta from various sources using Pyspark. Developed an automated process in Azure cloud which can ingest data daily from web service and load in to Azure SQL DB. Expert in optimizing the Pyspark jobs to run on different Cluster for faster data processing. Developed spark applications in python (Pyspark) on distributed environment to load huge number of CSV files with different schema in to Pyspark Dataframes and process them to reload in to Azure SQL DB tables. Analyzed data where it lives by Mounting Azure Data Lake and Blob to Databricks. Used Logic App to take decisional actions based on the workflow and developed custom alerts using Azure Data Factory, SQLDB and Logic App. Developed Databricks ETL pipelines using notebooks, Spark Dataframes, SPARK SQL and python scripting. Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns. Expert in understanding current production state of application and determine the impact of new implementation on existing business processes. Expert in ingesting streaming data with Databricks Delta tables and Delta Lake to enable ACID transaction logging. Expert in building Delta Lake On top Of Data Lake and performing transformations in Delta Lake. Thanks & Regards Sandeep Sharma Sr. Technical Recruiter : [email protected] , URL: www.siriinfo.com Siri InfoSolutions Inc. 3 Ethel Rd, Suite # 302, Edison NJ 08817. Disclaimer: We respect your online privacy. If you would like to be removed from our mailing list please reply with "Remove" in the subject and we will comply immediately. We apologize for any inconvenience caused. Please let us know if you have more than one domain. The material in this e-mail is intended only for the use of the individual to whom it is addressed and may contain information that is confidential, privileged, and exempt from disclosure under applicable law. If you are not the intended recipient, be advised that the unauthorized use, disclosure, copying, distribution, or the taking of any action in reliance on this information is strictly prohibited. We are an equal opportunity employer with a diverse workforce. Note : Any resume submitted by Siriinfo is presented with the understanding that the candidate is being considered for your direct end-client (end-client is the company where the work will be performed). If there is any other company involved between the end-client and your company, please do not submit this resume without our written approval. If you submit the resume to another third party, Siriinfo reserves the right to work with the third party directly. Keywords: database information technology Michigan New Jersey |
| [email protected] View all |
| Thu Sep 07 18:48:00 UTC 2023 |