| DataBricks Lead - Remote(EST)// Need GC/USC/H4 at Remote, Remote, USA |
| Email: [email protected] |
| Title: DataBricks Lead Duration: 12+ months Location: EST (Remote) Must Have: Databricks PL/SQL Python Any Cloud Experience Experience in implementation of Operational Data Model or Data Mart using Clinical & Operational data Experience in Databricks Data Engineering to create Data Lake solutions using AWS services. Knowledge of Databricks cluster and SQL warehouse, Experience in Delta and Parquet file handling Experience in Data Engineering and Data Pipeline creation on Databricks Extensive Experience in SQL, PL/SQL, complex Join, Aggregation function and DBT, Python, Data frames and Spark Experience in Airflow for Job Orchestration, dependency Setup and Job scheduling Knowledge of Databricks Unity Catalog and Consumption patterns Knowledge of GitHub and CI/CD Pipelines Role Responsible for authoring SQL and Python scripts on Databricks and DBT (Data Build tool) to create data pipelines to create Operational Data Mart. Responsible for creation of Data Pipelines for Data processing of Delta files into ODM format for downstream data consumption Responsible for identifying data set relationship, join criteria and implement it in code for ODM model development. Responsible for creation of Delta Lake for ODM model and setup of consumption pattern using Databricks Unity catalog Responsible for creation of Airflow DAGs for job orchestration and scheduling of data pipeline jobs Anuj Kaushik (732) 802-7547 [email protected] SwankTek, Inc | An E-Verified Firm 510 Franklin Avenue, Suite 6,7 & 8 Nutley, NJ 07110 | www.swanktek.com Keywords: continuous integration continuous deployment information technology procedural language New Jersey |
| [email protected] View all |
| Wed Feb 14 02:37:00 UTC 2024 |