Home

Teja R - Lead-Azure Data Engineer | ETL & Big Data Specialist | AI-Driven Data Solutions

roy@beaninfosys.com

Location: Dallas, Texas, USA

Relocation: Yes

Visa: H1B

TEJESWAR Contact: tejareddy4698@gmail.com || +1469-919-0104 Lead-Azure Data Engineer | ETL & Big Data Specialist | AI-Driven Data Solutions Professional Summary: Results-driven Azure Data Engineer with 10+ years of expertise in designing and optimizing data pipelines, ETL workflows, and cloud-based architectures. Proficient in Azure Data Factory (ADF), Databricks, Synapse Analytics, Snowflake, and AI-powered data automation. Passionate about integrating LLMs, Open-AI, and Generative AI into data engineering workflows for enhanced business intelligence, automation, and predictive analytics. Hands-on experience in Azure Cloud Services (PaaS & IaaS), including Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis Services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake, Snowflake, Azure Databricks, Apache Airflow, Azure Functions, and Power Automate. Strong working knowledge and understanding of Data Warehouse Concepts, Data Lake, Delta Lake, and Microsoft Fabric. Proficient in integrating Generative AI models into data engineering workflows, optimizing data pipelines, and enabling advanced data insights. Passionate about leveraging AI to solve complex data challenges and drive business intelligence in cloud-based environments. Developed and deployed Databricks Workflows/Jobs using Databricks Notebooks built on PySpark and Spark SQL. Implemented CI/CD pipelines using Azure DevOps to deploy ADF, Synapse, and Databases. Integrated Azure OpenAI and other LLMs for automating data transformation, anomaly detection, and metadata enrichment. Designed a Conversational AI assistant that allows business users to query the data warehouse using natural language. Used Snowpark for Python to execute ML models and complex transformations directly within Snowflake. Implemented Snowflake-based ETL workflows using Azure Data Factory, DBT, and Python for ingesting structured & semi-structured data. Developed Spark jobs to process JSON, CSV, Parquet, and text files using Spark Data Frames to perform joins, transformations, and data storage in ADLS. Created Python scripts for file validation in Databricks and automated the process using Azure Data Factory. Expertise in migrating SQL databases to Azure Data Lake, Azure Data Lake Analytics, Azure SQL Database, Databricks, and Azure SQL Data Warehouse (Synapse Analytics). Controlled and managed database access and migrated on-premise databases to Azure Data Lake Store using Azure Data Factory. Extensive experience using Azure DevOps for CI/CD pipelines and ARM Templates for deployments. Developed Data Forecasting Techniques using Predictive Analysis in Azure Machine Learning and presented insights via Power BI. Built and deployed classification, regression, and clustering models using Scikit-Learn for data-driven decision-making. Developed efficient data processing pipelines using NumPy and Pandas, leveraging vectorized operations, multi-threading, and memory optimization techniques to handle large-scale structured and unstructured datasets for ETL, analytics, and machine learning workflows. Created complex SQL queries using stored procedures, common table expressions (CTEs), and temporary tables to support Power BI reports. Implemented Change Data Capture (CDC) with Delta Lake MERGE, ensuring real-time updates for downstream analytics. Implemented dynamic DAG generation using Python enabling parameterized workflows for multiple data sources. Implemented Automated PySpark job execution using Airflow DAGs and Azure Data Factory. Built streaming data pipelines using PySpark Structured Streaming, enabling real-time processing of Kafka and event-driven data Core Technical Skills: Cloud & Data Platforms Azure Data Factory (ADF), Azure Synapse Analytic, Azure Data bricks, Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, Cosmos DB, Microsoft Fabric, Snowflake, AWS Redshift. BigData & ETL Tools Apache Spark, PySpark, Delta Lake, Apache Airflow, DBT (Data Build Tool), SSIS, Informatica, Data Orchestration, Data Integration, Data Transformation, Data Governance Programming & Scripting Python, SQL (T-SQL, PL/SQL, Spark SQL), Scala, Java, Shell Scripting, DAX (Power BI) AI & Machine Learning for Data Engineering OpenAI, Generative AI (GenAI), Large Language Models (LLMs), AI-powered ETL, Predictive Analytics, NLP, Anomaly Detection, Feature Engineering, Model Deployment DevOps & CI/CD Azure DevOps, Terraform, GitHub Actions, Jenkins, Docker, Kubernetes, Infrastructure as Code (IaC), ARM Templates Security & Compliance Role-Based Access Control (RBAC), Data Masking, Data Encryption, GDPR Compliance, Azure Key Vault, IAM (Identity & Access Management) Business Intelligence & Reporting Power BI, Tableau, KPI Reporting, Business Intelligence, Cross-functional Collaboration, Agile & Scrum. Certifications: ISTQB Foundation Level Microsoft Certified Azure Fundamentals (AZ-900) Microsoft (70-778) Analysing and Visualizing Data in Power BI. Microsoft Certified: Azure Data Engineer Associate Generative AI for Data Engineering and Data Professionals SnowPro Certification from Snowflake Education: Bachelor s Degree and Graduation Year: 2015 Work Experience: Client: Heartland Bank, Ohio Duration: Dec 2023 - Feb 2025 Role: Lead Azure Data Engineer Team Size: 12 Role and Responsibilities: Designed and optimized ETL/ELT pipelines across Snowflake and Azure using Databricks and DBT, ensuring data accuracy, consistency, and high-performance data processing. Designed and implemented an AI-powered agent application using ChatGPT-4 to automate SQL query generation based on DDL, enhancing data accessibility for non-technical users. Integrated natural language processing (NLP) capabilities to enable intuitive user interactions, reducing dependency on technical teams for data retrieval. Optimized the AI agent s performance for accurate SQL generation in complex financial data environments, ensuring compliance with data governance standards. Used LLMs to auto-generate data dictionary, lineage documentation, and column-level insights from existing datasets. Deployed and managed the AI agent in a cloud environment, ensuring high availability and scalability for enterprise-level usage. Designed and optimized ETL pipelines, improving data accuracy and consistency across Snowflake and Azure. Implemented DBT models and CI/CD pipelines, ensuring scalable and automated deployments. Developed and implemented DBT (Data Build Tool) models to enhance and modernize the existing codebase. Implemented version control and CI/CD practices for DBT projects, ensuring reproducibility and scalability. Conducted Databricks performance tuning and cost optimization for efficient data processing and storage utilization. Orchestrated DBT workflows using Azure Data Factory (ADF), ensuring timely execution and monitoring of data transformation processes. Conducted performance tuning and optimization of SQL queries and DBT models, improving overall data processing efficiency. Extensive experience with Snowflake Cloud Data Warehouse, with a deep understanding of Snowflake architecture and query optimization. Spearheaded efforts in Snowflake SQL query optimization, improving database performance and reducing query execution times. Diagnosed and resolved issues in SSAS cubes, ensuring data accuracy and reliability for business intelligence and reporting purposes. Technologies & Environment: Snowflake, DBT, Databricks, SSAS, SQL Server, Python, SQL, Azure DevOps, Power BI, Azure Data Factory, VS Code, OpenAI, Gen AI, Apache Airflow. Client: IAG, Australia Duration: Oct 2022 - Nov 2023 Role: Lead Azure Data Engineer Team Size: 6 Role and Responsibilities: Designed and implemented data pipelines using modern architecture for real-time data ingestion and processing. Implemented real-time data ingestion from Azure Data Lake Storage (ADLS) to Databricks using Delta Live Tables & Spark Streaming. Developed a metadata-driven ingestion framework, reducing ETL development time by 40%. Created and maintained data models in Databricks, leveraging distributed computing capabilities for big data processing. Optimized Databricks clusters and Spark jobs for efficient data processing and cost management. Created PySpark jobs to ingest structured and semi-structured data from CSV, JSON, Parquet, MySQL, and Oracle. Utilized PySpark's DataFrame APIs to extract and load data from Azure Blob, ADLS, and HDFS. Used SQL, Python scripts, and data mining techniques for data design, development, and extraction from legacy systems. Migrated on-prem data warehousing and reporting systems to Snowflake, ensuring seamless data transfer, minimal downtime, and performance improvements. Built and managed Apache Airflow DAGs for data orchestration across ADF, Databricks, and Synapse Analytics. Implemented task parallelization, retry mechanisms, and SLA monitoring to improve workflow efficiency and reliability. Connected Airflow with Azure Blob Storage, Synapse, Databricks, and REST APIs for seamless data movement and transformation. Configured Airflow task failure notifications using Slack, Teams, and email alerts for proactive issue resolution. Designed and orchestrated Airflow DAGs for complex ETL workflows, reducing manual intervention and ensuring 99.9% uptime Integrated Apache Airflow with Azure Data Factory and Databricks, enabling seamless end-to-end data pipeline automation. Implemented task parallelization, retry mechanisms, and SLA monitoring to improve workflow efficiency and reliability Connected Airflow with Azure Blob Storage, Synapse, Databricks, and REST APIs for seamless data movement and transformation. Integrated Delta Lake with Azure Synapse and Power BI, enabling direct querying of high-volume data with improved performance. Implemented Unity Catalog for fine-grained access control, ensuring secure and compliant data governance in a multi-user environment Integrated LLMs with Python for automated data cleansing, deduplication, and entity resolution, enhancing data quality. Technologies & Environment: Azure Data Factory, Azure Databricks, Snowflake, Apache Airflow, Delta Lake, PySpark, Python, SQL, Azure DevOps, Unity Catalog, Jira, Slack, REST APIs. Client: Kiwi Wealth, Wellington Duration: April 2021 - Oct 2022 Role: Sr Azure Data Engineer Team Size: 14 Role and Responsibilities: Created and designed an end-to-end solution to migrate multiple on-prem product databases into Azure SQL ODS (Operational Data Store). Designed and implemented a new architectural solution for a single Azure SQL database, consolidating multiple on-prem databases. Led the migration of multiple on-prem databases to Azure SQL Operational Data Store (ODS), ensuring seamless data transfer and minimal downtime. Developed and implemented end-to-end data integration workflows between Kiwi Wealth and vendor systems using Azure Data Factory. Integrated Azure Synapse with Data Lake Storage (ADLS Gen2), Power BI, and Databricks for end-to-end analytics solutions. Developed and automated ETL pipelines using Azure Synapse Pipelines, Dataflows, and Azure Data Factory (ADF) for seamless data ingestion and transformation. Implemented complex business logic using T-SQL stored procedures, functions, views, and advanced query concepts. Designed and built a data warehouse for reporting and analytics, establishing an integration between the operational database and data warehouse. Development level experience in Microsoft Azure providing data movement and scheduling functionality to cloud-based technologies such as Azure Blob Storage and Azure SQL Database Developed cloud-based ETL/ELT processes using Azure Data Factory and Azure Databricks, replacing on-prem ETL/ELT workflows with Azure Cloud solutions. Developed complex ETL workflows using Azure Data Factory, orchestrating data movement and transformations across on-premises and cloud-based data sources Created CI/CD pipelines in Azure DevOps using ARM templates for automated deployment and version control. Developed automated data ingestion processes in Azure Cloud, pulling data from web services and loading it into Azure SQL DB on a daily schedule. Developed parameterized Databricks notebooks to enable reusable ETL templates for different data ingestion scenarios. Designed metadata-driven and event-driven data ingestion into Data Lake, scheduling and orchestrating pipelines using Azure Data Factory. Technologies & Environment: Azure Data Factory, Azure SQL Database, Azure Synapse Analytics (Azure Data Warehouse), Azure Data Lake, Data Catalog, Power BI, Azure DevOps, Azure Databricks, Python, SQL. Client: Insight AI Duration: April 2019 - April 2021 Role: Data Engineer Role and Responsibilities: Supported Analytical Insights Consultancy specialists and Analytics & Modeling specialists by creating efficient data sets to enable customer-driven reporting and insights. Managed the data migration workstream within project budgets, advising Project Managers of potential deviations. Developed proof of concepts (POCs) and custom demos, evaluating Azure products like Azure Data Lake, Azure Databricks, and Azure Kubernetes to determine the best fit for the organization. Designed and developed database change processes for migrating on-premise databases to Azure Cloud. Extracted, transformed, and loaded (ETL) data from source systems into Azure Data Storage services using Azure Data Factory, T-SQL, Spark SQL, and U-SQL in Azure Data Lake Analytics. Ingested data into multiple Azure services (Azure Data Lake, Azure Storage, Azure SQL, Azure Data Warehouse/Synapse Analytics) and processed data using Azure Databricks. Developed and deployed Databricks notebooks using PySpark and SQL, automating data processing workflows. Developed and trained machine learning models using MLflow in Databricks, enabling versioning and tracking of model performance. Designed a serverless data lakehouse solution using Databricks SQL, enabling interactive ad-hoc analysis on structured and semi-structured data Developed Python scripts for file validation in Databricks and automated the validation process using Azure Data Factory (ADF). Built budget forecasting models using actual and planned budgets, providing actionable insights on employee work time and financial planning. Technologies & Environment: Azure Data Factory, Azure SQL Database, Azure Data Warehouse (Synapse Analytics), Azure Data Lake, Data Catalog, Power BI, Azure DevOps, Azure Databricks, Python, SQL, Spark SQL. Client: PriceTech, Wellington Duration: Sep 2016 - April 2019 Role: Data Engineer Role and Responsibilities: Created Power BI and Tableau visualization dashboards for over a dozen online reports, helping clients identify business opportunities and trends. Owned the design, development, and maintenance of ongoing metrics, reports, analyses, dashboards, and BI deliverables to support key business decisions. Gathered functional and non-functional client requirements, optimizing BI reporting through reports, dashboards, alerts, and visualizations. Developed and managed data integration workflows between client databases and MarginFuel s database using SSIS and Azure Data Factory. Designed and built the end-to-end MarginFuel database using Azure SQL Database. Designed and developed ETL workflows using SSIS and ADF, optimizing data integration across systems. Developed DAX-based calculations for complex business metrics, enhancing real-time insights for executive decision-making Created optimized Power BI reports using advanced DAX measures, improving dashboard performance Created and optimized incremental data loads using ETL SSIS Packages for efficient data processing. Developed complex SQL queries, stored procedures, common table expressions (CTEs), and views to support Power BI and SSRS reporting. Implemented custom-calculated measures using DAX in Power BI to satisfy business needs and enhance data analysis capabilities. Technologies & Environment: Power BI, SSIS, SSRS, Azure Data Factory, Azure SQL Server, SQL, Python, DAX, GIT. Client: TCS, India Duration: Aug 2015 - Feb 2016 Role: Database Engineer Role and Responsibilities: Configured and supported SQL Server replication and log shipping in SQL Server 2008 and 2012. Supervised code reviews of scripts and SSIS packages (2008), advising developers on code optimization for improved efficiency. Diagnosed job errors and implemented solutions, collaborating with developers and server administrators to resolve break/fix issues. Approved and executed scripts for Quality Assurance and production releases, ensuring database integrity and compliance. Maintained SQL Server instances, ensuring optimal performance, security patches, and version upgrades. Safeguarded database security by implementing data backup, user access management, and database security protocols. Developed and optimized load procedures using SQL, ensuring database consistency and up-to-date records. Created database objects including tables, stored procedures, views, triggers, user-defined data types, and functions to support business logic and reporting. Presented weekly progress updates to management, addressing data-related challenges and providing solutions. Technologies & Environment: MSSQL, SSIS, SSRS, GIT. Keywords: continuous integration continuous deployment artificial intelligence machine learning business intelligence database active directory information technology procedural language Arizona