Home

Adhi Rondla - Sr Data Engineer

[email protected]

Location: Stafford, Connecticut, USA

Relocation: Yes

Visa: Green Card

Name: Adhi Rondla : [email protected] : (660) 415-0018 Professional Summary: Over 10 years of professional AWS Certified Solution Architect and Certified Data bricks Lake House Fundamentals experience in information technology as Data Engineer with an expert hand in the areas of Database Development, ETL Development, Data modelling, Report Development and Big Data Technologies. Experienced in designing complex system architectures and integrating multiple modules and systems, including Big-Data Hadoop, Azure, and AWS. Proficient in hardware sizing, estimating, benchmarking, and data architecture. Expertise in writing and optimizing SQL queries across various databases, including Oracle 10g/11g/12c, DB2, Netezza, SQL Server 2008/2012/2016, and Teradata 13/14. Conducted data analysis and profiling using complex SQL on diverse source systems, including Oracle and Teradata. Proficiency in Teradata SQL queries, Teradata Indexes, and utilities like Mload, Tpump, Fast Load, and Fast Export. Integrated PySpark with data warehousing solutions like Amazon Redshift or Snowflake to support analytical workload Developed and maintained Delta Lake-based data pipelines for machine learning and artificial intelligence applications Experience with Delta Lake's integrations with other popular data processing frameworks, such as Spark, Hive, and Flink Experienced with Delta Lake's ACID transactions, streaming support, and time travel features Created and managed data catalog entities, including data sets, data sources, and data sources. Experience with Delta Lake's cloud integrations, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage Working experience on ETL implementation using AWS services like Glue, Lambda, EMR, Athena, S3, SNS, Kinesis, Data-Pipelines, Spark, etc. Proficient in designing and developing data models to meet the specific requirements of data engineering projects. Developed and implemented a Delta Lake-based data pipeline for a large e-commerce company, resulting in a 20% reduction in data processing time and a 15% increase in data accuracy. Designed and developed ETL (Extract, Transform, Load) pipelines in PySpark to ingest, cleanse, and transform data from various sources. Documented the Lambda Architecture and provided training to team members. Created and modified access database reports and queries for CDI and data quality. Good working knowledge on Snowflake and Teradata databases. Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle/Snowflake. Excellent Programming skills at a higher level of abstraction using Scala and Python. Analyzed data using Hadoop Ecosystem components like HDFS, Hive, Spark, Elastic Search, Kibana, Kafka, HBase, Zookeeper, PIG, Sqoop, and Flume. Proficient in data security tools and technologies, including SIEM, data forensics, threat intelligence, and security automation platforms. Skilled in Excel Pivot and VBA macros for business scenarios and data transformation using Pig scripts in AWS EMR, AWS RDS, and AWS Glue. Actively participated in Agile development methodologies and scrum processes for project management. Designed and implemented projects using various ETL/BI tools, including Big-Data, Azure, AWS, and Cloud Computing. Proficient in importing data using SQOOP from heterogeneous systems to HDFS and vice versa. Experienced in continuous Performance Tuning, System Optimization, and improvements for BI/OLAP systems and traditional databases. Enhanced query performance by optimizing Snowflake storage and virtual warehouse configurations. Good working knowledge on NoSQL databases such as HBase and Cassandra. Knowledge of job workflow scheduling and monitoring tools like Oozie (hive, pig) and dag (lambada). Delivered zero defect code for three large projects which involved changes to both front end (web services) and back-end (Oracle, snowflake, Teradata). Extensive knowledge of Data Modeling, Data Conversions, Data integration and Data Migration with specialization in Informatica Power Center. Expertise in extraction, transformation and loading data from heterogeneous systems like flat files, excel, Oracle, Teradata, MSSQL Server. Strong written and oral communication skills for giving presentations to non-technical stakeholders. TECHNICAL SKILLS: Databases: Microsoft SQL Server 2016/2014/2012, Teradata 15/14, Oracle 12c/11g/10g, MS Access, Poster SQL, Netezza, DB2, Snowflake, HBase, MongoDB and Cassandra. BI Tools: Business Objects XI, Tableau 9.1, Power BI Cloud computing: Amazon Web Services (AWS), Amazon Redshift, MS Azure, Azure blob storage, Azure Data Factory, Azure Synapse & Google cloud Platform (Big Query, Big Table, Datapost) ETL Tools: SSIS, Informatica Power 9.6/9.5 and SAP Business Objects. Scripting Languages: Unix, Python, Windows PowerShell Operating Systems: Linux, Windows, Ubuntu, Unix Big Data Technologies: Hadoop, HDFS, Hive, MapReduce, Pig, HBase, Sqoop, Flume, Oozie and No SQL Databases SDLC Methodology: Agile, Scrum, Waterfall, UML Spark Ecosystem: MapReduce, Hive/impala, Kafka, Flume, Sqoop, Oozie, Zookeeper, Spark PROFESSIONAL EXPERIENCE: Client: Brown & Brown Insurance, Dallas, TX Mar 2021 Till Date Role: Sr Cloud Data Engineer Responsibilities: Developed and maintained ETL scripts in Python and Perl for web data scraping and loading into a MySQL DB. Enhanced traditional data warehouse based on STAR schema, updated data models, performed Data Analytics and Reporting using Tableau, extracting data from MySQL, Azure, AWS into HDFS via Sqoop, and created Scheduling scripts in Python. Implemented AWS Lambda for data validation, transformation, and loading in DynamoDB tables, optimizing data handling. Developed and deployed a CatLog data model to manage the data for a large-scale cloud-based application. Maintained comprehensive documentation of data models, transformations, and processes in Palantir Foundry for knowledge sharing and auditing. Leveraged Hive Context for advanced querying, preferring HiveQL parser to read data from Hive tables (fact, syndicate). Developed efficient data pipelines for data extraction, transformation, and loading (ETL) into Snowflake, improving data accessibility and accuracy. Managed data solutions on Azure, AWS, creating Shell, Perl, and Python scripts for automation, Pig script control flows, and BI implementation. Created and scheduled DataBricks jobs for automated and scheduled data processing tasks, optimizing workflows and reducing manual intervention. Designed and implemented data governance frameworks for mortgage data, establishing data lineage, access controls, and audit trails for compliance purposes. Designed and developed ETL jobs in AWS Glue, enabling efficient and scalable data transformation workflows. Designed and implemented data pipelines on GCP using services like Cloud Dataflow, Dataflow SQL, or Apache Beam to handle large-scale data processing. Built and managed data lakes and data warehouses on AWS using Glue, optimizing data storage and accessibility. Designed and implemented Azure Gen2 Data Lake storage solutions to support data storage, processing, and analytics for the organization. Deployed, configured, and managed Hortonworks Data Platform (HDP) clusters Validate Databricks by developing python scripts and automated the process using ADF. Analyzed the SQL scripts and designed it by using Spark SQL for faster performance. Used Spark for reading and writing data formats such as JSON, Delta, and Parquet files from different sources. Developed and maintained data pipelines for mortgage loan origination, servicing, and securitization processes, optimizing for scalability and performance. Wrote custom scripts and workflows in Palantir Foundry to address specific data engineering requirements and automation. Implemented lazy loading of modules in Angular applications to improve load times. Developed and maintained Palantir Foundry pipelines to ingest, transform, and analyze large datasets. Developed and maintained DataBricks notebooks for data exploration, analysis, and collaboration among team members, documenting insights and methodologies. Implemented secure data encryption and access controls within AWS Glue to safeguard sensitive information. Conducted performance tuning and optimization of ETL jobs in AWS Glue to enhance efficiency and reduce costs. Integrated various data sources and systems by leveraging GCP tools like Data Fusion and Data Catalog. Experience with Azure Databricks security features such as role-based access control and encryption. Involved in writing T-SQL working on SSIS, SSRS, SSAS, Data Cleansing, Data Scrubbing and Data Migration. Developed MapReduce programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW. Expertise in identifying, assessing, and mitigating data security risks, ensuring the protection of sensitive data from unauthorized access, modification, disclosure, or destruction. Developed and implemented a data modeling standard that has been adopted by other companies in the industry. Implemented data ingestion processes in PySpark to acqdauire data from structured and unstructured sources, including databases, log files, and APIs. Used CatLog to integrate the data warehouse with other cloud-based services, such as AWS Redshift, Amazon S3, and Amazon Athena. Designed, developed, and implemented data models for data warehouses, data lakes, and other data storage and processing systems. Leveraged HDP's distributed computing capabilities to scale data processing workloads Used Azure Databricks to build and deploy scalable and performant data processing solutions. Wrote Sqoop Scripts for importing and exporting data from RDBMS to HDFS. Worked with Apache Spark SQL and data frame functions to perform data transformations and aggregations on complex semi structured data. Proven ability to implement and maintain effective data security controls, policies, and procedures aligned with industry standards and regulatory requirements, such as PCI DSS, HIPAA, and GDPR. Designed and implemented a data model for a data warehouse that stores data from multiple sources, including CRM, ERP, and marketing systems Implemented robust security measures and access controls within Databricks, ensuring data protection and compliance with industry standards. Optimized Java-based data processing pipelines for improved performance, including parallelization and multi-threading. Developed and maintained Palantir Foundry workspaces and dashboards. Developed and deployed AI-based solutions to meet the specific needs of data engineering teams. Used CatLog to create and manage materialized views to improve the performance of data queries. Used AWS EMR to transform and move large amounts of data into and out of other AWS Data stores and databases, such a S3 and Confidential Dynamo DB. Developed and deployed data engineering applications to Google Cloud Run, a serverless compute platform. Constructed dimensional data models and implemented on Teradata, achieving high performance without extensive tuning. Developed and implemented AI algorithms and models to solve complex data engineering problems. Ensured data privacy and compliance with data protection regulations, such as GDPR, when handling sensitive information in Palantir Foundry. Utilized Python, R, or SAS for data analysis and predictive modeling to assess credit risk, forecast loan defaults, and analyze mortgage portfolio performance. Designed Logical and Physical data models with Erwin, implementing Hadoop/Spark custom applications at scale. Built predictive models using Regression and Machine learning with SAS and Python, conducting Data analysis and statistical analysis. Implemented encryption and data security measures to protect sensitive mortgage information, ensuring confidentiality and data privacy compliance. Implemented optimized Databricks clusters configurations to maximize performance and resource utilization for data processing tasks. Established and enforced data governance policies within AWS Glue, ensuring data integrity and security. Collaborated with other data engineers and analysts to build and maintain the Palantir Foundry environment. Developed and deployed machine learning-based solutions to meet the specific needs of data engineering teams Utilized Airflow for creating, debugging, scheduling, and monitoring jobs in an AWS environment. Worked on building end-to-end data pipelines on Azure Data Platforms. Used Erwin to analyse and optimize a slow-performing database, resulting in a 40% improvement in query response times for an investment banking platform. Created and maintained data modeling documentation, including entity diagrams, relationship diagrams, and data dictionaries. Documented data engineering processes, data dictionaries, and best practices for mortgage data management and analysis, facilitating knowledge sharing within the team. Integrated HDP with other data engineering technologies, such as Spark, Hive, and Kafka Troubleshooted and resolved Kubernetes issues in the context of data engineering Developed and implemented machine learning (ML) algorithms to solve complex data engineering problems. Developing Data Extraction, Transformation and Loading jobs from flat files, Oracle, SAP, and Teradata Sources into Teradata using BTEQ, Fast Load, Fast Export, Multiload and stored procedure. Worked on different files like CSV, txt, and a fixed width to load data from various sources to raw tables. Experience and understanding of architecting, designing, and operationalization of data analytics solutions on Snowflake cloud data warehouse. Exporting of data from HDFS using Impala Shell with delimiter options and load into Teradata using BTEQ and TPT utilities Integrated Databricks with data lakes, such as AWS S3 or Azure Data Lake Storage, to store and manage data efficiently. Created a comprehensive Snowflake user guide to ensure the team's consistent usage of Snow SQL scripts and best practices. Developed and managed ETL processes using BigQuery and Dataflow to transform and load data efficiently. Proficient in writing Snow SQL scripts for automation and orchestration of data processes. Automated data pipeline workflows for efficient data processing and integration using tools like Apache Airflow or similar. Integrated Snowflake with tools like Apache Knife or Talend for seamless data movement and transformation. Monitored and maintained the health and performance of data warehouse infrastructure, including backups and disaster recovery plans. Implemented data partitioning and bucketing strategies within the Data Lake to optimize data retrieval and query performance. Integrated Data Lakes with data visualization tools like Tableau or Power BI to enable data-driven insights and reporting. Implemented robust security measures and access controls within DataBricks, ensuring data protection and compliance with company policies and regulations. Implemented data governance and quality initiatives within BigQuery, ensuring consistency and accuracy in analytics. Experience in the Creation of Data Shares, Secured Views, and UDFs (User defined functions). Experienced with AWS AZURE services to smoothly manage application in the cloud and creating or modifying the instances. A highly immersive Data Science program involving Data Manipulation & Visualization, Web Scraping, Machine Learning, Python programming, SQL, GIT, Unix Commands, NoSQL, MongoDB. Created data pipeline for different events of ingestion, aggregation and load consumer response data in AWS S3 bucket into Hive external tables in HDFS location to serve as feed for tableau dashboards. Used EMR (Elastic Map Reducing) to perform bigdata operations in AWS. Worked on Apache spark writing python applications to convert txt, axles files and parse. Developed Python scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop. Experience in CI and CD using Jenkins and Docker. Developed PIG UDFs to provide Pig capabilities for manipulating the data according to Business Requirements and worked on developing custom PIG Loaders and Implemented various requirements using Pig scripts. Client: Capital One, Chicago, IL Sep 2019 Feb 2021 Role: Data Engineer Responsibilities: Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Big Data technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive. Enforced referential integrity in the OLTP data model for consistent relationship between tables and efficient database design. Collaborated with data scientists to implement predictive analytics and machine learning models within BigQuery. Worked with Angular forms to create dynamic and interactive user input interfaces. Monitored and maintained the health and performance of Snowflake infrastructure, including query performance and resource utilization. Developed custom UDFs and transformations within Databricks to meet specific business requirements and optimize data processing. Leveraged Talend Big Data components to integrate with Hadoop, Spark, and other big data technologies for processing and analysis. Managed DataBricks clusters efficiently, optimizing resources and costs while ensuring high availability and reliability for critical data operations. Designed and developed scalable data models within Big Query, optimizing for performance and efficiency. Developed data governance frameworks on GCP, ensuring compliance with regulatory standards and best practices. Used CatLog to create and manage data quality rules for the data warehouse. Worked closely with the ETL Developers in designing and planning the ETL requirements for reporting, as well as with business and IT management in the dissemination of project progress updates, risks, and issues. Implemented Kafka producers create custom partitions, configured brokers and implemented High level consumers to implement data platform. Integrated DataBricks with data lakes or warehouses for efficient data storage, retrieval, and analytics, enabling a unified data ecosystem. Conducted extensive testing and validation of ETL processes in AWS Glue to ensure data accuracy and consistency. Received an award for exceptional performance and innovation in the implementation of data modeling techniques Designed and implemented CatLog data transformations to cleanse, enrich, and prepare data for analysis. Orchestrated ETL processes and data transformations within the Data Lake using tools like Apache Spark, Hadoop, or AWS Glue. Implemented data governance frameworks within Databricks to ensure data integrity, consistency, and compliance with regulations. Maintained NoSQL database to handle unstructured data, clean the data by removing invalidate data, unifying the format and rearranging the structure and load for following steps. Wrote Python scripts to parse XML documents and load the data in database. Worked on Normalization and Denormalization techniques for optimum performance in relational and dimensional databases environments. Collaborated with DevOps teams to design and implement robust CI/CD processes for AWS Glue jobs. Implemented monitoring and logging solutions for AWS Glue jobs to track performance and identify issues. Designed, developed, and tested Extract Transform Load (ETL) applications with different types of sources. Creating files and tuning the SQL queries in Hive Utilizing HUE. Implemented MapReduce jobs in Hive by querying the available data. Maintained documentation of data engineering processes, data dictionaries, and best practices for mortgage data management and analysis. Developed automated workflows and processes using tools like Apache Airflow or similar orchestration tools to streamline mortgage data processing and reporting. Implemented containerization strategies for data engineering workloads, ensuring consistent and reproducible deployments on Cloud Run. Integrated data quality checks and cleansing routines in Talend to identify and address data anomalies, ensuring data reliability and accuracy. Collaborated with business analysts and stakeholders to understand mortgage business requirements and translate them into scalable data engineering solutions. Written DDL and DML statements for creating, altering tables and converting characters into numeric values. Connected Databricks to data warehouses like Amazon Redshift or Snowflake to provide a unified query layer for analytics. Developed data quality checks and validation routines in Java to ensure data accuracy and integrity. Share an instance where you improved data quality through such checks. Facilitated deployment of multi-clustered environment using AWS EC2 and EMR apart from deploying Dockers for cross-functional deployment. Utilized IICS to perform data transformations, including data cleansing, enrichment, and aggregation, ensuring data is suitable for analysis. Conducted analysis and performance monitoring using AWS Glue's native capabilities to drive continuous improvements. Involved in converting Hive/SQL queries into Spark transformations using Spark data frames, Scala and Python. Create data ingestion modules using AWS Glue for loading data in various layers in S3 and reporting using Athena and Quick sight. Contributed to the development of reusable Cloud Run components and patterns for data engineering tasks, promoting code sharing and standardization. Proficient in using Azure Databricks for big data processing and analytics. Designed and implemented data loading processes in Talend to populate data warehouses, data marts, and other analytical repositories. Developed various Python scripts to find vulnerabilities with SQL Queries by doing SQL injection, permission checks and analysis. Developed and implemented Kubernetes-based continuous integration and continuous delivery (CI/CD) pipelines Developed and implemented Delta Lake pipelines for data ingestion, processing, and warehousing Designed and implemented data warehouse architectures to support business intelligence and reporting needs. Implemented logical and physical relational database and maintained Database Objects in the data model using ER Studio and used Star schema and Snowflake Schema methodologies in building and designing the Logical Data Model into Dimensional Models. Leveraged Kubernetes to orchestrate distributed machine learning and artificial intelligence (AI) workload Automated Kubernetes deployments and management for data engineering workloads using tools such as Ansible and Terraform Built and maintained a Spring Boot data governance and compliance system to ensure that data is managed and used in accordance with regulations. Migrate data into RV Data Pipeline using Data Bricks, Spark SQL and Scala and migrate Confidential Call center Data into RV data pipeline from Oracle into HDFS using Hive and Sqoop Developed several behavioural reports and data points creating complex SQL queries and stored procedures using SSRS and Excel. Spun up HDInsight clusters and used Hadoop ecosystem tools like Kafka, Spark and data bricks for real-time analytics streaming, Sqoop, pig, hive and Cosmos DB for batch jobs. Generated periodic reports based on the statistical analysis of the data using SQL Server Reporting Services (SSRS) and generated reports using Global Variables, Expressions and Functions using SSRS. Load terabytes of different level raw data into Spark RDD for data Computation to generate the Output response. Utilized SQL and data warehousing technologies such as Amazon Redshift, Snowflake, or Microsoft Azure SQL Data Warehouse to manage and query large datasets. Involved in Analysis, Design and Implementation/translation of Business User requirements. Performed Market Basket Analysis to identify the groups of assets moving together and recommended the client their risks Developed ETL (Extraction, Transformation and Loading) procedures and Data Conversion Scripts using Pre-Stage, Stage, Pre-Target and Target tables. Leadership of a major new initiative focused on Media Analytics and Forecasting will have the ability to deliver the sales lift associated the customer marketing campaign initiatives. Client: AT&T, Virginia, MN Jul 2017 Aug 2019 Role: Data Engineer Responsibilities: Worked on multiple Data Marts in Enterprise Data Warehouse Project (EDW) and involved in designing OLAP data models extensively used slowly changing dimensions (SCD). Designed 3rd normal form target data model and mapped to logical model and involved in extensive DATA validation using ANSI SQL queries and back-end testing Designed and maintained user interfaces with JSP to ensure a responsive and interactive user experience. Used Spring Boot to build and deploy a data quality and governance system to ensure the accuracy and integrity of data. Developed and deployed a Spring Boot data visualization application to provide insights into business data. Involve in preparation, distribution and collaboration of client specific quality documentation on developments for Big Data and Spark along with regular monitoring on reflecting the modifications or enhancements done in Confidential Schedulers. Migrate the Data from Teradata to Hadoop and data preparation using HIVE Tables. Developed custom scripts and components in SSIS to extend functionality and meet specific project requirements. Analyzed large amounts of data sets to determine optimal way to aggregate and report on it. Accessing the hive tables using Spark Hive context (Spark sql) and used Scala for interactive operations. Develop the Spark Sql logics which mimics the Teradata ETL logics and point the output Delta back to Newly Created Hive Tables and as well the existing TERADATA Dimensions, Facts, and Aggregated Tables. Developed PL/SQL procedures and used them in Stored Procedure Transformations. Wrote and executed unit, system, integration and UAT scripts in a Data Warehouse project. Deployed SSIS packages to SQL Server or other execution environments and scheduled them for automated execution. Developed RESTful web services and integrated them with JSP for data retrieval and presentation. Developed and deployed a Spring Boot-based data science application to predict customer churn. Implemented backup and disaster recovery strategies to ensure data integrity and availability within the Data Lake. Applied various data mining techniques: Linear Regression & Logistic Regression, classification, clustering. Devised PL/SQL Stored Procedures, Functions, Triggers, Views and packages. Made use of Indexing, Aggregation and Materialized views to optimize query performance. Designing Star schema and Snow Flake Schema on Dimensions and Fact Tables and worked with Data Vault Methodology Developed Normalized Logical and Physical database models. Transformed Logical Data Model to Physical Data Model ensuring the Primary Key and Foreign key relationships in PDM, Consistency of definitions of Data Attributes and Primary Index considerations. Client: iHeartMedia, San Antonio, TX Aug 2013 Jun 2017 Role: Data Analyst Responsibilities: Performed extensive Data Analysis and Data Validation on Teradata and designed Star and Snowflake Data Models for Enterprise Data Warehouse using ERWIN. Created and maintained Logical Data Model (LDM) for the project includes documentation of all entities, attributes, data relationships, primary and foreign key structures, allowed values, codes, business rules, glossary terms, etc. Integrated data from various Data sources like MS SQL Server, DB2, Oracle, Netezza and Teradata using Informatica to perform Extraction, Transformation, loading (ETL processes) Worked on ETL development and Data Migration using SSIS and (SQL Loader, PL/SQL). Created Entity/Relationship Diagrams, grouped and created the tables, validated the data, identified PKs for lookup tables. Designed data models and schema structures to leverage Snowflake's cluster key feature effectively. Created iterative macro in Alteryx to send Json request and download Json response from webservice and analyse the response data. Created and maintained data pipelines in Sisense, automating data updates and ensuring real-time access to critical information. Conducted performance optimization and troubleshooting of Sisense dashboards to ensure responsiveness and usability. Proficient in Sisense analytics platform, with hands-on experience in data modelling, dashboard creation, and data visualization. Enhance smooth transition from legacy to newer system, through change management process. Planned project activities for the team based on project timelines using Work Breakdown Structure. Built bi-directional ingestion pipelines in Hadoop and AWS S3 storage. Experience in building ingestion pipelines into AWS Redshift and DynamoDB. Compare data with original source documents and validate Data accuracy. Create and modify access database reports and queries for CDI and data quality. Create Business Objects Crystal reports for coding, ROI and Research Units using Epic Clarity Databases. Worked with BI teams in generating the reports and designing ETL workflows on Tableau Involved in loading data from UNIX file system and FTP to HDFS Used HIVE to do transformations, event joins and some pre-aggregations before storing the data onto HDFS. Developed UDF's in java for enhancing functionalities of Pig and Hive scripts. Exported data into excel for business meetings which made the discussions easier while looking at the data. Involved in the creation, maintenance of Data Warehouse and repositories containing Metadata. Wrote and executed SQL queries to verify that data has been moved from transactional system to DSS, Data Warehouse, and data mart reporting system in accordance with requirements. Education: Bachelors in Computer Science from Osmania University 2012 Keywords: continuous integration continuous deployment artificial intelligence machine learning business intelligence sthree database rlang information technology microsoft procedural language Illinois Minnesota Texas