Home

Afreen najam - Data egineer

[email protected]

Location: Kansas City, Kansas, USA

Relocation: no

Visa: H1B

Dear Recruiters, Hope you are doing great. We have the following experienced quality consultants for Sr. Big Data Engineer available for any c2c positions, and please let me know if you have any open C2C positions. Please send us C2C positions at [email protected] or reach me at 8325484963 Our consultant- having genuine experience. Lavanya Data Engineer BACKGROUND SUMMARY: Big Data Engineer/ Data Engineer with over 7+ years of overall experience as software developer in design, development, deploying and large scale supporting large scale distributed systems. In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, MR, Hadoop GEN2 Federation, High Availability and YARN architecture and good understanding of workload management, scalability and distributed platform architectures. Assisted Deployment team in setting up Hadoop cluster and services. Having good knowledge in Benchmarking & Performance Tuning of cluster. Designed and implemented a product search service using Apache Solr. Good experience in Oozie Framework and Automating daily import jobs. Implemented various algorithms for analytics using Cassandra with PySpark and Scala. Extensive implemented ETL design and development using Python, PySpark on Foundry (Palantir) Create data transformations in PySpark and Python code in the Palantir Foundry platform that is implemented into the data pipeline or used to create data analytics reports Have experience in using Contour and Data Preparation in Palantir Foundry platform Experienced of building Data Warehouse in Azure platform using Azure data bricks and data factory. Experienced in managing Hadoop clusters and services using Cloudera Manager. Experience in working with the Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HiveQL queries Expertise in using various Hadoop infrastructures such as Map Reduce, Pig, Hive, Zookeeper, Hbase, Sqoop, Oozie, Flume, Drill and PySpark for data storage and analysis. Experience in developing a data pipeline through Kafka-PySpark API. Proficient in data processing like collecting, aggregating, moving from various sources using Apache Flume and Kafka. Experience in developing custom UDFs for Pig and Hive to incorporate methods and functionality of Python/Java into Pig Latin and HQL (HiveQL) and Used UDFs from Piggybank UDF Repository. Experienced in running query - using Impala and used BI tools to run ad-hoc queries directly on Hadoop. Collected logs data from various sources and integrated in to HDFS using Flume. Experience using Job scheduling tools like Cron, Tivoli and Automic. Experienced in troubleshooting errors in Hbase Shell/API, Pig, Hive and MapReduce. Highly experienced in importing and exporting data between HDFS and Relational Database Management systems using Sqoop. Extensive experience as Hadoop and PySpark engineer and Big Data analyst. Excellent understanding of Hadoop architecture and underlying framework including storage management. Hands-on experience with Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, Cloud Front, CloudWatch, SNS, SES, SQS and other services of the AWS family. Selecting appropriate AWS services to design and deploy an application based on given requirements. Have experience in installing, configuring and administering Hadoop clusters for major Hadoop distributions like CDH4, and CDH5. Good understanding of NoSQL Databases and hands-on work experience in writing applications on No SQL databases like Cassandra and MongoDB. Experienced in Creating Viz boards for data visualization in Plat fora for real - time dashboard on Hadoop. Good knowledge in querying data from Cassandra for searching, grouping and sorting. Good Knowledge in Amazon AWS concepts like EMR and EC2 web services which provides fast and efficient processing of Big Data. Keywords: business intelligence sthree active directory