| Job Description: Big Data Engineer at New York City, New York, USA |
| Email: [email protected] |
| Hello partners, Job Description: Big Data Engineer - Data Warehousing and GCP Location New York City, NY(day1 onsite ) Passport number is mandatory Position Overview: We are seeking an experienced and skilled Big Data Engineer with a strong background in data warehousing and expertise in Google Cloud Platform (GCP) technologies. The ideal candidate will possess extensive hands-on experience with SQL, Hive, PySpark, and Data Frames, coupled with a proven track record of delivering complex data engineering solutions. As a Big Data Engineer, you will play a crucial role in designing, developing, and optimizing our data infrastructure to support our data warehousing initiatives and enhance data processing capabilities on GCP. Responsibilities: Collaborate with cross-functional teams including data scientists, data analysts, and software engineers to understand data requirements and design efficient data solutions. Develop and maintain scalable and high-performance data pipelines for data extraction, transformation, and loading (ETL) processes. Implement best practices for data modeling, storage, and partitioning in a data warehouse environment. Utilize GCP services, tools, and technologies such as BigQuery, Dataflow, Dataproc, and Storage to design and optimize data processing workflows. Build and maintain robust data architectures that ensure data quality, consistency, and accuracy throughout the data lifecycle. Perform data profiling, validation, and quality checks to identify and address anomalies or issues in the data. Optimize and tune ETL processes and data pipelines to achieve optimal performance and resource utilization. Troubleshoot and resolve data-related issues, ensuring smooth and uninterrupted data flow. Stay current with industry trends and emerging technologies, making recommendations for technology adoption and process improvements. Mentor and guide junior team members, sharing best practices and fostering a culture of continuous learning. Qualifications: Bachelor's or Master's degree in Computer Science, Information Technology, or a related field. 7+ years of professional experience in data engineering, with a strong focus on data warehousing and GCP. Proven expertise in designing, developing, and maintaining data pipelines using SQL, Hive, PySpark, and Data Frames. Extensive experience with Google Cloud Platform (GCP) services such as BigQuery, Dataflow, Dataproc, and Storage. Strong understanding of data modelling, schema design, and data partitioning strategies in a data warehouse environment. Proficiency in optimizing and tuning ETL processes for performance and scalability. Familiarity with data governance, security, and compliance best practices. Excellent problem-solving skills and the ability to troubleshoot complex data issues. Strong communication skills with the ability to collaborate effectively with technical and non-technical stakeholders. Experience with version control systems (e.g., Git) and Agile development methodologies. Certifications in GCP and relevant technologies are a plus. Thanks & Regards, Ganesh Talluri. Email ID: [email protected] Linked in : https://www.linkedin.com/in/talluri-ganesh-2a5847232/ Website: www.adeptconsultants.com -- Keywords: information technology New York |
| [email protected] View all |
| Tue Aug 15 00:31:00 UTC 2023 |