| Big Data Engineer with expertise in PySpark(RICHMOND VA OR DALLAS TX) at Richmond, Virginia, USA |
| Email: [email protected] |
| From: Debasish Pattnaik, MRTECHNOSOFT [email protected] Reply to: [email protected] RICHMOND VA OR DALLAS TX Big Data Engineer with expertise in PySpark Job Overview We are seeking a skilled and experienced Big Data Engineer with expertise in PySpark to join our dynamic data engineering team. The successful candidate will be responsible for developing, optimizing, and maintaining our big data infrastructure and pipelines. This role involves working closely with data scientists, analysts, and other stakeholders to ensure data availability and quality for various analytical and operational purposes.Key Responsibilities Design and Development: Design, develop, and optimize scalable big data processing pipelines using PySpark and other relevant big data technologies. Data Management: Manage and maintain large datasets, ensuring data integrity, quality, and security. ETL Processes: Develop and maintain ETL (Extract, Transform, Load) processes to facilitate data extraction from various sources, data transformation, and loading into target systems. Performance Optimization: Identify and resolve performance bottlenecks in large-scale data processing. Collaboration: Work closely with data scientists, analysts, and other engineering teams to understand data requirements and provide appropriate solutions. Documentation: Create and maintain comprehensive documentation for data pipelines, processes, and system architecture. Monitoring and Troubleshooting: Monitor data pipeline performance and troubleshoot any issues to ensure continuous data flow and availability.Required Qualifications Education: Bachelor's degree in Computer Science, Information Technology, or a related field. A masters degree is a plus. Experience: 3+ years of experience in big data engineering, with a strong focus on PySpark. Technical Skills: Proficiency in PySpark and other Apache Spark components. Strong programming skills in Python. Experience with big data technologies such as Hadoop, Hive, HBase, and Kafka. Proficiency in SQL and experience with relational and NoSQL databases. Familiarity with cloud platforms like AWS, Azure, or Google Cloud Platform. Tools and Platforms: Experience with tools like Databricks, Airflow, and other data orchestration and processing tools. Data Management: Strong understanding of data modeling, data warehousing, and data governance principles. Problem-Solving: Excellent problem-solving and analytical skills. Communication: Strong verbal and written communication skills, with the ability to collaborate effectively with cross-functional teams.Preferred Qualifications Experience with machine learning frameworks and libraries. Knowledge of data visualization tools such as Tableau or Power BI. Certification in big data technologies or cloud platforms. Thanks Debasish Pattnaik [email protected] www.mrtechnosoft.com Keywords: business intelligence Texas Virginia Big Data Engineer with expertise in PySpark(RICHMOND VA OR DALLAS TX) [email protected] |
| [email protected] View all |
| Mon Jun 24 14:14:00 UTC 2024 |