| : Sr Data Engineer at Miramar, Florida, USA |
| Email: [email protected] |
| From: Aniket Dhakar, Technocraft Solutions [email protected] Reply to: [email protected] Role: Sr Data Engineer Location - Location: Miramar, FL or Dallas, TX (Hybrid). Need local Duration- 6+ months contract Position Description We're a hard-working team of tech enthusiasts and professionals on a mission to reshape the future. We've got everything from pioneering projects that challenge your skills to an encouraging and inclusive work environment that fuels your growth. Plus, we offer unparalleled opportunities for professional development, empowering you to stay at the forefront of the ever-evolving tech landscape. We are Seeking a Sr. Data engineer who has designed and built framework-based API ingestion architecture for data extract and exchange and can successfully manage and enhance the data framework, ensuring secure and efficient data handling across various platforms and environments. Your future duties and responsibilities Data Integration Platform Design and Development: Design, build and implement a framework-based API data ingestion architecture with a low-code/no-code interface for data ingestion and processing using reusable plug-and-play components. Develop frameworks to consume API components such as JSON payloads, CSVs, etc. Abstract data to build canonical models and data pipelines to maintain them. Integrate and process private data securely across multiple AWS accounts with compliance to all security protocols, including PgP encryption and encryption at rest. Implement data transformations and curate ingested data to ensure high-quality and usable datasets. Integration and Ingestion: Utilize Databricks and Informatica as unified data integration platforms. Leverage Kafka (or similar messaging queue) as an enterprise integration tool to ingest and integrate near real time and large volumes of diverse data from multiple sources. Build frameworks around APIs to bring in data from various sources efficiently. Security and Compliance: Ensure secure handling of Personally Identifiable Information (PII) data in AWS cloud environments. Understand and implement secure zones and mechanisms for PII data handling. Manage data securely across different storage solutions like RedShift and other data warehouses. Multi-Cloud Expertise: Demonstrate an understanding of different cloud service models - PaaS, SaaS, IaaS, etc. across AWS, Azure, and GCP. Develop solutions that are compatible with multi-cloud environments, ensuring seamless integration and operation. Continuous Development and Deployment: Implement continuous development, deployment, testing, integration, and monitoring practices. Use tools like Airflow for workflow automation and scheduling. Required qualifications to be successful in this role Technical Skills: Proficiency in Python and Spark for data processing and manipulation. Hands-on experience with Databricks for data analytics and pipeline creation. Experience with Kafka or similar messaging queues for data ingestion and integration. Strong SQL skills for database querying and manipulation. Familiarity with Airflow for workflow management. Cloud Expertise: Advanced level understanding of the different cloud service models - PAAS, SAAS, IAAS etc. across AWS, Azure and GCP In-depth knowledge of AWS, including security, data handling, and storage solutions like RedShift and S3 Understanding of multi-cloud architectures and best practices. Security and Compliance: Experience handling PII data with a strong understanding of encryption standards (PgP, encryption at rest). Knowledge of secure zones and mechanisms for data protection in cloud environments. Analytical and Problem-Solving Skills: Ability to abstract and transform data to meet business requirements. Strong problem-solving skills to troubleshoot and optimize data pipelines. Communication and Collaboration: Excellent communication (verbal and written) skills to work effectively with cross-functional teams. Ability to document processes and frameworks clearly for future reference. Timeliness, accuracy, and professionalism. Minimum Education Required: A degree in Computer Science, Information Technology, or a related field. Keywords: sthree Florida Texas : Sr Data Engineer [email protected] |
| [email protected] View all |
| Sat Jun 15 00:15:00 UTC 2024 |