| Data Architect || Hybrid at CA at Remote, Remote, USA |
| Email: [email protected] |
| Role : Data Architect Total years of experience : 12 +yrs Role : Hybrid (Palo Alto , CA) Relocation is fine The Data Architect will be responsible for designing, implementing, and managing data workflows and orchestration solutions that integrate with various data storage and processing technologies. This role requires expertise in data orchestration tools, task queues, and modern data storage solutions to ensure the efficient flow and management of data within our organization. Key Responsibilities: Data Orchestration: Design, implement, and manage data workflows using Airflow to automate and orchestrate data processing tasks. Optimize Airflow DAGs (Directed Acyclic Graphs) for performance and scalability. Task Management: Develop and maintain distributed task processing using Celery and ensure robust task queue management with Redis or RabbitMQ. Database Management: Design and manage databases using Cosmos DB, MongoDB, and PostgreSQL. Develop and maintain efficient data models and ensure data consistency and integrity. API and Webhooks: oImplement and manage FastAPI webhooks to handle data ingestion and integration tasks. oDevelop and maintain Azure Functions to support webhook operations and integrate with cloud services. Streaming Data: oImplement and manage Kafka Streams to handle real-time data processing and streaming requirements. Data Lake Management: oWork with Iceberg to manage and optimize large-scale data lake storage and querying. Collaboration and Communication: oCollaborate with data scientists, engineers, and business analysts to understand data requirements and provide technical solutions. oDocument processes, architectures, and configurations to ensure knowledge sharing and compliance with best practices. Required Skills and Qualifications: Experience and Knowledge: oProven experience with Airflow for data orchestration and workflow management. oHands-on experience with Celery for task management and Redis or RabbitMQ for messaging. oProficiency with Cosmos DB, MongoDB, and PostgreSQL for data storage and management. oExperience developing and managing webhooks using FastAPI and integrating with Azure Functions. oKnowledge of Kafka Streams for real-time data processing. oFamiliarity with Iceberg for data lake management and optimization. Technical Skills: oStrong understanding of data pipelines, ETL processes, and data integration. oProficient in Python, with experience in building and maintaining data-oriented applications. oAbility to work with large datasets and optimize performance across distributed systems. Soft Skills: oExcellent problem-solving and analytical skills. oStrong communication and collaboration skills. oAbility to work independently and manage multiple priorities in a fast-paced environment. Healthcare domain experience is good to have . -- Samuel McCoy Account Manager [email protected] linkedin: https://www.linkedin.com/in/sai-krishna-putta-845518232/ Email is the Best way To reach me . -- Keywords: database information technology California Data Architect || Hybrid at CA [email protected] |
| [email protected] View all |
| Thu Aug 22 19:23:00 UTC 2024 |