| Azure Data Engineer at Minneapolis, Minnesota, USA |
| Email: [email protected] |
| From: Ashok, Bean Infosystems LLC [email protected] Reply to: [email protected] Job Title: Azure Data Engineer Line of Business: Business Banking Location: Minneapolis, MN - Hybrid Duration: 12 months with potential extension or conversion to hire Position Overview We are looking for a Lead Developer skilled in Python, Apache Spark, Azure Synapse, and Azure Data Engineering services to develop and manage ETL pipelines and data processing solutions supporting AI/ML initiatives. The ideal candidate will have strong expertise in Azure Cloud, including SQL, Data Factory, SQL Pools, Spark Pools, and Data Warehousing. Familiarity with CI/CD processes, cross-functional collaboration, and methodologies is essential. Additionally, knowledge of Data Science, Java, Kubernetes, Azure Data Lake Storage (ADLS) Gen2, and API development is required. Key Responsibilities: ETL and Data Pipeline Development Design, develop, and optimize scalable ETL processes using Python, Apache Spark, and Azure Synapse. Build and manage Azure Data Factory pipelines for orchestrating complex data workflows. Use SQL Pools and Spark Pools within Synapse for efficient large dataset processing. Implement Data Warehousing solutions with Azure Synapse Analytics for structured, queryable data. Support real-time and batch AI/ML data requirements. Azure Cloud Development & CI/CD Deployment Build, configure, and manage CI/CD pipelines on Azure DevOps for ETL and data processing. Automate infrastructure provisioning, testing, and deployment with Infrastructure-as-Code tools like ARM templates or Terraform. Optimize ADLS Gen2 for raw and processed data, ensuring access control and security. Cross-Functional Collaboration Work closely with Data Scientists, ML Engineers, and Business Analysts to translate business requirements into data solutions. Collaborate with DevOps and Security teams to ensure secure and smooth application deployment. Lead technical design, development, and implementation of data solutions, mentoring junior team members. Data Engineering and API Development Develop and integrate with APIs for data ingestion and exchange. Build, test, and deploy RESTful APIs for secure data access. Use Kubernetes for containerizing and deploying data applications. Manage data storage and transformation for advanced AI/ML models. Agile Project Management Participate and lead Agile ceremonies such as sprint planning, daily stand-ups, and retrospectives. Collaborate with teams in iterative development to ensure timely, high-quality feature delivery. Adapt to changing priorities in an Agile environment. Required Skills and Qualifications: Technical Skills: Proficient in Python and Apache Spark for large-scale data processing. Extensive experience with Azure Synapse Analytics, SQL Pools, and Spark Pools. Advanced Azure Data Factory expertise for ETL pipeline orchestration and management. Strong Data Warehousing background on Azure. Proficient in SQL, CI/CD tools like Azure DevOps, and managing infrastructure in Azure Cloud. Familiarity with Java for API integration and microservices architecture. Experience with Kubernetes and ADLS Gen2 for data storage and management. Proficient in API-based data integrations (REST, SOAP). Agile and Cross-Functional Skills: Experience working in Agile environments using Scrum or Kanban. Ability to lead, mentor, and coach junior developers. Strong collaboration skills to work with data scientists, analysts, and cross-functional teams. Behavioral Skills: Strong analytical and problem-solving skills. Excellent communication and presentation skills. Ability to work effectively in a dynamic, fast-paced environment. Self-motivated and detail-oriented. Preferred Qualifications: Azure certifications in data engineering or cloud architecture. Experience deploying AI/ML models on cloud platforms. Familiarity with Data Governance and compliance with data privacy regulations. Keywords: continuous integration continuous deployment artificial intelligence machine learning Minnesota Azure Data Engineer [email protected] |
| [email protected] View all |
| Thu Nov 14 00:20:00 UTC 2024 |