| senior data engineer at Boston, MA for Hybrid at Quincy, Massachusetts, USA |
| Email: [email protected] |
| From: alka, Allied Business Consulting [email protected] Reply to: [email protected] Sr Data Engineer Location: Quincy, MA - hybrid Open to any visa Key Responsibilities: Data Integration & API Development: Integrate cybersecurity data sources and build data APIs for real-time insights and streamlined access. Pipeline Engineering: Design and optimize large-scale ETL/ELT pipelines on Databricks using Python and PySpark. Data Quality & Governance: Implement automated data checks, manage data lineage, and ensure compliance with cybersecurity standards. Analytics & Visualization: Create data models, visualizations, and dashboards for threat and vulnerability insights using Databricks and React.js. Data Architecture: Develop secure, scalable environments on AWS (S3, ELB, Lambda) for data storage and processing. CI/CD for Data & Security: Use CI/CD pipelines to automate testing and deployment in alignment with Agile practices. ML Integration: Deploy ML models for threat detection, anomaly detection, and risk scoring. Mentorship: Guide junior engineers, establish best practices, and foster a high-performance, innovative culture. Qualifications: The CASM Platform Product & Engineering team is looking for a Sr. Data Engineer, Cyber AI App & Attack Surface Management. CASM Platform team is focused on building enterprise-level robust and scalable Cyber AI applications and Platform to manage vulnerability, risk, security configurations, and attack surfaces oversight across the bank. We are seeking an experienced, hands-on Senior Data Engineer to join our CASM Platform team. Ideal candidates bring a deep understanding of data engineering within a cybersecurity context, extensive experience with cloud-native tools (especially AWS), and a passion for automation, security, intelligent orchestration, and resilience. Youll work in high-paced Agile Product Engineering cross-functional teams, including Product, AI application, DevOps, and Data Quality Management (DQM), to transform the CASM platform into a single pane of glass for secure and insightful network asset and risk visibility. We have multiple openings for this role, and it is open to candidates with varying levels of experience. Role Responsibilities As a Senior Data Engineer, CASM Platform, you will: Data Integration, API Development: Integrate diverse cybersecurity data sources using varietyof API mechanisms and to standardize and streamline data across the data and user planes. Build and maintain data APIs for seamless access to data pipelines, enabling real-time insights for applications, machine learning models, and analytical layers. Data Pipeline Engineering Optimization: Design, develop, and optimize large-scale ETL/ELTpipelines on Databricks to efficiently process and transform cybersecurity data. Utilize Python,PySpark, and Databricks to automate and standardize data workflows across stages (raw,cleaned, curated), ensuring scalability and high performance. Data Quality & Governance: Implement automated data quality checks, leveraging DatabricksDQM tools and CI/CD pipelines to uphold data integrity and governance standards. Ensure datlineage, metadata management, and compliance with cybersecurity and privacy regulations,applying rigorous quality standards across data ingestion and processing workflows. Data Analytics & Visualization: Design centralized data models and perform in-depth data analysis to support cybersecurity and risk management objectives. Develop visualizations anddashboards using tools like Databricks, encapsulate data to spin up to React.js application layerto provide stakeholders with actionable insights into threat landscapes, vulnerability trends, andperformance metrics across the platform. Scalable & Secure Data Architecture: Architect and manage secure, high-performance dataenvironments on Databricks, utilizing AWS services such as S3, ELB, and Lambda. Ensure dataavailability, consistency, and security, aligning with AWS best practices and data encryptionstandards to safeguard sensitive cybersecurity data. Agile Product; Engineering Continuous Delivery: Collaborate with advanced Agile Product; Engineering cross-functional teams to deliver data-driven insights through analytics tools andcustom visualizations that inform strategy and decision-making. Empower stakeholders withtimely, actionable intelligence from complex data analyses, enhancing their ability to respond toevolving cybersecurity risks. Data Science & ML Integration: deploy machine learning models, including predictive analytics,anomaly detection, and risk scoring algorithms, into the CASM platform. Leverage Python andPySpark to enable real-time and batch processing of model outputs, enhancing CASM Platforms proactive threat detection and response capabilities. Mentorship; Best Practices Promotion: Mentor junior engineers, establishing best practices in data engineering, DevOps, data science, and analytics. Encourage high standards in modeldeployment, data security, performance optimization, and visualization practices, fostering aculture of innovation and excellence. Education Qualifications: Minimum Qualifications Education: B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience. Technical Expertise: 8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with Databricks and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context. Cloud Proficiency: Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance. Security Integration: Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high- stakes cybersecurity environment. Data Architecture: Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics. Data Quality ; Governance: Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines. Data Analytics; Visualization: Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams. CI/CD and Automation: Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery. Agile Experience: Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration. Preferred Experience: Advanced Data Modeling; Governance: Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures. Machine Learning; Predictive Analytics: Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platforms cybersecurity capabilities. High-Performance Engineering Culture: Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement. Keywords: continuous integration continuous deployment artificial intelligence machine learning javascript business intelligence sthree information technology Massachusetts senior data engineer at Boston, MA for Hybrid [email protected] |
| [email protected] View all |
| Thu Nov 07 19:16:00 UTC 2024 |