| Immediate Hire - ETL Developer - Remote at Remote, Remote, USA |
| Email: [email protected] |
| From: Gangadhar, Intellectt [email protected] Reply to: [email protected] Hello, Hope you are doing great. This is Gangadhar from Intellectt INC ; we have an immediate opportunity with one of our clients. Please find the below job description and if you are interested, please forward your resume to [email protected] Role: ETL Developer / Data Scientist / Data Engineer Location: Remote Duration: Long Term Description: Code Modernization and Refactoring: Analyze existing ETL processes and identify components that require modernization to be cloud native. Refactor legacy ETL jobs to utilize cloud-native services (e.g., AWS Glue). Ensure that all ETL jobs are containerized, if applicable, for better portability and scalability (e.g., using Docker, Kubernetes). Adopt server less technologies where appropriate to manage ETL jobs without the need to provision servers. Implement version control for all ETL scripts and configurations using tools like Bitbucket. Performance Optimization: Profile current ETL jobs to establish benchmarks for performance. Optimize data processing by implementing parallel processing and partitioning strategies. Use cloud-native auto-scaling features to handle variable workloads efficiently. Implement caching mechanisms where necessary to improve data retrieval times. Design and apply indexing strategies to databases to expedite query execution. Cost Optimization: Analyze current resource utilization and identify cost-saving opportunities through rightsizing and scheduling. Utilize cloud provider pricing models (e.g., spot instances, reserved instances) to reduce costs. Implement cost allocation tags to track ETL job costs for better budget management. Optimize data storage by archiving old data and transitioning to cost-effective storage solutions. Data Quality Assurance: Establish data validation frameworks to ensure accuracy and consistency of data through the ETL process. Design and implement data quality checks within the ETL pipeline. Automate error logging and notification mechanisms to quickly address any data quality issues. Document data lineage and transformations to maintain data integrity and auditability. Collaboration and Support: Work closely with data engineers, database administrators, and cloud architects for seamless integration of ETL processes into the cloud environment. Develop and maintain clear documentation for ETL processes and data models to support other developers and stakeholders. Provide training and support to other team members transitioning to cloud native ETL solutions. Security and Compliance: Implement security best practices, including encryption in transit and at rest, identity and access management (IAM) policies. Ensure that all ETL workflows comply with relevant data protection rules (e.g., GxP). Monitoring and Maintenance: Set up monitoring and logging for ETL jobs using cloud-native tools to ensure high availability and reliability. Create alerts for system anomalies, performance degradation, and cost overruns. Schedule regular reviews and updates of ETL processes to adapt to evolving data sources and business requirements. Automation and CI/CD: Automate ETL pipelines using continuous integration and continuous deployment (CI/CD) practices for smoother transitions and updates. Implement automated testing frameworks for ETL code to facilitate rapid iteration and deployment. Non-tech requirements: Proactive in communications. Have some business analyst mindset: Understanding requirements and identifying gaps. HAVE to be excellent at customer management. HAVE to play nice with others and leave their ego OUT of this job. Must have a consultative nature to the Keywords: continuous integration continuous deployment |
| [email protected] View all |
| Wed Jan 03 01:32:00 UTC 2024 |