| Onsite Title: Data Engineer at Miami, Florida, USA |
| Email: [email protected] |
| Must have Local DL Miami ,FL Client: Lennar Mortgage Title: Data Engineer Location: Miami ,FL onsite 5 days-5505 Blue Lagoon Dr Miami FL -local Only Duration: Contract to Hire ( conversion 140k) Work Authorization: US Citizen or Green Card holder Rate: $60-$62 MOI: Skype Need LinkedIn and Reference-2 candidate Max Data Engineering & ETL: Utilize 10+ years of experience in ETL tools, with at least 5 years dedicated to Azure Data Factory (ADF), to design, code, implement, and manage multiple parallel data pipelines. Additional knowledge about Fabric Pipelines and Data Flows Gen 2 usage is desirable. Data Manipulation: Develop and optimize data fetching, data wrangling, manipulation, and transformation processes, ensuring high-quality data delivery. SQL Proficiency: Leverage a strong SQL background, particularly in T-SQL, to work with Azure SQL Databases, performing complex queries, data transformations and able to perform basic database administration and access control configurations. Python & PySpark: Use Python, especially PySpark notebooks to convert and manage data. Understanding of Spark Streaming and Spark configuration is highly desirable. Hand on usage of Panda Libraries, Literal strings constants, user defined functions using Python. Experience in developing Python-based solutions is essential. Data Warehousing & Modeling: Apply a deep understanding of data warehousing concepts, including data modeling techniques like star and snowflake schemas, SCD Type 2, Change Data Feeds, Change Data Capture. Also demonstrates hands-on experience with Data Lake Gen 2, Delta Lake, Delta Parquet files, JSON files, big data storage layers, optimize and maintain big data storage using Partitioning, V-Order, Optimize, Vacuum and other techniques. Reporting Knowledge: Basic understanding of reporting tools, Semantic Layers, Tabular Models, Direct Lake Mode vs Incremental refreshes with Power BI experience considered a plus. API Integration & Event-Driven Architecture: Develop and manage REST API calls, webhooks, and event-driven architectures using Python. Experience with Azure Service Bus and Azure Event Hubs/ Event Grids/ Data Explorer is desirable. Schema Drift & ETL Architecture: Expertise in managing schema drift within ETL processes, ensuring robust and adaptable data integration solutions. Qualifications: 10+ years of experience in ETL tools, with a minimum of 5 years in Azure Data Factory (ADF),Dataflows Gen 2. Strong SQL skills with hand on experience with complex window functions, dynamic Sql, partitions, CDC,CDF particularly in T-SQL, and experience with Azure SQL Database, database administration and setup. Proficiency in Python, specifically with PySpark notebooks, for data processing tasks, configuration of Spark Pools, understanding of spark compute and consumption. Solid understanding of data warehousing concepts, data modeling (star and snowflake schemas), and SCD Type 2, CDC, CDF. Also implementation experience with Delta Lakes and Delta Parquet, JSON, data storage, optimization techniques for Big Data. Familiarity with REST API, webhooks, and event-driven architectures, Data Explorer, Functions Apps, Service Bus . Knowledge of reporting tools, query language, semantic models with a preference for Power BI experience. Ability to manage schema drift in semi structured files and adapt ETL architectures accordingly. Understanding of constructs like Lakehouse, EventHouse, KustoDB, KQL, Data Activator, Reflex is plus. Must-Have: ADF ETL SSIS Python& PySpark SQL Power Bi Data Warehousing & Modeling Data modeling (star and snowflake schemas), API Schema Drift & ETL Architecture: Data Lake Gen 2, Delta Lake, Delta Parquet -- Keywords: business intelligence information technology Florida Onsite Title: Data Engineer [email protected] |
| [email protected] View all |
| Tue Oct 29 03:29:00 UTC 2024 |