| Urgent, Senior Data Engineer, Onsite, Miami, FL at Miami, Florida, USA |
| Email: [email protected] |
| From: Gagan, TEK Inspirations LLC [email protected] Reply to: [email protected] Hello, Hope you are doing well! Please have a look on the given JD below. And let me know if you have any candidate for this role. Job Title: Senior Data Engineer Location: Onsite/Miami, FL(5 days) Duration: 12+ Months MOI: Skype Must: Need Local only LinkedIn profile must have Need 2 Manager professional reference with official email ID, contact number PV will verify, email shouldnt bounce back Need 4-5 page resume Job Description - Data Engineering & ETL: Utilize 10+ years of experience in ETL tools, with at least 5 years dedicated to Azure Data Factory (ADF), to design, code, implement, and manage multiple parallel data pipelines. Additional knowledge about Fabric Pipelines and Data Flows Gen 2 usage is desirable. Data Manipulation: Develop and optimize data fetching, data wrangling, manipulation, and transformation processes, ensuring high-quality data delivery. SQL Proficiency: Leverage a strong SQL background, particularly in T-SQL, to work with Azure SQL Databases, performing complex queries, data transformations and able to perform basic database administration and access control configurations. Python & PySpark: Use Python, especially PySpark notebooks to convert and manage data. Understanding of Spark Streaming and Spark configuration is highly desirable. Hand on usage of Panda Libraries, Literal strings constants, user defined functions using Python. Experience in developing Python-based solutions is essential. Data Warehousing & Modeling: Apply a deep understanding of data warehousing concepts, including data modeling techniques like star and snowflake schemas, SCD Type 2, Change Data Feeds, Change Data Capture. Also demonstrates hands-on experience with Data Lake Gen 2, Delta Lake, Delta Parquet files, JSON files, big data storage layers, optimize and maintain big data storage using Partitioning, V-Order, Optimize, Vacuum and other techniques. Reporting Knowledge: Basic understanding of reporting tools, Semantic Layers, Tabular Models, Direct Lake Mode vs Incremental refreshes with Power BI experience considered a plus. API Integration & Event-Driven Architecture: Develop and manage REST API calls, webhooks, and event-driven architectures using Python. Experience with Azure Service Bus and Azure Event Hubs/ Event Grids/ Data Explorer is desirable. Schema Drift & ETL Architecture: Expertise in managing schema drift within ETL processes, ensuring robust and adaptable data integration solutions. Qualifications: 10+ years of experience in ETL tools, with a minimum of 5 years in Azure Data Factory (ADF),Dataflows Gen 2. 8 year USA background is must Strong SQL skills with hand on experience with complex window functions, dynamic Sql, partitions, CDC,CDF particularly in T-SQL, and experience with Azure SQL Database, database administration and setup. Proficiency in Python, specifically with PySpark notebooks, for data processing tasks, configuration of Spark Pools, understanding of spark compute and consumption. Solid understanding of data warehousing concepts, data modeling (star and snowflake schemas), and SCD Type 2, CDC, CDF. Also implementation experience with Delta Lakes and Delta Parquet, JSON, data storage, optimization techniques for Big Data. Familiarity with REST API, webhooks, and event-driven architectures, Data Explorer, Functions Apps, Service Bus . Knowledge of reporting tools, query language, semantic models with a preference for Power BI experience. Ability to manage schema drift in semi structured files and adapt ETL architectures accordingly. Understanding of constructs like Lakehouse, EventHouse, KustoDB, KQL, Data Activator, Reflex is plus Must-Have: ADF, ETL, SSIS, Python& PySpark, SQL, Power Bi, Data Warehousing & Modeling, Data modeling (star and snowflake schemas), API Schema Drift & ETL Architecture:Data Lake Gen 2, Delta Lake, Delta Parquet Thanks & Regards, Gagan Shrivastava Technical Recruiter TEK Inspirations LLC | 13573 Tabasco Cat Trail, Frisco, TX 75035 Ph: +1 (469) 498-0305 Linkedin: linkedin.com/in/gagan-shrivastava-a7b15b22a Keywords: business intelligence Florida Idaho Texas Urgent, Senior Data Engineer, Onsite, Miami, FL [email protected] |
| [email protected] View all |
| Wed Oct 30 02:23:00 UTC 2024 |