Job Description
A fortune 100 client is seeking a highly skilled and experienced Senior Data Engineer to support an enterprise-level migration of the metadata layer from Presto to Unity Catalog within the Databricks Lakehouse Platform. This role requires deep expertise in Databricks, Spark, and modern data architecture, with a strong focus on data governance, performance optimization, and scalable pipeline development. The two additional data engineers will join the client's 2 current team members and will work on an individual basis to complete the migration effort.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Required Skills & Experience
• 8+ years experience as a Data Engineer with a minimum 5 years of experiences in Azure tool stack (Databricks, ADF, Synapse Analytics)
• Strong expertise in writing complex SQL and transformations
• Ability to write, review and examine code written in Python and PySpark
• Proficient in Devops CICD
• Expertise in RDBMS
• Fundamental knowledge of Data Warehousing concepts
• SQL and SQL tuning within Data Lake (Delta Format)
Nice to Have Skills & Experience
• Azure services: azure functions, event grid
• Tableau & PowerBI
• Prior PepsiCo experience
• Databricks certification
• Local to HYD
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.