Lead Data Engineer

Insight Global is seeking a Lead Data Engineer to join a large retail client based in Vancouver, BC. The data engineering team builds and maintains the data infrastructure that powers analytics, reporting, and data-driven decision making across the company’s digital platforms, retail operations, and business units. We work with cross-functional agile squads to develop reliable data pipelines, improve data quality, build self-serve analytics tools for our business stakeholders, and ensure timely access to accurate data for our analysts and data scientists. This Lead Engineer will be responsible for supporting several upcoming initiatives, including modernizing the organization’s data platform through the migration of data pipelines from Azure Data Factory to Databricks and the transition of legacy Hive tables to Unity Catalog. The ideal candidate will bring strong Databricks experience, including a solid understanding of Medallion Architecture and best practices for defining and managing Bronze, Silver, and Gold data layers to support scalable, governed data solutions.

Day to Day Activities:
• Architecting and implementing scalable data platforms using Azure, Databricks, and NoSQL databases
• Overseeing Unity Catalog adoption for centralized data governance and metadata management
• Designing integrations with ML infrastructure and collaborating with data science teams on model deployment
• Leading and mentoring a team of data engineers, balancing hands-on coding with strategic oversight
• Establishing coding standards, DevOps/SDLC workflows, and performance optimization practices
• Driving backlog prioritization, technical planning, and cross-functional alignment
• Implementing enterprise-grade data governance frameworks, including security (SecOps) and cost management (FinOps)
• Championing data quality initiatives and advanced monitoring solutions
• Evaluating emerging technologies (e.g., real-time processing, GenAI pipelines) for business impact
• Partnering with business leaders to translate requirements into technical roadmaps
• Leading hiring processes, performance evaluations, and team development
• Exemplifying the company’s leadership code (Lead Self, Lead Others, Lead Business) through innovation, accountability, and mentorship

Here are some of the things you could be working on in the future:
• Leading the technical implementation of data mesh architecture and domain-oriented data products
• Designing and implementing real-time data streaming solutions for critical business processes
• Establishing advanced MLOps practices to operationalize machine learning workflows at scale
• Developing comprehensive data observability solutions across the entire data ecosystem
• Creating specialized data acceleration frameworks to improve platform performance

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

• 6-10 years of experience in data engineering, with 3+ years in overseeing projects
• Strong Azure experience with Databricks, and pipeline orchestration tools (Spark)
• Strong experience with Unity Catalog, DLT and Asset bundles
• Advanced skills in SQL, Python, Pyspark and infrastructure-as-code (Terraform, ARM templates)
• Proven track record in designing large-scale data architectures (batch/streaming, lakehouse, NoSQL)
• Strong leadership skills with experience mentoring teams, supporting Jr Engineers and driving technical excellence
• Can communicate complex technical strategies to executives and stakeholders

• AWS Experience (they don’t use AWS just yet but potentially next year)
• ML Ops experience (Databricks, ML Flow)
• Retail experience in the past

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.