Sr Data Engineer

Insight Global is seeking a Senior Data Engineer to join a large retail client based in Vancouver, BC. The Data & Analytics team builds and maintains the data infrastructure that powers analytics, reporting, and data-driven decision making across the company's digital platforms, retail operations, and business units. This Senior Data Engineer will support the Guest Data Lake within the Platform Analytics space. The platform is responsible for enabling trusted, scalable access to customer and guest data that supports digital experiences, personalization, analytics, reporting, and downstream business initiatives.
The team is currently focused on the evolution of the Guest Data Lake, including building new data products for the Digital organization, developing integrations between the Guest Data Lake and other core enterprise systems, and delivering new data features that enhance customer insights and business capabilities. The ideal candidate will bring strong experience building and maintaining cloud-native data platforms, supporting customer and guest data domains, and designing scalable data solutions using Azure, Databricks, and PySpark. This individual will work closely with data engineers, analytics teams, product stakeholders, and business partners to enhance data quality, optimize platform performance, and ensure secure, governed access to critical guest data assets.

Day-to-Day Activities:
• Design and implement scalable data architectures (batch, streaming, and lakehouse) using Azure, Databricks, and PySpark
• Support and enhance the Guest Data Lake platform to ensure reliable, high-quality customer data is available across the organization
• Build data products, integrations, and new data features that support digital experiences, personalization, analytics, and reporting initiatives
• Develop and optimize ETL/ELT pipelines for structured and semi-structured data sources
• Work extensively within Databricks, leveraging advanced platform capabilities to build scalable, high-performance data solutions
• Collaborate closely with Product Delivery Managers (PDMs) and cross-functional stakeholders to refine requirements, groom and prioritize backlog tickets, and translate business needs into technical solutions
• Balance hands-on technical delivery (approximately 80%) with technical leadership, stakeholder collaboration, and team support (approximately 20%)
• Work with guest/customer data sets to support analytics, personalization, digital experiences, and reporting initiatives
• Lead migrations from legacy data solutions to cloud-native platforms while maintaining performance and reliability
• Establish and improve data quality frameworks, observability, monitoring, and governance controls
• Partner with Data Science and Analytics teams to provide ML-ready and analytics-ready datasets
• Define and improve CI/CD processes, Infrastructure-as-Code standards, and software development best practices
• Participate in architecture reviews, backlog prioritization, technical design discussions, and platform roadmap planning
• Mentor junior engineers through code reviews, knowledge sharing, and technical guidance
• Optimize platform performance, cost management, and security practices across the data ecosystem

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

• 5–8 years of experience in Data Engineering
• Strong Azure experience with Databricks and PySpark and Unity Catalog
• Advanced SQL and Python skills
• Experience with customer/guest data domains and understanding how customer data supports digital platforms, analytics, and business decision-making
• Experience working with structured and semi-structured data sources, including NoSQL databases (Cosmos DB preferred)
• Strong understanding of Data Governance, Data Quality, and observability best practices
• Experience building and supporting large-scale ETL/ELT pipelines and cloud-based data lakes
• Experience with CI/CD, Infrastructure-as-Code (Terraform), and software development lifecycle best practices
• Experience mentoring junior engineers and contributing to technical design and architecture discussions

• Experience with retail, e-commerce, loyalty, or customer data platforms
• Experience with Spark Declarative Pipelines and Metric Views in Databricks
• Experience supporting Machine Learning or AI-enabled data products
• Experience with Data Mesh architecture and domain-oriented data products
• AWS experience
• Experience implementing real-time streaming and event-driven architectures

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.