Senior Data Engineer

This Senior Data Engineer will be responsible for designing, building, and optimizing high-performance data pipelines and platforms within a Databricks and AWS environment. The team is currently in the middle of a major Databricks transformation initiative, focused on modernizing their data architecture while continuing to support and enhance components of an existing AWS-based data stack.

This initiative is expected to run through the end of the year, with multiple key deliverables still in progress. As a result, this role will be heavily involved in both net-new development within Databricks and the ongoing support and evolution of AWS data infrastructure, including pipelines that support enterprise BI and reporting systems.

The Senior Data Engineer will partner closely with Data Analytics leadership, including a recently established Head of Data & Analytics, as well as cross-functional stakeholders across business intelligence, reporting, and machine learning teams. This role will help bridge the gap between data engineering and advanced analytics by enabling reliable, production-ready data pipelines that support ML workflows and data-driven decision-making.

This is a high-impact, senior-level position requiring strong ownership, technical depth, and the ability to operate independently. The ideal candidate will be comfortable stepping into an environment with active initiatives underway and contributing immediately to both delivery and longer-term data platform improvements.

Key Responsibilities
• Design, build, and maintain scalable data pipelines using Python, Spark, and Databricks
• Support and enhance an existing AWS-based data stack, ensuring performance, scalability, and reliability
• Contribute to a large Databricks initiative, helping drive development and implementation efforts
• Partner with Data Analytics and BI teams to support enterprise reporting and analytics needs
• Enable and support machine learning workflows, including data preparation and pipeline integration
• Optimize data processing performance and ensure strong data quality practices
• Collaborate with stakeholders to translate business needs into technical data solutions
• Contribute to ongoing improvements in data architecture, tooling, and engineering best practices

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

• 5+ years of experience in Data Engineering or a related field
• Strong hands-on experience with Python and Apache Spark
• Proven experience working in AWS environments
• Hands-on experience with Databricks and building production-level data pipelines
• Experience supporting BI/reporting platforms and analytics use cases
• Exposure to or experience supporting machine learning workflows or ML pipelines
• Strong understanding of data architecture, ETL/ELT processes, and distributed systems
• Ability to operate independently in a senior-level capacity and drive deliverables

• Experience with AWS services such as Glue, Redshift, Lambda, or Step Functions
• Familiarity with MLOps tools (e.g., SageMaker or similar platforms)
• Experience working in large-scale enterprise data environments
• Exposure to data governance, data quality, or observability frameworks
• Experience mentoring or guiding junior engineers

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.