Senior Data Engineer

Insight Global is seeking a Senior Data Engineer for a leading insur-tech client. This individual will join a highly technical Data Engineering team focused on modernizing a core subrogation platform and revamping customer data integration capabilities. The ideal candidate is a deeply technical engineer who thrives in large-scale data environments, understands the business impact of data solutions, and can design highly efficient data pipelines within AWS. This role will be responsible for building and optimizing ETL/ELT frameworks, maintaining data lakes and feature stores, supporting machine learning initiatives, and enabling seamless customer integrations through APIs and Guidewire-based solutions. Success in this position requires strong Spark expertise, deep AWS knowledge, excellent communication skills, and the ability to connect technical decisions to measurable business outcomes.

Day-to-Day:

Design, build, and optimize ETL/ELT pipelines that transform customer and enterprise data into actionable datasets
Develop and maintain scalable AWS-based data platforms and data lakes
Partner with Product and Market teams to create reporting tools and dashboards that support business strategy
Build and maintain feature stores used by ML Engineering teams
Develop customer data integration pipelines via APIs, Guidewire, and other ingestion methods
Implement data quality controls and monitoring frameworks
Troubleshoot data issues across distributed systems and streaming architectures
Optimize Spark workloads and cloud resources for performance and cost savings
Collaborate with a globally distributed Data Engineering team
Gather business requirements and translate them into scalable data solutions

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Must-Haves:
Strong proficiency in Python and SQL
Recent, hands-on experience with Apache Spark in production environments
Deep expertise building and optimizing ETL/ELT pipelines
Extensive experience with AWS data services:
Glue
S3
EMR
Athena
Step Functions
DynamoDB
SQS
Experience with distributed processing technologies beyond managed platforms
Experience with data streaming technologies:
Kafka
Spark Streaming
SQS-based event architectures
Strong data modeling experience, preferably within insurance or financial services
Experience designing and maintaining data lakes
Knowledge of medallion architecture (Bronze/Silver/Gold)
Schema evolution and incremental processing experience
Familiarity with MongoDB and document-based data stores
Infrastructure as Code experience (Terraform or CloudFormation)
Experience troubleshooting and resolving data quality issues across distributed systems
Experience implementing data quality frameworks such as Great Expectations
Strong business acumen with the ability to translate technical solutions into business outcomes
Ability to optimize pipelines for performance, scalability, and cost efficiency
Excellent communication skills and ability to explain the business problem being solved

Plusses:
Experience in Insur-Tech or Technology
Experience handling insurance claims data and complex feature relationships
Experience supporting machine learning platforms or feature stores

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.