Insight Global is looking to add a Lead Data Engineer to our client's team who are headquartered in the St. Louis area but can sit remotely. Our client is currently going through a major acquisition with a deadline of May 2025 for the acquisition to be fully complete. This individual will be overseeing a team of 2 Data Engineers with 40% leadership responsibility and 60% hands on duties. They will be responsible for moving applications from one company to another in addition to ensuring every component has been upgraded and will also be responsible for modernizing the framework as well. They will need someone to come in and lead all development efforts across the enterprise and will interact with multiple different application owners & software vendors so communication is very important. They will be using Apache Spark, Apache Iceberg, and Apache Airflow for ETL pipelines. This person will transform and integrate EBCDIC Mainframe data into Hive and Impala tables using Precisely Connect for Big Data, optimize data transformation processes for performance, scalability, and reliability, and ensure data consistency, accuracy, and quality across the ETL pipelines. This person will function as the point of escalation for ETL loads, data quality issues, and production support issues, monitor ETL workflows and troubleshoot issues to ensure smooth production operations and implement proactive monitoring, alerting, and reporting solutions to improve pipeline reliability. This individual will be responsible for supporting and rebuilding legacy ETL jobs (currently not using ACID transactions) with modern solutions using Apache Spark and Apache Iceberg to support ACID transactions. They will be overseeing design, and implement workflows for automating data pipelines using Apache Airflow as well as establishing and enforcing best practices for ETL code development, version control, and deployment using Azure DevOps.
We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to
HR@insightglobal.com. The EEOC "Know Your Rights" Poster is available
here.
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:
https://insightglobal.com/workforce-privacy-policy/ .
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.