Job Description
We are seeking an experienced Data Engineer with strong expertise in Databricks and Palantir Foundry to support enterprise data modernization and analytics initiatives. This individual will be responsible for designing, building, and optimizing scalable data pipelines, integrating complex data sources, and enabling data-driven decision-making through robust data architecture and engineering best practices. Experience working within cloud-based environments and large-scale data ecosystems is essential.
Key Responsibilities:
Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL
Build and manage data ingestion frameworks for both structured and unstructured data sources
Develop and support data pipelines within Palantir Foundry, including ontology design, source-to-target mappings, lineage, and transformation workflows
Create and manage Bronze, Silver, and Gold data layers utilizing Medallion Architecture principles
Optimize data processing performance through Databricks cluster tuning, partitioning strategies, and Delta Lake best practices
Collaborate with business stakeholders, architects, data scientists, and analysts to translate business requirements into scalable data solutions
Implement data quality, governance, metadata management, and lineage tracking capabilities
Develop reusable frameworks and automation for data ingestion, transformation, monitoring, and validation
Maintain documentation of data flows, mappings, and technical specifications
Support production deployments, troubleshooting, and ongoing operational support of enterprise data platforms
Partner with cloud engineering and security teams to ensure compliance, scalability, and reliability of solutions
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Required Skills & Experience
8+ years of experience in Data Engineering
Strong hands-on experience with Databricks
Advanced proficiency in PySpark, Spark SQL, and Python
Experience building enterprise-scale ETL/ELT solutions
Hands-on experience with Palantir Foundry, including:
Data pipelines
Ontologies
Lineage tracking
Data integration workflows
Source-to-target mappings
Strong SQL development and performance tuning experience
Experience with Delta Lake and Lakehouse architectures
Knowledge of data modeling concepts, including dimensional and relational modeling
Experience working with cloud platforms (Azure, AWS, or GCP)
Familiarity with CI/CD processes, Git, and DevOps methodologies
Strong troubleshooting, analytical, and problem-solving skills
Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.