Databricks Developer

Insight Global is seeking a Databricks Developer for a large maritime transportation enterprise. This role is a hands‑on technical contributor responsible for building high‑performance data pipelines, optimizing Spark workloads, and enabling scalable data products across a modern Azure cloud environment. The Databricks Developer will work closely with data engineers, analysts, and architects to deliver reliable, trusted datasets that power enterprise analytics and reporting.

Responsibilities
Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
Apply best practices in data modeling and data warehousing concepts.
Perform data validation and quality checks.
Work with various data formats, including JSON, Parquet, and Avro.
Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, and SQL Server.
Create parameterized and dynamic ADF pipelines, and trigger Databricks notebooks from ADF.
Collaborate closely with Data Scientists, Data Analysts, Business Analysts, and Data Architects to deliver trusted, high-quality datasets.
Contribute to data governance, metadata documentation, and ensure adherence to data quality standards.
Use version control tools (e.g., Git) and CI/CD pipelines to manage code deployment and workflow changes.
Develop real-time and batch processing pipelines for streaming data sources such as MQTT, Kafka, and Event Hub.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

3+ years in data engineering or big data development
Strong Databricks + Apache Spark experience (PySpark, Spark SQL, Delta Lake)
Azure Data Factory + Azure Data Lake expertise
SQL & Python proficiency for transformation and validation
API integration experience using requests/http
Delta Lake architecture knowledge including performance tuning
Experience with real‑time and batch design patterns
Comfort with Git, CI/CD, and Agile delivery

Unity Catalog governance
dbt, Azure Synapse, or Microsoft Fabric
Azure/Databricks certifications
ML/IoT familiarity for anomaly detection and predictive modeling

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.