Job Description
Responsibilities:
• Design and maintain scalable ETL/ELT pipelines to process time-series data from thermostats, weather sources, schedules, and device analytics.
• Develop and validate HVAC and energy datasets (e.g., Run & Drift, Thermal Coefficients, Efficiency Drift) as trusted sources for analytics, modeling, and automation.
• Refactor legacy Scala/Akka workflows into PySpark or Databricks jobs, enhancing observability, automated testing, and CI/CD coverage for upstream data feeds.
• Ensure reliable ingestion and consistency of data from MongoDB telemetry, Kafka/Pub/Sub streams, and cloud storage (GCS).
• Partner with data scientists to create and operationalize features for HVAC runtime prediction, anomaly detection, and demand response optimization.
• Implement best practices for data lineage, schema documentation, and change management to safeguard production systems against regressions.
Day to Day:
One of Insight Global's clients is looking for a Data Engineer to join their team. This Individual will design and maintain scalable data pipelines that convert raw IoT telemetry into analytics-ready datasets, enabling energy and AI-driven applications. They will build and optimize data products that transform device telemetry into reliable, actionable intelligence. Additionally, they will manage integrations with streaming platforms and cloud storage. They will collaborate with data scientists to develop predictive models and demand response solutions, and enforce best practices for data governance. Additionally, the role may involve supporting AI/ML workflows for energy forecasting and other advanced applications. The ideal candidates is someone who has a startup mentality and isn’t afraid of working on projects with little to no predefined requirements.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Required Skills & Experience
Must-Have:
• 5+ years of data engineering or backend data systems experience
• Strong proficiency in Python, SQL, and distributed data frameworks (PySpark, Databricks)
• Hands-on experience with GCP, Kafka/Pub-Sub, and data lake architecture
• Ability to read and modernize legacy Scala/Akka codebases
• Experince building production-grade pipelines that deliver analytics-ready datasets
Nice to Have Skills & Experience
Pluses:
• Skilled in ML platforms and feature engineering (e.g., Vertex AI)
• Hands-on experience with AI/ML applications: LLMs, computer vision, energy forecasting
• Knowledge of Demand Response and home energy automation
• Strong IoT background: protocols, telemetry
• Proficient with specialized databases:
o Graph (Neo4j)
o Vector (Pinecone)
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.