Job Description
Insight Global is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.
The ideal candidate combines deep Teradata DBA expertise with hands-on Databricks engineering capability and applied AI/ML skills. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.
Key Responsibilities
Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Required Skills & Experience
-7+ years of Teradata DBA / Administration - Primary platform under analysis; deep knowledge of system internals required
-Databricks Platform (Workspaces, Clusters, Jobs) - Core analysis and solution delivery platform for the engagement
-Machine Learning (scikit-learn, MLflow) - Anomaly detection on access patterns, predictive cold-data classification
-LLM / Generative AI Integration
Nice to Have Skills & Experience
Familiarity with DataStage orchestration log parsing
Prior work on data platform cost optimization or cloud migration readiness assessments
Experience working in healthcare payer data environments (HIPAA awareness)
Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs)
Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.