Job Search Results for Big Data

The Data Engineer is responsible for architecting, developing, and maintaining enterprise-scale data platforms that support analytics, operational reporting, and machine learning initiatives. This... More role requires deep technical expertise in distributed systems, cloud platforms, and data modeling, combined with strong communication and leadership capabilities.Design and implement large-scale ETL/ELT pipelines using Python, Spark, and distributed processing frameworksDevelop and maintain big data infrastructure leveraging Hadoop, Spark, Kafka, and Kafka StreamsArchitect cloud-native data solutions on AWS or Azure, including serverless componentsBuild and optimize Snowflake data warehouses using dimensional modeling best practicesConduct data discovery and source analysis to support new integrations and transformationsModel complex datasets using normalized, denormalized, star, and snowflake schemasIntegrate external systems through RESTful APIs and automated ingestion frameworksImplement DevOps practices including CI/CD, containerization, and orchestrationDevelop real-time streaming applications using Kafka, Storm, Kinesis, or Pub/SubOperationalize machine learning models in collaboration with data science teamsOptimize performance across queries, pipelines, and distributed workloadsTroubleshoot and resolve complex data issues across upstream and downstream systemsMaintain comprehensive documentation for data pipelines, models, and integrationsWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Our leading entertainment client is expanding their data team and seeking a Data Engineer with deep expertise in GCP, Big Data, Python, SQL, DBT, and ERD architecture. This cross-functional team... More operates across three core areas: AI/ML, Data Engineering, and Data Analytics.? Collaborate with stakeholders across subscription, viewership, marketing, paid media, CRM, and more. ? Support data visualization initiatives and ensure data is accessible and actionable.? Work in a highly collaborative environment that values fast learners and team players. Spend approximately 80% of your time in hands-on development using Python and SQL.Team Dynamic:Data Analytics Engineering (Approx. 80% of time) - Primary Focus: Analyze and maintain data pipelines, conduct root-cause analysis to resolve data-related issues, and create data models leveraging DBT and Medallion architecture.Tools & Skills: ? Proficient in Python and SQL for pipeline development. ? Initially focused on learning and maintaining existing pipelines, with a gradual transition to building new ones. Mindset: Business-oriented with a strong analytical approach. AI/ML Initiatives (Approx. 5% of time) Objective: Support AI and machine learning efforts using modern platforms such as Vertex AI. Scope: Limited involvement, primarily assisting with existing initiatives and tools. Data Engineering (Approx. 15% of time) Responsibilities: Design and implement scalable data models, pipelines, and architecture.Technical Stack: Heavy use of Python and SQL to ensure robust and efficient data infrastructure.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

The Data Engineer is responsible for architecting, developing, and maintaining enterprise-scale data platforms that support analytics, operational reporting, and machine learning initiatives. This... More role requires deep technical expertise in distributed systems, cloud platforms, and data modeling, combined with strong communication and leadership capabilities.Design and implement large-scale ETL/ELT pipelines using Python, Spark, and distributed processing frameworksDevelop and maintain big data infrastructure leveraging Hadoop, Spark, Kafka, and Kafka StreamsArchitect cloud-native data solutions on AWS or Azure, including serverless componentsBuild and optimize Snowflake data warehouses using dimensional modeling best practicesConduct data discovery and source analysis to support new integrations and transformationsModel complex datasets using normalized, denormalized, star, and snowflake schemasIntegrate external systems through RESTful APIs and automated ingestion frameworksImplement DevOps practices including CI/CD, containerization, and orchestrationDevelop real-time streaming applications using Kafka, Storm, Kinesis, or Pub/SubOperationalize machine learning models in collaboration with data science teamsOptimize performance across queries, pipelines, and distributed workloadsTroubleshoot and resolve complex data issues across upstream and downstream systemsMaintain comprehensive documentation for data pipelines, models, and integrationsWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Insight Global is seeking a Sr. Consultant ? Enterprise Data & Records to join a leading financial services client in Toronto on a 3-month contract with potential to convert. This role sits within... More the data governance and risk function and focuses on improving how data is managed, governed, and leveraged across the organization.The ideal candidate will have a strong background in process improvement, data/analytics, and risk or compliance environments, with the ability to bridge business and technology teams. You will be responsible for driving end-to-end process transformation, enhancing data governance practices, and delivering insights through reporting and dashboards to support executive decision-making.This is a highly cross-functional role, requiring someone who can translate complex data and risk concepts into clear, actionable insights, influence stakeholders, and lead initiatives that improve efficiency, compliance, and overall data quality across the enterprise.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

The Senior Data Engineer / Streaming Data Engineer will design, build, and support enterprise-scale streaming and big data pipelines across AWS, Spark, Hadoop, Kafka, and cloud-native ingestion... More platforms. This role is hands-on and production-focused, with responsibility for reliable real-time data movement, ingestion modernization, distributed systems engineering, and scalable data processing in a large enterprise environment.?Hands-on streaming and real-time data engineering using Kafka, Spark, AWS Kinesis, and cloud-native data services.?Modernization opportunity focused on moving legacy ingestion patterns to scalable AWS-native services.?Production engineering role requiring strong troubleshooting, operational ownership, and distributed systems depth.?Enterprise-scale environment with complex data warehousing, big data, and cross-platform integration needs.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

? Build, maintain, and optimize ETL/ELT pipelines for ingesting, transforming, and processing data.? Ensure data pipelines are scalable, reliable, and efficient.? Design and implement data models,... More warehouses, and lakehouse architectures.? Optimize data storage solutions using relational (SQL) and NoSQL databases.? Work with big data technologies like Spark, Hadoop, and Kafka for large-scale data processing.? Deploy and manage data solutions on cloud platforms (AWS, Azure, GCP).? Improve query performance and storage efficiency.? Implement indexing, partitioning, and caching strategies for faster data retrieval.? Performs other duties as assigned.Main Projects: Converting from old JDE to new JDE- rewriting all integrations in SSIS to stored procedures- moving to Mulesoft and building a Lakehouse and FabricSQL version is 2014 - this person will be helping to move to SQL 2025Managed service provider helping in moving but this person will be leading effortsSalary: $120,000-130,000 annually plus 15% bonus depending on skills and experienceWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Responsible for translating business requirements into analytical constructs and using data to propose solutions for effective decision making. Collects, validates, transforms, and cleanses data, as... More well as performs quantitative analysis to derive insights. Runs analytical experiments in a methodical manner and regularly evaluates alternative models and techniques. Develops predictive models to forecast business performance metrics and provides recommendations for strategic decisions. Responsible for teaching others the tools, techniques and best practices in self-service reporting, data analysis and predictive analytics.?Performs in-depth analyses (e.g., cost-benefit, invest-divest, forecasting, predictive, what-if, impact analysis, etc.) to help the company focus on key decisions to improve safety, employee engagement, operational efficiency, product quality, and customer satisfaction ?Develops, modifies, and automates reports, builds and prototypes dashboards to provide insights, and provides analytical solutions ?Responsible for discovering insights from Big Data to help shape or meet specific business needs and goals. ?Develops and maintains analytical models through understanding data, evaluating technologies, optimizing algorithms, experimenting and validating models ?Delivers effective presentations that tell compelling stories about analytical insights ?Performs data cleansing and blending processes to produce analytic data sets for use by a variety of downstream purposes ?Implements new statistical, mathematical, machine learning or other methodologies for modeling or analysis ?Utilizes business knowledge to translate goals into data-based deliverables, such as predictive models, pattern detection analysis or optimization algorithms ?Trains and enables self-service reporting capability and use of Business Intelligence tools for supported business unit(s) ?Conducts research to identify relevant data for developing prototypes and proof-of-concepts ?Collaborates with cross-functional stakeholders to understand business needs, formulate complete end-to end analyses that includes business requirements, data gathering, analysis, scaleable solutions, and presentations $40/hrExact compensation may vary based on several factors, including skills, experience, and education. Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

We are seeking a highly skilled Data Platform Engineer to design, build, automate, and support enterprise-scale data platforms across hybrid cloud environments. This role will focus on platform... More engineering, infrastructure automation, observability, and the operational excellence of modern analytics ecosystems supporting large-scale data products.The ideal candidate will have deep experience managing distributed data platforms, cloud infrastructure, and big data technologies, while leveraging automation and engineering best practices to enhance reliability, scalability, and performance.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

- Bachelor?s degree in Computer Science, Software Engineering, MIS, or related field- 5+ years of experience building and supporting enterprise-scale data platforms on AWS- Strong hands-on... More programming skills in Python and SQL, with working knowledge of Shell scripting- Experience designing and implementing cloud-based data architectures, including data lakes and warehouses- Proficiency with core AWS services- S3, Glue, Redshift, Lambda. - Experience developing and orchestrating data pipelines using ETL/ELT frameworks- Familiarity with serverless architectures and modern data processing approaches (e.g., Spark/EMR)Understanding of data modeling, big data concepts, and cloud-native design patternsWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Must report onsite to the Bangalore Office 5 days a week**We are seeking a skilled Data Engineer to design, build, and optimize large-scale batch data pipelines in a cloud environment. This role... More focuses on reliability, performance, and data quality, supporting analytics and downstream consumers through well-engineered big-data solutions. The ideal candidate has strong experience with Apache Spark, cloud data platforms (GCP preferred), and writing performant SQL at scale.Key ResponsibilitiesDesign, develop, and maintain batch data pipelines using Apache Spark, Hadoop, Hive, or similar frameworks in a cloud environment.Build highly optimized, fault-tolerant, and SLA-driven data pipelines that operate reliably at scale.Leverage Google Cloud Platform (GCP) services such as BigQuery, GCS, Dataproc, and Pub/Sub to support data ingestion, processing, and storage.Write and optimize SQL queries (BigQuery SQL and/or Spark SQL) for data analysis, profiling, and performance tuning.Collaborate closely with analytics, data science, and downstream consumers to ensure data availability, correctness, and usability.Monitor and troubleshoot pipeline failures; implement alerting, retries, and data quality checks.Improve pipeline performance through partitioning, clustering, resource tuning, and query optimization.Follow software engineering best practices, including version control, testing, and documentation.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less