Job Search Results for Big Data

? Design and maintain the overall data platform architecture integrating Snowflake, AtScale, and dbt components? Establish data quality frameworks and monitoring systems? Define platform scalability... More and performance optimization strategies? Design and implement Snowflake warehouse configurations and resource management policies? Manage user access controls, roles, and security policies within Snowflake? Optimize query performance through warehouse sizing, clustering, and partitioning strategies? Implement data sharing and collaboration features across business units? Monitor and control Snowflake costs through usage optimization and resource scheduling? Design and maintain data pipeline architectures for ingestion and transformation? Establish connection protocols between AtScale and various BI tools (Tableau, Power BI, etc.)? Optimize query performance through aggregate tables and caching strategies? Manage AtScale cluster configurations and scaling policies? Implement security models that align with organizational data access requirements dbt Development & Operations? Design and maintain dbt project structures and development workflows? Create and enforce dbt coding standards and documentation requirements? Implement dbt testing frameworks for data quality validation? Manage dbt deployment pipelines and environment promotion processes? Design modular dbt models that support reusability and maintainability? Establish version control and collaboration practices for dbt development teams Integration & Data Flow Management? Design end-to-end data pipelines connecting source systems to Snowflake via dbt transformations? Implement data lineage tracking across all three platforms? Create monitoring and alerting systems for data pipeline health and performance? Establish disaster recovery and backup procedures for the integrated platform? Design API integrations and data exchange protocols between platforms Team Leadership & Collaboration? Mentor and train data engineers and analysts on platform best practices? Collaborate with data science teams to support ML/AI workload requirements? Coordinate with DevOps teams on infrastructure automation and deployment practices? Lead architectural review sessions and technical decision-making processes Performance & Cost Optimization? Monitor and optimize compute resource utilization across all platforms? Implement cost allocation and chargeback models for different business units? Analyze query patterns and usage metrics to identify optimization opportunities? Design and implement automated scaling policies based on workload demands? Establish SLA monitoring and reporting for platform availability and performance Security & Compliance? Implement data encryption, masking, and anonymization policies? Ensure compliance with regulatory requirements (GDPR, HIPAA, SOX, etc.)? Design and maintain audit trails and data access logging? Coordinate security assessments and vulnerability management? Establish data classification and handling procedures across all platformsPay Rate: $16.00-$22.00 depending on skills and experienceWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Insight Global is seeking a hands-on Principal Data Architect to sit hybrid at a cybersecurity client in Newport Beach, San Jose, or Frisco Texas. You bring a strong background in big data... More engineering, modern lakehouse architectures, and decentralized data governance. This is a unique opportunity to define the strategic vision for their next-generation data platform, transition our ecosystem to a Data Mesh paradigm, and implement robust security standards across a petabyte-scale environment.(This role can pay 220,000-245,000 based on years of experience).We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Insight Global is looking to hire a Hadoop Developer for one of our large financial clients in the Charlotte, NC area. We are looking fro a Mid-level to Senior software developer to design, develop,... More maintain and test software for the ECR RCF team, collaborating with the development team and business partners to ensure successful delivery and implementation of application solutions in a dynamic Agile environment. Candidate?s ability to achieve subject matter expertise quickly on new applications is needed. This role requires proven track record in strong data base skills working on Oracle/SQL Server/Exadata, performance tuning, ETL processes, Windows and Linux environments. Experience in multiple database platforms is preferred: SQL Server, Oracle, Exadata, Big data. Programming experience on Java, Hadoop / Big Data, Hadoop and Java scripting, Angular 5, React, NodeJS, a plus.Responsibilities include:o Develop, support Enterprise Credit Risk ETL platformo Develop unit / integration / regression / performance test scripts/ test suites for the frameworko Follow Agile development process - work refinement, estimation, retrospectives, etc. ritualso Interface with users and other team members, understand requirements and engineer and analyze solution optionso Communicate solution effectively to team / teams and play a mentor role for junior resourceso Engage Architecture team as needed during development and supporto Participate in POCs - evaluate tools and technologies as neededWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

The Data Engineer is responsible for architecting, developing, and maintaining enterprise-scale data platforms that support analytics, operational reporting, and machine learning initiatives. This... More role requires deep technical expertise in distributed systems, cloud platforms, and data modeling, combined with strong communication and leadership capabilities.Design and implement large-scale ETL/ELT pipelines using Python, Spark, and distributed processing frameworksDevelop and maintain big data infrastructure leveraging Hadoop, Spark, Kafka, and Kafka StreamsArchitect cloud-native data solutions on AWS or Azure, including serverless componentsBuild and optimize Snowflake data warehouses using dimensional modeling best practicesConduct data discovery and source analysis to support new integrations and transformationsModel complex datasets using normalized, denormalized, star, and snowflake schemasIntegrate external systems through RESTful APIs and automated ingestion frameworksImplement DevOps practices including CI/CD, containerization, and orchestrationDevelop real-time streaming applications using Kafka, Storm, Kinesis, or Pub/SubOperationalize machine learning models in collaboration with data science teamsOptimize performance across queries, pipelines, and distributed workloadsTroubleshoot and resolve complex data issues across upstream and downstream systemsMaintain comprehensive documentation for data pipelines, models, and integrationsWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

A healthcare client of ours is looking for multiple Data Engineers to join their growing team. This team is currently working on a project to modernize on prem SQL warehouses to snowflake. The data... More warehouse is built, so you will be supporting the build out of data marts on top of their Snowflake enterprise warehouse. This includes lifting and shifting existing data marts, building net new data marts, and further integrations. Additionally, the data engineering team designs, tests, deploys and maintains data pipelines and establishes a strong data reporting environment. This company operates within an AWS environment which is supported by their DevOps team; however, any cloud integration experience could be helpful. This position is currently slated as a 6-month contract, with high likelihood to extend for an additional 6+ months. Furthermore, this org has a large pipeline of project work for the next three years, so continuous extensions are likely. Insight Global offers full benefits while on contract. If you're interested, apply today!We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

Insight Global is looking for a Lead Data Engineer to come join their team. This candidate will be helping design and deliver modern data solutions using cloud-based platforms and AI-driven... More innovations. You will partner with business, technology, and governance teams to enable scalable, secure and compliant data platforms that power advanced analytics and AI. You will act as the enterprise expert for designing, developing, and optimizing data processing systems and pipelines used to populate data into transactional databases, data warehouses, data marts, or other data repositories. You will also coordinate with other IT teams to configure and implement security, monitoring, tooling, and related capabilities as they apply to their assigned systems. The Day-to-Day:?Use Agile practices to elicit and refine requirements through an iterative process of planning, defining acceptance criteria, prioritizing, developing and delivering enterprise data asset solutions in collaboration with business intelligence teams or other IT teams?Develop, configure, customize and manage integration tools, databases, warehouses and analytical systems?Design, build, and deploy data extraction, transformation, and loading processes and pipelines from various sources including databases, APIs, and data files?Develop and deploy structured, semi-structured, and unstructured data storage models such as data vault or dimensional modeling on a various platforms to meet business requirements?Coordinate security, application monitoring, development tooling, and related capabilities with responsible teams?Provide operational support and incident management for data pipelines and data flows, including proactively updating internal clients and relevant parties on status changes and issues encountered?Author and maintain accurate and up-to-date documentation for processes, procedures, and technical designs?Coach junior data and business intelligence personnel across the company, including developing material and guiding junior personnel on their technical skill growthWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

One of our clients within the Financial Services space is looking for a full-time Data Engineer to support their organization. This position is part of the Enterprise Data & Analytics Capability... More team under Global Technology Organization. In this role, you will lead the design, development, and optimization of large-scale data solutions on the Databricks platform. Day to day responsibilities include the following:?Design, build, and maintain scalable data pipelines on Databricks (using Spark, Delta Lake, etc.)?Write clean, efficient and maintainable PySpark or SQL code for data transformation?Design robust data models for analytics and reporting?Ensure data quality, consistency, and governance?Handle batch and streaming data workflows?Provide architectural guidance and support in platform usage?Drive best practices in data engineering across the team?Monitor and optimize performance of Spark jobs and cluster usage?Ensure compliance with security and data privacy standardsWe are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

We?re seeking a highly skilled Data Engineer to design, build, and maintain production-grade data pipelines that process and transform terabytes of data. In this role, you?ll collaborate closely with... More data scientists and other SWEs to ensure that our data infrastructure is scalable, reliable, and cost-effective.Salary Range - $150,000 - $220,000We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

?Design, build, and maintain highly scalable and reliable data pipelines to move and transform data across large-scale, distributed systems at exabyte scale.?Architect, implement, and deploy new data... More models and data processing workflows in production, ensuring data quality, integrity, and compliance with data privacy regulations.?Collaborate with cross-functional teams ? including software engineers, data scientists, and product managers ? to understand data requirements and deliver effective, high-impact solutions.?Develop and optimize data-driven systems and solutions to enhance operational scale, efficiency, and generate actionable business insights aligned with evolving product and business needs.?Work closely with engineering teams to plan and execute on half-yearly roadmaps that align with business priorities and available engineering capacity.?Perform data quality monitoring, root cause analysis, and proactive resolution of data pipeline issues to ensure high reliability and trust in data assets.?Develop comprehensive documentation for all data pipelines, models, and related processes, and cross-train team members on new and evolving data systems.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less

We are seeking a skilled Data Engineer to design, build, and optimize large-scale batch data pipelines in a cloud environment. This role focuses on reliability, performance, and data quality,... More supporting analytics and downstream consumers through well-engineered big-data solutions. The ideal candidate has strong experience with Apache Spark, cloud data platforms (GCP preferred), and writing performant SQL at scale.Key ResponsibilitiesDesign, develop, and maintain batch data pipelines using Apache Spark, Hadoop, Hive, or similar frameworks in a cloud environment.Build highly optimized, fault-tolerant, and SLA-driven data pipelines that operate reliably at scale.Leverage Google Cloud Platform (GCP) services such as BigQuery, GCS, Dataproc, and Pub/Sub to support data ingestion, processing, and storage.Write and optimize SQL queries (BigQuery SQL and/or Spark SQL) for data analysis, profiling, and performance tuning.Collaborate closely with analytics, data science, and downstream consumers to ensure data availability, correctness, and usability.Monitor and troubleshoot pipeline failures; implement alerting, retries, and data quality checks.Improve pipeline performance through partitioning, clustering, resource tuning, and query optimization.Follow software engineering best practices, including version control, testing, and documentation.We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.Less