You will be responsible for building scalable, efficient data solutions and ensuring data quality, reliability, and performance in large-scale environments.
* Design and implement ETL pipelines: Develop, optimize, and maintain ETL processes for the ingestion, transformation, and extraction of large datasets using Spark (Scala/Python).
* Data modeling and processing: Build scalable data processing applications that handle high-volume and complex data from various sources.
* Spark Application Development: Write optimized, scalable Spark applications in Scala or Python to process and analyze large datasets.
* Elasticsearch Integration: Utilize Elasticsearch for indexing, searching, and querying big data; ensure the data in Elasticsearch is well-structured and optimized for performance.
* SQL Expertise: Develop complex SQL queries and scripts to perform data analysis, transformations, and ensure data integrity across the pipeline.
* Data Architecture: Collaborate with architects to design data solutions that are resilient, efficient, and easy to scale.
* Performance Tuning: Optimize Spark jobs and queries for performance, including tuning parameters, resource management, and troubleshooting issues related to data quality and performance.
* Collaboration: Work with cross-functional teams, including data scientists, analysts, and software engineers, to deliver end-to-end data solutions.
* Monitoring and Maintenance: Set up monitoring tools and frameworks to ensure the stability and reliability of data pipelines in production environments.
We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to
Human Resources Request Form. The EEOC "Know Your Rights" Poster is available
here.
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:
https://insightglobal.com/workforce-privacy-policy/ .
We are seeking a highly skilled Big Data Engineer with a strong background in designing and implementing ETL data pipelines using Apache Spark and either Scala or Python. The ideal candidate will have a deep understanding of big data frameworks, advanced SQL skills, and hands-on experience with Elasticsearch
* 6+ years of experience in big data engineering or related fields.
* Strong expertise in Apache Spark for data processing, with hands-on experience developing Spark applications in Scala or Python.
* Experience in ETL pipeline development, including data ingestion, transformation, and processing of large-scale datasets.
* Proficient in Elasticsearch: Experience with Elasticsearch for querying and indexing large datasets, understanding of its architecture and performance optimization.
* Advanced SQL skills: Strong ability to write complex SQL queries for data extraction, aggregation, and analysis.
* Knowledge of distributed data processing and managing large data sets in a cloud or on-premise environment.
* Familiarity with Hadoop ecosystems, HDFS, and other big data technologies.
* Experience with version control (e.g., Git) and CI/CD pipelines.
* Strong problem-solving skills, attention to detail, and a mindset for performance tuning and optimization
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.