Insight Global is searching for a remote Data Engineer with proficient technical experience in Azure Databricks, Apache Spark, and cloud data engineering.
The role requires proficiency in designing, implementing, and optimizing data processing workflows on the Azure platform using Databricks. The candidate should have a deep understanding of big data technologies, data analytics, and machine learning concepts.
Responsibilities:
Design, implement, and maintain data processing pipelines using Azure Databricks, Apache Spark, and related technologies.
Collaborate with other data engineers, data scientists, and business stakeholders to understand data requirements and develop solutions that meet business objectives.
Architect scalable and reliable data solutions on the Azure platform, ensuring high performance and availability.
Optimize data processing workflows for performance, cost-effectiveness, and resource utilization.
Develop and deploy machine learning models and algorithms using Azure Databricks for predictive analytics and data-driven insights.
Implement data security and compliance measures to ensure data integrity and regulatory compliance.
Monitor and troubleshoot data processing jobs, identify performance bottlenecks, and implement optimizations to improve efficiency and reliability.
Collaborate with cross-functional teams to integrate data pipelines with downstream systems and applications.
Stay updated with the latest developments in big data technologies, machine learning frameworks, and cloud computing platforms.
Document technical designs, configurations, and best practices for Azure Databricks solutions.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to
HR@insightglobal.com.
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:
https://insightglobal.com/workforce-privacy-policy/ .
- 2+ years of Databricks experience in AWS or Azure.
- Proven experience in data engineering, big data processing, and analytics.
- Strong proficiency in Azure Databricks, Apache Spark, and Scala or Python programming languages.
- Experience with building and optimizing data pipelines using Azure services such as Azure Data Lake Storage, and Azure Cosmos DB.
- Knowledge of distributed computing concepts and parallel processing techniques.
- Familiarity with data modeling, ELT (Extract, Load, Transform) processes, and Delta Data Lake principles - Medallion Architecture.
- Experience with version control systems such as Git and CI/CD pipelines for automated deployment.
- Strong problem-solving skills and the ability to troubleshoot complex data engineering issues.
- Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
-Experience with Agile development methodologies and DevOps practices.
- Knowledge of data governance, data privacy, and regulatory compliance standards.
This position requires expertise commensurate with a high-level developer, with at least eight (8) years of experience, and a BS in Computer Science or related field.
Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.