Lead AWS Glue Developer

We are seeking a Lead AWS Glue Developer to design, build, and maintain scalable cloud-native data integration and ETL solutions within AWS. This role will lead the development of serverless data pipelines, automate cloud infrastructure deployment, and build robust data processing frameworks leveraging AWS Glue, AWS Lambda, DynamoDB, S3, and related AWS services. The ideal candidate will have deep expertise in AWS data engineering, Infrastructure as Code (IaC), Python development, and modern DevOps practices. Experience with AWS Glue, Lambda, CloudFormation, S3, and DynamoDB aligns closely with capabilities referenced in internal AWS engineering profiles.

Responsibilities
AWS Data Engineering & ETL Development
Design, develop, and optimize AWS Glue ETL jobs using Python and Apache Spark.
Build scalable batch and event-driven data processing solutions.
Develop and maintain data pipelines that ingest, transform, validate, and load data into AWS data stores.
Create and manage AWS Glue Data Catalogs, crawlers, and metadata management processes.
Implement data quality controls, monitoring, and exception handling throughout ETL workflows.
Serverless Application Development
Develop AWS Lambda functions to automate file processing and event-driven workflows.
Build solutions that process files dropped into S3 buckets and orchestrate movement of data into DynamoDB tables and downstream systems.
Implement integrations between S3, Lambda, DynamoDB, and other AWS services.
Troubleshoot, optimize, and enhance serverless applications for performance and reliability.
Cloud Infrastructure & DevOps
Create and maintain AWS CloudFormation templates to automate infrastructure provisioning and deployment.
Configure and manage AWS CodeCommit repositories and CI/CD pipelines.
Establish and maintain AWS environments following Infrastructure-as-Code best practices.
Set up and administer S3 buckets, SFTP services, access controls, encryption, and lifecycle management policies.
Collaborate with DevOps teams to automate deployments and improve platform reliability.
Technical Leadership
Provide technical leadership and architectural guidance to development teams.
Lead code reviews and establish development standards and best practices.
Mentor junior developers and data engineers.
Collaborate with architects, business stakeholders, and product teams to define technical solutions.
Drive performance tuning, scalability improvements, and cloud modernization initiatives.
Testing & Quality Engineering
Write comprehensive unit, integration, and regression tests for AWS Lambda functions and AWS Glue jobs.
Implement automated testing strategies within CI/CD pipelines.
Ensure high code quality, security compliance, and operational excellence.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Required Qualifications
8+ years of software engineering or data engineering experience.
5+ years of hands-on AWS development experience.
Strong expertise with:
AWS Glue
AWS Lambda
Amazon S3
Amazon DynamoDB
AWS CloudFormation
AWS CodeCommit
Python
Apache Spark / PySpark
Experience building serverless and event-driven architectures.
Strong understanding of JSON data structures and APIs.
Experience developing and deploying Infrastructure-as-Code solutions.
Experience building and supporting ETL and data integration pipelines.
Strong troubleshooting, debugging, and performance optimization skills.
Experience writing automated unit and integration tests.
Preferred Qualifications
AWS Certified Data Engineer, Solutions Architect, or Developer certification.
Experience with additional AWS services such as Step Functions, EventBridge, SNS, SQS, CloudWatch, or Redshift.
Experience working in Agile/Scrum environments.
Familiarity with GitHub and modern source control practices.
Knowledge of data governance, security, and compliance requirements.
Experience leading development teams and technical projects.
AI & Productivity Expectations
Comfortable using GitHub Copilot to accelerate software development and improve developer productivity.
Ability to leverage AI-assisted coding tools while maintaining code quality, security, and performance standards.
Interest in applying emerging AI-enabled development practices to cloud engineering and data engineering workflows.

Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.