Job Description
A rapidly expanding company supporting federal and military programs is seeking a fast-learning and self-motivated Data Engineer with Apache Kafka/NiFi experience to design, build, and maintain systems and infrastructure that collect, store, and transform data for a mission-critical federal program. This effort is primarily centered around Tactical Network data engineering support. This is a hybrid role in North Charleston, SC.
You Will:
- Support designing and building ETL (Extract, Transform, Load) processes and pipelines to move data from disparate sources into centralized repositories.
- Develop automated deployment pipelines and Infrastructure-as-Code templates using scripts and orchestration tools to automate repetitive data tasks, reducing manual intervention and error.
- Design and optimize data architectures to improve performance, cost-efficiency, security, and scalability.
- Design and implement large-scale processing systems and database architecture solutions (e.g., Data Lakes, Data Warehouses).
- Provide recommendations for data engineering best practices and process improvements.
- Contribute to team objectives and support data engineering initiatives.
- Prepare technical documentation and present architecture proposals to engineering teams.
- Partner with other engineering groups to understand their data requirements and with leadership to align technical infrastructure with mission objectives.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Required Skills & Experience
- Possession of and the ability to maintain a Secret Security Clearance.
- 3–5 years of experience supporting data engineering efforts and initiatives.
- Experience with containerization technologies (Docker, Kubernetes) is highly valued.
- Experience with Helm charts and deploying, managing, and scaling applications.
- Working knowledge of Agile methodology and DevOps practices.
- Experience with open-source, distributed event streaming platforms used to collect, process, store, and analyze data feeds (Apache Kafka).
- Experience with data orchestration tools (Apache NiFi).
- Experience with open-source monitoring and observability architectures including Grafana LGTM+ Stack and Elastic (ELK) Stack.
- Experience with high-performance, open-source networking infrastructure tools (Nginx, Apache APISIX).
- Experience with open-source, distributed event streaming platforms used to collect, process, store, and analyze data feeds (Apache Kafka, Debezium).
- Experience with data orchestration tools (Apache NiFi).
- Experience with open-source data storage and caching services (Postgres, Redis, MinIO).
- Strong documentation and communication skills to work effectively across technical teams.
- Bachelor's degree preferred or equivalent experience.
Nice to Have Skills & Experience
- Bachelor's degree in Computer Science, Engineering, or related field.
- 5+ years of experience in data engineering roles.
- Experience with CI/CD tools (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
- Proficiency with version control systems and collaborative development tools
- Experience with configuration management and automation platforms (Ansible, Chef, Puppet).
- Programming/scripting experience (Python, PowerShell, Bash, Go).
- Experience with data engineering design and visualization tools (Swagger).
- Knowledge of microservices architecture and serverless computing
Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.