Job Description
The scope of this effort spans the data life cycle and includes but is not limited to the planning, design, development, implementation and operational support of the data models, ETL, Data Sourcing processes and related software programs. The client will require support in the adoption of cloud-based technologies such as Spark/PySpark, Databricks, Scala, AWS Glue and S3. The contractor will work and cooperate with a wide range of stakeholders such as end users, SMEs, and technical staff from data vendors such as FINRA and other divisions within the organization.
Required Skills & Experience
The Senior Big Data (Cloud) Developer shall have a minimum of ten (10) years of IT experience with demonstrated hands-on expertise in:
- Development of complex Python scripts (at least 5 years of experience).
- Development and support activities in cloud-based environments using big data technologies and programming languages and tools such as Scala, Spark, PySpark, Presto, Ranger, Hadoop, and Hive etc. (at least 3 years of experience).
- Working in Amazon Cloud environment (AWS) utilizing tools such as S3, EMR, Databricks, Data Lakes, AWS Glue, Amazon StageMaker, Amazon Redshift etc.
- Design, development and tuning of complex ETL/ELT processes involving massive data (order of TB).
- Design, development and support of Web Services (RESTful/SOAP) & Web Scraping (HTTP, CSS and HTML) ETL/ELT processes.
- Working with complex file formats of structured, unstructured and semi-structured data including JSON, XML, CSV, Avro, Parquet etc.
- Data modelling using RDBMS such as Amazon Relation Database Services (RDS), and PostgreSQL, etc.
- Development of clear and concise technical documentations including data models, architecture and data workflow diagrams, design documents, and deployment documents.
- Working with Git, and GitLab.
Nice to Have Skills & Experience
- AWS developer certification
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.