Our customer is a global company operating in Europe and Asia. TerraAlto works closely with them on big data, IoT and data analysis projects.
As part of the ongoing work of the data science team they began to use Lambda for serverless data transformation and processing. It was necessary to enable the data scientists and data engineers with a Lambda deployment pipline for continous deployment and integration capabilities. The customer requested TerraAlto to provide an solution.
- Using the SAM framework for Lambda deployment.
- Code repository on CodeCommit, integrated with Code Pipeline, with staged deployments in Development, Test and Production.
Some of the challenges involved in this project were to:
- Aligning the data science team with usage of SAM framework and Lambda functions.
- Limitations on creating triggers on S3 buckets with Cloudformation, where the S3 bucket already exists.
- IAM access control for data science in an AWS account used by multiple teams.
- Consistent deployment pipeline that enables data scientists with varying level of technical capabilities to use Lambda for serverless data processing.