The client
iRobot is a global consumer robot company that designs and builds thoughtful robots and intelligent home innovations that make life better. iRobot introduced the first Roomba robot vacuum in 2002. Today,
iRobot is a global enterprise that has sold millions of robots worldwide. iRobot’s product portfolio features technologies and advanced concepts in cleaning, mapping and navigation. Working from this portfolio, iRobot engineers are building robots and smart home devices to help consumers make their homes easier to maintain and healthier places to live.
The challenge
With our data team continuing to help expand the iRobot data platform by creating dozens of data pipelines every year, keeping the platform infrastructure up-to-date manually became a particularly challenging and time-consuming task. Development machines run on various operating systems. They are also set up with different versions of development packages and other tools which sometimes impact manual deployment processes. In addition to that, monitoring such number of resources needed to get simpler and faster as unformatted emails received directly from Amazon CloudWatch were becoming ineffective and have begun to pile up in the spam folders.
The solution
To automate and improve these processes, our DevOps team has built over twenty unique CI/CD pipelines with each being deployed across multiple environments (AWS accounts). Each unique CI/CD pipeline is sourced from a different Bitbucket repository. The contents of those repositories are delivered into AWS using a Bitbucket hook that creates a .zip archive of the repository and pushes it into a designated S3 Bucket used specifically for storing these archives. With different teams using different tools to test and deploy their infrastructure, we had to adjust the pipeline’s testing and deployment processes as well. To easily support different infrastructure types and complexity levels, we utilized AWS SAM CLI, AWS CLI and CodeBuild’s CloudFormation action Provider.
Monitoring improvements were achieved by integrating with Slack. Each pipeline has its own Event Rule that sends failure notifications to a central Event Bus which then triggers a Lambda function. This Lambda function parses CodeBuild execution logs and forms a message containing a code snippet with logs entries which were created at the time of the failure. This function also fetches the name of the project whose CI/CD had failed and the account ID in which the failure had occurred. After forming it, the message is sent via a webhook to a Slack channel intended specifically for CI/CD pipeline failure alerts.
All CI/CD pipelines are built by combining AWS CodePipeline, AWS CodeBuild, AWS Lambda, AWS CloudFormation, Amazon Elastic Container Registry, Amazon Simple Notification Service (SNS), Amazon EventBridge, AWS Identity and Access Management (IAM).
Example: Production CI/CD Pipeline Architecture
The results
Developing CI/CD infrastructures not only automated the test, build and deployment processes, but it also greatly benefited our Data teams, as they could now focus on both existing and new features in the Data Platform instead of spending time on debugging failed manual deployments. The team’s predictivity was increased as test, build and deployment processes became standardized and automated across all environments. The monitoring improvements have also simplified and improved the response time of our DevOps team and the project teams.
