Our client operates within the biopharmaceutical industry, specialising in the development of innovative therapies for severe diseases. Dedicated to advancing scientific research and improving patient outcomes, they leverage cutting-edge drug discovery and development processes.

With a strong focus on collaboration and innovation, the client seeks to utilise advanced technologies and analytical methods to accelerate the delivery of life-changing treatments. Confidentiality and excellence are paramount as they actively pursue partnerships with industry-leading firms to further their mission.

Challenge

The client faced significant challenges in expanding their cloud infrastructure due to numerous internal legacy applications and tools deployed on on-premises servers. This complexity compounded their deployment processes, making it essential to find streamlined solutions. Initially, the client lacked CI/CD pipelines on AWS, making it difficult to deploy software quickly and efficiently while maintaining high standards for code quality and security.

They aimed to enhance their AWS deployment process by leveraging new cloud technologies for automated tests, security checks, and code reviews. However, without a robust CI/CD pipeline system, updating software promptly and ensuring consistent quality posed significant challenges.

Solution

To address these challenges, we engineered a highly reusable and maintainable CI/CD pipeline leveraging the Python AWS Cloud Development Kit (CDK), AWS CodePipeline, and AWS CodeBuild. This solution enabled the client to expedite deployments while adhering to best practices in software development. Our pipeline is capable of deploying CI/CD processes for CloudFormation, S3, and ECS, ensuring flexibility and scalability.

Pipeline Architecture

The pipeline is imported as a git submodule, ensuring ease of updates and customisation for different projects. Core functions can be updated without affecting individual project pipelines, which can be modified on a project-specific basis using dedicated branches.

Deployment Process:

1. Code Push: Developer pushes code to GitLab or Bitbucket, triggering the AWS CodePipeline.
2. Styling, Linting, Testing: The code undergoes styling (e.g., black, flake8), linting, unit tests (with 75% coverage), and security scans (e.g., Snyk).
3. Artifact Storage: Built artifacts are encrypted and stored in an S3 bucket.
4. Deployment to Development: Stack is deployed to the DEVELOPMENT account.
5. Approval for Test: Approval email is sent to the admin group before deployment to the TEST account.
6. Test Deployment: Stack is deployed to the TEST account.
7. Approval for Production: Another approval email is sent before deployment to production.
8. Production Deployment: Stack is deployed to the PRODUCTION account.

Pipeline Stages

Basic Pipeline: Test, Build, Approval for PROD deployment, CloudFormation stack deployment

Advanced Pipeline: Lint, Security Check, Test, Build, Approval for PROD deployment, CloudFormation stack deployment, Power Tuning

Final Pipeline: Lint, Security Scan (Optional), Test, Build, Documentation Generation (Optional), Deploy, Power Tuning

Hot-Fix Pipeline: Triggered by changes in the hotfix branch, skips development and test deployments, direct deployment to production with build lint tests and security scan

Key Features and Capabilities

• Comprehensive CI/CD Automation: Developed reusable CI/CD pipelines with AWS CodePipeline and CodeBuild for various deployment targets.
• Unit Test Integration: Enforces 75% code coverage in unit tests and 100% successful integration test execution.
• Customisable Test Stages: Multiple buildspec files provided, automatically generating optional test stages.
• Code Quality Assurance: Mandatory linting stage to enforce coding standards.
• Advanced Security Measures: Integrated Snyk for detecting vulnerabilities and encrypting artifacts stored in S3.
• Performance Optimisation: AWS Lambda power tuning tool for optimising function configurations.
• Automated Documentation: Generated documentation based on in-code docstrings published to internal Confluence pages.
• Real-Time Deployment Monitoring: Microsoft Teams integration for real-time updates on deployment statuses.
• Access to Bitbucket Metadata: Custom approach for accessing version control metadata within the pipeline using CodePipeline variables syntax.

Results and Benefits

• Accelerated Deployment Process: Over 20 CI/CD pipelines deployed, significantly speeding up the software deployment process.
• Enhanced Code Quality: Implementation of linting stages improved code quality across the organisation.
• Increased Test Coverage: Automated testing ensured higher software reliability.
• Improved Security: Snyk security scanning detected and mitigated critical vulnerabilities.
• Simplified Deployments: Streamlined processes empowered developers to focus on innovation.
• Enhanced Visibility and Collaboration: Real-time deployment status updates fostered transparency and improved team collaboration.

Cost Management

• Cost Optimisation: Utilised AWS CodePipeline and CodeBuild to manage and optimise costs through the use of spot instances and reserved instances.
• Budget Impact: The CI/CD pipeline implementation resulted in cost savings compared to previous manual deployment processes.

 Scalability and Future-Proofing

• Scalability: The CI/CD pipeline supports handling increased load and more frequent deployments using CodePipeline and CodeBuild.
• Future Enhancements: Planned enhancements include refactoring to use CDKv2, implementing hotfix stacks, adding integration testing stages, integrating optional ECS resource allocation, and implementing notifications for power tuning results.

Compliance and Best Practices

Compliance Standards

Through our implementation of linting, unit testing, and security scanning, we ensure adherence to several key compliance standards, including:

• ISO/IEC 27001: This standard outlines requirements for establishing, implementing, maintaining, and continually improving an information security management system (ISMS). Our CI/CD pipeline supports ISO/IEC 27001 compliance by incorporating security scanning with Snyk to identify and mitigate vulnerabilities, ensuring that security best practices are followed throughout the development lifecycle.
• SOC 2: This standard is designed for service providers storing customer data in the cloud. It defines criteria for managing customer data based on five “trust service principles”—security, availability, processing integrity, confidentiality, and privacy. By enforcing coding standards through linting, ensuring 75% code coverage in unit tests, and running comprehensive security scans, our pipeline aligns with SOC 2 requirements.
• GDPR: The General Data Protection Regulation is a legal framework that sets guidelines for the collection and processing of personal information from individuals who live in the European Union (EU). Our security scanning process helps in identifying and addressing potential data protection issues, which is crucial for GDPR compliance.
• HIPAA: The Health Insurance Portability and Accountability Act requires the protection and confidential handling of protected health information (PHI). Our CI/CD pipeline’s security measures, including the encryption of artifacts stored in S3 and the use of secure secrets management, help ensure HIPAA compliance.

AWS Best Practices for DevOps

Our CI/CD pipeline follows several AWS best practices for DevOps, which align with the AWS Well-Architected Framework:

Operational Excellence:

• Deployment Automation: Use of AWS CodePipeline and CodeBuild for automated deployments reduces human error and increases deployment speed and consistency.
• Monitoring and Logging: Integration with Microsoft Teams for real-time deployment status updates enhances visibility and operational awareness

Security:

• Security Scanning: Integration of Snyk for vulnerability scanning ensures early detection and remediation of security issues.
• Encryption: Encryption of built artifacts stored in S3 protects data at rest.
• Access Management: Use of IAM roles and policies to manage access to AWS resources securely.

Reliability:

• Automated Testing: Comprehensive unit tests with enforced 75% code coverage and integration tests ensure the reliability of deployed code.
• Approval Workflow: Manual approval steps before deploying to TEST and PROD environments ensure an additional layer of oversight and reliability.

Performance Efficiency:

• Lambda Power Tuning: Optimisation of AWS Lambda function configurations for performance efficiency.
• Scalable Architecture: The pipeline supports handling increased load and more frequent deployments, ensuring scalability.

Cost Optimisation:

• Cost Management: Utilisation of AWS CodePipeline and CodeBuild for cost-effective CI/CD processes, including the use of spot instances and reserved instances to manage costs effectively.

Continuous Improvement:

• Feedback Loops: Real-time notifications and automated documentation generation facilitate continuous improvement and learning.
• Customisation and Extensibility: The use of git submodules allows for easy updates and customisation of the pipeline, ensuring it can adapt to evolving project needs and technologies.

By adhering to these compliance standards and AWS best practices, our CI/CD pipeline not only meets the necessary regulatory requirements but also ensures a secure, reliable, and efficient development and deployment process. This alignment positions the client to maintain high standards of operational excellence and security, ultimately supporting their mission of delivering life-changing therapies more effectively.

Training and Knowledge Transfer

Training Programs

To ensure the client can maintain and extend the CI/CD pipeline effectively, we provided comprehensive training programs tailored to different roles within the development and operations teams. The training programs covered:

• Introduction to CI/CD Pipelines: Overview of continuous integration and continuous deployment principles, benefits, and best practices.
• AWS Services Overview: Detailed walkthrough of AWS services used in the pipeline, including AWS CodePipeline, CodeBuild, S3, and CloudFormation.
• Pipeline Architecture: Explanation of the pipeline architecture, including the use of git submodules and the separation of core functions from project-specific customisations.
• Hands-On Workshops: Interactive sessions where participants built and deployed sample applications using the CI/CD pipeline.
• Troubleshooting and Debugging: Techniques and tools for diagnosing and resolving issues within the pipeline, including interpreting build logs and error messages.

Documentation

We created comprehensive documentation to support the client’s ongoing use and extension of the CI/CD pipeline. The documentation includes:

User Guides:

• Pipeline Setup Guide: Step-by-step instructions for setting up the CI/CD pipeline for new projects, including repository configuration, integrating with GitLab/Bitbucket, and defining buildspec files.
• Deployment Guide: Detailed procedures for deploying applications using the pipeline, including triggering deployments, approving stages, and monitoring deployment progress.
• Testing Guide: Guidelines for writing and integrating unit tests, linting, and security scans into the pipeline.

Developer Documentation:

• Codebase Overview: Detailed explanation of the pipeline’s code structure, including the Python AWS Cloud Development Kit (CDK) scripts and their configurations.
• Customisation Guide: Instructions for customising the pipeline for specific project needs, including adding new stages, modifying existing ones, and creating project-specific branches.
• API Reference: Comprehensive reference for the APIs and CLI commands used in the pipeline, including examples and use cases.

Troubleshooting Manuals:

• Common Issues and Solutions: List of common issues encountered during pipeline setup and deployment, with detailed troubleshooting steps and solutions.
• Error Message Glossary: Explanation of common error messages and logs, including their causes and resolutions.
• Support Resources: Contact information for support channels and links to relevant AWS documentation and community forums.

FAQs:

• General FAQs: Answers to frequently asked questions about CI/CD principles, pipeline setup, and usage.
• Technical FAQs: Detailed responses to common technical queries, such as handling specific error codes, customising buildspec files, and optimising pipeline performance.

Video Tutorials:

• Getting Started Videos: Short videos demonstrating the initial setup and basic usage of the CI/CD pipeline.
• Advanced Features Videos: In-depth tutorials on advanced features like integrating Snyk for security scanning, using AWS Lambda power tuning, and setting up automated documentation generation.
• Troubleshooting Videos: Visual guides to troubleshooting common issues, including live debugging sessions and step-by-step problem resolution.

Knowledge Transfer Sessions

In addition to formal training programs and documentation, we conducted several knowledge transfer sessions to ensure a smooth handover and ongoing capability building within the client’s team. These sessions included:

• Kickoff Meeting: Initial meeting to introduce the project, set expectations, and outline the knowledge transfer plan.
• Regular Check-Ins: Weekly check-ins during the initial rollout phase to address questions, provide guidance, and ensure smooth adoption of the pipeline.
• Deep-Dive Sessions: In-depth technical sessions focused on specific aspects of the pipeline, such as security integrations, performance tuning, and deployment strategies.
• Post-Implementation Review: Review session after the initial implementation phase to evaluate the pipeline’s performance, gather feedback, and discuss potential improvements and future enhancements.

Conclusion

By implementing a robust CI/CD pipeline using AWS tools and integrating essential features like unit testing, security scanning, performance optimisation, and real-time monitoring, we significantly enhanced the client’s software deployment process.

This solution not only addressed the immediate challenges posed by legacy systems but also positioned the client to better leverage advanced cloud technologies, ultimately supporting their mission of delivering life-changing therapies to patients more efficiently.