Introduction to Release Engineering
Release engineering is a new and rapidly growing specialist subject of software engineering. The standards for application development are derived and will be seriously followed to get the reliable product when release happens. A release engineer has an excellent knowledge of multiple disciplines like source code management, configuration management, development process, and customer support. Release engineering is a flexible process which molded bases on the requirements of the product.
Development Guidelines for Release Engineering
This is the first part of the release engineering process definition, which enlists all the guidelines that must be adhered to by the developers.
This clause states that a proper git-flow branching model must be followed during development. Git branches are reasonable and affordable to create and maintain in the long run. Respective guidelines for every branch are listed below, reference to the working of the git-flow branching model.
Source code master branch represents the production-ready state. Master contains the most stable code and will be deployed to production. The master branch officially reserves the release history of any software application’s source code. All the commits on the master branch must be tagged with a version number. Developers should not commit anything directly into the master branch. Without the approval of the QA team, the code shouldn’t be merged into the master branch.
Instead of a single master branch, our workflow will use two branches to record the project's history. The develop branch serves as a branch that integrates various feature branches. Both developers and maintainers have the privilege to merge feature branches to develop branches. While merging branches into development, the developers should briefly describe the features they worked on that branch.
A feature branch is a child of the developed branch, i.e., instead of branching off of master, feature branches use to develop as their parent branch. Using feature branches, the developers can work on multiple features at the same time. Once new features are designed and tested, the feature branch will merge into the development branch.
- The code for each new feature for your application should reside in its particular feature branch, which can be pushed to the central source code repository for backup and collaboration.
- All the feature branches should exist only on developers' repos, not in origin.
- The feature branches never interact with or get merged into the master branch directly. When a feature is complete, its respective branch always gets merged back into the develop branch.
User Acceptance Testing is the pre-production branch, also known as the staging area. Once code is developed and tested in the develop branch, it gets merged into the uat branch.
- The UAT branch should be locked so that developers couldn’t commit directly to this branch. Only the maintainer should have permission to accept the merge/pull request in the UAT branch.
- The UAT branch should be used for creating a release branch once all testing is done. Bug fixes in UAT shouldn’t be directly merged into the uat branch, all the bug fixes branch should be merged into develop branch first.
The release branch will be derived from the UAT branch. The creation of this branch is done to start the next release cycle, and it indicates that no new features can be added after this—only documentation, bug fixes, and other tasks related to the release should be done in this branch. Source code tagging and release versioning are also done here. Once the code is ready to be shipped, the release branch is merged into the master branch. It is tagged with a version number too. Additionally, it should be merged back into the develop branch, which may have progressed since the release was initiated.
Hotfix branches, also known as maintenance branches, are used for quickly patching the production releases. The bugfix branch is derived from the master branch when there are some bugs in the production environment. As soon as the bug is fixed, the hotfix branch gets merged into both the master and the develop branches and the ongoing release branch, and then the master branch is tagged with an updated version number.
Read more about the Best Practices for Infrastructure as Code
Branch Naming Conventions
This clause states that proper naming conventions must be followed for each branch mentioned below. Branch names should only contain lowercase letters, hyphens, slash, and numbers, regexp: /[a-z0-9\-]+/.
You can use slash or hyphen as a separator while naming a feature branch but make sure the feature branch naming should be homogeneous, don’t mix multiple feature branch naming conventions. Before starting working on a project, decide on one naming convention and stick to that.
Developer should follow proper Merge Process/Request and Release Branching Process.
Before performing a merge, there are a couple of preparation steps to ensure that the merge goes smoothly. Use a short description before merging the code to any branch.
- Confirm the destination branch
- Fetch the latest remote commits
- Apply Merge
feature/branch_name → develop → UAT→ Release→ Master
Release branching refers to the concept of making a release reside within a particular branch. A branch is created when the team starts working on the new release (e.g., “Release 2.1”), and all work is done until the next release is stored in this branch.
Git Commit Guidelines
This clause states the guidelines for Git Commit. When you’re working in a group project or contributing to an open-source project, the commit message is the best way to communicate the context about a change to other developers working on that project, and indeed, to your future self.
But most of the time, developers don't focus on commit messages. They use weird commit messages that don't help anybody, and also, these messages can’t convey what changes are done and why. To make commit messages significant, a developer must follow the following guidelines. Commit message should be short and descriptive, try not to exceed 50 chars ( max 72 chars)
-Commit message should be imperative.
- “Add Dockerfile”
- “Added Dockerfile”
-Commit messages should describe why the change is made and how it addresses the issue.
-Capitalize your subject in the commit message.
- “Fix github integration error.”
- “fix github integration error.”
-If commit refers to an issue/bug, make sure to add the issue/bug number in the commit message.
-Use characters to make the commit message short.
- “Add Hindi & English translation.”
- “Add hindi and english translation.”
-The subject line should not end with a period/Full stop.
- “Add new Dockerfile.”
- “add new Dockerfile.”
Click to know more about Infrastructure as Code Tools to Boost Your Productivity
CI/CD Guidelines for Release Engineering
This is the second part of the release engineering process definition, which enlists all the guidelines that must be adhered to by the release engineers.
Reference to the required process flow diagram.
Every branch mentioned in Part-I above should have a CI/CD pipeline that strictly consists of the below-mentioned stages.
- Unit Testing
- Integration Testing
- Deploy code (if required)
- Code Quality Check
- Unit Testing
- Integration Testing
- Functional Testing
- Deploy to Dev
- Browser Performance Testing
- Load Testing
- Fuzz Testing
- Deploy to UAT
Release Branch: This branch should not contain any pipeline. This branch only has code ready for release. Only the documentation, final touches, and minor bug fixes to the code are focused on.
- Deploy to Prod
Definition of each stage
- Code Quality Check: This will help analyze the source code quality and ensure that the project code stays simple, readable, and easier to contribute to. Source code will be automatically analyzed to surface issues to check if the quality of the code is improving or getting worse with the latest commit.
- Unit Testing: Unit testing is a stage where discrete functions are tested at the source code level. It is a type of testing where we test the individual units or components of a software application. Unit testing fulfills the purpose of validating the performance of each unit of the software code.
- Integration Testing: The integration testing stage is included to perform a level of software testing where individual units/components are integrated and tested as a whole. It fulfills the purpose of exposing faults in the interaction between the integrated units.
- Functional Testing: Functional testing is a quality assurance process and a type of black-box software testing that validates the software system against the functional requirements/specifications. The purpose of functional tests is to test each function of the software application by providing appropriate input and verifying the output against the functional requirements. The internal program structure is rarely considered.
- SAST: SAST stands for Static Application Security Testing. It aims to scan the application source code and binaries to perform static code analysis. This helps to spot potential vulnerabilities before the application deployment. Every merge request triggers the scanning, collects the results, and presents the list of vulnerabilities in a single report.
- DAST: Dynamic Application Security Testing analyzes the running web application for known runtime vulnerabilities. It introduces enhanced security since it is a black box testing where communication with the web application is done through the front-end to identify potential security vulnerabilities in the web application and architectural weaknesses.
- Browser Performance Testing: This is a web performance testing where we ensure the performance of the software in-browser using automated browser performance testing. This would measure the performance of a web page using an open-source tool and create a report which should be uploaded as an artifact. This feature is introduced to provide the overall performance score of each web page.
- Load Testing: This is a typically important part of the CI process where developers can make performance decisions earlier in the development process by measuring how the software responds under load and when changes are introduced. This can be categorized as performance testing.
- Fuzz Testing: Fuzz testing, or fuzzing, measures the application’s response and stability by providing unexpected inputs like some malformed or random data. This helps in monitoring the unpredictable behavior or application crashes. Fuzz testing is important because it finds issues that traditional testing methods typically do not. It lets you discover software defects that should be addressed on priority as these defects may lead to highly exploitable vulnerabilities.
- Build: Regardless of the language, cloud-native software is typically deployed with Docker. A runnable instance of our product is built and shipped to the end-users by combining the application’s source code and its dependencies. This stage of the CI/CD pipeline packages the entire code and its dependencies into images and builds the Docker containers.
- Push: This is a stage where the built docker images are pushed to our secure and private harbor registry with relevant tags to be later retrieved for deployment as per the requirement.
- Deploy: This is the final stage of any pipeline where the application is deployed to an environment relevant to the branch, i.e., dev, uat, or prod.
Best Practices to Design CI/CD Stage
With reference to Design CI/CD Stage mentioned above, it must be ensured that:
- Build and push stages should be inculcated in the pipeline of every branch, namely feature/hotfix, develop, uat, and master except release.
- The deployment stage in the feature/hotfix branch pipeline shall be inculcated only if required for testing purposes. Further, code present in the develop branch must be deployed to a dedicated dev environment. Similarly, code present in the uat branch must be deployed to the uat environment for manual testing by the QA team. Final deployment to the production environment for end-users must be done within the pipeline of the master branch.
- Release branches must not have any pipeline as these branches should only be dedicated to documentation and final touches to the release.
- As far as testing is concerned, the following guideline should be followed:
-Pipeline in the Feature/Hotfix branch must have unit and integration testing only.
-Pipeline in the Develop branch must have unit, integration, and functional testing only.
-Pipeline in the UAT branch must be dedicated to all types of security and performance testings, namely, SAST, DAST, Browser Performance Testing, Load Testing, and Fuzz Testing.
-The pipeline in the master branch should not have any stage of testing.
Read more about Software Distribution solutions and Services
This clause states that there should be isolated deployment environments for DEV, UAT, and PROD.
- Isolation of each environment must be strictly considered to avoid any point of conflict. Ideally, the safest and most secure way to isolate different environments is to use a separate cluster for each environment. This option minimizes the production environment's risk due to potential human mistakes and machine failures. However, this increases the maintenance and administration overhead, reduced utilization and the cost of more infrastructure management.
- Alternatively, a single cluster with different namespaces can be used to isolate different environments. The K8s community has evolved and matured a lot, enabling us to address security issues through proper network policies, node selectors, seccomp, access control, etc.
- Still, to have a strong separation between production and non-production environment, we can have one cluster for DEV + UAT environments and one cluster wholly dedicated for the final production environment.
As a result, this choice completely depends upon the need of the hour. Considering all the probable pros and cons, the best option relevant to the use case shall be implemented.
Naming Conventions based on Environment
This clause states that proper naming conventions should be followed for accessing the applications deployed in different environments.
Suppose an application with the hostname ‘example.com’ is getting developed. So, it must be ensured that:
- The hostname of the application deployed in the DEV environment should be dev.example.com
- The hostname of the application deployed in the UAT environment should be uat.example.com
- The hostname of the application deployed finally in the PROD environment should be example.com
This clause states that proper artifactory management must be given prime importance
Every release in the DevOps CI/CD processes is a collection of artifacts that must be preserved and managed correctly. An artifact is referred to as the by-product of software development. It is considered a deployable component of an application, e.g., docker images, jar files, wheel files, etc.
- Source code, meeting notes, workflow diagrams, data models, risk assessments, use cases, prototypes, and the compiled application are all considered artifacts. There must be a list drawn up during the planning stage that covers all of the required artifacts produced.
- Stable tagging of artifacts like docker images must be ensured for proper segregation and easy identification. The tag might include the project name, repository name, branch name, pipeline ID, etc.
For eg: lifedata/stt-server:develop123456
- An Artifactory repository must be used for storing and managing artifacts. It is an application designed to store, version, and deploy artifacts for builds. It is both a source for artifacts needed for a build and a target to deploy artifacts generated in the build process.
Release Your Product to End Users
Product Shipping to end-Users
This clause states the process of product shipping to end-users.
Once your code is ready to be shipped to the live environment, the product team must follow the following clauses for the smooth release of the product to the end-users.
- The developer should create Changelog files for every microservices of the product. The developers must stick to the given link for the Changelog file structure.
- The production-ready code should also contain README files.
- The QA team should send the QA signoff to the developers and release engineers. Without the QA sign off the code should never be merged into the production.
- The product team should be ready with all the user documents and reference documents
- The sales and marketing team should have a basic idea of the product to talk about its features with the customers.
- The support team should be properly trained to counter any user issues in the product.
- The release engineers should create a checklist and cover all the things that are essential for the release.
- Once everything in the checklist is completed, the developer must create a release branch for the release versioning and the tagging.
- The developer should follow semantic versioning while creating a tag or a release.
- Once the release versioning is done, the branch should be merged into the production branch by the developers.
This clause states the importance of an appropriate application deployment strategy.
- Recreate: Version A is terminated, then version B is rolled out.
- Ramped (also known as rolling-update or incremental): Version B is slowly rolled out, replacing version A.
- Blue/Green: Version B is released alongside version A, then the traffic is switched to version B.
- Canary: Version B is released to a subset of users, then proceeds to a full rollout.
- A/B testing: Version B is released to a subset of users under specific conditions.
- Shadow: Version B receives real-world traffic alongside version A and doesn’t impact the response.
Choosing the Right Deployment Strategies
This type of deployment strategy states that the old pods will be killed all at once and replaced with the new ones. This deployment approach involves downtime while the old versions are being brought down and the latest ones are starting up.
- Your application does not support multiple versions and can withstand a short amount of downtime.
- You have a ReadWriteOnce volume mounted to your Pod, and you cannot share it with a Replica.
- You want to stop processing old data and run some prerequisites before starting up your new application.
It’s the default strategy in Kubernetes, in which new pods are slowly rolled out by replacing the old pods without causing any downtime. A rolling update doesn’t scale down the old pods unless it validates via the readiness probe and confirms that the new pods have become ready. You can abort the rolling update or deployment without bringing the whole cluster down if there is any problem.
The rolling update configuration consists of:
- max surge: Maximum numbers of pods that can be created over a desired no of pods.
- max unavailable: no of pods that can be unavailable during the update process
- Releasing the application/service to development/staging environments.
- Downtime in an application that processes a massive number of transactions per minute can cause lots of problems. In the Rolling update, the application continues to operate with Zero Downtime.
- Feature of “track/record” deployments, useful in rollback.
In this strategy, the old version and the new version of the application are deployed simultaneously. The users will be able to access only the old version, and the new version is firstly available for the QA team for testing, once they’re done testing, the users' traffic is shifted to the new version.
It helps in testing a production-quality environment before it is made public. In contrast to the rolling and canary deployment approaches, it also enables them to switch all users over to a news release at once.
- Quick rollbacks and easy disaster recovery.
- Blue/green deployment is zero-downtime, so the team can make the switch and let the load balancing system automatically shift all users to the green version. The old, blue version of the application is ready and waiting in case something goes wrong and requires it to be rolled back.
- Overhead: running two identical environments is expensive.
In the Canary deployment strategy, we shift a controlled percentage of user traffic to the new version of the application. We use this strategy mostly when we’re not confident about the stability of the new version. A key advantage of this app deployment strategy over blue/green deployment is early access to bug identification and feedback. It finds weaknesses and improves the update before the IT team rolls it out to all users.
- Great to try out new features and see how the application and system behave when the traffic is routed to a new version.
- Canary deployments will only be helpful to you if you can track their impact on your system.
A/B testing deployment strategy is about routing a subset of users to a new functionality under specific conditions. It is usually a technique for making business decisions based on statistics rather than a deployment strategy.
- GGG gives full control over traffic distribution, and you can track customer behavior and revenue results.
- Several versions run in parallel, and It’s easy to revert to the older version in case the new version does not provide significant benefits.
- Overhead: Expensive Setup
A shadow deployment consists of releasing version B alongside version A, fork version A’s incoming requests, and sending them to version B without impacting production traffic. This is particularly useful to test production load on a new feature. A rollout of the application is triggered when stability and performance meet the requirements.
- It frees you from setting up a dedicated load test environment. The load tests are based on actual traffic in your existing environment.
- Performance testing of the application can be done with production traffic for more accuracy.
- Overhead: Expensive and complexity of setup(duplicate transactions or requests).
Click to explore about Governance as a Code: Managing Infrastructure in Cloud
Rollback is an operation that returns the object to some previous state
We do the Rollback to an earlier Deployment revision if the current state of the Deployment is not stable or not giving expected results due to the application code or the configuration.
There are scenarios where even the perfect deployment gets your application halted when deployment is integrated with a newer version. At that time, we had to get back to the application version that was running fine with the help of revisions available for the deployments.
Steps to follow while performing a rollback:
- Firstly find the cause of the error because of which application is getting into problems. It can be reviewed through the logs and state of the deployment(its pods). Some error references: CrashLoopBack, ImagePullBackOff, etc.
- We follow the rollback process if the problem cannot be resolved and there is a need to roll back.
-Check the rollout history for the revisions available. If the --record flag was used while creating the deployment, you could check the change-cause.
kubectl rollout history deployment.v1.apps/<deployment-name>
-Pick the revision which you consider stable and working as expected, make its(revision) entry in the --to-revision flag
kubectl rollout undo deployment.v1.apps/<deployment-name> --to-revision=<number-of-the-revision>
-You can describe the rolled back deployment and verify its configuration.
kubectl describe deployment nginx-deployment
Note: By default, Kubernetes stores the last 10 ReplicaSets. But you can change the number of ReplicaSets to be retained by changing the spec. Revision history limit key in your Deployment file.
A release engineering is a fast-growing software engineering profile and is a systematic process from development to release of a reliable product. An organization must establish the release engineering process to deliver its product rapidly with high quality.