Generally speaking, a build pipeline is an automation setup which allows you to streamline the development, build and release processes of your projects. It consists of a CI (Continuous Integration) and a CD (Continuous Delivery & Deployment) phase. It ensures that every build is done in a reliable, repeatable way and generates a potential shippable product.
Olaf Spiewok, Senior Developer at Endava, takes a brief look at why you need a build pipeline and discusses some technological options for setting up a build pipeline for games.
Advantages of a build pipeline
These advantages have the highest impact on game development, no matter the build environment and platform:
- Create any release version at any time
It saves a lot of communication time if everyone knows how to use the ‘big red build button’ and what parameters they need.
- Minimise human errors
To prevent mistakes from happening everyone in the team needs to have identical platform settings (same XCode, same system, same Android SDK, same libraries, etc.) or must change them for a specific release build.
- Restrict credentials and user access
Sometimes you want to restrict access within your team, so that not everyone can upload, change, or view all platforms. With a specially configured build pipeline no team member needs to have full access to everything or to be informed of all changes.
- Reduce build time and resources to a minimum
Everyone simply uses a ‘red button’ build pipeline, which will address a set of nodes and thus create the requested builds. Besides some maintenance, everyone can work on the features and topics they have been planned for.
- Apply tracking and analytics
Creating a build pipeline in an early project phase allows you to inspect and analyse changes of the release packages over time. Analysing these changes can streamline your releases.
- Handle platform-specific requirements
Developing for multiple platforms is difficult since you have to make sure that every new feature you add does not break on specific platforms. A build pipeline informs you when a new feature might break the deployment process. A tester can then easily verify the changes.
- Ensure clean, complete and sanitised project integration
Different types of tests from the testing pyramid run at different phases in the build- and release process as post-commit testing, merge-request testing, integration or unit testing. Use this to exclude expected errors. Sanitise the project by rebuilding it from a clean workspace using the build pipeline.
Build pipeline options for game developers
There are many different tools and solutions to help game developers with their builds. Let’s take a look at some of the options I have experience with. Of course, other options could be mentioned as well, like Plastic SCM, but for this article I will stick to the ones I have used or am currently using.
CD server systems (Jenkins, Teamcity etc.)
Once installed and configured, these types of CI/CD systems are in my opinion the best and ‘easiest’ solution in game development. The hardware is in your own hands and can be adapted, updated, and maintained accordingly. You can check for errors and analyse them directly on the system, which – with the appropriate number of nodes – can scale well. However, the maintenance effort and the necessary adaptation in case of system/engine updates needs to be emphasised as well, as this is a recurring effort that should not be underestimated. Apart from that, such systems can easily be transferred to and re-used for other projects.
Engine build systems (Unity Cloud Build, Lumberyard WAF Build etc.)
Many engine vendors provide their own build pipeline solution for their engine. Unfortunately, this is often associated with extra costs, so that many refrain from using these solutions for indie or small projects. Furthermore, the in-house solutions are usually poorly extensible or designed only for certain target platforms.
Custom Solutions with remote server nodes
Most of the time, using purchased nodes as build nodes is a very good alternative which can be scaled perfectly. You can simply build your setup on an externally hosted system and upgrade/downgrade the nodes as needed. Unfortunately, there are always three big problems with this solution:
- Running costs demanded by the provider for the top-tier computers
- Rather mediocre network traffic capability of the nodes, which can only load the data with difficulty due to access and provider restrictions
- Internal security concerns since the data and, if applicable, credentials are openly stored on the external systems
Despite these disadvantages, custom solutions are useful if you maintain them properly and if they are not too specific.
CI Solutions based on OS virtualisation (Bitrise, Git CI, Docker etc.)
Bitrise and Git CI create completely new nodes to start the build process, on which you have to install your engine, its licence, and additional platform-specific tools. On top of the actual build process, which can take 20 minutes and longer, the setup gets added with every build job, which can get annoying pretty quickly.
Bitrise has been trying to find solutions together with Unity since 2017, but they are still struggling with it. Unreal also doesn’t leave a good impression in this regard due to its C++ framework, as build times quickly explode.
Docker can be useful in certain cases but gets very out of hand when serving multiple platforms. The many Docker images for the various platforms and the different engine version images make a gigantic image fleet necessary.
The downsides of build pipelines
Unfortunately, creating a good build pipeline also has some negative points that can put a damper on the initial euphoria.
Maintenance
Besides the time needed for the initial setup for all platforms, with each new engine version, plug-in, or system update, some time is needed for maintenance. Depending on the number of platforms and nodes involved, this can take several days or even weeks. Finding and fixing bugs is usually very difficult with a dedicated system, and it ends up taking several days to reconcile the issue logs with Stack Overflow.
Maintenance can be time-consuming and unexpected, but it’s only necessary for one setup, so it is still worth the effort considering the benefits. Just imagine having to maintain all individual build variants for each developer or tester – suddenly, maintaining your build pipeline doesn’t look that bad, right?
Extra internal costs
Depending on the build pipeline system and engine, additional costs may include:
- Provisioning costs for 1–x nodes (Mac/Windows/Linux machine)
- License costs for the respective nodes
- Monthly costs for the CD platform itself
- Maintenance and operating costs for your own nodes
These additional expenses may pay off for large projects or companies with many application areas but might be unnecessary overhead for indie or one-off projects. Looking at all these potential costs may lead you to want to skip build pipelines for small-scope projects. However, having a developer do the work necessary without a build pipeline multiple times a week will probably amount to similar costs.
Security issues
In a company where the data and credentials are highly sensitive, an external build pipeline can be ruled out from the start. To make it worse, having a shared account within a build account might not be in compliance with the company’s code of conduct. Those obstacles can reduce a fully automated system to its minimum, where everything has become safe but hard to maintain, for example, having internal-level access only or having to build up non-standardised, highly customised self-made systems.
Manual steps
Some plug-ins or tools might require manual steps which you cannot integrate in an automated build pipeline. Steam, for example, requires you to enter a 5-digit code as soon as you want to upload a build from a new environment. Unity wants to configure some internal packages before you can actually use them, which implies that you have to ‘open’ Unity at least once on the system you are using. Solving these problems with a build pipeline can be very frustrating – and might even be impossible in some cases.
The good thing is that at this point, you will know where such issues might appear and which may have to be solved manually.
Conclusion
Setting up a build pipeline always makes sense in the long run. Since you can easily use the pipeline over several projects, the initial effort is minimised with each additional project and a general standard can be established. As soon as the very first code of a project exists, you should set up a build pipeline.
Even if you spend some time setting up and maintaining your build pipeline, the overall time saving still is significant. For example, if I were to individually build all variations for one of our large game projects, I would have to spend around 8 hours! Having multiple nodes in our Jenkins pipeline shrinks the time spent on these tasks to around 2 hours, including building and uploading.
My rule of thumb therefore is: as soon as you have made or have more than 3 releases in front of you – or have to serve more than one platform – set up a build pipeline!