Unreal Engine automated build and release with Gitlab-CI

Table of Contents

A friend of mine have an Unreal Engine based game project, with the sources hosted at gitlab.com. Since I was setting up my own Gitlab instance and wanted to practice using Gitlab’s CI/CD system, I gave him a hand in automating his building and release system.

Before, to publish a beta of his game for playtesters to try out, he used to develop and build his game locally, push the sources to Gitlab, upload the build to Google Drive, and then post an announcement as well as a changelog to his Discord server.

I figured his workflow could use a little automation. Basically, all he would have to do is push his changes to Gitlab and create a release tag to get a working build posted on Discord as well as a semi-automated changelog.

Here’s how to do that.

Prerequisites
#

I do not really have time nor want to delve too deep in how Gitlab-CI works in this article, so it is best to have some experience with it (there is nothing overly complicated).

Some background with Unreal Engine would be best, though I’ll get that out of the way now: I have NO knowledge whatsoever of UE4. I found the commands used to build the game in this Github repo. So, thanks to caiRanN!

As for the runner, I have opted for a Windows Server, as while Epic’s FAQ lists Linux in the supported OS to run the SDK on, the recommended specs do not. It also needs a LOT of Microsoft components (such as DirectX). While Microsoft and/or the community have made most of these readily available on Linux, I figured it would be more trouble than it’s worth.

The engine being quite the handful, you will need a decently powerful computer (you can find the details on the aforementioned recommended specs page).

I used an old gaming computer myself, with a fifth generation i5, 8GB of RAM, an HD7850 GPU (beware, AMD drivers are a bit of a pain to install on Windows Server) and a very old HDD. As far as personal experience go, I noticed the GPU was barely, if even, used in the build process. My CPU and disk were the most solicited, so you may want to capitalize on that more.

The first couple automated builds took up to about half an hour, but afterwards (once everything was cached I suppose) went down to three to five minutes.

Finally, you will obviously need a Gitlab account on the official website or on your own instance.

I will not go too deep into git best practices, but you really want to create a .gitignore file to avoid pushing heavy binaries to your repository (it happened to my friend, which pumped up his repository size into the GB, and good luck cleaning up using BFS). Github’s official UE4 gitignore template is a very good start.

Runner preparation
#

Download and install UE4 on your server (you will need an Epic Games account)
Download and install Visual Studio 2019 with the C++ components for Unreal and .NET Framework SDK 4.6.0+ for VS (both are built-in, just need to tick the boxes)
Get the Gitlab-Runner executable and register a new runner to your repository (tag it as ue4)
Add your UE4 installation’s path to your system envvars as UE4ENGINE (you can make sure it is correctly set by opening a powershell prompt and checking the content of $env:UE4ENGINE: it should print out something like C:\Program Files\Epic Games\UE_x.xx)
Optionally, download and install Python3 and 7-Zip to upload your releases

Repository variables
#

If you want to use the release uploading feature, you will need to set two variables in your repository settings:

MEGA_EMAIL: The Mega account’s email.
MEGA_PASSWORD: The password used to login to this account.

NOTE: you may want to mark both variables as protected, and you absolutely should mark the password as masked, or it may get printed in the logs at one time or the other.

.gitlab-ci.yml
#

Here is the .gitlab-ci.yml file we are using:

variables:
  GIT_STRATEGY: none        # we disable fetch, clone or checkout for every job
  GIT_CHECKOUT: "false"     # as we only want to checkout and fetch in the preperation stage
  GIT_DEPTH: "1"

stages:
  - preparations
  - compile
  - build
  - beta-upload
  - release-upload

preparations:
  stage: preparations
  variables:
    GIT_STRATEGY: fetch
    GIT_CHECKOUT: "true"
  tags:
    - ue4
  script:
    - 'Write-Host "Executing pre-build instructions..."'
    - 'Write-Host "UE4 Path on runner: ${env:UE4ENGINE}"'
    - 'Write-Host "Cache dir: ${env:CI_PROJECT_DIR}"'
    - 'Write-Host "Creating `"${env:CI_PROJECT_DIR}\Release\Game`" dir if it does not exists..."'
    - if (-not (Test-Path "${env:CI_PROJECT_DIR}\Release\Game")) {New-Item -Path "${env:CI_PROJECT_DIR}\Release" -Name "Game" -ItemType "directory"}
    - 'Start-Process -FilePath "${env:UE4ENGINE}\Engine\Binaries\DotNET\UnrealBuildTool.exe" -ArgumentList "-projectfiles -project=`"${env:CI_PROJECT_DIR}\Game.uproject`" -game -rocket -progress" -NoNewWindow -Wait'
    - 'Write-Host "Finished preparations."'

compile:
  stage: compile
  tags:
    - ue4
  script:
    - 'Write-Host "Compiling the game..."'
    - 'Write-Host "UE4 Path on runner: ${env:UE4ENGINE}"'
    - 'Write-Host "Cache dir: ${env:CI_PROJECT_DIR}"'
    - 'Start-Process -FilePath "${env:UE4ENGINE}\Engine\Binaries\DotNET\UnrealBuildTool.exe" -ArgumentList "Game Development Win64 -project=`"$env:CI_PROJECT_DIR\Game.uproject`" -rocket -editorrecompile -progress -noubtmakefiles -NoHotReloadFromIDE" -NoNewWindow -Wait'
    - 'Write-Host "Compilation done."'

build:
  stage: build
  tags:
    - ue4
  script:
    - 'Write-Host "Building the game..."'
    - 'Write-Host "UE4 Path on runner: ${env:UE4ENGINE}"'
    - 'Write-Host "Cache dir: ${env:CI_PROJECT_DIR}"'
    - 'Start-Process -FilePath "${env:UE4ENGINE}\Engine\Build\BatchFiles\RunUAT.bat" -ArgumentList "BuildCookRun -project=`"$env:CI_PROJECT_DIR\Game.uproject`" -noP4 -platform=Win64 -clientconfig=Shipping -cook -allmaps -build -stage -pak -archive -archivedirectory=`"$env:CI_PROJECT_DIR\Release\Game`"" -NoNewWindow -Wait'
    - 'Write-Host "Done building the game!"'

beta-upload:
  stage: beta-upload
  only:
    - /^.*-beta.*$/
  except:
    - branches
  tags:
    - ue4
  script:
    - 'Write-Host "Compressing the game..."'
    - 'Start-Process -FilePath "C:\Program Files\7-Zip\7z.exe" -ArgumentList "a -tzip -mm=lzma -mmt8 -mx=9 -r Game-$env:CI_COMMIT_REF_NAME.zip `"$env:CI_PROJECT_DIR\Release\Game`"" -NoNewWindow -Wait'
    - 'Write-Host "Done compressing, now uploading to Mega..."'
    - 'Start-Process -FilePath "pip" -ArgumentList "install mega.py" -NoNewWindow -Wait'
    - >
      Start-Process -FilePath "python" 
      -ArgumentList "-c
      `"from mega import Mega;
      import os;
      import logging;
      logging.basicConfig(level=logging.INFO);
      EMAIL = os.getenv('MEGA_EMAIL');
      PASSWORD = os.getenv('MEGA_PASSWORD');
      REFNAME = os.getenv('CI_COMMIT_REF_NAME');
      mega = Mega();
      m = mega.login(str(EMAIL), str(PASSWORD));
      file = m.upload(f'Game-{REFNAME}.zip');
      print(m.get_upload_link(file))`"" -NoNewWindow -Wait      

release-upload:
  stage: release-upload
  only:
    - /^.*-release$/
  except:
    - branches
  tags:
    - ue4
  script:
    - 'Write-Host "Compressing the game..."'
    - 'Start-Process -FilePath "C:\Program Files\7-Zip\7z.exe" -ArgumentList "a -tzip -mm=lzma -mmt8 -mx=9 -r Game-$env:CI_COMMIT_REF_NAME.zip `"$env:CI_PROJECT_DIR\Release\Game`"" -NoNewWindow -Wait'
    - 'Write-Host "Done compressing, now uploading to Mega..."'
    - 'Start-Process -FilePath "pip" -ArgumentList "install mega.py" -NoNewWindow -Wait'
    - >
      Start-Process -FilePath "python" 
      -ArgumentList "-c
      `"from mega import Mega;
      import os;
      import logging;
      logging.basicConfig(level=logging.INFO);
      EMAIL = os.getenv('MEGA_EMAIL');
      PASSWORD = os.getenv('MEGA_PASSWORD');
      REFNAME = os.getenv('CI_COMMIT_REF_NAME');
      mega = Mega();
      m = mega.login(str(EMAIL), str(PASSWORD));
      file = m.upload(f'Game-{REFNAME}.zip');
      print(m.get_upload_link(file))`"" -NoNewWindow -Wait

As you can see, I mostly used Powershell (and a bit of Python) to do everything.

You may also notice that the first thing set is a git policy that disables clones at each stages. This is because UE4 projects can get very heavy, and each jobs are supposed to be executed on the same runner, so there is no point in fetching the source every time.

I recommend setting your UE4 runner with its own tag, so your other eventual runners do not pick up the jobs.

Now for some explanations and tips on each stages:

preparations
#

First the git policy is overridden to allow the runner to clone the sources. At the same time, it will clean up the eventual leftovers of previous builds.

Then it creates a release directory for our build to sit in, and runs the Unreal build tool to prepare for the build (again, I am in no way knowledgeable on UE4 so I am not even sure this is useful).

compile
#

The Unreal build tool is used again, this time to compile the game, with the targeted platform being Win64. Not much else to say on this stage.

build
#

The game is built for good in this stage. You may change the clientconfig parameter from Shipping to Development if you prefer a dev build.

beta-upload (optional)
#

Automatic build publication is optional, and I’ve set it to run only on tags with the -beta suffix (e.g. v1.2.3-beta).

It requires installing Python and 7-Zip on the runner, and registering them in the PATH. It also uses Mega as host: I chose it for it’s large storage quota and fairly easy to use API. Plus there is a handy python library available.

Anyway, the game is archived in a .zip file, then uploaded to mega. Once done, the file’s share link is printed out in the job’s logs and can be used by anybody.

release-upload (optional)
#

This stage is nearly identical to beta-upload, but runs on tags with a -release suffix and will, in the future, trigger a webhook (or similar) on completion to post a message on Discord, with a link to the game and a complete changelog (which is just the tag’s Release content).

And?
#

… well you’re good to go. Commit & push your project to Gitlab along with the .gitlab-ci.yml file tailored to your needs, check your runner’s logs to make sure the build is going smoothly, and if you are satisfied with it, create a beta/release tag to get your game uploaded to Mega.

As I have said in the introduction, my objective is to also have a webhook and/or bot post the release to a Discord channel. Depending on the solution I pick, I will update this article or create a new one.

On the subject of bots, if your project depends on a single runner, you may be interested in Sprinter, a Discord bot I made in Python to monitor my UE4 runner. I will surely write something about it someday, once I hit version 1.0.

While it is private at the moment, I will update this post with an invite to my friend’s Discord server once it goes public, so you may see the result of this setup first-hand.

Prerequisites #

Runner preparation #

Repository variables #

.gitlab-ci.yml #

preparations #

compile #

build #

beta-upload (optional) #

release-upload (optional) #

And? #