RISC OS Build service

CI Configuration

Introduction

RISC OS Build service can be used with Continuous Integration ('CI') systems to build RISC OS components automatically. The API documentation describes the API that is used by the service. The Build configuration describes configuration files that can be used to describe the RISC OS build process.

This document describes how these can be used together with CI systems. The CI systems that are discussed are:

These build tools can be used with the JSON API,and examples will be given for curl and jq. However, the service is easier to access with the 'robuild-client' tool. Both these methods will be described.

Assumptions

It is assumed that:

JSON API with curl

Simple build and status

The simplest example of submitting a file my-source-file to the service with curl would be a command like:

The result is written to the file /tmp/output. However this assumes that the build was successful - if it is unsuccessful the result will be a 400 response with the output written to the body. The 400 code is silently ignored by the response here.

It is, however, possible to capture the status code with a little a capture and assignment:

One caveat for submitting single files which you should be aware of is that the -F switch to specify the file to submit does not support passing filenames with commas in. This means that if you are submitting a filename encoded with the RISC OS filetype format you will need to copy the file before hand.

Extracting more information

The above examples suffice for simple submissions to the service, but you may wish to do more than this. The JSON interface allows you to extract more information about the response from the service.

The 'jq' tool can be used to extract information from the output JSON file. For example to extract the messages you might use:

The resulting data, if any, is returned in the 'data' key, and the RISC OS filetype in the 'filetype' as an integer. The returned data is Base64 encoded, so will need to be decoded before it can be used.

Of course, it is common to want to stop when the build fails. This is simplest if we just check the return code for non-0, and exit the build process at that point.

Handling multiple files

Whilst it's nice to be able to work with a single file, that's commonly not how projects are structured. Commonly there are multiple files that make up the project, and that's where the zip archives and '.robuild.yaml' build configuration becomes useful. If you have a repository containing files, together with a '.robuild.yaml' file, these can be transferred to the server as a Zip archive.

Bringing it all together

To script this in a generic way, I put together the fragments in a way that makes it easy to reuse in different cases. This could be in your own scripts or in a build system like Jenkins. The basic submission and data collection script I have used is:

This produces a set of files, in a temporary directory, and a couple of environment variables which can be used to decide what to do with the build.

To decide what to do you might print out the output messages, and report the failures if any:

'robuild-client' tool

The 'robuild-client' tool is intended to make it easier to drive the build service through the websockets interface by removing the need for manually manipulating the JSON files and extracting content, and to give a more interactive environment to see the output as it happens. The tool works on Linux, macOS and RISC OS. It should be trivial to port to Windows, but this has not been attempted as yet.

Unlike the manual method above, we must first obtain the build client tool, before we can submit the files to the service. Whilst the tool could be installed in your environment, when used from CI, it's commonly easier to just download it as needed.

The result will be written to the file '/tmp/build,XXX'. The job has been given a timeout of 60 seconds. This can be useful if you're concerned that it might get into an infinite loop. There is a timeout on the service, but this is system controlled, so it is better to specify a timeout to what you feel is appropriate. The system output and the build output will be written to the terminal, which means that it is not ncessary to do any more parsing of files.

Although this requires a download of the tool on each invocation, it's simpler and more maintainable in a CI environment than the shell code which did similar operations. Consult the 'robuild-client' repository for more details on the tool.

GitHub Workflows

GitHub can trigger builds when changes are pushed to branches or tags. The definition of what build is triggered is called a 'workflow', and the workflows can trigger multiple actions. The workflows are described in YAML, which is described in the GitHub documentation.

Basic workflow

The YAML file is held in the files '.github/workflows/NAME.yml'. There can be multiple independant workflows which will be run in parallel, differentiated by the filename 'NAME'. The YAML content looks like this:

Artifacts and version numbering

This configuration can be combined with the scripting used in the earlier examples to give a build of the RISC OS component. To be useful as a tool for building binaries that others may use, though, it is necessary to archive the output into an 'artifact'. In building an distributable component, it is common to want to distribute the results with a version number or other differentiating feature.

Version numbering can be performed in a number of ways, but it is common to have a 'VersionNum' file which contains '#define' statements to declare the version number. This is the way that the RISC OS source manages its versions. However, if no version is handled this way, it may be more useful to just use the commit identifier (aka 'Git SHA'). These operations can be performed with a YAML file that looks like this:

Releases

Artifacts stored by GitHub are only available for a short period. In order to be retained so that they can be used for distribution, it is necessary to create a 'Release' in GitHub. This can be achieved at the end of the build process. Commonly, releases are only created when a block of work has been completed and the author is happy with the results. This can be achieved by using git 'tags' to indicate that a release is required. Using any tag prefixed by a 'v' is a common way to achieve this.

Once released, it is necessary for the author to confirm the release (because it was a 'draft'). This allows the author a chance to test that what was built is actually suitable for distribution. Releases do not expire by default and will remain available to the public.

The CObey component gives an example of how you might build components and releases automatically. It is not required to follow the pattern given here - there are other ways to achieve this automation which you can use as you wish.

GitLab CI

GitLab CI is able to build on commits in much the same way as the GitHub actions. It is a very powerful system, but doesn't have the same 'action' infrastructure as the GitHub. Consequently, a simpler example is supplied here which shows how to use CI with GitLab. There is fuller documentation on the GitLab documentation site.

Basic structure

As with the RISC OS Build service build configuration, and GitHub workflow, the configuration for GitLab CI is YAML. GitLab CI organises the builds into stages, which may contain multiple builds. The builds will happen in parallel, unless a dependency is introduced. Each build may contain multiple scripted steps, which are run in the shell on the target system.

The basic structure of the GitLab CI file is thus: