RISC OS Build service

History

Introduction

JFPatch has a history that covers its many components. Each of the parts of the tool had its own history log at the top of the file, and the main application had a history that wrapped up the descriptions of each of the parts. Now, the service itself has another history. There's also the technologies that the service is built upon, which have separate histories covering about 18 months, which are not described here.

RISC OS Build service

Version Date Changes
121 2 Jun 2021 Changed the help menu's labels to be more helpful.

Coming off of the results of my survey, the help menu has been updated to hopefully make it easier to find what people may want.

The Help menu was referring to CI, and the RISC OS Build configuration, which wasn't very helpful. They now say explicitly 'Automation with git' and '.robuild.yaml format' to make it very clear what they talk about. And the documentation pages are prefixed by Docs to help that.

120 7 May 2021 Update the configuration to allow runtime configuration of input.

The input settings are relatively benign and can allow us to test some operations which would otherwise cause the system to terminate whilst processing user input. We may as well allow this, for the input.eof_effect setting.

The EOF default is now to cause a Reset, which should cause the system to exit.

115 9 Nov 2020 Update the site branding to build.riscos.online style.

The site is now switching over the build.riscos.online style, from the old JFPatch-as-a-service branding. The build tool is pretty much the same, but this is acknowledging that it's a generic build service.

The menu items have also moved around slightly to de-emphasise the JFPatch side of the service.

114 9 Nov 2020 Upgraded docker image to speed up builds; added config option to cli.py.

The build jobs were running very slowly. This was traced to the tracepoint support, which was triggered in OS_SynchroniseCodeAreas, which caused each call to take significantly longer. This has been made more efficient, and is now ~23,000x faster.

During debugging it was useful to add the extra configuration information to trigger the timings statistics. There is now a --config option to make it easier to test.

113 8 Nov 2020 Add download links for JFShared and JFPatch.

The JFPatch source has been updated so that the full application can now be built. It's really terrible, and honestly, the service is much better in general (and it's not great really!), but we may as well have a working binary release. I'm genuinely not sure how well it runs on RISC OS Classic, but I'm also not that fussed.

112 7 Nov 2020 Rework makefile builds to be able to return linkables.

Linkables are now able to be returned as artifacts in the automatically processed Makefiles in archives. There were some problems with the handling of the makefiles and we never actually returned the linkables that it had determined were present anyhow. This was work that had been deferred - it's now complete.

The examples have been expanded to include C application and C module code. The documentation has been expanded to give more explanation of the API and additional links.

111 7 Nov 2020 Fix for failure to build front images on linux.

The CI was failing to build images on Linux which uses ImageMagick 6; whilst it worked find on ImageMagick 7 on macOS. This appears to be a difference in how the large images are handled in 6 compared to 7, and the SVG to PNG conversion at high density appears to cause it to exceed the memory limits.

110 6 Nov 2020 Update service to be deployable through Ansible.

The service can now be deployed through Ansible, with configuration to pull the necessary docker container on service start, and with configurable overrides. This ensures that the system can theoretically keep up with the deployed versions from my docker repository. In reality, it probably won't need to be, because it's not expected to auto-scale.

109 6 Nov 2020 Update the docker configuration to push to the repository on build.

When we have built the docker image, if we're on master (and clean), we push the built image with the right name to the repository. This should allow it to be pulled down by the back end systems when they are started.

108 6 Nov 2020 Update the JFPatch-as-a-service examples repo with 32bit.

The submodule for JFPatch-as-a-service examples which we use to generate the examples when you click the button in the front end is now pointing at the examples that are able to be built 32bit.

107 5 Nov 2020 Added statistics page to the site.

The statistics that were on the resource site have been moved into here, as they directly relate to this service and we might as well. have them here.

105 4 Nov 2020 Tidied up the history page.

The history page had been made to look ugly by some misformatted commit messages. This should now look a lot better for the list of the variables and tools.

104 4 Nov 2020 Finish updating the CI documentation to give a guide on use.

The CI documentation describes how you might perform the build with the JSON API and the WebSockets API through robuild-client. The examples are given in unix shell.

CI wrappers are explained for the GitHub and GitLab CI systems. Jenkins use would depend on the type of job that was used, and the manner in which the author wishes to use it; it's easier to leave that to them - the shell scripting examples will be easy to transplant.

103 3 Nov 2020 Add initial documentation of the CI usage.

The beginnings of some documentation about how you can use the service in CI. At the moment this describes how you use the JSON API and the WS API through robuild-client within plain shell commands. There are placeholders for the GitHub and GitLab configuration, which will be documented later.

Shell colouring has been introduced so that we have a nicer to look at presentation of the script lines.

102 2 Nov 2020 Harmonise the stylesheet and code with the resource site.

The resource site provides a lot of information using the same basic style and libraries that were used in the JFPatch-as-a-service site. However, they have been extended and the style improved in some cases. We bring those changes back to this site, and add some more descriptions in some of the pages.

100 5 Oct 2020 Created diagrams for the front end interfaces and builder.

The front end now has diagrams generated for the external interfaces and the builder's structure so that we can describe the way that the service works in more detail for the ROUGOL talk. These are generated at the moment, but are not exported out to the main site.

99 6 Sep 2020 Update JFPatch tool to pull from Artifactory.

The JFPatch tool source is now pulled from the build version of canonical source, rather than being embedded in the service source. This should mean that we can update it independant of the service - keeping the two concerns quite separate!

The new JFPatch that's coming in should support 32bit module builds as well, which is also handy.

98 28 Jun 2020 Added support for ANSI Text mode.

The ANSI text mode is now supported, essentially just configuring whether the vdu.implementation confiuration setting is set to 'ansitext' or 'plain'.

97 28 Jun 2020 Add support for websocket options.

WebSocket connections can request options, and read the current options for the session. There are two features that I want to expose in this way - one is a global timeout, which is currently hard coded in the server, and the other is the ability to disable ansitext VDU implementation, reverting to just the plain text output, without any control codes.

The latter is not yet implemented, but the former is present in this change.

96 26 Jun 2020 Fix for the extension swapping duplicating directories.

The extension swapping code which was applying changes to files named *.c and *.h (for example) was accidentally appending the full path within the subdirectory. This meant that it only ever worked if the file was in the root of the zip archive. Oops.

95 26 Jun 2020 Fix for P2C header builds, permissions on Lib$Dir; add Alias command.

The P2C header file wasn't in the right place, so couldn't be found by the build.

Lib$Dir wasn't writable so libraries couldn't be written there, which was required by the riscos-build-online tool.

The *Alias command is now supported, to make it easier to mimic a RISC OS Classic system that has been booted.

94 17 Jun 2020 Reduced size of the robuild-service docker image.

The docker image was about 580MB. This has been reduced to about 276MB by removing the things we don't need and merging the layers. It's not a huge issue, but it will improve the deployment, and makes it a bit easier to manage.

93 17 Jun 2020 Update the RISC OS source processor to not touch buildables (optionally).

An internal option to not build the buildables files from the archive, has been added. The intention of touching those files was to force the rebuilding of the sources if the user uploads object files. Obviously if they sent the files to the build service, they want them to be built. But that might be wrong in the future, so there's an internal option to disable that behaviour. It's not exposed anywhere yet, but it's at least available for testing.

92 14 Jun 2020 Add extra headers and libraries; make cli timeout configurable for tests.

More headers and libraries are now included:

C:

  •  Standard C library (as before) with Stubs, StubsG and StubsGS (smaller).

  •  Fortify.

  •  Asm (Assembler functions for C)

  •  GetOpt.

  •  Toolbox libraries.

  •  RISC OSLib

  •  OSLib

  •  TCPIPLibs

  •  Interface headers for RISC OS.

Assembler:

  •  Global and interface headers.

Makefiles:

  •  My standard AMU makefiles for native and cross compilation.

Pascal:

  •  P2C support headers and libraries.

Variables are set:

  •  `Lib$Dir`: `$.Libs.C`

  •  `C$Path`: `<Lib$Dir>.CLib.,<Lib$Dir>.Interface.,<Lib$Dir>.`

  •  `Hdr$Path`: `$.Libs.Hdr.Global.,$.Libs.Hdr.Interface.`

  •  `RISCOSLib$Path`: `<Lib$Dir>.RISCOSLib.`

  •  `RISC_OSLib$Path`: `<Lib$Dir>.RISCOSLib.`

  •  `OSLib$Dir`: `<Lib$Dir>.OSLib`

  •  `OS$Path`: `<OSLib$Dir>.,<OSLib$Dir>.Core.,<OSLib$Dir>.Computer.,<OSLib$Dir>.User.,<OSLib$Dir>.Toolbox.`

  •  `TCPIPLibs$Path`: `<Lib$Dir>.TCPIPLibs.`

  •  `TBox$Path`: `<Lib$Dir>.TBoxLibs.`

  •  `Makefiles$Path`: `$.Libs.Makefiles.`

CLI now has a timeout of 10 minutes when running locally, which is the same as the service.

91 7 Jun 2020 Updated rozipinfo to handle sanitising unix paths.

If the archive contained paths like '../foo/bar' it would have been possible to escape the limited environment restrictions that were present in the service. Similarly, those names would not have been able to be used within the service itself. These are now checked for and stripped by the rozipinfo component.

Similarly, we weren't autovivifying directories, which is assumed by some (most?) archive tools.

90 7 Jun 2020 Disable the use of bold in the ANSIText output.

The bold text means that the output when rendered in the CI systems looks a bit odd - with it coming out brighter than the text surrounding it. Because the use of colours really isn't all that significant in the build system, I've disabled the bold usage.

This means that we will treat all the standard colours as if they have maximum brightness for the purposes of colour matching, and will never use the bold attribute. 24bit and 8bit colour sequences can still be generated, as they have not been disabled.

89 6 Jun 2020 Fix for Makefile builds failing to recognise makefile filename.

The Makefile filename was not being recognised unless it was in a subdirectory. We now recognise the Makefile when it is in the root of the archive.

88 6 Jun 2020 Make the runtime consistent between interfaces; add ping endpoint.

The maximum execution time has been pushed to 10 minutes, and made consistent between the JSON and WS interfaces.

There's an endpoint for 'ping' now, which just returns OK for the JSON service. This allows it to be used as a healthcheck, which is needed for load balancer use within the AWS ALB. The ALB is needed because the prior solution of using CloudFront was failing due to the maximum timeout of 60 seconds.

87 6 Jun 2020 Fix for multiple dependency rules not being appended properly.

When multiple dependency rules were present, as used by older versions of amu, the rules would fail to be appended properly - instead of the list accumulating the subsequent dependencies, it would nest them.

This caused exceptions because the system was not expecting that form of list - it should be a flat set of dependencies.

86 21 May 2020 Added .robuild.yaml file format documentation.

The .robuild.yaml file format is now documented, and an example given in the page. Some of the features of the build environment are documented.

85 21 May 2020 Update front end to recognise Zip archives, with a new icon.

The front end can now recognise a Zip archive being produced as the output, and has an SVG icon for the zip file, so that it looks nice. We have special case code that can turn the &a91 types into .zip with the correct media type. Although the files produced are in RISC OS format (ie, not NFS encoding), I would still expect the use to be for RISC OS, without any special handling, so using the RISC OS extra information is what users would expect, I think.

84 21 May 2020 Update the rozipinfo source with the latest version.

Import of the latest rozipinfo which handles a lot of extrafield and nfs encoding issues better.

83 20 May 2020 Add support for artifact definitions in .robuild.yaml files.

The artifact definitions are now honoured, exporting the results to a zip archive. This should retain the RISC OS filetypes from the NFS encoding on the filesystem.

82 19 May 2020 Moved robuild YAML parsing to its own class.

Rather than defining the YAML format inline, it is now in its own class, so that it can be isolated and (maybe) tested.

81 19 May 2020 Updated the module comments.

The module comments have been improved to better explain the interactions between different parts of the system.

80 19 May 2020 Initial support for the .robuild.yml build configuration.

If a build configuration file (.robuild.yml) is found, we'll process it and try to build using the definition given within the file. The file format is currently in flux, as I'm playing with it, but seemed to work with the LineEditor build - but without artifacts. They will need to be added separately.

79 19 May 2020 Replace the hardcoded RO Zip info parser with rozipinfo.

The rozipinfo module is a lot more complex and contains a lot more than we currently need for the reading of the Zip archives, but we should be able to use it for the creation of Zip archives for the outputs in the future.

Also fixed some zip file decoding problems when the Makefile does not supply any targets (in this case because it uses the includes to include the targets it expects to build).

78 18 May 2020 Fix for output accumulator not writing out properly on heartbeat.

The 'heartbeat' callback is meant to be invisible to the calling user, and causes output to flush to the stream if no further output to a newline has been seen in a period (usually a few seconds). The heartbeat was unfortunately being exposed as a 'heartbeat' action, with the output attached. That wasn't the intention - it should be written out as a plain 'output' action.

Updated the docker container name to match the container that we build.

75 4 Apr 2020 Improve rendering of the images; add a large icon.

The images weren't rendering well - the imagemagick conversion appears to be affected by whether a viewbox specification is given in the SVG, even though that viewbox is not used. These are now included, and this should make the icons appear better.

There is now a large patched icon generated. This was intended to be used for the repository card, but it's too much of a faff to get it generating in the right shape, so I've given up for now.

74 1 Apr 2020 Fix for failure to load the binary zip data properly in front end.

When we loaded the binary zip data in the front end it wasn't passing it through properly to the back end - it was ending up somehow mangled which meant that we couldn't use it. Now the data is passed through properly.

73 31 Mar 2020 Fix for failure to actually invoke the tool on zip archives.

When a zip archive is supplied we determine what the tool we should use from the filetypes of the files, and then we report the command line to use and execute it. Only, at some point, the list of buildable files in the zip archive changed from a tuple of name and filetype to a RISCOSName object that translates unix names to RISCOS names and filetypes (and vice-versa). But it was only changed in the single file code path.

This change ensures that we use the correct names, and introduces the example which performs the patching. You don't get an editor window when you've got a zip file (I had considered multiple editors, one for each file, but... complexity).

72 31 Mar 2020 Set the build timeout to 5 minutes.

5 minutes should be enough for anyone. Any longer than that and I'm sure that you're doing something wrong. Even DDEUtils, slow though it is to build, doesn't take that long.

71 30 Mar 2020 Update the JFPatch examples to all build, or fail interestingly.

The JFPatch examples now all build, except the SerialTCP which fails due to not finding the ESocket SWI. The EasySocket module is disabled on RISC OS because it might be a security issue. It at least demonstrates the throwback.

70 30 Mar 2020 Fix for syntax colouring of empty BASIC REM statements.

An empty REM statement was handled as a REM followed by anything (in two captures) being coloured as a comment followed by a comment. This appears to cause the syntax colourer to crash. Just making the REM and the following content a single capture appears to work.

69 30 Mar 2020 Fix for jfpatch syntax definition not colouring : separated fields.

JFPatch actually accepts '<field> : <value>' in the definition blocks, which I'd not supported in the syntax colouring. It's only used in some very old JFPatch source files, but it still shouldn't fail to colour properly.

68 30 Mar 2020 Add 'load example' button.

We now have a button to load files, in addition to the create new document and the uplaod document. That might be a confusing collection, so maybe they should be rationalised. However, for now this works, and I've not got time to be playing around with it.

We take the examples from the local submodule, so the state of the repository when the service was deployed is used. This could be reorganised in the future.

67 30 Mar 2020 Fix for returned binary being UTF-8 encoded.

UTF-8 encoding an ARM binary can only end badly.

66 30 Mar 2020 Add references to the supporting examples in github.

A github repository provides examples for how you access the service through a simple JSON request and the WebSocket interface. It also has some examples of JFPatch files. Eventually I'll make them selectable within the front end UI, but I'm getting pushed for time here, and I just need to make something work.

65 29 Mar 2020 Front end fixes to handle WebSocket disconnect better.

When we're disconnected this appears as a message in the output window, which helps to understand what's going on beyond the status that appears beside the help icon. The states where we've disconnected and have source which needs to be re-sent are now recognised and messages added to the output window. In the future this workflow should probably be updated to be more efficient - initially it was intended that we only transfer data when the user actually requests it, keeping them in control, but it seems that that's not a very useful flow.

It'd still be possible to repeatedly transfer lots of data if the connection kept going up and down at the moment but the user would have to actively send the data. Probably for the sizes we're talking about this is not especially relevant, and we could just take the hit of sending the data on reconnect.

64 29 Mar 2020 Added AWS deployment service and instructions.

The service can now be deployed on to an AWS instance, and has logging to track the service use.

Small fix for bug found during testing with a single line causing a failure to recognise the source code type.

63 28 Mar 2020 Added the structure diagram to the About page.

The structure diagram is now included in the About page, and the technologies used is expanded upon a little bit. It's kinda amusing that I've spent multiple days on getting the syntax colouring right, but haven't put the time into making JFPatch create 32bit modules yet. 'cos that bit is boring.

62 28 Mar 2020 Added icons to the parts of the diagram.

There are now little icons beside each of the technologies involved in the service, because they break the diagram up and make it look like I know what I'm doing.

61 28 Mar 2020 Create a structure diagram for how the service works.

I love diagrams. Pictures, they say, are worth a thousand words, and I don't like writing words, so pictures are cool. On the other hand, it's a lot easier to write words than to get Graphviz to do what you want.

Diagram shows the parts of the system, with the containers they live in. It can optionally show the JSON server, as well as the WS server, but it makes the diagram less clear, so I've turned the option off right now.

60 28 Mar 2020 Add help icon and guidance for how to use the service.

The service is relatively simple, to me, but others may not have much of a clue what the buttons do or the workflow. It's not as polished (by far) as Compiler Explorer, so people may need a bit more help. In the future, it might be nice to simplify the interface so that it doesn't follow the back end service interface, but right now it is what it is and we can help make it easier to use.

The help box toggles with the button, but also hides itself when you press one of the create or upload buttons.

This change also renames the 'Save' button to 'Send' because that better represents what it does, and I don't really want to confuse people with thinking that it's going to save to some storage.

59 26 Mar 2020 Add create document button.

Can now directly create a document and then start typing. This means that we've now got an extra button on the workflow bar, and that lets us create whatever source we want to.

55 24 Mar 2020 Pass the context colouring script through preprocessor

We now have context colouring for C files, Pascal files, and Perl files in addition to the JFPatch files. We have the ability to select the Basic Text and ObjAsm files for colouring, but currently I've not written the modes to do that colouring.

The colouring source is now built conditionally based on whether we have enabled the different supported modes. Strictly that's only for the front end. The back end is always able to support them.

The preprocessor I've used is one that I wrote for Doom, Heretic and Hexen, with additional features for the processing of the conditionals, which it turned out I hadn't implemented. I might as well have written a clean pre-processor rather than reusing that one, but hey-ho.

53 23 Mar 2020 Tidy up the pages to use paragraph elements in all sections.

Documentation should be inside the cells, or lists, or within a paragraph tag. These were not applied consistently through the pages, and have now been improved. Some of the FIXME marks have been cleaned up into a slightly tidier form too.

52 23 Mar 2020 Add ability to generate version information directly.

The versions information can now be directly generated from the git commit history. Some commit messages are elided as they don't offer any useful information, or are not relevant to the service.

The order of the history messages is now consistent - they all now go from newest to oldest.

51 23 Mar 2020 Clean up the history page.

The history page now omits the date on sections that we don't know, and has structured jfpatch sections so that the colouring is clearer for the properties.

CSS has been marked as UTF-8 so that it displays properly in Chrome.

50 23 Mar 2020 Add main JFPatch history.

There is now a history page that includes information about the componentst that make up JFPatch and how they were developed.

I've yet to fill in any history for this development version of the tool, and the version numbers appear to be in reverse order for some sections.

49 23 Mar 2020 Add the source code editor to the front end.

The front end is now able to show you the source code and let you edit it and then send it to the server for building. It's likely that the workflow isn't quite right for a real development environment, but... honestly, who cares, it's a freakin' build tool for RISC OS in a browser.

48 22 Mar 2020 Add automatic reconnection of the websocket.

The websocket now reports the service status into a little status icon which pops up when we're disconnected. If we fail to connect or there is an error, or we're disconnected, we now pop the icon up and set a timeout to reconnect in 15 seconds.

45 22 Mar 2020 Improve the documentation of the fileformat.

Fileformat now uses more of the colouring jfpatch sections.

44 22 Mar 2020 Begin integration of codemirror with the documentation.

The colouring has begin to be added to the file format documentation to make it possible to colour our examples.

43 22 Mar 2020 Simplify the colouring code for the example file.

Selecting the elements to colour using the class makes it easier to both style and name the elements we wish to provide examples for in the documentation.

42 21 Mar 2020 Add codemirror and JFPatch assembler mode.

The JFPatch assembler mode provides all the colouring for the tool, as it currently stands. I believe that most of the colouring is sane. It might not be the colours that everyone likes, but that's not necessarily the point. It's got the right bits for what we need - and includes ARM and BBC BASIC colouring within the sections that of the file as appropriate.

41 21 Mar 2020 Add missing documentation for some module features, macros.

The modules did not define a number of fields in the documentation. Many of the filters, the MessageFile, Resources and ImageFS/FS were not documented at all. These have now been added. Similarly the documentation has been updated to add in the Macros which were not documented at all (and are surprisingly non-obvious).

Some of the styling has been tidied up a little, and the SVG for file_xxx has been cleaned up so that it has the correct header.

40 17 Mar 2020 Add favicon for the site; refactor HTML HEAD element.

The favicon is now referenced from within the HTML HEAD element. The HEAD element is now handled in the main macros. This means that it's a bit more manageable.

39 16 Mar 2020 Clean up some more of the file format documentation.

I think most of the file format documentation now has some vaguely sensible styling present. It's still not complete, but it's definitely reaching the point at which it's now able to be used as a useful reference.

38 16 Mar 2020 Restructure to use semantic markup.

Might as well use semantic markup to make the documentation a bit more structured.

37 16 Mar 2020 Format the introduction section.

The introduction is now layed out properly, but doesn't have links.

36 15 Mar 2020 Checkpoint for the JFPatch file format documentation.

The JFPatch documentation is partially converted to HTML. There's a lot missing, and it could do with some examples littering through it, but it seems to be at least vaguely working as it stands.

This change only really covers the Format file; it doesn't cover the !Summary, which might contain more information to be included. Similarly the !Help itself may include information about the historic nature of the application that may fit into the about pages at some point.

35 15 Mar 2020 Add an 'about' page to describe what JFPatch was and how it works.

The About page talks about the service and about JFPatch.

33 15 Mar 2020 Add the site menu box.

The site menu lets you select where to go to, particularly it will take you to the API and file format documentation. Only the API documentation exists right now.

32 15 Mar 2020 Add example communications with the WebSocket server.

The WebSocket communications are not amazingly clear, so we can give some simple examples of the exchange.

31 15 Mar 2020 Added disclaimer footer, and added a 404 page.

The disclaimer is now present at the bottom of the page and has vaguely silly, but still pertinent messages. It uses the flex display to float to the end. That's cute.

The 404 page 'notfound.html' is now generated, and contains a simple message to go back to the root.

30 15 Mar 2020 Add introduction to API documentation.

A short introduction to give some general information on what the API can do.

29 15 Mar 2020 Move the macros out of the API file into macros.

Might as well put them in the right file.

28 15 Mar 2020 Added some protocol documentation.

Protocol documentation now exists, for anyone wishing to use the API to access the build service.

This uses a few of the HSC macros to control how we display the protocol definitions. These might be moved off into the macros file as they're probably a generally useful definition.

27 14 Mar 2020 Make some tidier workflow icons for the buttons.

The icons from the font that I had selected looked poor in Firefox. For simplicity, I've just replaced them with images. The layout has been tidied up a little to make the buttons more balanced.

26 14 Mar 2020 Replaced form buttons with icons for the user to press.

The buttons are pretty simple, taken from The Noun Project (and attributed). The icons for the filetypes are also shown when the filetype is recognised.

25 14 Mar 2020 Add building of the frontend with HSC.

We can now build the frontend content with HSC and a makefile. This may make it easier to put some parts of the site together. At the moment it's a bit of an overkill, but it'll help as I add more pages.

24 14 Mar 2020 Fix for CLI failing to report clipboard data properly.

After the functions were refactored, I didn't change the variable name.

23 14 Mar 2020 Fix the character set used by the frontend; add selection of wss:.

The front end's character set wasn't selecting UTF-8, so the build status was coming out in an ugly way. It's now set with a meta directive.

The use of ws: when connected by HTTPS is insecure and rejected by the browser. So I've made it select ws: or wss: depending on whether the HTTP or HTTPS was used.

22 14 Mar 2020 Split up the headings on the build and throwback windows.

The headings being implemented as ::before elements was kinda cute, but not as flexible as I'd like. For example, it precludes adding a close or 'iconise' type icon to the bar. This isn't a huge deal, but it makes redesigning the page harder.

21 13 Mar 2020 Add 'basic' HTML client that can communicate with the back end.

The 'basic' HTML client is actually a quite powerful WebSocket client which can pass the supplied source to the back end, display the messages from the build process, handle throwback and clipboard data and allow the user to download the output file afterward, if any.

It's surprising that it works quite so well!

20 13 Mar 2020 Report errors during the setup of the websocket build.

If the file you supplied wasn't recognised (or any other exception occurred during that setup), it would fail and never report any protocol errors back to you. We now report this as a failed build using the message, and the return code. We finally report that we are complete.

19 13 Mar 2020 Update the JFPatch.pyro configuration for ansitext.

As the output will be processed by an ANSIText processor, let's give it that as the main configuration. Maybe in the future that'll be an option to the API but right now, it's fixed.

18 13 Mar 2020 Moved the 'started build' message after the point we start the build.

We were reporting that we were starting the build, and then if it failed, we'd report that there was an error - that breaks the 'one response or error per action' rule'.

17 12 Mar 2020 Add documentation of the HTTP and WebSocket protocols.

The protocols are now documented, which means that they should be able to be made into webpages soon!

The command to start the build has been changed from 'go' to 'build'.

16 12 Mar 2020 Tidy up the websocket client and server, and JSON server for live use.

The live site is currently pointing at the test system, and with that, the configuration has been updated so that the tools and the commands that access the system are able to work that way.

The JSON server is now on a port alongside the WebSocket service. The responses no longer gives the arguments as lists for most of the values returned. This might be changed in the future, but it's simplest to just leave them as bare strings in most cases.

15 12 Mar 2020 Add WebSocket server and client.

We can now provide a websocket server which will allow the building of a source using the RISC OS Pyromaniac environment. This gives a set of operations that are updated live whilst the system is running.

14 12 Mar 2020 Update CI with allowance for empty resource files.
13 12 Mar 2020 Add ability to build the docker image in CI.

Might as well have CI to check that it's able to be built!

12 12 Mar 2020 Add support for the streamed output from the build.

Streamed output means that we get results as they happen through a callback. The results are streamed on to the same thread that did the request. Output lines are accumulated through a heartbeat, to ensure that we output data regularly and we don't supply single character callbacks for each letter (because that's how we tend to write things on RISC OS).

7 9 Mar 2020 Add option for debug to CLI; clean up better on failures.

The CLI tool can now specify a comma-separated list of debug flags for the pyromaniac system. This makes it easier to see what's going on when needed.

When we call the 'close' method on the builder, the rosource will clean up the temporary directory it created. This should make it less messy when something goes wrong.

5 8 Mar 2020 Add execution timeout; support for BASTXT files; tidier JSON and binary.

An execution timeout of 60 seconds is imposed by the 'timeout' tool. This should ensure that we never run away with code being left running.

Basic Text files are now included in the buildables, which means that you can submit a simple BASIC file and get out the text that it generated but not any binary, as there are no clipboard operations (unless the user adds them).

JSON output now comes out pretty printed if the data is small. The Binary output now works!

4 8 Mar 2020 Add a real service to return the content.

The service supports JSON and Binary outputs, and can report the results of the build in both cases when it fails (in text if the binary output was requested).

3 8 Mar 2020 Create a CLI tool to use the building functions.

The CLI tool is really just a testing tool to make sure that we can work with the back end system.

2 8 Mar 2020 Add a result collection class; rework all the building into a class.

The results are now in a single class, which is used with callbacks to accumulate the output so that we can write it out to a remote server if necessary. These functions are called by the docker and pyronativeserver objects.

The main code that was just in the body of the build.py is now in a class, and can be called in stages - this allows additional code to be inserted between the parts, such as to add debug or to debug issues.

1 8 Mar 2020 Initial import of the JFPatch as a service (RO Build service).

This has been developed over the last few days from the bare Pyromaniac build environment, and should now contain a base on which the service can actually be created. The JFPatch is just a snapshot. The Pyromaniac is the 'last docker version' so isn't ideal in its management, but this does mean less duplication.

In the future we'll pull in the different parts of the environment from the artifact server.

!JFPatch RISC OS application and back end

Version Changes
2.55ß Note: (partial releases were 2.54ß)
  • Multiple entries in 'Events' blocks now work - previously they did everything except claim the vector.
  • MemCpy routine added to AOF code.
  • EgCode added in case people want it.
  • Service call handler code isn't quite as intelligent as first thought. Produces things like :
    For &123456, which some might argue was slightly sub-optimal. Now slightly more optimal (move, and increased tap bits)
    which is kinda better.
    Also, instead of re-starting the subtractions afresh each time, the value will only be recalculated if it would generate more than two subtractions. This may not be amazingly optimal, but without a full search it will suffice. Test includes miscellaneous ridiculous service numbers to test the optimisation code.
    To facilitate these changes the services list is now sorted and each routine requiring a service must supply two pieces of code, one for when the user specifies a service handler (now decrecated), and one for the auto-generated form. Autogeneration is much more preferable.
  • Ursula style module service blocks added. As the current dispatch handler is the reject handler in JFPatch modules, it is not possible to seperate them. However it is intended that this situation be changed in the near future.
  • Added choice of relocation routine in AOF modules with
  • Added JB's cunning LDR/ADR :
    Note: These do NOT use LTORG as you may have used in other assemblers.
  • Multiple post filter reason codes can now be filtered off and combined with unknown reason code processors (which no longer worked in 2.53 :-( )
2.53ß partial releases were 2.52ß2
  • Filenames localised differently; most places that include filenames (eg In, Out, #Include, #Load) now use a generic 'localfile' routine meaning that any filename with no path information in it (ie no .'s or :'s) will be assumed to be in the source file directory; those with path information are assumed to be absolute, /except/ those starting with @ which are localised to the source file directory. It makes more sense when you use it than written down !
  • Added MessageFile fix (Johnathan Brady) - what did I write last time ? Was it complete cack or am I just imagining the code I saw ? :-)
  • Added Resources block (Johnathan Brady) :
    Note: 2nd parameter is the filename of the resource file, if you give this a different leaf name to the local file then it'll use that leaf name.
    Note 2: If you use a directory it'll include that directory as the resource file name and all of it's children.
    Local files follow the same regime as the In and Out directives described above.
    #Here Resources will include the resources at a particular location, or at the end of the code if not specified.
    JB's routines slightly optimised at basic sides.
  • Expanded list of people beta-testing to : Chris Johns, Phil Norman, Jonathan Brady, (Is matt interested?)
    Will other people wishing to test please contact me to be added to this list. Similarly, suggestions for people who wouldn't mind would be good. Testing means that you're happy to put up with more bugs than usual in JFPatch and suggest both ideas and possible fixes. Not all will be implemented but that's true of anything.
  • Workspace checks now include `len_wsname (Johnathan Brady), but I don't see much of a use for this except in the context of ADRW when creating an FD stack. These routines will soon be moved elsewhere to cope with CAS/RAS.
  • FNshowreg improved to use ConvertInteger instead of adding a - and calling ConvertCardinal; this saves all of three instructions per REM "%r#" call (!).
  • Output buffering doubled to 16k (text) and 4k (debug).
  • 'Flags Token' now added to definition of commands.
  • In Post-Filter code, the 'Code' entry may now be of two forms :
    The standard form, with usage as before, and also
    The extended form with all the checking of the conditions done for you. Mask/Accept need not be specified, but if it is /must/ preceed the Code entries. Multiple entries of the form /are/ allowed and will be cumulative. Use of the standard form and the extended form are allowed, but you will need to use 'Mask value' or 'Accept reason' if you wish to use these as otherwise you will never actually see the reasons. Similarly, the standard form will never see those codes catered for by the extended codes.
  • Tiny optimisation means that Modules with the same name as their SWI prefix will only include one instance of the name.
  • ImageFS handling blocks added :
    Only 'Type' is required, but without the rest it won't do much. 'Flags' may be given many times, and may be preceeded by - to negate.
    Flag bit to set (see manuals)
    TellWhenFlushing 27
    This has not been tested hardly; use at your own risk :-)
  • Normal filing system block added :
    'Name', 'Number' and 'Files' are required. Startup text of - will use Func 17 to display FS name.
    'Flags' may be given many times, and may be preceeded by - to negate. Flag bit to set (see manuals)
    SpecialFields 31
    InteractiveStreams 30
    NullFilenames 29
    AlwaysOpenFiles 28
    TellWhenFlushing 27
    SupportsFile9 26
    SupportsFunc20 25
    SupportsFunc18 24
    SupportsImageFS 23
    UseURDLib 22
    NoDirectories / NoDirs 21
    NeverLoad / UseOpenGetClose 20
    NeverSave / UseOpenPutClose 19
    UseFunc9 18
    ReadOnly 16
    SupportsFile34 ext 0
    SupportsCat ext 1
    SupportsEx ext 2
    This has not been tested hardly; use at your own risk :-)
    FS names has been optimised to use the module name if this is the same; similarly the startup text.
  • Fixed bug in module help worker-outer to append more tabs if required. The
  • code's been there since /really/ early versions but has only ever fired on single character module names because it was severely broken.
2.52ß
  • Changed the function of the ERR macro. This used to embed an 'OS_GenerateError' block into some code. Now it embeds an error block; syntax is :
  • Added REMF and REMFP variants of the REM macro; these /will/ preserve the
  • flags and the link register...
2.51ß
  • Output buffering added. This isn't at all impressive, but seemed like an
  • interesting and useful optimisation to have. Maybe it won't thrash the drive on Chris' machine, or maybe it will - I'll have to see.
  • Output buffering on debug data added. As above this appears to make very
  • little difference to the performance, but as I'm timing things using the seconds numbers on !Alarm I reserve the right to be wrong. For those of interest, the buffers are 8k for the main BasTxt file and 2k for the debug data. Since BasTxt files tend to be rather large I may up this some time soon.
  • Bug fixes in 2.50ß's dictionary encoding code, and optimised encoding's now
  • mean that entries including the token 0 will not fail to be compressed to their minimal form.
  • Release to Chris Johns and Phil Norman. I don't actually know who else uses
  • it... I wish people would contact me - if only to report bugs !
2.50ß
  • printf improved to include %d for cmj
  • memcpy added (suboptimal but works!) (currently only on cmj's machine)
  • Pre/PostFilter code handlers in AOF mode didn't work previously - now fixed
  • WimpSWIs didn't work since about 2.46ß where I tried optimising the code; both WimpSWIs and Filter code should be vaguely optimal
  • SWIs block now allows for 'Pre' and 'Post' handler code - this should allow you to do things before we check the SWI number and after we return from the SWI call
  • Extra info string has been added (for Module block) to allow you to describe a module more fully (eg, ARM3 variant, etc) on the help line.
  • Syntax strings will now have the OS dictionary for RO3.1 substituted into them
  • Looks like strings can have embedded variable expressions in them (I've no idea when I added this); things like Help blocks can have {expr} to indicate that the expr can be embedded - eg, This was compiled {TIME$} Just goes to show what you can learn when you read your own programs!
2.49ß
  • Added Phil Norman's divide routine to the 'Libraries' directory (at last!), as well as another Divide routine I got from 'jonboy' on IRC.
  • Partial messages file support added ('jonboy')
  • InsBranch library added; similar to Patch, but useful when you don't care what you've patched, or need to patch a branch table, etc... ('jonboy')
  • Strings library updated to use 'nice-upper' code extracted from RiscOS :-)
  • SWI out of range now uses MessageTrans rather than 'knowing' the error ('jonboy')
  • Added 'printf' library for making debugging in C easier (yay!)
  • Fixed and optimised a few string routines.
2.48ß
  • 'Type AOFModule' is now depreciated; you should use 'Type AOF Module' instead; new syntax is 'Type AOF [subtype [subtype]...]' where 'subtype' may be Module or Debug. The current implementation of Debug, whilst looking correct seems to crash DDT; I recommend that you avoid the debug option for the time being.
  • Released to Chris Johns, Phil Norman and 'jonboy' (who I've forgotten the name of!)
2.47ß
  • REM storage space moved onto stack rather than inside code. This should allow you to /really/ say READONLY in AOF with REM's in them.
  • WARNING: Do NOT use WimpSWI Post trap code prior to this version; pretrap code will be called instead !!!!
  • In AOF, any exported label followed by the word ENTRY will be made the execution entry point for this file.
2.46ß
  • Minor bugs in AOF handling fixed
  • Fixed bug with $$ - this now translates to $ correctly if macros are not in use
2.45ß
  • Arrgghh... fixed nasty bug in the WimpSWIve claim routines ;-( This should stop the table being included twice and r0 being corrupted randomly in the init code !
  • Don't even think about using |'s around WimpSWIve stuff - it doesn't work, I've got to work out how to get the right addresses in there - atm it's using offsets for normal variables, and addresses for |'s - I need a consistant interface internally - I don't want conversions on the fly if I can help it ;-(
  • CAS and RAS macros added. Don't expect anything special from them though.
  • I think CodePrefix actually works now... Need to check this really though.
  • TaskWindow error handling improved.
  • Includes now work correctly (inline, rather than at end)
2.44ß
  • Multiple areas now supported correctly.
  • AOFModule now allows |label| style labels for the most part - if there's one
  • I've missed please tell me :-)
  • Relocation of symbols where they are both exported and local now works if the
  • first instance was a reference and not a definition.
  • Filters /may/ not work on AOFModules. This is untested.
  • WimpSWIs /do/ work with AOFModules.
  • EQUD |routine| will store the /absolute/ address, not the relative one - ideas as to how to fiddle this are greatfully appreciated :-)
  • strdup added to j.memory.
  • astrcmp added - this is an assembler style string compare - it returns EQ,
  • LT, GT, etc rather than -1,0,+1 as in C. There is no C header for this.
  • Events, Services, Vectors, init, final & service have not been tested for use with AOF functions.
  • Some re-organisation of the internals of the filters code means that it is no longer restricted to the 64k previously available - this takes one extra word and I'm not happy with it - I'd rather the use of that style of function was restricted to AOFModule only, then standard Module types could have the luxury of an extra instruction with the knowledge that they cannot exceed 64k of code (!)
Notes:
  • Remember: strcpy copies from r1 to r0 NOT r0 to r1.
  • Filters using Accept are unstable - be careful to check explicitly for the reason you want. Unfortunately ToolBox passes ridiculously big reasons down to the application - it's difficult to mask reason 17 million if you've only got a 32 bit word to use :-(
  • Am considering making the jfplib functions into a seperate area each - the overheads will be minimal and non-existant when linked, but only the required routines will be linked and link can ditch those we don't want.
2.43ß
  • Added > macro command to embed function name before routine
  • #CodePrefix bool will modify this variable.
  • bool is now allowed to be =file which evaluates to TRUE if the file exists but is empty, and the boolean value of the contents of that file otherwise. The latter form is prefered.
  • Matthew Godbolt's main routine included in the jfplib library, SkipWhitespace and SkipNonWhitespace added to string.j, puts (writes a string), putnl (new line), updated headers.
2.41ß
  • Fixed bug in Event handler code (DON'T use workspace in events prior to this version :-( )
  • Added LO and HS to the list of conditionals accepted.
2.40ß Changes in 2.40ß over 2.33
  • Macros are slightly more stable - still not brilliant though.
  • AOF compilation now possible - again, not brilliant, but it works !
  • REM's now optimised; should cut quite a size off debugging code.
  • Minor modifications to allow tabs - not complete but getting there.
  • Error handling slightly improved - now allows errors returned using ABEX.

JFPatch back end

Version Date Changes
2.57 31 Aug 2020 Added -apcs command line option
2.56 02 Mar 2020 Added ClipboardHolder operation
2.55 09 Apr 1999 JB's modifications added
2.54 07 Apr 1998 Service entry fixed, Code-in fixed now. Other things.
2.53 26 Dec 1997 Fixed bugs in REM, Resources, Workspace changes, PostFilter improvements, optimisations for modules, ImageFS and FS blocks, Module help improved.
2.52 05 Dec 1997 ERR changed and REMF[P] added
2.51 18 Nov 1997 Output buffering added
2.50 15 Nov 1997 AOF module filter/wimpswis fix
2.49 22 Oct 1997 Many module changes
2.48 26 Sep 1997 AOF Debug support
2.47 09 Sep 1997 REM's now use stack as ws
2.46 11 Aug 1997 $$ now translates to $
2.45 29 May 1997 Includes work inline
2.44 28 May 1997 AOFModule header allows AOF
2.43 16 May 1997 > is a function, new bool
2.42 29 Apr 1997 > macro for code prefixes
2.41 25 Apr 1997 Cond Set fixed
2.40 09 Apr 1997 XBL/XSWI supports apcs
2.39 09 Apr 1997 Pre support (constants)
2.38 08 Apr 1997 AOF support
2.37 08 Apr 1997 ;'s for comments in Pre/Post
2.36 09 Mar 1997 FNmess improved
2.35 08 Mar 1997 version$ added to code
2.34 06 Mar 1997 PROCGetRegs 'r' bug fixed !
2.33 05 Feb 1997 Added E REM message
2.32 22 Jan 1997 ^ and # in XSWI supported
2.31 19 Jan 1997 fixed REM &, added OS_NewLine!
2.30 21 Dec 1996 Directory structure changed
2.29 14 Nov 1996 Modified Conditionals for 3.1
2.28 30 Oct 1996 Added export of locals
2.27 25 Jul 1996 Added hourglass option
2.26 28 May 1996 Added macro support
2.25 22 May 1996 Fixed no DDEUtils bug
2.24 22 May 1996 Fixed naff handling of TB
2.23 09 May 1996 Damned tiny bug fixed !
2.22 15 Apr 1996 Throwback support added
2.21 14 Apr 1996 Compile_A removed to allow vague Make support
2.20 19 Feb 1996 includes and Compile_A added
2.19 28 Jan 1996 Tabs in source file added
2.18 29 Dec 1995 SWAP instruction added
2.17 08 Sep 1995 -ve LMOVs implemented
2.16 05 Sep 1995 Library file reorganisation
2.15 05 Sep 1995 XLDMFD added for errors
2.14 02 Sep 1995 CAPTURE post assembly added
2.13 30 Aug 1995 EXAMINE post assembly added
2.12 20 Aug 1995 MODE instruction added
2.11 18 Aug 1995 LADD instruction added
2.10 18 Aug 1995 ; comments remove :'s
2.09 10 Aug 1995 NOP instructiXon added
2.08 09 Aug 1995 Flag setting added
2.07 20 Jul 1995 XSWI and XBL added
2.06 22 Jun 1995 VDUStream forcing
2.05 11 Jun 1995 Workspace added; Long MOV; Local labels
2.04 06 May 1995 bug fix for stamping
2.03 03 May 1995 internal version numbering
2.02 18 Mar 1995 application making added
2.00 25 Feb 1995 no application making

Module generation

Version Date Changes
2.34 30 Aug 2020 Added 32bit support
2.33 12 Aug 2020 Added (restored?) Date support
2.32 24 May 2001 Added support for 'duplicate service handlers' which may help when using auto-added handlers
2.31 24 May 2001 Service handler code completely re-written to cope correctly with high-service numbers. Totally removed code for encoding using dictionary
2.30 16 Mar 2001 WimpSWIve registration now uses multiple instructions (BASIC problem ?)
2.29 27 Feb 2001 Can explicitly set version with module_version$ (and date with module_date$)
2.28 13 Feb 2001 Added 'lightning fast service' code
2.27 13 Feb 2001 Added fast service reject code
2.26 09 Jun 2000 Removed the dictionary encoding as this changed between OS versions.
2.25 05 Sep 1998 Some ADRs changed to LADRs in initialisation code (JB)
2.24 14 Jul 1998 Rect, PostRect and PostIcon filters added.
2.23 27 Apr 1998 WimpSWIve AOF code fixed, new filter system AOF code fixed
2.22 23 Apr 1998 Wimp poll reason codes moved to file, new filter system, aof startcode, init, final, service and swihandler entries fixed.
2.21 01 Apr 1998 Module initialisation error handling improved
2.20 31 Mar 1998 Ursula style service table
2.19 30 Mar 1998 Service handler code improved
2.18 16 Dec 1997 ImageFS support added
2.17 15 Dec 1997 Filter 'Code' now allows reasons
2.16 14 Dec 1997 Messages file fix, resource file blocks
2.15 15 Nov 1997 Dictionary tokenisation added
2.14 15 Nov 1997 SWI Pre/Post, bug fixes for WimpSWI/Filter
2.13 22 Oct 1997 Messages files
2.12 09 Sep 1997 WimpSWIve post trap code fixed
2.11 28 May 1997 AOF imports for header added
2.10 26 Apr 1997 Event code was completely buggered
2.09 08 Apr 1997 AOF compliant code added
2.08 09 Mar 1997 Extra info added
2.07 05 Feb 1997 Events and Vectors added
2.06 21 Dec 1996 Services moved into Resources
2.05 05 Nov 1996 Services added
2.04 26 Aug 1995
2.03 08 Aug 1995
2.02 22 Jun 1995
2.01 20 Jun 1995
2.00 11 Jun 1995

Workspace

Version Date Changes
1.04 14 Dec 1997 `len_name in ADRW works
1.03 30 Oct 1996 Fix for MapWS returning default
1.02 13 Jan 1996 | Union loop id added
1.01 18 Aug 1995

AOF generation

Version Date Changes
1.09 02 Feb 2003 Added specifier for 32Bit areas
1.08 06 Nov 1998 LDR, conditional ADR (JB)
1.07 15 Apr 1998 ADR (JB)
1.07 22 Jul 1998 area data now works
1.06 26 Sep 1997 debug data
1.05 26 Sep 1997 entry points
1.04 28 May 1997 functions to do jumps/offsets
1.03 16 May 1997 multiple areas work
1.02 09 Apr 1997 EQUD works, Pre added
1.01 09 Apr 1997 fixed module code and swstk
1.00 08 Apr 1996 started, simple aof support

Macros

Version Date Changes
1.02 08 Jan 1998 Constants (JB)
1.01 28 May 1996 Embedded macros (ie one calls another)
1.00 28 May 1996 Base macros