Add: [Actions] release workflow for both releases and nightlies #8371

TrueBrain · 2020-12-12T15:45:43Z

A nightly run looks something like this:
https://github.com/TrueBrain/OpenTTD/actions/runs/420435699

A release run something like this:
https://github.com/TrueBrain/OpenTTD/actions/runs/420414947
(I tagged a fictional version in my own fork; you can ignore that completely)

Targets currently not included

ARM version of Mac OS. This is basically done and ready to ship, but we will wait till after this PR.
ARM version of Windows. This requires a few more patches before it compiles successful. The rest of the workflow is prepared for this.
Emscripten. This requires review of a few PRs, but otherwise this workflow is prepared to accept that too.
AppImage. There is a PR to introduces most basics of AppImage to the repo; it needs integration with the release workflow in such way we minimize diskspace and still have a workable image.

Choices made

NSIS is only running for releases, not for nightlies.
.deb builds are only made for releases, not for nightlies.
Universal MacOS builds will be a PR further down the line, if it turns out we want them / our user-base wants them.
We might attach our binaries to GitHub Releases for releases, but not right now. This will be an additional PR.

glx22 · 2020-12-12T17:59:13Z

I'll look at OPTION_TOOLS_ONLY

LordAro · 2020-12-13T22:33:04Z

My particular answers for To Consider:

Do we need -dbg.deb?
Yes - matches having PDB on Windows, and good for consistency
Do we want Debian builds? (we now only do Ubuntu)
Yes
Do we want NSIS runs for every nightly, or only for releases (like it is now).
IMO no
Do we want .deb builds for every nightly, or only for releases (like it is now).
IMO no
Do we want Mac OS universal builds?
Yes
Should we attach the binaries to GitHub Releases too? And if so, including the checksums? And if so, all three (MD5, SHA-1, SHA-256), or only SHA-256?
IMO no to all. No point mixing where people get downloads from.

orudge · 2020-12-13T22:34:08Z

I'll cover the macOS ARM builds and Universal builds, but will likely wait for this PR to be merged first.

michicc · 2020-12-13T22:41:32Z

Attaching binaries to GitHub for releases is probably useful as there will be people that look for them on the GitHub Release tab first. I don't think having nightlies here as well makes sense, will just clutter the list way too much.

TrueBrain · 2020-12-13T22:42:59Z

Attaching binaries to GitHub for releases is probably useful as there will be people that look for them on the GitHub Release tab first. I don't think having nightlies here as well makes sense, will just clutter the list way too much.

I tempt to agree with you. And yes, I meant for releases only :D Oops .. should have been more clear :P For nightlies it is not even possible.
But for now I will not implement it; we can add it if we like it in a later PR :)

Look at me, scoping the PR :D W00p!

TrueBrain · 2020-12-14T00:16:18Z

Unsure if .github/changelog.sh is a good location for that script ... but I also didn't find a better one. Suggestions are welcome!

TrueBrain · 2020-12-14T08:54:03Z

Fixed .deb dependencies, and postponed -dbg.deb (as it seems CPack doesn't support this currently for applications; only for "components"). So this is now ready for review!

.github/workflows/release.yml

LordAro

More questions than problems

LordAro · 2020-12-17T21:23:36Z

.github/workflows/release.yml

+  workflow_dispatch:
+    inputs:
+      ref:
+        description: 'Ref to build (for Pull Requests, use refs/pull/NNN/head)'
+        required: true
+  repository_dispatch:
+    # client_payload should be the same as the inputs for workflow_dispatch.
+    types:
+    - Build*


I don't quite understand what these are for, can you explain?
Relatedly, why are the nightlies not triggered with a cron, e.g. https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#onschedule ?

Like I don't know about schedule .. pfft :P See OpenTTD/workflows#10 :)

workflow_dispatch means we have a new button in the GitHub interface that allows us to make a binary release of anything, anytime. This makes rebuilding easier, but also building PRs possible.

repository_dispatch is similar, but for automated triggers. A while ago we decided to move all those workflows into https://github.com/OpenTTD/workflows . This has mostly to do with forks. If you fork this repo, these days by default schedules are disabled; but this was not always the case. Additionally, if you enable it once for what-ever reason, it remains on for 90 days. So that is pretty noisy. So we decided to move all the schedule stuff to https://github.com/OpenTTD/workflows, as it is very unlikely anyone would fork that.

Additionally, the schedule code is not straightforward. It does not support timezones, but also it cannot check if there was anything new. That complexity is now all in OpenTTD/workflows#10, as it really has nothing to do with OpenTTD.

In https://github.com/OpenTTD/workflows you will find more lovely glue-code we use for our infra :) It also triggers eints commits, publishes the docs, rebuilds the website, etc :)

LordAro · 2020-12-17T21:24:49Z

.github/workflows/release.yml

+    - name: Checkout (Release)
+      if: github.event_name == 'release'
+      uses: actions/checkout@v2
+      with:
+        # We generate a changelog; for this we need the full git log.
+        fetch-depth: 0
+
+    - name: Checkout (Manual)
+      if: github.event_name == 'workflow_dispatch'
+      uses: actions/checkout@v2
+      with:
+        ref: ${{ github.event.inputs.ref }}
+        # We generate a changelog; for this we need the full git log.
+        fetch-depth: 0
+
+    - name: Checkout (Trigger)
+      if: github.event_name == 'repository_dispatch'
+      uses: actions/checkout@v2
+      with:
+        ref: ${{ github.event.client_payload.ref }}
+        # We generate a changelog; for this we need the full git log.
+        fetch-depth: 0


Would it be possible to set a 'ref' variable to use rather than duplicating this 3 times?

Yes, kind of. But it would require a bash rule with the if logic in there to deduce which ref to use. The downside of that is, it is really easy to fuck that up. This now is trivially correct, and is why I prefer it. It does require a step extra .. but as you can see in a step later where we figure out the branch-name .. that code is not pretty, and really easy to mess up :) (which I also did, as it turns out :P)

So if you don't mind, I prefer this way. If you do mind, I can rewrite it .. but no guarantees :D

(I hope GitHub comes with some updates soon that I can also remove the other bash blob, and integrate it into these blobs .. but the actions/checkout needs some changes for that :D).

.github/workflows/release.yml

Co-authored-by: Owen Rudge <owen@owenrudge.net>

This in preparation for other architectures, like arm64.

This has several ways of being triggered: - When creating a new release via the GitHub interface. Fully automated that will produce new binaries, upload them, and it will even update the website to tell about the new version. - When triggered in an automated way from OpenTTD/workflows to start a nightly. - Manually via the Release workflow, which accepts branches, Pull Requests and tags to build. Rerunning a job is safe and should be without issues. Everything retriggers and updates what-ever might have been broken. In fact, except for dates, it should produce identical results. Co-authored-by: Charles Pigott <charlespigott@googlemail.com>

Azure Pipelines has build our releases for two years now, but we are finally switching to GitHub Actions. This to bring the full workflow to a single place, making it easier for people to see what is going on and how to influence the process.

LordAro

About time we had working nightlies again

glx22

Ok for me too.

TrueBrain force-pushed the release-workflow branch 2 times, most recently from 628a813 to e126cd9 Compare December 13, 2020 21:51

TrueBrain force-pushed the release-workflow branch 4 times, most recently from fc8b750 to 7bafbb6 Compare December 14, 2020 00:15

TrueBrain force-pushed the release-workflow branch from 7bafbb6 to fee7948 Compare December 14, 2020 08:52

TrueBrain marked this pull request as ready for review December 14, 2020 08:53

TrueBrain force-pushed the release-workflow branch 2 times, most recently from 7169bfe to a7977fd Compare December 14, 2020 09:45

TrueBrain added candidate: yes size: large size: small and removed size: large labels Dec 14, 2020

TrueBrain force-pushed the release-workflow branch from a7977fd to b0879c7 Compare December 15, 2020 14:29

glx22 reviewed Dec 17, 2020

View reviewed changes

.github/workflows/release.yml Show resolved Hide resolved

LordAro reviewed Dec 17, 2020

View reviewed changes

TrueBrain and others added 6 commits December 18, 2020 14:36

Fix: [bundle] custom launch.sh is no longer required for Mac OS

c9f7243

Co-authored-by: Owen Rudge <owen@owenrudge.net>

Fix: [bundle] postfix the architecture for Mac OS bundles

408ab82

This in preparation for other architectures, like arm64.

Fix: [bundle] postfix the distribution for Linux bundles

8193f34

Fix: [CMake] automatically detect dependencies for .deb packages

9063da1

TrueBrain force-pushed the release-workflow branch from b0879c7 to 324c01e Compare December 18, 2020 13:37

LordAro approved these changes Dec 19, 2020

View reviewed changes

glx22 approved these changes Dec 19, 2020

View reviewed changes

TrueBrain merged commit de61413 into OpenTTD:master Dec 19, 2020

TrueBrain deleted the release-workflow branch December 19, 2020 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add: [Actions] release workflow for both releases and nightlies #8371

Add: [Actions] release workflow for both releases and nightlies #8371

TrueBrain commented Dec 12, 2020 •

edited

Loading

glx22 commented Dec 12, 2020

LordAro commented Dec 13, 2020

orudge commented Dec 13, 2020

michicc commented Dec 13, 2020

TrueBrain commented Dec 13, 2020

TrueBrain commented Dec 14, 2020

TrueBrain commented Dec 14, 2020

LordAro left a comment

LordAro Dec 17, 2020

TrueBrain Dec 18, 2020

LordAro Dec 17, 2020

TrueBrain Dec 18, 2020

LordAro left a comment

glx22 left a comment

Add: [Actions] release workflow for both releases and nightlies #8371

Add: [Actions] release workflow for both releases and nightlies #8371

Conversation

TrueBrain commented Dec 12, 2020 • edited Loading

Targets currently not included

Choices made

glx22 commented Dec 12, 2020

LordAro commented Dec 13, 2020

orudge commented Dec 13, 2020

michicc commented Dec 13, 2020

TrueBrain commented Dec 13, 2020

TrueBrain commented Dec 14, 2020

TrueBrain commented Dec 14, 2020

LordAro left a comment

Choose a reason for hiding this comment

LordAro Dec 17, 2020

Choose a reason for hiding this comment

TrueBrain Dec 18, 2020

Choose a reason for hiding this comment

LordAro Dec 17, 2020

Choose a reason for hiding this comment

TrueBrain Dec 18, 2020

Choose a reason for hiding this comment

LordAro left a comment

Choose a reason for hiding this comment

glx22 left a comment

Choose a reason for hiding this comment

TrueBrain commented Dec 12, 2020 •

edited

Loading