Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change: [Actions] stop relying on external actions for trivial stuff, and rely on cmake tools to build, test and pack #8718

Merged
merged 5 commits into from Feb 24, 2021

Conversation

glx22
Copy link
Contributor

@glx22 glx22 commented Feb 21, 2021

Motivation / Problem

We used run-vcpkg to manage vcpkg for windows and macos targets and run-cmake to build windows targets. But doing the same without these actions is quite trivial, and easier to understand in the workflow.
Also we used to run make to build non windows targets, but it's better to let cmake handle the calls to native build tools.

Description

Implemented the above changes.

Limitations

Checklist for review

Some things are not automated, and forgotten often. This list is a reminder for the reviewers.

  • The bug fix is important enough to be backported? (label: 'backport requested')
  • This PR affects the save game format? (label 'savegame upgrade')
  • This PR affects the GS/AI API? (label 'needs review: Script API')
    • ai_changelog.hpp, gs_changelog.hpp need updating.
    • The compatibility wrappers (compat_*.nut) need updating.
  • This PR affects the NewGRF API? (label 'needs review: NewGRF')

Copy link
Member

@TrueBrain TrueBrain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks very nice :D

Often I mentioned something on a single place, but I am sure you understand I mean all instances of similar change ;)

vcpkgTriplet: '${{ matrix.arch }}-osx'
path: /usr/local/share/vcpkg/installed
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }}
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird restore-key, ending with a -?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It matches all keys if key is a miss, and use the most recent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I submitted a PR to run-vcpkg (lukka/run-vcpkg#75) which alters the caching there so that it uses the env.ImageVersion variable in the cache key, so that the cache is invalidated if the VM environment is changed. It would be worth us using this in our key (it did cause a problem on macOS before).

(I was waiting for the new release of run-vcpkg, which did happen but I then forgot about it, so never updated OpenTTD...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you remember the exact issue ?
I cache less than run-vcpkg (only vcpkg/installed folder) and I assume vcpkg exe provided in the image works. If vcpkg in image is upgraded, it should detect if installed libs need an upgrade (at least my local vcpkg does that)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the build logs will have expired now, but I think the error was relating to missing dependencies in libstdc++.dylib or something along those lines, due to a compiler change in the VM template.

I had pinned vcpkg to a particular commit for macOS, but I think the version in the image is new enough to support our arm64 workflow, so it should hopefully be fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well release workflow seems to work https://github.com/glx22/OpenTTD/runs/1946699269?check_suite_focus=true
And it seems run-vcpkg always rebuilt vcpkg executable (even for windows)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A closer look to how images generate their name etc, this cache-key will indeed be a bit of a problem when GitHub changes some things.

${{ runner.os } is just macOS for example. Not macos-latest or macos-10.15. This means that if they switch the base image, it is possible that the precompiled libraries no longer match the host system, resulting in compile errors, or, maybe worse, missing security fixes etc.

It seems that for cache keys the most correct approach would be to include ${{ env.ImageOs }}-${{ env.ImageVersion }}. This changes every change of the base image, and would trigger a full recompile for us, but those events are rare. It would also mean that this cache should in theory never break our compiles .. might save a lot of issues?

What do you think @glx22 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yes I can replace ${{ runner.os } with ${{ env.ImageOS }}-${{ env.ImageVersion }}.
That means cache is always rebuilt when a new image is used. It was partially the case when using the commit id in the key as it would use cached libs and upgrade them if needed, assuming binary compatibility.
Will be safer to always fully rebuild libs unconditionally.
I think I can remove commit id from the key, and also restore-keys.

.github/workflows/ci-build.yml Outdated Show resolved Hide resolved
.github/workflows/ci-build.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
.github/workflows/release.yml Show resolved Hide resolved
Copy link
Contributor Author

@glx22 glx22 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will address other comments

vcpkgTriplet: '${{ matrix.arch }}-osx'
path: /usr/local/share/vcpkg/installed
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }}
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It matches all keys if key is a miss, and use the most recent.

.github/workflows/ci-build.yml Outdated Show resolved Hide resolved
LordAro
LordAro previously approved these changes Feb 22, 2021
Copy link
Member

@LordAro LordAro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues in principle

vcpkgTriplet: '${{ matrix.arch }}-osx'
path: /usr/local/share/vcpkg/installed
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }}
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A closer look to how images generate their name etc, this cache-key will indeed be a bit of a problem when GitHub changes some things.

${{ runner.os } is just macOS for example. Not macos-latest or macos-10.15. This means that if they switch the base image, it is possible that the precompiled libraries no longer match the host system, resulting in compile errors, or, maybe worse, missing security fixes etc.

It seems that for cache keys the most correct approach would be to include ${{ env.ImageOs }}-${{ env.ImageVersion }}. This changes every change of the base image, and would trigger a full recompile for us, but those events are rare. It would also mean that this cache should in theory never break our compiles .. might save a lot of issues?

What do you think @glx22 ?

Copy link
Member

@TrueBrain TrueBrain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seriously nice work :)

@glx22 glx22 merged commit 70e4845 into OpenTTD:master Feb 24, 2021
@glx22 glx22 deleted the cmake_ci branch February 24, 2021 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants