New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change: [Actions] stop relying on external actions for trivial stuff, and rely on cmake tools to build, test and pack #8718
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That looks very nice :D
Often I mentioned something on a single place, but I am sure you understand I mean all instances of similar change ;)
.github/workflows/ci-build.yml
Outdated
vcpkgTriplet: '${{ matrix.arch }}-osx' | ||
path: /usr/local/share/vcpkg/installed | ||
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }} | ||
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
weird restore-key, ending with a -
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It matches all keys if key
is a miss, and use the most recent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I submitted a PR to run-vcpkg (lukka/run-vcpkg#75) which alters the caching there so that it uses the env.ImageVersion variable in the cache key, so that the cache is invalidated if the VM environment is changed. It would be worth us using this in our key (it did cause a problem on macOS before).
(I was waiting for the new release of run-vcpkg, which did happen but I then forgot about it, so never updated OpenTTD...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you remember the exact issue ?
I cache less than run-vcpkg
(only vcpkg/installed
folder) and I assume vcpkg exe provided in the image works. If vcpkg in image is upgraded, it should detect if installed libs need an upgrade (at least my local vcpkg does that)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suspect the build logs will have expired now, but I think the error was relating to missing dependencies in libstdc++.dylib or something along those lines, due to a compiler change in the VM template.
I had pinned vcpkg to a particular commit for macOS, but I think the version in the image is new enough to support our arm64 workflow, so it should hopefully be fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well release workflow seems to work https://github.com/glx22/OpenTTD/runs/1946699269?check_suite_focus=true
And it seems run-vcpkg
always rebuilt vcpkg
executable (even for windows)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A closer look to how images generate their name etc, this cache-key will indeed be a bit of a problem when GitHub changes some things.
${{ runner.os }
is just macOS
for example. Not macos-latest
or macos-10.15
. This means that if they switch the base image, it is possible that the precompiled libraries no longer match the host system, resulting in compile errors, or, maybe worse, missing security fixes etc.
It seems that for cache keys the most correct approach would be to include ${{ env.ImageOs }}-${{ env.ImageVersion }}
. This changes every change of the base image, and would trigger a full recompile for us, but those events are rare. It would also mean that this cache should in theory never break our compiles .. might save a lot of issues?
What do you think @glx22 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm yes I can replace ${{ runner.os }
with ${{ env.ImageOS }}-${{ env.ImageVersion }}
.
That means cache is always rebuilt when a new image is used. It was partially the case when using the commit id in the key as it would use cached libs and upgrade them if needed, assuming binary compatibility.
Will be safer to always fully rebuild libs unconditionally.
I think I can remove commit id from the key, and also restore-keys
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will address other comments
.github/workflows/ci-build.yml
Outdated
vcpkgTriplet: '${{ matrix.arch }}-osx' | ||
path: /usr/local/share/vcpkg/installed | ||
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }} | ||
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It matches all keys if key
is a miss, and use the most recent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No issues in principle
.github/workflows/ci-build.yml
Outdated
vcpkgTriplet: '${{ matrix.arch }}-osx' | ||
path: /usr/local/share/vcpkg/installed | ||
key: ${{ runner.os }}-${{ matrix.arch }}-vcpkg-${{ steps.key.outputs.commitid }} | ||
restore-keys: ${{ runner.os }}-${{ matrix.arch }}-vcpkg- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A closer look to how images generate their name etc, this cache-key will indeed be a bit of a problem when GitHub changes some things.
${{ runner.os }
is just macOS
for example. Not macos-latest
or macos-10.15
. This means that if they switch the base image, it is possible that the precompiled libraries no longer match the host system, resulting in compile errors, or, maybe worse, missing security fixes etc.
It seems that for cache keys the most correct approach would be to include ${{ env.ImageOs }}-${{ env.ImageVersion }}
. This changes every change of the base image, and would trigger a full recompile for us, but those events are rare. It would also mean that this cache should in theory never break our compiles .. might save a lot of issues?
What do you think @glx22 ?
56cd3bb
to
7ac062e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seriously nice work :)
Motivation / Problem
We used
run-vcpkg
to managevcpkg
for windows and macos targets andrun-cmake
to build windows targets. But doing the same without these actions is quite trivial, and easier to understand in the workflow.Also we used to run
make
to build non windows targets, but it's better to letcmake
handle the calls to native build tools.Description
Implemented the above changes.
Limitations
Checklist for review
Some things are not automated, and forgotten often. This list is a reminder for the reviewers.