snap: Build snaps on Travis via remote-build #506

ppd · 2019-11-24T15:24:18Z

This PR introduces building & publishing of snaps into the CI pipeline.
Implements #483

~~A bit of refactoring while at it:~~

~~Fix invalid osx_image xcode8.2~~
~~Drop depreceated sudo keyword~~
~~Run debian build on bionic image & re-enable building with ui~~
~~Split build stage into separate jobs for better separation of concerns~~

~~I guess the unrelated changes should be split out into separate PRs if the general approach is considered sane.~~

The above was merged in #510

What's left to do

Wait for snapcraft 3.9.x to make it into stable with remote-build included
Create a solvespace launchpad account and replace my personal credentials with it
Transfer ownership of the solvespace snap Transfer snap ownership to solvespace organization #482
Use someone else's snapcraft token for deploying:

snapcraft export-login --snaps=solvespace --channels=edge snapcraft-login
travis env set SNAP_TOKEN "$(cat snapcraft-login)"

Decide on what architectures we want to build snaps for

ppd · 2019-11-24T15:28:21Z

An exemplary build: https://travis-ci.org/ppd1990/solvespace/builds/616292318

.travis.yml

whitequark · 2019-11-24T15:29:14Z

Great work refactoring the Travis config! I would prefer if it went in as a separate PR so we can get the benefits of it before snapcraft releases their changes.

ppd · 2019-11-24T15:33:58Z

Do you think we should refine the Travis config further into test & deploy stages or do you feel that's overkill for now? At the moment, a OSX or Debian test fail wouldn't stop a snap package from being built & released (to edge that is).

whitequark · 2019-11-24T15:39:34Z

Do you think we should refine the Travis config further into test & deploy stages or do you feel that's overkill for now?

That would be great! We definitely shouldn't publish packages that don't pass tests.

ppd · 2019-11-24T16:31:16Z

We have no easy build artefact sharing between Travis stages though. Suppose we want something like this:

Stage: Test

Debian test build
OSX test build

Stage: Deploy (run only if all jobs in Test succeed)

OSX production build -> deploy
Snap production build -> deploy

Then we need to do the OSX build twice.

Also: Do we want to block snap builds if OSX fails? Do we want to block all deploy jobs when a single one of the test jobs fails?

whitequark · 2019-11-26T09:07:28Z

Do we want to block snap builds if OSX fails?

No.

Do we want to block all deploy jobs when a single one of the test jobs fails?

The tests are supposed to be completely platform-independent. (There's a few tricky parts, mostly having to do with FP precision issues and such.) So I think we should run tests on every platform, but as a redundant mechanism to alert us if they somehow are not platform-independent; they ideally always should be.

ppd · 2019-11-26T10:39:48Z

With the currently merged Travis config, we run sequentially like so:

1. Test on Debian 
2. Test & Deploy on OSX

Adding snap:

1. Test on Debian 

In parallel:
2.1 Test & Deploy on OSX
2.2 Build & Deploy Snap

So OSX & Snap are blocked by Debian/Ubuntu test fails and of course by their own fails. I feel that Snap deployments are rather uncritical as we'd only ever manually promote revisions to candidate, stable and beta.

vespakoen · 2019-11-26T21:15:04Z

.travis/install-macos.sh

@@ -2,3 +2,4 @@

 brew update
 brew install freetype cairo


I noticed the macOS build is failing, and it might be fixed (and become faster) by removing the brew update & brew install statements.

I don't think we need to update brew and also don't need freetype & cairo since they are already bundled as git submodules.

It seems we only need git & cmake which should already be installed by default on Travis.

I don't think we need to update brew and also don't need freetype & cairo since they are already bundled as git submodules.

My position is that libraries that are available on the system should not be linked as submodules, and also should be linked dynamically, if possible at all.

Agreed, however, cairo and freetype are not installed by default as far as I know, so I think we should make sure they are included (on macOS and windows deliverables) also, the CI should probably test the dependencies that are linked as submodules to make sure they work.

I have seen many OSS failing on macOS due to some missing library and otool workarounds (even on this repo if I remember correctly) and I guess that is not what we want.

As an added bonus, I guess the libraries that are in use are not that big at all.

I have seen many OSS failing on macOS due to some missing library and otool workarounds (even on this repo if I remember correctly) and I guess that is not what we want.

They're now linked statically instead, which is what I should have done in the first place.

the CI should probably test the dependencies that are linked as submodules to make sure they work.

Our CI is already quite slow and testing the dependency version that is updated once a few months at best is a waste of time.

ppd · 2019-11-27T12:28:48Z

Note to myself: --launchpad-user will not be necessary in the future

canonical/snapcraft#2825

ppd · 2019-11-28T17:03:04Z

Snapcraft 3.9.2 is in the stable channel

Edit: It was rolled back due to a bug. Still waiting for 3.9.3 to hit stable today: https://forum.snapcraft.io/t/call-for-testing-snapcraft-3-9/13944/30?u=ppd

Edit 2: And nothing yet. 3.9.3 seems to be blocked too.

Edit 2020-01-08: Still nothing. Rest assured, this PR is not forgotten...

.travis/build-snap.sh

.travis.yml

We invoke builds on Launchpad in stage "deploy" and release it into the edge channel of the Snap Store. The deploy stage is blocked on fails of the test stage, so we don't release snaps with failing tests.

ppd · 2020-01-21T13:00:40Z

A little closer to the finish line: https://travis-ci.org/ppd1990/solvespace/builds/639505699

Nothing much left to do, I guess. Snapcraft 3.9.7 is stable, Launchpad is happy, Travis is happy. Are we happy?

whitequark · 2020-01-21T13:22:50Z

Yup all looks good. Let me know how I should merge this and we're done.

ppd · 2020-01-21T13:47:22Z

We need a launchpad account for building. Just register one solely for solvespace.
Then you need to use it once via remote-build:

./pkg/snap/build.sh remote-build \
  --launchpad-user solvespace \
  --launchpad-accept-public-upload \

You will now have those credentials stored in $HOME/.local/share/snapcraft/provider/launchpad/credentials.

Encrypt them in Travis: travis encrypt-file credentials and replace my launchpad-credentials.enc with the encrypted file in this merge request. Adjust the openssl command in build-snap.sh accordingly. The --launchpad-user option must be adjusted too.
Login in snapcraft into the solvespace account: snapcraft login and then do

snapcraft export-login --snaps=solvespace --channels=beta,edge snapcraft-login
travis env set SNAP_TOKEN "$(cat snapcraft-login)"

Those are the credentials for pushing & releasing to the store.

ppd · 2020-01-22T12:03:16Z

Re the failed to deploy error: I found this in dpl (deploy tools for Travis):
in v1

      def push_app
        snaps = Dir.glob(snap)
        case snaps.length
        when 0
          error "No snap found matching '#{snap}'"
        when 1
          snap_path = snaps.first
          context.fold("Pushing snap") do
            context.shell "snapcraft push #{snap_path} --release=#{channel}"
          end
        else
          snap_list = snaps.join(', ')
          error "Multiple snaps found matching '#{snap}': #{snap_list}"
        end
      end

or v2

      def validate
        error :no_snaps if snaps.empty?
        error :multiple_snaps if snaps.size > 1
      end

I guess the deployment is meant to only handle one snap at a time.

ppd · 2020-01-22T13:18:17Z

@whitequark We can fix this by either a deployment script or by explicit deploy sections for each architecture (n = 4 x 2 = 8), which seems a little heavy-handed. What do you prefer?

whitequark · 2020-01-22T13:20:51Z

Deployment script seems fine.

ppd · 2020-01-22T15:41:13Z

See #543

whitequark reviewed Nov 24, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

ppd mentioned this pull request Nov 25, 2019

Refactor Travis config #510

Merged

vespakoen reviewed Nov 26, 2019

View reviewed changes

ppd force-pushed the travis-snap branch from eb7fb9e to bce7f0e Compare November 27, 2019 10:13

ppd commented Dec 2, 2019

View reviewed changes

.travis/build-snap.sh Show resolved Hide resolved

.travis.yml Outdated Show resolved Hide resolved

ppd force-pushed the travis-snap branch from fffeda4 to 31e3611 Compare January 20, 2020 12:32

ppd added 4 commits January 20, 2020 13:32

snap: Build snaps on Travis via remote-build

9c2218a

We invoke builds on Launchpad in stage "deploy" and release it into the edge channel of the Snap Store. The deploy stage is blocked on fails of the test stage, so we don't release snaps with failing tests.

use snapcraft from stable channel

8f04470

build on amd64, arm64, armhf and i386

31e3611

release tagged revisions to beta channel too

463b2ee

ppd changed the title ~~WIP: snap: Build snaps on Travis via remote-build~~ snap: Build snaps on Travis via remote-build Jan 21, 2020

whitequark closed this Jan 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

snap: Build snaps on Travis via remote-build #506

snap: Build snaps on Travis via remote-build #506

ppd commented Nov 24, 2019 •

edited

ppd commented Nov 24, 2019

whitequark commented Nov 24, 2019

ppd commented Nov 24, 2019

whitequark commented Nov 24, 2019

ppd commented Nov 24, 2019

whitequark commented Nov 26, 2019

ppd commented Nov 26, 2019

vespakoen Nov 26, 2019

whitequark Nov 26, 2019

vespakoen Nov 26, 2019

whitequark Nov 26, 2019

ppd commented Nov 27, 2019

ppd commented Nov 28, 2019 •

edited

ppd commented Jan 21, 2020

whitequark commented Jan 21, 2020

ppd commented Jan 21, 2020 •

edited

ppd commented Jan 22, 2020 •

edited

ppd commented Jan 22, 2020

whitequark commented Jan 22, 2020

ppd commented Jan 22, 2020

snap: Build snaps on Travis via remote-build #506

snap: Build snaps on Travis via remote-build #506

Conversation

ppd commented Nov 24, 2019 • edited

What's left to do

ppd commented Nov 24, 2019

whitequark commented Nov 24, 2019

ppd commented Nov 24, 2019

whitequark commented Nov 24, 2019

ppd commented Nov 24, 2019

Stage: Test

Stage: Deploy (run only if all jobs in Test succeed)

whitequark commented Nov 26, 2019

ppd commented Nov 26, 2019

vespakoen Nov 26, 2019

Choose a reason for hiding this comment

whitequark Nov 26, 2019

Choose a reason for hiding this comment

vespakoen Nov 26, 2019

Choose a reason for hiding this comment

whitequark Nov 26, 2019

Choose a reason for hiding this comment

ppd commented Nov 27, 2019

ppd commented Nov 28, 2019 • edited

ppd commented Jan 21, 2020

whitequark commented Jan 21, 2020

ppd commented Jan 21, 2020 • edited

ppd commented Jan 22, 2020 • edited

ppd commented Jan 22, 2020

whitequark commented Jan 22, 2020

ppd commented Jan 22, 2020

ppd commented Nov 24, 2019 •

edited

ppd commented Nov 28, 2019 •

edited

ppd commented Jan 21, 2020 •

edited

ppd commented Jan 22, 2020 •

edited