Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snap: Build snaps on Travis without remote-build & drop i386 + armhf #547

Merged
merged 1 commit into from Feb 8, 2020

Conversation

ppd
Copy link
Member

@ppd ppd commented Feb 7, 2020

To quote the commits:

snap: Support building snaps in Docker containers

Docker provides clean environments for snapcraft builds.
This commit adds Dockerfiles for architectures amd64, i386, arm64
and armhf.
They support cross-building to foreign architectures if the host
supports QEMU user mode emulation.

Support for building these images on Docker Hub is implemented
by the pre_build hook.

The main purpose of those images is to allow building of snaps
on Travis without relying on external services like Launchpad.
snap: Build snaps on Travis via Docker

Snapcraft's remote-build does not fit the requirements of CI,
so replace it with builds running in Docker containers:

1. Builds in Docker can run independently, whereas remote-build
competes for the same resource (Launchpad) and will potentially
block if multiple commits require building in succession.
2. Snapcrafts CLI for remote-build is not designed to be easily
scriptable.
Builds in Docker behave like any ordinary local build would.

Travis recently introduced building on arm64, so builds for
arm64 and armhf are now very fast and don't require emulation.

One reason to use remote-build was the fact that no easy way existed to cheaply build foreign CPU architectures on Travis.
This has changed though: https://docs.travis-ci.com/user/multi-cpu-architectures/

An example build: https://travis-ci.org/ppd1990/solvespace/builds/647288825

The WIP portion: The build-snap.sh pulls the images from my image on Docker Hub: mfederle/solvespace-snapcraft:<arch>. This would have to be changed to an official image like solvespace/snapcraft:<arch> or whatever is preferable.

I feel this is the most pragmatic and "idiomatic" way of doing what we want to do. Snapcraft's remote-build is still functional and a great tool for local builds, but it's a brittle solution for CI.

@ppd
Copy link
Member Author

ppd commented Feb 7, 2020

They support cross-building to foreign architectures if the host
supports QEMU user mode emulation.

Well, support cross-building on amd64, to be precise. That's the dominant use-case but we could or should be more specific here.

This would have to be changed to an official image like solvespace/snapcraft: or whatever is preferable.

I propose registering an account on hub.docker.com and then connecting this repo. I'd either disable auto-builds or limit auto-building to very specific tags like e.g. docker/ec63361 so Docker Hub does not hammer out build after build for every unrelated change to master.

@whitequark
Copy link
Contributor

whitequark commented Feb 8, 2020

Thanks for the quick fix. Could you explain the exact relationship to Docker Hub? Are SolveSpace builds actually generated on Travis CI or Docker Hub?

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

We have to build in clean & minimal Ubuntu environments that conform to our declared base: core18, i.e. Ubuntu 18.04.

On Travis, we can only get Ubuntu 18.04 environments on amd64 and on arm64. This is problematic, as we also want to build for i386 and armhf. We can build for amd64 or arm64 though by disabling snapcraft's default mode of launching a VM, as launching a VM doesn't work on Travis.

Enter Docker: We take upstream's example Dockerfile and modify it, so that it creates a sane build environment for all four architectures.

As the Dockerfile is a mere description/recipe for building a Docker image, we have to build it before using said image.
We can do that in multiple ways:
a) on Travis, right before using that image to build the snap
b) manually on our computers and push the resulting images to Docker Hub
c) automatically by Docker Hub, which accesses our Dockerfiles and saves the resulting image for us to use later

The obvious choice here is c. We use Docker Hub as CD for our Docker images so we don't have to worry about them and to accelerate the builds on Travis, as we can simply pull the image from Dockerhub from then on before building the snap.

TL/DR: Docker Hub builds our Docker images that we need for snapcraft builds later. Docker Hub builds will be very rare, i.e. only when our Dockerfiles change. SolveSpace builds run on Travis and use those previously built images for clean environments in their respective target architecture.

@whitequark
Copy link
Contributor

as we can simply pull the image from Dockerhub from then on before building the snap.

Do you think it would be possible to use the Travis cache instead? This has a few advantages: fewer points of failure, no need to manage more credentials, the images are more obviously built from our Dockerfiles because there is no manual step that may introduce error, etc.

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

Yes. I think that's possible. I haven't played with any caching stuff so far, but I'll quickly find out whether that's viable.

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

I wonder whether we could just configure a foreign architecture in dpkg and force snapcraft's env variable SNAP_ARCH to that architecture. Then we could lose the Docker images altogether...

@whitequark
Copy link
Contributor

I'm in favor. In fact, I'm not sure if there is much point in supporting i386 and armhf anymore. For armhf, we have Raspberry Pi... does anyone actually do CAD on rpi? It's painfully slow if you're trying to do far simpler things. For i386, I'm not actually sure if people still run Linux 32-bit x86 systems. (Even Windows 32-bit x86 systems are dying out, though much slower so we still have to support them for a few years.)

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

That would be dead easy then. We can do amd64 and arm64 with --destructive-mode just fine on those stock images. i386 hmm. I'd say that's almost dead for desktop users.

@whitequark
Copy link
Contributor

Let's just drop i386 and armhf then. If someone actually requests them, we can do additional work to support those.

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

Right. A new PR then or shall I modify this one?

@whitequark
Copy link
Contributor

Let's modify this one.

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

While I'm at it: snapcraft has the concept of branches:

A branch is an optional finer subdivision of a channel for a published snap that allows for the creation of a short-lived sequences of snaps that can be pushed on demand by snap developers to help with fixes or temporary experimentation.

We could release builds for every PR to edge/pr<number> and people could install them easily for local testing.

@whitequark
Copy link
Contributor

We could release builds for every PR to edge/pr<number> and people could install them easily for local testing.

I think this would effectively expose snapcraft credentials to everyone, since they can just echo them in a PR Travis config.

@nabijaczleweli
Copy link
Contributor

nabijaczleweli commented Feb 8, 2020

Better yet, secure env vars aren't enabled in Travis on foreign PRs, such as this one, so they'd have to be present in every build log anyway.

Snapcraft's remote-build does not fit the requirements of CI,
so replace it with builds running directly on Travis:

1. Builds on Travis can run independently, whereas remote-build
competes for the same resource (Launchpad) and will potentially
block if multiple commits require building in succession.
2. Snapcrafts CLI for remote-build is not designed to be easily
scriptable.
3. Travis recently introduced building on arm64, so builds for
arm64 are now very fast and don't require emulation.

We do not build for armhf and i386 any more because they are
of little relevance on the desktop.
@ppd ppd changed the title [WIP] snap: Build snaps on Travis via Docker snap: Build snaps on Travis without remote-build & drop i386 + armhf Feb 8, 2020
@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

I think that's it. Way easier and the required changes are minimal.

@whitequark
Copy link
Contributor

Wait--how are packages uploaded without credentials? Am I missing something?

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

Where do you see uploads?

@whitequark
Copy link
Contributor

I don't, that's what I don't understand. Shouldn't we upload the built packages to the Snap store?

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

From master, yes. From PRs, no.

@ppd
Copy link
Member Author

ppd commented Feb 8, 2020

Ah, you're talking about the launchpad credentials. Those were just for remote-build. SNAP_TOKEN ist still getting picked up in deploy-snap.sh

@whitequark
Copy link
Contributor

Ah sorry, was looking at an old checkout without deploy-snap.sh.

@whitequark whitequark merged commit 65ab595 into solvespace:master Feb 8, 2020
@ppd ppd deleted the travis-docker branch October 20, 2020 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants