Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[20.09] Backport changes to terraform and terraform-providers #110964

Closed
wants to merge 63 commits into from

Conversation

anandsuresh
Copy link
Contributor

@anandsuresh anandsuresh commented Jan 27, 2021

Motivation for this change

Terraform has undergone some major changes which makes it unusable with v20.09; mainly a change that prevents using symlinks to identify plugins. Terraform implemented this change to prevent infinite loops when trying to search for plugins. However, this breaks terraform since the nix-store relies heavily on symlinks.

This commit backports the latest changes from master to ensure terraform runs as expected on v20.09.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Myhlamaeus and others added 30 commits January 27, 2021 17:23
…#97590)

Add version to binaries produced via patchGoModVendor to allow run-time provider
version validation by terraform.

This was initially introduced in 4e63119 (terraform: add the version component to terraform provider paths (NixOS#34497), 2018-02-06)
but broken in 3e5149a (terraform-providers: fix the google and google-beta providers, 2020-03-24)
for terraform-providers.google and terraform-providers.google-beta, and in 20f55a9 (terraform-providers.ibm: move to update-all script, 2020-04-26)
for terraform-providers.ibm.

(cherry picked from commit 94d2aeb)
0.6.0 added support for Fedora CoreOS Config v1.1.0.

 - Add Fedora CoreOS Config v1.1.0 support (NixOS#63)
 -  - Accept FCC v1.1.0 and output Ignition v3.1.0
 -  - Continue to support FCC v1.0.0 and output Ignition v3.0.0
 -  - Support merging FCC snippets into v1.0.0 or v1.1.0 FCC content
 -  - Note: Version skew among snippets and content is not supported
 - Change Container Linux Config to render Ignition v2.3.0 (NixOS#60)
 - Add zip archive format with signed checksum

(cherry picked from commit 8cdc34c)
Terraform 0.13 adopts a new filesystem layout for plugins (illustrated below).

Terraform 0.12 and earlier
`plugins-dir/terraform-provider-aws_v3.7.0`

Terraform 0.13
`plugins-dir/registry.terraform.io/hashicorp/aws/3.7.0/linux_amd64/terraform-provider-aws_v3.7.0`

To support all packaged Terraform versions a shim is created at both locations.

This approach was inspired by https://github.com/numtide/generate-terraform-provider-shim

Terraform 0.13 provider documentation https://www.terraform.io/upgrade-guides/0-13.html#new-filesystem-layout-for-local-copies-of-providers

layout terraform providers filesystem in withPlugins

(cherry picked from commit cd1b594)
Added provider source address to providers hosted in the same GitHub
repository. Other providers have moved repository and would require a
patch to the package path.

(cherry picked from commit 72cd428)
We will be relying on the pname to figure out where to link the plugin.

(cherry picked from commit 3f49e5b)
Move the providerSourceAddress fallback to the terraform.withPlugins
function. Since plugins can be arbitrary derivations, there is no
guarantee that this attribute will be exposed.

While doing that I also cleaned the toDrv function to only pass
attributes to the builder which are required by the build.

The Terraform 0.13 fallback slug has changed from
`nixpkgs/<provider-owner>/<provider-name>` to `nixpkgs/<provider-name>`
as the owner is also not always available. As a nixpkgs user, all I know
is that the provider is in nixpkgs and his name, the owner information
is not necessarily easy to get by.

(cherry picked from commit c79ff8d)
(cherry picked from commit b0a0ffb)
…ixOS#103396)

When using `terraform.withPlugins (ps: [])` the returned derivation does
not expose the meta of the underlying package. This change inherits meta
from the terraform derivation in the wrapper derivation to expose it.

(cherry picked from commit 380337d)
timstott and others added 10 commits January 27, 2021 17:29
* terraform-providers: handle go modules

More and more terraform providers are switching from vendored
dependencies to Go modules.

Let's say that you update the "aws" provider and it fails. You can run
it again with:

    ./update-provider aws --vendor

Any package that has the "vendorSha256" key will be considered as a go
module package.

The script now also supports adding new providers by using the
<owner>/<repo> format. Eg:

     ./update-provider hetznercloud/hcloud --vendor

* address comment

Fixes NixOS#104667 (comment)

* support the null use-case

* escape provider name as well

* fix typo

(cherry picked from commit 5851c31)
This also changes the origin of the code. Hashicorp decided to stop
maintaining that provider and passed on the buck to cyrilgdn.

(cherry picked from commit 0dc2e6d)
@anandsuresh
Copy link
Contributor Author

Also dropped the cherry-pick of 51c28d6 which dropped support for Terraform 0.11. Should be good to go now.

@basvandijk
Copy link
Member

@GrahamcOfBorg build terraform terraform_0_12 terraform_0_13 terraform_0_14

@manveru
Copy link
Contributor

manveru commented Feb 26, 2021

This will not build without this patch:

diff --git a/pkgs/applications/networking/cluster/terraform/default.nix b/pkgs/applications/networking/cluster/terraform/default.nix
index 73366f509688..c9ee971c2027 100644
--- a/pkgs/applications/networking/cluster/terraform/default.nix
+++ b/pkgs/applications/networking/cluster/terraform/default.nix
@@ -18,6 +18,9 @@ let
 
       postConfigure = ''
         # speakeasy hardcodes /bin/stty https://github.com/bgentry/speakeasy/issues/22
+        vendor="$(readlink -f vendor)"
+        rm vendor
+        cp -r "$vendor" vendor
         substituteInPlace vendor/github.com/bgentry/speakeasy/speakeasy_unix.go \
           --replace "/bin/stty" "${coreutils}/bin/stty"
       '';

I'm not sure if this is the intended way to resolve the issue, but vendor is a symlink into the nix store, and immutable. So I'm curious how this ever built for you.

@mweinelt
Copy link
Member

mweinelt commented Jul 5, 2021

Unfortunately NixOS 20.09 has reached its end-of-life status on 2021-07-01, one
month after the release of NixOS 21.05.

Since we do not accept any changes to its branches anymore, I'm closing this pull
request.

@mweinelt mweinelt closed this Jul 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet