Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go_1_12: disable flakey TestGcSys test #70368

Closed

Conversation

d-goldin
Copy link
Contributor

@d-goldin d-goldin commented Oct 4, 2019

Motivation for this change

In our tests we have experienced failures of this test,
but it was otherwise not reproducible so far. A backported
upstream fix did not alleviate the issue either, so disabling
seems workable for now.

I'd suggest that we only apply this one to release-19.09 and leave master and staging as is, so that we're less likely to forget to check if the issue is fixed in newer releases.

Related: #68361

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @Mic92 @cstrahan @velovix @rvolosatovs @orivej

In our tests we have experienced failures of this test,
but it was otherwise not reproducible so far. A backported
upstream fix did not alleviate the issue either, so disabling
seems workable for now.
Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nix-review passes on NixOS (failures not related to PR)

[nix-shell:/home/jon/.cache/nix-review/pr-70368]$ nix-build ./nixpkgs/ -A pkgsi686Linux.go built fine

[740 built (4 failed), 872 copied (12729.7 MiB), 3291.5 MiB DL]
error: build of '/nix/store/c54f6sabh8d0ggqby2wc8wwc3g7prcky-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/70368
8 package are marked as broken and were skipped:
beegfs dcrd dcrwallet envoy facette gopherclient meguca storebrowse

6 package failed to build:
packetbeat sourcehut.gitsrht sourcehut.hgsrht vimPlugins.youcompleteme wal-g ycmd

660 package were build:
aerc amass amazon-ecr-credential-helper aminal antibody aptly archiver argo asmfmt assh autospotting awless aws-env aws-iam-authenticator aws-okta aws-rotate-key aws-vault azure-storage-azcopy azure-vhd-utils buildozer bazel-watcher bazelisk bettercap blockbook blsd boohu boringssl bosun brig brook browserpass browsh btops buildah buildkit buildkite-agent buildkite-agent3 c14 caddy cadvisor captive-browser cayley certigo certmgr certmgr-selfsigned certstrap cfssl chezmoi circleci-cli clair claws cli53 cloud-nuke cloud-print-connector cloud-sql-proxy cloudflared cloudfoundry-cli cni cni-plugins cockroachdb codesearch cointop compile-daemon confd conftest consul consul-alerts consul-template container-linux-config-transpiler containerd convoy coredns corgi coyim cri-o cri-tools croc ctop cue curl_unix_socket curlie dapper dashing datadog-agent datadog-process-agent dbmate deadcode deepin.dbus-factory deepin.dde-api deepin.dde-control-center deepin.dde-daemon deepin.dde-dock deepin.dde-file-manager deepin.dde-launcher deepin.dde-session-ui deepin.deepin-desktop-base deepin.deepin-desktop-schemas deepin.deepin-metacity deepin.deepin-mutter deepin.deepin-wallpapers deepin.deepin-wm deepin.go-dbus-generator deepin.go-gir-generator deis deisctl delve demoit dep dep2nix desync devd dex-oidc dgraph direnv diskrsync distrobuilder dive dnscrypt-proxy2 docker docker-credential-gcr docker-distribution docker-gc docker-ls docker-machine docker-machine-kvm docker-machine-kvm2 docker-proxy docker_18_09 doctl docui documize-community drive drone drone-cli duplicacy easyjson echoip ecs-agent ejson eksctl elfinfo elvish envconsul envdir envsubst errcheck etcd evmdis exercism fac filebeat filebeat7 filegive flannel fluxctl fly fscrypt-experimental fsql fzf gauge gawp gcsfuse gdrive gh-ost ghq git-lfs git-lfs1 git-sizer git-town gitAndTools.git-annex-remote-b2 gitAndTools.git-annex-remote-rclone gitAndTools.git-appraise gitAndTools.git-bug gitAndTools.grv gitAndTools.hub gitAndTools.lab gitaly gitea github-commenter gitlab-runner gitlab-shell gitlab-workhorse glide gllvm glock gnatsd go go-2fa go-bindata go-bindata-assetfs go-check go-dependency-manager go-ethereum go-jira go-jsonnet go-junit-report go-langserver go-md2man go-motion go-mtpfs go-outline go-protobuf go-repo-root go-sct go-symbols go-tools go-upower-notify go2nix goa gocode gocode-gomod goconst goconvey gocryptfs gocyclo godef gofumpt gogetdoc gogs gohai goklp golangci-lint golint gometalinter gomodifytags gomuks gopass gopkgs goreman gosec gosu gotags gotests gotestsum gotools gotop gotty govc govendor govers gox grafana grafana-loki grafana_reporter grobi gron grpcui grpcurl gsctl guardian-agent gucci gx gx-go harmonist hcloud heapster heartbeat heartbeat7 hecate helmfile heptio-ark hetzner-kube hivemind holochain-go hologram hound httplab hugo hydron hyperledger-fabric i3cat iamy ical2org iferr impl ineffassign influxdb interfacer interlock ipfs ipfs-cluster ipfs-migrator ipget jd jid jmespath joker journalbeat journalbeat7 jp json2hcl jump jx k6 k9s kail kapacitor kbfs kexpand keybase kind kompose kontemplate kops kops_1_12 kt kube-aws kube-prompt kube-router kube3d kubecfg kubectl kubectx kubeless kubernetes kubernetes-helm kubeval kubicorn kustomize lazydocker lazygit leaps lego lf librarian-puppet-go linkerd linuxkit livepeer lnd localtime loccount lxd madonctl mage magnetico mailhog maligned manul marathonctl massren matterbridge matterircd mattermost mautrix-whatsapp meme mesos-dns metricbeat metricbeat7 micro miniflux minify minikube minio minio-client minishift mirrorbits mkcert mm mmake mmark moby mod modd mongodb-tools mop morph morty mpd-mpris mqtt-bench mtail mustache-go mynewt-newt nats-streaming-server nfpm nginx-sso ngrok-1 nix-prefetch-docker nix-store-gcs-proxy nixops-dns nomad notary noti nsq nvidia-docker oauth2_proxy obfs4 oci-image-tool oh openshift out-of-tree overmind pachyderm packer packet packetbeat7 packr pcstat pdfcpu peco perkeep pet pgcenter pgmetrics phraseapp-client pigeon pipework platinum-searcher podman pond powerline-go pprof prom2json prometheus prometheus-alertmanager prometheus-aws-s3-exporter prometheus-bind-exporter prometheus-blackbox-exporter prometheus-collectd-exporter prometheus-consul-exporter prometheus-dnsmasq-exporter prometheus-dovecot-exporter prometheus-fritzbox-exporter prometheus-gitlab-ci-pipelines-exporter prometheus-haproxy-exporter prometheus-json-exporter prometheus-mail-exporter prometheus-mesos-exporter prometheus-minio-exporter prometheus-mysqld-exporter prometheus-nginx-exporter prometheus-node-exporter prometheus-openvpn-exporter prometheus-postfix-exporter prometheus-postgres-exporter prometheus-process-exporter prometheus-pushgateway prometheus-rabbitmq-exporter prometheus-snmp-exporter prometheus-statsd-exporter prometheus-surfboard-exporter prometheus-unifi-exporter prometheus-varnish-exporter proto-contrib protoc-gen-doc prototool prow pup syncthing-gtk python37Packages.jupyter-repo2docker qr-filetransfer qsyncthingtray quicktemplate rclone reflex reftools restic restic-rest-server reviewdog richgo rkt rootlesskit runc s3gof3r saml2aws sampler scaleway-cli sensu-go-agent sensu-go-backend sensu-go-cli serfdom shadowfox shfmt sift simplehttp2server singularity skaffold skopeo skydns slack-term sonobuoy sops sourcehut.buildsrht ssm-agent statik stern styx subfinder syncthing syncthing-cli syncthing-discovery syncthing-relay syncthing-tray systemd-journal2gelf systrayhelper tegola teleconsole telegraf teleport tendermint terminal-parrot termshark terracognita terraform terraform-docs terraform-full terraform-inventory terraform-lsp terraform-providers.acme terraform-providers.alicloud terraform-providers.ansible terraform-providers.archive terraform-providers.arukas terraform-providers.atlas terraform-providers.aws terraform-providers.azuread terraform-providers.azurerm terraform-providers.azurestack terraform-providers.bigip terraform-providers.bitbucket terraform-providers.brightbox terraform-providers.chef terraform-providers.circonus terraform-providers.clc terraform-providers.cloudflare terraform-providers.cloudscale terraform-providers.cloudstack terraform-providers.cobbler terraform-providers.consul terraform-providers.datadog terraform-providers.digitalocean terraform-providers.dme terraform-providers.dns terraform-providers.dnsimple terraform-providers.docker terraform-providers.dyn terraform-providers.elasticsearch terraform-providers.external terraform-providers.fastly terraform-providers.flexibleengine terraform-providers.gandi terraform-providers.github terraform-providers.gitlab terraform-providers.google terraform-providers.google-beta terraform-providers.grafana terraform-providers.hcloud terraform-providers.hedvig terraform-providers.helm terraform-providers.heroku terraform-providers.http terraform-providers.huaweicloud terraform-providers.ibm terraform-providers.icinga2 terraform-providers.ignition terraform-providers.influxdb terraform-providers.kubernetes terraform-providers.librato terraform-providers.libvirt terraform-providers.linode terraform-providers.local terraform-providers.logentries terraform-providers.logicmonitor terraform-providers.mailgun terraform-providers.matchbox terraform-providers.mysql terraform-providers.netlify terraform-providers.newrelic terraform-providers.nixos terraform-providers.nomad terraform-providers.ns1 terraform-providers.nsxt terraform-providers.null terraform-providers.nutanix terraform-providers.oci terraform-providers.oneandone terraform-providers.opc terraform-providers.openstack terraform-providers.opentelekomcloud terraform-providers.opsgenie terraform-providers.oraclepaas terraform-providers.ovh terraform-providers.packet terraform-providers.pagerduty terraform-providers.panos terraform-providers.postgresql terraform-providers.powerdns terraform-providers.profitbricks terraform-providers.rabbitmq terraform-providers.rancher terraform-providers.random terraform-providers.rightscale terraform-providers.rundeck terraform-providers.runscope terraform-providers.scaleway terraform-providers.secret terraform-providers.segment terraform-providers.selectel terraform-providers.skytap terraform-providers.softlayer terraform-providers.spotinst terraform-providers.statuscake terraform-providers.telefonicaopencloud terraform-providers.template terraform-providers.tencentcloud terraform-providers.terraform terraform-providers.tfe terraform-providers.tls terraform-providers.triton terraform-providers.ucloud terraform-providers.ultradns terraform-providers.vault terraform-providers.vcd terraform-providers.vsphere terraform-providers.yandex terraform_0_11 terraform_0_11-full terraform_plugins_test terragrunt tewisay textql tflint thanos tilt timescaledb-parallel-copy timescaledb-tune tmsu todoist todolist toxiproxy traefik trezord tychus ua uchiwa udocker unconvert up vale vault verifpal vgo2nix vimPlugins.coc-go vimPlugins.vim-go vndr vultr wallutils webhook websocketd wego ws wtf wuzz xmloscopy xmonad-log xmpp-client xurls yaml2json yj yq-go zabbixctl

@kalbasit
Copy link
Member

kalbasit commented Oct 7, 2019

@GrahamcOfBorg build go_1_12 jx pet

@kalbasit
Copy link
Member

kalbasit commented Oct 7, 2019

another try

@GrahamcOfBorg build go_1_12 jx pet

@d-goldin
Copy link
Contributor Author

d-goldin commented Oct 9, 2019

I just noticed that @vcunat applied the upstream patch in d8218de to master, which was previously tried in #69359

Let's see what it does this time, but I think this PR might still remain necessary.

@vcunat
Copy link
Member

vcunat commented Oct 9, 2019

Oh, I should have searched better, though perhaps you were unlucky or something.

@d-goldin
Copy link
Contributor Author

d-goldin commented Oct 9, 2019

@vcunat: Possibly - that's the problematic thing with this issue. Were you able to locally reproduce the issue and fix? If so, then maybe there is a good chance it will at least happen less.

@vcunat
Copy link
Member

vcunat commented Oct 9, 2019

Well, the build succeeded on Hydra: https://hydra.nixos.org/eval/1547624#tabs-now-succeed

@vcunat
Copy link
Member

vcunat commented Oct 9, 2019

No, the issue didn't happen locally for me, so I only checked it didn't break.

Copy link
Member

@vcunat vcunat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the failure happens again on Hydra, we should merge this, both to 19.09 and master. (And maybe even if it doesn't.)

@d-goldin
Copy link
Contributor Author

d-goldin commented Oct 9, 2019

@vcunat: Alright, nice. Then let's see how this goes. Thanks! :)

@vcunat
Copy link
Member

vcunat commented Oct 9, 2019

The patch re-enabled that test on aarch64, and that also seems green on Hydra 📈

@d-goldin
Copy link
Contributor Author

d-goldin commented Oct 9, 2019

@vcunat: Alright, that's great news! Thanks! A pity it didn't seem to work back then on ofBorg. Im going to close the PR then. But I'm still wondering what is possibly different in the ofBorg environment. Maybe someone could trigger the fixing commit on ofBorg to see if this still fails while succeeding on hydra?

@d-goldin d-goldin closed this Oct 9, 2019
@rnhmjoj
Copy link
Contributor

rnhmjoj commented Oct 15, 2019

I'm still seeing failures here: https://hydra.nixos.org/build/103322858

vcunat pushed a commit that referenced this pull request Oct 16, 2019
In our tests we have experienced failures of this test,
but it was otherwise not reproducible so far. A backported
upstream fix did not alleviate the issue either, so disabling
seems workable for now.
@vcunat vcunat self-assigned this Oct 16, 2019
vcunat pushed a commit that referenced this pull request Oct 16, 2019
In our tests we have experienced failures of this test,
but it was otherwise not reproducible so far. A backported
upstream fix did not alleviate the issue either, so disabling
seems workable for now.

(cherry picked from commit cffba01)
@vcunat
Copy link
Member

vcunat commented Oct 16, 2019

Right, so it's come to that. I pushed your commit and reverted mine.

@d-goldin
Copy link
Contributor Author

@vcunat: I'm not sure if reverting yours is necessary, as it might actually address some useful cases (the GO team has their reasons, after all), but I'm really not familiar with go internals to judge that.

@d-goldin d-goldin deleted the go_1_12_disable_testgcsys branch October 16, 2019 09:13
@vcunat
Copy link
Member

vcunat commented Oct 17, 2019

It was not necessary. I don't know... well after update it won't matter anyway, as that should have the change included.

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Jan 27, 2020
In our tests we have experienced failures of this test,
but it was otherwise not reproducible so far. A backported
upstream fix did not alleviate the issue either, so disabling
seems workable for now.

(cherry picked from commit cffba01)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants