Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hackagePackages: dontCheck cryptohash-sha512, dontCheck hnix-store-remote #108010

Merged

Conversation

Anton-Latukha
Copy link
Contributor

@Anton-Latukha Anton-Latukha commented Dec 31, 2020

Status: waiting on the commercialhaskell/stackage#5766 & haskell-nix/hnix-store#104

Motivation for this change

Bringing up to date configuration of a couple of packages.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@cdepillabout
Copy link
Member

Hmm, I tried building hnix-store-remote, but it doesn't look like it is able to build:

$ git rev-parse HEAD
2ad09b6771bc76f5691cbd612f847528daceff9a
$ NIXPKGS_ALLOW_BROKEN=1 nix-build -A haskellPackages.hnix-store-remote
these derivations will be built:
  /nix/store/fwg6v2p9f9mnrjvq6836v40pwbi69ai1-hnix-store-remote-0.3.0.0.drv
building '/nix/store/fwg6v2p9f9mnrjvq6836v40pwbi69ai1-hnix-store-remote-0.3.0.0.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/kfc4428g15gzv5xq4wi7vjs1m2qzkkqh-ghc-8.10.3.
unpacking sources
unpacking source archive /nix/store/vpkdq310w3bz1h3rml8106b1qanv2s9r-hnix-store-remote-0.3.0.0.tar.gz
source root is hnix-store-remote-0.3.0.0
setting SOURCE_DATE_EPOCH to timestamp 1000000000 of file hnix-store-remote-0.3.0.0/tests/Util.hs
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/build/setup-package.conf.d -j8 +RTS -A64M -RTS -threaded -rtsopts
[1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /build/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/0mzmlb7sx5ynlx76ylfcw1l6k3gz6a6f-hnix-store-remote-0.3.0.0 --libdir=$prefix/lib/$compiler --libsubdir=$abi/$libname --docdir=/nix/store/q97drw4qk265478rhjkv5kk13m25j06f-hnix-store-remote-0.3.0.0-doc/share/doc/hnix-store-remote-0.3.0.0 --with-gcc=gcc --package-db=/build/package.conf.d --ghc-options=-j8 +RTS -A64M -RTS --disable-split-objs --enable-library-profiling --profiling-detail=exported-functions --disable-profiling --enable-shared --disable-coverage --enable-static --disable-executable-dynamic --disable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --ghc-option=-split-sections --extra-lib-dirs=/nix/store/diaimgl27ri62wkf44jxan3m5vpj4f1y-ncurses-6.2/lib --extra-lib-dirs=/nix/store/zb3fx7zx5pafnqk1c6nxk1fy84k3cnxz-libffi-3.3/lib --extra-lib-dirs=/nix/store/gk72n6pafkh603q9syxgj0i3pwp4fqf4-gmp-6.2.1/lib
Using Parsec parser
Configuring hnix-store-remote-0.3.0.0...
...
building
Preprocessing library for hnix-store-remote-0.3.0.0..
Building library for hnix-store-remote-0.3.0.0..
[1 of 8] Compiling System.Nix.Store.Remote.Binary ( src/System/Nix/Store/Remote/Binary.hs, dist/build/System/Nix/Store/Remote/Binary.o, dist/build/System/Nix/Store/Remote/Binary.dyn_o )
[2 of 8] Compiling System.Nix.Store.Remote.Builders ( src/System/Nix/Store/Remote/Builders.hs, dist/build/System/Nix/Store/Remote/Builders.o, dist/build/System/Nix/Store/Remote/Builders.dyn_o )
[3 of 8] Compiling System.Nix.Store.Remote.Parsers ( src/System/Nix/Store/Remote/Parsers.hs, dist/build/System/Nix/Store/Remote/Parsers.o, dist/build/System/Nix/Store/Remote/Parsers.dyn_o )

src/System/Nix/Store/Remote/Parsers.hs:37:13: error:
    • Variable not in scope:
        decodeBase32 :: Data.Text.Text -> Either String (Digest 'SHA256)
    • Perhaps you meant ‘encodeBase32’ (imported from System.Nix.Hash)
   |
37 |   digest <- decodeBase32 @'SHA256 <$> parseHash
   |             ^^^^^^^^^^^^

src/System/Nix/Store/Remote/Parsers.hs:49:20: error:
    Variable not in scope:
      mkNamedDigest
        :: Data.Text.Text
           -> Data.Text.Text -> Either String SomeNamedDigest
   |
49 | parseTypedDigest = mkNamedDigest <$> parseHashType <*> parseHash
   |                    ^^^^^^^^^^^^^
[4 of 8] Compiling System.Nix.Store.Remote.Types ( src/System/Nix/Store/Remote/Types.hs, dist/build/System/Nix/Store/Remote/Types.o, dist/build/System/Nix/Store/Remote/Types.dyn_o )
[5 of 8] Compiling System.Nix.Store.Remote.Util ( src/System/Nix/Store/Remote/Util.hs, dist/build/System/Nix/Store/Remote/Util.o, dist/build/System/Nix/Store/Remote/Util.dyn_o )

src/System/Nix/Store/Remote/Util.hs:27:1: error:
    Could not find module ‘System.Nix.Build’
    Perhaps you meant System.Nix.Util (from hnix-store-core-0.2.0.0)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
27 | import           System.Nix.Build
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

builder for '/nix/store/fwg6v2p9f9mnrjvq6836v40pwbi69ai1-hnix-store-remote-0.3.0.0.drv' failed with exit code 1
error: build of '/nix/store/fwg6v2p9f9mnrjvq6836v40pwbi69ai1-hnix-store-remote-0.3.0.0.drv' failed

@Anton-Latukha
Copy link
Contributor Author

Anton-Latukha commented Jan 1, 2021

@cdepillabout
You are right, 0.3 does not build.

Please help to understand what is the best algorithm to do here:

I arrived because we made a hnix-store-core and hnix-store-remote both 0.4 releases to Hackage.

My friends released a broken 0.3 previously, so I went in the project and made sure to make things work further after, deployed CI both Cabal builds and Nix builds, and did stuff to made sure that the next releases would be assured to work.

Right after every Haskell project release I involved in - I do what is proper - I open Nixpkgs and check the descriptions of those projects we just released to Hackage, in Nixpkgs.
Knowing that Stackage almost fully automatically updates every midnight UTC. And that everything else gets managed by you and the team through the week, and that Peti merges it all on Friday, after he made his updates and work.

So what is a proper way to do it.

Then guys released the 0.4 on Hackage. The situation we have with *-remote is more complex then the classical case, because it is custom Nix-fueled monorepo, guys priorly made a number broken releases of dependent packages they so far not fixed, and on top of it guys on releases hit the Hackage Build Matrix bug. But our CI and scheduled builds show that releases are Ok.
https://github.com/haskell-nix/hnix-store/runs/1632365896?check_suite_focus=true
https://github.com/haskell-nix/hnix-store/runs/1632473259?check_suite_focus=true - this one tracks the Nixpkgs channel.
And we already use working 0.4 release, I migrated project downstream to it.

But all that complexity on top - does not touches the main core question and procedure:

  • Right after we made releases - I went and cleaned up things in the Nixpkgs haskell-updates branch, so on Friday after Hackage regeneration and the merges the hnix-store-remote 0.4 would work.

What do you recommend. I like doing it proactively and so people also can do it proactively right after releases, and I even went some way to check it directly during development: https://github.com/marketplace/actions/automatic-haskell-project-integration-test-to-nixpkgs. And I use it in normal Haskell repos, but *-store-* is a monorepo, and so far I need to adjust the action code to support Nix monorepos.

But overall. If person arrives with patches right after the release - we have *this problem. And doing/shipping it a 1.5 week later seems like an additional hustle to keep in mind and shedule, and with that it creates that timeframe in-between Hackage release and the Nixpkgs merge when the normal Hackage release may be still marked broken etc in Nixpkgs for a weak or so.

So far, what I made is this action - that moves the Nixpkgs store integration check before the project release, so in the upstream we always have a pulse of clean integration of Hackage release into Nixpkgs.

What we can do to ease the process we have with Hackage&haskell-updates misalignment.
Maybe some way to run a test that takes the Nixpkgs master, autoregenerates Hackage DB, and tries the patch on it.
I would think about these processes and interested in what thoughts you provide.

@Anton-Latukha
Copy link
Contributor Author

Sorry about rambling, sometimes I write too much and I do not know how I can compress it to Shannon information boundary more.

@cdepillabout
Copy link
Member

@Anton-Latukha Hey, sorry, I don't quite understand what needs to be done here.

Could you summarize what needs to happen for hnix-store-remote in order to get it building here in nixpkgs? It looks like hnist-store-remote-0.4 needs hnix-store-core-0.4, but hnix-store-core is locked to version 0.2 in Stackage nightly: commercialhaskell/stackage#5766

I'd suggest trying to fix commercialhaskell/stackage#5766 before trying to get hnix-store-remote working here in nixpkgs.

@Anton-Latukha
Copy link
Contributor Author

Anton-Latukha commented Jan 2, 2021

Ok. Thanks.

We are not even aware hnix-store-core is in Stackage Nightly, and I have no idea why it needs to be there, Store* has no reverse-deps except us, and it is stupid to publish the Nix parser/interpreter to Stackage, especially in this still rough state. And you probably know how it is when people try to prematurely sneak the projects you develop into the Stackage, project flops in and out of Stackage and you take the heat from that rapid oscillation.

base16-bytestring had new epoch (1.0) 3 month ago:

https://hackage.haskell.org/package/base16-bytestring-1.0.1.0/changelog

Which fixed the old return type to normal `Either String ByteString`.

`cryptohash-sha512` is stale for ~10-11 months and testsuite does not suport the
new `base16-bytesting` releases.
@Anton-Latukha Anton-Latukha force-pushed the 2020-12-31-upd-haskell-packages branch from 0730319 to cfce290 Compare January 7, 2021 20:43
@Anton-Latukha
Copy link
Contributor Author

Used hnix-store-core 0_4_0_0 around until Stackage unpin would happen.

sorki said the hnix-store-remote can run the test suite, so with hnix-store-core 0_4_0_0 supplied after the expression regeneration the hnix-store-remote should build fine.

Current patch state addresses the current situation in projects.

@Anton-Latukha Anton-Latukha marked this pull request as ready for review January 7, 2021 20:49
@peti peti merged commit 8e68a5e into NixOS:haskell-updates Jan 8, 2021
@Anton-Latukha Anton-Latukha deleted the 2020-12-31-upd-haskell-packages branch January 9, 2021 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants