wip: openstackclient: 1.7.1 -> 3.11.0 #25898

nlewo · 2017-05-18T17:17:42Z

Dependencies are private and have been generated by pypi2nix.

Motivation for this change

Upgrade

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-05-18T17:17:52Z

@nlewo, thanks for your PR! By analyzing the history of the files in this pull request, we identified @FRidh to be a potential reviewer.

FRidh · 2017-05-18T17:29:56Z

pkgs/applications/virtualization/openstack/openstackclient/default.nix

+  # We don't have build requires
+  doCheck = false;
+
+  propagatedBuildInputs = [


Might as well use propagatedBuildInputs = lib.attrValues generated; considering they'are all propagated anyhow.

yes, it would be better... but i get a strange error when I'm doing this:

nix-build ./ -A openstackclient --show-trace error: while evaluating the attribute ‘propagatedBuildInputs’ of the derivation ‘python-openstackclient-3.11.0’ at /.../nixpkgs/pkgs/development/interpreters/python/mk-python-derivation.nix:56:3: cannot coerce a function to a string, at /.../nixpkgs/pkgs/development/interpreters/python/mk-python-derivation.nix:56:3

I don't know for now why these expressions are not equivalent.

hmm, I have to use fix instead of fix' but I still don't understand (ok, I don't try too much to understand....).

FRidh · 2017-05-18T17:30:04Z

pkgs/applications/virtualization/openstack/openstackclient/default.nix

+    homepage = "http://docs.openstack.org/developer/python-openstackclient";
+    license = "License :: OSI Approved :: Apache Software License";
+    description = "OpenStack Command-line Client";
+  };


maintainer?

FRidh · 2017-05-18T17:31:41Z

pkgs/top-level/python-packages.nix

@@ -4453,7 +4453,7 @@ in {

    propagatedBuildInputs = with self; [
      six keystoneclient prettytable oslo-utils oslo-i18n lxml httplib2 cliff
-      dogpile_cache appdirs anyjson pbr openstackclient
+      dogpile_cache appdirs anyjson pbr pkgs.openstackclient


This is going to cause trouble. What will happen is that all the propagatedBuildInputs of openstackclient will also be added to the PYTHONPATH of this package. Is your intention that it should have openstackclient on PATH?

openstackclient brings python modules that are required by ironicclient. So, it had to be in propagatedBuildInputs. But ironicclient is currently broken because cliff is. I will simply mark ironicclient as broken and explain why in a commen.

In fact, ironicclient should be removed and added to the openstaclient one... but I personally really don't need it.

nlewo · 2017-05-24T15:28:04Z

@FRidh I think I addressed all of your requests. Could you have a look please?

FRidh · 2017-05-30T06:46:45Z

The dependencies here are generated. What is missing is the script/documentation that was used to generate it.

nlewo · 2017-05-30T07:07:35Z

Currently, the only missing thing is that pipy2nix doesn't generate dependencies in a seperated files. I prepared a PR for pypi2nix but I was waiting to get feedback on this current PR in order to see if this is pertinent or not from the nixpkgs point of view.

FRidh · 2017-05-30T07:34:38Z

@nlewo I'm not really fond of having applications use their own dependency package sets because now you also have to make sure (from a security point of view) to keep those dependencies up to date.
Its to me as if you get a statically compiled binary that may use insecure versions of libraries.
As it becomes more common that (larger) applications fix their dependencies, there sometimes is just no other way.

I've just sent a mail to nix-dev about this general issue. I prefer not to make a decision on this by myself because we may be setting a precedent then.

nlewo · 2017-05-30T07:46:50Z

@FRidh Ok, let's continue on nix-dev. FYI, I submitted the PR nix-community/pypi2nix#127

FRidh · 2017-05-30T07:49:35Z

Currently, the only missing thing is that pipy2nix doesn't generate dependencies in a seperated files.

What do you mean here? Do you want a separate file per package? There's no need for that when its generated.

nlewo · 2017-05-30T08:02:37Z

pypi2nix currently generates the python requirements nix expressions in a file that also contains a lot of pypi2nix specific expressions which are useless in the nixpkgs repo. So, I just proposed to generate all of these expressions in one dedicated file. This file is called requirements_generated.nix. It is imported by the main pypi2nix nix expression but could also be used alone in a nixpkgs application.

FRidh · 2017-05-30T08:31:37Z

From the mailing list:

As an example, openstackclient needs a version of the requests
library which is older than our pythonPackages.requests. We then need
to add another version but we don't want to mix different versions in
the python packages set (to avoid conflicts). In this context, it seems
the only way to bring openstackclient was to provide its dependencies
as 'private' dependencies.

If you just need a single (or a few) different dependencies, then its quite straigthforward.
You can define this version in your openstackclient expression, and override the python.pkgs package set to use that version.

nlewo · 2017-05-30T08:56:45Z

The main problem is that the request library is used by other openstack dependencies (directs and transitives) and I then need to also override them. I tryied to do this for keystone (which have a lot of common deps with openstackclient) but we concluded this is not the good way (#25752). This is why I started to use pypi2nix.

FRidh · 2017-05-30T09:11:37Z

@nlewo perhaps I should have been more clear then. Its not ok to add additional versions of packages to python-packages.nix, however, in your expression, you can override python-packages.nix and provide custom versions. E.g.

let
  mypython = python.pkgs.override {
    packageOverrides = self: super: {
      requests = super.buildPythonPackage {
        ...
      }; 
     };
  };
in ...

Then, mypython.pkgs uses the new requests for both direct and transitive dependencies.

nlewo · 2017-06-01T06:23:16Z

@FRidh As mentionned by chat, it seems more complicated to partially use generated expressions than using all of them.
Could you move forward on this PR?

FRidh · 2017-06-01T07:17:42Z

you need to include documentation on how to update these packages

- dependencies are private and have been generated by pypi2nix - ironicclient is marked as broken

nlewo · 2017-06-01T22:53:10Z

@FRidh I reworked how requirments are generated and how they are used in order to minimize manual actions to facilitate the upgrade of this package. I use a patched version of pypi2nix to generate a clean file. I'm currently discussing with pypi2nix maintainers to see how this could be integrated...
The way the pipy2nix file is generated could be largely improved but I would prefer to let these works for future PR:)
So please consider this PR as WIP until we have some news from pypi2nix.

nlewo · 2017-06-13T19:55:17Z

I finally put it into nixpkgs-python for now... Thanks @FRidh for all your comments.

nlewo mentioned this pull request May 18, 2017

Openstack Keystone from Liberty to Ocata #25752

Closed

FRidh requested changes May 18, 2017

View reviewed changes

nlewo force-pushed the pr/openstackclient branch from f01d96e to 7b57543 Compare May 19, 2017 12:01

joachifm added the 8.has: package (update) label May 21, 2017

nlewo force-pushed the pr/openstackclient branch from 7b57543 to 0d90772 Compare May 22, 2017 08:25

nlewo mentioned this pull request May 30, 2017

Python packages nix expressions are written in a separated file nix-community/pypi2nix#127

Closed

openstackclient: 1.7.1 -> 3.11.0

6a883f8

- dependencies are private and have been generated by pypi2nix - ironicclient is marked as broken

nlewo force-pushed the pr/openstackclient branch from 0d90772 to 6a883f8 Compare June 1, 2017 22:45

nlewo changed the title ~~openstackclient: 1.7.1 -> 3.11.0~~ wip: openstackclient: 1.7.1 -> 3.11.0 Jun 1, 2017

nlewo closed this Jun 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wip: openstackclient: 1.7.1 -> 3.11.0 #25898

wip: openstackclient: 1.7.1 -> 3.11.0 #25898

nlewo commented May 18, 2017

mention-bot commented May 18, 2017

FRidh May 18, 2017

nlewo May 19, 2017 •

edited

nlewo May 22, 2017

FRidh May 18, 2017

nlewo May 19, 2017

FRidh May 18, 2017

nlewo May 19, 2017

nlewo commented May 24, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented Jun 1, 2017 •

edited

FRidh commented Jun 1, 2017

nlewo commented Jun 1, 2017 •

edited

nlewo commented Jun 13, 2017

wip: openstackclient: 1.7.1 -> 3.11.0 #25898

wip: openstackclient: 1.7.1 -> 3.11.0 #25898

Conversation

nlewo commented May 18, 2017

Motivation for this change

Things done

mention-bot commented May 18, 2017

FRidh May 18, 2017

Choose a reason for hiding this comment

nlewo May 19, 2017 • edited

Choose a reason for hiding this comment

nlewo May 22, 2017

Choose a reason for hiding this comment

FRidh May 18, 2017

Choose a reason for hiding this comment

nlewo May 19, 2017

Choose a reason for hiding this comment

FRidh May 18, 2017

Choose a reason for hiding this comment

nlewo May 19, 2017

Choose a reason for hiding this comment

nlewo commented May 24, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented May 30, 2017

FRidh commented May 30, 2017

nlewo commented Jun 1, 2017 • edited

FRidh commented Jun 1, 2017

nlewo commented Jun 1, 2017 • edited

nlewo commented Jun 13, 2017

nlewo May 19, 2017 •

edited

nlewo commented Jun 1, 2017 •

edited

nlewo commented Jun 1, 2017 •

edited