Google removed a meta package that installed all of the packages (meta package google-cloud) without having to specify them individually. Would this be worth adding?

How do you envision maintaining, that is, updating these packages? @r-ryantm only updates the source, whereas we might want to update more things. Maybe it's an idea to generate with your tool generated.nix, and then use .overridePythonAttrs in default.nix? There is a memory consideration here but it would allow for more automated maintenance.

Meta packages are not worth adding until another package requires them.

Definitely something that I would be interested looking into. You are talking about doing something similar to how haskell is handled in nixpkgs? So as of right now I can think of two issues.

1. Currently there is no pattern followed for naming python packages e.g. is it (google-cloud-datastore vs google_cloud_datastore)? This may break some peoples setups but it would allow automation of package names and searching for packages easier if we had a rule for naming package attributes and folder names. I could see name.lowercase().replace('_', '-').replace('.', '-') . My tool had a hard time getting the package names correct.

2. Running tests. I had to manually add all the tests and the dependencies. There are just too many ways that testing is done in python. I can't think of any solution to this problem. But if we are handling updates this might not be a problem.

I think that 1 is the main blocker. We would just need to establish a convention. With that advice I will not add the meta package google-cloud.

1. the naming guideline is to use normalized names, but many don't follow it.
2. That we indeed cannot automate yet.

For Haskell we generate a whole package set in one file, and then have a file with overrides per Haskell/GHC version. This is something that was discussed in the past as well for Python. In the end, I gave up on that direction. For Haskell we base our package sets on the curated Stackage sets. For Python, such sets don't exist so we need to resolve a stable set first. When resolving, we need to consider all Python versions and platforms. Unfortunately, it is not possible to retrieve PyPI at a certain state, making it even harder to resolve. Now, these problems can be resolved, one by one.

What I was thinking of now was maybe a generated.nix and default.nix file per package, but I am not sure whether it is worth making this change.

I know this would require a lot of rebuilds and would possibly break people's configuration but I think that getting all the python package names in nixpkgs normalized is a first step before update automation could be considered.

I think that a generated.nix is very doable and a great idea! With correct normalized package naming the only thing this tool this tool misses is extra packages and checkInputs. default.nix would get a lot shorter and simpler to write.

I would be willing to work on normalizing python package attributes and directories. And later possibly a generated.nix for packages.

Bump. Ready for merge. Only added new packages and will allow access to the newer google-cloud python apis.

@Mic92 showed me nix-review was now able to confirm that all packages work without nix-build -A <attribute> over and over. Ready for merge.

Rebased to fix merge conflict. @FRidh is there anything that I can do to help speed up the python modules merge process? Should I ask others for reviews on python contributions? How can I help? I understand you are quite busy and I don't want to make work harder for you. I have tried to combine commits so that the review process is quicker but it doesn't always seem that works. I am committed to nixpkgs and I want to see it succeed.

nix-review has given me much more confidence in my PRs as before I was nix-build -A ... for each attribute manually. So I can say from this point on I will have a much greater assurance that my PRs are good.

@Mic92 do you mind if I ping you on python merge requests that take awhile to merge? Usually @FRidh is the only one that gets pinged.

@costrouc Since I have installed github-refined, I will see pull requests order by last update. This means I will also see older pull request that got updated. I take this list as a priority. I also look at pings but not as regular as this one. Apart from that you can also ask around in IRC. Usually it is also a good idea to keep your pull requests smaller then this one so one can review it faster.

Will do in the future!