Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mapproxy: init at 1.12.0 #68628

Merged
merged 1 commit into from Feb 6, 2020
Merged

mapproxy: init at 1.12.0 #68628

merged 1 commit into from Feb 6, 2020

Conversation

Rakesh4G
Copy link
Contributor

@Rakesh4G Rakesh4G commented Sep 13, 2019

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @CMCDragonkai

@Rakesh4G
Copy link
Contributor Author

The test cases are disabled because
Dependency list is huge.
https://github.com/mapproxy/mapproxy/blob/master/requirements-tests.txt
There are security issues with package Riak
#33876
#56480

@CMCDragonkai
Copy link
Member

I've done some checks in the source code regarding the usage of libraries. I found that:

  • There's a MAPPROXY_LIB_PATH that can be used to add paths to C .so objects or dlls so that the system can load C libraries to bind to.
  • The 2 C libraries that it makes use of are: libproj and libgdal. These are not necessary if the 2 python fallbacks are available: pyproj and gdal.

So in the example derivation I pasted above, it just has pyproj and gdal.

When I tried building locally with nix-build:

{
  pkgs ? (import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/83e01aa5e48a9107f3b988c9d4c1c21d610626fb.tar.gz) {}),
  pythonPath ? "python37"
}:
  with pkgs;
  let
    python = lib.getAttrFromPath (lib.splitString "." pythonPath) pkgs;
  in
    python.pkgs.buildPythonPackage rec {
      pname = "MapProxy";
      version = "1.12.0";

      src = python.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "622e3a7796ef861ba21e42231b49c18d00d75f03eaf3f01a2b7687be7568e2ec";
      };

      propagatedBuildInputs = with python.pkgs; [
        pillow
        pyyaml
        pyproj
        shapely
        gdal
        lxml
      ];

      doCheck = false;

      meta = with lib; {
        description = "MapProxy is an open source proxy for geospatial data";
        homepage = https://mapproxy.org/;
        license = licenses.asl20;
        maintainers = with maintainers; [ rakesh4g ];
      };
    }

Every works fine. But I use the callPackage pattern required by python-modules in Nixpkgs, I end up with this error:

nix-build ./default.nix --attr pkgs.python3Packages.mapproxy
»» ~/Projects/nixpkgs
 ♖ ./result/bin/mapproxy-util --version
Traceback (most recent call last):
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/bin/.mapproxy-util-wrapped", line 7, in <module>
    from mapproxy.script.util import main
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/script/util.py", line 28, in <module>
    from mapproxy.script.conf.app import config_command
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/script/conf/app.py", line 32, in <module>
    from .sources import sources
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/script/conf/sources.py", line 17, in <module>
    from mapproxy.srs import SRS
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/srs.py", line 26, in <module>
    from mapproxy.proj import Proj, transform, set_datapath
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/proj.py", line 262, in <module>
    raise ImportError('could not find libproj or pyproj')
ImportError: could not find libproj or pyproj

@CMCDragonkai
Copy link
Member

CMCDragonkai commented Oct 2, 2019

Why would using the call package pattern result in such a difference? Given that my out-of-tree default.nix uses an older nixpkgs hash, it just means setuptools is part of the package environment, whereas the current nixpkgs master doesn't have that.

Furthermore this is not the only problem. Even though my bin/mapproxy-util works, it fails to work as server due to probably the application considering that bin/mapproxy-util is a python script and trying to run it with python, but it fails with a syntax error because the bin/mapproxy-util is a wrapped shell script due to Nix.

»» ~/Projects/mapproxy/mymapproxy
 ♖ ../result/bin/mapproxy-util serve-develop mapproxy.yaml
[2019-10-02 13:22:18,621] mapproxy.config - INFO - reading: /home/cmcdragonkai/Projects/mapproxy/mymapproxy/mapproxy.yaml
[info]  * Running on http://127.0.0.1:8080/ (Press CTRL+C to quit)
[info]  * Restarting with reloader
  File "/nix/store/bfvw32p8c61jajjm7a8z98xbk1n6qiwz-python3.7-MapProxy-1.12.0/bin/mapproxy-util", line 2
    export PATH='/nix/store/kimimvhimclyfzlncpg36zjni3wn70nq-python3-3.7.3/bin:/nix/store/bfvw32p8c61jajjm7a8z98xbk1n6qiwz-python3.7-MapProxy-1.12.0/bin:/nix/store/rpayih9fb4ms2wjvmyn6kp7kmsizim49-python3.7-setuptools-40.8.0/bin:/nix/store/kp0g7z5h4kr6gx71jf9q0a9322p4jr5p-python3.7-numpy-1.16.2/bin:/nix/store/7vgnq00rxw9xh03920apdyj4ibgsmrix-gdal-2.4.0/bin'${PATH:+':'}$PATH
              ^
SyntaxError: invalid syntax

I've reported this problem on mapproxy: mapproxy/mapproxy#424

@CMCDragonkai
Copy link
Member

CMCDragonkai commented Oct 2, 2019

@Rakesh4G Also this is more of an application, not a package. Maybe we should be changing it to buildPythonApplication? Can you try changing this PR structure to instead mimic dvc's style: #54530

Notice where the default.nix is placed. It's not in python-modules.

@ofborg ofborg bot requested a review from CMCDragonkai October 2, 2019 05:02
@ofborg ofborg bot removed the 6.topic: python label Oct 24, 2019
buildInputs = [
pillow
pyyaml
python3Packages.pyproj
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for $python3Packages here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

};

prePatch = ''
substituteInPlace mapproxy/util/ext/serving.py --replace "args = [sys.executable] + sys.argv" "args = sys.argv"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

throwing away the location of the interpreter seems incorrect to me

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FRidh , Please suggest what should be the alternate to this. This done to get rid of the problem mapproxy/mapproxy#424

Thanks in advance.

pkgs/applications/misc/mapproxy/default.nix Show resolved Hide resolved
@sikmir
Copy link
Member

sikmir commented Jan 2, 2020

Is there any progress on it?

@Rakesh4G
Copy link
Contributor Author

Rakesh4G commented Jan 7, 2020

Is there any progress on it?

I am still waiting on some guidance to alternate options due to
#68628 (comment)

@Rakesh4G
Copy link
Contributor Author

Rakesh4G commented Jan 14, 2020

This can not be merged as is.
With Latest pyproj mapproxy breaks.
details here:
mapproxy/mapproxy#417

So we might need to add pyproj version - 1.9.6, With this version Mapproxy works.
@CMCDragonkai @FRidh

@CMCDragonkai
Copy link
Member

If pyproj 2 is way different from pyproj1, we might need to keep a pyproj1 around in a separate derivation.

@Rakesh4G
Copy link
Contributor Author

If pyproj 2 is way different from pyproj1, we might need to keep a pyproj1 around in a separate derivation.

@CMCDragonkai , OK. I will create a new PR to add support for pyproj1 as separate derivation.

@Rakesh4G
Copy link
Contributor Author

Added PR for pyproj1:
#78092

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, please use an indent width of 2, otherwise the package looks good to me

pkgs/applications/misc/mapproxy/default.nix Show resolved Hide resolved
@jonringer
Copy link
Contributor

should be just 1 commit, please squash.

git reset HEAD^^
git add pkgs/
git commit --amend --no-edit
git push .. .. --force

@Rakesh4G
Copy link
Contributor Author

Rakesh4G commented Feb 6, 2020

should be just 1 commit, please squash.

git reset HEAD^^
git add pkgs/
git commit --amend --no-edit
git push .. .. --force

ok.

@Rakesh4G Rakesh4G changed the title pythonPackages.mapproxy: init at 1.12.0 mapproxy: init at 1.12.0 Feb 6, 2020
@jonringer
Copy link
Contributor

@GrahamcOfBorg build mapproxy

@Rakesh4G
Copy link
Contributor Author

Rakesh4G commented Feb 6, 2020

Hi @jonringer , sorry i made one mistake. it is supposed to be a buildPythonApplication not buildPythonPackage. I am just updating this in a minute.

@jonringer
Copy link
Contributor

@GrahamcOfBorg build mapproxy

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
executables show usage
has tests 👍
upstream seems somewhat active

[2 built, 0.0 MiB DL]
https://github.com/NixOS/nixpkgs/pull/68628
1 package built:
mapproxy

@jonringer jonringer merged commit 21b11bf into NixOS:master Feb 6, 2020
@Rakesh4G Rakesh4G deleted the mapproxy-python branch February 7, 2020 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants