Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gnaf-loader: init at 2018-09-10 #46139

Closed
wants to merge 1 commit into from

Conversation

CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented Sep 6, 2018

Motivation for this change

I need some help with this. It's a python script that isn't a Python package. I thought I can use stdenv.mkDerivation but how do I get a python module dependency into it? It relies on psycopg2!

gnaf-loader is a script that loads the GNAF data into a PostgreSQL database. The GNAF database is a database of all postable addresses in Australia and it's used by a lot of GIS applications targeting Australia.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@CMCDragonkai CMCDragonkai changed the title gnaf-loader: init at 2018-08-28 WIP: gnaf-loader: init at 2018-08-28 Sep 6, 2018
@CMCDragonkai CMCDragonkai force-pushed the gnaf-loader branch 4 times, most recently from a9e5af4 to f426d11 Compare September 19, 2018 04:19
@CMCDragonkai CMCDragonkai changed the title WIP: gnaf-loader: init at 2018-08-28 gnaf-loader: init at 2018-08-28 Sep 19, 2018
@CMCDragonkai
Copy link
Member Author

So I have made it work as imports. However I had to do some funny things as makeWrapper doesn't have an option to set the interpreter of the executable it is wrapping. Furthermore the source file isn't even set to have the executable permission. While these things could be fixed upstream, I'm sure I'll deal with more of these sorts of things since I'm packaging a lot of scientific stuff and they are not always standards.

@CMCDragonkai
Copy link
Member Author

@FRidh Could you have a look?

@CMCDragonkai
Copy link
Member Author

One other thing, this script outputs a log file usually placed in the same location as the script. I changed that so it outputs that script to the current directory. I could also point it to /tmp. Not sure what the convention is.

Copy link
Member

@FRidh FRidh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend putting the modules in ${python3.sitePackages}, especially because psma is used as a module.

dontBuild = true;

installPhase = ''
mkdir -p $out/bin
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow FHS inside $out

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you mean here.

chmod u+x,g+x,o+x $out/load-gnaf.py
makeWrapper \
$out/load-gnaf.py \
$out/bin/load-gnaf.py \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

executables should not have an extension

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even when the source script is just load-gnaf.py? Are you suggesting we make the wrapper load-gnaf?

makeWrapper \
$out/load-gnaf.py \
$out/bin/load-gnaf.py \
--set PYTHONPATH "$PYTHONPATH" \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the scripts have a shebang? Add python3.withPackages(ps: with ps; [ psycopg2 ] to buildInputs and patchShebangs will handle it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't have a shebang, that's why I added the shebang in manually with sed.

@CMCDragonkai
Copy link
Member Author

CMCDragonkai commented Sep 19, 2018

The psma is just another python module in the same relative directory. What do you mean by putting it into site-packages? The source repo doesn't use setup.py or anything.

@FRidh
Copy link
Member

FRidh commented Sep 19, 2018

No, it indeed doesn't, and they don't have to. However, if one would take your expression, and install it in a profile, they end up with scripts in the root of their profile, instead of under /bin or /lib.

@CMCDragonkai
Copy link
Member Author

The problem is that the script relies on relative imports and relative paths to the SQL scripts. I'm not sure how to do that with buildPythonApplication or buildPythonPackage.

@FRidh
Copy link
Member

FRidh commented Sep 19, 2018

You don't need to use buildPython* for it. Just create a site-packages folder, and move the modules in there. If you want to use PYTHONPATH, you then add that folder to it.

@CMCDragonkai CMCDragonkai changed the title gnaf-loader: init at 2018-08-28 gnaf-loader: init at 2018-09-10 Sep 20, 2018
@CMCDragonkai
Copy link
Member Author

CMCDragonkai commented Sep 20, 2018

So I can do this these things.

Use: buildInputs = python3.withPackages (ps: with ps; [ psycopg2 ]);

Then inside the installPhase:

mkdir -p "$out/${python3.sitePackages}"
cp __init__.py "$out/${python3.sitePackages}"
cp load-gnaf.py "$out/${python3.sitePackages}"
cp psma.py "$out/${python3.sitePackages}"
cp -r postgres-scripts "$out/${python3.sitePackages}"

The load-gnaf.py still loads postgres-scripts relatively here: https://github.com/minus34/gnaf-loader/blob/master/load-gnaf.py#L276-L279

We still need something in bin to be load-gnaf I'm guessing? How should that be done? Like this?

sed \
  -i \
  '1s;^;#!/usr/bin/env python\n;' \
  "$out/${python3.sitePackages}/load-gnaf.py"
chmod u+x,g+x,o+x "$out/${python3.sitePackages}/load-gnaf.py"
makeWrapper \
  "$out/${python3.sitePackages}/load-gnaf.py" \
  $out/bin/load-gnaf \
  --set PATH '${stdenv.lib.makeBinPath [ postgis ]}'

Or maybe instead:

cat <<EOF > $out/bin/load-gnaf
#!/usr/bin/env sh
exec ${python3.interpreter} "$out/${python3.sitePackages}/load-gnaf.py"
EOF
chmod u+x,g+x,o+x $out/bin/load-gnaf
wrapProgram 
makeWrapper \
  $out/bin/load-gnaf \
  --set PATH '${stdenv.lib.makeBinPath [ postgis ]}'

@Ekleog
Copy link
Member

Ekleog commented Dec 8, 2018

(triage) @CMCDragonkai I personally prefer your first option, though I haven't checked whether it actually corresponded to @FRidh's request. The advantage of it over your second option is that you avoid loading one useless file in memory, a priori.

Actually, I think you may even get rid of the sed by wrapping directly the python executable and providing it default arguments set to pointing to the script.

@mmahut
Copy link
Member

mmahut commented Aug 12, 2019

Are there any updates on this pull request, please?

@CMCDragonkai
Copy link
Member Author

Feel free to take over the PR @mmahut, I'm not working on this atm.

@mmahut
Copy link
Member

mmahut commented Aug 19, 2019

Ok, going to close it for now, if anyone is interesting in taking over, feel free to reopen this.

@mmahut mmahut closed this Aug 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants