is this meant to be used an application or python package? (do you just do $ databricks-connect or do you need to do import databricks-connect in python?)

according to https://docs.databricks.com/dev-tools/databricks-connect.html it looks like it's meant to be used as a cli tool

if it's only meant to be used from the cli, please use https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#buildpythonapplication-function as this allows for separation of python environments

EDIT:
an example is ansible #95683 , where it's available both as a python package and as a cli utility

You use it as a python package. But have some applications for setup.

Normal user case is:

1. $ databricks-connect configure configure connection to databricks cluster
2. $ databricks-connect test test environment and connection to cluster
3. Then you use it as a package in your python script. But with the name pyspark.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

Please follow CONTRIBUTING.md and manual#submitting-changes-making-patches and squash the fix-up commits.

in this case, you should have two commits:

maintainers: ...
pythonPackages.databricks-connect: ...

Have now clean up the commits, so now there are three commits:

licenses: add databricks
maintainers: add kfollesdal
pythonPackages.databricks-connect: init at 7.1.0

@jonringer thanks for feedback and help so fare. I have one problem/question left.

If I make a nix-shell with the following script:

with import ./nixpkgs { };

pkgs.mkShell rec {
  buildInputs = [
    python37Packages.databricks-connect
  ];
}

then using the python package pyspark and running the cli cmd pyspark works fine.

But if I install an environment with nix-env -if and the file:

with import ./nixpkgs {};

python37.withPackages (ps: with ps; [ databricks-connect  ])

Then I have to manually set the environment variable JAVA_HOME to get pyspark python package and pyspark cli command to work.

Do you have any clue what the problem can be? I think it can be something with the wrapping.

UPDATE
Have now figured out that you do not have to set JAVA_HOME manually after installing pythonPackages.databricks-connect. If you install withnix-env and the following file:

with import ./nixpkgs {};
buildEnv {
  name = "bricks";
  paths = [
    (python37.withPackages (ps: with ps; [ databricks-connect  ]))
    jdk
  ];
}

Then everything works fine.

But as you can see I have to add jdk in the environment. That is strange since it is already added as an dependency in the expression for pythonPackages.databricks-connect. Any advice to how I can avoid this?

sorry, I sorted python-packages, can you rebase? :)

oh, i forgot to respond

JAVA_HOME currently will only be exposed to the builder.

For it to appear in a nix-shell, you would need to do:

mkShell {
  inputsFrom = [ python3Packages.databricks-connect ];
}

however, inputsFrom has other implications, it's easier to just add jdk like you did above

thanks @kfollesdal for opening your first nixpkgs PR :)

Thanks @jonringer, I hope I can contribute more.

Regarding the nix-shell and nix-env, I think you misunderstood my question.

nix-shell works as expected with the following expression:

with import ./nixpkgs { };

pkgs.mkShell {
  buildInputs = [
    python37Packages.databricks-connect
  ];
}

But if I install databricks-connect with nix-env with the
following expression:

with import ./nixpkgs {};
buildEnv {
  name = "bricks";
  paths = [
    (python37.withPackages (ps: with ps; [ databricks-connect  ]))
  ];
}

Then it do not work as expected. But if I add jdk to paths, like so:

with import ./nixpkgs {};
buildEnv {
  name = "bricks";
  paths = [
    (python37.withPackages (ps: with ps; [ databricks-connect  ]))
    jdk
  ];
}

Then everything works fine as in the nix-shell case. I think this is strange since
jdk is already added as an dependency to databricks-connect. Is it anyway we can fix this
so it works as expected?

I think your issue is buildEnv which I believe just symlink joins the outputs, and will disregard stuff passed to mkDerivation

Ok, thanks. I will look more into it :-)