Having lpgr, INLA, and ITKR available would certainly be useful.

The CRAN bump should probably be done separately.

Since Github is not even minimally reviewed unlike CRAN/bioc, the script to generate the package set should probably also take commits and hashes as inputs as well as owner and repo for at least security reasons.

Is there anything special about GitHub here or should this be generalized to arbitrary source repos?

Happy to split that out, it may resolve the current breakage with lgpr (although it would then break building posterior). Fixing the StanHeaders build failure is important because many packages depend on it.

I think I agree regarding freezing commit and sha256 for the github packages (instead of refreshing each time DESCRIPTION has a version bump). This logic could be separated to another script. Alternatively, since generate-r-packages.R doesn't delete old entries in e.g. github-packages.nix, we could make it so the list of packages to add or version bump needs to be passed explicitly on the command line (I think I like this idea best).

As for generalizing to other sources of git packages, some of the logic is github specific. For example downloading the raw DESCRIPTION file without cloning the repo is likely github (possibly gitlab) exclusive.

Addendum: This doesn't stop us from generalizing, since we're running nix-prefetch-git we have the DESCRIPTION file locally, so we could presumably use that copy, but we'd have to parse the store path out of the nix-prefetch-git verbose output. But this may be better saved for the next iteration.

I've added commandline handling, fixed the code so old packages are maintained if not passed on the commandline, modified the code to use the parallel map for the git fetches and parsing. And squashed the history.

@peti what are your thoughts on adding this feature?

Thanks for this! We also have a number of proprietary R packages in the internal git repository. Can the PR be extended to support that?

@alexvorobiev It could, but I think we're basically waiting for some signal from @peti that this will be merged at some point or what sort of modifications to make. I think we need to split the CRAN package regeneration into a separate PR at least but other than that this should be ready to go.

@bcdarwin Since the R packages in nixpkgs are updated so infrequently I am working on an overlay that uses the copies of the files from r-modules. I am going to apply the changes from this PR to my overlay because I have an immediate use case for them. I will also try to add our internal CRAN-like repository (we use Artifactory) and some way to install packages from multiple snapshots of MRAN (the new data.table doesn't build for some reason). Thanks!

Are there any updates to this PR or obstacles? In general I think it would be very useful to have mechanisms to

1. provide custom derive* functions (overriding the current ones if needed)
2. additional package derivation lists (my-own-packages.nix)
3. adjust the lists like packagesWithNativeBuildInput

without modifying https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/r-modules/default.nix or https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/r-modules/generate-r-packages.R. I have an internal mirror of CRAN and a number of proprietary packages, so every time there are updates to that file (I copied it to an overlay) I need to merge my changes to the new version.

Hi @peti,

I have discussed this with my colleague @cfhammill and we're willing to become co-maintainers. Realistically we can't promise to refactor this code in the near term but we'll certainly try to stop it from growing further out of control.

(Of course, this PR shouldn't be merged yet due to unresolved issues.)

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/automated-package-updates/12908/9

@peti @bcdarwin @cfhammill I'd also be willing to be a co-maintainer. I've reasonably frequently updated this package set and am a frequent user of R and it's packages in Nix.

As for this pull request, I also feel it's a bit too github specific and that it should be generalised, especially since the repository is cloned anyway to compute a hash so we may as well read the DESCRIPTION file from there.

I have discussed my thoughts about this issue in great detail at https://www.twitch.tv/videos/878608118?t=1h5m36s.

The gist of if it is that IMHO the complexity of generate-r-packages.R is getting out of hand and that I feel less and less able to maintain that code. I am a moderately competent user of R, but I am by no means a developer. I have never written an R package myself. I came up with some simple and straightforward solution to generate our package set a few years ago that worked, but my impression is that this code should really be taken over by someone more knowledgeable than myself.

So, I am perfectly happy to see these changes merged, but I would prefer someone else to take over the responsibility of maintaining of the R package set in that case because I don't want to find myself in a situation where that generate-r-packages.R script no longer works and I have to feel like I'm responsible for fixing it.

If anyone is up to the task, please let me know and I'll make the necessary changes in .github/CODEOWNERS to make sure that all pull requests go through you. Anyone?

Hi @peti -- any word on this? Three people in this thread have responded to your call to volunteer to co-maintain the R packages set. Admittedly no-one has generalized this PR yet but maybe out of a sense that merging of changes to the R infrastructure has somewhat stalled, r-packages is not being regularly evaluated on Hydra, etc. Thanks!!

cc @cfhammill @jbedo

I want to know if I understand this PR. Does the workflow for github packages look like this:

1. generate_r_packages.R github user/repo@commit user2/repo2@commit2 ....
2. Move the output to overwrite github-packages.nix
3. Push the changes to NixOS/nixpkgs

So users wanting to update the github package set do so on a per package basis, with no bulk update of github packages for security reasons?

I frequently use overlays to bring in a package from Github, sometimes specifically the development version of a CRAN package, which needs to override the CRAN version. How would I do that?

Similar to @alexvorobiev, I would like to see functions like buildRpackage exposed, so I can write things like

environment.systemPackages = [
pkgs.rstudioWrapper.override {
  packages = [
    (pkgs.buildRpackage { name="lgpr"; version="0.33.2"; sha256="14i1vf9119cypjnacy43flfzx3lsjhax5zg33h4kjsyjbfl68ngi"; depends=[MASS Rcpp bayesplot ggplot2 ggpubr rstan rstantools]; url = "https://github.com/jtimonen/lgpr"; rev = "5a25fa8a1b21a34f915461802e21f592184d6e67"; }; };)
  ]; 
];

Hi @peti -- any word on this? Three people in this thread have responded to your call to volunteer to co-maintain the R packages set.

Yeah, this is great! Is there anything you guys need from me to get started maintaining the package set? What is blocking you?

r-packages is not being regularly evaluated on Hydra, etc.

Hydra frequently evaluates https://hydra.nixos.org/jobset/nixpkgs/r-updates. One just needs to push the to-be-evaluated code to the r-updates branch, e.g. via a PR.

Yeah, this is great! Is there anything you guys need from me to get started maintaining the package set? What is blocking you?

I think you just need to add some of them to CODEOWNERS for R. It looks like you could add all of @cfhammill @jbedo @bcdarwin. Take yourself off if you want.

Hydra frequently evaluates https://hydra.nixos.org/jobset/nixpkgs/r-updates. One just needs to push the to-be-evaluated code to the r-updates branch, e.g. via a PR.

Thanks, I didn't know that it needed to go to a specific branch.

I added @jbedo and @bcdarwin in 430a043. I didn't add @cfhammill because that user name is not listed in maintainers/maintainer-list.nix and I wasn't sure whether that would be a problem or not.

https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-code-owners
The people you choose as code owners must have write permissions for the repository. When the code owner is a team, that team must have write permissions, even if all the individual members of the team already have write permissions directly, through organization membership, or through another team membership.

You need write access to the repo to get notifications via CODEOWNERS.

I am starting a hydra job for r-updates but it seems that Hydra's config causes it to check out @peti's r-updates branch which is somewhat stale, and I don't have permission to edit the config. Should this be changed to point to my own r-updates (or nixpkgs's [a bit less plausibly])?

I am starting a hydra job for r-updates but it seems that Hydra's config causes it to check out @peti's r-updates branch which is somewhat stale, and I don't have permission to edit the config. Should this be changed to point to my own r-updates (or nixpkgs's [a bit less plausibly])?

Haskell's update branch is off NixOS/nixpkgs, so maybe we should be the same?

how is this pr going on? I've used code from this pr and added a file like this, and source it to get github packages in generate-r-packages.R

I think file like this should be more convenient that call generate-r-packages.R with all github packages

I think it'd be great to have some support for github based r packages, but I'm now not sure they should be part of rPackages. Is there any reason to have them part of the rPackages tree? I'm concerned about conflicts and how to resolve them. Specifically, it's not unusual to have a github hosted R package get listed in CRAN/BIOC, at which point there is a conflict to resolve as it will show up as part of the CRAN/BIOC package set automatic update.

Another option to consider is to just manually manage these github packages outside of the automatically generated rPackages tree, like any other package in nixpkgs. This would allow us to leverage tools like nix-update instead to keep these github packages up to date, rather than reimplementing the same functionality in generate-r-packages. It also would allow an opportunity to properly fill out the metadata (including importantly maintainers), something that is currently missing from the generated rPackages and that I'm keen on fixing.

One final though is that many use cases might be covered by an r2nix tool, or with a recursive nix expression that parses DESCRIPTION. We can't consider recursive nix as part of nixpkgs but it would allow the easy addition of arbitrary packages as part of an R environment.

@jbedo I see where you're coming from. It was so long ago now, but I think the order preference was CRAN, BioC, then git for the case of cross-listed packages. This issue would already be occurring with many packages appearing both on CRAN and in BioC.

I think it is probably preferable to append the git packages to the rPackages tree, perhaps with a e.g. .gitPackages subtree. Nice to keep everything in one place in my opinion. Its tough to know at this moment the scale of non CRAN non BioC packages that people would want though, if it's 5 then maybe a nix file per package is fine, but if there are hundreds maybe not. What do you think?

This issue would already be occurring with many packages appearing both on CRAN and in BioC.

It's part of BioC naming policy to avoid naming collisions with CRAN so this should be very infrequent (from https://www.bioconductor.org/developers/package-submission/):

Packages should be named in a way that does not conflict (irrespective of case) with any current or past BIOCONDUCTOR package, nor any current CRAN package.

Currently there is one package (interacCircos) that is both in CRAN and BioC, and it probably should be reported upstream to resolve the issue.

I think it is probably preferable to append the git packages to the rPackages tree, perhaps with a e.g. .gitPackages subtree. Nice to keep everything in one place in my opinion. Its tough to know at this moment the scale of non CRAN non BioC packages that people would want though, if it's 5 then maybe a nix file per package is fine, but if there are hundreds maybe not. What do you think?

Even if it's hundreds I don't think it will be an issue, after all most of nixpkgs is handled this way. I still feel like generate-r-packages is the wrong machinery for updating github hosted packages and that other existing tooling (nix-update) is more suitable. This also already handles non-github hosted repositories (at least gitlab).

I also agree with you and think it's useful to have them in grouped in a subtree not top-level, maybe in another top-level tree called rPackagesExtra so there's a clear distinction between the CRAN/BioC repositories and packages hosted elsewhere. I'm not keen on a subtree of rPackages as we may end up with a conflict if a CRAN/BioC package with the same name appears.

Ok all sounds good to me. I will take a stab at redoing this PR, adding a top level rPackagesExtra tree with up to date versions of the packages above.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-there-a-way-to-add-r-packages-straight-from-github/17871/2

@cfhammill - do you have a branch started for adding a top-level rPackagesExtra? I'm interested in picking this up again.

@nviets I haven't been working on it, if you wanted to start a new PR with the feature I'd be happy to review

Thanks for confirming. I'll try to take a look soon.