Ok, got a clean build of apache-airflow. This is ready for review.

This looks to me better than the original as it removes the need for custom patching. LGTM

This is a semi-automatic executed nixpkgs-review which is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 106696 run on x86_64-linux 1

@bhipple @costrouc @ingenieroariel Can anyone of you review this? It's a big change and it worries me that it seems to bitrot already, while there a security issue with the whole thing.

This is a semi-automatic executed nixpkgs-review which is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 106696 run on x86_64-darwin 1

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

apache-airflow:

substituteStream(): WARNING: pattern 'jinja2>=2.10.1, <2.11.0' doesn't match anything in file 'setup.py'

This is a semi-automatic executed nixpkgs-review which is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 106696 run on x86_64-linux 1

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

apache-airflow:

substituteStream(): WARNING: pattern 'jinja2>=2.10.1, <2.11.0' doesn't match anything in file 'setup.py'

substituteStream(): WARNING: pattern 'jinja2>=2.10.1, <2.11.0' doesn't match anything in file 'setup.py'

@bhipple @costrouc @ingenieroariel Can anyone of you review this? It's a big change and it worries me that it seems to bitrot already, while there a security issue with the whole thing.

@mweinelt I can run the latest version this weekend to verify that the binary and libraries work as expected. Airflow has a lot of dependencies and I am afraid this kind of work will be needed before every NixOS release. :/

Airflow has a lot of dependencies and I am afraid this kind of work will be needed before every NixOS release. :/

This is not about a pre-release bump, this is about at least three CVEs that need to be addressed during the release cycle. Please see #106671.

@mweinelt thanks for clearing up my confusion.

@drewrisinger I had to add this commit for it to build in OSX: piensa@fa9fc16

I used this shell.nix to try things out:

let
  pkgs = import (fetchTarball "https://github.com/piensa/nixpkgs/archive/dr-pr-update-airflow.tar.gz") {};
in pkgs.stdenv.mkDerivation rec {
  name = "composer";
  src = ./.;
  buildInputs = with pkgs; [
    apache-airflow
  ];
}

Once it built, I created a sample dag, by creating a dags subfolder within it and this file in there simple.py:

from __future__ import print_function

import datetime

from airflow import models
from airflow.operators import bash_operator
from airflow.operators import python_operator


default_dag_args = {
    # The start_date describes when a DAG is valid / can be run. Set this to a
    # fixed point in time rather than dynamically, since it is evaluated every
    # time a DAG is parsed. See:
    # https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date
    'start_date': datetime.datetime(2021, 1, 18),
}

# Define a DAG (directed acyclic graph) of tasks.
# Any task you create within the context manager is automatically added to the
# DAG object.
with models.DAG(
        'daily_hello_world',
        schedule_interval=datetime.timedelta(days=1),
        default_args=default_dag_args) as dag:
    def greeting():
        import logging
        logging.info('Hello World!')

    # An instance of an operator is called a task. In this case, the
    # hello_python task calls the "greeting" Python function.
    hello_python = python_operator.PythonOperator(
        task_id='hola',
        python_callable=greeting)

    # Likewise, the goodbye_bash task calls a Bash script.
    goodbye_bash = bash_operator.BashOperator(
        task_id='chao',
        bash_command='echo Goodbye.')

    # Define the order in which the tasks complete by using the >> and <<
    # operators. In this example, hello_python executes before goodbye_bash.
    hello_python >> goodbye_bash

With that, I set the airflow home to the folder where I had the checkout (not a great practice, but works for this test):
export AIRFLOW_HOME="$(pwd)"
and created the sqlite database needed before starting airflow:

airflow db init

This creates a few files, but one of them has a value that does not work with the Werkzeug version in this PR, see: apache/airflow#11873

so we need to manually change the following in the generated airflow.cfg:

# Set samesite policy on session cookie
cookie_samesite = Lax

and then run the webserver and navigated to localhost:8080 to find this:

In order for the sample task to actually run, we need to run airflow scheduler on a new terminal and using the same folder / env vars.

Then we can trigger the task manually on the UI:

and wait for evidence on the scheduler output that the task is running:

Functionality wise, and with some small modifications this PR looks good to me.

Taking another look at this for #122042, @drewrisinger are you still available to rebase? It looks like the only outstanding merge concern a few months ago was missing maintainers on a few python dependencies. You can add me to those.

In the meantime there's been an Airflow 2.0.X release, which seems to change around some dependencies:
http://airflow.apache.org/docs/apache-airflow/stable/changelog.html

I'd be fine with rebasing/polishing this PR as-is and merging, since it seems strictly better than what we have currently; but going to 2.X would be nice at some point too. It's a shame the dependency closure of Airflow is such a mess.