Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Survey of installed Python dependencies (requirements.txt, commands.json, tox.ini, etc.) #28809

Closed
foolip opened this issue May 4, 2021 · 1 comment
Labels

Comments

@foolip
Copy link
Member

foolip commented May 4, 2021

Similar to #28801, I've had a look at what Python dependencies we install, rather than vendoring them into tool/third_party/. These are the mechanisms I'm aware of:

  • requirements.txt and requirements_*.txt files
  • install fields in commands.json files (and requirements fields which can refer to requirements.txt files)
  • deps sections in tox.ini files (which often refer to requirements.txt files)
  • direct calls to pip install in scripts or CI steps

Here are the dependencies, starting with some special groups and then alphabetically...

marionette_driver + mozcrash + mozdebug + mozinstall + mozleak + moznetwork + mozprofile + mozrunner + mozversion

These are dependencies for Firefox required only by tools/wptrunner/requirements_firefox.txt. (Some others are used more broadly, see below.)

flake8 + pep8-naming + pycodestyle + pyflakes

Dependencies for Python coding style in tools/tox.ini, via tools/requirements_flake8.txt. Not installed by any ./wpt command.

mypy + mypy-extensions + typed-ast

Dependencies for Python type checking in tools/tox.ini, via tools/requirements_mypy.txt. Not installed by any ./wpt command.

pytest + pytest-cov + hypothesis + mock

Dependencies for tests for tools/ in tools/tox.ini, via tools/requirements_pytest.txt. These are not the same as the pytest + dependencies used to run webdriver/tests/, those are separately vendored with different versions, see #28801.

requests + taskcluster + pyyaml + json-e + jsonschema

Dependencies in tools/tox.ini used in CI. Not installed by any ./wpt command. These aren't pinned, but I haven't seen things suddenly break because of them.

Sphinx + sphinx-argparse + recommonmark

These are in docs/requirements.txt, dependencies for building web-platform-tests.org, wired up to ./wpt build-docs but in CI that isn't used.

Template-Python + html5lib + lxml + mercurial + webencodings

These are the dependencies for css/build-css-testsuites.sh via css/requirements.txt. Seemingly not run in CI?

aioquic

Dependency for ./wpt serve-quic-transport in tools/quic/requirements.txt, but this is also a vendored dependency, see #28801. The versions aren't the same, which one ends up being used?

html5lib

In tools/wptrunner/requirements.txt, but is also a vendored dependency, see #28801. Notably, the webencodings dependency isn't in tools/wptrunner/requirements.txt...

Also in css/requirements.txt, see above.

Also in resources/test/tox.ini, used to extract inline test expectations.

mozinfo

In tools/wpt/requirements.txt and tools/wptrunner/requirements.txt, but seems like it's ultimately only used for Firefox, since it's related to mozinfo.json files that are probably only for Firefox.

mozinstall

Required by ./wpt install, via tools/wpt/commands.json.

mozlog

In tools/wptrunner/requirements.txt, used for logging in lots of places.

mozprocess

This is listed in many files:

tools/wptrunner/requirements_android_webview.txt
tools/wptrunner/requirements_chrome.txt
tools/wptrunner/requirements_chrome_android.txt
tools/wptrunner/requirements_chrome_ios.txt
tools/wptrunner/requirements_edge.txt
tools/wptrunner/requirements_edge_chromium.txt
tools/wptrunner/requirements_epiphany.txt
tools/wptrunner/requirements_firefox.txt
tools/wptrunner/requirements_ie.txt
tools/wptrunner/requirements_opera.txt
tools/wptrunner/requirements_safari.txt
tools/wptrunner/requirements_sauce.txt
tools/wptrunner/requirements_servo.txt
tools/wptrunner/requirements_webkit.txt

All the usage is with mozprocess.ProcessHandler, in these files:

tools/wptrunner/wptrunner/browsers/chrome_android.py
tools/wptrunner/wptrunner/browsers/firefox.py
tools/wptrunner/wptrunner/browsers/servodriver.py
tools/wptrunner/wptrunner/executors/executorservo.py
tools/wptrunner/wptrunner/webdriver_server.py

Maybe this can be simplified somehow?

pillow

A dependency in tools/wptrunner/requirements.txt needed for RefTestImplementation.

psutil

From tools/wptrunner/requirements_safari.txt, used for killing Safari processes.

Also in tools/wptrunner/requirements_firefox.txt but seemingly not used for running Firefox?

requests

In these files:
tools/wpt/requirements.txt
tools/wptrunner/requirements.txt
tools/wptrunner/requirements_sauce.txt

Used for requests.get and requests.post as you'd expect.

Also in resources/test/tox.ini and tools/tox.ini, but see #28777.

Also in tools/ci/commands.json, tools/docker/commands.json and tools/wpt/commands.json.

selenium

A dependency of anything built on tools/wptrunner/wptrunner/executors/executorselenium.py:

tools/wptrunner/requirements_edge.txt
tools/wptrunner/requirements_ie.txt
tools/wptrunner/requirements_opera.txt
tools/wptrunner/requirements_sauce.txt

six

In tools/wptrunner/requirements.txt, #28776 tracks getting rid of wpt-internal usage. (It would still reamain as a transitive in dependency.)

Also in css/requirements.txt, see above.

Also in resources/test/tox.ini, but see #28777.

ua-parser

The only dependency in tools/wave/requirements.txt, used in tools/wave/utils/user_agent_parser.py.

urllib3

Listed in tools/wptrunner/requirements.txt, it's a dependency of selenium and requests, never used directly in wpt.

virtualenv

The only dependency for the ./wpt CLI beyond Python itself. Installed in CI, but no specific version is required. #20019 discusses avoiding this dependency.

wptserve

Listed in annotation-protocol/requirements.txt and can presumably be removed since wptserve is part of wpt now, see #28810.

zstandard

Required by ./wpt run, via tools/wpt/commands.json.

@foolip foolip added the infra label May 4, 2021
foolip added a commit that referenced this issue May 4, 2021
This lists only wptserve, which is part of this repo. This
requirements.txt file does not seem to be referenced anywhere in the
repo, so there should be nothing more to clean up.

Discovered as part of #28809.
@foolip
Copy link
Member Author

foolip commented May 4, 2021

@jgraham pointed out install sections in commands.json files, so I updated the description, but didn't list every last dependency. pygithub and pyyaml are also in use.

@foolip foolip changed the title Survey of installed Python dependencies (requirements.txt, tox.ini, etc.) Survey of installed Python dependencies (requirements.txt, tox.ini, commands.json, etc.) May 4, 2021
@foolip foolip changed the title Survey of installed Python dependencies (requirements.txt, tox.ini, commands.json, etc.) Survey of installed Python dependencies (requirements.txt, commands.json, tox.ini, etc.) May 4, 2021
@foolip foolip closed this as completed May 4, 2021
Ms2ger pushed a commit that referenced this issue May 18, 2021
This lists only wptserve, which is part of this repo. This
requirements.txt file does not seem to be referenced anywhere in the
repo, so there should be nothing more to clean up.

Discovered as part of #28809.
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue May 20, 2021
…txt, a=testonly

Automatic update from web-platform-tests
Delete annotation-protocol/requirements.txt

This lists only wptserve, which is part of this repo. This
requirements.txt file does not seem to be referenced anywhere in the
repo, so there should be nothing more to clean up.

Discovered as part of web-platform-tests/wpt#28809.

--

wpt-commits: d3c0946cde393e0308bc34fffe4f60fdf5151be7
wpt-pr: 28810
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant