Further constrain dependency versions in setup.py and requirements.txt #934
+68
−16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some of the TFQ transitive dependencies have recently gotten Dependabot alerts to be updated for some security flaws. However running pip-compile again does not end up updating those dependencies. It seems that the recommended way to handle this is to use a constraint file. This updates `./scripts/generate_requirements.sh` to invoke pip-compile with a constraint file. It also tells pip-compile to write the name of this script into the header of the requirements.txt file it generates.
This increases the versions of a few transitive dependencies so that pip-compile generates higher versions in the requirements.txt, as a way of addressing some security warnings in the dependencies.
Latest version from running `./scripts/generate_requirements.sh`.
Use an array variable for the constraint argument, as a way of both keeping with better practices and using the same style for all command argument variables in this script.
mhucka
changed the title
Testing on the production version of Colab ran into some more version issues, such as packages whose installation led to pip trying to install NumPy 2.x instead of 1.x. This adds a few more version constraints.
mhucka
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses three needs with respect to dependency management and updates both
setup.pyandrequirements.txtto add some constraints on a few dependencies in order to make installations trouble-free.In some environments, we need to constrain the versions of certain Python packages required to pip-install TFQ in order to avoid pip dependency resolution errors. The two problems that I'm seeing on Linux and Colab are (a) pip may try to install a version of the dependency that requires a higher version of Python than what the user is running, an (b) pip tries to install or build NumPy 2 because some dependency version constraints NumPy to be >2. To deal with this, this PR adds a few additional version constraints to
setup.pyfor packages that end up needing to be installed when installing TFQ. These additions tosetup.pyare the minimum version constraints that produced error-free installations in my testing.Similar dependency version constraints need to be specified for developers installing dependencies using
requirements.txt. However, a better approach forrequirements.txtis to use a minimalrequirements.infile in combination withpip-compileto write the finalrequirements.txt. The resultingrequirements.txtfile leads to more predictable build & test environments.A shortcoming of generating
requirements.txtwith version pins is that the versions of some transitive dependencies may need to be updated to address security releases. Simply re-runningpip-compileto produce a newrequirements.txtmay not update the versions because the dependencies that bring in the transitive dependencies have not themselves changed. The recommended practice for this situation is to put the version constraints into a separate constraint file and pass it as an argument topip-compile. (pip-compilehas explicit support for this.) Some additions to the scriptscripts/generate_requirements.shencapsulate and document this process.