Add package versions + commit hash of repo to logs

[RobotSail]

When trying to debug why various training runs don't work, often times it's important to be able to replicate someone's training scenario.
To help with this, we should add a log step where we capture the exact packages being used (pip freeze) and also the commit hash of the training library that was ran at the time.
This way, we can go back and replicate someone's exact scenario.

[fynnsu]

While I think it could be useful to auto log some of this info, I think another route we can consider is asking users to run something like this if they're submitting a bug report:

Note: this is the script PyTorch uses for this purpose.

wget https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

[RobotSail]

@fynnsu While I appreciate your thoughts, a major issue arises when training many models at scale with an evolving codebase where there are many potential things which could fail. One thing we often find is that we need to replicate older experiments and struggle to recreate the same results. So having the exact hash of the repo + versions used at that point in time is absolutely essential for having reproducibility

[fynnsu]

@RobotSail Yeah that makes sense, but I think there are two different use cases for this kind of collection (and perhaps you just intended this issue to focus on the first one, which is fair):

1. Internal usage logging run config / package versions for reproducibility
2. External users raising bug reports

I agreed that for 1. we need automated log collection in a standardized format, etc.

For 2. I think it is easier and more likely to be successful if we have a consistent way for external users to report their environment alongside bug reports, however, not all external users will be collecting full logs, or want to share full training logs for the purpose of reporting a bug. And I think that's where a script like this is useful.

[RobotSail]

@fynnsu I don't see how this wouldn't be useful for both 1 and 2

[fynnsu]

It can be useful for both. I merely meant we may need something alongside that because it's not guaranteed that this "log step" will run for:

• application code that uses the library as an sdk and just calls a few functions
• application code that sets up logging differently / manages logging itself

[RobotSail]

The more logging we have the better

[JamesKunstle]

@fynnsu thinks we get this for free with some backends but it should be added manually into our logging so that's available consistently

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant!

[RobotSail]

Wandb provides this directly through training logs, but we may eventually want to move that over into this.