Runs with precision like FP4 converges much slower than RCP so how to deal with that?

[ShriyaRishab]

Since convergence is very different for FP4 vs say BF16/FP32 - how to check if convergence is indeed correct or super lucky or cherry-picked for some results?

[ShriyaRishab]

12/3 WG meeting: Reference code RCPs are generally Bf16 lately, but its quite different from say FP4.

1. Could we add reference RCPs for smaller precisions also?
2. For submitters who distribute their code to partners, if their submission takes more tokens to converge than RCPs, the distributer of the codebase would need to create RCPs using the distribution codebase and the lower precision. Instead of full RCPs, maybe create a statistical M0 runs at min BS and max BS which shows numerical relationship of how many more approx tokens are needed for convergence wrt RCPs.
3. If there are multiple similar BS submissions which all use the same codebase, then we can compare those against each other and it would be easier to flag extreme cherry-picking cases. On the other hand, if there are very few submissions at some BS, this method would not work that well.
4. Should all <=fp4 submissions require M>N submissions, maybe M=2N?

Gemini-generated summary
The FP4 Loophole: Lower precision runs (like FP4) can converge slower than the Reference Convergence Points (RCPs) but achieve faster throughput, creating a loophole. This is because the historical penalty for slower convergence is negated by the speed increase, meaning the current RCP system only effectively prevents faster-than-reference convergence.

The Cherry-Picking Concern: Dell's data from Llama 3 8B runs highlighted that submitters can exploit "lucky runs" with better convergence times from a large number of runs. While RCPs are designed to normalize or penalize these outliers, the discussion focused on ensuring fairness when this is combined with the FP4 loophole.

Proposed Countermeasures: Solutions to address the issue include:

1. Requiring owners of lower-precision code to submit a new RCP.
2. Establishing a simple convergence factor from a few baseline runs (instead of a full RCP) for new numerical formats.
3. Placing the burden of proof on submitters with outlier (unexpectedly faster or deviating) convergence.
4. Increasing the required number of submission runs for lower-precision models.

Conclusion: Given that none of these solutions are great, the group committed to continuing the discussion on these options the following week.