The following definitions are used throughout this document:

Performance always refers to execution speed.

Quality always refers to a model’s ability to produce “correct” outputs.

A system consists of a defined set of hardware resources such as processors, memories, disks, and interconnect. It also includes specific versions of all software such as operating system, compilers, libraries, and drivers that significantly influences the running time of a benchmark, excluding the ML framework.

A framework is a specific version of a software library or set of related libraries, possibly with associated offline compiler, for training ML models using a system. Examples include specific versions of Caffe2, MXNet, PaddlePaddle, pyTorch, or TensorFlow.

A benchmark is an abstract problem that can be solved using ML by training a model based on a specific dataset or simulation environment to a target quality level.

A suite is a specific set of benchmarks.

A division is a set of rules for implementing benchmarks from a suite to produce a class of comparable results.

A reference implementation is a specific implementation of a benchmark provided by the MLPerf organization.

A benchmark implementation is an implementation of a benchmark in a particular framework by a user under the rules of a specific division.

A submission implementation set is a set of benchmark implementations for one or more benchmarks from a suite under the rules of a specific division using the same framework.

A run is a complete execution of an implementation on a system, training a model from initialization to the quality target.

A run result is the wallclock time required for a run.

A reference result is the result provided by the MLPerf organization for each reference implementation

A benchmark result is the mean of a benchmark-specific number of run results, dropping the highest and lowest results. The result is then normalized to the reference result for that benchmark. Normalization is of the form (reference result / benchmark result) such that a better benchmark result produces a higher number.

A submission result set is a one benchmark result for each benchmark implementation in a submission implementation set.

A submission is a submission implementation set and a corresponding submission result set.

A custom summary result is the weighted geometric mean of an arbitrary set of results from a specific submission. MLPerf endorses this methodology for computing custom summary results but does not endorse any official summary result.

Benchmarking should be conducted to measure the framework and system performance as fairly as possible. Ethics and reputation matter.

The same system and framework must be used for a submission result set. Note that the reference implementations do not all use the same framework.

The framework and system should not detect and behave differently for benchmarks.

Unless part of the definition of a benchmark, the implementation should not encode any information about the content of the dataset or a successful model’s state in any form. High-level statistical information about the dataset, such as distribution of sizes, may be used.

For gpt3, manipulation of metadata which consists of the number of documents in the dataset and the size of each document is allowed as long as the data tokens are not accessed.

For benchmarks which are defined as starting from a fixed set of weights, such as a checkpoint or backbone, the implementation should start from the weights provided in the benchmark reference definition, or if that is not posssible, provide information and code sufficient for reproducing how those starting weights were obtained. For v0.7, sets of weights used in v0.6 are allowed.

The Closed division requires using the same preprocessing, model, training method, and quality target as the reference implementation.

The closed division models and quality targets are:

Closed division benchmarks must be referred to using the benchmark name plus the term Closed, e.g. “for the Image Classification Closed benchmark, the system achieved a result of 7.2.”

The Open division allows using arbitrary training data, preprocessing, model, and/or training method. However, the Open division still requires using supervised or reinforcement machine learning in which a model is iteratively improved based on training data, simulation, or self-play.

Open division benchmarks must be referred to using the benchmark name plus the term Open, e.g. “for the Image Classification Open benchmark, the system achieved a result of 7.2.”

CLOSED: Random numbers must be generated using stock random number generators.

Random number generators may be seeded from the following sources:

• Clock

• System source of randomness, e.g. /dev/random or /dev/urandom

• Another random number generator initialized with an allowed seed

Random number generators may be initialized repeatedly in multiple processes or threads. For a single run, the same seed may be shared across multiple processes or threads.

OPEN: Any random number generation may be used.

CLOSED: The numerical formats fp64, fp32, tf32, fp16, fp8, bfloat16, Graphcore FLOAT 16.16, int8, uint8, int4, and uint4 are pre-approved for use. Additional formats require explicit approval. Scaling may be added where required to compensate for different precision.

Reference Convergence Points must be obtained using FP32 precision, or FP32 emulation with explanation of the methodology for emulation.

OPEN: Any format and scaling may be used.

Epochs should always be numbered from 1.

Public results should be rounded normally.

CLOSED: Each reference implementation includes a script to download the input dataset and script to verify the dataset using a checksum. The data must then be preprocessed in a manner consistent with the reference implementation, excepting any transformations that must be done for each run (e.g. random transformations). The data may also be reformatted for the target system provided that the reformatting does not introduce new information or introduce duplicate copies of data.

OPEN: Any public dataset may be used for training the model, however the evaluation data must be drawn from the benchmark dataset in a manner consistent with the reference.

You must flush the cache or restart the system prior to benchmarking. Data can start on any durable storage system such as local disks and cloud storage systems. This explicitly excludes RAM.

Only preprocessing that must be done for each run (e.g. random transformations) must be timed.

CLOSED: The same preprocessing steps as the reference implementation must be used.

OPEN: Any preprocessing steps are allowed for training data. However, each datum must be preprocessed individually in a manner that is not influenced by any other data. The evaluation data must be preprocessed in a manner consistent with reference.

CLOSED: Images must have the same size as in the reference implementation. Mathematically equivalent padding of images is allowed.

CLOSED: For benchmarks with sequence inputs, you may choose a length N and either truncate all examples to length N or throw out all examples which exceed length N. This must be done uniformly for all examples. This may only be done on the training set and not the evaluation set.

CLOSED: Two ways to represent the Mask R-CNN mask are permitted. One is a polygon and the other is a scalable bitmask.

OPEN: The closed division data representations restrictions only apply at the start of the run. Data may be represented in an arbitrary fashion during the run.

Input encoding data, such as language vocabulary, or the set of possible labels may used during pre-processing or execution without counting as "touching the training data" for timing purposes. Same applies to processing metadata like the number of documents, or document sizes in a dataset.

CLOSED: If applicable, the dataset must be separated into training and test sets in the same manner as the reference implementation.

OPEN: If applicable, the test dataset must be extracted in the same manner as the reference implementation. The training data set may not contain data that appears in the test set.

CLOSED: the training and test data must be traversed in the same conceptual order as the reference implementation. For instance, the data might be traversed sequentially or randomly with uniform distribution. Batch size, shard size, and the random number generator will affect order.

Where data pipelines randomly order data, arbitrary sharding, batching, and packing are allowed provided that (1) the data is still overall randomly ordered and not ordered to improve convergence and (2) each datum still appears exactly once. Modifications to data order and/or batching must be presented to the SWG group in advance of the submission deadline for approval if they could affect the ability to borrow hyperparameters and/or approximately follow the learning rate schedule defined by the RCPs.

In the case of DLRMv2 benchmark, training dataset is shuffled during preprocessing (with a fixed seed) on a per-sample basis. The resulting order of samples should be then used during training and any other extra dataset shuffling is prohibited.

OPEN: The training data may be traversed in any order. The test data must be traversed in the same order as the reference implementation.

CLOSED: Each of the current frameworks has a graph that describes the operations performed during the forward propagation of training. The frameworks automatically infer and execute the corresponding back-propagation computations from this graph. Benchmark implementations must use the same graph as the reference implementation.

CLOSED: Weights and biases must be initialized using the same constant or random value distribution as the reference implementation, unless a pre-trained set of weights, such as a checkpoint or backbone, is used by the reference.

OPEN: Weights and biases must be initialized using a consistent constant or random value distribution.

CLOSED: Frameworks are free to optimize the non-weight parts of the computation graph provided that the changes are mathematically equivalent. So optimizations and graph / code transformations of the flavor of dead code elimination, common subexpression elimination, loop-invariant code motion, and recomputation of node state are entirely allowed.

OPEN: Frameworks are free to alter the graph.

CLOSED:

By default, the hyperparameters must be the same as the reference.

Hyperparameters include the optimizer used and values like the regularization norms and weight decays.

The implementation of the optimizer must match the optimizer specified in the Appendex: Allowed Optimizer. The Appendex lists which optimizers in the popular deep learning frameworks are compliant by default. If a submission uses an alternate implementation, the submitter must describe the optimizer’s equation and demonstrate equivalence with the approved optimizers on that list.

The following table lists the tunable hyperparameters for each allowed model,optimizer combination. The value of each tunable hyperparameter must meet the listed constraint.

The MLPerf verifier scripts checks all hyperparameters except those with names marked with asterisks. If a hyperparameter is marked with one asterisk, it must be checked manually. If a hyperparameter is marked with two asterisks, it is also not logged and it must be checked manually in the code. If the verifier and the constraints in this table differ, the verifier (specifically, the version on the date of submission unless otherwise decided by the review committee) is the source of truth.

OPEN: Hyperparameters and optimizer may be freely changed.

Submitters are expected to use their best efforts to submit with optimal hyperparameters for their system. The intent of Hyperparameter Borrowing is to allow a submitter to update their submission to reflect what they would have submitted had they known about more optimal hyperparameters before submitting, without knowing any other info (ie the performance of other submissions).

During the review period as described in the Submission Rules, a submitter may replace the hyperparameters, once per benchmark entry, in their implementation of a benchmark with hyperparameters from another submitter’s implementation of the same benchmark. By default, they may change batch size (local batch size, global batch size, batchnorm span), but must replace all other hyperparameters as a group.

With evidence that the resulting model, using the same batch size as the other submitter’s implementation, converges worse in terms of epochs required, the submitter may make a minimum number of additional hyperparameter changes for the purpose of improving convergence and achieving comparable, but not better, convergence in epochs compared to the other submitter’s implementation, but preserving any difference in convergence that may exist due to precision choices. In this situation, the other submitter’s implementation is considered the reference, and the new submitter must match the convergence behavior of the other submitter in a similar way as we compare any submission to the reference.

A resubmission of a benchmark with borrowed hyperparameters must use the same software (with the exceptions listed in the Software Adoption section of this document), system and system configuration (accelerators, NICs etc) as the original submission. The largest scale submission for a benchmark from a given system may be resubmitted with borrowed hyperparameters using a change of scale on that system, but only if the new scale is either larger, or enables the resubmission to achieve a faster run result. In addition, the new scale must not be larger than the largest scale used in an original submission of at least one of the benchmarks on that system in this round.

Since the hyperparameters are fixed for GPT3, hyperparameter borrowing is not allowed.

CLOSED: The same loss function used in the reference implementation must be used.

OPEN: Any loss function may be used. Do not confuse the loss function with target quality measure.

Each run must reach a target quality level on the reference implementation quality measure. By default, the time to evaluate the quality is included in the wallclock time. However, if the reference implementation generates timestamped checkpoints and evaluates the quality after the clock has been stopped, then an implementation may either perform evaluation on-the-clock or generate timestamped checkpoints, evaluate them after the clock has been stopped, and update the clock stopped time to the timestamp of the first passing checkpoint. The checkpoint timestamp may be any time after the last weight value included in the checkpoint is updated.

CLOSED: The same quality measure as the reference implementation must be used. The quality measure must be evaluated at the same frequency (in terms of number of training items between test sets) and at least as thoroughly (in terms of number of tests per set) as in the reference implementation. Where applicable, the required evaluation point may be rounded up to the nearest batch size. Typically, a test consists of comparing the output of one forward pass through the network with the desired output from the test set.

OPEN: An arbitrary stopping criteria may be used, including but not limited to the closed quality measure, a different quality measure, the number of epochs, or a fixed time. However, the reported results must include the geometric mean of the final quality as measured by the closed quality measure.

Exceptions for GPT3 OPEN: the open submissions are allowed to choose a language version that’s not English for the C4 dataset. When doing so, the submitter needs to make it clear that the dataset and convergence measures are different from the close division submissions.

Check points can be created at the discretion of submitter. No check points are required to be produced or retained.

The CLOSED division allows limited exemptions to mathematical equivalence between implementations for pragmatic purposes, including:

• Different methods can be used to add color jitter as long as the methods are of a similar distribution and magnitude to the reference.

• If data set size is not evenly divisible by batch size, one of several techniques may be used. The last batch in an epoch may be composed of the remaining samples in the epoch, may be padded, or may be a mixed batch composed of samples from the end of one epoch and the start of the next. If the mixed batch technique is used, quality for the ending epoch must be evaluated after the mixed batch. If the padding technique is used, the first batch may be padded instead of the last batch. Additionally, in the case of DLRMv2 benchmark, the last partial training batch may be dropped.

• Values introduced for padding purposes may be reflected in batch norm computations.

• Adam optimizer implementations may use the very small value epsilon to maintain mathematical stability in slightly different ways, provided that methods are reviewed and approved in advance. One such method involves squaring the value of epsilon and moving epsilon inside the square root in the parameter update equation.

• Distributed batch normalization is allowed.

Additional exemptions need to be explicitly requested and approved in advance. In general, exemptions may be approved for techniques that are common industry practice, introduce small differences that would be difficult to engineer around relative to their significance, and do not substantially decrease the required computation. Over time, MLPerf should seek to help the industry converge on standards and remove exemptions.

The OPEN division does not restrict mathematical equivalence.

An MLPerf submission score is intended to represent the median expected result across a large number of runs.

To reduce statistical variance and the potential to cherry pick results, each benchmark submission is composed of a set of N independent runs, with N chosen based on the observed variation of the benchmark, as described in the table above.

Running multiple iterations of N independent runs with the goal of validating that the submission is close to a median result is encouraged but not required. Running multiple iterations of N runs to try find the lowest one is against the spirit of MLPerf and is prohibited – see Section 2.1, “Strive to be fair”. Results that appear to be too far away from a median result may be rejected.

As a more computationally efficient method of validating that a submission is close to the median result, it is also allowed to run M>N independent runs as a group and to designate N consecutive runs from the group as the runs to be used for scoring, provided that the submitter chooses the N consecutive runs that are closest to the median result. For the purposes of calculating the median, sets of N consecutive runs that would create an invalid benchmark result should be included in the median calculation as "infinite" scores. If the median set would be an invalid benchmark result, the entire result is invalid. Submitting the full run set (vs just the N runs used for scoring) as a reference is optional, but may be required in the future. For purposes of this scoring, "consecutive" is defined as an objective and deterministic method, such as submission timestamps. Submitters are not allowed to pick different orderings to improve their score. Runs may go in parallel on the submitter’s compute resources, as long as there is a way to objectively and deterministically sort the runs, for example by timestamp.

An example could be for a benchmark with N=5 runs, a submitter could ahead of time pick M=10, launch 10 runs on their compute resources, sort the 10 runs by their launch time stamp, then take a sliding window of 5 consecutive runs over those 10 runs. That sliding window would create 6 possible sets of 5 runs. Each of those 6 sets would be olympically scored, and the set with the median runtime would be submitted as that submitter’s score. Any failed runs within those 10 runs would count as infinity time and need to be included in the olympic scoring (could be thrown away as the slowest score). It is recommended that a submitter keep the logs for all M runs, because the review committee may ask for the submitter to share the M logs during the review period.

The score of an MLPerf submission may be scaled if the training committee decides so during the review period. This scaling may be, but not limited to failing to meet the reference convergence limits imposed by the Reference Convergence Points (see following section). To facilitate the automatic generation of the scaled score the scaling factor must be provided in a json file under the name scaling.json in the directory whose scores are going to be scaled.

Submitters are encouraged to run the RCP checker script prior to their submission to make sure they do not violate RCP limits.

If a submission fails the RCP test, such as S2 in the Appendix, they have the option to submit with the --rcp_bypass parameter. This will allow the submission to upload, but the submitter must notify the results chair, and prepare for the audit process described in the next section where at review time the submitter should be able to justify why their submission is valid while it failed the RCP test.

If a submission is missing the RCP for the batch size they are submitting, such as S4 and S6 in the Appendix they must provide the missing convergence points by making a PR in the logger. All missing RCPs are due 24h after the submission deadline (Exception is GPT3: where RCPs are due 5 weeks before the submission deadline). RCPs are added by making a pull request into the RCP library in the logging repository. Since the RCP may arrive after the submission deadline, the submitter can use the --rcp_bypass parameter again to have their submission accepted.

During hyperparameter borrowing, borrowers can use hyperparameters from submissions that passed or failed the RCP test. If their submission fails to pass the RCP test they can have it upload by using --rcp-bypass and then prepare for the audit decribed in the next section.

To extract submission convergence points, logs should report epochs as follows.

In order to reduce the burden on the submitter as well as the Submitter’s Working Group (SWG) during the review period, submitters shall ensure compliance with RCP tests ahead of the submission deadline. Submissions that need new RCPs are required to supply those RCPs at the same time as their submission, as specified in the Training Rules document. While providing new RCPs, a submitter must also include reference run logs for the SWG and reference owner to review.

Submissions with failing RCP tests are rejected by default until the SWG approves the submission. Submitters shall notify the SWG in advance of a potential RCP failure, so they can prefetch requests for additional data and minimize churn during the review period. A submitter requesting approval for a submission with failing RCP test shall provide additional explanatory data to the SWG explaining why the WG should consider the non-compliant submission a fair comparison to compliant submissions. This list will be decided by the WG for each submission individually.

A non-exhaustive list of potential requests of data is:

1. Written statement from the submitter explaining the plausible cause of deviation. This should also be supported by data from A/B experiments.

2. Logs showing training loss of the submission vs training loss of the reference. Note that the reference run should be on reference hardware platform in FP32

3. Model summary showing number of trainable_parameters (weights) in the model vs the same.

4. Debugging via comparing intermediate activations, distributions of initialization weights, and/or compliant randomization on the reference vs the submission. The SWG may further request additional information, not listed above, at their discretion.

A submitter requesting approval for their RCP failing submission during the review period shall provide requested information in a timely manner. All evidence supporting the appeal is due at the latest by the end of Review Week 1. For resubmissions during the review period, all appeal evidence is due at the time of resubmission.

The SWG must come to majority consensus to approve a submission that fails the RCP test. If the SWG cannot come to majority consensus to approve a submission, then potential alternatives are:

1. Normalize submission run epochs to reference epochs to pass RCP test irrespective of accuracy achieved

2. Submission is withdrawn due to non-compliance

For v1.0 only, Mask-RCNN submitters may use the non-reference backbone located here with the understanding that it converges similarly to the reference backbone. If the non-reference backbone is shown to converge faster than the reference backbone at any scale on any submitted hyperparameter set, all uses of that backbone for any submitter are to be re-run with the reference backbone to have their submission published. For future rounds, the expectation is that all submitters will use the reference backbone, which will fixed at reference code freeze time.

For v1.0 only, BERT submissions may implement clip-norm either before or after inter-accelerator all-reduce. For future rounds, the expectation is that submissions must use clip-norm-after-reduce, to be consistent with most commonly used public BERT model repos.

For performance consistency of at scale BERT submissions for v1.0, submitters are disallowed from using clip-norm-after-reduce to enable additional overlap of communication and math. If a submitter plans to use clip-norm-after-reduce for v1.0, they must notify the committee before the submission deadline, and be prepared to show code in their submission proving that they do not do overlap as a result of clip-norm-after-reduce.

Furthermore, for simplicity, the RCPs for this round will use clip-norm-before-reduce. In theory, this could allow clip-norm-after-reduce submissions that converge faster than they should, but still not faster than clip-norm-before-reduce, but the Training Working Group feels that this is ok risk for v1.0, in interest of simplifying the RCPs for v1.0.

For v1.0 only, the allowed untimed compile time is increased from 20 minutes to 30 minutes. This is to enable new submitters to submit who were close to the 20 minute limit. The 20 minute number was chosen empirically for rounds prior to v1.0. For v1.1 and beyond, the training working group should make a data driven decision on what compile time is reasonable for real user applications.

For v1.0 only, BERT submissions may implement clip-norm either before or after inter-accelerator all-reduce. For future rounds, the expectation is that submissions must use clip-norm-after-reduce, to be consistent with most commonly used public BERT model repos. This exception from v1.0 was extended to v1.1 because of the tight schedule between rounds.

For v1.1, we changed the policy documentation to say that a Preview submission needs to be available at the next submission after 140 days, not 180 days like it was before. However, this does not apply to Preview submissions from v1.0, which will still follow the 180 day policy. For v1.1 Preview submissions and beyond, the 140 day rule will apply. This is not necessarily an "exception," but we are listing it here as a special case for the record.

Allowed model initialization compile time for GPT3 benchmark is increased to 60 minutes for Closed Division owing to the large memory footprint of initial checkpoints.