From d6c8f0524319e135b2e5b7b42ad2d90bb54341d3 Mon Sep 17 00:00:00 2001 From: Krzysztof Lecki Date: Mon, 28 Aug 2023 18:54:43 +0200 Subject: [PATCH] Enable AutoAugment and modernize DALI pipeline for ConvNets Update DALI implementation to use modern "fn" API instead of old class approach. Add a codepath using AutoAugment in DALI training pipeline. It can be easily extended to use other Automatic Augmentations. The integration of DALI Pipeline with PyTorch additionally skips the transposition when exposing NHWC data. Extract the DALI implementation to separate file. Update the readme and some configuration files for EfficientNet: * dali-gpu is the default one, instead of PyTorch * DALI supports AutoAugment (+ a mention of other Automatic Augmentations) Fix a typo in the readme files: --data-backends -> --data-backend This PR is a backport of the changes made to this example, when it was introduced into DALI codebase: https://github.com/NVIDIA/DALI/tree/main/docs/examples/use_cases/pytorch/efficientnet The changes were tested with the smallest EfficientNet only. The usage od DALI GPU pipeline in the training can remove the CPU bottlneck on both DGX-1V and DGX-A100 when running using AMP which was covered in the blogpost: https://developer.nvidia.com/blog/why-automatic-augmentation-matters/ Signed-off-by: Krzysztof Lecki --- PyTorch/Classification/ConvNets/configs.yml | 4 +- .../ConvNets/efficientnet/README.md | 31 +-- .../ConvNets/image_classification/dali.py | 91 ++++++++ .../image_classification/dataloaders.py | 215 +++++------------- PyTorch/Classification/ConvNets/main.py | 8 +- .../ConvNets/resnet50v1.5/README.md | 6 +- .../ConvNets/resnext101-32x4d/README.md | 6 +- .../ConvNets/se-resnext101-32x4d/README.md | 6 +- 8 files changed, 181 insertions(+), 186 deletions(-) create mode 100644 PyTorch/Classification/ConvNets/image_classification/dali.py diff --git a/PyTorch/Classification/ConvNets/configs.yml b/PyTorch/Classification/ConvNets/configs.yml index 2780517a6..f6a2f48a5 100644 --- a/PyTorch/Classification/ConvNets/configs.yml +++ b/PyTorch/Classification/ConvNets/configs.yml @@ -105,9 +105,9 @@ anchors: mixup: 0.2 lr_schedule: cosine momentum: 0.9 - warmup: 16 + warmup: 16 epochs: 400 - data_backend: pytorch + data_backend: dali-gpu augmentation: autoaugment num_classes: 1000 interpolation: bicubic diff --git a/PyTorch/Classification/ConvNets/efficientnet/README.md b/PyTorch/Classification/ConvNets/efficientnet/README.md index 7e10c9306..ae2cc9a24 100644 --- a/PyTorch/Classification/ConvNets/efficientnet/README.md +++ b/PyTorch/Classification/ConvNets/efficientnet/README.md @@ -92,7 +92,7 @@ This model uses SGD optimizer for B0 models and RMSPROP optimizer alpha=0.853 e * 0.0125 for 128 batch size for B0 models * 4.09e-06 for 32 batch size for B4 models scale the learning rate. -* Learning rate schedule: +* Learning rate schedule: * cosine LR schedule for B0 models * linear LR schedule for B4 models * Weight decay (WD): @@ -128,7 +128,7 @@ The following features are supported by this model: | Feature | EfficientNet |-----------------------|-------------------------- -|[DALI](https://docs.nvidia.com/deeplearning/dali/release-notes/index.html) | Yes (without autoaugmentation) +|[DALI](https://docs.nvidia.com/deeplearning/dali/release-notes/index.html) | Yes |[APEX AMP](https://nvidia.github.io/apex/amp.html) | Yes |[QAT](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization) | Yes @@ -143,10 +143,11 @@ We use [NVIDIA DALI](https://github.com/NVIDIA/DALI), which speeds up data loading when CPU becomes a bottleneck. DALI can use CPU or GPU, and outperforms the PyTorch native dataloader. -Run training with `--data-backends dali-gpu` or `--data-backends dali-cpu` to enable DALI. -For DGXA100 and DGX1 we recommend `--data-backends dali-cpu`. +Run training with `--data-backend dali-gpu` or `--data-backend dali-cpu` to enable DALI. +For DGXA100 and DGX1 we recommend `--data-backend dali-gpu`. -DALI currently does not support Autoaugmentation, so for best accuracy it has to be disabled. +DALI supports [AutoAugment](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/auto_aug/auto_augment.html), +as well as other [Automatic Augmentation schemes](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/auto_aug/auto_aug.html). **[APEX](https://github.com/NVIDIA/apex)** @@ -199,7 +200,7 @@ To enable mixed precision, you can: #### Enabling TF32 -TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. +TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations. @@ -319,9 +320,9 @@ For example: You can download pre-trained weights from NGC: ```bash -wget --content-disposition -O +wget --content-disposition -O -unzip +unzip ``` To run inference on ImageNet, run: @@ -441,12 +442,12 @@ URL for each model can be found in the following table: | **Model** | **NGC weights URL** | |:---------:|:-------------------:| -| efficientnet-b0 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b0_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-b0_210412.pth | -| efficientnet-b4 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b4_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-b4_210412.pth | -| efficientnet-widese-b0 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_widese_b0_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-widese-b0_210412.pth | -| efficientnet-widese-b4 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_widese_b4_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-widese-b4_210412.pth | -| efficientnet-quant-b0 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b0_pyt_qat_ckpt_fp32/versions/21.03.0/files/nvidia-efficientnet-quant-b0-130421.pth | -| efficientnet-quant-b4 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b4_pyt_qat_ckpt_fp32/versions/21.03.0/files/nvidia-efficientnet-quant-b4-130421.pth | +| efficientnet-b0 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b0_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-b0_210412.pth | +| efficientnet-b4 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b4_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-b4_210412.pth | +| efficientnet-widese-b0 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_widese_b0_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-widese-b0_210412.pth | +| efficientnet-widese-b4 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_widese_b4_pyt_amp/versions/20.12.0/files/nvidia_efficientnet-widese-b4_210412.pth | +| efficientnet-quant-b0 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b0_pyt_qat_ckpt_fp32/versions/21.03.0/files/nvidia-efficientnet-quant-b0-130421.pth | +| efficientnet-quant-b4 | https://api.ngc.nvidia.com/v2/models/nvidia/efficientnet_b4_pyt_qat_ckpt_fp32/versions/21.03.0/files/nvidia-efficientnet-quant-b4-130421.pth | To run inference on ImageNet, run: @@ -469,7 +470,7 @@ During the QAT process, evaluation is done in the same way as during standard tr or to evaluate a created checkpoint with the flag `--evaluate`: -`python ./quant_main.py --arch efficientnet-quant- --evaluate --epochs 1 --resume -b ` +`python ./quant_main.py --arch efficientnet-quant- --evaluate --epochs 1 --resume -b ` It also can run on multi-GPU in an identical way as the standard `main.py` script: diff --git a/PyTorch/Classification/ConvNets/image_classification/dali.py b/PyTorch/Classification/ConvNets/image_classification/dali.py new file mode 100644 index 000000000..5b963fbfe --- /dev/null +++ b/PyTorch/Classification/ConvNets/image_classification/dali.py @@ -0,0 +1,91 @@ +# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvidia.dali import fn +from nvidia.dali import types + +from nvidia.dali.pipeline.experimental import pipeline_def + +from nvidia.dali.auto_aug import auto_augment, trivial_augment + + +@pipeline_def(enable_conditionals=True) +def training_pipe(data_dir, interpolation, image_size, output_layout, automatic_augmentation, + dali_device="gpu", rank=0, world_size=1): + rng = fn.random.coin_flip(probability=0.5) + + jpegs, labels = fn.readers.file(name="Reader", file_root=data_dir, shard_id=rank, + num_shards=world_size, random_shuffle=True, pad_last_batch=True) + + if dali_device == "gpu": + decoder_device = "mixed" + resize_device = "gpu" + else: + decoder_device = "cpu" + resize_device = "cpu" + + # This padding sets the size of the internal nvJPEG buffers to be able to handle all images + # from full-sized ImageNet without additional reallocations + images = fn.decoders.image_random_crop(jpegs, device=decoder_device, output_type=types.RGB, + device_memory_padding=211025920, + host_memory_padding=140544512, + random_aspect_ratio=[0.75, 4.0 / 3.0], + random_area=[0.08, 1.0]) + + images = fn.resize(images, device=resize_device, size=[image_size, image_size], + interp_type=interpolation, antialias=False) + + # Make sure that from this point we are processing on GPU regardless of dali_device parameter + images = images.gpu() + + images = fn.flip(images, horizontal=rng) + + # Based on the specification, apply the automatic augmentation policy. Note, that from the point + # of Pipeline definition, this `if` statement relies on static scalar parameter, so it is + # evaluated exactly once during build - we either include automatic augmentations or not. + # We pass the shape of the image after the resize so the translate operations are done + # relative to the image size. + if automatic_augmentation is None: + output = images + elif automatic_augmentation == "autoaugment": + output = auto_augment.auto_augment_image_net(images, shape=[image_size, image_size]) + else: + raise ValueError(f"Automatic augmentation: '{automatic_augmentation}'" + f" is not supported for DALI") + + + output = fn.crop_mirror_normalize(output, dtype=types.FLOAT, output_layout=output_layout, + crop=(image_size, image_size), + mean=[0.485 * 255, 0.456 * 255, 0.406 * 255], + std=[0.229 * 255, 0.224 * 255, 0.225 * 255]) + + return output, labels + + +@pipeline_def +def validation_pipe(data_dir, interpolation, image_size, image_crop, output_layout, rank=0, + world_size=1): + jpegs, label = fn.readers.file(name="Reader", file_root=data_dir, shard_id=rank, + num_shards=world_size, random_shuffle=False, pad_last_batch=True) + + images = fn.decoders.image(jpegs, device="mixed", output_type=types.RGB) + + images = fn.resize(images, resize_shorter=image_size, interp_type=interpolation, + antialias=False) + + output = fn.crop_mirror_normalize(images, dtype=types.FLOAT, output_layout=output_layout, + crop=(image_crop, image_crop), + mean=[0.485 * 255, 0.456 * 255, 0.406 * 255], + std=[0.229 * 255, 0.224 * 255, 0.225 * 255]) + return output, label diff --git a/PyTorch/Classification/ConvNets/image_classification/dataloaders.py b/PyTorch/Classification/ConvNets/image_classification/dataloaders.py index 7f3249b4d..9d7e82886 100644 --- a/PyTorch/Classification/ConvNets/image_classification/dataloaders.py +++ b/PyTorch/Classification/ConvNets/image_classification/dataloaders.py @@ -41,13 +41,13 @@ DATA_BACKEND_CHOICES = ["pytorch", "synthetic"] try: from nvidia.dali.plugin.pytorch import DALIClassificationIterator - from nvidia.dali.pipeline import Pipeline - import nvidia.dali.ops as ops import nvidia.dali.types as types + from image_classification.dali import training_pipe, validation_pipe + DATA_BACKEND_CHOICES.append("dali-gpu") DATA_BACKEND_CHOICES.append("dali-cpu") -except ImportError: +except ImportError as e: print( "Please install DALI from https://www.github.com/NVIDIA/DALI to run this example." ) @@ -77,138 +77,26 @@ def load_jpeg_from_file(path, cuda=True): return input -class HybridTrainPipe(Pipeline): - def __init__( - self, - batch_size, - num_threads, - device_id, - data_dir, - interpolation, - crop, - dali_cpu=False, - ): - super(HybridTrainPipe, self).__init__( - batch_size, num_threads, device_id, seed=12 + device_id - ) - interpolation = { - "bicubic": types.INTERP_CUBIC, - "bilinear": types.INTERP_LINEAR, - "triangular": types.INTERP_TRIANGULAR, - }[interpolation] - if torch.distributed.is_initialized(): - rank = torch.distributed.get_rank() - world_size = torch.distributed.get_world_size() - else: - rank = 0 - world_size = 1 - - self.input = ops.FileReader( - file_root=data_dir, - shard_id=rank, - num_shards=world_size, - random_shuffle=True, - pad_last_batch=True, - ) - - if dali_cpu: - dali_device = "cpu" - self.decode = ops.ImageDecoder(device=dali_device, output_type=types.RGB) - else: - dali_device = "gpu" - # This padding sets the size of the internal nvJPEG buffers to be able to handle all images from full-sized ImageNet - # without additional reallocations - self.decode = ops.ImageDecoder( - device="mixed", - output_type=types.RGB, - device_memory_padding=211025920, - host_memory_padding=140544512, - ) - - self.res = ops.RandomResizedCrop( - device=dali_device, - size=[crop, crop], - interp_type=interpolation, - random_aspect_ratio=[0.75, 4.0 / 3.0], - random_area=[0.08, 1.0], - num_attempts=100, - antialias=False, - ) - - self.cmnp = ops.CropMirrorNormalize( - device="gpu", - dtype=types.FLOAT, - output_layout=types.NCHW, - crop=(crop, crop), - mean=[0.485 * 255, 0.456 * 255, 0.406 * 255], - std=[0.229 * 255, 0.224 * 255, 0.225 * 255], - ) - self.coin = ops.CoinFlip(probability=0.5) - - def define_graph(self): - rng = self.coin() - self.jpegs, self.labels = self.input(name="Reader") - images = self.decode(self.jpegs) - images = self.res(images) - output = self.cmnp(images.gpu(), mirror=rng) - return [output, self.labels] - - -class HybridValPipe(Pipeline): - def __init__( - self, batch_size, num_threads, device_id, data_dir, interpolation, crop, size - ): - super(HybridValPipe, self).__init__( - batch_size, num_threads, device_id, seed=12 + device_id - ) - interpolation = { - "bicubic": types.INTERP_CUBIC, - "bilinear": types.INTERP_LINEAR, - "triangular": types.INTERP_TRIANGULAR, - }[interpolation] - if torch.distributed.is_initialized(): - rank = torch.distributed.get_rank() - world_size = torch.distributed.get_world_size() - else: - rank = 0 - world_size = 1 - - self.input = ops.FileReader( - file_root=data_dir, - shard_id=rank, - num_shards=world_size, - random_shuffle=False, - pad_last_batch=True, - ) - - self.decode = ops.ImageDecoder(device="mixed", output_type=types.RGB) - self.res = ops.Resize( - device="gpu", - resize_shorter=size, - interp_type=interpolation, - antialias=False, - ) - self.cmnp = ops.CropMirrorNormalize( - device="gpu", - dtype=types.FLOAT, - output_layout=types.NCHW, - crop=(crop, crop), - mean=[0.485 * 255, 0.456 * 255, 0.406 * 255], - std=[0.229 * 255, 0.224 * 255, 0.225 * 255], - ) - - def define_graph(self): - self.jpegs, self.labels = self.input(name="Reader") - images = self.decode(self.jpegs) - images = self.res(images) - output = self.cmnp(images) - return [output, self.labels] - - class DALIWrapper(object): + def gen_wrapper(dalipipeline, num_classes, one_hot, memory_format): for data in dalipipeline: - input = data[0]["data"].contiguous(memory_format=memory_format) + if memory_format == torch.channels_last: + # If we requested the data in channels_last form, utilize the fact that DALI + # can return it as NHWC. The network expects NCHW shape with NHWC internal memory, + # so we can keep the memory and just create a view with appropriate shape and + # strides reflacting that memory layouyt + shape = data[0]["data"].shape + stride = data[0]["data"].stride() + + # permute shape and stride from NHWC to NCHW + def nhwc_to_nchw(t): + return t[0], t[3], t[1], t[2] + + input = torch.as_strided(data[0]["data"], size=nhwc_to_nchw(shape), + stride=nhwc_to_nchw(stride)) + else: + input = data[0]["data"].contiguous(memory_format=memory_format) target = torch.reshape(data[0]["label"], [-1]).cuda().long() if one_hot: target = expand(num_classes, torch.float, target) @@ -227,7 +115,7 @@ def __iter__(self): ) -def get_dali_train_loader(dali_cpu=False): +def get_dali_train_loader(dali_device="gpu"): def gdtl( data_path, image_size, @@ -249,21 +137,27 @@ def gdtl( rank = 0 world_size = 1 + interpolation = { + "bicubic": types.INTERP_CUBIC, + "bilinear": types.INTERP_LINEAR, + "triangular": types.INTERP_TRIANGULAR, + }[interpolation] + + output_layout = "HWC" if memory_format == torch.channels_last else "CHW" + traindir = os.path.join(data_path, "train") - if augmentation is not None: - raise NotImplementedError( - f"Augmentation {augmentation} for dali loader is not supported" - ) - - pipe = HybridTrainPipe( - batch_size=batch_size, - num_threads=workers, - device_id=rank % torch.cuda.device_count(), - data_dir=traindir, - interpolation=interpolation, - crop=image_size, - dali_cpu=dali_cpu, - ) + + pipeline_kwargs = { + "batch_size" : batch_size, + "num_threads" : workers, + "device_id" : rank % torch.cuda.device_count(), + "seed": 12 + rank % torch.cuda.device_count(), + } + + pipe = training_pipe(data_dir=traindir, interpolation=interpolation, image_size=image_size, + output_layout=output_layout, automatic_augmentation=augmentation, + dali_device=dali_device, rank=rank, world_size=world_size, + **pipeline_kwargs) pipe.build() train_loader = DALIClassificationIterator( @@ -299,17 +193,26 @@ def gdvl( rank = 0 world_size = 1 + interpolation = { + "bicubic": types.INTERP_CUBIC, + "bilinear": types.INTERP_LINEAR, + "triangular": types.INTERP_TRIANGULAR, + }[interpolation] + + output_layout = "HWC" if memory_format == torch.channels_last else "CHW" + valdir = os.path.join(data_path, "val") - pipe = HybridValPipe( - batch_size=batch_size, - num_threads=workers, - device_id=rank % torch.cuda.device_count(), - data_dir=valdir, - interpolation=interpolation, - crop=image_size, - size=image_size + crop_padding, - ) + pipeline_kwargs = { + "batch_size" : batch_size, + "num_threads" : workers, + "device_id" : rank % torch.cuda.device_count(), + "seed": 12 + rank % torch.cuda.device_count(), + } + + pipe = validation_pipe(data_dir=valdir, interpolation=interpolation, + image_size=image_size + crop_padding, image_crop=image_size, + output_layout=output_layout, **pipeline_kwargs) pipe.build() val_loader = DALIClassificationIterator( diff --git a/PyTorch/Classification/ConvNets/main.py b/PyTorch/Classification/ConvNets/main.py index 2135301e8..abb0e84be 100644 --- a/PyTorch/Classification/ConvNets/main.py +++ b/PyTorch/Classification/ConvNets/main.py @@ -101,7 +101,7 @@ def add_parser_arguments(parser, skip_arch=False): "--interpolation", metavar="INTERPOLATION", default="bilinear", - help="interpolation type for resizing images: bilinear, bicubic or triangular(DALI only)", + help="interpolation type for resizing images: bilinear, bicubic or triangular (DALI only)", ) if not skip_arch: model_names = available_models().keys() @@ -129,7 +129,7 @@ def add_parser_arguments(parser, skip_arch=False): default=2, type=int, metavar="N", - help="number of samples prefetched by each loader", + help="number of samples prefetched by each loader (PyTorch only)", ) parser.add_argument( "--epochs", @@ -498,10 +498,10 @@ def _worker_init_fn(id): get_train_loader = get_pytorch_train_loader get_val_loader = get_pytorch_val_loader elif args.data_backend == "dali-gpu": - get_train_loader = get_dali_train_loader(dali_cpu=False) + get_train_loader = get_dali_train_loader(dali_device="gpu") get_val_loader = get_dali_val_loader() elif args.data_backend == "dali-cpu": - get_train_loader = get_dali_train_loader(dali_cpu=True) + get_train_loader = get_dali_train_loader(dali_device="cpu") get_val_loader = get_dali_val_loader() elif args.data_backend == "synthetic": get_val_loader = get_synthetic_loader diff --git a/PyTorch/Classification/ConvNets/resnet50v1.5/README.md b/PyTorch/Classification/ConvNets/resnet50v1.5/README.md index 9af61a395..80d895dd6 100644 --- a/PyTorch/Classification/ConvNets/resnet50v1.5/README.md +++ b/PyTorch/Classification/ConvNets/resnet50v1.5/README.md @@ -143,8 +143,8 @@ We use [NVIDIA DALI](https://github.com/NVIDIA/DALI), which speeds up data loading when CPU becomes a bottleneck. DALI can use CPU or GPU, and outperforms the PyTorch native dataloader. -Run training with `--data-backends dali-gpu` or `--data-backends dali-cpu` to enable DALI. -For DGXA100 and DGX1 we recommend `--data-backends dali-cpu`, for DGX2 we recommend `--data-backends dali-gpu`. +Run training with `--data-backend dali-gpu` or `--data-backend dali-cpu` to enable DALI. +For DGXA100 and DGX1 we recommend `--data-backend dali-cpu`, for DGX2 we recommend `--data-backend dali-gpu`. ### Mixed precision training @@ -188,7 +188,7 @@ To enable mixed precision, you can: #### Enabling TF32 -TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. +TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations. diff --git a/PyTorch/Classification/ConvNets/resnext101-32x4d/README.md b/PyTorch/Classification/ConvNets/resnext101-32x4d/README.md index 5184c7cee..60ff44177 100644 --- a/PyTorch/Classification/ConvNets/resnext101-32x4d/README.md +++ b/PyTorch/Classification/ConvNets/resnext101-32x4d/README.md @@ -127,8 +127,8 @@ We use [NVIDIA DALI](https://github.com/NVIDIA/DALI), which speeds up data loading when CPU becomes a bottleneck. DALI can use CPU or GPU, and outperforms the PyTorch native dataloader. -Run training with `--data-backends dali-gpu` or `--data-backends dali-cpu` to enable DALI. -For DGXA100 and DGX1 we recommend `--data-backends dali-cpu`. +Run training with `--data-backend dali-gpu` or `--data-backend dali-cpu` to enable DALI. +For DGXA100 and DGX1 we recommend `--data-backend dali-cpu`. ### Mixed precision training @@ -172,7 +172,7 @@ To enable mixed precision, you can: #### Enabling TF32 -TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. +TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations. diff --git a/PyTorch/Classification/ConvNets/se-resnext101-32x4d/README.md b/PyTorch/Classification/ConvNets/se-resnext101-32x4d/README.md index 9c4ad6902..a3eea415c 100644 --- a/PyTorch/Classification/ConvNets/se-resnext101-32x4d/README.md +++ b/PyTorch/Classification/ConvNets/se-resnext101-32x4d/README.md @@ -128,8 +128,8 @@ We use [NVIDIA DALI](https://github.com/NVIDIA/DALI), which speeds up data loading when CPU becomes a bottleneck. DALI can use CPU or GPU, and outperforms the PyTorch native dataloader. -Run training with `--data-backends dali-gpu` or `--data-backends dali-cpu` to enable DALI. -For DGXA100 and DGX1 we recommend `--data-backends dali-cpu`. +Run training with `--data-backend dali-gpu` or `--data-backend dali-cpu` to enable DALI. +For DGXA100 and DGX1 we recommend `--data-backend dali-cpu`. ### Mixed precision training @@ -173,7 +173,7 @@ To enable mixed precision, you can: #### Enabling TF32 -TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. +TensorFloat-32 (TF32) is the new math mode in [NVIDIA A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs. TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations.