User:104gli/NAS edits

for DeepScale page...

Technology
[leave 1st paragraph alone]

In recent years, Neural architecture search (NAS) has begun to outperform humans at designing DNNs that produce high-accuracy results while running fast. In 2019, DeepScale published a SqueezeNAS, which uses [link: Neural architecture search#supernetwork] [supernetwork-based] NAS to design a family of fast and accurate DNNs for semantic segmentation of images. DeepScale's paper claimed that the SqueezeNAS-designed networks outperform the speed-accuracy tradeoff curve of Google's MobileNetV3 family of neural network models. Further, while Google used thousands of GPU-days to search for the design of MobileNetV3, DeepScale used just tens of GPU-days to design the DNNs presented in the SqueezeNAS paper.

for NAS page...

Supernetwork search
RL-based NAS has been shown to require thousands of GPU-days of searching/training to produce neural networks that achieve the state-of-the-art computer vision results described in the NASNet, mNASNet and MobileNetV3 papers. However, supernetwork-based NAS provides a more computationally-efficient solution. The essential idea is, rather than generating thousands of networks and training them independently, supernetwork-based search trains one supernetwork that contains many choices for what the final network design could look like. In addition to the learned parameters in the neural network modules, there is also a set of architecture parameters, which learn to "prefer" one neural network module over an other, leading to a highly-optimized neural network design. In a sense, supernetwork-based search is a self-designing neural network.

Supernetwork-based search has been shown to produce competitive results while using a fraction of the search-time required by RL-based search methods. For example, a paper called FBNet (which is short for Facebook Berkeley Network) demonstrated that supernetwork-based search produces neural networks that outperform the speed-accuracy tradeoff curve of mNASNet and MobileNetV2 on the ImageNet image-classification dataset, and FBNet accomplishes this using over 400x less search time than was used for mNASNet. Further, a paper called SqueezeNAS demonstrated that supernetwork-based NAS produces neural networks that outperform the speed-accuracy tradeoff curve of MobileNetV3 on the Cityscapes semantic segmentation dataset, and SqueezeNAS uses over 100x less search time than was used in the MobileNetV3 authors' RL-based search.