Memory Wall: Stories

Product Information

Iandola FN, Shaw AE, Krishna R, Keutzer KW. SqueezeBERT: What can computer vision teach NLP about efficient neural networks?. arXiv preprint arXiv:2006.11316. 2020 Jun 19. In addition to serving as temporary storage and working space for the operating system and applications, RAM is used in numerous other ways. CPU speed improvements slowed significantly partly due to major physical barriers and partly because current CPU designs have already hit the memory wall in some sense. Intel summarized these causes in a 2005 document. [33] Cai F, Correll J M, Lee S H, et al. A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations. Nat Electron, 2019, 2: 290–299 Hoefler T, Alistarh D, Ben-Nun T, Dryden N, Peste A. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102.00554. 2021 Jan 31.

Center collaborators are Cornell University; Georgia Tech; Pennsylvania State University; the University of California, Los Angeles; the University of California, San Diego; the University of Washington; the University of Wisconsin; and the University of Pennsylvania. Gholami A, Kwon K, Wu B, Tai Z, Yue X, Jin P, Zhao S, Keutzer K. Squeezenext: Hardware-aware neural network design. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2018 (pp. 1638–1647). Rajbhandari S, Rasley J, Ruwase O, He Y. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis 2020 Nov 9 (pp. 1–16). IEEE. and other types of non-volatile memories allow random access for read operations, but either do not allow write operations or have other kinds of limitations on them. These include most types of ROM and a type of flash memory called NOR-Flash. Samsung Announces the World's First 222 MHz 32Mbit SGRAM for 3D Graphics and Networking Applications". Samsung Semiconductor. Samsung. 12 July 1999 . Retrieved 10 July 2019.Samsung Electronics Develops Industry's First Ultra-Fast GDDR4 Graphics DRAM". Samsung Semiconductor. Samsung. October 26, 2005 . Retrieved 8 July 2019.

Coudrain P, Charbonnier J, Garnier A, et al. Active interposer technology for chiplet-based advanced 3D system architectures. In: Proceedings of 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), Las Vegas, 2019. 569–578 Ahmed Amine Jerraya and Wayne Wolf (2005). Multiprocessor Systems-on-chips. Morgan Kaufmann. pp.90–91. ISBN 9780123852519. Archived from the original on August 1, 2016 . Retrieved March 31, 2014.Angizi S, Fan D. ReDRAM: a reconfigurable processing-in-DRAM platform for accelerating bulk bit-wise operations. In: Proceedings of IEEE/ACM International Conference on Computer-Aided Design (ICCAD), Westminster, 2019. 1–8 IBM Archives -- FAQ's for Products and Services". ibm.com. Archived from the original on 2012-10-23. SOTA 模型带来了不少额外开销（overhead）。 这些问题大多是由于训练中使用的是一阶 SGD 优化方法。虽然 SGD 超参容易实现，却没有稳健的方法去调试超参，特别是对于那些还没得到正确超参集合的新模型，调参就更加困难了。 一个可能的解决方法是使用二阶 SGD 优化方法，如我们最近发表的 ADAHESSIAN 方法[4]。这类方法在超参调优时往往更加稳健，从而达到可以达到 SOTA。 但是，这种方法也有亟待解决的问题：目前占用的内存是原来的3-4倍。微软关于 Zero 论文种介绍了一个很有前景的工作：可以通过删除/切分冗余优化器状态参数[21, 3]，在保持内存消耗量不变的前提下，训练8倍大的模型。如果这些高阶方法的引入的 overhead 问题可以得到解决，那么可以显著降低训练大型模型的总成本。 Oliveira G F, Santos P C, Alves M A Z, et al. A generic processing in memory cycle accurate simulator under hybrid memory cube architecture. In: Proceedings of 2017 International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation (SAMOS), Pythagorion, 2017. 54–61 Murphy R (2007) On the effects of memory latency and bandwidth on supercomputer application performance. In: Proceedings of the IEEE International Symposium on Workload Characterization, Boston, 27–29 Sept 2007. IEEE, Piscataway, pp 35–43