"Roselektronika" Holding of Rostec State Corporation has developed the Fisher supercomputer with a peak capacity of 13.5 teraflops.
Compared to the GeForce RTX 2080 Ti graphics card, you should look at "FP64 (double) performance" (420.2 GFLOPS) instead of "FP32 (float) performance" (13.45 TFLOPS). Thus, the performance of the supercomputer is not actually equal to the performance of RTX 2080 Ti but exceeds it 32 times by FP64 (double), thanks for the specification of Pand5461.
The system was commissioned by the Joint Institute for High Temperatures of the Russian Academy of Sciences (RAS) to calculate digital models and predict the behavior of substances in extreme conditions (at ultra-high temperatures, pressures and energy densities).
The modular supercomputer is designed on the basis of the Angara network, which allows you to combine any number of computers into a single computing cluster, developers say.
The project was implemented by specialists of the Research Center for Electronic Computing (part of "Roselektroniki").
Angara network
Angara" network is the first Russian solution for supercomputers of high performance, computing clusters for processing large data and calculations on the basis of parallel algorithms, the developers write. The network allows you to combine thousands of servers in a single system, including those of different manufacturers and with different processor architecture.
The first generation of Angara saw the light of day in 2016, when the United Instrumentation Corporation, a part of Rostec, developed a network adapter designed to connect computing clusters. The adapter was a PCI Express x16 board equipped with an ultra-large integrated circuit (VLSI).
VLSI for a network adapter
Samtec HDLSP connectors (up to eight connections) are provided for communication with other devices. VLSI EC8430 was developed by the Moscow "Research Center of Electronic and Computer Engineering" (RRCEVT) and is manufactured by Taiwanese company TSMC using 65 nm technology. The board of the network adapter is manufactured by SICEVT at its own production facilities.
In 2018, Roselektronika introduced the second generation of the network, in which the speed of data transfer between the connected computers has tripled compared to the first generation.
Angara communication network (1st generation). Principles of construction (from documentation).
See also the user manual
The second generation of Hangars allows you to create clusters from servers and workstations using conventional copper or fiber optic cables. The developers claim that "any trained user" will be able to connect the machines in this way. The communication delay in data exchange between the nodes has been reduced to 0.85 s.
Features of the Fisher modular supercomputer and interconnect
Network topology: 4D-torus
Adapter based on VLSI
Different physical media for data transmission
8 communication channels
Delay on MPI: 0.85 s
Delay on hop: 130 ns
Scaling up to 32K nodes
Switchgear and switchgearless design
Support for RDMA and GPUDirect
Determined and adaptive routing
Routing bypassing failed nodes and communication channels
Reliable data transfer protocol via communication channel
Linux kernel OS support
Support for Astra Linux Special Edition
Support of parallel programming tools OpenSHMEM, MPI, OpenMP
Power consumption up to 20 W
"Fisher"
"Fisher" is the first supercomputer on the basis of "Angara" in the switch version, which allows you to create supercomputers with a higher density of layout (compared to the switchless version) and facilitates installation and further operation of the computer system by reducing the number of cables used for switching.
"Complex calculations are required for scientific researches, training of neural networks, processing of the big volumes of the data, modeling of characteristics of new products of the industrial enterprises. At the same time, computational resources of supercomputers are extremely in demand and expensive, - says Oleg Yevtushenko, Executive Director of Rostec State Corporation. - The "Angara" network successfully solves these tasks, allowing to form supercomputers on the basis of separate computational nodes at a relatively low cost and in a short time. For several years now, the OIVT RAS has been using a supercomputer DESMOS with the capacity of 52.24 Tflops, created on the basis of the previous generation of the Angara network.
Its computing power turned out to be so much in demand by scientists that it was decided to create a "younger brother" of this supercomputer on the basis of a new generation of switching network. Its performance is designed for specific tasks, but if necessary, Fisher's capabilities can be significantly expanded.
The previous cluster consisted of 32 nodes with VLSI-based adapters and Intel Xeon E5-1650 v3 processors.
"Fisher" consists of 24 computational nodes with 16-core processors. The computing cluster uses an immersion (immersion) cooling system that provides uniform and energy-efficient thermal regulation.
Due to this solution, the supercomputer does not require specially prepared premises, and the range of cluster operating temperatures is from -50°C to +50°C.
Previously, immersion cooling was considered a difficult to implement from an engineering point of view and poorly scalable solution, but now among the world's most high-performance systems (from the Top 500 ranking) includes several rather compact and extremely economical systems with immersion cooling.
Currently, the most powerful supercomputer in Russia is the Lomonosov-2 supercomputer installed in MSU, manufactured by T-Platforms (peak performance 4.9 petaflops, according to Linpack 2.5 petaflops test). In second place is a supercomputer manufactured by T-Platforms and CRAY, installed in the main computer center of the Federal Service for Hydrometeorology and Environmental Monitoring (1.2 petaflops for Linpack). The Lomonosov system (T-Platforms, MSU) with a capacity of 901.9 teraflops per Linpack closes the top three.