Turbulence is ubiquitous in environmental and engineering fluid flows, and is encountered routinely in on a regular basis life. A greater understanding of those turbulent processes may present beneficial insights throughout a wide range of analysis areas — enhancing the prediction of cloud formation by atmospheric transport and the spreading of wildfires by turbulent power alternate, understanding sedimentation of deposits in rivers, and enhancing the effectivity of combustion in plane engines to cut back emissions, to call a couple of. Nevertheless, regardless of its significance, our present understanding and our potential to reliably predict such flows stays restricted. That is primarily attributed to the extremely chaotic nature and the big spatial and temporal scales these fluid flows occupy, starting from energetic, large-scale actions on the order of a number of meters on the high-end, the place power is injected into the fluid move, all the way in which right down to micrometers (μm) on the low-end, the place the turbulence is dissipated into warmth by viscous friction.
A strong device to grasp these turbulent flows is the direct numerical simulation (DNS), which gives an in depth illustration of the unsteady three-dimensional flow-field with out making any approximations or simplifications. Extra particularly, this method makes use of a discrete grid with sufficiently small grid spacing to seize the underlying steady equations that govern the dynamics of the system (on this case, variable-density Navier-Stokes equations, which govern all fluid move dynamics). When the grid spacing is sufficiently small, the discrete grid factors are sufficient to signify the true (steady) equations with out the lack of accuracy. Whereas that is enticing, such simulations require super computational sources to be able to seize the right fluid-flow behaviors throughout such a variety of spatial scales.
The precise span in spatial decision to which direct numerical calculations should be utilized is determined by the duty and is decided by the Reynolds quantity, which compares inertial to viscous forces. Usually, the Reynolds quantity can vary between 102 as much as 107 (even bigger for atmospheric or interstellar issues). In 3D, the grid measurement for the decision required scales roughly with the Reynolds quantity to the facility of 4.5! Due to this robust scaling dependency, simulating such flows is mostly restricted to move regimes with average Reynolds numbers, and usually requires entry to high-performance computing programs with tens of millions of CPU/GPU cores.
In “A TensorFlow simulation framework for scientific computing of fluid flows on tensor processing items”, we introduce a brand new simulation framework that allows the computation of fluid flows with TPUs. By leveraging newest advances on TensorFlow software program and TPU-hardware structure, this software program device permits detailed large-scale simulations of turbulent flows at unprecedented scale, pushing the boundaries of scientific discovery and turbulence evaluation. We reveal that this framework scales effectively to accommodate the dimensions of the issue or, alternatively, improved run occasions, which is exceptional since most large-scale distributed computation frameworks exhibit diminished effectivity with scaling. The software program is on the market as an open-source mission on GitHub.
Massive-scale scientific computation with accelerators
The software program solves variable-density Navier-Stokes equations on TPU architectures utilizing the TensorFlow framework. The only-instruction, multiple-data (SIMD) method is adopted for parallelization of the TPU solver implementation. The finite distinction operators on a colocated structured mesh are forged as filters of the convolution operate of TensorFlow, leveraging TPU’s matrix multiply unit (MXU). The framework takes benefit of the low-latency high-bandwidth inter-chips interconnect (ICI) between the TPU accelerators. As well as, by leveraging the single-precision floating-point computations and extremely optimized executable by way of the accelerated linear algebra (XLA) compiler, it’s doable to carry out large-scale simulations with glorious scaling on TPU {hardware} architectures.
This analysis effort demonstrates that the graph-based TensorFlow together with new varieties of ML particular objective {hardware}, can be utilized as a programming paradigm to unravel partial differential equations representing multiphysics flows. The latter is achieved by augmenting the Navier-Stokes equations with bodily fashions to account for chemical reactions, heat-transfer, and density adjustments to allow, for instance, simulations of cloud formation and wildfires.
It’s value noting that this framework is the primary open-source computational fluid dynamics (CFD) framework for high-performance, large-scale simulations to completely leverage the cloud accelerators which have develop into widespread (and develop into a commodity) with the development of machine studying (ML) lately. Whereas our work focuses on utilizing TPU accelerators, the code will be simply adjusted for different accelerators, resembling GPU clusters.
This framework demonstrates a strategy to vastly cut back the fee and turn-around time related to working large-scale scientific CFD simulations and permits even better iteration pace in fields, resembling local weather and climate analysis. Because the framework is carried out utilizing TensorFlow, an ML language, it additionally permits the prepared integration with ML strategies and permits the exploration of ML approaches on CFD issues. With the final accessibility of TPU and GPU {hardware}, this method lowers the barrier for researchers to contribute to our understanding of large-scale turbulent programs.
Framework validation and homogeneous isotropic turbulence
Past demonstrating the efficiency and the scaling capabilities, it’s also crucial to validate the correctness of this framework to make sure that when it’s used for CFD issues, we get cheap outcomes. For this objective, researchers usually use idealized benchmark issues throughout CFD solver improvement, lots of which we adopted in our work (extra particulars within the paper).
One such benchmark for turbulence evaluation is homogeneous isotropic turbulence (HIT), which is a canonical and effectively studied move during which the statistical properties, resembling kinetic power, are invariant beneath translations and rotations of the coordinate axes. By pushing the decision to the bounds of the present state-of-the-art, we had been in a position to carry out direct numerical simulations with greater than eight billion levels of freedom — equal to a three-dimensional mesh with 2,048 grid factors alongside every of the three instructions. We used 512 TPU-v4 cores, distributing the computation of the grid factors alongside the x, y, and z axes to a distribution of [2,2,128] cores, respectively, optimized for the efficiency on TPU. The wall clock time per timestep was round 425 milliseconds and the move was simulated for a complete of 400,000 timesteps. 50 TB knowledge, which incorporates the speed and density fields, is saved for 400 timesteps (each 1,000th step). To our information, this is likely one of the largest turbulent move simulations of its sort performed thus far.
As a result of complicated, chaotic nature of the turbulent move discipline, which extends throughout a number of magnitudes of decision, simulating the system in excessive decision is important. As a result of we make use of a fine-resolution grid with eight billion factors, we’re in a position to precisely resolve the sphere.
 |
| Contours of x-component of velocity alongside the z midplane. The excessive decision of the simulation is crucial to precisely signify the turbulent discipline. |
The turbulent kinetic power and dissipation charges are two statistical portions generally used to research a turbulent move. The temporal decay of those properties in a turbulent discipline with out further power injection is because of viscous dissipation and the decay asymptotes observe the anticipated analytical energy regulation. That is in settlement with the theoretical asymptotes and observations reported within the literature and thus, validates our framework.
 |
| Stable line: Temporal evolution of turbulent kinetic power (ok). Dashed line: Analytical energy legal guidelines for decaying homogeneous isotropic turbulence (n=1.3) (Ⲧl: eddy turnover time). |
 |
| Stable line: Temporal evolution of dissipation charge (ε). Dashed line: Analytical energy legal guidelines for decaying homogeneous isotropic turbulence (n=1.3). |
The power spectrum of a turbulent move represents the power content material throughout wavenumber, the place the wavenumber ok is proportional to the inverse wavelength λ (i.e., ok ∝ 1/λ). Typically, the spectrum will be qualitatively divided into three ranges: supply vary, inertial vary and viscous dissipative vary (from left to proper on the wavenumber axis, under). The bottom wavenumbers within the supply vary correspond to the most important turbulent eddies, which have essentially the most power content material. These massive eddies switch power to turbulence within the intermediate wavenumbers (inertial vary), which is statistically isotropic (i.e., basically uniform in all instructions). The smallest eddies, equivalent to the most important wavenumbers, are dissipated into thermal power by the viscosity of the fluid. By advantage of the tremendous grid having 2,048 factors in every of the three spatial instructions, we’re in a position to resolve the move discipline as much as the size scale at which viscous dissipation takes place. This direct numerical simulation method is essentially the most correct because it doesn’t require any closure mannequin to approximate the power cascade under the grid measurement.
 |
| Spectrum of turbulent kinetic power at totally different time cases. The spectrum is normalized by the instantaneous integral size (l) and the turbulent kinetic power (ok). |
A brand new period for turbulent flows analysis
Extra just lately, we prolonged this framework to foretell wildfires and atmospheric flows, which is related for climate-risk evaluation. Aside from enabling high-fidelity simulations of complicated turbulent flows, this simulation framework additionally gives capabilities for scientific machine studying (SciML) — for instance, downsampling from a tremendous to a rough grid (mannequin discount) or constructing fashions that run at decrease decision whereas nonetheless capturing the right dynamic behaviors. It may additionally present avenues for additional scientific discovery, resembling constructing ML-based fashions to raised parameterize microphysics of turbulent flows, together with bodily relationships between temperature, strain, vapor fraction, and many others., and will enhance upon numerous management duties, e.g., to cut back the power consumption of buildings or discover extra environment friendly propeller shapes. Whereas enticing, a major bottleneck in SciML has been the supply of information for coaching. To discover this, we’ve got been working with teams at Stanford and Kaggle to make the information from our high-resolution HIT simulation accessible by way of a community-hosted web-platform, BLASTNet, to supply broad entry to high-fidelity knowledge to the analysis group through a network-of-datasets method. We hope that the supply of those rising high-fidelity simulation instruments at the side of community-driven datasets will result in vital advances in numerous areas of fluid mechanics.
Acknowledgements
We wish to thank Qing Wang, Yi-Fan Chen, and John Anderson for consulting and recommendation, Tyler Russell and Carla Bromberg for program administration.