Other Resources

OCCA Memory

TODO

deviceMemory occa::memory memory Pool

Mesh Class (mesh_t)

Each solver in NekRS owns a corresponding mesh_t object.

• For the fluid (velocity / pressure) solver, the mesh is stored in nrs->fluid->mesh (often also referred to as the fluid mesh).

• For scalar fields (e.g., temperature), meshes are accessed through the scalar, for example, nrs->scalar->mesh("temperature") or by index via nrs->scalar->mesh(idx), where the idx can be accessed via idx = nrs->scalar->nameToIndex.find("temperature")

In most cases, all scalar meshes point to the same fluid-domain mesh. However, in CHT setups, selected scalars (such as temperature) can live on a mesh that covers both fluid and solid regions. (See conjugate heat transfer turorial)

NekRS performs a domain decomposition to distribute the mesh across MPI ranks with approximately equal degrees of freedom, in order to balance the computational load. As a result, the members of mesh_t correspond to the local portion of the mesh assigned to a given MPI rank. For example, Nelements is the number of elements owned locally by that rank.

In the summary table below, we refer generically to the mesh object (e.g., mesh->Nelements) without distinguishing whether it is the fluid mesh, a scalar mesh, or a CHT mesh. The meaning of each member is the same, but applied to the local partition held by that particular solver instance.

If there are both host and device, the device one will have prefix o_

Table 27 Important mesh_t members

Variable Name	Size	Host or Device?	Meaning
dim = 3	1	Host	spatial dimension of mesh
Nfaces = 6	1	Host	number of faces per element
Nverts = 8	1	Host	number of vertices per element
NfaceVertices = 4	1	Host	number of vertices per face
N	1	Host	polynomial order for each dimension
Nq = N + 1	1	Host	number of quadrature points in each direction
Np = Nq * Nq * Nq	1	Host	number of quadrature points per element
Nfp = Nq * Nq	1	Host	number of quadrature points per face
Nfields	1	Host	number of fields passed to the PDE solver
Nelements	1	Host	number of local elements owned by current process
Nlocal = Nelements * Np	1	Host	number of local quadrature points owned by current process
NboundaryFaces	1	Host	total number of faces on a boundary (rank sum)
solid	1	Host	whether contains solid domain (1 = CHT)
elementInfo	Nelements	Both	phase of element (0 = fluid, 1 = solid)
EToV	Nelements * Nverts	Host	element to vertex connectivity
EToE	Nelements * Nfaces	Host	element to element connectivity
EToF	Nelements * Nfaces	Host	element to local face connectivity
EToP	Nelements * Nfaces	Host	element to partition/process connectivity
EToB	Nelements * Nfaces	Both	mapping of elements to type of boundary condition
vmapM	Nelements * Nfaces * Nfp	Both	quadrature point index for faces on boundaries
ogs	ogs_t	ogs_t	OCCA gather/scatter operation handle
oogs	oogs_t	oogs_t	(overlapped) OCCA gather/scatter operation handle
o_x	Nelements * Np	device	\(x\)-coordinates of physical quadrature points
o_y	Nelements * Np	device	\(y\)-coordinates of physical quadrature points
o_z	Nelements * Np	device	\(z\)-coordinates of physical quadrature points
gllz	Nq	Both	1D GLL points
gllw	Nq	Both	1D GLL quadratule weights
D	Nq * Nq	Both	1D Differentiation matrix
o_LMM (or o_Jw)	Nlocal	Device	lumped mass matrix
o_invLMM (or o_invAJw)	Nlocal	Device	inverse of assembled lumped mass matrix
o_vgeo	Nlocal * 12	Device	volume geometric factors
o_sgeo	Nelements * Nfaces * Nfp * 19	Device	surface geometric factors
o_ggeo	Nlocal * 7	Device	second-order volumetric geometric factors

Note

For the mapping from reference coordinates \((r,s,t)\) to physical coordinates \((x,y,z)\), the geometric factors are stored in three main device arrays:

• o_vgeo: volume geometric factors

  • the 9 components \(\partial (r,s,t) / \partial (x,y,z)\),

  • the Jacobian \(J\),

  • the weighted Jacobian \(J w\),

  • and its inverse \(1 / (J w)\).

• o_sgeo: surface geometric factors (on face nodes). Only 18 of the 19 stored entries are currently used:

  • the 3 components of the outward unit normal,

  • 6 components for the tangential and bitangential vectors,

  • the surface Jacobian and weighted surface Jacobian,

  • 3 entries for \(n \cdot \nabla r\), \(n \cdot \nabla s\), \(n \cdot \nabla t\),

  • 3 entries for averaged normal/tangential data.

• o_ggeo: geometric factors for the 3D Laplacian

  • the 6 unique components of the symmetric metric tensor \(G_{ij}\),

  • and the weighted Jacobian \(J w\).

Linear Algebra Functions (linAlg)

doxygen?

TODO

src/platform/linAlg/linAlg.tpp src/platform/linAlg/linAlg.hpp src/platform/linAlg/linAlg.cpp

SEM Operators (opSEM)

The opSEM namespace (see src/core/opSEM.hpp for arguments) collects commonly used spectral-element derivative-based operators. By default, the operators correspond to the weak form, while functions with a strong suffix implement the corresponding weighted strong form. Strong functions come with an optional argument avg (default: true) that returns the unweighted version.

Note

In the finite element setting, let \(u\) be a field, \(v\) a test function, and \(F\) a differential operator. The (weighted) strong form appears in integrals of the type

\[\int_\Omega v\, F[u] \,\mathrm{d}\vec{x}.\]

If \(F\) can be written in divergence form \(F[u] = \nabla \cdot \widehat{F}[u]\), with the associated flux \(\widehat{F}[u]\), integration by parts gives

\[\int_\Omega v\, F[u] \,\mathrm{d}\vec{x} = - \int_\Omega (\nabla v) \cdot \widehat{F}[u] \,\mathrm{d}\vec{x} + \int_{\partial\Omega} v\, \widehat{F}[u]\cdot\vec{n}\,\mathrm{d}s.\]

Neglecting the boundary term, the weak form is the volume term with a minus sign. In opSEM, the weak operators implement

\[\int_\Omega (\nabla v) \cdot \widehat{F}[u] \,\mathrm{d}\vec{x},\]

i.e., the same expression without the minus sign.

After discretization, testing with all basis functions turns this bilinear form into a linear system for the coefficients (nodal values) of \(u\). The test functions \(v\) no longer appear explicitly in the algebraic system.

Table 28 lists the opSEM functions and their corresponding discrete operators. The following notations are used:

• \(D_x, D_y, D_z\): first-order derivative operators in the \(x\), \(y\), and \(z\) directions.

• \(B\): multiplication by the diagonal (lumped) mass matrix.

• \(G\): gather–scatter averaging, \(G = M^{-1} Q Q^{T}\), where \(Q\) is the finite-element gather matrix, \(Q^{T}\) is the scatter, and \(M\) is a diagonal matrix of nodal multiplicities.

• \(\Lambda\): multiplication by a scalar field \(\lambda\) (a diagonal operator in the discrete setting).

Table 28 opSEM functions applied to a discretized scalar \(u\) or a vector \(\vec{u} = (u_x, u_y, u_z)\)

Function	In	Out	Math	Avg mode (avg=true)
grad	Scalar	Vector (3)	\([B D_x u,\; B D_y u,\; B D_z u]\)	NA
strongGrad	Scalar	Vector (3)	Same as grad	\([G D_x u,\; G D_y u,\; G D_z u]\)
strongGradVec	Vector (3)	Vector (9)	strongGrad of \(u_x\), then of \(u_y\) and of \(u_z\)	Applying strongGrad(avg=true) to 3 components
divergence	Vector (3)	Scalar	\(B (D_x^{T} u_x + D_y^{T} u_y + D_z^{T} u_z)\)	NA
strongDivergence	Vector (3)	Scalar	\(B (D_x u_x + D_y u_y + D_z u_z)\)	\(G (D_x u_x + D_y u_y + D_z u_z)\)
laplacian	Scalar	Scalar	\((D_x^{T} \Lambda B D_x + D_y^{T} \Lambda B D_y + D_z^{T} \Lambda B D_z)\, u\)	NA
strongLaplacian	Scalar	Scalar	\(B \big(D_x \Lambda G D_x + D_y \Lambda G D_y + D_z \Lambda G D_z\big) u\)	\(G \big(D_x \Lambda G D_x + D_y \Lambda G D_y + D_z \Lambda G D_z\big) u\)
strongCurl	Vector (3)	Vector (3)	\([B (D_y u_z - D_z u_y),\; B (D_z u_x - D_x u_z),\; B (D_x u_y - D_y u_x)]\)	\([G (D_y u_z - D_z u_y),\; G (D_z u_x - D_x u_z),\; G (D_x u_y - D_y u_x)]\)

The derivative operators commonly used in postprocessing are listed in Table 29.

Table 29 Lookup table for opSEM functions

Operators	opSEM function
\(\nabla u\)	strongGrad(avg=true)
\(\nabla \vec u\)	strongGradVec(avg=true)
\(\nabla \cdot \vec u\)	strongDivergence(avg=true)
\(\nabla \cdot (\lambda \nabla u)\)	strongLaplacian(avg=true)
\(\nabla\times\vec u\)	strongCurl(avg=true)

For example, the following call computes the gradient of the velocity field:

auto mesh = nrs->meshV;
auto o_gradU = opSEM::strongGradVec(mesh, nrs->fieldOffset, nrs->fluid->o_U); // default: avg = true

The output o_gradU is a device buffer from the memory pool, with type deviceMemory<dfloat> o_out(mesh->dim * mesh->dim * nrs->fieldOffset); and its entries ordered as

\[\partial_x u_x,\; \partial_y u_x,\; \partial_z u_x,\; \partial_x u_y,\; \partial_y u_y,\; \partial_z u_y,\; \partial_x u_z,\; \partial_y u_z,\; \partial_z u_z.\]

Building the Nek5000 Tools

NekRS does not package the legacy toolkit available with Nek5000, e.g., tools for creating or adapting meshes. It relies instead on the toolbox available with Nek5000, which must be installed separately. To build these scripts, clone the Nek5000 repository, and then navigate to the tools directory. The desired tools can be built here using ./maketools.

For example, if you want to make the genbox tool,

cd ~
git clone https://github.com/Nek5000/Nek5000.git
cd Nek5000/tools
./maketools genbox

This will create binary executables in the Nek5000/bin directory. You may want to add this folder to your PATH environment variable for quick access to these tools,

export PATH+=:$HOME/Nek5000/bin

Additional information about Nek5000 tools can be found here.

Note

The genbox tool creates simple 2D and 3D meshes that can be used for both Nek5000 and NekRS simulations. For NekRS applications, this is most likely the only tool that be of use to most users. Other Nek5000 tools will be rarely used.

Note

Some tools require additional dependencies such as cmake, wget (and internet access), or X11 libraries. Check the messages from the maketools script and the corresponding build.log files in each tool directory for any errors.