Case files
NekRS simulation requires a number of case definition files which are described in this page. An overview of these are presented in the image below. See also NekRS Files & Extensions for summarized tables of all NekRS files and their extensions.
There are a minimum of three files required to run a case:
Parameter file, with
.parextension. This sets simulation parameters used by the case and can be modified between runs.User-defined host file, with
.udfextension. This is used to set specific equations of the case, initial/boundary conditions, data outputs and other user definable behaviour.Mesh file, with
.re2extension. This file defines the geometry of the case.
Some optional files can also be included:
User-defined okl file, with
.oudfextension. This file conventionally has all the user defined device kernels. It must be included in OKL block of.udffile.Legacy Nek5000 user file, with
.usrextension. This file allows usage of Nek5000, legacy, Fortran 77 user routines.Session file, with
.sessextension. This file is for NekNek only. (See overlapping_overset_grids tutorial).
The case name is the default prefix applied to these files - for instance, a complete input description with a case name of “eddy” would be given by the files eddy.par, eddy.re2, eddy.udf.
Optionally, the user can also define the names of .udf, .oudf and .usr in the General Parameters section and name of .re2 file in Mesh Parameters section of .par file.
The following sections describe the structure and syntax for each of these files for a general case.
Parameter File (.par)
Tip
The user can access the manual containing specifications of the parameter file using nrsman as follows
nrsman par
Warning
nekRS .par file is not compatible or interchangeable with Nek5000 .par file.
Most information about the problem setup is defined in the parameter (.par) file.
This file is organized in a number of sections, each with a number of keys.
Values are assigned to these keys in order to control the simulation settings.
The general structure of the .par file is as follows, where FOO and BAR are both section names, with a number of (key, value) pairs.
# This is a comment
[FOO]
key = value # this is also a comment
baz = bat
[BAR]
alpha = beta
gamma = delta + keyword=value + ... # composed value with inline keywords
theta = value1, value2, value3 # comma-separated list
The valid sections for setting up a NekRS simulation are:
GENERAL: generic settings for the simulation (mandatory, see General Parameters)OCCA: backend OCCA device settings (optional, see OCCA section)PROBLEMTYPE: settings for the governing equations (optional, see Problem Type)MESH: mesh related settings (optional, see Mesh Parameters)GEOM: TODOField Settings (see Field Settings)
FLUID VELOCITY: settings for the velocity solverFLUID PRESSURE: settings for the pressure solverSCALAR: default scalar settingsSCALAR FOO: settings for theFOOscalar
BOOMERAMG: settings for the Hypre’s AMG solverNEKNEK: settings for the NekNek module in NekRS (see NekNek Parameters)CVODE: settings for the CVODE solver (see CVODE Parameters)
Note
Section name and key/value pairs are case insensitive
Values are words with all spaces removed
Values enclosed within quotes preserve case and whitespace
Values prefixed with ‘env::’ are interpreted as references to environment variables
Values separated by commas forms a list
User Sections
Custom sections may be added via the userSections key to pass additional keys into the code.
userSections = CASEDATA
...
[CASEDATA]
key = value
General Parameters
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
|
|
|
Enables/disables over-integration of convective term |
|
|
Polynomial order of |
|
|
|
|
|
String entry for the name of the logfile to direct NekRS output |
|
|
Restart from specified |
|
|
Order of time discretization for BDFk/EXTk scheme |
|
|
stop criterion |
|
|
Number of simulation time steps |
|
|
Simulation end time |
|
|
Simulation time in wall clock minutes |
|
|
Time step size |
|
|
Number of OIFS sub-steps for advection |
|
|
Specifies constant flow velocity |
|
|
Name of scalar fields to be solved |
|
|
Specifies engine to write field files |
|
|
Specifies precision of field files |
|
|
Specifies check point frequency control type |
|
|
Specifies check point frequency ( |
|
|
Optional name of user-defined host function file |
|
|
Optional name of user-defined OCCA kernel function file |
|
|
Optional name of user-defined legacy Nek5000 (fortran) function file |
|
Specifies regularization options for all fields |
OCCA Parameters
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Specifies the device for JIT compilation. Default is defined in |
|
|
Default is |
|
|
Only used by |
Problem Type Parameters
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Stokes solver |
Mesh Parameters
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Specifies mesh partitioner |
|
|
Map mesh boundary ids to 1,2,3,… |
|
|
Required for conjugate heat transfer cases |
|
|
Specifies mesh tolerance for partitioner |
|
|
Optional name of mesh ( |
|
|
Number of splits per direction along an element |
Field Settings
The sections for specific fields, including velocity (FLUID VELOCITY), pressure (FLUID PRESSURE) and scalars (SCALAR or SCALAR FOO) contain keys to describe linear solver setting for the corresponding field.
Most of the keys in the field sections are similar, described in Common Field Settings.
Some specific field keys are shown below:
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Fluid density |
|
|
Fluid dynamic viscosity |
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Specifies the mesh region where scalar |
|
|
Transport property for the scalar |
|
|
Diffusion coefficient for the scalar |
|
|
Transport property for the scalar |
|
|
Diffusion coefficient for the scalar |
Common Field Settings
The following table describes settings and corresponding keys for the linear solver.
The keys are common to all solution fields, including velocity, pressure and scalar fields.
These are to be included in the .par file under appropriate section for FLUID VELOCITY, FLUID PRESSURE, general SCALAR and specific scalar (SCALAR FOO).
Note
Linear solver settings for all scalar fields can be commonly specified under the SCALAR section.
Any setting under the specific SCALAR FOO section will override the common settings under SCALAR for FOO field
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Solve off |
|
|
absolute linear solver residual tolerance. Default = |
|
|
absolute solver tolerance (for CVODE only) |
|
|
. |
|
|
Default for velocity and scalars |
|
|
Linear finite element discretization. Default |
|
|
. |
|
|
custom polynomial order and Chebyshev order for each pMG level |
|
|
. |
|
|
Turns on/off checkpointing for specific field |
|
|
See Boundary conditions for details |
|
|
High-pass filter stabilization |
CVODE Parameters
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Linear solver |
|
|
|
|
|
relative tolerance |
|
|
ratio between linear and nonlinear tolerances |
|
|
step size for Jv difference quotient |
|
|
|
|
|
use same density field for all but the first scalar |
|
|
recycle property (freeze) evaluation for Jv |
|
|
NekNek Parameters
Key |
Value(s) |
Description/Note(s)/Default Value |
|---|---|---|
|
|
Boundary extrapolation order |
|
|
Default = |
User-Defined Host File (.udf)
A UDF (user-defined function) is an entry point NekRS calls during setup and time stepping, letting a case customize behavior without touching core code. Typical uses include setting initial and boundary conditions, defining custom materials, models, and source terms, and performing sampling and logging for post-processing. The available entry-point functions and their typical uses are as follows. All UDF functions are optional. If a function is not provided, NekRS uses a default no-op version.
OKL block
The .udf usually contains a preprocessor guard #ifdef __okl__ that encloses all OKL kernels and device functions compiled for the selected OCCA backend.
For an overview of the OKL language, see the OKL language guide.
NekRS provides default device functions for boundary conditions.
Details of the udfDirichlet and udfNeumann functions used for Dirichlet and Neumann boundary conditions, respectively, can be found in Boundary conditions.
#ifdef __okl__
void udfDirichlet(bcData *bc)
{
if(isField("fluid velocity")) {
bc->uxFluid = 1.0;
bc->uyFluid = 0.0;
bc->uzFluid = 0.0;
}
else if (isField("fluid pressure")) {
bc->pFluid = 0.0;
}
else if (isField("scalar temperature")) {
bc->sScalar = 0.0;
}
}
void udfNeumann(bcData *bc)
{
if(isField("fluid velocity")) {
bc->tr1 = 0.0;
bc->tr2 = 0.0;
}
else if (isField("scalar temperature")) {
bc->fluxScalar = 0.0;
}
}
#endif
Functions marked with the decorate @kernel are compiled as device kernels and can be launched from UDF host code directly as a regular function.
// -------- Device side (inside the OKL block) --------
#ifdef __okl__
@kernel void my_exact(const dlong Ntotal,
@ restrict const dfloat *X,
@ restrict dfloat *U)
{
for (dlong n = 0; n < Ntotal; ++n; @tile(p_blockSize, @outer, @inner)) {
if (n < Ntotal) {
U[n] = sin(X[n]);
}
}
}
#endif
// -------- Host side (UDF code) --------
{
auto mesh = nrs->meshV;
deviceMemory<dfloat> o_tmp(mesh->Nlocal);
my_exact(o_tmp.size(), mesh->o_x, o_tmp); // o_tmp = sin(x)
}
Tip
If the user-defined functions are sufficiently large, it is conventional practice to write them in a
.oudffile which is included within theifdefblock instead of the functions in the.udffile, as follows:
#ifdef __okl__
#include "case.oudf"
#endif
Tip
Many common operations are available in platform->linAlg and opSEM (e.g., fills, axpby, dot products, norms, gradient, divergence).
Prefer these utilities over writing custom kernels when possible.
See Linear Algebra Functions (linAlg) and SEM Operators (opSEM) for details.
UDF_Setup0
UDF_Setup0 is called once at startup, before NekRS initializes the mesh, solvers, or solution arrays.
It receives the NekRS MPI communicator (comm) and the user options object (options).
Because the simulation state is not yet constructed, this function is mainly for reading or adjusting user settings.
static dfloat P_GAMMA;
void UDF_Setup0(MPI_Comm comm, setupAide &options)
{
platform->par->extract("casedata","p_gamma",P_GAMMA);
}
void UDF_LoadKernels(deviceKernelProperties& kernelInfo)
{
kernelInfo.define("p_GAMMA") = P_GAMMA;
}
In this example, UDF_Setup0 reads the p_gamma key from the CASEDATA user section in the .par file (see User Sections) using the convenience helper platform->par->extract.
The value is stored in a global P_GAMMA and then exported as the device macro p_GAMMA in UDF_LoadKernels for JIT-compiled kernels.
Warning
Do not modify parameters used to compile legacy Nek5000 in UDF_Setup0 such as polynomialOrder.
UDF_LoadKernels
As shown in the example above, UDF_LoadKernels is primarily used to attach preprocessor macros (global defines) for device kernels.
It takes deviceKernelProperties& kernelInfo which holds compilation metadata.
Using kernelInfo.define("NAME") = value to expose constants to OKL.
Tip
For constants passed to device code (via UDF_LoadKernels), it’s recommended to use a p_ prefix (e.g., p_gamma, p_Pr, p_Re) for clarity and consistency.
UDF_Setup
UDF_Setup is called once at the beginning of the simulation after NekRS initializing the mesh, solution fields, material properties, and boundary mappings.
It is typically the .udf function users will use most to overwrite defaults and customize settings.
Various operations are performed within this routine, including, but not limited to:
Assign initial conditions (see Initial conditions).
Mesh manipulation (see RANS Channel tutorial).
Register function pointers for user-defined spatially varying material properties (see Physical properties).
Register function pointers for user-defined source terms (see Custom Source Terms).
Initialize and set up RANS turbulence models (see RANS models)
Initialize and set up the Low-Mach compressible model (see Low-Mach Compressible Model)
Initialize solution recycling routines and arrays (see Velocity recycling)
Allocate
bc->o_usrwrkfor user-defined boundary data (see TODO)Initialize time-averaging routines and buffers (see Time Averaging)
UDF_ExecuteStep
UDF_ExecuteStep offers the most flexibility.
It is called once just before time marching begins, and then once per time step.
The routine receives the current time (double t) and the step index (int tstep).
Typical operations include:
Run time-averaging updates (see Time Averaging)
Invoke solution recycling logic (see Velocity recycling).
Post-processing tasks, such as:
Extracting data over a line (see Data Extraction Along a Line).
Writing custom field files (see Adding Custom Output File).
Legacy Nek5000 User File (.usr)
NekRS includes an optional interface to Nek5000 for using legacy Fortran-77 user routines.
If <case>.usr is present in the case directory, the file is compiled and linked, though not all Nek5000 user subroutines are invoked by NekRS.
Many one-time setup routines are still honored so existing Nek5000 settings can be carried over.
Users can continually use subroutines usrdat0, usrdat, usrdat2, usrdat3, usrsetvert and useric to modify mesh geometry, tag boundary surfaces, adjust connectivity and set initial conditions.
Compute intensive subroutines such as userbc, userf, userq and uservp are not used by NekRS. Users need to convert them into UDF or OKL implementation.
The userchk is not called automatically.
If needed, call it manually from UDF (e.g., UDF_ExecuteStep) for post-processing or other setup tasks, for example:
void UDF_ExecuteStep(double time, int tstep)
{
if(nrs->checkpointStep) { // occasionally call when a checkpoint step
nrs->copyToNek(time, tstep); // push device field to Nek5000 arrays
nek::userchk(); // call userchk in .usr
nrs->copyFromNek(time); // only if userchk modify solutions
}
}
Note
nrs->copyToNekcopies all solution fields from OCCA device arrays to Nek5000 (Fortran) host arrays. Use it only whennek::userchkneeds current solutions.This transfer is expensive. Wrap the call under a suitable condition to avoid excessive overhead.
Call
nrs->copyFromNekonly if the legacy routine modifies solution data that must be synchronized back to the device.
For background on Nek5000, see the Nek5000 documentation.
Details on these initialization routines are documented here.
To migrate a case, follow the TODO tutorial for converting a Nek5000 case to NekRS.
In this section, we focus on the interface for referencing variables between .udf and .usr.
Legacy Data Interface
NekRS does not “send” values or arrays to Nek5000.
Instead, it shares them by registering pointers.
On the Fortran side, any variable or array you expose must have a stable address, so put it in a COMMON block or give it the SAVE attribute.
Use the Fortran routine nekrs_registerPtr to register variables/arrays so NekRS can reference them by a string key.
Here is an example .usr file:
c-----------------------------------------------------------------------
subroutine usrdat0
real gamma
save gamma
gamma = 1.4
call nekrs_registerPtr('gamma', gamma)
return
end
c-----------------------------------------------------------------------
subroutine userchk()
include 'SIZE'
include 'TOTAL'
common /exact/ uexact(lx1,ly1,lz1,lelt*3),
& texact(lx1,ly1,lz1,lelt)
real uexact, texact
call nekrs_registerPtr('uexact', uexact(1,1,1,1,1))
call nekrs_registerPtr('vexact', uexact(1,1,1,1,2))
call nekrs_registerPtr('wexact', uexact(1,1,1,1,3))
call nekrs_registerPtr('texact', texact)
! update uexact/texact here
return
end
In this example, COMMON /exact/ block holds two arrays store exact solutions (e.g., velocity and temperature).
We register the first-element addresses of each velocity components and of temperature with nekrs_registerPtr inside userchk.
The scalar gamma is registered in usrdat0 and kept alive with SAVE.
Any data you register must be in a COMMON block or SAVE so its address remains valid.
These pointers can then be looked up in the .udf and used to exchange data between NekRS and Nek5000. Example usage in .udf:
static dfloat gamma;
void UDF_Setup0(MPI_Comm comm, setupAide &options)
{
gamma = *nek::ptr<double>("gamma");
}
void UDF_LoadKernels(deviceKernelProperties& kernelInfo)
{
kernelInfo.define("p_GAMMA") = gamma;
}
void UDF_ExecuteStep(double time, int tstep)
{
if (nrs->lastStep) {
auto mesh = nrs->meshV;
auto Nlocal = mesh->Nlocal;
nek::userchk(); // compute exact solutions in .usr
// Host pointers exported from usr
auto *u_ex = nek::ptr<double>("uexact");
auto *v_ex = nek::ptr<double>("vexact");
auto *w_ex = nek::ptr<double>("wexact");
// Range constructor (copy values into std::vector)
std::vector<double> t_ex(nek::ptr<double>("texact"),
nek::ptr<double>("texact") + Nlocal);
// Host buffer
std::vector<dfloat> uexact(Nlocal), uexact(Nlocal), wexact(Nlocal), texact(Nlocal);
// Convert from double to dfloat
for (int i = 0; i < Nlocal; i++) {
u_exact[i] = static_cast<dfloat>(u_ex[i]);
v_exact[i] = static_cast<dfloat>(v_ex[i]);
w_exact[i] = static_cast<dfloat>(w_ex[i]);
t_exact[i] = static_cast<dfloat>(t_ex[i]);
}
// Device buffer
auto o_uexact = platform->device.malloc<dfloat>(nrs->fluid->fieldOffsetSum);
auto o_texact = platform->device.malloc<dfloat>(Nlocal);
// copyFrom(src pointer, count, dest offset, src offset)
o_uexact.copyFrom(uexact.data(), Nlocal, 0 * nrs->fluid->fieldOffset, 0);
o_uexact.copyFrom(vexact.data(), Nlocal, 1 * nrs->fluid->fieldOffset, 0);
o_uexact.copyFrom(wexact.data(), Nlocal, 2 * nrs->fluid->fieldOffset, 0);
o_texact.copyFrom(texact.data(), Nlocal);
//compute error here...
}
}
As shown, nek::ptr returns the registered pointers by the string keys defined in the .usr file.
In UDF_Setup0, the value referenced by gamma is read into a C++ static and later exported as a device macro in UDF_LoadKernels.
For the arrays (uexact, vexact, wexact, texact), we show two ways, viz., use a raw host pointer from nek::ptr<double>(...) for velocity components or use a range constructor to make a copy into std::vector for temperature.
Then, all are converted from double to dfloat before copying to OCCA arrays.
Note
NekRS compiles Fortran real as real*8 (i.e., C++ double).
In NekRS, dfloat may be double (default) or float (FP32 mode).
Converting on the host as shown works for both settings.
If you always run in FP64, you can skip the cast.
Alternatively, you can convert on device via platform->copyDoubleToDfloatKernel.
Note
Nek5000/NekRS sizes and offsets
mesh->Nlocal: number of local points on this rank (e.g.,lx1*ly1*lz1*nelv; fluid+solid mesh usesnelt).fieldOffset: padded device stride (alignment + max local size). It is not equal tolx1*ly1*lz1*lelv.fieldOffsetSum: sum of component strides.– For velocity:
fieldOffsetSum = 3 * fieldOffset.– For scalar:
fieldOffsetSum = nrs->Nscalar * fieldOffset.
Use fieldOffset/fieldOffsetSum for declaring multi-component device arrays and indexing.
Use Nlocal for loop lengths and buffer sizes.
Mesh File (.re2)
The .re2 mesh format originates from Nek5000, and NekRS uses the same binary format, so meshes made for Nek5000 remain compatible with NekRS.
This section summarizes the .re2 layout and conventions.
See Mesh & Meshing for mesh creation and conversion workflows.
You can obtain .re2 by:
Porting from an existing Nek5000 case,
Using a mesh converter from other format, or
Generating it with Nek5000 meshing tools.
There are three main limitations for the NekRS mesh:
3D hex only: meshes must be hexahedral and three-dimensional.
Contiguous boundary IDs: face boundary IDs must be 1, 2, 3, … without gaps.
Element types: the main supported types are HEX8 and HEX20.
Tip
Lower-dimensional (2D/1D) setups can run on a 3D mesh by enforcing zero-gradient boundary conditions on all solution variables in the out-of-interest directions. Material properties and forcing terms must likewise be constant (or zero-gradient) in those directions. See Connectivity for examples.
The header of a .re2 file is a single 80-character line.
While minor variants exist across versions, they all encode: (version) (total elements) (dimension) (fluid elements).
For example, the ethier.re2 has 32 elements:
#v003 32 3 32 this is the hdr
Note
In conjugate heat transfer (CHT) cases, one mesh file contains both fluid and solid regions, with the fluid mesh stored first, followed by the solid mesh. Consequently, the total element count exceeds the fluid count.
conj_ht example (96 fluid + 96 solid = 192 total, 3D):
#v002 192 3 96 this is the hdr
In CHT runs, make sure you use the correct mesh in your .udf.
Typically, reference the mesh object under the corresponding solver:
nrs->fluid->meshfor fluid solver meshnrs->scalar->mesh("temperature")or, if it’s the first scalar,nrs->scalar->mesh[0]for temperature mesh
See the Conjugate Heat Transfer tutorial for details, and Conjugate Heat Transfer Meshing for creating CHT meshes.
After the header and an endianness check, a .re2 file stores, for each element, the coordinates of its 8 vertices (HEX8).
If curvature is present, upto 12 mid-edge nodes are added, yielding HEX20.
Some legacy curved primitives (e.g., cylinders or spheres) use special curve records for higher-order accuracy, but in general edges are represented by second-order polynomials.
If needed, users can recover exact geometry in usrdat2 or by reading all higher-order grid points from a checkpoint file.
Boundary faces are tagged for boundary conditions.
The fluid mesh is always present, and scalars may have their own tag sets; in CHT cases, fluid and solid regions each have separate boundary maps.
These tags are later mapped to Nek5000 arrays CBC(f,e,ifield) and to boundary IDs boundaryID(f,e) (and boundaryIDt(f,e) for CHT).
NekNek Session File (.sess)
NekNek allows coupling of multiple overlapping subdomains, relaxing the need for conformal meshes.
Each NekRS instance uses its own .par file.
In the .sess file, each line lists one instance as:
<path-to-par-file>:<num-mpi-ranks>;
Paths may be absolute or relative, and the sum of ranks over all instances must equal the total MPI ranks requested. Here is the eddyNekNek.sess example:
inside/inside:1;
outside/outside:1;
Tip
Using a subfolder per case is optional but recommended. On HPC systems, try to size each session in units of a node. For example, if a node has 4 GPUs, it’s often best to make each session’s MPI ranks a multiple of 4.
Trigger File (nekrs.upd)
The trigger file lets you change selected parameters at runtime so you can adjust a simulation without killing the job.
Before launching, set a catchable signal in NEKRS_SIGNUM_UPD (see Runtime Control with Signals).
At runtime, write the requested options in nekrs.upd under the case directory.
Accepted keys include:
checkpoint = true
[GENERAL]
endTime = 10.0
numSteps = 10000
writeinterval = 1.0
When NekRS receives the configured signal, it reloads nekrs.upd and applies the changes on the next timestep. You can trigger this multiple times during a run.
NekRS Files & Extensions
Extension |
Name |
Required? |
Notes / Typical usage |
|---|---|---|---|
.sess |
NekNek sessions file |
Yes (Only for NekNek) |
|
.par |
Main parameter file |
Yes |
|
.udf |
User-defined functions |
Yes |
|
.oudf |
Extra OKL kernels file |
No |
See OKL block |
.upd |
Trigger file |
No |
|
.usr |
Legacy Nek5000 user file |
No |
|
.re2 |
Mesh file |
Yes |
See Mesh File (.re2) |
.co2 |
Legacy connectivity file |
No |
|
.yaml |
YAML config file |
No |
Used for |
Extension |
Name |
Notes / Typical usage |
|---|---|---|
*0.f0000 |
Nek format checkpoint file |
0-based index. See Visualization |
.nek5000 |
Nek format checkpoint metadata file |
See Visualization |
.bp/ |
ADIOS2 format checkpoint (folder) |
|
.log.* |
Log files |
From |
.vtu/.vtk |
VTK files |
Mainly for particle data |