17  Field Solver

View:

Space-charge effects are attached to the tracking setup through a FIELDSOLVER command and then referenced from RUN as described in the tracking chapter. The standard model is fully three-dimensional. The field solve is based on Poisson’s equation with open boundaries, either through FFT-based convolution, iterative irregular-domain solvers, or adaptive mesh refinement.

17.1 FFT Based Particle-Mesh (PM) Solver

The classical particle-mesh solver discretizes the charge density on a regular Cartesian mesh

\[ \Omega = [-L_x, L_x] \times [-L_y, L_y] \times [-L_t, L_t] \]

with the corresponding grid points. Instead of computing all pairwise particle-particle interactions, the charge is distributed across the mesh and Poisson’s equation is solved there.

For isolated bunches the potential satisfies

\[ \nabla^2 \phi = -\rho / \epsilon_0 \tag{17.1}\]

and can be written as a convolution of the charge density with the open-space Green function. The FFT-based solver uses the Hockney zero-padding construction so that the open-boundary convolution can be evaluated with periodic FFTs on an enlarged mesh.

17.1.1 FFT-based Convolutions and Zero Padding

The key idea is:

  1. deposit the physical charge density on the unpadded mesh
  2. enlarge the mesh so the convolution can be represented as a periodic one
  3. construct the periodic Green-function array
  4. solve the convolution in Fourier space
  5. retain the potential and field only in the physical region

This gives the physical open-boundary solution while preserving the efficiency of FFTs.

17.1.2 Algorithm

In practical terms, the FFT space-charge solve proceeds as:

  1. deposit charge to the mesh
  2. build or reuse the Green-function array
  3. perform the FFT-based convolution solve
  4. compute the electric field on the mesh
  5. interpolate the field back to particles

The method scales much better than direct particle-particle summation and is the standard stable space-charge path.

17.1.3 Interpolation Schemes

The interpolation from mesh fields back to particles follows the standard particle-mesh deposition/interpolation choices used. The legacy manual reserves the detailed interpolation discussion for later revisions, but the solver is intended for ordinary 3D PIC workflows.

17.2 Particle-Particle-Particle-Mesh (P3M) Solver

The P3M solver splits the total electrostatic force into a short-range direct part and a long-range mesh part,

\[ F = F_{\mathrm{sr}} + F_{\mathrm{lr}} \tag{17.2}\]

so that close encounters are handled by direct summation inside a cutoff radius, while the smooth long-range contribution is handled on the mesh.

In this formulation the Green function is likewise split into short-range and long-range components. OPAL supports two practical choices:

  • STANDARD: the Ewald-type splitting with interaction parameter ALPHA
  • INTEGRATED: an integrated polynomial form with a convenient closed-form cell integration

17.2.1 Use of P^3M solver

The legacy implementation notes are important:

  • P3M is only available in OPAL-T
  • emission must not be active
  • the solver uses OPEN boundary conditions
  • ALPHA is only used together with GREENSF=STANDARD

Example:

REAL inter_rad = 3.125e-5;

FS_P3M: FIELDSOLVER, FSTYPE="P3M", MX=64, MY=64, MT=64,
        PARFFTX=decx, PARFFTY=decy, PARFFTT=decz,
        RC=inter_rad, GREENSF=INTEGRATED;

17.3 Iterative Space Charge Solver

The iterative Poisson solver is intended for irregular geometries where the plain FFT open-boundary solve is not the best model. The discretization is based on finite differences, and the resulting linear system is solved by a preconditioned iterative method with smoothed-aggregation algebraic multigrid preconditioning.

Compilation requirements:

  • ENABLE_SAAMG_SOLVER=ON
  • Parmetis
  • Trilinos

This is the solver family behind FSTYPE=SAAMG.

17.4 Energy Binning

When the bunch energy spread is too large for a single boosted-frame electrostatic approximation, OPAL can split the bunch into several energy bins and perform multiple field solves. This is the basis of the multi-Lorentz-frame approximation.

In cyclotron multi-bunch mode the number of energy bins should be at least the number of neighboring bunches represented by NNEIGHBB. The companion control parameter MINSTEPFORREBIN defines after how many integration steps the energy bins are merged again.

17.5 The FIELDSOLVER Command

The FIELDSOLVER command selects the solver type, mesh resolution, domain decomposition, boundary conditions, and optional solver-specific parameters.

17.5.1 Core command attributes

Attribute Meaning
TYPE Solver type: NONE, FFT, OPEN, CG
PARFFTX Parallelized grid in x direction
PARFFTY Parallelized grid in y direction
PARFFTZ Parallelized grid in z direction
NX Number of mesh points in x
NY Number of mesh points in y
NZ Number of mesh points in z
BCFFTX Boundary condition in x
BCFFTY Boundary condition in y
BCFFTZ Boundary condition in z
GREENSF Green-function choice for FFT-based solvers
BBOXINCR Bounding-box enlargement factor in percent
BINS Name of a BINNING definition, or NONE for no binning

17.5.2 Fieldsolver type

TYPE selects the actual solver backend. At present FFT based solver is the most stable solver.

17.5.3 Domain Decomposition

PARFFTX, PARFFTY, and PARFFTZ decide whether the corresponding mesh direction is distributed over MPI processes.

Default:

  • PARFFTX = FALSE
  • PARFFTY = FALSE
  • PARFFTZ = TRUE

So the conventional decomposition is serial in x and y, parallel in z.

17.5.4 Number of Grid Points

NX, NY, and NT define the rectangular mesh resolution in x, y, and z. These values determine both the field resolution and the FFT problem size, so they are the main tuning parameters for accuracy and cost.

17.5.5 Boundary Conditions

Boundary conditions are specified independently per axis:

  • BCFFTX: OPEN (default), DIRICHLET or PERIODIC
  • BCFFTY: OPEN (default), DIRICHLET or PERIODIC
  • BCFFTZ: OPEN(default), DIRICHLET or PERIODIC

17.5.6 Core command attributes

Attribute Meaning
FSTYPE Solver type: FFT, FFTPERIODIC, SAAMG, P3M, NONE
PARFFTX Distribute the x direction over MPI ranks
PARFFTY Distribute the y direction over MPI ranks
PARFFTT Distribute the z direction over MPI ranks
MX Number of mesh points in x
MY Number of mesh points in y
MT Number of mesh points in z
BCFFTX Boundary condition in x
BCFFTY Boundary condition in y
BCFFTZ Boundary condition in z
GREENSF Green-function choice for FFT-based solvers
BBOXINCR Bounding-box enlargement factor in percent
GEOMETRY Geometry list for irregular-domain solves
ITSOLVER Iterative linear solver
INTERPL Boundary-point interpolation scheme
TOL Iterative solver tolerance
MAXITERS Maximum number of iterations
PRECMODE Preconditioner reuse policy
RC P3M short-range cutoff radius
ALPHA P3M interaction splitting parameter
ENBINS Number of energy bins
MINSTEPFORREBIN Merge energy bins after this many steps

17.5.7 Fieldsolver type

FSTYPE selects the actual solver backend.

  • FFT: standard open-boundary FFT-based particle-mesh solver
  • FFTPERIODIC: FFT solver with periodic longitudinal treatment
  • SAAMG: iterative irregular-domain solver
  • P3M: mixed mesh plus short-range direct solver
  • NONE: disable space-charge field solving

FFT solver is the most stable default path.

17.5.8 Domain Decomposition

PARFFTX, PARFFTY, and PARFFTT decide whether the corresponding mesh direction is distributed over MPI processes.

Legacy default:

  • PARFFTX = FALSE
  • PARFFTY = FALSE
  • PARFFTT = TRUE

So the conventional decomposition is serial in x and y, parallel in z.

17.5.9 Number of Grid Points

MX, MY, and MT define the rectangular mesh resolution in x, y, and z. These values determine both the field resolution and the FFT problem size, so they are the main tuning parameters for accuracy and cost.

17.5.10 Boundary Conditions

Boundary conditions are specified independently per axis:

  • BCFFTX: OPEN
  • BCFFTY: OPEN
  • BCFFTZ: OPEN or PERIODIC

Using PERIODIC in z is the standard way to model a DC beam rather than an isolated bunch.

17.5.11 Greens Function

GREENSF controls the Green-function model for FFT-based solvers and P3M.

Allowed values:

  • INTEGRATED
  • STANDARD

The default is INTEGRATED. This is the usual recommendation for FFT-based space-charge calculations.

17.5.12 Bounding Box Enlargement

The solver constructs a minimal rectangular box that encloses all particles. BBOXINCR enlarges that box by a user-specified percentage before the solve.

Default:

  • BBOXINCR = 2.0

This is often used to avoid an overly tight computational box around the bunch.

17.5.13 Geometry

GEOMETRY attaches a list of geometry objects that define the boundary of the computational domain for the irregular-domain solver. This option is relevant to SAAMG workflows.

17.5.14 Iterative Solver

ITSOLVER selects the Krylov or direct linear solver used by the iterative field-solver family.

For SAAMG, the documented choices are:

  • CG (default)
  • BICGSTAB
  • GMRES

17.5.15 Interpolation for Boundary Points

INTERPL controls how grid points near the irregular boundary are interpolated.

Allowed values:

  • CONSTANT
  • LINEAR (default)
  • QUADRATIC

This parameter is specific to SAAMG.

17.5.16 Tolerance

TOL is the convergence tolerance for the iterative solver.

Default:

  • TOL = 1e-8

17.5.17 Maximal Iterations

MAXITERS limits the number of iterations taken by the iterative solver.

Default:

  • MAXITERS = 100

17.5.18 Preconditioner Behavior

PRECMODE controls how aggressively the SAAMG preconditioner is rebuilt or reused.

Value Behavior
STD Rebuild the preconditioner every step
HIERARCHY Reuse the hierarchy only; this is the default
REUSE Reuse the full preconditioner

This parameter should only be changed deliberately, because it affects both performance and solver robustness.

17.5.19 Cut-off Radius

RC is the short-range cutoff radius for the P3M solver in the boosted frame. Particles separated by less than this radius are included in the direct particle-particle correction.

Default:

  • RC = 0

This corresponds to disabling the direct short-range correction.

17.5.20 Interaction splitting parameter

ALPHA controls the balance between the short-range and mesh parts of the P3M Green-function split. Large ALPHA gives greater weight to the mesh part; small ALPHA gives greater weight to the particle-particle part.

It is commonly chosen as

\[ \alpha = C / r_c \tag{17.3}\]

with C of order one and r_c the cutoff radius.

Default:

  • ALPHA = 1e8

This parameter is only used for GREENSF=STANDARD.

17.5.21 Number of Energy Bins

ENBINS selects how many energy bins are used when the bunch is split for multi-frame space-charge treatment. In cyclotron multi-bunch mode this should not be smaller than NNEIGHBB.

MINSTEPFORREBIN defines after how many steps the temporarily split energy-bin representation is merged back into one.

17.6 Adaptive Mesh Refinement (AMR) Solver

The AMR path extends the FIELDSOLVER command with hierarchy and refinement controls. It is enabled by compiling OPAL with ENABLE_AMR=ON and enabling OPTION, AMR=TRUE.

The legacy interface describes three AMR solver types:

  • FMG
  • ML
  • AMR_MG

The hierarchy is defined from the base grid by the number of AMR levels, refinement ratios, blocking factors, and maximum grid sizes.

17.6.1 AMR extensions to FIELDSOLVER

Attribute Meaning
AMR_MAXLEVEL Maximum AMR refinement level
AMR_REFX, AMR_REFY, AMR_REFZ Refinement ratio per coordinate
AMR_MAXGRIDX, AMR_MAXGRIDY, AMR_MAXGRIDZ Maximum base-level box size
AMR_BFX, AMR_BFY, AMR_BFZ Blocking factors
AMR_TAGGING Mesh-refinement rule
AMR_DENSITY Density threshold for CHARGE_DENSITY tagging
AMR_MAX_NUM_PART Threshold for MAX_NUM_PARTICLES tagging
AMR_MIN_NUM_PART Threshold for MIN_NUM_PARTICLES tagging
AMR_SCALING Scaling threshold for POTENTIAL, EFIELD, MOMENTA
AMR_DOMAIN_RATIO Computational-domain aspect ratio

17.6.2 Mesh refinement strategies

The manual documents these tagging modes:

AMR_TAGGING Refinement criterion
POTENTIAL refine where the potential exceeds a scaled level maximum
EFIELD refine where any field component exceeds a scaled level maximum
MOMENTA refine cells containing particles above a scaled momentum threshold
CHARGE_DENSITY refine cells whose density exceeds AMR_DENSITY
MIN_NUM_PARTICLES refine cells based on AMR_MIN_NUM_PART
MAX_NUM_PARTICLES refine cells based on AMR_MAX_NUM_PART

AMR_TAGGING is a string attribute and therefore must be quoted in the input deck.

17.6.3 Hardware-Architecture Independent AMR Poisson Solver

The AMR_MG path is the Trilinos-based AMR Poisson solver. It requires ENABLE_AMR_MG_SOLVER=ON and a working Trilinos installation. The legacy manual lists these core packages:

  • Tpetra
  • Ifpack2
  • Amesos2
  • Belos
  • MueLu

Some base-level linear solvers require additional third-party packages such as SUPERLU, UMFPACK, PARDISO_MKL, MUMPS, or LAPACK.

The AMR multigrid extensions are:

Attribute Meaning
AMR_MG_SMOOTHER GS, JACOBI, or SGS; default GS
AMR_MG_NSWEEPS Number of smoothing sweeps; default 8
AMR_MG_PREC Bottom-level preconditioner; default NONE
AMR_MG_INTERP Prolongation operator; default PC
AMR_MG_NORM Convergence norm; default LINF_NORM
AMR_MG_VERBOSE Write solver diagnostics; default FALSE
AMR_MG_REBALANCE Rebalance solver/preconditioner communicators; default FALSE
AMR_MG_REUSE MueLu reuse mode; default RAP
AMR_MG_TOL Solver tolerance; default 1e-10
ITSOLVER Base-level linear solver; default CG

For ITSOLVER on the AMR base level, the legacy interface documents:

  • BICGSTAB
  • MINRES
  • PCPG
  • CG
  • GMRES
  • STOCHASTIC_CG
  • RECYCLING_CG
  • RECYCLING_GMRES
  • KLU2
  • SUPERLU
  • UMFPACK
  • PARDISO_MKL
  • MUMPS
  • LAPACK
  • SA

17.6.4 Use of AMR

The legacy manual states that AMR is available only in OPAL-cycl multi-bunch mode. Once more than one bunch is present, the AMR hierarchy is built. In this setup the AMR backend manages the MPI parallelism itself, so the other Poisson-solver backends are not combined with the AMR run mode.

17.6.5 AMR Example

OPTION, AMR              = TRUE;
OPTION, AMR_REGRID_FREQ  = 10;
OPTION, AMR_YT_DUMP_FREQ = 100000;

REAL Edes      = .072;
REAL turns     = 8;
REAL nstep     = 360;
REAL frequency = 50.650;

ring: CYCLOTRON, ...;
rf0: RFCAVITY, ...;
rf1: RFCAVITY, ...;
rf2: RFCAVITY, ...;
rf3: RFCAVITY, ...;
rf4: RFCAVITY, ...;
l1: LINE = (ring, rf0, rf1, rf2, rf3, rf4);

Dist1: DISTRIBUTION, ...;

Fs1: FIELDSOLVER, FSTYPE=AMR_MG,
                  MX=128, MY=128, MT=128,
                  PARFFTX=TRUE, PARFFTY=TRUE, PARFFTT=TRUE,
                  BCFFTX=OPEN, BCFFTY=OPEN, BCFFTZ=OPEN,
                  BBOXINCR=20, AMR_MAXLEVEL=2,
                  AMR_MAXGRIDX=32, AMR_MAXGRIDY=32, AMR_MAXGRIDZ=32,
                  AMR_BFX=16, AMR_BFY=16, AMR_BFZ=16,
                  AMR_REFX=2, AMR_REFY=2, AMR_REFZ=2,
                  AMR_DOMAIN_RATIO={1.0, 0.75, 0.75},
                  AMR_TAGGING="CHARGE_DENSITY", AMR_DENSITY=1.0e-9,
                  AMR_MG_VERBOSE=TRUE, AMR_MG_REBALANCE=TRUE, AMR_MG_REUSE=FULL,
                  ITSOLVER=SA, AMR_MG_NORM=LINF_NORM, AMR_MG_NSWEEPS=12;

beam1: BEAM, PARTICLE=PROTON, PC=P0, NPART=1e5, BCURRENT=2.0E-3, CHARGE=1.0, BFREQ=frequency;

SELECT, LINE=l1;

TRACK, LINE=l1, BEAM=beam1, MAXSTEPS=nstep*turns, STEPSPERTURN=nstep;
RUN, METHOD="CYCLOTRON-T", BEAM=beam1, FIELDSOLVER=Fs1, DISTRIBUTION=Dist1,
    MBMODE=FORCE, TURNS=11, MB_BINNING=GAMMA_BINNING, MB_ETA=0.25;

ENDTRACK;
STOP;