INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN FLUIDS Int. J. Numer. Meth. Fluids (2009) Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/fld.2162 Coupled solution of the species conservation equations using unstructured finite-volume method Ankan Kumar and Sandip Mazumder∗, † Department of Mechanical Engineering, The Ohio State University, Columbus, OH 43210, U.S.A. SUMMARY A coupled solver was developed to solve the species conservation equations on an unstructured mesh with implicit spatial as well as species-to-species coupling. First, the computational domain was decomposed into sub-domains comprised of geometrically contiguous cells—a process similar to additive Schwarz decomposition. This was done using the binary spatial partitioning algorithm. Following this step, for each sub-domain, the discretized equations were developed using the finite-volume method, and solved using an iterative solver based on Krylov sub-space iterations, that is, the pre-conditioned generalized minimum residual solver. Overall (outer) iterations were then performed to treat explicitness at sub-domain interfaces and nonlinearities in the governing equations. The solver is demonstrated for both two-dimensional and three-dimensional geometries for laminar methane–air flame calculations with 6 species and 2 reaction steps, and for catalytic methane–air combustion with 19 species and 24 reaction steps. It was found that the best performance is manifested for sub-domain size of 2000 cells or more, the exact number depending on the problem at hand. The overall gain in computational efficiency was found to be a factor of 2–5 over the block (coupled) Gauss–Seidel procedure. All calculations were performed on a single processor machine. The largest calculations were performed for about 355 000 cells (4.6 million unknowns) and required 900 MB of peak runtime memory and 19 h of CPU on a single processor. Copyright q 2009 John Wiley & Sons, Ltd. Received 4 February 2009; Revised 17 July 2009; Accepted 18 July 2009 KEY WORDS: CFD; reacting flow; species conservation; coupled solution; unstructured; finite volume; GMRES 1. INTRODUCTION Computational fluid dynamics (CFD) calculations of reacting flows or flows in which multiple species mix finds widespread use in a large number of practical applications. Examples include ∗ Correspondence to: Sandip Mazumder, Department of Mechanical Engineering, The Ohio State University, Suite E410, Scott Laboratory, 201 West 19th Avenue, Columbus, OH 43210, U.S.A. † E-mail: [email protected] Contract/grant sponsor: ESI Group Copyright q 2009 John Wiley & Sons, Ltd. A. KUMAR AND S. MAZUMDER combustors, catalytic converters, mixing devices in the chemical industry and in biological applications, among many others. In such calculations, mass conservation equations for individual species must be solved. If the species interact with one another strongly, as in cases where chemical reactions or multi-component diffusion is present, it becomes numerically difficult to solve the species conservation equations in a sequential (or segregated) manner. These difficulties have been well documented [1] for reacting flow calculations. Recently, it has been demonstrated that segregated solution is also prohibitive when the full multi-component diffusion equations are solved [2]. Sample calculations have shown that a simple problem involving diffusive mixing of three species in a square box cannot be brought to convergence using a segregated solution approach even with strong under-relaxation of the species [2]. In this case too, as in cases with chemical reactions, the species are tightly coupled. In summary, any problem in which the species are tightly coupled through the physics/chemistry mandates coupled solution of the species conservation equations. In general, coupled solution of a set of partial differential equations (PDE) is the cornerstone for the simulation of wide range of problems. Examples include solution of Maxwell’s equation [3] for electromagnetic phenomena, the radiative transfer equation [4, 5] for heat transfer applications, and the Navier–Stokes equation [6, 7] for fluid-thermal-chemical phenomena, just to name a few. Thus, the volume of literature in this area is too large to discuss in a single article. Here, we will restrict ourselves to the discussion of pertinent methods and algorithms that have been prevalent for the simulation of fluid flow and associated phenomena, especially since the nature and technique for coupling the governing equations is, to a large degree, physics dependent. Traditionally, in CFD, coupled solution of PDE has its early roots in the aerospace area in which density-based time-marching methods are used. For compressible fluid flow, this results in a set of five (in three dimensions) coupled equations that are time-marched to steady state using either explicit or implicit methods. Additional equations, such as the species conservation equations are generally added as separate blocks, and can also be solved in a similar manner. Explicit methods require much less memory, but must obey the stability criteria, thereby necessitating very small time steps. Thus, they are not often desirable from an efficiency perspective, especially in scenarios where the steady-state solution is sought. Nonetheless, explicit methods have been widely used, and still continue to be used routinely in direct numerical simulation of turbulent flows, large eddy simulation of reacting turbulent flows, and other applications such as the simulation of fire and detonation waves [8]. In such applications, the unsteady nature of the flow field mandates highly accurate unsteady solutions. In addition, from a memory perspective, only explicit solution is feasible for such applications. On the other hand, in many engineering applications, such as catalytic converters, fuel cells, micro- (laminar) combustion, steady-state solution is of primary interest. For such applications, explicit time-marching methods are wasteful and undesirable. In addition, since any time-marching method introduces additional temporal discretization errors in the solution, direct solution of the steady-state equations is warranted. For low Mach number flows, as is the case for the aforementioned applications, the use of a time-marching method to attain the steady-state solution implies the use of artificial compressibility—a method that comes with additional problems [9]. The steady-state conservation equations in low Mach number flows are strongly elliptic in space, and thus, best convergence is attained through strong implicit coupling between spatial nodes or cells. Implicit spatial as well as variable-to-variable (species-to-species coupling is the subject of interest here) coupling has been demonstrated by many researchers for structured meshes. On such a mesh, because of the conformity, the discrete equations can be written in banded blockmatrix form, and then solved using linewise sweeps with block-tridiagonal matrix inversions. Such Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS methods are efficient and require very little memory because the resulting matrices are banded. For example, Vanka [6] used this method for implicit coupled solution of the Navier–Stokes equation in primitive variables. Duan et al. [10] has demonstrated this method for solution of the coupled electric potential and species conservations equations in the context of fuel cell simulations, while Mazumder [2] has demonstrated it for the solution of the multi-component species diffusion equations. In an unstructured mesh, the discrete equations, when written in block-implicit form, result in a matrix that is sparse but not banded. Thus, line-by-line sweeps with block-tridiagonal inversion, as used for structured meshes, cannot be employed directly in this case. Nevertheless, the idea of using line-like sub-domains to perform line-by-line sweeps on unstructured meshes has been used successfully by Cete et al. [11] and by Li et al. [12]—the so-called ‘unstructured ADI’ method. It requires re-ordering of the cells and the compilation of additional connectivity information, and is prohibitively tedious for three-dimensional (3D) computations. Point-by-point block (or vector) Gauss–Seidel is a straightforward extension of the scalar Gauss–Seidel method for the solution of a coupled set of PDE on an unstructured mesh. However, as with scalar Gauss– Seidel, the spectral radius of convergence rapidly approaches unity as the mesh is refined, and the convergence deteriorates. Thus, standalone block Gauss–Seidel (BGS) is not attractive for large-scale computations of reacting flows. One approach to circumvent this problem is to use a multigrid method with BGS as the smoother. In recent years, multigrid methods have become quite popular and are being widely used [13–22]. Of these, the algebraic multigrid (AMG) method [18–22] has become the frontrunner on account of its ability to elegantly accommodate complex geometry and arbitrary unstructured mesh topology. In such methods, the linear algebraic equations are agglomerated (or added) to develop equations that philosophically correspond to equations for coarser geometric entities. The agglomeration strategy involves comparison of link coefficients between adjacent cells. Research has now taken AMG methods for a single PDE (i.e. scalar case) to a level where the method has outshone most other methods because of its convergence characteristics and low memory requirements. In the case of coupled PDEs, however, problems remain. While the method has been used successfully for solution of coupled PDEs [19, 20, 22], in classical AMG solvers, questions remain as to the best strategy for agglomerating two cells (or block algebraic equations), since each PDE now has a different set of link coefficients. In this work, we proceed with a strategy that builds upon the ‘unstructured ADI’ idea. In the BGS method, only variable-to-variable coupling is attained and all neighboring spatial cells are treated explicitly. At the other end of the spectrum is the fully coupled solution approach in which the entire block matrix is assembled and solved using a direct solver, namely block Gaussian Elimination (BGE). In the latter case, all variables and spatial nodes are treated implicitly, and no iterations are necessary if the governing equations are linear. Direct solution using BGE is prohibitive on two accounts: (1) it requires extraordinarily large amounts of memory and (2) the computational time scales as O(M 3 ) [23], where M is the total number of unknowns, implying that the computational time would be prohibitively large for reasonably large-sized problems. A compromise between the afore-mentioned two extreme scenarios would be to decompose the computational domain into sub-domains comprised of contiguously located cell groups, and then solve the fully implicit set of equations for each sub-domain sequentially, while treating the other sub-domains explicitly. In essence the ‘unstructured ADI’ method may be thought of as a special case of the method just described, in which the sub-domains are long and snake-like in shape. Since it is already known that block ADI methods result in significantly improved convergence over BGS [2, 11], it was felt that the above idea should be generalized Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER and pursued further, and may be cited as the motivation for the current work. It is worth noting also that this general idea has been also used successfully for modeling of flow in random porous media [24]. In the specific case of the species conservation equations (often referred to in the literature as the advection–diffusion–reaction equation), if multi-component diffusion is to be neglected, as is appropriate for high mass transport Peclet number applications such as combustion, the equations are tightly coupled only through the nonlinear volumetric reaction source term [25]. In view of this, and also the fact that the computation of the reaction source is computationally expensive, if a complex multi-step reaction mechanism is to be used, one approach that has often been adopted is operator splitting [26–29]. In this approach, the reaction and transport operators are split and treated in sequential steps, rather than simultaneously. The rationale is that reaction and transport time scales are vastly different, and can therefore, be segregated. First, the chemical reaction sources are computed explicitly by time integration of the ordinary differential equations arising out of the chemical kinetics, and then substituted into the overall conservation equations, which are then solved in a segregated manner to account for the transport. In the recent years, increasingly complex reaction mechanisms are being used, and efficient means of computing the chemical reaction sources, such as in situ adaptive tabulation [30], has become necessary. The use of such methods mandates splitting of the reaction and transport operators. However, operator splitting introduces additional errors and difficulties [26, 31]. In addition, in CFD calculations for combustion applications, operator splitting is well justified since the reactions at combustion temperatures are indeed much faster than the transport processes in the vast majority of cases. However, for the applications of interest to the authors, such as fuel cells and catalytic converters, the afore-mentioned time-scale argument is not always valid, and therefore, the validity of operator splitting is questionable. Furthermore, even if the operators are split, in the presence of multicomponent diffusion, the species conservation equations are tightly coupled through the diffusion operator [2], and thus, the benefits of splitting reaction and transport operators are not manifested. The objective of the current study is to develop a solver that couples reaction and transport in a general framework. Strongly coupled treatment of both reaction and transport has been implemented by Shadid and co-workers in their code MPSalsa [32], which is a general-purpose implicit finite-element code for reacting flow applications [33, 34]. It uses a Jacobian-free inexact Newton iteration technique in conjunction with a parallel generalized minimum residual solver (GMRES) solver to solve the final discrete algebraic equations. The memory requirements of this solver are not clearly reported. Nonetheless, benchmark 2D flow problems, such as the driven cavity problem and the backward facing step problem with 100×100 mesh were solved using 20 Intel Paragon processors [33], while 3D calculations in a chemical vapor deposition reactor with 3 chemical species and approximately 48 000 nodes were solved using 220 Intel Paragon processors. The reason for using such high degree of parallelism for such simple problems is not clear. These calculations were performed on processors that are more than a decade old. A recent publication by the same group [34] reports 3D calculations of a chemical vapor deposition reactor with 3 chemical species (i.e. a total of 8 unknowns at each node: 3 velocity components, temperature, pressure, and three mass fractions). Calculations with approximately 75 000 total nodes (636 168 unknowns) were performed on a 64-processor machine, each processor being a 3 GHz Intel Pentium IV processor. Such calculations required approximately 4–5 min of CPU time, which translates to 256–320 min of CPU time with linear (best case) parallel scaling. The objective of the present study is to develop a solution strategy that will make such calculations possible on a single processor (i.e. without necessarily Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS relying on parallel architectures for memory) with a few hours of computational time, and is easily extendable to parallel architectures and multi-core processors for truly large-scale computations. 2. GOVERNING EQUATIONS Prior to the discussion of algorithms, it is necessary to discuss the governing equations and accompanying boundary conditions. These equations will lay the foundation for development of the new solver, to be discussed in later sections. Another motivation for detailed discussion of the governing equations is that in low-speed reacting flow applications, which happen to be of interest to the authors, diffusive transport of mass is critical. Reacting flow formulations outlined in various textbooks and published articles pay little or no attention to diffusive (molecular) transport of species since turbulent transport dominates in the vast majority of applications of such flows. Thus, there is a need to critically assess commonly used formulations for the treatment of mass transport in low Peclet number multi-component systems. 2.1. Conservation equations The governing equations are the equations of conservation of mass (both overall and individual species), momentum and energy, and are written as [25, 35, 36]: Overall mass : Momentum: (1) * (U)+∇ ◦(UU) = −∇ p +∇ ◦s+B *t Energy : Species mass: * ()+∇ ◦(U) = 0 *t * (h)+∇ ◦(Uh) = −∇ ◦q+ Ṡh *t * (Yk )+∇ ◦(UYk ) = −∇ ◦Jk + Ṡk *t (2) (3) ∀k = 1, 2, . . ., N (4) where is the mixture density, p is the pressure, s is the shear stress tensor, and B is the body force vector. Equations (1) and (2) are the well-known Navier–Stokes equations, and need no further discussion. In Equation (4), Yk is the mass fraction of the kth species, Jk is the mass diffusion flux of the kth species, and Ṡk is the production rate of the kth species due to homogeneous chemical reactions. The total number of gas-phase species in the system is denoted by N . In Equation (3), Ṡh represents the net source due to viscous dissipation and other work and heat interactions, and q denotes the net heat flux due to molecular conduction, radiation, and inter-species diffusion, and is written as [35]: q = qC +qR +qD = −kc ∇T +qR + N Jk h k (5) k=1 where h k is the enthalpy of the kth species, kc is the thermal conductivity of the mixture, and h N h k Yk ). In the above formulation, the enthalpy of the kth is the enthalpy of the mixture (= k=1 Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER species, h k , includes the enthalpy of the formation and the sensible enthalpy, and is written as T h k (T ) = h 0f,k + c p,k (T ) dT (6) T0 where h 0f,k is the enthalpy of formation of species k at the standard state, and c p,k is the specific heat capacity of species k. The species enthalpy is generally computed using standard thermodynamic databases, such as the JANNAF database. Equation (1) can be derived from Equation (4) if and only if the following two constraints are satisfied: At any point in space : N Yk = 1 (7) k=1 At any arbitrary cutting plane : N Jk ◦ n̂ = 0 (8) k=1 where n̂ is the surface normal to the cutting plane in question. Equation (7) is generally enforced either by solving only N −1 equations from the set in Equation (4), and then using Equation (7) directly to determine the mass fraction of the last species, or by normalizing the calculated mass fractions by their sum, resulting in an indirect correction strategy. In diffusion dominated (i.e. low mass-transport Peclet number) systems, failure to satisfy the constraint given by Equation (8) results in a serious inconsistency. The inconsistency is a result of the fact that the sum of the species conservation equations over all species does not result in the continuity equation (Equation (1)). Rather, it results in a continuity equation with a spurious mass source. In general, the mass diffusion flux, Jk , includes diffusion due to concentration gradients, temperature gradients (Soret diffusion), and pressure gradients (pressure diffusion) [35]. Here, only mass transport due to concentration gradients is considered for the sake of simplicity. The most common approach for modeling diffusive transport of species due to concentration gradients is to use Fick’s law of diffusion. In a binary system, consisting of species A and B, the diffusion flux is accurately described by Fick’s law of diffusion [35]: J A = −DAB ∇Y A , J B = −DBA ∇Y B (9) where DAB is the binary diffusion coefficient of species A into B, and is equal to DBA , which is the binary diffusion coefficient of species B into A. Using the mass fraction summation constraint (Equation (7)), it can be readily shown that J A = −J B , that is, Equation (8) is automatically obeyed. The same law, when used for multi-component systems with more than two species, leads to the violation of mass conservation due to non-satisfaction of the constraint given by Equation (8). In a multi-component system, diffusion is best described by the Stefan–Maxwell equation [35], which implicitly relates molar fluxes of species to mole fraction gradients: N Xi J j X j Ji M ∇ Xi = (10) − j =1 M j Dij Mi Dij j =i where M is the mixture molecular weight, Mk is the molecular weight of the kth species, and X k is the mole fraction of the kth species. The Stefan–Maxwell equation has been formulated in such a manner that Equation (8) is satisfied for an arbitrary multi-component system. Unfortunately, in Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS its primitive form, it can only be used for pure diffusion problems, and is not amenable for use in a CFD framework since it is not an equation of the generalized advection–diffusion form. Upon significant manipulation, Equation (10) can be re-written as [2, 37] Jk = − N Mk Mn Dkn ∇ X n M 2 n=1 (11) where Dkn is the ordinary multi-component diffusion coefficient [2, 37], and is different from the binary diffusion coefficient Dkn . Specifically, while the binary diffusion coefficients are independent of the mole fractions, the ordinary multi-component diffusion coefficients are strong nonlinear functions of the mole fractions. In addition, it is worth noting that while Dkk = 0, Dkn = Dnk . The ordinary multi-component diffusion coefficients can be computed using well-known relationships [35, 38]. Equation (11) essentially states that in a multi-component system, diffusion of a certain species is governed not only by its own concentration gradient, but also the concentration gradient of the other species in the system. In a reacting flow, mass is always conserved while moles are not. Thus, it is advantageous to re-write Equation (11) in terms of mass fractions rather than mole fractions. Using the conversion relation Yk = X k Mk /M, Equation (11) may be written as [2, 37] Jk = − N kn ∇Yn (12) n=1 where kn is a new tensor, written as [2, 37] [] = − 1 [M][D][M][C] M2 (13) In Equation (13), [M] = diag[M1 , M2 , . . ., M N ], [D] is the matrix notation for the ordinary multicomponent diffusion tensor Dkn , and [C] the Jacobian of the transformation between mass and mole fraction, and is written as M M Ckn = kn −Yk (14) Mk Mn where kn is the Kronecker delta. Substitution of Equation (12) into Equation (4) yields N * kn ∇Yn + Ṡk (Yk )+∇ ◦(UYk ) = ∇ ◦ *t n=1 ∀k = 1, 2, . . ., N (15) Equation (15) represents the governing equation for species transport in a multi-component system. It satisfies the constraint posed by Equation (8) automatically. It is important to note at this point that even in the absence of any homogeneous chemical reactions (i.e. Ṡk = 0), the mass fractions of the individual species are tightly coupled through the diffusion operator, that is, the first term on the right-hand side of Equation (15). Thus, segregated solution (i.e. one species at a time) of the equations represented by Equation (15) has inherent instabilities associated with it. Details pertaining to these numerical issues may be obtained from Reference [2] and are not discussed here for the sake of brevity. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER 2.2. Boundary conditions The boundary conditions for the mass and momentum conservation equations are the no-slip conditions at walls, and appropriate mass flux or pressure boundary conditions at inflow and outflow boundaries. These boundary conditions and their numerical implementation are well known [9] and need no further discussion. The focus of this subsection is the boundary conditions for species and energy associated with heterogeneous chemical reactions at fluid–solid interfaces. At a reacting surface, the diffusion flux of species is balanced by the reaction flux since the surface cannot store any mass. At the heart of surface reaction processes are adsorption and desorption of species at the surface, the treatment of which requires inclusion of the so-called surface-adsorbed species [39]. At steady state, the net production rate of the surface-adsorbed species is zero. In the absence of etching or deposition of material from the surface (i.e. zero Stefan flux), the reaction–diffusion balance equation at the surface may be written as [39, 40] Ṙk = Jk ◦ n̂ = Mk Ṙk ∀k ∈ gas-phase species dk /dt for unsteady 0 for steady (16a) ∀k ∈ surface-adsorbed species (16b) where Ṙk is the molar production rate of species k due to heterogeneous chemical reactions, k is the molar concentration of species k at the fluid–solid interface, and n̂ is the outward unit surface normal. Since Ṙk is an extremely nonlinear function of the molar concentrations (or mass fractions) [39, 40], Equation (20) represents a nonlinear set of differential algebraic equations (DAE). The solution of this stiff set of nonlinear DAE is generally obtained using the Newton method, but requires special pre-conditioning to address stiffness and ill-posedness in the case of steady-state solutions. Details pertaining to these numerical issues may be found elsewhere [40]. The solution of Equation (16) provides the near-wall mass fractions and mass fluxes (represented by the left-hand side of Equation (16a)) of all gas-phase species, which appear as sources/sinks for control volumes adjacent to the surface in a finite-volume formulation [40]. The balance of energy at the surface yields the following equation: N −k∇T +qR + Jk h k ◦ n̂ = [−k∇T +qR ]S ◦ n̂ (17) k=1 F where the subscript ‘F’ denotes quantities on the fluid side of the fluid–solid interface, while the subscript ‘S’ denotes quantities on the solid side of the same interface. The solution of Equation (17), which is also a nonlinear equation, yields the temperature at the fluid–solid interface, and subsequently provides the flux of energy at the interface, which can then be used as a source/sink for the cells adjacent to the interface after appropriate linearization. In this enthalpy formulation, the heat of surface reaction actually manifests itself through the Jk h k term. Equations (1)–(3) along with Equation (15), when solved along with the appropriate boundary conditions described in the preceding subsection, will produce flow, temperature, and mass fraction distributions of all species within the computational domain. In the present study, the equations were discretized using the standard conservative unstructured finite-volume procedure [41, 42], which is well documented in the literature and is, in fact, in use in the vast majority of commercial CFD codes. For the sake of completeness, the finite-volume integration procedure and the resulting block-implicit form of the species conservation equations are briefly outlined in Appendix A. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS 3. SOLVER DEVELOPMENT In terms of coupled solution of the species conservation equations, the best-case scenario is one in which all species conservation equations are solved at all grid points using a direct solver. In this scenario, the solution of all species at all grid points is obtainable by a single matrix inversion provided that the governing equations are linear. However, this requires extraordinarily large amounts of memory for even simple problems, as discussed earlier. This is particularly true for unstructured meshes for which the system of equations may be sparse but not necessarily perfectly banded. Another alternative is to sub-divide the domain into sub-domains that are small enough such that for each of these sub-domains a fully implicit solution of the governing equations is affordable from a memory standpoint. This is the approach used here. The rationale is that this approach will enable spatial as well as species-to-species implicit coupling, and the extent of spatial coupling is to be dictated by memory constraints. It is similar in principle to additive Schwarz domain decomposition methods, although parallel nodes are not used in this case. Thus, the performance of this approach is expected to be somewhere between a BGS solver (worst case), and a full block-implicit direct solver (best case). The development of the coupled solver entails three major steps, as follows: 1. The computational domain is decomposed into smaller groups of cells that are geometrically contiguous—a process termed internal domain decomposition (IDD). This is a one-time preprocessing step. The term ‘internal’ is used here to differentiate with decomposition used for parallel processing. 2. For each sub-domain, an iterative solver based on Krylov subspace iterations (i.e. the restarted and pre-conditioned GMRES [43])is employed to obtain the solution at all nodes within the sub-domain and for all species simultaneously. Other sub-domains adjacent to the sub-domain in question are treated explicitly, and ghost (or virtual) boundary conditions are applied at interfaces between sub-domains to transfer information between sub-domains. 3. An overall (or outer) iteration is performed within which the preceding step is repeated until convergence. Iterations are necessary not only to resolve the explicit coupling between sub-domains, but also to resolve nonlinearities in the governing equations. The overall algorithm is depicted in Figure 1. Details of the important first two steps are provided next. 3.1. Internal domain decomposition As depicted in Figure 1, and mentioned earlier, the first step in the development of the solver is the decomposition of the computational domain into smaller groups of cells that are contiguous. Domain decomposition is routinely performed for parallel processing, and several algorithms are available. However, the focus of such algorithms is often the balancing of domains [44], that is, creating domains with almost equal number of cells, as necessitated by parallel algorithms. For the present purpose, balancing of domains is not a priority since the sub-domains will be processed sequentially (see Step 2 above) rather than in parallel. Thus, existing domain decomposition algorithms, in their full glory, are not necessary here. We use an algorithm for domain decomposition that is foolproof, yet efficient. The algorithm uses binary spatial partitioning (BSP) trees [45, 46] to recursively bisect the domain into a set of hierarchically linked sub-domains. The idea is Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Figure 1. Block-diagram representation of the coupled implicit solver for the species conservation equations developed in the present study. If a flow solver is included, it will be a part of the outer loop iterations. depicted schematically in Figure 2, and the algorithm to create each sub-domain (cell group) is as follows: (A) The coordinates of all vertices in the computational domain are scanned to determine the bounds of the computational domain. The binary table is initiated, and six real numbers, namely, X min , X max , Ymin , Ymax, Z min , and Z max are stored for this first point. In addition, an integer list, consisting of all the cell indices, is also stored. (B) The box is bisected along its central X , Y , or Z plane. The decision as to what direction the bisection should be performed in depends on the current level of the tree. The definition of ‘level’ is as follows: the first entry corresponds to level 1, entries 2–3 in level 2, entries 4–7 in level 3, 8–15 in level 4, and so on. One approach to create an almost isotropic tree is to cycle the bisection directions with each level. If the box is bisected along the central X plane, for example, one of the children’s X max value will be replaced by (X min + X max )/2 of its mother, while the X min value of the other child will be replaced by (X min + X max )/2 of its mother. All other bounds are set to that of their mother. (C) The two children are incrementally numbered, such that each entry (or box) in the table has a unique number (Figure 2). For the mother, the entry numbers corresponding to her two children are stored. (D) The cell list of the mother is distributed between the children after checking to see which cells lie within the bounding box of the children. This is done by scanning the coordinates of the individual vertices. Cells that are shared by both children are assigned to either of the two children. Steps B–D are repeated recursively until the prescribed termination criteria are reached. The termination criterion is the number of cells within each sub-domain. This, in turn, is dictated by the number of species in the problem, and the available memory, as will be discussed in detail later. The actual result of applying this domain decomposition algorithm to a simple unstructured Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 2. Internal domain decomposition (IDD) using the Binary Spatial Partitioning (BSP) algorithm: in this particular example, only two levels of bisection are shown. Figure 3. An example of applying the BSP algorithm to domain decomposition of a two-dimensional geometry with triangular mesh. two-dimensional domain is depicted in Figure 3. An important point that is illustrated by Figure 3 is that although the domain is bisected recursively, the resulting sub-domains do not always end up with equal number of cells. Conversely, if it is specified within the bisection algorithm that any sub-domain should have a maximum number of N D cells, the final result will be sub-domains that will, on an average, have fewer cells than N D . Many of the operations, described above, and much more can be performed using software libraries such as METIS and PETSc. These libraries provide tremendous flexibility in domain partitioning and other functions. However, it was found that use of these libraries resulted in additional overheads. Thereafter, except for the GMRES solver which used the SPARSKIT Fortran90 routines directly, all other functionalities were developed in-house. 3.2. Linear algebraic equation solvers One critical issue in the development of this solver is the linear algebraic equation solver to be used for each sub-domain. The choice of the linear algebraic equation solver, that is, its memory Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER requirements and efficiency, is likely to dictate how many cells can be included in each sub-domain, and consequently the overall efficiency. During the preliminary stages of this study, two linear algebraic equation solvers with completely different methods of solution were chosen: (1) a direct solver: block Gaussian Elimination, and (2) an iterative solver: pre-conditioned restarted GMRES [43]. A benchmark multi-component pure diffusion (without flow) problem with 3 species in a square box (as outlined in Reference [2]) was solved using both solvers [47]. It was found that the GMRES-based solver was most efficient for sub-domain sizes of 200 cells or more, while the Gaussian Elimination-based solver was most efficient for sub-domain sizes of 50 cells or less. Overall, the GMRES-based solver was found to be more efficient than the Gaussian Eliminationbased solver. In part, this was due to the fact that the total number of overall iterations was significantly smaller with the GMRES-based solver in which more cells were treated implicitly within each sub-domain. Based on these preliminary tests, the GMRES-based solver was chosen for full-blown reacting flow calculations, and the Gaussian Elimination-based solver was abandoned. The factor that motivated the choice of GMRES over other iterative solvers is that it has extremely good convergence properties when preconditioned appropriately even for relatively large matrix sizes. In this solver, memory requirement can be limited by specifying the largest Krylov subspace size and the number of internal iterations can be limited by specifying the maximum number of GMRES iterations. In essence, the solver chosen is very flexible and appropriate for use in the current solution strategy. During the course of this study, several pre-conditioners, namely Jacobi, BGS, and incomplete LU (ILU), were tested. The ILU preconditioner’s performance was found to be superior to the others, and has been chosen for the final test calculations. The algebraic equations of each sub-domain were preconditioned using an ILU pre-conditioner without any off-diagonal fill in. Equations in each sub-domain were solved to a specified degree of convergence, that is, until the residuals decreased by two orders of magnitude. It was found that reducing the accuracy of sub-domain solutions any lesser lead to non-convergence in all test cases. A restart value of 30 has been used for the GMRES solver and maximum number of internal iterations was set equal to 200. Choosing too small a value of restart lead to stagnation of the solution and non-convergence, while choosing values that are too large was limited by the memory requirement. These particular values were dictated by the size of sub-domains under consideration and the desired level of accuracy. For the GMRES solver and pre-conditioner, the SPARSKIT library was used directly. Henceforth, the coupled implicit solver for the species conservation equations will be referred to as the IDD+GMRES solver. 4. SAMPLE CALCULATIONS AND RESULTS As our primary interest is in low-speed reacting flow applications, a pressure-based solver was chosen over a density-based solver for solution of the Navier–Stokes equations. The SIMPLEC algorithm [48] was used to resolve velocity–pressure coupling. A co-located mesh was used, and the pressure-weighted interpolation method [49, 50] was used to damp checkerboard pressure oscillations. In addition, since the present work focuses specifically on coupled implicit solution of the species conservation equations, the species equation solver was kept separate from solution of the Navier–Stokes and energy equations. The momentum equations, the pressure correction equation, and the enthalpy equation (in correction form) were solved in a segregated manner using full-field conjugate gradient (for momentum and energy equations) and algebraic multi-grid (for pressure correction equation) solvers. Overall outer iterations were performed to couple all the Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS conservation equations, and to address nonlinearities in the governing equations, which were all solved in appropriate linearized form. In order to judge the efficacy of the new species solver, a standalone BGS solver was also developed for the species equations. Thus, given the same flow and energy equation solver, it is possible to determine the efficacy of the IDD+GMRES solver for the species conservation equations by comparing it against a BGS solver. 4.1. Validation Prior to performance studies, a validation study was performed to assess the accuracy of the code. Three-way conversion of an automotive exhaust stream in a catalytic converter with circular channels of diameter 1 mm and length 29 mm was considered. The exhaust gas mixture enters the channel at a predefined velocity of 1.35 m/s and at a temperature 298 K. The wall of the catalytic converter channel is maintained at 700 K. Simulations have been performed for three air fuel ratios, namely 0.5 (lean), 0.9 (near-stoichiometric), and 1.8 (rich). Experimental data for this setup are available from the study of Chatterjee et al. [51]. The reaction mechanism used for this particular study is also adopted from the same paper, and is comprised of 61 reaction steps between 31 species (8 gas-phase species, 23 surface-adsorbed species). The comparison of simulated conversion percentages for three species, CO, C3 H6 and NO and experimental data is plotted in Figure 4. Open symbols represent simulated results, while solid symbols represent experimental data. It is seen that the simulated results show excellent match for the fuel-rich mixture for all species. For the near-stoichiometric mixture, the comparison is excellent for CO and NO, while the simulation results are within 20% of the experimental observed value for C3 H6 . For the fuel-lean mixture, while CO and NO results agree reasonably well, the results for C3 H6 Figure 4. Simulated (open symbols) and experimental (solid symbols) [51] results for a three-way catalytic conversion process at various fuel-air ratios. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER do not agree. The reason for this discrepancy is that the reaction mechanism is not designed to accurately address the chemistry in lean regimes, as described in Chatterjee et al. [51]. Following the validation study, the focus was shifted to the performance of the solver, which is the primary focus of this article. Five different test problems were considered for the current study. They are as follows: • Homogeneous combustion (laminar flame) of methane using 2-step chemistry (6 gas-phase species) in a 2D geometry with 19 953 triangular cells • Heterogeneous (catalytic) combustion of methane using 24-step chemistry (8 gas-phase species) in a 2D geometry with 19 953 triangular cells • Homogeneous combustion (laminar flame) of ethane combustion using 1-step chemistry (5 gas-phase species) in a 3D geometry with 40 000 hexahedral cells • Heterogeneous (catalytic) combustion of methane using 24-step chemistry (8 gas-phase species) in a 3D geometry with 59 819 tetrahedral cells • Heterogeneous (catalytic) combustion of methane using 24-step chemistry (8 gas-phase species) in a 3D catalytic converter geometry with 355 000 mixed (hexahedra and prisms) cells All 5 problems were solved using a single processor with a 2 GHz Intel Xeon 5130 processor with 4 GB of RAM. The first four problems are similar in size to coarse mesh problems considered in Reference [34], while the final problem is comparable to the finest mesh case considered in the same study. Extensive timing studies (e.g. comparison with BGS, effect of number of cells in each sub-domain, etc.) were conducted only for the first four test problems because such timing studies require solution of the problem multiple times, and is prohibitive for the fifth test problem. It is worth noting that although the code has been written for general unstructured mesh topology including hybrid meshes, for three out of the five test problems, triangular and tetrahedral meshes were chosen over quadrilateral or hexahedral meshes since triangular or tetrahedral meshes have higher levels of skewness. Since increased skewness adversely affects convergence [9], the chosen mesh topology represents the most challenging scenarios for the new solver. 4.2. Test problem 1: 2D laminar methane–air flame The first test problem considered for the present study is that of steady-state homogeneous combustion of a methane–air mixture under laminar flow conditions, that is, a laminar flame. A simple two-dimensional (2D) diffusion flame configuration, as shown in Figure 5(a), is considered. Based on the inlet conditions, the overall equivalence ratio is unity, and the Reynolds number based on the channel width is approximately 150. A 2-step reaction mechanism [52] involving 6 gas-phase species (CH4 , CO2 , H2 O, N2 , O2 , CO) was used for gas-phase chemistry calculations. The computational mesh comprised of 19 953 triangular cells (199 530 unknowns), and part of the mesh is shown in Figure 5(b). Convergence was deemed to have been attained when the L2-norm of all the conservation equations decreased by six orders of magnitude. The results of the computations are summarized in Table I. The numbers in parenthesis alongside the IDD+GMRES solver indicate the maximum prescribed number of cells in any sub-domain, that is, the value of N D . As discussed earlier, the actual number of cells after IDD may be lower. SUF is the speed-up factor relative to the BGS solver. It is clear from the data shown in Table I that optimum the performance is obtained when ∼ 1000 cells are considered within each sub-domain. For smaller sub-domain sizes, the overhead in setting up the GMRES solver is too high since that Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 5. (a) Geometry and boundary conditions for Test problem 1 (2D laminar methane–air flame) and (b) computational mesh: only one-third of the channel is shown for clarity. operation has to be performed for many more sub-domains. For sub-domain sizes that are too large, although fewer outer iterations are required, the time taken by the GMRES solver per iteration is larger, resulting in more overall CPU time consumption. One of the reasons why the GMRES solver gets progressively slower with larger number of cells in each sub-domain is that the number of specified Krylov dimensions is not adequate for the best performance. However, in order to keep the memory within bounds, such a restriction must be placed. It is worth noting that in this test problem, since only 6 gas-phase species are present, the species equation solver consumes only about 50% of the overall CPU, while the remainder is consumed by the flow and energy equation solver. Thus, an overall SUF of 3.56 actually translates to larger gains (>5) if the speed-up of the species solver alone were to be considered. A typical convergence plot obtained using the IDD+GMRES solver is shown in Figure 6, and results of the computations are shown in Figure 7. It is clear from Figure 6 that the species residuals (normalized shown here) actually decrease the fastest, and the overall convergence is held up by the flow solver. Overall, this calculation required only 17 min of CPU time. For the optimum case with 800 cells in each sub-domain, the overall solver required 181 MB of memory. Detailed results for the effect of domain size on memory will be presented later for a 3D case. 4.3. Test problem 2: 2D methane–air catalytic combustion The second test problem considered for the present study is that of steady-state heterogeneous (catalytic) combustion of a methane–air mixture on platinum. The computations were performed on the same geometry and mesh as test problem 1, but with a different set of boundary conditions, as shown in Figure 8. The Reynolds number in this case is 100, and the inlet equivalence ratio is unity. The reaction mechanism used for this test calculation is a 24-step surface reaction mechanism [53] involving reactions between 11 surface-adsorbed species and 8 gas-phase species (CH4 , CO2 , H2 O, N2 , O2 , CO, OH, H2 ). The implementation of a reacting surface boundary condition requires solution of a stiff set of nonlinear algebraic equations (19 in this case) at each boundary face. This requires special numerical techniques that are discussed elsewhere [40]. It is to be noted that the species conservation equations are solved only for the 8 gas-phase species, and not for the surface Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Table I. Number of outer iterations and CPU time taken for the test problem 1 (2D laminar methane–air flame). Solver used IDD+GMRES (100) IDD+GMRES (200) IDD+GMRES (400) IDD+GMRES (800) IDD+GMRES (1300) IDD+GMRES (2500) IDD+GMRES (5000) Block Gauss–Seidel (BGS) Outer iterations CPU time (s) SUF 1517 1069 1068 873 872 834 834 5211 1486 1198 1120 1020 1041 1039 1067 3631 2.44 3.03 3.24 3.56 3.49 3.49 3.40 1.00 Figure 6. Convergence of the IDD+GMRES solver for test problem 1 (2D laminar methane–air flame) with 800 cells in each sub-domain. species. Thus, this particular problem has a total of 239 436 unknowns. Once again, computations were performed with the IDD+GMRES solver with various sub-domain sizes and compared against the performance of the BGS solver. The results are summarized in Table II. In this case, optimum performance is obtained when ∼ 3000 cells are considered within each sub-domain, for which overall SUF of 2.51 is manifested. The convergence plot obtained using the IDD+GMRES solver is shown in Figure 9, and results of the computations are shown in Figure 10. Overall, this calculation required only 20 minutes of CPU time for six orders of magnitude convergence. For the optimum case with 3200 cells in each sub-domain, the overall solver required 224 MB of memory. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 7. Temperature and selected species mass fraction distributions for test problem 1 (2D laminar methane–air flame). Figure 8. Geometry and boundary conditions for test problem 2 (2D methane–air catalytic combustion). 4.4. Test problem 3: 3D laminar ethane–air flame The third test problem considered is that of steady-state homogeneous combustion of an ethane-air mixture under laminar flow conditions. A three-dimensional (3D) diffusion flame configuration, as shown in Figure 11, is considered. Based on the inlet conditions, the overall equivalence ratio is Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Table II. Number of outer iterations and CPU time taken for the test problem 2 (2D methane–air catalytic combustion). Solver used IDD+GMRES (100) IDD+GMRES (200) IDD+GMRES (500) IDD+GMRES (800) IDD+GMRES (1600) IDD+GMRES (3200) IDD+GMRES (6400) Block Gauss–Seidel (BGS) Outer iterations CPU time (s) SUF 1417 1108 1023 862 851 793 787 3404 1548 1477 1512 1271 1249 1201 1223 3019 1.95 2.04 1.99 2.37 2.42 2.51 2.47 1.00 Figure 9. Convergence of the IDD+GMRES solver for test problem 2 (2D methane–air catalytic combustion) with 3200 cells in each sub-domain. unity, and the Reynolds number is approximately 60, based on the channel’s hydraulic diameter. A 1-step reaction mechanism [54] involving 5 gas-phase species (C2 H6 , CO2 , H2 O, N2 , O2 ) was used for gas-phase chemistry calculations. The computational mesh comprised of 40 000 hexahedral cells, resulting in a total of 400 000 unknowns. The results of the computations are summarized in Table III. In this case, the optimum performance is obtained when ∼ 2500 cells are considered within each sub-domain, for which an overall SUF of 3.29 is attained. The convergence plot obtained using the IDD+GMRES solver with 2500 cells in each sub-domain is shown in Figure 12, and results of the computations are shown in Figure 13. Overall, this calculation required only 24 min of CPU time for six orders of magnitude Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 10. Temperature and selected species mass fraction distributions for test problem 2 (2D methane–air catalytic combustion). convergence. For the optimum case with 2500 cells in each sub-domain, the overall solver required 207 MB of memory. 4.5. Test problem 4: 3D methane–air catalytic combustion The fourth test problem considered is that of steady-state heterogeneous (catalytic) combustion of a premixed methane–air mixture on platinum. The geometry and boundary conditions are shown in Figure 14(a), and the mesh is shown in Figure 14(b). The Reynolds number in this case is 130, and the inlet equivalence ratio is unity. The reaction mechanism used for this test calculation is the same as that used for test problem 2. The computational mesh is comprised of 59 819 tetrahedral cells, leading to 777 647 unknowns. Computations were performed with the IDD+GMRES solver with various sub-domain sizes and compared against the performance of the BGS solver. The results are summarized in Table IV. In this case, the optimum performance is obtained when ∼ 3000 cells are considered within each sub-domain, for which overall SUF of 2.13 is manifested. The convergence plot obtained using the IDD+GMRES solver is shown in Figure 15, and results of the computations are shown in Figure 16. Overall, this calculation required only 55 min of Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Figure 11. Geometry and boundary conditions for test problem 3 (3D laminar ethane-air flame). Table III. Number of outer iterations and CPU time taken for the test problem 3 (3D laminar ethane–air flame). Solver used IDD+GMRES (200) IDD+GMRES (350) IDD+GMRES (625) IDD+GMRES (1250) IDD+GMRES (2500) IDD+GMRES (5000) Block Gauss–Seidel (BGS) Outer iterations CPU time (s) SUF 1006 750 731 612 567 567 2891 2176 1688 1682 1447 1415 1461 4662 2.14 2.76 2.77 3.22 3.29 3.19 1.00 CPU time for six orders of magnitude convergence. For the optimum case with 3000 cells in each sub-domain, the overall solver required 305 MB of memory. Table IV also shows the effect of sub-domain size on memory. Comparison between the BGS and the IDD solver with the most memory requirement shows that an additional 118 MB of memory is required to perform coupled implicit solution of the species conservation equations. The additional amount necessary depends strictly on the number of species and the number of cells in each sub-domain, and not on the total number of unknowns in the problem. As the problem size is increased, this additional amount of memory necessary for the IDD+GMRES solver will remain the same, and will constitute only a small fraction of the overall memory required by the overall flow solver. It is informative at this point to compare the performance of our solver with the performance reported in Reference [34] for a similar problem. In Reference [34], calculations with 636,168 unknowns were performed on a 64-processor machine, each processor being a 3 GHz Intel Pentium IV processor. Such calculations required approximately 4–5 min of CPU time, which translates to 256–320 min of CPU time on a single processor with linear parallel scaling. In comparison, the current solver solves a problem with 777 647 unknowns in 55 min of CPU on a comparable single processor. It is also worth noting that the test problem considered here involved a 24-step reaction mechanism, and about 32% of the CPU time was spent in performing surface reaction calculations alone. In contrast, the afore-mentioned case in Reference [34] used single-step Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 12. Convergence of the IDD+GMRES solver for test problem 3 (3D laminar ethane-air flame) with 2500 cells in each sub-domain. chemistry. Although these comparisons are not direct, they do bring to light the fact that the algorithm used here is at least comparable to solvers reported in the recent literature on chemically reacting flows. In addition, these results highlight the fact that the current solver does not require an exorbitant amount of memory. 4.6. Test problem 5: large-scale 3D catalytic converter The fifth and final test problem considered is that of steady-state heterogeneous (catalytic) combustion of a premixed methane–air mixture on platinum in a catalytic converter monolith with multiple channels. The geometry and boundary conditions are shown in Figure 17. The Reynolds number in this case is 130, and the inlet equivalence ratio is unity. In this particular test problem, conjugate heat transfer is also included. The reaction mechanism used for this test calculation is the same as that used for test problems 2 and 4. The computational mesh is comprised of 354 300 cells, leading to approximately 4.6 million unknowns. The mesh, as evident in Figure 17, is a hybrid mesh comprised of hexahedral and prismatic elements. Computations were performed with the IDD+GMRES solver with prescribed maximum sub-domain size of 4000 cells. The convergence plot obtained using the IDD+GMRES solver is shown in Figure 18, and results of the computations are shown in Figure 19. Overall, this calculation required 19 h of CPU time for five orders of magnitude convergence. The overall solver required 900 MB of memory. Once again, to put matters into perspective, a comparison with performances reported in Reference [34] is warranted. For example, in Reference [34], a chemical vapor deposition reactor was modeled with just 3 chemical species (i.e. a total of 8 unknowns at each node: 3 velocity components, temperature, pressure, and 3 mass fractions) on a fine mesh, resulting in a total of 4.8 million unknowns. Computations were performed on a 64-node parallel cluster (each node was Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Figure 13. Temperature and selected species mass fraction distributions for test problem 3 (3D laminar ethane–air flame). The cross-sectional dimensions have been scaled by a factor of 2 for better visualization. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 14. (a) Geometry and boundary conditions for test problem 4 (3D methane–air catalytic combustion) and (b) computational mesh: only the surface mesh is shown, and the cross-sectional dimensions have been scaled by a factor of 2 for clarity. a 3 GHz Intel Pentium IV processor), and required about 2200 s of CPU time. If linear parallel scaling is considered, this implies that such computations would require about 140 000 s (= 39 h) on a single processor. In contrast, the present calculation with 4.6 million unknowns required 19 h of CPU on a comparable single processor. With single-step chemistry described by 5 species, the same computation required only 6 h of CPU time with the IDD+GMRES solver. While the two problems, just described, are not the same, they are comparable in terms of the number of unknowns. It appears that the solver used in the present study is computationally more efficient than existing state-of-the-art reacting flow solvers reported in the literature. It is often argued in the literature [32–34] that Newton-type iterations are desirable because they produce quadratic Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Table IV. Number of outer iterations and CPU time taken for the test problem 4 (3D methane–air catalytic combustion). Solver used IDD+GMRES (200) IDD+GMRES (500) IDD+GMRES (800) IDD+GMRES (1000) IDD+GMRES (1600) IDD+GMRES (2000) IDD+GMRES (3000) IDD+GMRES (4000) Block Gauss–Seidel (BGS) Outer iterations CPU time (s) SUF Memory (MB) 574 488 444 424 414 399 377 375 1310 4000 3642 3509 3417 3381 3287 3259 3362 6953 1.74 1.91 1.98 2.03 2.06 2.12 2.13 2.07 1.00 206 216 225 231 248 265 298 299 181 Figure 15. Convergence of the IDD+GMRES solver for test problem 4 (3D methane–air catalytic combustion) with 3000 cells in each sub-domain. convergence and low number of outer iterations. Clearly, the current solver requires a lot more outer iterations than Newton iteration-based methods, as is evident from Table IV. However, one outer iteration, in this case, is significantly cheaper than one Newton iteration, as is evidenced by the fact that the overall CPU times are comparable or better. It is also worth noting that part of the reason for the large number of outer iterations in the current solver is that the flow and energy equations were solved in a segregated manner, separate from the species equation solver. If all equations are coupled, as was done in Reference [32], the performance of the current solver is likely to improve further. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 16. Temperature and selected species mass fraction distributions for test problem 4 (3D methane–air catalytic combustion). The cross-sectional dimensions have been scaled by a factor of 2 for better visualization. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Figure 17. (a) Geometry and boundary conditions for test problem 5 (large-scale 3D catalytic converter) and (b) computational mesh of the cross-sectional plane. The mesh was extruded in the third direction to create a mixture of hexahedral and prismatic cells. 5. SUMMARY AND CONCLUSIONS A new solver for implicit coupled solution of the species conservation equations, referred to as the IDD+GMRES solver, was developed. The solver facilitates both species-to-species as well as spatial (cell-to-cell) coupling in an implicit manner. The governing species conservation equations were discretized using the finite-volume procedure on arbitrary unstructured mesh topology, including mixed mesh topology. The solver is based on decomposition of the computational domain into sub-domains using the BSP algorithm, and subsequent solution of the block implicit equations within each sub-domain using an ILU-preconditioned restarted GMRES solver. Coupling at sub-domain interfaces and nonlinearities is addressed using outer (global) iterations. The solver was coupled to a standalone flow and energy equation solver, and subsequently demonstrated for several different test problems with varying levels of complexity in both 2D Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS Figure 18. Convergence of the IDD+GMRES solver for test problem 5 (large-scale 3D catalytic converter) with 4000 maximum cells in each sub-domain. and 3D geometries. In all cases, the solver was found to be 2–5 times more efficient than solvers based on the block Gauss–Seidel procedure. One of the primary emphases of the current study was to perform all computations on a single-processor machine. It was conclusively demonstrated that the solution strategy developed here enables that, and is attractive both from a memory as well as efficiency standpoint. Reacting flow calculations with complex chemistry and about 60 000 cells required less than an hour of CPU on a single processor, while the largest 3D calculation with approximately 355 000 cells (with 8 gas-phase species, 24-step surface reaction, 4.6 million unknowns) required 900 MB of memory and 19 h of CPU for five orders of magnitude convergence. With regard to the IDD+GMRES solver, the following specific conclusions may be drawn from this study: • From an efficiency standpoint, there is an optimum value for the number of cells that must be placed in each sub-domain. If the sub-domain size is too small, the overheads in setting of the GMRES solver are too large, while if the sub-domain size is too large, the inner solver (GMRES) efficiency deteriorates. Based on the test cases considered, it is clear that the optimum prescribed (N D ) domain size is one with about 2000 cells or more. It is worth noting that although there are optima in each case, the penalty paid in deviating from the optima is not severe. In other words, one could simply set the value of N D to about 2000, and not be concerned about optimizing the solver to achieve the best performance. • The memory required by the IDD+GMRES solver scales with the number of cells in each sub-domain (N D ). The fact that there is an optimum number of cells in each sub-domain for the best performance implies that memory requirement is finite for the best performance. In other words, it is not necessary to use the maximum available memory for the best performance. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Figure 19. Temperature and selected species mass fraction distributions for test problem 5 (large-scale 3D catalytic converter). Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS • The extra memory required by the coupled solver does not scale with the total number of cells in the computational domain. It is strictly a function of the number of species and the number of cells in each sub-domain. Thus, whether the problem is large or small, the extra memory requirement due to implicit coupled solution of the species equations is approximately the same, which implies that for large-scale problems, the extra memory required is a tiny fraction of the overall memory requirement. Future work will focus on using the idea of overlapping sub-domains (overlapping Schwarz algorithm) to remove explicitness at sub-domain boundaries [55] and in using domains of various shapes. The solver is also being parallelized for large-scale calculations, and results of large-scale parallel computations will be reported shortly in a companion paper. APPENDIX A: FINITE-VOLUME INTEGRATION AND BLOCK-IMPLICIT FORM OF EQUATIONS In this appendix, the finite-volume integration of the general scalar advection–diffusion equation is first outlined. Consider the advection–diffusion equation for a scalar, , at steady state: ∇ ◦(U) = ∇ ◦(∇) (A1) where the symbols have their usual meanings (see Nomenclature). Integration of Equation (A1) over an arbitrary control volume of volume and control surface area S yields ∇ ◦(U) dV = ∇ ◦(∇) dV (A2) which upon application of the Gauss-divergence theorem results in (U)◦ n̂ dA = (∇)◦ n̂ dA S (A3) S In the case where the control volume (or cell) is bounded by discrete faces, as shown in Figure A1, the surface integrals in Equation (A3) can be replaced by discrete summations, leading to (for the stencil shown in Figure A1) 3 (U)f ◦ n̂f Af = f=1 3 (∇)f ◦ n̂f Af (A4) f=1 The computation of the left-hand side is straightforward. Depending on what scheme is used for advection (e.g. upwind, central, etc.) the face scalar, f , can be expressed in terms of cell center scalars. Similarly, the face density, f , can be computed using weighted interpolation of the cell center densities. The velocity at any face is obtained using pressure-weighted interpolation [49, 50], as discussed earlier. And since the area and surface normal at each face are known from the mesh, the left-hand side of the equation can be expressed in terms of the cell center values of . Consider the right-hand side term Af (∇)f ◦ n̂f . In order to cast this term as an algebraic equation in terms of the values of at the cell centers, it is necessary to express ∇f ◦ n̂f in terms of cell center values of . In order to do so, we begin with the vector identity ∇f = (∇f ◦ n̂f )n̂f +(n̂f ×∇f )× n̂f Copyright q 2009 John Wiley & Sons, Ltd. (A5) Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER Figure A1. Unstructured stencil and explanation of the various notations used in the finite-volume integration. Performing a dot product with the vector lf (see Figure A1), we get ∇f ◦lf = (∇f ◦ n̂f )n̂f ◦lf +[(n̂f ×∇f )× n̂f ]◦lf (A6) which upon further simplification yields 2,f −1,f = (∇f ◦ n̂f )f +[(n̂f ×∇f )× n̂f ]◦lf (A7) where f is the distance, in the direction of the surface normal, between the two cells straddling the face f (Figure A1). Equation (A7) can be further rearranged as (∇f ◦ n̂f ) = 2,f −1,f [(n̂f ×∇f )× n̂f ]◦lf − f f (A8) Essentially, in the above derivation, the flux normal to a cell face has been expressed in terms of the cell center values straddling the face, and a separate tangential flux term, which will, henceforth, be denoted by f . It can be readily shown that if the vectors n̂f and lf are aligned, this term vanishes. Substitution of Equation (A8) into Equation (A4) yields 3 3 2,f −1,f f (A9) (U)f ◦ n̂f Af = f Af − f f f =1 f =1 In our scheme, the first term of the right-hand side of Equation (9) is treated implicitly, while the second term is treated explicitly, that is, it is computed from previous iteration values. This computation of the second term requires determination of vertex values, and further application of the Gauss–divergence theorem. The derivation of the exact expression for this term is rather lengthy, and is omitted here for the sake of brevity. It may be found elsewhere [41, 42]. If the procedure, just described, is applied to the set of species conservation equations (Equation (15)), after much tedious algebra, the final result will be a set of algebraic equations of the form (for node O) N n=1 Copyright q Ak,n,O Yn,O − N nb n=1 j =1 2009 John Wiley & Sons, Ltd. Ak,n, j Yn, j = Sk ∀k = 1, 2, . . ., N −1 (A10) Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS where Ak,n, j are the link coefficients (combined convective and diffusive) for the kth species conservation equation and the j th neighboring cell, Yn, j are the mass fractions of the nth species and the j th neighboring cell, nb is the number of neighboring cells of node O (Figure A1), and Sk is the source term of the kth species arising either out of chemical reactions or skewness in the mesh (tangential flux component, as described above). Equation (A10) represents a set of N −1 coupled equations for the node O. In conjunction with the mass fraction summation criterion (Equation (7)), it can be written in matrix form as [2]: ⎡ A1,1,O ⎢ .. ⎢ . ⎢ ⎢ ⎢ ⎣ A N −1,1,O A1,2,O ... .. . .. . ... ... ... ... 1 ⎡ A1,1, j ⎢ .. nb ⎢ . ⎢ − ⎢ ⎢ j =1 ⎣ A N −1,1, j 0 A1,N ,O ⎤⎡ Y1,O ⎤ ⎥ ⎥⎢ ⎥ ⎢ Y2,O ⎥ ⎥ ⎥⎢ ⎥⎢ . ⎥ ⎥⎢ . ⎥ A N −1,N ,O ⎦ ⎣ . ⎦ .. . Y N ,O ⎤⎡ 1 A1,2, j ... .. . .. . ... ... ... ... A1,N , j Y1, j ⎤ ⎡ S1 ⎤ ⎥ ⎢ ⎥⎢ ⎥ ⎥ ⎢ Y2, j ⎥ ⎢ .. ⎥ ⎥ ⎢ . ⎥ ⎥⎢ ⎥⎢ . ⎥=⎢ ⎥ ⎥ ⎢ ⎥⎢ ⎥ A N −1,N , j ⎦ ⎣ .. ⎦ ⎣ S N −1⎦ .. . 0 YN , j (A11) 1 When considered for all nodes (= NC,m ) in the mth sub-domain, that is, O ∈ 1, 2, . . ., NC,m , Equation (A11) represents a block matrix system of equations, where each element of the block matrix is an N × N matrix itself. These element matrices are shown explicitly in Equation (A11) for a representative interior node, O. NOMENCLATURE B c p,k Dkn Dkn Dkm , Dk h hk h 0f,k Jk kc M Mk n̂ N ND Copyright q body force vector (m s−2 ) specific heat capacity of species k (J kg−1 K−1 ) binary diffusion coefficient of species k into n (m2 s−1 ) ordinary multi-component diffusion coefficient (m2 s−1 ) effective diffusivity of species k into the mixture (m2 s−1 ) enthalpy of mixture (J kg−1 ) enthalpy of the kth species (J kg−1 ) enthalpy of formation of species k at standard state (J kg−1) diffusion mass flux of the kth species (kg m−2 s−1 ) thermal conductivity of mixture (W m−1K−1 ) molecular weight of the mixture (kg kmol−1) molecular weight of kth species (kg kmol−1) unit surface normal vector total number of gas-phase species maximum prescribed number of cells in each sub-domain 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER p q qC qD qR Ṙk Ṡk Ṡh T U Xk Yk pressure (Pa) heat flux (W m−2) heat flux due to conduction (W m−2) heat flux due to inter-diffusion of species (W m−2) heat flux due to radiation (W m−2) production rate of species due to heterogeneous reactions (kmolm−2 s−1 ) production rate of kth species due to homogeneous reactions (kg m−3 s−1 ) volumetric enthalpy production rate (W m−3) temperature (K) mass averaged velocity (m s−1 ) mole fraction of kth species mass fraction of kth species Greek symbols kn kn k s Kronecker delta transformed diffusion coefficient tensor (m2 s−1 ) molar concentration of species k at fluid-solid interface (kmolm−3 ) mixture density (kg m−3 ) shear stress (N m−2) ACKNOWLEDGEMENTS Financial support for this work was provided, in part, by ESI Group. The authors would also like to thank numerous colleagues and anonymous reviewers of the conference versions of their papers for providing suggestions for improvement. REFERENCES 1. Venkateswaran S, Olsen M. Stability analysis of fully-coupled and loosely-coupled schemes in combustion CFD. Proceedings of the 16th AIAA Computational Fluid Dynamics Conference, Orlando, FL, 23–26 June 2003. AIAA Paper Number 2003-3543. 2. Mazumder S. Critical assessment of the stability and convergence of the equations of multi-component diffusion. Journal of Computational Physics 2006; 212(2):383–392. 3. Bochev PB, Garasi CJ, Hu JJ, Robinson AC, Tuminaro RS. An improved algebraic multigrid method for solving Maxwell’s equations. SIAM Journal on Scientific Computing 2003; 25:623–642. 4. Mathur SR, Murthy JY. Acceleration of anisotropic scattering computations using coupled ordinates method (COMET). Journal of Heat Transfer 2001; 123(3):607–612. 5. Mazumder S. A new numerical procedure for coupling radiation in participating media with other modes of heat transfer. Journal of Heat Transfer 2005; 127(9):1037–1045. 6. Vanka SP. Block-implicit multigrid solution of Navier–Stokes equations in primitive variables. Journal of Computational Physics 1986; 65:138–158. 7. de Lemos MJS. A block-implicit numerical procedure for simulation of buoyant swirling flows in a model furnace. International Journal for Numerical Methods in Fluids 2003; 43:281–299. 8. Oran E, Boris JP. Numerical Simulation of Reactive Flows (2nd edn). Cambridge University Press: Cambridge, 2005. 9. Ferziger J, Peric M. Computational Methods for Fluid Dynamics (2nd edn). Springer: Berlin, 1999. 10. Duan T, Weidner JW, White RE. Extension of Newman’s method to electrochemical reaction–diffusion in a fuel cell catalyst layer. Journal of Power Sources 2002; 107(1):24–33. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld COUPLED SOLUTION OF THE SPECIES CONSERVATION EQUATIONS 11. Cete AR, Yukselen MA, Kanyak U. A unifying grid approach for solving potential flows applicable to structured and unstructured grid configurations. Computers and Fluids 2008; 37:35–50. 12. Li D, Venkateswaran S, Fakhari K, Merkle CL. Convergence assessment of general fluid equations on unstructured hybrid grids. Fifteenth AIAA Computational Fluid Dynamics Conference, Annaheim, CA, 11–14 June 2001. Paper Number AIAA 2001-2557. 13. Wesseling P. An Introduction to Multi-grid Methods. Edwards Inc.: Philadelphia, PA, U.S.A., 2004. 14. Jyotsna R, Vanka SP. Multigrid calculation of steady viscous flow in a triangular cavity. Journal of Computational Physics 1995; 122:107–117. 15. Gupta MM, Kouatchou J, Zhang J. A compact multigrid solver for convection–diffusion equations. Journal of Computational Physics 1997; 132:123–129. 16. Volker J, Tobiska L. Numerical performance of smoothers in coupled multigrid methods for the parallel solution of the incompressible Navier–Stokes equations. International Journal for Numerical Methods in Fluids 2000; 33:453–473. 17. Chow E. An unstructured multigrid method based on geometric smoothness. Numerical Linear Algebra with Applications 2003; 10:401–421. 18. Stüben K. A review of algebraic multigrid. Computational and Applied Mathematics 2001; 128:281–309. 19. Wabro M. Coupled algebraic multigrid methods for the Oseen problem. Computing and Visualization in Science 2004; 7:141–151. 20. Raw MJ. A coupled algebraic multigrid method for the 3D Navier–Stokes equations. Fast Solvers for Flow Problems. Proceedings of the Tenth GAMM-Seminar, Kiel, Germany, 1994. Notes on Numerical Fluid Mechanics, vol. 49. Vieweg: Wiesbaden, 1995; 204–215. 21. Heys JJ, ManTeuffel TA, McCormick SA, Olson LN. Algebraic multigrid for higher order finite elements. Journal of Computational Physics 2005; 204:520–532. 22. Webster R. Algebraic multigrid and incompressible fluid flow. International Journal for Numerical Methods in Fluids 2007; 53:669–690. 23. Cheney W, Kincaid W. Numerical Mathematics and Computing (2nd edn). Brooks-Cole Publishing Company: Pacific Grove, CA, U.S.A., 1985. 24. Liu G, Thompson KE. A domain decomposition method for modeling Stokes flow in porous materials. International Journal for Numerical Methods in Fluids 2002; 38:1009–1025. 25. Kuo KK. Principles of Combustion. Wiley: New York, 1986. 26. Pope SB, Ren Z. Efficient implementation of chemistry in computational combustion. Flow, Turbulence and Combustion 2008; DOI: 10.1007/s10494-008-9145-3.2008. 27. Singer MA, Pope SB, Najm HN. Operator-splitting with ISAT to model reacting flow with detailed chemistry. Combustion Theory and Modelling 2006; 10(2):199–217. 28. Kim SH, Huh KY, Fraser RA. Modeling autoignition of a turbulent methane jet by the conditional moment closure model. Symposium (International) on Combustion 2000; 28(1):185–191. 29. Wright YM, De Paola G, Boulouchos K, Mastorakos E. Simulations of spray autoignition and flame establishment two two-dimensional CMC. Combustion and Flame 2005; 143(4):402–419. 30. Pope S. Computationally efficient implementation of combustion chemistry using In Situ adaptive tabulation. Combustion Theory and Modelling 1997; 1:41–63. 31. Ropp DL, Shadid JN, Ober CC. Studies of the accuracy of time integration methods for reaction–diffusion equations. Journal of Computational Physics 2004; 194:544–574. 32. Shadid JN, Salinger A, Schmidt R, Smith T, Hutchinson S, Hennigan G, Devine K, Moffat H. MPSalsa: a finite element computer program for reacting flow problems; part 1—theoretical development. Technical Report, SAND98-2864, Sandia National Laboratories, 1998. 33. Shadid JN, Tuminaro RS, Walker HF. An inexact Newton method for fully coupled solution of the Navier–Stokes equations with heat and mass transport. Journal of Computational Physics 1997; 137:155–185. 34. Shadid JN, Tuminaro RS, Devine KD, Hennigan GL, Lin PT. Performance of fully coupled domain decomposition preconditioners for finite-element transport/reaction simulations. Journal of Computational Physics 2005; 205: 24–47. 35. Bird RB, Stewart WE, Lightfoot EN. Transport Phenomena (2nd edn). Wiley: New York, 2001. 36. Mazumder S. Modeling full-scale monolithic catalytic converters: challenges and possible solutions. Journal of Heat Transfer 2007; 129(4):526–535. 37. Wangard W, Dandy DS, Miller BJ. A numerically stable method for integration of the multi-component species diffusion equations. Journal of Computational Physics 2001; 174:460–472. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld A. KUMAR AND S. MAZUMDER 38. Sutton K, Gnoffo PA. Multicomponent diffusion with application to computational aerothermodynamics. AIAA Paper Number 98-2575, 1998. 39. Coltrin M, Kee RJ, Rupley F. Surface Chemkin: a general formalism and software for analyzing heterogeneous chemical kinetics at gas-solid interfaces. International Journal of Chemical Kinetics 1991; 23:1111–1128. 40. Mazumder S, Lowry SA. The treatment of reacting surfaces for finite-volume schemes on unstructured meshes. Journal of Computational Physics 2001; 173(2):512–526. 41. Mathur SR, Murthy JY. A pressure based method for unstructured meshes. Numerical Heat Transfer, Part B 1997; 31(2):195–216. 42. Date AW. Introduction to Computational Fluid Dynamics. Cambridge University Press: Cambridge, 2005. 43. Saad Y. Iterative Methods for Sparse Linear Systems (2nd edn). SIAM: Philadelphia, PA, 2003. 44. Toselli A, Widlund O. Domain Decomposition Methods—Algorithms and Theory. Springer Series in Computational Mathematics, vol. 34. Springer: New York, 2004. 45. Sung K, Shirley P. Ray tracing with a BSP tree. In Computer Graphics Gems III, Kirk D (ed.). AP Professional: San Diego, CA, 1992; 271–274. 46. Karypis G, Kumar V. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing 1998; 20:359–392. 47. Kumar A, Mazumder S. A low-memory block-implicit solver for coupled solution of the species conservation equations on an unstructured mesh. Proceedings of the International Mechanical Engineering Congress and Exposition IMECE2007, Seattle, WA, U.S.A., 2007. Paper Number 2007-42517. 48. Van Doormal J, Raithby GD. Enhancement of the SIMPLE method for predicting incompressible flows. Numerical Heat Transfer 1984; 7:147–163. 49. Rhie CM, Chow WL. A numerical study of the turbulent flow past an isolated airfoil with trailing edge separation. AIAA Journal 1983; 21:1525–1532. 50. Miller TF, Schmidt FW. Use of a pressure weighted interpolation method for the solution of incompressible Navier–Stokes equations with non-staggered grid system. Numerical Heat Transfer, Part B: Fundamentals 1988; 14:213–233. 51. Chatterjee D, Deutschamnn O, Warnatz J. Detailed surface reaction mechanism in a three-way catalyst. Faraday Discussions 2001; 119:371–384. 52. Selle L, Lartigue G, Poinsot T, Koch R, Schildmacher KU, Krebs W, Prade B, Kaufmann P, Veynante D. Compressible large eddy simulation of turbulent combustion in complex geometry on unstructured meshes. Combustion and Flame 2004; 137:489–505. 53. Deutschmann O, Behrendt F, Warnatz J. Modeling and simulation of heterogeneous oxidation of methane on a platinum foil. Catalysis Today 1994; 21:461–471. 54. Westbrook CK, Dryer FL. Simplified reaction mechanisms for the oxidation of hydrocarbon fuels in flames. Combustion Science and Technology 1981; 27:31–43. 55. Smith B, Bjorstad P, Gropp W. Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge University Press: Cambridge, England, 1996. Copyright q 2009 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Fluids (2009) DOI: 10.1002/fld
© Copyright 2026 Paperzz