a detailed description of flow in porous media is essential for building mathematical models with applications in , for example , water management , oil and gas recovery , carbon dioxide ( co@xmath0 ) sequestration or nuclear waste disposal . in order to set up a reliable numerical model
, one needs to have a good knowledge of the problem geometry and input parameters .
for example , the flow of water in granite rock , which is a suitable site for nuclear waste disposal , is driven by the complex system of vugs , cavities and fractures with various topology and sizes .
these alter the effective permeability , and therefore should be accurately accounted for in the numerical model .
there are two main approaches : either the fractures are considered as free - flow regions , or the fractures contain debris and are also modelled as porous media with specific permeabilities . in the first case , a  unified approach to modelling
free - flow and porous media regions can be provided by the so called stokes - brinkman equation , which reduces to either the stokes or darcy model in certain parameter limits , e.g. , within the multiscale mixed finite - element ( msmfe ) framework  @xcite . in this paper
, we consider the latter case , and apply darcy s law to the flow in the reservoir and in the fractures as well ; see  @xcite for a related approach . in either case , the preferential flow in large geological dislocations and their intersections should be considered as two- and one - dimensional flows , respectively . due to the quite complex structure of the domains
, the discretisation is performed using finite element methods  ( fem ) .
the resulting meshes are therefore unstructured , and they combine different spatial dimensions ( line elements in 1d , triangles in 2d , and tetrahedrons in 3d ) . the systems of linear equations obtained from the fem discretisation are often very large , so that using direct methods is prohibitive and iterative solvers are warranted .
the systems are typically also bad - conditioned due to the mixing of spatial dimensions , large jumps in permeability coefficients and presence of elements of considerably different sizes , and so they are challenging for iterative solvers as well .
the matrices have a saddle - point structure @xmath1{cc}a & \ol{b}^{t}\\ \ol{b } & -\ol{c } \end{array } \right ]   , \label{eq : sp - matrix}\ ] ] where @xmath2 is symmetric positive definite on the kernel of @xmath3 , @xmath4 is symmetric positive semi - definite , and it is positive definite on the kernel of @xmath5 .
the ` penalty ' block @xmath6 arises from connecting meshes of different spatial dimensions .
the iterative solution of systems with this structure is a frequently studied topic ; see , for example ,  @xcite , the monographs  @xcite or  toselli-2005-ddm and the references therein .
however , efficient methodologies for solving saddle - point problems are typically problem dependent .    in this paper
, we develop a robust and scalable solver for linear systems with the saddle - point structure as in  ( [ eq : sp - matrix ] ) with the block@xmath7 either zero or nonzero .
the solver is tailored to the mixed - hybrid formulation of flow in porous media using the lowest order raviart - thomas ( @xmath8 ) finite elements with combined mesh dimensions ( 1d , 2d and 3d ) .
in particular , we adapt the balancing domain decomposition by constraints ( bddc ) method to this type of problems .
the bddc method is currently one of the most popular methods of iterative substructuring .
it has been proposed independently in @xcite ; see  @xcite for the proof of equivalence .
even though bddc has been originally formulated for elliptic problems , it has been successfully extended , for example , beyond elliptic cases  @xcite and to multiple levels  @xcite .
an optimal set - up has been studied in  @xcite .
a closely related bddc preconditioner for vector field problems discretised with raviart - thomas finite elements has been studied in  @xcite .
we are interested in applications of the bddc method to saddle - point problems . if @xmath9 in  ( [ eq : sp - matrix ] ) , one possible approach is to use an algebraic trick and constrain the iterative solution of the indefinite problem into a _ balanced
_ subspace , which is sometimes also called _ benign _ , where the operator is positive definite ; see  @xcite for the stokes problem , and  @xcite for flow in porous media .
however , due to the mixed - hybrid formulation and possible coupling of meshes with different spatial dimensions , @xmath6 in general , and we will favour an alternative , _ dual _ approach here .
our methodology is as follows .
the mixed - hybrid formulation @xcite is used in order to modify the saddle - point problem to one which is symmetric and positive definite by means of iterative substructuring .
in particular , we introduce a symmetric positive definite schur complement with respect to interface lagrange multipliers , corresponding to a part of block  @xmath4 .
the reduced system is solved by the preconditioned conjugate gradient ( pcg ) method , and the bddc method is used as a preconditioner . from this perspective , our work can be viewed as a further extension of  @xcite .
our main effort here is in accommodating the bddc solver to flows in porous media with combined mesh dimensions .
in addition , the presentation of the bddc algorithm is driven more by an efficient implementation , while it is more oriented towards underlying theory in @xcite .
we take advantage of the special structure of the blocks in matrix  ( [ eq : sp - matrix ] ) studied in detail in  @xcite . in particular , the nonzero structure of block @xmath4 resulting from a combination of meshes with different spatial dimensions is considered in @xcite .
we describe our parallel implementation of the method and study its performance on several benchmark and real - world problems .
another original contribution of this paper is proposing a new scaling operator in the bddc method suitable for the studied problems .
we note that if there is no coupling of meshes with different spatial dimensions present in the discretisation , the block @xmath9  in  ( [ eq : sp - matrix ] ) and our method is almost identical to the one introduced in  @xcite .
the paper is organised as follows . in section  [ sec : model ] , we introduce the model problem . in section  [ sec : fractures ] , we describe the modelling of fractured porous media and combining meshes of different dimensions . in section  [ sec : substructuring ] , we introduce the substructuring components and derive the interface problem . in section  [ sec : bddc ] , we formulate the bddc preconditioner . in addition , the selection of interface weights for bddc is studied in detail in section  [ sec : scaling ] . in section  [ sec : solver ] , we describe our parallel implementation , and in section  [ sec : numerical ] we report the numerical results and parallel performance for benchmark and engineering problems .
finally , section  [ sec : conclusion ] provides a summary of our work .    our notation does not , for simplicity , distinguish between finite element functions and corresponding algebraic vectors of degrees of freedom , and between linear operators and matrices within a specific basis
the meaning should be clear from the context .
the transpose of a matrix is denoted by superscript @xmath10 and the energy norm of a vector@xmath11 is denoted  by @xmath12 , where  @xmath13 is a symmetric positive definite matrix .
let @xmath14 be an open bounded polyhedral domain in @xmath15 .
we are interested in the solution of the following problem , combining darcy s law and the equation of continuity written as @xmath16 subject to boundary conditions on @xmath17 , where @xmath18 stands for the part of the boundary with _ natural _ ( dirichlet ) boundary condition , and @xmath19 for the part with _ essential _ ( neumann ) boundary condition . in applications ,
the variable@xmath20 describes the velocity of the fluid and @xmath21 the pressure ( head ) in an aquifer @xmath14 , @xmath22 is a symmetric positive definite tensor of the hydraulic conductivity , @xmath23 is the gravity term , and @xmath24 is the outer unit normal vector of @xmath25 .
the term @xmath26 is present due to the fact , that @xmath27 satisfies @xmath28 , where @xmath29 is the piezometric head . for a thorough  discussion of application background
we refer , e.g. , to monographs  @xcite .
let @xmath30 be the triangulation of domain @xmath14 consisting of @xmath31 simplicial elements with characteristic size @xmath32 .
we introduce a space @xmath33 equipped with the standard norm .
let @xmath34 be the space consisting of the lowest order raviart - thomas ( @xmath8 ) functions and let @xmath35 be the space consisting of piecewise constant functions on the elements of the triangulation @xmath30 .
we refer , e.g. , to monograph  @xcite for a detailed description of the mixed finite elements and the corresponding spaces .    in the _ mixed finite element approximation _ of problem ( [ eq : problem-1])([eq : problem-4 ] )
we look for a pair @xmath36 that satisfies @xmath37    in the discrete formulation , we need @xmath38 and @xmath39 only sufficiently regular so that the integrals in the weak formulation  ( [ eq : mixed-1])([eq : mixed-2 ] ) make sense , namely @xmath40 , @xmath41 .
next , we recall the mixed - hybrid formulation .
it was originally motivated by an effort to modify the saddle - point problem ( [ eq : mixed-1])([eq : mixed-2 ] ) to one which leads to symmetric positive definite matrices . nevertheless , this formulation is also suitable for a combination of meshes with different spatial dimensions , which will be described in detail in the next section .
let @xmath42 denote the set of inter - element _ faces _ of the triangulation @xmath30 .
we now introduce several additional spaces .
first , let us define the space @xmath43 by relaxing the condition of continuity of the normal components in the space @xmath44 on inter - element boundaries @xmath42 .
more precisely , we define local spaces @xmath45 for each element @xmath46 , @xmath47 , by @xmath48 and put @xmath49 .
next , we define the space of lagrange multipliers @xmath50 consisting of functions that take constant values on individual inter - element faces in @xmath42 , @xmath51 in particular , @xmath52 on @xmath25 for any @xmath53 .    in the _ mixed - hybrid finite element approximation _ of problem ( [ eq : problem-1])([eq : problem-4 ] ) , we look for a triple @xmath54 that satisfies @xmath55    &   \label{eq : hybrid-1}\\ = -\int_{\partial\omega_{n}}p_{n}\mathbf{v}\cdot\mathbf{n}\,ds-\sum_{i=1}^{n_{e}}\int_{t^{i}}v_{z}\,dx,\quad\forall\mathbf{v }   &   \in\mathbf{v}^{-1},\nonumber\\ -\sum_{i=1}^{n_{e}}\left [   \int_{t^{i}}q\nabla\cdot\mathbf{u}\,dx\right ]    & = -\int_{\omega}fq\,dx,\quad\forall q\in q,\label{eq : hybrid-2}\\ \sum_{i=1}^{n_{e}}\left [   \int_{\partial t^{i}\setminus\partial\omega}\mu(\mathbf{u}\cdot\mathbf{n})|_{\partial t_{i}}\,ds\right ]    & = 0,\quad\forall\mu\in\lambda .
\label{eq : hybrid-3}\ ] ] equation ( [ eq : hybrid-3 ] ) imposes a continuity condition on the normal component of the velocity ( also called _ normal flux _ ) @xmath56 across @xmath42 which guarantees that @xmath57 .
this condition also implies the equivalence of the two formulations ( [ eq : mixed-1])([eq : mixed-2 ] ) and ( [ eq : hybrid-1])([eq : hybrid-3 ] ) .
we note that the lagrange multipliers @xmath58 can be interpreted as the approximation of the trace of @xmath21 on @xmath42 , see  @xcite for details .
let us now write the matrix formulation corresponding to ( [ eq : hybrid-1])([eq : hybrid-3 ] ) as @xmath1{ccc}a & b^{t } & b_{\mathcal{f}}^{t}\\ b & 0 & 0\\ b_{\mathcal{f } } & 0 & 0 \end{array } \right ]   \left [ \begin{array } [ c]{c}\mathbf{u}\\ p\\ \lambda \end{array } \right ]   = \left [ \begin{array } [ c]{c}g\\ f\\ 0 \end{array } \right ]   .
\label{eq : hybrid - m}\ ] ] it is important to note that @xmath2 is block diagonal with @xmath31 blocks , corresponding to elements @xmath59 , @xmath60 , and each of the blocks is symmetric positive definite , cf . the first term in  ( [ eq : hybrid-1 ] ) .
it was shown in  @xcite that the system of equations ( [ eq : hybrid - m ] ) can be reduced ( twice ) to the schur complement corresponding to the lagrange multipliers @xmath58 and solved efficiently by a direct or iterative solver . here
, we will look for an efficient solution of a slightly modified , and in general also block dense , system which is introduced in the next section .
in this section , we recall the main ideas of the discrete model of the flow in fractured porous media that is based on connection of meshes of different dimensions as described in  @xcite .
let us denote the full domain by @xmath61 .
next , consider lower - dimensional domains @xmath62 , @xmath63 , such that @xmath64 consists of polygons and @xmath65 consists of line segments .
we will also assume that @xmath66 .
the first condition requires that a domain of a lower dimension can not poke out of the domain of higher dimension , while the second condition prevents domains of lower dimension from having boundaries in the interior of domains of higher dimension .
we impose these conditions to avoid technical difficulties in the analysis .
however , numerical evidence suggests that these conditions are not necessary , and in fact , they are not satisfied for the real - world problems presented in section  [ sec : results_engineering ] .    for every dimension @xmath67 , we introduce a triangulation @xmath68 of the domain @xmath69 that consists of finite elements @xmath70  @xmath71 and satisfies the compatibility conditions @xmath72 for every @xmath73 , @xmath74 , and @xmath63 .
this means that elements of a lower dimension match faces of elements of the higher dimension .
we consider equations ( [ eq : problem-2])([eq : problem-4 ] ) on the domains @xmath69 , @xmath67 , completed by a slight modification of the darcy s law  ( [ eq : problem-1 ] ) : @xmath75 where @xmath76 stands for the velocity integrated over the cross - section for @xmath77 , i.e. the units of @xmath78 , @xmath79 , and @xmath80 are ms@xmath81 , m@xmath82s@xmath81 , and m@xmath83s@xmath81 , respectively .
in addition , @xmath84 , @xmath85 is the thickness of a fracture , and @xmath86 is the cross - section of a 1d preferential channel .
the effective fluid source @xmath87 on @xmath64 is given as @xmath88 where @xmath89 is the density of external fluid sources , and the normal fluxes from the two faces of the 3d continuum surrounding the fracture are given through the robin ( also called newton ) boundary conditions @xmath90 in the last formula , @xmath91 are the transition coefficients ( cf .
@xcite for possible choices ) and @xmath92 , @xmath93 are the traces of pressure @xmath94 on the two sides of the fracture .
the effective fluid source @xmath95 on @xmath65 is similar , @xmath96 where @xmath97 is the density of external fluid sources . in the 3d ambient space
, the 1d channel can be connected to @xmath98 faces of 2d fractures .
thus @xmath99 is the normal flux from the connected fracture @xmath98 , @xmath100 is the transition coefficient , and @xmath101 is the trace of pressure @xmath102 on the face of fracture @xmath98 .    in the following ,
we describe the discrete mixed - hybrid formulation of the problem . the formulation and discussion of the continuous problem
can be found in  @xcite .
let us consider spaces@xmath103 for the definition of the space @xmath50 , we can not follow ( [ eq : lambda_space ] ) directly , since e.g. on @xmath64 , we need to distinguish values of  @xmath104 on two sides of a fracture .
thus , we introduce a separate value for every non - boundary side of every element : @xmath105 where @xmath106 is defined in the same way as the space @xmath44 but on the domain @xmath69 .
further , we identify values on faces / points that are not aligned to the fractures / channels : @xmath107 finally , we redefine @xmath108 . in the _ mixed - hybrid finite element approximation _ of the flow in fractured porous media we seek a triple @xmath109 that satisfies @xmath110 with @xmath111   , \label{eq : weak_term_a}\\ b\left (   \mathbf{u},q\right )    &   = -\sum_{d=1}^{3}\sum_{i=1}^{n_{e}^{d}}\left [   \int_{t_{d}^{i}}q_{d}\,\left (   \nabla\cdot\mathbf{u}_{d}\right ) \,dx\right ]   , \\
b_{\mathcal{f}}\left (   \mathbf{u},\lambda\right )    &   = \sum_{d=1}^{3}\sum_{i=1}^{n_{e}^{d}}\left [   \int_{\partial t_{d}^{i}\setminus\partial \omega_{d}}\lambda|_{\partial t_{d}^{i}}\,\left (   \mathbf{u}_{d}\cdot\mathbf{n}\right )   \,ds\right ]   , \\
c\left (   p , q\right )    &   = \sum_{d=2}^{3}\sum_{i=1}^{n_{e}^{d}}\left [   \int_{\partial t_{d}^{i}\cap\omega_{d-1}}\sigma_{d}\,\,p_{d-1}\,q_{d-1}\,ds\right ]   , \label{eq : weak_term_cbar}\\ c_{\mathcal{f}}\left (   p,\mu\right )    &   = -\sum_{d=2}^{3}\sum_{i=1}^{n_{e}^{d}}\left [   \int_{\partial t_{d}^{i}\cap\omega_{d-1}}\sigma_{d}\ p_{d-1}\,\mu_{d}\,ds\right ]   , \label{eq : weak_term_cf}\\ \widetilde{c}\left (   \lambda,\mu\right )    &   = \sum_{d=2}^{3}\sum_{i=1}^{n_{e}^{d}}\left [   \int_{\partial t_{d}^{i}\cap\omega_{d-1}}\sigma _ { d}\,\,\lambda_{d}\,\mu_{d}\,ds\right ]   ,
\label{eq : weak_term_ctilde}\\ \left\langle g,\mathbf{v}\right\rangle   &   = -\sum_{d=1}^{3}\sum_{i=1}^{n_{e}^{d}}\int_{\partial t_{d}^{i}\cap\partial\omega_{n}}p_{n}\,\left ( \mathbf{v}\cdot\mathbf{n}\right )   \,ds   { -\sum_{d=1}^{3}\sum_{i=1}^{n_{e}^{d}}\int_{t_{d}^{i}}v_{z}\,dx},\\ \left\langle f , q\right\rangle   &   = -\sum_{d=1}^{3}\int_{\omega } \delta_{d}\,\tilde{f}_{d}\,q_{d}\,dx .
\label{eq : weak_term_f}\ ] ]    the system  ( [ eq : hybrid - frac-1])([eq : hybrid - frac-3 ] ) now leads to the matrix form @xmath1{ccc}a & b^{t } & b_{\mathcal{f}}^{t}\\ b & -c & -c_{\mathcal{f}}^{t}\\ b_{\mathcal{f } } & -c_{\mathcal{f } } & -\widetilde{c}\end{array } \right ]   \left [ \begin{array } [ c]{c}\mathbf{u}\\ p\\ \lambda \end{array } \right ]   = \left [ \begin{array } [ c]{c}g\\ f\\ 0 \end{array } \right ]   .
\label{eq : hybrid - m - fractured}\ ] ]    we note that  ( [ eq : hybrid - m - fractured ] ) is related to  ( [ eq : hybrid - frac-1])([eq : hybrid - frac-3 ] ) in the same way as  ( [ eq : hybrid - m ] ) is related to  ( [ eq : hybrid-1])([eq : hybrid-3 ] ) . the main difference in the structure of the matrices between ( [ eq : hybrid - m - fractured ] ) and  ( [ eq : hybrid - m ] ) is the additional block @xmath112{cc}c & c_{\mathcal{f}}^{t}\\ c_{\mathcal{f } } & \widetilde{c}\end{array } \right ]   $ ] related to the normal fluxes between dimensions and arising from  ( [ eq : newton-1])([eq : newton-2 ] ) and ( [ eq : newton-3 ] ) via ( [ eq : weak_term_cbar])([eq : weak_term_ctilde ] ) . in particular , the modified right - hand side of the continuity equation for two - dimensional and one - dimensional elements , @xmath87 and @xmath95 , incorporates pressure unknowns on 2d and 1d elements , and traces of pressure on 3d and 2d elements at the fracture , which are nothing but the lagrange multipliers on 3d and 2d elements in the mixed - hybrid method .
consequently , @xmath113 in  ( [ eq : newton-1])([eq : newton-2 ] ) , and @xmath114 in  ( [ eq : newton-3 ] ) .
assuming  @xmath115 bounded and greater than zero , and using the fact that  @xmath116 corresponds to a symmetric positive definite matrix , we see from ( [ eq : weak_term_a ] ) that block @xmath2 in  ( [ eq : hybrid - m - fractured ] ) is symmetric positive definite .
block @xmath4 is symmetric positive semi - definite since @xmath117   .\ ] ]    the following theorem is a standard result , e.g. brezzi-1991-mhf . here ,
we rewrite it in a form suitable for our setting and we verify the assumptions for the specific blocks of the matrix in ( [ eq : hybrid - m - fractured ] ) .
we will further denote @xmath118 , @xmath119 , @xmath120 , and @xmath121 .
[ th : regular ] let natural boundary conditions ( [ eq : problem-3 ] ) be prescribed at a certain part of the boundary , i.e. @xmath122 for at least one @xmath123 .
then the discrete mixed - hybrid problem  ( [ eq : hybrid - m - fractured ] ) has a unique solution .
let us first investigate the structure of the matrix in ( [ eq : hybrid - m - fractured ] ) more closely .
let us number the unknowns within each block of ( [ eq : hybrid - m - fractured ] ) with respect to spatial dimension @xmath123 .
the matrix then takes the form of 9@xmath1249 blocks , @xmath125.\ ] ]
suppose for a  moment that we solve a  problem only on domain @xmath69 , @xmath126 ( i.e. @xmath127 for @xmath128 ) .
if no natural boundary conditions are imposed , there is a single @xmath129 entry on each row of @xmath130 and a single @xmath131 entry on each row of @xmath132 . since @xmath69 is a simply connected set , the matrix @xmath133 $ ] has a nontrivial nullspace of constant vectors . enforcing natural boundary condition on a part of
@xmath134 changes the @xmath131 value on the corresponding row of matrix @xmath132 to @xmath135 , in which case @xmath136 has only a trivial nullspace , i.e. full column rank ( see e.g. ( * ? ? ?
* section iv.1 ) or ( * ? ? ? * lemma  3.2 ) ) .
the nullspace becomes more complicated for domains with fractures , in which case @xmath69 typically has more simply connected components separated by fractures ( cf .
[ fig : schemes ] ) .
let us denote them @xmath137 , @xmath138 , regardless of the dimension .
in particular , @xmath137 , @xmath139 will be components without natural condition boundary , i.e. @xmath140 , while for @xmath141 we get components with prescribed natural boundary condition .
we also denote @xmath142 the characteristic vector of the component @xmath137 that takes value @xmath131 for degrees of freedom associated with elements and faces of the component @xmath137 . with such notation ,
the basis of the nullspace of the whole matrix @xmath143\ ] ] consists of characteristic vectors @xmath144 , @xmath145 and has dimension @xmath146 .
next , we show that matrix @xmath4 is not only symmetric positive semi - definite , as seen from ( [ eq : c_semi_definite ] ) , but also positive definite on @xmath147 . to this end , take @xmath148 , a vector that is piecewise constant on components , having value @xmath149 on the component @xmath137 of dimension @xmath150 for @xmath139 , and value @xmath151 for other components .
then @xmath152 implies @xmath153 .
indeed , every component @xmath137 of dimension @xmath63 has some component @xmath154 of dimension @xmath155 on its boundary , and therefore all @xmath149 have the same value , cf .
( [ eq : c_semi_definite ] ) .
this value is zero , since there is at least one component with natural boundary condition .
applying the congruence transformation , @xmath156 = \left [ \begin{matrix } i                  &    \\
\ol{b } a^{-1 } & i \end{matrix } \right ] \left [ \begin{matrix } a &    \\ & -\left ( \ol{b}a^{-1}\ol{b}^t + \ol{c } \right ) \end{matrix } \right ] \left [ \begin{matrix } i     & a^{-1 } \ol{b}^t \\ & i \end{matrix } \right].\ ] ] matrix @xmath2 is symmetric positive definite ( spd ) from ( [ eq : weak_term_a ] ) and therefore @xmath157 is spd on range @xmath3 , which is the orthogonal complement of the nullspace of @xmath5 .
thus @xmath158 is spd on whole @xmath159 . from the sylvester law of inertia ,
the number of positive , negative and zero eigenvalues is preserved by the congruence transformation .
since the block diagonal matrix on the right - hand side of  ( [ eq : congruence1 ] ) has only ( strictly ) positive and ( strictly ) negative eigenvalues , the matrix on the left - hand side also must be nonsingular , and problem  ( [ eq : hybrid - m - fractured ] ) has a unique solution .
example of a two - dimensional problem : even single fracture gives rise to two components of the two - dimensional mesh ( left ) ; example of three ( shaded ) triangles sharing a  single lagrange multiplier in 3d ( right).,title="fig:",scaledwidth=25.0% ]   example of a two - dimensional problem : even single fracture gives rise to two components of the two - dimensional mesh ( left ) ; example of three ( shaded ) triangles sharing a  single lagrange multiplier in 3d ( right).,title="fig:",scaledwidth=30.0% ]
for our purposes of combining meshes with different spatial dimensions , we define _ substructures _ as subsets of finite elements in the mesh rather than parts of a physical domain , cf .
@xcite .
to begin , let us define the combined triangulation @xmath160 as the union of triangulations for each spatial dimension , i.e. , @xmath161 .
the triangulation @xmath160 is subsequently divided into substructures@xmath162 .
note that in general a substructure can contain finite elements of different dimensions .
we define the interface @xmath163 as the set of degrees of freedom shared by more than one substructure .
note that @xmath164 for the current setting .
thus , let us split the lagrange multipliers@xmath165 into two subsets .
first , we denote by@xmath166 the set of multipliers corresponding to the interface@xmath167 .
the remaining multipliers , corresponding to substructure interiors , will be denoted by@xmath168 .
the interface@xmath169 of substructure@xmath170 is defined as a subset of@xmath167 corresponding to@xmath171 .
next , let @xmath172 be defined as the space of lagrange multipliers corresponding to  @xmath173 , @xmath174 , and define a space @xmath175 the substructure problems are obtained by assembling contributions of finite elements in each@xmath170,@xmath1{cccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it } & b_{\mathcal{f},\gamma}^{it}\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it } & -c_{\mathcal{f},\gamma } ^{it}\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i } & -\widetilde{c}_{{\gamma}i}^{it}\\ b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i } & -\widetilde{c}_{{\gamma}{\gamma}}^{i}\end{array } \right ]   \left [ \begin{array } [ c]{c}\mathbf{u}^{i}\\ p^{i}\\ \lambda_{i}^{i}\\ \lambda_{\gamma}^{i}\end{array } \right ]   = \left [ \begin{array } [ c]{c}g^{i}\\ f^{i}\\ 0\\ 0 \end{array } \right ]   , \qquad i=1,\dots , n_{s } , \label{eq : hybrid - fractured - substructure}\ ] ] where the blocks @xmath176 are block diagonal with blocks corresponding to finite element matrices , and the blocks @xmath177 only if the substructure @xmath178 contains coupling of elements of different dimensions .
we note that the global problem ( [ eq : hybrid - m - fractured ] ) could be obtained from ( [ eq : hybrid - fractured - substructure ] ) by further assembly at the interface .    in _ the iterative substructuring _
( see e.g.  @xcite ) , we first reduce the problem to substructure interfaces . in our context
we can eliminate normal fluxes , pressure unknowns , and lagrange multipliers at interiors of substructures , and we can define the substructure schur complements@xmath179 , @xmath180 , formally as @xmath181{ccc}b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i}\end{array } \right ]    \left [ \begin{array } [ c]{ccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it}\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it}\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i}\end{array } \right]^{-1 } \left [ \begin{array } [ c]{c}b_{\mathcal{f},\gamma}^{it}\\ -c_{\mathcal{f},\gamma}^{it}\\ -\widetilde{c}_{{\gamma}i}^{it}\end{array } \right ]   .
\label{eq : substructure - schur}\ ] ] however , in an implementation of a krylov subspace iterative method , we only need to compute the matrix - vector product @xmath182 for a given vector@xmath183 .
therefore , the matrix is not constructed explicitly , and the multiplication is obtained as follows .
[ alg : s - mult ] given @xmath184 , determine @xmath185 in the following two steps :    1 .
solve a local _ dirichlet problem _ @xmath1{ccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it}\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it}\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i}\end{array } \right ]   \left [ \begin{array } [ c]{c}\mathbf{w}^{i}\\ q^{i}\\ \mu_{i}^{i}\end{array } \right ]   = -\left [ \begin{array } [ c]{c}b_{\mathcal{f},\gamma}^{it}\\ -c_{\mathcal{f},\gamma}^{it}\\ -\widetilde{c}_{{\gamma}i}^{it}\end{array } \right ]   \lambda_{\gamma}^{i } , \label{eq : discrete - dirichlet - problem - substructure}\ ] ] 2 .
perform two sparse matrix - vector multiplications @xmath186{ccc}b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i}\end{array } \right ]   \left [ \begin{array } [ c]{c}\mathbf{w}^{i}\\ q^{i}\\ \mu_{i}^{i}\end{array } \right ]   \right ) .
\label{eq : substructure - schur - multiple}\ ] ]    next , let @xmath187 be the space of global degrees of freedom , such that the values of degrees of freedom from @xmath188 coincide on@xmath167 .
the vectors of the local substructure degrees of freedom @xmath184 and the vector of the global degrees of freedom @xmath189 are related by @xmath190 where @xmath191 are the restriction operators .
more specifically , each @xmath191 is a @xmath135-@xmath192 matrix such that every row contains one entry equal to one , and @xmath193 .
the global schur complement @xmath194 can be obtained as @xmath195 equation ( [ eq : schur - assembly ] ) represents the formal assembly of substructure schur complements into the global schur complement .
the global schur complement system , which we would like to solve iteratively , reads @xmath196 where vector @xmath197 is obtained from substructure reduced right - hand sides @xmath198{ccc}b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i}\end{array } \right ]   \left [ \begin{array } [ c]{ccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it}\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it}\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i}\end{array } \right ]   ^{-1}\left [ \begin{array } [ c]{c}g^{i}\\ f^{i}\\ 0 \end{array } \right ]   .\label{eq : reduced - rhs}\ ] ] in our implementation , we factor and store the matrices from  ( [ eq : discrete - dirichlet - problem - substructure ] ) .
the factors are then used to compute the vectors@xmath199 in  ( [ eq : reduced - rhs ] ) , and especially in algorithm  [ alg : s - mult ] , which is used in connection to formula  ( [ eq : schur - assembly ] ) to evaluate @xmath200 within each iteration of a krylov subspace iterative method .
this algorithm allows a straightforward parallel implementation . after an approximate solution of  ( [ eq : s_hat - system ] ) is found , solution in interiors of substructures , including all primal variables , is recovered from ( [ eq : hybrid - fractured - substructure ] ) using the factors from  ( [ eq : discrete - dirichlet - problem - substructure ] ) .
there are other ways to derive the interface problem  ( [ eq : s_hat - system ] ) .
the authors of  @xcite use a mixed - hybrid formulation with lagrange multipliers introduced only at the interface @xmath163 as their starting point . while problem  ( [ eq : s_hat - system ] ) is equivalent to the interface problems considered in  @xcite , the substructure problems therein have a different structure from  ( [ eq : hybrid - fractured - substructure ] ) .
in particular , there are no blocks corresponding to  @xmath201 , and the matrices corresponding to  @xmath176 are no longer block diagonal element - wise .
next , the authors of  @xcite construct explicit schur complement with respect to the whole block of lagrange multipliers @xmath58 and they show that , due to the special structure of  @xmath2 , the complement is both sparse and reasonably cheap to construct .
if this was performed substructure by substructure , this could be again seen as an intermediate step in obtaining problem  ( [ eq : s_hat - system ] ) by additional elimination of the interior lagrange multipliers  @xmath202 . however
, this would again lead to different substructure problems based on explicit local schur complements .
the following result allows an application of the bddc  method to problems with fractures .
[ th : sspd ] let natural boundary conditions ( [ eq : problem-3 ] ) be prescribed at a certain part of the boundary , i.e. @xmath122 for at least one @xmath123 .
then the matrix  @xmath203 in ( [ eq : s_hat - system ] ) is symmetric positive definite .
using the notation of  ( [ eq : blocks_matrix])([eq : congruence1 ] ) , let us introduce a matrix @xmath204 .
the matrix  @xmath205 is symmetric positive definite by theorem  [ th : regular ] ( and its proof ) .
applying another congruence transformation to @xmath205 and denoting the rows corresponding to the interface lagrange multipliers by subscript @xmath206 and the interior by  @xmath207 , we obtain @xmath208 = \left [ \begin{matrix } i                                           &    \\
\mathcal{s}_{\gamma i } \mathcal{s}_{ii}^{-1 } & i \end{matrix } \right ] \left [ \begin{matrix } \mathcal{s}_{ii } &    \\ & \mathcal{s}_{\gamma \gamma } - \mathcal{s}_{\gamma i } \mathcal{s}_{ii}^{-1 } \mathcal{s}_{\gamma i}^{t } \end{matrix } \right ] \left [ \begin{matrix } i     & \mathcal{s}_{ii}^{-1 } \mathcal{s}_{\gamma i}^{t }   \\ & i \end{matrix } \right].\ ] ] since the matrix on the left - hand side is spd , both diagonal blocks @xmath209 and @xmath210 are also symmetric positive definite from the sylvester law of inertia .
it remains to note that the schur complement @xmath203 in ( [ eq : s_hat - system ] ) is symmetric positive definite because @xmath211 .
theorem  [ th : sspd ] allows us to use the conjugate gradient method for the iterative solution of  ( [ eq : s_hat - system ] ) . in the next section ,
we describe the bddc method used as a preconditioner for @xmath203 .
in this section , we formulate the bddc method for the solution of  ( [ eq : s_hat - system ] ) . the algorithm can be viewed as a generalisation of  @xcite .
however , we follow the original description from  @xcite , which better reflects our implementation .    one step of bddc provides a two - level preconditioner for the conjugate gradient method applied to solving problem ( [ eq : s_hat - system ] ) .
it is characterised by the selection of certain _ coarse degrees of freedom _ based on primary degrees of freedom at interface @xmath163 .
the main coarse degrees of freedom in this paper are arithmetic averages over _ faces _ , defined as subsets of degrees of freedom shared by the same two substructures .
in addition , _ corner _ coarse degrees of freedom , defined as any selected lagrange multiplier at the interface , are used .
substructure _ edges
_ , defined as subsets of degrees of freedom shared by several substructures , may also appear ( see remark  [ rem : constraints ] ) .
the bddc method introduces _ constraints _ which enforce continuity of functions from @xmath212 at coarse degrees of freedom among substructures .
this gives raise to the space @xmath213 , which is given as the subspace of @xmath212 of functions which satisfy these continuity constraints .
in particular , @xmath214    [ rem : constraints ] in three spatial dimensions , several triangular elements can be connected at a single lagrange multiplier in a star - like configuration ( cf . fig .
[ fig : schemes ] ) .
a similar statement holds for line elements considered in two- or three - dimensional space .
this fact may lead to the presence of substructure _ edges _ and even _ vertices _ ( defined as degenerate edges consisting of a single degree of freedom ) , and we may prescribe also edge averages as constraints . as mentioned above , we also select _ corners _ as coarse degrees of freedom . while essentially any degree of freedom at the interface  @xmath163 can be _ a corner _ , we select them by the face - based algorithm  @xcite .
this algorithm considers all vertices as corners , and , in addition , it selects three geometrically well distributed degrees of freedom from the interface between two substructures sharing a face into the set of _
although considering corners is not the standard practice with @xmath8 finite elements , in our experience , corners improve convergence for numerically difficult problems , as can be observed for the engineering applications presented in section  [ sec : numerical ] .
we now proceed to the formulation of operators used in the bddc method .
the choice of constraints determines the construction of matrices  @xmath215 .
each row of@xmath216 defines one coarse degree of freedom at substructure  @xmath178 , e.g. , a  corner corresponds to a single @xmath192 entry at a row and an arithmetic average to several @xmath192 s at a row .
the _ coarse basis functions _
@xmath217 , one per each substructure coarse degree of freedom , are computed by augmenting the matrices from  ( [ eq : hybrid - fractured - substructure ] ) with  @xmath215 , and solving the augmented systems with multiple right - hand sides @xmath1{ccccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it } & b_{\mathcal{f},\gamma}^{it } & 0\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it } & -c_{\mathcal{f},\gamma } ^{it } & 0\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i } & -\widetilde{c}_{{\gamma}i}^{it } & 0\\ b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i } & -\widetilde{c}_{{\gamma}{\gamma}}^{i } & d^{it}\\ 0 & 0 & 0 & d^{i } & 0 \end{array } \right ]   \left [ \begin{array } [ c]{c}x^{i}\\ z^{i}\\ \phi_{i}^{i}\\ \phi_{\gamma}^{i}\\ l^{i}\end{array } \right ]   = \left [ \begin{array } [ c]{c}0\\ 0\\ 0\\ 0\\ i \end{array } \right ]   , \quad i=1,\dots , n_{s } , \label{eq : hybrid - fractured - augmented - substructure}\ ] ] where @xmath218 is the identity matrix , and @xmath219 , @xmath220 , and @xmath221 are auxiliary matrices not used any further . as shown in @xcite , the _ local coarse matrix _
@xmath222 is obtained as a side product of solving  ( [ eq : hybrid - fractured - augmented - substructure ] ) as @xmath223   \left [ \begin{array } [ c]{cccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it } & b_{\mathcal{f},\gamma}^{it}\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it } & -c_{\mathcal{f},\gamma } ^{it}\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i } & -\widetilde{c}_{{\gamma}i}^{it}\\ b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i } & -\widetilde{c}_{{\gamma}{\gamma}}^{i } \end{array } \right ]   \left [ \begin{array } [ c]{c}x^{i}\\ z^{i}\\ \phi_{i}^{i}\\ \phi_{\gamma}^{i}\end{array } \right ]   = -l^{i}. \label{eq : scci_def}\ ] ] let us define , similarly to  ( [ eq : r ] ) , operators@xmath224  that relate vectors of local coarse degrees of freedom@xmath225 to the vector of global coarse degrees of freedom@xmath226 as @xmath227 the _ global coarse matrix _
@xmath228 is then obtained by assembling the local contributions as @xmath229 finally , let us define the scaling operators @xmath230 which are given as diagonal matrices of weights that satisfy @xmath231 more details on the selection of diagonal entries in @xmath232 are given in section  [ sec : scaling ] .    with this selection of spaces and operators , we are ready to formulate the bddc preconditioner .
[ alg : bddc]the bddc preconditioner @xmath233 is defined in the following steps :    1 .
compute the local residuals @xmath234 2 .
compute the _ substructure corrections _ @xmath235 by solving the local _ neumann problems _
@xmath1{ccccc}a^{i } & b^{it } & b_{\mathcal{f},i}^{it } & b_{\mathcal{f},\gamma}^{it } & 0\\ b^{i } & -c^{i } & -c_{\mathcal{f},i}^{it } & -c_{\mathcal{f},\gamma } ^{it } & 0\\ b_{\mathcal{f},i}^{i } & -c_{\mathcal{f},i}^{i } & -\widetilde{c}_{ii}^{i } & -\widetilde{c}_{{\gamma}i}^{it } & 0\\ b_{\mathcal{f},\gamma}^{i } & -c_{\mathcal{f},\gamma}^{i } & -\widetilde{c}_{{\gamma}i}^{i } & -\widetilde{c}_{{\gamma}{\gamma}}^{i } & d^{it}\\ 0 & 0 & 0 & d^{i } & 0 \end{array } \right ]
\left [ \begin{array } [ c]{c}x^{i}\\ z^{i}\\ \eta_{i \delta } ^{i}\\ \eta_{\gamma \delta}^{i}\\ l^{i}\end{array } \right ]   = \left [ \begin{array } [ c]{c}0\\ 0\\ 0\\ r_{\gamma}^{i}\\ 0 \end{array } \right ]   , \quad i=1,\dots , n_{s}.\label{eq : bddc - substructure - problem}\ ] ] 3 .
compute the _ coarse correction _
@xmath236 by collecting the coarse residual @xmath237  solving the global _ coarse problem _ @xmath238 and distributing the coarse correction@xmath239 4 .   combine and
average the corrections@xmath240    we note that the factorisations of the matrices from ( [ eq : hybrid - fractured - augmented - substructure ] ) are also used for each solution of ( [ eq : bddc - substructure - problem ] ) .    in order to apply the existing bddc theory for elliptic problems ( e.g. @xcite ) to the proposed preconditioner
, we introduce some additional notation and make a few observations .
due to  ( [ eq : substructure - schur ] ) , the substructure corrections in  ( [ eq : bddc - substructure - problem ] ) can be written equivalently as @xmath1{cc}-s^{i } & d^{it}\\ d^{i } & 0 \end{array } \right ]   \left [ \begin{array } [ c]{c}\eta_{\gamma \delta}^{i}\\ l^{i}\end{array } \right ]   = \left [ \begin{array } [ c]{c}r_{\gamma}^{i}\\ 0 \end{array } \right ] , \label{eq : bddc - substructure - problem - with - schur}\ ] ] and the construction of coarse basis functions  @xmath217 in ( [ eq : hybrid - fractured - augmented - substructure ] ) as @xmath1{cc}-s^{i } & d^{it}\\ d^{i } & 0 \end{array } \right ]   \left [ \begin{array } [ c]{c } \phi_{\gamma}^{i}\\ l^{i}\end{array } \right ]   = \left [ \begin{array } [ c]{c}0\\ i \end{array } \right ] .
\label{eq : bddc - substructure - problem - with - schur - for - coarse - basis}\ ] ]    next , let us formally write the operators and vectors in the block form @xmath241{c}\lambda_{\gamma}^{1}\\ \vdots\\ \lambda_{\gamma}^{n_{s}}\end{array } \right ]   , \
r=\left [ \begin{array } [ c]{c}r^{1}\\ \vdots\\ r^{n_{s}}\end{array } \right ]   , \ w=\left
[ \begin{array } [ c]{ccc}w^{1 } &   & \\ & \ddots & \\ &   & w^{n_{s}}\end{array } \right ]   , \
s=\left [ \begin{array } [ c]{ccc}s^{1 } &   & \\ & \ddots & \\ &   & s^{n_{s}}\end{array } \right ]   .\ ] ]    by grouping the steps of algorithm  [ alg : bddc ] and using ( [ eq : bddc - substructure - problem - with - schur ] ) , the operator of the bddc preconditioner can be formally written as @xmath242 where @xmath243{cc}i_{{\lambda}_{\gamma}^{i } } & 0   \end{array } \right ]    \left [ \begin{array } [ c]{cc}-s^{i } & d^{it}\\ d^{i } & 0 \end{array } \right]^{-1 }   \left [ \begin{array } [ c]{c}i_{{\lambda}_{\gamma}^{i } } \\   0   \end{array } \right ]    \right )    + \left ( \sum_{i=1}^{n_{s}}r_{c}^{it}\phi_{\gamma}^{it }   \right)^t s_{cc}^{-1 } \left ( \sum_{i=1}^{n_{s}}r_{c}^{it}\phi_{\gamma}^{it } \right ) \right\}.\ ] ] the first term in @xmath244 corresponds to substructure corrections and the second term to the coarse correction ( steps 2 and 3 of algorithm  [ alg : bddc ] ) , and @xmath245 is the identity matrix in @xmath246 . from ( [ eq : mbddc - formal ] ) and ( [ eq : stilde ] ) , one can readily see that @xmath247 is symmetric .
[ as : nulls]let us assume that @xmath248    in order to satisfy assumption  [ as : nulls ] , we must prescribe enough coarse degrees of freedom as constraints along with robin boundary conditions ( [ eq : newton-1])([eq : newton-2 ] ) and ( [ eq : newton-3 ] ) at each fracture within substructure@xmath170 . since constrains in @xmath215 are linearly independent , @xmath249 has full column rank .
in particular , assumption  [ as : nulls ] is satisfied when arithmetic averages are used on each substructure face ( and eventually edge ) as constraints .
[ lemma : positive_definite_preconditioner ] the operator @xmath244 in preconditioner @xmath247 is symmetric and positive definite on the space @xmath213 .    the space @xmath213 is decomposed into the substructure spaces and the coarse space , @xmath250 to achieve this splitting , each local space @xmath246 is decomposed into subspaces @xmath251 corresponding to the _ substructure space _ @xmath252 , and the _ coarse space _ on substructure @xmath178 , @xmath253 , respectively .
to analyse this decomposition , let us recall that @xmath254 is a positive semi - definite matrix and write ( [ eq : bddc - substructure - problem - with - schur - for - coarse - basis ] ) in detail as @xmath255 from ( [ eq : bddc - substructure - problem - with - schur - for - coarse - basis-1 ] ) , @xmath256 which in turn , similarly to ( * ? ? ?
* lemma  8) , gives for any @xmath257 and @xmath258 @xmath259 from ( [ eq : bddc - substructure - problem - with - schur - for - coarse - basis-2 ] ) , the matrix @xmath217 has full column rank , and @xmath260 finally , from assumption  [ as : nulls ] and ( [ eq : decomposition_of_subdomain_space ] ) , @xmath261    decomposition of the subdomain space ( [ eq : decomposition_of_subdomain_space ] ) implies decomposition of a function @xmath262 to @xmath263 , where @xmath264 , @xmath265 , and @xmath266 by ( [ eq : s - orthogonality2 ] ) .
let us first analyse the substructure corrections .
following ( * ? ? ?
* section  3.3 ) , the matrix of ( [ eq : bddc - substructure - problem - with - schur ] ) is invertible due to assumption  [ as : nulls ] .
if we define , in addition , a matrix @xmath267 with orthonormal columns forming a basis of @xmath268 , i.e. @xmath269 we have @xmath1{cc}i_{{\lambda}_{\gamma}^{i } } & 0   \end{array } \right ]    \left [ \begin{array } [ c]{cc}-s^{i } & d^{it}\\ d^{i } & 0 \end{array } \right]^{-1 }   \left [ \begin{array } [ c]{c}i_{{\lambda}_{\gamma}^{i } } \\   0   \end{array } \right ]    =   -q^{i}\left(q^{it } s^{i } q^{i}\right)^{-1}q^{it}.\ ] ] the matrix @xmath270 is symmetric positive definite , and consequently , for any @xmath262 , @xmath271 with equality if @xmath272 .    next ,
let us turn towards the coarse correction .
formula  ( [ eq : scci_def ] ) for@xmath273 can be written equivalently as @xmath274 since the term on the right - hand side is just a ( negative ) galerkin projection of the positive semi - definite matrix @xmath254 , matrix @xmath222 is symmetric negative semi - definite .
if at least one substructure is equipped with natural boundary conditions , the matrix @xmath228 assembled by ( [ eq : scc_def ] ) becomes symmetric negative definite , and so is @xmath275 .
we have just verified the negative definiteness of the principal parts of @xmath244 , and the desired positive definiteness is obtained through the change of sign in front of the braces in  ( [ eq : stilde ] ) .    in view of lemma  [ lemma : positive_definite_preconditioner ]
, the standard condition number bound follows from ( * ? ? ?
* lemma  2 ) .
[ th : bddc_bound ] the condition number  @xmath276 of the preconditioned operator  @xmath277 satisfies @xmath278    the norms in ( [ eq : cond - bound - algebraic ] ) are induced by the matrix @xmath279 defined in ( [ eq : block_operators ] ) for all functions @xmath280 .    in addition
, in the case of a single mesh dimension in either 2d or 3d , and under the assumption of substructure - wise constant hydraulic conductivities , it has been also derived in  ( * ? ? ? * lemma  5.5 and theorem  6.1 ) that the condition number bound@xmath281 satisfies @xmath282 where @xmath283 is the characteristic size of geometric substructures .
we note that the bound  ( [ eq : cond - bound - analytic ] ) implies that , for a fixed relative subdomain size  @xmath284 , the condition number is independent of the problem size .
it is worth emphasising that theorem  [ th : bddc_bound ] is also valid for combined mesh dimensions .
however , several simplifications are employed in @xcite to obtain ( [ eq : cond - bound - analytic ] ) , which are not satisfied in the set - up considered in this paper . in particular
( i ) hydraulic conductivity coefficient here is , in general , not substructure - wise constant nor isotropic , ( ii ) it is not clear whether , in presence of fractures , the interpolation operator onto a  conforming mesh introduced in @xcite can be constructed and bounded in the @xmath285 norm .
let us now discuss the choice of entries in the diagonal weight matrices @xmath232 .
these matrices play an important role in the bddc method , both in the theory , cf .
theorem  [ th : bddc_bound ] or @xcite , and in the computations , cf .
three possible choices are also studied numerically in section  [ sec : results_engineering ] . the basic choice is presented by the _
arithmetic average _ taken from values at the neighbouring substructures . in this simplest construction ,
the entry corresponding to lagrange multiplier @xmath286 is given by the inverse counting function as @xmath287 where @xmath288 is the number of substructures in the set @xmath289 of indices of substructures to which @xmath286 belongs .
for 2d or 3d meshes without fractures , @xmath290 for raviart - thomas elements . however
, since several two - dimensional fractures can meet in our setting , smaller weights can occasionally appear at such regions .    while arithmetic average is sufficient for problems with homogeneous coefficients , it is well known that for problems with large variations in material properties along the interface , it is necessary to incorporate their values into the ( weighted ) average to obtain a robust method .
this gives rise to the  @xmath291-scaling , for which @xmath292 where @xmath293 is a material characteristic for substructure  @xmath294 .
this choice is robust with respect to jumps in coefficients across the interface , cf .
@xcite ; however , coefficients are assumed constant for each substructure .
this requirement is very restrictive for practical computations with quickly varying coefficients , and we employ a generalisation which takes into account the material coefficient of the element to which the lagrange multiplier @xmath286 corresponds . in our case , we use @xmath295 , where @xmath123 is the dimension of the element @xmath59 .
this value can be seen as a representative hydraulic conductivity on the element .
finally , we propose a modification of the popular scaling by diagonal stiffness  @xcite . in the usual diagonal stiffness approach ,
the optimal weight , which is the diagonal entry of the schur complement , is approximated by the diagonal entry of the original substructure matrix .
however , this is not directly applicable to the indefinite system ( [ eq : hybrid - fractured - substructure ] ) , as , in general , matrix @xmath296 contains only seldom nonzeros on the diagonal .
for this reason , we approximate the diagonal of the schur complement as @xmath297 where the index @xmath98 corresponds to the row in block @xmath176 of the element face to which the lagrange multiplier  @xmath286 belongs .
using the diagonal stiffness scaling in connection with the standard lagrange finite elements may lead to poor convergence for problems with rough interface @xcite , for which the diagonal stiffness can vary quickly even for smooth problems with constant coefficients on uniform meshes .
this is a severe issue for practical computations , in which graph partitioners are typically used for creating substructures .
however , this issue is not as pronounced for raviart - thomas elements , for which only one element contributes to the stiffness on the diagonal at an interface degree of freedom , and thus irregularities caused by changing number of elements contributing to an interface weight can not occur . on the other hand ,
an advantage of the diagonal stiffness scaling is the fact , that  unlike the @xmath291-scaling  it takes into account the shape and relative sizes of elements , which vary considerably in engineering applications , as well as the effect of @xmath115 introduced in  ( [ eq : weak_term_a ] ) and  ( [ eq : weak_term_f ] ) . unless stated otherwise , scaling  ( [ eq : scaling_diagonal ] ) is used in the computations presented in section  [ sec : numerical ] .
the basis for an efficient parallel implementation of the method described in previous sections was obtained by combining two existing open - source software packages : the finite element package _ _ flow123d _ _ ( version 1.6.5 ) for underground fluid flow simulations and the bddc - based solver _ _ bddcml _ _ ( version 2.0 ) used for the solution of the resulting system of equations . however
, minor changes have been made to both codes to support the specific features , such as the weights ( [ eq : scaling_rho ] ) and  ( [ eq : scaling_diagonal ] ) .
the _ flow123d _ package has been developed for modelling complex behaviour of underground water flow and pollution transport .
however , only the simple flow in a fully saturated porous media described by darcy s law is considered in this paper . to accurately account for fractures in the medium , such as granite rock ,
the solver allows us to combine finite elements of different dimensions : the three - dimensional elements of porous media are combined with two - dimensional elements modelling planar fractures , which may be in turn connected in one - dimensional elements for channels .
raviart - thomas elements are consistently used throughout such discretisation .
although the fractures are also modelled as porous media , their hydraulic conductivity is by orders of magnitude higher than that of the main porous material of the domain .
in addition , the finite element discretisations are typically not uniform within the domain , and the relative sizes of elements may also vary by orders of magnitude .
both these aspects give rise to very poorly conditioned linear systems , which are very challenging for iterative solvers .
the _ flow123d _
solver has been developed for over 10 years and it is written in c / c++ programming language with object - oriented design and parallelism through mpi .
the _ bddcml _ is a library for solving algebraic systems of linear equations by means of the bddc method .
the package supports the adaptive - multilevel bddc method @xcite suitable for very high number of substructures and computer cores , although we only use the standard ( non - adaptive two - level ) bddc method from @xcite for the purpose of this paper .
the _ bddcml _
library is typically interfaced by finite element packages , which may provide the division into substructures .
this feature is used in our current implementation , in which the division into non - overlapping substructures is constructed within the _ flow123d _ using the _ metis _ ( version 5.0 ) package @xcite . one substructure is assigned to a processor core in the current set - up of the parallel solver , although _ bddcml _ is more flexible in this respect .
the library performs the selection of additional corners by the face - based algorithm from @xcite .
the _ bddcml _
package is written in fortran 95 and parallelised through mpi .
the _ bddcml _ solver relies on a serial instance of the _ mumps _ direct solver @xcite for the solution of each local discrete dirichlet problem ( [ eq : discrete - dirichlet - problem - substructure ] ) as well as for the solution of each local discrete neumann problem  ( [ eq : bddc - substructure - problem ] ) .
the coarse problem  ( [ eq : bddc - coarse - problem ] ) is solved by a parallel instance of _ mumps_. the main difference from using _
bddcml _ for symmetric positive definite problems is the need to use the @xmath298 factorisation of general symmetric matrices for problems ( [ eq : discrete - dirichlet - problem - substructure ] ) , which are saddle - point ( i.e. indefinite ) systems in the present setting .
although the original system ( [ eq : hybrid - m - fractured ] ) is indefinite , system ( [ eq : s_hat - system ] ) is symmetric positive definite , which allows the use of the preconditioned conjugate gradient ( pcg ) method .
one step of bddc is used as the preconditioner within the pcg method applied to problem ( [ eq : s_hat - system ] ) .
the matrix of problem ( [ eq : s_hat - system ] ) is not explicitly constructed in the solver , and only its actions on vectors are computed following ( [ eq : discrete - dirichlet - problem - substructure])([eq : schur - assembly ] ) .    in our implementation , we change the sign neither in the action of @xmath299 ( [ eq : substructure - schur - multiple ] ) nor in the action of the preconditioner @xmath247 ( [ eq : bddc - preconditioner ] ) .
since both are then strictly negative definite , the product @xmath277 is the same as if both signs were changed , and the pcg method runs correctly . in this way ,
no changes are necessary in an implementation developed for symmetric positive definite problems .
in this section , we investigate the performance of the algorithm and its parallel implementation on several benchmark problems in 2d and 3d , and on two geoengineering problems of existing localities in 3d .
for the two benchmark problems without fractures we perform weak scaling tests . for the benchmark problem with fractures and for the geoengineering problems ,
we perform strong scaling tests with the problem size fixed and increasing number of processor cores . in all cases ,
the pcg iterations are run until the relative norm of residual @xmath300 .
if not stated otherwise , the proposed scaling by diagonal stiffness ( [ eq : scaling_diagonal ] ) is used within the averaging operator of bddc .
first , the performance of the solver is investigated on a unit square and a unit cube discretised solely using two - dimensional and three - dimensional finite elements , respectively .
for this reason , block @xmath4 in system  ( [ eq : hybrid - m - fractured ] ) , which is related to combining elements of different dimension , is zero , and the problem reduces to the standard problem  ( [ eq : hybrid - m ] ) .
the sequence of unstructured meshes is approximately uniform for both problems , and the problems do not contain any jumps in material coefficients . in figs .
[ fig : square2d_problem ] and [ fig : cube3d_problem ] , example meshes and the resulting pressure head and velocity fields are presented . while gravity is present in the 3d case , its effect is not considered in the 2d case .
the results of the weak scaling tests are summarised in tables  [ tab:2d_square_weak_scaling ] and [ tab:3d_cube_100k_weak_scaling ] . to give a better view ,
the resulting solution times for different problem sizes are also visualised in fig .
[ fig : timing_100k ] . in these tables
, @xmath301 denotes the number of substructures and processors , @xmath302 is the size of the global problem ( [ eq : hybrid - m - fractured ] ) , @xmath303 is the size of the interface problem ( [ eq : s_hat - system ] ) , @xmath304 denotes number of faces , @xmath305 denotes number of corners , ` its . ' stands for resulting number of pcg iterations , and ` cond . '
is the approximate condition number computed from the lanczos sequence in pcg .
we report separately the time spent in preconditioner set - up , the time spent by pcg iterations , and the total time for the whole solve .    in these weak scaling tests ,
the number of unknowns per core is kept approximately constant around @xmath306 .
these weak scaling tests were performed using up to 64 cores of the sgi altix uv supercomputer at the supercomputing centre of the czech technical university in prague .
the computer contains twelve intel xeon processors , each with six cores at frequency 2.67 ghz .
intel compilers version 12.0 were used .
the numbers of pcg iterations and condition number estimates in tables  [ tab:2d_square_weak_scaling ] and [ tab:3d_cube_100k_weak_scaling ] confirm the expected numerical scalability of the bddc method , which is well known for symmetric positive definite problems as well as for darcy s flow problems @xcite .
the slight irregularities in the condition number in table  [ tab:3d_cube_100k_weak_scaling ] are probably caused by using non - nested unstructured meshes .
looking at times in these tables and in fig .
[ fig : timing_100k ] , we can see almost optimal scaling , with only mild growth of times with number of cores .
the numbers of pcg iterations are higher in 3d , and the time spent in pcg iterations grows proportionally , while the time spent in the set - up phase does not differ considerably between two - dimensional and three - dimensional setting and dominates the overall time .
example of solution to the model square problem containing only 2d elements , plot of pressure head with mesh ( left ) and velocity vectors ( right).,title="fig:",scaledwidth=47.0% ]   example of solution to the model square problem containing only 2d elements , plot of pressure head with mesh ( left ) and velocity vectors ( right).,title="fig:",scaledwidth=50.0% ]     example of solution to the model cube problem containing only 3d elements , plot of pressure head with mesh ( left ) and velocity vectors ( right).,title="fig:",scaledwidth=47.0% ]   example of solution to the model cube problem containing only 3d elements , plot of pressure head with mesh ( left ) and velocity vectors ( right).,title="fig:",scaledwidth=50.0% ]    [ c]|ccc|ccc|cc|ccc| & & & & & & & & + & & & & & & & & set - up & pcg & solve + 2 & 207k & 103k & 155 & 1 & 2 & 7 & 1.37 & 8.3 & 1.6 & 9.9 + 4 & 440k & 110k & 491 & 5 & 10 & 8 & 1.60 & 12.2 & 2.2 & 14.4 + 8 & 822k & 103k & 1.2k & 13 & 26 & 9 & 1.78 & 11.0 & 2.5 & 13.5 + 16 & 1.8 m & 111k & 2.8k & 33 & 66 & 8 & 1.79 & 14.3 & 2.7 & 17.0 + 32 & 3.3 m & 104k & 5.9k & 74 & 148 & 9 & 1.79 & 12.1 & 3.3 & 15.4 + 64 & 7.2 m & 113k & 13.0k & 166 & 332 & 9 & 1.85 & 14.8 & 4.4 & 19.2 +    [ c]|ccc|ccc|cc|ccc| & & & & & & & & + & & & & & & & & set - up & pcg & solve + 2 & 217k & 108k & 884 & 1 & 3 & 11 & 2.88 & 11.7 & 2.3 & 14.0 + 4 & 437k & 109k & 2.3k & 6 & 18 & 12 & 3.04 & 11.7 & 2.5 & 14.2 + 8 & 945k & 118k & 5.7k & 21 & 63 & 15 & 12.00 & 15.4 & 4.0 & 19.3 + 16 & 1.6 m & 103k & 12.8k & 56 & 168 & 16 & 6.58 & 12.9 & 4.0 & 17.0 + 32 & 3.4 m & 106k & 29.8k & 132 & 401 & 18 & 10.10 & 15.4 & 5.2 & 20.6 + 64 & 6.1 m & 95k & 59.6k & 307 & 931 & 19 & 16.58 & 13.7 & 6.3 & 20.0 +     weak scaling test for the 2d square problem ( left ) , and the 3d cube problem ( right ) , approx .
100k unknowns per core .
computational times separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]   weak scaling test for the 2d square problem ( left ) , and the 3d cube problem ( right ) , approx .
100k unknowns per core .
computational times separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]    the next benchmark problem is considerably more complicated .
it consists again of a unit cube , which now contains four planar fractures aligned with diagonals of a 2d cross - section .
these four planar fractures meet at a 1d channel in the centre of the cross - section .
therefore , the problem contains the full possible combination of 3d , 2d and 1d finite elements .
the tensor @xmath22 is isotropic , thus it is just a scalar multiple of identity . the corresponding scalar value is set to 10 , 1 , and 0.1 for 1d , 2d , and 3d elements , respectively .    we perform a strong scaling test with this problem , keeping the mesh size fixed with approximately 2.1 million elements and 14.6 million degrees of freedom . in fig .
[ fig : cube123d_problem ] , the computational mesh and the resulting pressure head and velocity fields are presented .
this scaling test was computed on the cray xe6 supercomputer _ hector _ at the edinburgh parallel computing centre .
this supercomputer is composed of 2816 nodes , each containing two amd opteron interlagos processors with 16 cores at 2.3 ghz .
gnu compilers version 4.6 were employed .
results of the strong scaling test are summarised in table  [ tab:123d_cube_strong_scaling ] , and the computing times are visualised also in fig .
[ fig : timing_strong_123d ] together with the parallel speed - up .
the reference value for computing speed - up is the time on 16 cores , and the speed - up on @xmath307 processors is computed as @xmath308 where @xmath309 is the time on @xmath307 processors .
we can see that the number of pcg iterations grows with the number of substructures for this problem which is also confirmed by the growing condition number estimate . while the time spent in set - up phase scales very well , the time spent in pcg grows together with the number of iterations .
the reason for this growth seem to be related to the larger interface , at which more numerical difficulties appear .
this seems to be related to more 1d-2d and 2d-3d connections at the interface and makes this difficult problem a good candidate for using the _ adaptive bddc _
. however , this will be the subject of a separate study .
example of solution to the model cube problem containing 1d , 2d , and 3d elements , plot of pressure head with mesh and fractures ( left ) and velocity vectors ( right).,title="fig:",scaledwidth=47.0% ]   example of solution to the model cube problem containing 1d , 2d , and 3d elements , plot of pressure head with mesh and fractures ( left ) and velocity vectors ( right).,title="fig:",scaledwidth=50.0% ]    [ c]|cc|ccc|cc|ccc| & & & & & & & + & & & & & & & set - up & pcg & solve + 16 & 912k & 47k & 53 & 159 & 26 & 59.3 & 171.6 & 84.5 & 256.2 + 32 & 456k & 65k & 126 & 380 & 48 & 2091.0 & 90.1 & 109.8 & 200.0 + 64 & 228k & 86k & 301 & 914 & 81 & 1436.1 & 36.8 & 77.1 & 114.0 + 128 & 114k & 116k & 689 & 2076 & 109 & 2635.8 & 14.3 & 43.1 & 57.4 + 256 & 57k & 151k & 1436 & 4365 & 164 & 1700.5 & 6.7 & 31.2 & 38.0 + 512 & 28k & 196k & 3021 & 9244 & 254 & 42614.5 & 4.0 & 26.9 & 30.9 +     strong scaling test for the cube problem with 1d , 2d , and 3d elements and 14.6 m unknowns , computational time ( left ) and speed - up ( right ) separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]   strong scaling test for the cube problem with 1d , 2d , and 3d elements and 14.6 m unknowns , computational time ( left ) and speed - up ( right ) separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]      the performance of the algorithm and its parallel implementation has been investigated on two engineering problems of underground flows within real geologic locations . for both problems ,
the porous medium is fractured granite rock , with the fractures modelled by two - dimensional elements .
the first problem is the _ melechov locality _ , which models one of the candidate sites for a nuclear waste deposit to be build within the czech republic in future .
the goal is to model the underground flow and estimate the speed at which an eventual radioactive pollution would spread .
the computational mesh contains 2.1 million finite elements resulting in 15 million unknowns .
the geometry of the problem with the resulting distribution of piezometric head and the finite element mesh is presented in fig .
[ fig : melechov_1 ] .
the problem contains vertical two - dimensional fractures visualised in fig .
[ fig : melechov_2 ] .
the maximal hydraulic conductivity within the fractures is 6.3@xmath31010@xmath311 ms@xmath81 , while the minimal conductivity of the outer material is 6.0@xmath31010@xmath312 ms@xmath81 , the transition coefficient @xmath313 = 1 s@xmath81 , and the effective thickness of fractures @xmath85 = 0.1 m.    we perform a strong scaling test for this problem , keeping the problem size fixed and increasing the number of substructures and computing cores .
an example of division into 64 substructures is presented in fig .
[ fig : melechov_2 ] .
the scaling test was computed on the _ hector _ supercomputer .
table  [ tab : melechov_large_strong_scaling ] summarises the results of this test .
we can still see some growth of the number of iterations with the number of substructures , which is however much milder than the growth observed for the unit cube with fractures in table  [ tab:123d_cube_strong_scaling ] .
correspondingly , the times reported in table  [ tab : melechov_large_strong_scaling ] and visualised in fig .  [ fig : timing_melechov_large ] show an optimal scaling of the solver over a large range of core counts .
graphics / melechov_image1_piezo ( 9,5)@xmath314 ( 9,9)@xmath315 ( 6,11.5)@xmath316 ( -10,18)@xmath3179940 m ( 25,45)@xmath31812100 m ( 87,35)@xmath3191410 m    the problem of the _ melechov locality _ ; the system of fractures ( left ) and an example division into 64 substructures ( right).,title="fig:",scaledwidth=40.0% ] the problem of the _ melechov locality _ ; the system of fractures ( left ) and an example division into 64 substructures ( right).,title="fig:",scaledwidth=40.0% ]    [ c]|cc|ccc|cc|ccc| & & & & & & & + & & & & & & & set - up & pcg & solve + 16 & 934k & 36k & 32 & 96 & 40 & 53.0 & 131.4 & 144.1 & 275.6 + 32 & 467k & 54k & 76 & 228 & 70 & 878.3 & 47.5 & 112.9 & 160.4 + 64 & 233k & 82k & 186 & 561 & 67 & 202.4 & 17.4 & 50.2 & 67.7 + 128 & 117k & 116k & 528 & 1592 & 69 & 237.6 & 7.9 & 23.1 & 31.1 + 256 & 58k & 155k & 1235 & 3747 & 96 & 5577.0 & 4.0 & 14.7 & 18.8 + 512 & 29k & 207k & 2699 & 8256 & 106 & 1658.1 & 2.2 & 8.3 & 10.5 + 1024 & 15k & 271k & 5711 & 17581 & 119 & 11554.5 & 2.1 & 7.0 & 9.2 +     strong scaling test for the problem of the _ melechov locality _
containing 2d and 3d elements and 15 m unknowns , computational time ( left ) and speed - up ( right ) separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]   strong scaling test for the problem of the _ melechov locality _ containing 2d and 3d elements and 15 m unknowns , computational time ( left ) and speed - up ( right ) separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]    the second engineering model is the locality around the _
bedichov tunnel_. the main purpose of this 2.1 km long tunnel near the city of liberec in the north of the czech republic to accommodate water pipes , which supply the city by drinking water from a reservoir in the mountains .
however , this locality is also a valuable site for experimental geological measurements performed inside the tunnel .
the model aims at describing the flow in the granite rock surrounding the tunnel .
the computational mesh consists of 1.1 million elements leading to 7.8 million unknowns .
the mesh with the plot of resulting piezometric head is presented in fig .
[ fig : tunel_1 ] .
the system of fractures and an example division into 256 substructures are visualised in fig .
[ fig : tunel_2 ] .
the hydraulic conductivity of the fractures is 10@xmath320 ms@xmath81 , while the conductivity of the outer material is 10@xmath321 ms@xmath81 , the transition coefficient @xmath313 = 1 s@xmath81 , and the effective thickness of fractures @xmath85 = 1.1 m.    although the mesh contains fewer finite elements than the one of the _ melechov locality _ model , this problem is considerably more complicated .
this is caused mainly by the presence of relatively very small and irregularly shaped finite elements in the vicinity of the tunnel and near the cross - sections of fractures ( see fig .
[ fig : tunel_3 ] ) generated by the mesh generator .
the results of a strong scaling test are summarised in table  [ tab : tunnel_strong_scaling ] . as before , the times are also plotted in fig .
[ fig : timing_tunnel ] .
although the number of iterations is not independent of the number of substructures , the growth is still small . consequently , the computing times , and especially the time for set - up , scale very well over a large range of numbers of substructures .
the observed super - optimal scaling may be related to faster factorisation of the smaller local problems by the direct solver for indefinite matrices .
table  [ tab : tunnel_corner_effect ] summarises an experiment performed to analyse the effect of using corners in the construction of the coarse space in bddc . as has been mentioned in section  [ sec : bddc ] , using raviart - thomas finite elements does not lead to ` natural ' corners as cross - points shared by several substructures . on the other hand ,
the notion of corners was generalised to any selected interface degree of freedom , at which continuity of functions from the coarse space is required .
such generalisation is important for the well - posedness of the local problems for unstructured meshes e.g. in elasticity analysis @xcite .
this is also the default option for _ bddcml _ , in which selection of corners is performed at each face between two substructures .
adding corners improves the approximation properties of the coarse space at the cost of increasing the size of the coarse problem .
table  [ tab : tunnel_corner_effect ] compares the convergence for variable number of substructures without and with constraints at corners .
column ` with corners ' in table  [ tab : tunnel_corner_effect ] corresponds to results in table  [ tab : tunnel_strong_scaling ] , and it is repeated for comparison .
we can see that while the effect of corners on convergence is small for smaller number of substructures , the improvement of the coarse problem and the approximation of bddc becomes more significant for higher numbers of cores . looking at times in table  [ tab : tunnel_corner_effect ] , the additional time spent in the set - up phase due to higher number of constraints when using corners is compensated by the lower number of pcg iterations , resulting in lower overall times .
thus , using additionally selected corners appears beneficial for complicated engineering problems like this one .
graphics / tunel_image1_piezo ( 2,12)1754 m ( width ) ( 16,57)2655 m ( length ) ( 98,43)458 m ( height )     the _ bedichov tunnel _
problem ; the system of fractures ( left ) and an example division into 256 substructures ( right).,title="fig:",scaledwidth=48.0% ]   the _ bedichov tunnel _ problem ; the system of fractures ( left ) and an example division into 256 substructures ( right).,title="fig:",scaledwidth=48.0% ]     example of difficulties in the mesh of the _ bedichov tunnel _ problem ; detail of the tunnel geometry with fine elements ( left ) and enforced refinement at an intersection of fractures ( right).,title="fig:",scaledwidth=45.5% ]    example of difficulties in the mesh of the _ bedichov tunnel _
problem ; detail of the tunnel geometry with fine elements ( left ) and enforced refinement at an intersection of fractures ( right).,title="fig:",scaledwidth=48.0% ]    [ c]|cc|ccc|cc|ccc| & & & & & & & + & & & & & & & set - up & pcg & solve + 32 & 245k & 20k & 106 & 322 & 112 & 1514.1 & 110.3 & 144.0 & 254.3 + 64 & 123k & 28k & 192 & 597 & 63 & 117.7 & 42.2 & 36.0 & 78.3 + 128 & 61k & 45k & 413 & 1293 & 75 & 194.4 & 13.4 & 16.8 & 30.3 + 256 & 31k & 72k & 902 & 2791 & 119 & 526.7 & 4.2 & 10.9 & 15.1 + 512 & 15k & 110k & 2009 & 6347 & 137 & 1143.4 & 1.8 & 7.1 & 9.0 + 1024 & 8k & 155k & 4575 & 14725 & 173 & 897.0 & 1.6 & 8.0 & 9.7 +     strong scaling test for the problem of the _ bedichov tunnel _ containing 2d and 3d elements and 7.8 m unknowns , computational time ( left ) and speed - up ( right ) separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]   strong scaling test for the problem of the _ bedichov tunnel _ containing 2d and 3d elements and 7.8 m unknowns , computational time ( left ) and speed - up ( right ) separately for set - up and pcg phases , and their sum ( solve).,title="fig:",scaledwidth=48.0% ]    [ c]|cc|ccccc|ccccc| & & & + & & & & & & & + & & & & set - up & pcg & solve & & & set - up & pcg & solve + 32 & 245k & 131 & 1789.6 & 107.5 & 175.0 & 282.5 & 112 & 1514.1 & 110.3 & 144.0 & 254.3 + 64 & 123k & 70 & 122.2 & 40.3 & 40.4 & 80.7 & 63 & 117.7 & 42.2 & 36.0 & 78.3 + 128 & 61k & 96 & 208.5 & 10.9 & 21.6 & 32.6 & 75 & 194.4 & 13.4 & 16.8 & 30.3 + 256 & 31k & 139 & 541.9 & 3.7 & 12.5 & 16.2 & 119 & 526.7 & 4.2 & 10.9 & 15.1 + 512 & 15k & 197 & 1418.9 & 1.4 & 10.0 & 11.4 & 137 & 1143.4 & 1.8 & 7.1 & 9.0 + 1024 & 8k & 312 & 3779.4 & 1.0 & 14.5 & 15.6 & 173 & 897.0 & 1.6 & 8.0 & 9.7 +    in the final experiment
, we compare the effect of different averaging techniques on the convergence of bddc . in table
[ tab : tunnel_averaging ] , results of the strong scaling test for arithmetic averaging  ( [ eq : scaling_arithmetic ] ) , the modified @xmath291-scaling  ( [ eq : scaling_rho ] ) , and the proposed scaling by diagonal stiffness  ( [ eq : scaling_diagonal ] ) are summarised .
the final column corresponds to the results from table  [ tab : tunnel_strong_scaling ] , which are repeated here for comparison .
table  [ tab : tunnel_averaging ] suggests , that while the simple arithmetic averaging does not lead to satisfactory convergence for this problem , the modified @xmath291-scaling and the diagonal scaling mostly lead to similar convergence .
however , while the former provides slightly better convergence for several cases , it also leads to irregularities for certain divisions , for which the bddc method with this averaging converges rather poorly .
therefore , the proposed scaling ( [ eq : scaling_diagonal ] ) can be recommended as the most robust choice among the three tested options .
[ c]|cc|ccc|cc|cc|cc| & & & & & & & + & & & & & its . &
cond . & its . & cond . & its . & cond .
+ 32 & 245k & 20k & 106 & 322 & 637 & 9811.7 & 110 & 1467.8 & 112 & 1514.1 + 64 & 123k & 28k & 192 & 597 & 618 & 10254.1 & 62 & 115.1 & 63 & 117.7 + 128 & 61k & 45k & 413 & 1293 & 2834 & 1.0e+11 & 206 & 401641.4 & 75 & 194.4 + 256 & 31k & 72k & 902 & 2791 & 799 & 11172.9 & 117 & 512.9 & 119 & 526.7 + 512 & 15k & 110k & 2009 & 6347 & 883 & 15449.6 & 136 & 1160.1 & 137 & 1143.4 + 1024 & 8k & 155k & 4575 & 14725 & n / a & 2.5e+10 & 504 & 99023.6 & 173 & 897.0 +
a parallel solver for the mixed - hybrid finite element formulation based on darcy s law has been presented .
the software combines an existing package _ flow123d _ developed for problems in geophysics with _ bddcml _ , a parallel library for solution of systems of linear algebraic equations by the bddc method .    in geoengineering applications ,
the mathematical model is applied to geometries with the presence of fractures . in the present approach , the flow in these fractures is also modelled by darcy s law , although the hydraulic conductivity of the porous media is considered by orders of magnitude higher .
these fractures are modelled by finite elements of a lower dimension . in the discretised model , 1d
, 2d and 3d finite elements are coupled together through robin boundary conditions .
these coupling terms lead to a modification of the usual saddle - point matrix of the system , in which a new non - zero block appears on the diagonal .
the bddc method is employed for the solution of the resulting system of linear algebraic equations .
bddc is based on iterative substructuring , in which the problem is first reduced to the interface among substructures .
the schur complement is not built explicitly .
instead , only multiplications by the matrix are performed through solving a discrete dirichlet problem on each substructure . in the setting of the mixed - hybrid problem ,
the interface is built only as a subset of the block of lagrange multipliers , while remaining unknowns belong to interiors of substructures .
although the original problem is symmetric indefinite , the system reduced to the interface is symmetric positive definite .
this is also shown to hold for the case with fractures in the present paper .
consequently , the pcg method is used for the solution of the reduced problem .
however , unlike the symmetric positive definite problems , a direct solver for symmetric indefinite matrices needs to be used for the factorisation and repeated solution of local problems on substructures .
one step of the bddc method is used as the preconditioner for the pcg method run on the interface problem .
a modification of the diagonal stiffness scaling has been introduced .
it is motivated by difficult engineering problems , for which it performs significantly better than other two applicable choices
the arithmetic averaging and the modified @xmath291-scaling .
arithmetic averages over faces between substructures are used as the basic constraints defining the coarse space .
in addition , corners are selected from unknowns at the interface using the face - based algorithm .
while corners are not required by the theory , they are shown to improve both the convergence and the computational times for complicated problems .
the performance of the resulting solver has been investigated on three benchmark problems in 2d and 3d .
both weak and strong scaling tests have been performed . on benchmark problems with single mesh dimension , the expected optimal convergence independent of number of substructures has been achieved .
correspondingly , the resulting parallel scalability has been nearly optimal for the weak scaling tests up to 64 computer cores .
the strong scaling tests were presented for a benchmark problem of a unit cube and for two engineering problems containing large variations in element sizes and hydraulic conductivities , using up to 1024 computer cores and containing up to 15 million degrees of freedom .
the convergence for the unit cube problem with all three possible dimensions of finite elements slightly deteriorated by using more substructures , and this translated to sub - optimal parallel performance . however , for the two engineering applications , in which only 3d and 2d elements are combined , the bddc method has also maintained good convergence properties with the growing number of substructures , resulting in optimal , or even super - optimal parallel scalability of the solver .
it has been also shown that the proposed modification of the diagonal stiffness scaling plays an important role in achieving such independence for the challenging engineering problems presented in the paper .
elman hc , silvester dj , wathen aj . _ finite elements and fast iterative solvers : with applications in incompressible fluid dynamics_. numerical mathematics and scientific computation , oxford university press , 2005 .
cros jm . a preconditioner for the schur complement domain decomposition method . _ domain decomposition methods in science and engineering
_ , herrera i , keyes de , widlund ob ( eds . ) .
national autonomous university of mexico ( unam ) , mxico , 2003 ; 373380 .
14th international conference on domain decomposition methods , cocoyoc , mexico , january 612 , 2002 .
fragakis y , papadrakakis m. the mosaic of high performance domain decomposition methods for structural mechanics : formulation , interrelation and numerical efficiency of primal and dual methods .
_ computer methods in applied mechanics and engineering _ 2003 ; * 192*:37993830 .
mandel j , sousedk b. adaptive selection of face coarse degrees of freedom in the bddc and the feti - dp iterative substructuring methods . _
computer methods in applied mechanics and engineering _ 2007 ; * 196*(8):13891399 .
oden j , lee j. dual - mixed hybrid finite element method for second - order elliptic problems .
_ mathematical aspects of finite element methods _ , _ lecture notes in mathematics _ , vol .
606 , galligani i , magenes e ( eds . ) .
springer berlin heidelberg , 1977 ; 275291 .
maryka j , rozlonk m , tma m. schur complement systems in the mixed - hybrid finite element approximation of the potential fluid flow problem .
_ siam journal on scientific computing _ 2000 ; * 22*(2):704723 .
bezina j , hokr m. mixed - hybrid formulation of multidimensional fracture flow .
_ proceedings of the 7th international conference on numerical methods and applications _ ,
nma10 , springer - verlag : berlin , heidelberg , 2011 ; 125132 .
ertkov m , stek j , burda p. on selection of interface weights in domain decomposition methods .
_ proceedings of programs and algorithms of numerical mathematics 16 , doln maxov , czech republic , june 38 , 2012_. institute of mathematics as cr , 2013 ; 3544 .