the computational effort needed to deal with large combinatorial structures considerably varies with the task to be performed and the resolution procedure used@xcite .
the worst case complexity of a task , more precisely an optimization or decision problem , is defined as the time required by the best algorithm to treat any possible inputs to the problem .
for instance , the sorting problem of a list of @xmath0 numbers has worst - case complexity @xmath1 : there exists several algorithms that can order any list in at most @xmath2 elementary operations , and none with asymptotically less operations .
unfortunately , the worst - case complexities of many important computational problems , called np - complete , is not known .
partitioning a list of @xmath0 numbers in two sets with equal partial sums is one among hundreds of such np - complete problems .
it is a fundamental conjecture of theoretical computer science that there exists no algorithm capable of partitioning any list of length @xmath0 , or of solving any other np - complete problem with inputs of size @xmath0 , in a time bounded by a polynomial of @xmath0 .
therefore , when dealing with such a problem , one necessarily uses algorithms which may takes exponential times on some inputs . quantifying how ` frequent ' these hard inputs are for a given algorithm
is the question answered by the analysis of algorithms . in this paper
, we will present an overview of recent works done by physicists to address this point , and more precisely to characterize the average performances , called hereafter complexity , of a given algorithm over a distribution of inputs to an optimization problem .
the history of algorithm analysis by physical methods / ideas is at least as old as the use of computers by physicists .
one well - established chapter in this history is , for instance , the analysis of monte carlo sampling algorithms for statistical mechanics models . in this context
, it is well known that phase transitions , _
i.e. _ abrupt changes in the physical properties of the model , can imply a dramatic increase in the time necessary to the sampling procedure .
this phenomenon is commonly known as critical slowing down .
the physicists insight in this problem comes mainly from the analogy between the dynamics of algorithms and the physical dynamics of the system .
this analogy is quite natural : in fact many algorithms mimick the physical dynamics itself .
a quite new idea is instead to abstract from physically motivated problems and use statistical mechanics ideas for analyzing the dynamics of algorithms . in effect
there are many reasons which suggest that analysis of algorithms and statistical physics should be considered close relatives . in both cases
one would like to understand the asymptotic behavior of dynamical processes acting on exponentially large ( in the size of the problem ) configuration spaces .
the differences between the two disciplines mainly lie in the methods ( and , we are tempted to say , the style ) of investigation .
theoretical computer science derives rigorous results based on probability theory
. however these results are sometimes too weak for a complete characterization of the algorithm .
physicists provide instead heuristic results based on intuitively sensible approximations .
these approximations are eventually validated by a comparison with numerical experiments . in some lucky cases , approximations are asymptotically irrelevant : estimates are turned into conjectures left for future rigorous derivations .
perhaps more interesting than stylistic differences is the _ point of view _ which physics brings with itself .
let us highlight two consequences of this point of view .
first , a particular importance is attributed to `` complexity phase transitions '' _ i.e. _ abrupt changes in the resolution complexity as some parameter defining the input distribution is varied@xcite .
we shall consider two examples in the next sections :    * random satisfiability of boolean constraints ( sat ) . in @xmath3-sat one
is given an instance , that is , a set of @xmath4 logical constraints ( clauses ) among @xmath0 boolean variables , and wants to find a truth assignment for the variables which fulfill all the constraints .
each clause is the logical or of @xmath3 literals , a literal being one of the @xmath0 variables or its negation e.g. @xmath5 for 3-sat .
random @xmath3-sat is the @xmath3-sat problem supplied with a distribution of inputs uniform over all instances having fixed values of @xmath0 and @xmath4 .
the limit of interest is @xmath6 at fixed ratio @xmath7 of clauses per variable@xcite . *
vertex cover of random graphs ( vc ) .
an input instance of the vc decision problem consists in a graph @xmath8 and an integer number @xmath9 .
the problem consists in finding a way to distribute @xmath9 covering marks over the vertices in such a way that every edge of the graph is covered , that is , has at least one of its ending vertices marked . a possible distribution of inputs is provided by drawing random graphs @xmath8 _  la _ erds - reny _ i.e. _ with uniform probability among all the graphs having @xmath0 vertices and @xmath10 edges .
the limit of interest is @xmath11 at fixed ratio @xmath12 of edges per vertex .
the algorithms for random sat and vc we shall consider in the next sections undergo a complexity phase transition as the input parameter @xmath13 ( @xmath14 for sat , @xmath15 for vc ) crosses some critical threshold @xmath16 . typically resolution of a randomly drawn instance
requires linear time below the threshold @xmath17 and exponential time above @xmath18 .
the observation that most difficult instances are located near the phase boundary confirms the relevance of the phase - transition phenomenon .
secondly , a key role is played by the intrinsic ( algorithm independent ) properties of the instance under study .
the intuition is that , underlying the dramatic slowing down of a particular algorithm , there can be some _ qualitative _ change in some structural property of the problem e.g. the geometry of the space of solutions . while there is no general understanding of this question
, we can further specify the above statements case - by - case .
let us consider , for instance , a local search algorithm for a combinatorial optimization problem .
if the algorithm never increases the value of the cost function @xmath19 where @xmath20 is the configuration ( assignment ) of variables to be optimized over , the number and geometry of the local minima of @xmath19 will be crucial for the understanding of the dynamics of the algorithm .
this example is illustrated in sec .
[ gradxor ] .
while the `` dynamical '' behavior of a particular algorithm is not necessarily related to any `` static '' property of the instance , this approach is nevertheless of great interest because it could provide us with some ` universal ' results
. some properties of the instance , for example , may imply the ineffectiveness of an entire class of algorithms .
while we shall mainly study in this paper the performances of search algorithms applied to hard combinatorial problems as sat , vc , we will also consider easy , that is , polynomial problems as benchmarks for these algorithms .
the reason is that we want to understand if the average hardness of resolution of solving np - complete problems with a given distribution of instances and a given algorithm truly reflects the intrinsic hardness of these combinatorial problems or is simply due to some lack of efficiency of the algorithm under study .
the benchmark problem we shall consider is random xorsat .
it is a version of a satisfiability problem , much simpler than sat from a computational complexity point of view@xcite .
the only but essential difference with sat is that a clause is said to be satisfied if the exclusive , and not inclusive , disjunction of its literals is true .
xorsat may be recast as a linear algebra problem , where a set of @xmath4 equations involving @xmath0 boolean variables must be satisfied modulo 2 , and is therefore solvable in polynomial time by various methods e.g. gaussian elimination .
nevertheless , it is legitimate to ask what are the performances of general search algorithms for this kind of polynomial computational problem . in particular
, we shall see that some algorithms requiring exponential times to solve random sat instances behave badly on random xorsat instances too . a related question we shall focus on in sec .
[ codesection ] is decoding , which may also , in some cases , be expressed as the resolution of a set of boolean equations .
the paper is organized as follows . in sec .
[ dpllsection ] we shall review backtracking search algorithms , which , roughly speaking , work in the space of instances .
we explain the general ideas and then illustrate them on random sat ( sec .
[ dpllsatsection ] ) and vc ( sec .
[ dpllvcsection ] ) . in sec .
[ dpllflucsection ] we consider the fluctuations in running times of these algorithms and analyze the possibility of exploiting these fluctuations in random restart strategies . in sec .
[ localsection ] we turn to local search algorithms , which work in the space of configurations .
we review the analysis of such algorithms for decoding problems ( sec .
[ codesection ] ) , random xorsat ( sec .
[ gradxor ] ) , and sat ( sec .  [ walksatsection ] ) .
finally in the conclusion we suggest some possible future developments in the field .
in this section , we briefly review the davis - putnam - loveland - logemann ( dpll ) procedure  @xcite .
a decision problem can be formulated as a constrained satisfaction problem , where a set of variables must be sought for to fulfill some given constraints . for simplicity , we suppose here that variables may take a finite set of values with cardinality @xmath21 e.g. @xmath22 for sat or vc .
dpll is an exhaustive search procedure operating by trials and errors , the sequence of which can be graphically represented by a search tree ( fig .  [ trees ] ) .
the tree is defined as follows : * ( 1 ) * a node in the tree corresponds to a choice of a variable . *
( 2 ) * an outgoing branch ( edge ) codes for the value of the variable and the logical implications of this choice upon not yet assigned variables and clauses . obviously a node gives birth to @xmath21 branches at most . *
( 3 ) * implications can lead to : * ( 3.1 ) * a violated constraint , then the branch ends with @xmath20 ( contradiction ) , the last choice is modified ( backtracking of the tree ) and the procedure goes on along a new branch ( point 2 above ) ; * ( 3.2 ) * a solution when all constraints are satisfied , then the search process is over ; * ( 3.3 ) * otherwise , some constraints remain and further assumptions on the variables have to be done ( loop back to point 1 ) .    a computer independent measure of computational complexity , that is , the amount of operations necessary to solve the instance , is given by the size @xmath23 of the search tree _
i.e. _ the number of nodes it contains .
performances can be improved by designing sophisticated heuristic rules for choosing variables ( point 1 ) . the resolution time ( or complexity ) is a stochastic variable depending on the instance under consideration and on the choices done by the variable assignment procedure .
its average value , @xmath24 , is a function of the input distribution parameters @xmath13 e.g. the ratio @xmath25 of clauses per variable for sat , or the average degree @xmath15 for the vc of random graphs , which can be measured experimentally and that we want to calculate theoretically .
more precisely , our aim is to determine the values of the input parameters for which the complexity is linear , @xmath26 or exponential , @xmath27 , in the size @xmath0 of the instance and to calculate the coefficients @xmath28 as functions of @xmath13 .
the dpll algorithm gives rise to a dynamical process .
indeed , the initial instance is modified during the search through the assignment of some variables and the simplification of the constraints that contain these variables . therefore , the parameters of the input distribution are modified as the algorithm runs .
this dynamical process has been rigorously studied and understood in the case of a search tree reducing to one branch ( tree a in figure  [ trees])@xcite .
study of trees with massive backtracking e.g. trees b and c in fig .
[ trees ] is much more difficult .
backtracking introduces strong correlations between nodes visited by dpll at very different times , but close in the tree .
in addition , the process is non markovian since instances attached to each node are memorized to allow the search to resume after a backtracking step .
the study of the operation of dpll is based on the following , elementary observation .
since instances are modified when treated by dpll , description of their statistical properties generally requires additional parameters with respects to the defining parameters @xmath13 of the input distribution .
our task therefore consists in    1 .   identifying these extra parameters @xmath29@xcite ; 2 .   deriving the phase diagram of this new , extended distribution @xmath30 to identify , in the @xmath30 space ,
the critical surface separating instances having solution with high probability ( satisfiable phase ) from instances having generally no solution ( unsatisfiable phase ) , see fig .
[ schemoins ] .
3 .   tracking the evolution of an instance under resolution with time @xmath31 ( number of steps of the algorithm ) , that is , the trajectory of its characteristic parameters @xmath32 in the phase diagram .    whether this trajectory remains confined to one of the two phases or crosses the boundary inbetween has dramatic consequences on the resolution complexity .
we find three average behaviours , schematized on fig .
[ schemoins ] :    * if the initial instance has a solution and the trajectory remains in the sat phase , resolution is typically linear , and almost no backtracking is present ( fig .
[ trees]a ) .
the coordinates of the trajectory @xmath32 of the instance in the course of the resolution obey a set of coupled ordinary differential equations accounting for the changes in the distribution parameters done by dpll .
* if the initial instance has no solution , solving the instance , that is , finding a proof of unsatisfiability , takes exponentially large time and makes use of massive backtracking ( fig .
[ trees]b ) .
analysis of the search tree is much more complicated than in the linear regime , and requires a partial differential equation that gives information on the population of branches with parameters @xmath30 throughout the growth of the search tree .
* in some intermediary regime , instances have solutions but finding one requires an exponentially large time ( fig .  [ trees]c ) .
this may be related to the crossing of the boundary between sat and unsat phases of the instance trajectory .
we have therefore a mixed behaviour which can be understood through combination of the two above cases .
we now explain how to apply concretly this approach to the cases of random sat and vc .     and @xmath29 are scalar and not vectorial parameters
. vertical axis is the instance distribution defining parameter @xmath13 .
instances are almost always satisfiable if @xmath33 , unsatisfiable if @xmath34 . under the action of dpll ,
the distribution of instances is modified and requires another parameter @xmath29 to be characterized ( horizontal axis ) , equal to , say , zero prior to any action of dpll .
for non zero values of @xmath29 , the critical value of the defining parameter @xmath13 obviously changes ; the line @xmath35 defines a boundary separating typically sat from unsat instances ( bold line ) .
when the instance is unsat ( point u ) , dpll takes an exponential time to go through the tree trajectory . for satisfiable and easy instances , dpll goes along a branch trajectory in a linear time ( point s ) .
the mixed case of hard sat instances ( point ms ) correspond to the branch trajectory crossing the boundary separating the two phases ( bold line ) , which leads to the exploration of unsat subtrees before a solution is finally found . ]
clauses involving @xmath36 variables @xmath37 , which can be assigned to true ( t ) or false ( f ) .
@xmath38 means ( not @xmath39 ) and @xmath40 denotes the logical or .
the search tree is empty .
dpll randomly selects a clause among the shortest ones , and assigns a variable in the clause to satisfy it , e.g. @xmath41 t ( splitting with the generalized unit clause guc
heuristic @xcite ) .
a node and an edge symbolizing respectively the variable chosen ( @xmath39 ) and its value ( t ) are added to the tree .
the logical implications of the last choice are extracted : clauses containing @xmath39 are satisfied and eliminated , clauses including @xmath38 are simplified and the remaining ones are left unchanged . if no unitary clause ( _ i.e. _ with a single variable ) is present , a new choice of variable has to be made . *
* splitting takes over .
another node and another edge are added to the tree . *
* same as step 2 but now unitary clauses are present .
the variables they contain have to be fixed accordingly .
the propagation of the unitary clauses results in a contradiction .
the current branch dies out and gets marked with c. * 6 .
* dpll backtracks to the last split variable ( @xmath42 ) , inverts it ( f ) and creates a new edge . *
* same as step 4 .
the propagation of the unitary clauses eliminates all the clauses .
a solution s is found and the instance is satisfiable . for an unsatisfiable instance , unsatisfiability is proven when backtracking ( see step 6 ) is not possible anymore since all split variables have already been inverted . in this case , all the nodes in the final search tree have two descendent edges and all branches terminate by a contradiction c. ]    the input distribution of 3-sat is characterized by a single parameter @xmath13 , the ratio @xmath25 of clauses per variable . the action of dpll on an instance of 3-sat , illustrated in fig
.  [ algo ] , causes the changes of the overall numbers of variables and clauses , and thus of @xmath25 .
furthermore , dpll reduces some 3-clauses to 2-clauses .
we use a mixed 2+p - sat distribution@xcite , where @xmath43 is the fraction of 3-clauses , to model what remains of the input instance at a node of the search tree .
using experiments and methods from statistical mechanics@xcite and rigorous calculations@xcite , the threshold line @xmath44 , separating sat from unsat phases , may be estimated with the results shown in fig .
[ sche ] . for @xmath45 , _
i.e. _ left to point t , the threshold line is given by @xmath46 , and saturates the upper bound for the satisfaction of 2-clauses . above @xmath47 , no exact value for @xmath44 is known .
the phase diagram of 2+p - sat is the natural space in which the dpll dynamics takes place .
an input 3-sat instance with ratio @xmath25 shows up on the right vertical boundary of fig .
[ sche ] as a point of coordinates @xmath48 . under the action of dpll , the representative point moves aside from the 3-sat axis and follows a trajectory in the @xmath49 plane .    in this section
, we show that the location of this trajectory in the phase diagram allows a precise understanding of the search tree structure and of complexity as a function of the ratio @xmath25 of the instance to be solved ( inset of fig .  [ sche ] ) .
in addition , we shall present an approximate calculation of trajectories accounting for the case of massive backtracking , that is for unsat instances , and slightly below the threshold in the sat phase .
our approach is based on a non rigorous extension of works by chao and franco who first studied the action of dpll ( without backtracking ) on easy , sat instances@xcite as a way to obtain lower bounds to the threshold @xmath50 , see @xcite for a recent review .
let us emphasize that the idea of trajectory is made possible thanks to an important statistical property of the heuristics of split we consider @xcite ,    * unit - clause ( uc ) heuristic : pick up randomly a literal among a unit clause if any , or any unset variable otherwise .
* generalized unit - clause ( guc ) heuristic : pick up randomly a literal among the shortest avalaible clauses . *
short clause with majority ( sc@xmath51 ) heuristic : pick up randomly a literal among unit clauses if any , or pick up randomly an unset variable @xmath21 , count the numbers of occurences @xmath52 of @xmath21 , @xmath53 in 3-clauses , and choose @xmath21 ( respectively @xmath53 ) if @xmath54 ( resp .
@xmath55 ) . when @xmath56 , @xmath21 and @xmath53 are equally likely to be chosen .
these heuristics do not induce any bias nor correlation in the instances distribution@xcite .
such a statistical `` invariance '' is required to ensure that the dynamical evolution generated by dpll remains confined to the phase diagram of fig .
[ sche ] . in the following
, the initial ratio of clauses per variable of the instance to be solved will be denoted by @xmath57 .
let us consider the first descent of the algorithm , that is the action of dpll in the absence of backtracking .
the search tree is a single branch ( fig .
[ trees]a ) .
the numbers of 2 and 3-clauses are initially equal to @xmath58 respectively . under the action of dpll , @xmath59 and
@xmath60 follow a markovian stochastic evolution process , as the depth @xmath61 along the branch ( number of assigned variables ) increases .
both @xmath59 and @xmath60 are concentrated around their average values , the densities @xmath62 $ ] ( @xmath63 ) of which obey a set of coupled ordinary differential equations ( ode)@xcite , @xmath64 where @xmath65 is the probability that dpll fixes a variable at depth @xmath31 through unit - propagation . function @xmath66 depends upon the heuristic : @xmath67 , @xmath68 ( if @xmath69 ) , @xmath70 with @xmath71 and @xmath72 denotes the @xmath73 modified bessel function . to obtain the single branch trajectory in the phase diagram of fig .
[ sche ] , we solve the odes ( [ ode ] ) with initial conditions @xmath74 , and perform the change of variables @xmath75    results are shown for the guc heuristics and starting ratios @xmath76 and 2.8 in fig .  [ sche ]
. trajectories , indicated by light dashed lines , first head to the left and then reverse to the right until reaching a point on the 3-sat axis at a small ratio .
further action of dpll leads to a rapid elimination of the remaining clauses and the trajectory ends up at the right lower corner s , where a solution is found .
frieze and suen @xcite have shown that , for ratios @xmath77 ( for the guc heuristics ) , the full search tree essentially reduces to a single branch , and is thus entirely described by the odes ( [ ode ] ) .
the number of backtrackings necessary to reach a solution is bounded from above by a power of @xmath78 .
the average size @xmath24 of the branch then scales linearly with @xmath0 with a multiplicative factor @xmath79 that can be analytically computed @xcite .
the boundary @xmath80 of this easy sat region can be defined as the largest initial ratio @xmath57 such that the branch trajectory @xmath81 issued from @xmath57 never leaves the sat phase in the course of dpll resolution .      for ratios above threshold ( @xmath82 ) ,
instances almost never have a solution but a considerable amount of backtracking is necessary before proving that clauses are incompatible .
as shown in fig .
[ trees]b , a generic unsat tree includes many branches .
the number of branches ( leaves ) , @xmath83 , or the number of nodes , @xmath84 , grow exponentially with @xmath0@xcite .
it is convenient to define its logarithm @xmath85 through @xmath86 .
contrary to the previous section , the sequence of points @xmath87 characterizing the evolution of the 2+p - sat instance solved by dpll does not define a line any longer , but rather a patch , or cloud of points with a finite extension in the phase diagram of fig .
[ schemoins ] .
we have analytically computed the logarithm @xmath85 of the size of these patches , as a function of @xmath88 , extending to the unsat region the probabilistic analysis of dpll .
this is , _ a priori _ , a very difficult task since the search tree of fig .
1b is the output of a complex , sequential process : nodes and edges are added by dpll through successive descents and backtrackings .
we have imagined a different building up , that results in the same complete tree but can be mathematically analyzed : the tree grows in parallel , layer after layer ( fig .
[ struct ] ) .     denotes the depth in the tree , that is the number of variables assigned by dpll along each ( living ) branch . at depth @xmath61
, one literal is chosen on each branch among 1-clauses ( unit - propagation , grey circles not represented on figure 1 ) , or 2,3-clauses ( split , black circles as in figure 1 ) .
if a contradiction occurs as a result of unit - propagation , the branch gets marked with c and dies out .
the growth of the tree proceeds until all branches carry c leaves .
the resulting tree is identical to the one built through the usual , sequential operation of dpll . ]
a new layer is added by assigning , according to dpll heuristic , one more variable along each living branch . as a result ,
a branch may split ( case 1 ) , keep growing ( case 2 ) or carry a contradiction and die out ( case 3 ) .
cases 1,2 and 3 are stochastic events , the probabilities of which depend on the characteristic parameters @xmath89 defining the 2+p - sat instance carried by the branch , and on the depth ( fraction of assigned variables ) @xmath31 in the tree .
we have taken into account the correlations between the parameters @xmath90 on each of the two branches issued from splitting ( case 1 ) , but have neglected any further correlation which appear between different branches at different levels in the tree@xcite .
this markovian approximation permits to write an evolution equation for the logarithm @xmath91 of the average number of branches with parameters @xmath90 as the depth @xmath31 increases ,    @xmath92 \qquad . \label{croi}\ ] ]    @xmath93 incorporates the details of the splitting heuristics . in terms of the partial derivatives
@xmath94 , @xmath95 , we have for the uc and guc heuristics @xmath96 \nonumber \\ { h } _ { guc }   % ( c_2 , c_3 , y_2 , y_3 , t )   & = & \log _ 2 \nu ( y_2 )    +   \frac{1}{\ln2 } \left [ \frac { 3\ , c_3}{1-t}\ ; \left ( e^{y_3 } \frac{1+e^{-y_2}}{2 } -1 \right)+ \frac{c_2}{1-t }   \ ; \left ( \nu(y_2 ) -2 \right ) \right ] \nonumber \\ \hbox{\rm where } & & \nu ( y_2 ) = \frac 12\ ; e^{y_2}\left (   1 + \sqrt{1 + 4 e^{-y_2 } } \right)\qquad .\end{aligned}\ ] ] partial differential equation ( pde ) ( [ croi ] ) is analogous to growth processes encountered in statistical physics @xcite .
the surface @xmath85 , growing with `` time '' @xmath31 above the plane @xmath97 , or equivalently from ( [ change ] ) , above the plane @xmath87 ( fig .  [ dome ] ) , describes the whole distribution of branches .
the average number of branches at depth @xmath31 in the tree equals @xmath98 , where @xmath99 is the maximum over @xmath100 of @xmath101 reached in @xmath102 . in other words ,
the exponentially dominant contribution to @xmath103 comes from branches carrying 2+p - sat instances with parameters @xmath102 , which define the tree trajectories on fig .
[ sche ] .    the hyperbolic line in fig .
[ sche ] indicates the halt points , where contradictions prevent dominant branches from further growing .
each time dpll assigns a variable through unit - propagation , an average number @xmath104 of new 1-clauses is produced , resulting in a net rate of @xmath105 additional 1-clauses .
as long as @xmath106 , 1-clauses are quickly eliminated and do not accumulate .
conversely , if @xmath107 , 1-clauses tend to accumulate . opposite 1-clauses @xmath42 and @xmath108 are likely to appear , leading to a contradiction @xcite .
the halt line is defined through @xmath109 . as far as dominant branches are concerned , the equation of the halt line reads @xmath110\;\frac 1{1-p } \simeq \frac{1.256}{1-p}\qquad .\ ] ]    along the tree trajectory , @xmath99 grows from 0 , on the right vertical axis , up to some final positive value , @xmath111 , on the halt line .
@xmath112 is our theoretical prediction for the logarithm of the complexity ( divided by @xmath0 ) .
values of @xmath113 obtained for @xmath114 by solving equation ( [ croi ] ) compare very well with numerical results @xcite .
.5 cm    we have plotted the surface @xmath85 above the @xmath87 plane , with the results shown in fig .
it must be stressed that , though our calculation is not rigorous , it provides a very good quantitative estimate of the complexity .
furthermore , complexity is found to scale asymptotically as @xmath115 ^ 2 \simeq \frac{0.292}{\alpha _ 0 } \qquad   ( \alpha _ 0 \gg   \alpha _ c ) .\ ] ] this result exhibits the expected scaling@xcite , and could indeed be exact . as @xmath57 increases ,
search trees become smaller and smaller , and correlations between branches , weaker and weaker .
the interest of the trajectory approach proposed in this paper is best seen in the upper sat phase , that is ratios @xmath57 ranging from @xmath80 to @xmath116 .
this intermediate region juxtaposes branch and tree behaviors , see fig .
[ trees]c .
the branch trajectory starts from the point @xmath117 corresponding to the initial 3-sat instance and hits the critical line @xmath118 at some point g with coordinates ( @xmath119 ) after @xmath120 variables have been assigned by dpll ( fig .
[ sche ] ) .
the algorithm then enters the unsat phase and generates 2+p - sat instances with no solution . a dense subtree , that dpll has to go through entirely , forms beyond g till the halt line ( fig .
[ sche ] ) .
the size of this subtree , @xmath121 , can be analytically predicted from our theory .
g is the highest backtracking node in the tree ( fig .
[ trees]c ) reached back by dpll , since nodes above g are located in the sat phase and carry 2+p - sat instances with solutions .
dpll will eventually reach a solution .
the corresponding branch ( rightmost path in fig .
[ trees]c ) is highly non typical and does not contribute to the complexity , since almost all branches in the search tree are described by the tree trajectory issued from g ( fig .
[ sche ] ) .
we have checked experimentally this scenario for @xmath122 .
the coordinates of the average highest backtracking node , @xmath123 ) , coincide with the analytically computed intersection of the single branch trajectory and the critical line @xmath118@xcite . as for complexity ,
experimental measures of @xmath85 from 3-sat instances at @xmath124 , and of @xmath125 from 2 + 0.78-sat instances at @xmath126 , obey the expected identity @xmath127 and are in very good agreement with theory@xcite .
therefore , the structure of search trees for 3-sat reflects the existence of a critical line for 2+p - sat instances .
we now consider the vc problem , where inputs are random graphs drawn from the @xmath128 ensemble@xcite .
in other words , graphs have @xmath0 vertices and the probability that a pair of vertices are linked through an edge is @xmath129 , independently of other edges .
when the number @xmath130 of covering marks is lowered , the model undergoes a cov / uncov transition at some critical density of covers @xmath131 for @xmath132 . for @xmath133
, vertex covers of size @xmath134 exist with probability one , for @xmath135 the available covering marks are not sufficient .
the statistical mechanics analysis of ref .
@xcite gave the result    @xmath136    where @xmath137 solves the equation @xmath138 .
this result is compatible with the bounds of refs .
@xcite , and was later shown to be exact @xcite . for @xmath139 , eq .
( [ critical_vc ] ) only gives an approximate estimate of @xmath131 .
more sophisticated calculations can be found in ref .
@xcite .
, high-@xmath15 uncov phase is separated by the dashed line , cf .
( [ critical_vc ] ) , from the high-@xmath42 , low-@xmath15 cov phase . the symbols ( numerics ) and continuous lines ( analytical prediction , cf .
( [ eqtrajvc ] ) ) refer to the simple search algorithm described in the text .
the dotted line is the separatrix between two types of trajectories . ]
let us consider a simple implementation of the dpll procedure for the present problem . during the computation
, vertices can be _ covered _ , _ uncovered _ or just _ free _ , meaning that the algorithm has not yet assigned any value to that vertex . at the beginning all the vertices are set _
free_. at each step the algorithm chooses a vertex @xmath140 at random among those which are _
free_. if @xmath140 has neighboring vertices which are either _ free _ or _ uncovered _ , then the vertex @xmath140 is declared _ covered _ first . in case
@xmath140 has only covered neighbors , the vertex is declared _
uncovered_. the process continues unless the number of covered vertices exceeds @xmath9 . in this case the algorithm backtracks and the opposite choice is taken for the vertex @xmath140 unless this corresponds to declaring _ uncovered _ a vertex that has one or more _ uncovered _ neighbors .
the algorithm halts if it finds a solution ( and declares the graph to be cov ) or after exploring all the search tree ( in this case it declares the graph to be uncov ) .    , is plotted versus the number of covering marks . here
we consider random instances with average connectivity @xmath141 .
the phase transition is at @xmath142 and corresponds to the peak in computational complexity . ]    of course one can improve over this algorithm by using smarter heuristics @xcite .
one remarkable example is the `` leaf - removal '' algorithm defined in ref .
@xcite . instead of picking any vertex randomly
, one chooses a connectivity - one vertex , declare it _ uncovered _ , and declare _ covered _ its neighbor .
this procedure is repeated iteratively on the subgraph of _ free _ nodes , until no connectivity - one nodes are left . in the low - connectivity , cov region @xmath143
, it stops only when the graph is completely covered . as a consequence , this algorithm can solve vc in linear time with high probability in all this region .
no equally good heuristics exists for higher connectivity , @xmath139 .      under the action of one of the above algorithms , the instance
is progressively modified and the number of variables is reduced .
in fact , at each step a vertex is selected and can be eliminated from the graph regardless whether it is declared _ covered _ or _
uncovered_. the analysis of the first algorithm is greatly simplified by the remark that , as long as backtracking has not begun , the new vertex is selected randomly .
this implies that the modified instance produced by the algorithm is still a random graph .
its evolution can be effectively described by a trajectory in the @xmath144 space .
if one starts from the parameters @xmath145 , @xmath146 , after @xmath147 steps of the algorithm , he will end up with a new instance of size @xmath148 and parameters @xcite    @xmath149    some examples of the two types of trajectories ( the ones leading to a solution and the ones which eventually enter the uncov region ) are shown in fig .
[ traj_vc ] .
the separatrix is given by    @xmath150    and corresponds to the dotted line in fig .
[ traj_vc ] . above this line
the algorithm solves the problem in linear time .    for more general heuristics
the analysis becomes less straightforward because the graph produced by the algorithm does not belong to the standard random - graph _
ensemble_. it may be necessary to augment the number of parameters which describe the evolution of the instance .
as an example , the leaf - removal algorithm mentioned in the previous section is conveniently described by keeping track of three numbers which parametrize the degree profile ( i.e. the fraction of vertices @xmath151 having a given degree @xmath152 ) of the graph @xcite .      below the critical line @xmath131 , cf .
( [ critical_vc ] ) , no solution exists to the typical random instance of vc .
our algorithm must explore a large backtracking tree to prove it and this takes an exponential time .
the size of the backtracking tree could be computed along the lines of sec .
however a good result can be obtained with a simple `` static '' calculation @xcite .    as explained in sec .
ii.b.2 , we imagine the evolution of the backtracking tree as proceeding `` in parallel '' . at the level @xmath4 of the tree
a set of @xmath4 vertices has been visited .
call @xmath153 the subgraph induced by these vertices .
since we always put a covering mark on a vertex which is surrounded by vertices declared _ uncovered _ , each node of the backtracking tree will carry a vertex cover of the associated subgraph @xmath153 .
therefore the number of backtracking nodes is given by    @xmath154    where @xmath155 is the number of vc s of @xmath153 using at most @xmath9 marks .
a very crude estimate of the right - hand side of the above equation is :    @xmath156    where we bounded the number of vc s of size @xmath157 on @xmath153 with the number of ways of placing @xmath157 marks on @xmath4 vertices .
the authors of @xcite provided a refined estimate based on the _ annealed approximation _ of statistical mechanics .
the results of this calculation are compared in fig .
[ time_vc ] with the numerics .
if the parameters which characterize an instance of vc lie in the region between @xmath131 , cf .
( [ critical_vc ] ) , and @xmath158 , cf .
( [ separatrix_vc ] ) , the problem is still soluble but our algorithm takes an exponential time to solve it . in practice ,
after a certain number of vertices has been visited and declared either _ covered _ or _ uncovered _ , the remaining subgraph @xmath159 can not be any longer covered with the leftover marks .
this happens typically when the first descent trajectory ( [ eqtrajvc ] ) crosses the critical line ( [ critical_vc ] ) .
it takes some time for the algorithm to realize this fact .
more precisely , it takes exactly the time necessary to prove that @xmath159 is uncoverable .
this time dominates the computational complexity in this region and can be calculated along the lines sketched in the previous section .
the result is , once again , reported in fig .
[ time_vc ] , which clearly shows a computational peak at the phase boundary .
finally , let us notice that this mixed behavior disappears in the entire @xmath160 region if the leaf - removal heuristics is adopted for the first descent .      up to now
we have studied the typical resolution complexity .
the study of fluctuations of resolution times is interesting too , particularly in the upper sat phase where solutions exist but are found at a price of a large computational effort .
we may expect that there exist lucky but rare resolutions able to find a solution in a time much smaller than the typical one . due to
the stochastic character of dpll complexity indeed fluctuates from run to run of the algorithm on the same instance . in fig .
[ historun ] we show this run - to - run distribution of the logarithm @xmath85 of the resolution complexity for four instances of random 3-sat with the same ratio @xmath161 .
the run to run distribution are qualitatively independent of the particular instances , and exhibit two bumps .
the wide right one , located in @xmath162 , correspond to the major part of resolutions .
it acquires more and more weight as @xmath0 increases and corresponds to the typical behavior analysed in section  ii.b.3 .
the left peak corresponds to much faster resolutions , taking place in linear time .
the weight of this peak ( fraction of runs with complexities falling in the peak ) decreases exponentially fast with @xmath0 , and can be numerically estimated to @xmath163 with @xmath164 .
therefore , instances at @xmath161 are typically solved in exponential time but a tiny ( exponentially small ) fraction of runs are able to find a solution in linear time only .    a systematic stop - and - restart procedure may be introduced to take advantage of this fluctuation phenomenon and speed up resolution . if a solution is not found before @xmath0 splits , dpll is stopped and rerun after some random permutations of the variables and clauses .
the expected number @xmath165 of restarts necessary to find a solution being equal to the inverse probability @xmath166 of linear resolutions , the resulting complexity scales as @xmath167 .    to calculate @xmath168
we have analyzed , along the lines of the study of the growth of the search tree in the unsat phase , the whole distribution of the complexity for a given ratio @xmath25 in the upper sat phase .
calculations can be found in @xcite .
linear resolutions are found to correspond to branch trajectories that cross the unsat phase without being hit by a contradiction , see fig .
results are reported in fig .
[ histolin ] and compare very well with the experimentally measured number @xmath165 of restarts necessary to find a solution . in the whole upper sat phase , the use of restarts offers an exponential gain with respect to usual dpll resolution ( see fig .
[ histolin ] for comparison between @xmath168 and @xmath85 ) , but the completeness of dpll is lost .
a slightly more general restart strategy consists in stopping the backtracking procedure after a fixed number of nodes @xmath169 has been visited . a new ( and statistically independent ) dpll procedure
is then started from the beginning . in this case one
exploits lucky , but still exponential , stochastic runs .
the tradeoff between the exponential gain of time and the exponential number of restarts , can be optimized by tuning the parameter @xmath170 .
this approach has been analyzed in ref .
@xcite taking vc as a working example . in fig .
[ time_rvc ] we show the computatonal complexity of such a strategy as a function of the restart parameter @xmath170 .
we compare the numerics with an approximate calculation @xcite .
the instances were random graphs with average connectivity @xmath171 , and @xmath172 covering marks per vertex .
the optimal choice of the parameter seems to be ( in this case ) @xmath173 , corresponding to polynomial runs .
the analytical prediction reported in fig .
[ time_rvc ] requires , as for 3-sat , an estimate of the execution - time fluctuations of the dpll procedure ( without restart ) .
it turns out that one major source of fluctuations is , in the present case , the location in the @xmath144 plane of the highest node in the backtracking tree . in the typical run
this coincides with the intersection @xmath174 between the first descent trajectory ( [ eqtrajvc ] ) and the critical line ( [ critical_vc ] ) .
one can estimate the probability @xmath175 for this node to have coordinates @xmath144 ( obviously @xmath176 ) .
when an upper bound @xmath170 on the backtracking time is fixed , the problem is solved in those lucky runs which are characterized by an atypical highest backtracking node . roughly speaking , this means that the algorithm has made some very good ( random ) choices in its first steps . in fig .
[ root_rvc ] we plot the position of the highest backtracking point in the ( last ) successful runs for several values of @xmath170 . once again the numerics compare favourably with an approximate calculation .
we now turn to the description and study of algorithms of another type , namely local search algorithms . as a common feature ,
these algorithms start from a configuration ( assignment ) of the variables , and then make successive improvements by changing at each step few of the variables in the configuration ( local move ) .
for instance , in the sat problem , one variable is flipped from being true to false , or _
vice versa _ , at each step .
whereas complete algorithms of the dpll type give a definitive answer to any instance of a decision problem , exhibiting either a solution or a proof of unsatisfiability , local search algorithms give a sure answer when a solution is found but can not prove unsatisfiability .
however , these algorithms can sometimes be turned into one - sided probabilistic algorithms , with an upper bound on the probability that a solution exists and has not been found after @xmath61 steps of the algorithm , decreasing to zero when @xmath177@xcite .
local search algorithms perform repeated changes of a configuration @xmath20 of variables ( values of the boolean variables for sat , status marked or unmarked of vertices for vc ) according to some criterion , usually based on the comparison of the cost function @xmath178 ( number of unsatisfied clauses for sat , of uncovered edges for vc ) evaluated at @xmath20 and over its neighborhood .
it is therefore clear that the shape of the multidimensional surface @xmath179 , called cost function landscape , is of high importance . on intuitive grounds , if this landscape is relatively smooth with a unique minimum , local procedures as gradient descent should be very efficient , while the presence of many local minima could hinder the search process ( fig .
[ landscape ] ) .
the fundamental underlying question is whether the performances of the dynamical process ( ability to find the global minimum , time needed to reach it ) can be understood in terms of an analysis of the cost function landscape only .
this question was intensively studied and answered for a limited class of cost functions , called mean field spin glass models , some years ago@xcite .
the characterization of landscapes is indeed of huge importance in physical systems .
there , the cost function is simply the physical energy , and local dynamics are usually low or zero temperature monte carlo dynamics , essentially equivalent to gradient descent . depending on the parameters of the input distribution ,
the minima of the cost functions may undergo structural changes , a phenomenon called clustering in physics .
clustering has been rigorously shown to take place in the random 3-xorsat problem@xcite , and is likely to exist in many other random combinatorial problems as 3-sat@xcite .
instances of the 3-xorsat problem with @xmath180 clauses and @xmath0 variables have almost surely solutions as long as @xmath181@xcite .
the clustering phase transition takes place at @xmath182 and is related to a change in the geometric structure of the space of solutions , see fig .
[ landscape ] :    * when @xmath183 , the space of solutions is connected . given a pair of solutions @xmath184 , _
i.e. _ two assignments of the @xmath0 boolean variables that satisfy the clauses , there almost surely exists a sequence of solutions , @xmath185 , with @xmath186 , @xmath187 , @xmath188 , connecting the two solutions such that the hamming distance ( number of different variables ) between @xmath189 and @xmath190 is bounded from above by some finite constant when @xmath132 .
* when @xmath191 , the space of solutions is not connected any longer .
it is made of an exponential ( in @xmath0 ) number of connected components , called clusters , each containing an exponentially large number of solutions .
clusters are separated by large voids : the hamming distance between two clusters , that is , the smallest hammming distance between pairs of solutions belonging to these clusters , is of the order of @xmath0 .    from intuitive grounds , changes of the statistical properties of the cost function landscape e.g. of the structure of the solutions space
may potentially affect the search dynamics .
this connection between dynamics and static properties was established in numerous works in the context of mean field models of spin glasses @xcite , and subsequently also put forward in some studies of local search algorithms in combinatorial optimization problems@xcite .
so far , there is no satisfying explanation to when and why features of _ a priori _ algorithm dependent dynamical phenomena should be related to , or predictable from some statistical properties of the cost function landscape .
we shall see some examples in the following where such a connection indeed exist ( sec .  iii.b ) and other ones where its presence is far less obvious ( sec .
iiic , d ) .    , while vertical axis is the associated cost @xmath19 .
left : smooth cost function , with a single minimum easily reachable with local search procedures e.g. gradient descent .
middle : rough cost function with a lot of local minima whose presence may damage the performances of local search algorithms .
the various global minima are spread out homogeneously over the configuration space .
right : rough cost function with global minima clustered in some portions of the configuration space only .
, title="fig : " ] , while vertical axis is the associated cost @xmath19 .
left : smooth cost function , with a single minimum easily reachable with local search procedures e.g. gradient descent .
middle : rough cost function with a lot of local minima whose presence may damage the performances of local search algorithms .
the various global minima are spread out homogeneously over the configuration space .
right : rough cost function with global minima clustered in some portions of the configuration space only .
, title="fig : " ] , while vertical axis is the associated cost @xmath19 .
left : smooth cost function , with a single minimum easily reachable with local search procedures e.g. gradient descent .
middle : rough cost function with a lot of local minima whose presence may damage the performances of local search algorithms .
the various global minima are spread out homogeneously over the configuration space .
right : rough cost function with global minima clustered in some portions of the configuration space only .
, title="fig : " ]      0 00        coding theory is a rich source of computational problems ( and algorithms ) for which the average case analysis is relevant @xcite .
let us focus , for sake of concreteness , on the decoding problem .
codewords are sequences of symbols with some built - in redundancy .
if we consider the case of linear codes on a binary alphabet , this redundancy can be implemented as a set of linear constraints . in practice ,
a codeword is a vector @xmath192 ( with @xmath193 ) which satisfies the equation    @xmath194    where @xmath195 is an @xmath196 binary matrix ( _ parity check matrix _ ) .
each one of the @xmath4 linear equations involved in eq .
( [ paritycheckmatrix ] ) is called a _
parity check_. this set of equation can be represented graphically by a _ tanner graph _ , cf .
[ tanner ] .
this is a bipartite graph highlighting the relations between the variables @xmath197 and the constraints ( parity checks ) acting on them .
the decoding problem consists in finding , among the solutions of eq .
( [ paritycheckmatrix ] ) , the `` closest '' one @xmath198 to the output @xmath199 of some communication channel .
this problem is , in general , np - hard @xcite .
the precise meaning of `` closest '' depends upon the nature of the communication channel .
let us make two examples :    * the binary symmetric channel ( bsc ) . in this case
the output of the communication channel @xmath199 is a codeword , i.e. a solution of ( [ paritycheckmatrix ] ) , in which a fraction @xmath200 of the entries has been flipped .
`` closest '' has to be understood in the hamming - distance sense .
@xmath198 is the solution of eq .
( [ paritycheckmatrix ] ) which minimizes the hamming distance from @xmath199 . * the binary erasure channel ( bec ) .
the output @xmath199 is a codeword in which a fraction @xmath200 of the entries has been erased .
one has to find a solution @xmath198 of eq .
( [ paritycheckmatrix ] ) which is compatible with the remaining entries .
such a problem has a _ unique _
solution for small enough erasure probability @xmath200 .
there are two sources of randomness in the decoding problem : @xmath201 the matrix @xmath195 which defines the code is usually drawn from some random _ ensemble _ ; @xmath202 the received message which is distributed according to some probabilistic model of the communication channel ( in the two examples above , the bits to be flipped / erased were chosen randomly ) . unlike many other combinatorial problems
, there is therefore a `` natural '' probability distribution defined on the instances .
average case analysis with respect to this distribution is of great practical relevance .
recently , amazingly good performances have been obtained by using low - density parity check ( ldpc ) codes @xcite .
ldpc codes are defined by parity check matrices @xmath195 which are large and sparse . as an example we can consider gallager _ regular _
codes @xcite . in this case
@xmath195 is chosen with flat probability distribution within the family of matrices having @xmath203 ones per column , and @xmath204 ones per row .
these are decoded using a suboptimal linear - time algorithm known as `` belief - propagation '' or `` sum - product '' algorithm @xcite .
this is an iterative algorithm which takes advantage of the locally tree - like structure of the tanner graph , see fig .
[ tanner ] , for ldpc codes .
after @xmath205 iterations it incorporates the information conveyed by the variables up to distance @xmath205 from the one to be decoded . this can be done in a recursive fashion allowing for linear - time decoding .
belief - propagation decoding shows a striking threshold phenomenon as the noise level @xmath200 crosses some critical ( code - dependent ) value @xmath206 .
while for @xmath207 the transmitted codeword is recovered with high probability , for @xmath208 decoding will fail almost always .
the threshold noise @xmath206 is , in general , smaller than the threshold @xmath209 for optimal decoding ( with unbounded computational resources ) .    the rigorous analysis of ref .
@xcite allows a precise determination of the critical noise @xmath206 under quite general circumstances .
nevertheless some important theoretical questions remain open : can we find some smarter linear - time algorithm whose threshold is greater than @xmath206 ?
is there any `` intrinsic '' ( i.e. algorithm independent ) characterization of the threshold phenomenon taking place at @xmath206 ? as a first step towards the answer to these questions , ref .
@xcite explored the dynamics of local optimization algorithms by using statistical mechanics techniques .
the interesting point is that `` belief propagation '' is by no means a local search algorithm .    for sake of concreteness , we shall focus on the binary erasure channel . in this case
we can treat decoding as a combinatorial optimization problem within the space of bit sequences of length @xmath210 ( the number of erased bits , the others being fixed by the received message ) .
the function to be minimized is the _ energy density _
@xmath211    where we denote as @xmath212 the hamming distance between two vectors @xmath213 and @xmath214 , and we introduced the normalizing factor for future convenience .
notice that both arguments of @xmath215 in eq .
( [ costfunction ] ) are vectors in @xmath216 .
we can define the @xmath217-neighborhood of a given sequence @xmath218 as the set of sequences @xmath219 such that @xmath220 , and we call @xmath217-stable states the bit sequences which are optima of the decoding problem within their @xmath217-neighborhood
.    one can easily invent local search algorithms @xcite for the decoding problem which use the @xmath217-neighborhoods .
the algorithm start from a random sequence and , at each step , optimize it within its @xmath217-neighborhood .
this algorithm is clearly suboptimal and halts on @xmath217-stable states .
let us consider , for instance , a @xmath221 regular code and decode it by local search in @xmath222-neighborhoods . in fig .
[ glauber ] we report the resulting energy density @xmath223 after the local search algorithm halts , as a function of the erasure probability @xmath200 .
we averaged over 100 different realizations of the noise and of the matrix @xmath195 .
for sake of comparison we recall that the threshold for belief - propagation decoding is @xmath224 @xcite , while the threshold for optimal decoding is at @xmath225 @xcite .
it is evident that local search by @xmath222-neighborhoods performs quite poorly .
a natural question is whether ( and how much ) , these performances are improved by increasing @xmath217 .
it is therefore quite natural to study _
metastable _ states .
these are @xmath217-stable states for any @xmath226 if @xmath227 . ] .
there exists no completely satisfying definition of such states : here we shall just suggest a possibility among others .
the tricky point is that we do not know how to compare @xmath217-stable states for different values of @xmath0 .
this forbids us to make use of the above asymptotic statement .
one possibility is to count without really defining them .
this can be done , at least in principle , by counting @xmath217-stable states , take the @xmath228 limit and , at the end , the @xmath229 limit@xcite . on physical grounds ,
we expect @xmath217-stable states to be exponentially numerous .
in particular , if we call @xmath230 the number of @xmath217-stable states taking a value @xmath223 of the cost function ( [ costfunction ] ) , we have    @xmath231    we can therefore define the so called ( physical ) complexity @xmath232 as follows ,    @xmath233    roughly speaking we can say that the number of metastable states is @xmath234 . of course there are several alternative ways of taking the limits @xmath229 , @xmath228 , and we do not yet have a proof that these procedures give the same result for @xmath232
nevertheless it is quite clear that the existence of an exponential number of metastable states should affect dramatically the behavior of local search algorithms .
statistical mechanics methods @xcite allows to determine the complexity @xmath232 @xcite . in `` difficult '' cases ( such as for error - correcting codes ) ,
the actual computation may involve some approximation , e.g. the use of a variational ansatz .
nevertheless the outcome is usually quite accurate . in fig .
[ complexity ] we consider a @xmath235 regular code on the binary erasure channel .
we report the resulting complexity for three different values of the erasure probability @xmath200 .
the general picture is as follows .
below @xmath206 there is no metastable state , except the one corresponding to the correct codeword . between @xmath206 and @xmath209
there is an exponential number of metastable states with energy density belonging to the interval @xmath236 ( @xmath232 is strictly positive in this interval ) . above @xmath209 , @xmath237 .
the maximum of @xmath232 is always at @xmath238 .
the above picture tell us that any local algorithm will run into difficulties above @xmath206 . in order to confirm this picture , the authors of ref .
@xcite made some numerical computations using simulated annealing as decoding algorithm for quite large codes ( @xmath239 bits ) .
for each value of @xmath200 , we start the simulation fixing a fraction @xmath240 of spins to @xmath241 ( this part will be kept fixed all along the run ) .
the remaining @xmath242 spins are the dynamical variables we change during the annealing in order to try to satisfy all the parity checks .
the energy of the system counts the number of unsatisfied parity checks .
the cooling schedule has been chosen in the following way : @xmath243 monte carlo sweeps ( mcs )   proposed spin flips .
each proposed spin flip is accepted or not accordingly to a standard metropolis test . ] at each of the 1000 equidistant temperatures between @xmath244 and @xmath245 .
the highest temperature is such that the system very rapidly equilibrates .
typical values for @xmath243 are from 1 to @xmath246 .
notice that , for any fixed cooling schedule , the computational complexity of the simulated annealing method is linear in @xmath0
. then we expect it to be affected by metastable states of energy @xmath238 , which are present for @xmath208 : the energy relaxation should be strongly reduced around @xmath238 and eventually be completely blocked .
some results are plotted in fig .
[ annealing ] together with the theoretical prediction for @xmath238 . the good agreement confirm our picture :
the algorithm gets stucked in metastable states , which have , in the great majority of cases , energy density @xmath238 .
both `` belief propagation '' and local search algorithms fail to decode correctly between @xmath206 and @xmath209 .
this leads naturally to the following conjecture : no linear time algorithm can decode in this regime of noise .
the ( typical case ) computational complexity changes from being linear below @xmath206 to superlinear above @xmath206 . in the case of the binary erasure channel
it remains polynomial between @xmath206 and @xmath209 ( since optimal decoding can be realized with linear algebra methods ) .
however it is plausible that for a general channel it becomes non - polynomial .      in this section the local procedure we consider is gradient descent ( gd ) .
gd is defined as follows .
* ( 1 ) * start from an initial randomly chosen configuration of the variables . call
@xmath10 the number of unsatisfied clauses . *
( 2 ) * if @xmath247 then stop ( a solution is found ) .
otherwise , pick randomly one variable , say @xmath248 , and compute the number @xmath249 of unsatisfied clauses when this variable is negated ; if @xmath250 then accept this change _
i.e. _ replace @xmath248 with @xmath251 and @xmath10 with @xmath249 ; if @xmath252 , do not do anything . then go to step 2 .
the study of the performances of gd to find the minima of cost functions related to statistical physics models has recently motivated various studies@xcite .
numerics indicate that gd is typically able to solve random 3-sat instances with ratios @xmath253 @xcite close to the onset of clustering @xcite
. we shall rigorously show below that this is not so for 3-xorsat .
let us apply gd to an instance of xorsat .
the instance has a graph representation explained in fig .
[ xorgr ] .
vertices are in one  to  one correspondence with variables .
a clause is fully represented by a plaquette joining three variables and a boolean label equal to the number of negated variables it contains modulo 2 ( not represented on fig .
[ xorgr ] ) .
once a configuration of the variables is chosen , each plaquette may be labelled by its status , s or u , whether the associated clause is respectively satisfied or unsatisfied .
a fundamental property of xorsat is that each time a variable is changed , _
i.e. _ its value is negated , the clauses it belongs to change status too .
this property makes the analysis of some properties of gd easy .
consider the hypergraph made of 15 vertices and 7 plaquettes in fig .
[ bi ] , and suppose the central plaquette is violated ( u ) while all other plaquettes are satisfied ( s ) .
the number of unsatisfied clauses is @xmath254 .
now run gd on this special instance of xorsat .
two cases arise , symbolized in fig .
[ bi ] , whether the vertex attached to the variable to be flipped belongs , or not , to the central plaquette .
it is an easy check that , in both cases , @xmath255 and the change is not permitted by gd .
the hypergraph of fig .
[ bi ] will be called hereafter island . when the status of the plaquettes is u for the central one and s for the other ones , the island
is called blocked .
though the instance of the xorsat problem encoded by a blocked island is obviously satisfiable ( think of negating at the same time one variable attached to a vertex @xmath256 of the central plaquette and one variable in each of the two peripherical plaquettes joining the central plaquette at @xmath256 ) , gd will never be able to find a solution and will be blocked forever in the local minimum with height @xmath254 .
the purpose of this section is to show that this situation typically happens for random instances of xorsat .
more precisely , while almost all instances of xorsat with a ratio of clauses per variables smaller than @xmath257 have a lot of solutions , gd is almost never able to find one .
even worse , the number of violated clauses reached by gd is bounded from below by @xmath258 where @xmath259 in other words , the number of clauses remaining unsatisfied at the end of a typical gd run is of the order of @xmath0 .
our demonstration , inspired from @xcite , is based on the fact that , with high probability , a random instance of xorsat contains a large number of blocked islands of the type of fig .
[ bi ] .    to make the proof easier
, we shall study the following fixed clause probability ensemble . instead of imposing the number of clauses to be equal to @xmath260 , any triplet @xmath243 of three vertices ( among @xmath0 )
is allowed to carry a plaquette with probability @xmath261 .
notice that this probability ensures that , on the average , the number of plaquettes equals @xmath262 .
let us now draw a hypergraph with this distribution .
for each triplet @xmath243 of vertices , we define @xmath263 if @xmath243 is the center of a island , 0 otherwise .
we shall show that the total number of islands , @xmath264 , is highly concentrated in the large @xmath0 limit , and calculate its average value .
the expectation value of @xmath265 is equal to @xmath266 = \frac{(n-3)\times ( n-4 ) \times \ldots \times   ( n-13)\times ( n-14)}{8\times 8\times 8 } \times \mu ^a \ , ( 1-\mu ) ^b   \ , \ ] ] where @xmath267 is the number of plaquettes in the island , and @xmath268 is the number of triplets with at least one vertex among the set of 15 vertices of the island that do not carry plaquette .
the significance of the terms in eq .
( [ expitau ] ) is transparent .
the central triplet @xmath243 occupying three vertices , we choose 2 vertices among @xmath269 to draw the first peripherical plaquette of the island , then other 2 vertices among @xmath270 for the other peripherical plaquette having a common vertex with the latter .
the order in which these two plaquettes are built does not matter and a factor @xmath271 permits us to avoid double counting .
the other four peripherical plaquettes have multiplicities calculable in the same way ( with less and less available vertices ) .
the terms in @xmath272 and @xmath273 correspond to the probability that such a 7 plaquettes configuration is drawn on the 15 vertices of the island , and is disconnected from the remaining @xmath274 vertices .
the expectation value of the number @xmath275 of islands per vertex thus reads , @xmath276 =   \lim _ { n\to\infty } \frac 1n \ , { n \choose 3}\ , e[i_{\tau } ] = \frac{729}{8 } \ , \alpha^7 \ , e^{-45\ ,   \alpha } \quad .\ ] ] chebyshev s inequality can be used to show that @xmath140 is concentrated around its above average value .
let us calculate the second moment of the number of islands , @xmath277= \sum _ { \tau , \sigma } e [ i_{\tau } i_{\sigma } ] $ ] .
clearly , @xmath278 $ ] depends only on the number @xmath279 of vertices common to triplets @xmath243 and @xmath280 .
it is obvious that no two triplets of vertices can be centers of islands when they have @xmath281 or @xmath282 common vertices . if @xmath283 , @xmath284 and @xmath285=e [ i_{\tau}]$ ] has been calculated above . for @xmath286 ,
a similar calculation gives @xmath287 finally , we obtain @xmath288 = \frac 1{n^2 } \left [ { n \choose 3 } e_{\ell =3 } + { n \choose 3}{n -3\choose 3 } e_{\ell = 0 } \right ] = e[i]^2 + o\left ( \frac 1n \right )   \quad .\ ] ] therefore the variance of @xmath140 vanishes and @xmath140 is , with high probability , equal to its average value given by ( [ eq89 ] ) . to conclude , an island has a probability @xmath289 to be blocked by definition .
therefore the number ( per vertex ) of blocked islands in a random xorsat instance with ratio @xmath25 is almost surely equal to @xmath290 given by eq .
( [ teh ] ) .
since each blocked island has one unsatisfied clause , this is also a lower bound to the number of violated clauses per variable .
notice however that @xmath291 is very small and bounded from above by @xmath292 over the range of interest , @xmath293 .
therefore , one would in principle need to deal with billions of variables not to reach solutions and be in the true asymptotic regime of gd .
the proof is easily generalizable to gradient descent with more than one look ahead . to extend the notion of blocked island to the case where gd is allowed to invert @xmath217 , and not only 1 , variables at a time
, it is sufficient to have @xmath294 , and not 2 , peripherical plaquettes attached to each vertex of the central plaquette .
the calculation of the lower bound @xmath295 to the number of violated clauses ( divided by @xmath0 ) reached by gd is straightforward and not reproduced here .
as a consequence , gd , even with @xmath217 simultaneous flips permitting to overcome local barriers , remains almost surely trapped at an extensive ( in @xmath0 ) level of violated clauses for any finite @xmath217 .
actually the lower bound @xmath296 tends to zero only if @xmath217 is of the order of @xmath78 .
we stress that the statistical physics calculation of physical ` complexity ' @xmath297 ( see sec .
[ codesection ] ) predicts there is no metastable states for @xmath298@xcite , while gd is almost surely trapped by the presence of blocked islands for any @xmath299 .
this apparent discrepancy comes from the fact that gd is sensible to the presence of configurations blocked for finite @xmath217 , while the physical ` complexity ' considers states metastable in the limit @xmath229 only@xcite .     and @xmath300 .
each clause or equation is represented by a plaquette whose vertices are the attached variables . when the variables are assigned some values
, the clauses can be satisfied ( s ) or unsatisfied ( u ) . ]          the pure random walksat ( prwsat ) algorithm for solving @xmath3-sat is defined by the following rules@xcite .    1 .
choose randomly a configuration of the boolean variables .
2 .   if all clauses are satisfied , output `` satisfiable '' .
3 .   if not , choose randomly one of the unsatisfied clauses , and one among the @xmath3 variables of this clause .
flip ( invert ) the chosen variable . notice that the selected clause is now satisfied , but the flip operation may have violated other clauses which were previously satisfied .
4 .   go to step 2 , until a limit on the number of flips fixed beforehand has been reached .
then output `` do nt know '' .
what is the output of the algorithm ?
either `` satisfiable '' and a solution is exhibited , or `` do nt know '' and no certainty on the status of the formula is achieved .
papadimitriou introduced this procedure for @xmath301 , and showed that it solves with high probability any satisfiable 2-sat instance in a number of steps ( flips ) of the order of @xmath302@xcite .
recently schning was able to prove the following very interesting result for 3-sat@xcite .
call ` trial ' a run of prwsat consisting of the random choice of an initial configuration followed by @xmath303 steps of the procedure .
if none of @xmath61 successive trials on a given instance has been successful ( has provided a solution ) , then the probability that this instance is satisfiable is lower than @xmath304 . in other words , after @xmath305 trials of prwsat , most of the configuration space has been ` probed ' : if there were a solution , it would have been found .
though this local search algorithm is not complete , the uncertainty on its output can be made as small as desired and it can be used to prove unsatisfiability ( in a probabilistic sense ) .
schning s bound is true for any instance .
restriction to special input distributions allows to strengthen this result .
alekhnovich and ben - sasson showed that instances drawn from the random 3-satisfiability ensemble described above are solved in polynomial time with high probability when @xmath25 is smaller than @xmath306@xcite .      in this section
, we briefly sketch the behaviour of prwsat , as seen from numerical experiments  @xcite and the analysis of @xcite . a dynamical threshold @xmath307 ( @xmath308 for 3-sat )
is found , which separates two regimes :    * for @xmath309 , the algorithm finds a solution very quickly , namely with a number of flips growing linearly with the number of variables @xmath0 .
figure  [ wsat_phen]a shows the plot of the fraction @xmath310 of unsatisfied clauses as a function of time @xmath31 ( number of flips divided by @xmath4 ) for one instance with ratio @xmath311 and @xmath312 variables .
the curve shows a fast decrease from the initial value ( @xmath313 in the large @xmath0 limit independently of @xmath25 ) down to zero on a time scale @xmath314 .
fluctuations are smaller and smaller as @xmath0 grows .
@xmath315 is an increasing function of @xmath25 .
this _ relaxation _
regime corresponds to the one studied by alekhnovich and ben - sasson , and @xmath316 as expected@xcite . * for instances in the @xmath317 range , the initial relaxation phase taking place on @xmath318 time scale is not sufficient to reach a solution ( fig .
[ wsat_phen]b ) .
the fraction @xmath319 of unsat clauses then fluctuates around some plateau value for a very long time . on the plateau ,
the system is trapped in a _
metastable _ state .
the life time of this metastable state ( trapping time ) is so huge that it is possible to define a ( quasi ) equilibrium probability distribution @xmath320 for the fraction @xmath319 of unsat clauses .
( inset of fig .
[ wsat_phen]b ) .
the distribution of fractions is well peaked around some average value ( height of the plateau ) , with left and right tails decreasing exponentially fast with @xmath0 , @xmath321 with @xmath322 .
eventually a large negative fluctuation will bring the system to a solution ( @xmath323 ) . assuming that these fluctuations are independant random events occuring with probability @xmath324 on an interval of time of order @xmath222 , the resolution time is a stochastic variable with exponential distribution .
its average is , to leading exponential order , the inverse of the probability of resolution on the @xmath325 time scale : @xmath326 \sim \exp ( n \zeta)$ ] with @xmath327 .
escape from the metastable state therefore takes place on exponentially large
in@xmath0 time scales , as confirmed by numerical simulations for different sizes .
schning s result@xcite can be interpreted as a lower bound to the probability @xmath328 , true for any instance .    the plateau energy , that is , the fraction of unsatisfied clauses reached by prwsat on the linear time scale is plotted on fig .  [ wsat_plateau ] .
notice that the `` dynamic '' critical value @xmath329 above which the plateau energy is positive ( prwsat stops finding a solution in linear time ) is strictly smaller than the `` static '' ratio @xmath330 , where formulas go from satisfiable with high probability to unsatisfiable with high probability . in the intermediate range @xmath331 ,
instances are almost surely satisfiable but prwsat needs an exponentially large time to prove so .
interestingly , @xmath332 and @xmath330 coincides for 2-sat in agreement with papadimitriou s result@xcite .
furthermore , the dynamical transition is apparently not related to the onset of clustering taking place at @xmath333 .    a b .5 cm    .5 cm      when prwsat finds easily a solution , the number of steps it requires is of the order of @xmath0 , or equivalently , @xmath4 .
let us call @xmath334 the average of this number divided by the number of clauses @xmath4 . by definition of the dynamic threshold ,
@xmath315 diverges when @xmath335 . assuming that @xmath336 can be expressed as a series of powers of @xmath25 , we find the following expansion@xcite @xmath337 around @xmath338 . as only a finite number of terms in this expansion have been computed
, we do not control its radius of convergence , yet as shown in fig .
[ wsat_fig_tres_q1 ] the numerical experiments provide convincing evidence in favour of its validity .
the above calculation is based on two facts .
first , for @xmath339 the instance under consideration splits into independent subinstances ( involving no common variable ) that contains a number of variables of the order of @xmath78 at most .
moreover , the number of the connected components containing @xmath340 clauses , computed with probabilistic arguments very similar to those of section  [ gradxor ] , contribute to a power expansion in @xmath25 only at order @xmath341 .
secondly , the number of steps the algorithm needs to solve the instance is simply equal to the sum of the numbers of steps needed for each of its independent subinstances .
this additivity remains true when one averages over the initial configuration and the choices done by the algorithm .
one is then left with the enumeration of the different subinstances with a given size and the calculation of the average number of steps for their resolution .
a detailed presentation of this method has been given in a general case in  @xcite , and applied more specifically to this problem in  @xcite ; the reader is referred to these previous works for more details .
equation ( [ dev_cluster_tresk ] ) is the output of the enumeration of subinstances with up to three clauses .
the above small @xmath25 expansion does not allow us to investigate the @xmath342 regime .
we turn now to an approximate method more adapted to this situation .
let us denote by @xmath20 an assignment of the boolean variables .
prwsat defines a markov process on the space of the configurations @xmath20 , a discrete set of cardinality @xmath343 .
it is a formidable task to follow the probabilities of all these configurations as a function of the number of steps @xmath61 of the algorithm so one can look for a simpler description of the state of the system during the evolution of the algorithm .
the simplest , and crucial , quantity to follow is the number of clauses unsatisfied by the current assignment of the boolean variables , @xmath344 . indeed , as soon as this value vanishes , the algorithm has found a solution and stops .
a crude approximation consists in assuming that , at each time step @xmath61 , all configurations with a given number of unsatisfied clauses are equiprobable , whereas the hamming distance between two configurations visited at step @xmath61 and @xmath345 of the algorithm is at most @xmath204 .
however , the results obtained are much more sensible that one could fear . within this simplification , a markovian evolution equation for the probability that @xmath346 clauses are unsatisfied after @xmath61 steps can be written . using methods similar to the ones in section  [ dpllsatsection ] , we obtain ( see @xcite for more details and @xcite for an alternative way of presenting the approximation ) :    * the average fraction of unsatisfied clauses , @xmath347 , after @xmath348 steps of the algorithm . for ratios @xmath349 ,
@xmath319 remains positive at large times , which means that typically a large formula will not be solved by prwsat , and that the fraction of unsat clauses on the plateau is @xmath350 .
the predicted value for @xmath351 , @xmath352 , is in good but not perfect agreement with the estimates from numerical simulations , around @xmath353 .
the plateau height , @xmath354 , is compared to numerics in fig .
[ wsat_plateau ] . * the probability @xmath321 that the fraction of unsatisfied clauses is @xmath310 .
it has been argued above that the distribution of resolution times in the @xmath355 phase is expected to be , at leading order , an exponential distribution of average @xmath356 , with @xmath357 .
predictions for @xmath358 are plotted and compared to experimental measures of @xmath168 in fig .
[ wsat_fig_zeta ] . despite the roughness of our markovian approximation , theoretical predictions are in qualitative agreement with numerical experiments .
a similar study of the behaviour of prwsat on xorsat problems has been also performed in  @xcite , with qualitatively similar conclusions : there exists a dynamic threshold @xmath329 for the algorithm , smaller both than the satisfiability and clustering thresholds ( known exactly in this case  @xcite ) . for low values of @xmath25 ,
the resolution time is linear in the size of the formula ; between @xmath329 and @xmath330 resolution occurs on exponentially large time scales , through fluctuations around a plateau value for the number of unsatisfied clauses . in the xorsat case , the agreement between numerical experiments and this approximate study ( which predicts @xmath359 ) is quantitatively better and seems to improve with growing @xmath3 .
in this article , we have tried to give an overview of the studies that physicists have devoted to the analysis of algorithms .
this presentation is certainly not exhaustive .
let us mention that use of statistical physics ideas have permitted to obtain very interesting results on related issues as number partitioning@xcite , binary search trees @xcite , learning in neural networks @xcite , extremal optimization @xcite
...    it may be objected that algorithms are mathematical and well defined objects and , as so , should be analysed with rigorous techniques only . though this point of view should ultimately prevail
, the current state of available probabilistic or combinatorics techniques compared to the sophisticated nature of algorithms used in computer science make this goal unrealistic nowadays .
we hope the reader is now convinced that statistical physics ideas , techniques , ... may be of help to acquire a quantitative intuition or even formulate conjectures on the average performances of search algorithms .
a wealth of concepts and methods familiar to physicists e.g. phase transitions and diagrams , dynamical renormalization flow , out - of - equilibrium growth phenomena , metastability , perturbative approaches ... are found to be useful to understand the behaviour of algorithms .
it is a simple bet that this list will get longer in next future and that more and more powerful techniques and ideas issued from modern theoretical physics will find their place in the field .
open questions are numerous .
variants of dpll with complex splitting heuristics , random backtrackings@xcite or applied to combinatorial problems with internal symmetries@xcite would be worth being studied .
as for local search algorithms , it would be very interesting to study refined versions of the pure walksat procedure that alternate random and greedy steps @xcite to understand the observed existence and properties of optimal strategies .
one of the main open questions in this context is to what extent performances are related to intrinsic features of the combinatorial problems and not to the details of the search algorithm@xcite .
this raises the question of how the structure of the cost function landscape may induce some trapping or slowing down of search algorithms@xcite .
last of all , the input distributions of instances we have focused on here are far from being realistic .
real instances have a lot of structure which will strongly reflect on the performances of algorithms .
going towards more realistic distributions or , even better , obtaining results true for any instance would be of great interest .
d. mitchell , b. selman , h. levesque , hard and easy distributions of sat problems , _ proc .  of the tenth natl .
conf .  on artificial intelligence ( aaai-92 )
_ , 440 - 446 , the aaai press / mit press , cambridge , ma ( 1992 ) .
j. gu , p.w .
purdom , j. franco , b.w .
wah , algorithms for satisfiability ( sat ) problem : a survey .
_ dimacs series on discrete mathematics and theoretical computer science _ * 35 * , 19 - 151 , american mathematical society ( 1997 )
.        for a review on the analysis of search heuristics in the absence of backtracking , see : + d. achlioptas , lower bounds for random 3-sat via differential equations , _ theor .
sci . _ * 265 * , 159 - 186 ( 2001 ) .
kaporis , l.m .
kirousis , e.g. lalas , the probabilistic analysis of a greedy satisfiability algorithm , _ proceedings of sat 2002 _ , pp 362376 ( 2002 ) , available at http://gauss.ececs.uc.edu/conferences/sat2002/abstracts/kaporis.ps    a.c .
kaporis , l.m .
kirousis , y.c .
stamatiou , how to prove conditional randomness using the principle of deferred decisions , technical report , computer technology institute , greece ( 2002 ) , available at http://www.ceid.upatras.fr/faculty/kirousis/kks-pdd02.ps      r. monasson , r. zecchina , s. kirkpatrick , b. selman , l. troyansky , 2+p - sat : relation of typical - case complexity to the nature of the phase transition , _ random structure and algorithms _ * 15 * , 414 ( 1999 ) .
s. cocco , r. monasson , trajectories in phase diagrams , growth processes and computational complexity : how search algorithms solve the 3-satisfiability problem , _ phys .
lett . _ * 86 * , 1654 ( 2001 ) ; analysis of the computational complexity of solving random satisfiability problems using branch and bound search algorithms , _ eur . phys . j. b _ * 22 * , 505 ( 2001 ) .
s. cocco , r. monasson , exponentially hard problems are sometimes polynomial , a large deviation analysis of search algorithms for the random satisfiability problem , and its application to stop - and - restart resolutions , _ phys .
e _ * 66 * , 037101 ( 2002 ) ; restart method and exponential acceleration of random 3-sat instances resolutions : a large deviation analysis of the davis  putnam  loveland  logemann algorithm , to appear in amai ( 2003 ) .
s. boettcher , m. grigni , jamming model for the extremal optimization heuristic , _ j. phys .
a _ * 35 * , 1109 - 1123 ( 2002 ) .
+ s. boettcher , a.g .
percus , extremal optimization : an evolutionary local - search algorithm , in computational modeling and problem solving in the networked world , eds .
h. m. bhargava and n. ye ( kluver , boston , 2003 ) .
l. baptista , j.p .
marques - silva , using randomization and learning to solve hard real - world instances of satisfiability , _ in international conference of principles and practice of contsraint programming _
, 489491 ( 2000 ) .