as the number of online resources and web documents continues to increase , the need for better organizational structures that guide readers towards the information they seek increases .
hierarchies and taxonomies are invaluable tools for this purpose .
taxonomies are widely used in libraries via the library of congress system or the dewey decimal system , and hierarchies were a fixture of the early world wide web ; perhaps the most famous example being the yahoo search engine , which originally was a taxonomic - collection of hyperlinks organized by topic .
these systems were developed because their effectiveness at topical organization and their logarithmic depth allowed users to quickly find the relevant documents for which they were searching .
unfortunately , taxonomy curation of documents , articles , books , etc .
is mostly a manual process , which is only possible when the number of curated documents is relatively small .
this process becomes increasingly impractical as the number of documents grows to web - scale , and has motivated research towards the automatic inference of taxonomies  @xcite .
most document repositories contain linkages between the documents creating a _ document - graph_. these links provide , among other things , proper context to the terms and topics in each document .
document - graphs are especially common in nonfiction and scientific literature , where citations are viewed as inter - document links .
similarly , the world wide web ( www ) can be considered to be a single , very large document - graph , where web pages represent documents and hyperlinks link documents .
web sites on the www could also be considered document graphs because a web site is simply a subgraph of the www .
web site subgraphs , in particular , are a collection of documents with a specific and purposeful organizational structure , which are often carefully designed to guide the user from the entry page , _
i.e. _ , a homepage , to progressively more specific web pages .
similarly , scientific literature can be categorized into a hierarchy of increasingly specific scientific topics by their citation links , and encyclopedia articles can be categorized into a hierarchy of increasingly specific articles by their cross references .
thus , we assert that most document - graphs , or , more generally , information networks , contain hidden node hierarchies .      in this paper
we draw specific distinctions between a _ hierarchy _ and
a _ taxonomy_. a taxonomy is defined to be a classification of objects into increasingly finer granularities , where each non - leaf node is a conceptual combination of its children .
a biological taxonomy is a canonical example of this definition because a classified species , say ( _ i.e. _ , humans ) , can only be placed at a leaf in the taxonomy ; the inner nodes , _
e.g. _ , , , , do not declare new species , rather they are conceptual agglomerations of species .
furthermore , each species is described by its path through the taxonomy .
for example , s , can be more generally described as s , s and as s ( among others ) .
a hierarchy , on the other hand , is an arrangement of objects where some objects are considered to be _ above _ , _ below _ or _ at the same level as _ others .
this necessarily means that objects of a hierarchy live at the internal nodes .
for example , a business , government or military chain of command is a hierarchy because the is a specific object that is above the s , which are above the s , and so on . in this case ,
high - level nodes like are not agglomerations of s just as the is not an aggregation of s or s.      in the similar studies , `` document hierarchies '' were not actually hierarchies of documents in the literal sense .
for example , hierarchical lda ( hlda )  @xcite , topicblock  @xcite , and the tree structured stick breaking ( tssb ) model  @xcite learn a conceptual taxonomy in which the non - leaf topics are a combination of words and does not represent a real document in the corpus ; the hierarchical pachinko allocation model ( hpam )  @xcite constructs a tree - like conceptual taxonomy like hlda , but where each topic can have multiple parents .    in these related models , only the leaves contained the actual , literal documents .
contrary to our perspective , the internal nodes of existing models contain ephemeral word - topic distributions , rather than actual documents .
see figure  [ fig : inh_relatedwork ] for a brief comparison of model outputs .
the hdtm model introduced in this paper requires that inner nodes , which in previous work are made of ephemeral distributions , be literal documents .
this requirement asserts that _ some documents are more general than others_. here we explore this assertion through examples and a review of similar assertions made in previous research .
a web site @xmath0 can be viewed as a directed graph with web pages as vertices @xmath1 and hyperlinks as directed edges @xmath2 between web pages @xmath3  excluding inter - site hyperlinks . in most cases , designating the web site entry page as the root @xmath4 allows for a web site to be viewed as a rooted directed graph .
web site creators and curators purposefully organize the hyperlinks between documents in a topically meaningful manner . as a result
, web documents further away from the root document typically contain more specific topics than web documents graphically close to the root document .
for example , the web site at the university of notre dame , shown in figure  [ fig : inh_webtree ] contains a root web document ( the entry page ) , and dozens of children web documents .
even with a very small subset of documents and edges , the corresponding web graph can be quite complicated and messy .
a breadth first traversal of the web graph starting with the root node is a simple way to distill a document hierarchy from the web graph .
unfortunately , a fixed breadth - first hierarchy can not account for many of the intricacies of real world web graphs .        for explanation purposes ,
let s assume there are four types of hyperlink edges in a web site : ( 1 ) parent - to - child links , ( 2 ) upward links , ( 3 ) shortcuts , and ( 4 ) cross - topic links .
parent - to - child links direct the user from one web page to a more topically specific web page ; _ e.g.
_ , a hyperlink from engineering.nd.edu to cse.nd.edu is a parent - to - child hyperlink because computer science is topically more specific than engineering .
upward links are hyperlinks that reference a more general document ; _ e.g. _ , there may exist a hyperlink from engineering.nd.edu to /the - arts because the engineering college would like to reference artistic happenings at the university .
shortcut links are hyperlinks that skip from very general web documents to very specific web documents as a way of featuring some specific topic ; _ e.g. _ , if a computer science professor wins a prestigious award or grant , the professor s web page may be linked to from the news section of the root web page . cross topic links are hyperlinks that move across topical subtrees ; _
e.g. _ , the college of science may reference some working relationship with the college of engineering by creating a hyperlink between the two web pages .    because the goal is to infer the document hierarchy ,
the hdtm model is , in a sense , trying to find parent - to - child links . in the event that there is more than one parent - to - child link to a particular web page , the goal is to find the best topically - relevant parent for each web document in the inferred hierarchy ( except the root ) .
web researchers and practitioners have used hyperlink structures to organize web documents for many years .
the pagerank and hits algorithms are two of the most famous examples of information propagation through links .
pagerank , for instance , uses the model of a random web surfer ( _ i.e. _ random walker ) , who randomly follows hyperlinks over the web ; a current measure of a web page s authority corresponds to the probability that a random surfer lands upon that web page  the pagerank score . in the hdtm model ,
pagerank s notion of authority loosely corresponds to topical generality , that is , web pages with a high random surfer probability are likely to be topically more general than others .
the document - graph structure is also used to enrich the document description by adding features to improve retrieval performance .
some of the intuition behind these previous works are helpful in framing the generative model .
a limitation of the random walker model is that it only looks at the graphical structure of the network .
the word distributions found in each document are clearly an important factor to consider when generating document hierarchies .
previous work by song , _
et al . _
@xcite and qin , _ et al . _
@xcite show that a given web page can be enriched by propagating information from its children .
their relevance propagation model modifies the language distribution of a web page to be a mixture of itself and its children according to the formula :    @xmath5    where @xmath6 is the frequency of term @xmath7 in a document @xmath8 before propagation , @xmath9 is the frequency of term @xmath7 in document @xmath8 after propagation , @xmath10 is a child page of @xmath8 in the sitemap @xmath11 , and @xmath12 is a parameter to control the mixing factor of the children .
this propagation algorithm assumes that the sitemap , @xmath11 , is constructed ahead of time .    for the purposes of web information retrieval language models
are often used to normalize and smooth word distributions . for illustration purposes ,
we apply a dirichlet prior smoothing function  @xcite to smooth the term distribution where the @xmath9 from above is used in place of the usual @xmath13 from the original dirichlet prior smoothing function yielding :    @xmath14    where @xmath15 is the distribution over all terms in @xmath1 , @xmath16 is the smoothing parameter , and the length is modified by the propagation algorithm to be @xmath17 .    as a result of the upward propagation @xmath18 function
, the root document ( web site entry page ) will contain all of the words from all of the web pages in the web site with different non - zero probabilities .
the most probable words are those that occur most frequently and most generally across all documents , and are thus propagated the most .    as a small , preliminary example
, table  [ tab : inh_lm ] shows the top six most probable words in the top document ( via text propagation ) and in root topics of hlda and hdtm of the computer science department s web site at the university of notre dame .
this small example reinforces the intuition that certain web sites have a hidden hierarchical topical structure .    in the previous term propagation work ,
web site sitemaps were constructed ahead of time using url heuristics or manually .
the goal of hdtm is to learn the document hierarchy automatically and in conjunction with the topical hierarchy .
documents from many different collections exist in hidden hierarchies . while technically a web site , wikipedia documents and categories form a unique document graph .
wikipedia categories are especially interesting because they provide a type of ontology wherein categories have more specific sub - categories and more general parent - categories .
most wikipedia articles are are represented by at least one category description ; this allows for users to drill down to relevant articles in a very few number of clicks by browsing the category graph .
a partial example of the wikipedia category graph is shown in figure  [ fig : inh_wikicat ] .
this figure illustrates how a document graph can be construed into a document hierarchy .
specifically , the wikipedia community has hand - crafted a category hierarchy , represented by colored circles and edges , on top of the article - graph , represented by grey squares and grey edges .
although the category graph is imperfect and incomplete , wikipedia browsers can get a sense of the granularity , topic and fit simply by viewing the article s placement in the category hierarchy . with the wikipedia construction in mind
, our goal can be loosely interpreted as automatically inferring the wikipedia category hierarchy from the wikipedia article graph ; this inference is tried and evaluated in section  [ sec : inh_experiments ] .
bibliographical networks may also be hierarchically structured . in a bibliographic network , papers or authors
( wherein each author could be a collection of documents ) are represented by nodes and each citation is represented by an edge in the graph .
apart from web and citation graphs , bioinformatics networks , for example protein networks , can also be hierarchically organized for protein fold prediction  @xcite .
the nodes in such network are proteins and two proteins are connected if they have structural or evolutionary relationships  @xcite .
the goal of this work is to construct node hierarchies from an information network using the node features ( _ e.g. _ , text ) and inter - node edges . for these purposes a node which appears _ above _ , _ below _ or _ at the same level as _ another node refers to the conceptual granularity of the nodes . in other words ,
given an information network with an explicitly identified root , such as a web site homepage , we aim to learn a node - hierarchy that best captures the conceptual hierarchy of the document - graph .
this problem poses three technical challenges :    1 .
* document - topic inference*. in document hierarchies the parent documents consist of topics that are more general than their children .
this requires that the parent documents are viewed as a mixture of the topics contained within its children , and children documents should topically fit underneath their selected parent . in this paper
we introduce the _ hierarchical document - topic model _ ( hdtm ) which generates a course - to - fine representation of the text information , wherein high - level documents live near the top of the hierarchy , and low - level , more specific documents live at or near the leaves .
2 .   * selecting document placement*. placement of a document within the hierarchy drives the topic mixing during inference . because links between edges hint at the context of the relationship between documents , the document placement in the inferred hierarchy is constrained by their edges within the original document - graph . in other words ,
if an edge exists in the final , inferred hierarchy , then it must also exist in original document - graph ( not vice versa ) .
unlike existing models , such as hlda  @xcite , that select topic paths using the nested chinese restaurant process ( ncrp ) , hdtm performs document placement based on a stochastic process resembling random walks with restart ( rwr ) over the original document - graph .
the use of a stochastic process over the document - graph frees the algorithm from rigid parameters ; perhaps most importantly the adoption of the rwr stochastic process instead of ncrp allows documents to live at non - leaf nodes , and frees the algorithm from the depth parameter of hlda .
* analysis at web site - scale*. in many document - graph collections , the number of edges grows quadratically with the number of nodes .
this limits the scalability of many topic diffusion algorithms  @xcite .
an important side - effect of the rwr process is the ability to adapt the hdtm model inference algorithm into a distributed graph processing system that is capable of processing billion - node graphs .
the remainder of this paper is organized as follows : after reviewing the related work we introduce the hdtm model and show how inference is performed .
next we show how the inference can be adapted to a large scale graph processing system in order to run on web - scale data sets .
the experiments section describes a myriad of tests that were run on various data sets and the quantitative and qualitative evaluations that were performed .
we conclude with a examples of real - world use cases , discuss avenues for future research , and provide a link to hdtm s source code .
initial efforts in hierarchical clustering used greedy heuristics such as single - link or complete - link agglomoration rules to infer dendrograms  @xcite , in which the root node is split into a series of branches that terminate with a single document at each leaf . ho , _ et al .
_ , point out that manually - curated web hierarchies like the open directory project are typically flatter and contain fewer inner nodes than agglomerative clustering techniques produce  @xcite .
other hierarchical clustering algorithms include top - down processes which iteratively partition the data  @xcite , incremental methods like cobweb  @xcite , classit  @xcite , and other algorithms optimized for hierarchical text clustering .
the processes that typically defines most hierarchical clustering algorithms can be made to fit in a probabilistic setting that build bottom - up hierarchies based on bayesian hypothesis testing  @xcite .
on the other hand , a lot of recent work uses bayesian generative models to find the most likely explanation of observed text and links .
the first of these hierarchical generative models was hierarchical latent dirichlet allocation ( hlda ) . in hlda
each document sits at a leaf in a tree of fixed depth @xmath19 as illustrated in figure  [ fig : rwhlda ] .
note that all non - leave nodes in figure  [ fig : rwhlda ] are conceptual topics containing word distribution instead of a document .
each document is represented by a mixture of multinomials along the path through the taxonomy from the document to the root .
documents are placed at their respective leaf nodes stochasically using the nested chinese restaurant process ( ncrp ) along side an lda - style word sampling process .
ncrp is a recursive version of the standard chinese restaurant process ( crp ) , which progresses according to the following analogy : an empty chinese restaurant has an infinite number of tables , and each table has an infinite number of chairs .
when the first customer arrives he sits in the first chair at the first table with probability of 1 .
the second customer can then chose to sit at an occupied table with probability of @xmath20 or sit at a new , unoccupied table with probability of @xmath21 , where @xmath22 is the current customer , @xmath23 is the number of customers currently sitting at table @xmath24 , and @xmath25 is a parameter that defines the affinity to sit at a previously occupied table .
the nested version of the crp extends the original analogy as follows : at each table in the chinese restaurant are cards with the name of another chinese restaurant .
when a customer sits at a given table , he reads the card , gets up and goes to that restaurant , where he is reseated according to the crp .
each customer visits @xmath19 restaurants until he is finally seated and is able to eat .
this process creates a tree with a depth of @xmath19 and a width determined by the @xmath25 parameter .
this process has also been called the chinese restaurant franchise because of this analogy  @xcite .
adams , _ et al .
_ proposed a hierarchical topic model called tree structured stick breaking ( tssb ) , illustrated in figure  [ fig : rwtssb ] , wherein documents can live at internal nodes , rather than exclusively at leaf nodes@xcite .
however , this process involves chaining together conjugate priors which makes inference more complicated , and it also does not make use of link data .    other work along this line include hierarchical labeled lda ( hllda ) by petinot _
et al . _
@xcite hllda , as well as fixed structure lda ( fslda ) by reisinger and pasca  @xcite which modify hlda by fixing the hierarchical structure and learning hierarchical topic distributions .
the hierarchical pachinko allocation model(hpam ) , shown in figure  [ fig : rwhpam ] , produces a directed acyclic graph ( dag ) of a fixed depth allowing for each internal ( non - document ) node to be represented a mixture of more abstract , _ i.e. _ , higher level , topics  @xcite .    in network - only data , community discovery is the process of finding self - similar group , or clusters .
the shrink algorithm creates hierarchical clusters by identifying tightly - knit communities and by finding disparate clusters by looking for hubs and other heuristics  @xcite .
clauset , _ et al _ , discover dendrograms by monte carlo sampling  @xcite ; however , dendrograms poorly represent the manually curated hierarchies and taxonomies that we are pursuing .
stochastic block models ( sbm ) are an alternative line of network clustering research that partitions nodes into communities in order to generatively infer link probabilities  @xcite .
several extensions to the original sbm have since been proposed ( for a survey see  @xcite ) .
one downside to block - model processes is that they assign probabilities to every possible edge requiring @xmath26 complexity in every sampling iteration .
furthermore , sbm methods typically are not concerned with topical / conceptual properties of the nodes .
because hdtm merges document text and inter - document links into a single model , we assume that the words and their latent topics fit within the link structure of the graph , and that the graph structure explains topical relationships between interlinked documents .
topic modeling with network structure ( tmn ) is similar in this regard because it regularizes a statistical topic model with a harmonic regularizer based on the graph structure in the data ; the result is that topic proportions of linked documents are similar to each other  @xcite .
however , hierarchical information is not discovered nor can be easily inferred from this model .
topic - sensitive pagerank combines document topics with the pagerank algorithm , by arguing that the pagerank score of a document ought to be influenced by its topical connection to the referring document  @xcite . like tmn model
, topic - senstive pagerank does not construct any type of information network hierarchy .
other work on generative models that combine text and links include : a probabilistic model for document connectivity  @xcite , the link - plsa - lda and pairwise - link - lda methods  @xcite , the latent topic model for hypertext ( lthm ) method  @xcite , role discovery in social networks  @xcite , the author - topic - model  @xcite , and others .
the above models operate by encoding link probability as a discrete random variable or a bernoulli trial that is parameterized by the topics of the documents .
the relational topic model ( rtm ) builds links between topics , where observed links are given a very high likelihood  @xcite .
the topicblock model combines the non - parametric hlda and stochastic block models  @xcite to generate document taxonomies from text and links  @xcite ; however , topicblock , like hlda , does not permit documents to reside at non - leaf nodes of the resulting tree .
to apply topic modeling algorithms on web - scale data , several parallel algorithms have been introduced .
newman , _ et al _ proposed an exact distributed gibbs sampling algorithm as well as an approximate distributed gibbs sampling algorithm that uses local gibbs sampling and global synchronization  @xcite .
smyth , _ et al _ introduced an asynchronous distributed algorithm that was capable of learning lda - style topics  @xcite ; and ahmed , _
et al _ recently released yahoo lda , which is a scalable approximate inference framework on large - scale streaming data  @xcite .
these parallel approaches use the conventional , procedural programming paradigms , and as a result can not guarantee statistically sound samples from their gibbs iterations . although conventional parallel and distributed algorithms
indeed divide the document set into smaller groups , the maximum number of subgroups is subject to the number of processors .
in contrast , the vertex - programming paradigm , _ e.g. _ , pregel , graphlab and graphx  @xcite , can distribute sampling operations at a much finer granularity by treating each graph - node as an independent computing unit .    although the distributed variants of topic inference have made significant contributions in large scale topic models on the one hand , and large scale graph processing on the other hand , we are unaware of any parallel algorithm capable of joining these two subjects to infer topical hierarchies on large scale information networks .
in contrast to the previous work , hdtm builds a hierarchy of documents from text and inter - document links . in this model , each node in the hierarchy contains a single document , and the hierarchy s width and depth is not fixed .
the distributed version of proposed algorithm has the ability to handle graph with millions of nodes and billions of tokens .
the problem of inferring the document hierarchy is a learning problem akin to finding the single , best parent for each document - node . unlike previous algorithms , which discover latent topic taxonomies ,
the hierarchical document - topic model ( hdtm ) finds hidden hierarchies by selecting edges in the document graph .
this section presents a detailed description of the model .
beginning with a document graph @xmath27 of documents @xmath1 and edges @xmath2 .
each document is a collection of words , where a word @xmath7 is an item in a vocabulary .
the basic assumption of hdtm and similar models is that each document can be generated by probabilistically mixing words from among topics .
distributions over topics are represented by @xmath28 , which is a multinomial variable with an associated set of distributions over words @xmath29 , where @xmath30 is a dirichlet hyper - parameter .
document - specific mixing proportions are denoted by the vector @xmath31 .
parametric - bayes topic models also include a @xmath32 parameter that denotes the number of topics , wherein @xmath28 is one of @xmath32 possible values and @xmath31 is a @xmath32-d vector .
hdtm , and other non - parametric bayesian models , do not require a @xmath32 parameter as input
. instead , in hdtm there exist @xmath33 topics , one for each graph node , and each document is a mixture of the topics on the path between itself and the root document .    in the original lda model ,
a single document mixture distribution is @xmath34 .
the process for generating a document is ( 1 ) choose a @xmath31 of topic proportions from a distribution @xmath35 , where @xmath35 is a dirichlet distribution ; ( 2 ) sample words from the mixture distribution @xmath36 for the @xmath31 chosen in step 1 .
hlda is an extension of lda in which the topics are situated in a taxonomy @xmath37 of fixed depth @xmath19 .
the hierarchy is generated by the nested chinese restaurant process ( ncrp ) which represents @xmath31 as an @xmath19-dimensional vector , defining an @xmath19-level path through @xmath37 from root to document .
because of the ncrp process , every document lives at a leaf and the words in each document are a mixture of the topic - words on the path from it to the root .      the ncrp stochastic process could not be used to infer document hierarchies because the ncrp process forces documents to the leaves in the tree .
hdtm replaces ncrp with random walk with restart ( rwr ) ( which is also known as personalized pagerank ( ppr ) )  @xcite .
in contrast , random walk with teleportation ( aka pagerank ) random walks by selecting a random starting point , and , with probability @xmath38 , the walker randomly walks to a new , connected location or chooses to jump to a random location with probability @xmath25 , where @xmath25 is called the jumping probability , however , this would be ambiguous with the dirichlet hyper - parameter @xmath12 . ] .    in hdtm ,
the root node is fixed , either as the entry page of a web site , by some other heuristic or manually .
therefore , for the purposes of hierarchy inference , the random walker is forced to start and restart at the root node .
[ alg : rwr ]    ( ) @xmath39 ( ) @xmath40 @xmath41    say we wish to find the rwr - probability between some node @xmath42 and some target node @xmath43 .
we model this by a random walker visiting document @xmath42 at time @xmath44 . in the next time step ,
the walker chooses a document @xmath45 from among @xmath42
s outgoing neighbors @xmath46 in the hierarchy @xmath37 uniformly at random . in other words , at time @xmath47 , the walker lands at node @xmath48 with probability @xmath49 , where @xmath50 is the outdegree of some document @xmath51 .
if at any time , there exists an edge to @xmath52 , _
i.e _ , an edge between the current node @xmath42 and the target node @xmath43 in the original graph @xmath0 , then we record the probability of that new path possibility for later sampling . alg .
[ alg : rwr ] describes this process algorithmically .
this procedure allows for new paths from the root @xmath53 to be probabilistically generated based on the current hierarchy effectively allowing for documents to migrate up , down and through the hierarchy during sampling .      because a document hierarchy is a tree
, each document - node can only have one parent . selecting a path for a document @xmath8 in the graph @xmath0
is akin to selecting a parent @xmath54 ( and grandparents , etc . ) from @xmath55 in the document graph @xmath0 .
hdtm creates and samples from a probability distribution over each documents parent , where the probability of document @xmath42 being the parent of @xmath8 is defined as :    @xmath56    where @xmath57 is the walkers current position at time @xmath44 , @xmath58 is the depth of @xmath8 in @xmath37 , and @xmath59 is the outdegree of @xmath57 in the hierarchy @xmath37 . in other words , the probability of landing at @xmath8 is the product of the emission probabilities from each document in the path through @xmath37 from @xmath4 to @xmath8 .
this random walker function assigns higher probabilities to parents that are at a shallower depth than those at deeper positions .
this is in line with the intuition that flatter hierarchies are easier for human understanding than deep hierarchies  @xcite .
simply put , the restart probability @xmath25 controls how much resistance there is to placing a document at successive depths .
algorithmically , hdtm infers document hierarchies by drawing paths @xmath60 from the @xmath4 to the document @xmath8 .
thus , the documents are drawn from the following generative process :    1 .
each document @xmath61 is assigned a topic @xmath62 dir(@xmath63 ) : 2 .   for each document @xmath61
: 1 .   draw a path @xmath64 rwr(@xmath25 ) 2 .   draw an @xmath19-dim topic proportion vector @xmath31 from dir(@xmath12 ) , where @xmath65len@xmath66 .
3 .   for each word
@xmath67 : 1
.   choose topic @xmath68 mult(@xmath69 ) .
choose word @xmath70 mult(@xmath71 ) , where @xmath72 is the topic in the @xmath28th position in @xmath60 .    in this
generative process hierarchical nodes represent documents _ and _ topics , where internal nodes contain the shared terminology of its descendants .
the illustration in figure  [ fig : topicdocuments ] shows the two potential outputs from a three - node graph of the wikipedia articles : , and .
clearly , data mining and machine learning could both be regarded as children of computer science , but there is indeed a wiki - link from to in the graph .
there are two key ideas conveyed in this illustration .
the first is that the hierarchy on the left picks edges so as to ignore the to wiki - link ( denoted as a thin - grey line ) , whereas the hierarchy on the right picks edges so as to ignore the to wiki - link .
thus , in this toy example , there are two possible hierarchy outcomes presented : the one on the left and the one on the right in figure  [ fig : topicdocuments ] .    the second idea
conveyed in this illustration is that the word - topic distributions ( in colored circles ) are stored along with documents words ( in black squares ) at the same position in the hierarchy .
the word distribution within each topic is constrained such that a document may only propagate terms upwards through the currently selected / sampled hierarchical topology .
thus , it is impossible , in the hierarchy on the left , for the topic distribution at to contain a word unique to or .
it is similarly impossible , in the hierarchy on the right , for the topic distribution in to contain a word unique to even though this is indeed possible in the hierarchical topology on the left .    like in earlier models
, there is statistical pressure to have more general terms in topics towards the root of the hierarchy .
this is because every path in the hierarchy includes the root node and there are more paths through nodes at higher levels than through nodes at lower levels .
moving down the tree , the topics , and therefore the documents , become more specific .
hyperparameters also play an important role in the shape and character of the hierarchy .
the @xmath12 parameter affects the smoothing on topic distributions , and the @xmath63 parameter affects the smoothing on word distributions .
the @xmath25 parameter is perhaps the most important parameter because it affects the depth of the hierarchy .
specifically , if @xmath25 is set to be large ( _ e.g. _ , @xmath73 ) then resulting hierarchy is shallow . low values ( _ e.g. _ , @xmath74 ) may result in deep hierarchies , because there is a smaller probabilistic penalty for each step that the random walker takes .
exact inference on this model is intractable , so an approximation technique for posterior inference is used .
the gibbs sampling algorithm is ideal in this situation because it simultaneously allows exploration of topic distributions and potential graphical hierarchies .
the variables needed by the gibbs sampler are : @xmath75 , the @xmath22th word in document @xmath8 ; @xmath76 , the assignment of the @xmath22th word in document @xmath8 ; and @xmath77 , the topic corresponding to document at the @xmath28th level .
the @xmath31 and @xmath30 variables are integrated out forming a collapsed gibbs sampler .
the sampling is performed in two parts : ( 1 ) given the current level allocations of each word @xmath76 sample the path @xmath77 , ( 2 ) given the current state of the hierarchy , sample @xmath76 .
in other words , we use the topic distributions to inform the path selections that make up the hierarchy , and the hierarchy topology to inform the topic distributions .
the first gibbs sampling step is to draw a path from each document to the root through the graph .
the sampling distribution for a path @xmath78 is @xmath79 where @xmath80 is the count of terms in document @xmath8 , and @xmath81 are the words without document @xmath8 .
this equation is an expression of bayes theorem where the first term represents the probability of data given some choice of path from the root , and the second term represents the probability of selecting some path .
specifically , the second term represents the probability of drawing the path @xmath82 to document @xmath8 at depth @xmath43 from the rwr process .
recall that each node has an emission probability of @xmath83 , and a restart probability of @xmath25 .
the probability is defined recursively : @xmath84 in other words , the probability of reaching @xmath8 is equal to the probability of a random walker with restart probability @xmath25 being at document @xmath8 at time @xmath43 .    the first term in eq .
[ eq : pathsample ] is the probability of a given word based on the current path @xmath85 and topic assignment @xmath86 : @xmath87 + w\eta } ) } { \prod_w { \gamma ( \#[\mathbf{c}_{-d , k } = c_{d , k } , \mathbf{w}_{-d } = w ] + \eta } ) } } \times \\ & \quad\hspace{1.1 cm } \frac { \prod_w{\gamma ( \#[\mathbf{z } = k , \mathbf{c}_{k } = c_{d , k } , \mathbf{w } = w ] + \eta ) } } { \gamma ( \sum { \#[\mathbf{z } = k , \mathbf{c}_{k } = c_{d , k } , \mathbf{w } = w ] } + w\eta ) } ,   \label{eq : pwgivenc } \end{split}\ ] ]    where @xmath88 $ ] counts the elements of an array that satisfy the given condition , and @xmath89 is the maximum depth of the current hierarchy state .
the expression @xmath90 $ ] counts : ( ii ) @xmath91 , _ i.e. _ , the number of words @xmath7 that do not appear in @xmath8 , for each ( i ) @xmath92 , _
i.e. _ , the number of paths to the current document @xmath8 except those where the path length is @xmath43 . the expression @xmath93 counts : ( iii ) @xmath94 , _
i.e. _ , the number of words @xmath7 such that , ( ii ) @xmath95 , _ i.e. _ , the words appear in document @xmath8 and are situated at the end of a path of length @xmath43 , where ( i ) @xmath96 , _ i.e. _ , @xmath43 is one of the topics in @xmath86 .
@xmath97 is the size of the vocabulary .
[ eq : pwgivenc ] is adapted from the standard ratio of normalizing constants for the dirichlet distribution  @xcite .
given the current state of all the variables , the word sampler must first pick an assignment @xmath28 for word @xmath22 in document @xmath8 . the sampling distribution of @xmath76 is @xmath98 where @xmath99 and @xmath100 . the first term is a distribution over word assignments : @xmath101 + \eta \end{split}\ ] ] which is the @xmath63-smoothed frequency of seeing word @xmath75 in the topic at level @xmath76 in the path @xmath78 .
the second term is the distribution over levels @xmath102}{\#[\mathbf{z}_{d ,- n } \ge j ] } } \right ) \times \\ & \frac{1-\gamma}{\mathrm{deg}_t(d_{k-1 } ) } \frac{\#[\mathbf{z}_{d ,- n } = k]}{\#[\mathbf{z}_{d ,-
n } \ge k ] } , \end{split } \label{eq : abuse}\ ] ] where @xmath88 $ ] is the number of elements in the vector which satisfy the given condition .
[ eq : abuse ] abuses notation so that the product from @xmath103 to @xmath104 combines terms representing nodes at the @xmath105th level in the path @xmath85 down to the parent of @xmath106 , and the second set of terms represents document @xmath106 at level @xmath43 .
the @xmath107 symbol in eq .
[ eq : abuse ] refers to terms representing all ancestors of a particular node , and @xmath108 refers to the ancestors of a node including itself .
a common complaint among data science practitioners is that graphical models , especially , non - parametric bayesian graphical models , do not perform well at scale . with this in mind
, we also implemented the hdtm inference algorithm in the scalable , distributed vertex - programming paradigm .
the mechanism behind gibbs sampling , and other markov chain monte carlo methods , requires sequential sampling steps , and execution of each step depends on the results of the previous step making gibbs samplers , and mcmc method in general , difficult to parallelize .
approximate distributed lda ( ad - lda ) is one attempt to find approximate , distributed solutions to the serial inference problem by dividing documents into @xmath109 parts where @xmath109 is the number of processors and initializes the topic distribution @xmath28 globally .
then , for every gibbs iteration , each processor samples @xmath110 of the dataset using the @xmath111 from last gibbs sampling iteration .
when all processors are finished , a global synchronization is performed and @xmath28 is updated  @xcite .    following the distribution lessons from ad - lda we sought to also implement a scalable , distributed version of hdtm . however , a major difference between the lda / hlda and hdtm is that hlda uses the ncrp stochastic process to assign terms to topics , while hdtm samples paths from a graph of documents using graph - based random walk with restart method .
the process of random walks over the network topology combined with the term sampling process described above is a good candidate for the vertex - programming paradigm using frameworks like pregel  @xcite or graphlab  @xcite .
although mapreduce is a widely used , general purpose parallel scheme that can easily deal with scalable data , it is not optimized for _ iterative _ computational tasks such as statistical inference or logistic regression  @xcite .
this is because mapreduce materializes all intermediate results and to disk in order to tolerate task failures .
mapreduce , therefore , has relatively high i / o costs compared to other designs that keep data in memory across iterations  @xcite .
apart from mapreduce , another scalable solution is to build a custom distributed system using message passing interface ( mpi ) .
custom approaches are usually closer to optimal because developers can tune the code based on their own needs and minimize the unnecessary overhead .
however , the drawbacks are also significant : because mpi is a barebone communication specification developers need to write their own code for job dispatching , load balancing , and dealing with node failure  @xcite .    after evaluating aforementioned approaches
, we decided to use an emerging computational paradigm called vertex programming .
vertex programming aims to improve the performance of graph / network computing by automatically distributing in - memory computation and with vertex - centric scheduling . unlike the mapreduce paradigm , which writes every intermediate result to disk , vertex programming keeps data up - to - date , in - memory and reduces i / o overhead by only materializing data to disk through periodic checkpoints .
vertex - centric scheduling views every graph - vertex as an elementary computing unit and uses mpi ( or some other message passing system ) to transfer data over graph - edges .
typically , vertex - programs are rather easy to implement , can be distributed easily , and are much more computationally efficient than conventional , procedural programming when working with iterative computational tasks  @xcite .
several vertex programming frameworks , including spark - graphx , hama , graphlab , graphlab create and graphlab 2.0 ( powergraph ) , were evaluated , and the powergraph framework was ultimately chosen .
the decision was based , in part , from insurmountable troubles experienced during several months of implementation attempts on spark s graphx framework .
ultimately , we concluded that c++ and mpi based powergraph was faster , more scalable and had a much smaller memory footprint than other frameworks .
[ alg : dist_rwr ]    ( ) vertex @xmath112    [ alg : dist_upate ]    ( ) vertex @xmath112    the distributed hdtm inference algorithm is similar to procedural hdtm .
we do not detail the entire distributed hdtm inference algorithm in this paper ; however , the source code is referenced in the section  [ sec : conclusions ] . to fit hdtm to the vertex - programming model , changes in sampling sequence and attention to global synchronization were required .
firstly , random walk with restart must be executed during each gibbs iteration so that every visited node can have a random walk probability .
next , every node gathers @xmath81 , @xmath86 , and @xmath85 separately and decides their new path @xmath85 according to eq  [ eq : pathsample ] and shown in alg  [ alg : dist_rwr ] .
after a path is sampled , each node will pick sample assignments @xmath28 for each word @xmath22 across all documents / nodes @xmath8 in parallel according to eq  [ eq : wordsample ] .
a global synchronization step is required so that each document / node can update the number of words @xmath22 and the topic assignments @xmath28 globally ; fortunately , because the topics and words are sampled according to the information provided in the _ path _ from root @xmath4 to the each document @xmath106 ( document @xmath8 at level @xmath43 ) , it suffices to update the nodes on the path from @xmath4 to @xmath106 instead of an actual global update to all nodes in @xmath1 .
this vertex - programming based path - global update function is shown in alg  [ alg : dist_upate ] .
furthermore , this update is executed during the _ synchronization barrier _ , which is built - in to most vertex - programming frameworks , is highly optimized , and does not lock the global system any more than the synchronization barrier already does .
this section describes the method and results for evaluating the hdtm model .
a quantitative and qualitative analysis of the hierarchical document - topic model s ability to learn accurate and interpretable hierarchies of document graphs is shown .
the main evaluations explore the empirical likelihood of the data and a very large case study wherein human judges are asked to evaluate the constructed hierarchies .
hdtm is evaluated on four corpora : the wikipedia category graph , the wikipedia document graph , a computer science web site from the university of illinois , and a bibliographic network of all cikm and sigir conferences .
the wikipedia data set has been used several times in the past for topic modeling purpose . however , because of the computational resources needed to infer topic models , prior studies have severely constricted the dataset size .
, crawled 105 pages starting with the article on the nips conference finding 799 links  @xcite . ho _
et al . _ performed a larger evaluation of their topicblock model using 14,675 document with 152,674 links ; however , they truncated each article to only the first 100 terms and limited the vocabulary to the 10,000 most popular words  @xcite .
the wikipedia category data set is a crawl of the _ category _ graph of wikipedia , beginning at the category as shown in figure  [ fig : inh_wikicat ] . in wikipedia
each category has a collection of articles and a set of links to other categories ; however , categories do nt typically have text associated with them , so the text of each article associated with a particular category is associated to the category s text . for example , the category includes articles : , etc . in total the crawled category graph consisted of 609 categories with text from 6,745 articles .
the category graph is rather sparse with only 2,014 edges between categories , but has vocabulary size of 146,624 with 5,570,868 total tokens .
we did not perform any text preprocessing procedure , including stop word removal and stemming , in any of the experiments due to empirical findings that these models are robust to the presents of stop words , etc  @xcite .
such settings can also explore hdtm s robustness in presence of a large and noisy corpus .
a computer science department web site from the university of illinois urbana - champaign was chosen as the second data set because it represents a rooted web graph with familiar topics . by inferring the document hierarchy ,
the goal is to find the organizational structure of the computer science department .
the intuition is that web sites reflect the business organization of the underlying entity ; thus it is expected that subtrees consisting of courses , faculty , news , research areas , etc .
are found at high levels , and specific web pages are found at lower levels in the hierarchy .
web site was crawled starting at the entry page and captured 1,078 web pages and 63,052 hyperlinks . in total
there were 15,101 unique terms from 771,309 tokens .
the bibliographic network consists of documents and titles from 4,713 articles from the sigir and cikm conferences .
there exist 3,908 terms across 43,345 tokens in the document collection . in this collection ,
links include citations between papers within the cikm and sigir conferences .
due to different citation styles , vendor abbreviations , and ambiguous names , without human interference most bibliographical meta - data extraction algorithms can only extract a portion of correct data  @xcite .
here we choose to use the most complete citation data set available from by the authors of the arnetminer project  @xcite , however the citation graph is not guaranteed to be complete .
a sigir 1998 paper by ponte and croft  @xcite was chosen to be the root document because , in the data set , it had the most in - collection citations .    to construct a larger data set with finer granularity , the english wikipedia article graph is used . in wikipedia , every wiki - article has two different link types : 1 ) category links , which point from the article to its categories , and 2 ) wiki - links , which point from the article to other wiki - articles .
the category links are important because they hint at some encoded hierarchical structure that was codified on top of wikipedia after the article - graph was largely developed .
for example , the wiki - article has a category link to and wiki - links to and among others . in later experiments we infer a document hierarchy from wikipedia and compare it to the crowd - encoded wikipedia category hierarchy .
our wikipedia snapshot has 4,606,884 documents with 1,878,158,318 tokens , 44,739,242 links , and a vocabulary of 6,064,216 words .
because the document graph is not a single connected component , we picked the largest component in that graph as data set .
this giant component contains 1,957,268 documents and 44,673,134 document to document links ; 42.49% of nodes and 70% of edges are in this component . additionally ,
1,316,879,537 terms with 4,225,765 unique tokens appear in this component , which is approximately 70% of the total number of terms . as a graph - preprocessing step
, we replaced all edges pointing to redirection pages with edges that point directly to the actual wiki - article .      using either the traditional inference model from sec .
[ sec : inference ] or the high - throughput distributed sampling algorithm from sec .
[ sec : distinference ] we basically sample a full hierarchy @xmath113 by sampling paths for each node @xmath78 except the root and the words in the document @xmath76 . given the state of the sampler at some time @xmath44 , _ i.e. _ , @xmath114 and @xmath115
, we iteratively sample each variable conditioned on the others as illustrated in  [ sec : generating ] .    the conditional distribution of the latent variables in hdtm model the given document - network .
after running the markov chain a sufficient number of times we approach the stationary distribution ; the process of approaching the stationary distribution is called the `` burn - in '' period . after the burn - in we collect samples at a selected interval , _
i.e. _ , the sampling lag .
the collected `` samples '' from the markov chain are full hierarchies constructed from the selection of a path for each node @xmath78 and a word for each document @xmath76 .
therefore each sampled hierarchy contains one estimation about position of each document in the hierarchy and the position of each word in a document . for a given sampled hierarchy
, we can assess the goodness of the hierarchy by measuring the log probability of that hierarchy and the observed words conditioned on the hyperparameters :    @xmath116    using this log likelihood function it is possible to pick the most sampled hierarchy that maximizes the log likelihood as the final result .
later , we will relax this assumption and use a more robust measurement that computes the mode of the sampled hierarchies and determines the _ certainty _ of the result .
hdtm has some distinct qualities that make apples to apples comparison difficult .
because hdtm is the first model to generate document hierarchies based on graphs , there is nothing to _ directly _ compare against .
however , some of the models in the related work perform similar tasks , and so comparisons were performed when applicable .
the related models typically perform quantitative evaluation by measuring the log likelihood on held out data or by performing some other task like link prediction .
log likelihood analysis looks at the goodness of fit on held out data .
unfortunately , the creation of a `` hold out '' data set is not possible , because each document , especially documents on the first or second level document , is very important to the resulting hierarchy .
removing certain documents might even cause the graph to separate , which would make hierarchy inference impossible . instead ,
for the first quantitative evaluation , we compare the highest log likelihood generated from each model s sampler .
quantitative experiments were performed on many of the aforementioned algorithms including : hlda  @xcite , topicblock @xcite , tssb  @xcite , and fslda  @xcite .
the fixed structure in fslda is determined by a breadth first iteration over the document graph because url heuristics used in the original paper do not exist in most of our data sets .
the depth of hlda and topicblock is 4 .    in all cases ,
a gibbs sampler was run for 5,000 iterations ; with a burn - in of 2,000 and a sampling lag of 20 .
figure  [ fig : lliterations ] shows the log likelihood for each sample on the computer science web site ; the other data sets showed similarly - shaped results .
interestingly , figure  [ fig : llavgdepth ] shows that higher likelihood values are strongly correlated with hierarchies of deeper average depth .
lower @xmath25 values result in deeper hierarchies . ]
the gibbs sampling algorithm was run on hdtm for various values of @xmath25 , and figure  [ fig : bestcumll ] shows the best cumulative log likelihood for each of the tested values of @xmath25 .
we observe that hdtm with @xmath74 achieved the best likelihood score .
likelihood scores decreased steadily for increasing @xmath25 values , and hdtm with @xmath73 achieved the worst likelihood score .
table  [ tab : llresults ] shows the results of the different algorithms on the three smaller data sets .
the topicblock and tssb clearly infer models with the best likelihood .
the remaining algorithms , including hdtm , have mixed results .
although these log likelihood results initially seem as if hdtm performs poorly , we demonstrate in section  [ subsec : discussion ] that these likelihood results are expected and preferred .
recall that the goal of hdtm is to generate document hierarchies from some original document - graph .
thus , a positive result should somehow demonstrate that hdtm demonstrates hierarchies that are similar to the hierarchy that a human ( or many humans ) would create given the same document graph . with this in mind we performed a large scale experiment on the full english wikipedia article graph .
also recall that the wikipedia category graph establishes a hierarchy that was human - built on top of wikipedia that maintains a pseudo - hierarchical classification of almost every document .
the structure of categories is described to be a _ tree _ rooted by or ; however , we find that 87.01% of categories have an in - degree larger than 1 indicating the actual category is more complex .
we find that although the wikipedia category graph _ mostly _ resembles a top - down ( coarse to fine ) hierarchy , it is technically an undirected loopy graph . as for wiki - articles
, they almost always belong to more than one category because the idea or object described in each article can rightfully have multiple classification perspectives .
for example , the wiki - article belongs to a category of the same name , but it also belongs to and among others .    for a large scale evaluation
our task is to infer a hierarchy from the full wikipedia article graph and see how well it matches the category graph .
unfortunately , it is difficult to measure similarity between category graph and hdtm - hierarchies using conventional evaluation methods like the log likelihood .
consider , for example , the category , which contains wiki - articles of faculty members from vastly different topical fields . in such cases ,
the bond is described primarily by the links rather than the content ; nevertheless , the log likelihood score judges models based on the goodness of fit of words . thus it is unreasonable to measure hierarchy quality by likelihood scores alone ; indeed , the hierarchy with the best possible likelihood score may be very different from actual , human created hierarchy . in order to perform a more thorough evaluation we should also compare the the graph structure .    graph topologies can be compared in many different ways . among many options we chose deltacon  @xcite to measure the similarity between hdtm s hierarchy and the wikipedia category hierarchy three reasons : 1 )
deltacon can calculate the similarity between two graphs with nodes that partially overlap , as observed in the messy wikipedia category graph , 2 ) the underlying metric in deltacon is the affinity score calculated by fast belief propagation ( fabp ) , which models the connectivity of nodes , and 3 ) deltacon can process millions of nodes in only a few hours .
the deltacon metric returns a similarity score between the two compared graphs ranging between 0 and 1 inclusive where 0 is completely dissimilar and 1 is completely similar .
although comparing with hlda and its variants is important , all existing hierarchical topic models are incapable of processing on data sets at this scale .
we attempted to use the hlda implementation in the mallet package  @xcite , but 130 hours was needed in order to compute a single gibbs sampling iteration .
fortunately , we were able to use graphlab s implementation of lda which is based on models developed by ahmed _
et al . _
@xcite and smyth _ et al . _
although lda does not directly generate a graph , we are able to compare it with the category hierarchy by connecting the documents within a single lda - cluster to a made - up root node to simulate a hierarchy .
the large scale models were inferred using 400 gibbs sampling iterations with a burn - in of 100 and a lag of 10 .
in addition to hdtm and lda , we also use deltacon to compare the original wikipedia article graph and a randomly generated article - hierarchy as baselines . in the case of wikipedia article graph
, we expect the wikipedia article graph ( the full graph , not a hierarchy ) to perform reasonably well because the category hierarchy is largely built on top of the article graph topology and they currently coexist and coevolve . in the case of the randomly generated article hierarchy , we randomly select a single parent for each node in the original article graph .
.comparison of the wikipedia category graph to other generated hierarchies .
deltacon scoring means higher is better .
the random article hierarchy is generated by randomly picking one parent among all possible parents , _
i.e. _ , references , for each article .
the differences in deltacon scores show that hdtm can preserve crucial connectivity information when constructing a new hierarchical structure from the original graph . [
cols=">,^ " , ]     table  [ tab : deltacon ] shows that the hierarchy inferred by hdtm is indeed most similar to the wikipedia category graph , followed by the random article hierarchy .
recall that deltacon looks at graph topology similarity , thus the hdtm hierarchy is expected to more topologically similar to another hierarchy ( even if random ) than the article graph .
these results demonstrate that hdtm can identify and preserve critical topological features when inferring a hierarchy from the graph . as usual ,
an increase in the number of lda topics increases the goodness of fit score ( in this case measured by deltacon instead of likelihood ) .
interestingly , the @xmath25 parameter and the selection of different roots did not significantly influence deltacon score . to understand why this is , recall that different root - nodes and @xmath25 values result in different classification perspectives , but the different perspectives are still subsets of the same category graph with similar topological properties .
hence they should have similar scores using the deltacon metric .
recall that the hdtm , in its most basic form , generates a hierarchy by picking the best parent for each node / document ( except the root ) . in the iterative gibbs sampling process it is an almost certainty that a given node will pick different parents during different iterations .
for example , say node @xmath8 has two parents @xmath117 and @xmath118 , it is possible that during iterations 15 node @xmath8 samples node @xmath117 as its parent , but then in iterations 620 node @xmath8 samples node @xmath118 to be its parent .
two questions come to mind : 1 ) which parent should be ultimately picked for node @xmath8 in the final output graph ? and
2 ) what can the distribution of samples say about the certainty of our inferred graph ?    in classic gibbs sampling , maximum a posteriori estimation is determined by the the mode of the samples . for some node @xmath8 , this translates to choosing the parent that appeared in the most samples as the final parent ; in the above case node @xmath8 would chose node @xmath118 as its parent because node @xmath118 was sampled 15 times compared to only 5 samples of node @xmath117 . to answer the second question , and therefore model the certainty of an inferred hierarchy
, we calculate the number of times some node @xmath8 picks its final parent over the total number of samples normalized by the number of possible choices , _
i.e. _ , parents , node @xmath8 has .
this results in a certainty score for node @xmath8 :    @xmath119    where @xmath22 represents the total number of samples , @xmath120 is the number of times the final parent @xmath121 is sampled , and @xmath122 is the indegree of node @xmath8 representing the total number of parents that node @xmath8 could pick from . in the certainty
score the outside fraction is used to measure the normalized difference between the raw probability and probability of random guess . applying this function to the above example
we would calculate that the certainty score of node @xmath8 would be @xmath123 .
figure  [ fig : wiki_certainty_density ] shows the probability density functions for the certainties in hdtms parent sampling process . in the figures on both left and right we find that the probability densities appear to be polynomial distributions with interesting plateaus that end just before the 50% , 66% , and 75% certainty scores corresponding to the near - certain ( but not perfectly - certain ) scores for nodes with indegree values of 2 , 3 , and 4 respectively . in these results , and others not displayed in this paper , we find that when the root is a general document like , then changing @xmath25 does not affect the certainty distribution . given a relatively specific root like , larger @xmath25 values increase certainty overall .
as discussed earlier , measuring topical similarity in hierarchies can be precarious because , in many cases , documents that are correctly situated under a topically unrelated parent may still have a strong contextual association with the parent document that outweighs the topical / language - oriented similarity as determined by the @xmath25 parameter .
as an example , consider the wiki - articles and : even though these two articles are topically dissimilar , hdtm parameterized with a high @xmath25 value is likely to place both articles as children of the hierarchy s root because the high @xmath25 values weigh the topological link as more important that the documents inferred topicality . if , on the other hand , hdtm was parameterized with a very low @xmath25 value , then is more likely to be situated with other state senators , and the is more likely to be situated with other legislation .    to quantitatively evaluate the topical and topological fitness of hierarchy together ,
we compare the similarily among the sets of parents and children in the inferred hierarchy against sets of wiki - articles and their wikipedia categories . because we are comparing sets of items we use the jaccard coefficient for this task .
specifically , the jaccard coefficient is calculated by    @xmath124    for each document @xmath8 that belongs to a set of categories @xmath125 , and where @xmath126 is the union of the corresponding categories of the nodes in @xmath8 s ancestors @xmath60 .
the jaccard coefficient is useful here because it gives an quantitative measure describing how well the inferred hierarchical structure matches with the human - annotated categorical structure of wikipedia .
figure  [ fig : wiki_jaccard ] shows the relationship between jaccard coefficients of the hierarchies . for illustration clarity , in this figure
all jaccard coefficients are binned into 10 equal intervals .
we observe a clear correlation between the node s certainty function and its jaccard coefficient .
this means that decisions that hdtm is most certain about tend to be the most similar to the human - created hierarchies .
recall that hdtm uses the @xmath25 parameter to balance the weight of topological links and topical similarities .
hence when specific nodes like are chosen as the root , higher @xmath25 will reduce jaccard coefficient for high certainty nodes .
this is because higher @xmath25 values favor topology rather than content during the path and parent selection process .
these results also underscore the importance of the selection of the root node . when provided a root node , which can be any node in the document graph , the inferred hierarchy is created from the perspective of that root .
the hierarchy rooted at would therefore organize , say , congressional legislation differently than a hierarchy generated by the wiki - article on . yet , as we ve seen , the graph similarity metrics views the drastically different document - topic hierarchies as topologically similar .      in order to properly understand the results captured in table  [ tab : llresults ] from above , recall that log likelihood is a metric on the _ fit _ of the observations on the configuration of the model . the original work on lda  @xcite found that likelihood increases as the number of topics increases . along those lines , chang , _ et al .
_ demonstrated that more fine grained topics , which appear in models with a larger number of topics have a lower interpretability , despite having higher likelihood scores  @xcite .
simply put , there exists a negative correlation between likelihood scores and human interpretability in lda and similar topic models .    applying those lessons to our experiments ,
recall that hdtm has as many topics as there are documents , and non - root document topics are mixtures of the topics on the path to the root . also recall that hlda , topicblock and tssb all generate a large number of latent topics . in hlda and topicblock , there are infinitely many topics / tables in the ncrp ; and practically speaking , the number of topics in the final model is much larger than the number of documents ( conditioned on the @xmath25 parameter ) . in tssb , the topic generation is said to be an interleaving of two stick breaking processes ; practically , this generates even larger topic hierarchies .
the fslda algorithm has as many topics as there are in hlda , however , the fslda hierarchy is not redrawn during gibbs iterations to fit the word distributions resulting in a lower likelihood .
simply put , the number of topics in hdtm and fslda @xmath127 @xmath128 hpam , hlda and topicblock @xmath128 tssb .
similarly , figure  [ fig : llavgdepth ] shows that deeper hierarchies have higher likelihood scores .
this is because long document - to - root paths , found in deep hierarchies , are able to provide a more fine grained fit for the words in the document resulting in a higher likelihood .
therefore , the better likelihood values of hlda , topicblock and tssb are due to the larger number of topics that these models infer and not necessarily due to better model outputs . in this case
, a better way to evaluate model accuracy is by some external task or by manually judging the coherence of the topics .      to measure the coherence of the inferred groupings , the _ word intrusion _
task developed by chang _
et al _  @xcite
is slightly modified to create the _ document intrusion _ task . in this task ,
a human subject is presented with a randomly ordered set of eight document titles .
the task for the human judge is to find the intruder , that is , the judge is asked to find which document is out of place or does not belong .
if the set of documents without the intruder document all make sense together , then the human judge should easily be able to find the intruder .
for example , given a set of computer science documents with titles \ { } , most people , even non - computer scientists , would pick as the intruder because the remaining words make sense together  they are all computer science disciplines .    for the set \ { } , identifying a single intruder is more difficult .
human judges , when forced to make a choice , will choose an intruder at random , indicating that the grouping has poor coherence .        to construct a set of document titles to present to the human judge , a grouping from the hierarchy is selected at random , and select seven documents are further selected at random from the grouping . if the there are fewer than 7 documents available in the selected grouping , then we select all of the documents available ; groupings of size less than 4 are thrown out .
in addition to these documents , an intruder document is selected at random from among the entire collection of documents minus the documents in the test group .
titles are then shuffled and presented to the human judges .
extracting document groupings for evaluation is slightly different for each model .
hdtm and fslda store a document at each node in the hierarchy .
a grouping is selected by first picking a document at random , and then choosing its siblings .
topicblock and hlda store documents at the leaves of the taxonomy , which often include several documents .
a grouping is selected from these models by first picking a document at random , and then choosing the other documents in the leaf - topic .
the hierarchies that the tssb model constructed allowed multiple documents to live at inner nodes .
attempts to evaluate groupings on inner nodes with more than 4 documents were unsuccessful .
nodes with 4 or more siblings were also difficult to find because the hierarchies that were generated were too sparse to find practical groupings .
thus human judges with tssb groupings could not be found .
each document - graph collection had different types of labels presented to the judges .
the compsci web site collection was labeled by the web page title and url ; the wikipedia collection was labeled by the category title as shown in figure  [ fig : intruder ] ; the bibliography network was labeled by the title of the paper .
the intruder detection tasks described above were offered on amazon mechanical turk .
no specialized training is expected of the judges .
50 tasks were created for each data set and model combination ; each user was presented with 5 tasks at a time at a cost of $ 0.07 per task .
each task was evaluated by 15 separate judges . in order to measure the trustworthiness of a judge ,
5 easy tasks were selected , _
i.e. _ , groupings with clear intruders , and gold - standard answers were created .
judges who did not answer 80% of the gold - standard answers correctly are thrown out and not paid . in total the solicitation attracted 31,494 judgments , across 14 models of 50 tasks each .
of these , 13,165 judgments were found to be from trustworthy judges .
the _ model precision _ is measured based on how well the intruders were detected by the judges . specifically , if the intruder word @xmath129 is from model @xmath130 and task @xmath43 , and @xmath131 is the intruder selected by the human judge @xmath105 on task @xmath43 in model @xmath130 then    @xmath132    where @xmath133 is the indicator function and @xmath134 is the number of judges .
the model precision is basically the fraction of judges agreeing with the model .
figure  [ fig : turks ] shows boxplots of the precision for the four models on three corpora . in most cases , hdtm performs the best . as in  @xcite
, the likelihood scores do not necessarily correspond to human judgments .
paired , two - tailed t - tests of statistical significants ( @xmath135 ) performed between hdtm @xmath136 and @xmath137 and the other models are represented by @xmath138 and @xmath139 in figure  [ fig : turks ] respectively .    .
words at the root document represent the most probable words in the root topic .
most probable words for other documents are not shown due to space constraints . ]
the bibliography network data had relatively low precision scores .
this is almost certainly because it was more difficult for the judges , who were probably not computer scientists , to differentiate between the topics in research paper titles .
figure  [ fig : dblphierarchy ] shows a small portion of the document hierarchy for the bibliographic network data set constructed with hdtm @xmath140 .
the root document has 20 children in the hierarchy despite having 145 in - collection links .
the remaining 120 documents live deeper in the hierarchy because hdtm has determined that they are too specific to warrant a first level position , and have a better fit in one of the subtrees .
recall that each document is associated with the topics from itself to the root , where the root is a single , general topic .
the seven most probable terms at the root level are also shown adjacent to the root s title in figure  [ fig : dblphierarchy ] .
these terms , like in hlda and topicblock , are terms that are general to the entire collection .
similar sets of words exist at each node in the hierarchy , but are not shown in this illustration to maintain clarity .
hdtm source code , analysis code and the scripts which generated the results found in this paper can be downloaded from https://github.com/nddsg/hdtm/releases/tag/kais .
the wikipedia , web site and bibliographical data are all publicly available and free to download and are replicated in our own data repository .
hierarchical document - topic model ( hdtm ) , is a bayesian generative model that creates document and topic hierarchies from rooted document graphs .
the initial hypothesis was that document graphs , such as web sites , wikipedia and bibliographic networks contain a hidden hierarchy .
unlike most previous work , hdtm allows documents to live at non - leaf nodes in the hierarchy , which requires the random walk with restart path sampling technique .
an interesting side - effect of the random walker adaptation is that the path sampling step , eq .
[ eq : pathsample ] , is much faster and easier to scale than the ncrp because rwr only creates a sampling distribution for the parents of a document , whereas the ncrp process creates a sampling distribution over all possible paths in the taxonomy .
several quantitative experiments comparing hdtm with related models were performed .
however , the results show that likelihood scores are a poor indicator of hierarchy interpretability , especially when the number of topics are different between comparison models .
a large qualitative case study was also performed which showed that the cohesiveness of the document groupings generated by hdtm were statistically better than many of the comparison models despite the poor likelihood scores .
55ifxundefined [ 1 ] ifx#1 ifnum [ 1 ] # 1firstoftwo secondoftwo ifx [ 1 ] # 1firstoftwo secondoftwo `` `` # 1''''@noop [ 0]secondoftwosanitize@url [ 0 ]
+ 12$12  & 12#1212_12%12@startlink[1]@endlink[0]@bib@innerbibempty in http://dblp.uni-trier.de/db/conf/nips/nips2010.html#adamsgj10[__ ]  ( )  pp .   http://dl.acm.org/citation.cfm?id=944919.944937
[ * * ,   ( ) ] in link:\doibase 10.1145/2187836.2187936 [ _ _ ]  ( ,  )  p.   in http://dblp.uni-trier.de/db/conf/nips/nips2010.html#chambersss10[__ ]  ( )  pp .
http://dblp.uni-trier.de/db/journals/jmlr/jmlrp5.html#changb09 [ * * ,   ( ) ] in  link:\doibase 10.1145/1273496.1273576 [ _ _ ] ( ,  )  pp . in @noop _ _  ( )  pp .
@noop * * ,   ( ) in  http://dblp.uni-trier.de/db/conf/trec/trec2004.html#songwsxlqzzxm04[__ ]  ( ) in  link:\doibase    10.1145/1076034.1076105 [ _ _ ]  ( ,  )  p.
link:\doibase 10.1145/984321.984322 [ * * ,   ( ) ] @noop * * ,   ( ) @noop * * ,   ( ) in  http://dblp.uni-trier.de/db/journals/jmlr/jmlrp15.html#nallapatimm11[__ ] ,  vol .
( )  pp .   in  http://dblp.uni-trier.de/db/conf/icwsm/icwsm2008.html#furukawamoui08[__ ]
( ) link:\doibase 10.1016/0306 - 4573(88)90027 - 1 [ * * ,   ( ) ] link:\doibase 10.1007/s10618 - 005 - 0361 - 3 [ * * ,   ( ) ] \doibase 10.1023/a:1022852608280 [ * * , ( ) ] link:\doibase 10.1016/0004 - 3702(89)90046 - 5 [ * * , ( ) ] in  link:\doibase 10.1145/1102351.1102389 [ _ _ ]  ( ,  )  pp .   in http://dl.acm.org/citation.cfm?id=2002736.2002866[__ ]
( )  pp .
http://dl.acm.org/citation.cfm?id=1690219.1690233 [ ( ) ] in  link:\doibase 10.1145/1871437.1871469 [ _ _ ]  ( ,  )  p.
link:\doibase 10.1038/nature06830 [ * * ,   ( ) ] link:\doibase 10.1016/0378 - 8733(83)90021 - 7 [ * * , ( ) ] http://arxiv.org/abs/0912.5410 [ ,  * * ,   ( ) ] in  link:\doibase    10.1145/1367497.1367512 [ _ _ ]  ( ,  )  pp .   in
link:\doibase 10.1145/511446.511513 [ _ _ ]  ( ,  )  pp .   in  http://dblp.uni-trier.de/db/conf/nips/nips2000.html#cohnh00[__ ]
( )  pp . in
link:\doibase 10.1145/1401890.1401957 [ _ _ ] ( ,  )  pp . in  http://dblp.uni-trier.de/db/conf/uai/uai2008.html#gruberrw08[__ ]  ( )  pp .   in  http://dblp.uni-trier.de/db/conf/ijcai/ijcai2005.html#mccallumcw05[__ ]  ( )
pp .   in http://dblp.uni-trier.de/db/conf/uai/uai2004.html#rosen-zvigss04[__ ]
( )  pp .   in @noop _
_  ( )  pp .   in  @noop _ _  ( )  pp .   in  @noop _ _  ( ,  )  pp .   in  @noop _ _  ( ,  ) @noop ( ) http://dblp.uni-trier.de/db/journals/pvldb/pvldb4.html#bahmanicg10 [ * * ,   ( ) ] in  @noop _ _  ( , )  pp .   in  @noop
_ _ ,  vol .
( )  p.   link:\doibase    10.1145/2094114.2094118 [ * * ,   ( ) ] @noop ( ) @noop ( ) in  @noop _ _  ( )  pp .   in  @noop _ _  ( ,  )  pp .   in  @noop _ _  ( ,  )  pp .   in  link:\doibase 10.1145/1401890.1402008 [ _ _ ]  ( ,  )  p.   in  link:\doibase 10.1145/290941.291008
[ _ _ ]  ( , )  pp .
@noop ( ) @noop ( ) in  @noop _ _ ( )  pp .