*   * \1 . data * 1.1 general considerations in regard to high dimensional and massive datasets * 1.2 characteristics of the data studied * 1.3 distribution of terms * \2 .
objective of computational efficiency : using aggregated subsets to construct the factor space * 2.1 introduction * 2.2 projection into the factor space * 2.3 the practical benefit of a selected word set * 2.4 conclusion *   * \3 . about the data ; followed by correspondence analysis of all recipes using a 247 word set * \4 .
search using solr * 4.1 setting up data file and its description * 4.2 running the server and updating the index * 4.3 querying and other operations * 4.3.1 web browser user interface * 4.3.2 command line querying * \5 .
consistency of solr searches * references * appendix 1 : sample recipe * appendix 2 : 247 ingredients used as attributes in the correspondence analysis * appendix 3 : alternative presentation of plot * appendix 4 : correspondence analysis for singular / plural association and potentially for disambiguation
very high dimensional ( or equivalently , very low sample size or low @xmath0 ) data sets , by virtue of high relative dimensionality alone , have points mostly lying at the vertices of a regular simplex or polygon @xcite .    in @xcite , the following issues are posed .
firstly , what should we do , for our analytics , when we have thousands of documents ? correspondence analysis outputs ( and this is the same for other kindred methods , like lsa , latent semantic analysis ; plsa , probabilistic adaptation of lsa using a mixture of multinomial distributions ; factorial correspondence analysis ( ca ) ; kohonen maps ; or clustering methods ) are very huge ( much more than the original dataset ) .
what we must therefore do is to display the output in an `` intelligent '' and friendly way .
we explore the following .
* we have 306 collections , labelled by `` recipe via meal - master ( tm ) v8.05 '' , each with , in principle , 500 recipes . *
( an additional set of 20 collections , relating to `` v8.06 '' , were considered as different , and set aside from this work . )
* there is one exception here : one of the collection files had two recipes less than the usual 500 complement . *
hence we are dealing with 306 @xmath1 500 recipes ( less 2 ) , i.e.  152,998 .
* appendix 1 shows a sample recipe . * this data , in text ,
contained : 5,948,739 lines of text ; 30,199,625 ( whitespace - demarcated ) words ; and 206,993,672 characters .
* on the 306 recipe collections , comprising 152,998 recipes , we used our term extractor .
this listed all terms of one character or more , having first removed punctuation and numeric character .
we will use the term `` words '' for what was extracted .
all upper case had been first set to lower case .
abbreviations and acronyms showed up as words , some character combinations did so also following punctuation removal , and all forms of stemming were retained .
+ various misspellings are in the given data . in collection
mm066001.mmf , there is : `` grate the zuccini into a calendar ( sp ) ... '' .
what is intended is `` zucchini '' and `` colander '' .
interestingly there are 57 times that `` zuccini '' appears , and once that `` zuccinis '' appears .
+ while spelt correctly , `` zucchini '' is present 7155 times , and `` zucchinis '' is present 56 times .
there are various other incorrect words present , with one occurrence each : `` azucchini '' , `` brushingzucchini '' , `` lambzucchini '' , `` ofzucchini '' , `` pzucchini '' , `` zucchinia '' , `` zucchiniabout '' , `` zucchinii '' , `` zzzucchini '' .
+ since we take the data as such , notwithstanding such inaccuracies , it is to be noted that the correct spellings are predominant , and that the misspellings only show up if quite atypical features in the data ( such as rare occurrence ) are looked into .
let us formulate this as a working principle , or perhaps even a working hypothesis that we have verified in all cases that we have looked into .
in the case of large , or very large , numbers of occurrences of entities , the syntactically correct form will predominate .
* from the 152,998 recipes we obtained 101,060 unique words .
* in this list , ranked by decreasing frequency of occurrence , the 5000th ranked word had 162 occurrences ; the 15,000th ranked word had 18 occurrences ; and at the 56,229th ranked word ( 56% through the total list of 101,060 ranked words ) , the frequency of occurrence became thereafter ( for higher ranked words ) , 1 .
* the top frequencies of occurrence are as follows : + .... and 890294 the 842542 to 531633 in 469003 with 313588 of 281320 recipe 280035 or 247912 ts 238309 for 226514 until 224481 add 207146 tb 200221 from 193659 minutes 190880 by 190815 on 183287 salt 162811 yield 162082 servings 161098 meal 155040 master 152980 via 149162 title 148802 tm 148766 categories 148612 into 145565 sugar 138670 heat 131846 water 130211 pepper 127901 oil 122997 chopped 119530 over 115212 butter 112329 at 106300 sauce 102881 is 101060 com 99841 flour 96793 stir 91580 mixture 91484 about 91279 it 89491 cook 87097 pan 86314 oz 83908 garlic 82776 cream 81633 place 80825 cheese 78986 bowl 76209 mix 76182 chicken 75807 onion 71516 cut 70808 lb 68376 posted 67769 fresh 66125 baking 64731 .... * the very final ranked words were as follows : + .... zwiebelfleisch 1 zwirek 1 zws 1 zygielbaum 1 zyla 1 zylman 1 zza 1 zzwc 1 zzzingers 1 zzzucchini 1 .... + the second last here arose from this entry in a recipe : `` title : wing zzzingers '' .
the last word comes from this entry in a recipe : `` add zzzucchini , peppers and onion . '' .
a power law ( see @xcite ) is a distribution ( e.g.  of frequency of occurrence ) of the form @xmath2 where constant , for the data , @xmath3 ; and an exponential law is of the form @xmath4 . for a power law , @xmath5 , @xmath6 .
a power law has heavier tails than an exponential distribution . in practice @xmath7 . for such values
, @xmath8 has infinite ( i.e.  arbitrarily large ) variance ; and if @xmath9 then the mean of @xmath8 is infinite .
the density function of a power law is @xmath10 , and so @xmath11 , where @xmath12 is a constant offset .
hence a log - log plot shows a power law as linear .
power laws have been of great importance for modelling networks and other complex data sets ( see @xcite ) .
figure [ fig1 ] shows a plot of rank ( most frequent , through to least frequent , the latter being a very large number of words that occur once only ) against the value of the frequency of occurrence . to understand the plot
, we use log - log scaling .    in a very similar way to the power law properties of large networks ( or file sizes , etc . )
we find an approximately linear regime , ending ( at the lower right ) in a large fan - out region .
the slope of the linear region characterizes the power law .
we find that the probability of having more than @xmath8 occurrences per word to be approximately @xmath13 for large @xmath8 ( since we find the slope of the fitted line in figure [ fig1 ] to be @xmath14 ) .
just how well can correspondence analysis carry out the euclidean , factor determining mapping , if recipes are first aggregated together ?
one reason for this question is computational time . for analysis of an @xmath15 data set , with @xmath16 words and @xmath0 recipes or recipes collections
, then the dual spaces means that we analyze either the data set or its transpose .
eigen - reduction is a cubic process .
the computational time for analysis of the @xmath15 data set is @xmath17 , or for its transpose , @xmath18 . for a selected word set , if we can carry out the main analysis on a smaller @xmath0 , so much the better for us .
aggregation is just concatenating the recipes . in the entire set of recipes , the number of words ( to repeat : of one or more characters in length ;
punctuation deleted ; upper case set to lower case ; numerical and accented characters ignored ) was 101,060 . for initial assessment
, we took the top - ranking 1000 words .
so this illustrative case , using just 500 recipes , has the recipe set crossed by this 1000-word set , with the frequencies of occurrence tabulated .
we aggregated the 500 recipes into 5 recipe - sets .
these comprised recipes 1100 , 101200 , 201300 , 301400 , and 401 - 500 .
each of these 5 recipe - sets had the frequencies of occurrences on the 1000-word set .
here we analyze two cross - tabulation tables , of dimensions respectively @xmath19 , and @xmath20 . by construction of the latter ,
the column sums are identical . then due to the use of profiles ,
the average profile is the same in each case .
if we represent the data matrix in frequency terms ( i.e.  the frequencies of occurrences each divided by the grand total ) , the matrix can be denoted @xmath21 where @xmath22 is set of recipes , or of recipe - sets , @xmath23 is the word - set , @xmath24 is a recipe or a recipe - set , and @xmath25 is a word ) the column marginal distribution is @xmath26 , or in summation terms , @xmath27 .    in correspondence analysis ,
the centre or origin of the euclidean factor space is given by @xmath28 in the recipe space , and by @xmath29 in the word space , such that we have the following view : starting with a probability distribution @xmath30 we want to explore how it differs from the product of marginal distributions , @xmath31 , and this in the @xmath32 metric of centre , the product @xmath31 .
this is a decomposition of inertia of clouds in recipe and/or in word space .
we have : @xmath33 where @xmath34 .
figure [ cawithagg ] shows the principal factor display resulting from the correspondence analysis of ( 1 ) the @xmath35 recipes times words data , and ( 2 ) the @xmath20 recipe - sets times words data .
note again that the analyses ( the eigen - reductions  determining the factors ) were completely independent .
the results of figure [ cawithagg ] can be explained in the following terms .
we have a set of 500 items ( the recipes ) with quantitative description in a 1000-dimensional space .
we have a derived , through concatenating , sets of 5 items in the same 1000-dimensional space .
so the resultant euclidean mappings illustrate well the shared provenance in the two cases .
the word analysis is less clear - cut . given the inherent dimensionality of the euclidean , factor space is min(@xmath36 ) , we have , on the one hand 1000 words projected into a 499-dimensional space , and on the other hand 1000 words projected into a 4-dimensional space .
the locations of a given word in these two spaces are not directly related .
we investigate supplementary elements to expedite the computation .
carry out analysis in a small dimensional space , that nonetheless takes account of all relationships in an aggregated way . then complete the analysis on all the data by projecting into the euclidean , factor space .
we took the 306 collected sets of , each , 500 recipes ( save for the case of 2 recipes short of this in one such collected set : hence we were dealing with 152,998 recipes ) .
the number of words found in this data ( to repeat : of one or more characters in length ; punctuation deleted ; upper case set to lower case ; numerical and accented characters ignored ) was 101,060 . for initial assessment
, we took the top - ranking 1000 words .
the following principles and practices of correspondence analysis will now be availed of ( see @xcite ) :    * in correspondence analysis , all frequency of occurrence values are divided by the grand total ( so that allows the viewpoint of empirical probabilities ) .
if the row @xmath24 mass is then @xmath37 , the row @xmath24 profile is the row vector @xmath38 with each element divided by @xmath37 .
now consider the 360-set , which is just all 500 recipes , @xmath38 , summed .
the profile of this set is @xmath39 where @xmath40 .
if one recipe is a multiplicative constant of another ( i.e.  elementwise , for each word ) , then their profiles are the same .
* the dual space relations allows supplementary rows or columns to be projected , after the fact , into the analysis .
+ ( in the latent semantic indexing context , `` folding in '' of newly presented rows or columns , that can be considered as a row or column set that is juxtaposed to the original @xmath41 matrix , is described in @xcite ) .
* profiles are analyzed in correspondence analysis , meaning that row vector , and column vector , values have been divided by the associated row , or column , total . consider the row or column total as a mass , as is done in this context . by having all values first divided by the grand total of the rows
/ columns array , we have all values bounded by 0 and 1 .
( note that we require non - negative values for the mass to be workable for us here . ) in this way our 306-set of ( 500 ) recipes is the _ weighted mean _ of the associated recipes + to see this , consider the given frequency of occurrence data divided by the grand total ( over all recipes and words ) , on the set of recipes , that we will represent by matrix @xmath8 .
consider the 500 recipes that are associated with one of the 306-sets . without loss of generality , call them @xmath42 .
we will denote one such recipe by @xmath43 and just right now we are most interested in the set @xmath44 .
@xmath23 is the set of words .
then the _ profile _ of recipe @xmath45 is @xmath46 . + now consider the 360-set .
it is the aggregation ( concatenation ) of the recipes .
so , it is @xmath47 .
its profile is @xmath48 .
figure [ proj1fig ] presents an initial look at the use of the top ranked words ( in terms of frequency of occurrence in the entire data collection , hence : 306 collections of recipes , or 152,998 recipes ) .    .
lower left and right : these two sets of 500 recipes are projected into the analysis of the 306 recipe collections .
what enables this to be accomplished is that the same 1000-word set is used in all cases here.,width=529 ]    in figure [ proj2fig ] we look at the potential for handling massive data sets by projecting into a `` primary '' analysis .
we have this analysis here on the @xmath49 recipe collections times 1000 highest ranked words .
we project into the analysis the first @xmath35 recipe set , and the second @xmath19 recipe set .
the scheme used is as follows .
we take as a main matrix the one that crosses 306 rows with 1000 columns .
then we have a supplementary row set , of 500 rows crossed with 1000 columns .
finally we have a further supplementary row set , of 500 rows crossed with 1000 columns .
the supplementary rows are projected into the correspondence analysis of the @xmath49 matrix , post factum .
we note that it is best to carry out the eigen - reduction on the smaller of the row set and the column set . hence , relative to how we have presented our processing steps above , in fact we worked on the transposed matrices .
( for the @xmath49 matrix , diagonalization is carried out on a @xmath50 matrix . )
dataset , and black `` x '' are the projections , relative to principal factor plane projection of using the original @xmath35 data .
upper right : a black dot , `` . '' , represents projections as supplementary elements of the 500 recipes , relative to the derived 5 recipe - sets ; and a red dot , `` . '' , represents the active correspondence analysis of these @xmath35 data
. so the black dots are efficiently determined , but the red dots show the `` ground truth '' .
lower panel : different set of 5 recipe - collections , as used previously ( e.g.upper left panel here ) , and then independently defined in order to enhance having `` like with like '' in the same recipe - collection .
note that in all cases we are using a 2-dimensional ( albeit optimal ) display.,width=529 ]    in figure [ bothways - diff5 ] , in the upper left we show with `` x '' ( i ) the projection as supplementary elements of a @xmath20 set of data derived from a @xmath35 set of data ; and shown with a red `` + '' ( ii ) the projections of the active correspondence analysis of the @xmath20 set of data .
the @xmath35 set of data was the set of 500 recipes used before  in fact the sequentially first of these batches of recipes ( and as indicated there were 306 of these batches ) .
the word set is 1000 , being the top - ranked words in the entire word - set derived from all recipes . from the @xmath35 set of data
, we took the first 100 recipes , then recipes 201300 , 301400 , and 401500 .
we aggregated the recipes , which is equivalent to concatenating the recipes .
this provided a frequency of occurrence cross - tabulation of @xmath20 .    in figure
[ bothways - diff5 ] , upper right , we show with a dot , `` . '' , ( i ) the 500 recipes with their projections found from using the @xmath35 data as supplementary elements relative to the @xmath20 data .
we show also , using red dots , `` . '' , ( ii ) the active correspondence analysis of the @xmath35 data .    in figure
[ bothways - diff5 ] , lower panel , there is , with an `` x '' , ( i ) the projections of the active correspondence analysis of the @xmath20 set of data , as in the upper left .
then , with a `` o '' , there is ( ii ) the same analysis on a matrix of the same dimensions , but this time with the 5 concatenated sets of recipes being based not on the given sequential order , but rather on the first factor projections of the full @xmath35 matrix .
this was done in order to assess a different derived frequency of occurrence set of data .
arising out of this discussion , we will draw a balance sheet on projections , through assessment of similarity of outcomes .
we use the sums of squared euclidean distances , taken from the principal plane factor projections , which are euclidean .
* quality of results based on sum of squared euclidean distances . *    1 .   * concatenated recipes projected onto full recipe set , versus concatenated recipes : * @xmath20 projected onto @xmath35 , and active analysis of @xmath20 : * 0.03306077 * 2 .   * full recipe set projected onto concatenated recipes , versus full recipe set : * @xmath35 projected onto @xmath20 , and active analysis of @xmath35 : * 0.283562 *    next ,
@xmath20 is `` ordered '' i.e.  based on a chosen sequencing and not an arbitrarily given one .
the chosen sequencing is that of projections on the first factor .
so , in the foregoing experiment , the 5 groups of recipes , each of 100 recipes , had their sets of 100 recipes in the given , arbitrary , sequence .
now , the 5 groups of recipes , each of 100 recipes , have their sets of 100 recipes in a somewhat clustered sequence that is provided by these recipes projections on the first factor .    1 .
* concatenated clustered recipes projected onto full recipe set , versus concatenated clustered recipes : * @xmath20 projected onto @xmath35 , and active analysis of @xmath20 : * 0.9963667 * 2 .   *
full recipe set projected onto concatenated clustered recipes , versus full recipe set : * @xmath35 projected onto @xmath20 , and active analysis of @xmath35 : * 1.097960 *    what is noteworthy here is that the `` ordered '' , i.e.  somewhat more clustered 100-strong recipe - sets , preform less well than arbitrarily - given 100-strong recipe sets .
latter two results above vis -  - vis the former two . )
we draw the conclusion that the lesser coherence associated with the latter is better when it comes to projecting other recipes into this space .
we see a certain parallel between this outcome and the importance of incoherence in compressed sampling in signal processing .
such incoherence is when @xcite ( p.  278 ) , there should be as much spread as possible between the sensing vector , on the one hand , and on the other hand , the `` sparsity atoms '' or basic components that we can use for the signal to be acquired .
we also have an indication from the above that the bigger the better the analysis , in terms of observations , when it is a matter of projecting into the factor space .
first result above , out of the four , being better than the second result . )    we conclude : the bigger the set of observations that we can work on , so much the better .
also if we do need to work on a limited , representative set of aggregates , then lack of coherence ( i.e.  lack of clustering ) is best .
we use again the first 500-recipe set . with the word set , as before , derived from the entire set of recipes , we look at ( i ) the top ranked ( i.e.  most frequently occurring ) 1000 words , ( ii ) the 42,052 word list such that word lengths are greater than 2 , and ( iii ) the full word set , with 101,060 words . in each case
, we therefore used the 500 recipes cross - tabulated with the words in terms of frequency of occurrences . recall again that the word list used was derived from the entire set of recipes ( and not just these 500 ) .
figures [ figevals1 ] and [ figevals2 ] portray the rapid falloff in inherent embedding dimensionality , based on the word set , and associated attribute dimensionality .    in figure [ figevals1 ] , the ( effectively superimposed ) 42,052 and 101,060 word sets are on top , and the 1000 word set on the bottom .
the reason for the two top curves being superimposed is the following .
the full word set is just the words that occur twice or once in the recipes .
hence they are very rare words .
hence , too , while the 500 recipes here are characterized by , on the one hand , 42,052 words , and on the other hand , 101,060 words , nonetheless the extra data is extremely sparse  consisting mostly of zero values .
it is small wonder therefore that these two datasets , relating to words with more than two occurrences , and all words , give practically the same outcome .
the top 8 actual eigenvalues are as follows , if we choose , on the basis of figure [ figevals1 ] to select , e.g.  8 factors as bearing the most important information :    .... xx1c$evals[1:8 ] [ 1 ] 0.3441775 0.2973126 0.2854613 0.2463559 0.2445217 0.2348758
0.2255849 0.2212731 x1c$evals[1:8 ] [ 1 ] 0.2722012 0.1816320 0.1526613 0.1341513 0.1278123 0.1133759       0.1084320 0.1009735 ....    our preference is to focus our interest on just the first three factors , given these eigenvalues .         displaying the eigenvalues themselves . here
the top curve relates to the set of 1000 words , and the bottom  more or less fully superimposed
curves relate to the 42,052 word set and using the entire word set of 101,060 words.,width=529 ]      in section [ sectchar ] , we observed that : in the case of large , or very large , numbers of occurrences of entities , the syntactically correct form will predominate .    in section [ sectproj ]
, we concluded that : the bigger the set of observations that we can work on , so much the better .
also if we do need to work on a limited , representative set of aggregates , then lack of coherence ( i.e.  lack of clustering ) is best .
finally , in section [ sectwordset ] , we concluded that : large , sparse word sets lead to a similar outcome ; and a well selected , smaller sized word set is important for best , i.e.  smallest , reduced dimensionality mapping .
the 152,998 recipes were found to have 101,060 unique words of 1 character or longer , having set all upper case to lower case , and having ignored punctuation , numeric and accented characters .
the distribution of terms follows a power law ( zipf s law ) : the probability of having more than @xmath8 occurrences per word was found to be approximately @xmath13 .
in total we have 152,998 recipes .
a set of 247 recipe ingredients was assembled .
they were chosen from a word list ( unique words ) drawn from the recipes and ordered by decreasing frequency of occurrence .
there were just 122 recipes that did not contain at least one of these 247 ingredient terms .
( that could be due to use of less common ingredients , not figuring in our list of 247 ; or misspellings , of which there were a considerable number ; or some unusual text instead of a recipe .
an example of the latter was a short overview of appropriate wines for types of food . )
figure [ f1f2rec ] shows the projections of recipes on the principal factor plane .
figure [ f1f2ing ] shows the ingredients used , projected onto the principal factor plane .
the ingredients with the strongest contributions to these factors are noted .
looking further , we found that the strongest contribution by ingredients to factor 3 is `` chocolate '' ; and the strongest contribution to factor 4 is `` cheese '' .
the solr ( lucence.apache.org/solr ) search server was used .    1 .
the data to be indexed , and supported for search , is put into xml with the following structure : * ` < add > ` and ` < /add > ` at beginning and end . *
each entry defined by ` < doc > ` and `
< /doc > ` . * a required field providing a unique identifier : + ` < field name=''id''>mm000001102.txt</field > ` . * other fields , such as the following for bounding box search : + ` < field name=''xcoord''>-0.7341409</field > ` + ` < field name=''ycoord''>-0.09961348</field > ` + ( note that these coordinates are the principal factor coordinates resulting from a correspondence analysis .
i.e. , they are the factor 1 and factor 2 projections , respectively . ) * other fields such as : + ` < field name=''name''>''21 '' club rice pudding</field > ` * the main text field , tagged by : ` < field name=''recipe''> ... </field > ` .
+ the xml file was of size over 205.5 mb .
it was put in directory ` example / exampledocs ` .
2 .   two files in directory ` example
/ solr / collection1/conf ` need to be fully cognizant of the xml data to be used .
these are ` schema.xml ` and ` solrconfig.xml ` . in the latter ,
changes made included the following , with regard to ` mlt ` or ` mlt ` , `` more like this '' option .
+ in the ` requesthandler ` there is : + ....         < arr name="components " >           < str > query</str >           < str > facet</str >           < str > mlt</str >           < str > highlight</str >           < str > stats</str >           < str > debug</str >         <
/arr >      < /requesthandler
>    ...
< ! -- query settings -- >         < str name="deftype" > edismax</str >         < str name="qf " >            recipe^1.0         < /str
>         < str name="df" > text</str >         < str name="mm">100%</str >         < str name="q.alt">*:*</str >         < str name="rows">10</str >         < str name="fl">*,score</str >           < str name="mlt.qf " >            recipe^1.0         <
/str >         < str name="mlt.fl" > recipe</str >         < ! -- following : no . of nearest neigbours returned in mlt -- >         < int name="mlt.count">1</int >           < ! -- faceting defaults fm changed to off -- >         < str name="facet" > off</str > .... 3 .   in file `
schema.xml ` the fields used and their definitions need to be defined .
this included : + ` < field name="recipe " type="text_general " indexed="true " stored="true " ` + ` termvectors="true"/ > ` + also : + ....     < field name="xcoord " type="double " indexed="true " stored="true " / >
< field name="ycoord " type="double " indexed="true " stored="true " / > ....
4 .   for the browser - based querying ,
this is management by the ` velocity ` component , that is in subdirectory ` example / solr / collection1/conf / velocity ` .
+ the files changed there , to be appropriate for the data analyzed , were : ` browse.vm , footer.vm , header.vm , product-doc.vm , query.vm ` .
display images ( see the principal factor plane in figures [ screen1 ] , [ screen2 ] ) used there are located in the named image directory , which is currently + ` example / solr - webapp / webapp / img ` .
the solr server is started thus , in directory ` example ` , where in this example , 1 gb of memory is provided :    ` java -xmx1024 m -jar start.jar `    to update , or commence , indexing , in the directory containing the xml data , ` example / exampledocs ` , the following command is issued .
this supposes a running server .    `
post.jar recipes-f1f2-1.xml `    the unique identifier field is the crucial aspect of what gets taken into the index , or updated .
the following access address is used , based on the running server :    ` http://localhost:8983/solr / browse `    through the upper right hand corner link to the admin screen , or directly using ` http://localhost:8983/solr/#/collection1 ` , there is availability of the log file ; querying can be carried out ; statistics of use and of the data can be accessed .
note that as currently configured in this work , ` collection1 ` contains the indexed data .
example of query : `` sugar beer pasta '' .
a required term is specified in the query with a preceding plus sign , and a requirement not to have a term is specified with a minus sign preceding the term .
the `` more like this '' option gives a number of nearest neighbours of the document , and its parameters are defined in the settings in ` schema.xml ` .
figures [ screen1 ] and [ screen2 ] show examples of use .
the correspondence analysis figure at the top is static .
it and the table to its left are provided as navigation aids in search and discovery .
the table to the left of the correspondence analysis principal factor plane is also static .
an alternative could be a text cloud or cloudmap ( see @xcite ) .
an example follows ( all to be placed on one line ) . for this ,
three `` more like this '' near neighbours were required ( i.e. , in file ` solrconfig.xml ` , there was this setting : ` < int name=''mlt.count''>3</int > ` ) .
.... curl -o out.xml ' http://localhost:8983/solr / collection1/        browse?&q = id : mm078001428.txt&wt = xml&mlt = true '   ....    in this case , results are saved to file ` out.xml ` . from a search for recipe ` mm078001428.txt ` , a `` more like this '' request is submitted .    from the file ` out.xml ` here is a little utility to write out just the recipe identifiers ( all on one line ) :    .... awk -v srch="\"id\ " > " ' begin{l = length(srch ) } end{print " \n " }   { t = match($0,srch);if(!t){next}extr = substr($0,t+5,15);printf " % s " , extr } '   < out.xml ....    in this case , this gives :    .... mm078001428.txt mm110501451.txt mm158501305.txt mm161501159.txt   ....
using identifier ( and recipe ) `` mm078001428.txt '' , with name `` no fat , no salt , no sugar vanilla ice cream '' , we found its `` more like this '' best match to be `` mm110501451.txt '' , with name `` sugar - free cappuccino ice cream '' . using the latter
, we found its best match to be the former , thereby showing consistency .
more specifically , the two recipes in this case were mutual or reflexive nearest neighbours .
see @xcite for the use of this principle ( of mutual nearest neighbours , and also nearest neighbour chains ) in agglomerative hierarchical clustering algorithms .
we also sought best match recipes using the correspondence analysis factor space .
this latter is of course euclidean .
we used the full space dimensionality ( hence 247 less 1 due to linear dependence through centring the cloud of recipes , and the dual cloud of ingredients ) . for recipe ``
mm078001428.txt '' , we got its closest neighbour as `` mm048501554.txt '' , `` fruit flavor milk shakes '' . for recipe ``
mm110501451.txt '' , we got its closest neighbour as `` mm161501159.txt '' , `` vanilla ice cream - diabetic * ww no.2 ''     is the nearest neighbour of point @xmath51 .
point @xmath52 is the nearest neighbour of point @xmath53 .
point @xmath54 is the nearest neighbour of point @xmath52 , and reciprocally @xmath52 is the nearest neighbour of point @xmath54.,width=377 ]    in figure [ nns ] the geometric situation is depicted , irrespective of how nearest neighbour is defined ( it need not be a distance but some dissimilarity satisfying @xmath55 ; @xmath56 ; and the closer @xmath51 is to @xmath53 , i.e.  the more alike they are , the smaller the value of @xmath57 ) .
the squared euclidean distance in the full dimensional euclidean factor space gives for the pair of best match recipes furnished by solr , 18.94413 .
then in the full dimensional factor space , the best match distance squared of `` mm078001428.txt '' , as noted above , was 4.934411 ; and the best match distance squared of `` mm110501451.txt '' was 6.103886 .
to note that the correspondence analysis best match distances are supported by the data is to be unfair to what is , with solr , a different framework for determining best matches .
this includes using all words , with stemming and lemmatization , compared to the 247 ingredients used in the correspondence analysis case .
we note that the solr best match information lends itself to looking for clusters such that nearest neighbour chains are followed , and once a ( mutual or reciprocal ) nearest neighbour pair is found , they can be agglomerated with no impact on close - by recipes .
this is due to bruynooghe s reducibility property .
background on nearest neighbour chain , and reciprocal nearest neighbour based hierarchical agglomerative clustering can be found in @xcite .
99    s. deerwester , g.w .
dumais , s.t .
furnas , t.k .
landauer and r. harshman , `` indexing by latent semantic analysis '' , _ journal of the american society for information science _
, 41 , 391407 , 1990 .
n. eiron and k.s .
mccurley , `` link structure of hierarchical information networks '' , proc .
third workshop on algorithms and models for the web - graph ( waw 2004 ) , _ lecture notes in computer science _ , pp . 143155 , 2004 .
i. fellows , word clouds , software in r written by ian fellows , version 2.2 , 2012 - 09 - 05 , http://log.fellstat.com/?cat=11    p. hall , j.s . marron and a. neeman , `` geometric representation of high dimension , low sample size data '' , _ journal of the royal statistical society b _ , 67 , 427444 , 2005 .
b. le roux and h. rouanet , geometric data analysis , kluwer , 2004 .
b. le roux and h. rouanet , multiple correspondence analysis , sage , 2010 .
m. mitzenmacher , `` a brief history of generative models for power law and lognormal distributions '' , _ internet mathematics _ , 1 , 226251 , 2004 .
a. morin , `` interactive text mining and applications '' , _
ntts 2011  new techniques and technologies for statistics 2011 _ , presentation ( 78 pp . ) at http://www.cros-portal.eu/book/export/html/17    f. murtagh , _ multidimensional clustering algorithms _ , physica - verlag , 1985 .
f. murtagh and a. heck , _ multivariate data analysis _ , kluwer , 1987 .
f. murtagh , `` on ultrametricity , data coding , and computation '' , _ journal of classification _ , * 21 * , 167184 , 2004 .
f. murtagh , `` identifying the ultrametricity of time series '' , _
european physical journal b _ , * 43 * , 573579 , 2005 .
f. murtagh , _ correspondence analysis and data coding with r and java _ , chapman & hall / crc , 2005 .
f. murtagh , a. ganz , s. mckie , j. mothe and k. englmeier , `` tag clouds for displaying semantics : the case of filmscripts '' , _ information visualization journal _ , 9 , 253 - 262 , 2010 .
e. neuwirth and l. reisinger , `` dissimilarity and distance coefficients in automation - supported thesauri '' , _ information systems _ , * 7 * , 4752 , 1982 .
newman , `` the structure and function of complex networks '' , _ siam review _
, 45 , 167256 , 2003 .
starck , f. murtagh and j. fadili , _ sparse image and signal processing : wavelets , curvelets , morphological diversity _ , cambridge university press , 2010 .
( 2nd edition forthcoming 2015 . )
recipe mm160001 collection , number 245 out of 500 in sequence from this collection , is as follows .    ....
title : the perfect roast chicken with roasted shallots and portob   categories : lifetime tv , life5        yield : 4 servings          4 lb young roaster             salt and pepper ; to taste        1 md onion ; halved and peeled        3     cloves garlic ; peeled and             -smashed        1 bn fresh herbs ; rosemary thyme             ; flat - leaf parsley      1/4 c   olive oil        2 c   chicken broth ; divided        8 lg shallots        8     portobello mushrooms ; stems             -removed and             ; cut in half      1/2 c   dry white wine      how to prepare the chicken :       1 .
preheat the oven to 350 f. season the skin and the inside of the cavity    of the chicken generously with salt and pepper .       2 .
place the onion , garlic , and fresh herbs in the cavity and truss the    chicken . in a roasting pan over medium - high heat , heat the oil until it    begins to smoke .
place the chicken , breast - side - down , in the oil and cook until all the    sides off the chicken are completely browned .
place the bird , breast - side - up into the oven and baste with 1/4 cup of    the chicken stock .
continue basting with 3/4 cup of the chicken stock every    10 minutes until the chicken is done , approximately 1 hour , or until the    chicken reaches an internal temperature of 170 f.       5 .
add the shallots 20 minutes after the chicken has been put into the oven    and the mushrooms 40 minutes after the chicken has been put into the oven .
remove the cooked chicken , shallots , and mushrooms from the oven and set    on a platter , cover and let rest for 10 minutes .       how to prepare the sauce :       1 .
place the roasting pan back on the stove over medium - high heat and bring    the juices in the pan to a boil . using a small ladle remove any of the fat    that has risen to the top of the pan .       2 .
add the white wine to the pan and , using a wooden spoon , scrape up the    browned bits from the bottom of the pan . reduce the wine by half and add    the remaining 1 cup of chicken stock .
reduce the stock by half , season with salt and pepper and add the fresh    thyme .
carve the chicken and arrange on a platter with the shallots and    mushrooms .
spoon the pan juices over the meat .
the perfect roast chicken with roasted shallots and portobello
< a9 > 1997 lifetime entertainment services .
all rights reserved .
mc formatted using mc buster by barb at pk         converted by mm_buster v2.0l .
----- ....    we note the following in this recipe text : abbreviations ( `` lg '' , large ; `` lb '' , pounds ; `` c '' , cup ; etc . ) .
source and other summarized or abbreviated data . a control character ( `` @xmath58a9@xmath59 '' ) .
list of 247 ingredients used , with their frequencies of occurrence .    ....            salt           sugar           water          pepper             oil          butter           167636          142624          134454          132012          126242          115387            sauce           flour          garlic           cream          cheese         chicken           105909           99870           85080           83859           80835           78635            onion           juice             egg            milk           lemon            eggs            73933           65773           62000           61125           55408           50086            bread          onions            rice       chocolate           olive         vanilla            48789           44169           41645           40737
39539           38386             cake        tomatoes         parsley        potatoes         vinegar       vegetable            35317           34894           32217           31423
31340           30735             wine          tomato           beans            beef      vegetables          cloves            29192           28747           28180           28761
27203           27034             soup          orange        cinnamon       margarine       mushrooms           salad            26558           25976           25656           24266
23693           23612             fish            corn           broth          celery         mustard             pie            23267           21212           20832
21193           20192           20081            pasta           fruit         peppers             soy           chili           basil            19743           18989           18695           18036           17148
16354           shrimp            soda         cookies           syrup         carrots           honey            16332           16257           16216           15652
15197           14733           sodium          cookie        parmesan             ice        dressing      cornstarch            14034           14173           13878           13928
13886           13848            thyme           bacon          pastry            lime           yeast          potato            13636           13653           13309           13274
13347           12957            apple         protein         spinach       casserole         oregano           cumin            12642           12129           11889           12106
11855           11996           nutmeg     cholesterol         raisins           clove         coconut       pineapple            11756           11376           11019           11042           10892           10864            roast           chips            chop           puree         topping        marinade            10758           10667           10637           10638
10542           10586          noodles            loaf        desserts        cilantro           yolks          peanut            10441           10456           10371           10405
10362           10258          italian           chile       seasoning          apples         almonds            peas            10285           10293           10350           10280           10216           10459           sesame          turkey             ham         cabbage         paprika            leaf            10184            9975            9725            9736            9689            9703            mixer          yogurt       coriander    carbohydrate         sausage         cayenne             9211            9067            8903            8765            8918            8892
lamb           wheat            bean         walnuts           cakes            mint             8571            8639            8573            8470
8398            8256          lettuce      mayonnaise          pecans          sherry           cocoa         pudding             8159            8000            7893            7950            7835            7843          cheddar           grain          salmon          olives          carrot           shell             7877            7806            7724            7744            7767            7511       vegetarian        broccoli        zucchini           salsa          flakes          grease             7405            7413            7290            7268            7104            7079          chinese        shallots         poultry        mushroom           steak        rosemary             7079            6877            6969            6857            6927            6699         eggplant          chiles            rind           curry          coffee            dill             6708            6804            6696            6798
6602            6683           spices          breads      buttermilk worcestershire          starch         seafood             6648            6670            6351            6273            6031            6135          pumpkin         gelatin   carbohydrates       scallions          almond          chives             5820            5899            5749            5815
5662            5621            spice           meats           herbs            tofu         dessert           pizza             5576            5591            5496            5409            5466
5398     strawberries          juices         muffins         mexican       tortillas           chops             5373            5318            5290            5176            5139
4996              rum           icing           soups        cornmeal          emeril       asparagus             5014            4917            5084            5032            4874            4740           sauces          muffin          fruits        stuffing           jelly          salads             4792            4778            4737            4704            4756            4731           shells        jalapeno      mozzarella          banana          steaks          greens             4645            4581            4484            4387            4354
4390        spaghetti         broiler        cucumber        cherries           toast          cherry             4343            4346            4679            4351            4307
4358          chilies            yolk           gravy           cider            root         tabasco             4324            4289            4258            4244
4304            4272             oats           fiber            veal        tortilla            sage           dijon             4225            4135            4061            4085            4040
4040          bananas           broil            tuna        molasses         peaches     peppercorns             3970            3970            3914            3949
3939            3928            candy            duck        tarragon          fennel            tart         custard             3884            3805            3851            3789            3814            3774            maple        scallops            stew          brandy         berries           pears
3749            3871            3802            3780            3777            3559             crab   confectioners        crackers        biscuits        bouillon         peanuts             3611            3571            3570            3464            3526            3440            leeks            beer        turmeric          stalks         ricotta         oranges             3392            3450            3385            3393            3274
3278          lentils       raspberry         cracker            herb     raspberries      strawberry             3260            3216            3128            3113            3083            3093              jam             3151   ....
figure [ textplot ] shows an improved presentation of figure [ f1f2ing ] , through movement of projections ( from the red dots , indicated by the stick lines ) , as implemented in the textplot program in the word cloud package in r @xcite .
in the recipe set , there were 101,060 unique words .
words were as defined by us : length @xmath60 ; all upper case set to lower case ; punctuation removed ; numeric and accented characters ignored .
variations on spelling included the following example , with numbers of occurrence in the recipe set :    .... zucchini 7155 zuchinni 86 zuccini 57 zucchinis 56 zuchini 40 zucchine 14 zucchinni 7 zucchin 6 zuckinni 5 zuchinnis 4 zuccinni 3 zuccchini 2 zuccihini 2 zuchhini 2 azucchini 1 brushingzucchini 1 lambzucchini 1 ofzucchini 1 pzucchini 1 zucchiniabout 1 zucchinii 1 zucchinnis 1 zucchni 1 zucchnini 1 zuccinis 1 zuchhinis 1 zuchine 1 zuchinis 1 zucinni 1 zzzucchini 1 ....      we examine here , using the 152,998 @xmath1 247 ingredients set , the projections in the principal factor plane of singulars and plurals .
recall that the principal factor plane , while being the best visualization of the data , with inherent dimensionality 246 , only accounts for just over 3.8% of the total inertia .
the singulars used were as follows , with additionally their 23 corresponding plurals : `` strawberry '' , `` carrot '' , `` muffin '' , `` egg '' , `` apple '' , `` yolk '' , `` tortilla '' , `` bread '' , `` cookie '' , `` pepper '' , `` juice '' , `` vegetable '' , `` clove '' , `` carbohydrate '' , `` spice '' , `` soup '' , `` peanut '' , `` onion '' , `` olive '' , `` cracker '' , `` almond '' , `` bean '' , `` banana '' .
figures [ sings ] , [ plurs ] , and also [ sandp ] with links drawn for all singular and plural pairs , show the close association in the principal factor plane . some singulars and plurals
are admittedly somewhat separated , e.g.  `` cracker '' , `` yolk '' , pointing to some difference in semantic  clearly possible in practice  and contextual use of singular and plural .
overall the association between the terms is well exemplified in the figures .