online social networks ( osn ) like facebook and twitter have become an essential part of our daily life , e.g. facebook has over 1 billion monthly active users  1/7 of the world population . despite their overwhelming success ,
the centralized control of these services have led to serious concerns about user privacy , censorship @xcite and operational robustness .
they are now obvious targets or even vehicles of many totalitarian regimes which constantly seek to monitor and control information dissemination among their people .
some even argues that the centralized and absolute control power associated with osn services have resulted in many non - user - friendly management styles or policies , e.g. the real world cases reported by indiewebcamp participants .
the aforementioned concerns have motivated the active development of decentralized social networks ( dsn ) in the recent years with the goal to let users regain better control over their personal data .
although numerous dsn projects have been launched in the past few years , to date , very few of them have managed to proceed beyond the prototyping or paper - publishing stage .
even the most successful , often cited example of dsn , namely , diaspora @xcite , has only about 0.4 million users , let alone active ones .
while this may not be surprising given the wide range of challenges in implementing , bootstrapping and operating a dsn @xcite , we believe that the biggest hurdle for the widespread adoption of dsn services is the lack of a gradual transition path for a user to migrate to a dsn without leaving behind a large portion of his / her friends who are likely to stick with only existing osns for convenience . as such , even for the minority early adopters who are determined to move to a dsn service for its enhanced privacy protection and possibly richer functionalities , the ability to perform cross - platform socialization will be critical . towards this end , we propose snsapi , a middleware which enables an end - user to aggregate and stitch together all of his / her online social activities and become a node in a `` meta social network '' @xcite as illustrated in fig . [
fig : instantiated - metasn ] .
in fact , many existing sns users already consciously or subconsciously `` stitch '' the heterogeneous platforms together by selectively relaying messages across different platforms after some manual filtering and editing .
for example , after reading a `` juicy '' gossip from a blog , you may forward it manually to your personal friends on facebook .
such manual cross - platform forwarding operations can be viewed as the formation of a meta social network which overlays on top of the existing sns .
our objective is to provide the tools and systems that can help users to better perform cross - platform socialization .
we expect those tools and systems can gradually detach users from existing centralized osns and allows a smooth transition to the decentralized ones . in summary , this paper has made the following technical contributions :    @xmath0 after reviewing related work in section [ sec : related - works ] , we analyze the challenges of building a dsn and propose a meta social networking approach in section [ sec : a - meta - social - networking - approach - for - decentralization ] .    @xmath0 as described in section [ sec : current - design - and - implementation ] , we have designed , implemented and released multiple iterations of an open - source middleware called snsapi to support cross - platform socialization over existing sns .
sample applications built using snsapi are also presented to demonstrate its flexibility and extensibility .
@xmath0 design choices , observations and maintenance experience of snsapi are discussed in section [ sec : design - maintenance - and - refactor - experience ] .
@xmath0 we have conducted an experimental deployment of a 6000-node dsn over planetlab based on the snsapi middleware and measured its empirical performance . as a comparison
, we also provide an analytical model to characterize the performance of the system with results presented in section [ sec : a - medium - scale - dsn - in - the - wild ] .
@xmath0 we conclude our findings and propose follow - up work for the future in section [ sec : conclusions ] .
to overcome the drawbacks of centralized sns , decentralized social networks ( dsn ) such as diaspora @xcite , musubi @xcite and onesocialweb @xcite have recently been proposed and implemented . according to our classification in section [ sec : a - meta - social - networking - approach - for - decentralization ] , most works are distributed social network ( dissn ) .
two examples of federation protocols are onesocialweb @xcite and ostatus @xcite , which are not bound to any particular software implementation . for the dissns , there are several different system architectures .
prpl @xcite and musubi @xcite are two examples of fully decentralized systems .
each user corresponds to a dsn node and they only have the view of their direct connections .
diaspora @xcite is a super - node based system .
users can setup their own `` pod '' ( server ) or register on other `` pods '' .
peoplenet @xcite and rflex @xcite are systems combining heterogeneous communication technologies .
peoplenet @xcite first routes messages to the neighbourhood of the target using cellular infrastructures and then leverages opportunistic forwarding to complete the last hop(s ) of the delivery .
rflex @xcite targets spontaneous group messaging and transparently switches between the cloud backend and nfc/ d2d connections .
epost @xcite and peerson @xcite are two dht - based solutions .
while epost @xcite follows dht loop for message delivery , peerson @xcite only uses dht to locate users before establishing direct connections for data transfer .
these systems and protocols adopt a clean - slate design and have few or none provisions to support migration .
although some systems have import functions from existing service providers , they still have the lock - in effect because they do not give a common data structure abstraction to support convenient export and inter - operation . in terms of the ability to perform cross - platform operation , there are many existing aggregation services on the internet . for example
, ifttt @xcite abstracts internet - based information services as `` channels '' and allows users to define `` if - this - happens - then - do - that '' ( ifttt ) forwarding recipes .
yahoo pipes @xcite supports fewer platforms but allows more sophisticated processing by designing a story - board of logical operations ( e.g. condition , loop , etc ) .
yoono @xcite , socialoomph @xcite , and hootsuite @xcite are other example services which provide partial interoperability between heterogeneous platforms with some basic personalization functions . while these services are useful for novice users , they have at least one of the following problems : 1 ) only configurable but not programmable , which severely limits their functions ; 2 ) non open - source and thus less reusable for automation ; 3 ) not readily extensible to other platforms .
the open - source projects , opensocial @xcite provides an abstraction for different osns but it requires a steep learning for simple tasks .
thinkup @xcite , an open - source web service , reads messages from several osns and stores them in a local database to facilitate the mining of important information .
however , it does not provide a comprehensive abstraction of heterogeneous sns like snsapi and its web service nature also requires more infrastructure support ( e.g. lamp environment ) , making it hard to run under resource critical environment , e.g. mobile devices .
before delving into the details of our approach for the decentralization of sns , we briefly discuss three alternative paths towards this goal . according to the degree of decentralization , there are three types of dsn as illustrated in fig .
[ fig : degree - of - decentralization ] :    @xmath0 distributed social network ( dissn ) . in a distributed social network , nodes are homogeneous and runs the same software package  the solid circles .
they use the same protocol  the solid lines .
most dsn proposals and implementations are actually dissn , e.g. diaspora @xcite , musubi @xcite etc . in this sense ,
the dissn approach is * software - package oriented*.    @xmath0 federated social network ( fedsn ) .
nodes in a fedsn can adopt different software packages  the `` solid '' and `` dashed '' circles in fig .
[ fig : fedsn ] as long as they support a common protocol , a.k.a . the federation protocol represented by solid lines in fig . [
fig : fedsn ] .
email and ostatus @xcite are two examples of this approach . in short , fedsn is * protocol oriented*.    @xmath0 meta social network ( metasn ) @xcite .
metasn is the network of different social networks : in this case , 1 ) there are many different software packages ( server/ client implementations ) and 2 ) nodes may use different protocols  the solid or dashed lines in fig . [
fig : metasn ] .
information diffusion on metasn does not rely on a single software package or a single protocol .
instead , it is done via multi - hop bilateral communications .
there is a common object to hold information and it can have different representations on different links . in other words , metasn is * object oriented*. one can see that metasn is highly reminiscent of our real life social network in which different people may talk different languages .
socialization is not dependent on a single language but on multi - hop bilateral communications .
we have designed a cross - platform middleware , snsapi , to realize the metasn approach described above .
[ fig : instantiated - metasn ] illustrates a metasn formed by snsapi and existing osns .
users running an snsapi - based app can socialize with users of mainstream osns , e.g. twitter and facebook , using the corresponding protocol .
they can also conduct the same types of online social activities with each other in a transparent manner via other communication channels supported by snsapi , e.g. ( e)mail and ( r)ss .
in contrast , the twitter user and the facebook user at bottom of the figure can not communicate with each other due to the lack of a common protocol .
snsapi users also have the ability to aggregate , filter and/or relay messages in this metasn . as such
, the snsapi - based nodes can enable multi - hop social interactions among users who are originally locked into their own silo of existing osns .
in short , the snsapi can help to jumpstart and then grow a dsn in an incremental , gradual manner .
our cross - platform middleware , snsapi , is designed based on the following principles : 1 ) focusing on solving the 80% problems ; 2 ) staying open to support future service evolution ; 3 ) trading execution performance for script development efficiency . in this section ,
we first introduce the overall architecture and the core abstractions .
after that , we present some sample applications to show the flexibility and programmability of snsapi .
figure [ fig : the - snsapi - architecture ] depicts the architecture of snsapi , which consists of the following three layers :    @xmath0 interface layer ( il ) .
is the base class for all kinds of sns and can be derived to implement real logic that interfaces with those platforms . in snsapi terminology ,
the module containing derived classes is called `` plugin '' ; the derived class from is called `` platform '' ; the instance of the class is called `` channel '' . message and message list classes are also defined in this layer to provide the core abstraction to support metasn .
@xmath0 physical layer ( pl ) .
there are many common operations when interfacing with different sns , e.g. http , oauth , error definition , etc .
we implement them in the pl so that plugins can reuse them . to enhance flexibility ,
we provide a wrapper class for most 3rd party modules so that users can substitute ( all or part of ) them with others without modifying the core of snsapi .    @xmath0 application layer ( al ) . to reduce repeated works in batch operations
, we developed a `` pocket '' class in al .
it is a container to hold multiple channels . for most applications
, pocket should be the service access point ( sap ) to snsapi . in this way
, end users can enable new channels by simple configurations and no intervention from app developer is needed .
, title="fig : " ]      we abstract the interfaces of different sns platforms to support a set of common primitives . since not all platforms have direct support for those primitives , we often have to translate the functions in order to provide a unified view to the upper layers .
the three fundamental primitives and some examples are as follows :    @xmath0  this name is short for either `` authorization '' or `` authentication '' .
for example , in order to access osns like facebook , we need to go through the oauth flow . in order to access email platform ,
the usual approach is to authenticate via username/ password .
@xmath0  this function gets the latest messages targeted to the user on a certain channel . for an osn
, it retrieves the home timeline . for email platform
, this function gets latest messages from inbox , which resembles the home timeline of osns .
is the basic `` read '' function for a platform .
@xmath0  it provides the basic `` write '' function for a platform . on an osn , this can be a status update , blog update , or others , depending on the platform .
on rss platform , this function writes a new entry to the feed .
although and can be named as `` read '' and `` write '' from a modeling viewpoint we stick to the osn convention .
this philosophy is adopted throughout snsapi : instead of providing a unification scheme from scratch and requiring other existing/ new osn platforms to follow ( as in the case of opensocial @xcite ) , we adapt snsapi to existing platforms and work out a common divisor . from the basic primitives , two more
`` write '' functions can be derived : reply and forward .
the definition is different across osns due to differences in the positioning of each platform .
the model of `` write '' operations observed on existing osns are further discussed in section [ sec : the - model - of - write - in - osn - platforms ] . following are the definitions within the context of snsapi :    @xmath0  by replying a message , the user is able to add comments to the original message .
if bob replies a message posted by alice , she will get a notification in some way .
where to post the comment is not regularized and can be different across platforms .
@xmath0  by forwarding a message , the user is also able to add comments to the original message .
if bob forwards a message posted by alice , she may or may not get a notification , depending on the platform .
however , the forward message together with the original message will appear in the message update list of bob . note that can be readily implemented via and we have already included this cross - platform forwarding function in the base class .
plugins can implement platform - specific forwarding functions to enrich the functionalities .      as is discussed in section [ sec : a - meta - social - networking - approach - for - decentralization ] , we need to design a common object to allow the formation of metasn .
message is the most important data structure sns but comes in different forms , e.g. json object returned by the api of one osn , rfc822 formatted texts of an email , or an xml entry of a rss feed .
even if we only consider conventional osns , the json objects returned can still be quite different . in order to facilitate cross - platform operations , we abstract a object .
it has the following four components :    @xmath0  it contains sufficient information for one to identify a across platforms .
it is only designed to be used by snsapi plugins .
users are not supposed to tap into the fields of .
@xmath0 mandatory fields  this includes , , , , and .
those fields are actually the `` greatest common divisor '' of different sns platforms according to our year - long refactoring experience .
the field is used to convey non - text - based information like url , image , and video files and can be an empty list .
@xmath0 optional fields  although mandatory fields provide a common base to perform cross - platform socialization , we may sometimes expect a richer set of functionalities .
examples are like the `` like count '' and `` share count '' on some osns .
those fields are also unified via re - formatting and re - assembling of the raw response from osn providers .
app developers should test the existence before using it since they are optional .    @xmath0 raw data  this is the original data obtained from an osn service provider .
it gives the largest amount of information .
however , app developer should refer to the manual of specific platforms in order to use it .
plugin implements the logic to enable transaction with any specific sns platform .
a standard plugin implements the 5 interfaces defined in section [ sec : abstraction - of - interfaces ] .
it also performs essential data conversions from raw response to the data structures regularized in section [ sec : abstraction - of - data - structures ] .
following is the list of plugins we have already developed :    @xmath0 renren , sina weibo , and tencent weibo  we started with these platforms , which are the three largest osn services in mainland china .
the plugins use restful apis of the corresponding osn .
based on the experience of developing these plugins , we have derived some common building blocks , e.g. http , oauth , and put them in the physical layer ( pl ) .
@xmath0 rss , email and sqlite  these three platforms are not regarded as social networking services in the traditional sense .
however , they are fundamentally the same as the osns mentioned above , i.e. a means to read/ write messages , so it is easy to make them snsapi - compliant .
@xmath0 twitter and facebook  these two platforms are developed based on existing 3rd - party wrappers .
their existence shows that one can easily adapt existing wrappers to be snsapi - compliant , so as to enable seamless cross - platform operation on new platforms .
pros/ cons of using 3rd - party wrappers are further discussed in section [ sec : user-3rd - party - wrappers - for - plugins ] .
the so - called snscli , a command - line interface in form of a python shell , has been developed as an application ( app ) of snsapi .
some typical usages include : 1 ) use snscli to interface with programmes / scripts written in other languages via stdin/ stdout ; 2 ) support interactive debugging for a platform .
researchers have also developed crawlers in snscli using the vertical interfaces ( section [ sec : vertical - interface - and - horizontal - interface ] ) to facilitate the collection of data over large - scale osns .
after our third round of refactoring at the beginning of 2013 , an open - source contributor from github , tommy alex , built a graphical user interface ( gui ) called snsgui ( fig .
[ fig : screenshots - of - gui - on - pc - and - android ] ( a ) ) which runs on major computing platforms including linux , windows , and os x. later tommy also ported snsapi to android with a new ui ( snsdroid in fig .
[ fig : screenshots - of - gui - on - pc - and - android ] ( b ) ) .
it is noteworthy that tommy managed to develop snsgui and snsdroid in 10 days and 7 days , respectively . before developing snsgui
, he has no prior knowledge of snsapi .
this demonstrates the flexibility and portability of snsapi .
we have also developed another app called snsrouter @xcite which provides a web ui and a ranking framework for cross - platform socialization .
based on the unified abstraction of snsapi , the ui was developed in one week .
user efficiency is improved due to the convenient cross - platform operations plus prioritization of incoming messages . among all kinds of benefits ,
it is worth to note that snsrouter can readily become a node in an ad hoc dsn by properly configuring some channels supported by snsapi .    due to space limit
, we can not present other existing apps that have been developed for snsapi , e.g. automatic forwarder , automatic replier , automatic backup script .
they can be found in our open - source repository .
they all share one distinguishing feature when comparing to their traditional counterparts : these apps all work in a platform - independent fashion and are ready to operate on new platforms .
since the initiation of snsapi project , we have gone through three major rounds of refactoring .
the project is still under active development and there will be more forthcoming features . in this section ,
we provide a comprehensive discussions of our experience in designing , maintaining and refactoring snsapi .
the philosophical shifts and key observations extracted from our experience may shed light on future design of similar systems .
we categorize the functions provided by different social platforms into three groups .
as is described in section [ sec : abstraction - of - interfaces ] , base functions include , and ; derived functions include and .
any other osn - specific functions are regarded as extra functions , e.g. album list query and like .
extra functions are not very common across platforms , so we do not try to unify them .
[ fig : interface - horizontal - vertical.eps ] illustrates the three types of functions .
based on this categorization , we can deal the aforementioned functions in a horizontal manner .
while providing horizontal interfaces is the original goal of snsapi , we found that some users expected snsapi to be a full - fledged wrapper for a given osn platform .
such mis - matched perception roots from the fact that there exists many ad hoc wrappers in all kinds of languages for a wide range of osn platforms .
we realized that the support of the horizontal interfaces alone will not be adequate : in order to provide a unified set of horizontal interfaces , we are forced to trim down existing operations in some platforms which results in a loss of functionalities . to overcome this dilemma
, we have to expose an additional set of vertical interfaces which support a richer set of functions but are allowed to be drastically different across various platforms .
the horizontal interfaces can be used to systematically stitch different social platforms together and they require a very mild learning curve .
the vertical interfaces must be used with the assistance of documents from service providers .
they are not the recommended way of using snsapi because apps have to deal with upcoming changes from the osn providers .
nevertheless , developers may find it useful when they are dealing with a specific platform , e.g. large - scale crawling as is used in section [ sec : a - medium - scale - dsn - in - the - wild ] .
although snsapi tried to conform to existing services from the very beginning , the models provided by existing platforms may not be ultimately ideal .
we observed different models of `` write '' operations , i.e. , and , from existing osns .
a summary of the write models is presented in table [ tab : write - models - of - different - osn - platforms ] and we discuss the advantages/ disadvantages as follows .
@xmath0 twitter : there is only one operation
twitter has a `` retweet '' built - in ( official retweet , ort ) but users find it not very useful because they can not put comments while retweeting . towards this end , many twitter users are still using the old user - invented conventions ( user - invented retweet , urt ) by updating a status in the form : .
this user invented `` rt '' notation is closer to the `` forward '' operation defined in section [ sec : abstraction - of - interfaces ] .
if the `` '' appears at the beginning of a status update , it is regarded as a `` reply '' . in other words ,
the reply operation on twitter is still an update operation in essence . @xmath0
facebook : there are three operations regarding a status
update , share , and reply .
facebook s reply is the same as our definition in section [ sec : abstraction - of - interfaces ] .
we will term this type of reply as facebook - style reply ( fbsr ) , in contrast to the above mentioned twitter - style reply ( tsr ) .
fbsr is more privacy preserving but snsapi will lose track of those reply messages because they do not enter the timeline of the replier .
when `` share '' one object on facebook , users can add comments to it . at first glance
, it appears as a forward operation in our context .
however , all share operations go to the original status and are threaded there . compared with our notion of `` forward '' , this `` share '' can not track the forwarding traces . in this case
, it is better to textually construct a `` forward'' message by the method directly ( i.e. a urt ) on facebook .
in fact , the general method in is designed to perform a urt if the platform does not have a built - in `` forward '' function in our sense .
@xmath0 renren , sina weibo , tencent weibo : there are three `` write '' operations  update , forward , and reply . in terms of product positioning , renren is a facebook - like service in china ; sina weibo and tencent weibo are two twitter - like ones .
interestingly , their write models are the same and partly differ from both twitter and facebook as presented in table [ tab : write - models - of - different - osn - platforms ] .
.write models of different osn platforms[tab : write - models - of - different - osn - platforms ] [ cols="<,<,<",options="header " , ]      + ( the more * s the better )      friend - list management for osns is so far only available via platform - specific vertical interfaces ( section [ sec : vertical - interface - and - horizontal - interface ] ) . as the designers of snsapi , we have two strong reasons not to horizontally abstract friend - list management functions .
firstly , friend - list management is not a common and frequent operation ( so - called 80% problems ) , so out - of - band ( oob ) management is acceptable .
secondly , the channel - list in snsapi is actually an implementation of friend - list under the fully decentralized scenario .
consider a social network formed using the rss platform ( section [ sec : a - medium - scale - dsn - in - the - wild ] ) .
one user can update status to a rss feed and publish it to some common location known by his/ her friends .
the other users can instantiate a rss channel with the url pointing to this location . in this way
, the snsapi users form a dsn and the channel - list of snsapi is actually the friend - list . instead of unifying friend - list management functions on different osns ,
a better way is to build a channel management application on snsapi to provide a more elegant abstraction .
although our primary goal is to evolve towards a decentralized social networks structure , centralized osn platforms are ( and will remain to be ) very important components under the bigger meta social network paradigm . as discussed before
, one reason is that those centralized osn services are already established and we must avoid link loss during the migration process .
another reason is that centralized osn platforms can actually serve as good message aggregators .
consider a special platform we built in snsapi  . with this platform , one can directly configure the list of users he / she wants to follow . without explicitly adding the target parties as friends via the request/ approval process
, the snsapi user can readily get their public status updates .
this approach gives us a clean model for friend - list management as is discussed in section [ sec : friend - list - management ] .
however , more resource is used for querying those `` friends '' one by one .
the alternative approach , adding those friends on an osn first and configure only one normal channel in snsapi , is more resource saving because a single query is enough to retrieve an aggregated timeline .
this section describes our experimental deployment of a large - scale dsn on planetlab using snsapi .
after instantiating a network of 6000 snsapi - based nodes on planetlab , we replay real - world update/ forward activity traces collected from sina weibo , one of the top osns in china . to our best knowledge , this is not only the largest experimental validation and systematic evaluation of any dsn solutions in the literature but also the first one which involves the replaying of real osn traces in this scale .
our experiment platform has three major components :    @xmath0 snsapi : the cross - platform middleware described in section [ sec : current - design - and - implementation ]  the basic building block for this dsn .
@xmath0 bot : when the experiment starts , a bot running on each instance of a snsapi - based node will continuously perform four actions : 1 ) periodically query timeline from their friends ( ) ; 2 ) update sns status according to a schedule ( ) ; 3 ) forward status according to a data - driven user behavior model to be described in section [ sec : data - driven - user - action - model ] ( ) ; 4 ) monitor nodal resource usage .
we define an important system parameter called query gap ( qg ) which is the interval between two rounds of the timeline query action . in our experiment
, we use a system - wide constant value for qg and show that this simple strategy suffices for the initial formation of a medium sized dsn ( e.g. 6000 nodes ) . to avoid synchronization problems , every bot sleeps for a uniformly random duration between 0 and qg before the issuing its first timeline - query to its neighbours .
@xmath0 planetlab manager ( plm ) : planetlab is a global research infrastructure widely used to evaluate new networking systems .
planetlab nodes provide no guarantee on their service availability and can go online / offline at any time .
this is ideal to evaluate the robustness and elasticity of our proposed dsn architecture .
we have also developed a toolbox to manage our experiment on pl which includes modules for 1 ) base environment deployment such as the nodal installation of python2.7 and virtualenv ; 2 ) node - screening which help us to avoid machines with large time - drift and those which are behind incompatible firewall ; 3 ) experiment workflow management like workspace cleaning , bot distribution , initialization/ starting/ terminating each experiment , and nodal log collection .    in principle
, we can connect our bots through all kinds of platforms supported by snsapi .
in reality , most osn platforms require user s intervention before snsapi can access it : app - key application , configuration on the osn , and completion of an authorization flow . to allow automatic deployment of thousands of bots , we choose to use rss in this experiment . for practical deployment in the wild , each pair of snsapi users can choose their preferred communication channel as our metasn model only requires bilateral communications .
we collected real data from sina weibo , the largest microblogging service in china , to evaluate the proposed ad hoc dsn based on snsapi .
we leverage the vertical interface ( section [ sec : vertical - interface - and - horizontal - interface ] ) of snsapi to crawl the profile , status update , forwarding history , as well as the followee list of each user . in order to get a meaningful trace to evaluate our system ,
we need a tightly knitted group of active users with content frequently originated from and forwarded within the group .
the following heuristic to used to identify such group of candidate users : 1 ) we use 60 + users categorized as `` programmers '' by sina weibo as our seed ; 2 ) from the forwarding history of these seed users , we extract 6000 + more users . the overall development plus execution time of this snsapi - based sina weibo crawler is less than two days .
this shows the remarkable development and execution efficiency provided by snsapi . in total , we collected 33 gb of raw json data , which includes 12 million status update / forward actions , involving 480,000 users in the forwarding chain .
one of the primary reasons for users to use osn is to identify and disseminate interesting messages . using only passive measurements , it is impossible to determine what messages are seen and what messages are interesting to each user . as a surrogate , we consider the forwarding behaviors , which are strong indicators of user interest and message relevance .
a key objective is to determine the additional forwarding latency caused by our snsapi - based dsn when comparing to that of a centralized osn .
consider the process illustrated in fig .
[ fig : user - model.eps ] : user @xmath1 post the original message @xmath2 at @xmath3 ; user @xmath4 forward this message by posting @xmath5 at @xmath6 .
the following factors will affect the end - to - end delay from @xmath3 to @xmath6 :    @xmath0 user online / offline pattern : from @xmath7 to @xmath8 the user is offline . the original message @xmath2 is posted during this period but can not be forwarded by @xmath4 .
@xmath0 system delay : this is the time required by @xmath2 to go from @xmath1 s database to @xmath4 s database . for a centralized osn ,
there is conceptually only one database so this delay component should be negligible .
that is , as soon as @xmath1 post @xmath2 , @xmath4 is able to get @xmath2 in his / her database . for dsn , it can take considerable time for @xmath2 to go to @xmath4 s database , which may cause a major increase in latency .
@xmath0 application delay : users rely on certain application to retrieve their home timeline , e.g. the web ui or the mobile app of facebook .
additional delay is added due to various reasons like refreshing frequency or message re - ranking strategy .
users rely on many different applications to access social networks , so this part of delay can be highly variant .
@xmath0 user delay : after the above stages , @xmath4 already has @xmath2 in the home timeline . in order to identify the interesting messages , @xmath4 will spend time on reading the messages , watching attached videos , or even following the embedded links .
additional delay is added until @xmath5 is finally posted at time @xmath6 .
our analysis of the user action cycle shows that : 1 ) system delay causes the major difference between centralized and decentralized social networks ; 2 ) factors like application delay and user delay are too complex to be characterised by simple models ; 3 ) precise user online / offline pattern is only available to the sns providers ; 4 ) the end - to - end delay , i.e. @xmath9 , is observable via passive measurement . towards this end
, we adopt an overall data - driven model , established based on two assumptions : 1 ) a bot is always online , trying to identify and disseminate interesting messages ; 2 ) there is an intrinsic delay for each message to be forwarded .
we use @xmath10 to model the intrinsic delay for each message .
suppose one bot in our dsn see the message @xmath2 at time @xmath11 , then the forward time @xmath12 is determined by the two rules :    @xmath0 1 ) if @xmath13 , let @xmath14    @xmath0 2 ) if @xmath15 , let @xmath16    the first rule is to respect the delay observed from real traces . if our dsn delivers messages quickly enough , then the bot waits until the intrinsic delay is passed .
the second rule comes from the bot assumption .
since the bot is always online and the message is seen after the intrinsic delay period , it forwards the message immediately .
[ fig : user - model - bot.eps ] depicts this data - driven user behavior model .
based on this model , our core performance metric is the extra forwarding delay ( efd ) , defined as @xmath17 .      to launch our experiment ,
we first distribute 6733 bots to 450 planetlab nodes randomly .
the experiment lasts for one day wall - clock time .
we filter out update and forward activities of those users from aug 1 , 2013 0:00:00 to aug 1 , 2013 23:59:59 ( utc ) . during this period
, there are 1168 messages originated by the 6733 sina weibo users and are further forwarded within this group . the query gap ( qg ) is empirically set to 5 minutes . in real life
, users can manually trigger the function if one wants to get the updates of his friends immediately .    after a replaying a 24-hour real trace ,
we collect 57 gb of logs from the planetlab nodes .
we then compute the average resources consumed by each bot .
the histograms and summary statistics are presented from fig .
[ fig : avg_cpu_hist.eps ] to fig .
[ fig : sqlite_db_size_hist.eps ] .
notice that the nodal memory usage includes both the bot and snsapi . without the extra bot logic ,
the typical value is 10 mb , i.e. for running the snscli .
there are two major disk - space consumption in this experiment .
first is the rss feed file : each bot writes its feed to a directory which is then served by a lightweight http server .
second is a sqlite database to support asynchronous operations via the buffering of incoming and outgoing messages . from fig .
[ fig : atom_size_hist.eps ] and fig .
[ fig : sqlite_db_size_hist.eps ] , we see an exponential decay and the typical values are smaller enough to hold in the ram so that disk storage and access can be avoided in real applications .    the c.d.f . and
some percentiles of extra forwarding delay ( efd ) are shown fig .
[ efd_cdf.eps ] .
we see that 68% , 73% , 88% and 99% messages are forwarded with no efd , @xmath18 1min efd , @xmath18 10min efd and @xmath18 1hour efd , respectively .
since social networking is not a real - time service and there are already considerable delay other than system delay , we remark that this efd is acceptable .
the mild resource consumption and small efd shows that snsapi is a viable solution to enable rapid and incremental deployment of an ad hoc dsn .
furthermore , users can run snsapi not only on all major desktop operating systems ( as demonstrated in section [ sec : applications ] ) but also resource - constrained devices like smartphones .      although snsapi supports both pull and push channels ( section [ sec : pull - channels - vs - push - channels ] ) , we only used pull channels , namely rss , in our experiment .
the major parameter for this pull - based dsn is the timeline query gap ( qg ) and we denote it by @xmath19 .
we want to characterise the relation between @xmath19 and the following random variables ( r.v . ) :    @xmath0 @xmath20 : an abstract notation for certain resource .
it is an r.v .
parameterized by @xmath19 .
the expectation of @xmath20 is inversely proportional to @xmath19 . without loss of generality , we let @xmath21 = \frac{1}{h}$ ] .
@xmath0 @xmath22 : cpu usage .
@xmath24 is the cpu usage of the bot logic , i.e. checking and forwarding messages every second .
it is constant w.r.t .
the poll parameter @xmath19 .
@xmath25 is the resource consumption of snsapi .
@xmath26 is the coefficient to be fit later . taking expectation
, we have the relationship : @xmath27 = { \mathrm{e}}[c_b ] + \beta_c { \mathrm{e}}[r ] = { \mathrm{e}}[c_b ] + \frac{1}{h } \beta_c$ ] .
however , we already know from the system design that bot logic will take up the major portion of cpu usage . we expect @xmath26 to be a near - zero coefficient .
@xmath0 @xmath28 : memory usage .
as @xmath19 varies , only query frequency is changed .
memory consumption is not affected by it .
it is a r.v .
independent of @xmath20 and @xmath19 .
@xmath0 @xmath29 : disk i / o .
there are two parts of major disk i / o as discussed in section [ sec : a - case - study - of-6k - node - dsn - based - on - snsapi ] .
the maximum size of rss feed and sqlite db are 20 kb and 8 mb , respectively .
in real applications , they can be fully held in ram , thus eliminating all disk i / o . since this only adds a constant to @xmath28 , we do not build an individual model for disk i / o .
@xmath0 @xmath31 : http usage . in our experiment ,
the major network i / o is caused by http .
others like domain name lookup only create very little network load .
we measure http usage from the followee s point of view .
@xmath32 is the number of http query one bot receives and @xmath33 is the size of http query it serves . since the traces are from a micro - blogging service , the size of messages are upper bounded .
thus , we can assume @xmath34 and formulate the following relationships : @xmath35 = \beta_{h_s } { \mathrm{e}}[r ] = \frac{1}{h } \beta_{h_s}$ ] and @xmath36 = \beta_{h_q } { \mathrm{e}}[r ] = \frac{1}{h } \beta_{h_q}$ ] .
@xmath0 @xmath37 : the delay from the original message to the forward message in our dsn system .
this depends on @xmath38 , the intrinsic delay .
most of the relationships are clear and ready for model fitting .
the end - to - end forwarding delay @xmath37 can be characterized based on the following additional assumptions :    @xmath0 a1 : @xmath19 is the dominant delay component in this system so that other small delays can be neglected .
@xmath0 a2 : we assume an ideal internet connection . given current typical access rate and the size of rss feeds in fig .
[ fig : atom_size_hist.eps ] , one bot can finish the query to all its neighbours within seconds .
this is also very small compared to typical values of @xmath19 so that we can further assume one round of query can be completed immediately upon its issuing .    in reality
, the forwarding traces form a tree .
since our target is @xmath39 $ ] , we can invoke the linearity of expectation , namely @xmath39 $ ] is the summation of per - segment delay . in this way
, we essentially decompose the tree into individual forwarding chains .
denote the length of forwarding sequence by @xmath40 , which will be fit later using data from real traces .
then the expectation of end - to - end delay can be expressed as : ( total expectation ) @xmath41 = \int { \mathrm{e}}[\delta |l = l ] \pr \ { l = l \ } { \mathrm{d}}l\ ] ]    next , we focus on the analysis of a chain of length @xmath42 .
we denote the original message as @xmath43 and the forward messages as @xmath44 .
we denote the absolute time when message @xmath45 is posted as @xmath46 .
let @xmath47 be the per - segment delay so that @xmath48 according to definitions .
the per - segment intrinsic delay is denoted as @xmath49
.    denote the absolute time that a bot starts to pull messages by @xmath31 .
note that the bots sleep a uniformly random duration before the first pull .
then we have @xmath50 $ ] for any fixed time @xmath51 .
we use the deferred decision principle to study the forwarding behaviour of @xmath45 .
that is , we first fix @xmath52 , and defer the realization of r.v . @xmath31 .
@xmath53 is the duration from first pull to the post of @xmath54 . by modular @xmath19
, we get the time elapse from last pull to the post of @xmath54 .
then @xmath55 is the waiting time until next pull and we have a compact expression for @xmath47 according to the data - driven user behavior model described in section [ sec : data - driven - user - action - model ] : @xmath56 if we fix @xmath57 , @xmath58 $ ] and further @xmath59 $ ] .
then the overall delay conditioned on the length being @xmath42 is : @xmath60 we now calculate the extra forwarding delay ( efd ) , denoted by @xmath61 .
this is to use end - to - end delay in our system minus intrinsic delay ( the delay expected in a centralized system ) : @xmath62\ ] ] taking expectation , we have : @xmath63 = \sum_{k = 1}^l { \mathrm{e}}[\delta_k -     i_k ] = \sum_{k = 1}^l { \mathrm{e}}[\max \ { i_k , w \ } - i_k]\ ] ] where @xmath64 is a uniform random variable in @xmath65 $ ] .
the overall efd is : @xmath66 = \int { \mathrm{e}}[\delta_{{\ensuremath{\operatorname{efd } } } } |l = l ]     \pr \ { l = l \ } { \mathrm{d}}l\ ] ] we assume @xmath49 to be i.i.d . and use @xmath38 for a shorthand .
then the expression is simplified to : @xmath66 = { \mathrm{e}}[l ] { \mathrm{e}}[\max \ { i_k , w \ } -     i_k]\ ] ] we can fit @xmath38 and @xmath40 from real traces and give the final expression for efd .
in this section , we run 20 experiments parameterized by @xmath19 and compare the corresponding empirical measurement results with those predicted by the analytical model .
we replay 5 days worth of traces from aug 1 , 2013 0:00:00 to aug 5 , 2013 23:59:59 ( utc ) to drive the experiment in order to reduce diurnal variation . to make the experiment tractable and
also reduce planetlab machine variance , we accelerate the experiment time by 12 times .
that is , the 5-day traces are replayed in 10 real - life hours .
we also trim the dsn topology to 4881 nodes by excluding those nodes having no update or forward activities during this period .
the empirical distribution of @xmath38 and @xmath40 are plotted in figs .
[ fig : distri_intrinsic_delay.eps]-[fig : distri_sequence_length.eps ] . observe that the intrinsic delay of messages follows a power - law decay while exponential decay is exhibited by forwarding sequence length with their distributions given by : @xmath67 where @xmath68 , @xmath69 , @xmath70 and @xmath71 based on fitting against experimental measurements .
the normalization factors can be calculated as : @xmath72     \nonumber\end{aligned}\ ] ] and @xmath73 where @xmath74 and @xmath75 are the maximum and minimum intrinsic delay from observed from the trace data .      the expression for efd
, @xmath76 = { \mathrm{e}}[l ] { \mathrm{e}}[\max \ { i , w \ } - i]$ ] , can be derived in two steps . instead of going through the fitted parameters , we estimate @xmath77 $ ] directly by averaging over the forwarding sequence lengths and get @xmath77 = 1.14 $ ] .
this is to reduce possible errors in parameter fitting .
the second part can be solved by double integral : @xmath78\\    & = & \int_1^{i_{(\max ) } } \int_0^h p ( i , w ) [ \max \ { i , w \ } - i ] { \mathrm{d}}w    { \mathrm{d}}i\\    & = & \frac{1}{h } \frac{1}{z_i}10^b   [   \frac{1}{(a + 1 ) ( a + 2 ) ( a +    3 ) } h^{a + 3 }     - \frac{1}{2 } h^2   \frac{1}{a + 1 }     \\    & &    +    \frac{h}{a + 2 } -    \frac{2-a(a+3)}{(a + 1 ) ( a + 2 ) ( a + 3 ) }    ] \end{aligned}\ ] ]      fig .
[ fig : h_vs_efd.eps ] depicts the scattered plots of the experiment results .
the analytical prediction is plotted as a solid line .
one can see that the analytical result matches experimental result when @xmath19 is relatively large ( @xmath79 10 mins ) . for smaller @xmath19 ,
the analytical result under estimates the efd which can be explained by the following reasons : firstly , we assume @xmath19 is the only delay but there are other delays .
those small delays become dominating when @xmath19 is small . secondly
, we assume the statuses are obtained immediately if one bot pulls its neighbour .
however , given the fully decentralized construction of our dsn , one needs to pull rss feeds from multiple friends .
since the network performance between planetlab nodes varies widely , some bots may take much longer time than qg to finish one round of pull .
this also makes the experiment results deviate from analytical ones with small @xmath19 .
one extreme case is when @xmath80 , the analytical model , which assumes @xmath19 to be the only delay factor , will predict efd=0 .
this is obviously not the case and we are working on the refinement of the model for efd .      various resource consumption is depicted from fig .
[ fig : h_vs_cpu.eps ] to fig .
[ fig : h_vs_http_size.eps ] .
the results align well with our analysis in section [ sec : analytical - model - for - the - pull - based - backbone ] , namely 1 ) cpu and memory usage are just a constant plus experimental variance ; 2 ) http query and size are reciprocal of query gap , @xmath19 .
in this paper , we analyze the challenge towards a decentralized paradigm of social networking services and proposed a meta social networking approach to solve the migration problem .
we have built a cross - platform middleware , which is lightweight , programmable , and flexible , to support the transition from centralized osns to decentralized ones .
we have presented the design and implementation of snsapi and discussed design choices and observations based on our year - long experience in maintaining and refactoring the corresponding open - source project .
we deployed a 6000-node dsn on planetlab based on snsapi to demonstrate that it is a viable approach to form a dsn . using real traces collected from a mainstream osn ,
we show that with only mild resource consumption , we can achieve acceptable forwarding latency comparable to that of a centralized osn .
we have also developed an analytical model to characterize the tradeoffs between resource consumption and message forwarding delay in such a dsn . via 20 parameterized experiments on planetlab , we have found that the empirical measurement results match reasonably well with performance predicted by our analytical model .
future works include using non - uniform and even adaptive query gap , improving the analytical model , formation of a dsn using push channels ( e.g. email platform also supported by snsapi ) , developing a distributed back - off protocol to alleviate the congestion in the dsn . during the deployment of the 6000-node dsn and the series of parameterized experiments , we have also built a full set of tools for conducting and managing of similar large - scale dsn experiments . like the snsapi and its apps , this toolbox as well as
the osn activity traces we collected from sina weibo will be open - sourced in the near future .
we hope they will provide a foundation to support large - scale experimental deployment and performance evaluation for dsn and the associated algorithms .