we are living in the age of `` big data '' , generally characterized by a series of `` vs '' .
data are generated at an unprecedented _ volume _ by scientific instruments observing the macrocosm  @xcite and the microcosm  @xcite , or by humans connected around - the - clock to mobile platforms such as facebook and twitter .
these data come in a _ variety _ of formats , ranging from delimited text to semi - structured json and multi - dimensional binaries such as fits .
the volume and variety of `` big data '' pose serious problems to traditional database systems .
before it is even possible to execute queries over a dataset , a relational schema has to be defined and data have to be loaded inside the database .
schema definition imposes a strict structure on the data , which is expected to remain stable .
however , this is rarely the case for rapidly evolving datasets represented using key - value and other semi - structured data formats , e.g. , json .
data loading is a schema - driven process in which data are duplicated in the internal database representation to allow for efficient processing . even though storage is relatively cheap ,
generating and storing multiple copies of the same data can easily become a bottleneck for massive datasets .
moreover , it is quite often the case that many of the attributes in the schema are never used in queries .    motivated by the flexibility of nosql systems to access schema - less data and by the hadoop functionality to directly process data in any format ,
we have recently witnessed a sustained effort to bring these capabilities inside relational database management systems ( rdbms ) . starting with version 9.3 ,
postgresql includes support for json data type and corresponding functions .
vertica flex zone and sinew  @xcite implement flex table and column reservoir , respectively , for storing key - value data serialized as maps in a blob column . in both systems
, certain keys can be promoted to individual columns , in storage as well as in a dynamically evolving schema . with regards to directly processing raw data ,
several query - driven extensions have been proposed to the loading and external table  @xcite mechanisms . instead of loading all the columns before querying , in adaptive partial loading  @xcite data
are loaded only at query time , and only the attributes required by the query .
this idea is further extended in invisible loading  @xcite , where only a fragment of the queried columns are loaded , and in nodb  @xcite , data vaults  @xcite , sds / q  @xcite , and raw  @xcite , where columns are loaded only in memory , but not into the database .
scanraw  @xcite is a super - scalar pipeline operator that loads data speculatively , only when spare i / o resources are available .
while these techniques enhance the rdbms flexibility to process schema - less raw data , they have several shortcomings , as the following examples show .
* example 1 : twitter data . *
the twitter api provides access to several objects in json format through a well - defined interface .
the schema of the objects is , however , not well - defined , since it includes `` nullable '' attributes and nested objects .
the state - of - the - art rdbms solution to process semi - structured json data  @xcite is to first load the objects as tuples in a blob column .
essentially , this entails complete data duplication , even though many of the object attributes are never used .
the internal representation consists of a map of key - values that is serialized / deserialized into / from persistent storage .
the map can be directly queried from sql based on the keys , treated as virtual attributes . as an optimization , certain columns  chosen by the user or by the system based on appearance frequency  are promoted to physical status .
the decision on which columns to materialize is only an heuristic , quite often sub - optimal .
* example 2 : sloan digital sky survey ( sdss ) data .
* sdss is a decade - long astronomy project having the goal to build a catalog of all the astrophysical objects in the observable universe .
images of the sky are taken by a high - resolution telescope , typically in binary fits format .
the catalog data summarize quantities measured from the images for every detected object .
the catalog is stored as binary fits tables .
additionally , the catalog data are loaded into an rdbms and made available through standard sql queries .
the loading process replicates multi - terabyte data three times  in ascii csv and internal database representation  and it can take several days  if not weeks  @xcite . in order to evaluate the effectiveness of the loading , we extract a workload of 1 million sql queries executed over the sdss catalog in 2014 .
the most frequent table in the workload is ` photoprimary ` , which appears in more than 70% of the queries . `
photoprimary ` has 509 attributes , out of which only 74 are referenced in queries .
this means that 435 attributes are replicated three times without ever being used  a significantly sub - optimal storage utilization
.    * problem statement . * inspired by the above examples , in this paper , we study the raw data processing with partial loading problem .
_ given a dataset in some raw format , a query workload , and a limited database storage budget , find what data to load in the database such that the overall workload execution time is minimized .
_ this is a standard database optimization problem with bounded constraints , similar to vertical partitioning in physical database design  @xcite .
however , while physical design investigates what non - overlapping partitions to build over internal database data , we focus on what data to load , i.e. , replicate , in a columnar database with support for multiple storage formats .
existing solutions for loading and raw data processing are not adequate for our problem .
complete loading not only requires a significant amount of storage and takes a prohibitively long time , but is also unnecessary for many workloads .
pure raw data processing solutions  @xcite are not adequate either , because parsing semi - structured json data repeatedly is time - consuming . moreover ,
accessing data from the database is clearly optimal in the case of workloads with tens of queries .
the drawback of query - driven , adaptive loading methods  @xcite is that they are greedy , workload - agnostic .
loading is decided based upon each query individually .
it is easy to imagine a query order in which the first queries access non - frequent attributes that fill the storage budget , but have limited impact on the overall workload execution time .
* contributions . * to the best of our knowledge , this is the first paper that incorporates query workload in raw data processing .
this allows us to model raw data processing with partial loading as fully - replicated binary vertical partitioning .
our contributions are guided by this equivalence .
they can be summarized as follows :    we provide a linear mixed integer programming optimization formulation that we prove to be np - hard and inapproximable .
we design a two - stage heuristic that combines the concepts of query coverage and attribute usage frequency .
the heuristic comes within close range of the optimal solution in a fraction of the time .
we extend the optimization formulation and the heuristic to a restricted type of pipelined raw data processing . in the pipelined scenario , data access and extraction
are executed concurrently .
we evaluate the performance of the heuristic and the accuracy of the optimization formulation over three real data formats
csv , fits , and json  processed with a state - of - the - art pipelined operator for raw data processing .
the results confirm the superior performance of the proposed heuristic over related vertical partitioning algorithms and the accuracy of the formulation in capturing the execution details of a real operator .
* outline . * the paper is organized as follows .
raw data processing , the formal statement of the problem , and an illustrative example are introduced in the preliminaries ( section  [ sec : preliminaries ] ) .
the mixed integer programming formulation and the proof that the formulation is np - hard are given in section  [ sec : mip ] .
the proposed heuristic is presented in detail in section  [ sec : heuristic ] .
the extension to pipelined raw data processing is discussed in section  [ sec : pipeline ] .
extensive experiments that evaluate the heuristic and verify the accuracy of the optimization formulation over three real data formats are presented in section  [ sec : experiments ] .
related work on vertical partitioning and raw data processing is briefly discussed in section  [ sec : rel - work ] , while section  [ sec : conclusions ] concludes the paper .
in this section , we introduce query processing over raw data . then , we provide a formal problem statement and an illustrative example .
query processing over raw data is depicted in figure  [ fig : scanraw ] .
the input to the process is a raw file from a non - volatile storage device , e.g. , disk or ssd , a schema that can include optional attributes , a procedure to extract tuples with the given schema from the raw file , and a driver query .
the output is a tuple representation that can be processed by the query engine and , possibly , is materialized ( i.e. , loaded ) on the same storage device . in the ` read ` stage , data are read from the original raw file , page - by - page , using the file system s functionality . without additional information about the structure or the content  stored inside the file or in some external structure
the entire file has to be read the first time it is accessed . `
extract ` transforms tuples
one per line  from raw format into the processing representation , based on the schema provided and using the extraction procedure given as input to the process .
there are two stages in ` extract``tokenize ` and ` parse ` .
[ fig : scanraw ]    ` tokenize ` identifies the schema attributes and outputs a vector containing the starting position for every attribute in the tuple  or a subset , if the driver query does not access all the attributes . in ` parse ` , attributes are converted from raw format to the corresponding binary type and mapped to the processing representation of the tuple
the record in a row - store , or the array in column - stores , respectively .
multiple records or column arrays are grouped into a chunk  the unit of processing . at the end of `
extract ` , data are loaded in memory and ready for query processing .
multiple paths can be taken at this point . in external tables
@xcite , data are passed to the query engine and discarded afterwards . in nodb
@xcite and in - memory databases  @xcite , data are kept in memory for subsequent processing . in standard database loading  @xcite
, data are first written to the database and only then query processing starts .
scanraw  @xcite invokes ` write ` concurrently with the query execution , only when spare i / o - bandwidth is available .
the interaction between ` read ` and ` write ` is carefully scheduled in order to minimize interference .
consider a relational schema @xmath0 and an instantiation of it that contains @xmath1 tuples .
semi - structured json data can be mapped to the relational model by linearizing nested constructs  @xcite . in order to execute queries over @xmath2
, tuples have to be read in memory and converted from the storage format into the processing representation .
two timing components correspond to this process .
@xmath3 is the time to read data from storage into memory .
@xmath3 can be computed straightforwardly for a given schema and storage bandwidth @xmath4 .
a constraint specific to raw file processing  and row - store databases , for that matter  is that all the attributes are read in a query  even when not required .
@xmath5 is the second timing component .
it corresponds to the conversion time . for every attribute @xmath6 in the schema ,
the conversion is characterized by two parameters , defined at tuple level .
the _ tokenizing time @xmath7 _ is the time to locate the attribute in a tuple in storage format .
the _ parsing time @xmath8 _ is the time to convert the attribute from storage format into processing representation .
a limited amount of storage @xmath9 is available for storing data converted into the processing representation .
this eliminates the conversion and replaces it with an i / o process that operates at column level  only complete columns can be saved in the processing format .
the time to read an attribute @xmath6 in processing representation , @xmath10 , can be determined when the type of the attribute and @xmath1 are known .
.query access pattern to raw data attributes .
[ cols="<,^,^,^,^,^,^,^,^",options="header " , ]     [ tbl : mip : params ]    while written as a single constraint , @xmath11 decomposes into three separate constraints  one corresponding to each `` @xmath12 '' operator  for a total of @xmath13 constraints .
@xmath14 is a reduced form of @xmath11 , applicable to query processing .
the largest number of constraints , i.e. , @xmath15 , in the mip formulation are of type @xmath16 .
they enforce that it is not possible to tokenize an attribute in a tuple without tokenizing all the preceding schema attributes in the same tuple .
@xmath16 applies strictly to raw files without direct access to individual attributes .
constraint @xmath17 guarantees that every attribute accessed in a query is either extracted from the raw file or read from the processing representation .
there are @xmath18 binary 0/1 variables in the linear mip formulation , where @xmath19 is the number of queries in the workload and @xmath20 is the number of attributes in the schema .
solving the mip directly is , thus , impractical for workloads with tens of queries over schemas with hundreds of attributes , unless the number of variables in the search space can be reduced .
we prove that this is not possible by providing a reduction from a well - known np - hard problem to a restricted instance of the mip formulation .
moreover , we also show that no approximation exists .
[ def : k - elem - cover ] given a set of @xmath20 elements @xmath21 , @xmath19 subsets @xmath22 of @xmath2 , such that @xmath23 , and a value @xmath24 , the objective in the k - element cover problem is to find a size @xmath24 subset @xmath25 of @xmath2 that covers the largest number of subsets @xmath26 , i.e. , @xmath27 , @xmath28 .
for the example in table  [ tbl : prelim : problem : example ] , @xmath29 is the single 2-element cover solution ( covering @xmath30 ) .
while many 3-element cover solutions exist , they all cover only one query .
the k - element cover problem is a restricted instance of the mip formulation , in which parameters @xmath7 , @xmath8 , and the loading and reading time to / from database are set to zero , i.e. , @xmath31 , while the raw data reading time is set to one , i.e. , @xmath32 .
the objective function is reduced to counting how many times raw data have to be accessed .
the bounding constraint limits the number of attributes that can be loaded , i.e. , @xmath33 , while the functional constraints determine the value of the other variables .
the optimal solution is given by the configuration that minimizes the number of queries accessing raw data .
a query does not access raw data when the @xmath34 variables corresponding to its attributes are all set to one .
when the entire workload is considered , this equates to finding those attributes that cover the largest number of queries , i.e. , finding the k - attribute cover of the workload .
given this reduction , it suffices to prove that k - element cover is np - hard for the mip formulation to have only exponential - time solutions .
we provide a reduction to the well - known minimum k - set coverage problem  @xcite that proves k - element cover is np - hard .    [ def : min - k - set - cover ] given a set of @xmath20 elements @xmath21 , @xmath19 subsets @xmath22 of @xmath2 , such that @xmath23 , and a value @xmath24 , the objective in the minimum k - set coverage problem is to choose @xmath24 sets @xmath35 from @xmath36 whose union has the smallest cardinality , i.e. , @xmath37 .
set @xmath21 and @xmath19 subsets @xmath22 of @xmath2 ; number @xmath38 of sets @xmath26 to choose in minimum set coverage minimum number @xmath24 of elements from @xmath2 covered by choosing @xmath38 subsets from @xmath36 @xmath39 = * _ k - element cover_*(@xmath36 , @xmath40 ) * if * @xmath41 * then * * return * @xmath40    algorithm  [ alg : reduction - cover - coverage ] gives a reduction from k - element cover to minimum k - set coverage .
the solution to minimum k - set coverage is obtained by invoking k - element cover for any number of elements in @xmath2 and returning the smallest such number for which the solution to k - element cover contains at least @xmath38 subsets of @xmath36 .
since we know that minimum k - set coverage is np - hard  @xcite and the solution is obtained by solving k - element cover , it implies that k - element cover can not be any simpler , i.e. , k - element cover is also np - hard .
the following theorem formalizes this argument .
[ thm : reduction ] the reduction from k - element cover to minimum k - set coverage given in algorithm  [ alg : reduction - cover - coverage ] is correct and complete .    * _ proof . _
* in order to prove the theorem , we have to show that if the answer to the minimum k - set coverage problem is @xmath42 , algorithm  [ alg : reduction - cover - coverage ] returns @xmath42 and if algorithm  [ alg : reduction - cover - coverage ] returns @xmath42 , the answer to minimum k - set coverage problem is @xmath42 .
we start with the first implication .
let the optimal solution to the minimum k - set coverage problem be @xmath43 and @xmath44 be the set of elements in @xmath2 covered by @xmath45 , where @xmath46 and @xmath47 .
suppose there exists a subset @xmath48 of @xmath2 and let the sets covered by @xmath48 be @xmath49 , where @xmath50 and @xmath51 .
when @xmath51 , the union of any @xmath38 sets in @xmath49 is no larger than @xmath52 , which is smaller than @xmath42 .
we get a contradiction .
thus , there is no subset of @xmath2 whose size is smaller than @xmath42 that covers at least @xmath38 sets in @xmath36 . as a result ,
algorithm  [ alg : reduction - cover - coverage ] does not return when @xmath53 . by the problem definition , we can use @xmath54 to cover at least @xmath38 sets @xmath45 .
therefore , algorithm  [ alg : reduction - cover - coverage ] returns when @xmath55 .    for the second implication ,
let the elements chosen by the _ * k - element cover * _ function be @xmath54 and @xmath45 be the sets covered by @xmath54 , where @xmath47 .
suppose the optimal solution to minimum k - set coverage is @xmath49 , which covers elements @xmath48 , where @xmath56 and @xmath57 .
then , @xmath49 is the answer to the function @xmath58 . in this case ,
algorithm  [ alg : reduction - cover - coverage ] returns @xmath52 before @xmath42 .
we have a contradiction .
therefore , we know that the optimal solution to minimum k - set coverage can not be smaller than @xmath59 .
we can choose any @xmath38 sets in @xmath45 as the solution to the minimum k - set coverage problem .
the union of the sets in @xmath45 is not larger than @xmath59 .
therefore , the second implication holds .    based on these two implications ,
we conclude that the reduction is correct .
the fact that algorithm  [ alg : reduction - cover - coverage ] has linear time complexity @xmath60 guarantees the completeness of the reduction .
@xmath61    [ thm :
reduction : corolar ] the mip formulation is np - hard and can not be approximated unless np - complete problems can be solved in randomized sub - exponential time .
the np - hardness is a direct consequence of the reduction to the k - element cover problem and theorem  [ thm : reduction ] .
in addition ,  @xcite and  @xcite prove that minimum k - set coverage can not be approximated within an absolute error of @xmath62 , for any @xmath63 , unless p = np .
consequently , the mip formulation can not be approximated .
in this section , we propose a novel heuristic algorithm for raw data processing with partial loading that has as a starting point a greedy solution for the k - element cover problem .
the algorithm also includes elements from vertical partitioning  a connection we establish in the paper .
the central idea is to combine _ query coverage _ with _ attribute usage frequency _ in order to determine the best attributes to load . at a high level
, query coverage aims at reducing the number of queries that require access to the raw data , while usage frequency aims at eliminating the repetitive extraction of the heavily - used attributes .
our algorithm reconciles between these two conflicting criteria by optimally dividing the available loading budget across them , based on the format of the raw data and the query workload .
the solution found by the algorithm is guaranteed to be as good as the solution corresponding to each criterion , considered separately .    in the following ,
we make the connection with vertical partitioning clear .
then , we present separate algorithms based on query coverage and attribute usage frequency .
these algorithms are combined into the proposed heuristic algorithm for raw data processing with partial loading .
we conclude the section with a detailed comparison between the proposed heuristic and algorithms designed specifically for vertical partitioning .      vertical partitioning  @xcite of a relational schema
@xmath64 splits the schema into multiple schemas  possibly overlapping  each containing a subset of the columns in @xmath2 . for example
, @xmath65 is the atomic non - overlapping vertical partitioning of @xmath2 in which each column is associated with a separate partition .
tuple integrity can be maintained either by sorting all the partitions in the same order , i.e. , positional equivalence , or by pre - pending a tuple identifier ( @xmath66 ) column to every partition .
vertical partitioning reduces the amount of data that have to be accessed by queries that operate on a small subset of columns since only the required columns have to be scanned  when they form a partition .
however , tuple reconstruction  @xcite can become problematic when integrity is enforced through @xmath66 values because of joins between partitions .
this interplay between having partitions that contain only the required columns and access confined to a minimum number of partitions , i.e. , a minimum number of joins , is the objective function to minimize in vertical partitioning .
the process is always workload - driven .
raw data processing with partial loading can be mapped to _ fully - replicated binary vertical partitioning _ as follows .
the complete raw data containing all the attributes in schema @xmath2 represent the _ raw partition_. the second partition  _ loaded partition _
is given by the attributes loaded in processing representation .
these are a subset of the attributes in @xmath2 .
the storage allocated to the loaded partition is bounded .
the asymmetric nature of the two partitions differentiates raw data processing from standard vertical partitioning .
the raw partition provides access to all the attributes , at the cost of tokenizing and parsing .
the loaded partition provides faster access to a reduced set of attributes . in vertical partitioning , all the partitions are equivalent .
while having only two partitions may be regarded as a simplification , all the top - down algorithms we are aware of  @xcite apply binary splits recursively in order to find the optimal partitions .
the structure of raw data processing with partial loading limits the number of splits to one .
workload @xmath67 ; storage budget @xmath9 set of attributes @xmath68 to be loaded in processing representation    [ a2:l1 ] @xmath69 ; @xmath70 [ a2:l2 ] [ a2:l3 ] @xmath71 [ a2:l4 ] * if * @xmath72 * then * * break * [ a2:l5 ] @xmath73 [ a2:l6 ] @xmath74 [ a2:l8 ] @xmath75    a query that can be processed without accessing the raw data is said to be _ covered_. in other words , all the attributes accessed by the query are loaded in processing representation .
these are the queries whose attributes are contained in the solution to the k - element cover problem .
intuitively , increasing the number of covered queries results in a reduction to the objective function , i.e. , total query workload execution time , since only the required attributes are accessed .
moreover , access to the raw data and conversion are completely eliminated .
however , given a limited storage budget , it is computationally infeasible to find the optimal set of attributes to load  the k - element cover problem is np - hard and can not be approximated ( corollary  [ thm : reduction : corolar ] ) .
thus , heuristic algorithms are required .
we design a standard greedy algorithm for the k - element cover problem that maximizes the number of covered queries within a limited storage budget .
the pseudo - code is given in algorithm  [ alg : query - coverage ] .
the solution @xmath76 and the covered queries @xmath77 are initialized with the empty set in line  [ a2:l1 ] . as long as the storage budget is not exhausted ( line  [ a2:l2 ] ) and the value of the objective function @xmath78 decreases ( line  [ a2:l4 ] ) , a query to be covered is selected at each step of the algorithm ( line  [ a2:l3 ] ) .
the criterion we use for selection is the reduction in the cost function normalized by the storage budget , i.e. , we select the query that provides the largest reduction in cost , while using the smallest storage .
this criterion gives preference to queries that access a smaller number of attributes and is consistent with our idea of maximizing the number of covered queries .
an alternative selection criterion is to drop the cost function and select the query that requires the least number of attributes to be added to the solution .
the algorithm is guaranteed to stop when no storage budget is available or all the queries are covered .
* example .
* we illustrate how the _ * query coverage * _ algorithm works on the workload in table  [ tbl : prelim : problem : example ] . without loss of generality ,
assume that all the attributes have the same size and the time to access raw data is considerably larger than the extraction time and the time to read data from processing representation , respectively .
these is a common situation in practice , specific to delimited text file formats , e.g. , csv .
let the storage budget be large enough to load three attributes , i.e. , @xmath79 .
in the first step , only queries @xmath30 , @xmath80 , and @xmath81 are considered for coverage in line  [ a2:l3 ] , due to the storage constraint .
while the same objective function value is obtained for each query , @xmath30 is selected for loading because it provides the largest normalized reduction , i.e. , @xmath82 .
the other two queries have a normalized reduction of @xmath83 , where @xmath3 is the time to read the raw data . in the second step of the algorithm , @xmath84 .
this also turns to be the last step since no other query can be covered in the given storage budget .
notice that , although @xmath80 and @xmath81 make better use of the budget , the overall objective function value is hardly different , as long as reading raw data is the dominating cost component .
workload @xmath22 of @xmath2 ; storage budget @xmath9 ; set of loaded attributes @xmath85 set of attributes @xmath86 to be loaded in processing representation    @xmath87 [ a3:l2 ] [ a3:l3 ] @xmath88
@xmath89 @xmath75    the query coverage strategy operates at query granularity .
an attribute is always considered as part of the subset of attributes accessed by the query . it is never considered individually .
this is problematic for at least two reasons .
first , the storage budget can be under - utilized , since a situation where storage is available but no query can be covered , can appear during execution .
second , a frequently - used attribute or an attribute with a time - consuming extraction may not get loaded if , for example , is part of only long queries .
the assumption that accessing raw data is the dominant cost factor does not hold in this case .
we address these deficiencies of the query coverage strategy by introducing a simple greedy algorithm that handles attributes individually . as the name implies
, the intuition behind the attribute usage frequency algorithm is to load those attributes that appear frequently in queries .
the rationale is to eliminate the extraction stages that incur the largest cost in the objective function .
the pseudo - code for the attribute usage frequency strategy is given in algorithm  [ alg : freq ] .
in addition to the workload and the storage budget , a set of attributes already loaded in the processing representation is passed as argument . at each step
( line  [ a3:l3 ] ) , the attribute that generates the largest decrease in the objective function is loaded . in this case , the algorithm stops only when the entire storage budget is exhausted ( line  [ a3:l2 ] ) .
* example .
* we illustrate how the _ * attribute usage frequency * _ algorithm works by continuing the example started in the query coverage section .
recall that only two attributes @xmath90 out of a total of three are loaded .
@xmath91 is chosen as the remaining attribute to be loaded since it appears in five queries , the largest number between unloaded attributes .
given that all the attributes have the same size and there is no cost for tuple reconstruction , @xmath92 is the optimal loading configuration for the example in table  [ tbl : prelim : problem : example ] .
workload @xmath67 ; storage budget @xmath9 set of attributes @xmath68 to be loaded in processing representation    @xmath93 [ a4:l2 ] [ a4:l3 ] @xmath94 [ a4:l4 ] @xmath95 @xmath96 @xmath97 @xmath98
@xmath99 @xmath100    the heuristic algorithm for raw data processing with partial loading unifies the query coverage and attribute usage frequency algorithms . the pseudo - code is depicted in algorithm  [ alg : combined ] . given a storage budget @xmath9 , _ * query coverage * _
is invoked first ( line  [ a4:l3 ] ) . _
* attribute usage frequency * _ ( line  [ a4:l4 ] ) takes as input the result produced by _ * query coverage * _ and the unused budget @xmath101 . instead of invoking these algorithms only once , with the given storage budget @xmath9
, we consider a series of allocations .
@xmath9 is divided in @xmath102 increments ( line  [ a4:l2 ] ) .
each algorithm is assigned anywhere from @xmath103 to @xmath9 storage , in @xmath102 increments .
a solution is computed for each of these configurations .
the heuristic algorithm returns the solution with the minimum objective .
the increment @xmath102 controls the complexity of the algorithm .
specifically , the smaller @xmath102 is , the larger the number of invocations to the component algorithms .
notice , though , that as long as @xmath104 remains constant with respect to @xmath19 and @xmath20 , the complexity of the heuristic remains @xmath105 .    the rationale for using several budget allocations between query coverage and attribute usage frequency lies in the limited view they take for solving the optimization formulation .
query coverage assumes that the access to the raw data is the most expensive cost component , i.e. , processing is i / o - bound , while attribute usage frequency focuses exclusively on the extraction , i.e. , processing is cpu - bound .
however , the actual processing is heavily - dependent on the format of the data and the characteristics of the system .
for example , binary formats , e.g. , fits , do not require extraction , while hierarchical text formats , e.g. , json , require complex parsing . moreover , the extraction complexity varies largely across data types .
the proposed heuristic algorithm recognizes these impediments and solves many instances of the optimization formulation in order to identify the optimal solution .      as discussed in section  [ sec : heuristic : vertical - part ] , raw data processing with partial loading is a special case of vertical partitioning  binary vertical partitioning with full replication .
however , there is a fundamental difference between the problem addressed in this paper and standard vertical partitioning .
the amount of storage allocated to partitions is not a parameter in vertical partitioning because it is constant and independent of the layout  all the partitions use the same storage , plus - minus metadata .
the bounded storage constraint is what makes raw data processing with partial loading a considerably more complicated problem , to which standard vertical partitioning algorithms are not directly applicable .
a comprehensive comparison of vertical partitioning methods is given in  @xcite .
with few exceptions  @xcite , vertical partitioning algorithms consider only the non - replicated case . when replication is considered , it is only partial replication .
the bounded scenario  limited storage budget for replicated attributes  is discussed only in  @xcite . at a high level
, vertical partitioning algorithms can be classified along several axes  @xcite .
we discuss the two most relevant axes for the proposed heuristic .
based on the direction in which partitions are built , we have top - down and bottom - up algorithms .
a top - down algorithm  @xcite starts with the complete schema and , at each step , splits it into two partitioned schemas .
the process is repeated recursively for each resulting schema .
a bottom - up algorithm  @xcite starts with a series of schemas , e.g. , one for each attribute or one for each subset of attributes accessed in a query , and , at each step , merges a pair of schemas into a new single schema . in both cases ,
the process stops when the objective function can not be improved further .
a second classification axis is given by the granularity at which the algorithm works .
an attribute - level algorithm  @xcite considers the attributes independent of the queries in which they appear .
the interaction between attributes across queries still plays a significant role , though . a query or transaction - level algorithm  @xcite works at query granularity .
a partition contains either all or none of the attributes accessed in a query .
based on the classification of vertical partitioning algorithms , the proposed heuristic qualifies primarily as a top - down query - level attribute - level algorithm .
however , the recursion is only one - level deep , with the loaded partition at the bottom .
the partitioning process consists of multiple steps , though . at each step , a new partition extracted from the raw data
is merged into the loaded partition  similar to a bottom - up algorithm .
the query coverage algorithm gives the query granularity characteristic to the proposed heuristic , while attribute usage frequency provides the attribute - level property .
overall , the proposed heuristic combines ideas from several classes of vertical partitioning algorithms , adapting their optimal behavior to raw data processing with partial loading . an experimental comparison with specific algorithms is presented in the experiments ( section  [ sec : experiments ] ) and a discussion on their differences in the related work ( section  [ sec : rel - work ] ) .
in this section , we discuss on the feasibility of mip optimization in the case of pipelined raw data processing with partial loading .
we consider a super - scalar pipeline architecture in which raw data access and the extraction stages  tokenize and parse  can be executed concurrently by overlapping disk i / o and cpu processing .
this architecture is introduced in  @xcite , where it is shown that , with a sufficiently large number of threads , raw data processing is an i / o - bound task .
loading and accessing data from the processing representation are not considered as part of the pipeline since they can not be overlapped with raw data access due to i / o interference .
we show that , in general , pipelined raw data processing with partial loading can not be modeled as a linear mip . however , we provide a linear formulation for a scenario that is common in practice , e.g. , binary fits and json format . in these cases , tokenization is atomic .
it is executed for all or none of the attributes .
this lets parsing as the single variable in the extraction stage .
the mip formulation can not be solved efficiently , due to the large number of variables and constraints  much larger than in the sequential formulation .
we handle this problem by applying a simple modification to the heuristic introduced in section  [ sec : heuristic ] that makes the algorithm feasible for pipelined processing .
since raw data access and extraction are executed concurrently , the objective function corresponding to pipelined query processing has to include only the maximum of the two : @xmath106 this is the only modification to the mip formulation for sequential processing given in section  [ sec : mip ] .
since the @xmath107 function is non - linear , solving the modified formulation becomes impossible with standard mip solvers , e.g. , cplex , which work only for linear problems .
the only alternative is to eliminate the @xmath107 function and linearize the objective .
however , this can not be achieved in the general case .
it can be achieved , though , for specific types of raw data  binary formats that do not require tokenization , e.g. , fits , and text formats that require complete tuple tokenization , e.g. , json . as discussed in the introduction
, these formats are used extensively in practice .
queries over raw data can be classified into two categories based on the pipelined objective function in eq .
( [ eq : obj : pipeline : max ] ) . in i / o - bound queries ,
the time to access raw data is the dominant factor , i.e. , @xmath107 returns the first argument . in cpu - bound queries , the extraction time dominates , i.e.
, @xmath107 returns the second argument .
if the category of the query is known , @xmath107 can be immediately replaced with the correct argument and the mip formulation becomes linear .
our approach is to incorporate the category of the query in the optimization as 0/1 variables . for each query @xmath40 , there is a variable for cpu - bound ( @xmath108 ) and one for io - bound ( @xmath109 ) . only one of them can take value 1 .
moreover , these variables have to be paired with the variables for raw data access and extraction , respectively .
variables of the form @xmath110 , @xmath111 , and @xmath112 correspond to the variables in table  [ tbl : mip : vars ] , in the case of a cpu - bound query . variables
@xmath113 , @xmath114 , and @xmath115 are for the io - bound case , respectively .    with these variables , we can define the functional * constraints * for pipelined raw data processing : @xmath116 constraint @xmath117 forces a query to be either cpu - bound or io - bound .
constraints @xmath118 tie the new family of cpu / io variables to their originals in the serial formulation . for example , the raw data is accessed in a cpu / io query only if it is accessed in the stand - alone query .
the same holds for tokenizing / parsing a column @xmath119 in query @xmath40 .
constraints @xmath120 and @xmath121 , respectively , tie the value of the cpu / io variables to the value of the corresponding query variable .
for example , only when a query @xmath40 is cpu - bound , it makes sense for @xmath111 and @xmath112 to be allowed to take value 1 .
if the query is io - bound , @xmath114 and @xmath115 can be set , but not @xmath111 and @xmath112 .    at this point
, we have still not defined when a query is cpu - bound and when is io - bound .
this depends on the relationship between the time to access the raw data and the time to extract the referenced attributes .
while the parsing time is completely determined by the attributes accessed in the query , the tokenizing time is problematic since it depends not only on the attributes , but also on their position in the schema .
for example , in the sdss ` photoprimary ` table containing 509 attributes , the time to tokenize the @xmath122 attribute is considerably smaller than the time to tokenize the @xmath123 attribute .
moreover , there is no linear relationship between the position in the schema and the tokenize time . for this reason
, we can not distinguish between cpu- and io - bound queries in the general case .
however , if there is no tokenization  the case for binary formats such as fits  or the tokenization involves all the attributes in the schema  the case for hierarchical json format  we can define a threshold @xmath124 that allows us to classify queries .
@xmath125 is given by the ratio between the time to access raw data less the constant tokenize time and the average time to parse an attribute .
intuitively , @xmath125 gives the number of attributes that can be parsed in the time required to access the raw data .
if a query has to parse more than @xmath125 attributes , it is cpu - bound .
otherwise , it is io - bound .
the threshold * constraints * @xmath126 and @xmath127 make these definitions formal : @xmath128 for the atomic tokenization to hold , constraint @xmath16 in the serial formulation has to be replaced with @xmath129 .
the complete pipelined mip includes the constraints in the serial formulation ( eq .  ( [ eq : mip : formulation ] ) ) and the constraints @xmath130 .
the linear * objective function * corresponding to query processing is re - written using the newly introduced variables as follows : @xmath131      since the number of variables and constraints increases with respect to the serial mip formulation , the task of a direct linear solver becomes even harder .
it is also important to notice that the problem remains np - hard and can not be approximated since the reduction to the k - element cover still applies . in these conditions , heuristic algorithms are the only solution .
we design a simple modification to the heuristic introduced in section  [ sec : heuristic ] specifically targeted at pipelined raw data processing .    given a configuration of attributes loaded in processing representation , the category of a query can be determined by evaluating the objective function .
what is more important , though , is that the evolution of the query can be traced precisely as attributes get loaded .
an i / o - bound query remains i / o - bound as long as not all of its corresponding attributes are loaded . at that point , it is not considered by the heuristic anymore .
a cpu - bound query has the potential to become i / o - bound if the attributes that dominate the extraction get loaded .
once i / o - bound , a query can not reverse to the cpu - bound state .
thus , the only transitions a query can make are from cpu - bound to i / o - bound , and to loaded from there . if an io - bound query is not covered in the _ * query coverage * _ section of the heuristic , its contribution to the objective function can not be improved since it can not be completely covered by _ * attribute usage frequency*_. based on this observation , the only strategy to reduce the cost is to select attributes that appear in cpu - bound queries .
we enforce this by limiting the selection of the attributes considered in line  [ a3:l3 ] of _ * attribute usage frequency * _ to those attributes that appear in at least one cpu - bound query .
the objective of the experimental evaluation is to investigate the accuracy and performance of the proposed heuristic across a variety of datasets and workloads executed sequentially and pipelined . to this end
, we explore the accuracy of predicting the execution time for complex workloads over three raw data formats
csv , fits , and json .
additionally , the sensitivity of the heuristic is quantified with respect to the various configuration parameters .
specifically , the experiments we design are targeted to answer the following questions :    what is the impact of each stage in the overall behavior of the heuristic ?
how accurate is the heuristic with respect to the optimal solution ?
with respect to vertical partitioning algorithms ?    how much faster is the heuristic compared to directly solving the mip formulation ? compared to other vertical partitioning algorithms ?    can the heuristic exploit pipeline processing in partitioning ?
do the mip model and the heuristic reflect reality across a variety of raw data formats ?
* implementation .
* we implement the heuristic and all the other algorithms referenced in the paper in ` c++ ` .
we follow the description and the parameter settings given in the original paper as closely as possible . the loading and query execution plans returned by the optimization routine
are executed with the scanraw  @xcite operator for raw data processing .
scanraw supports serial and pipelined execution .
the real results returned by scanraw are used as reference .
we use ibm cplex 12.6.1 to implement and solve the mip formulations .
cplex supports parallel processing .
the number of threads used in the optimization is determined dynamically at runtime .
* we execute the experiments on a standard server with 2 amd opteron 6128 series 8-core processors ( 64 bit )  16 cores
64 gb of memory , and four 2 tb 7200 rpm sas hard - drives configured raid-0 in software .
each processor has 12 mb l3 cache while each core has 128 kb l1 and 512 kb l2 local caches .
the storage system supports 240 , 436 and 1600 mb / second minimum , average , and maximum read rates , respectively  based on the ubuntu disk utility .
the cached and buffered read rates are 3 gb / second and 565 mb / second , respectively .
ubuntu 14.04.2 smp @xmath132-bit with linux kernel 3.13.0 - 43 is the operating system .
* methodology .
* we perform all experiments at least 3 times and report the average value as the result .
we enforce data to be read from disk by cleaning the file system buffers before the execution of every query in the workload .
this is necessary in order to maintain the validity of the modeling parameters .
* we use three types of real data formats in our experiments
csv , fits , and json .
the csv and fits data are downloaded from the sdss project using the cas tool .
they correspond to the complete schema of the ` photoprimary ` table , which contains 509 attributes .
the csv and fits data are identical .
only their representation is different .
csv is delimited text , while fits is in binary format .
there are 5 million rows in each of these files .
csv is 22 gb in size , while fits is only 19 gb . json is a lightweight semi - structured key - value data format .
the twitter api provides access to user tweets in this format .
tweets have a hierarchical structure that can be flattened into a relational schema .
we acquire 5,420,000 tweets by making requests to the twitter api .
there are at most 155 attributes in a tweet .
the size of the data is 19 gb .
* workloads . *
we extract a real workload of 1 million sql queries executed over the sdss catalog in 2014 . out of these , we select the most popular 100 queries over table ` photoprimary ` and their corresponding frequency . these represent approximately 70% of the 1 million queries .
we use these 100 queries as our workload in the experiments over csv and fits data .
the weight of a query is given by its relative frequency .
furthermore , we extract a subset of the 32 most popular queries and generate a second workload .
the maximum number of attributes referenced in both workloads is 74 .
we create the workload for the tweets data synthetically since we can not find a real workload that accesses more than a dozen of attributes .
the number of attributes in a query is sampled from a normal distribution centered at 20 and having a standard deviation of 20 .
the attributes in a query are randomly selected out of all the attributes in the schema or , alternatively , out of a subset of the attributes .
the smaller the subset , the more attributes are not accessed in any query .
the same weight is assigned to all the queries in the workload .      in this set of experiments ,
we evaluate the sensitivity of the proposed heuristic with respect to the parameters of the problem , specifically , the number of queries in the workload and the storage budget .
we study the impact each stage in the heuristic has on the overall accuracy .
we measure the error incurred by the heuristic with respect to the optimal solution computed by cplex and the decrease in execution time .
we also compare against several top - down vertical partitioning algorithms .
we use the sdss data and workload in our evaluation .
we consider the following vertical partitioning algorithms in our comparison : agrawal  @xcite , navathe  @xcite , and chu  @xcite . the _ agrawal algorithm _
@xcite is a pruning - based algorithm in which all the possible column groups are generated based on the attribute co - occurrence in the query workload . for each column group ,
an interestingness measure is computed . since there is an exponential number of such column groups , only the `` interesting '' ones
are considered as possible partitions .
a column group is interesting if the interestingness measure , i.e. , cg - cost , is larger than a specified threshold . the interesting column groups are further ranked based on another measure , i.e. , vp - confidence , which quantifies the frequency with which the entire column group is referenced in queries .
the attributes to load are determined by selecting column groups in the order given by vp - confidence , as long as the storage budget is not filled .
while many strategies can be envisioned , our implementation is greedy .
it chooses those attributes in a column group that are not already loaded and that minimize the objective function , one - at - a - time .
the agrawal algorithm has exponential complexity @xmath133 since this is the number of potential column groups .
this can be reduced by selecting the cg - cost threshold intelligently . however , this results in a corresponding accuracy decrease .
trojan layouts  @xcite are a newer version of the agrawal algorithm in which a different interestingness measure is defined and the selection of the partitions from the interesting column groups is done using an optimal exponential algorithm . since the authors admit that `` finding the right trojan layouts for scientific data sets ( having hundreds of attributes ) , like sdss , becomes a difficult task to achieve ''  @xcite , we use the original agrawal algorithm in our implementation .    the _ navathe algorithm _
@xcite starts with an affinity matrix that quantifies the frequency with which any pair of two attributes appear together in a query .
the main step of the algorithm consists in finding a permutation of the rows and columns that groups attributes that co - occur together in queries .
this extends upon the affinity of two attributes to a larger number of attributes .
while finding the optimal permutation is exponential in the number of attributes , a quadratic greedy algorithm that starts with two random attributes and then chooses the best attribute to add and the best position , one - at - a - time , is given .
these are computed based on a benefit function that is independent of the objective .
the attributes are ordered on the benefit function in the resulting matrix .
the final step of the algorithm consists in finding a split point along the attribute axis that generates two partitions with minimum objective function value across the query workload .
an additional condition that we have to consider in our implementation is the storage budget  we find the optimal partition that also fits in the available storage space .    the _ chu algorithm _  @xcite considers only those partitions supported by at least one query in the workload , i.e. , a column group can be a partition only if it is accessed entirely by a query .
moreover , a column group supported by a query is never split into smaller sub - parts .
the algorithm enumerates all the column groups supported by any number of queries in the workload  from a single query
to all the queries  and chooses the partition that minimizes the objective function .
the remaining attributes  not supported by the query  form the second partition .
this algorithm is exponential in the number of queries in the workload @xmath134 .
the solution proposed in  @xcite is to limit the number number of query combinations to a relatively small constant , e.g. , 5 . in our implementation
, we let the algorithm run for a limited amount of time , e.g. , one hour , and report the best result at that time  if the algorithm has not finished by that time .    *
heuristic stage analysis .
* figure  [ fig : combined_compare ] and  [ fig : error ] depict the impact each stage in the heuristic  query * coverage * and attribute usage * frequency *  has on the accuracy , when taken separately and together , i.e. , * heuristic*. we measure both the absolute value ( figure  [ fig : combined_compare ] ) and the relative error with respect to the optimal value ( figure  [ fig : error ] ) .
we depict these values as a function of the storage budget , given as the number of attributes that can be loaded .
we use the 32 queries workload .
as expected , when the budget increases , the objective decreases . in terms of relative error , though , the heuristic is more accurate at the extremes  small budget or large budget . when the budget is medium , the error is the highest
the reason for this behavior is that , at the extremes , the number of choices for loading is considerably smaller and the heuristic finds a good enough solution .
when the storage budget is medium , there are many loading choices and the heuristic makes only local optimal decisions that do not necessarily add - up to a good global solution .
the two - stage heuristic has better accuracy than each stage taken separately .
this is more clear in the case of the difficult problems with medium budget . between the two separate stages ,
none of them is dominating the other in all the cases .
this proves that our integrated heuristic is the right choice since it always improves upon the best stage taken individually .    *
serial heuristic accuracy .
* figure  [ fig : sequential_100_obj ] depicts the accuracy as a function of the storage budget for several algorithms in the case of serial raw data processing .
the workload composed of 100 queries is used . out of the heuristic algorithms , the proposed heuristic is the most accurate .
as already mentioned , the largest error is incurred when the budget is medium . between the vertical partitioning algorithms , the query - level granularity algorithm  @xcite
is the most accurate .
the other two algorithms  @xcite do not improve as the storage budget increases .
this is because they are attribute - level algorithms that are not optimized for covering queries .
* serial heuristic execution time .
* figure  [ fig : sequential_100_time ] depicts the execution time for the same scenario as in figure  [ fig : sequential_100_obj ] .
it is clear that the proposed heuristic is always the fastest , even by three orders of magnitude in the best case .
surprisingly , calculating the exact solution using cplex is faster than all the vertical partitioning algorithms almost in all the cases .
if an algorithm does not finish after one hour , we stop it and take the best solution at that moment .
this is the case for chu  @xcite and agrawal  @xcite .
however , the solution returned by chu is accurate  a known fact from the original paper .
* pipelined heuristic accuracy . *
the objective function value for pipelined processing over fits data is depicted in figure  [ fig : pipeline_100_obj ] . the same 100 query workload is used .
the only difference compared to the serial case is that cplex can not find the optimal solution in less than one hour .
however , it manages to find a good - enough solution in most cases .
the proposed heuristic achieves the best accuracy for all the storage budgets .
* pipelined heuristic execution time . *
the proposed heuristic is the only solution that achieves sub - second execution time for all the storage budgets ( figure  [ fig : pipeline_100_time ] ) .
cplex finishes execution in the alloted time only when the budget is large .
the number of variables and constraints in the pipeline mip formulation increase the search space beyond what the cplex algorithms can handle .
we provide a series of case studies over different data formats in order to validate that the raw data processing architecture depicted in figure  [ fig : scanraw ] is general and the mip models corresponding to this architecture fit reality .
we use the implementation of the architecture in the scanraw operator  @xcite as a baseline .
for a given workload and loading plan , we measure the cumulative execution time after each query and compare the result with the estimation computed by the mip formulation .
if the two match , this is a good indication that the mip formulation models reality accurately .
the csv format maps directly to the raw data processing architecture . in order to apply the mip formulation ,
the value of the parameters has to be calibrated for a given system and a given input file .
the time to tokenize @xmath7 and parse @xmath8 an attribute are the only parameters that require discussion .
this can be done by executing the two stages on a sample of the data and measuring the average value of the parameter for each attribute .
as long as accurate estimates are obtained , the model will be accurate .
figure  [ fig : csv ] confirms this on the sdss workload of 32 queries . in this case
, there is a perfect match between the model and the scanraw execution .
[ fig : csv ]      since fits is a binary format , there is no extraction phase , i.e. , tokenizing and parsing , in the architecture .
moreover , data can be read directly in the processing representation , as long as the file access library provides such a functionality .
cfitsio  the library we use in our implementation
 can read a range of values of an attribute in a pre - allocated memory buffer .
however , we observed experimentally that , in order to access any attribute , there is a high startup time .
essentially , the entire data are read in order to extract the attribute .
the additional time is linear in the number of attributes .
[ fig : fits ]    based on these observations  that may be specific to cfitsio
the following parameters have to be calibrated : the time to read the raw data corresponds to the startup time ; an extraction time proportional with the number of attributes in the query is the equivalent of @xmath8 .
@xmath7 is set to zero .
although pipelining is an option for fits data , due to the specifics of the cfitsio library , it is impossible to apply it .
the result for the sdss data confirms that the model is a good fit for fits data since there is almost complete overlap in figure  [ fig : fits ] .      at first sight
, it seems impossible to map json data on the raw data processing architecture and the mip model .
looking deeper , we observe that json data processing is even simpler than csv processing .
this is because every object is fully - tokenized and parsed in an internal map data structure , independent of the requested attributes .
at least this is how the jsoncpp library works .
+    once the map is built , it can be queried for any key in the schema . for schemas with a reduced number of hierarchical levels  the case for tweets
there is no difference in query time across levels . essentially , the query time is proportional only with the number of requested keys , independent of their existence or not . based on these observations
, we set the model parameters as follows .
@xmath7 is set to the average time to build the map divided by the maximum number of attributes in the schema .
@xmath8 is set to the map data structure query time .
since @xmath7 is a constant , the pipelined mip formulation applies to the json format .
the results in figure  [ fig : json ] confirm the accuracy of the model over a workload of 32 queries executed in scanraw .     [
fig : json ]      the experimental evaluation provides answers to each of the questions raised at the beginning of the section .
the two - stage heuristic improves over each of the component parts .
it is not clear which of the query coverage and attribute usage frequency is more accurate .
using them together guarantees the best results .
the proposed heuristic comes close to the optimal solution whenever the storage budget is either small or large .
when many choices are available  the case for a medium budget  the accuracy decreases , but remains superior to the accuracy of the other vertical partitioning methods . in terms of execution time ,
the proposed heuristic is the clear winner  by as much as three orders of magnitude .
surprisingly , cplex outperforms the other heuristics in the serial case .
this is not necessarily unexpected , given that these algorithms have been introduced more than two decades ago .
the case studies confirm the applicability of the mip formulation model to several raw data formats .
the mip model fits the reality almost perfectly both for serial and pipelined raw data processing .
two lines of research are most relevant to the work presented in this paper  raw data processing and vertical partitioning as a physical database design technique .
our contribution is to integrate workload information in raw data processing and model the problem as vertical partitioning optimization . to the best of our knowledge ,
this is the first paper to consider the problem of optimal vertical partitioning for raw data processing with partial loading .    *
raw data processing .
* several methods have been proposed for processing raw data within a database engine .
the vast majority of them bring enhancements to the external table functionality , already supported by several major database servers  @xcite .
a common factor across many of these methods is that they do not consider loading converted data inside the database . at most ,
data are cached in memory on a query - by - query basis .
this is the approach taken in nodb  @xcite , data vaults  @xcite , sds / q  @xcite , raw  @xcite , and impala  @xcite .
even when loading is an option , for example in adaptive partial loading  @xcite , invisible loading  @xcite , and scanraw  @xcite , the workload is not taken into account and the storage budget is unlimited . the decision on what to load is local to every query , thus , prone to be acutely sub - optimal over the entire workload .
the heuristic developed in this paper requires workload knowledge and aims to identify the optimal data to load such that the execution time of the entire workload is minimized . as in standard database processing
, loading is executed offline , before query execution . however , the decision on what data to load is intelligent and the time spent on loading is limited by the allocated storage budget .
notice that the heuristic is applicable both to secondary storage - based loading as well as to one - time in - memory caching without subsequent replacement .    * vertical partitioning . * vertical partitioning has a long - standing history as a physical database design strategy , dating back to the 1970 s .
many types of solutions have been proposed over the years , ranging from integer programming formulations to top - down and bottom - up heuristics that operate at the granularity of a query or of an attribute .
a comparative analysis of several vertical partitioning algorithms is presented in  @xcite .
the serial mip formulation for raw data processing is inspired from the formulations for vertical partitioning given in  @xcite . while both are non - linear , none of these formulations considers pipeline processing .
we prove that even the linear mip formulation is np - hard .
the scale of the previous results for solving mip optimizations have to be taken with a grain of salt , given the extensive enhancements to integer programming solvers over the past two decades . as explained in section  [ sec : heuristic : comparison - vertical ] , the proposed heuristic combines ideas from several classes of vertical partitioning algorithms , adapting their optimal behavior to raw data processing with partial loading .
the top - down transaction - level algorithm given in  @xcite is the closest to the query coverage stage .
while query coverage is a greedy algorithm , @xcite employs exhaustive search to find the solution .
as the experimental results show , this is time - consuming .
other top - down heuristics  @xcite consider the interaction between attributes across the queries in the workload .
the partitioning is guided by a quantitative parameter that measures the strength of the interaction . in  @xcite
, only the interaction between pairs of attributes is considered .
the attribute usage frequency phase of the proposed heuristic treats each attribute individually , but only after query coverage is executed .
the objective in  @xcite is to find a set of vertical partitions that are subsequently evaluated for index creation .
since we select a single partitioning scheme , the process is less time - consuming .
finally , the difference between the proposed heuristic and bottom - up algorithms  @xcite is that the latter can not guarantee that only two partitions are generated at the end .
this is a requirement for raw data processing with partial loading .
all these algorithms are offline .
they are executed only once , before query processing , over a known workload .
online vertical partitioning algorithms form a separate class . in  @xcite
, the entire workload is known in advance . however , the order of the queries is fixed and the vertical partitioning evolves .
another series of algorithms  @xcite operates over an unknown workload , given one query at a time .
their goal is to gather evidence from the past workload in order to determine the optimal vertical partitioning at each query .
in this paper , we study the problem of workload - driven raw data processing with partial loading .
we model loading as binary vertical partitioning with full replication . based on this equivalence ,
we provide a linear mixed integer programming optimization formulation that we prove to be np - hard and inapproximable .
we design a two - stage heuristic that combines the concepts of query coverage and attribute usage frequency .
the heuristic comes within close range of the optimal solution in a fraction of the time .
we extend the optimization formulation and the heuristic to a restricted type of pipelined raw data processing . in the pipelined scenario ,
data access and extraction are executed concurrently .
we evaluate the performance of the heuristic and the accuracy of the optimization formulation over three real data formats
csv , fits , and json  processed with a state - of - the - art pipelined operator for raw data processing .
the results confirm the superior performance of the proposed heuristic over related vertical partitioning algorithms and the accuracy of the formulation in capturing the execution details of a real operator .
following the steps of database physical design , we envision several avenues to extend the proposed research in the future .
we can move from the offline loading setting to online loading , where query processing and loading are intertwined .
we can assume that the workload is known beforehand or it is given one query at a time .
we can drop the strict requirement of atomic attribute loading and allow for portions  horizontal partitions  of an attribute to be loaded .
finally , we can consider a multi - query processing environment in which raw data access and attribute extraction can be shared across several queries .