tutorial file organization fundamental

File Organisation

! _{The database is stored as a collection of files. Each file is a}

sequence of records. A record is a sequence of fields.

! _{First approach:}

" _{assume record size is fixed}

" _{each file has records of one particular type only}

" _{different files are used for different relations}

Fixed-Length Records

! _{Simple approach:}

" _{Store record i starting from byte n ! (i – 1), where n is the size of}

each record.

" _{Record access is simple. But records may cross blocks!}

! Modification: do not allow records to cross block boundaries

! _{Deletion of record i:}

alternatives:

" _{move records i + 1, . . ., n}

to i, . . . , n – 1

" _{move record n to i}

" _{do not move records, but}

mark deleted records and

Free Lists

! _{Store the address of the first deleted record in the file header.}

! _{Use this first record to store the address of the second deleted record,}

and so on

! _{Can think of these stored addresses as pointers since they “point” to}

the location of a record.

! _{More space efficient representation: reuse space for normal attributes}

Variable-Length Records

! _{Variable-length records arise in database systems in several}

ways:

" _{Storage of multiple record types in a file.}

" _{Record types that allow variable lengths for one or more}

fields.

! _{Store several records, with variable length, in blocks (slotted}

page)

! _{A record cannot be bigger than a slotted page}

! _{This limits the size of records in a database, which is usually}

the (default) case

" _{There are special types for big records, that are treated}

Variable-Length Records: Slotted Page Structure

! _{Slotted page are usually the size of a block} ! _{Header contains:}

" _{number of record entries}

" _{end of free space in the block}

" _{location and size of each record}

! _{Records can be moved around within a page to keep them contiguous}

with no empty space between them; entry in the header must be updated.

Organization of Records in Files

! _{Heap – a record can be placed anywhere in the file where there}

is space

! _{Sequential – store records in sequential order, based on the}

value of the search key of each record

! _{Hashing – a hash function computed on some attribute of each}

record; the result specifies in which block of the file the record should be placed

! _{Records of each relation may be stored in a separate file. In a}

multitable clustering file organization records of several

different relations can be stored in the same file

" _{Motivation: store related records on the same block to}

minimize I/O

! _{The choice of a proper organisation of records in a file is}

Sequential File Organization

! _{Suitable for applications that require sequential processing of}

the entire file

Sequential File Organization (Cont.)

! _{Deletion – use pointer chains}

! _{Insertion –locate the position where the record is to be inserted}

" _{if there is free space insert there}

" _{if no free space, insert the record in an overflow block}

" _{In either case, pointer chain must be updated}

! _{Need to reorganise the file}

from time to time to restore sequential order

Multitable Clustering File Organization

Store several relations in one file using a multitable clustering

Multitable Clustering File Organization (cont.)

Multitable clustering organization of customer and depositor:

" _{good for queries involving depositor customer, and for queries}

involving one single customer and his accounts

" _{bad for queries involving only customer}

" _{but one can add pointer chains to link records of a particular relation} " _{results in variable size records}

File System

! _{In sequential file organisation, each relation is stored in a file}

" _{One may rely in the file system of the underlying operating system}

! _{Multitable clustering may have significant gains in efficiency}

" _{But this may not be compatible with the file system of the}

operating system

! _{Several large scale database management systems do not rely}

directly on the underlying operating system

" _{The relations are all stored in a single (multitable) file}

" _{The database management system manages the file by itself}

" _{This requires the implementation of an own file system inside the}

Data Dictionary Storage

! _{Information about relations}

" _{names of relations}

" _{names and types of attributes of each relation}

" _{names and definitions of views}

" _{integrity constraints}

! _{User and accounting information, including passwords} ! _{Statistical and descriptive data}

" _{number of tuples in each relation}

! _{Physical file organisation information}

" _{How relation is stored (sequential/hash/…)}

" _{Physical location of relation}

! _{Information about indices (next lecture)}

Data dictionary (also called system catalogue) stores metadata; that is, data about data, such as

Data Dictionary Storage (Cont.)

! _{Catalog structure}

" _{Relational representation on disk}

" _{specialized data structures designed for efficient access, in}

memory

! _{A possible catalog representation:}

Relation_metadata = (relation_name, number_of_attributes, storage_organization, location)

Attribute_metadata = (attribute_name, relation_name, domain_type,

position, length)

User_metadata = (user_name, encrypted_password, group)

Index_metadata = (index_name, relation_name, index_type,

index_attributes)

File Organisation in Oracle

! _{Oracle has its own buffer management, with complex policies}

! _{Oracle doesn!t rely on the underlying operating system!s file system} ! _{A database in Oracle consists of tablespaces:}

" _{System tablespace: contains catalog meta-data}

" _{User data tablespaces}

! _{The space in a tablespace is divided into segments:}

" _{Data segment}

" _{Index segment}

" _{Temporary segment (for sort operations)}

" _{Rollback segment (for processing transactions)}

! _{Segments are divided into extents, each extent being a set of}

contiguous database blocks.

" _{A database block need not be the same size of an operating}

File Organization in Oracle (cont.)

! _{A standard table is organised in a heap (no sequence is imposed)}

! _{Partitioning of tables is possible for optimisation}

" _{Range partitioning (e.g by dates)} " _{Hash partitioning}

" _{Composite partitioning}

! _{Table data in Oracle can also be (multitable) clustered}

" _{One may tune the clusters to significantly improve the efficiency of query}

to frequently used joins.

! _{Hash file organisation (to be studied in the sequel) is also possible for fetching}

the appropriate cluster

! _{A database can be tuned by an appropriate choice for the organisation of data:}

" _{Choosing partitions}

" _{Appropriate choice of clusters} " _{Hash or sequential}

José Alferes

Versão modificada de Database System Concepts, 5th Ed.

Chapter 12: Indexing and Hashing

! _{Basic Concepts} ! _{Ordered Indices} ! _B+_{-Tree Index Files}

" _{B-Tree Index Files}

! _Hashing

" _{Static Hashing}

" _{Dynamic Hashing}

! _{Comparison of Ordered Indexing and Hashing} ! _{Multiple-Key Access and Bitmap indices}

! _{Index Definition in SQL} ! _{Indexing in Oracle 10g}

Basic Concepts

! _{Indexing mechanisms are used to speed up access to desired data.}

! _{Search Key - attribute to set of attributes used to look up records in a}

file.

! _{An index file consists of records (called index entries) of the form}

! _{Index files are typically much smaller than the original file} ! _{Two basic kinds of indices:}

" _{Ordered indices: search keys are stored in sorted order}

" _{Hash indices: search keys are distributed uniformly across}

“buckets” using a “hash function”. search-key pointer

Index Evaluation Metrics

! _{Access time} ! _{Insertion time} ! _{Deletion time} ! _{Space overhead}

! _{Access types supported efficiently. E.g.,}

" _{records with a specified value in the attribute}

" _{or records with an attribute value falling in a specified range of}

values.

" _{This strongly influences the choice of index, and depends on}

Ordered Indices

! _{In an ordered index, index entries are stored sorted on the search key}

value. E.g., author catalog in library.

! _{Primary index: in a sequentially ordered file, the index whose search}

key specifies the sequential order of the file.

" _{Also called clustering index}

" _{The search key of a primary index is usually but not necessarily the}

primary key.

! _{Secondary index: an index whose search key specifies an order}

different from the sequential order of the file. Also called non-clustering index.

Dense Index Files

! _{Dense index — Index record appears for every search-key value in}

Sparse Index Files

! _{Sparse Index: contains index records for only some search-key}

values.

" _{Only applicable when records are sequentially ordered on}

search-key

! _{To locate a record with search-key value K we:}

" _{Find index record with largest search-key value < K}

" _{Search file sequentially starting at the record to which the index}

Multilevel Index

!

_{If primary index does not fit in memory, access becomes}

expensive.

!

_{Solution: treat primary index kept on disk as a sequential file}

and construct a sparse index on it.

"

_{outer index – a sparse index of primary index}

"

_{inner index – the primary index file}

!

_{If even outer index is too large to fit in main memory, yet}

another level of index can be created, and so on.

!

_{Indices at all levels must be updated on insertion or deletion}

Index Update: Deletion

! _{If deleted record was the only record in the file with its particular}

search-key value, the search-search-key is deleted from the index also.

! _{Single-level index deletion:}

" _{Dense indices – deletion of search-key: similar to file record}

deletion.

" _{Sparse indices –}

! if an entry for the search key exists in the index, it is deleted by replacing the entry in the index with the next search-key value in the file (in search-key order).

! If the next search-key value already has an index entry, the entry is deleted instead of being replaced.

Index Update: Insertion

! _{Single-level index insertion:}

" _{Perform a lookup using the search-key value appearing in the}

record to be inserted.

" _{Dense indices – if the search-key value does not appear in the}

index, insert it.

" _{Sparse indices – if index stores an entry for each block of the file,}

no change needs to be made to the index unless a new block is created.

! If a new block is created, the first search-key value appearing in the new block is inserted into the index.

! _{Multilevel insertion (as well as deletion) algorithms are simple}

Secondary Indices

! _{Frequently, one wants to find all the records whose values in a}

certain field (which is not the search-key of the primary index) satisfy some condition.

" _{Example 1: In the account relation stored sequentially by}

account number, we may want to find all accounts in a particular branch

" _{Example 2: as above, but where we want to find all accounts}

with a specified balance or range of balances

! _{We can have a secondary index with an index record for each}

Secondary Indices Example

! _{Index record points to a bucket that contains pointers to all the actual}

records with that particular search-key value.

! _{Secondary indices have to be dense, since the file is not sorted by the}

Primary and Secondary Indices

! _{Indices offer substantial benefits when searching for records, but}

updating indices imposes overhead on database modification - when a file is modified, every index on the file must be updated,

! _{Sequential scan using primary index is efficient, but a sequential scan}

using a secondary index is expensive

" _{Each record access may fetch a new block from disk}

" _{Block fetch requires about 5 to 10 micro seconds, versus about}

B

+

_{-Tree Index Files}

! _{Disadvantage of indexed-sequential files}

" _{performance degrades as file grows, since many overflow blocks}

get created.

" _{Periodic reorganisation of entire file is required.}

! _{Advantage of B}+_{-treeindex files:}

" _{automatically reorganises itself with small, local, changes, in the}

face of insertions and deletions.

" _{Reorganisation of entire file is not required to maintain}

performance.

! _{(Minor) disadvantage of B}+_-trees:

" _{extra insertion and deletion overhead, space overhead.}

! _{Advantages of B}+_{-trees outweigh disadvantages}

" _B+_{-trees are used extensively}

B

+

_{-Tree Node Structure}

! _{Typical node}

" K

i are the search-key values

" P

i are pointers to children (for non-leaf nodes) or pointers to

records or buckets of records (for leaf nodes).

! _{The search-keys in a node are ordered}

" " K₁ < K₂ < K₃ < . . .< K_n–1

Example of a B

+

_-tree

B

+

_{-Tree Index File}

! _{All paths from root to leaf are of the same length}

! _{Each node that is not a root or a leaf has between "n/2# and n}

children.

! _{A leaf node has between "(n–1)/2# and n–1 values} ! _{Special cases:}

" _{If the root is not a leaf, it has at least 2 children.}

" _{If the root is a leaf (that is, there are no other nodes in the}

tree), it can have between 0 and (n–1) values.

Leaf Nodes in B

+

_-Trees

! _{For i = 1, 2, . . ., n–1, pointer P}

i either points to a file record with

search-key value K_i, or to a bucket of pointers to file records, each record having search-key value K_i. Only need bucket structure if search-key does not form a primary key.

! _{If L}

i, Lj are leaf nodes and i < j, Li!s search-key values are less than Lj!s

search-key values

! _P

n points to next leaf node in search-key order

Non-Leaf Nodes in B

+

_-Trees

! _{Non leaf nodes form a multi-level sparse index on the leaf nodes. For}

a non-leaf node with m pointers:

" All the search-keys in the subtree to which P

1 points are less than

K₁

" For 2 $ i $ n – 1, all the search-keys in the subtree to which P

i

points have values greater than or equal to K_i–1 and less than K_i

" All the search-keys in the subtree to which P

n points have values

Example of B

+

_-tree

! _{Leaf nodes must have between 2 and 4 values}

("(n–1)/2# and n –1, with n = 5).

! _{Non-leaf nodes other than root must have between 3 and 5}

children ("(n/2# and n with n =5).

! _{Root must have at least 2 children.}

12. José Alferes - Adaptado de Database System Concepts - 5th _Edition

B

+

_{-Tree Node Structure}

! _{Typical node}

" K

i are the search-key values

" P

i are pointers to children (for non-leaf nodes) or pointers to

records or buckets of records (for leaf nodes).

! _{The search-keys in a node are ordered}

" " K₁ < K₂ < K₃ < . . .< K_n–1

12. José Alferes - Adaptado de Database System Concepts - 5th _Edition

Example of a B

+

_-tree

12. José Alferes - Adaptado de Database System Concepts - 5th _Edition

B

+

_{-Tree Index File}

! _{All paths from root to leaf are of the same length}

! _{Each node that is not a root or a leaf has between "n/2# and n}

children.

! _{A leaf node has between "(n–1)/2# and n–1 values}

! _{Special cases:}

" _{If the root is not a leaf, it has at least 2 children.}

" _{If the root is a leaf (that is, there are no other nodes in the}

tree), it can have between 0 and (n–1) values.

12. José Alferes - Adaptado de Database System Concepts - 5th _Edition

Leaf Nodes in B

+

_-Trees

! _{For i = 1, 2, . . ., n–1, pointer P}

i either points to a file record with

search-key value K_i, or to a bucket of pointers to file records, each record

having search-key value K_i. Only need bucket structure if search-key

does not form a primary key.

! _{If L}

i, Lj are leaf nodes and i < j, Li!s search-key values are less than Lj!s

search-key values

! _P

n points to next leaf node in search-key order

12. José Alferes - Adaptado de Database System Concepts - 5th _Edition

Non-Leaf Nodes in B

+

_-Trees

! _{Non leaf nodes form a multi-level sparse index on the leaf nodes. For}

a non-leaf node with m pointers:

" All the search-keys in the subtree to which P

1 points are less than

K₁

" For 2 $ i $ n – 1, all the search-keys in the subtree to which P

i

points have values greater than or equal to K_i–1 and less than K_i

" All the search-keys in the subtree to which P

n points have values

12. José Alferes - Adaptado de Database System Concepts - 5th _Edition

Example of B

+

_-tree

! _{Leaf nodes must have between 2 and 4 values}

("(n–1)/2# and n –1, with n = 5).

! _{Non-leaf nodes other than root must have between 3 and 5}

children ("(n/2# and n with n =5).

! _{Root must have at least 2 children.}