Lec 1 Introduction Data Structures Algorithm

  Introduction Data Structures & Algorithm Data Structures & Algorithm Data Structures & Algorithm Algorithm: Outline, the essence of a

• computational procedure, step-by-step instructions Program: an implementation of an algorithm in
• some programming language Data structure: Organization of data needed to
• solve the problem

  2012-2013 ²

  History

  Name: Persian mathematician Mohammed al-

• Khowarizmi, in Latin became Algorismus First algorithm: Euclidean Algorithm, greatest
• common divisor, 400-300 B.C. _{th th} 19 century – Charles Babbage, Ada Lovelace.
• 20 century – Alan Turing, Alonzo Church, John • von Neumann

  2012-2013 ³

• Infinite number of input instances satisfying the specification. For example:
• A sorted, non-decreasing sequence of natural numbers. The sequence is of non-zero, finite length: _• 1, 20, 908, 909, 100000, 1000000000. _• 3.

  2012-2013 ⁴ Algorithmic problem

  Specification of input ?

  Specification of output as a function of input

Algorithmic Solution

  Input instance, Output Algorithm adhering to related to the the input as specification required

  Algorithm describes actions on the input instance

• Infinitely many correct algorithms for the same algorithmic

• problem

  2012-2013 ⁵

• Efciency:
• Running time
• Space used
• Efciency as a function of input size:

• Number of data elements (numbers, points)

• A number of bits in an input number

  2012-2013 ⁶ What is a Good Algorithm?

• How should we measure the running time of an algorithm ?
• Approach 1: Experimental Study
• Write a program that implements the algorithm
• Run the program with data sets of varying size and composition.
• Use a method like System.currentTimeMillis() to get an accurate measure of the actual running time.

  2012-2013 ⁷ Measuring the Running Time

  50 100 ^{t (ms) n 10 20 30 40 50 60}

  Limitation Experimental Studies

  Experimental studies have several limitations:

• It is necessary to implement and test
• the algorithm in order to determine its running time. Experiments can be done only on a
• limited set of inputs , and may not be

  

indicative of the running time on

other inputs not included in the experiment. In order to compare two algorithms,

• the same hardware and software

  _2012-2013 environments should be used.
₈

  Beyond Experimental Studies

  We will now develop a general methodology for

• analyzing the running time of algorithms. In contrast to the "experimental approach", this methodology:

  Uses a high-level description of the

• algorithm instead of testing one of its implementations. Takes into account all possible inputs.
• Allows one to evaluate the efciency

• of any algorithm in a way that is

  independent from the hardware and . software environment

  2012-2013 ⁹

  Pseudo-code Pseudo-code is a description of an algorithm that is more

• structured than usual prose but less formal than a programming language. Example: finding the maximum element of an array.
• Algorithm arrayMax(A, n): Input: An array A storing n integers.

  Output: The maximum element in A.

   A[0] currentMax for i  1 to n -1 do

if currentMax < A[i] then currentMax  A[i]

return currentMax

  Pseudo-code is our preferred notation for describing

• algorithms. However, pseudo-code hides program design issues.
• 2012-2013
¹⁰

  What is Pseudo-code?

• A mixture of natural language and high-level programming

  concepts that describes the main ideas behind a generic implementation of a data structure or algorithm.

  Expressions:

• use standard mathematical symbols to describe numeric and boolean
• _• expressions use  for assignment (“=” in Java)
• use = for the equality relationship (“==” in Java)

  Method Declarations: _• Algorithm name(param1, param2)

  2012-2013 ¹¹

  returns:return value 2012-2013 ¹²

  What is Pseudo-code?

• Programming Constructs:
• decision structures: if ... then ... [else ]
• while-loops: while ... Do • repeat-loops: repeat ... until …
• for-loop: for ... Do • array indexing: A[i]
• Functions/Method calls: object method(args)

  Analysis of Algorithms Primitive Operations: Low-level computations

• independent from the programming language can be identified in pseudocode. Examples:
• calling a function/method and returning from a
• function/method arithmetic operations (e.g. addition)
• comparing two numbers
• indexing into an array, etc.
• By inspecting the pseudo-code, we can count the
• number of primitive operations executed by an algorithm.

  2012-2013 ¹³

OUTPUT

INPUT

  For any given input the algorithm halts with the output:

  Depends on

  Running time

  Depends on

  , a ₂ , a ₃ ,….,a

_n

Running time

  is a permutation of a ₁

  < b ₂ < b ₃ < …. < b _n

  For any given input the algorithm halts with the output:

  2 4 5 7 10 Correctness

  2012-2013 ¹⁴ Sort Example: Sorting

  a permutation of the sequence of numbers 2 5 4 10 7

  3 ,….,b n

  2 ,b

  1 ,b

  3 ,….,a n b

  2 , a

  1 , a

  sequence of numbers a

• number of elements ( n )
• number of elements (n)
• how (partially) sorted they are
• b
1<
• how (partially) sorted they are
• b
₁ &lt; b ₂ &lt; b ₃ &lt; …. &lt; b n<
• b
₁ , b ₂ , b ₃ , …., b n<
• b
₁ , b ₂ , b ₃ , …., b _n is a permutation of a ₁ , a ₂ , a ₃ ,….,a

_n

Correctnes
• algorithm
• algorithm

• Start “empty handed”
• Insert a card in the right position of the already sorted hand
• Continue until all cards are inserted/sorted

• Start “empty handed”
• Insert a card in the right position of the already sorted hand
• Continue until all cards are inserted/sorted

  2012-2013 ¹⁵ Insertion Sort

  1 n j

  3 6 8

  4 9 7 2 5 1 i

A

  Strategy

  Strategy

  for j=2 to length(A) do key=A[j] “insert A[j] into the sorted sequence A[1..j-1]” i=j-1 while i&gt;0 and A[i]&gt;key do A[i+1]=A[i] i--

  A[i+1]:=key for j=2 to length(A) do key=A[j]

  “ insert A[j] into the sorted sequence A[1..j-1]” i=j-1 while i&gt;0 and A[i]&gt;key do A[i+1]=A[i] i--

  A[i+1]:=key

• Time to compute the running time as a function of the input size

• Total Time= n(c1+c2+c3+c7)+

  (c4+c5+c6)-(c2+c3+c5+c6+c7)
^{2 n j j} _{ } t

  2012-2013 ¹⁶ Analysis of Insertion Sort for j=2 to length(A) do key=A[j]

  “ insert A[j] into the sorted sequence A[1..j-1]” i=j-1 while i&gt;0 and A[i]&gt;key do A[i+1]=A[i] i--

  A[i+1]:=key cost c ₁ c ₂ c ₃ c ₄ c ₅ c ₆ c ₇ times n n-1 n-1 n-1 n-1 ^{2 n j j}

  _{ } t ₂ ( 1) ⁿ _{j j} t _ 

   ₂ ( 1) ⁿ _{j j} t _ 

   Best/Worst/Average Case _j =1,

  Best case: elements already sorted ® t

• running time = f(n), i.e., linear time.

  Worst case: elements are sorted in inverse

• order
₂

  =j, running time = f(n ), i.e., quadratic time

  ® t _{j 2 j}

  Average case: t =j/2, running time = f(n ), i.e.,

• quadratic time _{n j}
₂ t _j Total Time= n(c1+c2+c3+c7)+ 

   (c4+c5+c6)-(c2+c3+c5+c6+c7)

  2012-2013 ¹⁷

  Best/Worst/Average Case (2) For a specific size of input n, investigate running times for

• different input instances:

  6n 5n 4n 3n 2n 1n

  2012-2013 ¹⁸

• For inputs of all sizes:

  2012-2013 ¹⁹ Best/Worst/Average Case (3)

  1n 2n 3n 4n 5n 6n

  Input instance size R un ni ng ti m e 1 2 3 4 5 6 7 8 9 10 11 12 ….. best-case average-case worst-case Best/Worst/Average Case (4) Worst case is usually used:

• It is an upper-bound and in certain application
• domains (e.g., air trafc control, surgery) knowing the worst-case time complexity is of crucial importance For some algorithms worst case occurs fairly
• often The average case is often as bad as the
• worst case

  Finding the average case can be very

• difcult

  2012-2013 ²⁰

Asymptotic Notation

  Goal: to simplify analysis by getting rid of unneeded

• information (like “rounding” 1,000,001≈1,000,000)
_{2 2} We want to say in a formal way 3n &asy
• The “Big-Oh” Notation:
• given functions f(n) and g(n), we say that f(n) is O(g(n))
• if and only if there are positive constants c and n such that f(n)≤ c g(n) for n ≥ n

  Asymptotic Notation

• asymptotic upper bound

  The “big-Oh” O-Notation

• f(n) = O(g(n)), if there exists
• constants c and n , s.t. f(n) £ c g(n) for n ³ n

  f(n) and g(n) are functions over non-

• c g n ( )

   negative integers

  f n

  ( ) ng e

  Used for worst-case analysis

• ni im T un R

  n Input _2012-2013

  Size ₂₂

Example

  g(n) = n c g(n) =

  4n n f(n) = 2n + 6

  For functions f(n) and g(n) (to the right) there are positive constants c and n such that: f(n)≤ c g(n) for n ≥ n conclusion:

  2n+6 is O(n).

Another Example

  On the other hand… n

  2 is not O( n ) because there is no c and n such that: n

  2 ≤ cn for n ≥ n

  (As the graph to the right illustrates, no matter how large a c is chosen there is an n big enough that n

  2 &gt; cn ) .

Asymptotic Notation (cont.)

• 3

  “7n - 3 Note : Even though it is correct to say

  

is O(n )”, a better statement is “7n - 3 is O(n)”,

that is, one should make the approximation as

tight as possible (for all values of n).

  Simple Rule : Drop lower order terms and

• constant factors

  7n-3 is O(n)

  2

  2

  

2

8n log n + 5n + n is O(n log n)

Asymptotic Analysis of The Running Time

  Use the Big-Oh notation to express the number of

• primitive operations executed as a function of the input size. For example, we say that the arrayMax algorithm
• runs in O(n) time. Comparing the asymptotic running time

• -an algorithm that runs in O(n) time is better than one that runs in

₂ O(n )
• similarly, O(log n) is better than O(n)
₂ 3 n<
• hierarchy of functions: log n &lt;&lt; n &lt;&lt; n &lt;&lt; n &lt;&lt; 2

  

Caution! Beware of very large constant factors. An

• algorithm running in time 1,000,000 n is still O(n) but might be less efcient on your data set than one

  2

  2 running in time 2n , which is O(n )

Example of Asymptotic Analysis

  An algorithm for computing prefix averages Algorithm prefixAverages1(X): Input: An n-element array X of numbers.

  

Output: An n -element array A of numbers such that A[i] is the

average of elements X[0], ... , X[i].

  Let A be an array of n numbers. for i  0 to n - 1 do a  0 for j  0 to i do

   a + X[j] a i iterations n iterations

  1 step A[i]  a/(i+ 1) with return array A i=0,1,2...n-1 Analysis ...

Another Example

  A better algorithm for computing prefix

• averages:

  Algorithm prefixAverages2(X): Input: An n-element array X of numbers.

  Output: An n -element array A of numbers such that A[i] is the average of elements X[0], ... , X[i].

  Let A be an array of n numbers.

   0 s for i  0 to n do s  s + X[i]  s/(i+ 1) A[i] return array A Analysis ...

Asymptotic Notation (terminology)

• logarithmic : O(log n)

  Special classes of algorithms:

  linear : O(n)

  2 quadratic : O(n ) k polynomial: O(n ), k ≥ 1 n exponential : O(a ), n &gt; 1

•  (f(n)): Big Omega --asymptotic lower bound
•  (f(n)): Big Theta --asymptotic tight bound

  “Relatives” of the Big-Oh

  _1,00E+10 Growth Functions _1,00E+08 1,00E+09 _{n 1,00E+06 1,00E+07 n log n sqrt n} log n _{1,00E+05 n^2} 100n ) _n^3 (n

  T _{1,00E+03 1,00E+04 1,00E+01} 1,00E+02 1,00E+00 _{1,00E-01 2 4 8 16}

₃₂

_{64 128 256 512 1024 2012-2013} n ₃₀

  2012-2013 ³¹ Growth Functions (2) 1,00E-01 ^{1,00E+11 1,00E+23 1,00E+35 1,00E+47 1,00E+59 1,00E+71 1,00E+83 1,00E+95 1,00E+107 1,00E+119 1,00E+131 1,00E+143 1,00E+155} _{2 4 8 16} ^T

₃₂

_{64 128 256 512 1024 n} ^{(n ) 100n n log n sqrt n n log n n^2 n^3 2^n}

  Asymptotic Notation (2)

  The “big-Omega” W-Notation

• asymptotic lower bound
• f(n) = W(g(n)) if there exists
• constants c and n , s.t. c g(n) £

  f(n) for n ³ n

  Used to describe best-case running

• f ( n )

  ng

  e

  times or lower bounds of algorithmic

  ni c g n ( ) im 

  problems

  T un R

  E.g., lower-bound of searching in

• an unsorted array is W(n).

  n

Input Size

  2012-2013 ³²

  Asymptotic Notation (4)

  The “big-Theta” Q-Notation

• asymptoticly tight bound
• f(n) = Q(g(n)) if there exists
• constants c , c , and n , s.t. c
_{1 2 1}

  g(n) £ f(n) £ c g(n) for n ³ n ₂ g (n ) c  ₂ f(n) = Q(g(n)) if and only if f(n) = ng

• f ( n )

  e O(g(n)) and f(n) = W(g(n)) ni im c g ( n ) ₁ 

  T un O(f(n)) is often misused instead of

• R

  Q(f(n))

  n

Input Size

  2012-2013 ³³

• Two more asymptotic notations
• "Little-Oh" notation f(n)=o(g(n)) non-tight analogue of Big-Oh
• For every c, there should exist n , s.t. f(n) £ c g(n) for n ³ n

• Used for comparisons of running times. If

  f(n)=o(g(n)), it is said that g(n) dominates f(n).
• "Little-omega" notation f(n)= w(g(n)) non-tight analogue of Big-Omega

  2012-2013 ³⁴ Asymptotic Notation (5)

• Analogy with real numbers
• f(n) = O(g(n)) @ f
• f(n) = W(g(n)) @ f ³ g
• f(n) = Q(g(n)) @ f = g
• f(n) = o(g(n))
• f(n) = w(g(n)) @ f &gt; g
• Abuse of notation: f(n) = O(g(n)) actually means

  2012-2013 ³⁵ Asymptotic Notation (6)

  

£ g

  @ f &lt; g

  f(n) ÎO(g(n)) Comparison of Running Times Running Maximum problem size (n) Time in

  

1 second 1 minute 1 hour

ms 400 n 2500 150000 9000000 20 n log n 4096 166666 7826087

  2 2 n 707 5477 42426

  4 n

  31 88 244 n _2012-2013

  2

  19

  25

  31 ₃₆

  Math You Need to Review Logarithms and Exponents properties of logarithms:

• log (xy) = log x + log y _{b b b} log (x/y) = log x - log y _{b b b a} log x = alog x _{b b}

  Log a = log a/log b _{b x x (b+c) b c}

  properties of exponentials:

• a = a a _{bc b c} a = (a ) _{b c (b-c)} a /a = a _{log b} b = a a _{c c*log b} b = a a
• Geometric progression
• given an integer n and a real number 0&lt; a ¹ 1
• geometric progressions exhibit exponential growth
• Arithmetic progression

  2012-2013 ³⁸ A Quick Math Review

  1

  2

  1 1 ...

  1 n n i n i a a a a a

  _ a ^       

   

  (1 ) 1 2 3 ...

  2 n i n n i n _

          Summations

  The running time of insertion sort is determined

• by a nested loop

  for j¬2 to length(A) for j¬2 to length(A) key¬A[j] key¬A[j] i¬j-1 i¬j-1 while i&gt;0 and A[i]&gt;key while i&gt;0 and A[i]&gt;key

  A[i+1]¬A[i] A[i+1]¬A[i] i¬i-1 i¬i-1

  A[i+1]¬key A[i+1]¬key

• n

  Nested loops correspond to summations

  2 ( j 1) O n ( )   j

  2  

  2012-2013 ³⁹

  Proof by Induction

  We want to show that property P is true for all

• integers n ³ n

  Basis: prove that P is true for n

• Inductive step: prove that if P is true for all k

• such that n £ k £ n – 1 then P is also true for n Example •

  n n n ( 1)

   S n ( ) i for n

  1    

  2 i

  

• 1

  Basis

  1(1 1)  S (1) i

    

  2 i _2012-2013  ₄₀ Proof by Induction (2)

• k

  Inductive Step

  k k ( 1) 

  S k ( ) i for 1 k n

  1      

  2 i _ n n 

           i  i 

  

2

( n 1 1) ( n n 2 ) n    

  ( n 1) n     

  2

  2 n n ( 1)

   

  2 2012-2013 ⁴¹

Thank You

  2012-2013 ⁴²