Matrix Multiplication

1.3.1 Matrix Multiplication

A matrix 8 A is a collection of m × n numbers ordered in m rows and n columns:

We write A ∈ R m ×n , where R ×n denotes the set of all real m × n matrices. If m = n, then The product of two matrices A and B is defined if and only if the number of columns

in A equals the number of rows in B. If A ∈ R m ×p and B ∈ R ×n , then C = AB ∈ R ×n , where

c ij =

a ik b kj ,

1 ≤ i ≤ m, 1 ≤ j ≤ n. (1.3.1)

k =1

The product BA is defined only if m = n and then BA ∈ R p ×p . Clearly, matrix multipli- cation is in general not commutative. In the exceptional case where AB = BA holds, the

matrices A and B are said to commute. Matrix multiplication satisfies the associative and distributive rules:

A(BC) = (AB)C, A(B + C) = AB + AC. However, the number of arithmetic operations required to compute the left- and right-hand

sides of these equations can be very different!

Example 1.3.1.

be given. Then computing the product ABC as (AB)C requires mn(p+q) multiplications, whereas A(BC)

Let the three matrices A ∈ R m ×p

p ,B∈R n ×n , and C ∈ R ×q

requires pq(m + n) multiplications.

8 The first to use the term “matrix” was the English mathematician James Sylvester in 1850. Arthur Cayley then published Memoir on the Theory of Matrices in 1858, which spread the concept.

1.3. Matrix Computations

27 If A and B are square n × n matrices and C = x ∈ R n ×1 , a column vector of length

n , then computing (AB)x requires n 2 (n + 1) multiplications, whereas A(Bx) only requires 2n 2 multiplications. When n ≫ 1 this makes a great difference!

It is often useful to think of a matrix as being built up of blocks of lower dimensions. The great convenience of this lies in the fact that the operations of addition and multiplication can be performed by treating the blocks as noncommuting scalars and applying the definition (1.3.1). Of course the dimensions of the blocks must correspond in such a way that the operations can be performed.

Example 1.3.2.

Assume that the two n × n matrices are partitioned into 2 × 2 block form,

where A 11 and B 11 are square matrices of the same dimension. Then the product C = AB equals

C 11 B 11 +A 12 21 A 11 B 12 +A 12 B = 22 .

A 21 B 11 +A 22 B 21 A 21 B 12 +A 22 B 22 Be careful to note that since matrix multiplication is not commutative the order of the factors

in the products cannot be changed . In the special case of block upper triangular matrices this reduces to

R 11 R 12 S 11 S 12 R 11 S 11 R 11 S 12 +R 12 S 22

0 R 22 0 S 22 0 R 22 S 22 Note that the product is again block upper triangular, and its block diagonal simply equals

the products of the diagonal blocks of the factors. It is important to know roughly how much work is required by different matrix algo-

rithms. By inspection of (1.3.1) it is seen that computing the mp elements c ij in the product

C = AB requires mnp additions and multiplications. In matrix computations the number of multiplicative operations (×, /) is usually about

the same as the number of additive operations (+, −). Therefore, in older literature, a flop was defined to mean roughly the amount of work associated with the computation

s := s + a ik b kj ,

i.e., one addition and one multiplication (or division). In more recent textbooks (e.g., Golub and Van Loan [169])a flop is defined as one floating-point operation, doubling the older flop counts. 9

Hence, multiplication C = AB of two square matrices of order n requires 2n 3

9 Stewart [335, p. 96] uses flam (floating-point addition and multiplication) to denote an “old” flop.

28 Chapter 1. Principles of Numerical Calculations

flops. The matrix-vector multiplication y = Ax, where A ∈ R n ×n and x ∈ R , requires 2mn flops. 10

Operation counts are meant only as a rough appraisal of the work and one should not assign too much meaning to their precise value. On modern computer architectures the rate of transfer of data between different levels of memory often limits the actual performance. Also usually ignored is the fact that on many computers a division is five to ten times slower than a multiplication.

An operation count still provides useful information and can serve as an initial basis of comparison of different algorithms. It implies that the running time for multiplying two square matrices on a computer will increase roughly cubically with the dimension n. Thus, doubling n will approximately increase the work by a factor of eight; this is also apparent from (1.3.2).

A faster method for matrix multiplication would give more efficient algorithms for many linear algebra problems such as inverting matrices, solving linear systems, and solv- ing eigenvalue problems. An intriguing question is whether it is possible to multiply two

matrices A, B ∈ R n ×n (or solve a linear system of order n) in less than n 3 (scalar) multiplica- tions. The answer is yes! Strassen [341] developed a fast algorithm for matrix multiplication

which, if used recursively to multiply two square matrices of dimension n = 2 k , reduces the number of multiplications from n 3 to n log 2 7 =n 2.807... . The key observation behind the

algorithm is that the block matrix multiplication (1.3.2) can be performed with only seven block matrix multiplications and eighteen block matrix additions. Since for large dimen-

sions matrix multiplication is much more expensive (2n 3 flops) than addition (2n 2 flops), this will lead to a savings in operations. It is still an open (difficult!) question what the minimum exponent ω is such that matrix multiplication can be done in O(n ω ) operations. The fastest known algorithm, devised in 1987 by Coppersmith and Winograd [79], has ω < 2.376. Many believe that an optimal

algorithm can be found which reduces the number to essentially n 2 . For a review of recent efforts in this direction using group theory, see Robinson [306]. (Note that for many of the theoretically “fast” methods large constants are hidden in the O notation.)