§1.2 Algorithm Details 10
The algorithm starts with a number of elements N assumed to be distributed over P
processing nodes. No particular distribution of elements is assumed and the only restrictions on the size of N and P are the physical constraints of the machine.
The algorithm presented here is similar in some respects to parallel shellsort [Fox et al. 1988], but contains a number of new features. For example, the memory over-
head of the algorithm is considerably reduced.
1.2.3 Infinity Padding
In order for a parallel sorting algorithm to be useful as a general-purpose routine, arbitrary restrictions on the number of elements that can be sorted must be removed.
It is unreasonable to expect that the number of elements N should be a multiple of the number of nodes P.
The proof given in [Knuth 1981, solution to problem 5.3.4.38] shows that sort- ing networks will correctly sort lists of elements provided the number of elements in
each list is equal, and the comparison-exchange operation is replaced with a merge- exchange operation. The restriction to equal-sized lists is necessary, as small examples
show
5
. However, a simple extension of the algorithm, which will be referred to as in- finity padding, can remove this restriction
6
. First let us define M to be the maximum number of elements in any one node. It
is clear that it would be possible to pad each node with M − N
p
dummy elements so that the total number of elements would become M
× P. After sorting is complete the padding elements could be found and removed from the tail of the sorted list.
Infinity padding is a variation on this theme. We notionally pad each node with M
− N
p
“infinity” elements. These elements are assumed to have the property that they compare greater than any elements in any possible data set. If we now consider
one particular step in the sorting algorithm, we see that these infinity elements need only be represented implicitly.
Say nodes p
1
and p
2
have N
1
and N
2
elements respectively before being merged in
5
A small example where unequal sized lists fails with Batchers’s merge-exchange sorting network is a 4 way sort with elements [1] [0] [1] [0 0] which results in the unsorted data set [0] [0] [1] [0 1].
6
A method for avoiding infinity-padding using balancing is given in Section 1.2.5 so infinity-padding is not strictly needed but it provides some useful concepts nonetheless.
§1.2 Algorithm Details 11
procedure hypercube_balanceinteger base, integer num if num = 1 return
for all i in [0..num2 pair_balance base+i, base+i+num+12
hypercube_balance base+num2, num+12 hypercube_balance base, num - num+12
end
Figure 1.2 : Pseudo-code for load balancing
our algorithm, with node p
1
receiving the smaller elements. Then the addition of in- finity padding elements will result in M
−N
1
and M −N
2
infinity elements being added to nodes p
1
and p
2
respectively. We know that, after the merge, node p
2
must contain the largest M elements, so we can be sure that it will contain all of the infinity elements
up to a maximum of M. From this we can calculate the number of real elements which each node must contain after merging. If we designate the number of real elements
after merging as N
′ 1
and N
′ 2
then we find that N
′ 2
= max0, N
1
+ N
2
− M and
N
′ 1
= N
1
+ N
2
− N
′ 2
This means that if at each merge step we give node p
1
the first N
′ 1
elements and node p
2
the remaining elements, we have implicitly performed padding of the nodes with infinity elements, thus guaranteeing the correct behavior of the algorithm.
1.2.4 Balancing
