Benchmarking of mathematical co-processors

While working on speed optimizations of fitspng, I encountered that the power function pow() slows extremely full run. (pow() is used in conversion from CIE XYZ to CIE Luv).

My inspection of glibc sources revealed that numerical functions provided by GNU glibc just only wraps implementation of ones by mathematical co-processor. No own rational approximation is implemented. The hardware support is directly assured by C99 IEEE standard.

Following the track, I created a small test code to test duration of execution various computing functions in GNU system math library. One is practically tests implementation of the functions in mathematical co-processors.


Fortran


program funs


  integer, parameter :: double = selected_real_kind(15)
  integer, parameter :: nsteps = 10000
  integer :: i,j
  real(double) :: x,y, s, xsteps = nsteps


  s = 0.0
  do j = 1,nsteps
     do i = 1,nsteps
        x = i/xsteps
        y = x*(1.0_double/3.0_double)
        s = s + y
     end do
  end do


  write(*,*) s
end program funs




C

#include

int main()

{

  const int nsteps = 10000;

  int i,j;

  double xsteps = (double) nsteps;

  double x,y,s;

  s = 0.0;

  for(j = 1; j <= nsteps; j++)

    for(i = 1; i <= nsteps; i++) {

      x =  (double) i /xsteps;

/*!*/ y = pow(x,1.0/3.0);

      s += y;

    }

  return s > 0.0 ? 1 : 0;

}

(note that no-effect summations and return values are added to prevent compiler's over-optimizations)

The function in line /*!*/ have been changed.

Computations has been done on Intel(R) Core Quad and verified on Intel(R) Xeon machines (cca 30% faster). The gcc and gfortran compilers gives the same results, gfortran may give a litte bit faster code).

Results are summarized in following table.


                    Core Quad
                 [nsec]   [nsec]   ratio fortran  i686
                   -O      -O4            -O4 
pow(x,1.0/3.0)  128.56   123.01    7.2   123.06  287.68
exp(1/3*log(x)) 120.21   116.14    6.8   117.14  269.20
cbtr(x)          90.15    85.79    5.0   -       -
sinh(x)          95.15    91.80    5.4    91.99  235.34
log10(x)         95.74    91.14    5.4    91.73   95.35
atan2(x,2.0)     83.94    79.22    4.6    79.09  106.00
log(x)           79.30    74.98    4.4    74.88   95.22
cosh(x)          78.13    74.34    4.4    74.61  198.99
tan(x)           78.89    73.29    4.3    73.50  118.17
atan(x)          64.27    58.02    3.4    57.99  101.92
exp(x)           58.97    54.58    3.2    54.55  143.93
asin(x)          58.52    53.87    3.2    53.44  165.31
acos(x)          58.70    55.30    3.2    54.94  165.32
cos(x)           51.63    45.57    2.7    45.64   93.98
sin(x)           50.03    44.29    2.6    44.77   92.30
sqrt(x)          47.68    37.23    2.2    39.35   34.11
fabs(x)          20.93    16.14    1.0    16.11   17.25
(1.0/3.0)*x      23.02    17.16    1.0    17.22   17.87

(i686 = Intel(R) Pentium(R) 4 CPU 2.66GHz, 0.376 nsec)
(Core Quad = Intel(R) Core(TM)2 Quad CPU Q6600  @ 2.40GHz, 0.4167 ns )

As expected, the optimization flag -O4 just only marginally speed-up the computation, because ones are actually passed to be computed by the co-processor.

From the table, It is directly visible that pow(x,a) is computed as exp(a*log(x)) because 5.5+7.5 is approximately 13.0. Moreover, the results perhaps shows that computing of pow() by direct use of the identity exp(..log) gives a little bit faster computation.
Also log10 is computed perhaps as log(x)/log(10) because 1.7+7.5 is 9.5.

Computation times divides functions onto groups by duration:

• fast: arithmetical operations, fabs
•  medium: sin,cos, tan, atan, cbtr, sqrt, asin, acos, sinh, cosh, log(10), exp, atan2
• slow:  pow

By the way, I changed computation of x^(1/3) from pow(x,1.0/3.0) to cbtr(x) which speeds-up my code just only about few percents. Of course, the slower computation of a general power is not solved  by the way.

Finally, computation times looks horribly but I think that a method which computes a function with precision of 15 decimals just only 7 times slower (!) than basic arithmetical operations is a miracle. The real world miracle.:)

Fitting of Straight Line

One from most trivial problems of statistical regression analysis is fitting of a straight line. I selected this well-known problem to illustrate

• usage of Minpack library to least square fitting by the least square method (LS) together with
• usage of robust statistical methods and also to prepare
• a simple reference example for testing of any software.

All required code can be found in the archive. Please read README for detailed description of included files.

Reference Data and Solution

As data for a working example, I selected a tabulated values for a straight line from excellent mathematical handbook: Survey of Applicable Mathematics by K. Rektorys et al. (ISBN 0-7923-0681-3, Kluwer Academic Publishers, 1994). The data set is included in the archive as line.dat.

Normal equations:

 14*a +  125*b = 170
125*a + 1309*b = 1148.78

and LS solution:

a   = 29.223
b   = -1.913
S0  = 82.6997
rms =  2.625
sa  =  1.827
sb  =  0.189

Using Minpack

Simple usage of Minpack in LS case is straightforward. One calls hybrd (Jacobian is approximated by numerical differences) or hybrj (must specify second derivatives) and one pass a subroutine to compute vector of residuals in a Minpack required point. Minpack uses Powell's method which combines location of minimum with conjugate gradient method (locate minimum in direction of most steeper slope) far from minimum and Newton's method (fit the function with multidimensional paraboloid and locate minimum by intersection of tangent plane with coordinate axis) near of minimum.

For the straight line, we define

a + b * x_i

and minimizing of sum for i = 1 to N:

S = ∑ (a + b * x_i - y_i)²

The vector for Minpack is

∂S/∂a = ∑ (a + b * x_i - y_i)
∂S/∂b = ∑ (a + b * x_i - y_i)*x_i.

The Jacobian is than

∂²S/∂a²    ∂²S/∂a∂b
∂²S/∂b∂a   ∂²S/∂b²

or

N          ∑ xi
∑ xi       ∑ xi²

All the sums can be found in minfunj in straightlinej.f90.

The call of hybrj search for a minimum of the function. On output, the located minimum is included in fvec and I added a code to compute covariance matrix to estimate statistical deviations of parameters and their correlation.

With gfortran I get the solution on 64bit machine:

....

minfun: par =   29.22344    -1.91302  sum =  82.68976
minfun: par =   29.22344    -1.91302  sum =  82.68976
hybrj finished with code           1
Exit for:
algorithm estimates that the relative error between x and the solution is at most tol.
qr factorized jacobian in minimum:
q:

0.10769513068469699       0.99418396628934158
-0.99418396628934158       0.10769513068469688

r:
152.97596122677001        1371.8039727109779
1371.8039727109779        17.656368881644468

inverse of r (cov):

0.25799238244686612      -2.87651125847342495E-002
-2.87651125847342495E-002  3.20772561895261684E-003

covariance:

1.7777772679919355      -0.19821501231688973
-0.19821501231688973      2.21038374592466315E-002

No. of data =           14
solution =    29.223435764531654       -1.9130248056275454
with dev =    1.3333331421636287       0.14867359368511487
residual sum  =    82.689756238430220
rms =    2.6250358130641160

The results must correspond (within precision of tree digits) to the reference solution. As we can see, there is a great discrepancy in deviations of parameters. The Minpack's estimation is little bit optimistic. I think that is due to difference between matrix inversion (which is usually used) and Minpack's covariance estimation. On the other side, the values are the same from practical point of view.

Just for information. The inverse matrix (all by Octave) of Jacobian in minimum is (inv(.))

 0.4846353  -0.0462792
-0.0462792   0.0051833

and the QR factorization ([q,r,.]=qr(.)):

q =
-0.995472   0.095060
-0.095060  -0.995472

r =
-2.0541e+00 -1.2576e+02
0.0000e+00  -1.3150e+03

The Q matrix columns are base vectors (eigenvectors) of solution (the principal axes of covariance ellipsoid) and the diagonal elements are estimates of eigenvalues values (major and minor semiaxes of the ellipse) [l,v]=eig(.).

l =
-0.995457   0.095208
0.095208   0.995457

v =
2.0447e+00   0.0000e+00
0.0000e+00   1.3210e+03

The second supplied routine straightline.f90 does the same work but without explicit knowledge of the second derivatives. The Jacobian is estimated by numerical differences.

The solution can be also done via lmdef, lmder routines in Minpack. It is equivalent to presented solution but doesn't offers generalization toward robust methods.

Robust Fitting

The reference robust fitting procedure is included in rstraightline.f90.

The fitting is logically divided onto two parts. The first part implements minimizing of sum of absolute deviations to get robust estimation of proper solution and MAD (mean of absolute deviations). There is little change with respect on LS because minimizing function have no derivation in minimum. We need another method without using of derivatives. I'm using code prepared by John Burkardt, namely using Nelder-Mead Minimization Algorithm (simplex method). I slightly rearranged the code to nelmin.f90.

The resultant parameters are used to obtain MAD by looking for its median by a quick way algorithm described in Niklaus Wirth's Algorithms + Data Structures = Programs.

The solution is than passed as start point for hybrd which is the second part. The minfun is similar to non-robust version. Only difference between predicted and computed solution (residual) is not directly used, but a cut-off function is used (Tukey's function). This small change does robust fitting itself.

.....
medfun: par=   30.57930  -1.918842   sum=   29.9467137
medfun: par=   30.57930  -1.920842   sum=   29.9506577
ifault=           0   29.940157123526994
t=   30.579306941153348       -1.9198428764730142
4.3939266204833984        2.9615066
minfun: par =  30.57931  -1.91984   sum =  106.17691
.....

minfun: par =  29.24548  -1.91425   sum =   82.69177
hybrd finished with code:           1
Exit for:
algorithm estimates that the relative error between x and the solution is at most tol.
qr factorized jacobian in minimum:
q:

-0.10942034931424138      -0.99399556696996860
0.99399556696996860      -0.10942034931424133

r:

29.315789045435952        295.17538616058539    
295.17538616058539           4.3667770003656630

inverse:

5.3177769129754946      -0.52802747846866527
-0.52802747846866527      5.24418460845494372E-002

covariance:

2.7756865626932967      -0.27561118127052436
-0.27561118127052436      2.73727405045027551E-002

No. of data =           14
solution =    29.245477844988198       -1.9142493591979692
with dev =    1.6660391840209812       0.16544709276533920
residual sum  =    82.691772460937500
rms =    2.6250678159642766

The output values are practically the same as in non-robust case. Only the difference is estimation of parameter's deviation. I'm using the formula recommended by Hubber (1980), eq. (6.6) p. 173.

The real power of the robust fitting can be easy demonstrated by adding any outlier (point with really different value) to the set, for example, a point with coordinate 10,100. Try to see the robust algorithm in action. It should be practically the same while non-robust solution gives some strange values.

Minpack Fortran Interface

The original Minpack is written in Fortran 77. I'm using modern Fortran (Fortran 90, 95 or 2003) which supports better type checking via interfaces. I prepared such interface which is included in Archive as minpack.f90 and must be passed to compiler during compilation. The module did not changed original API to Minpack routines to prevent any programming errors. So you also must pass to the routines "working arrays" (wa). One is used in modern Fortran more elegant way as automatic arrays (arrays allocated automatically when subroutine is entered and deallocatedon its exit).

Estimation of an initial solution

The solution will not depend on starting point only in linear case. Every complex real) case will lead to non-linear solution with a lot of local minimums which will attract the simplex or the gradient (Powell's method) to a "wrong" solution. To locate global minimum (eg. required solution), I recommends use of genetic algorithms as predictors of a global minimum. The genetic algorithms will locate right minimum with a low precision and we can use some modification of above codes to determine the minimum with required precision.