use autoconf macro acx_blas?

[Peter]

ticket:435 and ticket:511 are related

​http://autoconf-archive.cryp.to/acx_blas.html

I am not sure the macro does everything that we want (i.e. the checks we do today), but it should be worthwhile to give a glance...

[Peter]

ticket:435 was marked as related

[Peter]

ticket:511 was marked as related

[Peter]

I would like to use this macro now in the trunk.

I've tried it and it solves ticket:511 and ticket:524.

The current tests for cblas are not good and need to be rewritten anyway. The macro does not set -m64 like we do; I think that is a good thing because setting -m64 like we do make the test fails on every 32bit system. If we want support for that kind of flags, it should be outside library detection. In worst case, user have to provide the flags via ./configure LDFLAGS=-m64.

Is that OK with everyone?

[Jari Häkkinen]

Go ahead.

[Peter]

(In [1914]) Using ACX_BLAS macro for blas detection. The behavior will likely be a bit different from before. The major difference is that there are no -m64 flags in the atlas tests, which implies you might need to add -m64 to LDFLAGS (and possibly CXXFLAGS) at configure time. It also implies that you might detect atlas on your 32bit machine, which previously was not possible. fixes #349

There is a new configure option: '--with-blas' that can be used to specify which blas should be used. fixes #524

refs #518

[Peter]

(In [1916]) activate message about gslcblas. refs #349

[Jari Häkkinen]

configure fails on my machine. The acx_blas macro does not work for me since I have no fortran compiler installed on my machine. Removing the three first fortran specific calls in acx_blas.m4 allwos me to have a successful finish of configure.

[Jari Häkkinen]

I accidentally committed the patched acx_blas.m4 in changeset:1954. The commit at least highlights the troublesome rows.

[Peter]

I'm pretty sure it is the first line of those that is troublesome: AC_F77_LIBRARY_LDFLAGS.

Possibly, the problem is redundant, i.e., either of the lines would cause similar problems.

[Peter]

I suggest that we replace:

AC_REQUIRE([AC_F77_LIBRARY_LDFLAGS])

with

AC_PROG_FC
AS_IF([test "x$FC" != "x"],
      [AC_F77_LIBRARY_LDFLAGS])

I can submit the patch but perhaps it's better if someone (jari) without fortran tests it.

[Peter]

(In [1961]) refs #349 - the patch as discussed

[Jari Häkkinen]

Replying to peter:

AC_PROG_FC
AS_IF([test "x$FC" != "x"],
      [AC_F77_LIBRARY_LDFLAGS])

I now fail on the fortran compiler test AC_PROG_FC. There is no compiler and the macro fails.

[Peter]

(In [1976]) - acx_blas.m4 reversing back to pristine version

• configure.ac check if fortran compiler is available and if not ignore ACX_BLAS macro. If ACX_BLAS was not called or no libblas was not founda generic test to libcblas is called. The purpose of this is to detect the stand-alone libcblas available on e.g. Mac OS X, but I'm not sure the syntax is correct because currently I don't have access to test on a mac.

refs #349: if the detection works on mac I think we can close this ticket (again).

[Jari Häkkinen]

Replying to peter:

refs #349: if the detection works on mac I think we can close this ticket (again).

It works on my mac. There is a flaw though, it cannot detect -lcblas -latlas installation on machines that have no fortran compiler but use atlas based cblas. In such a scenario it will revert to GSL's blas implementation.

It is complicated to locate the proper BLAS library. Previously we had one search order that we tried to replace with acx_blas. Unfortunately, acx_blas does not implement the search order we prefer, or at least it expects fortran to exist. We need to create something new based on acx_blas to cover all cases we currently have. And the new thing must support a preference order.

I suggest that there should be a well defined order for searching for blas support. It should start with what we had before, but we should add a search for fortran blas aswell. If we can do this with modules/functions it may be easy to change the search order; some sort of search sequence like

cblas atlas fblas

would trigger a search where one tries to use a single cblas lib, then an atlas based cblas, and finally a fblas lib. If one sets the sequence to

atlas cblas

atlas would be the first lib to look for, the cblas but never fblas.

The first sequence should be the default string and the docs could explain how to change the search order.

The above would require us to rewrite acx_blas but we could use large part of it.

Is the above something we want to pursue to resolve the unsupported detection of the use case of atlas base cblas but no fortran compiler?

[Peter]

Replying to jari:

It works on my mac. There is a flaw though, it cannot detect -lcblas -latlas installation on machines that have no fortran compiler but use atlas based cblas. In such a scenario it will revert to GSL's blas implementation.

That is not really true. While testing my hack, I moved the fortran compiler away from my PATH to see that things worked as desired. In that case when providing LDFLAGS=-L/usr/lib/atlas the test for cblas will succeed and we get YAT_CBLAS_LIB=-lcblas. This is not optimal because it will not work when linking statically; you need to link with -latlas too (issue #511).

[...]

The above would require us to rewrite acx_blas but we could use large part of it.

Is the above something we want to pursue to resolve the unsupported detection of the use case of atlas base cblas but no fortran compiler?

The simple fix to support atlas without fortran would be to add test for this between ACX_BLAS and the cblas test. We could essentially copy the test from acx_blas.m4 but remove fortran related stuff.

[Jari Häkkinen]

Replying to peter:

That is not really true. While testing my hack, I moved the fortran compiler away from my PATH to see that things worked as desired. In that case when providing LDFLAGS=-L/usr/lib/atlas the test for cblas will succeed and we get YAT_CBLAS_LIB=-lcblas. This is not optimal because it will not work when linking statically; you need to link with -latlas too (issue #511).

YAT_CBLAS_LIB=-lcblas works on your machine because the operating system keeps track of the cblas library dependencies. Is this true in general? Maybe it is for dynamic libraries but as you point out static linking will not work. I use static linking on the BASE job servers ...

[Peter]

Another flaw is that --with-blas is only honored when fortran is available.

[Peter]

If I understand correctly, ACX_BLAS is adding to much for what is needed for GSL. We only need the C API and there is no need for the fortran API. GSL ​Manual says that one can link with the ATLAS package as: 'gcc example.o -lgsl -lcblas -latlas -lm', which supports that we do not need the fortran interface.

The question is whether the other tests in ACX_BLAS are also adding too much for the other blas packages. Who knows?

Perhaps we can move the test for atlas out from ACX_BLAS and remove fortran thingies from the test, which implies we would get the flags GSL suggests.

It is important that we test for atlas before testing for generic cblas, since when atlas is available the test for cblas is successful.

[Jari Häkkinen]

Replying to peter:

If I understand correctly, ACX_BLAS is adding to much for what is needed for GSL. We only need the C API and there is no need for the fortran API. GSL ​Manual says that one can link with the ATLAS package as: 'gcc example.o -lgsl -lcblas -latlas -lm', which supports that we do not need the fortran interface.

Well, we do not need a Fortran interface to BLAS and GSL obviously does not support it. GSL is written in C and expacts a C version of BLAS. I interpret your statement that we do not need a fortran interface such that you'd like to drop detection of fblas ... you mean, don't solve problems that isn't even hurting us.

The question is whether the other tests in ACX_BLAS are also adding too much for the other blas packages. Who knows?

Perhaps we can move the test for atlas out from ACX_BLAS and remove fortran thingies from the test, which implies we would get the flags GSL suggests.

Well, my objection is that we should not try to get what GSL suggest but rather get the flags proper for the system where you try to use yat et al.

It is important that we test for atlas before testing for generic cblas, since when atlas is available the test for cblas is successful.

I do not agree, I'd like to see a smarter search for cblas. Just finding a cblas is not enough, if it is then we can settle with gslcblas. The way I see it is that if there is an atlas independent cblas, it should be detected first (or at least used wihtout atlas). Such a cblas is probably better tuned for your hardware since it is provided by the OS you are compiling in. Of course, we can look for atlas first but should try to find another independent cblas also and use the independent cblas if found.

[Peter]

Replying to jari:

Replying to peter:

Perhaps we can move the test for atlas out from ACX_BLAS and remove fortran thingies from the test, which implies we would get the flags GSL suggests.

Well, my objection is that we should not try to get what GSL suggest but rather get the flags proper for the system where you try to use yat et al.

I just wanted to point to the GSL docs to support my suggestion on how to deal with the ATLAS detection (that is drop the fortran stuff).

It is important that we test for atlas before testing for generic cblas, since when atlas is available the test for cblas is successful.

I do not agree, I'd like to see a smarter search for cblas. Just finding a cblas is not enough, if it is then we can settle with gslcblas. The way I see it is that if there is an atlas independent cblas, it should be detected first (or at least used wihtout atlas). Such a cblas is probably better tuned for your hardware since it is provided by the OS you are compiling in. Of course, we can look for atlas first but should try to find another independent cblas also and use the independent cblas if found.

The order is not important to me so if you prefer cblas before atlas, fine. I just wanted to underscore that using simple tests it is dangerous to look for -lcblas and conclude that success implies presence of independent libcblas. As you say, we need to come up with something more clever if we wanna put cblas first in line. When using gcc one could perhaps try linking with -static, but I don't know if that solution would be portable to other compilers.

[Jari Häkkinen]

Replying to peter:

The order is not important to me so if you prefer cblas before atlas, fine. I just wanted to underscore that using simple tests it is dangerous to look for -lcblas and conclude that success implies presence of independent libcblas. As you say, we need to come up with something more clever if we wanna put cblas first in line. When using gcc one could perhaps try linking with -static, but I don't know if that solution would be portable to other compilers.

Yes, of course. You have already reported that you find -lcblas and it implicitly (shared libs) also finds atlas.

[Peter]

Replying to peter:

When using gcc one could perhaps try linking with -static, but I don't know if that solution would be portable to other compilers.

Following lines in the libtool script

# Compiler flag to prevent dynamic linking.
link_static_flag="-static"

indicate to me that the flag is not portable. The flag is set in configure.

One could perhaps find corresponding variable in configure and use it, but I doubt that is a documented variable and its name might therefore vary between different versions of libtool.

I need to understand if that variable name originates in a file that is in the repository or if it comes from libtoolize (that is the system script). If it comes from libtoolize, the behavior might vary between different versions of libtoolize, i.e., it varies for different developers. If it comes from e.g. libtool.m4 it feels more safe.

[Peter]

Replying to peter:

I need to understand if that variable name originates in a file that is in the repository or if it comes from libtoolize (that is the system script). If it comes from libtoolize, the behavior might vary between different versions of libtoolize, i.e., it varies for different developers. If it comes from e.g. libtool.m4 it feels more safe.

The libtool macro files are not checked in so the question above is irrelevant. Sorry for the confusion.

[Peter]

Using -static via libtool is very related to ticket:496.

[Peter]

We should not test for BLAS API we should test for CBLAS API, in other words, instead of looking for function sgemm, we should look for, e.g., function cblas_sgemm. This implies that macro call AC_F77_FUNC is no longer needed because the purpose of that macro is to find out if the function is named sgemm or something else.

As mentioned above, we should modify tests to look for CBLAS functions and not BLAS functions. Several of the libraries in acx_blas.m4 do not provide CBLAS library. Therefore, I think we should re-write ACX_BLAS in a such a way that we check for cblas_sgemm or some other cblas function that gsl uses, and we should remove detection of libraries that only have blas and not a C api. This is significant changes for ACX_BLAS so I suggest that we call the modified macro something else to avoid confusion (say YAT_CBLAS).

Then the issue in which order we should detect cblas libraries. First in line should be a void detection, i.e., if user has already defined LIBS appropriately we don't need to add anything more. Ideally, we should then try -lcblas followed by -lcblas -latlas. The problem with that, as noted before, is that -lcblas also works with an ATLAS provided cblas because lib holds information for the linker that -latlas is needed to, and this will cause problem when we wanna link statically (see ticket:511). I, therefore, think we should try atlas before trying -lcblas. One could argue that cblas is probably optimized harder, so in the situation that we have both cblas and atlas, cblas is probably preferable. We should remember, however, that atlas is typically installed outside the standard path, i.e., the user typically needs to provide something like LDFLAGS=-L/usr/lib/atlas, or the atlas package will be hidden for the linker. So it is very unlikely that atlas is stealing any detection from cblas without having the blessing from the user.

I went through the libraries in ACX_BLAS to see if they provide a C api:

BLAS linked to by default? (happens on some supercomputers)

We need the C API so this is not very useful.

BLAS in ATLAS library? (​http://math-atlas.sourceforge.net/)

Well known.

BLAS in PhiPACK libraries? (requires generic BLAS lib, too)

No C API ​http://www.icsi.berkeley.edu/~bilmes/phipac/ -lsgemm -ldgemm -lblas

BLAS in Intel MKL library? (​http://software.intel.com/en-us/intel-mkl/)

libmlk_core contains cblas interface according to ​here -lmlk_core (see IT++)

BLAS in Apple vecLib library?

No C API

BLAS in Alpha CXML library?

CXML is a collection of mathematical routines optimized for Alpha systems. There seems to be no C API.

BLAS in Alpha DXML library? (now called CXML, see above)

Predecessor to CXML so consequently no C API

BLAS in Sun Performance library?

No C API

BLAS in SCSL library? (SGI/Cray Scientific Library)

Does contain C API. ​http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi/srch4@scsl/linux/bks/SGI_EndUser/books/SCSL_UG/sgi_html/ch01.html#IG35746943 -lscs

BLAS in SGIMATH library?

No C API

BLAS in IBM ESSL library? (requires generic BLAS lib, too)

No C API

[Jari Häkkinen]

Replying to peter:

We should not test for BLAS API we should test for CBLAS API, in other words, instead of looking for function sgemm, we should look for, e.g., function cblas_sgemm. This implies that macro call AC_F77_FUNC is no longer needed because the purpose of that macro is to find out if the function is named sgemm or something else.

Of course, you are right. GSL uses cblas.

Then the issue in which order we should detect cblas libraries. First in line should be a void detection, i.e., if user has already defined LIBS appropriately we don't need to add anything more. Ideally, we should then try -lcblas followed by -lcblas -latlas. The problem with that, as noted before, is that -lcblas also works with an ATLAS provided cblas because lib holds information for the linker that -latlas is needed to, and this will cause problem when we wanna link statically (see ticket:511). I, therefore, think we should try atlas before trying -lcblas.

How do you go ahead with this? If you try -lcblas -latlas and everything is in a independent cblas (no atlas requires) then compilation will succed. Unless there is no atlas, then the compiler may fail on atlas not found.

One could argue that cblas is probably optimized harder, so in the situation that we have both cblas and atlas, cblas is probably preferable. We should remember, however, that atlas is typically installed outside the standard path, i.e., the user typically needs to provide something like LDFLAGS=-L/usr/lib/atlas, or the atlas package will be hidden for the linker. So it is very unlikely that atlas is stealing any detection from cblas without having the blessing from the user.

Well, on macs atlas is a part of a framework providing cblas. This is in the standard path. Just a comment.

I can live with your outline.

[Peter]

Replying to jari:

How do you go ahead with this? If you try -lcblas -latlas and everything is in a independent cblas (no atlas requires) then compilation will succed. Unless there is no atlas, then the compiler may fail on atlas not found.

Yes, if the linker sees no libatlas, I assume it fails. If atlas is in the link path, I guess the test will succeed and we choose to link against '-lcblas -latlas'. I that bad? If atlas appears before the independent cblas in the link path, the linker will link against libcblas in atlas dir, which implies we are linking against atlas. Fine. The other case is that we are linking against the independent libcblas and on top of that a libatlas that is not needed nor used. I don't see a problem with this except that it is a bit ugly. On the other hand, I would say it is quite unlikely to happen.

You could come up with some other strange case, when one, for example, have one libcblas that is dependent on libatlas and one wrapper libcblas that is dependent on say libblas. In this case weird things might happen depending on the exact circumstances, but in my humble opinion I think it's a bit strange to have two different libraries with the same name visible in the link path.

AFAICS, the bottom line is that if the linking works in the configure test, the actual linking of the app (using yat) will also work (unless the system has changed). The exception is, of course, that static linking might fail, and the best way to solve this problem is to also try static linking in configure. This is, however, a quite complicated problem (ticket #496). Without having #496 in our hands, the goal must be to minimize the risk for this to happen.

Well, on macs atlas is a part of a framework providing cblas. This is in the standard path. Just a comment.

In this context I don't consider that atlas, because there is no libatlas. But of course you could have atlas in the standard path, especially if you build atlas yourself and install it. Never done that though.

I can live with your outline.

OK, good.

[Peter]

(In [2017]) refs #349. Adding macro that checks for CBLAS library and using this rather than ACX_BLAS that checks for BLAS library.

[Peter]

(In [2018]) remove acx_blas.m4. closes #349