Writing and Running Applications on Xeon Phi
Xeon Phi 7120P co-processor cards based on the Knights Corner chipset are available for MIC evaluation and development. The cards are installed in the interactive node phi.ph.liv.ac.uk, accessible from the HEP network.
Accessing the Co-processors
The co-processors can be accessed directly from phi.ph.liv.ac.uk using ssh. Log in with your usual HEP Linux username/password. Your default shell must be bash. Home directories are in /user, please contact the admins about moving your home directory to the default location if it can't be found (only new users are put into the default location).
There are three cards, with the following hostnames and assignments
Name |
IP Address |
Allocation |
hep-mic |
192.168.178.202 |
General development |
lhcb-mic |
192.168.178.203 |
LHCb development |
neutrino-mic |
192.168.178.204 |
T2K /Snoplus development |
If you need access to more than one card for a project please contact the admins.
The standard filestores are available (/scratch, /hepstore, /batchsoft) along with user home directories.
The cards run a limited Linux distribution, compiled for the MIC architecture. Standard x86_64 binaries won't run (scripts should be fine).
Acceleration modes (local or offload)
The cards can run native MIC code locally eg by ssh-ing to the card and running the application as on a normal node.
Applications can also run on the host phi.ph.liv.ac.uk, and offload certain functions to the co-processors, called off-loading. The code must be compiled with the off-load compiler and with the correct use of extra compiler directives. Any libraries that the code links to on the x86_64 host must also be made available in MIC format on the co-processor.
See the developer documentation for more information.
Cross-compiling for the MIC architecture
Intel Compiler
The
IntelĀ® Parallel Studio XE for Linux compiler suite is installed for evaluation and student development. Academic staff working on projects specifically funded for Xeon Phi development will need to acquire a full license for themselves.
The suite includes C (icc), C++ (icpc) and Fortran (ifort) compilers. On phi.ph.liv.ac.uk configure your shell to use these compilers by including their setup with
source /opt/intel/bin/compilervars.sh intel64
To enable cross compiling for the MIC architecture add -mmic to the compiler arguments eg
icc -mmic -o code code.c
Any libraries that the code links against will also need to be cross-compiled with -mmic, standard x86_64 libraries cannot be linked against MIC binaries. Link with the Intel compiler.
If using autotools eg ./configure, the host type can be specified to force cross-compiling with --host=x86_64-k1om-linux eg
CC=icc CFLAGS=-mmic LDFLAGS=-mmic ./configure --host=x86_64-k1om-linux
Again ensuring external libraries are compiled for MIC.
GCC
Currently (16/05/2014) GCC does not natively support compiling for the MIC architecture used on Knights Corner so cannot be used for MIC development. It is probable that support will be added in the future, although this may only be for newer MIC architectures (eg Knights Landing).
The MIC software suite includes a patched GCC 4.7 which is used for producing the Linux distribution that runs on the co-processors. This does not include any options for vectorisation and so does not produce binaries that exploit the full Xeon Phi performance, we do not recommend this be used for performance-critical work, if at all.
Monitoring activity
Standard tools like
top
,
ps
,
free
etc are available on the co-processors directly. Please use these sparingly as they can be a performance drain in some circumstances.
Real time co-processor activity can be viewed with the
micsmc
utilitity. This produces a window with processor and RAM utilisation, as well as general system states and errors.
This utilitity can also be used to show the processor state on the command line with eg
micsmc -a
Historical activity metrics are generated using Ganglia, adding them to the phi.ph.liv.ac.uk entry. See the
phi ganglia page. Metrics are recorded as phi-METRIC-micN.
Further Reading
There is a
Liverpool MIC development mailing list which we strongly encourage any interested developers to subscribe to. Contact the admins if you have trouble subscribing.
Some quick start guides for MIC development are attached to this topic.
The
Intel website has numerous documents for MIC development, debugging and optimisation.
Administration
After a kernel update the 'mic' module needs to be rebuilt for the new kernel otherwise the system becomes unstable. The module sources are kept in
/batchsoft/mpss/mpss-3.2.1/src/
. The new modules can be built with
-
rpmbuild --rebuild mpss-modules-3.2.1-1.el6.src.rpm
The new modules should be installed and the system rebooted.
--
JohnBland - 15 May 2014