Using Containers on HEP Linux

Software Containers can be run in a secure environment on HEP Linux systems. No special access or authorisation is required.

Note the use of containers and the HEP Singularity library is still in beta and may change with short notice.

Container Technology

The container technology provided and supported is Singularity. Singularity has some key advantages over Docker for use on our shared use systems:
  1. Users can run 'untrusted' containers without any special requirements or permissions.
  2. The containers run in a more secure environment, protecting both the user and the system from malicious or broken code.
  3. Local user accounts and storage areas are integrated seamlessly into the container environment.
These features allow containers to be run by any user locally on their desktop, on interactive nodes or in batch jobs just like any other application both textual and graphical.

Singularity supports both its own container format but also Docker images (which are converted locally into Singularity format on the fly).

Obtaining Containers

Containers can be downloaded and/or run from a number of sources
  • Local file: eg /bundle/data/Containers/hep_sl6.sif
  • CVMFS repository: eg /cvmfs/\:latest/
  • Singularity Library: eg library://library/default/alpine
  • Docker Library: eg docker://ubuntu
Containers can be run directly from their source (the image will be downloaded and/or cached in a local scratch area first).

Container images can be saved to a local image file with the pull command eg
  • singularity pull /local/path/alpine.sif library://alpine:latest
When running a single container any source can be used. If many containers are going to be run at the same time (eg in batch jobs), it is requested that container images are downloaded to local shared storage first with singularity pull (eg to hepstore or bundle data). This reduces load on the libraries and reduces the chances of issues with containers changing over time or local caches not updating properly.

Running Containers

Containers can be instantiated in three different ways
  • Run: If the container has a default action this will be run, usually to start a shell or execute the main application in the container.
    • singularity run library://sylabsed/examples/lolcow
  • Exec: Attempt to run a specific command in the container. Singularity will exit once the command has finished. The command can be used to run a specific shell within the container eg tcsh.
    • singularity exec docker://centos cat /etc/os-release
  • Shell: Log in to the default shell in the container
    • singularity shell /bundle/data/Containers/hep_sl6.sif

Singularity caches images locally in $HEPTMP/ after they are first downloaded. Some sources don't check if the image has changed since it has last been cached. If the images don't update properly if a new image is released you can clean the cache completely with
  • singularity cache clean --all
By default Singularity will mount standard user storage mounts (eg /user, /hepstore) inside the container. If you have any extra storage you wish to use within the container, eg experiment data stores, manually add it to be bound with the -B switch eg to add /mydata to the container
  • singularity shell -B /mydata /bundle/data/Containers/hep_sl6
Singularity containers will inherit any environment variables from the shell you start it in. This may interfere with some software environments within the container. To start the container with a completely clean environment use the -e switch eg
  • singularity shell -e /bundle/data/Containers/hep_sl6

Example Containers

These containers are intended to be generic environments with enough base libraries for running software sourced elsewhere (eg CVMFS or your user area). You can also use them as base for creating your own containers.



This container has a modest install of required SL6 packages intended for use with software provided via CVMFS. We will only add packages necessary to support CVMFS software collections to this image. The default environment when started with 'run' sources standard development tools using LCG Views from CVMFS. When started with 'shell' the environment has no extra configuration, ready for adding your own setup.

This container may be used to provide SL6 environments for development and batch jobs, allowing the aging SL6 systems to be retired or reinstalled with Centos 7.

HEP SL6 Full


A more complete image with more libraries and utilities installed for use either stand alone or with CVMFS. The image file is much larger.

HEP Centos7


This container has a modest install of required Centos7 packages intended for use with software provided via CVMFS. We will only add packages necessary to support CVMFS software collections to this image. The default environment when started with 'run' sources standard development tools using LCG Views from CVMFS. When started with 'shell' the environment has no extra configuration, ready for adding your own setup.

HEP Ubuntu


Similar to the SL6 containers this is a minimal install based on Ubuntu 18.04LTS for use with software sourced from CVMFS. We will only add packages necessary to support CVMFS software collections to this image. The default environment when started with 'run' sources standard development tools using LCG Views from CVMFS. When started with 'shell' the environment has no extra configuration, ready for adding your own setup.

Depending on which LCG views you use you may see an error message like this:

Singularity hep_ubuntu_latest.sif:~> source /cvmfs/
/cvmfs/ symbol lookup error: /cvmfs/ undefined symbol: UP

This can be fixed by forcing the correct readline library to be used with LD_PRELOAD eg
  • LD_PRELOAD=/usr/lib/x86_64-linux-gnu/ source /cvmfs/

HEP Ubuntu Full


Similar to the HEP Ubuntu container but with a lot more utilities and development libraries installed. The image file is much larger.

Example Usage

These are some examples of how to use Singularity in the HEP Linux environment.

Run in Ubuntu environment

Start an Ubuntu session with development tools from CVMFS and run a python command:
  • user@host mydir/ $ singularity run /bundle/data/Containers/hep_ubuntu.sif
  • Singularity> user@host mydir/ $ python
  • Singularity> user@host mydir/ $ exit
  • user@host mydir/ $

Run SL6 software on Centos7 Batch

SL6 software can be run on Centos 7 nodes. For an SL6 package installed in /hepstore/user/mydir and submitted to a Centos 7 Slurm batch queue:
  • Pull relevant container to local storage if needed (hep_sl6-full is recommended for general SL6 compatibility)
    • singularity pull /hepstore/user/hep_sl6-full.sif /bundle/data/Containers/hep_sl6-full.sif
  • Create job submission script
singularity -q exec /hepstore/user/hep_sl6-full.sif /hepstore/user/mydir/
  • should be executable and source whatever CVMFS or other environment is required (or source the default with 'source /etc/profile')
  • Submit job to the batch system
    • sbatch

Customising Containers

New containers can be built from scratch or using an existing container (either Singularity or Docker) as a base. This is relatively simple to do but does require root access. If you are running Singularity (or Docker) on your personal system you can use your normal sudo access to do this. On managed HEP Linux systems sudo is not normally available, but Singularity allows containers to be run in a secure environment with effective "fake" root access. This is activated by passing the --fakeroot option eg
  • singularity shell --fakeroot /bundle/data/Containers/hep_sl6.sif
Within the shell you will be logged in as root, giving you full access to modify the container environment eg add packages, change config files.

To use --fakeroot your HEP account will need to be given permission to do this. Request this via HEP helpdesk.

Fakeroot can also be used for building new containers. A typical workflow is to create a local sandbox container from an existing container, modify it, then build the final immutable image file eg
  • singularity build --sandbox --fakeroot /tmp/my_box /bundle/data/Containers/hep_sl6.sif
  • singularity shell --fakeroot --writable /tmp/my_box/
  • ...make modifications, add packages, then tidy up (remove tarballs, yum clean etc) and exit shell...
  • singularity build --fakeroot /tmp/my_box.sif /tmp/my_box/
The container my_box.sif can now be used directly or uploaded to a registry.

If you need assistance running, building or uploading containers to the library, or wish new features to be added to the existing HEP containers please contact HEP helpdesk.

Further Reading

The User and Admin guides are available online at the Sylabs website: (current version on HEP Linux is 3.3). Be careful not to follow guides on as these are for older versions and are very out of date now.

The singularity commands have man pages (eg man singularity-pull).

The main Singularity library can be browsed online at

The sregistry client has documentation about general usage and the various backends eg for Singularity Registries

HEP Container Library

Due to stability and compatibility issues this service is currently suspended. Many experiment distribute their containers through /cvmfs now.

We maintain a local library of containers in a Singularity registry at available both on and off-site.

Access is via Gitlab, meaning anyone with a HEP Linux account or an authorised external account can log in and add containers to existing or new collections.

The Library uses Singularity Registry which can be accessed via the web interface, or via the sregistry client.

Users can be grouped in Teams. Teams can be open to anyone that wants to join if they are public. If they are private then they must be invited with a web link generated in the interface under Users>Invite users.

Collections can be private (only viewable and readable by owners) or public (viewable and readable by anyone). To add containers to a collection users must be added to a Team, and then users from that Team added as owners for that collection. Teams themselves cannot be added.

Using sregistry Client

The sregistry client is available in local Python 3 installs eg
  • pythonSetup 3.6.3
To push images to the HEP Library using the client you will need to export your Sregistry Token. Log in to the web interface at Click on your username and then on Token. Paste this token into your $HOME/.sregistry file.

The client has a number of backends available for accessing different libraries, to enable the HEP library registry use
  • sregistry backend enable registry
Any commands will now use the HEP library for pull/push.

To upload an image file "localimage.sif" to the "myexpt" collection and name the container "mynewimage" use
  • sregistry push --name myexpt/mynewimage localimage.sif
If the myexpt collection doesn't already exist it will be created as a private collection.
Topic revision: r14 - 04 Nov 2020, JohnBland
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback