FAQs.

ASPIRE1 is a supercomputer operated by the National Supercomputing Centre (NSCC), and built on x86 based architecture with initial benchmarks of 1.01 PFLOPS of compute throughput, 13 PetaBytes storage, and I/O performance of up to 500GBytes/sec. This supercomputer has 1,288 compute nodes with more than 30,000 cores, including 128 GPU nodes, and 9 large memory nodes ranging from 1TB to 6TB RAM. ASPIRE1 has login nodes in NUS, NTU and A*STAR extended with DTN-InfiniBand-longrange link.

ASPIRE1 is built on a CentOS Linux platform and currently does not have any Microsoft Windows OS.

PBS Compute Manager is a graphical user interface which facilitates users to run, monitor, and manage the jobs with ASPIRE1. The PBS Compute Manager can be accessed from here:
https://aspireweb.nscc.sg

What is PBS Display Manager
PBS Display Manager is a graphical user interface which facilitates users to run graphics intensive applications in ASPIRE1. The PBS Display Manager can be accessed from here:
For A*STAR users – https://astar.nscc.sg
For NTU users – https://ntuweb.nscc.sg
For NUS users – https://nusweb.nscc.sg
For SUTD users – https://sutd.nscc.sg
For other users – https://aspireweb.nscc.sg

PBS Display Manager is a graphical user interface which facilitates users to run graphics intensive applications in ASPIRE1. The PBS Display Manager can be accessed from here:

For A*STAR users – https://astar.nscc.sg

For NTU users – https://ntuweb.nscc.sg

For NUS users – https://nusweb.nscc.sg

For SUTD users – https://sutd.nscc.sg

For other users – https://aspireweb.nscc.sg

NSCC’s enrollment system is federated with SingAREN to facilitate users from NUS, NTU, SUTD, SMU and A*STAR to register seamlessly to ASPIRE1. Please follow the instructions presented in "Enrollment user guide" from "User Guide" page for the users from above mentioned organizations. If you don’t belong to one of the organisations above, please contact [email protected] with the following details:

User ID (desired, preferrably same login ID which you are using in your organization, limited to 8 characters):

First Name:

Last name:

E-mailaddress:

Organization name:

Note: Login ID creation is subjected to availability and approvals.

I'm an NUS user and my NUS ID is in NUSEXT domain, and I cannot register through the enrollment system. What is the reason?

NUSEXT domain users are considered external users for NUS. Please contact [email protected] for registration to ASPIRE1.  

To utilize ASPIRE1 resources, you need to login to the cluster. Please follow the below table to determine how to access ASPIRE1.

Follow the below instructions to connect to ASPIRE1 through portal.

• Open the web browser and browse for the URL provided above.
• Use your ASPIRE1 credentials to access the web portal.

Below are instructions to connect using SSH/SCP on UNIX/Linux or Mac:

• Open any SSH terminal
• Type “ssh <userid>@<Host/FQDN>”
• Enter the password (characters are invisible while typing the password)
• Once logged in successfully you should be able to see a “$” prompt that allows you to execute Linux commands

If you want to use the X11 interface, replace the above ssh command with ssh -X.

For X11 – ssh -Y, make sure you have XQuartz installed for OS X 10.8 or higher.

Below are instructions to connect using SSH/SCP on Windows:

• Open any SSH terminal software (e.g. putty, MobaXterm)
• In case of putty, type the login host name and click on Open button (Putty may allow you to login automatically without asking for password. In case if putty asks for username/password, you need to type the login ID and password from your university.)
• In case of MobaXterm, type to command at the prompt “ssh [email protected]”
• Upon successful login, you will be prompted with “$” to use NSCC Supercomputer

Please follow the Password Reset guide and follow the step-by-step instructions.

I forgot my password. Can I reset it?

Please follow the Password Reset guide and follow the step-by-step instructions.

What are the prerequisites for my password?

The complexity requirement is below:
Minimum 8 characters
Mixture of upper and lower case
Must contain numbers  (0-9)
At least include one special character: [email protected]#$%^&*+=?><

Where can I find more information about NSCC and its movements?

http://nscc.sg - Corporate website

https://www.nscc.sg/hpc-calendar - Information on workshops

https://help.nscc.sg - Technical details of ASPIRE1 and NSCC’s facilities.

Internet access from remote login nodes are disabled due to security policies at each entity. However, users will still be able to access files on the Internet using the following methods.

A*Star Users - login to astar.nscc.sg and download files from Internet

NUS users - ssh to nscc04-ib0 after logging in to nus.nscc.sg, and download files from Internet

NTU users - ssh to nscc04-ib0 after logging in to ntu.nscc.sg, and download files from Internet

Users accessing through aspire.nscc.sg will be able to access the Internet, once logged in.

NSCC password expires every 90 days. Upon expiry of the account you will not be able to login to NSCC system any more, the below message will be displayed when tried ssh connection. 

Your account has expired; please contact your system administrator
You are required to change your LDAP password immediately.
Connection closed by xxx.xxx.xxx.xxx

Solution:

In order to resolve the issue you need to login to https://user.nscc.sg/saml with your organization credentials and click on set/reset your password. One new password is set you can login to NSCC with the new password. 

CPU resource in ASPIRE1 supercomputer can be utilized through Personal-project.

The quota is fixed and 100000.00 CPU hours is granted for every personal project for a period of ASPIRE1 Life time. If the personal quota is exceeded, users can only submit jobs through a valid project. 

The Personal-project grant of CPU hours will not be extended. 

Please send an email to [email protected] with the updated request to get your request approved and provisioned at the earliest. 

You can check your current CPU usage by executing the “myusage” command.

$ myusage

Usage in 7 days to 2017/09/19:
Number of jobs on CPU nodes: 300
Core hours on CPU nodes: 81,592.3
Number of jobs on GPU nodes: 0
Core hours on GPU nodes: 0.0

Usage in month to 2017/08/31:
Number of jobs on CPU nodes: 131
Core hours on CPU nodes: 778,539.6
Number of jobs on GPU nodes: 0
Core hours on GPU nodes: 0.0

Usage since system start to 2017/08/31:
Number of jobs on CPU nodes: 6,368
Core hours on CPU nodes: 15,577,192.9
Number of jobs on GPU nodes: 0
Core hours on GPU nodes: 0.0

The disk quota allocated is limited as stated below respectively :

/home    - Home directory    - 50GB.
/scratch - Scratch directory - 100 TB

This will not be extended on any terms. If more disk space is needed, you should raise a project request through https://user.nscc.sg/project.

You can check your current disk usage on /home by executing the “myquota” command.

$ myquota

===============================================================
|File System               Usage (GB)               Limit (GB) |
|nscc_home                      30.10                    50.00 |
|/scratch                    43204.67                102400.00 |
===============================================================

Purging policy is implemented on scratch directory and files unused for more than 30 days will be purged automatically.  

File under home or project directories are untouched.

The word MPI stands for Message Passing Interface. One of the most used technologies to parallelize the applications. There are various implementation of MPI such as OpenMPI, MVAPICH, IntelMPI, etc.,
MPI Interface is a wrapper to C/Fortran compiler, which means that all the codes which are used in C/Fortran can be used with MPI in addition to penalization techniques.
ASPIRE1 is powered with IntelMPI for better performance of applications.

One of the techniques to run your code in parallel is OpenMP. Codes using OpenMP are normally restricted to run in one physical server, unless you are running a hybrid code made out of MPI and OpenMP. OpenMP codes themselves are not capable of communicating through a network infrastructure such as Infiniband. OpenMP code is simple to use and does not need any wrappers, standard compilers like GCC/Gfortran or Intel C/Fortran can use OpenMP. However, the program must have OpenMP directives.

A serial application is an application which can use only one process at a time to perform calculations, while a parallel application can scale to multiple processes and make use of cores/CPUs in one or multiple servers using high speed Interconnects. For this reason, parallel applications produce faster results in general.
To promote the HPC experience, it is strongly recommended to use parallel codes on ASPIRE1 if applicable.

Checkpoint is a technique used to save the output periodically during a runtime of an application. The advantage using checkpoint is that it is possible to restart the program from the last checkpoint, in case if the execution of program terminates abruptly due to any reason.
You are advised to use the checkpoint and restart technique while writing your application so that in case of eventualities, you should be able to restart the job from the last checkpoint.

The available compilers are as follows:

• gcc (v4.9 & v4.4)
• Intel (v17.0 & v16.1)
• NAG (???)

The applications, Libraries, and compilers are very dynamic and powered with Environment modules in NSCC Supercomputer, to list the available modules, use “"module avail"” from at the prompt in NSCC Supercomputer login nodes.

NSCC does not support compilers other than the versions mentioned above at the moment. If you’re keen to use other compilers, you are more than welcome to install it into your own home directory. Please contact our service desk (access https://servicedesk.nscc.sg/ or contact [email protected]sg) if you face any difficulties during the installation.

Yes, you can request for additional libraries or applications through the Service Desk portal. However the installation of libraries is subjected to various conditions such as compatibility, the time required to make the library available, dependencies and our software policies. Please contact Service Desk for further clarification. Our Technical specialist will respond to make the most of your request.

I want to use a commercial application. How can I use it on ASPIRE1?

Commercial applications can be installed in ASPIRE1, but they need to be installed and used following the licensing terms of each software application. First of all, the license and the software modules of the intended application need to be provided by you to NSCC for installation. Based on the license terms and conditions, we will determine whether and how the software should be installed in ASPIRE1. The same rule applies to academic licenses.

Please note that the accountability on the usage of the software should be held by the owner of the license. For more information, please contact our service desk.

Jupyter notebook is not in the default supported applications. However, Jupyter notebook is available with anaconda package. Follow the below instructions to use Jupyter notebook.

1. Login to display manager (Please see the display manager instructions provided under user guides)
2. Double click on GlxSpheres app to launch the application 
3. Close the GlxSpheres by clicking on "x" on top right corner
4. Open terminal through Menu -> Tools -> Terminal
5. Execute the below commands at the terminal 
   module load anaconda/3
   export PYTHONHOME=/app/anaconda/3
   export PATH=/app/firefox:$PATH
6. Open jupyter notebook with the command "jupyter notebook"

There are many reasons jobs may be prevented from starting. The first thing to do is to run “qstat -s ”; this will print the comments from the job scheduler about your job.

If you see a “–”"Q" in the column "S", it means the scheduler has not yet executed your job. Be patient.

There could be several reasons for this, few common reasons listed below:

1. The operating system that you are using is different from that in ASPIRE1
   Solution:
   • if you are running in your local Linux machine, a simple recompilation on the ASPIRE1 login nodes will solve the issue.
   • if you are running in Windows PC and you want to run inASPIRE1, you need to either obtain a copy of the software for Linux or port the code from Windows to Linux.
2. Different compiler/library stack: if  your job is not running although your operating system in your own environment is the same as ASPIRE1, it means the compiler/libraries are not compatible. You need to recompile your codes on ASPIRE1 login nodes.

Please contact our service desk (access https://servicedesk.nscc.sg/ or contact [email protected]) if you still have any issues with running your job on ASPIRE1.

This error could be because you have forgotten to source the appropriate system .rc in your personal .rc file.

• If you are using an sh-derived shell for your jobs, edit the .bashrc file to ensure it contains the line “. /etc/bashrc”.
• If you are using a csh-derived shell for your jobs, edit the .cshrc file to ensure it contains the line “source /etc/csh.cshrc”.

If you have accidentally deleted the .bashrc file, you can copy back the original .bashrc file from /etc/skel/.bashrc .

Example:

$ cp /etc/skel/.bash* ~/

If your batch job is named, runjob.sh and your output is not redirected in the batch script, then your job output will appear in runjob.sh.o****, where the digits at the end of the file name represent your job ID. The final entries in the .o file give you the details on wall time and virtual memory used by the job.

Similarly, error messages of your jobs will be recorded in runjob.sh.e****, where the digits at the end of the file name represent your job ID.

In a PBS job script, the memory you specify using the -lmem= option is the total memory across all nodes. However, this value is internally converted into the per-node equivalent, and this is how it is monitored.
For example, since NSCC Supercomputer has 24 cores per node, if you request -l slect=2:ncpus=24, mem=10GB, the actual limit will be 10GB on each of the two nodes. If you exceed this on either of the nodes, your job will be killed.
Please note, if a job runs for less than a few minutes, the memory use reported in your .o file once the job completes may be inaccurate. We strongly discourage people running short jobs of, e.g. less than 1 hour. This is because there is significant overhead in setting up and tearing down a job and you may end up wasting large amounts of your grant. Instead, if you have many short jobs to run, considering merging them together into a few longer jobs

Please note that you are not supposed to run any job on the Login nodes. We will automatically kill if we see any user running a job on the login node. If you want to run an interactive session, please submit and interactive job into the compute nodes using the qsub command.

$qsub -I

The queues available for job submission are :

• normal
• gpu
• largemem

There are several queues in the system with different priorities which works for different workloads. Below are some of the queues in the system:

However these queues does not accept jobs directly rather jobs will be routed once submitted to normal/gpu queue. 

Please raise a project request through https://user.nscc.sg/project. Please note that all project requests are subject to approval.

E : Job is exiting after having run
F : Job is finished. Job has completed execution, job failed during execution, or job was deleted.
H : Job is held. A job is put into a held state by the server or by a user or administrator. A job stays in a held state until it is released by a user or administrator.
M : Job was moved to another server
Q : Job is queued, eligible to run or be routed
R : Job is running
S : Job is suspended by server. A job is put into the suspended state when a higher priority job needs the resources.
T : Job is in transition (being moved to a new location)
U : Job is suspended due to workstation becoming busy
W : Job is waiting for its requested execution time to be reached or job specified a stagein request which failed for some reason.
X : Subjobs only; subjob is finished (expired.)

Please refer to the PBSPro manual for more details:
http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf

E : Job is exiting after having run
F : Job is finished. Job has completed execution, job failed during execution, or job was deleted.
H : Job is held. A job is put into a held state by the server or by a user or administrator. A job stays in a held state until it is released by a user or administrator.
M : Job was moved to another server
Q : Job is queued, eligible to run or be routed
R : Job is running
S : Job is suspended by server. A job is put into the suspended state when a higher priority job needs the resources.
T : Job is in transition (being moved to a new location)
U : Job is suspended due to workstation becoming busy
W : Job is waiting for its requested execution time to be reached or job specified a stagein request which failed for some reason.
X : Subjobs only; subjob is finished (expired.)

Please refer to the PBSPro manual for more details:
http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf

How do I submit array jobs on ASPIRE1?

We recommend using loops like this:

#!/bin/bash
#PBS ...
for i in {1..10}; do
qsub -v PBS_ARRAY_INDEX=$i <job-script>
done

You could also run 24 single core jobs in parallel if they all use similar resources and will finish around the same time by using the following in the jobscript:

#!/bin/bash
#PBS -l select=1:ncpus=1, walltime=24:00:00
#PBS ...
for i in {1..24}; do
./run_my_program args ... &
donewait

Please note that there is a ‘&’ at the end of the command line, and the ‘wait’ for all background tasks to finish.

By default, any file is restricted to individual and the project/group. In case you need to let other users access your files , please contact the service desk (access https://servicedesk.nscc.sg/ or contact [email protected]). 

Please refer to our software list (https://help.nscc.sg/software-list/). Once you’re logged in to ASPIRE1, you can load the module of the application using the following commands:

$ module avail                                             # check on available modules

$ module load <application name>        # load application

Yes, you can install applications by yourself. Please feel free to install your intended application into your own home directory.

Yes, but in order to do so, you will need to have a project registered with NSCC. When requesting, please make sure to let us know your project ID. In general, we will ask you to install applications in your home directory by yourself, unless there are any dependencies that can only be solved by the system administrators.

You can use one of the file transfer protocols which supports copying over SSH (e.g. scp).Please note the speed of copy may depend on the network speed of your local environment.

The easiest way to transfer files from your organization HPC is to use rsync. For example, to transfer /scratch/johnsmit/project1 directory in your existing cluster to ASPIRE1’s /project/johnsmit/ directory, you can run the following command on your cluster:
$ rsync -arvz /scratch/johnsmit/project1 login.nscc.sg:/project/johnsmit/
*Please replace login with the respective login node mentioned in the table access methods

Please use one of the tools mentioned in the access methods table above, and connect to the right login node. If you intend to use rsync, you can use the below command on ASPIRE1.

$ rsync -arvz /<projectdir>/<project name>/<username>/<copy from> myorgnizationcluster:/<my destination directory>

Please use Filezilla software to use GUI for file transfer

Please refer to the Linux Tutorial.