FAQs

ASPIRE1 is a supercomputer operated by the National Supercomputing Centre (NSCC), and built on x86 based architecture with initial benchmarks of 1.01 PFLOPS of compute throughput, 13 PetaBytes storage, and I/O performance of up to 500GBytes/sec. This supercomputer has 1,288 compute nodes with more than 30,000 cores, including 128 GPU nodes, and 9 large memory nodes ranging from 1TB to 6TB RAM. ASPIRE1 has login nodes in NUS, NTU and A*STAR extended with DTN-InfiniBand-longrange link.

ASPIRE1 is built on a CentOS Linux platform and currently does not have any Microsoft Windows OS.

PBS Compute Manager is a graphical user interface which facilitates users to run, monitor, and manage the jobs with ASPIRE1. The PBS Compute Manager can be accessed from here:
https://aspireweb.nscc.sg

PBS Display Manager is a graphical user interface which facilitates users to run graphics intensive applications in ASPIRE1. The PBS Display Manager can be accessed from here:

For A*STAR users – https://astar.nscc.sg

For NTU users – https://ntuweb.nscc.sg

For NUS users – https://nusweb.nscc.sg

For SUTD users – https://sutd.nscc.sg

For other users – https://aspireweb.nscc.sg

NSCC’s enrollment system is federated with SingAREN to facilitate users from NUS, NTU, SUTD, SMU and A*STAR to register seamlessly to ASPIRE1. Please follow the instructions presented in "Enrollment user guide" from "User Guide" page for the users from above mentioned organizations.

If you don’t belong to one of the organizations above, please contact [email protected] with the following details:

• User ID (preferably the same login ID used in your organization, limited to 8 characters):
• First name:
• Last name:
• E-mail address:
• Organization name:

*Note: Login ID creation is subjected to availability and approvals.

NUSEXT domain users are considered external users for NUS. Please contact [email protected] for registration to ASPIRE1.  

To utilize ASPIRE1 resources, you need to login to the cluster. Please follow the below table to determine how to access ASPIRE1.

Follow the below instructions to connect to ASPIRE1 through portal.

• Open the web browser and browse for the URL provided above.
• Use your ASPIRE1 credentials to access the web portal.

Below are instructions to connect using SSH/SCP on UNIX/Linux or Mac:

• Open any SSH terminal
• Type “ssh <userid>@<Host/FQDN>”
• Enter the password (characters are invisible while typing the password)
• Once logged in successfully you should be able to see a “$” prompt that allows you to execute Linux commands

If you want to use the X11 interface, replace the above ssh command with ssh -X.

For X11 – ssh -Y, make sure you have XQuartz installed for OS X 10.8 or higher.

Below are instructions to connect using SSH/SCP on Windows:

• Open any SSH terminal software (e.g. putty, MobaXterm)
• In case of putty, type the login host name and click on Open button (Putty may allow you to login automatically without asking for password. In case if putty asks for username/password, you need to type the login ID and password from your university.)
• In case of MobaXterm, type to command at the prompt “ssh [email protected]”
• Upon successful login, you will be prompted with “$” to use NSCC Supercomputer

Please follow the Password Reset guide and follow the step-by-step instructions.

Note that the password must contain:

• a minimum of 8 characters;
• a mixture of upper and lower cases;
• numbers (0-9); and
• at least one special character: [email protected]#$%^&*+=?><

Internet access from remote login nodes are disabled due to security policies at each entity. However, users will still be able to access files on the Internet using the following methods.

A*Star Users - login to astar.nscc.sg and download files from Internet

NUS users - ssh to nscc04-ib0 after logging in to nus.nscc.sg, and download files from Internet

NTU users - ssh to nscc04-ib0 after logging in to ntu.nscc.sg, and download files from Internet

Users accessing through aspire.nscc.sg will be able to access the Internet, once logged in.

NSCC password expires every 90 days. Upon expiry of the account you will not be able to login to NSCC system any more, the below message will be displayed when using the ssh connection. 

• Your account has expired; please contact your system administrator
• You are required to change your LDAP password immediately.
• Connection closed by xxx.xxx.xxx.xxx

In order to resolve the issue you need to login to https://user.nscc.sg/ with your organization credentials and click on set/reset your password. Once a new password is set you can login to NSCC with the new password. 

If your account is still not re-activated after resetting your password, please contact our help desk at [email protected]

The CPU resource in ASPIRE1 can be utilized through Personal Project.

The quota is fixed and 100,000.00 CPU hours is granted for every personal project for a period of ASPIRE1 life time. If the personal quota is exceeded, users can only submit jobs through a valid project. 

The Personal Project grant of CPU hours will not be extended. 

Please refer to the Project Portal User Guideline. 

1. Government agency submission
2. Industry submission
3. Group submission
4. Individual submission

Please note that only the requester will be able to see the project in the Project Portal.

Only the PI and project requester can transfer the ownership of a project.

Please email [email protected], copying the PI, previous owner and new owner of the project to request for a transfer.

Please send an email to [email protected] with the updated request. New requests outside the project allocation calls will be reviewed by the Resource Allocation Committee within 2 weeks to 3 months depending on the scale of the request.

1. All resources granted in the previous project cycle will expire by 30 June 2018.
2. On 1^st July 2018, all project quota will be set to zero (0) hours and you will not be able to dispatch any jobs. All project data will remain in the system for 31 days.
3. On 1^st August 2018, all project data will be archived and scratch disk data will be purged.
4. Charges will be incurred for retrieval of archived data from 1 August 2018.
5. All archived data will be retained until 31 Dec 2018.
6. Project allocation management:
   1. Plan your resource utilisation for your project from now until 30 June 2018.
   2. Use your allocated resources as early as possible to avoid encountering resource bottlenecks towards the end of the project cycle.
7. Please note that all project allocation grantees must acknowledge and cite NSCC for any published manuscript / presentations and submit proof of deliverables achieved from utilizing resources granted. All academic journal or conference papers shall cite NSCC with the following: “The computational work for this article was (fully/partially) performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).”

Projects applying for a renewal from previous project cycle:

1. Please indicate the project ID assigned previously under section 1.1 in the form.
2. Please submit deliverables achieved from previous project allocation(s) through the Deliverables link in the Project Portal.
3. Unutilised resources may be brought forward only for this project allocation cycle (1 July 2018 – 30 June 2019), subject to approval by the Resource Allocation Committee and depending on justified reasons for the unutilised resources. However, please note that resource request brought forward may be capped at unused allocations.
4. All renewals are subject to approval by the Resource Allocation Committee. 

Upon logging in you are informed of the amount of core hours remaining in the different projects you are a member of

Core hours remaining for project: Total Grant: 1000000.00
Total Used: 200000.00
Total Pending: 50000.00
Total Avail: 7500000.00

“Total Grant” correspond to the amount of resource that was granted to you upon project submission
“Total Used” correspond to the amount consumed so far
“Total pending” corresponds to the current amount requested : #running_jobs * per_job_core_requested * per_job_walltime_requested
“Total Avail” corresponds to “Total Grant” - “Total Used” - “Total Pending”

Be aware that once the (Used+Pending >= Granted), the subsequent jobs will be on-HOLD, and no new jobs will be allowed.
The new jobs will be allowed when it fit into the Avail, which is (Avail >= Granted-Used-Pending ).

For example, if you request 50 hours for your job but your job actually only needs 10 hours to complete, the AMS will hold back the requested 50 hours to itself (and mark them as “pending core hours”) until the 10 hour-job is completed. The remaining 40 hours will only be released and brought back to the user at that point of time. What is categorized as “pending core hours” will be deducted from your “available core hours”, so if you’re not requesting for the right amount of walltime in your job, you might be temporarily seeing less available core hours than you actually have.

When the user logged into NSCC login nodes, in the MOTD page, you can see the utilization of projects assigned to the user however in case if you wanted to see the usage at any point of time, please execute project_summary project ID command. For example, to view the project summary of the project 10000001, issue the command project_summary 10000001. The output should be similar to the below output:

Resource: CPU-hrs, System: aspire1
Total Grant: 7000000.00
Total Used: 51392.84
Total Pending: 7488.00
Total Avail: 6941119.16
Resource: CPU-hrs-budget, System: aspire1
Total Grant: Infinite

To know the disk usage, issue the command myquota -p project id

Project Quota for 10000001

===============================================================
|File System Usage (GB) Limit (GB) |
|nscc_home 8572.90 30000.00 |
===============================================================

You can check your current CPU usage by executing the “myusage” command.

$ myusage

Usage in 7 days to 2017/09/19:
Number of jobs on CPU nodes: 300
Core hours on CPU nodes: 81,592.3
Number of jobs on GPU nodes: 0
Core hours on GPU nodes: 0.0

Usage in month to 2017/08/31:
Number of jobs on CPU nodes: 131
Core hours on CPU nodes: 778,539.6
Number of jobs on GPU nodes: 0
Core hours on GPU nodes: 0.0

Usage since system start to 2017/08/31:
Number of jobs on CPU nodes: 6,368
Core hours on CPU nodes: 15,577,192.9
Number of jobs on GPU nodes: 0
Core hours on GPU nodes: 0.0

The disk quota allocated is limited as stated below respectively :

/home    - Home directory    - 50GB.
/scratch - Scratch directory - 100 TB

This will not be extended on any terms. If more disk space is needed, you should raise a project request through https://user.nscc.sg/project.

You can check your current disk usage on /home by executing the “myquota” command.

$ myquota

===============================================================
|File System               Usage (GB)               Limit (GB) |
|nscc_home                      30.10                    50.00 |
|/scratch                    43204.67                102400.00 |
===============================================================

Purging policy is implemented on scratch directory and files unused for more than 30 days will be purged automatically.  

File under home or project directories are untouched.

The word MPI stands for Message Passing Interface. One of the most used technologies to parallelize the applications. There are various implementation of MPI such as OpenMPI, MVAPICH, IntelMPI, etc.,
MPI Interface is a wrapper to C/Fortran compiler, which means that all the codes which are used in C/Fortran can be used with MPI in addition to penalization techniques.
ASPIRE1 is powered with IntelMPI for better performance of applications.

One of the techniques to run your code in parallel is OpenMP. Codes using OpenMP are normally restricted to run in one physical server, unless you are running a hybrid code made out of MPI and OpenMP. OpenMP codes themselves are not capable of communicating through a network infrastructure such as Infiniband. OpenMP code is simple to use and does not need any wrappers, standard compilers like GCC/Gfortran or Intel C/Fortran can use OpenMP. However, the program must have OpenMP directives.

A serial application is an application which can use only one process at a time to perform calculations, while a parallel application can scale to multiple processes and make use of cores/CPUs in one or multiple servers using high speed Interconnects. For this reason, parallel applications produce faster results in general.
To promote the HPC experience, it is strongly recommended to use parallel codes on ASPIRE1 if applicable.

Checkpoint is a technique used to save the output periodically during a runtime of an application. The advantage using checkpoint is that it is possible to restart the program from the last checkpoint, in case if the execution of program terminates abruptly due to any reason.
You are advised to use the checkpoint and restart technique while writing your application so that in case of eventualities, you should be able to restart the job from the last checkpoint.

The available compilers are as follows:

• gcc (v4.9 & v4.4)
• Intel (v17.0 & v16.1)
• NAG (???)

The applications, Libraries, and compilers are very dynamic and powered with Environment modules in NSCC Supercomputer, to list the available modules, use “"module avail"” from at the prompt in NSCC Supercomputer login nodes.

NSCC does not support compilers other than the versions mentioned above at the moment. If you’re keen to use other compilers, you are more than welcome to install it into your own home directory. Please contact our service desk (access https://servicedesk.nscc.sg/ or contact [email protected]) if you face any difficulties during the installation.

Yes, you can request for additional libraries or applications through the Service Desk portal. However the installation of libraries is subjected to various conditions such as compatibility, the time required to make the library available, dependencies and our software policies. Please contact Service Desk for further clarification. Our Technical specialist will respond to make the most of your request.

I want to use a commercial application. How can I use it on ASPIRE1?

Commercial applications can be installed in ASPIRE1, but they need to be installed and used following the licensing terms of each software application. First of all, the license and the software modules of the intended application need to be provided by you to NSCC for installation. Based on the license terms and conditions, we will determine whether and how the software should be installed in ASPIRE1. The same rule applies to academic licenses.

Please note that the accountability on the usage of the software should be held by the owner of the license. For more information, please contact our service desk.

Yes. Jupyter notebook is not in the default supported applications, but it is available with anaconda package. Follow the instructions below to use the Jupyter notebook.

1. Login to display manager (Please see the display manager instructions provided under user guides)
2. Double click on GlxSpheres app to launch the application 
3. Close the GlxSpheres by clicking on "x" on top right corner
4. Open terminal through Menu -> Tools -> Terminal
5. Execute the below commands at the terminal 
   module load anaconda/3
   export PYTHONHOME=/app/anaconda/3
   export PATH=/app/firefox:$PATH
6. Open jupyter notebook with the command "jupyter notebook"

There are many reasons jobs may be prevented from starting. The first thing to do is to run “qstat -s ”; this will print the comments from the job scheduler about your job.

If you see a “–”"Q" in the column "S", it means the scheduler has not yet executed your job. Be patient.

There could be several reasons for this, few common reasons listed below:

1. The operating system that you are using is different from that in ASPIRE1
   Solution:
   • if you are running in your local Linux machine, a simple recompilation on the ASPIRE1 login nodes will solve the issue.
   • if you are running in Windows PC and you want to run inASPIRE1, you need to either obtain a copy of the software for Linux or port the code from Windows to Linux.
2. Different compiler/library stack: if  your job is not running although your operating system in your own environment is the same as ASPIRE1, it means the compiler/libraries are not compatible. You need to recompile your codes on ASPIRE1 login nodes.

Please contact our service desk (access https://servicedesk.nscc.sg/ or contact [email protected]) if you still have any issues with running your job on ASPIRE1.

This error could be because you have forgotten to source the appropriate system .rc in your personal .rc file.

• If you are using an sh-derived shell for your jobs, edit the .bashrc file to ensure it contains the line “. /etc/bashrc”.
• If you are using a csh-derived shell for your jobs, edit the .cshrc file to ensure it contains the line “source /etc/csh.cshrc”.

If you have accidentally deleted the .bashrc file, you can copy back the original .bashrc file from /etc/skel/.bashrc .

Example:

$ cp /etc/skel/.bash* ~/

If your batch job is named, runjob.sh and your output is not redirected in the batch script, then your job output will appear in runjob.sh.o****, where the digits at the end of the file name represent your job ID. The final entries in the .o file give you the details on wall time and virtual memory used by the job.

Similarly, error messages of your jobs will be recorded in runjob.sh.e****, where the digits at the end of the file name represent your job ID.

In a PBS job script, the memory you specify using the -lmem= option is the total memory across all nodes. However, this value is internally converted into the per-node equivalent, and this is how it is monitored.
For example, since NSCC Supercomputer has 24 cores per node, if you request -l slect=2:ncpus=24, mem=10GB, the actual limit will be 10GB on each of the two nodes. If you exceed this on either of the nodes, your job will be killed.
Please note, if a job runs for less than a few minutes, the memory use reported in your .o file once the job completes may be inaccurate. We strongly discourage people running short jobs of, e.g. less than 1 hour. This is because there is significant overhead in setting up and tearing down a job and you may end up wasting large amounts of your grant. Instead, if you have many short jobs to run, considering merging them together into a few longer jobs

Please note that you are not supposed to run any job on the Login nodes. We will automatically kill if we see any user running a job on the login node. If you want to run an interactive session, please submit and interactive job into the compute nodes using the qsub command.

$qsub -I

The queues available for job submission are :

• normal
• gpu
• largemem

There are several queues in the system with different priorities which works for different workloads. Below are some of the queues in the system:

However these queues does not accept jobs directly rather jobs will be routed once submitted to normal/gpu queue. 

Please raise a project request through https://user.nscc.sg/project. Please note that all project requests are subject to approval.

E : Job is exiting after having run
F : Job is finished. Job has completed execution, job failed during execution, or job was deleted.
H : Job is held. A job is put into a held state by the server or by a user or administrator. A job stays in a held state until it is released by a user or administrator.
M : Job was moved to another server
Q : Job is queued, eligible to run or be routed
R : Job is running
S : Job is suspended by server. A job is put into the suspended state when a higher priority job needs the resources.
T : Job is in transition (being moved to a new location)
U : Job is suspended due to workstation becoming busy
W : Job is waiting for its requested execution time to be reached or job specified a stagein request which failed for some reason.
X : Subjobs only; subjob is finished (expired.)

Please refer to the PBSPro manual for more details:
http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf

E : Job is exiting after having run
F : Job is finished. Job has completed execution, job failed during execution, or job was deleted.
H : Job is held. A job is put into a held state by the server or by a user or administrator. A job stays in a held state until it is released by a user or administrator.
M : Job was moved to another server
Q : Job is queued, eligible to run or be routed
R : Job is running
S : Job is suspended by server. A job is put into the suspended state when a higher priority job needs the resources.
T : Job is in transition (being moved to a new location)
U : Job is suspended due to workstation becoming busy
W : Job is waiting for its requested execution time to be reached or job specified a stagein request which failed for some reason.
X : Subjobs only; subjob is finished (expired.)

Please refer to the PBSPro manual for more details:
http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf

How do I submit array jobs on ASPIRE1?

We recommend using loops like this:

#!/bin/bash
#PBS ...
for i in {1..10}; do
qsub -v PBS_ARRAY_INDEX=$i <job-script>
done

You could also run 24 single core jobs in parallel if they all use similar resources and will finish around the same time by using the following in the jobscript:

#!/bin/bash
#PBS -l select=1:ncpus=1, walltime=24:00:00
#PBS ...
for i in {1..24}; do
./run_my_program args ... &
donewait

Please note that there is a ‘&’ at the end of the command line, and the ‘wait’ for all background tasks to finish.

By default, any file is restricted to individual and the project/group. In case you need to let other users access your files , please contact the service desk (access https://servicedesk.nscc.sg/ or contact [email protected]). 

Please refer to our software list (https://help.nscc.sg/software-list/). Once you’re logged in to ASPIRE1, you can load the module of the application using the following commands:

$ module avail                                             # check on available modules

$ module load <application name>        # load application

Yes, you can install applications by yourself. Please feel free to install your intended application into your own home directory.

Yes, but in order to do so, you will need to have a project registered with NSCC. When requesting, please make sure to let us know your project ID. In general, we will ask you to install applications in your home directory by yourself, unless there are any dependencies that can only be solved by the system administrators.

You can use one of the file transfer protocols which supports copying over SSH (e.g. scp).Please note the speed of copy may depend on the network speed of your local environment.

The easiest way to transfer files from your organization HPC is to use rsync. For example, to transfer /scratch/johnsmit/project1 directory in your existing cluster to ASPIRE1’s /project/johnsmit/ directory, you can run the following command on your cluster:
$ rsync -arvz /path/to/the/files <login>.nscc.sg:/project/johnsmit/
*Please replace login with the respective login node mentioned in the table access methods

From PC: 

For Microsoft Windows/Mac the preferred/suggested method is to use FileZilla or any other file transfer utility to transfer your local files to desired location in NSCC. 

From any Linux machine/Mac terminal, you can either use scp or rsync command to copy files from your local pc to NSCC. The simple syntax for rsync should be:

<localpc>: rsync -arvz /path/to/files <login>.nscc.sg:/destination/path/in/nscc

From a cluster in your organization:

This is tricky, and unless your organization cluster is published in internet, there's no straight forward method. But if your cluster is in A*Star extranet, this can be possible. Please try the below syntax:

<your cluster>: rsync -arvz /path/to/files astar.nscc.sg:/destination/path/in/nscc

For all other users, apologies, you need to copy your files to your PC and transfer back to NSCC. 

Please use Filezilla software to use GUI for file transfer

Please refer to the Linux Tutorial.