{"id":1002,"date":"2018-10-22T13:23:41","date_gmt":"2018-10-22T05:23:41","guid":{"rendered":"http:\/\/help.nscc.sg\/?page_id=1002"},"modified":"2018-11-20T12:00:23","modified_gmt":"2018-11-20T04:00:23","slug":"ime-user-guide-for-nscc","status":"publish","type":"page","link":"https:\/\/help-2626.nscc.sg\/ime-user-guide-for-nscc\/","title":{"rendered":"IME User Guide for NSCC"},"content":{"rendered":"\r\n<p class=\"has-medium-font-size\" style=\"text-align: center;\"><strong>I\/O workloads acceleration at NSCC with IME<\/strong><\/p>\r\n\r\n\r\n\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"906\" height=\"546\" class=\"wp-image-1004\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image.png 906w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-300x181.png 300w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-768x463.png 768w\" sizes=\"(max-width: 906px) 100vw, 906px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"944\" height=\"537\" class=\"wp-image-1005\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-1.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-1.png 944w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-1-300x171.png 300w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-1-768x437.png 768w\" sizes=\"(max-width: 944px) 100vw, 944px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"913\" height=\"464\" class=\"wp-image-1007\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-2.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-2.png 913w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-2-300x152.png 300w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-2-768x390.png 768w\" sizes=\"(max-width: 913px) 100vw, 913px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"674\" height=\"486\" class=\"wp-image-1008\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-3.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-3.png 674w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-3-300x216.png 300w\" sizes=\"(max-width: 674px) 100vw, 674px\" \/><\/figure>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>FAST ACCESS: JUST USE MOUNTPOINT \/ime<\/strong><\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>IME is mounted on all compute node: just store your files in \/ime<\/strong><br \/><br \/><strong>[ddnsupport@mon02 IO500]$ df -h<\/strong><br \/>Filesystem \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0 Size\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0Used Avail \u00a0Use%\u00a0\u00a0\u00a0\u00a0\u00a0 Mounted on<br \/>\/dev\/sda3                                             273G\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0216G\u00a0 44G\u00a0\u00a0 84% \u00a0\u00a0\u00a0\u00a0\u00a0 \/<br \/>tmpfs \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0                                          63G\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 136K\u00a0 63G\u00a0\u00a0 1% \u00a0\u00a0\u00a0\u00a0\u00a0  \/dev\/shm<br \/>\/dev\/sda\u00a0\u00a0\u00a0\u00a0                                          1488M      33M\u00a0\u00a0 431M\u00a0 7% \u00a0\u00a0     \/boot<br \/>192.168.156.29@o2ib,192.168.156.30@o2ib:\/scratch\u00a0\u00a0\u00a0\u00a0\u00a0 2.8P\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a02.2P\u00a0 664T\u00a0 77%\u00a0      \/scratch <br \/>192.168.156.29@o2ib,192.168.156.30@o2ib:\/seq\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1.2P\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0892T  240T\u00a0 79%\u00a0      \/seq <br \/>192.168.160.104:\/home\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 3.4P\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a02.7P\u00a0 775T\u00a0 78%\u00a0      \/home <br \/>192.168.160.101:\/data\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 5.3P\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a03.9P\u00a0 1.4P\u00a0 74%\u00a0      \/data <br \/><strong>imefs\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2.8P\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a02.2P\u00a0 664T\u00a0 77%\u00a0      \/ime<\/strong><br \/><strong>\u00a0<\/strong><br \/>%ls \/ime\/users<br \/>academy\u00a0adm\u00a0 astar\u00a0 create\u00a0gov\u00a0 hackathon\u00a0 industry\u00a0ntu\u00a0 nus\u00a0 smu\u00a0sutd<\/pre>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"931\" height=\"458\" class=\"wp-image-1011\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-5.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-5.png 931w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-5-300x148.png 300w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-5-768x378.png 768w\" sizes=\"(max-width: 931px) 100vw, 931px\" \/><\/figure>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>FAST ACCESS: examples<\/strong><\/p>\r\n\r\n\r\n\r\n<p>(Example only, change to relevant directory as per your home directory path. For example if your home directory is \/home\/users\/nscc\/user1 your relative path in ime willl be \/ime\/users\/nscc\/user1)<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>% cd \/ime\/users\/industry<\/strong><br \/><strong>% my_big_application &gt;result.out<\/strong><br \/>Or change the file name in your code \/ configurationfile from <strong>\/scratch\/users\/industry\/&lt;my_dir&gt;<\/strong><br \/>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 to <br \/><strong>\/ime\/users\/industry\/&lt;my_dir&gt;<\/strong><br \/><strong>From to 2 to 5 GB\/s per clientnode<\/strong><\/pre>\r\n\r\n\r\n\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"722\" height=\"410\" class=\"wp-image-1012\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-6.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-6.png 722w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-6-300x170.png 300w\" sizes=\"(max-width: 722px) 100vw, 722px\" \/><\/figure>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>FASTER ACCESS: USE DDN MPI <\/strong><\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>% MPI_DDN_ROOT=\/opt\/ddn\/mvapich<\/strong><br \/><strong>% export LD_LIBRARY_PATH=$MPI_DDN_ROOT\/lib64 <\/strong><br \/><strong>% export LD_LIBRARY_PATH+=:\/usr\/local\/lib<\/strong><br \/><strong>% mpirun -genv MV2_NUM_HCAS 1 -genvMV2_CPU_BINDING_LEVEL core -genv\u00a0MV2_CPU_BINDING_POLICY scatter -ppn 12\/home\/adm\/sup\/ddnsupport\/jacquaviva\/mdtest -r -t -F -w 3901 -e 3901 -d \/ime\/ddnsupport\/IO500\/datafiles\/mdt_hard -n 5000<\/strong><br \/><strong>\u00a0<\/strong><br \/>\u00a0<br \/><strong># Up to 10 GB\/s per client node<\/strong><\/pre>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>Better integration: native support of IME in major MPI distros<\/strong><\/p>\r\n\r\n\r\n\r\n<ul>\r\n<li><strong>Patch merged inMaster tree of OpenMPI<\/strong>\r\n<ul>\r\n<li>Expected to be part of 3.1.1 release<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul>\r\n<li><strong>Patch merged in Master tree of Mvapich<\/strong>\r\n<ul>\r\n<li>Expected to be part of v3.3b3 release<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1052\" height=\"611\" class=\"wp-image-1046\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-10.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-10.png 1052w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-10-300x174.png 300w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-10-768x446.png 768w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-10-1024x595.png 1024w\" sizes=\"(max-width: 1052px) 100vw, 1052px\" \/><\/figure>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"757\" height=\"629\" class=\"wp-image-1048\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-12.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-12.png 757w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-12-300x249.png 300w\" sizes=\"(max-width: 757px) 100vw, 757px\" \/><\/figure>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"554\" height=\"656\" class=\"wp-image-1049\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-13.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-13.png 554w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-13-253x300.png 253w\" sizes=\"(max-width: 554px) 100vw, 554px\" \/><\/figure>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"751\" height=\"624\" class=\"wp-image-1050\" src=\"https:\/\/help.nscc.sg\/wp-content\/uploads\/image-14.png\" alt=\"\" srcset=\"https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-14.png 751w, https:\/\/help-2626.nscc.sg\/wp-content\/uploads\/image-14-300x249.png 300w\" sizes=\"(max-width: 751px) 100vw, 751px\" \/><\/figure>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>Integration with PBS: pre-staging data<\/strong><\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">#!\/bin\/sh<br \/>#PBS-l walltime=3:30:00<br \/>#PBS-l select=32:ncpus=24:mpiprocs=24<br \/>#PBS-N io500_large<br \/>#PBS-o io500_large.o<br \/>#PBS-e io500_large.err<br \/>MPI_DDN_ROOT=\/opt\/ddn\/mvapich<br \/>exportLD_LIBRARY_PATH=$MPI_DDN_ROOT\/lib64 <br \/>exportLD_LIBRARY_PATH+=:\/usr\/local\/lib<br \/><strong>\/opt\/ddn\/ime\/bin\/ime-prestage -bime:\/\/\/scratch\/ime\/&lt;my_dir&gt;\/&lt;my_file&gt;<\/strong><br \/>\/home\/adm\/sup\/ddnsupport\/jacquaviva\/IO500\/io-500-dev\/io500.sh<strong>\u00a0 <\/strong>\u00a0<br \/>echo\"qstat -f ${PBS_JOBID}\"<br \/>echo\" \"<br \/>qstat-f ${PBS_JOBID}<strong><br \/><\/strong><\/pre>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>Integration with PBS: Staging-out<\/strong><\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">#!\/bin\/sh<br \/>#PBS -l walltime=3:30:00<br \/>#PBS -lselect=32:ncpus=24:mpiprocs=24<br \/>#PBS -N io500_large<br \/>#PBS -o io500_large.o<br \/>#PBS -e io500_large.err<br \/>MPI_DDN_ROOT=\/opt\/ddn\/mvapich<br \/>exportLD_LIBRARY_PATH=$MPI_DDN_ROOT\/lib64 <br \/>export LD_LIBRARY_PATH+=:\/usr\/local\/lib<br \/>\/opt\/ddn\/ime\/bin\/ime-prestage-b ime:\/\/\/scratch\/ime\/&lt;my_dir&gt;\/&lt;my_file&gt;<br \/>\/home\/adm\/sup\/ddnsupport\/jacquaviva\/IO500\/io-500-dev\/io500.sh<br \/><strong>\/opt\/ddn\/ime\/bin\/ime-sync-b ime:\/\/\/scratch\/ime\/&lt;my_dir&gt;\/&lt;my_file&gt; <\/strong><br \/><strong>cp\/scratch\/ime\/&lt;my_dir&gt;\/&lt;my_file&gt; \/home\/&lt;my_dir&gt;\/result <\/strong><br \/>echo \"qstat -f${PBS_JOBID}\"<br \/>echo \" \"<br \/>qstat -f ${PBS_JOBID}<\/pre>\r\n\r\n\r\n\r\n<p style=\"text-align: center;\"><strong>Template PBS job scripts<\/strong><\/p>\r\n\r\n<p>Example PBS job scripts can also be found on the system in the directory:<\/p>\r\n<!-- \/wp:paragraph -->\r\n\r\n<!-- wp:preformatted -->\r\n<pre class=\"wp-block-preformatted\">\/app\/examples\/ime<\/pre>\r\n<!-- \/wp:preformatted -->\r\n\r\n<!-- wp:paragraph -->\r\n<p>&nbsp;<\/p>\r\n<!-- \/wp:paragraph -->\r\n\r\n<!-- wp:paragraph -->\r\n<p><strong>End of Document<\/strong><\/p>\r\n<!-- \/wp:paragraph -->","protected":false},"excerpt":{"rendered":"<p>I\/O workloads acceleration at NSCC with IME &nbsp; &nbsp; &nbsp; &nbsp; FAST ACCESS: JUST USE MOUNTPOINT \/ime IME is mounted on all compute node: just store your files in \/ime[ddnsupport@mon02\u2026 <a href=\"https:\/\/help-2626.nscc.sg\/ime-user-guide-for-nscc\/\" class=\"read-more-link\">read more &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"_links":{"self":[{"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/pages\/1002"}],"collection":[{"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/comments?post=1002"}],"version-history":[{"count":34,"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/pages\/1002\/revisions"}],"predecessor-version":[{"id":1259,"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/pages\/1002\/revisions\/1259"}],"wp:attachment":[{"href":"https:\/\/help-2626.nscc.sg\/wp-json\/wp\/v2\/media?parent=1002"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}