-
Notifications
You must be signed in to change notification settings - Fork 3
/
notes
104 lines (69 loc) · 4.15 KB
/
notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
for coreneuron
mkdir coredat
mkdir modcore
cp mod/*.mod modcore
rm modcore/ri.mod
coreneuron nrniv/main1.cpp needs
nrn_finitialize( 0, input_params.voltage );
several changes to nrntraub can be reverted when global variables
are properly handled. ie. secondorder, v_init, cad.mod:ceiling
build coreneuron using (viz cluster for gpu) (note -DADDITIONAL_MECHPATH
and -DADDITIONAL_MECHS) Basic for normal machine is
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/bb/install -DENABLE_OPENACC=OFF
cmake .. -DADDITIONAL_MECHPATH=/gpfs/bbp.cscs.ch/home/hines/models/nrntraub/modcore -DCMAKE_INSTALL_PREFIX=/gpfs/bbp.cscs.ch/home/hines/bbgpu/install_viz -DCMAKE_C_FLAGS:STRING="-I/gpfs/bbp.cscs.ch/apps/viz/tools/pgi/15.10/linux86-64/2015/include -acc -Minfo=acc -Minline=size:200,levels:10 -O3 -DSWAP_ENDIAN_DISABLE_ASM -DLAYOUT=0 -DDISABLE_HOC_EXP" -DCMAKE_CXX_FLAGS:STRING="-I/gpfs/bbp.cscs.ch/apps/viz/tools/pgi/15.10/linux86-64/2015/include -acc -Minfo=acc -Minline=size:200,levels:10 -O3 -DSWAP_ENDIAN_DISABLE_ASM -DLAYOUT=0 -DDISABLE_HOC_EXP" -DCOMPILE_LIBRARY_TYPE=STATIC
rather:
corenrnbuild -m modcore -s $HOME/bb/coreneuron -i $HOME/bb/install -b buildgpu
nrnivmodl mod
export HOC_LIBRARY_PATH=`pwd`/hoc
nrniv -c mytstop=100 temp.hoc
#with command line options
mpirun -n 3 ./x86_64/special -c mytstop=1 temp.hoc -mpi
mpirun -n 1 coreneuron_exec -mpi -d coredat --gpu -e 100 --voltage=1000 --cell_permute=2 --nwarp=10
#mpirun is needed to get solve time
sh compare.sh out1.dat out0.dat|more
#Time profiling
module load cuda/6.0
mpirun -n 1 nvprof coreneuron_exec -mpi -d coredat -e 1 --cell_permute=2 --nwarp=10 --voltage=1000 --gpu
#CUDA api call history/timeline
mpirun -n 1 nvprof --print-api-trace coreneuron_exec -mpi -d coredat -e 0.1 --cell_permute=2 --nwarp=10 --voltage=1000 --gpu &> log
#Timeline view with NVVP
mpirun -n 1 nvprof -o traub.nvprof coreneuron_exec -mpi -d coredat -e 0.1 --cell_permute=2 --nwarp=10 --voltage=1000 --gpu
nvvp traub.nvprof
#Selective profiling of only psolve, add below cmake option
-DENABLE_SELECTIVE_GPU_PROFILING=ON
#CPU in case
mpirun -n 1 pgprof coreneuron_exec -mpi -d coredat -e 1 --cell_permute=1 --nwarp=10 --voltage=1000
#METRICS
nvprof --devices 0 --query-metrics
#EVENTS
nvprof --devices 0 --query-events > events
mpirun -n 1 nvprof --devices 0 --events warps_launched coreneuron_exec -mpi -d coredat -e 1 --cell_permute=2 --nwarp=10 --voltage=1000 --gpu
#selective profiling
module load cuda/6.0
add options to enable selective profiling psolve =>
-DCUDA_HOST_COMPILER=`which gcc` -DCUDA_PROPAGATE_HOST_FLAGS=OFF
To disable selective profiling, add cmake option :
-DENABLE_SELECTIVE_GPU_PROFILING=OFF
https://bbpteam.epfl.ch/project/spaces/pages/viewpage.action?spaceKey=BBPHPC&title=CoreNeuron+GPU+Development
On office machine, need mpich
export PATH=/opt/pgi/linux86-64/2016/bin:$HOME/soft/mpich/bin:$PATH
Discover maximum difference between two prcellstate files
python pcsdiff.py 242_cpu_t0.025.corenrn 242_acc_gpu_t0.025.corenrn
Build so that coreneuron prcellstate files are conistent with nrniv
(proper values for _v_unused and _g_unused)
corenrnbuild -m modcore -s $HOME/bb/coreneuron -i $HOME/bb/install \
-b buildgpu -cflags DNRN_PRCELLSTATE=1
Example run
mpirun -n 1 buildgpu/bin/coreneuron_exec -mpi -d coredat -e 0.025 \
--prcellgid 242 --celsius=6.3 --voltage=1000 --cell_permute=0 --gpu
[10:43:59 AM] Pramod Kumbhar: PGI_ACC_SYNCHRONOUS=1
[11:01:00 AM] Pramod Kumbhar: 2. race condition
[11:01:02 AM] Pramod Kumbhar: cuda-memcheck --tool racecheck [memcheck_options
app_name [app_options]
[11:02:56 AM] Pramod Kumbhar: http://docs.nvidia.com/cuda/cuda-memcheck/#axzz4F3VVLp8w
[11:06:03 AM] Pramod Kumbhar: 3. Unified memory : add compiler flag => -ta=tesla:managed
[11:06:21 AM] Pramod Kumbhar: here is the article: https://www.pgroup.com/lit/articles/insider/v6n2a4.htm
[11:16:10 AM] Pramod Kumbhar: https://www.pgroup.com/support/download_pgi2015.php?view=current
[8:05:00 AM] Pramod Kumbhar: https://bbpteam.epfl.ch/project/issues/login.jsp?permissionViolation=true&os_destination=%2Fbrowse%2FCNEUR-206&page_caps=&user_role=
Saturday, July 23, 2016
[11:08:54 AM] Pramod Kumbhar: https://www.pgroup.com/lit/articles/insider/v6n2a2.htm