Having Trouble using two Nvidia Jetson nano developer kits using Chapel - nvidia

Question

I'm using Chapel for a project in my class and I am trying to do multilocale executution with my two Nvidia Jetson nano boards.
Following this tutorial https://chapel-lang.org/docs/usingchapel/multilocale.html#readme-multilocale.
I keep getting errors when I try to run the hello executable.
I run the code by doing "./hello -nl 2"
Here is the error message I keep getting
*** GASNET WARNING(Node 0): int sendPacket(ep_t, amudp_msg_t*, size_t, en_t, packet_type) returning an error code: AM_ERR_RESOURCE (Problem with requested resource)
from function sendPacket
at /home/chico/chapel-1.20.0/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:112
reason: Invalid argument
*** GASNET WARNING(Node 0): int AMUDP_RequestGeneric(amudp_category_t, ep_t, amudp_node_t, handler_t, void*, size_t, uintptr_t, int, va_list, uint8_t, uint8_t) returning an error code: AM_ERR_RESOURCE (Problem with requested resource)
at /home/chico/chapel-1.20.0/third-party/gasnet/gasnet-src/other/amudp/amudp_reqrep.cpp:1045
GASNet gasnetc_AMRequestShort encountered an AM Error: AM_ERR_RESOURCE(3)
at /home/chico/chapel-1.20.0/third-party/gasnet/gasnet-src/udp-conduit/gasnet_core.c:827
*** WARNING (proc 0): GASNet gasnetc_AMRequestShort returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
at /home/chico/chapel-1.20.0/third-party/gasnet/gasnet-src/udp-conduit/gasnet_core.c:829
*** FATAL ERROR(Node 1): An active message was returned to sender,
and trapped by the default returned message handler (handler 0):
Error Code: ECONGESTION: Congestion at destination endpoint
Message type: AM_REQUEST_M
Destination: (127.0.0.1:52171) (0)
Handler: 64
Tag: 0x7f0001010000629e
Arguments(5): 0x00000000 0x00000001 0x00000000 0x00000000 0x00000009
Aborting...
*** Caught a fatal signal (proc 1): SIGABRT(6)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace.
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
bash: line 1: 1658 Aborted (core dumped) env 'AMUDP_SLAVE_ARGS=1,JetsonNano:52561,' './hello_real' '-nl' '2' '-E' 'LD_LIBRARY_PATH=:/usr/local/cuda/lib64' '-E' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' '-E' 'SSH_CONNECTION=132.241.216.227 8794 192.168.1.20 22' '-E' 'LESSCLOSE=/usr/bin/lesspipe %s %s' '-E' 'LANG=en_US.UTF-8' '-E' 'CHPL_REGEXP=none' '-E' 'OLDPWD=/home/chico/chapel-1.20.0' '-E' 'CHPL_GMP=none' '-E' 'LLVM_CONFIG=/usr/bin/llvm-config-7' '-E' 'XDG_SESSION_ID=132' '-E' 'USER=chico' '-E' 'PWD=/home/chico' '-E' 'HOME=/home/chico' '-E' 'SSH_CLIENT=132.241.216.227 8794 22' '-E' 'CHPL_COMM=gasnet' '-E' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' '-E' 'CHPL_MEM=cstdlib' '-E' 'GASNET_SSH_SERVERS=JetsonNano JetsonNano2' '-E' 'SPARK_HOME=/opt/spark' '-E' 'SSH_TTY=/dev/pts/4' '-E' 'MAIL=/var/mail/chico' '-E' 'TERM=xterm-256color' '-E' 'SHELL=/bin/bash' '-E' 'CHPL_TASKS=fifo' '-E' 'CHPL_LLVM=none' '-E' 'SHLVL=1' '-E' 'GASNET_SPAWNFN=S' '-E' 'MANPATH=/home/chico/chapel-1.20.0/man:' '-E' 'CHPL_HOME=/home/chico/chapel-1.20.0' '-E' 'LOGNAME=chico' '-E' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus' '-E' 'XDG_RUNTIME_DIR=/run/user/1001' '-E' 'PATH=/home/chico/.cargo/bin:/home/chico/.local/bin:/home/chico/chapel-1.20.0/bin/linux64-aarch64:/home/chico/chapel-1.20.0/util:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda-10.0/bin/:/opt/spark/bin:/opt/spark/sbin' '-E' 'LESSOPEN=| /usr/bin/lesspipe %s' '-E' '_=./hello'
It works if I only use the localhost.
Note: I have been using mpy4py with python across my two boards and it works. Also, I'm using an ansible playbook to sync both boards across the network.

Answer 1

The symptoms reported here relate to a failure occurring during GASNet initialization in which communication fails to be established between the multiple processes that implement the Chapel program. Unfortunately, there are a number of factors that could cause this sort of failure mode, such as incorrect /etc/hosts entries, firewalls preventing communication, etc. Users running into this sort of issue should feel free to open up an issue on the Chapel GitHub issues page to try and resolve it.
To see the discussion and resolution of the problem in this specific case, readers may wish to refer to Chapel issue #14581 and specifically this comment.