nvprof Warning: The path to CUPTI and CUDA Injection libraries might not be set in LD_LIBRARY_PATH - nvidia

Question

I get the message in the subject when I try to run a program I developed with OpenACC through Nvidia's nvprof profiler like this:
nvprof ./SFS 4
If I run nvprof with -o [output_file] the warning message doesn't appear, but the output file is not created. What could be wrong here?
The LD_LIBRARY_PATH is set in my .bashrc to: /opt/nvidia/hpc_sdk/Linux_x86_64/20.7/cuda/11.0/lib64/ because there I have found these files there (they have "cupti" and "inj" in their names and I thought they are the needed ones):
lrwxrwxrwx 1 root root 19 Aug 4 05:27 libaccinj64.so -> libaccinj64.so.11.0
lrwxrwxrwx 1 root root 23 Aug 4 05:27 libaccinj64.so.11.0 -> libaccinj64.so.11.0.194
...
lrwxrwxrwx 1 root root 16 Aug 4 05:27 libcupti.so -> libcupti.so.11.0
lrwxrwxrwx 1 root root 20 Aug 4 05:27 libcupti.so.11.0 -> libcupti.so.2020.1.0
...
I am on Ubuntu 18.04. workstation with Nvidia GeForce RTX 2070, and have CUDA version 11 installed.
nvidia-smi command gives me this:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66 Driver Version: 450.66 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 Off | 00000000:02:00.0 On | N/A |
| 30% 40C P2 58W / 185W | 693MiB / 7981MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
The compilers I have (nvidia and portland) are from the latest Nvidia HPC-SDK, version 20.7-0
I compile my programs with -acc -Minfo=accel options, not sure how could I set -ta= and if it is needed at all?
P.S. I am also not sure if running my code, with or without nvprof uses GPUs at all, although I did set ACC_DEVICE_TYPE to nvidia.
Any advice would be very welcome.
Cheers

Answer 1

Which nvprof are you using? The one that ships with NV HPC 20.7 or your own install?
This looks very similar to an issue reported yesterday on the NVIDIA DevTalk user forums:
https://forums.developer.nvidia.com/t/new-20-7-version-where-is-the-detail-release-bugfix/146168/4
Granted this was for Nsight-systems, but it may be the same issue. It appears to be a problem with the 2020.3 version of the profilers which is the version we ship with the NV HPC 20.7 SDK. As I note, the Nsight-Systems 2020.4 release should have this fixed, so the work around would be download and install 2020.4 or use a prior release.
https://developer.nvidia.com/nsight-systems
There does seem to be a temporary issue with the Nsight-systems download that hopefully be corrected before you see this note.
Also, nvprof is in the process of being deprecated so you should consider moving to use Nsight-systems and Nsight-compute.
https://developer.nvidia.com/blog/migrating-nvidia-nsight-tools-nvvp-nvprof/