Ask questions about the codebase here

[s66104444]

[mtijanic] Please use this issue to ask simple questions about the codebase, such as what a certain feature is, how a given codepath works, what a given acronym means, etc. Please keep your questions as specific as possible.

-- Original user post below this line --

I see a lot of RM in the code related to the RMAPI module, what does this RM and GSP-RM mean exactly?

[mtijanic]

Hello and thank you for your interest!

RM stands for "Resource Manager" and it is the internal name of the driver that gets compiled into nvidia.ko module - both the open source one (sometimes referred to as "OpenRM" internally) and the proprietary version. It can also refer to the team working on this code.

Depending on the context, GSP-RM can refer either to the new driver architecture that uses the "GSP" microcontroller, or it can refer to the code running on said microcontroller. To avoid ambiguity, the latter is often called "Physical RM" instead, and code running in kernel is called "Kernel RM".

RMAPI is the interface nvidia.ko (RM) exposes to other kernel modules and to userspace.

[mtijanic]

Meta: Do you think it makes sense to group these questions into a single issue, which we could then pin?

[s66104444]

Thanks! Yes, I think it's a good idea to group these questions into a single issue.

[mtijanic]

Okay, this is now a pinned generic code-related Q&A issue.

[antdking]

Would it be possible to enable Github Discussions, which has a cleaner structure for quick questions?

[mtijanic]

@antdking Please see #44 . I do agree Discussions would be a much cleaner option, and I expect we'll enable them soon enough. Will gladly retire this issue when that happens, but until such a time this is the best we have for these quick one-off questions.

[DemiMarie]

Will this driver support GPU virtualization with untrusted guests on consumer hardware? This would be a huge win for Qubes OS, which is currently forced to rely on software rendering. Support for CUDA or for enterprise GPUs would not be as helpful, as Qubes OS is an end-user operating system focused on the desktop use-case.

[mtijanic]

Hello @DemiMarie , thanks for the interest. The currently published driver does not support virtualization, neither as a host nor a guest. I currently don't have any roadmap information to share regarding this.

[mtijanic]

@DemiMarie At this point all I can say is that the current codebase does not support virtualization - neither as a host nor a guest. I don't currently have any information about future changes in this regard.

[aritger]

We've now enabled Discussions in this github repository. I've moved this issue to Discussions, though maybe we should close this and start a separate discussion for each question...

[RealAstolfo]

As many of us have noticed so far. NVIDIA has defaulted to one of the age old standards of C90. Moving forward, would NVIDIA be open to adopting more modern standards such as C11? or is there a very specific reason as to why we must stay C90?

My argument for adopting the newer standard would be the newer features, most prominently seen in C++ via operators, templates, and constexpr. constexpr in particular leads to less hardcoded variables in favor of a compile time resolved one.

What do you think?

[johnnynunez]

It would be important for the standard to be upgraded to c11.
https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.18-C11-Plan

[ElijahPepe]

At least it's not C89.

I'll submit a draft PR for this. I'm sure NVIDIA knows about C11 though; it's probably because they need these kernel modules to be able to be developed on any platform.

On second thought, C11 doesn't have many killer features. C99 should be the next logical step from C90, because the difference between C90 and C99 is huge and would be most desirable for those aforementioned compatibility considerations and for modern features.

[mtijanic]

Thank you for wanting to bring us into the 21st century! Unfortunately, much of the code published here is shared by various other projects and is compiled for a wide variety of platforms - from Windows to custom NV-internal ISAs and OSes - which means it must support a wide variety of toolchains and their quirks.

Generally, this means that we need to stick with C90. Many (but not all) C99 features are supported, but in some cases they are avoided due to style guidelines. We hope to publish a version of these guidelines in the future.

[Mattwmaster58]

I'm curious what the reason was for retaining whitespace in what look like deleted sections of the source eg https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/nv.c#L623. Was this intentional or just a side effect on how it was done?

[ElijahPepe]

Likely just a side effect of how they chose to get rid of the lines and nothing else.

[ptr1337]

You can see it here:
https://gist.github.com/ptr1337/2e361f8f87abd57b1f6c1ea443f87f46

[mtijanic]

This is an issue with our packaging scripts that removed parts of the codebase that are not relevant to this driver. Successive blank lines should have been collapsed into a single one. We will fix the scripts.

[cdknight]

Considering that NVIDIA does not currently have plans to open-source the userspace components of the driver, at the minimum, will there be any officiial documentation or libraries to help create a userspace RM client (I think this constitutes opening and controlling /dev/nvidiactl and /dev/nvidia%d)? What I mean something like a FOSS NvAPI or nvml, but not even to that level, but just simply documenting how to call methods defined in for example (but it could be any of the NVXXXX_CTRL functions): src/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080bus.h. Such documentation may aid (community) userspace driver development, but even at the minimum, help with GPU monitoring and control.

[mtijanic]

There are currently no plans to open source NVML or NVAPI, nor to maintain a third library, sorry.

[DemiMarie]

That’s unfortunate. Such a library really ought to be in this repository, so that it can be kept in sync with kernel driver changes.

[alexflint]

I am also interested in writing a userspace RM client. Is there any documentation or example code that might help me accomplish this? I am aware that there is no ABI stability and I'm willing to target a particular driver version.

[t-nakamura-dev]

Certainly we can provide these docs and would be interested in what tools the community makes with them. We ask for a bit of patience though while we find the time to write such docs.

@mtijanic
It's been 2.5 years since this discussion started. We are waiting patiently, so please let us know if there is any progress. This is an area that we are interested in, not only as individuals but also as a company and organization, so we would be happy to cooperate if we can.

[mtijanic]

Hey there! See this post from a few months ago: #530 (comment)

Probably best to continue the discussion on that thread if you want some additional clarification or examples. I don't think we'll be able to staff writing actual docs any time soon, sorry.

[edisionnano]

Since the other topic got locked Ill ask here too. While this Repo provides kernel drivers for turing, ampere and upcoming generations will PMU firmware be published to allow nouveau to reclock Maxwell 2 and Pascal GPUs?

[cdknight]

Skimming through the codebase, I see references to something called NVOS and NVOC. Their similar names (perhaps incorrectly) leads me to believe they are related. What are they?

[mtijanic]

See here for NVOC.
NVOS is part of the SDK which exposes the interface to RM from (mostly userspace) clients. It defines ioctl numbers and the like.

similar names

Almost everything starts with "NV" or "RM", so you can safely just treat those as namespace and not a meaningful part of the name :)

[cdknight]

Thanks for the information! (Should have looked around more…)

[fengyuanyu1]

How GPU fetch the push buffer? by DMA or others?
As far as I know, CPU side maintain a push buffer, its entry is the pointers to the GPU commands.
When I kick off the doorbell-register, GPU launch a DMA on PCIe to read the GPU command?

[aaronp24]

Yes, that's correct.

[fengyuanyu1]

Hello, @aaronp24
I have another issue about the communication mechanisms about the CPU/GPU. How CPU knows GPU's status? I mean, how can CPU knows the GPU have finished its commands?

[aaronp24]

It depends on what the CPU is looking for. If it just wants to see if there's room for more data in the push buffer or the GPFIFO, it can read the Get and GPGet fields of the KeplerBControlGPFifo structure. The other option is to use a host semaphore release to track progress. For example, see the InsertProgressTracker function in nvidia-push.c. Note the comment at the top about whether or not the semaphore triggers a wait for idle (WFI): if that's disabled, the semaphore only indicates that the host method processor has gotten past that particular method. If you enable WFI then it indicates that the GPU has finished all of its prior methods.

There other types of progress tracking indicators available as well. See, for example, the NV*97_SET_REPORT_SEMAPHORE_* methods.

[CarloRamponi]

Can the driver intercept a kernel launch or other high-level operations such as context creation or module loading?
Perhaps also accessing user-mode structures such as CUContext, CUModule, or CUFunction?

[mtijanic]

Something like cuCtxCreate() involves dozens of separate calls into the kernel driver. You can certainly catch any one of them and poke around. You can also try to do this in userspace by overriding ioctl(), or by using strace, or something like bpftrace.

However, actual kernel launch and stuff is mostly handled directly from userspace to GPU via shared memory, and the kernel is not involved. To instrument that, I believe envytools has something based on userfaultfd that will log all the writes to this memory that the UMD makes.

[mtijanic]

Here's a list of all the syscalls a CUDA app runs as part of cuInit() + cuCtxCreate(): https://gist.github.com/mtijanic/aabdfd00d9c73491c74638da826ed6d4 (gathered with bpftrace)

[Garrybest]

[Garrybest]

Hi @mtijanic @aaronp24 @gauravjuvekar, I'm tring to enable and disable the channel every 2s, but I found sometimes the cuda kernel will be stuck and the GPU utilization decreases to 0. Do you have any ideas about this?

[SeungsuBaek]

Why does this code exist, and when will it be resolved? Earlier versions don't have that code, is it okay to use Access counter?

[wyfs4321]

I'm a newbie of the open-gpu-kernel-modules, when i read the code, i got into trouble when understanding "pRmApi->Control()", e,g,
status = pRmApi->Control(pRmApi, pCtx->hClient, pCtx->hChannel, NVC56F_CTRL_CMD_GET_KMB, &getKmbParams, sizeof(getKmbParams));
i wonder how 'control' implement the function of "getKmbParams"? Can I think of ‘control’ as an interface in which a function about "getKmbParams" is invoked?

[mtijanic]

Hey there. Yes, your understanding is right. It's an object oriented design, and "controls" are just a method on a given object (which is identified with a {hClient,hObject} pair). The params are passed as basically (uint32_t cmd, void* params) and then the params pointer is cast to the appropriate structure depending on the cmd value. Here cmd is NVC56F_CTRL_CMD_GET_KMB and the param structure is NVC56F_CTRL_CMD_GET_KMB_PARAMS (the two usually follow this naming scheme, but there's exceptions).

Now, you can think of pRmApi->Control() as just a bunch of routing magic that resolves the objects in question and then calls the correct handler function. If you can tolerate black boxes, you can completely ignore all that and just jump straight into the handler function, which is: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/565.57.01/src/nvidia/src/kernel/gpu/fifo/kernel_channel.c#L4648-L4654

The easiest way to find this handler function is to search the all g_*_nvoc.c files NVC56F_CTRL_CMD_GET_KMB_PARAMS and find the export block that looks like: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/565.57.01/src/nvidia/generated/g_kernel_channel_nvoc.c#L570-L584

This will tell you what the function name is. If there is no such function found in the codebase, then that means the function is implemented on the GSP instead, and the pRmApi->Control() will invoke a blocking RPC to execute the function.

Technically, any control where the flags field in the export has this bit set

#define RMCTRL_FLAGS_ROUTE_TO_PHYSICAL                        0x000000040

will be RPC'd, even if the implementation is present, but that's very rare.

Hope this helps.

[zhizhi10]

I just got into open-gpu-kernel-modules. When I checked utilization.memory through the nvidia-smi command, the result was not the percentage of used memory, and I could not find relevant information in the code. Can anyone tell me what the utilization.memory parameter represents?

nvidia-smi -lms --query-gpu=utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv
utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
0 %, 0 %, 24096 MiB, 23093 MiB, 626 MiB
0 %, 0 %, 24096 MiB, 23093 MiB, 626 MiB
0 %, 0 %, 24096 MiB, 23093 MiB, 626 MiB
0 %, 0 %, 24096 MiB, 23093 MiB, 626 MiB
0 %, 0 %, 24096 MiB, 23093 MiB, 626 MiB

[t-nakamura-dev]

@zhizhi10 According to the document, it is percent of time over the past sample period during which global (device) memory was being read or written.

https://docs.nvidia.com/deploy/nvml-api/structnvmlUtilization__t.html

[X547]

What is the difference between NV01_MEMORY_VIRTUAL and NV50_MEMORY_VIRTUAL?

@aritger, @mtijanic

[aritger]

I'm sorry, I'm not very familiar with either.

From code inspection, it looks like:

NV01_MEMORY_VIRTUAL:

• allocated with param struct NV_MEMORY_VIRTUAL_ALLOCATION_PARAMS
• corresponds to the struct VirtualMemoryRange
• initialized by vmrangeConstruct_IMPL

NV50_MEMORY_VIRTUAL:

• allocated with param struct NV_MEMORY_ALLOCATION_PARAMS
• corresponds to the struct VirtualMemory
• initialized by virtmemConstruct_IMPL

Conceptually, NV01_MEMORY_VIRTUAL is "derived" from NV50_MEMORY_VIRTUAL.

I hope that helps.

[Sur-Ring]

When I called cuMemAlloc, I noticed that there are 3 ioctls, namely 0x2b of nvidia_ioctl, 73 of uvm_ioctl and 33 of uvm_ioctl. I wonder how cuda decides what parameters to pass to them. They are 'hObjectNew' in nvidia_ioctl(0x2b) (it is the hMemory of uvm_ioctl(33)) and the 'base' in uvm_ioctl(both 73 and 33). I guess it has something to do with cuContext, but I don't know how to continue tracing them, can you give me some hints?

[zhangboyue]

Happened to looking at the same topic. To my understanding, they are all relevant to a single memory management action, so it's not a problem to feed correct handles to uvm_ioctl(33) within CUDA driver. Nvidia_ioctl(0x2b) is used to allocated video memory, uvm_ioctl(73) is for uvm to device the proper address range in UVA, and uvm_ioctl(33) does the mapping works.

[Garrybest]

What is the relationship between UserD, Doorbell, GP_PUT and PushBuffer?

[X547]

My code that use channels may help with understanding: https://github.com/X547/mesa/blob/mesa-nvk/src/nouveau/vulkan/nvkmd/nvrm/nvkmd_nvrm_ctx.c.

[Xubbbb]

My code that use channels may help with understanding: https://github.com/X547/mesa/blob/mesa-nvk/src/nouveau/vulkan/nvkmd/nvrm/nvkmd_nvrm_ctx.c.

Hello, I would like to ask how to compile and build the 'mesa-nvk-r2' branch of this project.

[YusufKhan-gamedev]

See https://docs.mesa3d.org/install.html for a guide on compiling mesa

[Xubbbb]

See https://docs.mesa3d.org/install.html for a guide on compiling mesa

OK, thank you. By modifying some build configurations, I have successfully built it. I noticed that it seems there was no 'KernelChannelGroup' created in this project, is that not necessary?

[X547]

What is "swap group" concept in nvidia-modeset? That is the point of supporting swap group and non-swap group parts on the same screen? What is better to use (lower latency etc.) if there are no legacy constraints?

[aritger]

"Swap group" refers to this:
https://registry.khronos.org/OpenGL/extensions/NV/GLX_NV_swap_group.txt
multiple separate drawables who all are supposed to do their SwapBuffers atomically. It requires a lot of extra complexity in the driver to achieve without native hardware support. Unless you have a specific need for the functionality, it is simpler to use non-swapgroup.

[rajesh-s]

I would like to understand the sequence of calls that occur in the runtime upon calling cudaLaunchKernel from the user space. What would be the best way to get to this?

[YusufKhan-gamedev]

There is this tool:

https://gitlab.freedesktop.org/nouveau/envyhooks

There is also code in nvk:

https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/nouveau/vulkan/nvk_cmd_dispatch.c?ref_type=heads#L289
P_MTHD sets the function name, and the rest gives the function paramaters, this function is somewhat similar to cudaLaunchKernel, see the vulkan documentaton for more details
https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/nouveau/vulkan/nvk_shader.c?ref_type=heads
This shows the shader data and how it gets loaded

Look at the haiku code for how the NVIDIA driver does it, this bit in nvkmd is much more similar to how NIVIDA should do things
https://github.com/X547/mesa/tree/mesa-nvk/src/nouveau/vulkan/nvkmd/nvrm

you can also LD_PRELOAD a shared object with ioctl() and run a program to see how the ioctls are behaving.

[adarshpradhan-jmt]

Question: once I get the physical address table from nvidia_p2p_get_pages function, is there a way to remap them to a new virtual address area in gpu for user space access?

[Xubbbb]

Hello, based on my tests of CUDA and Vulkan, I found:

When initializing the CUDA Context's 'GP_FIFO', the user-space driver first allocates a block of 'VideoMemory', then calls the NV_ESC_RM_MAP_MEMORY ioctl, and finally uses mmap to map it to user space. In contrast, the Vulkan user-space driver create 'GP_FIFO' by allocating a block of 'SystemMemory' first during initialization, then calls the NV_ESC_RM_MAP_MEMORY ioctl, followed by mmap to map it to user space, and then allocates a block of 'VirtualMemory', mapping the virtual memory and system memory together using the NV_ESC_RM_MAP_MEMORY_DMA ioctl. What is the difference between these two methods? Moreover, when creating a channel, the 'gpFifoOffset' parameter passed in is a user-space virtual address in CUDA, while it is a GPU virtual address in Vulkan, which seems a bit strange.

Could you explain these?

[X547]

Moreover, when creating a channel, the 'gpFifoOffset' parameter passed in is a user-space virtual address in CUDA, while it is a GPU virtual address in Vulkan, which seems a bit strange.

I suppose that CUDA use the same virtual address space for CPU and GPU to allow transparent arbitrary memory migration between CPU and GPU memory.

[X547]

VideoMemory (NV01_MEMORY_LOCAL_USER) is a memory physically located on GPU. SystemMemory (NV01_MEMORY_SYSTEM) is a regular CPU memory (for example DDR 4 memory modules installed to motherboard) that is accessible by GPU via PCIe bus.

[Xubbbb]

Thank you for your reply, got it.

[RixinLiu]

Hi folks, I’m currently studying how GPU faults are handled and I’m trying to understand whether there is a practical way to trigger a UVM non-replayable fault.

As I understand it, UVM categorizes faults into replayable and non-replayable. Roughly speaking, faults coming from the Graphics Engine (SM) are replayable, while faults coming from the Copy Engine or PBDMA are non-replayable. So far, the only detailed explanation I’ve found is in the comments inside kernel-open/nvidia-uvm/uvm_gpu_non_replayable_faults.c (if there is any official documentation elsewhere, I’d really appreciate pointers).

The comment gives an example:
“An example of a Copy Engine non-replayable fault is a memory copy between two virtual addresses on a GPU, in which either the source or destination pointers are not currently mapped to a physical address in the page tables of the GPU.”

I tried to reproduce this in two ways:

• Using cudaMallocManaged and then applying cuMemAdvise to make the destination pages preferred on the CPU, this way does not guarantee the physical page on GPU has been evicted.
• Using the VMM API (cuMemCreate etc.) to create a valid GPU VA range without backing it with physical memory, this way should guarantee it.

But none of these attempts triggered a non-replayable fault. I monitored schedule_non_replayable_faults_handler in kernel-open/nvidia-uvm/uvm_gpu_isr.c and it never returned one(means one handler is scheduled). Instead, for the first way, i only got replayable fault, because UVM trying to migrate page from CPU to GPU. For the second way, I only got a segmentation fault from the CPU side :(

Before I keep digging, I wanted to ask:
Has anyone successfully triggered a UVM non-replayable fault, or has insights into conditions that reliably cause one?
Any suggestions or thoughts would be greatly appreciated!

[Xubbbb]

Hello, some gpu commands' subchannel is 5-7. Open-gpu-doc states that "Subchannels 5-7 are for software methods. Any methods on these subchannels (including SetObject methods) are kicked back to software for handling via the SW method dispatch mechanism using the NV_PPBDMA_INTR_*_DEVICE interrupt. SW may choose to send a SetObject method to each engine subchannel before sending any methods on that particular subchannel in order to support multiple software classes."
What's this mean? Are these gpu commands executed by the gpu driver on the Host CPU rather than by the GPU hardware itself? Could you give a more detailed introduction to its mechanism?

[mtijanic]

Hi! Yeah, you mostly figured it out. Userspace can put these "SW method" commands in the pushbuffer, and when the GPU hits one of those, it triggers an interrupt and the method is handled in the driver. This is done synchronously, and the engine is stalled until this is processed.

With the GSP architecture, the GSP handles all these interrupts, not the host CPU. GSP does, in some cases, forward them to the CPU, but this forwarding is async and the GPU won't wait for the CPU to finish processing.

Looks like we shipped a bit of dead code in this repo that shows one of these methods. Most of this code only ever runs on GSP, but you can read up on it to get a better understanding: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/590.44.01/src/nvidia/src/kernel/gpu/timed_semaphore.c

Specifically:

• mapping between command IDs and functions
• place where GSP sends an event over to CPU
• place where CPU receives the event
• place where CPU ends up servicing the event

Because all SW method interrupts go through GSP, I don't think you can actually add new ones. But I'm sure you can find new and exciting ways to use existing ones :)

[Xubbbb]

Thank you for your reply. Is there any public source that defines all existing software method classes, or are they entirely internal to the GSP?

[mtijanic]

There is no central place, sorry (not even internally). You can search for _SET_OBJECT in this repo or open-gpu-doc and you'll find some classes that support this. Typically the method at 0x100 is NO_OPERATION and then the real first method is 0x104.

Outside of those, the one interesting method might be Nv50DeferredApi that basically allows you to invoke some driver control commands inline with the GPU work being submitted.

But note that these are entirely software defined and are sometimes used for debugging or instrumentation, and may come and go between driver releases. I think envyhooks can tell you which ones are used by our UMDs in production.

[Xubbbb]

Got it, thanks a lot for your detailed explanation.