RadarAI· MarkTechPost（RSS）·· 3 小时前AI 评分60
NVIDIA Nsight Python 教程：如何在 Colab T4 上剖析 GPU kernel
AI 导读
MarkTechPost 发布教程，演示用 NVIDIA 开源的 Nsight Python 在免费 Colab T4 上剖析 GPU kernel：用 nsight.annotate 标注代码、用 nsight.analyze.kernel 装饰基准函数，直接得到含每次运行硬件指标的 pandas DataFrame。
正文 · AI 翻译
MarkTechPost· Sana Hassan·2026-10-22 11:59· 1 小时前AI 评分59
NVIDIA Nsight Python 教程：如何剖析 GPU kernel
AI 导读
教程演示如何用 NVIDIA 开源的 Nsight Python 库在 Colab T4 上剖析 GPU kernel：用 nsight.annotate 标注代码、用 nsight.analyze.kernel 装饰基准函数，直接得到包含每次运行硬件指标的 pandas DataFrame。
正文 · 原文
在本教程中，我们实现的是 Nsight Python，这是 NVIDIA 为其 kernel 剖析器提供的开源 Python 接口。我们不再需要在 Nsight Compute 命令行下运行脚本、事后再翻查报告，而是用 nsight.annotate 标记我们关心的代码，用 nsight.analyze.kernel 装饰基准函数，然后就能得到一个 pandas DataFrame，其中包含按标注区域、按配置、按每次运行记录的硬件指标。自 1.0 版本起，该库会将 Nsight Compute 附加到正在运行的 Python 进程上，使其可在 Jupyter 和 Colab 中使用；即将发布的 1.1 版本新增了实验性的 CUPTI 后端、显式工具管理和自定义信息收集器。我们在免费的 Colab T4 上运行全部内容：在 Colab 自带的 Nsight Compute 之外安装一个当前版本的 Nsight Compute，剖析第一个 kernel，确定一次测量需要多少次运行，测量缓存和时钟控制对同一 kernel 的影响，以 fp32 和 fp16 扫描矩阵尺寸并计算派生的 TFLOP/s 指标，使用硬件计数器区分计算受限与内存受限的工作负载，以字节移动量衡量 kernel 融合，记录每次运行的时钟和温度，比较 CUPTI 后端与 Nsight Compute，并保留原始报告以供深入分析。
import os
import sys
import glob
import json
import time
import shutil
import textwrap
import traceback
import subprocess
RESULTS = {}
def banner(title):
print("\n" + "=" * 78)
print(title)
print("=" * 78)
def section(name):
def wrap(fn):
def run(*a, **kw):
banner(name)
try:
out = fn(*a, **kw)
RESULTS[name] = out if isinstance(out, str) else "ok"
return out
except Exception as e:
RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {str(e).splitlines()[0][:160]}"
print(f"\n[!] {name} did not complete: {type(e).__name__}: {e}")
traceback.print_exc(limit=3)
return None
return run
return wrap
def sh(cmd):
"""Run a shell command and return its stdout, stripped."""
return subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout.strip()
banner("1. Install a current Nsight Compute next to Colab's, then import nsight before CUDA")
print(f" GPU: {sh('nvidia-smi --query-gpu=name,driver_version,memory.total,compute_cap --format=csv,noheader')}")
preinstalled = shutil.which("ncu")
print(f" ncu already on PATH: {preinstalled} -> {sh('ncu --version | tail -1') if preinstalled else 'none'}")
# nsight-python needs Nsight Compute 2026.2.1 or newer. Colab ships an older ncu, but its image already
# carries NVIDIA's CUDA apt repository, so a current release is one apt-get away. It installs into
# /opt/nvidia/nsight-compute/<version>/ and leaves the preinstalled one untouched.
subprocess.run("apt-get update -qq > /tmp/ncu_install.log 2>&1", shell=True)
for NCU_VERSION in ("2026.3.1", "2026.2.1"):
if subprocess.run(f"apt-get install -y -qq nsight-compute-{NCU_VERSION} >> /tmp/ncu_install.log 2>&1",
shell=True).returncode == 0:
break
else:
print(sh("tail -5 /tmp/ncu_install.log"))
raise RuntimeError("could not install Nsight Compute 2026.2.1+ from NVIDIA's apt repository")
# ask dpkg where the package put the ncu launcher rather than assuming /opt/nvidia/nsight-compute/<version>/
ncu_paths = sh(f"dpkg -L nsight-compute-{NCU_VERSION} | grep -E '/ncu$'").split()
if not ncu_paths:
raise RuntimeError(f"dpkg lists no ncu executable in nsight-compute-{NCU_VERSION}")
NCU_DIR = os.path.dirname(min(ncu_paths, key=len)) # the top-level launcher, not a per-platform copy
os.environ["PATH"] = NCU_DIR + os.pathsep + os.environ["PATH"] # nsight-python looks ncu up on PATH
print(f" installed: {shutil.which('ncu')} -> {sh('ncu --version | tail -1')}")
# nsight-python 1.1.0 (CUPTI backend, tool APIs, info collectors) is tagged but not yet on PyPI;
# pin the commit this notebook was written against. The [cupti] extra is used in step 9.
NSIGHT_COMMIT = "bb96db4260b1823d86ecab9acfed0646d7af79ad"
subprocess.run([sys.executable, "-m", "pip", "install", "-q",
f"nsight-python @ git+https://github.com/NVIDIA/nsight-python@{NSIGHT_COMMIT}"], check=True)
# The CUPTI backend needs cupti-python 13.4, which wants a newer libcupti than Colab's torch is pinned to.
# Install it into a separate directory that only step 9's child process puts on its path.
CUPTI_DIR = "/content/cupti_env"
CUPTI_EXTRA = subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--target", CUPTI_DIR, "--no-deps",
"cupti-python>=13.4.0", "nvidia-cuda-cupti~=13.4", "pyelftools"]).returncode == 0
import nsight # activates Nsight Compute's injection library: must come before anything initializes CUDA
from nsight.collection import ncu as ncu_backend
import numpy as np
import pandas as pd
import torch
SILENT = nsight.VerbosityLevel.SILENT
pd.set_option("display.width", 200)
pd.set_option("display.max_columns", 30)
print(f" nsight-python {nsight.__version__} | active tool: {nsight.get_active_tool()}"
f" | injection error at import: {ncu_backend.injection_load_error}")
print(f" torch {torch.__version__} (CUDA {torch.version.cuda}) | {torch.cuda.get_device_name(0)}"
f" | cupti-python installed: {CUPTI_EXTRA}")
print("\n Order matters: `import nsight` loads Nsight Compute into this Python process, and it can only")
print(" attach to CUDA work started after that. Every profiled function below runs in this same")
print(" process; nsight-python starts `ncu --mode attach` on its own PID for each one.")
我们首先检查 GPU 和剖析器。Nsight Python 要求 Nsight Compute 命令行工具 ncu 的版本为 2026.2.1 或更高，并通过在 PATH 中查找来定位它。Colab 的 GPU 镜像自带较旧的 Nsight Compute。不过，它已经配置好了 NVIDIA 的 CUDA 软件包仓库，因此我们用 apt-get 安装 Nsight Compute 2026.3.1，用 dpkg 查询 ncu 启动器安装到了哪里，并把该目录放在 PATH 的最前面。接着我们安装 Nsight Python，将其固定到本 notebook 编写时所依据的 1.1.0 预发布版本的提交，并把 cupti-python 包安装到单独目录，因为它需要的 CUPTI 库比 Colab 的 PyTorch 所固定的版本更新。导入顺序很重要：import nsight 会将 Nsight Compute 的注入库加载到 Python 进程中，而它只能附加到此后启动的 CUDA 工作，因此必须在 torch 接触 GPU 之前导入 nsight。
ADD = lambda x, y: x + y # sums a metric if cuBLAS splits one matmul into several kernels
@nsight.analyze.kernel(thermal_mode="off", combine_kernel_metrics=ADD)
def first_matmul(n: int) -> None:
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
with nsight.annotate("matmul"):
_ = a @ b
@section("2. Profile one kernel: what a ProfileResults row contains")
def first_profile():
df = first_matmul(2048).to_dataframe()
print(f"\n to_dataframe() columns: {list(df.columns)}\n")
r = df.iloc[0]
t = float(r["AvgValue"])
print(f" annotation {r['Annotation']!r}, n = {r['n']}, kernel {r['Kernel']}")
print(f" {r['Metric']} = {t:,.0f} {r['Unit']} (runs {r['NumRuns']}, stable: {r['StableMeasurement']})")
print(f" GPU {r['GPU']}, ComputeClock {r['ComputeClock']}, MemoryClock {r['MemoryClock']}")
print(f" -> {2 * 2048 ** 3 / (t * 1e-9) / 1e12:.2f} TFLOP/s for a 2048 x 2048 fp32 matmul")
print("\n Only the kernel launched inside nsight.annotate() is profiled; torch.randn above it is not.")
print(" With a single run, StableMeasurement is <NA>: one sample has no spread to judge.")
return f"fp32 2048 matmul {t / 1e6:.3f} ms ({2 * 2048 ** 3 / (t * 1e-9) / 1e12:.2f} TFLOP/s)"
first_profile()
第一个 profile 是单次 fp32 矩阵乘法。基准测试函数创建输入，然后只把乘法包裹在 nsight.annotate 中，因此构建输入的随机数内核不会被测量。调用被装饰的函数会在 notebook 自身进程上以 attach 模式启动 ncu，运行该函数，并返回一个 ProfileResults 对象，其 to_dataframe 方法为每个注释、配置和指标各给出一行。我们打印列并读取该行：内核名称、以纳秒为单位的实测 gpu__time_duration.sum 及其单位、运行次数、GPU 名称和时钟属性，然后手动把时间换算成 TFLOP/s。combine_kernel_metrics 参数会在 cuBLAS 将一次乘法拆分成多个内核时对时间求和；在单次运行的情况下，StableMeasurement 标志未定义，因为单个样本没有离散度。
def show(df, cols, scale=None, fmt="{:,.2f}"):
out = df[cols].copy()
for c, f in (scale or {}).items():
out[c] = out[c] * f
print(textwrap.indent(out.to_string(index=False, float_format=lambda v: fmt.format(v)), " "))
@nsight.analyze.kernel(configs=[1 << 14, 1 << 24], runs=20, thermal_mode="off", verbosity=SILENT,
combine_kernel_metrics=ADD)
def noisy_or_not(n: int) -> None:
x = torch.randn(n, device="cuda")
y = torch.randn(n, device="cuda")
a = torch.randn(2048, 2048, device="cuda")
with nsight.annotate("vector add"):
_ = x + y
with nsight.annotate("matmul 2048"):
_ = a @ a
@section("3. How many runs? Spread, confidence intervals and the stability flag")
def stability():
df = noisy_or_not().to_dataframe()
print(" 20 runs per configuration; times in microseconds:\n")
show(df, ["Annotation", "n", "AvgValue", "StdDev", "CI95_Lower", "CI95_Upper", "RelativeStdDevPct",
"StableMeasurement"], scale={"AvgValue": 1e-3, "StdDev": 1e-3, "CI95_Lower": 1e-3, "CI95_Upper": 1e-3})
tiny = df[(df.Annotation == "vector add") & (df.n == 1 << 14)].iloc[0]
print(f"\n StableMeasurement is simply RelativeStdDevPct < 2. The smallest kernel ({tiny['AvgValue'] / 1e3:.1f} us)"
f" varies by {tiny['RelativeStdDevPct']:.1f}%:")
print(" at microsecond scale, launch and clock jitter are a large share of the time. Judge small kernels")
print(" with more runs, and compare their confidence intervals rather than single averages.")
return f"16K-element add RSD {tiny['RelativeStdDevPct']:.1f}%; " + \
f"2048 matmul RSD {df[df.Annotation == 'matmul 2048'].RelativeStdDevPct.max():.1f}%"
stability()
单次测量无法说明噪声情况，因此下一步将每种配置运行二十次。此时 DataFrame 会包含均值、标准差、95% 置信区间、以百分比表示的相对标准差，以及 StableMeasurement 标志，在源码中该标志就是相对标准差低于百分之二。我们对两种规模的向量加法进行 profile，一种是运行时间以微秒计的 16000 元素规模，另一种是 1600 万元素规模，同时还有 2048 乘 2048 的矩阵乘法，并比较各自的波动程度。实用规则是：对短内核运行更多次，并比较置信区间，而不是比较单次平均值。
def make_scale_bench(cache_control, clock_control):
@nsight.analyze.kernel(configs=[1 << 18, 1 << 24], runs=10, cache_control=cache_control,
clock_control=clock_control, thermal_mode="off", verbosity=SILENT,
metrics=["gpu__time_duration.sum", "gpc__cycles_elapsed.avg.per_second"])
def scale(n: int) -> None:
x = torch.randn(n, device="cuda")
with nsight.annotate("x * 2"):
_ = x * 2.0
return scale
@section("4. Cache and clock control: two defaults that change the answer")
def cache_and_clocks():
rows = []
for cache in ("all", "none"):
for clock in ("none", "base"):
try:
df = make_scale_bench(cache, clock)().to_dataframe()
except Exception as e:
print(f" cache_control={cache!r}, clock_control={clock!r}: failed -> {type(e).__name__}: {str(e)[:120]}")
continue
t = df[df.Metric == "gpu__time_duration.sum"].set_index("n").AvgValue
hz = df[df.Metric == "gpc__cycles_elapsed.avg.per_second"].set_index("n").AvgValue
for n in t.index:
rows.append({"cache_control": cache, "clock_control": clock, "MB": 4 * n / 2 ** 20,
"time_us": t[n] / 1e3, "GB/s": 2 * 4 * n / t[n], "SM_MHz": hz[n] / 1e6})
tab = pd.DataFrame(rows)
print(textwrap.indent(tab.to_string(index=False, float_format=lambda v: f"{v:,.1f}"), " "))
print("\n The T4's L2 cache is 4 MB. A 1 MB input stays in L2 between runs unless caches are flushed,")
print(" so cache_control='none' measures a warm-cache kernel and 'all' (the default) a cold one.")
print(" clock_control='base' locks the SM clock to base frequency for every replay (the plain ncu CLI")
print(" default); nsight-python defaults to 'none', so the GPU boosts. SM_MHz is measured per kernel")
print(" (gpc__cycles_elapsed.avg.per_second). The ComputeClock column is not: it is the device's")
print(" rated clock attribute and stays the same whatever the kernel ran at.")
globals()["CLOCK_TABLE"] = tab
small = tab[tab.MB < 2].set_index(["cache_control", "clock_control"]).time_us
return (f"1 MB scale: cold/boost {small.get(('all', 'none'), float('nan')):.1f} us, "
f"warm/boost {small.get(('none', 'none'), float('nan')):.1f} us, "
f"cold/base {small.get(('all', 'base'), float('nan')):.1f} us")
cache_and_clocks()
两个 profiler 默认设置决定了测量的含义，本步骤在同一个内核上测量两者：将向量乘以二。cache_control 决定是否在每次内核重放前刷新缓存：Nsight Python 默认会刷新它们，因此原本会留在 T4 的 4 MB L2 缓存中的小输入每次都会从内存读取。clock_control 决定是否锁定 GPU 时钟：Nsight Python 默认不锁定它们，而普通的 ncu 命令行会将它们锁定到基础时钟。我们在 1 MB 和 64 MB 下运行全部四种组合，并额外收集 gpc__cycles_elapsed.avg.per_second，它测量内核实际运行时的时钟。这一点很重要，因为正如源码所示，DataFrame 的 ComputeClock 列是设备的额定时钟属性，而不是测量值。
SIZES = [(512,), (1024,), (2048,), (4096,)]
def tflops(time_ns: float, n: int) -> tuple[float, str]:
return 2 * n ** 3 / (time_ns * 1e-9) / 1e12, "TFLOP/s"
def make_precision_body():
def precision(n: int) -> None: # a fresh function object for each decorator
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
a16, b16 = a.half(), b.half()
with nsight.annotate("fp32"):
_ = a @ b
with nsight.annotate("fp16"):
_ = a16 @ b16
return precision
precision_tflops = nsight.analyze.kernel(configs=SIZES, runs=5, derive_metric=tflops, thermal_mode="off",
verbosity=SILENT, combine_kernel_metrics=ADD)(make_precision_body())
precision_ratio = nsight.analyze.kernel(configs=SIZES, runs=5, normalize_against="fp32", thermal_mode="off",
verbosity=SILENT, combine_kernel_metrics=ADD)(make_precision_body())
@section("5. A size sweep, a derived TFLOP/s metric, and normalisation")
def precision_sweep():
df = precision_tflops().to_dataframe()
tf = df[df.Metric == "tflops"].pivot(index="n", columns="Annotation", values="AvgValue")
print(" derive_metric=tflops receives (gpu__time_duration.sum, n) and returns (value, 'TFLOP/s'):\n")
print(textwrap.indent(tf[["fp32", "fp16"]].to_string(float_format=lambda v: f"{v:,.2f}"), " "))
print("\n T4 datasheet peaks: 8.1 TFLOP/s fp32, 65 TFLOP/s fp16 on tensor cores.")
rd = precision_ratio().to_dataframe()
fp16 = rd[rd.Annotation == "fp16"][["n", "AvgValue", "StdDev", "Unit", "Normalized"]]
print("\n normalize_against='fp32' divides each AvgValue by the fp32 AvgValue of the same configuration:\n")
print(textwrap.indent(fp16.to_string(index=False, float_format=lambda v: f"{v:,.3f}"), " "))
print("\n Read the row carefully: AvgValue is now a ratio (fp16 time / fp32 time), but StdDev and Unit are")
print(" still those of the raw nanosecond measurement, because only AvgValue is divided.")
best = tf.loc[4096]
return f"n=4096: fp32 {best['fp32']:.2f}, fp16 {best['fp16']:.2f} TFLOP/s"
precision_sweep()
接下来，我们将矩阵大小从 512 扫描到 4096，并在一个基准测试函数中同时运行 fp32 和 fp16，每种精度对应一个注释。derive_metric 函数接收实测时间，随后是基准测试的参数，并返回一个带单位的值，这里是 TFLOP/s，DataFrame 会将其作为额外指标存储在时间旁边。T4 的数据表峰值为 fp32 下 8.1 TFLOP/s，以及其张量核心上 fp16 下 65 TFLOP/s，这为实测数字提供了尺度。然后我们使用 normalize_against 设置为 fp32 注释来运行相同的扫描，这会将每个 AvgValue 除以相同配置的 fp32 值。正如该转换的源码所示，只有 AvgValue 被除：StdDev、置信区间和 Unit 列仍然描述原始的纳秒测量值，这很容易被误读。
BOUND_METRICS = ["gpu__time_duration.sum",
"sm__throughput.avg.pct_of_peak_sustained_elapsed",
"dram__throughput.avg.pct_of_peak_sustained_elapsed"]
@nsight.analyze.kernel(metrics=BOUND_METRICS, runs=3, thermal_mode="off", verbosity=SILENT)
def what_limits(n: int) -> None:
a = torch.randn(n, n, device="cuda").half()
x = torch.randn(64 * 2 ** 20, device="cuda")
s = torch.randn(8192, 1024, device="cuda")
with nsight.annotate("fp16 matmul 4096"):
_ = a @ a
with nsight.annotate("x * 2 (256 MB)"):
_ = x * 2.0
with nsight.annotate("softmax 8192x1024"):
_ = torch.softmax(s, dim=-1)
@section("6. Compute-bound or memory-bound? Three metrics from one replayed kernel")
def bound():
df = what_limits(4096).to_dataframe()
tab = df.pivot(index="Annotation", columns="Metric", values="AvgValue")
tab = tab.rename(columns={"gpu__time_duration.sum": "ms",
"sm__throughput.avg.pct_of_peak_sustained_elapsed": "SM % of peak",
"dram__throughput.avg.pct_of_peak_sustained_elapsed": "DRAM % of peak"})
tab["ms"] = tab["ms"] / 1e6
tab["limited by"] = np.where(tab["SM % of peak"] > tab["DRAM % of peak"], "compute", "memory")
print(textwrap.indent(tab.to_string(float_format=lambda v: f"{v:,.2f}"), " "))
x2 = tab.loc["x * 2 (256 MB)"]
gbs = 2 * 4 * 64 * 2 ** 20 / (x2["ms"] * 1e6)
print(f"\n Cross-check: x * 2 moves 2 x 256 MB in {x2['ms']:.2f} ms = {gbs:.0f} GB/s, against the T4's 320 GB/s")
print(f" datasheet peak ({100 * gbs / 320:.0f}%); Nsight Compute reports {x2['DRAM % of peak']:.0f}% DRAM throughput.")
print(" Collecting several metrics makes Nsight Compute replay each kernel once per pass it needs;")
print(" nsight-python still returns one row per annotation and metric.")
return f"x*2: {gbs:.0f} GB/s ({x2['DRAM % of peak']:.0f}% DRAM); fp16 matmul SM {tab.loc['fp16 matmul 4096', 'SM % of peak']:.0f}%"
bound()
硬件计数器能告诉我们某个内核为何耗时如此之久。传入一组指标会让 Nsight Compute 反复重放每个内核，直到收集齐所有指标，而 Nsight Python 仍然为每个标注和指标返回一行。我们收集三个不同特性内核的持续时间、SM 吞吐量和 DRAM 吞吐量（均以持续峰值的百分比表示）：一个 fp16 矩阵乘法、一个 256 MB 向量的缩放，以及对 8192 行、每行 1024 个值的 softmax，并按较高的吞吐量给每个内核打标签。对于向量缩放，我们还根据它必须搬运的字节数和测得的时间计算有效带宽，并将其与 T4 的 320 GB/s 数据手册峰值以及 Nsight Compute 报告的 DRAM 百分比进行比较。
BYTES = ["gpu__time_duration.sum", "dram__bytes_read.sum", "dram__bytes_write.sum"]
@nsight.analyze.kernel(metrics=BYTES, runs=5, thermal_mode="off", verbosity=SILENT,
combine_kernel_metrics=lambda x, y: x + y)
def fusion(n: int) -> None:
a, b, c = (torch.randn(n, device="cuda") for _ in range(3))
with nsight.annotate("a + b * c (unfused)"):
_ = a + b * c
with nsight.annotate("torch.addcmul (fused)"):
_ = torch.addcmul(a, b, c)
@nsight.analyze.kernel(runs=1, thermal_mode="off", verbosity=SILENT)
def fusion_without_combine(n: int) -> None:
a, b, c = (torch.randn(n, device="cuda") for _ in range(3))
with nsight.annotate("a + b * c"):
_ = a + b * c
@section("7. Kernel fusion, measured in bytes: combine_kernel_metrics")
def fused():
n = 32 * 2 ** 20
try:
fusion_without_combine(n)
print(" (the unfused expression profiled without combine_kernel_metrics)")
except Exception as e:
print(f" Profiled without combine_kernel_metrics, the unfused expression raises:\n {type(e).__name__}: {str(e).splitlines()[0][:200]}\n")
df = fusion(n).to_dataframe()
tab = df.pivot(index="Annotation", columns="Metric", values="AvgValue")
tab = pd.DataFrame({"ms": tab["gpu__time_duration.sum"] / 1e6,
"MB read": tab["dram__bytes_read.sum"] / 2 ** 20,
"MB written": tab["dram__bytes_write.sum"] / 2 ** 20})
print(f" {n:,} float32 elements per tensor (128 MB each); metrics summed over the kernels in each region:\n")
print(textwrap.indent(tab.to_string(float_format=lambda v: f"{v:,.1f}"), " "))
u, f = tab.loc["a + b * c (unfused)"], tab.loc["torch.addcmul (fused)"]
print(f"\n The unfused version writes b * c to memory and reads it back: {u['MB read'] + u['MB written']:,.0f} MB of")
print(f" traffic against {f['MB read'] + f['MB written']:,.0f} MB, and {u['ms'] / f['ms']:.2f}x the time.")
return f"fusion: {u['ms'] / f['ms']:.2f}x faster, {(u['MB read'] + u['MB written']) / (f['MB read'] + f['MB written']):.2f}x less traffic"
fused()
内核融合通常以时间来描述；这里我们以字节来衡量。表达式 a + b * c 在 eager PyTorch 中会先在内存中物化 b * c，然后再与 a 相加，而 torch.addcmul 在一次操作中计算出相同的结果。我们首先在不使用 combine_kernel_metrics 的情况下对未融合的表达式进行性能分析，以展示当一个标注区域包含多个内核时 Nsight Python 抛出的错误，然后使用 combine_kernel_metrics 对两个版本进行性能分析，对每个区域中每个内核的持续时间和读写的 DRAM 字节数求和，张量大小为 2^25（约 3350 万）个 float32 元素。
import pynvml
pynvml.nvmlInit()
NVML = pynvml.nvmlDeviceGetHandleByIndex(0)
@nsight.experimental.collect(scope=nsight.CollectionScope.ONCE)
def driver_version(*args):
return pynvml.nvmlSystemGetDriverVersion()
@nsight.experimental.collect(scope=nsight.CollectionScope.RUN)
def sm_clock_mhz(*args):
return pynvml.nvmlDeviceGetClockInfo(NVML, pynvml.NVML_CLOCK_SM)
@nsight.experimental.collect(scope=nsight.CollectionScope.RUN)
def gpu_temp_c(*args):
return pynvml.nvmlDeviceGetTemperature(NVML, pynvml.NVML_TEMPERATURE_GPU)
def make_heat_body():
def heat(n: int) -> None:
a = torch.randn(n, n, device="cuda").half()
with nsight.annotate("fp16 matmul"):
_ = a @ a
return heat
@section("8. Custom info collectors and thermal control (experimental API)")
def collectors():
out = {}
for mode in ("off", "auto"):
bench = nsight.analyze.kernel(configs=[4096], runs=30, thermal_mode=mode, thermal_timeout=60,
info_collectors=[driver_version, sm_clock_mhz, gpu_temp_c],
verbosity=SILENT, output_prefix=f"/content/nspy_thermal_{mode}/",
output_csv=True)(make_heat_body())
t0 = time.time()
try:
df = bench().to_dataframe()
except Exception as e:
print(f" thermal_mode={mode!r}: failed -> {type(e).__name__}: {str(e).splitlines()[0][:150]}")
continue
wall = time.time() - t0
csvs = sorted(glob.glob(f"/content/nspy_thermal_{mode}/profiled_data-*.csv"))
raw = pd.read_csv(csvs[-1]) if csvs else pd.DataFrame()
out[mode] = (df.iloc[0], wall, raw)
r = df.iloc[0]
print(f" thermal_mode={mode!r}: {r['AvgValue'] / 1e6:.3f} ms +/- {r['RelativeStdDevPct']:.1f}% "
f"wall {wall:.0f}s driver {r['driver_version']}")
print(f"\n The collectors became DataFrame columns; RUN-scoped ones were called before every run.")
if "off" not in out:
return "info-collector run failed"
raw = out["off"][2]
tcol = "Value" if "Value" in raw.columns else raw.columns[-1]
print(f" raw per-run CSV columns: {list(raw.columns)}")
if {"sm_clock_mhz", "gpu_temp_c"} <= set(raw.columns):
corr = raw[[tcol, "sm_clock_mhz", "gpu_temp_c"]].corr().iloc[0]
print(f" over 30 runs: SM clock {raw.sm_clock_mhz.min()}-{raw.sm_clock_mhz.max()} MHz, "
f"temperature {raw.gpu_temp_c.min()}-{raw.gpu_temp_c.max()} C")
print(f" correlation of kernel time with SM clock {corr['sm_clock_mhz']:+.2f}, with temperature {corr['gpu_temp_c']:+.2f}")
print("\n thermal_mode='auto' (the default) pauses between runs when the GPU's thermal headroom runs low;")
print(" the wall-time difference is the price of that, the spread difference what it buys.")
return "; ".join(f"{m} {out[m][0]['RelativeStdDevPct']:.1f}% RSD in {out[m][1]:.0f}s" for m in out)
collectors()
1.1 版本新增了自定义信息收集器，这是一个实验性 API，用于将应用特定的值附加到结果中。用 nsight.experimental.collect 和收集作用域装饰的函数会在每个会话、每个配置、每次运行或每个标注时被调用一次，其返回值成为一列。我们记录一次驱动版本，并在长时间 fp16 乘法的每次运行前通过 NVML 读取 SM 时钟和 GPU 温度，同时启用 output_csv，以便将每次运行的行写在聚合行旁边。我们运行基准测试两次，一次关闭热控制，一次使用默认自动模式（在该模式下，当 GPU 的热余量不足时，Nsight Python 会在运行之间暂停），并比较两者的离散程度和实际耗时。
CUPTI_SCRIPT = r'''
import os, json, time
os.environ["NSPY_NCU_INIT_AT_IMPORT"] = "0" # do not load Nsight Compute in this process
import nsight
nsight.activate(nsight.Tool.CUPTI)
import torch
@nsight.analyze.kernel(configs=[512, 1024, 2048, 4096], runs=10, thermal_mode="off",
verbosity=nsight.VerbosityLevel.SILENT, combine_kernel_metrics=lambda x, y: x + y)
def matmul(n):
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
with nsight.annotate("matmul"):
_ = a @ b
t0 = time.time()
df = matmul().to_dataframe()
wall = time.time() - t0
nsight.deactivate()
print(json.dumps({"wall": wall, "rows": df[["n", "AvgValue", "RelativeStdDevPct"]].to_dict("records")}))
'''
@nsight.analyze.kernel(configs=[512, 1024, 2048, 4096], runs=10, thermal_mode="off", verbosity=SILENT,
combine_kernel_metrics=ADD)
def matmul_ncu(n: int) -> None:
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
with nsight.annotate("matmul"):
_ = a @ b
@section("9. The experimental CUPTI backend, in its own process, against Nsight Compute")
def cupti_vs_ncu():
t0 = time.time()
ncu_df = matmul_ncu().to_dataframe().set_index("n")
ncu_wall = time.time() - t0
if not CUPTI_EXTRA:
return f"cupti-python could not be installed; Nsight Compute run took {ncu_wall:.0f}s"
with open("/content/cupti_matmul.py", "w") as f:
f.write(CUPTI_SCRIPT)
env = {k: v for k, v in os.environ.items() if k not in ("NVTX_INJECTION64_PATH", "NV_COMPUTE_PROFILER_PERFWORKS_DIR",
"NV_COMPUTE_PROFILER_IN_PROCESS")}
env["PYTHONPATH"] = CUPTI_DIR + os.pathsep + env.get("PYTHONPATH", "")
p = subprocess.run([sys.executable, "/content/cupti_matmul.py"], capture_output=True, text=True, env=env, timeout=900)
if p.returncode != 0:
print(f" CUPTI run failed (exit {p.returncode}); last lines of its stderr:")
print(textwrap.indent("\n".join(p.stderr.strip().splitlines()[-8:]), " "))
return f"CUPTI backend failed (exit {p.returncode}); Nsight Compute run took {ncu_wall:.0f}s"
res = next((json.loads(line) for line in reversed(p.stdout.strip().splitlines()) if line.startswith('{"wall"')), None)
if res is None:
return f"CUPTI run printed no result; Nsight Compute run took {ncu_wall:.0f}s"
cu = pd.DataFrame(res["rows"]).set_index("n")
print(f" {'n':>5s} {'ncu ms':>8s} {'cupti ms':>9s} {'ratio':>6s} {'ncu RSD%':>9s} {'cupti RSD%':>11s}")
for n in ncu_df.index:
a, b = ncu_df.loc[n, "AvgValue"] / 1e6, cu.loc[n, "AvgValue"] / 1e6
print(f" {n:5d} {a:8.3f} {b:9.3f} {b / a:6.2f} {ncu_df.loc[n, 'RelativeStdDevPct']:9.1f} {cu.loc[n, 'RelativeStdDevPct']:11.1f}")
print(f"\n wall time for 4 sizes x 10 runs: Nsight Compute {ncu_wall:.1f}s, CUPTI {res['wall']:.1f}s"
f" ({ncu_wall / res['wall']:.1f}x)")
print(" The two tools are mutually exclusive in one process and Nsight Compute cannot be unloaded, so")
print(" CUPTI runs in a child process with NSPY_NCU_INIT_AT_IMPORT=0. CUPTI records one execution per")
print(" launch, with no replay, no cache flush and no clock control: cheaper, and warmer caches.")
return f"CUPTI {ncu_wall / res['wall']:.1f}x faster to profile; durations within {abs(1 - cu.AvgValue.div(ncu_df.AvgValue)).max():.0%}"
cupti_vs_ncu()
CUPTI 后端是 1.1 的另一个主要特性：通过 CUPTI 的活动 API 测量内核持续时间，而无需 Nsight Compute。这两个工具在单个进程内互斥。Nsight Compute 一旦激活便无法卸载，因此我们编写一个小脚本，在导入 nsight 之前将 NSPY_NCU_INIT_AT_IMPORT 设为 0，激活 CUPTI 工具，对相同的四种矩阵大小进行性能分析，并在路径中包含 cupti-python 目录的子进程中运行它。CUPTI 仅支持持续时间指标，每次启动记录一次执行，不进行重放、缓存刷新或时钟控制，因此我们将性能分析的持续时间和实际耗时与 Nsight Compute 下的同一基准测试进行比较。
@section("10. Keep the evidence: CSVs and the full Nsight Compute report")
def evidence():
files = sorted(glob.glob("/content/nspy_thermal_off/*"))
for f in files:
print(f" {os.path.basename(f):60s} {os.path.getsize(f) / 1024:9.1f} KB")
import ncu_report
rep_path = next((f for f in files if f.endswith(".ncu-rep")), None)
if rep_path is None:
raise RuntimeError("no .ncu-rep in /content/nspy_thermal_off/ (step 8 did not complete)")
rep = ncu_report.load_report(rep_path)
actions = [rep.range_by_idx(i).action_by_idx(j) for i in range(rep.num_ranges())
for j in range(rep.range_by_idx(i).num_actions())]
a0 = actions[0]
print(f"\n the .ncu-rep holds {len(actions)} profiled kernel launches; the first is")
print(f" {a0.name()}")
print(f" gpu__time_duration.sum = {a0['gpu__time_duration.sum'].value():,.0f} {a0['gpu__time_duration.sum'].unit()}")
print(f" {len(a0.metric_names())} metrics available in the report for this launch")
print("\n output_csv=True writes profiled_data (one row per run) and processed_data (the aggregated")
print(" DataFrame). The .ncu-rep opens in the Nsight Compute GUI for source-level analysis.")
return f"{len(actions)} launches in the report, {len(a0.metric_names())} metrics each"
evidence()
每次被分析的调用都会留下证据。使用输出前缀和 output_csv 时，该目录中会保存 Nsight Compute 报告、其日志、每次运行的 profiled_data CSV 以及聚合后的 processed_data CSV。我们列出这些文件，并用 ncu_report（Nsight Compute 的 Python 报告接口，Nsight Python 自身也使用它）打开报告：它会列出被分析的 kernel 启动，每次启动都带有收集到的所有指标。同一个 .ncu-rep 文件也可以在 Nsight Compute 界面中打开，进行源码级分析。
banner("SUMMARY")
for name, res in RESULTSitems():
print(f" {name:<76s} {res}")
print("""
What to carry over
- import nsight first; it loads Nsight Compute into the process before CUDA starts.
- Annotate exactly the work you mean to measure; one kernel per region unless you pass
combine_kernel_metrics.
- Decide cache and clock control on purpose: nsight-python flushes caches and leaves clocks
unlocked by default, which is not what a plain ncu run does.
- Use runs and the CI95 / RelativeStdDevPct columns; one run tells you nothing about noise.
- After normalize_against, only AvgValue is a ratio.
Where to go next
- Triton and cuTile kernels: examples/07_triton_minimal.py and examples/12_cutile.py.
- Plots: @nsight.analyze.plot(...) on top of @nsight.analyze.kernel, with row/col panels.
- Docs: docs.nvidia.com/nsight-python Code: github.com/NVIDIA/nsight-python
""")
摘要会打印每一步的一行结果，以及值得保留的习惯：在任何东西初始化 CUDA 之前先导入 nsight，只标注你真正想测量的工作，有意识地选择缓存和时钟控制，使用足够多的运行次数并阅读置信区间，并记住只有 AvgValue 会被归一化。最后还会给出仓库中 Triton 和 cuTile 示例以及绘图装饰器的指引。
总之，Nsight Python 将 kernel 性能分析从单独的命令行会话变成了被测代码中的几行：一个装饰器、一个标注，以及一个携带测量值、其离散程度和上下文的 DataFrame。在 Colab T4 上实际操作它也表明，这些便利背后隐藏着会改变数字的决策。性能分析器必须比 Colab 自带的版本更新，并且必须在 CUDA 启动之前加载。缓存刷新和时钟锁定的默认值与普通 ncu 运行相反；时钟列报告的是显卡的额定值，而不是 kernel 实际运行时的时钟；归一化只除以平均值。每个被标注的区域都预期恰好包含一个 kernel，除非你说明如何合并它们。一旦这些选择被明确，同样的几行代码就能回答实践中重要的问题：计时有多稳定、是什么限制了 kernel、融合实际节省了什么，以及像新的 CUPTI 后端这样更便宜的性能分析器为降低开销放弃了什么。
查看完整代码。所有功劳归于该项目的研究者。另外，欢迎在Twitter上关注我们，别忘了加入我们的15万+机器学习 SubReddit并订阅我们的新闻通讯。等等！你在 Telegram 上吗？现在你也可以在 Telegram 上加入我们。
文章NVIDIA Nsight Python 教程：如何分析 GPU Kernel首次出现在MarkTechPost上。
来源：MarkTechPost · marktechpost.com
来源：RadarAI · aihot.news