在 OpenWrt 编译 ConnectX-5 驱动以及创建软件包的尝试

开篇废话

在上篇文章「一次适配 OpenWrt 的尝试」中,我们尝试了如何简单使用相同芯片家族家族现成的资源来编译 OpenWrt,不过 偷驱动文件 部分没有介绍,当时没有折腾到这一步,这里我们来继续介绍此部分。

本文中将介绍我是如何在 OpenWrt 适配起夜级的 ConnectX-5 网卡并且制作内核包 kmod-mlnx-ofed 的。

项目已经开源至 https://github.com/makeding/kmod-mlnx-ofed 如果您只需要固件请直接按照 readme 克隆然后勾选即可。

写下来感觉吧并非教程向,简单带过的地方太多了

基础知识简单带入

编译 OpenWrt

请先看官方 wiki https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem

这里我们还是要补充一下背景知识,先看看一般向的编译过程:

  • 准备一台 debian 硬盘至少给个 32G 核心越多越好 内存越大越好 然后需要一个健壮的网络,使你可以下载各种文件。

我这里使用了 Ryzen 5 2600(6核心12线程,实际使用 -j10) + 16GB内存 凑合用

  • apt 安装各种开发工具链,本文不复制粘贴,请看上面提到的文章
  • 克隆,我写这篇文章最新发布的是 23.05.4 当然不怕死上 snapshot 也可以
git clone https://git.openwrt.org/openwrt/openwrt.git -b 23.05.4
  • 拉取 & 安装 ./feeds.conf.default 里面提到了的仓库至 ./feeds
./scripts/feeds update -a
./scripts/feeds install -a

当你觉得 feeds 有问题导致包是炸的时候,可以尝试 make clean 并且删除 ./feeds 说不定有奇迹,比如:
Not overriding core package" ‘xxx’ “use -f to force WARNING: Makefile ‘package/feeds/xxxxx/xxxxx/Makefile’ has a dependency on ’libpcre’, which does not exist
ERROR: module ‘/home/h/openwrt/build_dir/target-x86_64_musl/xxxxxx’ is missing.

  • 编译 make -j

编写 Makefile

这里也是简单介绍一下,了解更多请搜索别的教程

这里我还是推荐直接找官方仓库抄抄改改,比如我在下文找到的类似的项目,但是就是软件版本低了,就很适合我们练手抄袭
总的来说我们需要什么类型的包,如果没有类似的项目的话就去官方仓库找个同类型的包即可开始:
比如我这次是需要 kmod 里面加个网卡的包,那我们就很自然地找到了:
https://github.com/openwrt/openwrt/blob/9dbd45c18702cdd55fcfb0f71dc505afa1ff64d6/package/kernel/r8125/Makefile

满足:

  1. 是内核拓展包
  2. 是网卡
    是学生

所以在找不到的情况下可以拿这这种项目为基础开始写,不需要改太多东西。

然后 Makefile 细节内容我就不解释了,请配合 GPT 以及高中英语老师食用,他们更专业。

不要尝试从头开始写 Makefile,因为这个是描述编译过程,你是无法自己从头开始写的,必须套模版

正文

寻找现成的类似项目

首先还是先看看有没有类似的项目,如果可以直接用的话就不需要自己折腾了

这里我找到了 https://github.com/allegro0132/Openwrt-mlnx-ofed 的项目,上次更新是3年前,看了下 Makefile 看上去还算比较好操作,我们只需要把 Makefile 里面的源码换成最新版本即可,站在巨人的肩膀上还是方便一点。

这里「好操作」的意思是 Nvidia 是光明正大开源的,不需要找什么野路子就可以下载到源码,具体做法参考下文。

寻找驱动源码

首先 Google 找到 Nvidia 的 MLNX OFED 驱动源码下载页面: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

当然如果你想移植的驱动没有找到厂商公开提供的下载链接。。。。
那就看您是否能凭本事什么神秘 ftp 服务器之类的找到相关文件了。
不过即使找到了也只能适配很久以前的 kernel ,可能会缺目前比较流行的 epbf 什么的,看个人取舍吧。

然后 Nvidia 这个驱动发布页面就让我非常迷惑。
首先 SRC- 里面没有源码,而是 .rpm 文件:
https://content.mellanox.com/ofed/MLNX_OFED-5.8-5.1.1.2/MLNX_OFED_SRC-5.8-5.1.1.2.tgz

❯ tree MLNX_OFED_SRC-5.8-5.1.1.2/
MLNX_OFED_SRC-5.8-5.1.1.2/
├── BUILD_ID
├── LICENSE
├── RPMS
├── SRPMS
│   ├── dapl-2.1.10.1.mlnx-OFED.4.9.0.1.5.58511.src.rpm
│   ├── ibdump-6.0.0-1.58511.src.rpm
│   ├── ibsim-0.10-1.58511.src.rpm
│   ├── iser-5.8-OFED.5.8.5.1.1.1.src.rpm
│   ├── isert-5.8-OFED.5.8.5.1.1.1.src.rpm
│   ├── kernel-mft-4.22.1-417.src.rpm
│   ├── knem-1.1.4.90mlnx3-OFED.23.10.0.2.1.1.src.rpm
│   ├── libvma-9.7.2-1.src.rpm
│   ├── libxlio-2.0.7-1.src.rpm
│   ├── mlnx-ethtool-5.18-1.58511.src.rpm
│   ├── mlnx-iproute2-5.19.0-1.58511.src.rpm
│   ├── mlnx-nfsrdma-5.8-OFED.5.8.5.1.1.1.src.rpm
│   ├── mlnx-nvme-5.8-OFED.5.8.5.1.1.1.src.rpm
│   ├── mlnx-ofa_kernel-5.8-OFED.5.8.5.1.1.1.src.rpm
│   ├── mlnx-tools-5.8.0-1.lts.58511.src.rpm
│   ├── mlx-steering-dump-1.0.0-0.58511.src.rpm
│   ├── mpi-selector-1.0.3-1.58511.src.rpm
│   ├── mpitests-3.2.20-de56b6b.58511.src.rpm
│   ├── mstflint-4.16.1-2.58511.src.rpm
│   ├── multiperf-3.0-3.0.58511.src.rpm
│   ├── ofed-docs-5.8-OFED.5.8.5.1.1.src.rpm
│   ├── ofed-scripts-5.8-OFED.5.8.5.1.1.src.rpm
│   ├── openmpi-4.1.5a1-1.58511.src.rpm
│   ├── openvswitch-2.17.2-1.58511.src.rpm
│   ├── perftest-4.5-0.18.gfcddfe0.58511.src.rpm
│   ├── rdma-core-58mlnx43-1.58511.src.rpm
│   ├── rshim-2.0.19-0.gbf7f1f2.src.rpm
│   ├── sockperf-3.10-0.git5ebd327da983.58511.src.rpm
│   ├── srp-5.8-OFED.5.8.5.1.1.1.src.rpm
│   ├── ucx-1.14.0-1.58511.src.rpm
│   └── xpmem-2.7.3-1.58511.src.rpm
├── common.pl
├── install.pl
└── uninstall.sh

3 directories, 36 files

然后 SRC-debian 里面躺着源码:

https://content.mellanox.com/ofed/MLNX_OFED-5.8-5.1.1.2/MLNX_OFED_SRC-debian-5.8-5.1.1.2.tgz

❯ tree MLNX_OFED_SRC-debian-5.8-5.1.1.2
├── BUILD_ID
├── DEBS
├── LICENSE
├── SOURCES
│   ├── dapl_2.1.10.1.mlnx.orig.tar.gz
│   ├── ibdump_6.0.0.orig.tar.gz
│   ├── ibsim_0.10.orig.tar.gz
│   ├── iser_5.8.orig.tar.gz
│   ├── isert_5.8.orig.tar.gz
│   ├── kernel-mft_4.22.1.orig.tar.gz
│   ├── knem_1.1.4.90mlnx3.orig.tar.gz
│   ├── libvma_9.7.2.orig.tar.gz
│   ├── libxlio_2.0.7.orig.tar.gz
│   ├── mlnx-ethtool_5.18.orig.tar.gz
│   ├── mlnx-iproute2_5.19.0.orig.tar.gz
│   ├── mlnx-nfsrdma_5.8.orig.tar.gz
│   ├── mlnx-nvme_5.8.orig.tar.gz
│   ├── mlnx-ofed-kernel_5.8.orig.tar.gz
│   ├── mlnx-tools_5.8.0.orig.tar.gz
│   ├── mlx-steering-dump_1.0.0.orig.tar.gz
│   ├── mpitests_3.2.20.orig.tar.gz
│   ├── mstflint_4.16.1.orig.tar.gz
│   ├── ofed-scripts_5.8.orig.tar.gz
│   ├── openmpi_4.1.5a1.orig.tar.gz
│   ├── openvswitch_2.17.2.orig.tar.gz
│   ├── perftest_4.5.orig.tar.gz
│   ├── rdma-core_58mlnx43.orig.tar.gz
│   ├── rshim_2.0.19.orig.tar.gz
│   ├── sockperf_3.10.orig.tar.gz
│   ├── srp_5.8.orig.tar.gz
│   └── ucx_1.14.0.orig.tar.gz
├── common.pl
├── install.pl
└── uninstall.sh

3 directories, 32 files

然后就是压缩文件里面还有压缩文件,所以我们要先解压(我首先被第一个 SRC-debian 坑了很久,直到下载到本地才发现问题。

总之能下载到就不错了,总比没有强

最初的东西收集完毕了,然后我们可以找其他的编译教程配套食用,不然我们可能不知道编译参数还有大坑什么的。

这里我找了这个教程:https://insujang.github.io/2020-01-25/building-mellanox-ofed-from-source/

(然后发现教程第一步就是教你如何正确下载到源码)

接下来我们来正式修改 makeFile 也就是造软件包

改名字

以之前我们提到的 r8125/Makefile 为例子,我们需要改成自己喜欢的名字:

include $(TOPDIR)/rules.mk

PKG_NAME:=r8125
PKG_VERSION:=9.013.02
PKG_RELEASE:=2

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.bz2
PKG_SOURCE_URL:=https://github.com/Noltari/rtl8125/releases/download/$(PKG_VERSION)
PKG_HASH:=d36410ee99c956f250d9cd08340d8c36567d190f420a8ee128ff6e51225aac0c

PKG_BUILD_PARALLEL:=1
PKG_LICENSE:=GPLv2
PKG_MAINTAINER:=Alvaro Fernandez Rojas <noltari@gmail.com>

include $(INCLUDE_DIR)/kernel.mk
include $(INCLUDE_DIR)/package.mk

define KernelPackage/r8125
  SUBMENU:=Network Devices
  TITLE:=Realtek RTL8125 PCI 2.5 Gigabit Ethernet driver
  DEPENDS:=@PCI_SUPPORT
  FILES:=$(PKG_BUILD_DIR)/src/r8125.ko
  AUTOLOAD:=$(call AutoProbe,r8125)
  PROVIDES:=kmod-r8169
  VARIANT:=regular
endef

define KernelPackage/r8125-rss
$(call KernelPackage/r8125)
  TITLE+= (RSS)
  VARIANT:=rss
endef

ifeq ($(BUILD_VARIANT),rss)
  PKG_MAKE_FLAGS += ENABLE_RSS_SUPPORT=y
endif

define Build/Compile
	+$(KERNEL_MAKE) $(PKG_JOBS) \
		$(PKG_MAKE_FLAGS) \
		M="$(PKG_BUILD_DIR)/src" \
		modules
endef

$(eval $(call KernelPackage,r8125))
$(eval $(call KernelPackage,r8125-rss))

这里我们可以直接批量替换,把 r8125 改成自己想要的名字,比如我改成了 mlnx-ofed

然后再把多余的 $(eval $(call KernelPackage,r8125-rss)) 删了
实际我们的软件包会因为最后 call 了什么而有自己的 prefix 比如说,KernelPackage,r8125 那么就会自动附加上 kmod- 最后包的名称会变成 kmod-r8125
具体逻辑在 includes/kernel.mk 可以找到

然后 TITLE 什么的自己乱改就是了,看得顺眼就行,有问题多问 GPT

指定源码下载 URL 以及校验

从这里开始我们只拿关键行出来解释 这部分应该非常好理解,就是软件从哪里来?
我们是从 Nvidia 官方渠道下载的,因此我们的 URL 直接指定即可:

PKG_VERSION:=9.013.02
PKG_RELEASE:=2
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.bz2
PKG_SOURCE_URL:=https://github.com/Noltari/rtl8125/releases/download/$(PKG_VERSION)
PKG_HASH:=d36410ee99c956f250d9cd08340d8c36567d190f420a8ee128ff6e51225aac0c

这里的 PKG_VERSION 也就是目前这包的版本号,我们一般遵循上游的,给什么用什么。
因此可以看到 PKG_SOURCE / PKG_SOURCE_URL 也是遵循版本号的,这样写起来也方便。
PKG_RELEASE:=2 的 2 的意思是这个软件在这个版本的 OpenWrt 版本的修改版本号,我们如果是第一次发布的话就用 1 即可,后面加了 patch 什么的再改成 2 3 4 5 。 然后 PKG_HASH 需要我们手动下载文件算出来,当然不算出来的话,在后面 make V=s 中我们还是可以看到报错然后知道现在的 HASH 的。

手动生成请用 sha256 < xxx.tar.gz 不过 md5 好像也可以,安全一点更好。

校验主要是为了避免供应链投毒,因为我们不可能保证上游是可信的,但是我们可以基本保证在编译这个包时刻的 HASH 是可信的。

prepare

其实刚才的步骤已经是准备完源码了,OpenWrt 会帮你自动解压下载完并且校验通过的文件到 $(PKG_BUILD_DIR) ,无需做此处的操作。

不过我🚪亲爱的英伟达采取了压缩包里面的压缩包才是本体,因此需要我们手动解压:
我们重定义个 Build/Prepare 步骤:

define Build/Prepare
	$(TAR) -xzf $(DL_DIR)/$(PKG_SOURCE) -C $(PKG_BUILD_DIR)
	$(TAR) -zxf $(PKG_BUILD_DIR)/MLNX_OFED_SRC*/SOURCES/mlnx-ofed-kernel_*.tar.gz -C $(PKG_BUILD_DIR)
endef

此操作第一步是解压下载完的源码包,然后是解压源码包的 /MLNX_OFED_SRC*/SOURCES/mlnx-ofed-kernel_*.tar.gz 文件,也就是我们需要的文件。

这里我们可以偷偷塞个 ls $(PKG_BUILD_DIR) 什么的方便偷窥一下文件是否正常解压,目录对不对。

configure

这个步骤也不用说了,经常编译过的群友都知道是什么 这里我们由于之前解压的问题,所以要先 cd 一下:

define Build/Configure
	cd $(PKG_BUILD_DIR)/mlnx-ofed-kernel* && ./configure --xxxx(你的参数)
endef

当然我还没试过不定义 Build/Configure 会不会智能执行 ./configure ,懒得试。

然后 mlnx-ofed 这个检查过程花了10分钟以上的时间,怪不得编译东西是浪费生命

compile

这部分也无需多言,还是定义过程,然后塞点奇怪命令就可以了:

define Build/Compile
	cd $(PKG_BUILD_DIR)/mlnx-ofed-kernel* && $(MAKE) $(mlnx_MAKEOPTS) -j4 kernel
endef

这里我使用了一些奇怪的参数以及 -j4 进行,之前用 -j 把我宿主机都编译冒烟了,慎用。

值得注意的是根据你包的编译顺序之类的,可能需要禁用 PKG_BUILD_PARALLEL 也就是并行编译的选项,这里如果一直失败可以看看改成 0。

收集产物文件

编译完了,我们需要收集文件然后拿来用了,这里我们只需要修改:

define KernelPackage/r8125
	FILES:=$(PKG_BUILD_DIR)/src/r8125.ko

里面的文件为你想要安装到系统的,也就是最终产物文件即可。

比较懒的可以把全家都带进去,比如:

FILES:= `find $(PKG_BUILD_DIR)/ \( -name "*.ko" -o -name "*.ko.gz" \)`

这其实是有风险的,因为上游可能在 src 里面塞个 .ko 然后就炸了,然后 autoload 里面的话也麻烦了,因为即使全家都带进去了还是要手动加载。

开机自动加载

然后我们需要设置此驱动开机自动加载,这里我们也是一样安排:

define KernelPackage/r8125
	AUTOLOAD:=$(call AutoProbe,r8125)

这样就加载了 r8125 的驱动,这里我们改成我们现在需要的:

AUTOLOAD:=$(call AutoLoad,35,mlx5_core mlx4_core ib_ipoib)

其他

这里我们没有介绍安装时的行为以及卸载之前的行为,还有如何给源码打 patch
不过您都看到这了,动手搜索一下应该很快就有答案了。

编译

这部分又是一门技术体力活,比如这玩意 configure 时间长达10分钟,可能会让你以为东西是坏的(

放文件

OpenWrt 创建包还是很简单的,我们往 ./package 随便放就行:

mkdir package/huggy/kmod-mlnx-ofed/ -p
cp xxxxx/Makefie package/huggy/kmod-mlnx-ofed/

这里的目录架构是这样的:

#Put this source to 'package/yourfolder' folder of OpenWRT/LEDE SDK
#Build(make menuconfig, make defconfig, make)

因此 package 里面的文件夹的改成你喜欢的名字就行,然后再创建个文件夹才是真的名字,这里我取名成了 kmod-mlnx-ofed 感觉规范了点

单独编译

我们在调试的时候直接跑 make 太浪费时间了,其实是可以单独编译的:

make package/huggy/kmod-mlnx-ofed/compile V=s

比如这是我的用法, V=s 查看编译详细日志什么的。

缩进问题

Makefile 是吃缩进的,如果出现了这样的问题:

Makefile:119: *** missing separator.  Stop.

那就是缩进炸了,反正文件行数都告诉你了,使用 tab 缩进,然后保证上下都一样即可。

Linux 编译时交互

开了一些选项后可能会卡在:

 make[1] world
 make[2] package/cleanup
 make[2] target/compile
 make[3] -C target/linux compile
    ERROR: target/linux failed to build.
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
make: *** [/home/h/openwrt/include/toplevel.mk:248: world] Error 1

V=s 后跳出这种提示,看不懂的话默认回车咯?:

make[5]: Entering directory '/home/h/openwrt/build_dir/target-x86_64_musl/linux-x86_64/linux-6.6.44'
  SYNC    include/config/auto.conf.cmd
.config:1170:warning: override: CPU_FREQ_DEFAULT_GOV_SCHEDUTIL changes choice state
.config:3908:warning: override: M686 changes choice state
*
* Restart config...
*
*
* Networking options
*
Packet socket (PACKET) [Y/n/m/?] y
  Packet: sockets monitoring interface (PACKET_DIAG) [N/m/y/?] n
Unix domain sockets (UNIX) [Y/n/?] y
  UNIX: socket monitoring interface (UNIX_DIAG) [N/m/y/?] n
Transport Layer Security support (TLS) [N/m/y/?] n
Transformation user configuration interface (XFRM_USER) [N/m/y/?] n
PF_KEY sockets (NET_KEY) [N/m/y/?] n
SMC socket protocol family (SMC) [N/m/?] (NEW) XDP sockets (XDP_SOCKETS) [N/y/?] n
TCP/IP networking (INET) [Y/n/?] y
  IP: multicasting (IP_MULTICAST) [Y/n/?] y
  IP: advanced router (IP_ADVANCED_ROUTER) [Y/n/?] y
    FIB TRIE statistics (IP_FIB_TRIE_STATS) [N/y/?] n
    IP: policy routing (IP_MULTIPLE_TABLES) [Y/n/?] y
    IP: equal cost multipath (IP_ROUTE_MULTIPATH) [Y/n/?] y
    IP: verbose route monitoring (IP_ROUTE_VERBOSE) [Y/n/?] y
  IP: kernel level autoconfiguration (IP_PNP) [N/y/?] n
  IP: tunneling (NET_IPIP) [M/n/y/?] m
  IP: GRE demultiplexer (NET_IPGRE_DEMUX) [N/m/y/?] n
  IP: multicast routing (IP_MROUTE) [Y/n/?] y
    IP: multicast policy routing (IP_MROUTE_MULTIPLE_TABLES) [Y/n/?] y
    IP: PIM-SM version 1 support (IP_PIMSM_V1) [Y/n/?] y
    IP: PIM-SM version 2 support (IP_PIMSM_V2) [Y/n/?] y
  IP: TCP syncookie support (SYN_COOKIES) [Y/n/?] y
  Virtual (secure) IP: tunneling (NET_IPVTI) [N/m/y/?] n
  IP: Foo (IP protocols) over UDP (NET_FOU) [N/m/y/?] n
  IP: FOU encapsulation of IP tunnels (NET_FOU_IP_TUNNELS) [N/y/?] n
  IP: AH transformation (INET_AH) [N/m/y/?] n
  IP: ESP transformation (INET_ESP) [N/m/y/?] n
  IP: IPComp transformation (INET_IPCOMP) [N/m/y/?] n
  INET: Source port perturbation table size (as power of 2) (INET_TABLE_PERTURB_ORDER) [16] 16
  INET: socket monitoring interface (INET_DIAG) [M/n/y/?] m
    UDP: socket monitoring interface (INET_UDP_DIAG) [M/n/?] m
    RAW: socket monitoring interface (INET_RAW_DIAG) [M/n/?] m
    INET: allow privileged process to administratively close sockets (INET_DIAG_DESTROY) [N/y/?] n
  TCP: MD5 Signature Option support (RFC2385) (TCP_MD5SIG) [N/y/?] n
  MPTCP: Multipath TCP (MPTCP) [N/y/?] n
Security Marking (NETWORK_SECMARK) [N/y/?] n
Timestamping in PHY devices (NETWORK_PHY_TIMESTAMPING) [N/y/?] n
The Reliable Datagram Sockets Protocol (RDS) [N/m/y/?] n
Asynchronous Transfer Mode (ATM) (ATM) [N/m/y/?] n
802.1d Ethernet Bridging (BRIDGE) [Y/n/m/?] y
  IGMP/MLD snooping (BRIDGE_IGMP_SNOOPING) [Y/n/?] y
  VLAN filtering (BRIDGE_VLAN_FILTERING) [Y/n/?] y
  MRP protocol (BRIDGE_MRP) [N/y/?] n
  CFM protocol (BRIDGE_CFM) [N/y/?] n
802.1Q/802.1ad VLAN Support (VLAN_8021Q) [Y/n/m/?] y
  GVRP (GARP VLAN Registration Protocol) support (VLAN_8021Q_GVRP) [N/y/?] n
  MVRP (Multiple VLAN Registration Protocol) support (VLAN_8021Q_MVRP) [N/y/?] n
ANSI/IEEE 802.2 LLC type 2 Support (LLC2) [N/m/y/?] n
Appletalk protocol support (ATALK) [N/m/y/?] n
CCITT X.25 Packet Layer (X25) [N/m/y/?] n
LAPB Data Link Driver (LAPB) [N/m/y/?] n
Phonet protocols family (PHONET) [N/m/y/?] n
Data Center Bridging support (DCB) [N/y/?] n
DNS Resolver support (DNS_RESOLVER) [N/m/y/?] n
B.A.T.M.A.N. Advanced Meshing Protocol (BATMAN_ADV) [N/m/y/?] n
Open vSwitch (OPENVSWITCH) [N/m/?] n
Virtual Socket protocol (VSOCKETS) [Y/n/m/?] y
  Virtual Sockets monitoring interface (VSOCKETS_DIAG) [N/m/y/?] n
  Virtual Sockets loopback transport (VSOCKETS_LOOPBACK) [Y/n/m/?] y
  VMware VMCI transport for Virtual Sockets (VMWARE_VMCI_VSOCKETS) [Y/n/m/?] y
  virtio transport for Virtual Sockets (VIRTIO_VSOCKETS) [N/m/y/?] n
Hyper-V transport for Virtual Sockets (HYPERV_VSOCKETS) [N/m/y/?] n
NETLINK: socket monitoring interface (NETLINK_DIAG) [N/m/y/?] n
High-availability Seamless Redundancy (HSR & PRP) (HSR) [N/m/y/?] n
Switch (and switch-ish) device support (NET_SWITCHDEV) [Y/n/?] y
L3 Master device support (NET_L3_MASTER_DEV) [N/y/?] n
Qualcomm IPC Router support (QRTR) [N/m/y/?] n
NCSI interface support (NET_NCSI) [N/y/?] n
Use percpu variables to maintain network device refcount (PCPU_DEV_REFCNT) [N/y/?] n
Maximum number of fragments per skb_shared_info (MAX_SKB_FRAGS) [17] 17
Network priority cgroup (CGROUP_NET_PRIO) [N/y/?] n
Network classid cgroup (CGROUP_NET_CLASSID) [N/y/?] n
enable BPF STREAM_PARSER (BPF_STREAM_PARSER) [N/y/?] n
*
* NVME Support
*
NVM Express block device (BLK_DEV_NVME) [Y/n/m/?] y
NVMe multipath support (NVME_MULTIPATH) [Y/n/?] y
NVMe verbose error reporting (NVME_VERBOSE_ERRORS) [N/y/?] n
NVMe hardware monitoring (NVME_HWMON) [Y/n/?] y
NVM Express over Fabrics RDMA host driver (NVME_RDMA) [N/m/?] (NEW) y 

CONFIG_NVME_RDMA:

This provides support for the NVMe over Fabrics protocol using
the RDMA (Infiniband, RoCE, iWarp) transport.  This allows you
to use remote block devices exported using the NVMe protocol set.

To configure a NVMe over Fabrics controller use the nvme-cli tool
from https://github.com/linux-nvme/nvme-cli.

If unsure, say N.

Symbol: NVME_RDMA [=n]
Type  : tristate
Defined at drivers/nvme/host/Kconfig:47
  Prompt: NVM Express over Fabrics RDMA host driver
  Depends on: INFINIBAND [=m] && INFINIBAND_ADDR_TRANS [=y] && BLOCK [=y]
  Location:
    -> Device Drivers
      -> NVME Support
        -> NVM Express over Fabrics RDMA host driver (NVME_RDMA [=n])
Selects: NVME_FABRICS [=n] && SG_POOL [=y]



NVM Express over Fabrics RDMA host driver (NVME_RDMA) [N/m/?] (NEW) y

CONFIG_NVME_RDMA:

This provides support for the NVMe over Fabrics protocol using
the RDMA (Infiniband, RoCE, iWarp) transport.  This allows you
to use remote block devices exported using the NVMe protocol set.

To configure a NVMe over Fabrics controller use the nvme-cli tool
from https://github.com/linux-nvme/nvme-cli.

If unsure, say N.

Symbol: NVME_RDMA [=n]
Type  : tristate
Defined at drivers/nvme/host/Kconfig:47
  Prompt: NVM Express over Fabrics RDMA host driver
  Depends on: INFINIBAND [=m] && INFINIBAND_ADDR_TRANS [=y] && BLOCK [=y]
  Location:
    -> Device Drivers
      -> NVME Support
        -> NVM Express over Fabrics RDMA host driver (NVME_RDMA [=n])
Selects: NVME_FABRICS [=n] && SG_POOL [=y]



NVM Express over Fabrics RDMA host driver (NVME_RDMA) [N/m/?] (NEW) y

CONFIG_NVME_RDMA:

This provides support for the NVMe over Fabrics protocol using
the RDMA (Infiniband, RoCE, iWarp) transport.  This allows you
to use remote block devices exported using the NVMe protocol set.

To configure a NVMe over Fabrics controller use the nvme-cli tool
from https://github.com/linux-nvme/nvme-cli.

If unsure, say N.

Symbol: NVME_RDMA [=n]
Type  : tristate
Defined at drivers/nvme/host/Kconfig:47
  Prompt: NVM Express over Fabrics RDMA host driver
  Depends on: INFINIBAND [=m] && INFINIBAND_ADDR_TRANS [=y] && BLOCK [=y]
  Location:
    -> Device Drivers
      -> NVME Support
        -> NVM Express over Fabrics RDMA host driver (NVME_RDMA [=n])
Selects: NVME_FABRICS [=n] && SG_POOL [=y]



NVM Express over Fabrics RDMA host driver (NVME_RDMA) [N/m/?] (NEW) n
NVM Express over Fabrics FC host driver (NVME_FC) [N/m/y/?] n
NVM Express over Fabrics TCP host driver (NVME_TCP) [N/m/y/?] n
NVM Express over Fabrics In-Band Authentication (NVME_AUTH) [N/y/?] n
*
* InfiniBand support
*
InfiniBand support (INFINIBAND) [M/n/y/?] m
  InfiniBand userspace MAD support (INFINIBAND_USER_MAD) [M/n/?] m
  InfiniBand userspace access (verbs and CM) (INFINIBAND_USER_ACCESS) [M/n/?] m
    InfiniBand on-demand paging support (INFINIBAND_ON_DEMAND_PAGING) [Y/n/?] y
  RDMA/CM (INFINIBAND_ADDR_TRANS) [Y/n/?] y
  Amazon Elastic Fabric Adapter (EFA) support (INFINIBAND_EFA) [N/m/?] (NEW) 
  Alibaba Elastic RDMA Adapter (ERDMA) support (INFINIBAND_ERDMA) [N/m/?] (NEW) 
  Mellanox ConnectX HCA support (MLX4_INFINIBAND) [N/m/?] (NEW) 
  Mellanox 5th generation network adapters (ConnectX series) support (MLX5_INFINIBAND) [M/n/?] m
  Mellanox HCA support (INFINIBAND_MTHCA) [N/m/?] (NEW) 
  Emulex One Connect HCA support (INFINIBAND_OCRDMA) [N/m/?] (NEW) 
  Verbs support for Cisco VIC (INFINIBAND_USNIC) [N/m/?] (NEW) 
  VMware Paravirtualized RDMA Driver (INFINIBAND_VMWARE_PVRDMA) [N/m/?] (NEW) 
  RDMA verbs transport library (INFINIBAND_RDMAVT) [N/m/?] (NEW) 
  Software RDMA over Ethernet (RoCE) driver (RDMA_RXE) [N/m/?] (NEW) 
  Software RDMA over TCP/IP (iWARP) driver (RDMA_SIW) [N/m/?] (NEW) 
  IP-over-InfiniBand (INFINIBAND_IPOIB) [M/n/?] m
    IP-over-InfiniBand Connected Mode support (INFINIBAND_IPOIB_CM) [Y/n/?] y
    IP-over-InfiniBand debugging (INFINIBAND_IPOIB_DEBUG) [Y/n/?] y
      IP-over-InfiniBand data path debugging (INFINIBAND_IPOIB_DEBUG_DATA) [N/y/?] (NEW) 
  InfiniBand SCSI RDMA Protocol (INFINIBAND_SRP) [M/n/?] m
  iSCSI Extensions for RDMA (iSER) (INFINIBAND_ISER) [N/m/?] (NEW) 
  RTRS client module (INFINIBAND_RTRS_CLIENT) [N/m/?] (NEW) 
  RTRS server module (INFINIBAND_RTRS_SERVER) [N/m/?] (NEW) 
  Cornelis OPX VNIC support (INFINIBAND_OPA_VNIC) [N/m/?] (NEW) 
*
* Compile-time checks and compiler options
*
Debug information
  1. Disable debug information (DEBUG_INFO_NONE)
> 2. Rely on the toolchain's implicit default DWARF version (DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT)
  3. Generate DWARF Version 4 debuginfo (DEBUG_INFO_DWARF4)
  4. Generate DWARF Version 5 debuginfo (DEBUG_INFO_DWARF5)
choice[1-4?]: 2
Reduce debugging information (DEBUG_INFO_REDUCED) [N/y/?] n
Compressed Debug information
> 1. Don't compress debug information (DEBUG_INFO_COMPRESSED_NONE)
  2. Compress debugging information with zlib (DEBUG_INFO_COMPRESSED_ZLIB)
  3. Compress debugging information with zstd (DEBUG_INFO_COMPRESSED_ZSTD)
choice[1-3?]: 1
Produce split debuginfo in .dwo files (DEBUG_INFO_SPLIT) [N/y/?] n
Generate BTF type information (DEBUG_INFO_BTF) [Y/n/?] y
Generate BTF type information for kernel modules (DEBUG_INFO_BTF_MODULES) [Y/n/?] (NEW) 
  Allow loading modules with non-matching BTF type info (MODULE_ALLOW_BTF_MISMATCH) [N/y/?] (NEW) 
Provide GDB scripts for kernel debugging (GDB_SCRIPTS) [N/y/?] n
Warn for stack frames larger than (FRAME_WARN) [2048] 2048
Strip assembler-generated symbols during link (STRIP_ASM_SYMS) [Y/n/?] y
Generate readable assembler code (READABLE_ASM) [N/y/?] n
Install uapi headers to usr/include (HEADERS_INSTALL) [N/y/?] n
Enable full Section mismatch analysis (DEBUG_SECTION_MISMATCH) [N/y/?] n
Make section mismatch errors non-fatal (SECTION_MISMATCH_WARN_ONLY) [Y/n/?] y
Force all function address 64B aligned (DEBUG_FORCE_FUNCTION_ALIGN_64B) [N/y/?] n
Compile the kernel with frame pointers (FRAME_POINTER) [Y/?] y
Compile-time stack metadata validation (STACK_VALIDATION) [Y/n/?] y
Generate vmlinux.map file when linking (VMLINUX_MAP) [N/y/?] n
Force weak per-cpu definitions (DEBUG_FORCE_WEAK_PER_CPU) [N/y/?] n
*
* Tracers
*
Tracers (FTRACE) [Y/n/?] y
  Boot-time Tracing support (BOOTTIME_TRACING) [N/y/?] n
  Kernel Function Tracer (FUNCTION_TRACER) [N/y/?] n
  Trace max stack (STACK_TRACER) [N/y/?] n
  Interrupts-off Latency Tracer (IRQSOFF_TRACER) [N/y/?] n
  Scheduling Latency Tracer (SCHED_TRACER) [N/y/?] n
  Tracer to detect hardware latencies (like SMIs) (HWLAT_TRACER) [N/y/?] n
  OS Noise tracer (OSNOISE_TRACER) [N/y/?] n
  Timerlat tracer (TIMERLAT_TRACER) [N/y/?] n
  Memory mapped IO tracing (MMIOTRACE) [N/y/?] n
  Trace process context switches and events (ENABLE_DEFAULT_TRACERS) [N/y/?] n
  Trace syscalls (FTRACE_SYSCALLS) [N/y/?] n
  Create a snapshot trace buffer (TRACER_SNAPSHOT) [N/y/?] n
  Branch Profiling
  > 1. No branch profiling (BRANCH_PROFILE_NONE)
    2. Trace likely/unlikely profiler (PROFILE_ANNOTATED_BRANCHES)
  choice[1-2?]: 1
  Support for tracing block IO actions (BLK_DEV_IO_TRACE) [N/y/?] n
  Support BTF function arguments for probe events (PROBE_EVENTS_BTF_ARGS) [Y/n/?] (NEW) 
  Enable kprobes-based dynamic events (KPROBE_EVENTS) [Y/n/?] y
  Enable uprobes-based dynamic events (UPROBE_EVENTS) [N/y/?] n
  Synthetic trace events (SYNTH_EVENTS) [N/y/?] n
  User trace events (USER_EVENTS) [N/y/?] n
  Histogram triggers (HIST_TRIGGERS) [N/y/?] n
  Trace event injection (TRACE_EVENT_INJECT) [N/y/?] n
  Add tracepoint that benchmarks tracepoints (TRACEPOINT_BENCHMARK) [N/y/?] n
  Ring buffer benchmark stress tester (RING_BUFFER_BENCHMARK) [N/m/y/?] n
  Show eval mappings for trace events (TRACE_EVAL_MAP_FILE) [N/y/?] n
  Ring buffer startup self test (RING_BUFFER_STARTUP_TEST) [N/y/?] n
  Verify ring buffer time stamp deltas (RING_BUFFER_VALIDATE_TIME_DELTAS) [N/y/?] n
  Test module to create a preempt / IRQ disable delay thread to test latency tracers (PREEMPTIRQ_DELAY_TEST) [N/m/?] n
  Test module for in-kernel kprobe event generation (KPROBE_EVENT_GEN_TEST) [N/m/?] n

  CC      scripts/mod/empty.o
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/modpost.o
  CC      scripts/mod/devicetable-offsets.s
  HOSTCC  scripts/mod/file2alias.o
  HOSTCC  scripts/mod/sumversion.o
  HOSTCC  scripts/mod/symsearch.o
  HOSTLD  scripts/mod/modpost
  CC      kernel/bounds.s
  CC      arch/x86/kernel/asm-offsets.s
  DESCEND objtool
  INSTALL libsubcmd_headers

(常见于开启某个功能后 没有 defconfig 什么的)

总结

我们走完了大部分流程,可以发现其实都是换汤不换药的,还算简单 复制粘贴 OpenWrt 这套写起来还是挺舒服的。
然后本文还是省略掉了非常多的细节,有任何问题建议咨询 GPT
好久没有写一张图都没有的文章了,在这里附几张:
luci-package
windows menuconfig-1
menuconfig-2