slurm 23.11.0集群 debian 11.5 安装

阿里云国内75折回扣微信号：monov8

阿里云国际，腾讯云国际，低至75折。AWS 93折免费开户实名账号代冲值优惠多多微信号：monov8 飞机：@monov6

slurm 23.11.0集群 debian 11.5 安装

用途

Slurm(Simple Linux Utility for Resource Management http://slurm.schedmd.com/ )是开源的、具有容错性和高度可扩展的Linux集群超级计算系统资源管理和作业调度系统。超级计算系统可利用Slurm对资源和作业进行管理以避免相互干扰提高运行效率。所有需运行的作业无论是用于程序调试还是业务计算都可以通过交互式并行 srun 、批处理式 sbatch 或分配式 salloc 等命令提交提交后可以利用相关命令查询作业状态等。

架构

Slurm采用slurmctld服务守护进程作为中心管理器用于监测资源和作业为了提高可用性还可以配置另一个备份冗余管理器。各计算节点需启动slurmd守护进程以便被用于作为远程shell使用等待作业、执行作业、返回状态、再等待更多作业。slurmdbd(Slurm DataBase Daemon)数据库守护进程非必需建议采用也可以记录到纯文本中等可以将多个slurm管理的集群的记账信息记录在同一个数据库中。还可以启用slurmrestd(Slurm REST API Daemon)服务非必需该服务可以通过REST API与Slurm进行交互所有功能都对应的API。用户工具包含 srun 运行作业、 scancel 终止排队中或运行中的作业、 sinfo 查看系统状态、 squeue 查看作业状态、 sacct 查看运行中或结束了的作业及作业步信息等命令。 sview 命令可以图形化显示系统和作业状态可含有网络拓扑。 scontrol 作为管理工具可以监控、修改集群的配置和状态信息等。用于管理数据库的命令是 sacctmgr 可认证集群、有效用户、有效记账账户等。

xxxxxxxxxx10 1#192.168.86.134 - 192.168.86.1362cd3ssh-keygen 4sed -i ‘s/#PermitRootLogin prohibit-password/PermitRootLogin yes/’ /etc/ssh/sshd_config5systemctl restart ssh6passwd7#192.168.86.134 8ssh-copy-id slurm-head9ssh-copy-id slurm-db10ssh-copy-id slurm-computebash

SlurmDBD Node
- slurm-smd
- slurm-smd-slurmdbd
Head Node (slurmctld node)
- slurm-smd
- slurm-smd-slurmctld
Compute Nodes (slurmd node)
- slurm-smd
- slurm-smd-slurmd

192.168.86.134 slurm-head # 控制节点 Head Node
192.168.86.135 slurm-db #数据节点 SlurmDBD Node
192.168.86.136 slurm-compute #计算节点 Compute Nodes

注意如果是老服务器已有服务在运行可不改具体的hostname只要对应名称进行替换

修改主机名

# 192.168.86.134 
hostnamectl set-hostname  slurm-head
# 192.168.86.135
hostnamectl set-hostname  slurm-db
# 192.168.86.136
hostnamectl set-hostname  slurm-compute

修改/etc/hosts

#192.168.86.134 - 192.168.86.136
echo "192.168.86.134  slurm-head 
192.168.86.135  slurm-db
192.168.86.136  slurm-compute" >>/etc/hosts
cat /etc/hosts

修改debian的apt源

#192.168.86.134 - 192.168.86.136
mv /etc/apt/sources.list{,.bak}

echo "deb http://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware
# deb-src http://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware

deb http://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware
# deb-src http://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware

deb http://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware
# deb-src http://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware

deb https://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
# deb-src https://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware" >/etc/apt/sources.list

apt update
apt -y install vim wget

同步时间

#192.168.86.134 - 192.168.86.136
apt update
apt install ntpdate -y
ntpdate ntp1.aliyun.com

远程免密

#192.168.86.134 - 192.168.86.136
cd
ssh-keygen 
 sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
systemctl restart ssh
#192.168.86.134 
ssh-copy-id slurm-head
ssh-copy-id slurm-db
ssh-copy-id slurm-compute

安装munge

#192.168.86.134 - 192.168.86.136
export MUNGEUSER=1120
groupadd -g $MUNGEUSER munge 
useradd -m -c "MUNGE Uid 'N' Gid Emporium" -d /var/lib/munge -u $MUNGEUSER -g munge -s /sbin/nologin munge 
 
3.安装munge软件
#192.168.86.134 - 192.168.86.136
apt-get install -y munge libmunge-dev libmunge2 rng-tools make hwloc libhwloc-dev git gcc build-essential fakeroot devscripts debhelper libncurses-dev libgtk2.0-dev libpam0g-dev libperl-dev liblua5.3-dev libhwloc-dev dh-exec librrd-dev libipmimonitoring-dev hdf5-helpers libfreeipmi-dev libhdf5-dev man2html libcurl4-openssl-dev libpmix-dev libhttp-parser-dev libyaml-dev libjson-c-dev libjwt-dev liblz4-dev libdbus-1-dev librdkafka-dev libreadline-dev perl libpam0g-dev liblua5.3-dev libhwloc-dev


#192.168.86.135
apt-get install mariadb-server libmariadb-dev-compat libmariadb-dev -y
4.添加配置文件 
rngd -r /dev/urandom
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key  
chown munge: /etc/munge/munge.key 
chmod 400 /etc/munge/munge.key
chown -R munge: /var/lib/munge
chown -R munge: /var/run/munge
chown -R munge: /var/log/munge


scp /etc/munge/munge.key root@slurm-db:/etc/munge/
scp /etc/munge/munge.key root@slurm-compute:/etc/munge/
6.启动服务
 
systemctl restart munge
systemctl enable munge
systemctl status munge
#192.168.86.135 - 192.168.86.136

rngd -r /dev/urandom
chmod 700 /etc/munge
chown -R munge: /etc/munge
chown -R munge: /var/lib/munge
chown -R munge: /var/run/munge
chown -R munge: /var/log/munge


systemctl start munge
systemctl enable munge
systemctl status munge

安装slurm

#添加用户
#192.168.86.134 - 192.168.86.136
groupadd slurm
useradd -r -M -g slurm slurm  

## 编译安装
#192.168.86.134  - 192.168.86.136
wget https://download.schedmd.com/slurm/slurm-23.11.0.tar.bz2
tar -xf slurm-23.11.0.tar.bz2
cd slurm-23.11.0/
./configure --enable-debug  --prefix=/usr/local/slurm 
 make && make install
 #查看是否缺少插件
#192.168.86.135 
ls /usr/local/slurm/lib/slurm|grep accounting_storag|grep mysql  
 
##启动并且登陆创建数据库
systemctl start mariadb
mysql -u root

CREATE DATABASE slurm_acct_db;
CREATE USER 'slurm'@'192.168.86.135';
SET PASSWORD FOR 'slurm'@'192.168.86.135' = PASSWORD('mypassword');
GRANT ALL PRIVILEGES ON slurm_acct_db.* TO 'slurm'@'192.168.86.135';
FLUSH PRIVILEGES;
EXIT;

CREATE DATABASE slurm_acct_db;
CREATE USER 'slurm'@'192.168.86.134';
SET PASSWORD FOR 'slurm'@'192.168.86.134' = PASSWORD('mypassword');
GRANT ALL PRIVILEGES ON slurm_acct_db.* TO 'slurm'@'192.168.86.134';
FLUSH PRIVILEGES;
EXIT;

## 创建文件夹并且赋权
mkdir -p /var/log/slurm/
chown slurm:  /var/log/slurm/
## 编写配置文件
vim slurmdbd.conf
AuthType=auth/munge


DbdAddr=localhost
DbdHost=localhost
#DbdPort=7031
SlurmUser=slurm                      
MessageTimeout=300                  
DebugLevel=debug5                    
DefaultQOS=normal                  
LogFile=/var/log/slurm/slurmdbd.log 
PidFile=/var/run/slurmdbd.pid
StorageType=accounting_storage/mysql
StorageHost=localhost              
StoragePort=3306                    
StoragePass=mypassword               
StorageUser=slurm                    
StorageLoc=slurm_acct_db             

##更改文件权限
chown slurm: /usr/local/slurm/etc/slurmdbd.conf
chmod 600 /usr/local/slurm/etc/slurmdbd.conf
systemctl start slurmdbd
systemctl status slurmdbd
 systemctl enable  slurmdbd

#192.168.86.134 - 192.168.86.136 ##控制节点或计算节点
cp -rf etc/ /usr/local/slurm/
cp etc/slurm*.service /lib/systemd/system/

cd /usr/local/slurm/etc
cp slurm.conf.example slurm.conf
cp cgroup.conf.example cgroup.conf
cp slurmdbd.conf.example slurmdbd.conf  

## 修改cgroup配置
echo "CgroupMountpoint=/sys/fs/cgroup"  >> cgroup.conf
cat  cgroup.conf

vim slurm.conf

ClusterName=myCluster
SlurmctldHost=slurm-head
#SlurmctldHost=
#
MpiDefault=none
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmdUser=root
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/affinity
#TaskPlugin=task/cgroup

#
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300

Waittime=0

# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
#
#
# JOB PRIORITY
AccountingStorageEnforce=qos,limits
AccountingStorageHost=slurm-db
AccountingStoragePass=/var/run/munge/munge.socket.2
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageUser=slurm
#AccountingStorageTRES=gres/gpu
JobCompHost=slurm-db
JobCompLoc=slurm_acct_db
JobCompPass=mypassword
JobCompType=jobcomp/none
JobCompUser=slurm
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
#GresTypes=gpu

NodeName=slurm-head      RealMemory=1935 State=UNKNOWN
NodeName=slurm-db        RealMemory=1935 State=UNKNOWN
NodeName=slurm-compute   RealMemory=1935 State=UNKNOWN
PartitionName=compute Nodes=slurm-head,slurm-compute  Default=YES MaxTime=168:00:00 State=UP


## 创建一些文件赋权
mkdir /var/log/slurm/

touch /var/log/slurm/slurmctld.log
chown slurm: /var/log/slurm/slurmctld.log
chmod u+rw /var/log/slurm/slurmctld.log
touch /var/log/slurm/slurmd.log
chown slurm:   /var/log/slurm/slurmd.log
chmod u+rw /var/log/slurm/slurmd.log
mkdir -p /var/spool/slurmctld
chown slurm: /var/spool/slurmctld

mkdir -p /var/spool/slurmd 
chown slurm:  /var/spool/slurmd 


chown slurm:  /var/log/slurm/



systemctl restart slurmctld
systemctl status slurmctld

systemctl restart slurmd
systemctl status slurmd
 systemctl enable   slurmctld
 systemctl enable  slurmd
***有问题,根据提示执行以下命令看下面的踩坑环节***
journalctl  -xeu slurmd
journalctl  -xeu slurmctld
journalctl  -xeu slurmdbd

展示

集群状态验证

1702024509513

任务运行验证

数据库写入验证与slurmdbd通信 192.168.86.135机器

踩坑环节

Couldn’t find the specified plugin name for cgroup/v2 looking at all files

░░ The job identifier is 1747.
Dec 07 06:28:09 slurm-head slurmd[59887]: slurmd: error: Couldn't find the specified plugin name for cgroup/v2 looking at all files
Dec 07 06:28:09 slurm-head slurmd[59887]: slurmd: error: cannot find cgroup plugin for cgroup/v2
Dec 07 06:28:09 slurm-head slurmd[59887]: slurmd: error: cannot create cgroup context for cgroup/v2
Dec 07 06:28:09 slurm-head slurmd[59887]: slurmd: error: Unable to initialize cgroup plugin
Dec 07 06:28:09 slurm-head slurmd[59887]: slurmd: error: slurmd initialization failed
Dec 07 06:28:09 slurm-head systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited

这是因为缺少/usr/local/slurm/lib/slurm/cgroup_v2.so不要看网上的很扯淡基本上没有参考价值

解决

#老老实实吧以下命令执行一遍
apt-get install -y munge libmunge-dev libmunge2 rng-tools make hwloc libhwloc-dev git gcc build-essential fakeroot devscripts debhelper libncurses-dev libgtk2.0-dev libpam0g-dev libperl-dev liblua5.3-dev libhwloc-dev dh-exec librrd-dev libipmimonitoring-dev hdf5-helpers libfreeipmi-dev libhdf5-dev man2html libcurl4-openssl-dev libpmix-dev libhttp-parser-dev libyaml-dev libjson-c-dev libjwt-dev liblz4-dev libdbus-1-dev librdkafka-dev libreadline-dev perl libpam0g-dev liblua5.3-dev libhwloc-dev
#然后重新config make
cd slurm-23.11.0/
make uninstall
make clean
./configure --enable-debug  --prefix=/usr/local/slurm 
 make && make install

slurmdbd无法运行排除数据库信息不对后

需要make前安装mariadb/mysql哪怕你的mysql/mariadb不跟slurmdbd在一台都是要装不然缺少组件

缺少/usr/local/slurm/lib/slurm/accounting_storage_mysql.so

error: cgroup namespace ‘freezer’ not mounted. aborting

说明你方向错了没有按照我的配置先按“Couldn’t find the specified plugin name for cgroup/v2 looking at all files”解决方案解决然后cgrep.conf如下配置

#CgroupAutomount=yes
CgroupMountpoint=/sys/fs/cgroup
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes

阿里云国内75折回扣微信号：monov8

阿里云国际，腾讯云国际，低至75折。AWS 93折免费开户实名账号代冲值优惠多多微信号：monov8 飞机：@monov6

返回列表

上一篇：SQL事务管理

下一篇：Win32 HIWORD和LOWORD宏学习

“slurm 23.11.0集群 debian 11.5 安装” 的相关文章

Python调用Go语言编译的动态链接库（CGO）【待续】1年前 (2023-02-02)

字符串不可变1年前 (2023-02-02)

（六）elasticsearch 源码之选主流程分析1年前 (2023-02-02)

寒假不躺平，假期很精彩——2022 年寒假计划1年前 (2023-02-02)

Atcoder dp I Coins 题解1年前 (2023-02-02)

Spring61年前 (2023-02-02)

go语言写http踩得坑1年前 (2023-02-02)

T-SQL 模拟彩票预测 1年前 (2023-02-02)

OpenMP 原子指令设计与实现1年前 (2023-02-02)

VMware虚拟机安装Ubuntu 2022最新版详细图文安装教程(VMware虚拟机安装+Ubuntu下载+VMware虚拟机配置运行)1年前 (2023-02-02)

slurm 23.11.0集群 debian 11.5 安装