Nodejs 实现监控告警

作者: UMU @ MEET.ONE 实验室

钉钉

  1. 选择要接受通知的群,群设置 - 群机器人 - 添加机器人;

  2. 复制 webhook URL,记为 webhook_url;

  3. 发送通知的代码:

1
const fetch = require('node-fetch')

fetch(webhook_url, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    "msgtype": "text",
    "text": {"content": text}
  })
}).then(function(res) {
  if (res.ok) {
    console.log('Dingtalk message sent!')
  } else {
   console.log('status = ' + res.status)
  }
})

Telegram

  1. 添加 @BotFather,发送 /newbot 命令,随提示逐步建立一个机器人,得到这个机器人的 token,记为 bot_token。

  2. 选择要接受通知的 Group 或 Channel,按以下任一方式取得 chat_id:

(1) 转发 Group 或 Channel 内的消息到 @getidsbot

(2) 通过 Web 版查看 Group 或 Channel 的 URL 中,p 参数 的值。

  • 如果是 Group,chat_id 为把 g 前缀替换为负号的负整数。比如 p=g268787210,则 chat_id = '-268787210'

  • 如果是 Channel,chatid 为 前部分,并把 c 前缀替换为 -100 的负整数。比如 p=c1383705039_968667419389618100,则 chat_id = '-1001383705039'

  1. 发送通知的代码:
1
2
3
4
const TelegramBot = require('node-telegram-bot-api')
const bot = new TelegramBot(bot_token, {polling: false})

bot.sendMessage(chat_id, text)

为 EOSIO MongoDB 插件搭建高可用集群

选型

系统:CentOS7。 正像大部分国人喜欢用免费的 Windows 旗舰版,采用 RedHat 社区版,既有“企业级待遇”,又免费。实在是解决选择恐惧症必备良药……

MongoDB:4.0.4。 4.0 之前的版本不支持一些类型转换的函数,后期使用起来很麻烦。举个例子:

$toDate

New in version 4.0.

文件系统:XFS。 4.0 已经抛弃 MMAPv1 Storage Engine,官方文档强烈建议
WiredTiger Storage Engine
和 XFS 配套使用。

With the WiredTiger storage engine, using XFS is strongly recommended for data bearing nodes to avoid performance issues that may occur when using EXT4 with WiredTiger.

副本数:1。 数据可以很容易重新获取,丢失的代价不高,所以副本不是很重要(有钱请搞三副本)。另外,目前 nodeos 较常把数据弄脏,在它本身没高可用时,不宜对数据库投入太多成本。

机器配置:某云服务器一台。 16 Cores,256G RAM,启动盘 10G,额外八个 1T Disk。

1
2
3
4
5
6
7
8
9
10
11
12
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /
sdb 8:16 0 1T 0 disk
sdc 8:32 0 1T 0 disk
sdd 8:48 0 1T 0 disk
sde 8:64 0 1T 0 disk
sdf 8:80 0 1T 0 disk
sdg 8:96 0 1T 0 disk
sdh 8:112 0 1T 0 disk
sdi 8:128 0 1T 0 disk

环境配置

1. 设置 SE Linux

安装过程中,若您需要 reboot 系统,则每次 reboot 之后都要做一次:

1
setenforce Permissive

2. 关闭 TPH

以下命令不是持久化改变,但比较容易说明改了啥,仅供参考:

1
2
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

根据 Disable Transparent Huge Pages (THP),真正使用的是:

1
2
3
4
5
6
7
8
9
mkdir -p /etc/tuned/no-thp

echo '[main]
include=virtual-guest

[vm]
transparent_hugepages=never' > /etc/tuned/no-thp/tuned.conf


tuned-adm profile no-thp

3. TCP 优化

以下命令不是持久化改变,但比较容易说明优化了啥,仅供参考:

1
2
3
echo 120 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 3 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 3 > /proc/sys/net/ipv4/tcp_orphan_retries

真正使用的是:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
FILE=/etc/sysctl.conf
cp $FILE ${FILE}_`date +%Y%m%d%H%M`

KEY=tcp_keepalive_time
VALUE=120

egrep "net.ipv4.$KEY" $FILE && sed -i -c "s/net\.ipv4\.$KEY.*/net\.ipv4\.$KEY = $VALUE/g" $FILE || echo "net.ipv4.$KEY = $VALUE" >> $FILE

VALUE=3
KEY=tcp_fin_timeout

egrep "net.ipv4.$KEY" $FILE && sed -i -c "s/net\.ipv4\.$KEY.*/net\.ipv4\.$KEY = $VALUE/g" $FILE || echo "net.ipv4.$KEY = $VALUE" >> $FILE

KEY=tcp_orphan_retries

egrep "net.ipv4.$KEY" $FILE && sed -i -c "s/net\.ipv4\.$KEY.*/net\.ipv4\.$KEY = $VALUE/g" $FILE || echo "net.ipv4.$KEY = $VALUE" >> $FILE

sysctl -p 2>/tmp/sysctl.tmp

安装步骤

1. 全局设置

1
2
3
4
5
6
7
8
PASSWORD=MEETONE_FAKE_PASSWORD

PREFIX=/disk
BIND_IP=10.140.0.10
D_PORT=17089
CS_PORT=17088
S_PORT=17087
NUM_SHARD=7

2. 防火墙例外

1
semanage port -a -t mongod_port_t -p tcp 17087-17095

3. 分区

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
yum install -y xfsprogs

function InitDisk {
umount ${PREFIX}*
sed -i "s:^LABEL=${PREFIX}:#LABEL=${PREFIX}:" /etc/fstab
i=0
for drv in {b..i}; do
/sbin/parted /dev/sd"$drv" -s mklabel gpt mkpart primary 2048s 100%
sleep 1
mkfs.xfs -f -b size=4096 -d su=64k,sw=4,agcount=2000 /dev/sd"$drv"1
xfs_admin -L ${PREFIX}$i /dev/sd"$drv"1
mkdir -p ${PREFIX}$i
echo "LABEL=${PREFIX}$i ${PREFIX}$i xfs rw,noatime,nodiratime,allocsize=16M,inode64,logbsize=256k,delaylog,nobarrier,nolargeio,swalloc 0 0" >> /etc/fstab

i=$[i+1]
done
mount -a
}

InitDisk

4. 安装 MongoDB Community Edition

参考官网的安装文档:Install MongoDB Community Edition on Red Hat Enterprise or CentOS Linux

1
2
3
4
5
6
7
8
9
10
echo '[mongodb-org-4.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc' > /etc/yum.repos.d/mongodb-org-4.0.repo


yum install -y mongodb-org

echo 'exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools' >> /etc/yum.conf

5. 初始化数据库目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
function InitDir {
rm -rf ${PREFIX}0/mongod_conf_data ${PREFIX}0/mongo_log
for ((i = 1; i <= $NUM_SHARD; ++i)); do
rm -rf ${PREFIX}$i/mongo*
done

mkdir -p ${PREFIX}0/mongod_conf_data
chown -R mongod. ${PREFIX}0/mongod_conf_data
mkdir -p ${PREFIX}0/mongo_log
chown -R mongod. ${PREFIX}0/mongo_log
for ((i = 1; i <= $NUM_SHARD; ++i)); do
mkdir -p ${PREFIX}$i/mongod_data
chown mongod. ${PREFIX}$i/mongo*
done
}

InitDir

6. 配置 MongoD 分片服务器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
function CreateShardConfig {
for ((i = 1; i <= $NUM_SHARD; ++i)); do
echo "shardsvr=true
replSet=shard$i
bind_ip=127.0.0.1,$BIND_IP
port=$[D_PORT+i-1]
dbpath=${PREFIX}$i/mongod_data
logpath=${PREFIX}0/mongo_log/shard$i.log
pidfilepath=/var/run/mongodb/mongod_shard$i.pid
logappend=true
logRotate=reopen
fork=true
wiredTigerCacheSizeGB=10
#keyFile=/etc/mongodb-keyfile
#verbose=true
directoryperdb=true
wiredTigerDirectoryForIndexes=true" > /etc/mongod_shard$i.conf

chown mongod. /etc/mongod_shard$i.conf
done
}

function CreateShardService {
for ((i = 1; i <= $NUM_SHARD; ++i)); do
echo "[Unit]
Description=MongoD
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongod
Group=mongod
Environment=\"OPTIONS=-f /etc/mongod_shard$i.conf\"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod \$OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongod_shard$i.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target" > /usr/lib/systemd/system/mongod_shard$i.service

done

systemctl daemon-reload
}

function InitShard {
SH=/tmp/mongod_shard.sh
echo "" > $FILE
for ((i = 1; i <= $NUM_SHARD; ++i)); do
PORT=$[D_PORT+i-1]
echo "echo \"rs.initiate({_id: 'shard$i', members:[{_id: 0, host: '$BIND_IP:$PORT'}]})\" | mongo --host 127.0.0.1 --port $PORT" >> $SH
done
bash $SH
}

function CreateShardUser {
JS=/tmp/mongod_shard.js

echo "use admin;
db.createUser({ \"user\": \"MongoAdmin\", \"pwd\": \"${PASSWORD}\", \"roles\": [\"root\"]});
cfg = rs.conf();
cfg.members[0].priority = 10;
rs.reconfig(cfg);" > $JS


for ((i = 1; i <= $NUM_SHARD; ++i)); do
mongo --port $[CS_PORT+i-1] < $JS
done
}

CreateShardConfig
CreateShardService
for ((i = 1; i <= $NUM_SHARD; ++i)); do systemctl restart mongod_shard$i.service; done
InitShard
CreateShardUser

7. 配置 MongDB Config Server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
function CreateConfigServerConfig {
FILE=/etc/mongod.conf

echo "replSet=csReplSet
configsvr=true
bind_ip=127.0.0.1,$BIND_IP
port=$CS_PORT
dbpath=${PREFIX}0/mongod_conf_data
logpath=${PREFIX}0/mongo_log/config.log
pidfilepath=/var/run/mongodb/mongod.pid
logappend=true
logRotate=reopen
fork=true
#keyFile=/etc/mongodb-keyfile" > $FILE


chown mongod. $FILE
}

function CreateConfigServerService {
echo '[Unit]
Description=MongoCS
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongod.conf"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod $OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongod.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/mongod.service


systemctl daemon-reload
}

CreateConfigServerConfig
CreateConfigServerService

systemctl restart mongod.service

sleep 5
echo "rs.initiate({_id: 'csReplSet', members:[{_id: 0, host: '$BIND_IP:$CS_PORT'}]})" | mongo --port $CS_PORT
echo "use admin;
db.createUser({ \"user\": \"MongoAdmin\", \"pwd\": \"${PASSWORD}\", \"roles\": [\"root\"]});" | mongo --port $CS_PORT

8. 配置 MongS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
function CreateMongoSConfig {
FILE=/etc/mongos.conf

echo "configdb=csReplSet/$BIND_IP:$CS_PORT
bind_ip_all=true
port=17087
logpath=${PREFIX}0/mongos_log/mongos.log
pidfilepath=/var/run/mongodb/mongos.pid
logappend=true
logRotate=reopen
fork=true
#keyFile=/etc/mongodb-keyfile" > $FILE


chown mongod. $FILE

mkdir -p ${PREFIX}0/mongos_log
chown -R mongod. ${PREFIX}0/mongos_log
}

function CreateMongoSService {
echo '[Unit]
Description=MongoS
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongos.conf"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongos $OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongos.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/mongos.service


systemctl daemon-reload
}

function AddShards {
FILE=/tmp/mongo_add_shards.js
echo "use admin;" > $FILE
for ((i = 1; i <= $NUM_SHARD; ++i)); do
echo "db.runCommand({addshard:\"shard$i/$BIND_IP:$[D_PORT+i-1]\"});" >> $FILE
done

mongo --port $S_PORT < $FILE
}

CreateMongoSConfig
CreateMongoSService

systemctl restart mongos.service

AddShards

9. 【可选】配置 keyfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
function ConfigKeyfile {
FILE=/etc/mongodb-keyfile
openssl rand -base64 745 > $FILE
chown mongod. $FILE
chmod 600 $FILE

grep -P '^#keyFile' /etc/mongod*.conf && sed -i 's/#keyFile/keyFile/g' /etc/mongod*.conf

grep -P '^#keyFile' /etc/mongos.conf && sed -i 's/#keyFile/keyFile/g' /etc/mongos.conf
}

function RestartMongo {
systemctl restart mongod.service
for ((i = 1; i <= $NUM_SHARD; ++i)); do
systemctl restart mongod_shard$i.service;
done
systemctl restart mongos.service
}

ConfigKeyfile
RestartMongo

10. 【可选】配置集合和创建索引

1
use EOS
db.action_traces.createIndex({"act.account": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.name": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.receiver": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.from": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.to": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.name": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.voter": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.authorization.actor": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.account": 1, "block_time": 1, "_id":1},{"background": true, "sparse" : true})

sh.enableSharding("EOS")
sh.shardCollection("EOS.action_traces", {"_id" : 1},  true)
sh.shardCollection("EOS.transaction_traces", {"id" : "hashed"})

11. 【可选】安装 EOSIO 1.5

1
2
wget https://github.com/eosio/eos/releases/download/v1.5.0/eosio-1.5.0-1.el7.x86_64.rpm
rpm -ivh ./eosio-1.5.0-1.el7.x86_64.rpm --nodeps

MongoDB db.stats() 的各种 Size

现象

db.stats() 的各种 Size 需要理理,先看例子:

1
2
3
4
5
6
7
8
9
10
11
> db.action_traces.dataSize()
12489840963

> db.action_traces.totalSize()
5391249408

> db.action_traces.storageSize()
3684032512

> db.action_traces.totalIndexSize()
1707216896

概念解释

db.action_traces.stats() 里的 size 就是 db.action_traces.dataSize(),也就是数据本身的逻辑大小。

由于数据库引擎有压缩概念,所以存储到介质时,可能占用的空间并没有逻辑大小那么多,比如 WiredTiger Storage Engine 的压缩率就挺不错的,dataSize = 12,489,840,963 字节的数据,存到硬盘只有 storageSize = 5,391,249,408 字节。

其中 totalIndexSize 是索引占存储器的大小,所以 totalSize = storageSize + totalIndexSize。

注意:索引有时会比数据本身还大……

参考

db.collection.stats() — MongoDB Manual

db.collection.totalIndexSize() — MongoDB Manual

db.collection.dataSize() — MongoDB Manual

db.collection.storageSize() — MongoDB Manual

db.collection.totalSize() — MongoDB Manual

学习 MongoDB 选举机制

为了快速了解 MongoDB 选举机制,在网上找了一些文章来学习,后来发现里面提到的一些机制都过时了,尝试看代码了解,发现协议有 PV0 和 PV1 两种。

代码:https://github.com/mongodb/mongo/blob/r3.6.5/src/mongo/db/repl/topology_coordinator.cpp

一篇比较新的参考文章:https://blog.csdn.net/wentyoon/article/details/78986174

如果新选举出的主节点立马挂掉,至少需要 30s 重新选主,这个是由 leaseTime 常量决定的:

const Seconds TopologyCoordinator::VoteLease::leaseTime = Seconds(30);

PV0 时,一个反对会将最终票数减 10000,即在绝大多数情况下,只要有节点反对,请求的节点就不能成为主节点,由 prepareElectResponse 函数实现,里面有不少 vote = -10000;,PV1 版本取消了否决票。

MongoDB Shard ID hash 算法 std::hash 的跨平台性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <functional>
#include <iomanip>
#include <iostream>
#include <string>


int main()
{

std::string str = "Meet the new boss...";
std::size_t str_hash = std::hash<std::string>{}(str);
std::cout << "hash(" << std::quoted(str) << ") = " << str_hash << std::endl;

str = "Meet the new boss..;";
str_hash = std::hash<std::string>{}(str);
std::cout << "hash(" << std::quoted(str) << ") = " << str_hash << std::endl;

str = "Meet the new boss../";
str_hash = std::hash<std::string>{}(str);
std::cout << "hash(" << std::quoted(str) << ") = " << str_hash << std::endl;

str = "Meet the new boss..,";
str_hash = std::hash<std::string>{}(str);
std::cout << "hash(" << std::quoted(str) << ") = " << str_hash << std::endl;

return 0;
}

Windows, VS 2017 的结果:

hash(“Meet the new boss…”) = 5935324269489717502

hash(“Meet the new boss..;”) = 5935347359233909933

hash(“Meet the new boss../“) = 5935325369001345713

hash(“Meet the new boss..,”) = 5935322070466461080

Ubuntu 16.04, g++ 5.4.0 20160609 的结果:

hash(“Meet the new boss…”) = 10656026664466977650

hash(“Meet the new boss..;”) = 12509209616339026574

hash(“Meet the new boss../“) = 6552276210272946664

hash(“Meet the new boss..,”) = 15639609178671340058

还好我们不会在生产环境,使用 Windows 部署 MongoDB……

1
2
3
std::size_t ShardId::Hasher::operator()(const ShardId& shardId) const {
return std::hash<std::string>()(shardId._shardId);
}

详见:https://github.com/mongodb/mongo/blob/master/src/mongo/s/shard_id.cpp

这个 std::hash 在 x86 和 x64 下都不一样,所以,让我们看看 MongoDB 如何解决这个问题:

MongoDB 3.4 no longer supports 32-bit x86 platforms.

好样的!

Mongo Shell 下批量更新集合

需求

延长 mongodb 某集合里的“过期时间”字段。

风险分析

update 一下是很简单,主要怕在 Shell 下操作可能改变数字类型。
先做了实验,发现 3.2 的版本下,并没有这个问题,之前看书,说数字可能被改为双精度,看来是旧版本的不足。

1
2
3
4
db.UMU.find().forEach(function (doc) {
doc.expireDate = NumberLong(doc.updateTime + 180*24*60*60*1000);
db.UMU.save(doc);
})

其中 NumberLong 是必要的,不然更新后,expireDate 的类型并不是和 updateTime 一样的 NumberLong。