为 EOSIO MongoDB 插件搭建高可用集群

选型

系统:CentOS7。 正像大部分国人喜欢用免费的 Windows 旗舰版,采用 RedHat 社区版,既有“企业级待遇”,又免费。实在是解决选择恐惧症必备良药……

MongoDB:4.0.4。 4.0 之前的版本不支持一些类型转换的函数,后期使用起来很麻烦。举个例子:

$toDate

New in version 4.0.

文件系统:XFS。 4.0 已经抛弃 MMAPv1 Storage Engine,官方文档强烈建议
WiredTiger Storage Engine
和 XFS 配套使用。

With the WiredTiger storage engine, using XFS is strongly recommended for data bearing nodes to avoid performance issues that may occur when using EXT4 with WiredTiger.

副本数:1。 数据可以很容易重新获取,丢失的代价不高,所以副本不是很重要(有钱请搞三副本)。另外,目前 nodeos 较常把数据弄脏,在它本身没高可用时,不宜对数据库投入太多成本。

机器配置:某云服务器一台。 16 Cores,256G RAM,启动盘 10G,额外八个 1T Disk。

1
2
3
4
5
6
7
8
9
10
11
12
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /
sdb 8:16 0 1T 0 disk
sdc 8:32 0 1T 0 disk
sdd 8:48 0 1T 0 disk
sde 8:64 0 1T 0 disk
sdf 8:80 0 1T 0 disk
sdg 8:96 0 1T 0 disk
sdh 8:112 0 1T 0 disk
sdi 8:128 0 1T 0 disk

环境配置

1. 设置 SE Linux

安装过程中,若您需要 reboot 系统,则每次 reboot 之后都要做一次:

1
setenforce Permissive

2. 关闭 TPH

以下命令不是持久化改变,但比较容易说明改了啥,仅供参考:

1
2
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

根据 Disable Transparent Huge Pages (THP),真正使用的是:

1
2
3
4
5
6
7
8
9
mkdir -p /etc/tuned/no-thp

echo '[main]
include=virtual-guest

[vm]
transparent_hugepages=never' > /etc/tuned/no-thp/tuned.conf


tuned-adm profile no-thp

3. TCP 优化

以下命令不是持久化改变,但比较容易说明优化了啥,仅供参考:

1
2
3
echo 120 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 3 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 3 > /proc/sys/net/ipv4/tcp_orphan_retries

真正使用的是:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
FILE=/etc/sysctl.conf
cp $FILE ${FILE}_`date +%Y%m%d%H%M`

KEY=tcp_keepalive_time
VALUE=120

egrep "net.ipv4.$KEY" $FILE && sed -i -c "s/net\.ipv4\.$KEY.*/net\.ipv4\.$KEY = $VALUE/g" $FILE || echo "net.ipv4.$KEY = $VALUE" >> $FILE

VALUE=3
KEY=tcp_fin_timeout

egrep "net.ipv4.$KEY" $FILE && sed -i -c "s/net\.ipv4\.$KEY.*/net\.ipv4\.$KEY = $VALUE/g" $FILE || echo "net.ipv4.$KEY = $VALUE" >> $FILE

KEY=tcp_orphan_retries

egrep "net.ipv4.$KEY" $FILE && sed -i -c "s/net\.ipv4\.$KEY.*/net\.ipv4\.$KEY = $VALUE/g" $FILE || echo "net.ipv4.$KEY = $VALUE" >> $FILE

sysctl -p 2>/tmp/sysctl.tmp

安装步骤

1. 全局设置

1
2
3
4
5
6
7
8
PASSWORD=MEETONE_FAKE_PASSWORD

PREFIX=/disk
BIND_IP=10.140.0.10
D_PORT=17089
CS_PORT=17088
S_PORT=17087
NUM_SHARD=7

2. 防火墙例外

1
semanage port -a -t mongod_port_t -p tcp 17087-17095

3. 分区

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
yum install -y xfsprogs

function InitDisk {
umount ${PREFIX}*
sed -i "s:^LABEL=${PREFIX}:#LABEL=${PREFIX}:" /etc/fstab
i=0
for drv in {b..i}; do
/sbin/parted /dev/sd"$drv" -s mklabel gpt mkpart primary 2048s 100%
sleep 1
mkfs.xfs -f -b size=4096 -d su=64k,sw=4,agcount=2000 /dev/sd"$drv"1
xfs_admin -L ${PREFIX}$i /dev/sd"$drv"1
mkdir -p ${PREFIX}$i
echo "LABEL=${PREFIX}$i ${PREFIX}$i xfs rw,noatime,nodiratime,allocsize=16M,inode64,logbsize=256k,delaylog,nobarrier,nolargeio,swalloc 0 0" >> /etc/fstab

i=$[i+1]
done
mount -a
}

InitDisk

4. 安装 MongoDB Community Edition

参考官网的安装文档:Install MongoDB Community Edition on Red Hat Enterprise or CentOS Linux

1
2
3
4
5
6
7
8
9
10
echo '[mongodb-org-4.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc' > /etc/yum.repos.d/mongodb-org-4.0.repo


yum install -y mongodb-org

echo 'exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools' >> /etc/yum.conf

5. 初始化数据库目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
function InitDir {
rm -rf ${PREFIX}0/mongod_conf_data ${PREFIX}0/mongo_log
for ((i = 1; i <= $NUM_SHARD; ++i)); do
rm -rf ${PREFIX}$i/mongo*
done

mkdir -p ${PREFIX}0/mongod_conf_data
chown -R mongod. ${PREFIX}0/mongod_conf_data
mkdir -p ${PREFIX}0/mongo_log
chown -R mongod. ${PREFIX}0/mongo_log
for ((i = 1; i <= $NUM_SHARD; ++i)); do
mkdir -p ${PREFIX}$i/mongod_data
chown mongod. ${PREFIX}$i/mongo*
done
}

InitDir

6. 配置 MongoD 分片服务器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
function CreateShardConfig {
for ((i = 1; i <= $NUM_SHARD; ++i)); do
echo "shardsvr=true
replSet=shard$i
bind_ip=127.0.0.1,$BIND_IP
port=$[D_PORT+i-1]
dbpath=${PREFIX}$i/mongod_data
logpath=${PREFIX}0/mongo_log/shard$i.log
pidfilepath=/var/run/mongodb/mongod_shard$i.pid
logappend=true
logRotate=reopen
fork=true
wiredTigerCacheSizeGB=10
#keyFile=/etc/mongodb-keyfile
#verbose=true
directoryperdb=true
wiredTigerDirectoryForIndexes=true" > /etc/mongod_shard$i.conf

chown mongod. /etc/mongod_shard$i.conf
done
}

function CreateShardService {
for ((i = 1; i <= $NUM_SHARD; ++i)); do
echo "[Unit]
Description=MongoD
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongod
Group=mongod
Environment=\"OPTIONS=-f /etc/mongod_shard$i.conf\"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod \$OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongod_shard$i.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target" > /usr/lib/systemd/system/mongod_shard$i.service

done

systemctl daemon-reload
}

function InitShard {
SH=/tmp/mongod_shard.sh
echo "" > $FILE
for ((i = 1; i <= $NUM_SHARD; ++i)); do
PORT=$[D_PORT+i-1]
echo "echo \"rs.initiate({_id: 'shard$i', members:[{_id: 0, host: '$BIND_IP:$PORT'}]})\" | mongo --host 127.0.0.1 --port $PORT" >> $SH
done
bash $SH
}

function CreateShardUser {
JS=/tmp/mongod_shard.js

echo "use admin;
db.createUser({ \"user\": \"MongoAdmin\", \"pwd\": \"${PASSWORD}\", \"roles\": [\"root\"]});
cfg = rs.conf();
cfg.members[0].priority = 10;
rs.reconfig(cfg);" > $JS


for ((i = 1; i <= $NUM_SHARD; ++i)); do
mongo --port $[CS_PORT+i-1] < $JS
done
}

CreateShardConfig
CreateShardService
for ((i = 1; i <= $NUM_SHARD; ++i)); do systemctl restart mongod_shard$i.service; done
InitShard
CreateShardUser

7. 配置 MongDB Config Server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
function CreateConfigServerConfig {
FILE=/etc/mongod.conf

echo "replSet=csReplSet
configsvr=true
bind_ip=127.0.0.1,$BIND_IP
port=$CS_PORT
dbpath=${PREFIX}0/mongod_conf_data
logpath=${PREFIX}0/mongo_log/config.log
pidfilepath=/var/run/mongodb/mongod.pid
logappend=true
logRotate=reopen
fork=true
#keyFile=/etc/mongodb-keyfile" > $FILE


chown mongod. $FILE
}

function CreateConfigServerService {
echo '[Unit]
Description=MongoCS
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongod.conf"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod $OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongod.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/mongod.service


systemctl daemon-reload
}

CreateConfigServerConfig
CreateConfigServerService

systemctl restart mongod.service

sleep 5
echo "rs.initiate({_id: 'csReplSet', members:[{_id: 0, host: '$BIND_IP:$CS_PORT'}]})" | mongo --port $CS_PORT
echo "use admin;
db.createUser({ \"user\": \"MongoAdmin\", \"pwd\": \"${PASSWORD}\", \"roles\": [\"root\"]});" | mongo --port $CS_PORT

8. 配置 MongS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
function CreateMongoSConfig {
FILE=/etc/mongos.conf

echo "configdb=csReplSet/$BIND_IP:$CS_PORT
bind_ip_all=true
port=17087
logpath=${PREFIX}0/mongos_log/mongos.log
pidfilepath=/var/run/mongodb/mongos.pid
logappend=true
logRotate=reopen
fork=true
#keyFile=/etc/mongodb-keyfile" > $FILE


chown mongod. $FILE

mkdir -p ${PREFIX}0/mongos_log
chown -R mongod. ${PREFIX}0/mongos_log
}

function CreateMongoSService {
echo '[Unit]
Description=MongoS
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongos.conf"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongos $OPTIONS
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongos.pid
Type=forking
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target' > /usr/lib/systemd/system/mongos.service


systemctl daemon-reload
}

function AddShards {
FILE=/tmp/mongo_add_shards.js
echo "use admin;" > $FILE
for ((i = 1; i <= $NUM_SHARD; ++i)); do
echo "db.runCommand({addshard:\"shard$i/$BIND_IP:$[D_PORT+i-1]\"});" >> $FILE
done

mongo --port $S_PORT < $FILE
}

CreateMongoSConfig
CreateMongoSService

systemctl restart mongos.service

AddShards

9. 【可选】配置 keyfile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
function ConfigKeyfile {
FILE=/etc/mongodb-keyfile
openssl rand -base64 745 > $FILE
chown mongod. $FILE
chmod 600 $FILE

grep -P '^#keyFile' /etc/mongod*.conf && sed -i 's/#keyFile/keyFile/g' /etc/mongod*.conf

grep -P '^#keyFile' /etc/mongos.conf && sed -i 's/#keyFile/keyFile/g' /etc/mongos.conf
}

function RestartMongo {
systemctl restart mongod.service
for ((i = 1; i <= $NUM_SHARD; ++i)); do
systemctl restart mongod_shard$i.service;
done
systemctl restart mongos.service
}

ConfigKeyfile
RestartMongo

10. 【可选】配置集合和创建索引

1
use EOS
db.action_traces.createIndex({"act.account": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.name": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.receiver": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.from": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.to": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.name": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.data.voter": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"act.authorization.actor": 1, "_id":1},{background: true, sparse: true})
db.action_traces.createIndex({"receipt.receiver": 1, "_id":1},{background: true, sparse: true})

db.action_traces.createIndex({"block_num": 1, "_id":1},{background: true})
db.action_traces.createIndex({"block_time": 1, "_id":1},{background: true})

db.transaction_traces.createIndex({"id": 1, "_id":1},{background: true})

sh.enableSharding("EOS")
sh.shardCollection("EOS.action_traces", {"_id" : 1},  true)
sh.shardCollection("EOS.transaction_traces", {"_id" : 1},  true)

11. 【可选】安装 EOSIO 1.5

1
2
wget https://github.com/eosio/eos/releases/download/v1.5.0/eosio-1.5.0-1.el7.x86_64.rpm
rpm -ivh ./eosio-1.5.0-1.el7.x86_64.rpm --nodeps