Compare commits

3 Commits

Author SHA1 Message Date
Yifan Gao
4e24cefa5e Refactor project with justfile 2025-11-25 12:03:49 +08:00
Yifan Gao
c100789431 Revise README.md for clarity and accuracy
- Updated project description for better readability and coherence.
- Enhanced explanations regarding BGP data and its significance.
- Adjusted the status of China Telecom's IP collection to reflect its deprecation.
- Reformatted the operator IP lists section into a clearer table format.
- Added acknowledgments and license section for better project documentation.
2025-11-25 11:18:28 +08:00
Yifan Gao
f9e4799665 bgptool 0.2.2 (fix #53) 2025-11-25 02:03:42 +08:00
9 changed files with 258 additions and 203 deletions

View File

@@ -13,44 +13,32 @@ on:
jobs:
build:
runs-on: ubuntu-latest
env:
BGPTOOLS_VERSION: 0.0.3
GO111MODULE: on
steps:
- uses: actions/checkout@v3
- uses: dtolnay/rust-toolchain@stable
- uses: cargo-bins/cargo-binstall@main
- uses: actions/setup-go@v2
with:
go-version: 1.16
- run: sudo apt-get install -y lftp bgpdump tree jq axel
- run: sudo apt-get install -y tree jq axel
- uses: extractions/setup-just@v3
- uses: actions/cache@v3
with:
key: ${{ runner.os }}-build-dependencies
path: |
~/go/bin
~/.cargo/bin
- run: ./dependency.sh
- run: just dependency
- uses: nick-fields/retry@v3
with:
timeout_minutes: 60
max_attempts: 3
command: ./generate.sh
- run: ./stat.sh
- run: ./guard.sh
command: just
- run: just guard
- name: Checkout ip-lists branch
uses: actions/checkout@v3
if: github.event_name == 'schedule' && github.ref == 'refs/heads/master'
with:
ref: ip-lists
path: ip-lists
- run: ./upload.sh
- run: just upload
if: github.event_name == 'schedule' && github.ref == 'refs/heads/master'
- name: Refresh CDN cache
- run: just refresh_jsdelivr ${{ github.repository }}
if: github.event_name == 'schedule' && github.ref == 'refs/heads/master'
run: |
cd ip-lists
for file in *; do
curl -i "https://purge.jsdelivr.net/gh/${{ github.repository }}@ip-lists/${file}"
done

View File

@@ -13,32 +13,32 @@
依据中国网络运营商分类的IP地址库
## 为什么创造这个项目
## 为什么要创建这个项目
国内BGP/ASN数据分析的商业服务只有一个[ipip.net](https://www.ipip.net),是目前运营商IP库准确度最高的服务商,我认为没有之一
国内BGP/ASN数据分析和应用方面,目前主要有[ipip.net](https://www.ipip.net)等商业服务,其运营商IP库准确度较高
随着互联网规模的增加,为了处理大批量的路由数据,边界网关协议(BGP下同应运而生是互联网的基础协议之一。为了保证了全球网络路由的可达性但凡需要在互联网中注册一个IP都需要借助BGP协议对外告,这样互联网中的其他自治域才能学习到这段地址的路由信息其它主机才能成功访问这个IP。因此可以说BGP数据是最适合分析运营商IP地址的数据来源之一
随着互联网的持续发展边界网关协议BGP成为处理大规模路由数据不可或缺的基础协议之一。通过BGP新的IP地址或前缀可以在全球互联网上对外告,并被其他自治系统学习和访问。因此BGP数据为分析归属和运营商IP分类提供了宝贵的数据基础
但是,目前国内绝大多数IP库都由[WHOIS数据库](https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz)作为基础数据源。WHOIS数据仅表示某个IP被哪个机构注册但无从知晓该IP被用在何处,这导致许多非运营商自己注册的IP地址无法被正确分类。ipip.net是最早开始BGP/ASN数据分析的公司之一数据准确性甩其它库几条街。但很可惜是ipip.net作为商业公司绝大多数高质量的IP数据都是收费的且价格不菲
不过,目前国内大部分IP库依赖[WHOIS数据库](https://ftp.apnic.net/apnic/whois/apnic.db.inetnum.gz)作为数据源。WHOIS虽然能标明IP的注册机构但无法体现实际使用情况,这导致一些并非运营商亲自宣告的IP地址被分类不准确。像ipip.net这样较早开始结合BGPASN数据进行分析的公司,能够提供较为丰富和准确的数据服务,但其高质量数据部分需要付费
由于在做其他课题时需要处理BGP数据本着开源精神我将这部分代码重新封装创造了这个项目。至于如何使用大家可以自己发挥想象力。如[@ustclug](https://github.com/ustclug)将其用在权威DNS服务器上做分域解析我则借助这个IP库做了一个多出口的网关访问不同的运营商时走不同的线路如果都不匹配则走国外vps原因你懂的
在其他项目中我曾用到BGP数据因此基于开源的想法整理和公布了这些相关代码形成了本项目。该IP库可以灵活应用于多种场景例如[@ustclug](https://github.com/ustclug)利用它在权威DNS服务器中进行分域解析,或者作为按运营商出口分流的参考等
但由于个人精力有限IP库的覆盖率并不及ipip.net尤其是一些骨干节点的地址,这些地址往往是核心路由设备或企业托管给运营商的地址,对普通用户影响不大
个人精力限制本项目的IP覆盖率难以与商业服务商持平特别是在部分骨干节点相关的地址上,可能会有遗漏,但这些情况一般对大多数用户影响较小
果大家有任何建议或问,欢迎提交issue
建议或问,欢迎通过[issue](https://github.com/gaoyifan/china-operator-ip/issues)反馈
## 收录的运营商
* 中国电信(chinanet)
* 中国移动(cmcc)
* 中国联通(unicom)
* ~~中国铁通(tietong)~~<即将废弃>
* ~~中国铁通(tietong)~~<废弃>
* 教育网(cernet)
* 科技网(cstnet)
* 鹏博士(drpeng) <试验阶段>
* 谷歌中国(googlecn) <试验阶段>
*P.S. 由于移动与铁通已合并,铁通集合即将废弃,详见[issue #10](https://github.com/gaoyifan/china-operator-ip/issues/10)。处于兼容性考虑,当前铁通的预生成数据同中国移动,未来将择机移除铁通。*
*P.S. 由于移动与铁通已合并,铁通集合废弃,详见[issue #10](https://github.com/gaoyifan/china-operator-ip/issues/10)。*
*P.S. 鹏博士集团包括鹏博士数据、北京电信通、长城宽带、宽带通的IP地址并非全都由独立的自治域做宣告目前大部分地址仍由电信、联通、科技网代为宣告。故[列表](https://github.com/gaoyifan/china-operator-ip/blob/ip-lists/drpeng.txt)中的地址仅为鹏博士拥有的部分IP地址且这些IP同时具有电信、联通两个上级出口。详见[issue #2](https://github.com/gaoyifan/china-operator-ip/issues/2).*
@@ -56,63 +56,53 @@ git clone -b ip-lists https://github.com/gaoyifan/china-operator-ip.git
亦可通过以下站点获取:
| 运营商 | [EdgeOne Pages](https://china-operator-ip.yfgao.com) | [GitHub Pages](https://gaoyifan.github.io/china-operator-ip) | [jsDelivr](https://www.jsdelivr.com/package/gh/gaoyifan/china-operator-ip) |
|---|---|---|---|
| 中国 | [IPv4](https://china-operator-ip.yfgao.com/china.txt) \| [IPv6](https://china-operator-ip.yfgao.com/china6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/china.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/china6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/china.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/china6.txt) |
| 中国电信 | [IPv4](https://china-operator-ip.yfgao.com/chinanet.txt) \| [IPv6](https://china-operator-ip.yfgao.com/chinanet6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/chinanet.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/chinanet6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/chinanet.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/chinanet6.txt) |
| 中国移动 | [IPv4](https://china-operator-ip.yfgao.com/cmcc.txt) \| [IPv6](https://china-operator-ip.yfgao.com/cmcc6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/cmcc.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/cmcc6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/cmcc.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/cmcc6.txt) |
| 中国联通 | [IPv4](https://china-operator-ip.yfgao.com/unicom.txt) \| [IPv6](https://china-operator-ip.yfgao.com/unicom6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/unicom.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/unicom6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/unicom.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/unicom6.txt) |
| 中国铁通 | [IPv4](https://china-operator-ip.yfgao.com/tietong.txt) \| [IPv6](https://china-operator-ip.yfgao.com/tietong6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/tietong.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/tietong6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/tietong.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/tietong6.txt) |
| 教育网 | [IPv4](https://china-operator-ip.yfgao.com/cernet.txt) \| [IPv6](https://china-operator-ip.yfgao.com/cernet6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/cernet.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/cernet6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/cernet.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/cernet6.txt) |
| 科技网 | [IPv4](https://china-operator-ip.yfgao.com/cstnet.txt) \| [IPv6](https://china-operator-ip.yfgao.com/cstnet6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/cstnet.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/cstnet6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/cstnet.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/cstnet6.txt) |
| 鹏博士 | [IPv4](https://china-operator-ip.yfgao.com/drpeng.txt) \| [IPv6](https://china-operator-ip.yfgao.com/drpeng6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/drpeng.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/drpeng6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/drpeng.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/drpeng6.txt) |
| 谷歌中国 | [IPv4](https://china-operator-ip.yfgao.com/googlecn.txt) \| [IPv6](https://china-operator-ip.yfgao.com/googlecn6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/googlecn.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/googlecn6.txt) | [IPv4](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/googlecn.txt) \| [IPv6](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/googlecn6.txt) |
| 统计 | [stat](https://china-operator-ip.yfgao.com/stat) | [stat](https://gaoyifan.github.io/china-operator-ip/stat) | [stat](https://cdn.jsdelivr.net/gh/gaoyifan/china-operator-ip@ip-lists/stat) |
| 运营商 | [EdgeOne Pages](https://china-operator-ip.yfgao.com) | [GitHub Pages](https://gaoyifan.github.io/china-operator-ip) |
|---|---|---|
| 中国 | [IPv4](https://china-operator-ip.yfgao.com/china.txt) \| [IPv6](https://china-operator-ip.yfgao.com/china6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/china.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/china6.txt) |
| 中国电信 | [IPv4](https://china-operator-ip.yfgao.com/chinanet.txt) \| [IPv6](https://china-operator-ip.yfgao.com/chinanet6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/chinanet.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/chinanet6.txt) |
| 中国移动 | [IPv4](https://china-operator-ip.yfgao.com/cmcc.txt) \| [IPv6](https://china-operator-ip.yfgao.com/cmcc6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/cmcc.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/cmcc6.txt) |
| 中国联通 | [IPv4](https://china-operator-ip.yfgao.com/unicom.txt) \| [IPv6](https://china-operator-ip.yfgao.com/unicom6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/unicom.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/unicom6.txt) |
| 中国铁通 | [IPv4](https://china-operator-ip.yfgao.com/tietong.txt) \| [IPv6](https://china-operator-ip.yfgao.com/tietong6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/tietong.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/tietong6.txt) |
| 教育网 | [IPv4](https://china-operator-ip.yfgao.com/cernet.txt) \| [IPv6](https://china-operator-ip.yfgao.com/cernet6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/cernet.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/cernet6.txt) |
| 科技网 | [IPv4](https://china-operator-ip.yfgao.com/cstnet.txt) \| [IPv6](https://china-operator-ip.yfgao.com/cstnet6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/cstnet.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/cstnet6.txt) |
| 鹏博士 | [IPv4](https://china-operator-ip.yfgao.com/drpeng.txt) \| [IPv6](https://china-operator-ip.yfgao.com/drpeng6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/drpeng.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/drpeng6.txt) |
| 谷歌中国 | [IPv4](https://china-operator-ip.yfgao.com/googlecn.txt) \| [IPv6](https://china-operator-ip.yfgao.com/googlecn6.txt) | [IPv4](https://gaoyifan.github.io/china-operator-ip/googlecn.txt) \| [IPv6](https://gaoyifan.github.io/china-operator-ip/googlecn6.txt) |
| 统计 | [stat](https://china-operator-ip.yfgao.com/stat) | [stat](https://gaoyifan.github.io/china-operator-ip/stat) |
镜像说明:
* **EdgeOne Pages**: 中国大陆境内完整镜像
* **GitHub Pages**: 海外完整镜像
* **jsDelivr**: 海外CDN缓存
### 方法2从BGP数据生成
#### 安装依赖
* [bgptools](https://github.com/gaoyifan/bgptools) (`cargo install bgptools --version 0.0.3`)
* [bgpdump](https://bitbucket.org/ripencc/bgpdump-hg/wiki/Home) (`apt install bgpdump`)
* [cidr-merger](https://github.com/zhanhb/cidr-merger) (`go get github.com/zhanhb/cidr-merger`)
* [just](https://github.com/casey/just?tab=readme-ov-file#installation)
* [Rust Toolchain](https://www.rust-lang.org/tools/install)
* [bgpkit-broker](https://github.com/bgpkit/bgpkit-broker) (`cargo install bgpkit-broker@0.7.0`)
* [bgptools](https://github.com/gaoyifan/bgptools) (`cargo install bgptools --version 0.2.2`)
#### 生成IP列表
```shell
./generate.sh
just
```
#### 统计IP数量
```shell
./stat.sh
```
注:执行 `just --list` 查看所有可用的命令。
## 社区关联项目
- [OneOhCloud/One-GeoIP](https://github.com/OneOhCloud/one-geoip): 每日更新的适用于 sing-box 的规则集
- [fcshark-org/route-list](https://github.com/fcshark-org/route-list): 每日更新的适用于 dnsmasq 的规则集
- [OneOhCloud/One-GeoIP](https://github.com/OneOhCloud/one-geoip): 适用于 sing-box 的规则集
- [fcshark-org/route-list](https://github.com/fcshark-org/route-list): 适用于 dnsmasq 的规则集
- [zxlhhyccc/smartdns-list-scripts](https://github.com/zxlhhyccc/smartdns-list-scripts): smartdns 使用的规则集
## 致谢
## Acknowledgments
* 感谢[boj](https://ring0.me)师兄提出的[设计建议](https://github.com/ustclug/discussions/issues/79#issuecomment-267958775)
* 感谢[University of Oregon Route Views Archive Project](http://archive.routeviews.org)项目提供BGP数据源
* 感谢[Travis CI](https://travis-ci.org)提供优秀的持续集成平台
* 感谢[GitHub Action](https://github.com/features/actions)提供计算资源
* 感谢[cidr-merger](https://github.com/zhanhb/cidr-merger)项目提供高效的IP地址合并工具
* 感谢[bgpdump](https://bitbucket.org/ripencc/bgpdump/wiki/Home)项目提供rib数据的读取工具
* 感谢[Tencent EdgeOne](https://edgeone.ai/zh?from=github)为本项目提供 CDN 加速及安全防护赞助
<a href="https://edgeone.ai/zh?from=github">
<img src="https://edgeone.ai/media/34fe3a45-492d-4ea4-ae5d-ea1087ca7b4b.png" width="270" height="38" alt="34fe3a45-492d-4ea4-ae5d-ea1087ca7b4b">
</a>
* 感谢[boj](https://ring0.me)师兄提出的[设计思路](https://github.com/ustclug/discussions/issues/79#issuecomment-267958775)
* [bgpkit](https://bgpkit.com)
* [University of Oregon Route Views Archive Project](http://archive.routeviews.org)
* [GitHub Action](https://github.com/features/actions)
* [Tencent EdgeOne](https://edgeone.ai/zh?from=github)
## 协议
## License
[MIT License](LICENSE)

View File

@@ -1,58 +0,0 @@
#!/usr/bin/env bash
[[ $DEBUG == true ]] && set -x
log_info(){
>&2 echo "INFO>" $@
}
get_asn(){
local CONF_FILE=$1
unset PATTERN
unset COUNTRY
unset EXCLUDE
source $CONF_FILE
EXCLUDE=${EXCLUDE:-"^$"}
grep -P "${COUNTRY}\$" asnames.txt |
grep -Pi "$PATTERN" |
grep -vPi "$EXCLUDE" |
awk '{gsub(/AS/, ""); print $1 }'
}
prepare_data_v4(){
rm -f rib.gz
bgpkit-broker latest -c rrc00 --json | jq -c '.[] | select( .data_type | contains("rib")) | .url' | head -n 1 | xargs axel -q -o rib.gz
stat rib.gz
log_info "runing bgpdump v4 ..."
bgpdump -m -O rib.txt rib.gz
stat rib.txt
log_info "finish bgpdump v4"
}
prepare_data_v6(){
rm -f rib6.gz
bgpkit-broker latest -c route-views6 --json | jq -c '.[] | select( .data_type | contains("rib")) | .url' | head -n 1 | xargs axel -q -o rib6.bz2
stat rib6.bz2
log_info "runing bgpdump v6 ..."
bgpdump -m -O rib6.txt rib6.bz2
stat rib6.txt
log_info "finish bgpdump v6"
}
prepare_data(){
curl -sSL https://bgp.potaroo.net/cidr/autnums.html | awk '-F[<>]' '{print $3,$5}' | grep '^AS' > asnames.txt &
prepare_data_v4 &
prepare_data_v6 &
wait_exit
}
wait_exit(){
local oldstate=$(set +o)
set +e
local s=0
while [[ $s -ne 127 ]]; do
[[ $s -ne 0 ]] && exit $s
wait -n
s=$?
done
eval "$oldstate"
return 0
}

View File

@@ -1,20 +0,0 @@
#!/usr/bin/env bash
set -e
cidr-merger --version || \
go install github.com/zhanhb/cidr-merger@v1.1.2
bgptools --version | grep -F $BGPTOOLS_VERSION || \
cargo install --version $BGPTOOLS_VERSION bgptools
bgpkit-parser --version || \
cargo binstall --secure --no-confirm bgpkit-parser@0.10.6
bgpkit-broker --version || \
cargo binstall --secure --no-confirm bgpkit-broker@0.7.0
cidr-merger --version
bgptools --version
bgpkit-parser --version
bgpkit-broker --version

View File

@@ -1,16 +0,0 @@
#!/usr/bin/env bash
set -e
source common.sh
[[ $SKIP_DATA_PREPARATION != true ]] && prepare_data
mkdir -p result
for file in operator/*.conf; do
operator=${file%.*}
operator=${operator##*/}
log_info "generating IP list of $operator ..."
get_asn $file
get_asn $file | xargs bgptools -b rib.txt | cidr-merger -s | grep -Fv : | cat > result/${operator}.txt &
get_asn $file | xargs bgptools -b rib6.txt | grep -v '^::/0$' | cidr-merger -s | grep -F : | cat > result/${operator}6.txt &
done
wait_exit

View File

@@ -1,7 +0,0 @@
#!/bin/bash
[[ $(wc -l result/china.txt) < 3000 ]] && exit 1
[[ $(wc -l result/china6.txt) < 1000 ]] && exit 2
exit 0

215
justfile Normal file
View File

@@ -0,0 +1,215 @@
set unstable
set script-interpreter := ['bash']
bgptools_version := "0.2.2"
default: prepare all stat
[doc('Install or update bgp tooling dependencies')]
[script]
dependency:
set -euo pipefail
if ! bgptools --version 2>/dev/null | grep -F "{{bgptools_version}}" >/dev/null; then
cargo install --version "{{bgptools_version}}" bgptools
fi
if ! bgpkit-broker --version >/dev/null 2>&1; then
cargo binstall --secure --no-confirm bgpkit-broker@0.7.0
fi
bgptools --version
bgpkit-broker --version
[doc('Download and normalize latest autnums list')]
[script]
prepare_autnums:
set -euo pipefail
axel -q -o autnums.html https://bgp.potaroo.net/cidr/autnums.html
awk -F'[<>]' '{print $3,$5}' autnums.html | grep '^AS' > asnames.txt
rm -f autnums.html
echo "INFO> asnames.txt updated ($(wc -l < asnames.txt) entries)" >&2
[doc('Download the latest IPv4 RIB snapshot')]
[script]
prepare_rib_v4:
set -euo pipefail
url="$(bgpkit-broker latest -c rrc00 --json \
| jq -r '.[] | select(.data_type | contains("rib")) | .url' \
| head -n 1)"
if [[ -z "${url}" ]]; then
echo "Unable to determine IPv4 RIB download url" >&2
exit 1
fi
rm -f rib.gz
axel -q -o rib.gz "${url}"
stat rib.gz
echo "INFO> rib.gz ready for bgptools" >&2
[doc('Download the latest IPv6 RIB snapshot')]
[script]
prepare_rib_v6:
set -euo pipefail
url="$(bgpkit-broker latest -c route-views6 --json \
| jq -r '.[] | select(.data_type | contains("rib")) | .url' \
| head -n 1)"
if [[ -z "${url}" ]]; then
echo "Unable to determine IPv6 RIB download url" >&2
exit 1
fi
rm -f rib6.bz2
axel -q -o rib6.bz2 "${url}"
stat rib6.bz2
echo "INFO> rib6.bz2 ready for bgptools" >&2
[doc('Prepare data for generation')]
[parallel]
prepare: prepare_autnums prepare_rib_v4 prepare_rib_v6
[doc('Print ASN list for OPERATOR based on operator/*.conf')]
[script]
get_asn operator:
set -euo pipefail
config="operator/{{operator}}.conf"
if [[ ! -f "${config}" ]]; then
echo "Unknown operator: {{operator}}" >&2
exit 1
fi
if [[ ! -s asnames.txt ]]; then
echo "Missing asnames.txt. Run 'just prepare_autnums' first." >&2
exit 1
fi
# shellcheck disable=SC1090
source "${config}"
: "${COUNTRY:?COUNTRY must be set in ${config}}"
EXCLUDE="${EXCLUDE:-^$}"
PATTERN="${PATTERN:-}"
grep -P "${COUNTRY}\$" asnames.txt \
| grep -Pi "${PATTERN}" \
| grep -vPi "${EXCLUDE}" \
| awk '{gsub(/AS/, ""); print $1 }'
[doc('Generate IP lists for a single operator')]
[parallel]
gen operator: (gen4 operator) (gen6 operator)
[script]
gen4 operator:
set -euo pipefail
mkdir -p result
echo "INFO> generating IPv4 prefixes for {{operator}}" >&2
just get_asn "{{operator}}" \
| tee >(awk 'END { if (NR == 0) exit 1 }') \
| xargs bgptools --mrt-file rib.gz \
| grep -Fv ':' \
> "result/{{operator}}.txt"
echo "INFO> {{operator}}.txt generated ($(wc -l < result/{{operator}}.txt) entries)" >&2
[script]
gen6 operator:
set -euo pipefail
mkdir -p result
echo "INFO> generating IPv6 prefixes for {{operator}}" >&2
just get_asn "{{operator}}" \
| tee >(awk 'END { if (NR == 0) exit 1 }') \
| xargs bgptools --mrt-file rib6.bz2 \
| grep -v '^::/0$' \
| grep -F ':' \
> "result/{{operator}}6.txt" || true # ignore empty output, since drpeng has no IPv6 prefixes
echo "INFO> {{operator}}6.txt generated ($(wc -l < result/{{operator}}6.txt) entries)" >&2
[doc('Generate IP lists for all operators sequentially')]
[parallel]
all: (gen "china") (gen "cernet") (gen "chinanet") (gen "cmcc") (gen "unicom") (gen "cstnet") (gen "drpeng") (gen "googlecn")
[script]
guard:
set -euo pipefail
if [[ $(wc -l < result/china.txt) -lt 3000 ]]; then
echo "china.txt too small" >&2
exit 1
fi
if [[ $(wc -l < result/china6.txt) -lt 1000 ]]; then
echo "china6.txt too small" >&2
exit 2
fi
echo "INFO> guard checks passed" >&2
[doc('Summarize total IPv4/IPv6 address space per operator')]
[script]
stat:
set -euo pipefail
cd result
for file in *.txt; do
name="${file%.*}"
echo "${name}"
if [[ "${file}" == *6.txt ]]; then
base=48
else
base=32
fi
sum=0
while IFS=/ read -r _ mask; do
if [[ -z "${mask}" ]]; then
continue
fi
if (( mask <= base )); then
((s=base-mask))
((sum+=1<<s))
fi
done < "${file}"
echo "${sum}"
echo
done | tee stat
[doc('Publish generated results into the ip-lists branch')]
[script]
upload: guard
set -euo pipefail
rm -f ip-lists/*.txt
mv result/* ip-lists
cd ip-lists
tree -H . -P "*.txt|stat" -T "China Operator IP - prebuild results" > index.html
git config user.name "GitHub Actions"
git config user.email noreply@github.com
git add .
git commit -m "update $(date +%Y-%m-%d)"
git push -q
[doc('Refresh CDN cache for all files in ip-lists directory')]
[script]
refresh_jsdelivr repository:
set -euo pipefail
if [[ ! -d ip-lists ]]; then
echo "ip-lists directory not found" >&2
exit 1
fi
cd ip-lists
for file in *; do
if [[ -f "${file}" ]]; then
echo "INFO> purging CDN cache for ${file}" >&2
curl -i "https://purge.jsdelivr.net/gh/{{repository}}@ip-lists/${file}"
fi
done

26
stat.sh
View File

@@ -1,26 +0,0 @@
#!/usr/bin/env bash
source common.sh
cd result
for file in *.txt; do
echo ${file%.*}
if [[ $file == *6.txt ]]; then
#statistics IPv6 /48 blocks
base=48
else
base=32
fi
cat $file |
awk -F\/ '{print $2}' |
(
sum=0
while read n; do
if [[ -n $n ]] && [[ $n -le $base ]]; then
((s=base-n))
((sum+=1<<s))
fi
done
echo $sum
)
echo
done | tee stat

View File

@@ -1,11 +0,0 @@
#!/usr/bin/env bash
rm ip-lists/*.txt
mv result/* ip-lists
cd ip-lists
tree -H . -P "*.txt|stat" -T "China Operator IP - prebuild results" > index.html
git config user.name "GitHub Actions"
git config user.email noreply@github.com
git add .
git commit -m "update $(date +%Y-%m-%d)"
git push -q