Merge branch 'main' into admin-page

This commit is contained in:
felinae98 2021-11-18 19:23:31 +08:00
commit ea08f1e681
No known key found for this signature in database
GPG Key ID: 00C8B010587FF610
43 changed files with 574 additions and 9233 deletions

View File

@ -9,6 +9,7 @@ orbs:
node: circleci/node@4.7.0
# poetry: frameio/poetry@0.21.0
swissknife: roopakv/swissknife@0.59.0
docker: circleci/docker@1.7.0
workflows:
build-test-publish:
@ -31,6 +32,21 @@ workflows:
ignore: /.*/
tags:
only: /^v.*/
- docker/publish:
requires:
- test
filters:
branches:
ignore: /.*/
tags:
only: /^v.*/
context:
- docker
image: $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
tag: latest,${CIRCLE_TAG}
update-description: true
docker-username: DOCKERHUB_USERNAME
docker-password: DOCKERHUB_PASSWORD
jobs:
build-frontend:
@ -54,7 +70,7 @@ jobs:
- image: cimg/python:3.9
- image: browserless/chrome
environment:
HK_REPORTER_BROWSER: ws://localhost:3000
BISON_BROWSER: ws://localhost:3000
steps:
- checkout
# - run: sed -e '41,45d' -i pyproject.toml

View File

@ -17,7 +17,13 @@
- 发送RSS订阅的title
- 修复浏览器渲染问题
## [0.3.2]
## [0.3.2] - 2021-09-28
- 增加NoTargetGroup
- 增加1x3拼图的支持
- 增加网易云
## [0.3.3] - 2021-09-28
- 修复拼图问题
## [0.4.0] - 2021-11-18
- 项目更名为nonebot-bison

View File

@ -1,10 +1,10 @@
FROM python:3.9
RUN python3 -m pip config set global.index-url https://mirrors.aliyun.com/pypi/simple
RUN python3 -m pip install poetry && poetry config virtualenvs.create false
WORKDIR /app
COPY ./pyproject.toml ./poetry.lock* /app/
RUN poetry install --no-root --no-dev
# RUN PYPPETEER_DOWNLOAD_HOST='http://npm.taobao.org/mirrors' pyppeteer-install
COPY . /app/
ADD src /app/src
ADD bot.py /app/
ENV HOST=0.0.0.0
CMD ["python", "bot.py"]

View File

@ -6,6 +6,7 @@ WORKDIR /app
COPY ./pyproject.toml ./poetry.lock* /app/
RUN poetry install --no-root --no-dev
# RUN PYPPETEER_DOWNLOAD_HOST='http://npm.taobao.org/mirrors' pyppeteer-install
COPY . /app/
ENV HK_REPORTER_BROWSER=local:/usr/bin/chromium
ADD src /app/src
ADD bot.py /app/
ENV HOST=0.0.0.0
CMD ["python", "bot.py"]

11
Dockerfile_tuna Normal file
View File

@ -0,0 +1,11 @@
FROM python:3.9
RUN python3 -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
RUN python3 -m pip install poetry && poetry config virtualenvs.create false
WORKDIR /app
COPY ./pyproject.toml ./poetry.lock* /app/
RUN poetry install --no-root --no-dev
# RUN PYPPETEER_DOWNLOAD_HOST='http://npm.taobao.org/mirrors' pyppeteer-install
ADD src /app/src
ADD bot.py /app/
ENV HOST=0.0.0.0
CMD ["python", "bot.py"]

View File

@ -1,12 +0,0 @@
module.exports = {
title: 'Nonebot HK Reporter',
description: 'Docs for Nonebot HK Reporter',
themeConfig: {
nav: [
{ text: '主页', link: '/' },
{ text: '部署与使用', link: '/usage/' },
{ text: '开发', link: '/dev/' },
{ text: 'Github', link: 'https://github.com/felinae98/nonebot-hk-reporter' }
]
}
}

View File

@ -1,15 +0,0 @@
---
home: true
heroText: Nonebot HK Reporter
tagline: 本bot励志做全世界跑得最快的搬运机器人
actionText: 快速部署
actionLink: /usage/
features:
- title: KISS
details: 作为插件可以Simple和Stupid作为插件可以Simple和Stupid作为Bot提供适用的功能
- title: 拓展性强
details: 没有自己想要的网站?只要简单的爬虫知识就可以给它适配一个新的网站
- title: 通用,强大
details: 社交媒体?网站更新?游戏开服?只要能爬就都能推,还支持自定义过滤
footer: MIT Licensed
---

View File

@ -1,53 +0,0 @@
---
sidebar: auto
---
# 开发指南
本插件需要你的帮助!只需要会写简单的爬虫,就能给本插件适配新的网站。
## 基本概念
* `nonebot_hk_reporter.post.Post`: 可以理解为推送内容,其中包含需要发送的文字,图片,链接,平台信息等
* `nonebot_hk_reporter.types.RawPost`: 从站点/平台中爬到的单条信息
* `nonebot_hk_reporter.types.Target`: 目标账号Bilibili微博等社交媒体中的账号
* `nonebot_hk_reporter.types.Category`: 信息分类,例如视频,动态,图文,文章等
* `nonebot_hk_reporter.types.Tag`: 信息标签例如微博中的超话或者hashtag
## 快速上手
上车!我们走
先明确需要适配的站点类型,先明确两个问题:
#### 我要发送什么样的推送
* `nonebot_hk_reporter.platform.platform.NewMessage` 最常见的类型,每次爬虫向特定接口爬取一个消息列表,
与之前爬取的信息对比,过滤出新的消息,再根据用户自定义的分类和标签进行过滤,最后处理消息,把
处理过后的消息发送给用户
例如微博Bilibili
* `nonebot_hk_reporter.platform.platform.StatusChange` 每次爬虫获取一个状态,在状态改变时发布推送
例如:游戏开服提醒,主播上播提醒
* `nonebot_hk_reporter.platform.platform.SimplePost``NewMessage`相似,但是不过滤新的消息
,每次发送全部消息
例如:每日榜单定时发送
#### 这个平台是否有账号的概念
* `nonebot_hk_reporter.platform.platform.TargetMixin` 有账号的概念
例如Bilibili用户微博用户
* `nonebot_hk_reporter.platform.platform.NoTargetMixin` 没有账号的概念
例如:游戏公告,教务处公告
现在你已经选择了两个类,现在你需要在`src/plugins/nonebot_hk_reporter/platform`下新建一个py文件
在里面新建一个类,继承你刚刚选择的两个类,重载一些关键的函数,然后……就完成了,不需要修改别的东西了。
例如要适配微博微博有账号并且我希望bot搬运新的消息所以微博的类应该这样定义
```python
class Weibo(NewMessage, TargetMixin):
...
```
当然我们非常希望你对自己适配的平台写一些单元测试,你可以模仿`tests/platforms/test_*.py`中的内容写
一些单元测试。为保证多次运行测试的一致性可以mock http的响应测试的内容包括获取RawPost处理成Post
测试分类以及提取tag等当然最好和rsshub做一个交叉验证。
::: danger
Nonebot项目使用了全异步的处理方式所以你需要对异步Python asyncio的机制有一定了解当然
依葫芦画瓢也是足够的
:::
## 类的方法与成员变量
## 方法与变量的定义

View File

@ -1,100 +0,0 @@
---
sidebar: auto
---
# 部署和使用
本节将教你快速部署和使用一个nonebot-hk-reporter如果你不知道要选择哪种部署方式推荐使用[docker-compose](#docker-compose部署-推荐)
## 部署
本项目可以作为单独的Bot使用可以作为nonebot2的插件使用
### 作为Bot使用
额外提供自动同意超级用户的好友申请和同意超级用户的加群邀请的功能
#### docker-compose部署推荐
1. 在一个新的目录中下载[docker-compose.yml](https://raw.githubusercontent.com/felinae98/nonebot-hk-reporter/main/docker-compose.yml)
将其中的`<your QQ>`改成自己的QQ号
```bash
wget https://raw.githubusercontent.com/felinae98/nonebot-hk-reporter/main/docker-compose.yml
```
2. 运行配置go-cqhttp
```bash
docker-compose run go-cqhttp
```
通信方式选择:`3: 反向 Websocket 通信`
编辑`bot-data/config.yml`,更改下面字段:
```
account: # 账号相关
uin: <QQ号> # QQ账号
password: "<QQ密码>" # 密码为空时使用扫码登录
message:
post-format: array
............
servers:
- ws-reverse:
universal: ws://nonebot:8080/cqhttp/ws # 将这个字段写为这个值
```
3. 登录go-cqhttp
再次
```bash
docker-compose run go-cqhttp
```
参考[go-cqhttp文档](https://docs.go-cqhttp.org/faq/slider.html#%E6%96%B9%E6%A1%88a-%E8%87%AA%E8%A1%8C%E6%8A%93%E5%8C%85)
完成登录
4. 确定完成登录后启动bot
```bash
docker-compose up -d
```
#### docker部署
本项目的docker镜像为`felinae98/nonebot-hk-reporter`可以直接pull后run进行使用
相关配置参数可以使用`-e`作为环境变量传入
#### 直接运行(不推荐)
可以参考[nonebot的运行方法](https://v2.nonebot.dev/guide/getting-started.html)
::: danger
本项目中使用了Python 3.9的语法如果出现问题请检查Python版本
:::
1. 首先安装poetry[安装方法](https://python-poetry.org/docs/#installation)
2. clone本项目在项目中`poetry install`安装依赖
3. 编辑`.env.prod`配置各种环境变量,见[Nonebot2配置](https://v2.nonebot.dev/guide/basic-configuration.html)
4. 运行`poetry run python bot.py`启动机器人
### 作为插件使用
本部分假设大家会部署nonebot2
#### 手动安装
1. 安装pip包`nonebot-hk-reporter`
2. 在`bot.py`中导入插件`nonebot_hk_reporter`
### 自动安装
使用`nb-cli`执行:`nb plugin install nonebot_hk_reporter`
## 配置
可参考[源文件](https://github.com/felinae98/nonebot-hk-reporter/blob/main/src/plugins/nonebot_hk_reporter/plugin_config.py)
* `HK_REPORTER_CONFIG_PATH`: 插件存放配置文件的位置,如果不设定默认为项目目录下的`data`目录
* `HK_REPORTER_USE_PIC`: 将文字渲染成图片后进行发送,多用于规避风控
* `HK_REPORTER_BROWSER`: 在某些情况下需要使用到chrome进行渲染
* 使用browserless提供的Chrome管理服务设置为`ws://xxxxxxxx`值为Chrome Endpoint推荐
* 使用本地安装的Chrome设置为`local:<chrome path>`,例如`local:/usr/bin/google-chrome-stable`
* 如果不进行配置那么会在使用到Chrome的时候自动进行安装不推荐
### 需要使用Chrome的情况
* 设置了`HK_REPORTER_USE_PIC`,需要将文字渲染成图片
* 渲染明日方舟游戏内公告
## 使用
::: warning
本节假设`COMMAND_START`设置中包含`''`如果出现bot不响应的问题请先
排查这个设置
:::
### 命令
#### 在本群中进行配置
所有命令都需要@bot触发
* 添加订阅仅管理员和群主和SUPERUSER`添加订阅`
* 查询订阅:`查询订阅`
* 删除订阅仅管理员和群主和SUPERUSER`删除订阅`
#### 私聊机器人进行配置需要SUPERUER权限
* 添加订阅:`管理-添加订阅`
* 查询订阅:`管理-查询订阅`
* 删除订阅:`管理-删除订阅`
### 所支持平台的uid
#### Weibo
* 对于一般用户主页`https://weibo.com/u/6441489862?xxxxxxxxxxxxxxx``/u/`后面的数字即为uid
* 对于有个性域名的用户如:`https://weibo.com/arknights`,需要点击左侧信息标签下“更多”,链接为`https://weibo.com/6279793937/about`其中中间数字即为uid
#### Bilibili
主页链接一般为`https://space.bilibili.com/161775300?xxxxxxxxxx`数字即为uid
#### RSS
RSS链接即为uid

View File

@ -1,17 +0,0 @@
{
"name": "nonebot-hk-reporter-docs",
"version": "1.0.0",
"description": "Docs for nonebot-hk-reporter",
"main": "index.js",
"repository": "git@github.com:felinae98/nonebot-hk-reporter.git",
"author": "felinae98 <731499577@qq.com>",
"license": "MIT",
"private": false,
"devDependencies": {
"vuepress": "^1.8.2"
},
"scripts": {
"docs:dev": "vuepress dev docs",
"docs:build": "vuepress build docs"
}
}

1060
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,15 +1,15 @@
[tool.poetry]
name = "nonebot-hk-reporter"
version = "0.3.2"
name = "nonebot-bison"
version = "0.4.0"
description = "Subscribe message from social medias"
authors = ["felinae98 <felinae225@qq.com>"]
license = "MIT"
homepage = "https://github.com/felinae98/nonebot-hk-reporter"
homepage = "https://github.com/felinae98/nonebot-bison"
keywords = ["nonebot", "nonebot2", "qqbot"]
readme = "README.md"
packages = [
{ include = "nonebot_hk_reporter/*.py", from = "./src/plugins/" },
{ include = "nonebot_hk_reporter/platform/*.py", from = "./src/plugins/" }
{ include = "nonebot_bison/*.py", from = "./src/plugins/" },
{ include = "nonebot_bison/platform/*.py", from = "./src/plugins/" }
]
classifiers = [
"Development Status :: 2 - Pre-Alpha",
@ -39,18 +39,13 @@ jinja2 = "^3.0.1"
ipdb = "^0.13.4"
pytest = "^6.2.4"
pytest-asyncio = "^0.15.1"
respx = "^0.17.0"
respx = "^0.19.0"
coverage = "^5.5"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
[[tool.poetry.source]]
name = "aliyun"
url = "https://mirrors.aliyun.com/pypi/simple/"
default = true
[tool.pytest.ini_options]
markers = [
"compare: compare fetching result with rsshub",

View File

@ -14,14 +14,18 @@ from .platform import platform_manager
supported_target_type = platform_manager.keys()
def get_config_path() -> str:
if plugin_config.hk_reporter_config_path:
data_dir = plugin_config.hk_reporter_config_path
if plugin_config.bison_config_path:
data_dir = plugin_config.bison_config_path
else:
working_dir = os.getcwd()
data_dir = path.join(working_dir, 'data')
if not path.isdir(data_dir):
os.makedirs(data_dir)
return path.join(data_dir, 'hk_reporter.json')
old_path = path.join(data_dir, 'hk_reporter.json')
new_path = path.join(data_dir, 'bison.json')
if os.path.exists(old_path) and not os.path.exists(new_path):
os.rename(old_path, new_path)
return new_path
class NoSuchUserException(Exception):
pass

View File

@ -10,10 +10,14 @@ from nonebot.matcher import Matcher
from .config import Config, NoSuchSubscribeException
from .platform import platform_manager, check_sub_target
from .send import send_msgs
from .utils import parse_text
from .types import Target
def _gen_prompt_template(prompt: str):
if hasattr(Message, 'template'):
return Message.template(prompt)
return prompt
common_platform = [p.platform_name for p in \
filter(lambda platform: platform.enabled and platform.is_common,
platform_manager.values())
@ -29,7 +33,7 @@ async def send_help(bot: Bot, event: Event, state: T_State):
def do_add_sub(add_sub: Type[Matcher]):
@add_sub.handle()
async def init_promote(bot: Bot, event: Event, state: T_State):
state['_prompt'] = '请输入想要订阅的平台,目前支持\n' + \
state['_prompt'] = '请输入想要订阅的平台,目前支持,请输入冒号左边的名称\n' + \
''.join(['{}{}\n'.format(platform_name, platform_manager[platform_name].name) \
for platform_name in common_platform]) + \
'要查看全部平台请输入:“全部”'
@ -46,11 +50,11 @@ def do_add_sub(add_sub: Type[Matcher]):
else:
await add_sub.reject('平台输入错误')
@add_sub.got('platform', '{_prompt}', parse_platform)
@add_sub.got('platform', _gen_prompt_template('{_prompt}'), parse_platform)
@add_sub.handle()
async def init_id(bot: Bot, event: Event, state: T_State):
if platform_manager[state['platform']].has_target:
state['_prompt'] = '请输入订阅用户的id详情查阅https://nonebot-hk-reporter.vercel.app/usage/#%E6%89%80%E6%94%AF%E6%8C%81%E5%B9%B3%E5%8F%B0%E7%9A%84uid'
state['_prompt'] = '请输入订阅用户的id详情查阅https://nonebot-bison.vercel.app/usage/#%E6%89%80%E6%94%AF%E6%8C%81%E5%B9%B3%E5%8F%B0%E7%9A%84uid'
else:
state['id'] = 'default'
state['name'] = await platform_manager[state['platform']].get_target_name(Target(''))
@ -63,14 +67,14 @@ def do_add_sub(add_sub: Type[Matcher]):
state['id'] = target
state['name'] = name
@add_sub.got('id', '{_prompt}', parse_id)
@add_sub.got('id', _gen_prompt_template('{_prompt}'), parse_id)
@add_sub.handle()
async def init_cat(bot: Bot, event: Event, state: T_State):
if not platform_manager[state['platform']].categories:
state['cats'] = []
return
state['_prompt'] = '请输入要订阅的类别,以空格分隔,支持的类别有:{}'.format(
','.join(list(platform_manager[state['platform']].categories.values())))
' '.join(list(platform_manager[state['platform']].categories.values())))
async def parser_cats(bot: Bot, event: Event, state: T_State):
res = []
@ -80,7 +84,7 @@ def do_add_sub(add_sub: Type[Matcher]):
res.append(platform_manager[state['platform']].reverse_category[cat])
state['cats'] = res
@add_sub.got('cats', '{_prompt}', parser_cats)
@add_sub.got('cats', _gen_prompt_template('{_prompt}'), parser_cats)
@add_sub.handle()
async def init_tag(bot: Bot, event: Event, state: T_State):
if not platform_manager[state['platform']].enable_tag:
@ -94,7 +98,7 @@ def do_add_sub(add_sub: Type[Matcher]):
else:
state['tags'] = str(event.get_message()).strip().split()
@add_sub.got('tags', '{_prompt}', parser_tags)
@add_sub.got('tags', _gen_prompt_template('{_prompt}'), parser_tags)
@add_sub.handle()
async def add_sub_process(bot: Bot, event: Event, state: T_State):
config = Config()
@ -118,7 +122,6 @@ def do_query_sub(query_sub: Type[Matcher]):
if platform.enable_tag:
res += ' {}'.format(', '.join(sub['tags']))
res += '\n'
# send_msgs(bot, event.group_id, 'group', [await parse_text(res)])
await query_sub.finish(Message(await parse_text(res)))
def do_del_sub(del_sub: Type[Matcher]):

View File

@ -1,14 +1,13 @@
from typing import Any
import httpx
import json
from typing import Any
from bs4 import BeautifulSoup as bs
import httpx
from ..types import Category, RawPost, Target
from .platform import NewMessage, NoTargetMixin, CategoryNotSupport, StatusChange
from ..utils import Render
from ..post import Post
from ..types import Category, RawPost, Target
from ..utils import Render
from .platform import CategoryNotSupport, NewMessage, NoTargetMixin, StatusChange
class Arknights(NewMessage, NoTargetMixin):
@ -41,6 +40,7 @@ class Arknights(NewMessage, NoTargetMixin):
async def parse(self, raw_post: RawPost) -> Post:
announce_url = raw_post['webUrl']
text = ''
async with httpx.AsyncClient() as client:
raw_html = await client.get(announce_url)
soup = bs(raw_html, 'html.parser')
@ -50,12 +50,15 @@ class Arknights(NewMessage, NoTargetMixin):
render = Render()
viewport = {'width': 320, 'height': 6400, 'deviceScaleFactor': 3}
pic_data = await render.render(announce_url, viewport=viewport, target='div.main')
pics.append(pic_data)
if pic_data:
pics.append(pic_data)
else:
text = '图片渲染失败'
elif (pic := soup.find('img', class_='banner-image')):
pics.append(pic['src'])
else:
raise CategoryNotSupport()
return Post('arknights', text='', url='', target_name="明日方舟游戏内公告", pics=pics, compress=True, override_use_pic=False)
return Post('arknights', text=text, url='', target_name="明日方舟游戏内公告", pics=pics, compress=True, override_use_pic=False)
class AkVersion(NoTargetMixin, StatusChange):
@ -82,9 +85,13 @@ class AkVersion(NoTargetMixin, StatusChange):
def compare_status(self, _, old_status, new_status):
res = []
if old_status.get('preAnnounceType') == 2 and new_status.get('preAnnounceType') == 0:
res.append(Post('arknights', text='开始维护!', target_name='明日方舟更新信息'))
res.append(Post('arknights',
text='登录界面维护公告上线(大概是开始维护了)',
target_name='明日方舟更新信息'))
elif old_status.get('preAnnounceType') == 0 and new_status.get('preAnnounceType') == 2:
res.append(Post('arknights', text='维护结束!冲!(可能不太准确)', target_name='明日方舟更新信息'))
res.append(Post('arknights',
text='登录界面维护公告下线(大概是开服了,冲!)',
target_name='明日方舟更新信息'))
if old_status.get('clientVersion') != new_status.get('clientVersion'):
res.append(Post('arknights', text='游戏本体更新(大更新)', target_name='明日方舟更新信息'))
if old_status.get('resVersion') != new_status.get('resVersion'):
@ -96,3 +103,45 @@ class AkVersion(NoTargetMixin, StatusChange):
async def parse(self, raw_post):
return raw_post
class MonsterSiren(NewMessage, NoTargetMixin):
categories = {3: '塞壬唱片新闻'}
platform_name = 'arknights'
name = '明日方舟游戏信息'
enable_tag = False
enabled = True
is_common = False
schedule_type = 'interval'
schedule_kw = {'seconds': 30}
async def get_target_name(self, _: Target) -> str:
return '明日方舟游戏信息'
async def get_sub_list(self, _) -> list[RawPost]:
async with httpx.AsyncClient() as client:
raw_data = await client.get('https://monster-siren.hypergryph.com/api/news')
return raw_data.json()['data']['list']
def get_id(self, post: RawPost) -> Any:
return post['cid']
def get_date(self, _) -> None:
return None
def get_category(self, _) -> Category:
return Category(3)
async def parse(self, raw_post: RawPost) -> Post:
url = f'https://monster-siren.hypergryph.com/info/{raw_post["cid"]}'
async with httpx.AsyncClient() as client:
res = await client.get(f'https://monster-siren.hypergryph.com/api/news/{raw_post["cid"]}')
raw_data = res.json()
content = raw_data['data']['content']
content = content.replace('</p>', '</p>\n')
soup = bs(content, 'html.parser')
imgs = list(map(lambda x: x['src'], soup('img')))
text = f'{raw_post["title"]}\n{soup.text.strip()}'
return Post('monster-siren', text=text, pics=imgs,
url=url, target_name="塞壬唱片新闻", compress=True,
override_use_pic=False)

View File

@ -134,7 +134,7 @@ class MessageProcessMixin(PlatformNameMixin, CategoryMixin, ParsePostMixin, abst
# if post_id in exists_posts_set:
# continue
if (post_time := self.get_date(raw_post)) and time.time() - post_time > 2 * 60 * 60 and \
plugin_config.hk_reporter_init_filter:
plugin_config.bison_init_filter:
continue
try:
self.get_category(raw_post)
@ -157,7 +157,7 @@ class NewMessageProcessMixin(StorageMixinProto, MessageProcessMixin, abstract=Tr
filtered_post = await self.filter_common(raw_post_list)
store = self.get_stored_data(target) or self.MessageStorage(False, set())
res = []
if not store.inited and plugin_config.hk_reporter_init_filter:
if not store.inited and plugin_config.bison_init_filter:
# target not init
for raw_post in filtered_post:
post_id = self.get_id(raw_post)
@ -241,7 +241,7 @@ class Platform(PlatformNameMixin, UserCustomFilterMixin, base=True):
...
class NewMessage(
Platform,
Platform,
NewMessageProcessMixin,
UserCustomFilterMixin,
abstract=True
@ -306,6 +306,33 @@ class StatusChange(
logger.warning("network connection error: {}, url: {}".format(type(err), err.request.url))
return []
class SimplePost(
Platform,
MessageProcessMixin,
UserCustomFilterMixin,
StorageMixinProto,
abstract=True
):
"Fetch a list of messages, dispatch it to different users"
async def fetch_new_post(self, target: Target, users: list[UserSubInfo]) -> list[tuple[User, list[Post]]]:
try:
new_posts = await self.get_sub_list(target)
if not new_posts:
return []
else:
for post in new_posts:
logger.info('fetch new post from {} {}: {}'.format(
self.platform_name,
target if self.has_target else '-',
self.get_id(post)))
res = await self.dispatch_user_post(target, new_posts, users)
self.parse_cache = {}
return res
except httpx.RequestError as err:
logger.warning("network connection error: {}, url: {}".format(type(err), err.request.url))
return []
class NoTargetGroup(
Platform,
NoTargetMixin,

View File

@ -0,0 +1,22 @@
from pydantic import BaseSettings
import warnings
import nonebot
class PlugConfig(BaseSettings):
bison_config_path: str = ""
bison_use_pic: bool = False
bison_use_local: bool = False
bison_browser: str = ''
bison_init_filter: bool = True
bison_use_queue: bool = True
bison_outer_url: str = 'http://localhost:8080/bison/'
class Config:
extra = 'ignore'
global_config = nonebot.get_driver().config
plugin_config = PlugConfig(**global_config.dict())
if plugin_config.bison_use_local:
warnings.warn('BISON_USE_LOCAL is deprecated, please use BISON_BROWSER')

View File

@ -29,7 +29,7 @@ class Post:
def _use_pic(self):
if not self.override_use_pic is None:
return self.override_use_pic
return plugin_config.hk_reporter_use_pic
return plugin_config.bison_use_pic
async def _pic_url_to_image(self, data: Union[str, bytes]) -> Image.Image:
pic_buffer = BytesIO()
@ -110,7 +110,10 @@ class Post:
msgs = []
text = ''
if self.text:
text += '{}'.format(self.text if len(self.text) < 500 else self.text[:500] + '...')
if self._use_pic():
text += '{}'.format(self.text)
else:
text += '{}'.format(self.text if len(self.text) < 500 else self.text[:500] + '...')
text += '\n来源: {}'.format(self.target_type)
if self.target_name:
text += ' {}'.format(self.target_name)

View File

@ -11,6 +11,7 @@ from .platform import platform_manager
from .send import do_send_msgs
from .send import send_msgs
from .types import UserSubInfo
from .plugin_config import plugin_config
scheduler = AsyncIOScheduler()
@ -43,7 +44,7 @@ async def fetch_and_send(target_type: str):
if not bot:
logger.warning('no bot connected')
else:
send_msgs(bot, user.user, user.user_type, await send_post.generate_messages())
await send_msgs(bot, user.user, user.user_type, await send_post.generate_messages())
for platform_name, platform in platform_manager.items():
if platform.schedule_type in ['cron', 'interval', 'date']:
@ -52,16 +53,17 @@ for platform_name, platform in platform_manager.items():
fetch_and_send, platform.schedule_type, **platform.schedule_kw,
args=(platform_name,))
scheduler.add_job(do_send_msgs, 'interval', seconds=0.3, coalesce=True)
if plugin_config.bison_use_queue:
scheduler.add_job(do_send_msgs, 'interval', seconds=0.3, coalesce=True)
class SchedulerLogFilter(logging.Filter):
class SchedulerLogFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
logger.debug("logRecord", record, record.getMessage())
return not (record.name == "apscheduler" and 'skipped: maximum number of running instances reached' in record.getMessage())
def filter(self, record: logging.LogRecord) -> bool:
logger.debug("logRecord", record, record.getMessage())
return not (record.name == "apscheduler" and 'skipped: maximum number of running instances reached' in record.getMessage())
aps_logger = logging.getLogger("apscheduler")
aps_logger.setLevel(30)
aps_logger.addFilter(SchedulerLogFilter())
aps_logger.handlers.clear()
aps_logger.addHandler(LoguruHandler())
aps_logger = logging.getLogger("apscheduler")
aps_logger.setLevel(30)
aps_logger.addFilter(SchedulerLogFilter())
aps_logger.handlers.clear()
aps_logger.addHandler(LoguruHandler())

View File

@ -0,0 +1,43 @@
import time
from nonebot import logger
from nonebot.adapters.cqhttp.bot import Bot
from .plugin_config import plugin_config
QUEUE = []
LAST_SEND_TIME = time.time()
async def _do_send(bot: 'Bot', user: str, user_type: str, msg):
if user_type == 'group':
await bot.call_api('send_group_msg', group_id=user, message=msg)
elif user_type == 'private':
await bot.call_api('send_private_msg', user_id=user, message=msg)
async def do_send_msgs():
global LAST_SEND_TIME
if time.time() - LAST_SEND_TIME < 1.5:
return
if QUEUE:
bot, user, user_type, msg, retry_time = QUEUE.pop(0)
try:
await _do_send(bot, user, user_type, msg)
except Exception as e:
if retry_time > 0:
QUEUE.insert(0, (bot, user, user_type, msg, retry_time - 1))
else:
msg_str = str(msg)
if len(msg_str) > 50:
msg_str = msg_str[:50] + '...'
logger.warning(f'send msg err {e} {msg_str}')
LAST_SEND_TIME = time.time()
async def send_msgs(bot, user, user_type, msgs):
if plugin_config.bison_use_queue:
for msg in msgs:
QUEUE.append((bot, user, user_type, msg, 2))
else:
for msg in msgs:
await _do_send(bot, user, user_type, msg)

View File

@ -1,15 +1,20 @@
import asyncio
import base64
from html import escape
import os
from time import asctime
import re
from typing import Awaitable, Callable, Optional
from urllib.parse import quote
from nonebot.adapters.cqhttp.message import MessageSegment
from nonebot.adapters.cqhttp.message import MessageSegment
from nonebot.log import logger
from pyppeteer import connect, launch
from pyppeteer.browser import Browser
from pyppeteer.chromium_downloader import check_chromium, download_chromium
from pyppeteer.page import Page
from bs4 import BeautifulSoup as bs
from .plugin_config import plugin_config
class Singleton(type):
@ -19,6 +24,10 @@ class Singleton(type):
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
if not plugin_config.bison_browser and not plugin_config.bison_use_local \
and not check_chromium():
os.environ['PYPPETEER_DOWNLOAD_HOST'] = 'http://npm.taobao.org/mirrors'
download_chromium()
class Render(metaclass=Singleton):
@ -29,15 +38,15 @@ class Render(metaclass=Singleton):
self.remote_browser = False
async def get_browser(self) -> Browser:
if plugin_config.hk_reporter_browser:
if plugin_config.hk_reporter_browser.startswith('local:'):
path = plugin_config.hk_reporter_browser.split('local:', 1)[1]
if plugin_config.bison_browser:
if plugin_config.bison_browser.startswith('local:'):
path = plugin_config.bison_browser.split('local:', 1)[1]
return await launch(executablePath=path, args=['--no-sandbox'])
if plugin_config.hk_reporter_browser.startswith('ws:'):
if plugin_config.bison_browser.startswith('ws:'):
self.remote_browser = True
return await connect(browserWSEndpoint=plugin_config.hk_reporter_browser)
raise RuntimeError('HK_REPORTER_BROWSER error')
if plugin_config.hk_reporter_use_local:
return await connect(browserWSEndpoint=plugin_config.bison_browser)
raise RuntimeError('bison_BROWSER error')
if plugin_config.bison_use_local:
return await launch(executablePath='/usr/bin/chromium', args=['--no-sandbox'])
return await launch(args=['--no-sandbox'])
@ -60,8 +69,7 @@ class Render(metaclass=Singleton):
# self.lock.release()
def _inter_log(self, message: str) -> None:
# self.interval_log += asctime() + '' + message + '\n'
logger.debug(message)
self.interval_log += asctime() + '' + message + '\n'
async def do_render(self, url: str, viewport: Optional[dict] = None, target: Optional[str] = None,
operation: Optional[Callable[[Page], Awaitable[None]]] = None) -> Optional[bytes]:
@ -111,8 +119,20 @@ class Render(metaclass=Singleton):
async def parse_text(text: str) -> MessageSegment:
'return raw text if don\'t use pic, otherwise return rendered opcode'
if plugin_config.hk_reporter_use_pic:
if plugin_config.bison_use_pic:
render = Render()
return await render.text_to_pic_cqcode(text)
else:
return MessageSegment.text(text)
def html_to_text(html: str, query_dict: dict = {}) -> str:
html = re.sub(r'<br\s*/?>', '<br>\n', html)
html = html.replace('</p>', '</p>\n')
soup = bs(html, 'html.parser')
if query_dict:
node = soup.find(**query_dict)
else:
node = soup
assert node is not None
return node.text.strip()

View File

@ -1,38 +0,0 @@
from typing import Any
import httpx
from .platform import NewMessage, NoTargetMixin
from ..types import RawPost
from ..post import Post
class MonsterSiren(NewMessage, NoTargetMixin):
categories = {}
platform_name = 'monster-siren'
enable_tag = False
enabled = True
is_common = False
schedule_type = 'interval'
schedule_kw = {'seconds': 30}
name = '塞壬唱片官网新闻'
@staticmethod
async def get_target_name(_) -> str:
return '塞壬唱片新闻'
async def get_sub_list(self, _) -> list[RawPost]:
async with httpx.AsyncClient() as client:
raw_data = await client.get('https://monster-siren.hypergryph.com/api/news')
return raw_data.json()['data']['list']
def get_id(self, post: RawPost) -> Any:
return post['cid']
def get_date(self, _) -> None:
return None
async def parse(self, raw_post: RawPost) -> Post:
url = f'https://monster-siren.hypergryph.com/info/{raw_post["cid"]}'
return Post('monster-siren', text=raw_post['title'],
url=url, target_name="塞壬唱片新闻", compress=True,
override_use_pic=False)

View File

@ -1,21 +0,0 @@
from pydantic import BaseSettings
import warnings
import nonebot
class PlugConfig(BaseSettings):
hk_reporter_config_path: str = ""
hk_reporter_use_pic: bool = False
hk_reporter_use_local: bool = False
hk_reporter_browser: str = ''
hk_reporter_init_filter: bool = True
hk_reporter_outer_url: str = 'http://localhost:8080/hk_reporter/'
class Config:
extra = 'ignore'
global_config = nonebot.get_driver().config
plugin_config = PlugConfig(**global_config.dict())
if plugin_config.hk_reporter_use_local:
warnings.warn('HK_REPORTER_USE_LOCAL is deprecated, please use HK_REPORTER_BROWSER')

View File

@ -1,30 +0,0 @@
from nonebot import logger
import time
QUEUE = []
LAST_SEND_TIME = time.time()
async def do_send_msgs():
global LAST_SEND_TIME
if time.time() - LAST_SEND_TIME < 1.5:
return
if QUEUE:
bot, user, user_type, msg, retry_time = QUEUE.pop(0)
try:
if user_type == 'group':
await bot.call_api('send_group_msg', group_id=user, message=msg)
elif user_type == 'private':
await bot.call_api('send_private_msg', user_id=user, message=msg)
except:
if retry_time > 0:
QUEUE.insert(0, (bot, user, user_type, msg, retry_time - 1))
else:
logger.warning('send msg err {}'.format(msg))
LAST_SEND_TIME = time.time()
def send_msgs(bot, user, user_type, msgs):
for msg in msgs:
QUEUE.append((bot, user, user_type, msg, 2))

View File

@ -5,18 +5,18 @@ import typing
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
@pytest.fixture#(scope="module")
def plugin_module(tmpdir):
nonebot.init(hk_reporter_config_path=str(tmpdir))
nonebot.init(bison_config_path=str(tmpdir))
nonebot.load_plugins('src/plugins')
plugins = nonebot.get_loaded_plugins()
plugin = list(filter(lambda x: x.name == 'nonebot_hk_reporter', plugins))[0]
plugin = list(filter(lambda x: x.name == 'nonebot_bison', plugins))[0]
return plugin.module
@pytest.fixture
def dummy_user_subinfo(plugin_module: 'nonebot_hk_reporter'):
def dummy_user_subinfo(plugin_module: 'nonebot_bison'):
user = plugin_module.types.User('123', 'group')
return plugin_module.types.UserSubInfo(
user=user,

View File

@ -7,12 +7,12 @@ import feedparser
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
from .utils import get_json, get_file
@pytest.fixture
def arknights(plugin_module: 'nonebot_hk_reporter'):
def arknights(plugin_module: 'nonebot_bison'):
return plugin_module.platform.platform_manager['arknights']
@pytest.fixture(scope='module')
@ -23,17 +23,27 @@ def arknights_list_0():
def arknights_list_1():
return get_json('arknights_list_1.json')
@pytest.fixture(scope='module')
def monster_siren_list_0():
return get_json('monster-siren_list_0.json')
@pytest.fixture(scope='module')
def monster_siren_list_1():
return get_json('monster-siren_list_1.json')
@pytest.mark.asyncio
@respx.mock
async def test_fetch_new(arknights, dummy_user_subinfo, arknights_list_0, arknights_list_1):
async def test_fetch_new(arknights, dummy_user_subinfo, arknights_list_0, arknights_list_1, monster_siren_list_0, monster_siren_list_1):
ak_list_router = respx.get("https://ak-conf.hypergryph.com/config/prod/announce_meta/IOS/announcement.meta.json")
detail_router = respx.get("https://ak-fs.hypergryph.com/announce/IOS/announcement/675.html")
version_router = respx.get('https://ak-conf.hypergryph.com/config/prod/official/IOS/version')
preannouncement_router = respx.get('https://ak-conf.hypergryph.com/config/prod/announce_meta/IOS/preannouncement.meta.json')
monster_siren_router = respx.get("https://monster-siren.hypergryph.com/api/news")
ak_list_router.mock(return_value=Response(200, json=arknights_list_0))
detail_router.mock(return_value=Response(200, text=get_file('arknights-detail-675.html')))
version_router.mock(return_value=Response(200, json=get_json('arknights-version-0.json')))
preannouncement_router.mock(return_value=Response(200, json=get_json('arknights-pre-0.json')))
monster_siren_router.mock(return_value=Response(200, json=monster_siren_list_0))
target = ''
res = await arknights.fetch_new_post(target, [dummy_user_subinfo])
assert(ak_list_router.called)
@ -51,4 +61,5 @@ async def test_fetch_new(arknights, dummy_user_subinfo, arknights_list_0, arknig
assert(post.target_name == '明日方舟游戏内公告')
assert(len(post.pics) == 1)
assert(post.pics == ['https://ak-fs.hypergryph.com/announce/images/20210623/e6f49aeb9547a2278678368a43b95b07.jpg'])
print(res3[0][1])
r = await post.generate_messages()

View File

@ -5,7 +5,7 @@ from httpx import Response
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
from .utils import get_json
@ -14,7 +14,7 @@ def bing_dy_list():
return get_json('bilibili_bing_list.json')['data']['cards']
@pytest.fixture
def bilibili(plugin_module: 'nonebot_hk_reporter'):
def bilibili(plugin_module: 'nonebot_bison'):
return plugin_module.platform.platform_manager['bilibili']
@pytest.mark.asyncio

View File

@ -1,44 +0,0 @@
import pytest
import typing
import respx
from httpx import Response
import feedparser
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
from .utils import get_json, get_file
@pytest.fixture
def monster_siren(plugin_module: 'nonebot_hk_reporter'):
return plugin_module.platform.platform_manager['monster-siren']
@pytest.fixture(scope='module')
def monster_siren_list_0():
return get_json('monster-siren_list_0.json')
@pytest.fixture(scope='module')
def monster_siren_list_1():
return get_json('monster-siren_list_1.json')
@pytest.mark.asyncio
@respx.mock
async def test_fetch_new(monster_siren, dummy_user_subinfo, monster_siren_list_0, monster_siren_list_1):
ak_list_router = respx.get("https://monster-siren.hypergryph.com/api/news")
ak_list_router.mock(return_value=Response(200, json=monster_siren_list_0))
target = ''
res = await monster_siren.fetch_new_post(target, [dummy_user_subinfo])
assert(ak_list_router.called)
assert(len(res) == 0)
mock_data = monster_siren_list_1
ak_list_router.mock(return_value=Response(200, json=mock_data))
res3 = await monster_siren.fetch_new_post(target, [dummy_user_subinfo])
assert(len(res3[0][1]) == 1)
post = res3[0][1][0]
assert(post.target_type == 'monster-siren')
assert(post.text == '#D.D.D.PHOTO')
assert(post.url == 'https://monster-siren.hypergryph.com/info/241303')
assert(post.target_name == '塞壬唱片新闻')
assert(len(post.pics) == 0)

View File

@ -8,10 +8,10 @@ from httpx import Response
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
@pytest.fixture
def ncm_artist(plugin_module: 'nonebot_hk_reporter'):
def ncm_artist(plugin_module: 'nonebot_bison'):
return plugin_module.platform.platform_manager['ncm-artist']
@pytest.fixture(scope='module')

View File

@ -7,9 +7,9 @@ import pytest
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
from nonebot_hk_reporter.types import *
from nonebot_hk_reporter.post import Post
import nonebot_bison
from nonebot_bison.types import *
from nonebot_bison.post import Post
from time import time
now = time()
@ -26,18 +26,18 @@ raw_post_list_2 = raw_post_list_1 + [
]
@pytest.fixture
def dummy_user(plugin_module: 'nonebot_hk_reporter'):
def dummy_user(plugin_module: 'nonebot_bison'):
user = plugin_module.types.User('123', 'group')
return user
@pytest.fixture
def user_info_factory(plugin_module: 'nonebot_hk_reporter', dummy_user):
def user_info_factory(plugin_module: 'nonebot_bison', dummy_user):
def _user_info(category_getter, tag_getter):
return plugin_module.types.UserSubInfo(dummy_user, category_getter, tag_getter)
return _user_info
@pytest.fixture
def mock_platform_without_cats_tags(plugin_module: 'nonebot_hk_reporter'):
def mock_platform_without_cats_tags(plugin_module: 'nonebot_bison'):
class MockPlatform(plugin_module.platform.platform.NewMessage,
plugin_module.platform.platform.TargetMixin):
@ -76,7 +76,7 @@ def mock_platform_without_cats_tags(plugin_module: 'nonebot_hk_reporter'):
return MockPlatform()
@pytest.fixture
def mock_platform(plugin_module: 'nonebot_hk_reporter'):
def mock_platform(plugin_module: 'nonebot_bison'):
class MockPlatform(plugin_module.platform.platform.NewMessage,
plugin_module.platform.platform.TargetMixin):
@ -123,7 +123,7 @@ def mock_platform(plugin_module: 'nonebot_hk_reporter'):
return MockPlatform()
@pytest.fixture
def mock_platform_no_target(plugin_module: 'nonebot_hk_reporter'):
def mock_platform_no_target(plugin_module: 'nonebot_bison'):
class MockPlatform(plugin_module.platform.platform.NewMessage,
plugin_module.platform.platform.NoTargetMixin):
@ -174,7 +174,7 @@ def mock_platform_no_target(plugin_module: 'nonebot_hk_reporter'):
return MockPlatform()
@pytest.fixture
def mock_platform_no_target_2(plugin_module: 'nonebot_hk_reporter'):
def mock_platform_no_target_2(plugin_module: 'nonebot_bison'):
class MockPlatform(plugin_module.platform.platform.NewMessage,
plugin_module.platform.platform.NoTargetMixin):
@ -230,7 +230,7 @@ def mock_platform_no_target_2(plugin_module: 'nonebot_hk_reporter'):
return MockPlatform()
@pytest.fixture
def mock_status_change(plugin_module: 'nonebot_hk_reporter'):
def mock_status_change(plugin_module: 'nonebot_bison'):
class MockPlatform(plugin_module.platform.platform.StatusChange,
plugin_module.platform.platform.NoTargetMixin):
@ -359,7 +359,7 @@ async def test_status_change(mock_status_change, user_info_factory):
assert(len(res4) == 0)
@pytest.mark.asyncio
async def test_group(plugin_module: 'nonebot_hk_reporter', mock_platform_no_target, mock_platform_no_target_2, user_info_factory):
async def test_group(plugin_module: 'nonebot_bison', mock_platform_no_target, mock_platform_no_target_2, user_info_factory):
group_platform = plugin_module.platform.platform.NoTargetGroup([mock_platform_no_target, mock_platform_no_target_2])
res1 = await group_platform.fetch_new_post('dummy', [user_info_factory(lambda _: [1,4], lambda _: [])])
assert(len(res1) == 0)

View File

@ -9,12 +9,12 @@ import feedparser
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
from .utils import get_json, get_file
@pytest.fixture
def weibo(plugin_module: 'nonebot_hk_reporter'):
def weibo(plugin_module: 'nonebot_bison'):
return plugin_module.platform.platform_manager['weibo']
@pytest.fixture(scope='module')

View File

@ -4,14 +4,14 @@ import typing
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
@pytest.fixture
def config(plugin_module):
plugin_module.config.start_up()
return plugin_module.config.Config()
def test_create_and_get(config: 'nonebot_hk_reporter.config.Config', plugin_module: 'nonebot_hk_reporter'):
def test_create_and_get(config: 'nonebot_bison.config.Config', plugin_module: 'nonebot_bison'):
config.add_subscribe(
user='123',
user_type='group',

View File

@ -4,7 +4,7 @@ import typing
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
merge_source_9 = [
'https://wx1.sinaimg.cn/large/0071VPLMgy1gq0vib7zooj30dx0dxmz5.jpg',
@ -23,7 +23,7 @@ merge_source_9 = [
]
@pytest.mark.asyncio
async def test_9_merge(plugin_module: 'nonebot_hk_reporter'):
async def test_9_merge(plugin_module: 'nonebot_bison'):
post = plugin_module.post.Post('', '', '', pics=merge_source_9)
await post._pic_merge()
assert len(post.pics) == 5

View File

@ -4,11 +4,11 @@ import typing
if typing.TYPE_CHECKING:
import sys
sys.path.append('./src/plugins')
import nonebot_hk_reporter
import nonebot_bison
@pytest.mark.asyncio
@pytest.mark.render
async def test_render(plugin_module: 'nonebot_hk_reporter'):
async def test_render(plugin_module: 'nonebot_bison'):
render = plugin_module.utils.Render()
res = await render.text_to_pic('''a\nbbbbbbbbbbbbbbbbbbbbbb\ncd
<h1>中文</h1>

8008
yarn.lock

File diff suppressed because it is too large Load Diff