mirror of
https://github.com/suyiiyii/nonebot-bison.git
synced 2026-05-10 18:57:56 +08:00
Compare commits
15 Commits
cookie-init
...
cookie
| Author | SHA1 | Date | |
|---|---|---|---|
|
40f4490b74
|
|||
| 3bdc79162e | |||
| 8f86f61802 | |||
| ab5154d523 | |||
| 77a4dcd70e | |||
|
a1843bb98a
|
|||
|
f08ab9f926
|
|||
|
fd349eefed
|
|||
|
e8f0d578e1
|
|||
|
60dd2c4bab
|
|||
|
d6b8d3b44e
|
|||
|
3f7a9bf8a3
|
|||
|
955a06d9e9
|
|||
| d4f45571b3 | |||
| a671bd0c61 |
+6
-1
@@ -1,11 +1,16 @@
|
|||||||
# Change Log
|
# Change Log
|
||||||
|
|
||||||
## 最近更新
|
## v0.9.5
|
||||||
|
|
||||||
### 新功能
|
### 新功能
|
||||||
|
|
||||||
- :sparkles: 更新默认UA为Windows平台 [@suyiiyii](https://github.com/suyiiyii) ([#643](https://github.com/MountainDash/nonebot-bison/pull/643))
|
- :sparkles: 更新默认UA为Windows平台 [@suyiiyii](https://github.com/suyiiyii) ([#643](https://github.com/MountainDash/nonebot-bison/pull/643))
|
||||||
|
|
||||||
|
### Bug 修复
|
||||||
|
|
||||||
|
- 🐛 修复微博更换长内容接口 [@phidiaLam](https://github.com/phidiaLam) ([#645](https://github.com/MountainDash/nonebot-bison/pull/645))
|
||||||
|
- :bug: 修复B站获取匿名Cookie逻辑 [@suyiiyii](https://github.com/suyiiyii) ([#644](https://github.com/MountainDash/nonebot-bison/pull/644))
|
||||||
|
|
||||||
### 文档
|
### 文档
|
||||||
|
|
||||||
- 📝 小刻食堂剪彩文档 [@phidiaLam](https://github.com/phidiaLam) ([#636](https://github.com/MountainDash/nonebot-bison/pull/636))
|
- 📝 小刻食堂剪彩文档 [@phidiaLam](https://github.com/phidiaLam) ([#636](https://github.com/MountainDash/nonebot-bison/pull/636))
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 74 KiB After Width: | Height: | Size: 70 KiB |
+22
-33
@@ -5,7 +5,7 @@ prev: /usage/
|
|||||||
|
|
||||||
# Cookie 开发须知
|
# Cookie 开发须知
|
||||||
|
|
||||||
本项目将大部分 Cookie 相关逻辑提出到了 Site 及 ClientManger 模块中,你只需要继承相关类即可获得使用 Cookie 的能力。
|
本项目将大部分 Cookie 相关逻辑提出到了 Site 及 ClientManger 模块中,只需要继承相关类即可获得使用 Cookie 的能力。
|
||||||
|
|
||||||
::: tip
|
::: tip
|
||||||
|
|
||||||
@@ -18,11 +18,10 @@ prev: /usage/
|
|||||||
- `nonebot_bison.config.db_model.Cookie`: 用于存储 Cookie 的实体类,包含了 Cookie 的名称、内容、状态等信息
|
- `nonebot_bison.config.db_model.Cookie`: 用于存储 Cookie 的实体类,包含了 Cookie 的名称、内容、状态等信息
|
||||||
- `nonebot_bison.config.db_model.CookieTarget`: 用于存储 Cookie 与订阅的关联关系
|
- `nonebot_bison.config.db_model.CookieTarget`: 用于存储 Cookie 与订阅的关联关系
|
||||||
- `nonebot_bison.utils.site.CookieClientManager`: 添加了 Cookie 功能的 ClientManager,是 Cookie 管理功能的核心,调度 Cookie 的功能就在这里实现
|
- `nonebot_bison.utils.site.CookieClientManager`: 添加了 Cookie 功能的 ClientManager,是 Cookie 管理功能的核心,调度 Cookie 的功能就在这里实现
|
||||||
- `nonebot_bison.utils.site.CookieSite`: 添加了 Cookie 功能的 Site 类,根据需求添加了和 Site 强相关的 Cookie 功能实现方法
|
|
||||||
|
|
||||||
## 快速上手
|
## 快速上手
|
||||||
|
|
||||||
例如,你现在有一个这样子的 Site 类:
|
例如,现在有一个这样子的 Site 类:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
class WeiboSite(Site):
|
class WeiboSite(Site):
|
||||||
@@ -31,39 +30,30 @@ class WeiboSite(Site):
|
|||||||
schedule_setting = {"seconds": 3}
|
schedule_setting = {"seconds": 3}
|
||||||
```
|
```
|
||||||
|
|
||||||
简而言之,要让你的站点获得 Cookie 能力,只需要:
|
简而言之,要让站点获得 Cookie 能力,只需要:
|
||||||
|
|
||||||
1. 将父类从`Site`改为`CookieSite`
|
为 Site 类添加一个`client_mgr`字段,值为`CookieClientManager.from_name(name)`,其中`name`为站点名称,这是默认的 Cookie 管理器。
|
||||||
|
|
||||||
```python {1}
|
|
||||||
class WeiboSite(CookieSite):
|
|
||||||
name = "weibo.com"
|
|
||||||
schedule_type = "interval"
|
|
||||||
schedule_setting = {"seconds": 3}
|
|
||||||
```
|
|
||||||
|
|
||||||
2. 为你的 Site 类添加一个`client_mgr`属性,值为`create_cookie_client_manager(name)`,其中`name`为你的站点名称,这是默认的 Cookie 管理器。
|
|
||||||
|
|
||||||
```python {5}
|
```python {5}
|
||||||
class WeiboSite(CookieSite):
|
class WeiboSite(Site):
|
||||||
name = "weibo.com"
|
name = "weibo.com"
|
||||||
schedule_type = "interval"
|
schedule_type = "interval"
|
||||||
schedule_setting = {"seconds": 3}
|
schedule_setting = {"seconds": 3}
|
||||||
client_mgr = create_cookie_client_manager(name)
|
client_mgr = CookieClientManager.from_name(name)
|
||||||
```
|
```
|
||||||
|
|
||||||
至此,你的站点就可以使用 Cookie 了!
|
至此,站点就可以使用 Cookie 了!
|
||||||
|
|
||||||
## 更好的体验
|
## 更好的体验
|
||||||
|
|
||||||
为了给用户提供更好的体验,你还可以为你的 Site 重写`validate_cookie`和`get_target_name`方法。
|
为了给用户提供更好的体验,还可以创建自己的 `ClientManager`:继承 `CookieClientManager` 并重写`validate_cookie`和`get_target_name`方法。
|
||||||
|
|
||||||
- `async def validate_cookie(cls, content: str) -> bool`该方法将会在 Cookie 添加时被调用,你可以在这里验证 Cookie 的有效性
|
- `async def validate_cookie(cls, content: str) -> bool`该方法将会在 Cookie 添加时被调用,可以在这里验证 Cookie 的有效性
|
||||||
- `async def get_cookie_name(cls, content: str) -> str`该方法将会在验证 Cookie 成功后被调用,你可以在这里设置 Cookie 的名字并展示给用户
|
- `async def get_cookie_name(cls, content: str) -> str`该方法将会在验证 Cookie 成功后被调用,可以在这里设置 Cookie 的名字并展示给用户
|
||||||
|
|
||||||
## 我要自己调度 Cookie
|
## 自定义 Cookie 调度策略
|
||||||
|
|
||||||
当默认的 Cookie 调度逻辑无法满足你的需求时,你可以重写`CookieClientManager`的`_choose_cookie`方法。
|
当默认的 Cookie 调度逻辑无法满足需求时,可以重写`CookieClientManager`的`_choose_cookie`方法。
|
||||||
|
|
||||||
目前整体的调度逻辑是:
|
目前整体的调度逻辑是:
|
||||||
|
|
||||||
@@ -94,14 +84,12 @@ sequenceDiagram
|
|||||||
|
|
||||||
- `refresh_anonymous_cookie(cls)` 移除已有的匿名 cookie,添加一个新的匿名 cookie,应该在 CCM 初始化时调用
|
- `refresh_anonymous_cookie(cls)` 移除已有的匿名 cookie,添加一个新的匿名 cookie,应该在 CCM 初始化时调用
|
||||||
- `add_user_cookie(cls, content: str)` 添加用户 cookie,在这里对 Cookie 进行检查并获取 cookie_name,写入数据库
|
- `add_user_cookie(cls, content: str)` 添加用户 cookie,在这里对 Cookie 进行检查并获取 cookie_name,写入数据库
|
||||||
- `_generate_hook(self, cookie: Cookie) -> callable` hook 函数生成器,用于回写请求状态到数据库
|
- `_generate_hook(self, cookie: Cookie) -> Callable` hook 函数生成器,用于回写请求状态到数据库
|
||||||
- `_choose_cookie(self, target: Target) -> Cookie` 选择 cookie 的具体算法
|
- `_choose_cookie(self, target: Target) -> Cookie` 选择 cookie 的具体算法
|
||||||
- `add_user_cookie(cls, content: str, cookie_name: str | None = None) -> Cookie` 对外的接口,添加用户 cookie,内部会调用 Site 的方法进行检查
|
- `add_user_cookie(cls, content: str, cookie_name: str | None = None) -> Cookie` 对外的接口,添加用户 cookie,内部会调用 Site 的方法进行检查
|
||||||
- `get_client(self, target: Target | None) -> AsyncClient` 对外的接口,获取 client,根据 target 选择 cookie
|
- `get_client(self, target: Target | None) -> AsyncClient` 对外的接口,获取 client,根据 target 选择 cookie
|
||||||
- `_assemble_client(self, client, cookie) -> AsyncClient` 组装 client,可以自定义 cookie 对象的 content 装配到 client 中的方式
|
- `_assemble_client(self, client, cookie) -> AsyncClient` 组装 client,可以自定义 cookie 对象的 content 装配到 client 中的方式
|
||||||
|
|
||||||
CookieSite 的方法见上文
|
|
||||||
|
|
||||||
::: details 大致流程
|
::: details 大致流程
|
||||||
|
|
||||||
1. `Platfrom` 调用 `CookieClientManager.get_client` 方法,传入 `Target` 对象
|
1. `Platfrom` 调用 `CookieClientManager.get_client` 方法,传入 `Target` 对象
|
||||||
@@ -112,18 +100,19 @@ CookieSite 的方法见上文
|
|||||||
|
|
||||||
简单来说:
|
简单来说:
|
||||||
|
|
||||||
- 如果你需要修改 Cookie 的默认参数,可以重写`add_user_cookie`方法,这里设置你需要的属性
|
- 如果需要修改 Cookie 的默认参数,可以重写`add_user_cookie`方法,这里设置需要的字段
|
||||||
- 如果你需要修改选择 Cookie 的逻辑,可以重写`_choose_cookie`方法,使用你自己的算法选择合适的 Cookie 并返回
|
- 如果需要修改选择 Cookie 的逻辑,可以重写`_choose_cookie`方法,使用自己的算法选择合适的 Cookie 并返回
|
||||||
- 如果你需要自定义 Cookie 的格式,可以重写`valid_cookie`方法,自定义验证 Cookie 的逻辑,并重写`_assemble_client`方法,自定义将 Cookie 装配到 Client 中的逻辑
|
- 如果需要自定义 Cookie 的格式,可以重写`valid_cookie`方法,自定义验证 Cookie 的逻辑,并重写`_assemble_client`方法,自定义将 Cookie 装配到 Client 中的逻辑
|
||||||
- 如果你要在请求结束后做一些操作(例如保存此次请求的结果/状态),可以重写`_response_hook`方法,自定义请求结束后的行为
|
- 如果要在请求结束后做一些操作(例如保存此次请求的结果/状态),可以重写`_response_hook`方法,自定义请求结束后的行为
|
||||||
|
- 如果需要跳过一次请求,可以在 `get_client` 方法中抛出 `SkipRequestException` 异常,调度器会捕获该异常并跳过此次请求
|
||||||
|
|
||||||
## 实名 Cookie 和匿名 Cookie
|
## 实名 Cookie 和匿名 Cookie
|
||||||
|
|
||||||
部分站点所有接口都需要携带 Cookie,对于匿名用户(未登录)也会发放一个临时 Cookie,我们称为匿名 Cookie。
|
部分站点所有接口都需要携带 Cookie,对于匿名用户(未登录)也会发放一个临时 Cookie,本项目称为匿名 Cookie。
|
||||||
|
|
||||||
在此基础上,我们添加了用户上传 Cookie 的功能,这种 Cookie 我们称为实名 Cookie。
|
在此基础上,我们添加了用户上传 Cookie 的功能,这种 Cookie 本项目称为实名 Cookie。
|
||||||
|
|
||||||
匿名 Cookie 和实名 Cookie 在同一个框架下统一调度,实名 Cookie 优先级高于匿名 Cookie。为了调度,Cookie 对象有以下属性:
|
匿名 Cookie 和实名 Cookie 在同一个框架下统一调度,实名 Cookie 优先级高于匿名 Cookie。为了调度,Cookie 对象有以下字段:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# 最后使用的时刻
|
# 最后使用的时刻
|
||||||
@@ -148,7 +137,7 @@ CookieSite 的方法见上文
|
|||||||
|
|
||||||
- **无 Target 平台的 Cookie 处理方式**
|
- **无 Target 平台的 Cookie 处理方式**
|
||||||
|
|
||||||
对于不存在 Target 的平台,如小刻食堂,可以重写 add_user_cookie 方法,为用户 Cookie 设置 is_universal 属性。这样,在获取 Client 时,由于传入的 Target 为空,就只会选择 is_universal 的 cookie。实现了无 Target 平台的用户 Cookie 调度。
|
对于不存在 Target 的平台,如小刻食堂,可以重写 add_user_cookie 方法,为用户 Cookie 设置 is_universal 字段。这样,在获取 Client 时,由于传入的 Target 为空,就只会选择 is_universal 的 cookie。实现了无 Target 平台的用户 Cookie 调度。
|
||||||
|
|
||||||
## 默认的调度策略
|
## 默认的调度策略
|
||||||
|
|
||||||
|
|||||||
@@ -40,15 +40,15 @@ Cookie 全局生效,这意味着,通过你的 Cookie 获取到的内容,
|
|||||||
|
|
||||||
## :nerd_face: 如何获取 Cookie?
|
## :nerd_face: 如何获取 Cookie?
|
||||||
|
|
||||||
对于大部分平台,Bison支持JSON格式的Cookie,你可以通过浏览器的开发者工具获取。
|
对于大部分平台,Bison 支持 JSON 格式的 Cookie,你可以通过浏览器的开发者工具获取。
|
||||||
|
|
||||||
- RSS: 对于各种RSS订阅,你需要自行准备需要的Cookie,以JSON格式添加即可
|
- RSS: 对于各种 RSS 订阅,你需要自行准备需要的 Cookie,以 JSON 格式添加即可
|
||||||
- 微博: Bison兼容RSSHub的Cookie,以下方法引用自[RSSHub的文档](https://docs.rsshub.app/zh/deploy/config#%E5%BE%AE%E5%8D%9A)
|
- 微博:Bison 兼容 RSSHub 的 Cookie,以下方法引用自[RSSHub 的文档](https://docs.rsshub.app/zh/deploy/config#%E5%BE%AE%E5%8D%9A)
|
||||||
> 1. 打开并登录 https://m.weibo.cn (确保打开页面为手机版,如果强制跳转电脑端可尝试使用可更改 UserAgent 的浏览器插件)
|
> 1. 打开并登录 https://m.weibo.cn(确保打开页面为手机版,如果强制跳转电脑端可尝试使用可更改 UserAgent 的浏览器插件)
|
||||||
> 2. 按下F12打开控制台,切换至Network(网络)面板
|
> 2. 按下 F12 打开控制台,切换至 Network(网络)面板
|
||||||
> 3. 在该网页切换至任意关注分组,并在面板打开最先捕获到的请求 (该情形下捕获到的请求路径应包含/feed/group)
|
> 3. 在该网页切换至任意关注分组,并在面板打开最先捕获到的请求(该情形下捕获到的请求路径应包含/feed/group)
|
||||||
> 4. 查看该请求的Headers(请求头), 找到Cookie字段并复制内容
|
> 4. 查看该请求的 Headers(请求头), 找到 Cookie 字段并复制内容
|
||||||
- Bilibili: Bison兼容RSSHub的Cookie,以下方法引用自[RSSHub的文档](https://docs.rsshub.app/zh/deploy/config#bilibili)
|
- Bilibili: Bison 兼容 RSSHub 的 Cookie,以下方法引用自[RSSHub 的文档](https://docs.rsshub.app/zh/deploy/config#bilibili)
|
||||||
> 1. 打开 https://api.vc.bilibili.com/dynamic_svr/v1/dynamic_svr/dynamic_new?uid=0&type=8
|
> 1. 打开 https://api.vc.bilibili.com/dynamic_svr/v1/dynamic_svr/dynamic_new?uid=0&type=8
|
||||||
> 2. 打开控制台,切换到 Network 面板,刷新
|
> 2. 打开控制台,切换到 Network 面板,刷新
|
||||||
> 3. 点击 dynamic_new 请求,找到 Cookie
|
> 3. 点击 dynamic_new 请求,找到 Cookie
|
||||||
|
|||||||
@@ -15,10 +15,10 @@ from .jwt import load_jwt, pack_jwt
|
|||||||
from ..scheduler import scheduler_dict
|
from ..scheduler import scheduler_dict
|
||||||
from ..types import Target as T_Target
|
from ..types import Target as T_Target
|
||||||
from ..utils.get_bot import get_groups
|
from ..utils.get_bot import get_groups
|
||||||
|
from ..platform import platform_manager
|
||||||
from .token_manager import token_manager
|
from .token_manager import token_manager
|
||||||
from ..config.db_config import SubscribeDupException
|
from ..config.db_config import SubscribeDupException
|
||||||
from ..platform import site_manager, platform_manager
|
from ..utils.site import CookieClientManager, site_manager, is_cookie_client_manager
|
||||||
from ..utils.site import CookieClientManager, is_cookie_client_manager
|
|
||||||
from ..config import NoSuchUserException, NoSuchTargetException, NoSuchSubscribeException, config
|
from ..config import NoSuchUserException, NoSuchTargetException, NoSuchSubscribeException, config
|
||||||
from .types import (
|
from .types import (
|
||||||
Cookie,
|
Cookie,
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ from pkgutil import iter_modules
|
|||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
from importlib import import_module
|
from importlib import import_module
|
||||||
|
|
||||||
from ..utils import Site
|
|
||||||
from ..plugin_config import plugin_config
|
from ..plugin_config import plugin_config
|
||||||
from .platform import Platform, make_no_target_group
|
from .platform import Platform, make_no_target_group
|
||||||
|
|
||||||
@@ -36,10 +35,3 @@ def _get_unavailable_platforms() -> dict[str, str]:
|
|||||||
|
|
||||||
# platform => reason for not available
|
# platform => reason for not available
|
||||||
unavailable_paltforms: dict[str, str] = _get_unavailable_platforms()
|
unavailable_paltforms: dict[str, str] = _get_unavailable_platforms()
|
||||||
|
|
||||||
|
|
||||||
site_manager: dict[str, type[Site]] = {}
|
|
||||||
for site in Site.registry:
|
|
||||||
if not hasattr(site, "name"):
|
|
||||||
continue
|
|
||||||
site_manager[site.name] = site
|
|
||||||
|
|||||||
@@ -24,9 +24,8 @@ B = TypeVar("B", bound="Bilibili")
|
|||||||
|
|
||||||
class BilibiliClientManager(CookieClientManager):
|
class BilibiliClientManager(CookieClientManager):
|
||||||
|
|
||||||
_default_cookie_cd: int = timedelta(seconds=120)
|
_default_cookie_cd = timedelta(seconds=120)
|
||||||
|
|
||||||
@classmethod
|
|
||||||
async def _get_cookies(self) -> list[Cookie]:
|
async def _get_cookies(self) -> list[Cookie]:
|
||||||
browser = await get_browser()
|
browser = await get_browser()
|
||||||
async with await browser.new_page() as page:
|
async with await browser.new_page() as page:
|
||||||
@@ -38,7 +37,6 @@ class BilibiliClientManager(CookieClientManager):
|
|||||||
|
|
||||||
return cookies
|
return cookies
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def _gen_json_cookie(self, cookies: list[Cookie]):
|
def _gen_json_cookie(self, cookies: list[Cookie]):
|
||||||
cookie_dict = {}
|
cookie_dict = {}
|
||||||
for cookie in cookies:
|
for cookie in cookies:
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ from nonebot_plugin_saa import PlatformTarget
|
|||||||
from ..post import Post
|
from ..post import Post
|
||||||
from ..utils import Site, ProcessContext
|
from ..utils import Site, ProcessContext
|
||||||
from ..plugin_config import plugin_config
|
from ..plugin_config import plugin_config
|
||||||
from ..types import Tag, Target, RawPost, SubUnit, Category, RegistryMeta
|
from ..types import Tag, Target, RawPost, SubUnit, Category
|
||||||
|
|
||||||
|
|
||||||
class CategoryNotSupport(Exception):
|
class CategoryNotSupport(Exception):
|
||||||
@@ -29,6 +29,21 @@ class CategoryNotRecognize(Exception):
|
|||||||
"""raise in get_category, when you don't know the category of post"""
|
"""raise in get_category, when you don't know the category of post"""
|
||||||
|
|
||||||
|
|
||||||
|
class RegistryMeta(type):
|
||||||
|
def __new__(cls, name, bases, namespace, **kwargs):
|
||||||
|
return super().__new__(cls, name, bases, namespace)
|
||||||
|
|
||||||
|
def __init__(cls, name, bases, namespace, **kwargs):
|
||||||
|
if kwargs.get("base"):
|
||||||
|
# this is the base class
|
||||||
|
cls.registry = []
|
||||||
|
elif not kwargs.get("abstract"):
|
||||||
|
# this is the subclass
|
||||||
|
cls.registry.append(cls)
|
||||||
|
|
||||||
|
super().__init__(name, bases, namespace, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
P = ParamSpec("P")
|
P = ParamSpec("P")
|
||||||
R = TypeVar("R")
|
R = TypeVar("R")
|
||||||
|
|
||||||
|
|||||||
@@ -10,14 +10,14 @@ from ..post import Post
|
|||||||
from .platform import NewMessage
|
from .platform import NewMessage
|
||||||
from ..types import Target, RawPost
|
from ..types import Target, RawPost
|
||||||
from ..utils import text_similarity
|
from ..utils import text_similarity
|
||||||
from ..utils.site import Site, create_cookie_client_manager
|
from ..utils.site import Site, CookieClientManager
|
||||||
|
|
||||||
|
|
||||||
class RssSite(Site):
|
class RssSite(Site):
|
||||||
name = "rss"
|
name = "rss"
|
||||||
schedule_type = "interval"
|
schedule_type = "interval"
|
||||||
schedule_setting = {"seconds": 30}
|
schedule_setting = {"seconds": 30}
|
||||||
client_mgr = create_cookie_client_manager(name)
|
client_mgr = CookieClientManager.from_name(name)
|
||||||
|
|
||||||
|
|
||||||
class RssPost(Post):
|
class RssPost(Post):
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
import re
|
import re
|
||||||
import json
|
import json
|
||||||
|
from typing import Any
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from typing import Any, override
|
|
||||||
from urllib.parse import unquote
|
from urllib.parse import unquote
|
||||||
|
from typing_extensions import override
|
||||||
|
|
||||||
from yarl import URL
|
from yarl import URL
|
||||||
from lxml.etree import HTML
|
from lxml.etree import HTML
|
||||||
@@ -175,7 +176,7 @@ class Weibo(NewMessage):
|
|||||||
try:
|
try:
|
||||||
client = await self.ctx.get_client()
|
client = await self.ctx.get_client()
|
||||||
weibo_info = await client.get(
|
weibo_info = await client.get(
|
||||||
"https://m.weibo.cn/statuses/show",
|
"https://m.weibo.cn/statuses/extend",
|
||||||
params={"id": weibo_id},
|
params={"id": weibo_id},
|
||||||
headers=_HEADER,
|
headers=_HEADER,
|
||||||
)
|
)
|
||||||
@@ -189,7 +190,7 @@ class Weibo(NewMessage):
|
|||||||
|
|
||||||
async def _parse_weibo(self, info: dict) -> Post:
|
async def _parse_weibo(self, info: dict) -> Post:
|
||||||
if info["isLongText"] or info["pic_num"] > 9:
|
if info["isLongText"] or info["pic_num"] > 9:
|
||||||
info["text"] = (await self._get_long_weibo(info["mid"]))["text"]
|
info["text"] = (await self._get_long_weibo(info["mid"]))["longTextContent"]
|
||||||
parsed_text = self._get_text(info["text"])
|
parsed_text = self._get_text(info["text"])
|
||||||
raw_pics_list = info.get("pics", [])
|
raw_pics_list = info.get("pics", [])
|
||||||
pic_urls = [img["large"]["url"] for img in raw_pics_list]
|
pic_urls = [img["large"]["url"] for img in raw_pics_list]
|
||||||
|
|||||||
@@ -40,8 +40,8 @@ class Scheduler:
|
|||||||
logger.error(f"scheduler config [{self.name}] not found, exiting")
|
logger.error(f"scheduler config [{self.name}] not found, exiting")
|
||||||
raise RuntimeError(f"{self.name} not found")
|
raise RuntimeError(f"{self.name} not found")
|
||||||
self.scheduler_config = scheduler_config
|
self.scheduler_config = scheduler_config
|
||||||
self.scheduler_config_obj = self.scheduler_config()
|
|
||||||
self.client_mgr = scheduler_config.client_mgr()
|
self.client_mgr = scheduler_config.client_mgr()
|
||||||
|
self.scheduler_config_obj = self.scheduler_config()
|
||||||
|
|
||||||
self.schedulable_list = []
|
self.schedulable_list = []
|
||||||
self.batch_platform_name_targets_cache = defaultdict(list)
|
self.batch_platform_name_targets_cache = defaultdict(list)
|
||||||
|
|||||||
@@ -30,10 +30,12 @@ add_sub_matcher = on_command(
|
|||||||
add_sub_matcher.handle()(set_target_user_info)
|
add_sub_matcher.handle()(set_target_user_info)
|
||||||
do_add_sub(add_sub_matcher)
|
do_add_sub(add_sub_matcher)
|
||||||
|
|
||||||
|
|
||||||
query_sub_matcher = on_command("查询订阅", rule=configurable_to_me, priority=5, block=True)
|
query_sub_matcher = on_command("查询订阅", rule=configurable_to_me, priority=5, block=True)
|
||||||
query_sub_matcher.handle()(set_target_user_info)
|
query_sub_matcher.handle()(set_target_user_info)
|
||||||
do_query_sub(query_sub_matcher)
|
do_query_sub(query_sub_matcher)
|
||||||
|
|
||||||
|
|
||||||
del_sub_matcher = on_command(
|
del_sub_matcher = on_command(
|
||||||
"删除订阅",
|
"删除订阅",
|
||||||
rule=configurable_to_me,
|
rule=configurable_to_me,
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ def admin_permission():
|
|||||||
async def generate_sub_list_text(
|
async def generate_sub_list_text(
|
||||||
matcher: type[Matcher],
|
matcher: type[Matcher],
|
||||||
state: T_State,
|
state: T_State,
|
||||||
user_info: PlatformTarget = None,
|
user_info: PlatformTarget | None = None,
|
||||||
is_index=False,
|
is_index=False,
|
||||||
is_show_cookie=False,
|
is_show_cookie=False,
|
||||||
is_hide_no_cookie_platfrom=False,
|
is_hide_no_cookie_platfrom=False,
|
||||||
|
|||||||
@@ -58,18 +58,3 @@ class ApiError(Exception):
|
|||||||
class SubUnit(NamedTuple):
|
class SubUnit(NamedTuple):
|
||||||
sub_target: Target
|
sub_target: Target
|
||||||
user_sub_infos: list[UserSubInfo]
|
user_sub_infos: list[UserSubInfo]
|
||||||
|
|
||||||
|
|
||||||
class RegistryMeta(type):
|
|
||||||
def __new__(cls, name, bases, namespace, **kwargs):
|
|
||||||
return super().__new__(cls, name, bases, namespace)
|
|
||||||
|
|
||||||
def __init__(cls, name, bases, namespace, **kwargs):
|
|
||||||
if kwargs.get("base"):
|
|
||||||
# this is the base class
|
|
||||||
cls.registry = []
|
|
||||||
elif not kwargs.get("abstract"):
|
|
||||||
# this is the subclass
|
|
||||||
cls.registry.append(cls)
|
|
||||||
|
|
||||||
super().__init__(name, bases, namespace, **kwargs)
|
|
||||||
|
|||||||
+39
-18
@@ -2,16 +2,17 @@ import json
|
|||||||
from typing import Literal
|
from typing import Literal
|
||||||
from json import JSONDecodeError
|
from json import JSONDecodeError
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
|
from collections.abc import Callable
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from httpx import AsyncClient
|
from httpx import AsyncClient
|
||||||
from nonebot.log import logger
|
from nonebot.log import logger
|
||||||
|
|
||||||
|
from ..types import Target
|
||||||
from ..config import config
|
from ..config import config
|
||||||
from .http import http_client
|
from .http import http_client
|
||||||
from ..config.db_model import Cookie
|
from ..config.db_model import Cookie
|
||||||
from ..types import Target, RegistryMeta
|
|
||||||
|
|
||||||
|
|
||||||
class ClientManager(ABC):
|
class ClientManager(ABC):
|
||||||
@@ -42,8 +43,14 @@ class DefaultClientManager(ClientManager):
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class SkipRequestException(Exception):
|
||||||
|
"""跳过请求异常,如果需要在选择 Cookie 时跳过此次请求,可以抛出此异常"""
|
||||||
|
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
class CookieClientManager(ClientManager):
|
class CookieClientManager(ClientManager):
|
||||||
_default_cookie_cd: int = timedelta(seconds=15)
|
_default_cookie_cd = timedelta(seconds=15)
|
||||||
_site_name: str = ""
|
_site_name: str = ""
|
||||||
|
|
||||||
async def _generate_anonymous_cookie(self) -> Cookie:
|
async def _generate_anonymous_cookie(self) -> Cookie:
|
||||||
@@ -98,7 +105,7 @@ class CookieClientManager(ClientManager):
|
|||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
||||||
def _generate_hook(self, cookie: Cookie) -> callable:
|
def _generate_hook(self, cookie: Cookie) -> Callable:
|
||||||
"""hook 函数生成器,用于回写请求状态到数据库"""
|
"""hook 函数生成器,用于回写请求状态到数据库"""
|
||||||
|
|
||||||
async def _response_hook(resp: httpx.Response):
|
async def _response_hook(resp: httpx.Response):
|
||||||
@@ -132,7 +139,7 @@ class CookieClientManager(ClientManager):
|
|||||||
return await self._assemble_client(client, cookie)
|
return await self._assemble_client(client, cookie)
|
||||||
|
|
||||||
async def _assemble_client(self, client, cookie) -> AsyncClient:
|
async def _assemble_client(self, client, cookie) -> AsyncClient:
|
||||||
"""组装 client,可以自定义 cookie 对象的 content 装配到 client 中的方式"""
|
"""组装 client,可以自定义 cookie 对象装配到 client 中的方式"""
|
||||||
cookies = httpx.Cookies()
|
cookies = httpx.Cookies()
|
||||||
if cookie:
|
if cookie:
|
||||||
cookies.update(json.loads(cookie.content))
|
cookies.update(json.loads(cookie.content))
|
||||||
@@ -140,6 +147,15 @@ class CookieClientManager(ClientManager):
|
|||||||
client.event_hooks = {"response": [self._generate_hook(cookie)]}
|
client.event_hooks = {"response": [self._generate_hook(cookie)]}
|
||||||
return client
|
return client
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_name(cls, site_name: str) -> type["CookieClientManager"]:
|
||||||
|
"""创建一个平台特化的 CookieClientManger"""
|
||||||
|
return type(
|
||||||
|
"CookieClientManager",
|
||||||
|
(CookieClientManager,),
|
||||||
|
{"_site_name": site_name},
|
||||||
|
)
|
||||||
|
|
||||||
async def get_client_for_static(self) -> AsyncClient:
|
async def get_client_for_static(self) -> AsyncClient:
|
||||||
return http_client()
|
return http_client()
|
||||||
|
|
||||||
@@ -154,7 +170,25 @@ def is_cookie_client_manager(manger: type[ClientManager]) -> bool:
|
|||||||
return issubclass(manger, CookieClientManager)
|
return issubclass(manger, CookieClientManager)
|
||||||
|
|
||||||
|
|
||||||
class Site(metaclass=RegistryMeta, base=True):
|
site_manager: dict[str, type["Site"]] = {}
|
||||||
|
|
||||||
|
|
||||||
|
class SiteMeta(type):
|
||||||
|
def __new__(cls, name, bases, namespace, **kwargs):
|
||||||
|
return super().__new__(cls, name, bases, namespace)
|
||||||
|
|
||||||
|
def __init__(cls, name, bases, namespace, **kwargs):
|
||||||
|
if kwargs.get("base"):
|
||||||
|
# this is the base class
|
||||||
|
cls._key = kwargs.get("key")
|
||||||
|
elif not kwargs.get("abstract"):
|
||||||
|
# this is the subclass
|
||||||
|
if hasattr(cls, "name"):
|
||||||
|
site_manager[cls.name] = cls
|
||||||
|
super().__init__(name, bases, namespace, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
class Site(metaclass=SiteMeta):
|
||||||
schedule_type: Literal["date", "interval", "cron"]
|
schedule_type: Literal["date", "interval", "cron"]
|
||||||
schedule_setting: dict
|
schedule_setting: dict
|
||||||
name: str
|
name: str
|
||||||
@@ -166,15 +200,6 @@ class Site(metaclass=RegistryMeta, base=True):
|
|||||||
return f"[{self.name}]-{self.name}-{self.schedule_setting}"
|
return f"[{self.name}]-{self.name}-{self.schedule_setting}"
|
||||||
|
|
||||||
|
|
||||||
def create_cookie_client_manager(site_name: str) -> type[CookieClientManager]:
|
|
||||||
"""创建一个平台特化的 CookieClientManger"""
|
|
||||||
return type(
|
|
||||||
"CookieClientManager",
|
|
||||||
(CookieClientManager,),
|
|
||||||
{"_site_name": site_name},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def anonymous_site(schedule_type: Literal["date", "interval", "cron"], schedule_setting: dict) -> type[Site]:
|
def anonymous_site(schedule_type: Literal["date", "interval", "cron"], schedule_setting: dict) -> type[Site]:
|
||||||
return type(
|
return type(
|
||||||
"AnonymousSite",
|
"AnonymousSite",
|
||||||
@@ -185,7 +210,3 @@ def anonymous_site(schedule_type: Literal["date", "interval", "cron"], schedule_
|
|||||||
"client_mgr": DefaultClientManager,
|
"client_mgr": DefaultClientManager,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
class SkipRequestException(Exception):
|
|
||||||
pass
|
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[tool.poetry]
|
[tool.poetry]
|
||||||
name = "nonebot-bison"
|
name = "nonebot-bison"
|
||||||
version = "0.9.4"
|
version = "0.9.5"
|
||||||
description = "Subscribe message from social medias"
|
description = "Subscribe message from social medias"
|
||||||
authors = ["felinae98 <felinae225@qq.com>"]
|
authors = ["felinae98 <felinae225@qq.com>"]
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
|
|||||||
@@ -10,12 +10,11 @@ from nonebug import App
|
|||||||
async def test_cookie(app: App, init_scheduler):
|
async def test_cookie(app: App, init_scheduler):
|
||||||
from nonebot_plugin_saa import TargetQQGroup
|
from nonebot_plugin_saa import TargetQQGroup
|
||||||
|
|
||||||
from nonebot_bison.platform import site_manager
|
|
||||||
from nonebot_bison.config.db_config import config
|
from nonebot_bison.config.db_config import config
|
||||||
from nonebot_bison.scheduler import scheduler_dict
|
from nonebot_bison.scheduler import scheduler_dict
|
||||||
from nonebot_bison.types import Target as T_Target
|
from nonebot_bison.types import Target as T_Target
|
||||||
from nonebot_bison.utils.site import CookieClientManager
|
|
||||||
from nonebot_bison.config.utils import DuplicateCookieTargetException
|
from nonebot_bison.config.utils import DuplicateCookieTargetException
|
||||||
|
from nonebot_bison.utils.site import CookieClientManager, site_manager
|
||||||
|
|
||||||
target = T_Target("weibo_id")
|
target = T_Target("weibo_id")
|
||||||
platform_name = "weibo"
|
platform_name = "weibo"
|
||||||
|
|||||||
+4
-11493
File diff suppressed because it is too large
Load Diff
+3
-11498
File diff suppressed because it is too large
Load Diff
@@ -45,7 +45,7 @@ async def test_fetch_new(weibo, dummy_user_subinfo):
|
|||||||
from nonebot_bison.types import Target, SubUnit
|
from nonebot_bison.types import Target, SubUnit
|
||||||
|
|
||||||
ak_list_router = respx.get("https://m.weibo.cn/api/container/getIndex?containerid=1076036279793937")
|
ak_list_router = respx.get("https://m.weibo.cn/api/container/getIndex?containerid=1076036279793937")
|
||||||
detail_router = respx.get("https://m.weibo.cn/statuses/show?id=4649031014551911")
|
detail_router = respx.get("https://m.weibo.cn/statuses/extend?id=4649031014551911")
|
||||||
ak_list_router.mock(return_value=Response(200, json=get_json("weibo_ak_list_0.json")))
|
ak_list_router.mock(return_value=Response(200, json=get_json("weibo_ak_list_0.json")))
|
||||||
detail_router.mock(return_value=Response(200, text=get_file("weibo_detail_4649031014551911")))
|
detail_router.mock(return_value=Response(200, text=get_file("weibo_detail_4649031014551911")))
|
||||||
image_cdn_router.mock(Response(200, content=b""))
|
image_cdn_router.mock(Response(200, content=b""))
|
||||||
@@ -77,7 +77,7 @@ async def test_fetch_new(weibo, dummy_user_subinfo):
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@respx.mock
|
@respx.mock
|
||||||
async def test_fetch_repost(weibo):
|
async def test_fetch_repost(weibo):
|
||||||
repost_detail_router = respx.get("https://m.weibo.cn/statuses/show?id=4645748019299849")
|
repost_detail_router = respx.get("https://m.weibo.cn/statuses/extend?id=4645748019299849")
|
||||||
repost_detail_router.mock(return_value=Response(200, text=get_file("weibo_detail_4645748019299849")))
|
repost_detail_router.mock(return_value=Response(200, text=get_file("weibo_detail_4645748019299849")))
|
||||||
image_cdn_router.mock(Response(200, content=b""))
|
image_cdn_router.mock(Response(200, content=b""))
|
||||||
raw_post = get_json("weibo_ak_list_1.json")["data"]["cards"][3]
|
raw_post = get_json("weibo_ak_list_1.json")["data"]["cards"][3]
|
||||||
@@ -121,7 +121,7 @@ async def test_fetch_repost(weibo):
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@respx.mock
|
@respx.mock
|
||||||
async def test_video_cover(weibo):
|
async def test_video_cover(weibo):
|
||||||
router = respx.get("https://m.weibo.cn/statuses/show?id=4645748019299849")
|
router = respx.get("https://m.weibo.cn/statuses/extend?id=4645748019299849")
|
||||||
router.mock(return_value=Response(200, text=get_file("weibo_detail_4645748019299849")))
|
router.mock(return_value=Response(200, text=get_file("weibo_detail_4645748019299849")))
|
||||||
image_cdn_router.mock(Response(200, content=b""))
|
image_cdn_router.mock(Response(200, content=b""))
|
||||||
raw_post = get_json("weibo_ak_list_1.json")["data"]["cards"][0]
|
raw_post = get_json("weibo_ak_list_1.json")["data"]["cards"][0]
|
||||||
@@ -152,7 +152,7 @@ async def test_classification(weibo):
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@respx.mock
|
@respx.mock
|
||||||
async def test_parse_long(weibo):
|
async def test_parse_long(weibo):
|
||||||
detail_router = respx.get("https://m.weibo.cn/statuses/show?id=4645748019299849")
|
detail_router = respx.get("https://m.weibo.cn/statuses/extend?id=4645748019299849")
|
||||||
detail_router.mock(return_value=Response(200, text=get_file("weibo_detail_4645748019299849")))
|
detail_router.mock(return_value=Response(200, text=get_file("weibo_detail_4645748019299849")))
|
||||||
raw_post = get_json("weibo_ak_list_1.json")["data"]["cards"][0]
|
raw_post = get_json("weibo_ak_list_1.json")["data"]["cards"][0]
|
||||||
post = await weibo.parse(raw_post)
|
post = await weibo.parse(raw_post)
|
||||||
|
|||||||
Reference in New Issue
Block a user