Compare commits

..

233 Commits

Author SHA1 Message Date
MadDogOwner e11b8a82e7 fix(drivers/cloudreve_v4): remove token check for share (#2274)
Fixed the issue of token verification for shared links.
2026-04-04 18:43:08 +08:00
sdvcrx 12c9bdbd56 fix(offline_download): prevent infinite retry on status update failure (#2294) 2026-04-03 15:47:40 +08:00
Seven 9e49adc353 fix(drivers/openlist): pass through frontend refresh flag (#2307)
* fix(drivers/openlist): pass through frontend refresh flag
* fix(drivers/openlist): gate refresh flag forwarding by config
2026-04-03 15:41:25 +08:00
Pikachu Ren 9fdba3a730 feat(drivers/123_open): support 123 official app api (#2293)
* feat(driver): support 123 official app api

* fix(123_open): migrate api refresh to token.go

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/123_open): trigger proactive refresh with client credentials

* fix(drivers/123_open): use client-credential token endpoint for local refresh

Keep renewapi parsing for expires_in and map it to internal expiry time handling.

* fix(drivers/123_open): limit proactive refresh to client credentials

* fix(drivers/123_open): allow renewapi refresh token proactive init

* fix(drivers/123_open): update API address to use renewapi endpoint

* fix(drivers/123_open): simplify token refresh parsing

* fix(drivers/123_open): unify token expiration to expiredAt

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Suyunmeng <Susus0175@proton.me>
Co-authored-by: Suyunjing <suyunmeng@oplist.org.cn>
2026-04-03 15:23:26 +08:00
Suyunjing 7bea29c18e refactor(db): migrate sqlite to pure-go and add mips compatibility switch (#2296)
- Switch default SQLite path to github.com/glebarez/sqlite to reduce CGO dependency pressure.
- Introduce a unified openSQLite entry in bootstrap and split driver selection by build tags.
- Add sqlite_cgo_compat fallback for linux mips, mips64, loong64 and mipsle to keep legacy target builds working.
- Update build.sh musl build flow to apply compatibility tag for mips-family targets.
- Update beta_release workflow to pass compatibility tag cleanly and avoid conflicting flag composition.
2026-04-03 01:27:02 +08:00
ShenLin 29447a41a3 revert(db)!: replace SQLite Driver with glebarez/sqlite to avoid CGO (#2269)
Revert "refactor(db)!: replace SQLite Driver with glebarez/sqlite to avoid CG…"

This reverts commit d598ef7569.
2026-03-26 21:01:48 +08:00
Jealous d85f084acb feat(permissions): implement fine-grained permission control (#2145)
* refactor(permission): rename permission check functions for clarity

- User.CanWrite() → User.CanCreateFilesOrFolders()
- common.CanWrite() → common.CanWriteContentBypassUserPerms()
- common.IsApply() → common.MetaCoversPath()

Improves code readability by making function names more descriptive.
The new MetaCoversPath name clearly indicates it checks if a meta rule
covers a specific path. It better conveys that it's a query function
rather than an action, and the applyToSubFolder parameter is more
explicit than applySub.

Also adds comprehensive test coverage:
- 10 tests for MetaCoversPath core logic
- 6 tests for CanWriteContent
UserPerms
- 7 tests for getReadme
- 5 tests for getHeader
- 6 tests for isEncrypt
- 9 tests for whetherHide

Total: 43 test scenarios covering all path matching and permission
inheritance logic. Tests verify both normal behavior and bug fixes
for Readme/Header information leakage and write permission bypass.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat(permission): implement fine-grained user permissions for read/write operations

Add per-user read and write permission controls at the meta level to enable
more granular access control beyond the existing permission flags.

Key changes:
- Add ReadUsers/WriteUsers fields to Meta model with sub-directory inheritance flags
- Implement CanRead and CanWrite permission check functions in server/common
- Filter file list results based on user read permissions
- Add permission checks across all file operations (FTP, HTTP handlers, WebDAV)
- Simplify error handling pattern for MetaNotFound errors throughout codebase

This allows administrators to restrict specific users from accessing or modifying
certain paths, providing finer control over file system permissions.

Note: Batch and recursive operations (FsMove, FsCopy, FsRemove, FsRecursiveMove,
FsBatchRename, FsRegexRename) currently check parent directory permissions only.
Individual item permission checks are not performed for performance reasons.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* test(permission): add comprehensive tests for CanRead, CanWrite, and combined permission checks

Add TestCanRead, TestCanWrite, TestCanAccessWithReadPermissions, and
TestWritePermissionCombinations to validate the three-layer permission
system including nil user/meta, sub-path inheritance, user whitelists,
and root-level restrictions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(webdav): use safe type assertion for MetaPassKey to prevent panic

Bearer-token and OPTIONS auth paths do not set MetaPassKey in context,
causing a panic when handlers perform a forced type assertion on nil.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(permission): treat nil user as system context in CanRead/CanWrite

Previously, CanRead/CanWrite returned false for nil user, causing
filterReadableObjs to return an empty list when fs.List is called from
internal contexts without a user (e.g. context.Background()). A nil user
represents an internal/system call and should bypass per-user restrictions,
consistent with how whetherHide already handles nil user.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(fsmanage): prevent path traversal in FsRemove

The previous check only skipped names that resolved to "/", but did not
prevent traversal to sibling directories (e.g. "../secret"), which could
bypass the CanWrite permission check that is only applied to req.Dir.

Replace with a post-join prefix check to ensure each resolved path stays
within reqPath.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(webdav): align MetaPassKey behavior with FTP auth logic

For guest users, the WebDAV password input serves as the meta folder
password (consistent with FTP anonymous/guest handling). For authenticated
users, MetaPassKey is set to empty string since their login password is
not the meta folder password.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(permission): require write auth for fs list refresh

* refactor(permission): use MetaCoversPath in CanRead/CanWrite for consistency

Replace inline `(Sub || meta.Path == path)` logic with MetaCoversPath,
consistent with CanWriteContentBypassUserPerms. Also fix a copy-paste
error in the CanWrite comment (read → write).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-03-26 14:42:35 +08:00
Pikachu Ren d598ef7569 refactor(db)!: replace SQLite Driver with glebarez/sqlite to avoid CGO (#2211)
* mod(db): driver/sqlite ->glebarez/sqlite

* mod(db): driver/sqlite ->glebarez/sqlite

* [WIP] Refactor SQLite Driver with glebarez/sqlite to avoid CGO (#2213)

* Initial plan

* fix: address review comments - fix import order, update test, remove CGO sqlite deps

Co-authored-by: PIKACHUIM <40362270+PIKACHUIM@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: PIKACHUIM <40362270+PIKACHUIM@users.noreply.github.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-25 17:45:37 +08:00
Elegant1E e41b683efb feat(drivers): add doubao_new driver (#2114) 2026-03-23 17:19:45 +08:00
Jealous 9a2ba1dabe fix(server): add missing return after error responses (#2150)
In BeginAuthnRegistration (webauthn.go), missing return statements after
error responses caused the function to continue executing with a nil
authnInstance, potentially leading to a nil pointer panic.

In OIDCLoginCallback and SSOLoginCallback (ssologin.go), missing return
statements after GenerateToken/autoRegister errors caused the handler to
send a second response, resulting in a superfluous response write.

In SetThunderBrowser (offline_download.go), the default case of the
storage type switch sent an error response but did not return, causing
SaveSettingItems and tool initialization to continue executing even when
driver type validation failed.
2026-03-16 22:22:55 +08:00
Copilot f3428e65bc fix(net): honor proxy settings when uploading to 115/115 Open/PikPak OSS (#2222)
* Initial plan

* fix: honor HTTPS proxy for OSS uploads

Co-authored-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>

* Honor HTTPS proxy settings for 115/115 Open/PikPak OSS uploads

Co-authored-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>

* revert

* chore

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: jyxjjj <773933146@qq.com>
2026-03-16 22:15:00 +08:00
Roy 5eaef96078 fix(azure): remove properties and fix prefix (#2209)
Remove properties from azure blob response

fix azure blob prefix filter: prefix should be empty if it is "/"
2026-03-09 21:25:30 +08:00
ShenLin b5626b275b chore(ci)!: update issue CI configuration [skip ci] (#2166) 2026-02-25 16:32:09 +08:00
MadDogOwner 795a18b565 feat(drivers/thunder*): implement GetDetails (#2113)
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-02-21 15:21:05 +08:00
MadDogOwner 82ae2d5890 chore(handles/auth): improve error response (#2148)
* chore(handles/auth): improve error response

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* Apply suggestion from @xrgzs

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-02-19 17:15:40 +08:00
MadDogOwner db0e2ec103 feat(security): add SECURITY.md (#2147)
[skip ci]

Add SECURITY.md

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-02-19 17:14:55 +08:00
mkitsdts e0ee737080 fix(driver/wps): fetch all files via multiple API invocations (#2139)
fix(driver/wps): wps list all files in one request
2026-02-19 14:48:43 +08:00
LXY 0673a74302 chore(typo): fix typo in UpdateFileReq field name (#2133)
Fix typo in UpdateFileReq field name

ref: https://github.com/OpenListTeam/115-sdk-go/pull/3

Signed-off-by: LXY <767763591@qq.com>
2026-02-16 22:57:43 +08:00
jenfonro a8d1c0ddbf fix(‎drivers/quark_uc_tv) : Update error code judgment (#2080)
* fix error code

* add ErrorInfo check
2026-02-08 21:15:00 +08:00
renovate[bot] 8431c1b1e3 fix(deps): update go4.org digest to a507140 (#2095)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-02-08 16:26:54 +08:00
gdm257 a121f861dc feat(drivers/123open): support sha1 reuse api (#2089)
* feat(drivers/123open): support sha1 reuse api

* fix(drivers/123open): fix typos
2026-02-08 14:01:26 +08:00
MadDogOwner 6861cb4c3b chore(ci): add breaking change guideline to PR title check (#2087)
* feat(ci): add PR title validation for breaking changes

Updated regex to allow '!' for breaking changes in PR titles.

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* chore(pr): Update PR template

Add bilingual instructions for PR title formatting.

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-02-06 21:52:06 +08:00
Shelton Zhu 5d9fc8359d fix(115_share): adjust 115 share driver for official API update (#2068)
* fix(115_share): add user agent support and update driver dependency

* fix(115): fix download error

* feat: add thumbnail support for 115 driver and 115 share

- Add Thumb() method to FileObj in 115 driver to return thumbnail URL
- Add ThumbURL field to FileObj struct in 115 share utility
- Update 115driver dependency from v1.2.2 to v1.2.3 to support thumbnail functionality
- Implement Thumb() method for 115 share FileObj to return thumbnail URL
2026-02-02 18:43:26 +08:00
Chaloemchai f542187654 fix(drivers/teldrive): enhance file listing and upload functionality with pagination and random chunk naming (#2034)
* fix(drivers/teldrive): enhance file listing and upload functionality with pagination and random chunk naming

* fix(drivers/teldrive): optimize file listing by removing unnecessary mutex and restructuring data handling

* Update drivers/teldrive/meta.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Chaloemchai <chaloemchai.yy@gmail.com>

---------

Signed-off-by: Chaloemchai <chaloemchai.yy@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-01 20:39:25 +08:00
ShenLin e3c664f81d Merge commit from fork
Co-authored-by: KirCute <951206789@qq.com>
2026-01-31 16:52:20 +08:00
ShenLin 7b78fed106 Merge commit from fork
Co-authored-by: KirCute <951206789@qq.com>
2026-01-31 16:50:32 +08:00
Hu Yuantao d685bbfa9a fix(api/remove): add validation for empty items in delete file list (#1617)
* fix(FsRemove): add validation for empty items in delete file list

If Req.Names contains an empty string item, the whole directory will be removed. As a result we need add a simple guard to prevent such cases.

Signed-off-by: huyuantao <huyuantao@ultrarisc.com>

* fix(FsRemove): enhance validation to prevent unintended directory deletion

1. Use `utils.FixAndCleanPath` to correctly identify and block invalid names.
2. Change error handling from `return` to `continue`.

Signed-off-by: huyuantao <huyuantao@ultrarisc.com>

---------

Signed-off-by: huyuantao <huyuantao@ultrarisc.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-01-29 21:48:16 +08:00
mkitsdts 27732ccc88 fix(drivers/quark): apply html escaping in quark (#2046)
* fix(drivers/quark): apply html escaping in quark
2026-01-29 21:30:17 +08:00
MadDogOwner 29fcf5904a fix(drivers/cloudreve_v4): add IsFolder attribute to Getter response (#2035)
* fix(drivers/cloudreve_v4): add IsFolder attribute to Getter response

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* refactor(drivers/cloudreve_v4): implement File.fileToObject method

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/cloudreve_v4): implement 404 not found for getter

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-01-28 19:30:52 +08:00
KirCute f0e53d18a8 fix(drivers/alias): default sort & substitute link (#1917)
* fix(drivers/alias): default sort & substitute link

* fix

* fix
2026-01-26 19:31:09 +08:00
Rui Huang a2573fb285 fix(drivers/seafile): object not found when RootFolderPath != "/" (#2010)
* fix(driver/seafile): object not found when RootFolderPath != "/"

* refactor(seafile): restructure Seafile driver for improved library handling and error management

* add IsDir method to LibraryInfo type

* improve initialization

* add repoID to RepoItemResp and update List method to set repoID

---------

Co-authored-by: Khoray <hhkorm@gmail.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
2026-01-25 17:00:32 +08:00
hcrgm 031b719bb0 perf(baidu_netdisk): reduce the number of file listing requests (#2016)
* 本优化减少了百度网盘驱动下文件列表的请求次数,能加快文件浏览速度。此前只要文件夹下有至少一个文件,都会至少发出2次列表请求。

Signed-off-by: hcrgm <hcrgm@qq.com>
2026-01-25 16:47:03 +08:00
mkitsdts d8417e050c fix(webdav/move): fix source file still exist after moving file by webdav (#1979) 2026-01-21 17:27:19 +08:00
Cp0204 02aec71383 fix(driver/quark_uc): Fix display of non-compliant filenames (#2000) 2026-01-20 19:30:44 +08:00
MadDogOwner fca993a830 feat(drivers/cloudreve): implement GetDetails (#1960) 2026-01-20 00:49:00 +08:00
Tron a79d8347bd fix(fs): handle non-existent destination directory in file transfer (#1898)
* fix(FileTransferTask): skip copying if destination directory does not exist

* pass only object not found error

---------

Co-authored-by: cyk <dr_arc@163.com>
Co-authored-by: KirCute <951206789@qq.com>
2026-01-17 20:12:31 +08:00
KirCute f057846e01 feat(drivers): add autoindex driver (#1978)
* feat(drivers): add autoindex driver

* fix

* add NoUpload

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* add TestParseSize

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* go mod tidy

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* use base.RestyClient

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix: support evaluate size and modified time

* fix apache

* perf

* feat: support ignore size and modified time

* rename driver

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2026-01-17 20:08:32 +08:00
KirCute 27fdd03ec3 fix(ci): change unchecked regex (#1993)
fix(ci/issue): change unchecked regex

Signed-off-by: KirCute <951206789@qq.com>
2026-01-17 20:00:43 +08:00
Seven ea4b5e3251 fix(drivers/strm): delete extra local directories in sync mode (#1980) 2026-01-16 15:52:04 +08:00
KirCute 378e37b082 fix(drivers/github): failed to get objs (#1915)
fix(drivers/github): failed get objs
2026-01-16 12:07:27 +08:00
MadDogOwner 5c82f55200 feat(drivers/cloudreve_v4): implement Getter interface (#1937)
feat(cloudreve_v4): implement Getter interface

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-01-16 12:05:53 +08:00
KirCute b553dc35b6 fix(drivers/ftp): failed to get objs (#1948)
fix(drivers/ftp): failed get objs
2026-01-16 12:04:55 +08:00
Ljcbaby e6db0bd6ed chore(.github): update issue templates [no ci] (#1976)
chore(.github): update issue templates
2026-01-16 12:02:45 +08:00
KirCute 85c69d853f fix(fs): panic when failed to get storage details (#1964) 2026-01-13 22:01:35 +08:00
MadDogOwner c6bd437242 feat(drivers/webdav): add support for 302 redirects (#1952)
* Remove the `OnlyProxy` restriction and obtain the redirected link to support 302

* Add `driver.Config` `PreferProxy` to recommend user to enable the proxy by default

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-01-11 15:15:42 +08:00
KirCute e192149480 fix(drivers/alist_v3): failed to unmarshal MeResp (#1931) 2026-01-08 10:54:14 +08:00
TwoOnefour 0d99e16b59 fix(cloudreve): Manually set path to avoid empty path (#1933) 2026-01-08 10:53:09 +08:00
KirCute 744dbd5e26 feat(drivers): support getting disk usage of some drivers (#1905)
* feat(drivers): support getting disk usage of some drivers

* feat(drivers/degoo): implement GetDetails

* fix(fs/storage-details): fill used space rather than free space

* fix mega

* fix bsize type
2026-01-08 10:47:22 +08:00
Seven 7088fa1f69 chore(strm): remove redundant param (#1929) 2026-01-07 18:34:33 +08:00
mcxiedidi 642acf8bca feat(123pan): add offline download (#1911)
* feat(123网盘): 添加123网盘离线下载功能

- 新增123网盘离线下载实现
- 添加相关API接口和常量配置
- 在路由和工具集中集成123网盘支持

* refactor(offline_download): 重构123网盘离线下载状态处理和类型定义

- 将离线下载相关类型定义从util.go移至types.go
- 更新状态获取api

* 移除了备选方案(/offline_download/task/status)
2026-01-04 23:13:38 +08:00
KirCute 747993e2ca feat(drivers/mega): support permanently deleting files (#1913) 2026-01-04 23:07:59 +08:00
KirCute 1f9a7e3a91 feat(drivers/mega): support getting disk usage (#1903)
* feat(drivers/mega): support getting disk usage

* fix
2026-01-04 13:26:26 +08:00
MadDogOwner 874234449b fix(doubao_share): use new download info (#1890)
fix(doubao_share): update file URL retrieval to use new download info structure

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-01-01 22:29:03 +08:00
Edward 5fe267089a fix(123_open): infinite recursive call (#1854)
fix(123_open): token refresh logic

Fix token handling logic to avoid deadlock. Token method took reference of Alist's implementation.
2025-12-31 00:46:54 +08:00
KirCute 2442e302ad ci(lang): sync only new fields (#1881) 2025-12-30 15:43:30 +08:00
Tron 0612271732 fix(driver): fix file copy failure to 123pan due to incorrect etag (#1874) 2025-12-29 23:54:33 +08:00
我怎么就不是一只猫呢? c261ce78fb fix(s3): use current time as default modified time (#1860) 2025-12-29 23:52:35 +08:00
KirCute 7398e7d45e feat(alias): support load balance (#1767)
* feat(alias): support load balance

* feat(alias): support storage match for load balance

* feat(patch): add alias addition upgrade patch

* fix bugs

* fix(op/balance): optimize compatibility

* chore: change default read conflict policy

* feat(alias): refactor Alias initialization and enhance path handling

* feat(alias): enhance object masking and add support for operation restrictions

* feat(alias): enhance object masking

* feat(fs): add permission checks

* improve parsing

* update object masks

* feat(fs): enhance virtual file handling

* feat(storage): enhance virtual file retrieval and path handling

* refactor(alias): rename path handling functions for clarity and consistency

* fix(alias): update path handling in Other method to use balanced path

* fix bug

* feat(alias): add file size validation

* feat(alias): add hash consistency check

* 移除哈希合并,

* fix(alias): wrong behavior for all_strict/deterministic_or_all

* Revert "fix(alias): wrong behavior for all_strict/deterministic_or_all"

This reverts commit f001f2dcd7.

* fix(alias): wrong behavior for all_strict/deterministic_or_all

* feat(alias): support part-based read load balance

* fix(alias): list panic when leak conflict path

* fix(alias): remove Other load balance

* fix(alias): 修复 Link 方法中 resultLink 的返回类型和内容复制问题

* fix(alias): 更好的下载并发?

* chore(alias): all tips

* fix(alias): moving paths mismatch

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: ShenLin <773933146@qq.com>
2025-12-29 17:16:07 +08:00
KirCute 6e2d499ca9 refactor(bootstrap): fix OpenList-Mobile compile failed (#1857) 2025-12-24 18:46:13 +08:00
绎泽 4680ece2d9 docs(readme): add demo site (#1850)
Last Sync: 2025-12-22 12:39
2025-12-22 12:52:41 +08:00
Seven 8a4f3769d8 feat(strm): add save local mode (#1814)
* feat(strm): add KeepSameNameOnly logic

* chore(strm): skip update strm file when keepLocalDownloadFile

* feat(strm): add save local mode
2025-12-22 10:12:21 +08:00
foxxorcat cc5172e70b fix(weiyun): update sdk and support getDetails (#1845) 2025-12-22 00:20:07 +08:00
XZB-1248 a32ae97860 docs: update README for zh-CN (#1844) 2025-12-22 00:16:33 +08:00
TwoOnefour d6dd62dfe5 fix(s3): incorrect copy key with plus sign (#1820) 2025-12-22 00:15:58 +08:00
XZB-1248 216f071e64 docs: add VPS.Town as sponsor to all README (#1842)
Co-authored-by: XZB-1248 <i@1248.ink>
2025-12-21 12:26:03 +08:00
hshpy f47df5f9b2 feat(115_open): support custom pagesize (#1822) 2025-12-20 13:57:02 +08:00
KirCute ff3c4b885c fix(strm): support generate strm with sign (#1832) 2025-12-20 13:55:51 +08:00
MadDogOwner f86c7c844c feat(cloudreve_v4): add ks3 support (#1828)
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-12-19 18:05:11 +08:00
Mako (XSpy) 5db2172ed6 feat(driver): add personal / business wps drive support (#1802)
* feat(driver): add wps drive support

* feat(driver): add wps drive support

* fix(wps): update personal mode string to English

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(wps): remove trailing slash from drive origin URL

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(wps): correct order of options in mode selection

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(wps): enable local sort and upload overwrite

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(wps): resolve put bugs, fix file op problems and optimize list logic

- Fix uploading bugs. Support all uploading methods based on 8825.85d3c864.js
- Fix issues in delete/copy/move while opearting big folders.
- Use cache to optimize performance of list, especially in a deep path.

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-12-15 21:49:01 +08:00
wongz c4c121befc fix(139): disk-usage unmarshal failed when used capacity overflow (#1718)
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2025-12-15 21:30:08 +08:00
UcnacDx2 b4542753ba feat(drivers/139): user authentication and file batch operations (#1534)
* feat(139): Enhance 139 driver with password login and root path handling

- Added support for password-based login in the 139 driver.
- Introduced RootPath field to store the root directory path.
- Updated Init method to handle family and group types more effectively.
- Implemented new methods for handling file operations in family and group contexts.
- Enhanced error handling and logging for better debugging.
- Added new request and response structures for batch operations and document modifications.
- Improved encryption and decryption methods for secure communication.

* Update drivers/139/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>

* Update drivers/139/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>

* Update drivers/139/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>

* Update drivers/139/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>

* Update drivers/139/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>

---------

Signed-off-by: UcnacDx2 <127503808+UcnacDx2@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-15 19:38:33 +08:00
KirCute 2a99c97d52 feat(ldap): support webdav, ftp and sftp login (#1746)
* feat(ldap): support webdav, ftp and sftp login

* fix: apply suggestions of Copilot

* feat(ldap) support ftp, sftp and webdav auto-register
2025-12-15 16:53:38 +08:00
MadDogOwner 0a407c3d8b fix(openlist): disable status check for openlist driver (#1757)
* fix(openlist): disable status check to avoid network stability issues

* fix(alist_v3): disable status check to avoid network stability issues

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-12-15 16:48:05 +08:00
KirCute b2596fdc24 refactor(bootstrap): move booting logic to bootstrap package (#1773)
* refactor(bootstrap): move booting to bootstrap package

* chore(log): reduce level of some callings of `utils.Log.Fatal`

* fix(s3): no shutdown after SIGTERM received

* fix: add handle hook
2025-12-15 16:47:50 +08:00
zzzhr1990 2dbe1b00d3 fix(halalcloud_open): halal-cloud upload issues (#1800)
fix halal-cloud upload issues
2025-12-15 16:47:09 +08:00
KirCute 1fc9c83df1 fix(ilanzou): parse vip size (#1792) 2025-12-12 12:13:42 +08:00
j2rong4cn d31e1a333d feat(model): add object mask support and enhance cache/task handling (#1743) 2025-12-11 15:10:42 +08:00
jenfonro e1bba7072b fix(task): tasks keep being cancelled (#1745)
* fix_cancel

* update(go.mod): update tache version

* tache v0.2.2

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-12-10 19:09:16 +08:00
MadDogOwner 94c7d68413 feat(utils): add support for ignoring '@eaDir' system files (#1779) 2025-12-10 13:45:14 +08:00
KirCute 9ed77a5875 feat(driver): add AList v3 (#1721)
* feat(driver/openlist): compatible with AList v3

* Revert "feat(driver/openlist): compatible with AList v3"

This reverts commit 90f3f80186.

* feat(driver): add AList v3

* Revert "feat(patch): add migration from Alist V3 driver to OpenList (#919)"

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-12-08 22:25:10 +08:00
varg1714 7d6d3b8f55 feat(fs): Support customizing the cache time for a specific path (#1533)
* feat(fs): Support customizing the cache time for a specific path

* feat(fs): Get the cache rule for driver information.

* feat(fs): Support globbing.

* feat(fs): Add log.

---------

Signed-off-by: ShenLin <773933146@qq.com>
Co-authored-by: ShenLin <773933146@qq.com>
2025-12-04 09:59:27 +08:00
ShenLin 5480d61f70 refactor!(userAgent): merge most userAgent into base (#1722)
refactor!(userAgent): merge all userAgent into base

1. change var to const
2. remove duplicated ua definetion after original Resty R
3. upgrade Chrome and OS versions
2025-12-04 09:54:14 +08:00
j2rong4cn 96cd714385 refactor(op): remove automatic Path assignment (#1734)
* refactor: 移除 ObjResp 中的 Id 和 Path 字段

* 移除op.List的自动设置Path
Path和Id只在驱动内使用,不应由op.List设置Path

* cnb_releases:将 Addition 结构体中的 RootPath 字段为 RootID
当List方法加载二级目录时,若使用的是Id,应对使用driver.RootID

* doubao_share: 添加潜在bug注释

* 添加 GetRootPath 方法到多个驱动
2025-12-03 00:55:40 +08:00
VXTLS e29d92f92e fix(mediafire): enable automatic session token acquisition and fix gzip parsing (#1661)
* fix(mediafire): enable automatic session token acquisition and fix gzip parsing

- Fix Init() method to allow automatic session token retrieval from cookie
- Change SessionToken from required to optional in configuration
- Add proper gzip decompression support for API responses
- Improve error handling for session token acquisition failures
- Update help text to clarify authentication requirements

Resolves initialization failure and JSON parsing errors when session token
can be automatically obtained from browser cookie.

* fix(mediafire): ensure driver files end with newline

* chore: gofmt drivers/mediafire/*.go
2025-12-02 12:13:39 +08:00
ShenLin c5f57bbcc5 fix(drivers/crypt): remove hard dependency on RemotePath (#1713) 2025-11-28 01:21:22 +08:00
j2rong4cn 9835afc645 refactor: improve upload handling (#1455)
* fix(quark): refactor upPart to use http.NewRequest

* fix(quark): improved upload handling

* fix(quark_open): improved upload handling

* fix: add retry context to multiple upload functions

* fix: optimize hash calculation in multipart upload to avoid blocking

* fix: update error handling in lifecycle functions for better clarity

* fix: update upload progress calculation to improve accuracy

* fix: simplify error handling in lifecycle functions for improved readability

* fix: remove unnecessary mutex for part uploads to simplify code

* fix(stream): simplify file handling in NewStreamSectionReader and improve error messages

* fix(terabox): optimize chunk count calculation in Put method

* perf(chaoxing): 表单上传文件0拷贝

* fix(cnb_releases): improve file upload progress tracking

* fix(baidu_netdisk): improve upload handling

* fix(upload): optimize buffer initialization for file uploads

* fix(baidu_netdisk): add retry condition to skip ErrUploadIDExpired in upload loop
2025-11-27 19:34:03 +08:00
Seven 1f373eac8d chore(strm): avoid generating empty folders (#1720)
chore(strm): empty folders are not generated locally
2025-11-27 14:02:11 +08:00
jenfonro ede96a314c fix(onedrive_shareurl): Reduce temporary file errors (#1686)
* fix onedrive_shareurl

* .
2025-11-25 21:43:17 +08:00
Seven 72206ac9f6 feat(strm): keep local download file (#1707) 2025-11-25 18:08:48 +08:00
ShenLin 62dedb2a2e fix(pkg/aria2): use pointer receivers for Call methods (#1706) 2025-11-25 13:25:42 +08:00
ShenLin 7189c5b461 chore(pkg/aria2): simplify context cancellation handling in RPC calls (#1705) 2025-11-25 13:09:04 +08:00
ShenLin 1a445f9d3f chore(archive): fix struct literal uses unkeyed fields (#1704) 2025-11-25 13:05:36 +08:00
ShenLin aa22884079 fix(search): fix duplicated variable init (#1703) 2025-11-25 12:23:25 +08:00
ImoutoHeaven 316d4caf37 feat(search): Add task queue for Meilisearch to prevent race conditions (#1423)
* Add task queue for Meilisearch to prevent race conditions

- Implement TaskQueueManager for async index operations
- Queue update tasks and process them in batches every 30 seconds
- Check pending task status before executing new operations
- Optimize batch indexing and deletion logic
- Fix type assertion bug in buildSearchDocumentFromResults

* fix(search): re-enqueue skipped tasks to prevent task loss

When tasks are skipped due to pending dependencies, they are now
re-enqueued if not already in queue. This prevents task loss while
avoiding overwriting newer snapshots for the same parent.

* fix(copilot-comment): Invoke Stop() & err of SliceConvert

---------

Co-authored-by: ImoutoHeaven <noreply@imoutoheaven.org>
Co-authored-by: jyxjjj <773933146@qq.com>
2025-11-25 11:38:27 +08:00
KirCute 60a489eb68 feat(archive): support non-overwrite decompress (#1701) 2025-11-25 10:27:29 +08:00
varg1714 b22e211044 feat(fs): Add skipExisting option to move and copy, merge option to copy (#1556)
* fix(fs): Add skipExisting option to move and copy.

* feat(fs): Add merge option to copy.

* feat(fs): Code smell.

* feat(fs): Code smell.
2025-11-24 14:20:24 +08:00
KirCute ca401b9af9 fix(local): assign non-CoW copy requests to the task module (#1669)
* fix(local): assign non-CoW copy requests to the task module

* fix build

* fix cross device
2025-11-24 14:14:53 +08:00
jenfonro addce8b691 feat(baidu_netdisk): Add shard upload timeout setting (#1682)
add timeout
2025-11-24 14:14:34 +08:00
Seven 42fc841dc1 feat(strm): custom path prefixes (#1697)
fix(strm): custom path prefixes

Signed-off-by: ShenLin <773933146@qq.com>
Co-authored-by: ShenLin <773933146@qq.com>
2025-11-24 14:11:59 +08:00
varg1714 4c0916b64b fix(strm): fix the name and type issue (#1630)
* fix(strm): fix the name and type issue

* fix(strm): update version
2025-11-24 14:05:49 +08:00
VXTLS 3989d35abd fix(misskey): folderId format validation and root directory handling (#1647)
fix(misskey): Fix folderId format validation and root directory handling
2025-11-21 12:18:54 +08:00
KirCute 72e2ae1f14 feat(fs): support manually trigger objs update hook (#1620)
* feat(fs): support manually trigger objs update hook

* fix: support driver internal copy & move case

* fix

* fix: apply suggestions of Copilot
2025-11-21 12:18:20 +08:00
Seven 3e37f575d8 fix(openlist_driver): ensure UA is correctly propagated (#1679) 2025-11-21 12:13:41 +08:00
MoYan c0d480366d fix(driver/123): initialize Platform field (#1644)
* fix(driver/123): initialization the Platform field

Signed-off-by: MoYan <1561515308@qq.com>

* Fix formatting of Platform field in Pan123

Signed-off-by: MoYan <1561515308@qq.com>

---------

Signed-off-by: MoYan <1561515308@qq.com>
2025-11-14 18:49:44 +08:00
Copilot 9de7561154 feat(upload): add optional system file filtering for uploads (#1634) 2025-11-14 14:45:39 +08:00
MadDogOwner 0866b9075f fix(link): correct link cache mode bitwise comparison (#1635)
* fix(link): correct link cache mode bitwise comparison

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* refactor(link): use explicit flag equality for link cache mode bitmask checks

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-11-13 13:52:33 +08:00
KirCute 055696f576 feat(s3): support frontend direct upload (#1631)
* feat(s3): support frontend direct upload

* feat(s3): support custom direct upload host

* fix: apply suggestions of Copilot
2025-11-13 13:22:17 +08:00
ShenLin 854415160c chore(issue templates): require logs (#1626) 2025-11-12 13:04:13 +08:00
varg1714 8f4f7d1291 feat(doubao): Add rate limiting (#1618) 2025-11-11 21:59:10 +08:00
KirCute ee2c77acd8 fix(archive/zip): user specific encoding for non-EFS zips (#1599)
* fix(archive/zip): user specific encoding for non-EFS zips

* fix(stream): simplify head cache initialization and improve reader retrieval logic

* fix: support multipart zips (.z01)

* chore(deps): update github.com/KirCute/zip to v1.0.1

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2025-11-10 19:08:50 +08:00
yuyamionini fc90ec1b53 fix(terabox): wrong return code used (#1547)
fix(terabox): rename, delete, copy operations sometimes failed

Signed-off-by: yuyamionini <46483865+yuyamionini@users.noreply.github.com>
2025-11-10 13:40:00 +08:00
jenfonro 7d78944d14 fix(baidu_netdisk): Fix Baidu Netdisk resume uploads sticking to the same upload host (#1609)
Fix Baidu Netdisk resume uploads sticking to the same upload host
2025-11-09 20:43:02 +08:00
jenfonro f2e0fe8589 refactor(fs): implement immediate retry within task execution cycle (#1575) 2025-11-07 19:11:11 +08:00
ASLant 39dcf9bd19 feat(onedrive): support frontend direct upload (#1532)
* OneDrive添加直连上传

* refactor

* fix: duplicate root path join

---------

Co-authored-by: KirCute <951206789@qq.com>
2025-11-06 23:22:02 +08:00
Seven 25f38df4ca fix(strm): non-specified type generates strm (#1585)
* fix(strm): non-specified type generates strm

* fix(strm): only insert to strmTrie if SaveStrmToLocal is enabled

* fix(strm): update suffix handling in convert2strmObjs function

* fix(strm): refactor generateStrm to use range reader

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-11-06 20:58:43 +08:00
j2rong4cn a1f1f98f94 refactor(stream): simplify code (#1590)
* refactor(stream): simplify Close method and update SeekableStream to use RangeReader interface

* refactor(stream):  improve RangeRead comments for clarity
2025-11-06 20:06:48 +08:00
KirCute affc499913 fix(189): disk-usage unmarshal failed when used capacity overflow (#1577) 2025-11-05 12:35:51 +08:00
KirCute c7574b545c feat(github_release): support Source code (zip/tar.gz) (#1581)
* support Github Release Source code (zip/tar.gz)

* fix TarballUrl and ZipballUrl

* fix show source code by allversion

---------

Co-authored-by: nibazshab <44338441+nibazshab@users.noreply.github.com>
2025-11-05 12:30:05 +08:00
hcrgm 9e852ba12d fix(baidu_netdisk): improve upload experience (#1562)
* fix(baidu_netdisk): improve upload experience

* fix(typo): URL should be uppercase, apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: ShenLin <773933146@qq.com>

* fix(typo): URL should be uppercase, apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: ShenLin <773933146@qq.com>

* fix(baidu_netdisk): use "UploadAPI" as a fallback when using dynamic upload api

* fix(baidu_netdisk): all uploads share the same upload url cache

* fix(drivers/baidu_netdisk): defer uploadUrlMu unlock

* update driver.go to main

---------

Signed-off-by: ShenLin <773933146@qq.com>
Signed-off-by: jenfonro <799170122@qq.com>
Co-authored-by: ShenLin <773933146@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: jenfonro <799170122@qq.com>
2025-11-05 12:21:32 +08:00
j2rong4cn 174eae802a perf(stream): optimize CacheFullAndWriter for better memory management (#1584)
* perf(stream): optimize CacheFullAndWriter for better memory management

* fix(stream): ensure proper seek handling in CacheFullAndWriter for improved data integrity
2025-11-05 12:16:09 +08:00
KirCute b9f058fcc9 fix(backup-restore): add shares (#1500) 2025-11-05 12:11:20 +08:00
j2rong4cn 6de15b6310 feat(stream): enhance GetRangeReaderFromLink rate limiting (#1528)
* feat(stream): enhance GetRangeReaderFromLink rate limiting

* refactor(stream): update GetRangeReaderFromMFile to return *model.FileRangeReader

* refactor(stream): simplify context error handling in RateLimitReader, RateLimitWriter, and RateLimitFile

* refactor(net): replace custom LimitedReadCloser with readers.NewLimitedReadCloser

* fix(model): update Link.ContentLength JSON tag for correct serialization

* docs(model): add clarification to FileRangeReader usage comment
2025-11-04 23:56:09 +08:00
jenfonro 2844797684 fix(baidu_netdisk): support resuming uploads when an error occurs (#1279)
support resuming uploads when an error occurs
2025-11-04 13:33:33 +08:00
Seven 9f4e439478 chore(strm): Built-in file types support modification (#1483) 2025-11-04 10:33:16 +08:00
jenfonro 9d09ee133d fix(google_driver): fix google link file display size (#1335)
* fix file link display size

* fix performance and field

* cn to en notes

---------

Co-authored-by: ShenLin <773933146@qq.com>
2025-11-04 09:26:20 +08:00
jenfonro d88f0e8f3c feat(net): support proxy configuration via config file (#1359)
* support proxy

* debug

* debug2

* del debug

* add proxy configuration with env var fallback

* comments to en

* refactor(env): fallback env

---------

Co-authored-by: jyxjjj <773933146@qq.com>
2025-11-04 09:01:35 +08:00
ex-hentai 0857478516 feat(thunder): allow setting space (#1219)
allows access to files on remote devices via Thunder's tunneling service.
2025-11-03 10:53:38 +08:00
Seven 66d9809057 feat(strm): strm local file (#1127)
* feat(strm): strm local file

* feat: 代码优化

* feat: 访问被strm挂载路径时也更新

* fix: 路径最后带/判断缺失

* fix: 路径最后带/判断缺失

* refactor

* refactor

* fix: close seekable-stream in `generateStrm`

* refactor: lazy create local file

* 优化路径判断

---------

Co-authored-by: KirCute <kircute@foxmail.com>
2025-11-03 10:48:15 +08:00
MoYan db8a7e8caf feat(123): allow modification of the platform header (#1542)
* feat(drivers/123): Allow modification of the platform field

* feat(drivers/123): Set login platfrom as web

* fix(drivers/123): update platform field help value
2025-11-03 10:02:52 +08:00
walloo 8f18e34da0 fix(alias): nil panic in ResolveLinkCacheMode (#1527)
* fix(alias): Check the driver path during initialization

* fix(alias): Don't check the driver path during initialization anymore.
2025-10-23 22:18:56 +08:00
ShenLin 525f26dc23 feat(command): add --config flag to set custom config path (#1479) 2025-10-22 19:35:34 +08:00
NewbieOrange a0fcfa3ed2 fix(aliyundrive_open): use safe disk usage calculation (#1510) 2025-10-20 22:05:14 +08:00
ILoveScratch 15f276537c fix(share): remove share when user delete (#1493) 2025-10-19 22:48:21 +08:00
MadDogOwner 623a12050e feat(openlist): add PassIPToUpsteam to driver (#1498) 2025-10-19 22:45:40 +08:00
D@' 3z K!7 ae2d2d1021 feat(drivers): add ProtonDrive driver (#1368)
* feat(drivers): add ProtonDrive driver

- Implement complete ProtonDrive storage driver with end-to-end encryption support
- Add authentication via username/password with credential caching and reusable login
- Support all core operations: List, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include encrypted file operations with PGP key management and node passphrase handling
- Add temporary HTTP server for secure file downloads with range request support
- Support media streaming using temp server range requests
- Implement progress tracking for uploads and downloads
- Support directory operations with circular move detection
- Add proper error handling and panic recovery for external library integration
- Support buffered upload for specific sequential and encrypted, but optimized transmission.

* Update drivers/proton_drive/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>

* chore

* feat(drivers): enhance ProtonDrive temp server

- Implement separate listen and public port configuration for complex network deployments
- Add intelligent port detection with 8080 as preferred default, fallback to auto-assignment
- Support Container/NAT/VM environments through configurable external host and port mapping
- Add port availability validation with graceful fallback to listen port
- Enable users to specify external domain/IP for client connections (e.g., 192.168.1.5)
- Follow FTP server configuration patterns for network flexibility
- Maintain localhost development simplicity while supporting production deployments

* feat(proton_drive): refactor directory handling and improve link retrieval

* fix(proton_drive): add NoLinkURL configuration option

* fix(proton_drive): update file size retrieval and enforce TwoFACode requirement

* feat(proton_drive): add expiration to link response

* fix(proton_drive): handle empty RootFolderID in Init method

* fix(proton_drive): update credential handling to use email and reusable login

* fix(proton_drive): update credential handling to use reusableCredential variable

* fix(proton_drive): update DirectRename to use GetLink for source object retrieval

* fix(proton_drive): refactor uploadFile to return model.Obj and handle errors correctly

* fix(proton_drive): refactor DirectMove to use getLink for source retrieval and simplify destination handling

* fix(proton_drive): simplify Copy method by removing temporary file creation and directly using FileStream

* refactor(proton_drive): remove unused temporary server and related code

* chore

* fix(proton_drive): fix driver

- Handle fresh login if ProtonDrive rejects AccessToken or RefreshToken
- Update stored credentials

* fix(proton_drive): simplify reusable login handling in Init method

* fix(proton_drive): fix driver

- Update stored credentials, now is failing

* feat(proton_drive): improve authentication handling and remove unused variables

* fix(proton_drive): fix driver

- Update stored credentials, now is failing

* fix(proton_drive): improve authentication handling

* refactor(proton_drive): move client initialization to initClient method

* feat(proton_drive): move addrs and addrKRs

* feat(proton_drive): optimize upload threads

- Change ConcurrentBlockUploadCount to user configured upload threads number
- Comment ConcurrentFileCryptoCount, default is runtime.GOMAXPROCS(0)

---------

Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: KirCute <kircute@foxmail.com>
2025-10-19 12:07:11 +08:00
jerrita a109152a13 feat(onedrive): add ref support (#1435)
* feat(onedrive): add ref support

* fix(onedrive): remove redundant token assignment from reference

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-10-18 21:48:26 +08:00
ILoveScratch febbcd6027 feat(cache): improve cache management (#1339)
* feat(cache): improve cache management

* feat(disk-usage): add cache

* feat(disk-usage): add refresh

* fix(disk-usage): cache with ttl

* feat(cache): implement KeyedCache and TypedCache for improved caching mechanism

* fix(copy): update object retrieval to use Get instead of GetUnwrap

* refactor(cache): simplify DirectoryCache structure and improve object management

* fix(cache): correct cache entry initialization and key deletion logic in TypedCache

* refactor(driver): remove GetObjInfo interface and simplify Link function logic
https://github.com/OpenListTeam/OpenList/pull/888/files#r2430925783

* fix(link): optimize link retrieval and caching logic

* refactor(cache): consolidate cache management and improve directory cache handling

* fix(cache): add cache control based on storage configuration in List function

* .

* refactor: replace fmt.Sprintf with strconv for integer conversions

* refactor(cache): enhance cache entry management with Expirable interface

* fix(cache): improve link reference acquisition logic to handle expiration

* refactor: replace OnlyLinkMFile with NoLinkSF in driver configurations and logic

* refactor(link): enhance link caching logic with dynamic type keys based on IP and User-Agent

* feat(drivers): add LinkCacheType to driver configurations for enhanced caching

* refactor(cache): streamline directory object management in cache operations

* refactor(cache): remove unnecessary 'dirty' field from CacheEntry structure

* refactor(cache): replace 'dirty' field with bitwise flags

* refactor(io): 调高SyncClosers.AcquireReference的优先级

* refactor(link): 优化链接获取逻辑,增加重

* refactor(link): 添加RequireReference字段以增强链接管理

* refactor(link): 移除MFile字段,改用RangeReader

* refactor: 移除不必要的NoLinkSF字段

* refactor(cache): 修改目录缓存的脏标志定义和更新逻辑

* feat(cache): add expiration gc

---------

Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: KirCute <kircute@foxmail.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-10-18 21:47:18 +08:00
jenfonro 549e60136b fix(fs):fix retry task after restart (#1467)
* fix retry task after restart

* fix: initialize SrcStorage and DstStorage in tasks to prevent nil pointer dereference

* feat: implement storage load signal mechanism for improved synchronization

* fix:  update StoragesLoaded logic

* refactor: reorganize storage loading logic and improve synchronization handling

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-10-18 20:52:02 +08:00
Tursom K. Ulefits 14d2b8290a feat(local): move deleted files to corresponding locations (#1281) 2025-10-17 18:01:02 +08:00
ShenLin cdc069d8e7 feat(http3|quic): Set Alt-Svc header only for HTTPS requests (#1469) 2025-10-14 18:07:29 +08:00
zzzhr1990 fb5094f688 feat(drivers): add halalcloud_open driver (#1430)
* 新增清真云Open驱动,支持最新的轻量SDK

* Change Go version in go.mod

Downgrade Go version from 1.24.2 to 1.23.4

Signed-off-by: zzzhr1990 <zzzhr@hotmail.com>

* Apply suggestions from code review

* Removed unnecessary comments
* Downgraded the Go version to 1.23.4.
* Not sure whether FileStream supports concurrent read and write operations, so currently using single-threaded upload to ensure safety.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: zzzhr1990 <zzzhr@hotmail.com>

* feat(halalcloud_open): support disk usage

* Set useSingleUpload to true for upload safety

Not sure whether FileStream supports concurrent read and write operations, so currently using single-threaded upload to ensure safety.

Signed-off-by: zzzhr1990 <zzzhr@hotmail.com>

* Update meta.go

Change required for RefreshToken, If using a personal API approach, the RefreshToken is not required.

Signed-off-by: zzzhr1990 <zzzhr@hotmail.com>

* remove debug logs

* bump halalcloud SDK version

* fix unnecessary params

* Update drivers/halalcloud_open/driver_init.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: zzzhr1990 <zzzhr@hotmail.com>

* Fixed spelling errors; changed hardcoded retry parameters to constants.

* remove pointer in get link function in utils.go

---------

Signed-off-by: zzzhr1990 <zzzhr@hotmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: KirCute <951206789@qq.com>
2025-10-14 16:03:42 +08:00
KirCute 670e0bdc45 chore(frontend): optimize user operations (#1449) 2025-10-14 16:02:50 +08:00
ShenLin 89235012af feat(http3|quic): add http3|quic support (#1466)
* feat(http3|quic): add http3|quic support

* revert(ai): fix ai error

* fix(shutdown): shutdown was using close

* feat(http3|quic): add config if needs h3

* feat(http3|quic): add Alt-Svc to expose h3
2025-10-14 15:57:16 +08:00
KirCute 2bfbad2874 feat(offline_download): add 123 open (#1427) 2025-10-07 01:13:25 +08:00
KirCute 4ba7696032 feat(pikpak): support disk usage (#1426)
* feat(pikpak): support disk usage

* fix(alias): cannot list with details

* refactor: rename `NewDiskUsageFromUsedAndTotal`

* fix(disk-usage): get details of storages that is not initialized
2025-10-06 16:37:00 +08:00
KirCute 66645516e5 fix(ilanzou): wrong total capacity (#1433) 2025-10-06 16:35:10 +08:00
KirCute eb2ff2d2ca fix(189pc/189tv): request panic when login failed (#1428) 2025-10-05 22:44:50 +08:00
NewbieOrange 4153245f2c fix(drivers): free space underflow if used larger than total space (#1407) 2025-10-04 00:41:45 +08:00
KirCute 6fe9af7819 fix(alias): stabilize list root result (#1401) 2025-10-01 21:05:04 +08:00
j2rong4cn 2edc446ced feat(stream): support using temporary files as large buffer (#1399)
feat(stream): refactor StreamSectionReader to support using temporary files as large buffer
2025-10-01 18:43:20 +08:00
j2rong4cn c3c7983f7b perf(sync-closers): improve Close method (#1395)
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-01 18:06:10 +08:00
HG-ha 22deb4df30 fix(lanzou): auto handle acw_sc__v2 and down_ip=1 for all requests, improve secondary validation (#1394)
* Refactor upload and request handling in util.go

重构请求部分以避免偶现failed link: failed get link:的情况

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

* Format and clean up code in util.go

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

* go fmt util.go

---------

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>
2025-10-01 17:38:50 +08:00
j2rong4cn da0c734aa3 fix(net): unable to pass HttpStatusCode (#1397) 2025-10-01 00:16:36 +08:00
ILoveScratch 189cebe4c9 feat(drivers): add MediaFire driver support (#1322)
* feat(drivers): add MediaFire driver support (#9319)

- Implement complete MediaFire storage driver
- Add authentication via session_token and cookie
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload API with multi-unit transfers
- Add proper error handling and progress reporting

Co-authored-by: Da3zKi7 <da3zki7@duck.com>

* fix(mediafire): fix code errors in mediafire

* fix(mediafire): fix code errors in mediafire

* fix(drivers): add session renewal cron for MediaFire driver (#9321)

- Implement automatic session token renewal every 6-9 minutes
- Add validation for required SessionToken and Cookie fields in Init
- Handle session expiration by calling renewToken on validation failure
- Prevent storage failures due to MediaFire session timeouts

Fixes session closure issues that occur after server restarts or extended periods.

Co-authored-by: Da3zKi7 <da3zki7@duck.com>

* docs: restore README changes

Signed-off-by: ILoveScratch <ilovescratch@foxmail.com>

* fix

* fix

* fix: add stream upload limit

* fix

* fix: clear action token on drop and refactor header setting

* feat(drivers/mediafire): optimize file caching - support direct stream processing

- Remove forced caching to *os.File type
- Support generic model.File interface for better flexibility
- Improve upload efficiency by avoiding unnecessary file conversions
- Fix return type to use model.Object instead of model.ObjThumb

* feat(drivers/mediafire): improve global rate limiting

- Ensure all API methods properly use context for rate limiting
- Fix context parameter usage in getDirectDownloadLink, getActionToken, getFileByHash
- Maintain consistent rate limiting across all MediaFire API calls

* feat(drivers/mediafire): unify return types - remove unnecessary ObjThumb

- Change MakeDir, Rename, Copy methods to return model.Object instead of model.ObjThumb
- Remove empty Thumbnail fields where not meaningful
- Keep ObjThumb only for fileToObj (List operations) which provides actual thumbnail URLs
- Improve code consistency and reduce unnecessary wrapper objects

* refactor(drivers/mediafire): extract common error handling logic

- Add checkAPIResult helper function to reduce code duplication
- Replace repetitive MediaFire API error checks with centralized function
- Maintain specific error messages for unique cases (token, upload, search)
- Improve code maintainability and consistency

* enhance(drivers/mediafire): improve quick upload implementation

- Add null check for existingFile to prevent potential issues
- Improve error handling in quick upload - continue normal upload if search fails
- Add detailed comments explaining quick upload logic
- Optimize getExistingFileInfo with clearer fallback strategy
- Ensure upload reliability even when file search encounters issues

* refactor(drivers/mediafire): optimize request method reusability

- Extract common HTTP request logic into apiRequest method
- Reduce code duplication between getForm and postForm methods
- Maintain backward compatibility with existing method signatures
- Centralize rate limiting and header management
- Support extensible HTTP method handling

* docs(drivers/mediafire): add comprehensive English comments

- Add function-level comments for all major driver methods
- Document Init, List, Link, MakeDir, Move, Rename, Copy, Remove, Put methods
- Add comments for key utility functions including session token management
- Improve code readability and maintainability for community collaboration
- Follow Go documentation conventions with clear, concise descriptions

* perf(mediafire): optimize memory allocation and type assertion performance

- Pre-allocate slice capacity in getFiles and bitmap conversion to reduce reallocations
- Cache file type check in uploadUnits to avoid repeated type assertions
- Add uploadSingleUnitOptimized for os.File to eliminate redundant type checks
- Optimize string to int conversion with proper error handling
- Improve memory efficiency in file upload operations

* fix(mediafire): upload without cache

* feat(mediafire): add rate limiting to all API methods

- Add WaitLimit(ctx) calls to all driver methods: List, Link, MakeDir, Move, Rename, Copy, Remove, Put
- Ensure consistent rate limiting across all MediaFire API interactions
- Follow project standard pattern used by other drivers

* feat(mediafire): improve error handling consistency

- Add context parameter to all HTTP API functions for proper context propagation
- Update getForm, postForm and apiRequest to accept context parameter
- Fix rate limiting to use caller context instead of background context
- Ensure consistent error handling patterns across all API calls
- Improve cancellation and timeout support

* feat(mediafire): refactor resumableUpload to use io.ReadSeeker and improve upload handling

* fix(mediafire): release section reader

* feat: add disk usage

* feat(drivers/mediafire): support concurrent upload (#1387)

* feat(drivers): add MediaFire driver with concurrent upload support

- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

* feat(stream): add DiscardSection method to StreamSectionReader for skipping data

* feat(mediafire): refactor resumableUpload logic for improved upload handling and error management

* fix(mediafire): stop cron job and clear action token in Drop method

* .

* fix(mediafire): optimize buffer sizing logic in uploadUnits method

* fix(docs): remove duplicate MediaFire

* fix(mediafire): revert 'optimization', large files should not be fully chached.

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Da3zKi7 <da3zki7@duck.com>
Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* fix(mediafire): optimize hash calculation in uploadUnits function

* feat(drivers/mediafire): support concurrent upload  (#1366)

* feat(drivers): add MediaFire driver with concurrent upload support

- Implement complete MediaFire storage driver with session token authentication
- Support all core operations: List, Get, Link, Put, Copy, Move, Remove, Rename, MakeDir
- Include thumbnail generation for media files
- Handle MediaFire's resumable upload with intelligent and multi-unit transfers
- Support concurrent chunk uploads using errgroup.NewOrderedGroupWithContext, using splitted file caching for large files
- Optimize memory usage with adaptive buffer sizing (10MB-100MB (default))
- Include rate limiting and retry logic for API requests
- Add proper error handling and progress reporting
- Handle MediaFire's bitmap-based resumable upload protocol

Closes PR #1322

* feat(stream): add DiscardSection method to StreamSectionReader for skipping data

* feat(mediafire): refactor resumableUpload logic for improved upload handling and error management

* fix(mediafire): stop cron job and clear action token in Drop method

* .

* fix(mediafire): optimize buffer sizing logic in uploadUnits method

* fix(docs): remove duplicate MediaFire

* fix(mediafire): revert 'optimization', large files should not be fully chached.

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

---------

Signed-off-by: ILoveScratch <ilovescratch@foxmail.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Signed-off-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: D@' 3z K!7 <99719341+Da3zKi7@users.noreply.github.com>
Co-authored-by: Da3zKi7 <da3zki7@duck.com>
Co-authored-by: KirCute <951206789@qq.com>
Co-authored-by: Suyunmeng <Susus0175@proton.me>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
2025-09-30 21:55:41 +08:00
j2rong4cn 9d3da44a99 fix(chunk): ensure NumListWorkers is set during Chunk initialization (#1396) 2025-09-30 21:39:19 +08:00
j2rong4cn 8f17d35ed5 feat(chunk): add ChunkPrefix and ChunkLargeFileOnly options (#1321)
* fix(chunk): move chunk existence check to Link method

* feat(chunk): add chunk prefix configuration

* feat(chunk): add chunk_large_file_only configuration

* feat(chunk): concurrently list chunk folder

* refactor(chunk): remove unnecessary mutex for result handling in List method

---------

Co-authored-by: KirCute <kircute@foxmail.com>
2025-09-30 21:26:06 +08:00
HG-ha 89759b6e3b fix(lanzou): support acw_sc__v2 and secondary validation for download link (#1379)
* 重定向链接添加acw_sc__v2验证

在最新的蓝奏云解析中,最后重定向获取真实地址时也需要添加acw_sc__v2信息

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

* 重定向链接添加acw_sc__v2验证

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

* 重定向真实下载链接添加acw_sc__v2验证

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

* fix: CalcAcwScV2

* Add error handling for response body read

Handle error when reading response body.

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

* Improve response handling and validation logic

优化重定向资源管理,添加二次人机验证acw_sc__v2处理

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>

---------

Signed-off-by: HG-ha <60115106+HG-ha@users.noreply.github.com>
Co-authored-by: foxxorcat <foxxorcat@foxmail.com>
2025-09-30 17:07:30 +08:00
KirCute a2fc38be8d perf(disk-usage): concurrently get details (#1326) 2025-09-29 23:26:56 +08:00
KirCute e0414e7110 fix(offline_download/http): attach UA to the request (#1347)
fix(simple-http): attach UA to the request
2025-09-29 22:13:00 +08:00
KirCute b486af0031 feat(sftp-server): support disable password login (#1357) 2025-09-29 21:46:55 +08:00
j2rong4cn ea09ce4b8f fix(fs): improve error handling in op.Get (#1323) 2025-09-29 21:38:34 +08:00
KirCute d465da43e3 fix(ftp-server): cannot get obj in uploading state inconsistency window (#1293)
* fix(ftp-server): cannot get obj in uploading state inconsistency window

* fix

* fix: duplicate obj when upload completed but client access does not

* fix

* feat: support stat remove and move
2025-09-23 01:00:18 +08:00
Dgs 84ed487950 feat(quark_uc_tv): add order by and direction options for file listing (#1325) 2025-09-21 20:43:32 +08:00
ILoveScratch 3c07144211 chore(announcement): change default announcement text (#1319)
* chore(announcement): change default announcement text for better clarity  

- Improve site announcement style for a more polished look
- Adjust default value to avoid unclear meaning with only repo or single link
2025-09-20 23:06:40 +08:00
KirCute 3936e736e6 feat(drivers): add a driver that divides large files into multiple chunks (#1153) 2025-09-19 19:27:35 +08:00
KirCute 68433d4f5b fix(local): cannot mkdir on specific platforms (#1304) 2025-09-19 15:34:58 +08:00
KirCute cc16cb35bf feat(style): add driver icons and disk usage (#1274)
* feat(style): add driver icons and disk usage

* feat(driver): add disk usage for 115_open, 123_open, aliyundrive_open and baidu_netdisk

* feat(driver): add disk usage for crypt, sftp and smb

* chore: clean unused variable

* feat(driver): add disk usage for cloudreve_v4

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(local): disk label check when getting disk usage

* feat(style): return details when accessing the manage page

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-09-19 11:59:11 +08:00
Yinan Qin d3bc6321f4 chore(build): update Go version to 1.25.0 across workflows and build scripts (#1290)
build: update Go version to 1.25.0 across workflows and build scripts

fixes #1286
2025-09-16 18:44:29 +08:00
TwoOnefour cbbb5ad231 fix(stream): http chucked upload issue (#1152)
* fix(stream): http chucked upload issue

* fix(stream): use MmapThreshold

* fix(stream): improve caching mechanism and handle size=0 case

* fix bug

* fix(buffer): optimize ReadAt method for improved performance

* fix(upload): handle Content-Length and File-Size headers for better size management

* fix(189pc): 移除重复限速

* fix(upload): handle negative file size during streaming uploads

* fix(upload): update header key from File-Size to X-File-Size for size retrieval

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-09-15 19:36:16 +08:00
hshpy c1d03c5bcc fix(security): zip slip (#1228)
* fix(security): Zip Slip

* chore:remove repeat clean

* fix: archives,iso9660 and rardecode module

---------

Co-authored-by: ILoveScratch <ilovescratch@foxmail.com>
2025-09-15 13:25:21 +08:00
KirCute 61a8ed515f fix(123): add get and list hash info (#1278) 2025-09-14 21:36:54 +08:00
KirCute bbb7c06504 feat(alias): support pass through provider (#1269) 2025-09-14 21:36:38 +08:00
ILoveScratch 8bbdb272d4 docs(readme): extend driver list with newest support (#1271) 2025-09-13 20:41:17 +08:00
foxxorcat c15ae94307 feat(189PC,189TV): add refreshToken and qrcode login (#1205)
### Key Changes
- **189PC**: Add QR code login and refresh token support
- **189TV**: Add session refresh mechanism and fix TempUuid persistence issue
- **Both**: Implement session keep-alive with cron jobs (5min interval)

### Features
- QR code authentication for 189PC as alternative to password login
- Automatic token refresh to avoid frequent re-authentication
- Session keep-alive to maintain long-term connections
- Retry logic with max attempts to prevent infinite loops

### Fixes
- Fixed 189TV TempUuid causing storage corruption on QR code reload
- Enhanced error handling for token expiration scenarios
2025-09-13 13:59:47 +08:00
MadDogOwner f1a5048558 feat(drivers): add cnb_releases (#1033)
* feat(drivers): add cnb_releases

* feat(cnb_release): implement reference

* refactor(cnb_releases): get release info by ID instead of tag name

* feat(cnb_releases): add option to use tag name instead of release name

* fix(cnb_releases): set default root and improve release info retrieval

* feat(cnb_releases): implement Put

* perf(cnb_release): use io.Pipe to stream file upload

* perf(cnb_releases): add context timeout for file upload request

* feat(cnb_releases): implement Remove

* feat(cnb_releases): implement MakeDir

* feat(cnb_releases): implement Rename

* feat(cnb_releases): require repo and token in Addition

* chore(cnb_releases): remove unused code

* Revert 'perf(cnb_release): use io.Pipe to stream file upload'

* perf(cnb_releases): optimize upload with MultiReader

* feat(cnb_releases): add DefaultBranch

---------

Co-authored-by: ILoveScratch <ilovescratch@foxmail.com>
2025-09-11 18:11:32 +08:00
Tursom K. Ulefits 1fe26bff9a feat(local): auto create recycle dir if not exists (#1244) 2025-09-10 20:57:21 +08:00
Suyunjing 433dcd156b fix(ci): add tag_name to upload assets step (#1234)
fix(release): add tag_name to upload assets step
2025-09-06 22:51:05 +08:00
MadDogOwner e97f0a289e feat(cloudreve_v4): enhance token management (#1171)
* fix(cloudreve_v4): improve error handling in request method

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* feat(cloudreve_v4): enhance token management with expiration checks and refresh logic

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* feat(cloudreve_v4): add JWT structures for access and refresh tokens; validate access token on initialization

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(cloudreve_v4): improve error messages

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-09-04 19:41:41 +08:00
walloo 89f35170b3 fix(fs): clear cache after directory rename to ensure consistency (#1193)
Clear cache after renaming the directory.
2025-09-01 18:47:54 +08:00
MadDogOwner 8188fb2d7d fix(123open): get direct link (#1185)
* fix(123open): correct query parameter name from 'fileId' to 'fileID' in getDirectLink function

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(123open): change SpaceTempExpr type from 'string' to 'int64' in UserInfoResp struct

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(123open): comment out unused fields in UserInfoResp struct

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(123open): add getUID method and cache UID

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-31 15:47:38 +08:00
MadDogOwner 87cf95f50b fix(139): refactor part upload logic (#1184)
* fix(139): refactor part upload logic

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(139): handle upload errors

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(139): sort upload parts by PartNumber before uploading

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(139): improve error handling

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(139): add validation for upload part index to prevent out of bounds errors

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-31 15:47:12 +08:00
MadDogOwner 8ab26cb823 fix(123open): change DirectLink type from 'boolean' to 'bool' (#1180)
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-29 19:06:37 +08:00
j2rong4cn 5880c8e1af fix(189tv): use rate-limited upload stream in OldUpload function (#1176)
* fix(189tv): use rate-limited upload stream in OldUpload function

* fix(189tv): wrap tempFile with io.NopCloser to prevent premature closure in OldUpload function

* .
2025-08-29 16:01:50 +08:00
KirCute 14bf4ecb4c fix(share): support custom proxy url (#1130)
* feat(share): support custom proxy url

* fix(share): count access

* fix: maybe a path traversal vulnerability?
2025-08-28 22:11:19 +08:00
hshpy 04a5e58781 fix(server): can't edit .md source files (#1159)
* fix(server): can't edit .md source files

* chore

* add ignore direct link args
2025-08-28 16:19:57 +08:00
MadDogOwner bbd4389345 fix(wopan): use fixed timezone for parsing time (#1170)
fix(wopan): update getTime function to use fixed timezone for parsing

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-28 13:02:02 +08:00
MadDogOwner f350ccdf95 fix(189pc): sliceSize must not be equal to fileSize (#1169)
* fix(189pc): sliceSize not equal to fileSize

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* Update comment for sliceSize parameter

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-28 11:32:40 +08:00
Caspian 4f2de9395e feat(degoo): token improvement (#1149)
* Update driver.go

Signed-off-by: Caspian <app@caspian.im>

* Update meta.go

Signed-off-by: Caspian <app@caspian.im>

* Update util.go

Signed-off-by: Caspian <app@caspian.im>

* Update util.go

Signed-off-by: Caspian <app@caspian.im>

* Update util.go

Signed-off-by: Caspian <app@caspian.im>

* Update util.go

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* make account optional

* ensure username and password

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: Caspian <app@caspian.im>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-26 01:22:59 +08:00
TwoOnefour b0dbbebfb0 feat(drivers): add Teldrive driver (#1116)
https://github.com/tgdrive/teldrive

https://teldrive-docs.pages.dev/docs/api

实现:
* copy
* move
* link (302 share and local proxy)
* chunked uploads
* rename

未实现:
- openlist扫码登陆
- refresh token

https://github.com/OpenListTeam/OpenList-Docs/pull/155


* feat(Teldrive): Add driver Teldrive

* fix(teldrive): force webproxy and memory optimized

* chore(teldrive): go fmt

* chore(teldrive): remove TODO

* chore(teldrive): organize code

* feat(teldrive): add UseShareLink option and support 302

* fix(teldrive): standardize API path construction

* fix(teldrive): trim trailing slash from Address in Init method

* chore(teldrive): update help text for UseShareLink field in Addition struct

* fix(teldrive): set 10 MiB as default chunk size

---------

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: ILoveScratch <ilovescratch@foxmail.com>
2025-08-25 01:34:08 +08:00
MadDogOwner 0c27b4bd47 docs(contributing): update guidelines (#983)
[skip ci]

* docs(contributing): update guidelines

* docs(contributing): clarify fork

* docs(contributing): sync translation

Co-authored-by: Yinan Qin <39023210+elysia-best@users.noreply.github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* docs(contributing): add label and cc reminder

* docs(contributing): remove ensure new branch from checklist

* docs(contributing): replace generic GitHub URLs with user-specific ones

* docs(contributing): make branch deletion after PR merge optional

* docs(contributing): keep --recurse-submodules

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Yinan Qin <39023210+elysia-best@users.noreply.github.com>
2025-08-24 20:13:11 +08:00
power721 736cd9e5f2 fix(quark): fix getTranscodingLink (#1136)
The first video info may not contain url

* fix(quark): fix getTranscodingLink

* fix(quark_tv): fix getTranscodingLink
2025-08-24 19:55:10 +08:00
power721 c7a603c926 fix(115): fix get 115 app version (#1137) 2025-08-24 19:50:21 +08:00
Edward a28d6d5693 fix(123_open): fix token refresh (#1121) 2025-08-23 23:01:41 +08:00
Caspian e59d2233e2 feat(drivers): add Degoo driver (#1097)
* Create driver.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Create util.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Create types.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Create meta.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update drivers/degoo/driver.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: CaspianGUAN <app@caspian.im>

* Update drivers/degoo/driver.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: CaspianGUAN <app@caspian.im>

* Update driver.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update meta.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update types.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update util.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update driver.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update util.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* Update drivers/degoo/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: CaspianGUAN <app@caspian.im>

* Update util.go

Signed-off-by: CaspianGUAN <app@caspian.im>

* refactor(degoo): add Degoo driver integration and update API handling

* fix(degoo): apply suggestions

---------

Signed-off-by: CaspianGUAN <app@caspian.im>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-23 22:47:02 +08:00
我怎么就不是一只猫呢? 01914a06ef refactor(ci): add permissions check at docker's entrypoint (#1128)
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-22 19:35:48 +08:00
Shelton Zhu 6499374d1c fix(deps): update 115driver to v1.1.1 (close SheltonZhu/115driver#57) (#1115) 2025-08-20 21:33:21 +08:00
MadDogOwner b054919d5c feat(ilanzou): add support for rapid upload and fix duplication handling (#1065)
* feat(ilanzou): add support for rapid upload token handling

* feat(ilanzou): add NoOverwriteUpload option
2025-08-19 19:19:44 +08:00
KirCute 048ee9c2e5 feat(server): adapting #1099 to #991 (#1102) 2025-08-19 15:48:59 +08:00
MadDogOwner 23394548ca feat(123_open): add DirectLink option (#1045)
* feat(123_open): add `UseDirectLink` option

* feat(123_open): update rate limit rules

* fix(123_open): update api

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* feat(123_open): enhance direct link functionality with private key and expiration

* refactor(123_open): use UUID for random generation

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-19 15:23:10 +08:00
MadDogOwner b04677b806 feat(server): add error page and status code (#1099) 2025-08-19 15:18:12 +08:00
KirCute e4c902dd93 feat(share): support more secure file sharing (#991)
提供一种类似大多数网盘的文件分享操作,这种分享方式可以通过强制 Web 代理隐藏文件源路径,可以设置分享码、最大访问数和过期时间,并且不需要启用 guest 用户。

在全局设置中可以调整:
- 是否强制 Web 代理
- 是否允许预览
- 是否允许预览压缩文件
- 分享文件后,点击“复制链接”按钮复制的内容

前端部分:OpenListTeam/OpenList-Frontend#156
文档部分:OpenListTeam/OpenList-Docs#130

Close #183
Close #526
Close #860
Close #892
Close #1079


* feat(share): support more secure file sharing

* feat(share): add archive preview

* fix(share): fix some bugs

* feat(openlist_share): add openlist share driver

* fix(share): lack unwrap when get virtual path

* fix: use unwrapPath instead of path for virtual file name comparison

* fix(share): change request method of /api/share/list from GET to Any

* fix(share): path traversal vulnerability in sharing path check

* 修复分享alias驱动的文件 没开代理时无法获取URL

* fix(sharing): update error message for sharing root link extraction

---------

Co-authored-by: Suyunmeng <69945917+Suyunmeng@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
2025-08-19 15:10:02 +08:00
MadDogOwner 5d8bd258c0 refactor(docker): reduce docker image size (#1091)
* fix(docker): reduce image size

* refactor(docker): update user and group creation

* Update Dockerfile

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-19 10:27:33 +08:00
huancun _- 08c5283c8c feat(docker): Update docker-compose configuration (#1081)
* feat(docker): Update docker-compose configuration

* Update docker-compose.yml

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
Signed-off-by: huancun _- <huancun@hc26.org>

---------

Signed-off-by: huancun _- <huancun@hc26.org>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-18 14:29:59 +08:00
我怎么就不是一只猫呢? 10a14f10cd fix(docker): improve startup process and SIGTERM handling (#1089)
* fix(ci): Modify the way of star OpenList.

* fix(ci): start runsvdir
2025-08-18 11:13:05 +08:00
j2rong4cn f86ebc52a0 refactor(123_open): improve upload (#1076)
* refactor(123_open): improve upload

* optimize buffer initialization for multipart form

* 每次重试生成新的表单

* .
2025-08-17 14:25:23 +08:00
j2rong4cn 016ed90efa feat(stream): fast buffer freeing for large cache (#1053)
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-16 17:19:52 +08:00
tuthanika d76407b201 fix(dropbox): incorrect path error during upload (#1052)
* Fix incorrect path error during upload on Dropbox

* Add RootNamespaceId to the config for direct modification

* Refactor Dropbox header logic: extract JSON marshaling into helper method

* Fix Dropbox: replace marshalToJSONString with utils.Json.MarshalToString
2025-08-16 14:18:02 +08:00
yuyamionini 5de6b660f2 fix(terabox): user not exists error (#1056)
* fix user location error when upload file
2025-08-15 21:25:57 +08:00
Suyunjing 71ada3b656 fix(ci-sync): fix workflow for syncing Repository (#1062) 2025-08-15 18:48:55 +08:00
Suyunjing dc42f0e226 [skip ci]fix(ci): update sync workflow (#1061) 2025-08-15 18:36:52 +08:00
Suyunjing 74bf9f6467 [skip ci]feat(sync): add workflow to sync GitHub repository (#1060)
feat(sync): add workflow to sync GitHub repository to Gitee
2025-08-15 18:12:29 +08:00
我怎么就不是一只猫呢? d0c22a1ecb feat(ci): add the default user for docker image (#1036)
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-12 09:51:40 +08:00
j2rong4cn 57fceabcf4 perf(stream): improve file stream range reading and caching mechanism (#1001)
* perf(stream): improve file stream range reading and caching mechanism

* 。

* add bytes_test.go

* fix(stream): handle EOF and buffer reading more gracefully

* 注释

* refactor: update CacheFullAndWriter to accept pointer for UpdateProgress

* update tests

* Update drivers/google_drive/util.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>

* 更优雅的克隆Link

* 修复stream已缓存但无法重复读取

* 将Bytes类型重命名为Reader

* 修复栈溢出

* update tests

---------

Signed-off-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-11 23:41:22 +08:00
MadDogOwner 8c244a984d refactor(assets): migrate to resource domain (#975)
* refactor(assets): migrate to resource domain

* feat(bootstrap): add migration value for logo and favicon settings
2025-08-10 09:57:33 +08:00
火星大王 df479ba806 fix(aliyundrive_open): limit rate for every request (close #724) (#1011)
* fix(aliyundrive_open): limit rate for `Remove` and `MakeDir`; reduce limit for `List` and `Link` (close #724)

* Update drivers/aliyundrive_open/driver.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: 火星大王 <34576789+huoxingdawang@users.noreply.github.com>

* Update drivers/aliyundrive_open/driver.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: 火星大王 <34576789+huoxingdawang@users.noreply.github.com>

* fix(aliyundrive_open): limit rate for every request

* fix(aliyundrive_open): fix limiter not work on reference driver

* fix(aliyundrive_open): typo

* fix(aliyundrive_open): limiter not set to nil after free

* fix(aliyundrive_share): limit rate for every request

---------

Signed-off-by: 火星大王 <34576789+huoxingdawang@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-10 09:55:20 +08:00
MadDogOwner 5ae8e96237 feat(123_open): update Put method to return model.Obj (#1008)
* feat(123_open): update Put method to return model.Obj

* fix(123_open): declear time zones

* chore(123_open): fix typo

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(123_open): use fixed timezone

* fix(123_open): implement PutResult interface for Open123 driver

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Suyunmeng <69945917+Suyunmeng@users.noreply.github.com>
2025-08-09 15:09:12 +08:00
Suyunjing aa0ced47b0 fix(webdav): Handle HEAD requests for directories with appropriate headers (#1015)
Implement handling of HEAD requests for directories by setting the correct Content-Type and Content-Length headers.
2025-08-09 13:57:09 +08:00
Suyunjing ab747d9052 feat(config): Add PWA manifest.json endpoint for web app installation (#990)
* feat(config): Add PWA manifest.json endpoint for web app installation

* fix: Update comment to English in manifest handler

* fix: fix EOL

* fix: Remove unused fmt import from manifest handler

* feat: use site settings for manifest name and icon

* fix(manifest): Move manifest.json route to static handler for proper CDN handling

* feat: move manifest.json handler to static package and improve path handling

* feat: Add custom static file handler to prevent manifest.json conflicts

* fix: Integrate manifest.json handling into static file serving routes

* fix: Simplify PWA manifest scope handling and static file serving

- Remove CDN-specific logic for PWA manifest scope and start_url
- Always use base path for PWA scope regardless of CDN configuration
- Replace manual file serving logic with http.FileServer for static assets

* fix: Ensure consistent base path handling in site configuration and manifest path construction

* fix: Refactor trailing slash handling in site configuration

* feat(static): update manifest path handling and add route for manifest.json
2025-08-08 20:07:51 +08:00
LmanTW 93c06213d4 feat(local): add directory size support (#624)
* feat(local): add directory size support

* fix(local): fix and improve directory size calculation

* style(local): fix code style

* style(local): fix code style

* style(local): fix code style

* fix(local): refresh directory size when force refresh

Signed-off-by: 我怎么就不是一只猫呢? <26274059+dezhishen@users.noreply.github.com>

* fix:(local): Avoid traversing the parent's parent, which leads to an endless loop

Signed-off-by: 我怎么就不是一只猫呢? <26274059+dezhishen@users.noreply.github.com>

* fix(local:) refresh dir size only enabled

Signed-off-by: 我怎么就不是一只猫呢? <26274059+dezhishen@users.noreply.github.com>

* fix(local): logical error && add RecalculateDirSize && cleaner code for int64

* feat(local): add Benchmark for CalculateDirSize

* refactor(local): 优化移动中对于错误的判断。

---------

Signed-off-by: 我怎么就不是一只猫呢? <26274059+dezhishen@users.noreply.github.com>
Co-authored-by: 我怎么就不是一只猫呢? <26274059+dezhishen@users.noreply.github.com>
2025-08-08 16:59:16 +08:00
Suyunjing b9b8eed285 [skip ci]feat(ci): add FRONTEND_REPO variable to workflows and build script (#1006) 2025-08-08 16:36:22 +08:00
j2rong4cn 317d190b77 fix(ftp): create a new connection for each download (#989) 2025-08-06 20:35:01 +08:00
Sakana 52d7d819ad feat(lenovonas_share): add thumb (#986) 2025-08-06 17:34:43 +08:00
RedSTAR.Cheng 0483e0f868 feat(driver_strm): also shown some files with strm (#969)
* feat(driver_strm): Also shown some files with strm

Allow user set some file types that need to shown with strm, usually subtitles

Most of code was copy and managed from drivers/alias

* 优化

* 优化

* 。

* 添加注释

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
2025-08-06 15:40:48 +08:00
MadDogOwner 08dae4f55f feat(123_open): update upload api v2 (#976) 2025-08-06 15:27:13 +08:00
j2rong4cn 9ac0484bc0 perf(ftp): improve concurrent Link response; fix alias/local driver issues (#974) 2025-08-06 13:32:37 +08:00
j2rong4cn 8cf15183a0 perf: optimize upload (#554)
* pref(115,123): optimize upload

* chore

* aliyun_open, google_drive

* fix bug

* chore

* cloudreve, cloudreve_v4, onedrive, onedrive_app

* chore(conf): add `max_buffer_limit` option

* 123pan multithread upload

* doubao

* google_drive

* chore

* chore

* chore: 计算分片数量的代码

* MaxBufferLimit自动挡

* MaxBufferLimit自动挡

* 189pc

* errorgroup添加Lifecycle

* 查缺补漏

* Conf.MaxBufferLimit单位为MB

* 。

---------

Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2025-08-05 21:42:54 +08:00
MadDogOwner c8f2aaaa55 feat(cmd): add delete command for storage (#952) 2025-08-04 17:30:43 +08:00
j2rong4cn 1208bd0a83 fix(fs): nil interface not equal to nil (#971)
https://go.dev/doc/faq#nil_error
2025-08-03 23:51:11 +08:00
j2rong4cn 6b096bcad4 fix(fs): deadlock when get link error (#963) 2025-08-02 17:49:53 +08:00
Seven 58dbf088f9 fix(fs): forget cache when get link error (#956) 2025-08-02 11:03:34 +08:00
Seven 05ff7908f2 fix(strm): encoded path is ineffective (#951) 2025-08-02 00:23:18 +08:00
Suyunjing a703b736c9 feat(offline_download): filter empty URLs in offline download requests (#948) 2025-08-01 16:12:21 +08:00
MadDogOwner e458f2ab53 fix(bootstrap): add newline after initial admin password output (#943)
fix(bootstrap): add newline after initial admin  password output
2025-08-01 13:43:41 +08:00
445 changed files with 30442 additions and 5764 deletions
+11 -8
View File
@@ -13,7 +13,7 @@ body:
attributes:
label: 请确认以下事项
description: |
您必须勾选以下内容,否则您的问题可能会被直接关闭。
您必须阅读、检查、确认、同意以下内容,否则您的问题一定会被直接关闭。
或者您可以去[讨论区](https://github.com/OpenListTeam/OpenList/discussions)。
options:
- label: |
@@ -34,7 +34,8 @@ body:
我认为此问题必须由`OpenList`处理,而非第三方。
- label: |
我已确认这个问题在最新版本中没有被修复。
- label: |
我没有阅读这个清单,只是闭眼选中了所有的复选框,请关闭这个 Issue 。
- type: input
id: version
attributes:
@@ -59,6 +60,14 @@ body:
label: 问题描述(必填)
validations:
required: true
- type: textarea
id: logs
attributes:
label: 日志(必填)
description: |
请复制粘贴错误日志,或者截图。(可隐藏隐私字段) [查看方法](https://doc.oplist.org/faq/howto#%E5%A6%82%E4%BD%95%E5%BF%AB%E9%80%9F%E5%AE%9A%E4%BD%8Dbug)
validations:
required: true
- type: textarea
id: config
attributes:
@@ -67,12 +76,6 @@ body:
请提供您的`OpenList`应用的配置文件,并截图相关存储配置。(可隐藏隐私字段)
validations:
required: true
- type: textarea
id: logs
attributes:
label: 日志(可选)
description: |
请复制粘贴错误日志,或者截图。(可隐藏隐私字段) [查看方法](https://doc.oplist.org/faq/howto#%E5%A6%82%E4%BD%95%E5%BF%AB%E9%80%9F%E5%AE%9A%E4%BD%8Dbug)
- type: textarea
id: reproduction
attributes:
+11 -8
View File
@@ -13,7 +13,7 @@ body:
attributes:
label: Please confirm the following
description: |
You must check all the following, otherwise your issue may be closed directly.
You must read, check, confirm, and agree to all the following, otherwise your issue will definitely be closed directly.
Or you can go to the [discussions](https://github.com/OpenListTeam/OpenList/discussions).
options:
- label: |
@@ -34,7 +34,8 @@ body:
I believe this issue must be handled by `OpenList` and not by a third party.
- label: |
I confirm this issue is not fixed in the latest version.
- label: |
I have not read these checkboxes and therefore I just ticked them all, Please close this issue.
- type: input
id: version
attributes:
@@ -59,6 +60,14 @@ body:
label: Bug Description (required)
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs (required)
description: |
Please copy and paste any relevant log output or screenshots. (You may mask sensitive fields) [Guide](https://doc.oplist.org/faq/howto#how-to-quickly-locate-bugs)
validations:
required: true
- type: textarea
id: config
attributes:
@@ -67,12 +76,6 @@ body:
Please provide your `OpenList` application's configuration file and a screenshot of the relevant storage configuration. (You may mask sensitive fields)
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs (optional)
description: |
Please copy and paste any relevant log output or screenshots. (You may mask sensitive fields) [Guide](https://doc.oplist.org/faq/howto#how-to-quickly-locate-bugs)
- type: textarea
id: reproduction
attributes:
@@ -7,7 +7,7 @@ body:
attributes:
label: 请确认以下事项
description: |
您必须勾选以下内容,否则您的问题可能会被直接关闭。
您必须阅读、检查、确认、同意以下内容,否则您的问题可能会被直接关闭。
或者您可以去[讨论区](https://github.com/OpenListTeam/OpenList/discussions)。
options:
- label: |
@@ -28,6 +28,8 @@ body:
我已确认此功能尚未被实现。
- label: |
我已确认此功能是合理的,且有普遍需求,并非我个人需要。
- label: |
我没有阅读这个清单,只是闭眼选中了所有的复选框,请关闭这个 Issue 。
- type: textarea
id: feature-description
attributes:
@@ -1,13 +1,13 @@
name: "Feature Request"
description: Feature Request / Enhancement
title: "[Feature] Please change the title to your feature name"
title: "[Feature] Please modify the title to your feature name"
labels: [enhancement]
body:
- type: checkboxes
attributes:
label: Please confirm the following
description: |
You must check all the following, otherwise your request may be closed directly.
You must read, check, confirm, and agree to all the following, otherwise your request may be closed directly.
Or you can go to the [discussions](https://github.com/OpenListTeam/OpenList/discussions).
options:
- label: |
@@ -28,6 +28,8 @@ body:
I confirm this feature has not been implemented yet.
- label: |
I confirm this feature is reasonable and has general demand, not just my personal need.
- label: |
I have not read these checkboxes and therefore I just ticked them all, Please close this issue.
- type: textarea
id: feature-description
attributes:
+58
View File
@@ -0,0 +1,58 @@
<!--
Provide a general summary of your changes in the Title above.
The PR title must start with `feat(): `, `docs(): `, `fix(): `, `style(): `, or `refactor(): `, `chore(): `. For example: `feat(component): add new feature`.
If it spans multiple components, use the main component as the prefix and enumerate in the title, describe in the body.
For breaking changes, add `!` after the type, e.g., `feat(component)!: breaking change`.
-->
<!--
在上方标题中提供您更改的总体摘要。
PR 标题需以 `feat(): `, `docs(): `, `fix(): `, `style(): `, `refactor(): `, `chore(): ` 其中之一开头,例如:`feat(component): 新增功能`。
如果跨多个组件,请使用主要组件作为前缀,并在标题中枚举、描述中说明。
如果是破坏性变更,请在类型后添加 `!`,例如 `feat(component)!: 破坏性变更`。
-->
## Description / 描述
<!-- Describe your changes in detail -->
<!-- 详细描述您的更改 -->
## Motivation and Context / 背景
<!-- Why is this change required? What problem does it solve? -->
<!-- 为什么需要此更改?它解决了什么问题? -->
<!-- If it fixes an open issue, please link to the issue here. -->
<!-- 如果修复了一个打开的issue,请在此处链接到该issue -->
Closes #XXXX
<!-- or -->
<!-- 或者 -->
Relates to #XXXX
## How Has This Been Tested? / 测试
<!-- Please describe in detail how you tested your changes. -->
<!-- 请详细描述您如何测试更改 -->
## Checklist / 检查清单
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- 检查以下所有要点,并在所有适用的框中打`x` -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
<!-- 如果您对其中任何一项不确定,请不要犹豫提问。我们会帮助您! -->
- [ ] I have read the [CONTRIBUTING](https://github.com/OpenListTeam/OpenList/blob/main/CONTRIBUTING.md) document.
我已阅读 [CONTRIBUTING](https://github.com/OpenListTeam/OpenList/blob/main/CONTRIBUTING.md) 文档。
- [ ] I have formatted my code with `go fmt` or [prettier](https://prettier.io/).
我已使用 `go fmt` 或 [prettier](https://prettier.io/) 格式化提交的代码。
- [ ] I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
我已为此 PR 添加了适当的标签(如无权限或需要的标签不存在,请在描述中说明,管理员将后续处理)。
- [ ] I have requested review from relevant code authors using the "Request review" feature when applicable.
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
- [ ] I have updated the repository accordingly (If it’s needed).
我已相应更新了相关仓库(若适用)。
- [ ] [OpenList-Frontend](https://github.com/OpenListTeam/OpenList-Frontend) #XXXX
- [ ] [OpenList-Docs](https://github.com/OpenListTeam/OpenList-Docs) #XXXX
+25 -3
View File
@@ -61,20 +61,38 @@ jobs:
strategy:
matrix:
include:
- target: "!(*musl*|*windows-arm64*|*windows7-*|*android*|*freebsd*)" # xgo and loongarch
- target: "!(*musl*|*windows-arm64*|*windows7-*|*android*|*freebsd*)" # xgo and loongarch (exclude mips64le)
hash: "md5"
- target: "linux-!(arm*)-musl*" #musl-not-arm
flags: ""
goflags: ""
- target: "linux-(mips|mips64|mipsle|mips64le|loong64)-musl*" # musl-compat-family
hash: "md5-linux-musl-mips"
flags: ""
goflags: ""
- target: "linux-!(arm*|mips|mips64|mipsle|mips64le|loong64)-musl*" # musl-not-arm (exclude compat-family)
hash: "md5-linux-musl"
flags: ""
goflags: ""
- target: "linux-arm*-musl*" #musl-arm
hash: "md5-linux-musl-arm"
flags: ""
goflags: ""
- target: "windows-arm64" #win-arm64
hash: "md5-windows-arm64"
flags: ""
goflags: ""
- target: "windows7-*" #win7
hash: "md5-windows7"
flags: ""
goflags: "-tags=sqlite_cgo_compat"
- target: "android-*" #android
hash: "md5-android"
flags: ""
goflags: ""
- target: "freebsd-*" #freebsd
hash: "md5-freebsd"
flags: ""
goflags: ""
name: Beta Release
runs-on: ubuntu-latest
@@ -87,17 +105,19 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.24.5"
go-version: "1.25.0"
- name: Setup web
run: bash build.sh dev web
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Build
uses: OpenListTeam/cgo-actions@v1.2.2
with:
targets: ${{ matrix.target }}
flags: ${{ matrix.flags || '-ldflags=' }}
musl-target-format: $os-$musl-$arch
github-token: ${{ secrets.GITHUB_TOKEN }}
out-dir: build
@@ -109,6 +129,8 @@ jobs:
github.com/OpenListTeam/OpenList/v4/internal/conf.GitCommit=$git_commit
github.com/OpenListTeam/OpenList/v4/internal/conf.Version=$tag
github.com/OpenListTeam/OpenList/v4/internal/conf.WebVersion=rolling
env:
GOFLAGS: ${{ matrix.goflags }}
- name: Compress
run: |
+2 -1
View File
@@ -33,12 +33,13 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.24.5"
go-version: "1.25.0"
- name: Setup web
run: bash build.sh dev web
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Build
uses: OpenListTeam/cgo-actions@v1.2.2
+50 -10
View File
@@ -19,11 +19,49 @@ jobs:
uses: actions/github-script@v7
with:
script: |
let comment = "";
const issueTitle = context.payload.issue.title || "";
const titleNotEdited = /(请修改标题|Please modify the title)/i.test(issueTitle);
if (titleNotEdited) {
comment = "⚠️ 请修改标题以更好地描述您的问题或需求,并删除示例提示。当前 Issue 将被自动关闭。如需继续提交,请创建新的 Issue。\n";
comment += "⚠️ Please modify the title to better describe your issue or request, and remove the example prompt. This issue will be automatically closed. If you wish to proceed, please create a new issue.\n";
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: comment
});
await github.rest.issues.update({
...context.repo,
issue_number: context.issue.number,
state: 'closed',
state_reason: 'not_planned',
labels: ['invalid']
});
return;
}
const issueBody = context.payload.issue.body || "";
const unchecked = /- \[ \] /.test(issueBody);
let comment = "感谢您联系OpenList。我们会尽快回复您。\n";
comment += "Thanks for contacting OpenList. We will reply to you as soon as possible.\n\n";
if (unchecked) {
const confirmHasRead = /- \[ \] (?!我没有阅读这个清单|I have not read these checkboxes)/.test(issueBody);
const confirmNotRead = /- \[[xX]\] (?:我没有阅读这个清单|I have not read these checkboxes)/.test(issueBody);
if (confirmNotRead) {
comment = "⚠️ 你的 Issue 不符合提交规则。请先阅读相关规范后再重新提交。当前 Issue 将被自动关闭。如需继续提交,请确认已了解规则后重新打开或创建新的 Issue。\n";
comment += "⚠️ Your issue does not comply with the submission rules. Please read the guidelines before submitting again. This issue will be automatically closed. If you wish to proceed, please confirm that you have reviewed the rules before reopening or creating a new issue.\n";
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: comment
});
await github.rest.issues.update({
...context.repo,
issue_number: context.issue.number,
state: 'closed',
state_reason: 'not_planned',
labels: ['invalid']
});
return;
}
if (confirmHasRead) {
comment = "感谢您联系OpenList。我们会尽快回复您。\n";
comment += "Thanks for contacting OpenList. We will reply to you as soon as possible.\n\n";
comment += "由于您提出的 Issue 中包含部分未确认的项目,为了更好地管理项目,在人工审核后可能会直接关闭此问题。\n";
comment += "如果您能确认并补充相关未确认项目的信息,欢迎随时重新提交。我们会及时关注并处理。感谢您的理解与支持!\n";
comment += "Since your issue contains some unchecked tasks, it may be closed after manual review.\n";
@@ -31,12 +69,12 @@ jobs:
comment += "We will pay attention and handle it in a timely manner.\n\n";
comment += "感谢您的理解与支持!\n";
comment += "Thank you for your understanding and support!\n";
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: comment
});
}
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: comment
});
pr-title-check:
runs-on: ubuntu-latest
@@ -47,12 +85,14 @@ jobs:
with:
script: |
const title = context.payload.pull_request.title || "";
const ok = /^(feat|docs|fix|style|refactor|chore)\(.+?\): /i.test(title);
const ok = /^(feat|docs|fix|style|refactor|chore)\(.+?\)!?: /i.test(title);
if (!ok) {
let comment = "⚠️ PR 标题需以 `feat(): `, `docs(): `, `fix(): `, `style(): `, `refactor(): `, `chore(): ` 其中之一开头,例如:`feat(component): 新增功能`。\n";
comment += "⚠️ The PR title must start with `feat(): `, `docs(): `, `fix(): `, `style(): `, or `refactor(): `, `chore(): `. For example: `feat(component): add new feature`.\n\n";
comment += "如果跨多个组件,请使用主要组件作为前缀,并在标题中枚举、描述中说明。\n";
comment += "If it spans multiple components, use the main component as the prefix and enumerate in the title, describe in the body.\n\n";
comment += "如果是破坏性变更,请在类型后添加 `!`,例如 `feat(component)!: 破坏性变更`。\n";
comment += "For breaking changes, add `!` after the type, e.g., `feat(component)!: breaking change`.\n\n";
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
+3 -1
View File
@@ -46,7 +46,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.24'
go-version: '1.25.0'
- name: Checkout
uses: actions/checkout@v4
@@ -66,10 +66,12 @@ jobs:
bash build.sh release ${{ matrix.build-type == 'lite' && 'lite' || '' }} ${{ matrix.target-platform }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Upload assets
uses: softprops/action-gh-release@v2
with:
files: build/compress/*
prerelease: false
tag_name: ${{ github.event.release.tag_name }}
+4 -2
View File
@@ -47,7 +47,7 @@ jobs:
- uses: actions/setup-go@v5
with:
go-version: 'stable'
go-version: '1.25.0'
- name: Cache Musl
id: cache-musl
@@ -66,6 +66,7 @@ jobs:
run: bash build.sh release docker-multiplatform
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Upload artifacts
uses: actions/upload-artifact@v4
@@ -86,7 +87,7 @@ jobs:
- uses: actions/setup-go@v5
with:
go-version: 'stable'
go-version: '1.25.0'
- name: Cache Musl
id: cache-musl
@@ -105,6 +106,7 @@ jobs:
run: bash build.sh release lite docker-multiplatform
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Upload artifacts
uses: actions/upload-artifact@v4
+38
View File
@@ -0,0 +1,38 @@
name: Sync to Gitee
on:
push:
branches:
- main
workflow_dispatch:
jobs:
sync:
runs-on: ubuntu-latest
name: Sync GitHub to Gitee
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup SSH
run: |
mkdir -p ~/.ssh
echo "${{ secrets.GITEE_SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan gitee.com >> ~/.ssh/known_hosts
- name: Create single commit and push
run: |
git config user.name "GitHub Actions"
git config user.email "actions@github.com"
# Create a new branch
git checkout --orphan new-main
git add .
git commit -m "Sync from GitHub: $(date)"
# Add Gitee remote and force push
git remote add gitee ${{ vars.GITEE_REPO_URL }}
git push --force gitee new-main:main
+2 -1
View File
@@ -36,7 +36,7 @@ jobs:
- uses: actions/setup-go@v5
with:
go-version: 'stable'
go-version: '1.25.0'
- name: Cache Musl
id: cache-musl
@@ -55,6 +55,7 @@ jobs:
run: bash build.sh beta docker-multiplatform
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Upload artifacts
uses: actions/upload-artifact@v4
+40 -70
View File
@@ -2,106 +2,76 @@
## Setup your machine
`OpenList` is written in [Go](https://golang.org/) and [React](https://reactjs.org/).
`OpenList` is written in [Go](https://golang.org/) and [SolidJS](https://www.solidjs.com/).
Prerequisites:
- [git](https://git-scm.com)
- [Go 1.20+](https://golang.org/doc/install)
- [Go 1.24+](https://golang.org/doc/install)
- [gcc](https://gcc.gnu.org/)
- [nodejs](https://nodejs.org/)
Clone `OpenList` and `OpenList-Frontend` anywhere:
## Cloning a fork
Fork and clone `OpenList` and `OpenList-Frontend` anywhere:
```shell
$ git clone https://github.com/OpenListTeam/OpenList.git
$ git clone --recurse-submodules https://github.com/OpenListTeam/OpenList-Frontend.git
$ git clone https://github.com/<your-username>/OpenList.git
$ git clone --recurse-submodules https://github.com/<your-username>/OpenList-Frontend.git
```
## Creating a branch
Create a new branch from the `main` branch, with an appropriate name.
```shell
$ git checkout -b <branch-name>
```
You should switch to the `main` branch for development.
## Preview your change
### backend
```shell
$ go run main.go
```
### frontend
```shell
$ pnpm dev
```
## Add a new driver
Copy `drivers/template` folder and rename it, and follow the comments in it.
## Create a commit
Commit messages should be well formatted, and to make that "standardized".
### Commit Message Format
Each commit message consists of a **header**, a **body** and a **footer**. The header has a special
format that includes a **type**, a **scope** and a **subject**:
Submit your pull request. For PR titles, follow [Conventional Commits](https://www.conventionalcommits.org).
```
<type>(<scope>): <subject>
<BLANK LINE>
<body>
<BLANK LINE>
<footer>
```
https://github.com/OpenListTeam/OpenList/issues/376
The **header** is mandatory and the **scope** of the header is optional.
Any line of the commit message cannot be longer than 100 characters! This allows the message to be easier
to read on GitHub as well as in various git tools.
### Revert
If the commit reverts a previous commit, it should begin with `revert: `, followed by the header
of the reverted commit.
In the body it should say: `This reverts commit <hash>.`, where the hash is the SHA of the commit
being reverted.
### Type
Must be one of the following:
* **feat**: A new feature
* **fix**: A bug fix
* **docs**: Documentation only changes
* **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing
semi-colons, etc)
* **refactor**: A code change that neither fixes a bug nor adds a feature
* **perf**: A code change that improves performance
* **test**: Adding missing or correcting existing tests
* **build**: Affects project builds or dependency modifications
* **revert**: Restore the previous commit
* **ci**: Continuous integration of related file modifications
* **chore**: Changes to the build process or auxiliary tools and libraries such as documentation
generation
* **release**: Release a new version
### Scope
The scope could be anything specifying place of the commit change. For example `$location`,
`$browser`, `$compile`, `$rootScope`, `ngHref`, `ngClick`, `ngView`, etc...
You can use `*` when the change affects more than a single scope.
### Subject
The subject contains succinct description of the change:
* use the imperative, present tense: "change" not "changed" nor "changes"
* don't capitalize first letter
* no dot (.) at the end
### Body
Just as in the **subject**, use the imperative, present tense: "change" not "changed" nor "changes".
The body should include the motivation for the change and contrast this with previous behavior.
### Footer
The footer should contain any information about **Breaking Changes** and is also the place to
[reference GitHub issues that this commit closes](https://help.github.com/articles/closing-issues-via-commit-messages/).
**Breaking Changes** should start with the word `BREAKING CHANGE:` with a space or two newlines.
The rest of the commit message is then used for this.
It's suggested to sign your commits. See: [How to sign commits](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits)
## Submit a pull request
Push your branch to your `openlist` fork and open a pull request against the
`main` branch.
Please make sure your code has been formatted with `go fmt` or [prettier](https://prettier.io/) before submitting.
Push your branch to your `openlist` fork and open a pull request against the `main` branch.
## Merge your pull request
Your pull request will be merged after review. Please wait for the maintainer to merge your pull request after review.
At least 1 approving review is required by reviewers with write access. You can also request a review from maintainers.
## Delete your branch
(Optional) After your pull request is merged, you can delete your branch.
---
Thank you for your contribution! Let's make OpenList better together!
+16 -7
View File
@@ -1,3 +1,6 @@
### Default image is base. You can add other support by modifying BASE_IMAGE_TAG. The following parameters are supported: base (default), aria2, ffmpeg, aio
ARG BASE_IMAGE_TAG=base
FROM alpine:edge AS builder
LABEL stage=go-builder
WORKDIR /app/
@@ -7,21 +10,27 @@ RUN go mod download
COPY ./ ./
RUN bash build.sh release docker
### Default image is base. You can add other support by modifying BASE_IMAGE_TAG. The following parameters are supported: base (default), aria2, ffmpeg, aio
ARG BASE_IMAGE_TAG=base
FROM openlistteam/openlist-base-image:${BASE_IMAGE_TAG}
LABEL MAINTAINER="OpenList"
ARG INSTALL_FFMPEG=false
ARG INSTALL_ARIA2=false
LABEL MAINTAINER="OpenList"
ARG USER=openlist
ARG UID=1001
ARG GID=1001
WORKDIR /opt/openlist/
COPY --chmod=755 --from=builder /app/bin/openlist ./
COPY --chmod=755 entrypoint.sh /entrypoint.sh
RUN addgroup -g ${GID} ${USER} && \
adduser -D -u ${UID} -G ${USER} ${USER} && \
mkdir -p /opt/openlist/data
COPY --from=builder --chmod=755 --chown=${UID}:${GID} /app/bin/openlist ./
COPY --chmod=755 --chown=${UID}:${GID} entrypoint.sh /entrypoint.sh
USER ${USER}
RUN /entrypoint.sh version
ENV PUID=0 PGID=0 UMASK=022 RUN_ARIA2=${INSTALL_ARIA2}
ENV UMASK=022 RUN_ARIA2=${INSTALL_ARIA2}
VOLUME /opt/openlist/data/
EXPOSE 5244 5245
CMD [ "/entrypoint.sh" ]
+13 -5
View File
@@ -1,18 +1,26 @@
ARG BASE_IMAGE_TAG=base
FROM ghcr.io/openlistteam/openlist-base-image:${BASE_IMAGE_TAG}
LABEL MAINTAINER="OpenList"
ARG TARGETPLATFORM
ARG INSTALL_FFMPEG=false
ARG INSTALL_ARIA2=false
LABEL MAINTAINER="OpenList"
ARG USER=openlist
ARG UID=1001
ARG GID=1001
WORKDIR /opt/openlist/
COPY --chmod=755 /build/${TARGETPLATFORM}/openlist ./
COPY --chmod=755 entrypoint.sh /entrypoint.sh
RUN addgroup -g ${GID} ${USER} && \
adduser -D -u ${UID} -G ${USER} ${USER} && \
mkdir -p /opt/openlist/data
COPY --chmod=755 --chown=${UID}:${GID} /build/${TARGETPLATFORM}/openlist ./
COPY --chmod=755 --chown=${UID}:${GID} entrypoint.sh /entrypoint.sh
USER ${USER}
RUN /entrypoint.sh version
ENV PUID=0 PGID=0 UMASK=022 RUN_ARIA2=${INSTALL_ARIA2}
ENV UMASK=022 RUN_ARIA2=${INSTALL_ARIA2}
VOLUME /opt/openlist/data/
EXPOSE 5244 5245
CMD [ "/entrypoint.sh" ]
+17 -2
View File
@@ -64,7 +64,9 @@ Thank you for your support and understanding of the OpenList project.
- [x] [UPYUN Storage Service](https://www.upyun.com/products/file-storage)
- [x] [WebDAV](https://en.wikipedia.org/wiki/WebDAV)
- [x] Teambition([China](https://www.teambition.com), [International](https://us.teambition.com))
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
@@ -74,7 +76,6 @@ Thank you for your support and understanding of the OpenList project.
- [x] [Thunder](https://pan.xunlei.com)
- [x] [Lanzou](https://www.lanzou.com)
- [x] [ILanzou](https://www.ilanzou.com)
- [x] [Aliyundrive share](https://www.alipan.com)
- [x] [Google photo](https://photos.google.com)
- [x] [Mega.nz](https://mega.nz)
- [x] [Baidu photo](https://photo.baidu.com)
@@ -85,6 +86,15 @@ Thank you for your support and understanding of the OpenList project.
- [x] [FeijiPan](https://www.feijipan.com)
- [x] [dogecloud](https://www.dogecloud.com/product/oss)
- [x] [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs)
- [x] [Chaoxing](https://www.chaoxing.com)
- [x] [CNB](https://cnb.cool/)
- [x] [Degoo](https://degoo.com)
- [x] [Doubao](https://www.doubao.com)
- [x] [Febbox](https://www.febbox.com)
- [x] [GitHub](https://github.com)
- [x] [OpenList](https://github.com/OpenListTeam/OpenList)
- [x] [Teldrive](https://github.com/tgdrive/teldrive)
- [x] [Weiyun](https://www.weiyun.com)
- [x] Easy to deploy and out-of-the-box
- [x] File preview (PDF, markdown, code, plain text, ...)
- [x] Image preview in gallery mode
@@ -112,12 +122,17 @@ Thank you for your support and understanding of the OpenList project.
## Demo
N/A (to be rebuilt)
- 🌎 [Global Demo](https://demo.oplist.org)
- 🇨🇳 [CN Demo](https://demo.oplist.org.cn)
## Discussion
Please refer to [*Discussions*](https://github.com/OpenListTeam/OpenList/discussions) for raising general questions, ***Issues* is for bug reports and feature requests only.**
## Sponsor
[![VPS.Town](https://vps.town/static/images/sponsor.png)](https://vps.town "VPS.Town - Trust, Effortlessly. Your Cloud, Reimagined.")
## License
The `OpenList` is open-source software licensed under the [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.txt) license.
+17 -2
View File
@@ -64,7 +64,9 @@ OpenList 是一个由 OpenList 团队独立维护的开源项目,遵循 AGPL-3
- [x] [又拍云对象存储](https://www.upyun.com/products/file-storage)
- [x] [WebDAV](https://en.wikipedia.org/wiki/WebDAV)
- [x] Teambition([中国](https://www.teambition.com), [国际](https://us.teambition.com))
- [x] [MediaFire](https://www.mediafire.com)
- [x] [分秒帧](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [和彩云](https://yun.139.com)(个人、家庭、群组)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [百度网盘](http://pan.baidu.com)
@@ -74,7 +76,6 @@ OpenList 是一个由 OpenList 团队独立维护的开源项目,遵循 AGPL-3
- [x] [迅雷网盘](https://pan.xunlei.com)
- [x] [蓝奏云](https://www.lanzou.com)
- [x] [蓝奏云优享版](https://www.ilanzou.com)
- [x] [阿里云盘分享](https://www.alipan.com)
- [x] [Google 相册](https://photos.google.com)
- [x] [Mega.nz](https://mega.nz)
- [x] [百度相册](https://photo.baidu.com)
@@ -85,6 +86,15 @@ OpenList 是一个由 OpenList 团队独立维护的开源项目,遵循 AGPL-3
- [x] [飞机盘](https://www.feijipan.com)
- [x] [多吉云](https://www.dogecloud.com/product/oss)
- [x] [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs)
- [x] [超星](https://www.chaoxing.com)
- [x] [CNB](https://cnb.cool/)
- [x] [Degoo](https://degoo.com)
- [x] [豆包](https://www.doubao.com)
- [x] [Febbox](https://www.febbox.com)
- [x] [GitHub](https://github.com)
- [x] [OpenList](https://github.com/OpenListTeam/OpenList)
- [x] [Teldrive](https://github.com/tgdrive/teldrive)
- [x] [微云](https://www.weiyun.com)
- [x] 部署方便,开箱即用
- [x] 文件预览(PDF、markdown、代码、纯文本等)
- [x] 画廊模式下的图片预览
@@ -112,12 +122,17 @@ OpenList 是一个由 OpenList 团队独立维护的开源项目,遵循 AGPL-3
## 演示
N/A(待重建)
- 🇨🇳 [国内演示站](https://demo.oplist.org.cn)
- 🌎 [海外演示站](https://demo.oplist.org)
## 讨论
如有一般性问题请前往 [*Discussions*](https://github.com/OpenListTeam/OpenList/discussions) 讨论区,***Issues* 仅用于错误报告和功能请求。**
## 赞助者
[![VPS.Town](https://vps.town/static/images/sponsor.png)](https://vps.town "VPS.Town - Trust, Effortlessly. Your Cloud, Reimagined.")
## 许可证
`OpenList` 是基于 [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.txt) 许可证的开源软件。
+17 -2
View File
@@ -65,6 +65,7 @@ OpenListプロジェクトへのご支援とご理解をありがとうござい
- [x] [WebDAV](https://en.wikipedia.org/wiki/WebDAV)
- [x] Teambition([中国](https://www.teambition.com), [国際](https://us.teambition.com))
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com)(個人、家族、グループ)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
@@ -74,7 +75,6 @@ OpenListプロジェクトへのご支援とご理解をありがとうござい
- [x] [Thunder](https://pan.xunlei.com)
- [x] [Lanzou](https://www.lanzou.com)
- [x] [ILanzou](https://www.ilanzou.com)
- [x] [Aliyundrive share](https://www.alipan.com)
- [x] [Google photo](https://photos.google.com)
- [x] [Mega.nz](https://mega.nz)
- [x] [Baidu photo](https://photo.baidu.com)
@@ -85,6 +85,16 @@ OpenListプロジェクトへのご支援とご理解をありがとうござい
- [x] [FeijiPan](https://www.feijipan.com)
- [x] [dogecloud](https://www.dogecloud.com/product/oss)
- [x] [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs)
- [x] [Chaoxing](https://www.chaoxing.com)
- [x] [CNB](https://cnb.cool/)
- [x] [Degoo](https://degoo.com)
- [x] [Doubao](https://www.doubao.com)
- [x] [Febbox](https://www.febbox.com)
- [x] [GitHub](https://github.com)
- [x] [OpenList](https://github.com/OpenListTeam/OpenList)
- [x] [Teldrive](https://github.com/tgdrive/teldrive)
- [x] [Weiyun](https://www.weiyun.com)
- [x] [MediaFire](https://www.mediafire.com)
- [x] 簡単にデプロイでき、すぐに使える
- [x] ファイルプレビュー(PDF、markdown、コード、テキストなど)
- [x] ギャラリーモードでの画像プレビュー
@@ -112,12 +122,17 @@ OpenListプロジェクトへのご支援とご理解をありがとうござい
## デモ
N/A(再構築中)
- 🌎 [グローバルデモ](https://demo.oplist.org)
- 🇨🇳 [CNデモ](https://demo.oplist.org.cn)
## ディスカッション
一般的な質問は [*Discussions*](https://github.com/OpenListTeam/OpenList/discussions) をご利用ください。***Issues* はバグ報告と機能リクエスト専用です。**
## スポンサー
[![VPS.Town](https://vps.town/static/images/sponsor.png)](https://vps.town "VPS.Town - Trust, Effortlessly. Your Cloud, Reimagined.")
## ライセンス
「OpenList」は [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.txt) ライセンスの下で公開されているオープンソースソフトウェアです。
+17 -2
View File
@@ -64,7 +64,9 @@ Dank u voor uw ondersteuning en begrip
- [x] [UPYUN Storage Service](https://www.upyun.com/products/file-storage)
- [x] [WebDAV](https://en.wikipedia.org/wiki/WebDAV)
- [x] Teambition([China](https://www.teambition.com), [Internationaal](https://us.teambition.com))
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Persoonlijk, Familie, Groep)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
@@ -74,7 +76,6 @@ Dank u voor uw ondersteuning en begrip
- [x] [Thunder](https://pan.xunlei.com)
- [x] [Lanzou](https://www.lanzou.com)
- [x] [ILanzou](https://www.ilanzou.com)
- [x] [Aliyundrive share](https://www.alipan.com)
- [x] [Google photo](https://photos.google.com)
- [x] [Mega.nz](https://mega.nz)
- [x] [Baidu photo](https://photo.baidu.com)
@@ -85,6 +86,15 @@ Dank u voor uw ondersteuning en begrip
- [x] [FeijiPan](https://www.feijipan.com)
- [x] [dogecloud](https://www.dogecloud.com/product/oss)
- [x] [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs)
- [x] [Chaoxing](https://www.chaoxing.com)
- [x] [CNB](https://cnb.cool/)
- [x] [Degoo](https://degoo.com)
- [x] [Doubao](https://www.doubao.com)
- [x] [Febbox](https://www.febbox.com)
- [x] [GitHub](https://github.com)
- [x] [OpenList](https://github.com/OpenListTeam/OpenList)
- [x] [Teldrive](https://github.com/tgdrive/teldrive)
- [x] [Weiyun](https://www.weiyun.com)
- [x] Eenvoudig te implementeren en direct te gebruiken
- [x] Bestandsvoorbeeld (PDF, markdown, code, platte tekst, ...)
- [x] Afbeeldingsvoorbeeld in galerijweergave
@@ -112,12 +122,17 @@ Dank u voor uw ondersteuning en begrip
## Demo
N.v.t. (wordt opnieuw opgebouwd)
- 🌎 [Global Demo](https://demo.oplist.org)
- 🇨🇳 [CN Demo](https://demo.oplist.org.cn)
## Discussie
Stel algemene vragen in [*Discussions*](https://github.com/OpenListTeam/OpenList/discussions), ***Issues* zijn alleen voor bugmeldingen en feature requests.**
## Sponsoren
[![VPS.Town](https://vps.town/static/images/sponsor.png)](https://vps.town "VPS.Town - Trust, Effortlessly. Your Cloud, Reimagined.")
## Licentie
`OpenList` is open-source software onder de [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.txt) licentie.
+89
View File
@@ -0,0 +1,89 @@
# Security Policy
## Supported Versions
Only the latest stable release receives security patches. We strongly recommend always keeping OpenList up to date.
| Version | Supported |
| -------------------- | ------------------ |
| Latest stable (v4.x) | :white_check_mark: |
| Older versions | :x: |
## Reporting a Vulnerability
**Please do NOT report security vulnerabilities through public GitHub Issues.**
If you discover a security vulnerability in OpenList, please report it responsibly by using one of the following channels:
- **GitHub Private Security Advisory** (preferred): [Submit here](https://github.com/OpenListTeam/OpenList/security/advisories/new)
- **Telegram**: Contact a maintainer privately via [@OpenListTeam](https://t.me/OpenListTeam)
When reporting, please include as much of the following as possible:
- A description of the vulnerability and its potential impact
- The affected version(s)
- Step-by-step instructions to reproduce the issue
- Any proof-of-concept code or screenshots (if applicable)
- Suggested mitigation or fix (optional but appreciated)
## Security Best Practices for Users
To keep your OpenList instance secure:
- Always update to the latest release.
- Use a strong, unique admin password and change it after first login.
- Enable HTTPS (TLS) for your deployment — do **not** expose OpenList over plain HTTP on the public internet.
- Limit exposed ports using a reverse proxy (e.g., Nginx, Caddy).
- Set up access controls and avoid enabling guest access unless necessary.
- Regularly review mounted storage permissions and revoke unused API tokens.
- When using Docker, avoid running the container as root if possible.
## Acknowledgments
We sincerely thank all security researchers and community members who responsibly disclose vulnerabilities and help make OpenList safer for everyone.
---
# 安全政策
## 支持的版本
我们仅对最新稳定版本提供安全补丁。强烈建议始终保持 OpenList 为最新版本。
| 版本 | 是否支持 |
| ------------------ | ------------------ |
| 最新稳定版(v4.x) | :white_check_mark: |
| 旧版本 | :x: |
## 报告漏洞
**请勿通过公开的 GitHub Issues 报告安全漏洞。**
如果您在 OpenList 中发现安全漏洞,请通过以下渠道之一负责任地进行报告:
- **GitHub 私密安全公告**(推荐):[点击提交](https://github.com/OpenListTeam/OpenList/security/advisories/new)
- **Telegram**:通过 [@OpenListTeam](https://t.me/OpenListTeam) 私信联系维护者
报告时,请尽量提供以下信息:
- 漏洞描述及其潜在影响
- 受影响的版本
- 复现问题的详细步骤
- 概念验证代码或截图(如有)
- 建议的缓解措施或修复方案(可选,但非常欢迎)
## 用户安全最佳实践
为保障您的 OpenList 实例安全:
- 始终更新至最新版本。
- 使用强且唯一的管理员密码,并在首次登录后立即修改。
- 为您的部署启用 HTTPS(TLS)—— **请勿**在公网上以明文 HTTP 方式暴露 OpenList。
- 使用反向代理(如 Nginx、Caddy)限制对外暴露的端口。
- 配置访问控制,非必要情况下不要开启访客访问。
- 定期检查已挂载存储的权限,并撤销未使用的 API 令牌。
- 使用 Docker 部署时,尽可能避免以 root 用户运行容器。
## 致谢
我们衷心感谢所有负责任地披露漏洞、帮助 OpenList 变得更加安全的安全研究人员和社区成员。
+35 -13
View File
@@ -4,6 +4,9 @@ builtAt="$(date +'%F %T %z')"
gitAuthor="The OpenList Projects Contributors <noreply@openlist.team>"
gitCommit=$(git log --pretty=format:"%h" -1)
# Set frontend repository, default to OpenListTeam/OpenList-Frontend
frontendRepo="${FRONTEND_REPO:-OpenListTeam/OpenList-Frontend}"
githubAuthArgs=""
if [ -n "$GITHUB_TOKEN" ]; then
githubAuthArgs="--header \"Authorization: Bearer $GITHUB_TOKEN\""
@@ -25,7 +28,7 @@ else
git tag -d beta || true
# Always true if there's no tag
version=$(git describe --abbrev=0 --tags 2>/dev/null || echo "v0.0.0")
webVersion=$(eval "curl -fsSL --max-time 2 $githubAuthArgs \"https://api.github.com/repos/OpenListTeam/OpenList-Frontend/releases/latest\"" | grep "tag_name" | head -n 1 | awk -F ":" '{print $2}' | sed 's/\"//g;s/,//g;s/ //g')
webVersion=$(eval "curl -fsSL --max-time 2 $githubAuthArgs \"https://api.github.com/repos/$frontendRepo/releases/latest\"" | grep "tag_name" | head -n 1 | awk -F ":" '{print $2}' | sed 's/\"//g;s/,//g;s/ //g')
fi
echo "backend version: $version"
@@ -45,8 +48,21 @@ ldflags="\
-X 'github.com/OpenListTeam/OpenList/v4/internal/conf.WebVersion=$webVersion' \
"
# Keep sqlite driver tag selection centralized to avoid target drift.
GetBuildTagsForTarget() {
local target="$1"
case "$target" in
linux-loong64|linux-mips|linux-mips64|linux-mips64le|linux-mipsle|linux-musl-loong64|linux-musl-mips|linux-musl-mips64|linux-musl-mips64le|linux-musl-mipsle|windows-386|windows7-386|windows7-amd64)
echo "jsoniter,sqlite_cgo_compat"
;;
*)
echo "jsoniter"
;;
esac
}
FetchWebRolling() {
pre_release_json=$(eval "curl -fsSL --max-time 2 $githubAuthArgs -H \"Accept: application/vnd.github.v3+json\" \"https://api.github.com/repos/OpenListTeam/OpenList-Frontend/releases/tags/rolling\"")
pre_release_json=$(eval "curl -fsSL --max-time 2 $githubAuthArgs -H \"Accept: application/vnd.github.v3+json\" \"https://api.github.com/repos/$frontendRepo/releases/tags/rolling\"")
pre_release_assets=$(echo "$pre_release_json" | jq -r '.assets[].browser_download_url')
# There is no lite for rolling
@@ -59,7 +75,7 @@ FetchWebRolling() {
}
FetchWebRelease() {
release_json=$(eval "curl -fsSL --max-time 2 $githubAuthArgs -H \"Accept: application/vnd.github.v3+json\" \"https://api.github.com/repos/OpenListTeam/OpenList-Frontend/releases/latest\"")
release_json=$(eval "curl -fsSL --max-time 2 $githubAuthArgs -H \"Accept: application/vnd.github.v3+json\" \"https://api.github.com/repos/$frontendRepo/releases/latest\"")
release_assets=$(echo "$release_json" | jq -r '.assets[].browser_download_url')
if [ "$useLite" = true ]; then
@@ -107,6 +123,7 @@ BuildWin7() {
# Build for both 386 and amd64 architectures
for arch in "386" "amd64"; do
echo "building for windows7-${arch}"
build_tags=$(GetBuildTagsForTarget "windows7-${arch}")
export GOOS=windows
export GOARCH=${arch}
export CGO_ENABLED=1
@@ -121,7 +138,7 @@ BuildWin7() {
fi
# Use the patched Go compiler for Win7 compatibility
$(pwd)/go-win7/bin/go build -o "${1}-${arch}.exe" -ldflags="$ldflags" -tags=jsoniter .
$(pwd)/go-win7/bin/go build -o "${1}-${arch}.exe" -ldflags="$ldflags" -tags="$build_tags" .
done
}
@@ -190,11 +207,12 @@ BuildDockerMultiplatform() {
cgo_cc=${CGO_ARGS[$i]}
os=${os_arch%%-*}
arch=${os_arch##*-}
build_tags=$(GetBuildTagsForTarget "$os_arch")
export GOOS=$os
export GOARCH=$arch
export CC=${cgo_cc}
echo "building for $os_arch"
go build -o build/$os/$arch/"$appName" -ldflags="$docker_lflags" -tags=jsoniter .
go build -o build/$os/$arch/"$appName" -ldflags="$docker_lflags" -tags="$build_tags" .
done
DOCKER_ARM_ARCHES=(linux-arm/v6 linux-arm/v7)
@@ -233,7 +251,9 @@ BuildRelease() {
BuildLoongGLIBC() {
local target_abi="$2"
local output_file="$1"
local oldWorldGoVersion="1.24.3"
local oldWorldGoVersion="1.25.0"
local loong_tags
loong_tags=$(GetBuildTagsForTarget "linux-loong64")
if [ "$target_abi" = "abi1.0" ]; then
echo building for linux-loong64-abi1.0
@@ -251,13 +271,13 @@ BuildLoongGLIBC() {
# Download and setup patched Go compiler for old-world
if ! curl -fsSL --retry 3 -H "Authorization: Bearer $GITHUB_TOKEN" \
"https://github.com/loong64/loong64-abi1.0-toolchains/releases/download/20250722/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
"https://github.com/loong64/loong64-abi1.0-toolchains/releases/download/20250821/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
-o go-loong64-abi1.0.tar.gz; then
echo "Error: Failed to download patched Go compiler for old-world ABI1.0"
if [ -n "$GITHUB_TOKEN" ]; then
echo "Error output from curl:"
curl -fsSL --retry 3 -H "Authorization: Bearer $GITHUB_TOKEN" \
"https://github.com/loong64/loong64-abi1.0-toolchains/releases/download/20250722/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
"https://github.com/loong64/loong64-abi1.0-toolchains/releases/download/20250821/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
-o go-loong64-abi1.0.tar.gz || true
fi
return 1
@@ -308,7 +328,7 @@ BuildLoongGLIBC() {
CXX="$(pwd)/gcc8-loong64-abi1.0/bin/loongarch64-linux-gnu-g++" \
CGO_ENABLED=1 \
GOCACHE="$abi1_cache_dir" \
$(pwd)/go-loong64-abi1.0/bin/go build -a -o "$output_file" -ldflags="$ldflags" -tags=jsoniter .; then
$(pwd)/go-loong64-abi1.0/bin/go build -a -o "$output_file" -ldflags="$ldflags" -tags="$loong_tags" .; then
echo "Error: Build failed with patched Go compiler"
echo "Attempting retry with cache cleanup..."
env GOCACHE="$abi1_cache_dir" $(pwd)/go-loong64-abi1.0/bin/go clean -cache
@@ -317,7 +337,7 @@ BuildLoongGLIBC() {
CXX="$(pwd)/gcc8-loong64-abi1.0/bin/loongarch64-linux-gnu-g++" \
CGO_ENABLED=1 \
GOCACHE="$abi1_cache_dir" \
$(pwd)/go-loong64-abi1.0/bin/go build -a -o "$output_file" -ldflags="$ldflags" -tags=jsoniter .; then
$(pwd)/go-loong64-abi1.0/bin/go build -a -o "$output_file" -ldflags="$ldflags" -tags="$loong_tags" .; then
echo "Error: Build failed again after cache cleanup"
echo "Build environment details:"
echo "GOOS=linux"
@@ -363,11 +383,11 @@ BuildLoongGLIBC() {
# Use standard Go compiler for new-world build
echo "Building with standard Go compiler for new-world ABI2.0..."
if ! go build -a -o "$output_file" -ldflags="$ldflags" -tags=jsoniter .; then
if ! go build -a -o "$output_file" -ldflags="$ldflags" -tags="$loong_tags" .; then
echo "Error: Build failed with standard Go compiler"
echo "Attempting retry with cache cleanup..."
go clean -cache
if ! go build -a -o "$output_file" -ldflags="$ldflags" -tags=jsoniter .; then
if ! go build -a -o "$output_file" -ldflags="$ldflags" -tags="$loong_tags" .; then
echo "Error: Build failed again after cache cleanup"
echo "Build environment details:"
echo "GOOS=$GOOS"
@@ -388,6 +408,7 @@ BuildReleaseLinuxMusl() {
mkdir -p "build"
muslflags="--extldflags '-static -fpic' $ldflags"
BASE="https://github.com/OpenListTeam/musl-compilers/releases/latest/download/"
# Keep mips-family targets enabled; sqlite driver selection is handled by Go build tags.
FILES=(x86_64-linux-musl-cross aarch64-linux-musl-cross mips-linux-musl-cross mips64-linux-musl-cross mips64el-linux-musl-cross mipsel-linux-musl-cross powerpc64le-linux-musl-cross s390x-linux-musl-cross loongarch64-linux-musl-cross)
for i in "${FILES[@]}"; do
url="${BASE}${i}.tgz"
@@ -400,12 +421,13 @@ BuildReleaseLinuxMusl() {
for i in "${!OS_ARCHES[@]}"; do
os_arch=${OS_ARCHES[$i]}
cgo_cc=${CGO_ARGS[$i]}
build_tags=$(GetBuildTagsForTarget "$os_arch")
echo building for ${os_arch}
export GOOS=${os_arch%%-*}
export GOARCH=${os_arch##*-}
export CC=${cgo_cc}
export CGO_ENABLED=1
go build -o ./build/$appName-$os_arch -ldflags="$muslflags" -tags=jsoniter .
go build -o ./build/$appName-$os_arch -ldflags="$muslflags" -tags="$build_tags" .
done
}
+7 -6
View File
@@ -6,6 +6,7 @@ package cmd
import (
"fmt"
"github.com/OpenListTeam/OpenList/v4/internal/bootstrap"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/setting"
@@ -20,8 +21,8 @@ var AdminCmd = &cobra.Command{
Aliases: []string{"password"},
Short: "Show admin user's info and some operations about admin user's password",
Run: func(cmd *cobra.Command, args []string) {
Init()
defer Release()
bootstrap.Init()
defer bootstrap.Release()
admin, err := op.GetAdmin()
if err != nil {
utils.Log.Errorf("failed get admin user: %+v", err)
@@ -61,8 +62,8 @@ var ShowTokenCmd = &cobra.Command{
Use: "token",
Short: "Show admin token",
Run: func(cmd *cobra.Command, args []string) {
Init()
defer Release()
bootstrap.Init()
defer bootstrap.Release()
token := setting.GetStr(conf.Token)
utils.Log.Infof("show admin token from CLI")
fmt.Println("Admin token:", token)
@@ -70,8 +71,8 @@ var ShowTokenCmd = &cobra.Command{
}
func setAdminPassword(pwd string) {
Init()
defer Release()
bootstrap.Init()
defer bootstrap.Release()
admin, err := op.GetAdmin()
if err != nil {
utils.Log.Errorf("failed get admin user: %+v", err)
+3 -2
View File
@@ -6,6 +6,7 @@ package cmd
import (
"fmt"
"github.com/OpenListTeam/OpenList/v4/internal/bootstrap"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/spf13/cobra"
@@ -16,8 +17,8 @@ var Cancel2FACmd = &cobra.Command{
Use: "cancel2fa",
Short: "Delete 2FA of admin user",
Run: func(cmd *cobra.Command, args []string) {
Init()
defer Release()
bootstrap.Init()
defer bootstrap.Release()
admin, err := op.GetAdmin()
if err != nil {
utils.Log.Errorf("failed to get admin user: %+v", err)
+2 -10
View File
@@ -6,24 +6,16 @@ import (
"strconv"
"github.com/OpenListTeam/OpenList/v4/internal/bootstrap"
"github.com/OpenListTeam/OpenList/v4/internal/bootstrap/data"
"github.com/OpenListTeam/OpenList/v4/internal/db"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
log "github.com/sirupsen/logrus"
)
func Init() {
bootstrap.InitConfig()
bootstrap.Log()
bootstrap.InitDB()
data.InitData()
bootstrap.InitStreamLimit()
bootstrap.InitIndex()
bootstrap.InitUpgradePatch()
bootstrap.Init()
}
func Release() {
db.Close()
bootstrap.Release()
}
var pid = -1
+2 -4
View File
@@ -1,19 +1,17 @@
package cmd
import (
log "github.com/sirupsen/logrus"
"io"
"os"
"path"
"path/filepath"
"strings"
"github.com/spf13/cobra"
rcCrypt "github.com/rclone/rclone/backend/crypt"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/obscure"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)
// encryption and decryption command format for Crypt driver
+1
View File
@@ -2,6 +2,7 @@ package flags
var (
DataDir string
ConfigPath string
Debug bool
NoPrefix bool
Dev bool
+20 -3
View File
@@ -8,7 +8,6 @@ import (
"fmt"
"io"
"os"
"reflect"
"strings"
_ "github.com/OpenListTeam/OpenList/v4/drivers"
@@ -69,15 +68,33 @@ func writeFile(name string, data interface{}) {
log.Errorf("failed to unmarshal json: %+v", err)
return
}
if reflect.DeepEqual(oldData, newData) {
if mergeJson(newData, oldData) {
log.Infof("%s.json no changed, skip", name)
} else {
log.Infof("%s.json changed, update file", name)
//log.Infof("old: %+v\nnew:%+v", oldData, data)
utils.WriteJsonToFile(fmt.Sprintf("lang/%s.json", name), newData, true)
utils.WriteJsonToFile(fmt.Sprintf("lang/%s.json", name), oldData, true)
}
}
func mergeJson(source, target map[string]interface{}) bool {
equal := true
for k, v := range source {
tgtV, tgtOk := target[k]
if !tgtOk {
equal = false
target[k] = v
} else {
srcMap, srcIsMap := v.(map[string]interface{})
tgtMap, tgtIsMap := tgtV.(map[string]interface{})
if srcIsMap && tgtIsMap {
equal = mergeJson(srcMap, tgtMap) && equal
}
}
}
return equal
}
func generateDriversJson() {
drivers := make(Drivers)
drivers["drivers"] = make(KV[interface{}])
+2 -1
View File
@@ -27,7 +27,8 @@ func Execute() {
}
func init() {
RootCmd.PersistentFlags().StringVar(&flags.DataDir, "data", "data", "data folder")
RootCmd.PersistentFlags().StringVar(&flags.DataDir, "data", "data", "data directory (relative paths are resolved against the current working directory)")
RootCmd.PersistentFlags().StringVar(&flags.ConfigPath, "config", "", "path to config.json (relative to current working directory; defaults to [data directory]/config.json, where [data directory] is set by --data)")
RootCmd.PersistentFlags().BoolVar(&flags.Debug, "debug", false, "start with debug mode")
RootCmd.PersistentFlags().BoolVar(&flags.NoPrefix, "no-prefix", false, "disable env prefix")
RootCmd.PersistentFlags().BoolVar(&flags.Dev, "dev", false, "start with dev mode")
+4 -209
View File
@@ -1,32 +1,13 @@
package cmd
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"os"
"os/signal"
"strconv"
"sync"
"syscall"
"time"
"github.com/OpenListTeam/OpenList/v4/cmd/flags"
"github.com/OpenListTeam/OpenList/v4/internal/bootstrap"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/fs"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/server"
"github.com/OpenListTeam/OpenList/v4/server/middlewares"
"github.com/OpenListTeam/sftpd-openlist"
ftpserver "github.com/fclairamb/ftpserverlib"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"golang.org/x/net/http2"
"golang.org/x/net/http2/h2c"
)
// ServerCmd represents the server command
@@ -36,142 +17,9 @@ var ServerCmd = &cobra.Command{
Long: `Start the server at the specified address
the address is defined in config file`,
Run: func(cmd *cobra.Command, args []string) {
Init()
if conf.Conf.DelayedStart != 0 {
utils.Log.Infof("delayed start for %d seconds", conf.Conf.DelayedStart)
time.Sleep(time.Duration(conf.Conf.DelayedStart) * time.Second)
}
bootstrap.InitOfflineDownloadTools()
bootstrap.LoadStorages()
bootstrap.InitTaskManager()
if !flags.Debug && !flags.Dev {
gin.SetMode(gin.ReleaseMode)
}
r := gin.New()
// gin log
if conf.Conf.Log.Filter.Enable {
r.Use(middlewares.FilteredLogger())
} else {
r.Use(gin.LoggerWithWriter(log.StandardLogger().Out))
}
r.Use(gin.RecoveryWithWriter(log.StandardLogger().Out))
server.Init(r)
var httpHandler http.Handler = r
if conf.Conf.Scheme.EnableH2c {
httpHandler = h2c.NewHandler(r, &http2.Server{})
}
var httpSrv, httpsSrv, unixSrv *http.Server
if conf.Conf.Scheme.HttpPort != -1 {
httpBase := fmt.Sprintf("%s:%d", conf.Conf.Scheme.Address, conf.Conf.Scheme.HttpPort)
fmt.Printf("start HTTP server @ %s\n", httpBase)
utils.Log.Infof("start HTTP server @ %s", httpBase)
httpSrv = &http.Server{Addr: httpBase, Handler: httpHandler}
go func() {
err := httpSrv.ListenAndServe()
if err != nil && !errors.Is(err, http.ErrServerClosed) {
utils.Log.Fatalf("failed to start http: %s", err.Error())
}
}()
}
if conf.Conf.Scheme.HttpsPort != -1 {
httpsBase := fmt.Sprintf("%s:%d", conf.Conf.Scheme.Address, conf.Conf.Scheme.HttpsPort)
fmt.Printf("start HTTPS server @ %s\n", httpsBase)
utils.Log.Infof("start HTTPS server @ %s", httpsBase)
httpsSrv = &http.Server{Addr: httpsBase, Handler: r}
go func() {
err := httpsSrv.ListenAndServeTLS(conf.Conf.Scheme.CertFile, conf.Conf.Scheme.KeyFile)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
utils.Log.Fatalf("failed to start https: %s", err.Error())
}
}()
}
if conf.Conf.Scheme.UnixFile != "" {
fmt.Printf("start unix server @ %s\n", conf.Conf.Scheme.UnixFile)
utils.Log.Infof("start unix server @ %s", conf.Conf.Scheme.UnixFile)
unixSrv = &http.Server{Handler: httpHandler}
go func() {
listener, err := net.Listen("unix", conf.Conf.Scheme.UnixFile)
if err != nil {
utils.Log.Fatalf("failed to listen unix: %+v", err)
}
// set socket file permission
mode, err := strconv.ParseUint(conf.Conf.Scheme.UnixFilePerm, 8, 32)
if err != nil {
utils.Log.Errorf("failed to parse socket file permission: %+v", err)
} else {
err = os.Chmod(conf.Conf.Scheme.UnixFile, os.FileMode(mode))
if err != nil {
utils.Log.Errorf("failed to chmod socket file: %+v", err)
}
}
err = unixSrv.Serve(listener)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
utils.Log.Fatalf("failed to start unix: %s", err.Error())
}
}()
}
if conf.Conf.S3.Port != -1 && conf.Conf.S3.Enable {
s3r := gin.New()
s3r.Use(gin.LoggerWithWriter(log.StandardLogger().Out), gin.RecoveryWithWriter(log.StandardLogger().Out))
server.InitS3(s3r)
s3Base := fmt.Sprintf("%s:%d", conf.Conf.Scheme.Address, conf.Conf.S3.Port)
fmt.Printf("start S3 server @ %s\n", s3Base)
utils.Log.Infof("start S3 server @ %s", s3Base)
go func() {
var err error
if conf.Conf.S3.SSL {
httpsSrv = &http.Server{Addr: s3Base, Handler: s3r}
err = httpsSrv.ListenAndServeTLS(conf.Conf.Scheme.CertFile, conf.Conf.Scheme.KeyFile)
}
if !conf.Conf.S3.SSL {
httpSrv = &http.Server{Addr: s3Base, Handler: s3r}
err = httpSrv.ListenAndServe()
}
if err != nil && !errors.Is(err, http.ErrServerClosed) {
utils.Log.Fatalf("failed to start s3 server: %s", err.Error())
}
}()
}
var ftpDriver *server.FtpMainDriver
var ftpServer *ftpserver.FtpServer
if conf.Conf.FTP.Listen != "" && conf.Conf.FTP.Enable {
var err error
ftpDriver, err = server.NewMainDriver()
if err != nil {
utils.Log.Fatalf("failed to start ftp driver: %s", err.Error())
} else {
fmt.Printf("start ftp server on %s\n", conf.Conf.FTP.Listen)
utils.Log.Infof("start ftp server on %s", conf.Conf.FTP.Listen)
go func() {
ftpServer = ftpserver.NewFtpServer(ftpDriver)
err = ftpServer.ListenAndServe()
if err != nil {
utils.Log.Fatalf("problem ftp server listening: %s", err.Error())
}
}()
}
}
var sftpDriver *server.SftpDriver
var sftpServer *sftpd.SftpServer
if conf.Conf.SFTP.Listen != "" && conf.Conf.SFTP.Enable {
var err error
sftpDriver, err = server.NewSftpDriver()
if err != nil {
utils.Log.Fatalf("failed to start sftp driver: %s", err.Error())
} else {
fmt.Printf("start sftp server on %s", conf.Conf.SFTP.Listen)
utils.Log.Infof("start sftp server on %s", conf.Conf.SFTP.Listen)
go func() {
sftpServer = sftpd.NewSftpServer(sftpDriver)
err = sftpServer.RunServer()
if err != nil {
utils.Log.Fatalf("problem sftp server listening: %s", err.Error())
}
}()
}
}
bootstrap.Init()
defer bootstrap.Release()
bootstrap.Start()
// Wait for interrupt signal to gracefully shutdown the server with
// a timeout of 1 second.
quit := make(chan os.Signal, 1)
@@ -180,60 +28,7 @@ the address is defined in config file`,
// kill -9 is syscall. SIGKILL but can"t be catch, so don't need add it
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
utils.Log.Println("Shutdown server...")
fs.ArchiveContentUploadTaskManager.RemoveAll()
Release()
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
var wg sync.WaitGroup
if conf.Conf.Scheme.HttpPort != -1 {
wg.Add(1)
go func() {
defer wg.Done()
if err := httpSrv.Shutdown(ctx); err != nil {
utils.Log.Fatal("HTTP server shutdown err: ", err)
}
}()
}
if conf.Conf.Scheme.HttpsPort != -1 {
wg.Add(1)
go func() {
defer wg.Done()
if err := httpsSrv.Shutdown(ctx); err != nil {
utils.Log.Fatal("HTTPS server shutdown err: ", err)
}
}()
}
if conf.Conf.Scheme.UnixFile != "" {
wg.Add(1)
go func() {
defer wg.Done()
if err := unixSrv.Shutdown(ctx); err != nil {
utils.Log.Fatal("Unix server shutdown err: ", err)
}
}()
}
if conf.Conf.FTP.Listen != "" && conf.Conf.FTP.Enable && ftpServer != nil && ftpDriver != nil {
wg.Add(1)
go func() {
defer wg.Done()
ftpDriver.Stop()
if err := ftpServer.Stop(); err != nil {
utils.Log.Fatal("FTP server shutdown err: ", err)
}
}()
}
if conf.Conf.SFTP.Listen != "" && conf.Conf.SFTP.Enable && sftpServer != nil && sftpDriver != nil {
wg.Add(1)
go func() {
defer wg.Done()
if err := sftpServer.Close(); err != nil {
utils.Log.Fatal("SFTP server shutdown err: ", err)
}
}()
}
wg.Wait()
utils.Log.Println("Server exit")
bootstrap.Shutdown(1 * time.Second)
},
}
+50 -13
View File
@@ -8,7 +8,9 @@ import (
"os"
"strconv"
"github.com/OpenListTeam/OpenList/v4/internal/bootstrap"
"github.com/OpenListTeam/OpenList/v4/internal/db"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/charmbracelet/bubbles/table"
tea "github.com/charmbracelet/bubbletea"
"github.com/charmbracelet/lipgloss"
@@ -22,27 +24,60 @@ var storageCmd = &cobra.Command{
}
var disableStorageCmd = &cobra.Command{
Use: "disable",
Short: "Disable a storage",
Use: "disable [mount path]",
Short: "Disable a storage by mount path",
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("mount path is required")
}
mountPath := args[0]
Init()
defer Release()
bootstrap.Init()
defer bootstrap.Release()
storage, err := db.GetStorageByMountPath(mountPath)
if err != nil {
return fmt.Errorf("failed to query storage: %+v", err)
} else {
storage.Disabled = true
err = db.UpdateStorage(storage)
if err != nil {
return fmt.Errorf("failed to update storage: %+v", err)
} else {
fmt.Printf("Storage with mount path [%s] have been disabled\n", mountPath)
}
storage.Disabled = true
err = db.UpdateStorage(storage)
if err != nil {
return fmt.Errorf("failed to update storage: %+v", err)
}
utils.Log.Infof("Storage with mount path [%s] has been disabled from CLI", mountPath)
fmt.Printf("Storage with mount path [%s] has been disabled\n", mountPath)
return nil
},
}
var deleteStorageCmd = &cobra.Command{
Use: "delete [id]",
Short: "Delete a storage by id",
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) < 1 {
return fmt.Errorf("id is required")
}
id, err := strconv.Atoi(args[0])
if err != nil {
return fmt.Errorf("id must be a number")
}
if force, _ := cmd.Flags().GetBool("force"); force {
fmt.Printf("Are you sure you want to delete storage with id [%d]? [y/N]: ", id)
var confirm string
fmt.Scanln(&confirm)
if confirm != "y" && confirm != "Y" {
fmt.Println("Delete operation cancelled.")
return nil
}
}
bootstrap.Init()
defer bootstrap.Release()
err = db.DeleteStorageById(uint(id))
if err != nil {
return fmt.Errorf("failed to delete storage by id: %+v", err)
}
utils.Log.Infof("Storage with id [%d] have been deleted from CLI", id)
fmt.Printf("Storage with id [%d] have been deleted\n", id)
return nil
},
}
@@ -89,8 +124,8 @@ var listStorageCmd = &cobra.Command{
Use: "list",
Short: "List all storages",
RunE: func(cmd *cobra.Command, args []string) error {
Init()
defer Release()
bootstrap.Init()
defer bootstrap.Release()
storages, _, err := db.GetStorages(1, -1)
if err != nil {
return fmt.Errorf("failed to query storages: %+v", err)
@@ -152,6 +187,8 @@ func init() {
storageCmd.AddCommand(disableStorageCmd)
storageCmd.AddCommand(listStorageCmd)
storageCmd.PersistentFlags().IntVarP(&storageTableHeight, "height", "H", 10, "Table height")
storageCmd.AddCommand(deleteStorageCmd)
deleteStorageCmd.Flags().BoolP("force", "f", false, "Force delete without confirmation")
// Here you will define your flags and configuration settings.
// Cobra supports Persistent Flags which will work for this command
+2 -3
View File
@@ -6,10 +6,9 @@ services:
ports:
- '5244:5244'
- '5245:5245'
user: '0:0'
environment:
- PUID=0
- PGID=0
- UMASK=022
- TZ=UTC
- TZ=Asia/Shanghai
container_name: openlist
image: 'openlistteam/openlist:latest'
+32 -15
View File
@@ -1,43 +1,60 @@
package _115
import (
"errors"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
driver115 "github.com/SheltonZhu/115driver/pkg/driver"
log "github.com/sirupsen/logrus"
)
var (
md5Salt = "Qclm8MGWUv59TnrR0XPg"
appVer = "27.0.5.7"
appVer = "35.6.0.3"
)
func (d *Pan115) getAppVersion() ([]driver115.AppVersion, error) {
result := driver115.VersionResp{}
resp, err := base.RestyClient.R().Get(driver115.ApiGetVersion)
err = driver115.CheckErr(err, &result, resp)
func (d *Pan115) getAppVersion() (string, error) {
result := VersionResp{}
res, err := base.RestyClient.R().Get(driver115.ApiGetVersion)
if err != nil {
return nil, err
return "", err
}
return result.Data.GetAppVersions(), nil
err = utils.Json.Unmarshal(res.Body(), &result)
if err != nil {
return "", err
}
if len(result.Error) > 0 {
return "", errors.New(result.Error)
}
return result.Data.Win.Version, nil
}
func (d *Pan115) getAppVer() string {
// todo add some cache?
vers, err := d.getAppVersion()
ver, err := d.getAppVersion()
if err != nil {
log.Warnf("[115] get app version failed: %v", err)
return appVer
}
for _, ver := range vers {
if ver.AppName == "win" {
return ver.Version
}
if len(ver) > 0 {
return ver
}
return appVer
}
func (d *Pan115) initAppVer() {
appVer = d.getAppVer()
log.Debugf("use app version: %v", appVer)
}
type VersionResp struct {
Error string `json:"error,omitempty"`
Data Versions `json:"data"`
}
type Versions struct {
Win Version `json:"win"`
}
type Version struct {
Version string `json:"version_code"`
}
+15 -5
View File
@@ -68,8 +68,7 @@ func (d *Pan115) Link(ctx context.Context, file model.Obj, args model.LinkArgs)
return nil, err
}
userAgent := args.Header.Get("User-Agent")
downloadInfo, err := d.
DownloadWithUA(file.(*FileObj).PickCode, userAgent)
downloadInfo, err := d.client.DownloadWithUA(file.(*FileObj).PickCode, userAgent)
if err != nil {
return nil, err
}
@@ -186,9 +185,7 @@ func (d *Pan115) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
preHash = strings.ToUpper(preHash)
fullHash := stream.GetHash().GetHash(utils.SHA1)
if len(fullHash) != utils.SHA1.Width {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
_, fullHash, err = streamPkg.CacheFullInTempFileAndHash(stream, cacheFileProgress, utils.SHA1)
_, fullHash, err = streamPkg.CacheFullAndHash(stream, &up, utils.SHA1)
if err != nil {
return nil, err
}
@@ -247,4 +244,17 @@ func (d *Pan115) DeleteOfflineTasks(ctx context.Context, hashes []string, delete
return d.client.DeleteOfflineTasks(hashes, deleteFiles)
}
func (d *Pan115) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
info, err := d.client.GetInfo()
if err != nil {
return nil, err
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: info.SpaceInfo.AllTotal.Size,
UsedSpace: info.SpaceInfo.AllUse.Size,
},
}, nil
}
var _ driver.Driver = (*Pan115)(nil)
+3 -4
View File
@@ -15,10 +15,9 @@ type Addition struct {
}
var config = driver.Config{
Name: "115 Cloud",
DefaultRoot: "0",
// OnlyProxy: true,
// NoOverwriteUpload: true,
Name: "115 Cloud",
DefaultRoot: "0",
LinkCacheMode: driver.LinkCacheUA,
}
func init() {
+4
View File
@@ -22,6 +22,10 @@ func (f *FileObj) GetHash() utils.HashInfo {
return utils.NewHashInfo(utils.SHA1, f.Sha1)
}
func (f *FileObj) Thumb() string {
return f.ThumbURL
}
type UploadResult struct {
driver.BasicResp
Data struct {
+7 -62
View File
@@ -9,7 +9,6 @@ import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
@@ -20,13 +19,12 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
netutil "github.com/OpenListTeam/OpenList/v4/internal/net"
"github.com/OpenListTeam/OpenList/v4/pkg/http_range"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
cipher "github.com/SheltonZhu/115driver/pkg/crypto/ec115"
crypto "github.com/SheltonZhu/115driver/pkg/crypto/m115"
driver115 "github.com/SheltonZhu/115driver/pkg/driver"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"github.com/pkg/errors"
)
@@ -108,60 +106,6 @@ func (d *Pan115) getUA() string {
return fmt.Sprintf("Mozilla/5.0 115Browser/%s", appVer)
}
func (d *Pan115) DownloadWithUA(pickCode, ua string) (*driver115.DownloadInfo, error) {
key := crypto.GenerateKey()
result := driver115.DownloadResp{}
params, err := utils.Json.Marshal(map[string]string{"pick_code": pickCode})
if err != nil {
return nil, err
}
data := crypto.Encode(params, key)
bodyReader := strings.NewReader(url.Values{"data": []string{data}}.Encode())
reqUrl := fmt.Sprintf("%s?t=%s", driver115.AndroidApiDownloadGetUrl, driver115.Now().String())
req, _ := http.NewRequest(http.MethodPost, reqUrl, bodyReader)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Cookie", d.Cookie)
req.Header.Set("User-Agent", ua)
resp, err := d.client.Client.GetClient().Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
if err := utils.Json.Unmarshal(body, &result); err != nil {
return nil, err
}
if err = result.Err(string(body)); err != nil {
return nil, err
}
b, err := crypto.Decode(string(result.EncodedData), key)
if err != nil {
return nil, err
}
downloadInfo := struct {
Url string `json:"url"`
}{}
if err := utils.Json.Unmarshal(b, &downloadInfo); err != nil {
return nil, err
}
info := &driver115.DownloadInfo{}
info.PickCode = pickCode
info.Header = resp.Request.Header
info.Url.Url = downloadInfo.Url
return info, nil
}
func (c *Pan115) GenerateToken(fileID, preID, timeStamp, fileSize, signKey, signVal string) string {
userID := strconv.FormatInt(c.client.UserID, 10)
userIDMd5 := md5.Sum([]byte(userID))
@@ -279,7 +223,7 @@ func (c *Pan115) UploadByOSS(ctx context.Context, params *driver115.UploadOSSPar
if err != nil {
return nil, err
}
ossClient, err := oss.New(driver115.OSSEndpoint, ossToken.AccessKeyID, ossToken.AccessKeySecret)
ossClient, err := netutil.NewOSSClient(driver115.OSSEndpoint, ossToken.AccessKeyID, ossToken.AccessKeySecret)
if err != nil {
return nil, err
}
@@ -309,7 +253,8 @@ func (c *Pan115) UploadByOSS(ctx context.Context, params *driver115.UploadOSSPar
// UploadByMultipart upload by mutipart blocks
func (d *Pan115) UploadByMultipart(ctx context.Context, params *driver115.UploadOSSParams, fileSize int64, s model.FileStreamer,
dirID string, up driver.UpdateProgress, opts ...driver115.UploadMultipartOption) (*UploadResult, error) {
dirID string, up driver.UpdateProgress, opts ...driver115.UploadMultipartOption,
) (*UploadResult, error) {
var (
chunks []oss.FileChunk
parts []oss.UploadPart
@@ -321,7 +266,7 @@ func (d *Pan115) UploadByMultipart(ctx context.Context, params *driver115.Upload
err error
)
tmpF, err := s.CacheFullInTempFile()
tmpF, err := s.CacheFullAndWriter(&up, nil)
if err != nil {
return nil, err
}
@@ -339,7 +284,7 @@ func (d *Pan115) UploadByMultipart(ctx context.Context, params *driver115.Upload
return nil, err
}
if ossClient, err = oss.New(driver115.OSSEndpoint, ossToken.AccessKeyID, ossToken.AccessKeySecret, oss.EnableMD5(true), oss.EnableCRC(true)); err != nil {
if ossClient, err = netutil.NewOSSClient(driver115.OSSEndpoint, ossToken.AccessKeyID, ossToken.AccessKeySecret, oss.EnableMD5(true), oss.EnableCRC(true)); err != nil {
return nil, err
}
+30 -22
View File
@@ -53,6 +53,12 @@ func (d *Open115) Init(ctx context.Context) error {
if d.Addition.LimitRate > 0 {
d.limiter = rate.NewLimiter(rate.Limit(d.Addition.LimitRate), 1)
}
if d.PageSize <= 0 {
d.PageSize = 200
} else if d.PageSize > 1150 {
d.PageSize = 1150
}
return nil
}
@@ -69,7 +75,7 @@ func (d *Open115) Drop(ctx context.Context) error {
func (d *Open115) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
var res []model.Obj
pageSize := int64(200)
pageSize := int64(d.PageSize)
offset := int64(0)
for {
if err := d.WaitLimit(ctx); err != nil {
@@ -131,23 +137,6 @@ func (d *Open115) Link(ctx context.Context, file model.Obj, args model.LinkArgs)
}, nil
}
func (d *Open115) GetObjInfo(ctx context.Context, path string) (model.Obj, error) {
if err := d.WaitLimit(ctx); err != nil {
return nil, err
}
resp, err := d.client.GetFolderInfoByPath(ctx, path)
if err != nil {
return nil, err
}
return &Obj{
Fid: resp.FileID,
Fn: resp.FileName,
Fc: resp.FileCategory,
Sha1: resp.Sha1,
Pc: resp.PickCode,
}, nil
}
func (d *Open115) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) (model.Obj, error) {
if err := d.WaitLimit(ctx); err != nil {
return nil, err
@@ -187,7 +176,7 @@ func (d *Open115) Rename(ctx context.Context, srcObj model.Obj, newName string)
}
_, err := d.client.UpdateFile(ctx, &sdk.UpdateFileReq{
FileID: srcObj.GetID(),
FileNma: newName,
FileName: newName,
})
if err != nil {
return nil, err
@@ -239,9 +228,7 @@ func (d *Open115) Put(ctx context.Context, dstDir model.Obj, file model.FileStre
}
sha1 := file.GetHash().GetHash(utils.SHA1)
if len(sha1) != utils.SHA1.Width {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
_, sha1, err = stream.CacheFullInTempFileAndHash(file, cacheFileProgress, utils.SHA1)
_, sha1, err = stream.CacheFullAndHash(file, &up, utils.SHA1)
if err != nil {
return err
}
@@ -339,6 +326,27 @@ func (d *Open115) OfflineList(ctx context.Context) (*sdk.OfflineTaskListResp, er
return resp, nil
}
func (d *Open115) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
userInfo, err := d.client.UserInfo(ctx)
if err != nil {
return nil, err
}
total, err := ParseInt64(userInfo.RtSpaceInfo.AllTotal.Size)
if err != nil {
return nil, err
}
used, err := ParseInt64(userInfo.RtSpaceInfo.AllUse.Size)
if err != nil {
return nil, err
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: total,
UsedSpace: used,
},
}, nil
}
// func (d *Open115) GetArchiveMeta(ctx context.Context, obj model.Obj, args model.ArchiveArgs) (model.ArchiveMeta, error) {
// // TODO get archive file meta-info, return errs.NotImplement to use an internal archive tool, optional
// return nil, errs.NotImplement
+4 -2
View File
@@ -12,13 +12,15 @@ type Addition struct {
OrderBy string `json:"order_by" type:"select" options:"file_name,file_size,user_utime,file_type"`
OrderDirection string `json:"order_direction" type:"select" options:"asc,desc"`
LimitRate float64 `json:"limit_rate" type:"float" default:"1" help:"limit all api request rate ([limit]r/1s)"`
PageSize int64 `json:"page_size" type:"number" default:"200" help:"list api per page size of 115open driver"`
AccessToken string `json:"access_token" required:"true"`
RefreshToken string `json:"refresh_token" required:"true"`
}
var config = driver.Config{
Name: "115 Open",
DefaultRoot: "0",
Name: "115 Open",
DefaultRoot: "0",
LinkCacheMode: driver.LinkCacheUA,
}
func init() {
+19 -9
View File
@@ -9,6 +9,8 @@ import (
sdk "github.com/OpenListTeam/115-sdk-go"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
netutil "github.com/OpenListTeam/OpenList/v4/internal/net"
streamPkg "github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"github.com/avast/retry-go"
@@ -35,7 +37,7 @@ func calPartSize(fileSize int64) int64 {
}
func (d *Open115) singleUpload(ctx context.Context, tempF model.File, tokenResp *sdk.UploadGetTokenResp, initResp *sdk.UploadInitResp) error {
ossClient, err := oss.New(tokenResp.Endpoint, tokenResp.AccessKeyId, tokenResp.AccessKeySecret, oss.SecurityToken(tokenResp.SecurityToken))
ossClient, err := netutil.NewOSSClient(tokenResp.Endpoint, tokenResp.AccessKeyId, tokenResp.AccessKeySecret, oss.SecurityToken(tokenResp.SecurityToken))
if err != nil {
return err
}
@@ -69,10 +71,7 @@ func (d *Open115) singleUpload(ctx context.Context, tempF model.File, tokenResp
// }
func (d *Open115) multpartUpload(ctx context.Context, stream model.FileStreamer, up driver.UpdateProgress, tokenResp *sdk.UploadGetTokenResp, initResp *sdk.UploadInitResp) error {
fileSize := stream.GetSize()
chunkSize := calPartSize(fileSize)
ossClient, err := oss.New(tokenResp.Endpoint, tokenResp.AccessKeyId, tokenResp.AccessKeySecret, oss.SecurityToken(tokenResp.SecurityToken))
ossClient, err := netutil.NewOSSClient(tokenResp.Endpoint, tokenResp.AccessKeyId, tokenResp.AccessKeySecret, oss.SecurityToken(tokenResp.SecurityToken))
if err != nil {
return err
}
@@ -86,6 +85,13 @@ func (d *Open115) multpartUpload(ctx context.Context, stream model.FileStreamer,
return err
}
fileSize := stream.GetSize()
chunkSize := calPartSize(fileSize)
ss, err := streamPkg.NewStreamSectionReader(stream, int(chunkSize), &up)
if err != nil {
return err
}
partNum := (stream.GetSize() + chunkSize - 1) / chunkSize
parts := make([]oss.UploadPart, partNum)
offset := int64(0)
@@ -98,20 +104,24 @@ func (d *Open115) multpartUpload(ctx context.Context, stream model.FileStreamer,
if i == partNum {
partSize = fileSize - (i-1)*chunkSize
}
rd := utils.NewMultiReadable(io.LimitReader(stream, partSize))
rd, err := ss.GetSectionReader(offset, partSize)
if err != nil {
return err
}
err = retry.Do(func() error {
_ = rd.Reset()
rateLimitedRd := driver.NewLimitedUploadStream(ctx, rd)
part, err := bucket.UploadPart(imur, rateLimitedRd, partSize, int(i))
rd.Seek(0, io.SeekStart)
part, err := bucket.UploadPart(imur, driver.NewLimitedUploadStream(ctx, rd), partSize, int(i))
if err != nil {
return err
}
parts[i-1] = part
return nil
},
retry.Context(ctx),
retry.Attempts(3),
retry.DelayType(retry.BackOffDelay),
retry.Delay(time.Second))
ss.FreeSectionReader(rd)
if err != nil {
return err
}
+13 -1
View File
@@ -1,3 +1,15 @@
package _115_open
// do others that not defined in Driver interface
import "encoding/json"
func ParseInt64(v json.Number) (int64, error) {
i, err := v.Int64()
if err == nil {
return i, nil
}
f, e1 := v.Float64()
if e1 == nil {
return int64(f), nil
}
return int64(0), err
}
+18 -3
View File
@@ -3,6 +3,7 @@ package _115_share
import (
"context"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
@@ -49,9 +50,16 @@ func (d *Pan115Share) List(ctx context.Context, dir model.Obj, args model.ListAr
if err := d.WaitLimit(ctx); err != nil {
return nil, err
}
var ua string
// TODO: will use user agent from header
// if args.Header != nil {
// ua = args.Header.Get("User-Agent")
// }
if ua == "" {
ua = base.UserAgentNT
}
files := make([]driver115.ShareFile, 0)
fileResp, err := d.client.GetShareSnap(d.ShareCode, d.ReceiveCode, dir.GetID(), driver115.QueryLimit(int(d.PageSize)))
fileResp, err := d.client.GetShareSnapWithUA(ua, d.ShareCode, d.ReceiveCode, dir.GetID(), driver115.QueryLimit(int(d.PageSize)))
if err != nil {
return nil, err
}
@@ -77,7 +85,14 @@ func (d *Pan115Share) Link(ctx context.Context, file model.Obj, args model.LinkA
if err := d.WaitLimit(ctx); err != nil {
return nil, err
}
downloadInfo, err := d.client.DownloadByShareCode(d.ShareCode, d.ReceiveCode, file.GetID())
var ua string
if args.Header != nil {
ua = args.Header.Get("User-Agent")
}
if ua == "" {
ua = base.UserAgent
}
downloadInfo, err := d.client.DownloadByShareCodeWithUA(ua, d.ShareCode, d.ReceiveCode, file.GetID())
if err != nil {
return nil, err
}
+8 -3
View File
@@ -5,6 +5,7 @@ import (
"strconv"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
driver115 "github.com/SheltonZhu/115driver/pkg/driver"
@@ -20,6 +21,7 @@ type FileObj struct {
FileName string
isDir bool
FileID string
ThumbURL string
}
func (f *FileObj) CreateTime() time.Time {
@@ -54,6 +56,10 @@ func (f *FileObj) GetPath() string {
return ""
}
func (f *FileObj) Thumb() string {
return f.ThumbURL
}
func transFunc(sf driver115.ShareFile) (model.Obj, error) {
timeInt, err := strconv.ParseInt(sf.UpdateTime, 10, 64)
if err != nil {
@@ -74,15 +80,14 @@ func transFunc(sf driver115.ShareFile) (model.Obj, error) {
FileName: string(sf.FileName),
isDir: isDir,
FileID: fileID,
ThumbURL: sf.ThumbURL,
}, nil
}
var UserAgent = driver115.UA115Browser
func (d *Pan115Share) login() error {
var err error
opts := []driver115.Option{
driver115.UA(UserAgent),
driver115.UA(base.UserAgentNT),
}
d.client = driver115.New(opts...)
if _, err := d.client.GetShareSnap(d.ShareCode, d.ReceiveCode, ""); err != nil {
+17 -5
View File
@@ -41,7 +41,9 @@ func (d *Pan123) GetAddition() driver.Additional {
}
func (d *Pan123) Init(ctx context.Context) error {
_, err := d.Request(UserInfo, http.MethodGet, nil, nil)
_, err := d.Request(UserInfo, http.MethodGet, func(req *resty.Request) {
req.SetHeader("platform", "web")
}, nil)
return err
}
@@ -74,7 +76,6 @@ func (d *Pan123) Link(ctx context.Context, file model.Obj, args model.LinkArgs)
"type": f.Type,
}
resp, err := d.Request(DownloadInfo, http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
}, nil)
if err != nil {
@@ -182,9 +183,7 @@ func (d *Pan123) Put(ctx context.Context, dstDir model.Obj, file model.FileStrea
etag := file.GetHash().GetHash(utils.MD5)
var err error
if len(etag) < utils.MD5.Width {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
_, etag, err = stream.CacheFullInTempFileAndHash(file, cacheFileProgress, utils.MD5)
_, etag, err = stream.CacheFullAndHash(file, &up, utils.MD5)
if err != nil {
return err
}
@@ -256,4 +255,17 @@ func (d *Pan123) APIRateLimit(ctx context.Context, api string) error {
return limiter.Wait(ctx)
}
func (d *Pan123) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
userInfo, err := d.getUserInfo(ctx)
if err != nil {
return nil, err
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: userInfo.Data.SpacePermanent + userInfo.Data.SpaceTemp,
UsedSpace: userInfo.Data.SpaceUsed,
},
}, nil
}
var _ driver.Driver = (*Pan123)(nil)
+11 -2
View File
@@ -11,17 +11,26 @@ type Addition struct {
driver.RootID
//OrderBy string `json:"order_by" type:"select" options:"file_id,file_name,size,update_at" default:"file_name"`
//OrderDirection string `json:"order_direction" type:"select" options:"asc,desc" default:"asc"`
AccessToken string
AccessToken string
UploadThread int `json:"UploadThread" type:"number" default:"3" help:"the threads of upload"`
Platform string `json:"platform" type:"string" default:"web" help:"the platform header value, sent with API requests"`
}
var config = driver.Config{
Name: "123Pan",
DefaultRoot: "0",
LocalSort: true,
PreferProxy: true,
}
func init() {
op.RegisterDriver(func() driver.Driver {
return &Pan123{}
// 新增默认选项 要在RegisterDriver初始化设置 才会对正在使用的用户生效
return &Pan123{
Addition: Addition{
UploadThread: 3,
Platform: "web",
},
}
})
}
+57 -1
View File
@@ -28,7 +28,7 @@ func (f File) CreateTime() time.Time {
}
func (f File) GetHash() utils.HashInfo {
return utils.HashInfo{}
return utils.NewHashInfo(utils.MD5, f.Etag)
}
func (f File) GetPath() string {
@@ -122,3 +122,59 @@ type S3PreSignedURLs struct {
PreSignedUrls map[string]string `json:"presignedUrls"`
} `json:"data"`
}
type UserInfoResp struct {
Data struct {
Uid int64 `json:"UID"`
Nickname string `json:"Nickname"`
SpaceUsed int64 `json:"SpaceUsed"`
SpacePermanent int64 `json:"SpacePermanent"`
SpaceTemp int64 `json:"SpaceTemp"`
FileCount int `json:"FileCount"`
} `json:"data"`
}
type offlineResolveResp struct {
Data struct {
List []struct {
Result int `json:"result"`
ID int64 `json:"id"`
ErrCode int `json:"err_code"`
ErrMsg string `json:"err_msg"`
Files []struct {
ID int64 `json:"id"`
} `json:"files"`
} `json:"list"`
} `json:"data"`
}
type offlineSubmitResp struct {
Data struct {
TaskList []struct {
TaskID int64 `json:"task_id"`
Result int `json:"result"`
} `json:"task_list"`
} `json:"data"`
}
type offlineTaskListResp struct {
Data struct {
HasRun bool `json:"has_run"`
List []offlineTask `json:"list"`
Total int `json:"total"`
} `json:"data"`
}
type offlineTask struct {
TaskID int64 `json:"task_id"`
Name string `json:"name"`
Status int `json:"status"`
Size int64 `json:"size"`
ThirdTask string `json:"third_task_id"`
Downloaded int64 `json:"downloaded"`
Progress float64 `json:"progress"`
UploadIDR int64 `json:"upload_idr"`
UploadName string `json:"upload_name"`
Type string `json:"type"`
Speed int64 `json:"speed"`
}
+86 -61
View File
@@ -6,11 +6,16 @@ import (
"io"
"net/http"
"strconv"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/errgroup"
"github.com/OpenListTeam/OpenList/v4/pkg/singleflight"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/avast/retry-go"
"github.com/go-resty/resty/v2"
)
@@ -69,18 +74,21 @@ func (d *Pan123) completeS3(ctx context.Context, upReq *UploadResp, file model.F
}
func (d *Pan123) newUpload(ctx context.Context, upReq *UploadResp, file model.FileStreamer, up driver.UpdateProgress) error {
tmpF, err := file.CacheFullInTempFile()
// fetch s3 pre signed urls
size := file.GetSize()
chunkSize := int64(16 * utils.MB)
chunkCount := 1
if size > chunkSize {
chunkCount = int((size + chunkSize - 1) / chunkSize)
}
ss, err := stream.NewStreamSectionReader(file, int(chunkSize), &up)
if err != nil {
return err
}
// fetch s3 pre signed urls
size := file.GetSize()
chunkSize := min(size, 16*utils.MB)
chunkCount := int(size / chunkSize)
lastChunkSize := size % chunkSize
if lastChunkSize > 0 {
chunkCount++
} else {
if lastChunkSize == 0 {
lastChunkSize = chunkSize
}
// only 1 batch is allowed
@@ -90,73 +98,90 @@ func (d *Pan123) newUpload(ctx context.Context, upReq *UploadResp, file model.Fi
batchSize = 10
getS3UploadUrl = d.getS3PreSignedUrls
}
thread := min(int(chunkCount), d.UploadThread)
threadG, uploadCtx := errgroup.NewOrderedGroupWithContext(ctx, thread,
retry.Attempts(3),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay))
for i := 1; i <= chunkCount; i += batchSize {
if utils.IsCanceled(ctx) {
return ctx.Err()
if utils.IsCanceled(uploadCtx) {
break
}
start := i
end := min(i+batchSize, chunkCount+1)
s3PreSignedUrls, err := getS3UploadUrl(ctx, upReq, start, end)
s3PreSignedUrls, err := getS3UploadUrl(uploadCtx, upReq, start, end)
if err != nil {
return err
}
// upload each chunk
for j := start; j < end; j++ {
if utils.IsCanceled(ctx) {
return ctx.Err()
for cur := start; cur < end; cur++ {
if utils.IsCanceled(uploadCtx) {
break
}
offset := int64(cur-1) * chunkSize
curSize := chunkSize
if j == chunkCount {
if cur == chunkCount {
curSize = lastChunkSize
}
err = d.uploadS3Chunk(ctx, upReq, s3PreSignedUrls, j, end, io.NewSectionReader(tmpF, chunkSize*int64(j-1), curSize), curSize, false, getS3UploadUrl)
if err != nil {
return err
}
up(float64(j) * 100 / float64(chunkCount))
var reader io.ReadSeeker
threadG.GoWithLifecycle(errgroup.Lifecycle{
Before: func(ctx context.Context) (err error) {
reader, err = ss.GetSectionReader(offset, curSize)
return
},
Do: func(ctx context.Context) (err error) {
reader.Seek(0, io.SeekStart)
uploadUrl := s3PreSignedUrls.Data.PreSignedUrls[strconv.Itoa(cur)]
if uploadUrl == "" {
return fmt.Errorf("upload url is empty, s3PreSignedUrls: %+v", s3PreSignedUrls)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPut, uploadUrl, driver.NewLimitedUploadStream(ctx, reader))
if err != nil {
return err
}
req.ContentLength = curSize
//req.Header.Set("Content-Length", strconv.FormatInt(curSize, 10))
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode == http.StatusForbidden {
_, err, _ = singleflight.AnyGroup.Do(fmt.Sprintf("Pan123.newUpload_%p", threadG), func() (any, error) {
newS3PreSignedUrls, err := getS3UploadUrl(ctx, upReq, cur, end)
if err != nil {
return nil, err
}
s3PreSignedUrls.Data.PreSignedUrls = newS3PreSignedUrls.Data.PreSignedUrls
return nil, nil
})
if err != nil {
return err
}
return fmt.Errorf("upload s3 chunk %d failed, status code: %d", cur, res.StatusCode)
}
if res.StatusCode != http.StatusOK {
body, err := io.ReadAll(res.Body)
if err != nil {
return err
}
return fmt.Errorf("upload s3 chunk %d failed, status code: %d, body: %s", cur, res.StatusCode, body)
}
progress := 100 * float64(threadG.Success()+1) / float64(chunkCount+1)
up(progress)
return nil
},
After: func(err error) {
ss.FreeSectionReader(reader)
},
})
}
}
if err := threadG.Wait(); err != nil {
return err
}
defer up(100)
// complete s3 upload
return d.completeS3(ctx, upReq, file, chunkCount > 1)
}
func (d *Pan123) uploadS3Chunk(ctx context.Context, upReq *UploadResp, s3PreSignedUrls *S3PreSignedURLs, cur, end int, reader *io.SectionReader, curSize int64, retry bool, getS3UploadUrl func(ctx context.Context, upReq *UploadResp, start int, end int) (*S3PreSignedURLs, error)) error {
uploadUrl := s3PreSignedUrls.Data.PreSignedUrls[strconv.Itoa(cur)]
if uploadUrl == "" {
return fmt.Errorf("upload url is empty, s3PreSignedUrls: %+v", s3PreSignedUrls)
}
req, err := http.NewRequest("PUT", uploadUrl, driver.NewLimitedUploadStream(ctx, reader))
if err != nil {
return err
}
req = req.WithContext(ctx)
req.ContentLength = curSize
//req.Header.Set("Content-Length", strconv.FormatInt(curSize, 10))
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode == http.StatusForbidden {
if retry {
return fmt.Errorf("upload s3 chunk %d failed, status code: %d", cur, res.StatusCode)
}
// refresh s3 pre signed urls
newS3PreSignedUrls, err := getS3UploadUrl(ctx, upReq, cur, end)
if err != nil {
return err
}
s3PreSignedUrls.Data.PreSignedUrls = newS3PreSignedUrls.Data.PreSignedUrls
// retry
reader.Seek(0, io.SeekStart)
return d.uploadS3Chunk(ctx, upReq, s3PreSignedUrls, cur, end, reader, curSize, true, getS3UploadUrl)
}
if res.StatusCode != http.StatusOK {
body, err := io.ReadAll(res.Body)
if err != nil {
return err
}
return fmt.Errorf("upload s3 chunk %d failed, status code: %d, body: %s", cur, res.StatusCode, body)
}
return nil
}
+130 -2
View File
@@ -14,6 +14,7 @@ import (
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/go-resty/resty/v2"
jsoniter "github.com/json-iterator/go"
@@ -43,9 +44,16 @@ const (
S3Auth = MainApi + "/file/s3_upload_object/auth"
UploadCompleteV2 = MainApi + "/file/upload_complete/v2"
S3Complete = MainApi + "/file/s3_complete_multipart_upload"
//AuthKeySalt = "8-8D$sL8gPjom7bk#cY"
OfflineResolve = MainApi + "/v2/offline_download/task/resolve"
OfflineSubmit = MainApi + "/v2/offline_download/task/submit"
OfflineTaskList = MainApi + "/offline_download/task/list"
OfflineTaskDelete = MainApi + "/offline_download/task/delete"
// AuthKeySalt = "8-8D$sL8gPjom7bk#cY"
)
var ErrOfflineTaskNotFound = errors.New("offline task not found")
func signPath(path string, os string, version string) (k string, v string) {
table := []byte{'a', 'd', 'e', 'f', 'g', 'h', 'l', 'm', 'y', 'i', 'j', 'n', 'o', 'p', 'k', 'q', 'r', 's', 't', 'u', 'b', 'c', 'v', 'w', 's', 'z'}
random := fmt.Sprintf("%.f", math.Round(1e7*rand.Float64()))
@@ -203,7 +211,7 @@ do:
"referer": "https://www.123pan.com/",
"authorization": "Bearer " + d.AccessToken,
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) openlist-client",
"platform": "web",
"platform": d.Platform,
"app-version": "3",
//"user-agent": base.UserAgent,
})
@@ -238,6 +246,115 @@ do:
return body, nil
}
func (d *Pan123) OfflineDownload(ctx context.Context, uri string, dstDir model.Obj) (int64, error) {
var resolveResp offlineResolveResp
_, err := d.Request(OfflineResolve, http.MethodPost, func(req *resty.Request) {
req.SetContext(ctx).SetBody(base.Json{
"urls": uri,
})
}, &resolveResp)
if err != nil {
return 0, err
}
if len(resolveResp.Data.List) == 0 {
return 0, fmt.Errorf("offline resolve failed: empty response")
}
if resolveResp.Data.List[0].Result != 0 {
msg := resolveResp.Data.List[0].ErrMsg
if msg == "" {
msg = "offline resolve failed"
}
return 0, fmt.Errorf("%s", msg)
}
resourceID := resolveResp.Data.List[0].ID
if resourceID == 0 {
return 0, fmt.Errorf("offline resolve failed: empty resource id")
}
selectFileIDs := make([]int64, 0, len(resolveResp.Data.List[0].Files))
for _, f := range resolveResp.Data.List[0].Files {
if f.ID > 0 {
selectFileIDs = append(selectFileIDs, f.ID)
}
}
if len(selectFileIDs) == 0 {
return 0, fmt.Errorf("offline resolve failed: empty file list")
}
uploadDir, err := strconv.ParseInt(dstDir.GetID(), 10, 64)
if err != nil {
return 0, fmt.Errorf("invalid destination dir id: %s", dstDir.GetID())
}
var submitResp offlineSubmitResp
_, err = d.Request(OfflineSubmit, http.MethodPost, func(req *resty.Request) {
req.SetContext(ctx).SetBody(base.Json{
"resource_list": []base.Json{
{
"resource_id": resourceID,
"select_file_id": selectFileIDs,
},
},
"upload_dir": uploadDir,
})
}, &submitResp)
if err != nil {
return 0, err
}
if len(submitResp.Data.TaskList) == 0 {
return 0, fmt.Errorf("offline submit failed: empty task list")
}
if submitResp.Data.TaskList[0].Result != 0 {
return 0, fmt.Errorf("offline submit failed")
}
if submitResp.Data.TaskList[0].TaskID == 0 {
return 0, fmt.Errorf("offline submit failed: empty task id")
}
return submitResp.Data.TaskList[0].TaskID, nil
}
func (d *Pan123) GetOfflineTask(ctx context.Context, taskID int64) (*offlineTask, error) {
if taskID == 0 {
return nil, fmt.Errorf("invalid task id")
}
page := 1
pageSize := 100
statusArr := []int{0, 1, 2, 3}
for {
var listResp offlineTaskListResp
_, err := d.Request(OfflineTaskList, http.MethodPost, func(req *resty.Request) {
req.SetContext(ctx).SetBody(base.Json{
"current_page": page,
"page_size": pageSize,
"status_arr": statusArr,
})
}, &listResp)
if err != nil {
return nil, err
}
for i := range listResp.Data.List {
if listResp.Data.List[i].TaskID == taskID {
return &listResp.Data.List[i], nil
}
}
if len(listResp.Data.List) == 0 || page*pageSize >= listResp.Data.Total {
break
}
page++
}
return nil, ErrOfflineTaskNotFound
}
func (d *Pan123) DeleteOfflineTasks(ctx context.Context, taskIDs []int64) error {
if len(taskIDs) == 0 {
return nil
}
_, err := d.Request(OfflineTaskDelete, http.MethodPost, func(req *resty.Request) {
req.SetContext(ctx).SetBody(base.Json{
"task_ids": taskIDs,
})
}, nil)
return err
}
func (d *Pan123) getFiles(ctx context.Context, parentId string, name string) ([]File, error) {
page := 1
total := 0
@@ -282,3 +399,14 @@ func (d *Pan123) getFiles(ctx context.Context, parentId string, name string) ([]
}
return res, nil
}
func (d *Pan123) getUserInfo(ctx context.Context) (*UserInfoResp, error) {
var resp UserInfoResp
_, err := d.Request(UserInfo, http.MethodGet, func(req *resty.Request) {
req.SetContext(ctx)
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
+4
View File
@@ -39,6 +39,10 @@ func (d *Pan123Link) Drop(ctx context.Context) error {
return nil
}
func (Addition) GetRootPath() string {
return "/"
}
func (d *Pan123Link) Get(ctx context.Context, path string) (model.Obj, error) {
node := GetNodeFromRootByPath(d.root, path)
return nodeToObj(node, path)
+158 -12
View File
@@ -2,7 +2,9 @@ package _123_open
import (
"context"
"fmt"
"strconv"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
@@ -15,6 +17,8 @@ import (
type Open123 struct {
model.Storage
Addition
UID uint64
tm *tokenManager
}
func (d *Open123) Config() driver.Config {
@@ -30,6 +34,24 @@ func (d *Open123) Init(ctx context.Context) error {
d.UploadThread = 3
}
if (d.UseOnlineAPI && d.RefreshToken != "" && len(d.APIAddress) > 0) || (d.ClientID != "" && d.ClientSecret != "") {
// proactive refresh by renewapi or client credentials
d.AccessToken = ""
d.tm = &tokenManager{}
} else {
// 避免个人 token 刷新产生的多个登录,被动刷新
// 默认过期时间90天,jwt exp 不可靠
d.tm = &tokenManager{
// accessToken: d.AccessToken,
expiredAt: time.Now().Add(90 * 24 * time.Hour),
}
}
_, err := d.getAccessToken(false)
if err != nil {
return fmt.Errorf("init get access token error: %w", err)
}
return nil
}
@@ -67,13 +89,45 @@ func (d *Open123) List(ctx context.Context, dir model.Obj, args model.ListArgs)
func (d *Open123) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
fileId, _ := strconv.ParseInt(file.GetID(), 10, 64)
if d.DirectLink {
res, err := d.getDirectLink(fileId)
if err != nil {
return nil, err
}
if d.DirectLinkPrivateKey == "" {
duration := 365 * 24 * time.Hour // 缓存1年
return &model.Link{
URL: res.Data.URL,
Expiration: &duration,
}, nil
}
uid, err := d.getUID(ctx)
if err != nil {
return nil, err
}
duration := time.Duration(d.DirectLinkValidDuration) * time.Minute
newURL, err := d.SignURL(res.Data.URL, d.DirectLinkPrivateKey,
uid, duration)
if err != nil {
return nil, err
}
return &model.Link{
URL: newURL,
Expiration: &duration,
}, nil
}
res, err := d.getDownloadInfo(fileId)
if err != nil {
return nil, err
}
link := model.Link{URL: res.Data.DownloadUrl}
return &link, nil
return &model.Link{URL: res.Data.DownloadUrl}, nil
}
func (d *Open123) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) error {
@@ -95,6 +149,22 @@ func (d *Open123) Rename(ctx context.Context, srcObj model.Obj, newName string)
}
func (d *Open123) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
// 尝试使用上传+MD5秒传功能实现复制
// 1. 创建文件
// parentFileID 父目录id,上传到根目录时填写 0
parentFileId, err := strconv.ParseInt(dstDir.GetID(), 10, 64)
if err != nil {
return fmt.Errorf("parse parentFileID error: %v", err)
}
etag := srcObj.(File).Etag
createResp, err := d.create(parentFileId, srcObj.GetName(), etag, srcObj.GetSize(), 2, false)
if err != nil {
return err
}
// 是否秒传
if createResp.Data.Reuse {
return nil
}
return errs.NotSupport
}
@@ -104,27 +174,103 @@ func (d *Open123) Remove(ctx context.Context, obj model.Obj) error {
return d.trash(fileId)
}
func (d *Open123) Put(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress) error {
func (d *Open123) Put(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress) (model.Obj, error) {
// 1. 创建文件
// parentFileID 父目录id,上传到根目录时填写 0
parentFileId, err := strconv.ParseInt(dstDir.GetID(), 10, 64)
etag := file.GetHash().GetHash(utils.MD5)
if err != nil {
return nil, fmt.Errorf("parse parentFileID error: %v", err)
}
// 尝试 SHA1 秒传
sha1Hash := file.GetHash().GetHash(utils.SHA1)
if len(sha1Hash) == utils.SHA1.Width {
resp, err := d.sha1Reuse(parentFileId, file.GetName(), sha1Hash, file.GetSize(), 2)
if err == nil && resp.Data.Reuse {
return File{
FileName: file.GetName(),
Size: file.GetSize(),
FileId: resp.Data.FileID,
Type: 2,
SHA1: sha1Hash,
}, nil
}
}
// etag 文件md5
etag := file.GetHash().GetHash(utils.MD5)
if len(etag) < utils.MD5.Width {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
_, etag, err = stream.CacheFullInTempFileAndHash(file, cacheFileProgress, utils.MD5)
_, etag, err = stream.CacheFullAndHash(file, &up, utils.MD5)
if err != nil {
return err
return nil, err
}
}
createResp, err := d.create(parentFileId, file.GetName(), etag, file.GetSize(), 2, false)
if err != nil {
return err
return nil, err
}
// 是否秒传
if createResp.Data.Reuse {
return nil
// 秒传成功才会返回正确的 FileID,否则为 0
if createResp.Data.FileID != 0 {
return File{
FileName: file.GetName(),
Size: file.GetSize(),
FileId: createResp.Data.FileID,
Type: 2,
Etag: etag,
}, nil
}
}
return d.Upload(ctx, file, createResp, up)
// 2. 上传分片
err = d.Upload(ctx, file, createResp, up)
if err != nil {
return nil, err
}
// 3. 上传完毕
for range 60 {
uploadCompleteResp, err := d.complete(createResp.Data.PreuploadID)
// 返回错误代码未知,如:20103,文档也没有具体说
if err == nil && uploadCompleteResp.Data.Completed && uploadCompleteResp.Data.FileID != 0 {
up(100)
return File{
FileName: file.GetName(),
Size: file.GetSize(),
FileId: uploadCompleteResp.Data.FileID,
Type: 2,
Etag: etag,
}, nil
}
// 若接口返回的completed为 false 时,则需间隔1秒继续轮询此接口,获取上传最终结果。
time.Sleep(time.Second)
}
return nil, fmt.Errorf("upload complete timeout")
}
var _ driver.Driver = (*Open123)(nil)
func (d *Open123) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
userInfo, err := d.getUserInfo(ctx)
if err != nil {
return nil, err
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: userInfo.Data.SpacePermanent + userInfo.Data.SpaceTemp,
UsedSpace: userInfo.Data.SpaceUsed,
},
}, nil
}
func (d *Open123) OfflineDownload(ctx context.Context, url string, dir model.Obj, callback string) (int, error) {
return d.createOfflineDownloadTask(ctx, url, dir.GetID(), callback)
}
func (d *Open123) OfflineDownloadProcess(ctx context.Context, taskID int) (float64, int, error) {
return d.queryOfflineDownloadStatus(ctx, taskID)
}
var (
_ driver.Driver = (*Open123)(nil)
_ driver.PutResult = (*Open123)(nil)
)
+14 -4
View File
@@ -6,16 +6,20 @@ import (
)
type Addition struct {
// refresh_token方式的AccessToken 【对个人开发者暂未开放】
RefreshToken string `json:"RefreshToken" required:"false"`
// 通过 https://www.123pan.com/developer 申请
ClientID string `json:"ClientID" required:"false"`
ClientSecret string `json:"ClientSecret" required:"false"`
// 直接写入AccessToken
// 直接写入AccessToken, AccessToken有过期时间,不建议直接填写
AccessToken string `json:"AccessToken" required:"false"`
// refresh_token方式的AccessToken 【对个人开发者暂未开放】
RefreshToken string `json:"RefreshToken" required:"false"`
// 使用在线API
UseOnlineAPI bool `json:"use_online_api" default:"true"`
APIAddress string `json:"api_url_address" default:"https://api.oplist.org/123cloud/renewapi"`
// 用户名+密码方式登录的AccessToken可以兼容
//Username string `json:"username" required:"false"`
//Password string `json:"password" required:"false"`
@@ -23,6 +27,11 @@ type Addition struct {
// 上传线程数
UploadThread int `json:"UploadThread" type:"number" default:"3" help:"the threads of upload"`
// 使用直链
DirectLink bool `json:"DirectLink" type:"bool" default:"false" required:"false" help:"use direct link when download file"`
DirectLinkPrivateKey string `json:"DirectLinkPrivateKey" required:"false" help:"private key for direct link, if URL authentication is enabled"`
DirectLinkValidDuration int64 `json:"DirectLinkValidDuration" type:"number" default:"30" required:"false" help:"minutes, if URL authentication is enabled"`
driver.RootID
}
@@ -30,6 +39,7 @@ var config = driver.Config{
Name: "123 Open",
DefaultRoot: "0",
LocalSort: true,
PreferProxy: true,
}
func init() {
+129
View File
@@ -0,0 +1,129 @@
package _123_open
import (
"errors"
"fmt"
"net/http"
"sync"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/op"
)
var (
AccessToken = "https://open-api.123pan.com/api/v1/access_token"
)
func expiresInToExpiredAt(expiresIn int64) (time.Time, error) {
if expiresIn <= 0 {
return time.Time{}, errors.New("invalid expires_in from official API")
}
return time.Now().UTC().Add(time.Duration(expiresIn) * time.Second), nil
}
type tokenManager struct {
// accessToken string
expiredAt time.Time
mu sync.Mutex
blockRefresh bool
}
func (d *Open123) getAccessToken(forceRefresh bool) (string, error) {
tm := d.tm
tm.mu.Lock()
defer tm.mu.Unlock()
if tm.blockRefresh {
return "", errors.New("Authentication expired")
}
if !forceRefresh && d.AccessToken != "" && time.Now().Before(tm.expiredAt.Add(-5*time.Minute)) {
return d.AccessToken, nil
}
if err := d.flushAccessToken(); err != nil {
// token expired and failed to refresh, block further refresh attempts
tm.blockRefresh = true
return "", err
}
return d.AccessToken, nil
}
func (d *Open123) flushAccessToken() error {
// Official app renewapi response contains access_token, refresh_token and expires_in.
if d.UseOnlineAPI && d.RefreshToken != "" && len(d.APIAddress) > 0 {
var resp RefreshTokenResp
_, err := base.RestyClient.R().
SetResult(&resp).
SetQueryParams(map[string]string{
"refresh_ui": d.RefreshToken,
"server_use": "true",
"driver_txt": "123cloud_oa",
}).
Get(d.APIAddress)
if err != nil {
return err
}
if resp.AccessToken == "" || resp.RefreshToken == "" {
errMessage := resp.ErrorDescription
if errMessage == "" {
errMessage = resp.Text
}
if errMessage == "" {
errMessage = resp.Message
}
if errMessage == "" {
errMessage = resp.Error
}
if errMessage != "" {
return fmt.Errorf("failed to refresh token: %s", errMessage)
}
return fmt.Errorf("empty access_token or refresh_token returned from official API")
}
expiredAt, err := expiresInToExpiredAt(resp.ExpiresIn)
if err != nil {
return err
}
d.AccessToken = resp.AccessToken
d.RefreshToken = resp.RefreshToken
d.tm.expiredAt = expiredAt
op.MustSaveDriverStorage(d)
d.tm.blockRefresh = false
return nil
}
// Developer API response contains code/message/data(accessToken, expiredAt).
if d.ClientID != "" && d.ClientSecret != "" {
req := base.RestyClient.R()
req.SetHeaders(map[string]string{
"platform": "open_platform",
"Content-Type": "application/json",
})
var resp AccessTokenResp
req.SetBody(base.Json{
"clientID": d.ClientID,
"clientSecret": d.ClientSecret,
})
req.SetResult(&resp)
_, err := req.Execute(http.MethodPost, AccessToken)
if err != nil {
return err
}
if resp.Code != 0 {
return fmt.Errorf("get access token failed: %s", resp.Message)
}
if resp.Data.AccessToken == "" || resp.Data.ExpiredAt == "" {
return errors.New("invalid token payload from developer API")
}
expiredAt, err := time.Parse(time.RFC3339, resp.Data.ExpiredAt)
if err != nil {
return fmt.Errorf("parse expire time failed: %w", err)
}
d.AccessToken = resp.Data.AccessToken
d.tm.expiredAt = expiredAt.UTC()
op.MustSaveDriverStorage(d)
d.tm.blockRefresh = false
return nil
}
return errors.New("no valid authentication method available")
}
+62 -46
View File
@@ -19,6 +19,7 @@ func (a *ApiInfo) Require() {
a.token <- struct{}{}
}
}
func (a *ApiInfo) Release() {
if a.qps > 0 {
time.AfterFunc(time.Second, func() {
@@ -26,13 +27,16 @@ func (a *ApiInfo) Release() {
})
}
}
func (a *ApiInfo) SetQPS(qps int) {
a.qps = qps
a.token = make(chan struct{}, qps)
}
func (a *ApiInfo) NowLen() int {
return len(a.token)
}
func InitApiInfo(url string, qps int) *ApiInfo {
return &ApiInfo{
url: url,
@@ -54,9 +58,13 @@ type File struct {
Category int `json:"category"`
Status int `json:"status"`
Trashed int `json:"trashed"`
SHA1 string
}
func (f File) GetHash() utils.HashInfo {
if len(f.SHA1) == utils.SHA1.Width && len(f.Etag) != utils.MD5.Width {
return utils.NewHashInfo(utils.SHA1, f.SHA1)
}
return utils.NewHashInfo(utils.MD5, f.Etag)
}
@@ -73,7 +81,9 @@ func (f File) GetName() string {
}
func (f File) CreateTime() time.Time {
parsedTime, err := time.Parse("2006-01-02 15:04:05", f.CreateAt)
// 返回的时间没有时区信息,默认 UTC+8
loc := time.FixedZone("UTC+8", 8*60*60)
parsedTime, err := time.ParseInLocation("2006-01-02 15:04:05", f.CreateAt, loc)
if err != nil {
return time.Now()
}
@@ -81,7 +91,9 @@ func (f File) CreateTime() time.Time {
}
func (f File) ModTime() time.Time {
parsedTime, err := time.Parse("2006-01-02 15:04:05", f.UpdateAt)
// 返回的时间没有时区信息,默认 UTC+8
loc := time.FixedZone("UTC+8", 8*60*60)
parsedTime, err := time.ParseInLocation("2006-01-02 15:04:05", f.UpdateAt, loc)
if err != nil {
return time.Now()
}
@@ -113,29 +125,32 @@ type AccessTokenResp struct {
}
type RefreshTokenResp struct {
AccessToken string `json:"access_token"`
ExpiresIn int `json:"expires_in"`
RefreshToken string `json:"refresh_token"`
Scope string `json:"scope"`
TokenType string `json:"token_type"`
AccessToken string `json:"access_token"`
RefreshToken string `json:"refresh_token"`
ExpiresIn int64 `json:"expires_in"`
Code int `json:"code"`
Message string `json:"message"`
ErrorDescription string `json:"error_description"`
Error string `json:"error"`
Text string `json:"text"`
}
type UserInfoResp struct {
BaseResp
Data struct {
UID int64 `json:"uid"`
Username string `json:"username"`
DisplayName string `json:"displayName"`
HeadImage string `json:"headImage"`
Passport string `json:"passport"`
Mail string `json:"mail"`
SpaceUsed int64 `json:"spaceUsed"`
SpacePermanent int64 `json:"spacePermanent"`
SpaceTemp int64 `json:"spaceTemp"`
SpaceTempExpr string `json:"spaceTempExpr"`
Vip bool `json:"vip"`
DirectTraffic int64 `json:"directTraffic"`
IsHideUID bool `json:"isHideUID"`
UID uint64 `json:"uid"`
// Username string `json:"username"`
// DisplayName string `json:"displayName"`
// HeadImage string `json:"headImage"`
// Passport string `json:"passport"`
// Mail string `json:"mail"`
SpaceUsed int64 `json:"spaceUsed"`
SpacePermanent int64 `json:"spacePermanent"`
SpaceTemp int64 `json:"spaceTemp"`
// SpaceTempExpr int64 `json:"spaceTempExpr"`
// Vip bool `json:"vip"`
// DirectTraffic int64 `json:"directTraffic"`
// IsHideUID bool `json:"isHideUID"`
} `json:"data"`
}
@@ -154,52 +169,53 @@ type DownloadInfoResp struct {
} `json:"data"`
}
type DirectLinkResp struct {
BaseResp
Data struct {
URL string `json:"url"`
} `json:"data"`
}
// 创建文件V2返回
type UploadCreateResp struct {
BaseResp
Data struct {
FileID int64 `json:"fileID"`
PreuploadID string `json:"preuploadID"`
Reuse bool `json:"reuse"`
SliceSize int64 `json:"sliceSize"`
FileID int64 `json:"fileID"`
PreuploadID string `json:"preuploadID"`
Reuse bool `json:"reuse"`
SliceSize int64 `json:"sliceSize"`
Servers []string `json:"servers"`
} `json:"data"`
}
type UploadUrlResp struct {
BaseResp
Data struct {
PresignedURL string `json:"presignedURL"`
}
}
// 上传完毕V2返回
type UploadCompleteResp struct {
BaseResp
Data struct {
Async bool `json:"async"`
Completed bool `json:"completed"`
FileID int64 `json:"fileID"`
} `json:"data"`
}
type UploadAsyncResp struct {
type SHA1ReuseResp struct {
BaseResp
Data struct {
Completed bool `json:"completed"`
FileID int64 `json:"fileID"`
FileID int64 `json:"fileID"`
Reuse bool `json:"reuse"`
} `json:"data"`
}
type UploadResp struct {
type OfflineDownloadResp struct {
BaseResp
Data struct {
AccessKeyId string `json:"AccessKeyId"`
Bucket string `json:"Bucket"`
Key string `json:"Key"`
SecretAccessKey string `json:"SecretAccessKey"`
SessionToken string `json:"SessionToken"`
FileId int64 `json:"FileId"`
Reuse bool `json:"Reuse"`
EndPoint string `json:"EndPoint"`
StorageNode string `json:"StorageNode"`
UploadId string `json:"UploadId"`
TaskID int `json:"taskID"`
} `json:"data"`
}
type OfflineDownloadProcessResp struct {
BaseResp
Data struct {
Process float64 `json:"process"`
Status int `json:"status"`
} `json:"data"`
}
+143 -91
View File
@@ -1,21 +1,28 @@
package _123_open
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"strconv"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/errgroup"
"github.com/OpenListTeam/OpenList/v4/pkg/http_range"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/avast/retry-go"
"github.com/go-resty/resty/v2"
)
// 创建文件 V2
func (d *Open123) create(parentFileID int64, filename string, etag string, size int64, duplicate int, containDir bool) (*UploadCreateResp, error) {
var resp UploadCreateResp
_, err := d.Request(UploadCreate, http.MethodPost, func(req *resty.Request) {
@@ -34,21 +41,136 @@ func (d *Open123) create(parentFileID int64, filename string, etag string, size
return &resp, nil
}
func (d *Open123) url(preuploadID string, sliceNo int64) (string, error) {
// get upload url
var resp UploadUrlResp
_, err := d.Request(UploadUrl, http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"preuploadId": preuploadID,
"sliceNo": sliceNo,
})
}, &resp)
// 上传分片 V2
func (d *Open123) Upload(ctx context.Context, file model.FileStreamer, createResp *UploadCreateResp, up driver.UpdateProgress) error {
uploadDomain := createResp.Data.Servers[0]
size := file.GetSize()
chunkSize := createResp.Data.SliceSize
ss, err := stream.NewStreamSectionReader(file, int(chunkSize), &up)
if err != nil {
return "", err
return err
}
return resp.Data.PresignedURL, nil
uploadNums := (size + chunkSize - 1) / chunkSize
thread := min(int(uploadNums), d.UploadThread)
threadG, uploadCtx := errgroup.NewOrderedGroupWithContext(ctx, thread,
retry.Attempts(3),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay))
for partIndex := range uploadNums {
if utils.IsCanceled(uploadCtx) {
break
}
partIndex := partIndex
partNumber := partIndex + 1 // 分片号从1开始
offset := partIndex * chunkSize
size := min(chunkSize, size-offset)
var reader io.ReadSeeker
var rateLimitedRd io.Reader
sliceMD5 := ""
// 表单
b := bytes.NewBuffer(make([]byte, 0, 2048))
threadG.GoWithLifecycle(errgroup.Lifecycle{
Before: func(ctx context.Context) (err error) {
reader, err = ss.GetSectionReader(offset, size)
return
},
Do: func(ctx context.Context) (err error) {
reader.Seek(0, io.SeekStart)
if sliceMD5 == "" {
// 把耗时的计算放在这里,避免阻塞其他协程
sliceMD5, err = utils.HashReader(utils.MD5, reader)
if err != nil {
return err
}
reader.Seek(0, io.SeekStart)
}
b.Reset()
w := multipart.NewWriter(b)
// 添加表单字段
err = w.WriteField("preuploadID", createResp.Data.PreuploadID)
if err != nil {
return err
}
err = w.WriteField("sliceNo", strconv.FormatInt(partNumber, 10))
if err != nil {
return err
}
err = w.WriteField("sliceMD5", sliceMD5)
if err != nil {
return err
}
// 写入文件内容
_, err = w.CreateFormFile("slice", fmt.Sprintf("%s.part%d", file.GetName(), partNumber))
if err != nil {
return err
}
headSize := b.Len()
err = w.Close()
if err != nil {
return err
}
head := bytes.NewReader(b.Bytes()[:headSize])
tail := bytes.NewReader(b.Bytes()[headSize:])
rateLimitedRd = driver.NewLimitedUploadStream(ctx, io.MultiReader(head, reader, tail))
token, err := d.getAccessToken(false)
if err != nil {
return err
}
// 创建请求并设置header
req, err := http.NewRequestWithContext(ctx, http.MethodPost, uploadDomain+"/upload/v2/file/slice", rateLimitedRd)
if err != nil {
return err
}
// 设置请求头
req.Header.Add("Authorization", "Bearer "+token)
req.Header.Add("Content-Type", w.FormDataContentType())
req.Header.Add("Platform", "open_platform")
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode != 200 {
return fmt.Errorf("slice %d upload failed, status code: %d", partNumber, res.StatusCode)
}
b.Reset()
_, err = b.ReadFrom(res.Body)
if err != nil {
return err
}
var resp BaseResp
err = json.Unmarshal(b.Bytes(), &resp)
if err != nil {
return err
}
if resp.Code != 0 {
return fmt.Errorf("slice %d upload failed: %s", partNumber, resp.Message)
}
progress := 100 * float64(threadG.Success()+1) / float64(uploadNums+1)
up(progress)
return nil
},
After: func(err error) {
ss.FreeSectionReader(reader)
},
})
}
if err := threadG.Wait(); err != nil {
return err
}
return nil
}
// 上传完毕
func (d *Open123) complete(preuploadID string) (*UploadCompleteResp, error) {
var resp UploadCompleteResp
_, err := d.Request(UploadComplete, http.MethodPost, func(req *resty.Request) {
@@ -62,11 +184,16 @@ func (d *Open123) complete(preuploadID string) (*UploadCompleteResp, error) {
return &resp, nil
}
func (d *Open123) async(preuploadID string) (*UploadAsyncResp, error) {
var resp UploadAsyncResp
_, err := d.Request(UploadAsync, http.MethodPost, func(req *resty.Request) {
// SHA1 秒传
func (d *Open123) sha1Reuse(parentFileID int64, filename string, sha1Hash string, size int64, duplicate int) (*SHA1ReuseResp, error) {
var resp SHA1ReuseResp
_, err := d.Request(UploadSHA1Reuse, http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"preuploadID": preuploadID,
"parentFileID": parentFileID,
"filename": filename,
"sha1": strings.ToLower(sha1Hash),
"size": size,
"duplicate": duplicate,
})
}, &resp)
if err != nil {
@@ -74,78 +201,3 @@ func (d *Open123) async(preuploadID string) (*UploadAsyncResp, error) {
}
return &resp, nil
}
func (d *Open123) Upload(ctx context.Context, file model.FileStreamer, createResp *UploadCreateResp, up driver.UpdateProgress) error {
size := file.GetSize()
chunkSize := createResp.Data.SliceSize
uploadNums := (size + chunkSize - 1) / chunkSize
threadG, uploadCtx := errgroup.NewGroupWithContext(ctx, d.UploadThread,
retry.Attempts(3),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay))
for partIndex := int64(0); partIndex < uploadNums; partIndex++ {
if utils.IsCanceled(uploadCtx) {
return ctx.Err()
}
partIndex := partIndex
partNumber := partIndex + 1 // 分片号从1开始
offset := partIndex * chunkSize
size := min(chunkSize, size-offset)
limitedReader, err := file.RangeRead(http_range.Range{
Start: offset,
Length: size})
if err != nil {
return err
}
limitedReader = driver.NewLimitedUploadStream(ctx, limitedReader)
threadG.Go(func(ctx context.Context) error {
uploadPartUrl, err := d.url(createResp.Data.PreuploadID, partNumber)
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, "PUT", uploadPartUrl, limitedReader)
if err != nil {
return err
}
req = req.WithContext(ctx)
req.ContentLength = size
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
_ = res.Body.Close()
progress := 10.0 + 85.0*float64(threadG.Success())/float64(uploadNums)
up(progress)
return nil
})
}
if err := threadG.Wait(); err != nil {
return err
}
uploadCompleteResp, err := d.complete(createResp.Data.PreuploadID)
if err != nil {
return err
}
if uploadCompleteResp.Data.Async == false || uploadCompleteResp.Data.Completed {
return nil
}
for {
uploadAsyncResp, err := d.async(createResp.Data.PreuploadID)
if err != nil {
return err
}
if uploadAsyncResp.Data.Completed {
break
}
}
up(100)
return nil
}
+115 -54
View File
@@ -1,42 +1,51 @@
package _123_open
import (
"context"
"crypto/md5"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/go-resty/resty/v2"
"github.com/google/uuid"
log "github.com/sirupsen/logrus"
)
var ( //不同情况下获取的AccessTokenQPS限制不同 如下模块化易于拓展
var ( // 不同情况下获取的AccessTokenQPS限制不同 如下模块化易于拓展
Api = "https://open-api.123pan.com"
AccessToken = InitApiInfo(Api+"/api/v1/access_token", 1)
RefreshToken = InitApiInfo(Api+"/api/v1/oauth2/access_token", 1)
UserInfo = InitApiInfo(Api+"/api/v1/user/info", 1)
FileList = InitApiInfo(Api+"/api/v2/file/list", 4)
DownloadInfo = InitApiInfo(Api+"/api/v1/file/download_info", 0)
Mkdir = InitApiInfo(Api+"/upload/v1/file/mkdir", 2)
Move = InitApiInfo(Api+"/api/v1/file/move", 1)
Rename = InitApiInfo(Api+"/api/v1/file/name", 1)
Trash = InitApiInfo(Api+"/api/v1/file/trash", 2)
UploadCreate = InitApiInfo(Api+"/upload/v1/file/create", 2)
UploadUrl = InitApiInfo(Api+"/upload/v1/file/get_upload_url", 0)
UploadComplete = InitApiInfo(Api+"/upload/v1/file/upload_complete", 0)
UploadAsync = InitApiInfo(Api+"/upload/v1/file/upload_async_result", 1)
UserInfo = InitApiInfo(Api+"/api/v1/user/info", 1)
FileList = InitApiInfo(Api+"/api/v2/file/list", 3)
DownloadInfo = InitApiInfo(Api+"/api/v1/file/download_info", 5)
DirectLink = InitApiInfo(Api+"/api/v1/direct-link/url", 5)
Mkdir = InitApiInfo(Api+"/upload/v1/file/mkdir", 2)
Move = InitApiInfo(Api+"/api/v1/file/move", 1)
Rename = InitApiInfo(Api+"/api/v1/file/name", 1)
Trash = InitApiInfo(Api+"/api/v1/file/trash", 2)
UploadCreate = InitApiInfo(Api+"/upload/v2/file/create", 2)
UploadComplete = InitApiInfo(Api+"/upload/v2/file/upload_complete", 0)
UploadSHA1Reuse = InitApiInfo(Api+"/upload/v2/file/sha1_reuse", 2)
OfflineDownload = InitApiInfo(Api+"/api/v1/offline/download", 1)
OfflineDownloadProcess = InitApiInfo(Api+"/api/v1/offline/download/process", 5)
)
func (d *Open123) Request(apiInfo *ApiInfo, method string, callback base.ReqCallback, resp interface{}) ([]byte, error) {
retryToken := true
for {
token, err := d.getAccessToken(false)
if err != nil {
return nil, err
}
req := base.RestyClient.R()
req.SetHeaders(map[string]string{
"authorization": "Bearer " + d.AccessToken,
"authorization": "Bearer " + token,
"platform": "open_platform",
"Content-Type": "application/json",
})
@@ -66,9 +75,9 @@ func (d *Open123) Request(apiInfo *ApiInfo, method string, callback base.ReqCall
if baseResp.Code == 0 {
return body, nil
} else if baseResp.Code == 401 && retryToken {
retryToken = false
if err := d.flushAccessToken(); err != nil {
} else if baseResp.Code == 401 {
// 强制刷新Token, 有小概率会 race condition 导致多次刷新Token,但不影响正确运行
if _, err := d.getAccessToken(true); err != nil {
return nil, err
}
} else if baseResp.Code == 429 {
@@ -78,52 +87,59 @@ func (d *Open123) Request(apiInfo *ApiInfo, method string, callback base.ReqCall
return nil, errors.New(baseResp.Message)
}
}
}
func (d *Open123) flushAccessToken() error {
if d.Addition.ClientID != "" {
if d.Addition.ClientSecret != "" {
var resp AccessTokenResp
_, err := d.Request(AccessToken, http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"clientID": d.ClientID,
"clientSecret": d.ClientSecret,
})
}, &resp)
if err != nil {
return err
}
d.AccessToken = resp.Data.AccessToken
op.MustSaveDriverStorage(d)
} else if d.Addition.RefreshToken != "" {
var resp RefreshTokenResp
_, err := d.Request(RefreshToken, http.MethodPost, func(req *resty.Request) {
req.SetQueryParam("client_id", d.ClientID)
req.SetQueryParam("grant_type", "refresh_token")
req.SetQueryParam("refresh_token", d.Addition.RefreshToken)
}, &resp)
if err != nil {
return err
}
d.AccessToken = resp.AccessToken
d.RefreshToken = resp.RefreshToken
op.MustSaveDriverStorage(d)
}
func (d *Open123) SignURL(originURL, privateKey string, uid uint64, validDuration time.Duration) (newURL string, err error) {
// 生成Unix时间戳
ts := time.Now().Add(validDuration).Unix()
// 生成随机数(建议使用UUID,不能包含中划线(-))
rand := strings.ReplaceAll(uuid.New().String(), "-", "")
// 解析URL
objURL, err := url.Parse(originURL)
if err != nil {
return "", err
}
return nil
// 待签名字符串,格式:path-timestamp-rand-uid-privateKey
unsignedStr := fmt.Sprintf("%s-%d-%s-%d-%s", objURL.Path, ts, rand, uid, privateKey)
md5Hash := md5.Sum([]byte(unsignedStr))
// 生成鉴权参数,格式:timestamp-rand-uid-md5hash
authKey := fmt.Sprintf("%d-%s-%d-%x", ts, rand, uid, md5Hash)
// 添加鉴权参数到URL查询参数
v := objURL.Query()
v.Add("auth_key", authKey)
objURL.RawQuery = v.Encode()
return objURL.String(), nil
}
func (d *Open123) getUserInfo() (*UserInfoResp, error) {
func (d *Open123) getUserInfo(ctx context.Context) (*UserInfoResp, error) {
var resp UserInfoResp
if _, err := d.Request(UserInfo, http.MethodGet, nil, &resp); err != nil {
if _, err := d.Request(UserInfo, http.MethodGet, func(req *resty.Request) {
req.SetContext(ctx)
}, &resp); err != nil {
return nil, err
}
return &resp, nil
}
func (d *Open123) getUID(ctx context.Context) (uint64, error) {
if d.UID != 0 {
return d.UID, nil
}
resp, err := d.getUserInfo(ctx)
if err != nil {
return 0, err
}
d.UID = resp.Data.UID
return resp.Data.UID, nil
}
func (d *Open123) getFiles(parentFileId int64, limit int, lastFileId int64) (*FileListResp, error) {
var resp FileListResp
@@ -138,7 +154,6 @@ func (d *Open123) getFiles(parentFileId int64, limit int, lastFileId int64) (*Fi
"searchData": "",
})
}, &resp)
if err != nil {
return nil, err
}
@@ -161,6 +176,21 @@ func (d *Open123) getDownloadInfo(fileId int64) (*DownloadInfoResp, error) {
return &resp, nil
}
func (d *Open123) getDirectLink(fileId int64) (*DirectLinkResp, error) {
var resp DirectLinkResp
_, err := d.Request(DirectLink, http.MethodGet, func(req *resty.Request) {
req.SetQueryParams(map[string]string{
"fileID": strconv.FormatInt(fileId, 10),
})
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
func (d *Open123) mkdir(parentID int64, name string) error {
_, err := d.Request(Mkdir, http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
@@ -215,3 +245,34 @@ func (d *Open123) trash(fileId int64) error {
return nil
}
func (d *Open123) createOfflineDownloadTask(ctx context.Context, url string, dirID, callback string) (taskID int, err error) {
body := base.Json{
"url": url,
"dirID": dirID,
}
if len(callback) > 0 {
body["callBackUrl"] = callback
}
var resp OfflineDownloadResp
_, err = d.Request(OfflineDownload, http.MethodPost, func(req *resty.Request) {
req.SetBody(body)
}, &resp)
if err != nil {
return 0, err
}
return resp.Data.TaskID, nil
}
func (d *Open123) queryOfflineDownloadStatus(ctx context.Context, taskID int) (process float64, status int, err error) {
var resp OfflineDownloadProcessResp
_, err = d.Request(OfflineDownloadProcess, http.MethodGet, func(req *resty.Request) {
req.SetQueryParams(map[string]string{
"taskID": strconv.Itoa(taskID),
})
}, &resp)
if err != nil {
return .0, 0, err
}
return resp.Data.Process, resp.Data.Status, nil
}
+1
View File
@@ -19,6 +19,7 @@ var config = driver.Config{
LocalSort: true,
NoUpload: true,
DefaultRoot: "0",
PreferProxy: true,
}
func init() {
+1 -1
View File
@@ -24,7 +24,7 @@ type File struct {
}
func (f File) GetHash() utils.HashInfo {
return utils.HashInfo{}
return utils.NewHashInfo(utils.MD5, f.Etag)
}
func (f File) GetPath() string {
+195 -83
View File
@@ -14,6 +14,7 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
streamPkg "github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/cron"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
@@ -28,6 +29,7 @@ type Yun139 struct {
Account string
ref *Yun139
PersonalCloudHost string
RootPath string
}
func (d *Yun139) Config() driver.Config {
@@ -41,7 +43,16 @@ func (d *Yun139) GetAddition() driver.Additional {
func (d *Yun139) Init(ctx context.Context) error {
if d.ref == nil {
if len(d.Authorization) == 0 {
return fmt.Errorf("authorization is empty")
if d.Username != "" && d.Password != "" {
log.Infof("139yun: authorization is empty, trying to login with password.")
newAuth, err := d.loginWithPassword()
log.Debugf("newAuth: Ok: %s", newAuth)
if err != nil {
return fmt.Errorf("login with password failed: %w", err)
}
} else {
return fmt.Errorf("authorization is empty and username/password is not provided")
}
}
err := d.refreshToken()
if err != nil {
@@ -54,7 +65,8 @@ func (d *Yun139) Init(ctx context.Context) error {
"userInfo": base.Json{
"userType": 1,
"accountType": 1,
"accountName": d.Account},
"accountName": d.Account,
},
"modAddrType": 1,
}, &resp)
if err != nil {
@@ -91,7 +103,22 @@ func (d *Yun139) Init(ctx context.Context) error {
if len(d.Addition.RootFolderID) == 0 {
d.RootFolderID = d.CloudID
}
_, err := d.groupGetFiles(d.RootFolderID)
if err != nil {
return err
}
case MetaFamily:
if len(d.Addition.RootFolderID) == 0 {
// Attempt to obtain data.path as the root via a query and persist it.
if root, err := d.getFamilyRootPath(d.CloudID); err == nil && root != "" {
d.RootFolderID = root
op.MustSaveDriverStorage(d)
}
}
_, err := d.familyGetFiles(d.RootFolderID)
if err != nil {
return err
}
default:
return errs.NotImplement
}
@@ -278,6 +305,42 @@ func (d *Yun139) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.Obj,
return nil, err
}
return srcObj, nil
case MetaFamily:
pathname := "/isbo/openApi/createBatchOprTask"
var contentList []string
var catalogList []string
if srcObj.IsDir() {
catalogList = append(catalogList, path.Join(srcObj.GetPath(), srcObj.GetID()))
} else {
contentList = append(contentList, path.Join(srcObj.GetPath(), srcObj.GetID()))
}
body := base.Json{
"catalogList": catalogList,
"accountInfo": base.Json{
"accountName": d.getAccount(),
"accountType": "1",
},
"contentList": contentList,
"destCatalogID": dstDir.GetID(),
"destGroupID": d.CloudID,
"destPath": path.Join(dstDir.GetPath(), dstDir.GetID()),
"destType": 0,
"srcGroupID": d.CloudID,
"srcType": 0,
"taskType": 3,
}
var resp CreateBatchOprTaskResp
_, err := d.isboPost(pathname, body, &resp)
if err != nil {
return nil, err
}
log.Debugf("[139] Move MetaFamily CreateBatchOprTaskResp.Result.ResultCode: %s", resp.Result.ResultCode)
if resp.Result.ResultCode != "0" {
return nil, fmt.Errorf("failed to move in family cloud: %s", resp.Result.ResultDesc)
}
return srcObj, nil
default:
return nil, errs.NotImplement
}
@@ -352,19 +415,27 @@ func (d *Yun139) Rename(ctx context.Context, srcObj model.Obj, newName string) e
var data base.Json
var pathname string
if srcObj.IsDir() {
// 网页接口不支持重命名家庭云文件夹
// data = base.Json{
// "catalogType": 3,
// "catalogID": srcObj.GetID(),
// "catalogName": newName,
// "commonAccountInfo": base.Json{
// "account": d.getAccount(),
// "accountType": 1,
// },
// "path": srcObj.GetPath(),
// }
// pathname = "/orchestration/familyCloud-rebuild/photoContent/v1.0/modifyCatalogInfo"
return errs.NotImplement
pathname = "/modifyCloudDocV2"
data = base.Json{
"catalogType": 3,
"cloudID": d.CloudID,
"commonAccountInfo": base.Json{
"account": d.getAccount(),
"accountType": "1",
},
"docLibName": newName,
"docLibraryID": srcObj.GetID(),
"path": path.Join(srcObj.GetPath(), srcObj.GetID()),
}
var resp ModifyCloudDocV2Resp
_, err = d.andAlbumRequest(pathname, data, &resp)
if err != nil {
return err
}
if resp.Result.ResultCode != "0" {
return fmt.Errorf("failed to rename family folder: %s", resp.Result.ResultDesc)
}
return nil
} else {
data = base.Json{
"contentID": srcObj.GetID(),
@@ -420,6 +491,33 @@ func (d *Yun139) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
}
pathname := "/orchestration/personalCloud/batchOprTask/v1.0/createBatchOprTask"
_, err = d.post(pathname, data, nil)
case MetaGroup:
err = d.handleMetaGroupCopy(ctx, srcObj, dstDir)
case MetaFamily:
pathname := "/copyContentCatalog"
var sourceContentIDs []string
var sourceCatalogIDs []string
if srcObj.IsDir() {
sourceCatalogIDs = append(sourceCatalogIDs, srcObj.GetID())
} else {
sourceContentIDs = append(sourceContentIDs, srcObj.GetID())
}
body := base.Json{
"commonAccountInfo": base.Json{
"accountType": "1",
"accountUserId": d.ref.UserDomainID,
},
"destCatalogID": dstDir.GetID(),
"destCloudID": d.CloudID,
"sourceCatalogIDs": sourceCatalogIDs,
"sourceCloudID": d.CloudID,
"sourceContentIDs": sourceContentIDs,
}
var resp base.Json // Assuming a generic JSON response for success/failure
_, err = d.andAlbumRequest(pathname, body, &resp)
// For now, we assume no error means success.
default:
err = errs.NotImplement
}
@@ -522,32 +620,27 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
var err error
fullHash := stream.GetHash().GetHash(utils.SHA256)
if len(fullHash) != utils.SHA256.Width {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
_, fullHash, err = streamPkg.CacheFullInTempFileAndHash(stream, cacheFileProgress, utils.SHA256)
_, fullHash, err = streamPkg.CacheFullAndHash(stream, &up, utils.SHA256)
if err != nil {
return err
}
}
size := stream.GetSize()
var partSize = d.getPartSize(size)
part := size / partSize
if size%partSize > 0 {
part++
} else if part == 0 {
part = 1
partSize := d.getPartSize(size)
part := int64(1)
if size > partSize {
part = (size + partSize - 1) / partSize
}
// 生成所有 partInfos
partInfos := make([]PartInfo, 0, part)
for i := int64(0); i < part; i++ {
if utils.IsCanceled(ctx) {
return ctx.Err()
}
start := i * partSize
byteSize := size - start
if byteSize > partSize {
byteSize = partSize
}
byteSize := min(size-start, partSize)
partNumber := i + 1
partInfo := PartInfo{
PartNumber: partNumber,
@@ -595,17 +688,20 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
// resp.Data.RapidUpload: true 支持快传,但此处直接检测是否返回分片的上传地址
// 快传的情况下同样需要手动处理冲突
if resp.Data.PartInfos != nil {
// 读取前100个分片的上传地址
uploadPartInfos := resp.Data.PartInfos
// Progress
p := driver.NewProgress(size, up)
rateLimited := driver.NewLimitedUploadStream(ctx, stream)
// 获取后续分片的上传地址
for i := 101; i < len(partInfos); i += 100 {
end := i + 100
if end > len(partInfos) {
end = len(partInfos)
}
// 先上传前100个分片
err = d.uploadPersonalParts(ctx, partInfos, resp.Data.PartInfos, rateLimited, p)
if err != nil {
return err
}
// 如果还有剩余分片,分批获取上传地址并上传
for i := 100; i < len(partInfos); i += 100 {
end := min(i+100, len(partInfos))
batchPartInfos := partInfos[i:end]
moredata := base.Json{
"fileId": resp.Data.FileId,
"uploadId": resp.Data.UploadId,
@@ -621,45 +717,13 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
if err != nil {
return err
}
uploadPartInfos = append(uploadPartInfos, moreresp.Data.PartInfos...)
}
// Progress
p := driver.NewProgress(size, up)
rateLimited := driver.NewLimitedUploadStream(ctx, stream)
// 上传所有分片
for _, uploadPartInfo := range uploadPartInfos {
index := uploadPartInfo.PartNumber - 1
partSize := partInfos[index].PartSize
log.Debugf("[139] uploading part %+v/%+v", index, len(uploadPartInfos))
limitReader := io.LimitReader(rateLimited, partSize)
// Update Progress
r := io.TeeReader(limitReader, p)
req, err := http.NewRequest("PUT", uploadPartInfo.UploadUrl, r)
err = d.uploadPersonalParts(ctx, partInfos, moreresp.Data.PartInfos, rateLimited, p)
if err != nil {
return err
}
req = req.WithContext(ctx)
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("Content-Length", fmt.Sprint(partSize))
req.Header.Set("Origin", "https://yun.139.com")
req.Header.Set("Referer", "https://yun.139.com/")
req.ContentLength = partSize
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
_ = res.Body.Close()
log.Debugf("[139] uploaded: %+v", res)
if res.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status code: %d", res.StatusCode)
}
}
// 全部分片上传完毕后,complete
data = base.Json{
"contentHash": fullHash,
"contentHashAlgorithm": "SHA256",
@@ -713,6 +777,8 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
return nil
case MetaPersonal:
fallthrough
case MetaGroup:
fallthrough
case MetaFamily:
// 处理冲突
// 获取文件列表
@@ -760,13 +826,18 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
},
}
pathname := "/orchestration/personalCloud/uploadAndDownload/v1.0/pcUploadFileRequest"
if d.isFamily() {
if d.isFamily() || d.Addition.Type == MetaGroup {
uploadPath := path.Join(dstDir.GetPath(), dstDir.GetID())
// if dstDir is root folder
if dstDir.GetID() == d.RootFolderID {
uploadPath = d.RootPath
}
data = d.newJson(base.Json{
"fileCount": 1,
"manualRename": 2,
"operation": 0,
"path": path.Join(dstDir.GetPath(), dstDir.GetID()),
"seqNo": random.String(32), //序列号不能为空
"path": uploadPath,
"seqNo": random.String(32), // 序列号不能为空
"totalSize": reportSize,
"uploadContentList": []base.Json{{
"contentName": stream.GetName(),
@@ -777,6 +848,7 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
pathname = "/orchestration/familyCloud-rebuild/content/v1.0/getFileUploadURL"
}
var resp UploadResp
log.Debugf("[139] upload request body: %+v", data)
_, err = d.post(pathname, data, &resp)
if err != nil {
return err
@@ -788,12 +860,10 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
size := stream.GetSize()
// Progress
p := driver.NewProgress(size, up)
var partSize = d.getPartSize(size)
part := size / partSize
if size%partSize > 0 {
part++
} else if part == 0 {
part = 1
partSize := d.getPartSize(size)
part := int64(1)
if size > partSize {
part = (size + partSize - 1) / partSize
}
rateLimited := driver.NewLimitedUploadStream(ctx, stream)
for i := int64(0); i < part; i++ {
@@ -807,12 +877,10 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
limitReader := io.LimitReader(rateLimited, byteSize)
// Update Progress
r := io.TeeReader(limitReader, p)
req, err := http.NewRequest("POST", resp.Data.UploadResult.RedirectionURL, r)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, resp.Data.UploadResult.RedirectionURL, r)
if err != nil {
return err
}
req = req.WithContext(ctx)
req.Header.Set("Content-Type", "text/plain;name="+unicode(stream.GetName()))
req.Header.Set("contentSize", strconv.FormatInt(size, 10))
req.Header.Set("range", fmt.Sprintf("bytes=%d-%d", start, start+byteSize-1))
@@ -872,4 +940,48 @@ func (d *Yun139) Other(ctx context.Context, args model.OtherArgs) (interface{},
}
}
func (d *Yun139) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
if d.UserDomainID == "" {
return nil, errs.NotImplement
}
var total, used int64
if d.isFamily() {
diskInfo, err := d.getFamilyDiskInfo(ctx)
if err != nil {
return nil, err
}
totalMb, err := strconv.ParseInt(diskInfo.Data.DiskSize, 10, 64)
if err != nil {
return nil, fmt.Errorf("failed convert disk size into integer: %+v", err)
}
usedMb, err := strconv.ParseInt(diskInfo.Data.UsedSize, 10, 64)
if err != nil {
return nil, fmt.Errorf("failed convert used size into integer: %+v", err)
}
total = totalMb * 1024 * 1024
used = usedMb * 1024 * 1024
} else {
diskInfo, err := d.getPersonalDiskInfo(ctx)
if err != nil {
return nil, err
}
totalMb, err := strconv.ParseInt(diskInfo.Data.DiskSize, 10, 64)
if err != nil {
return nil, fmt.Errorf("failed convert disk size into integer: %+v", err)
}
freeMb, err := strconv.ParseInt(diskInfo.Data.FreeDiskSize, 10, 64)
if err != nil {
return nil, fmt.Errorf("failed convert free size into integer: %+v", err)
}
total = totalMb * 1024 * 1024
used = total - (freeMb * 1024 * 1024)
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: total,
UsedSpace: used,
},
}, nil
}
var _ driver.Driver = (*Yun139)(nil)
+4
View File
@@ -8,9 +8,13 @@ import (
type Addition struct {
//Account string `json:"account" required:"true"`
Authorization string `json:"authorization" type:"text" required:"true"`
Username string `json:"username" required:"true"`
Password string `json:"password" required:"true" secret:"true"`
MailCookies string `json:"mail_cookies" required:"true" type:"text" help:"Cookies from mail.139.com used for login authentication."`
driver.RootID
Type string `json:"type" type:"select" options:"personal_new,family,group,personal" default:"personal_new"`
CloudID string `json:"cloud_id"`
UserDomainID string `json:"user_domain_id" help:"ud_id in Cookie, fill in to show disk usage"`
CustomUploadPartSize int64 `json:"custom_upload_part_size" type:"number" default:"0" help:"0 for auto"`
ReportRealSize bool `json:"report_real_size" type:"bool" default:"true" help:"Enable to report the real file size during upload"`
UseLargeThumbnail bool `json:"use_large_thumbnail" type:"bool" default:"false" help:"Enable to use large thumbnail for images"`
+76
View File
@@ -312,3 +312,79 @@ type RefreshTokenResp struct {
AccessToken string `xml:"accessToken"`
Desc string `xml:"desc"`
}
type PersonalDiskInfoResp struct {
BaseResp
Data struct {
FreeDiskSize string `json:"freeDiskSize"`
DiskSize string `json:"diskSize"`
IsInfinitePicStorage *bool `json:"isInfinitePicStorage"`
} `json:"data"`
}
type FamilyDiskInfoResp struct {
BaseResp
Data struct {
UsedSize string `json:"usedSize"`
DiskSize string `json:"diskSize"`
} `json:"data"`
}
type AndAlbumUploadResp struct {
Result struct {
ResultCode string `json:"resultCode"`
ResultDesc string `json:"resultDesc"`
} `json:"result"`
UploadResult struct {
UploadTaskID string `json:"uploadTaskID"`
RedirectionURL string `json:"redirectionUrl"`
NewContentIDList []struct {
ContentID string `json:"contentID"`
ContentName string `json:"contentName"`
} `json:"newContentIDList"`
} `json:"uploadResult"`
}
type ModifyCloudDocV2Req struct {
CatalogType int `json:"catalogType"`
CloudID string `json:"cloudID"`
CommonAccountInfo struct {
Account string `json:"account"`
AccountType string `json:"accountType"`
} `json:"commonAccountInfo"`
DocLibName string `json:"docLibName"`
DocLibraryID string `json:"docLibraryID"`
Path string `json:"path"`
}
type ModifyCloudDocV2Resp struct {
Result struct {
ResultCode string `json:"resultCode"`
ResultDesc string `json:"resultDesc"`
} `json:"result"`
}
type CreateBatchOprTaskReq struct {
CatalogList []string `json:"catalogList"`
CommonAccountInfo struct {
Account string `json:"account"`
AccountType string `json:"accountType"`
} `json:"commonAccountInfo"`
ContentList []string `json:"contentList"`
DestCatalogID string `json:"destCatalogID"`
DestGroupID string `json:"destGroupID"`
DestPath string `json:"destPath"`
DestType int `json:"destType"`
SourceCatalogType int `json:"sourceCatalogType"`
SourceCloudID string `json:"sourceCloudID"`
SourceType int `json:"sourceType"`
TaskType int `json:"taskType"`
}
type CreateBatchOprTaskResp struct {
Result struct {
ResultCode string `json:"resultCode"`
ResultDesc string `json:"resultDesc"`
} `json:"result"`
TaskID string `json:"taskID"`
}
+791 -17
View File
@@ -1,18 +1,29 @@
package _139
import (
"bytes"
"context"
"crypto/aes"
"crypto/cipher"
"crypto/md5"
crypto_rand "crypto/rand"
"crypto/sha1"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"path"
"regexp"
"sort"
"strconv"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
@@ -22,6 +33,11 @@ import (
log "github.com/sirupsen/logrus"
)
const (
KEY_HEX_1 = "73634235495062495331515373756c734e7253306c673d3d" // 第一层 AES 解密密钥
KEY_HEX_2 = "7150714477323633586746674c337538" // 第二层 AES 解密密钥
)
// do others that not defined in Driver interface
func (d *Yun139) isFamily() bool {
return d.Type == "family"
@@ -93,19 +109,22 @@ func (d *Yun139) refreshToken() error {
SetBody(reqBody).
SetResult(&resp).
Post(url)
if err != nil {
return err
}
if resp.Return != "0" {
return fmt.Errorf("failed to refresh token: %s", resp.Desc)
if err != nil || resp.Return != "0" {
log.Warnf("139yun: failed to refresh token with old token: %v, desc: %s. trying to login with password.", err, resp.Desc)
newAuth, loginErr := d.loginWithPassword()
log.Debugf("newAuth: Ok: %s", newAuth)
if loginErr != nil {
return fmt.Errorf("failed to login with password after refresh failed: %w", loginErr)
}
return nil
}
d.Authorization = base64.StdEncoding.EncodeToString([]byte(splits[0] + ":" + splits[1] + ":" + resp.Token))
op.MustSaveDriverStorage(d)
return nil
}
func (d *Yun139) request(pathname string, method string, callback base.ReqCallback, resp interface{}) ([]byte, error) {
url := "https://yun.139.com" + pathname
func (d *Yun139) request(url string, method string, callback base.ReqCallback, resp interface{}) ([]byte, error) {
req := base.RestyClient.R()
randStr := random.String(16)
ts := time.Now().Format("2006-01-02 15:04:05")
@@ -144,10 +163,29 @@ func (d *Yun139) request(pathname string, method string, callback base.ReqCallba
var e BaseResp
req.SetResult(&e)
log.Debugf("[139] request: %s %s, body: %s", method, url, string(body))
res, err := req.Execute(method, url)
log.Debugln(res.String())
if err != nil {
log.Debugf("[139] request error: %v", err)
return nil, err
}
log.Debugf("[139] response body: %s", res.String())
if !e.Success {
return nil, errors.New(e.Message)
// Always try to unmarshal to the specific response type first if 'resp' is provided.
if resp != nil {
err = utils.Json.Unmarshal(res.Body(), resp)
if err != nil {
log.Debugf("[139] failed to unmarshal response to specific type: %v", err)
return nil, err // Return unmarshal error
}
if createBatchOprTaskResp, ok := resp.(*CreateBatchOprTaskResp); ok {
log.Debugf("[139] CreateBatchOprTaskResp.Result.ResultCode: %s", createBatchOprTaskResp.Result.ResultCode)
if createBatchOprTaskResp.Result.ResultCode == "0" {
goto SUCCESS_PROCESS
}
}
}
return nil, errors.New(e.Message) // Fallback to original error if not handled
}
if resp != nil {
err = utils.Json.Unmarshal(res.Body(), resp)
@@ -155,6 +193,7 @@ func (d *Yun139) request(pathname string, method string, callback base.ReqCallba
return nil, err
}
}
SUCCESS_PROCESS:
return res.Body(), nil
}
@@ -216,7 +255,7 @@ func (d *Yun139) requestRoute(data interface{}, resp interface{}) ([]byte, error
}
func (d *Yun139) post(pathname string, data interface{}, resp interface{}) ([]byte, error) {
return d.request(pathname, http.MethodPost, func(req *resty.Request) {
return d.request("https://yun.139.com"+pathname, http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
}, resp)
}
@@ -265,7 +304,7 @@ func (d *Yun139) getFiles(catalogID string) ([]model.Obj, error) {
HashInfo: utils.NewHashInfo(utils.MD5, content.Digest),
},
Thumbnail: model.Thumbnail{Thumbnail: content.ThumbnailURL},
//Thumbnail: content.BigthumbnailURL,
// Thumbnail: content.BigthumbnailURL,
}
files = append(files, &f)
}
@@ -309,6 +348,9 @@ func (d *Yun139) familyGetFiles(catalogID string) ([]model.Obj, error) {
return nil, err
}
path := resp.Data.Path
if catalogID == d.RootFolderID {
d.RootPath = path
}
for _, catalog := range resp.Data.CloudCatalogList {
f := model.Object{
ID: catalog.CatalogID,
@@ -332,7 +374,7 @@ func (d *Yun139) familyGetFiles(catalogID string) ([]model.Obj, error) {
Path: path, // 文件所在目录的Path
},
Thumbnail: model.Thumbnail{Thumbnail: content.ThumbnailURL},
//Thumbnail: content.BigthumbnailURL,
// Thumbnail: content.BigthumbnailURL,
}
files = append(files, &f)
}
@@ -364,6 +406,9 @@ func (d *Yun139) groupGetFiles(catalogID string) ([]model.Obj, error) {
return nil, err
}
path := resp.Data.GetGroupContentResult.ParentCatalogID
if catalogID == d.RootFolderID {
d.RootPath = path
}
for _, catalog := range resp.Data.GetGroupContentResult.CatalogList {
f := model.Object{
ID: catalog.CatalogID,
@@ -387,7 +432,7 @@ func (d *Yun139) groupGetFiles(catalogID string) ([]model.Obj, error) {
Path: path, // 文件所在目录的Path
},
Thumbnail: model.Thumbnail{Thumbnail: content.ThumbnailURL},
//Thumbnail: content.BigthumbnailURL,
// Thumbnail: content.BigthumbnailURL,
}
files = append(files, &f)
}
@@ -415,6 +460,7 @@ func (d *Yun139) getLink(contentId string) (string, error) {
}
return jsoniter.Get(res, "data", "downloadURL").ToString(), nil
}
func (d *Yun139) familyGetLink(contentId string, path string) (string, error) {
data := d.newJson(base.Json{
"contentID": contentId,
@@ -491,11 +537,13 @@ func (d *Yun139) personalRequest(pathname string, method string, callback base.R
var e BaseResp
req.SetResult(&e)
log.Debugf("[139] personal request: %s %s, body: %s", method, url, string(body))
res, err := req.Execute(method, url)
if err != nil {
log.Debugf("[139] personal request error: %v", err)
return nil, err
}
log.Debugln(res.String())
log.Debugf("[139] personal response body: %s", res.String())
if !e.Success {
return nil, errors.New(e.Message)
}
@@ -507,12 +555,20 @@ func (d *Yun139) personalRequest(pathname string, method string, callback base.R
}
return res.Body(), nil
}
func (d *Yun139) personalPost(pathname string, data interface{}, resp interface{}) ([]byte, error) {
return d.personalRequest(pathname, http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
}, resp)
}
func (d *Yun139) isboPost(pathname string, data interface{}, resp interface{}) ([]byte, error) {
url := "https://group.yun.139.com/hcy/mutual/adapter" + pathname
return d.request(url, http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
}, resp)
}
func getPersonalTime(t string) time.Time {
stamp, err := time.ParseInLocation("2006-01-02T15:04:05.999-07:00", t, utils.CNLoc)
if err != nil {
@@ -542,7 +598,7 @@ func (d *Yun139) personalGetFiles(fileId string) ([]model.Obj, error) {
}
nextPageCursor = resp.Data.NextPageCursor
for _, item := range resp.Data.Items {
var isFolder = (item.Type == "folder")
isFolder := (item.Type == "folder")
var f model.Obj
if isFolder {
f = &model.Object{
@@ -554,7 +610,7 @@ func (d *Yun139) personalGetFiles(fileId string) ([]model.Obj, error) {
IsFolder: isFolder,
}
} else {
var Thumbnails = item.Thumbnails
Thumbnails := item.Thumbnails
var ThumbnailUrl string
if d.UseLargeThumbnail {
for _, thumb := range Thumbnails {
@@ -597,7 +653,7 @@ func (d *Yun139) personalGetLink(fileId string) (string, error) {
if err != nil {
return "", err
}
var cdnUrl = jsoniter.Get(res, "data", "cdnUrl").ToString()
cdnUrl := jsoniter.Get(res, "data", "cdnUrl").ToString()
if cdnUrl != "" {
return cdnUrl, nil
} else {
@@ -611,15 +667,733 @@ func (d *Yun139) getAuthorization() string {
}
return d.Authorization
}
func (d *Yun139) getAccount() string {
if d.ref != nil {
return d.ref.getAccount()
}
return d.Account
}
func (d *Yun139) getPersonalCloudHost() string {
if d.ref != nil {
return d.ref.getPersonalCloudHost()
}
return d.PersonalCloudHost
}
func (d *Yun139) uploadPersonalParts(ctx context.Context, partInfos []PartInfo, uploadPartInfos []PersonalPartInfo, rateLimited *driver.RateLimitReader, p *driver.Progress) error {
// 确保数组以 PartNumber 从小到大排序
sort.Slice(uploadPartInfos, func(i, j int) bool {
return uploadPartInfos[i].PartNumber < uploadPartInfos[j].PartNumber
})
for _, uploadPartInfo := range uploadPartInfos {
index := uploadPartInfo.PartNumber - 1
if index < 0 || index >= len(partInfos) {
return fmt.Errorf("invalid PartNumber %d: index out of bounds (partInfos length: %d)", uploadPartInfo.PartNumber, len(partInfos))
}
partSize := partInfos[index].PartSize
log.Debugf("[139] uploading part %+v/%+v", index, len(partInfos))
limitReader := io.LimitReader(rateLimited, partSize)
r := io.TeeReader(limitReader, p)
req, err := http.NewRequestWithContext(ctx, http.MethodPut, uploadPartInfo.UploadUrl, r)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("Content-Length", fmt.Sprint(partSize))
req.Header.Set("Origin", "https://yun.139.com")
req.Header.Set("Referer", "https://yun.139.com/")
req.ContentLength = partSize
err = func() error {
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
log.Debugf("[139] uploaded: %+v", res)
if res.StatusCode != http.StatusOK {
body, _ := io.ReadAll(res.Body)
return fmt.Errorf("unexpected status code: %d, body: %s", res.StatusCode, string(body))
}
return nil
}()
if err != nil {
return err
}
}
return nil
}
func (d *Yun139) getPersonalDiskInfo(ctx context.Context) (*PersonalDiskInfoResp, error) {
data := map[string]interface{}{
"userDomainId": d.UserDomainID,
}
var resp PersonalDiskInfoResp
_, err := d.request("https://user-njs.yun.139.com/user/disk/getPersonalDiskInfo", http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
req.SetContext(ctx)
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
func (d *Yun139) getFamilyDiskInfo(ctx context.Context) (*FamilyDiskInfoResp, error) {
data := map[string]interface{}{
"userDomainId": d.UserDomainID,
}
var resp FamilyDiskInfoResp
_, err := d.request("https://user-njs.yun.139.com/user/disk/getFamilyDiskInfo", http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
req.SetContext(ctx)
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
func getMd5(dataStr string) string {
hash := md5.Sum([]byte(dataStr))
return fmt.Sprintf("%x", hash)
}
func (d *Yun139) step1_password_login() (string, error) {
log.Debugf("--- 执行步骤 1: 登录 API ---")
loginURL := "https://mail.10086.cn/Login/Login.ashx"
// 密码 SHA1 哈希
hashedPassword := sha1Hash(fmt.Sprintf("fetion.com.cn:%s", d.Password))
log.Debugf("DEBUG: 原始密码: %s", d.Password)
log.Debugf("DEBUG: SHA1 输入: fetion.com.cn:%s", d.Password)
log.Debugf("DEBUG: 生成的 Password 哈希: %s", hashedPassword)
cguid := strconv.FormatInt(time.Now().UnixMilli(), 10) // 随机生成 cguid
loginHeaders := map[string]string{
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"accept-language": "zh-CN,zh;q=0.9,zh-TW;q=0.8,en-US;q=0.7,en;q=0.6,en-GB;q=0.5",
"cache-control": "max-age=0",
"content-type": "application/x-www-form-urlencoded",
"dnt": "1",
"origin": "https://mail.10086.cn",
"priority": "u=0, i",
"referer": fmt.Sprintf("https://mail.10086.cn/default.html?&s=1&v=0&u=%s&m=1&ec=S001&resource=indexLogin&clientid=1003&auto=on&cguid=%s&mtime=45", base64.StdEncoding.EncodeToString([]byte(d.Username)), cguid),
"sec-ch-ua": "\"Microsoft Edge\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Windows\"",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "same-origin",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0",
"Cookie": d.MailCookies,
}
loginData := url.Values{}
loginData.Set("UserName", d.Username)
loginData.Set("passOld", "")
loginData.Set("auto", "on")
loginData.Set("Password", hashedPassword)
loginData.Set("webIndexPagePwdLogin", "1")
loginData.Set("pwdType", "1")
loginData.Set("clientId", "1003")
loginData.Set("authType", "2")
log.Debugf("DEBUG: 登录请求 URL: %s", loginURL)
log.Debugf("DEBUG: 登录请求 Headers: %+v", loginHeaders)
log.Debugf("DEBUG: 登录请求 Body: %s", loginData.Encode())
// 设置客户端不跟随重定向
client := base.RestyClient.SetRedirectPolicy(resty.NoRedirectPolicy())
res, err := client.R().
SetHeaders(loginHeaders).
SetFormDataFromValues(loginData).
Post(loginURL)
if err != nil {
// 如果是重定向错误,则不作为失败处理,因为我们禁止了自动重定向
if res != nil && res.StatusCode() >= 300 && res.StatusCode() < 400 {
log.Debugf("DEBUG: 登录响应 Status Code: %d (Redirect)", res.StatusCode())
} else {
return "", fmt.Errorf("step1 login request failed: %w", err)
}
} else {
log.Debugf("DEBUG: 登录响应 Status Code: %d", res.StatusCode())
}
// 恢复客户端的默认重定向策略,以免影响后续请求
base.RestyClient.SetRedirectPolicy(resty.FlexibleRedirectPolicy(10))
log.Debugf("DEBUG: 登录响应 Headers: %+v", res.Header())
var sid, extractedCguid string
// 从 Location 头部提取 sid 和 cguid
locationHeader := res.Header().Get("Location")
if locationHeader != "" {
sidMatch := regexp.MustCompile(`sid=([^&]+)`).FindStringSubmatch(locationHeader)
cguidMatch := regexp.MustCompile(`cguid=([^&]+)`).FindStringSubmatch(locationHeader)
if len(sidMatch) > 1 {
sid = sidMatch[1]
log.Debugf("DEBUG: 从 Location 提取到 sid: %s", sid)
}
if len(cguidMatch) > 1 {
extractedCguid = cguidMatch[1]
log.Debugf("DEBUG: 从 Location 提取到 cguid: %s", extractedCguid)
}
}
// 如果 Location 中没有,尝试从 Set-Cookie 中提取
if sid == "" || extractedCguid == "" {
setCookieHeaders := res.Header().Values("Set-Cookie")
for _, cookieStr := range setCookieHeaders {
ssoSidMatch := regexp.MustCompile(`Os_SSo_Sid=([^;]+)`).FindStringSubmatch(cookieStr)
cookieCguidMatch := regexp.MustCompile(`cguid=([^;]+)`).FindStringSubmatch(cookieStr)
if len(ssoSidMatch) > 1 && sid == "" {
sid = ssoSidMatch[1]
log.Debugf("DEBUG: 从 Set-Cookie 提取到 sid: %s", sid)
}
if len(cookieCguidMatch) > 1 && extractedCguid == "" {
extractedCguid = cookieCguidMatch[1]
log.Debugf("DEBUG: 从 Set-Cookie 提取到 cguid: %s", extractedCguid)
}
}
}
if sid == "" || extractedCguid == "" {
return "", errors.New("failed to extract sid or cguid from login response")
}
// 提取并记录 cookies
loginUrlObj, _ := url.Parse(loginURL)
cookies := base.RestyClient.GetClient().Jar.Cookies(loginUrlObj)
var cookieStrings []string
for _, cookie := range cookies {
cookieStrings = append(cookieStrings, cookie.Name+"="+cookie.Value)
}
cookieStr := strings.Join(cookieStrings, "; ")
log.Debugf("DEBUG: 提取到的 Cookies: %s", cookieStr)
d.MailCookies = cookieStr
return sid, nil
}
func (d *Yun139) step2_get_single_token(sid string) (string, error) {
log.Debugf("\n--- 执行步骤 2: 换artifact API ---")
cguid := strconv.FormatInt(time.Now().UnixMilli(), 10)
exchangeArtifactURL := fmt.Sprintf("https://smsrebuild1.mail.10086.cn/setting/s?func=%s&sid=%s&cguid=%s", url.QueryEscape("umc:getArtifact"), sid, cguid)
// 从 MailCookies 中提取 RMKEY
var rmkey string
cookies := strings.Split(d.MailCookies, ";")
for _, cookie := range cookies {
cookie = strings.TrimSpace(cookie)
if strings.HasPrefix(cookie, "RMKEY=") {
rmkey = cookie
break
}
}
if rmkey == "" {
return "", errors.New("RMKEY not found in MailCookies")
}
exchangePassidHeaders := map[string]string{
"Host": "smsrebuild1.mail.10086.cn",
"Cookie": rmkey,
"Content-Type": "text/xml; charset=utf-8",
"Accept-Encoding": "gzip",
"User-Agent": "okhttp/4.12.0",
}
log.Debugf("DEBUG: 换passid 请求 URL: %s", exchangeArtifactURL)
log.Debugf("DEBUG: 换passid 请求 Headers: %+v", exchangePassidHeaders)
res, err := base.RestyClient.R().
SetHeaders(exchangePassidHeaders).
Post(exchangeArtifactURL)
if err != nil {
return "", fmt.Errorf("step2 exchange artifact request failed: %w", err)
}
log.Debugf("DEBUG: 换passid 响应 Status Code: %d", res.StatusCode())
log.Debugf("DEBUG: 换passid 响应 Headers: %+v", res.Header())
log.Debugf("DEBUG: 换passid 响应 Body: %s...", res.String()[:min(len(res.String()), 500)])
dycpwd := jsoniter.Get(res.Body(), "var", "artifact").ToString()
if dycpwd == "" {
return "", errors.New("failed to extract dycpwd from artifact exchange response")
}
log.Debugf("DEBUG: 提取到 dycpwd: %s", dycpwd)
return dycpwd, nil
}
// --- 辅助函数:加密/解密 ---
// sha1Hash 计算 SHA1 哈希值,返回十六进制字符串。
func sha1Hash(data string) string {
h := sha1.New()
h.Write([]byte(data))
return hex.EncodeToString(h.Sum(nil))
}
// pkcs7_pad PKCS7 填充
func pkcs7_pad(data []byte, blockSize int) []byte {
padding := blockSize - len(data)%blockSize
padtext := bytes.Repeat([]byte{byte(padding)}, padding)
return append(data, padtext...)
}
// pkcs7_unpad PKCS7 去填充
func pkcs7_unpad(data []byte) ([]byte, error) {
length := len(data)
if length == 0 {
return nil, errors.New("pkcs7: data is empty")
}
unpadding := int(data[length-1])
if unpadding > length {
return nil, errors.New("pkcs7: invalid padding")
}
return data[:(length - unpadding)], nil
}
// aes_ecb_decrypt AES/ECB/Pkcs7 解密,输入为十六进制字符串。
func aes_ecb_decrypt(ciphertext []byte, key []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(ciphertext)%block.BlockSize() != 0 {
return nil, errors.New("AES ECB decrypt: ciphertext is not a multiple of the block size")
}
decrypted := make([]byte, len(ciphertext))
blockSize := block.BlockSize()
for bs, be := 0, blockSize; bs < len(ciphertext); bs, be = bs+blockSize, be+blockSize {
block.Decrypt(decrypted[bs:be], ciphertext[bs:be])
}
return pkcs7_unpad(decrypted)
}
// 以下提供 camelCase 的 AES CBC 加解密,供文件中其它位置调用(并支持传入 IV)。
func aesCbcEncrypt(plaintext []byte, key []byte, iv []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(iv) != block.BlockSize() {
return nil, fmt.Errorf("aesCbcEncrypt: iv length %d does not match block size %d", len(iv), block.BlockSize())
}
padded := pkcs7_pad(plaintext, block.BlockSize())
ciphertext := make([]byte, len(padded))
mode := cipher.NewCBCEncrypter(block, iv)
mode.CryptBlocks(ciphertext, padded)
return ciphertext, nil
}
func aesCbcDecrypt(ciphertext []byte, key []byte, iv []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(iv) != block.BlockSize() {
return nil, fmt.Errorf("aesCbcDecrypt: iv length %d does not match block size %d", len(iv), block.BlockSize())
}
if len(ciphertext)%block.BlockSize() != 0 {
return nil, errors.New("aesCbcDecrypt: ciphertext is not a multiple of the block size")
}
decrypted := make([]byte, len(ciphertext))
mode := cipher.NewCBCDecrypter(block, iv)
mode.CryptBlocks(decrypted, ciphertext)
return pkcs7_unpad(decrypted)
}
// sortedJsonStringify 对 JSON 对象进行排序并字符串化。
func sortedJsonStringify(obj interface{}) (string, error) {
if obj == nil {
return "null", nil
}
switch v := obj.(type) {
case string:
// 尝试解析为 JSON,如果成功则递归处理
var parsed interface{}
if err := jsoniter.Unmarshal([]byte(v), &parsed); err == nil {
return sortedJsonStringify(parsed)
}
// 如果不是 JSON 字符串,则直接返回 JSON 字符串化的结果
return jsoniter.MarshalToString(v)
case int, float64, bool:
return fmt.Sprintf("%v", v), nil
case []interface{}:
var items []string
for _, item := range v {
s, err := sortedJsonStringify(item)
if err != nil {
return "", err
}
items = append(items, s)
}
return fmt.Sprintf("[%s]", strings.Join(items, ",")), nil
case map[string]interface{}:
sortedKeys := make([]string, 0, len(v))
for key := range v {
sortedKeys = append(sortedKeys, key)
}
sort.Strings(sortedKeys)
var pairs []string
for _, key := range sortedKeys {
value := v[key]
s, err := sortedJsonStringify(value)
if err != nil {
return "", err
}
// Use jsoniter.MarshalToString for the key to ensure it's quoted correctly
keyStr, err := jsoniter.MarshalToString(key)
if err != nil {
return "", err
}
pairs = append(pairs, fmt.Sprintf("%s:%s", keyStr, s))
}
return fmt.Sprintf("{%s}", strings.Join(pairs, ",")), nil
default:
// Fallback for other types, e.g., numbers, booleans, or unhandled complex types
// Use jsoniter's default marshalling for these
return jsoniter.MarshalToString(v)
}
}
// yun139EncryptedRequest handles the common encrypted request/response flow.
func (d *Yun139) yun139EncryptedRequest(url string, body interface{}, headers map[string]string, aesKeyHex string, resp interface{}) ([]byte, error) {
// 1. Decode AES key
aesKey, err := hex.DecodeString(aesKeyHex)
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: failed to decode AES key: %w", err)
}
// 2. Marshal and sort the request body
sortedJson, err := sortedJsonStringify(body)
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: failed to marshal and sort body: %w", err)
}
log.Debugf("yun139EncryptedRequest: Request Body (plaintext): %s", sortedJson)
// 3. Encrypt the body using AES/CBC
iv := make([]byte, 16) // 16 bytes for AES-128
if _, err := crypto_rand.Read(iv); err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: failed to generate IV: %w", err)
}
encryptedBody, err := aesCbcEncrypt([]byte(sortedJson), aesKey, iv)
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: failed to encrypt body: %w", err)
}
payload := base64.StdEncoding.EncodeToString(append(iv, encryptedBody...))
// 4. Make the request
res, err := base.RestyClient.R().
SetHeaders(headers).
SetBody(payload).
Post(url)
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: http request failed: %w", err)
}
if res.StatusCode() != 200 {
return nil, fmt.Errorf("yun139EncryptedRequest: unexpected status code %d: %s", res.StatusCode(), res.String())
}
// 5. Decrypt the response
respBody := res.Body()
var decryptedBytes []byte
if len(respBody) > 0 && respBody[0] == '{' {
log.Warnf("yun139EncryptedRequest: received a plain JSON response, not an encrypted string. Body: %s", string(respBody))
decryptedBytes = respBody
} else {
decodedResp, err := base64.StdEncoding.DecodeString(string(respBody))
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: response base64 decode failed: %w. Body: '%s'", err, string(respBody))
}
if len(decodedResp) < 16 {
return nil, fmt.Errorf("yun139EncryptedRequest: decoded response is too short to be encrypted. Length: %d", len(decodedResp))
}
respIv := decodedResp[:16]
respCiphertext := decodedResp[16:]
decryptedBytes, err = aesCbcDecrypt(respCiphertext, aesKey, respIv)
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: response aes decrypt failed: %w", err)
}
}
log.Debugf("yun139EncryptedRequest: Response Body (decrypted): %s", string(decryptedBytes))
// 6. Unmarshal to the final response struct
if resp != nil {
err = utils.Json.Unmarshal(decryptedBytes, resp)
if err != nil {
return nil, fmt.Errorf("yun139EncryptedRequest: failed to unmarshal decrypted response: %w", err)
}
}
return decryptedBytes, nil
}
func (d *Yun139) step3_third_party_login(dycpwd string) (string, error) {
log.Debugf("\n--- 执行步骤 3: 单点登录 API ---")
ssoLoginURL := "https://user-njs.yun.139.com/user/thirdlogin"
// 构建原始请求体
ssoRequestBodyRaw := base.Json{
"clientkey_decrypt": "l3TryM&Q+X7@dzwk)qP",
"clienttype": "886",
"cpid": "507",
"dycpwd": dycpwd,
"extInfo": base.Json{"ifOpenAccount": "0"},
"loginMode": "0",
"msisdn": d.Username,
"pintype": "13",
"secinfo": strings.ToUpper(sha1Hash(fmt.Sprintf("fetion.com.cn:%s", dycpwd))),
"version": "20250901",
}
ssoLoginHeaders := map[string]string{
"hcy-cool-flag": "1",
"x-huawei-channelSrc": "10246600",
"x-sdk-channelSrc": "",
"x-MM-Source": "0",
"x-UserAgent": "android|23116PN5BC|android15|1.2.6|||1440x3200|10246600",
"x-DeviceInfo": "4|127.0.0.1|5|1.2.6|Xiaomi|23116PN5BC||02-00-00-00-00-00|android 15|1440x3200|android|||",
"Content-Type": "text/plain;charset=UTF-8",
"Host": "user-njs.yun.139.com",
"Connection": "Keep-Alive",
"Accept-Encoding": "gzip",
"User-Agent": "okhttp/3.12.2",
}
// 使用通用加密请求函数
decryptedLayer1StrBytes, err := d.yun139EncryptedRequest(ssoLoginURL, ssoRequestBodyRaw, ssoLoginHeaders, KEY_HEX_1, nil)
if err != nil {
return "", fmt.Errorf("step3 encrypted request failed: %w", err)
}
hexInner := jsoniter.Get(decryptedLayer1StrBytes, "data").ToString()
if hexInner == "" {
return "", errors.New("missing data field in first layer decryption result")
}
log.Debugf("DEBUG: 第一层解密提取到 hex_inner: %s...", hexInner[:min(len(hexInner), 50)])
// 第二层解密
key2, err := hex.DecodeString(KEY_HEX_2)
if err != nil {
return "", fmt.Errorf("failed to decode KEY_HEX_2: %w", err)
}
hexInnerBytes, err := hex.DecodeString(hexInner)
if err != nil {
return "", fmt.Errorf("failed to decode hex_inner: %w", err)
}
finalJsonStrBytes, err := aes_ecb_decrypt(hexInnerBytes, key2)
if err != nil {
return "", fmt.Errorf("step3 response layer2 aes ecb decrypt failed: %w", err)
}
log.Debugf("DEBUG: 最终解密结果: %s", string(finalJsonStrBytes))
// 提取 authToken
authToken := jsoniter.Get(finalJsonStrBytes, "authToken").ToString()
if authToken == "" {
return "", errors.New("failed to extract authToken from final decryption result")
}
log.Debugf("DEBUG: 提取到 authToken: %s", authToken)
// 提取 account 和 userDomainId
account := jsoniter.Get(finalJsonStrBytes, "account").ToString()
userDomainId := jsoniter.Get(finalJsonStrBytes, "userDomainId").ToString()
if account == "" || userDomainId == "" {
return "", errors.New("failed to extract account or userDomainId from final decryption result")
}
d.UserDomainID = userDomainId
newAuthorization := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("pc:%s:%s", account, authToken)))
return newAuthorization, nil
}
func (d *Yun139) loginWithPassword() (string, error) {
if d.Username == "" || d.Password == "" || d.MailCookies == "" {
return "", errors.New("username, password or mail_cookies is empty")
}
passId, err := d.step1_password_login()
if err != nil {
return "", err
}
log.Infof("Step 1 success, passId: %s", passId)
token, err := d.step2_get_single_token(passId)
if err != nil {
return "", err
}
log.Infof("Step 2 success, token: %s", token)
newAuth, err := d.step3_third_party_login(token)
if err != nil {
return "", err
}
log.Infof("Step 3 success, new authorization generated.")
d.Authorization = newAuth // Ensure Authorization is also updated before saving
op.MustSaveDriverStorage(d)
return newAuth, nil
}
func (d *Yun139) andAlbumRequest(pathname string, body interface{}, resp interface{}) ([]byte, error) {
url := "https://group.yun.139.com/hcy/family/adapter/andAlbum/openApi" + pathname
headers := map[string]string{
"Host": "group.yun.139.com",
"authorization": "Basic " + d.getAuthorization(),
"x-svctype": "2",
"hcy-cool-flag": "1",
"api-version": "v2",
"x-huawei-channelsrc": "10246600",
"x-sdk-channelsrc": "",
"x-mm-source": "0",
"x-deviceinfo": "1|127.0.0.1|1|12.3.2|Xiaomi|23116PN5BC||02-00-00-00-00-00|android 15|1440x3200|android|zh||||032|0|", //重要参数
"content-type": "application/json; charset=utf-8",
"user-agent": "okhttp/4.11.0",
"accept-encoding": "gzip",
}
return d.yun139EncryptedRequest(url, body, headers, KEY_HEX_1, resp)
}
func (d *Yun139) handleMetaGroupCopy(ctx context.Context, srcObj, dstDir model.Obj) error {
pathname := "/copyContentCatalog"
var sourceContentIDs []string
var sourceCatalogIDs []string
if srcObj.IsDir() {
sourceCatalogIDs = append(sourceCatalogIDs, path.Join("root:/", srcObj.GetPath(), srcObj.GetID()))
} else {
sourceContentIDs = append(sourceContentIDs, path.Join("root:/", srcObj.GetPath(), srcObj.GetID()))
}
destCatalogID := path.Join("root:/", dstDir.GetPath(), dstDir.GetID())
log.Debugf("[139Yun Group Copy] srcObj ID: %s, srcObj Path: %s, dstDir ID: %s, dstDir Path: %s, destCatalogID: %s", srcObj.GetID(), srcObj.GetPath(), dstDir.GetID(), dstDir.GetPath(), destCatalogID)
body := base.Json{
"commonAccountInfo": base.Json{
"accountType": "1",
"accountUserId": d.UserDomainID,
},
"destCatalogID": destCatalogID,
"destCloudID": d.CloudID,
"sourceCatalogIDs": sourceCatalogIDs,
"sourceCloudID": d.CloudID,
"sourceContentIDs": sourceContentIDs,
}
var resp base.Json
_, err := d.andAlbumRequest(pathname, body, &resp)
return err
}
// getGroupRootByCloudID 查询 group 上层信息,优先返回 parentCatalogID,回退到 catalogList[0].path
func (d *Yun139) getGroupRootByCloudID(cloudID string) (string, error) {
pathname := "/orchestration/group-rebuild/catalog/v1.0/queryGroupContentList"
body := base.Json{
"groupID": cloudID,
"commonAccountInfo": base.Json{
"account": d.getAccount(),
"accountType": 1,
},
"pageInfo": base.Json{
"pageNum": 1,
"pageSize": 1,
},
}
var resp base.Json
_, err := d.post(pathname, body, &resp)
if err != nil {
return "", err
}
dataObj, _ := resp["data"].(map[string]interface{})
if dataObj == nil {
return "", fmt.Errorf("invalid group response data")
}
if gcr, ok := dataObj["getGroupContentResult"].(map[string]interface{}); ok {
if pid, ok := gcr["parentCatalogID"].(string); ok && pid != "" {
return pid, nil
}
if cl, ok := gcr["catalogList"].([]interface{}); ok && len(cl) > 0 {
if first, ok := cl[0].(map[string]interface{}); ok {
if p, ok := first["path"].(string); ok && p != "" {
return p, nil
}
}
}
}
return "", fmt.Errorf("no root found in group response")
}
// getFamilyRootPath 查询 family 的上层 path(data.path)
// 返回值已去除前缀 "root:/"(或 "root:"),直接返回纯 ID 或 path 部分,便于持久化为 RootFolderID。
func (d *Yun139) getFamilyRootPath(cloudID string) (string, error) {
// 使用 v1.2 接口(代码日志中已有该请求),pageSize 取 1 足够获取 path 字段
pathname := "/orchestration/familyCloud-rebuild/content/v1.2/queryContentList"
body := base.Json{
"catalogID": "",
"catalogType": 3,
"cloudID": cloudID,
"cloudType": 1,
"commonAccountInfo": base.Json{
"account": d.getAccount(),
"accountType": 1,
},
"contentSortType": 0,
"pageInfo": base.Json{
"pageNum": 1,
"pageSize": 1,
},
"sortDirection": 1,
}
var resp base.Json
_, err := d.post(pathname, body, &resp)
if err != nil {
return "", err
}
dataObj, _ := resp["data"].(map[string]interface{})
if dataObj == nil {
return "", fmt.Errorf("invalid family response data")
}
// helper to strip "root:/" or "root:" prefix
stripRoot := func(s string) string {
s = strings.TrimSpace(s)
s = strings.TrimPrefix(s, "root:/")
s = strings.TrimPrefix(s, "root:")
return s
}
if p, ok := dataObj["path"].(string); ok && p != "" {
return stripRoot(p), nil
}
// 回退:有时 path 在 cloudCatalogList.catalogList 中
if cl, ok := dataObj["cloudCatalogList"].([]interface{}); ok && len(cl) > 0 {
if first, ok := cl[0].(map[string]interface{}); ok {
if p, ok := first["path"].(string); ok && p != "" {
return stripRoot(p), nil
}
}
}
return "", fmt.Errorf("no path found in family response")
}
+13
View File
@@ -194,4 +194,17 @@ func (d *Cloud189) Put(ctx context.Context, dstDir model.Obj, stream model.FileS
return d.newUpload(ctx, dstDir, stream, up)
}
func (d *Cloud189) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
capacityInfo, err := d.getCapacityInfo(ctx)
if err != nil {
return nil, err
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: capacityInfo.CloudCapacityInfo.TotalSize,
UsedSpace: capacityInfo.CloudCapacityInfo.UsedSize,
},
}, nil
}
var _ driver.Driver = (*Cloud189)(nil)
+18
View File
@@ -66,3 +66,21 @@ type DownResp struct {
ResMessage string `json:"res_message"`
FileDownloadUrl string `json:"downloadUrl"`
}
type CapacityResp struct {
ResCode int `json:"res_code"`
ResMessage string `json:"res_message"`
Account string `json:"account"`
CloudCapacityInfo struct {
FreeSize int64 `json:"freeSize"`
MailUsedSize int64 `json:"mail189UsedSize"`
TotalSize int64 `json:"totalSize"`
UsedSize int64 `json:"usedSize"`
} `json:"cloudCapacityInfo"`
FamilyCapacityInfo struct {
FreeSize int64 `json:"freeSize"`
TotalSize int64 `json:"totalSize"`
UsedSize int64 `json:"usedSize"`
} `json:"familyCapacityInfo"`
TotalSize uint64 `json:"totalSize"`
}
+19 -9
View File
@@ -107,7 +107,7 @@ import (
// res, err = d.client.R().
// SetHeaders(map[string]string{
// "lt": lt,
// "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
// "User-Agent": base.UserAgentNT,
// "Referer": "https://open.e.189.cn/",
// "accept": "application/json;charset=UTF-8",
// }).SetFormData(map[string]string{
@@ -157,7 +157,7 @@ func (d *Cloud189) request(url string, method string, callback base.ReqCallback,
if err != nil {
return nil, err
}
//log.Debug(res.String())
// log.Debug(res.String())
if e.ErrorCode != "" {
if e.ErrorCode == "InvalidSessionKey" {
err = d.newLogin()
@@ -186,8 +186,8 @@ func (d *Cloud189) getFiles(fileId string) ([]model.Obj, error) {
"mediaType": "0",
"folderId": fileId,
"iconOption": "5",
"orderBy": "lastOpTime", //account.OrderBy
"descending": "true", //account.OrderDirection
"orderBy": "lastOpTime", // account.OrderBy
"descending": "true", // account.OrderDirection
})
}, &resp)
if err != nil {
@@ -311,7 +311,7 @@ func (d *Cloud189) newUpload(ctx context.Context, dstDir model.Obj, file model.F
}
d.sessionKey = sessionKey
const DEFAULT int64 = 10485760
var count = int64(math.Ceil(float64(file.GetSize()) / float64(DEFAULT)))
count := int64(math.Ceil(float64(file.GetSize()) / float64(DEFAULT)))
res, err := d.uploadRequest("/person/initMultiUpload", map[string]string{
"parentFolderId": dstDir.GetID(),
@@ -340,10 +340,10 @@ func (d *Cloud189) newUpload(ctx context.Context, dstDir model.Obj, file model.F
if DEFAULT < byteSize {
byteSize = DEFAULT
}
//log.Debugf("%d,%d", byteSize, finish)
// log.Debugf("%d,%d", byteSize, finish)
byteData := make([]byte, byteSize)
n, err := io.ReadFull(file, byteData)
//log.Debug(err, n)
// log.Debug(err, n)
if err != nil {
return err
}
@@ -365,11 +365,10 @@ func (d *Cloud189) newUpload(ctx context.Context, dstDir model.Obj, file model.F
log.Debugf("uploadData: %+v", uploadData)
requestURL := uploadData.RequestURL
uploadHeaders := strings.Split(decodeURIComponent(uploadData.RequestHeader), "&")
req, err := http.NewRequest(http.MethodPut, requestURL, driver.NewLimitedUploadStream(ctx, bytes.NewReader(byteData)))
req, err := http.NewRequestWithContext(ctx, http.MethodPut, requestURL, driver.NewLimitedUploadStream(ctx, bytes.NewReader(byteData)))
if err != nil {
return err
}
req = req.WithContext(ctx)
for _, v := range uploadHeaders {
i := strings.Index(v, "=")
req.Header.Set(v[0:i], v[i+1:])
@@ -396,3 +395,14 @@ func (d *Cloud189) newUpload(ctx context.Context, dstDir model.Obj, file model.F
}, nil)
return err
}
func (d *Cloud189) getCapacityInfo(ctx context.Context) (*CapacityResp, error) {
var resp CapacityResp
_, err := d.request("https://cloud.189.cn/api/portal/getUserSizeInfo.action", http.MethodGet, func(req *resty.Request) {
req.SetContext(ctx)
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
+38 -11
View File
@@ -1,7 +1,6 @@
package _189_tv
import (
"container/ring"
"context"
"net/http"
"strconv"
@@ -12,18 +11,20 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/cron"
"github.com/go-resty/resty/v2"
)
type Cloud189TV struct {
model.Storage
Addition
client *resty.Client
tokenInfo *AppSessionResp
uploadThread int
familyTransferFolder *ring.Ring
cleanFamilyTransferFile func()
storageConfig driver.Config
client *resty.Client
tokenInfo *AppSessionResp
uploadThread int
storageConfig driver.Config
TempUuid string
cron *cron.Cron // 新增 cron 字段
}
func (y *Cloud189TV) Config() driver.Config {
@@ -68,7 +69,7 @@ func (y *Cloud189TV) Init(ctx context.Context) (err error) {
// 避免重复登陆
if !y.isLogin() || y.Addition.AccessToken == "" {
if err = y.login(); err != nil {
return
return err
}
}
@@ -79,10 +80,17 @@ func (y *Cloud189TV) Init(ctx context.Context) (err error) {
}
}
return
y.cron = cron.NewCron(time.Minute * 5)
y.cron.Do(y.keepAlive)
return err
}
func (y *Cloud189TV) Drop(ctx context.Context) error {
if y.cron != nil {
y.cron.Stop()
y.cron = nil
}
return nil
}
@@ -236,7 +244,6 @@ func (y *Cloud189TV) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
FileName: srcObj.GetName(),
IsFolder: BoolToNumber(srcObj.IsDir()),
})
if err != nil {
return err
}
@@ -270,5 +277,25 @@ func (y *Cloud189TV) Put(ctx context.Context, dstDir model.Obj, stream model.Fil
}
return y.OldUpload(ctx, dstDir, stream, up, isFamily, overwrite)
}
func (y *Cloud189TV) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
capacityInfo, err := y.getCapacityInfo(ctx)
if err != nil {
return nil, err
}
var total, used int64
if y.isFamily() {
total = capacityInfo.FamilyCapacityInfo.TotalSize
used = capacityInfo.FamilyCapacityInfo.UsedSize
} else {
total = capacityInfo.CloudCapacityInfo.TotalSize
used = capacityInfo.CloudCapacityInfo.UsedSize
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: total,
UsedSpace: used,
},
}, nil
}
-1
View File
@@ -8,7 +8,6 @@ import (
type Addition struct {
driver.RootID
AccessToken string `json:"access_token"`
TempUuid string
OrderBy string `json:"order_by" type:"select" options:"filename,filesize,lastOpTime" default:"filename"`
OrderDirection string `json:"order_direction" type:"select" options:"asc,desc" default:"asc"`
Type string `json:"type" type:"select" options:"personal,family" default:"personal"`
+18
View File
@@ -316,3 +316,21 @@ type BatchTaskConflictTaskInfoResp struct {
TaskInfos []BatchTaskInfo
TaskType int `json:"taskType"`
}
type CapacityResp struct {
ResCode int `json:"res_code"`
ResMessage string `json:"res_message"`
Account string `json:"account"`
CloudCapacityInfo struct {
FreeSize int64 `json:"freeSize"`
MailUsedSize int64 `json:"mail189UsedSize"`
TotalSize int64 `json:"totalSize"`
UsedSize int64 `json:"usedSize"`
} `json:"cloudCapacityInfo"`
FamilyCapacityInfo struct {
FreeSize int64 `json:"freeSize"`
TotalSize int64 `json:"totalSize"`
UsedSize int64 `json:"usedSize"`
} `json:"familyCapacityInfo"`
TotalSize uint64 `json:"totalSize"`
}
+97 -17
View File
@@ -5,17 +5,19 @@ import (
"encoding/base64"
"encoding/xml"
"fmt"
"github.com/skip2/go-qrcode"
"io"
"net/http"
"strconv"
"strings"
"time"
"github.com/skip2/go-qrcode"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/go-resty/resty/v2"
@@ -64,6 +66,13 @@ func (y *Cloud189TV) AppKeySignatureHeader(url, method string) map[string]string
}
func (y *Cloud189TV) request(url, method string, callback base.ReqCallback, params map[string]string, resp interface{}, isFamily ...bool) ([]byte, error) {
return y.requestWithRetry(url, method, callback, params, resp, 0, isFamily...)
}
func (y *Cloud189TV) requestWithRetry(url, method string, callback base.ReqCallback, params map[string]string, resp interface{}, retryCount int, isFamily ...bool) ([]byte, error) {
if y.tokenInfo == nil {
return nil, fmt.Errorf("login failed")
}
req := y.client.R().SetQueryParams(clientSuffix())
if params != nil {
@@ -89,7 +98,22 @@ func (y *Cloud189TV) request(url, method string, callback base.ReqCallback, para
if strings.Contains(res.String(), "userSessionBO is null") ||
strings.Contains(res.String(), "InvalidSessionKey") {
return nil, errors.New("session expired")
// 限制重试次数,避免无限递归
if retryCount >= 3 {
y.Addition.AccessToken = ""
op.MustSaveDriverStorage(y)
return nil, errors.New("session expired after retry")
}
// 尝试刷新会话
if err := y.refreshSession(); err != nil {
// 如果刷新失败,说明AccessToken也已过期,需要重新登录
y.Addition.AccessToken = ""
op.MustSaveDriverStorage(y)
return nil, errors.New("session expired")
}
// 如果刷新成功,则重试原始请求(增加重试计数)
return y.requestWithRetry(url, method, callback, params, resp, retryCount+1, isFamily...)
}
// 处理错误
@@ -129,6 +153,7 @@ func (y *Cloud189TV) put(ctx context.Context, url string, headers map[string]str
}
}
// 请求完成后http.Client会Close Request.Body
resp, err := base.HttpClient.Do(req)
if err != nil {
return nil, err
@@ -151,6 +176,7 @@ func (y *Cloud189TV) put(ctx context.Context, url string, headers map[string]str
}
return body, nil
}
func (y *Cloud189TV) getFiles(ctx context.Context, fileId string, isFamily bool) ([]model.Obj, error) {
fullUrl := ApiUrl
if isFamily {
@@ -208,7 +234,7 @@ func (y *Cloud189TV) login() (err error) {
var erron RespErr
var tokenInfo AppSessionResp
if y.Addition.AccessToken == "" {
if y.Addition.TempUuid == "" {
if y.TempUuid == "" {
// 获取登录参数
var uuidInfo UuidInfoResp
req.SetResult(&uuidInfo).SetError(&erron)
@@ -216,9 +242,8 @@ func (y *Cloud189TV) login() (err error) {
req.SetHeaders(y.AppKeySignatureHeader(ApiUrl+"/family/manage/getQrCodeUUID.action",
http.MethodGet))
_, err = req.Execute(http.MethodGet, ApiUrl+"/family/manage/getQrCodeUUID.action")
if err != nil {
return
return err
}
if erron.HasError() {
return &erron
@@ -227,7 +252,7 @@ func (y *Cloud189TV) login() (err error) {
if uuidInfo.Uuid == "" {
return errors.New("uuidInfo is empty")
}
y.Addition.TempUuid = uuidInfo.Uuid
y.TempUuid = uuidInfo.Uuid
op.MustSaveDriverStorage(y)
// 展示二维码
@@ -255,10 +280,10 @@ func (y *Cloud189TV) login() (err error) {
// Signature
req.SetHeaders(y.AppKeySignatureHeader(ApiUrl+"/family/manage/qrcodeLoginResult.action",
http.MethodGet))
req.SetQueryParam("uuid", y.Addition.TempUuid)
req.SetQueryParam("uuid", y.TempUuid)
_, err = req.Execute(http.MethodGet, ApiUrl+"/family/manage/qrcodeLoginResult.action")
if err != nil {
return
return err
}
if erron.HasError() {
return &erron
@@ -267,7 +292,6 @@ func (y *Cloud189TV) login() (err error) {
return errors.New("E189AccessToken is empty")
}
y.Addition.AccessToken = accessTokenResp.E189AccessToken
y.Addition.TempUuid = ""
}
}
// 获取SessionKey 和 SessionSecret
@@ -279,7 +303,7 @@ func (y *Cloud189TV) login() (err error) {
reqb.SetQueryParam("e189AccessToken", y.Addition.AccessToken)
_, err = reqb.Execute(http.MethodGet, ApiUrl+"/family/manage/loginFamilyMerge.action")
if err != nil {
return
return err
}
if erron.HasError() {
@@ -288,7 +312,45 @@ func (y *Cloud189TV) login() (err error) {
y.tokenInfo = &tokenInfo
op.MustSaveDriverStorage(y)
return
return err
}
// refreshSession 尝试使用现有的 AccessToken 刷新会话
func (y *Cloud189TV) refreshSession() (err error) {
var erron RespErr
var tokenInfo AppSessionResp
reqb := y.client.R().SetQueryParams(clientSuffix())
reqb.SetResult(&tokenInfo).SetError(&erron)
// Signature
reqb.SetHeaders(y.AppKeySignatureHeader(ApiUrl+"/family/manage/loginFamilyMerge.action",
http.MethodGet))
reqb.SetQueryParam("e189AccessToken", y.Addition.AccessToken)
_, err = reqb.Execute(http.MethodGet, ApiUrl+"/family/manage/loginFamilyMerge.action")
if err != nil {
return err
}
if erron.HasError() {
return &erron
}
y.tokenInfo = &tokenInfo
return nil
}
func (y *Cloud189TV) keepAlive() {
_, err := y.get(ApiUrl+"/keepUserSession.action", func(r *resty.Request) {
r.SetQueryParams(clientSuffix())
}, nil)
if err != nil {
utils.Log.Warnf("189tv: Failed to keep user session alive: %v", err)
// 如果keepAlive失败,尝试刷新session
if refreshErr := y.refreshSession(); refreshErr != nil {
utils.Log.Errorf("189tv: Failed to refresh session after keepAlive error: %v", refreshErr)
}
} else {
utils.Log.Debugf("189tv: User session kept alive successfully.")
}
}
func (y *Cloud189TV) RapidUpload(ctx context.Context, dstDir model.Obj, stream model.FileStreamer, isFamily bool, overwrite bool) (model.Obj, error) {
@@ -311,11 +373,14 @@ func (y *Cloud189TV) RapidUpload(ctx context.Context, dstDir model.Obj, stream m
// 旧版本上传,家庭云不支持覆盖
func (y *Cloud189TV) OldUpload(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress, isFamily bool, overwrite bool) (model.Obj, error) {
tempFile, err := file.CacheFullInTempFile()
if err != nil {
return nil, err
fileMd5 := file.GetHash().GetHash(utils.MD5)
tempFile := file.GetFile()
var err error
if len(fileMd5) != utils.MD5.Width {
tempFile, fileMd5, err = stream.CacheFullAndHash(file, &up, utils.MD5)
} else if tempFile == nil {
tempFile, err = file.CacheFullAndWriter(&up, nil)
}
fileMd5, err := utils.HashFile(utils.MD5, tempFile)
if err != nil {
return nil, err
}
@@ -328,6 +393,10 @@ func (y *Cloud189TV) OldUpload(ctx context.Context, dstDir model.Obj, file model
// 网盘中不存在该文件,开始上传
status := GetUploadFileStatusResp{CreateUploadFileResp: *uploadInfo}
// driver.RateLimitReader会尝试Close底层的reader
// 但这里的tempFile是一个*os.File,Close后就没法继续读了
// 所以这里用io.NopCloser包一层
rateLimitedRd := driver.NewLimitedUploadStream(ctx, io.NopCloser(tempFile))
for status.GetSize() < file.GetSize() && status.FileDataExists != 1 {
if utils.IsCanceled(ctx) {
return nil, ctx.Err()
@@ -345,7 +414,7 @@ func (y *Cloud189TV) OldUpload(ctx context.Context, dstDir model.Obj, file model
header["Edrive-UploadFileId"] = fmt.Sprint(status.UploadFileId)
}
_, err := y.put(ctx, status.FileUploadUrl, header, true, io.NopCloser(tempFile), isFamily)
_, err := y.put(ctx, status.FileUploadUrl, header, true, rateLimitedRd, isFamily)
if err, ok := err.(*RespErr); ok && err.Code != "InputStreamReadError" {
return nil, err
}
@@ -408,7 +477,6 @@ func (y *Cloud189TV) OldUploadCreate(ctx context.Context, parentID string, fileM
})
}
}, &uploadInfo, isFamily)
if err != nil {
return nil, err
}
@@ -562,3 +630,15 @@ func (y *Cloud189TV) WaitBatchTask(aType string, taskID string, t time.Duration)
time.Sleep(t)
}
}
func (y *Cloud189TV) getCapacityInfo(ctx context.Context) (*CapacityResp, error) {
fullUrl := ApiUrl + "/portal/getUserSizeInfo.action"
var resp CapacityResp
_, err := y.get(fullUrl, func(req *resty.Request) {
req.SetContext(ctx)
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
+45 -11
View File
@@ -12,6 +12,7 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/cron"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/go-resty/resty/v2"
"github.com/google/uuid"
@@ -21,12 +22,12 @@ type Cloud189PC struct {
model.Storage
Addition
identity string
client *resty.Client
loginParam *LoginParam
tokenInfo *AppSessionResp
loginParam *LoginParam
qrcodeParam *QRLoginParam
tokenInfo *AppSessionResp
uploadThread int
@@ -35,6 +36,7 @@ type Cloud189PC struct {
storageConfig driver.Config
ref *Cloud189PC
cron *cron.Cron
}
func (y *Cloud189PC) Config() driver.Config {
@@ -84,14 +86,22 @@ func (y *Cloud189PC) Init(ctx context.Context) (err error) {
})
}
// 避免重复登陆
identity := utils.GetMD5EncodeStr(y.Username + y.Password)
if !y.isLogin() || y.identity != identity {
y.identity = identity
// 先尝试用Token刷新,之后尝试登陆
if y.Addition.RefreshToken != "" {
y.tokenInfo = &AppSessionResp{RefreshToken: y.Addition.RefreshToken}
if err = y.refreshToken(); err != nil {
return err
}
} else {
if err = y.login(); err != nil {
return
return err
}
}
// 初始化并启动 cron 任务
y.cron = cron.NewCron(time.Duration(time.Minute * 5))
// 每5分钟执行一次 keepAlive
y.cron.Do(y.keepAlive)
}
// 处理家庭云ID
@@ -114,7 +124,7 @@ func (y *Cloud189PC) Init(ctx context.Context) (err error) {
utils.Log.Errorf("cleanFamilyTransferFolderError:%s", err)
}
})
return
return err
}
func (d *Cloud189PC) InitReference(storage driver.Driver) error {
@@ -128,6 +138,10 @@ func (d *Cloud189PC) InitReference(storage driver.Driver) error {
func (y *Cloud189PC) Drop(ctx context.Context) error {
y.ref = nil
if y.cron != nil {
y.cron.Stop()
y.cron = nil
}
return nil
}
@@ -291,7 +305,6 @@ func (y *Cloud189PC) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
FileName: srcObj.GetName(),
IsFolder: BoolToNumber(srcObj.IsDir()),
})
if err != nil {
return err
}
@@ -397,3 +410,24 @@ func (y *Cloud189PC) Put(ctx context.Context, dstDir model.Obj, stream model.Fil
return y.StreamUpload(ctx, dstDir, stream, up, isFamily, overwrite)
}
}
func (y *Cloud189PC) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
capacityInfo, err := y.getCapacityInfo(ctx)
if err != nil {
return nil, err
}
var total, used int64
if y.isFamily() {
total = capacityInfo.FamilyCapacityInfo.TotalSize
used = capacityInfo.FamilyCapacityInfo.UsedSize
} else {
total = capacityInfo.CloudCapacityInfo.TotalSize
used = capacityInfo.CloudCapacityInfo.UsedSize
}
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: total,
UsedSpace: used,
},
}, nil
}
+14
View File
@@ -80,6 +80,20 @@ func timestamp() int64 {
return time.Now().UTC().UnixNano() / 1e6
}
// formatDate formats a time.Time object into the "YYYY-MM-DDHH:mm:ssSSS" format.
func formatDate(t time.Time) string {
// The layout string "2006-01-0215:04:05.000" corresponds to:
// 2006 -> Year (YYYY)
// 01 -> Month (MM)
// 02 -> Day (DD)
// 15 -> Hour (HH)
// 04 -> Minute (mm)
// 05 -> Second (ss)
// 000 -> Millisecond (SSS) with leading zeros
// Note the lack of a separator between the date and hour, matching the desired output.
return t.Format("2006-01-0215:04:05.000")
}
func MustParseTime(str string) *time.Time {
lastOpTime, _ := time.ParseInLocation("2006-01-02 15:04:05 -07", str+" +08", time.Local)
return &lastOpTime
+5 -3
View File
@@ -6,9 +6,11 @@ import (
)
type Addition struct {
Username string `json:"username" required:"true"`
Password string `json:"password" required:"true"`
VCode string `json:"validate_code"`
LoginType string `json:"login_type" type:"select" options:"password,qrcode" default:"password" required:"true"`
Username string `json:"username" required:"true"`
Password string `json:"password" required:"true"`
VCode string `json:"validate_code"`
RefreshToken string `json:"refresh_token" help:"To switch accounts, please clear this field"`
driver.RootID
OrderBy string `json:"order_by" type:"select" options:"filename,filesize,lastOpTime" default:"filename"`
OrderDirection string `json:"order_direction" type:"select" options:"asc,desc" default:"asc"`
+40 -9
View File
@@ -68,15 +68,7 @@ func (e *RespErr) Error() string {
return ""
}
// 登陆需要的参数
type LoginParam struct {
// 加密后的用户名和密码
RsaUsername string
RsaPassword string
// rsa密钥
jRsaKey string
type BaseLoginParam struct {
// 请求头参数
Lt string
ReqId string
@@ -88,6 +80,27 @@ type LoginParam struct {
CaptchaToken string
}
// QRLoginParam 用于暂存二维码登录过程中的参数
type QRLoginParam struct {
BaseLoginParam
UUID string `json:"uuid"`
EncodeUUID string `json:"encodeuuid"`
EncryUUID string `json:"encryuuid"`
}
// 登陆需要的参数
type LoginParam struct {
// 加密后的用户名和密码
RsaUsername string
RsaPassword string
// rsa密钥
jRsaKey string
BaseLoginParam
}
// 登陆加密相关
type EncryptConfResp struct {
Result int `json:"result"`
@@ -396,3 +409,21 @@ func (p Params) Encode() string {
}
return buf.String()
}
type CapacityResp struct {
ResCode int `json:"res_code"`
ResMessage string `json:"res_message"`
Account string `json:"account"`
CloudCapacityInfo struct {
FreeSize int64 `json:"freeSize"`
MailUsedSize int64 `json:"mail189UsedSize"`
TotalSize int64 `json:"totalSize"`
UsedSize int64 `json:"usedSize"`
} `json:"cloudCapacityInfo"`
FamilyCapacityInfo struct {
FreeSize int64 `json:"freeSize"`
TotalSize int64 `json:"totalSize"`
UsedSize int64 `json:"usedSize"`
} `json:"familyCapacityInfo"`
TotalSize uint64 `json:"totalSize"`
}
+351 -94
View File
@@ -7,6 +7,7 @@ import (
"encoding/hex"
"encoding/xml"
"fmt"
"hash"
"io"
"net/http"
"net/http/cookiejar"
@@ -28,6 +29,7 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/errgroup"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/skip2/go-qrcode"
"github.com/avast/retry-go"
"github.com/go-resty/resty/v2"
@@ -53,6 +55,9 @@ const (
MAC = "TELEMAC"
CHANNEL_ID = "web_cloud.189.cn"
// Error codes
UserInvalidOpenTokenError = "UserInvalidOpenToken"
)
func (y *Cloud189PC) SignatureHeader(url, method, params string, isFamily bool) map[string]string {
@@ -85,6 +90,9 @@ func (y *Cloud189PC) EncryptParams(params Params, isFamily bool) string {
}
func (y *Cloud189PC) request(url, method string, callback base.ReqCallback, params Params, resp interface{}, isFamily ...bool) ([]byte, error) {
if y.getTokenInfo() == nil {
return nil, fmt.Errorf("login failed")
}
req := y.getClient().R().SetQueryParams(clientSuffix())
// 设置params
@@ -184,6 +192,7 @@ func (y *Cloud189PC) put(ctx context.Context, url string, headers map[string]str
}
return body, nil
}
func (y *Cloud189PC) getFiles(ctx context.Context, fileId string, isFamily bool) ([]model.Obj, error) {
res := make([]model.Obj, 0, 100)
for pageNum := 1; ; pageNum++ {
@@ -263,7 +272,14 @@ func (y *Cloud189PC) findFileByName(ctx context.Context, searchName string, fold
}
}
func (y *Cloud189PC) login() (err error) {
func (y *Cloud189PC) login() error {
if y.LoginType == "qrcode" {
return y.loginByQRCode()
}
return y.loginByPassword()
}
func (y *Cloud189PC) loginByPassword() (err error) {
// 初始化登陆所需参数
if y.loginParam == nil {
if err = y.initLoginParam(); err != nil {
@@ -277,10 +293,15 @@ func (y *Cloud189PC) login() (err error) {
// 销毁登陆参数
y.loginParam = nil
// 遇到错误,重新加载登陆参数(刷新验证码)
if err != nil && y.NoUseOcr {
if err1 := y.initLoginParam(); err1 != nil {
err = fmt.Errorf("err1: %s \nerr2: %s", err, err1)
if err != nil {
if y.NoUseOcr {
if err1 := y.initLoginParam(); err1 != nil {
err = fmt.Errorf("err1: %s \nerr2: %s", err, err1)
}
}
y.Status = err.Error()
op.MustSaveDriverStorage(y)
}
}()
@@ -325,7 +346,7 @@ func (y *Cloud189PC) login() (err error) {
SetQueryParam("redirectURL", loginresp.ToUrl).
Post(API_URL + "/getSessionForPC.action")
if err != nil {
return
return err
}
if erron.HasError() {
@@ -333,16 +354,106 @@ func (y *Cloud189PC) login() (err error) {
}
if tokenInfo.ResCode != 0 {
err = fmt.Errorf(tokenInfo.ResMessage)
return
return err
}
y.Addition.RefreshToken = tokenInfo.RefreshToken
y.tokenInfo = &tokenInfo
return
op.MustSaveDriverStorage(y)
return err
}
/* 初始化登陆需要的参数
* 如果遇到验证码返回错误
*/
func (y *Cloud189PC) initLoginParam() error {
func (y *Cloud189PC) loginByQRCode() error {
if y.qrcodeParam == nil {
if err := y.initQRCodeParam(); err != nil {
// 二维码也通过错误返回
return err
}
}
var state struct {
Status int `json:"status"`
RedirectUrl string `json:"redirectUrl"`
Msg string `json:"msg"`
}
now := time.Now()
_, err := y.client.R().
SetHeaders(map[string]string{
"Referer": AUTH_URL,
"Reqid": y.qrcodeParam.ReqId,
"lt": y.qrcodeParam.Lt,
}).
SetFormData(map[string]string{
"appId": APP_ID,
"clientType": CLIENT_TYPE,
"returnUrl": RETURN_URL,
"paramId": y.qrcodeParam.ParamId,
"uuid": y.qrcodeParam.UUID,
"encryuuid": y.qrcodeParam.EncryUUID,
"date": formatDate(now),
"timeStamp": fmt.Sprint(now.UTC().UnixNano() / 1e6),
}).
ForceContentType("application/json;charset=UTF-8").
SetResult(&state).
Post(AUTH_URL + "/api/logbox/oauth2/qrcodeLoginState.do")
if err != nil {
return fmt.Errorf("failed to check QR code state: %w", err)
}
switch state.Status {
case 0: // 登录成功
var tokenInfo AppSessionResp
_, err = y.client.R().
SetResult(&tokenInfo).
SetQueryParams(clientSuffix()).
SetQueryParam("redirectURL", state.RedirectUrl).
Post(API_URL + "/getSessionForPC.action")
if err != nil {
return err
}
if tokenInfo.ResCode != 0 {
return fmt.Errorf(tokenInfo.ResMessage)
}
y.Addition.RefreshToken = tokenInfo.RefreshToken
y.tokenInfo = &tokenInfo
op.MustSaveDriverStorage(y)
return nil
case -11001: // 二维码过期
y.qrcodeParam = nil
return errors.New("QR code expired, please try again")
case -106: // 等待扫描
return y.genQRCode("QR code has not been scanned yet, please scan and save again")
case -11002: // 等待确认
return y.genQRCode("QR code has been scanned, please confirm the login on your phone and save again")
default: // 其他错误
y.qrcodeParam = nil
return fmt.Errorf("QR code login failed with status %d: %s", state.Status, state.Msg)
}
}
func (y *Cloud189PC) genQRCode(text string) error {
// 展示二维码
qrTemplate := `<body>
state: %s
<br><img src="data:image/jpeg;base64,%s"/>
<br>Or Click here: <a href="%s">Login</a>
</body>`
// Generate QR code
qrCode, err := qrcode.Encode(y.qrcodeParam.UUID, qrcode.Medium, 256)
if err != nil {
return fmt.Errorf("failed to generate QR code: %v", err)
}
// Encode QR code to base64
qrCodeBase64 := base64.StdEncoding.EncodeToString(qrCode)
// Create the HTML page
qrPage := fmt.Sprintf(qrTemplate, text, qrCodeBase64, y.qrcodeParam.UUID)
return fmt.Errorf("need verify: \n%s", qrPage)
}
func (y *Cloud189PC) initBaseParams() (*BaseLoginParam, error) {
// 清除cookie
jar, _ := cookiejar.New(nil)
y.client.SetCookieJar(jar)
@@ -356,17 +467,30 @@ func (y *Cloud189PC) initLoginParam() error {
}).
Get(WEB_URL + "/api/portal/unifyLoginForPC.action")
if err != nil {
return err
return nil, err
}
param := LoginParam{
return &BaseLoginParam{
CaptchaToken: regexp.MustCompile(`'captchaToken' value='(.+?)'`).FindStringSubmatch(res.String())[1],
Lt: regexp.MustCompile(`lt = "(.+?)"`).FindStringSubmatch(res.String())[1],
ParamId: regexp.MustCompile(`paramId = "(.+?)"`).FindStringSubmatch(res.String())[1],
ReqId: regexp.MustCompile(`reqId = "(.+?)"`).FindStringSubmatch(res.String())[1],
// jRsaKey: regexp.MustCompile(`"j_rsaKey" value="(.+?)"`).FindStringSubmatch(res.String())[1],
}, nil
}
/* 初始化登陆需要的参数
* 如果遇到验证码返回错误
*/
func (y *Cloud189PC) initLoginParam() error {
y.loginParam = nil
baseParam, err := y.initBaseParams()
if err != nil {
return err
}
y.loginParam = &LoginParam{BaseLoginParam: *baseParam}
// 获取rsa公钥
var encryptConf EncryptConfResp
_, err = y.client.R().
@@ -377,18 +501,17 @@ func (y *Cloud189PC) initLoginParam() error {
return err
}
param.jRsaKey = fmt.Sprintf("-----BEGIN PUBLIC KEY-----\n%s\n-----END PUBLIC KEY-----", encryptConf.Data.PubKey)
param.RsaUsername = encryptConf.Data.Pre + RsaEncrypt(param.jRsaKey, y.Username)
param.RsaPassword = encryptConf.Data.Pre + RsaEncrypt(param.jRsaKey, y.Password)
y.loginParam = &param
y.loginParam.jRsaKey = fmt.Sprintf("-----BEGIN PUBLIC KEY-----\n%s\n-----END PUBLIC KEY-----", encryptConf.Data.PubKey)
y.loginParam.RsaUsername = encryptConf.Data.Pre + RsaEncrypt(y.loginParam.jRsaKey, y.Username)
y.loginParam.RsaPassword = encryptConf.Data.Pre + RsaEncrypt(y.loginParam.jRsaKey, y.Password)
// 判断是否需要验证码
resp, err := y.client.R().
SetHeader("REQID", param.ReqId).
SetHeader("REQID", y.loginParam.ReqId).
SetFormData(map[string]string{
"appKey": APP_ID,
"accountType": ACCOUNT_TYPE,
"userName": param.RsaUsername,
"userName": y.loginParam.RsaUsername,
}).Post(AUTH_URL + "/api/logbox/oauth2/needcaptcha.do")
if err != nil {
return err
@@ -400,8 +523,8 @@ func (y *Cloud189PC) initLoginParam() error {
// 拉取验证码
imgRes, err := y.client.R().
SetQueryParams(map[string]string{
"token": param.CaptchaToken,
"REQID": param.ReqId,
"token": y.loginParam.CaptchaToken,
"REQID": y.loginParam.ReqId,
"rnd": fmt.Sprint(timestamp()),
}).
Get(AUTH_URL + "/api/logbox/oauth2/picCaptcha.do")
@@ -428,10 +551,38 @@ func (y *Cloud189PC) initLoginParam() error {
return nil
}
// getQRCode 获取并返回二维码
func (y *Cloud189PC) initQRCodeParam() (err error) {
y.qrcodeParam = nil
baseParam, err := y.initBaseParams()
if err != nil {
return err
}
var qrcodeParam QRLoginParam
_, err = y.client.R().
SetFormData(map[string]string{"appId": APP_ID}).
ForceContentType("application/json;charset=UTF-8").
SetResult(&qrcodeParam).
Post(AUTH_URL + "/api/logbox/oauth2/getUUID.do")
if err != nil {
return err
}
qrcodeParam.BaseLoginParam = *baseParam
y.qrcodeParam = &qrcodeParam
return y.genQRCode("please scan the QR code with the 189 Cloud app, then save the settings again.")
}
// 刷新会话
func (y *Cloud189PC) refreshSession() (err error) {
return y.refreshSessionWithRetry(0)
}
func (y *Cloud189PC) refreshSessionWithRetry(retryCount int) (err error) {
if y.ref != nil {
return y.ref.refreshSession()
return y.ref.refreshSessionWithRetry(retryCount)
}
var erron RespErr
var userSessionResp UserSessionResp
@@ -448,37 +599,102 @@ func (y *Cloud189PC) refreshSession() (err error) {
return err
}
// 错误影响正常访问,下线该储存
defer func() {
if err != nil {
y.GetStorage().SetStatus(fmt.Sprintf("%+v", err.Error()))
op.MustSaveDriverStorage(y)
}
}()
// token生效刷新token
if erron.HasError() {
if erron.ResCode == "UserInvalidOpenToken" {
if err = y.login(); err != nil {
return err
}
if erron.ResCode == UserInvalidOpenTokenError {
return y.refreshTokenWithRetry(retryCount)
}
return &erron
}
y.tokenInfo.UserSessionResp = userSessionResp
return
return nil
}
// refreshToken 刷新token,失败时返回错误,不再直接调用login
func (y *Cloud189PC) refreshToken() (err error) {
return y.refreshTokenWithRetry(0)
}
func (y *Cloud189PC) refreshTokenWithRetry(retryCount int) (err error) {
if y.ref != nil {
return y.ref.refreshTokenWithRetry(retryCount)
}
// 限制重试次数,避免无限递归
if retryCount >= 3 {
if y.Addition.RefreshToken != "" {
y.Addition.RefreshToken = ""
op.MustSaveDriverStorage(y)
}
return errors.New("refresh token failed after maximum retries")
}
var erron RespErr
var tokenInfo AppSessionResp
_, err = y.client.R().
SetResult(&tokenInfo).
ForceContentType("application/json;charset=UTF-8").
SetError(&erron).
SetFormData(map[string]string{
"clientId": APP_ID,
"refreshToken": y.tokenInfo.RefreshToken,
"grantType": "refresh_token",
"format": "json",
}).
Post(AUTH_URL + "/api/oauth2/refreshToken.do")
if err != nil {
return err
}
// 如果刷新失败,返回错误给上层处理
if erron.HasError() {
if y.Addition.RefreshToken != "" {
y.Addition.RefreshToken = ""
op.MustSaveDriverStorage(y)
}
// 根据登录类型决定下一步行为
if y.LoginType == "qrcode" {
return errors.New("QR code session has expired, please re-scan the code to log in")
}
// 密码登录模式下,尝试回退到完整登录
return y.login()
}
y.Addition.RefreshToken = tokenInfo.RefreshToken
y.tokenInfo = &tokenInfo
op.MustSaveDriverStorage(y)
return y.refreshSessionWithRetry(retryCount + 1)
}
func (y *Cloud189PC) keepAlive() {
_, err := y.get(API_URL+"/keepUserSession.action", func(r *resty.Request) {
r.SetQueryParams(clientSuffix())
}, nil)
if err != nil {
utils.Log.Warnf("189pc: Failed to keep user session alive: %v", err)
// 如果keepAlive失败,尝试刷新session
if refreshErr := y.refreshSession(); refreshErr != nil {
utils.Log.Errorf("189pc: Failed to refresh session after keepAlive error: %v", refreshErr)
}
} else {
utils.Log.Debugf("189pc: User session kept alive successfully.")
}
}
// 普通上传
// 无法上传大小为0的文件
func (y *Cloud189PC) StreamUpload(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress, isFamily bool, overwrite bool) (model.Obj, error) {
size := file.GetSize()
sliceSize := partSize(size)
// 文件大小
fileSize := file.GetSize()
// 分片大小,不得为文件大小
sliceSize := partSize(fileSize)
params := Params{
"parentFolderId": dstDir.GetID(),
"fileName": url.QueryEscape(file.GetName()),
"fileSize": fmt.Sprint(file.GetSize()),
"sliceSize": fmt.Sprint(sliceSize),
"fileSize": fmt.Sprint(fileSize),
"sliceSize": fmt.Sprint(sliceSize), // 必须为特定分片大小
"lazyCheck": "1",
}
@@ -487,7 +703,7 @@ func (y *Cloud189PC) StreamUpload(ctx context.Context, dstDir model.Obj, file mo
params.Set("familyId", y.FamilyID)
fullUrl += "/family"
} else {
//params.Set("extend", `{"opScene":"1","relativepath":"","rootfolderid":""}`)
// params.Set("extend", `{"opScene":"1","relativepath":"","rootfolderid":""}`)
fullUrl += "/person"
}
@@ -500,66 +716,95 @@ func (y *Cloud189PC) StreamUpload(ctx context.Context, dstDir model.Obj, file mo
return nil, err
}
threadG, upCtx := errgroup.NewGroupWithContext(ctx, y.uploadThread,
ss, err := stream.NewStreamSectionReader(file, int(sliceSize), &up)
if err != nil {
return nil, err
}
threadG, upCtx := errgroup.NewOrderedGroupWithContext(ctx, y.uploadThread,
retry.Attempts(3),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay))
count := int(size / sliceSize)
lastPartSize := size % sliceSize
if lastPartSize > 0 {
count++
} else {
count := 1
if fileSize > sliceSize {
count = int((fileSize + sliceSize - 1) / sliceSize)
}
lastPartSize := fileSize % sliceSize
if lastPartSize == 0 {
lastPartSize = sliceSize
}
fileMd5 := utils.MD5.NewFunc()
silceMd5 := utils.MD5.NewFunc()
silceMd5Hexs := make([]string, 0, count)
teeReader := io.TeeReader(file, io.MultiWriter(fileMd5, silceMd5))
byteSize := sliceSize
silceMd5 := utils.MD5.NewFunc()
var writers io.Writer = silceMd5
fileMd5Hex := file.GetHash().GetHash(utils.MD5)
var fileMd5 hash.Hash
if len(fileMd5Hex) != utils.MD5.Width {
fileMd5 = utils.MD5.NewFunc()
writers = io.MultiWriter(silceMd5, fileMd5)
}
for i := 1; i <= count; i++ {
if utils.IsCanceled(upCtx) {
break
}
offset := int64((i)-1) * sliceSize
partSize := sliceSize
if i == count {
byteSize = lastPartSize
}
byteData := make([]byte, byteSize)
// 读取块
silceMd5.Reset()
if _, err := io.ReadFull(teeReader, byteData); err != io.EOF && err != nil {
return nil, err
partSize = lastPartSize
}
partInfo := ""
var reader io.ReadSeeker
threadG.GoWithLifecycle(errgroup.Lifecycle{
Before: func(ctx context.Context) (err error) {
reader, err = ss.GetSectionReader(offset, partSize)
if err != nil {
return err
}
silceMd5.Reset()
w, err := utils.CopyWithBuffer(writers, reader)
if w != partSize {
return fmt.Errorf("failed to read all data: (expect =%d, actual =%d) %w", partSize, w, err)
}
// 计算块md5并进行hex和base64编码
md5Bytes := silceMd5.Sum(nil)
silceMd5Hexs = append(silceMd5Hexs, strings.ToUpper(hex.EncodeToString(md5Bytes)))
partInfo = fmt.Sprintf("%d-%s", i, base64.StdEncoding.EncodeToString(md5Bytes))
return nil
},
Do: func(ctx context.Context) (err error) {
reader.Seek(0, io.SeekStart)
uploadUrls, err := y.GetMultiUploadUrls(ctx, isFamily, initMultiUpload.Data.UploadFileID, partInfo)
if err != nil {
return err
}
// 计算块md5并进行hex和base64编码
md5Bytes := silceMd5.Sum(nil)
silceMd5Hexs = append(silceMd5Hexs, strings.ToUpper(hex.EncodeToString(md5Bytes)))
partInfo := fmt.Sprintf("%d-%s", i, base64.StdEncoding.EncodeToString(md5Bytes))
threadG.Go(func(ctx context.Context) error {
uploadUrls, err := y.GetMultiUploadUrls(ctx, isFamily, initMultiUpload.Data.UploadFileID, partInfo)
if err != nil {
return err
}
// step.4 上传切片
uploadUrl := uploadUrls[0]
_, err = y.put(ctx, uploadUrl.RequestURL, uploadUrl.Headers, false,
driver.NewLimitedUploadStream(ctx, bytes.NewReader(byteData)), isFamily)
if err != nil {
return err
}
up(float64(threadG.Success()) * 100 / float64(count))
return nil
})
// step.4 上传切片
uploadUrl := uploadUrls[0]
_, err = y.put(ctx, uploadUrl.RequestURL, uploadUrl.Headers, false, driver.NewLimitedUploadStream(ctx, reader), isFamily)
if err != nil {
return err
}
up(float64(threadG.Success()+1) * 100 / float64(count+1))
return nil
},
After: func(err error) {
ss.FreeSectionReader(reader)
},
},
)
}
if err = threadG.Wait(); err != nil {
return nil, err
}
defer up(100)
fileMd5Hex := strings.ToUpper(hex.EncodeToString(fileMd5.Sum(nil)))
if fileMd5 != nil {
fileMd5Hex = strings.ToUpper(hex.EncodeToString(fileMd5.Sum(nil)))
}
sliceMd5Hex := fileMd5Hex
if file.GetSize() > sliceSize {
if fileSize > sliceSize {
sliceMd5Hex = strings.ToUpper(utils.GetMD5EncodeStr(strings.Join(silceMd5Hexs, "\n")))
}
@@ -620,15 +865,16 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
cache = tmpF
}
sliceSize := partSize(size)
count := int(size / sliceSize)
count := 1
if size > sliceSize {
count = int((size + sliceSize - 1) / sliceSize)
}
lastSliceSize := size % sliceSize
if lastSliceSize > 0 {
count++
} else {
if lastSliceSize == 0 {
lastSliceSize = sliceSize
}
//step.1 优先计算所需信息
// step.1 优先计算所需信息
byteSize := sliceSize
fileMd5 := utils.MD5.NewFunc()
sliceMd5 := utils.MD5.NewFunc()
@@ -679,14 +925,14 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
if isFamily {
fullUrl += "/family"
} else {
//params.Set("extend", `{"opScene":"1","relativepath":"","rootfolderid":""}`)
// params.Set("extend", `{"opScene":"1","relativepath":"","rootfolderid":""}`)
fullUrl += "/person"
}
// 尝试恢复进度
uploadProgress, ok := base.GetUploadProgress[*UploadProgress](y, y.getTokenInfo().SessionKey, fileMd5Hex)
if !ok {
//step.2 预上传
// step.2 预上传
params := Params{
"parentFolderId": dstDir.GetID(),
"fileName": url.QueryEscape(file.GetName()),
@@ -738,12 +984,13 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
}
// step.4 上传切片
_, err = y.put(ctx, uploadUrl.RequestURL, uploadUrl.Headers, false, io.NewSectionReader(cache, offset, byteSize), isFamily)
rateLimitedRd := driver.NewLimitedUploadStream(ctx, io.NewSectionReader(cache, offset, byteSize))
_, err = y.put(ctx, uploadUrl.RequestURL, uploadUrl.Headers, false, rateLimitedRd, isFamily)
if err != nil {
return err
}
up(float64(threadG.Success()) * 100 / float64(len(uploadUrls)))
up(float64(threadG.Success()+1) * 100 / float64(len(uploadUrls)+1))
uploadProgress.UploadParts[i] = ""
return nil
})
@@ -755,6 +1002,7 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
}
return nil, err
}
defer up(100)
}
// step.5 提交
@@ -820,9 +1068,7 @@ func (y *Cloud189PC) GetMultiUploadUrls(ctx context.Context, isFamily bool, uplo
// 旧版本上传,家庭云不支持覆盖
func (y *Cloud189PC) OldUpload(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress, isFamily bool, overwrite bool) (model.Obj, error) {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
tempFile, fileMd5, err := stream.CacheFullInTempFileAndHash(file, cacheFileProgress, utils.MD5)
tempFile, fileMd5, err := stream.CacheFullAndHash(file, &up, utils.MD5)
if err != nil {
return nil, err
}
@@ -916,7 +1162,6 @@ func (y *Cloud189PC) OldUploadCreate(ctx context.Context, parentID string, fileM
})
}
}, &uploadInfo, isFamily)
if err != nil {
return nil, err
}
@@ -1226,3 +1471,15 @@ func (y *Cloud189PC) getClient() *resty.Client {
}
return y.client
}
func (y *Cloud189PC) getCapacityInfo(ctx context.Context) (*CapacityResp, error) {
fullUrl := API_URL + "/portal/getUserSizeInfo.action"
var resp CapacityResp
_, err := y.get(fullUrl, func(req *resty.Request) {
req.SetContext(ctx)
}, &resp)
if err != nil {
return nil, err
}
return &resp, nil
}
+405 -248
View File
@@ -5,6 +5,8 @@ import (
"errors"
"fmt"
"io"
"math/rand"
"net/url"
stdpath "path"
"strings"
@@ -12,8 +14,10 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/fs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/sign"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/http_range"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/server/common"
)
@@ -21,9 +25,9 @@ import (
type Alias struct {
model.Storage
Addition
pathMap map[string][]string
autoFlatten bool
oneKey string
rootOrder []string
pathMap map[string][]string
root model.Obj
}
func (d *Alias) Config() driver.Config {
@@ -35,267 +39,400 @@ func (d *Alias) GetAddition() driver.Additional {
}
func (d *Alias) Init(ctx context.Context) error {
if d.Paths == "" {
return errors.New("paths is required")
}
paths := strings.Split(d.Paths, "\n")
d.rootOrder = make([]string, 0, len(paths))
d.pathMap = make(map[string][]string)
for _, path := range strings.Split(d.Paths, "\n") {
for _, path := range paths {
path = strings.TrimSpace(path)
if path == "" {
continue
}
k, v := getPair(path)
d.pathMap[k] = append(d.pathMap[k], v)
}
if len(d.pathMap) == 1 {
for k := range d.pathMap {
d.oneKey = k
temp, ok := d.pathMap[k]
if !ok {
d.rootOrder = append(d.rootOrder, k)
}
d.autoFlatten = true
} else {
d.oneKey = ""
d.autoFlatten = false
d.pathMap[k] = append(temp, v)
}
switch len(d.rootOrder) {
case 0:
return errors.New("paths is required")
case 1:
paths := d.pathMap[d.rootOrder[0]]
roots := make(BalancedObjs, 0, len(paths))
roots = append(roots, &model.Object{
Name: "root",
Path: paths[0],
IsFolder: true,
Modified: d.Modified,
Mask: model.Locked,
})
for _, path := range paths[1:] {
roots = append(roots, &model.Object{
Path: path,
})
}
d.root = roots
default:
d.root = &model.Object{
Name: "root",
Path: "/",
IsFolder: true,
Modified: d.Modified,
Mask: model.ReadOnly,
}
}
if !utils.SliceContains(ValidReadConflictPolicy, d.ReadConflictPolicy) {
d.ReadConflictPolicy = FirstRWP
}
if !utils.SliceContains(ValidWriteConflictPolicy, d.WriteConflictPolicy) {
d.WriteConflictPolicy = DisabledWP
}
if !utils.SliceContains(ValidPutConflictPolicy, d.PutConflictPolicy) {
d.PutConflictPolicy = DisabledWP
}
return nil
}
func (d *Alias) Drop(ctx context.Context) error {
d.rootOrder = nil
d.pathMap = nil
d.root = nil
return nil
}
func (d *Alias) Get(ctx context.Context, path string) (model.Obj, error) {
if utils.PathEqual(path, "/") {
return &model.Object{
Name: "Root",
IsFolder: true,
Path: "/",
}, nil
func (d *Alias) GetRoot(ctx context.Context) (model.Obj, error) {
if d.root == nil {
return nil, errs.StorageNotInit
}
root, sub := d.getRootAndPath(path)
dsts, ok := d.pathMap[root]
if !ok {
return d.root, nil
}
// 通过op.Get调用的话,path一定是子路径(/开头)
func (d *Alias) Get(ctx context.Context, path string) (model.Obj, error) {
roots, sub := d.getRootsAndPath(path)
if len(roots) == 0 {
return nil, errs.ObjectNotFound
}
for _, dst := range dsts {
obj, err := d.get(ctx, path, dst, sub)
if err == nil {
return obj, nil
for idx, root := range roots {
rawPath := stdpath.Join(root, sub)
obj, err := fs.Get(ctx, rawPath, &fs.GetArgs{NoLog: true})
if err != nil {
continue
}
mask := model.GetObjMask(obj) &^ model.Temp
if sub == "" {
// 根目录
mask |= model.Locked | model.Virtual
}
ret := model.Object{
Path: rawPath,
Name: obj.GetName(),
Size: obj.GetSize(),
Modified: obj.ModTime(),
IsFolder: obj.IsDir(),
HashInfo: obj.GetHash(),
Mask: mask,
}
obj = &ret
if d.ProviderPassThrough && !obj.IsDir() {
if storage, err := fs.GetStorage(rawPath, &fs.GetStoragesArgs{}); err == nil {
obj = &model.ObjectProvider{
Object: ret,
Provider: model.Provider{
Provider: storage.Config().Name,
},
}
}
}
roots = roots[idx+1:]
var objs BalancedObjs
if idx > 0 {
objs = make(BalancedObjs, 0, len(roots)+2)
} else {
objs = make(BalancedObjs, 0, len(roots)+1)
}
objs = append(objs, obj)
if idx > 0 {
objs = append(objs, nil)
}
for _, d := range roots {
objs = append(objs, &tempObj{model.Object{
Path: stdpath.Join(d, sub),
}})
}
return objs, nil
}
return nil, errs.ObjectNotFound
}
func (d *Alias) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
path := dir.GetPath()
if utils.PathEqual(path, "/") && !d.autoFlatten {
return d.listRoot(), nil
}
root, sub := d.getRootAndPath(path)
dsts, ok := d.pathMap[root]
dirs, ok := dir.(BalancedObjs)
if !ok {
return nil, errs.ObjectNotFound
return d.listRoot(ctx, args.WithStorageDetails && d.DetailsPassThrough, args.Refresh), nil
}
var objs []model.Obj
fsArgs := &fs.ListArgs{NoLog: true, Refresh: args.Refresh}
for _, dst := range dsts {
tmp, err := d.list(ctx, dst, sub, fsArgs)
if err == nil {
objs = append(objs, tmp...)
// 因为alias是NoCache且Get方法不会返回NotSupport或NotImplement错误
// 所以这里对象不会传回到alias,也就不需要返回BalancedObjs了
objMap := make(map[string]model.Obj)
for _, dir := range dirs {
if dir == nil {
continue
}
dirPath := dir.GetPath()
tmp, err := fs.List(ctx, dirPath, &fs.ListArgs{
NoLog: true,
Refresh: args.Refresh,
WithStorageDetails: args.WithStorageDetails && d.DetailsPassThrough,
})
if err != nil {
continue
}
for _, obj := range tmp {
name := obj.GetName()
if _, exists := objMap[name]; exists {
continue
}
mask := model.GetObjMask(obj) &^ model.Temp
objRes := model.Object{
Name: name,
Path: stdpath.Join(dirPath, name),
Size: obj.GetSize(),
Modified: obj.ModTime(),
IsFolder: obj.IsDir(),
Mask: mask,
}
var objRet model.Obj
if thumb, ok := model.GetThumb(obj); ok {
objRet = &model.ObjThumb{
Object: objRes,
Thumbnail: model.Thumbnail{
Thumbnail: thumb,
},
}
} else {
objRet = &objRes
}
if details, ok := model.GetStorageDetails(obj); ok {
objRet = &model.ObjStorageDetails{
Obj: objRet,
StorageDetails: details,
}
}
objMap[name] = objRet
}
}
objs := make([]model.Obj, 0, len(objMap))
for _, obj := range objMap {
objs = append(objs, obj)
}
if d.OrderBy == "" {
sort := getAllSort(dirs)
if sort.OrderBy != "" {
model.SortFiles(objs, sort.OrderBy, sort.OrderDirection)
}
if d.ExtractFolder == "" && sort.ExtractFolder != "" {
model.ExtractFolder(objs, sort.ExtractFolder)
}
}
return objs, nil
}
func (d *Alias) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
root, sub := d.getRootAndPath(file.GetPath())
dsts, ok := d.pathMap[root]
if !ok {
return nil, errs.ObjectNotFound
}
for _, dst := range dsts {
reqPath := stdpath.Join(dst, sub)
link, file, err := d.link(ctx, reqPath, args)
if d.ReadConflictPolicy == AllRWP && !args.Redirect {
files, err := d.getAllObjs(ctx, file, getWriteAndPutFilterFunc(AllRWP))
if err != nil {
continue
return nil, err
}
var resultLink *model.Link
if link != nil {
resultLink = &model.Link{
URL: link.URL,
Header: link.Header,
RangeReader: link.RangeReader,
SyncClosers: utils.NewSyncClosers(link),
ContentLength: link.ContentLength,
linkClosers := make([]io.Closer, 0, len(files))
rrf := make([]model.RangeReaderIF, 0, len(files))
for _, f := range files {
link, fi, err := d.link(ctx, f.GetPath(), args)
if err != nil {
continue
}
if link.MFile != nil {
resultLink.RangeReader = &model.FileRangeReader{
RangeReaderIF: stream.GetRangeReaderFromMFile(file.GetSize(), link.MFile),
}
if fi.GetSize() != files.GetSize() {
_ = link.Close()
continue
}
} else {
resultLink = &model.Link{
URL: fmt.Sprintf("%s/p%s?sign=%s",
common.GetApiUrl(ctx),
utils.EncodePath(reqPath, true),
sign.Sign(reqPath)),
l := *link // 复制一份,避免修改到原始link
if l.ContentLength == 0 {
l.ContentLength = fi.GetSize()
}
}
if !args.Redirect {
if d.DownloadConcurrency > 0 {
resultLink.Concurrency = d.DownloadConcurrency
l.Concurrency = d.DownloadConcurrency
}
if d.DownloadPartSize > 0 {
resultLink.PartSize = d.DownloadPartSize * utils.KB
l.PartSize = d.DownloadPartSize * utils.KB
}
rr, err := stream.GetRangeReaderFromLink(l.ContentLength, &l)
if err != nil {
_ = link.Close()
continue
}
linkClosers = append(linkClosers, link)
rrf = append(rrf, rr)
}
return resultLink, nil
rr := func(ctx context.Context, httpRange http_range.Range) (io.ReadCloser, error) {
return rrf[rand.Intn(len(rrf))].RangeRead(ctx, httpRange)
}
return &model.Link{
RangeReader: stream.RangeReaderFunc(rr),
SyncClosers: utils.NewSyncClosers(linkClosers...),
}, nil
}
return nil, errs.ObjectNotFound
var link *model.Link
var fi model.Obj
var err error
files := file.(BalancedObjs)
if d.ReadConflictPolicy == RandomBalancedRP || d.ReadConflictPolicy == AllRWP {
rand.Shuffle(len(files), func(i, j int) {
files[i], files[j] = files[j], files[i]
})
}
for _, f := range files {
if f == nil {
continue
}
link, fi, err = d.link(ctx, f.GetPath(), args)
if err == nil {
if link == nil {
// 重定向且需要通过代理
return &model.Link{
URL: fmt.Sprintf("%s/p%s?sign=%s",
common.GetApiUrl(ctx),
utils.EncodePath(f.GetPath(), true),
sign.Sign(f.GetPath())),
}, nil
}
break
}
}
if err != nil {
return nil, err
}
resultLink := *link // 复制一份,避免修改到原始link
resultLink.Expiration = nil
resultLink.SyncClosers = utils.NewSyncClosers(link)
if args.Redirect {
return &resultLink, nil
}
if resultLink.ContentLength == 0 {
resultLink.ContentLength = fi.GetSize()
}
if d.DownloadConcurrency > 0 {
resultLink.Concurrency = d.DownloadConcurrency
}
if d.DownloadPartSize > 0 {
resultLink.PartSize = d.DownloadPartSize * utils.KB
}
return &resultLink, nil
}
func (d *Alias) Other(ctx context.Context, args model.OtherArgs) (interface{}, error) {
// Other 不应负载均衡,这是因为前端是否调用 /fs/other 的判断条件是返回的 provider 的值
// 而 ProviderPassThrough 开启时,返回的 provider 固定为第一个 obj 的后端驱动
storage, actualPath, err := op.GetStorageAndActualPath(args.Obj.GetPath())
if err != nil {
return nil, err
}
return op.Other(ctx, storage, model.FsOtherArgs{
Path: actualPath,
Method: args.Method,
Data: args.Data,
})
}
func (d *Alias) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) error {
if !d.Writable {
return errs.PermissionDenied
}
reqPath, err := d.getReqPath(ctx, parentDir, true)
objs, err := d.getWriteObjs(ctx, parentDir)
if err == nil {
for _, path := range reqPath {
err = errors.Join(err, fs.MakeDir(ctx, stdpath.Join(*path, dirName)))
for _, obj := range objs {
err = errors.Join(err, fs.MakeDir(ctx, stdpath.Join(obj.GetPath(), dirName)))
}
return err
}
if errs.IsNotImplement(err) {
return errors.New("same-name dirs cannot make sub-dir")
}
return err
}
func (d *Alias) Move(ctx context.Context, srcObj, dstDir model.Obj) error {
if !d.Writable {
return errs.PermissionDenied
}
srcPath, err := d.getReqPath(ctx, srcObj, false)
if errs.IsNotImplement(err) {
return errors.New("same-name files cannot be moved")
}
if err != nil {
return err
}
dstPath, err := d.getReqPath(ctx, dstDir, true)
if errs.IsNotImplement(err) {
return errors.New("same-name dirs cannot be moved to")
}
if err != nil {
return err
}
if len(srcPath) == len(dstPath) {
for i := range srcPath {
_, e := fs.Move(ctx, *srcPath[i], *dstPath[i])
srcs, dsts, err := d.getMoveObjs(ctx, srcObj, dstDir)
if err == nil {
for i, dst := range dsts {
src := srcs[i]
_, e := fs.Move(ctx, src.GetPath(), dst.GetPath())
err = errors.Join(err, e)
}
srcs = srcs[len(dsts):]
for _, src := range srcs {
e := fs.Remove(ctx, src.GetPath())
err = errors.Join(err, e)
}
return err
} else {
return errors.New("parallel paths mismatch")
}
return err
}
func (d *Alias) Rename(ctx context.Context, srcObj model.Obj, newName string) error {
if !d.Writable {
return errs.PermissionDenied
}
reqPath, err := d.getReqPath(ctx, srcObj, false)
objs, err := d.getWriteObjs(ctx, srcObj)
if err == nil {
for _, path := range reqPath {
err = errors.Join(err, fs.Rename(ctx, *path, newName))
for _, obj := range objs {
err = errors.Join(err, fs.Rename(ctx, obj.GetPath(), newName))
}
return err
}
if errs.IsNotImplement(err) {
return errors.New("same-name files cannot be Rename")
}
return err
}
func (d *Alias) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
if !d.Writable {
return errs.PermissionDenied
}
srcPath, err := d.getReqPath(ctx, srcObj, false)
if errs.IsNotImplement(err) {
return errors.New("same-name files cannot be copied")
}
if err != nil {
return err
}
dstPath, err := d.getReqPath(ctx, dstDir, true)
if errs.IsNotImplement(err) {
return errors.New("same-name dirs cannot be copied to")
}
if err != nil {
return err
}
if len(srcPath) == len(dstPath) {
for i := range srcPath {
_, e := fs.Copy(ctx, *srcPath[i], *dstPath[i])
srcs, dsts, err := d.getCopyObjs(ctx, srcObj, dstDir)
if err == nil {
for i, src := range srcs {
dst := dsts[i]
_, e := fs.Copy(ctx, src.GetPath(), dst.GetPath())
err = errors.Join(err, e)
}
return err
} else if len(srcPath) == 1 || !d.ProtectSameName {
for _, path := range dstPath {
_, e := fs.Copy(ctx, *srcPath[0], *path)
err = errors.Join(err, e)
}
return err
} else {
return errors.New("parallel paths mismatch")
}
return err
}
func (d *Alias) Remove(ctx context.Context, obj model.Obj) error {
if !d.Writable {
return errs.PermissionDenied
}
reqPath, err := d.getReqPath(ctx, obj, false)
objs, err := d.getWriteObjs(ctx, obj)
if err == nil {
for _, path := range reqPath {
err = errors.Join(err, fs.Remove(ctx, *path))
for _, obj := range objs {
err = errors.Join(err, fs.Remove(ctx, obj.GetPath()))
}
return err
}
if errs.IsNotImplement(err) {
return errors.New("same-name files cannot be Delete")
}
return err
}
func (d *Alias) Put(ctx context.Context, dstDir model.Obj, s model.FileStreamer, up driver.UpdateProgress) error {
if !d.Writable {
return errs.PermissionDenied
}
reqPath, err := d.getReqPath(ctx, dstDir, true)
objs, err := d.getPutObjs(ctx, dstDir)
if err == nil {
if len(reqPath) == 1 {
return fs.PutDirectly(ctx, *reqPath[0], &stream.FileStream{
Obj: s,
Mimetype: s.GetMimetype(),
WebPutAsTask: s.NeedStore(),
Reader: s,
})
} else {
file, err := s.CacheFullInTempFile()
if len(objs) == 1 {
storage, reqActualPath, err := op.GetStorageAndActualPath(objs.GetPath())
if err != nil {
return err
}
for _, path := range reqPath {
err = errors.Join(err, fs.PutDirectly(ctx, *path, &stream.FileStream{
Obj: s,
Mimetype: s.GetMimetype(),
WebPutAsTask: s.NeedStore(),
Reader: file,
return op.Put(ctx, storage, reqActualPath, &stream.FileStream{
Obj: s,
Mimetype: s.GetMimetype(),
Reader: s,
}, up)
} else {
file, err := s.CacheFullAndWriter(nil, nil)
if err != nil {
return err
}
count := float64(len(objs) + 1)
up(100 / count)
for i, obj := range objs {
err = errors.Join(err, fs.PutDirectly(ctx, obj.GetPath(), &stream.FileStream{
Obj: s,
Mimetype: s.GetMimetype(),
Reader: file,
}))
up(float64(i+2) / float64(count) * 100)
_, e := file.Seek(0, io.SeekStart)
if e != nil {
return errors.Join(err, e)
@@ -304,55 +441,40 @@ func (d *Alias) Put(ctx context.Context, dstDir model.Obj, s model.FileStreamer,
return err
}
}
if errs.IsNotImplement(err) {
return errors.New("same-name dirs cannot be Put")
}
return err
}
func (d *Alias) PutURL(ctx context.Context, dstDir model.Obj, name, url string) error {
if !d.Writable {
return errs.PermissionDenied
}
reqPath, err := d.getReqPath(ctx, dstDir, true)
objs, err := d.getPutObjs(ctx, dstDir)
if err == nil {
for _, path := range reqPath {
err = errors.Join(err, fs.PutURL(ctx, *path, name, url))
for _, obj := range objs {
err = errors.Join(err, fs.PutURL(ctx, obj.GetPath(), name, url))
}
return err
}
if errs.IsNotImplement(err) {
return errors.New("same-name files cannot offline download")
}
return err
}
func (d *Alias) GetArchiveMeta(ctx context.Context, obj model.Obj, args model.ArchiveArgs) (model.ArchiveMeta, error) {
root, sub := d.getRootAndPath(obj.GetPath())
dsts, ok := d.pathMap[root]
if !ok {
return nil, errs.ObjectNotFound
reqPath := d.getBalancedPath(ctx, obj)
if reqPath == "" {
return nil, errs.NotFile
}
for _, dst := range dsts {
meta, err := d.getArchiveMeta(ctx, dst, sub, args)
if err == nil {
return meta, nil
}
meta, err := d.getArchiveMeta(ctx, reqPath, args)
if err == nil {
return meta, nil
}
return nil, errs.NotImplement
}
func (d *Alias) ListArchive(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) ([]model.Obj, error) {
root, sub := d.getRootAndPath(obj.GetPath())
dsts, ok := d.pathMap[root]
if !ok {
return nil, errs.ObjectNotFound
reqPath := d.getBalancedPath(ctx, obj)
if reqPath == "" {
return nil, errs.NotFile
}
for _, dst := range dsts {
l, err := d.listArchive(ctx, dst, sub, args)
if err == nil {
return l, nil
}
l, err := d.listArchive(ctx, reqPath, args)
if err == nil {
return l, nil
}
return nil, errs.NotImplement
}
@@ -361,53 +483,88 @@ func (d *Alias) Extract(ctx context.Context, obj model.Obj, args model.ArchiveIn
// alias的两个驱动,一个支持驱动提取,一个不支持,如何兼容?
// 如果访问的是不支持驱动提取的驱动内的压缩文件,GetArchiveMeta就会返回errs.NotImplement,提取URL前缀就会是/ae,Extract就不会被调用
// 如果访问的是支持驱动提取的驱动内的压缩文件,GetArchiveMeta就会返回有效值,提取URL前缀就会是/ad,Extract就会被调用
root, sub := d.getRootAndPath(obj.GetPath())
dsts, ok := d.pathMap[root]
if !ok {
return nil, errs.ObjectNotFound
reqPath := d.getBalancedPath(ctx, obj)
if reqPath == "" {
return nil, errs.NotFile
}
for _, dst := range dsts {
link, err := d.extract(ctx, dst, sub, args)
if err == nil {
return link, nil
}
link, err := d.extract(ctx, reqPath, args)
if err != nil {
return nil, errs.NotImplement
}
return nil, errs.NotImplement
if link == nil {
return &model.Link{
URL: fmt.Sprintf("%s/ap%s?inner=%s&pass=%s&sign=%s",
common.GetApiUrl(ctx),
utils.EncodePath(reqPath, true),
utils.EncodePath(args.InnerPath, true),
url.QueryEscape(args.Password),
sign.SignArchive(reqPath)),
}, nil
}
resultLink := *link
resultLink.SyncClosers = utils.NewSyncClosers(link)
return &resultLink, nil
}
func (d *Alias) ArchiveDecompress(ctx context.Context, srcObj, dstDir model.Obj, args model.ArchiveDecompressArgs) error {
if !d.Writable {
return errs.PermissionDenied
}
srcPath, err := d.getReqPath(ctx, srcObj, false)
if errs.IsNotImplement(err) {
return errors.New("same-name files cannot be decompressed")
}
if err != nil {
return err
}
dstPath, err := d.getReqPath(ctx, dstDir, true)
if errs.IsNotImplement(err) {
return errors.New("same-name dirs cannot be decompressed to")
}
if err != nil {
return err
}
if len(srcPath) == len(dstPath) {
for i := range srcPath {
_, e := fs.ArchiveDecompress(ctx, *srcPath[i], *dstPath[i], args)
srcs, dsts, err := d.getCopyObjs(ctx, srcObj, dstDir)
if err == nil {
for i, src := range srcs {
dst := dsts[i]
_, e := fs.ArchiveDecompress(ctx, src.GetPath(), dst.GetPath(), args)
err = errors.Join(err, e)
}
return err
} else if len(srcPath) == 1 || !d.ProtectSameName {
for _, path := range dstPath {
_, e := fs.ArchiveDecompress(ctx, *srcPath[0], *path, args)
err = errors.Join(err, e)
}
return err
} else {
return errors.New("parallel paths mismatch")
}
return err
}
func (d *Alias) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
if !d.DetailsPassThrough {
return nil, errs.NotImplement
}
if len(d.rootOrder) != 1 {
return nil, errs.NotImplement
}
backends := d.pathMap[d.rootOrder[0]]
var storage driver.Driver
for _, backend := range backends {
s, err := fs.GetStorage(backend, &fs.GetStoragesArgs{})
if err != nil {
return nil, errs.NotImplement
}
if storage == nil {
storage = s
} else if storage.GetStorage().MountPath != s.GetStorage().MountPath {
return nil, errs.NotImplement
}
}
if storage == nil { // should never access
return nil, errs.NotImplement
}
return op.GetStorageDetails(ctx, storage)
}
func (d *Alias) ResolveLinkCacheMode(path string) driver.LinkCacheMode {
roots, sub := d.getRootsAndPath(path)
if len(roots) == 0 {
return 0
}
for _, root := range roots {
storage, actualPath, err := op.GetStorageAndActualPath(stdpath.Join(root, sub))
if err != nil {
continue
}
if storage.Config().CheckStatus && storage.GetStorage().Status != op.WORK {
continue
}
mode := storage.Config().LinkCacheMode
if mode == -1 {
return storage.(driver.LinkCacheModeResolver).ResolveLinkCacheMode(actualPath)
} else {
return mode
}
}
return 0
}
var _ driver.Driver = (*Alias)(nil)
+11 -14
View File
@@ -6,15 +6,15 @@ import (
)
type Addition struct {
// Usually one of two
// driver.RootPath
// define other
Paths string `json:"paths" required:"true" type:"text"`
ProtectSameName bool `json:"protect_same_name" default:"true" required:"false" help:"Protects same-name files from Delete or Rename"`
ParallelWrite bool `json:"parallel_write" type:"bool" default:"false"`
DownloadConcurrency int `json:"download_concurrency" default:"0" required:"false" type:"number" help:"Need to enable proxy"`
DownloadPartSize int `json:"download_part_size" default:"0" type:"number" required:"false" help:"Need to enable proxy. Unit: KB"`
Writable bool `json:"writable" type:"bool" default:"false"`
Paths string `json:"paths" required:"true" type:"text"`
ReadConflictPolicy string `json:"read_conflict_policy" type:"select" options:"first,random,all" default:"first"`
WriteConflictPolicy string `json:"write_conflict_policy" type:"select" options:"disabled,first,deterministic,deterministic_or_all,all,all_strict" default:"disabled" help:"How the driver handles identical backend paths when renaming, removing, or making directories."`
PutConflictPolicy string `json:"put_conflict_policy" type:"select" options:"disabled,first,deterministic,deterministic_or_all,all,all_strict,random,quota,quota_strict" default:"disabled" help:"How the driver handles identical backend paths when uploading, copying, moving, or decompressing."`
FileConsistencyCheck bool `json:"file_consistency_check" type:"bool" default:"false"`
DownloadConcurrency int `json:"download_concurrency" default:"0" required:"false" type:"number" help:"Need to enable proxy"`
DownloadPartSize int `json:"download_part_size" default:"0" type:"number" required:"false" help:"Need to enable proxy. Unit: KB"`
ProviderPassThrough bool `json:"provider_pass_through" type:"bool" default:"false"`
DetailsPassThrough bool `json:"details_pass_through" type:"bool" default:"false"`
}
var config = driver.Config{
@@ -24,14 +24,11 @@ var config = driver.Config{
NoUpload: false,
DefaultRoot: "/",
ProxyRangeOption: true,
LinkCacheMode: driver.LinkCacheAuto,
}
func init() {
op.RegisterDriver(func() driver.Driver {
return &Alias{
Addition: Addition{
ProtectSameName: true,
},
}
return &Alias{}
})
}
+77
View File
@@ -1 +1,78 @@
package alias
import (
"time"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/pkg/errors"
)
const (
DisabledWP = "disabled"
FirstRWP = "first"
DeterministicWP = "deterministic"
DeterministicOrAllWP = "deterministic_or_all"
AllRWP = "all"
AllStrictWP = "all_strict"
RandomBalancedRP = "random"
BalancedByQuotaP = "quota"
BalancedByQuotaStrictP = "quota_strict"
)
var (
ValidReadConflictPolicy = []string{FirstRWP, RandomBalancedRP, AllRWP}
ValidWriteConflictPolicy = []string{DisabledWP, FirstRWP, DeterministicWP, DeterministicOrAllWP, AllRWP,
AllStrictWP}
ValidPutConflictPolicy = []string{DisabledWP, FirstRWP, DeterministicWP, DeterministicOrAllWP, AllRWP,
AllStrictWP, RandomBalancedRP, BalancedByQuotaP, BalancedByQuotaStrictP}
)
var (
ErrPathConflict = errors.New("path conflict")
ErrSamePathLeak = errors.New("leak some of same-name dirs")
ErrNoEnoughSpace = errors.New("none of same-name dirs has enough space")
ErrNotEnoughSrcObjs = errors.New("cannot move fewer objs to more paths, please try copying")
)
type BalancedObjs []model.Obj
func (b BalancedObjs) GetSize() int64 {
return b[0].GetSize()
}
func (b BalancedObjs) ModTime() time.Time {
return b[0].ModTime()
}
func (b BalancedObjs) CreateTime() time.Time {
return b[0].CreateTime()
}
func (b BalancedObjs) IsDir() bool {
return b[0].IsDir()
}
func (b BalancedObjs) GetHash() utils.HashInfo {
return b[0].GetHash()
}
func (b BalancedObjs) GetName() string {
return b[0].GetName()
}
func (b BalancedObjs) GetPath() string {
return b[0].GetPath()
}
func (b BalancedObjs) GetID() string {
return b[0].GetID()
}
func (b BalancedObjs) Unwrap() model.Obj {
return b[0]
}
var _ model.Obj = (BalancedObjs)(nil)
type tempObj struct{ model.Object }
+444 -124
View File
@@ -2,98 +2,98 @@ package alias
import (
"context"
"fmt"
"net/url"
"math/rand"
stdpath "path"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/fs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/sign"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/server/common"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
)
func (d *Alias) listRoot() []model.Obj {
type detailWithIndex struct {
idx int
val *model.StorageDetails
}
func (d *Alias) listRoot(ctx context.Context, withDetails, refresh bool) []model.Obj {
var objs []model.Obj
for k := range d.pathMap {
obj := model.Object{
detailsChan := make(chan detailWithIndex, len(d.pathMap))
workerCount := 0
for _, k := range d.rootOrder {
obj := &model.Object{
Name: k,
Path: "/" + k,
IsFolder: true,
Modified: d.Modified,
Mask: model.Locked | model.Virtual,
}
idx := len(objs)
objs = append(objs, obj)
v := d.pathMap[k]
if !withDetails || len(v) != 1 {
continue
}
remoteDriver, err := op.GetStorageByMountPath(v[0])
if err != nil {
continue
}
obj.Modified = remoteDriver.GetStorage().Modified
_, ok := remoteDriver.(driver.WithDetails)
if !ok {
continue
}
objs[idx] = &model.ObjStorageDetails{
Obj: objs[idx],
StorageDetails: nil,
}
workerCount++
go func(dri driver.Driver, i int) {
details, e := op.GetStorageDetails(ctx, dri, refresh)
if e != nil {
if !errors.Is(e, errs.NotImplement) && !errors.Is(e, errs.StorageNotInit) {
log.Errorf("failed get %s storage details: %+v", dri.GetStorage().MountPath, e)
}
}
detailsChan <- detailWithIndex{idx: i, val: details}
}(remoteDriver, idx)
}
for workerCount > 0 {
select {
case r := <-detailsChan:
objs[r.idx].(*model.ObjStorageDetails).StorageDetails = r.val
workerCount--
case <-time.After(time.Second):
workerCount = 0
}
objs = append(objs, &obj)
}
return objs
}
// do others that not defined in Driver interface
func getPair(path string) (string, string) {
//path = strings.TrimSpace(path)
if strings.Contains(path, ":") {
pair := strings.SplitN(path, ":", 2)
if !strings.Contains(pair[0], "/") {
return pair[0], pair[1]
}
if name, path, ok := strings.Cut(path, ":"); ok && !strings.Contains(name, "/") {
return name, path
}
return stdpath.Base(path), path
}
func (d *Alias) getRootAndPath(path string) (string, string) {
if d.autoFlatten {
return d.oneKey, path
func (d *Alias) getRootsAndPath(path string) (roots []string, sub string) {
if len(d.rootOrder) == 1 {
return d.pathMap[d.rootOrder[0]], path
}
path = strings.TrimPrefix(path, "/")
parts := strings.SplitN(path, "/", 2)
if len(parts) == 1 {
return parts[0], ""
before, after, ok := strings.Cut(path, "/")
if !ok {
return d.pathMap[path], ""
}
return parts[0], parts[1]
}
func (d *Alias) get(ctx context.Context, path string, dst, sub string) (model.Obj, error) {
obj, err := fs.Get(ctx, stdpath.Join(dst, sub), &fs.GetArgs{NoLog: true})
if err != nil {
return nil, err
}
return &model.Object{
Path: path,
Name: obj.GetName(),
Size: obj.GetSize(),
Modified: obj.ModTime(),
IsFolder: obj.IsDir(),
HashInfo: obj.GetHash(),
}, nil
}
func (d *Alias) list(ctx context.Context, dst, sub string, args *fs.ListArgs) ([]model.Obj, error) {
objs, err := fs.List(ctx, stdpath.Join(dst, sub), args)
// the obj must implement the model.SetPath interface
// return objs, err
if err != nil {
return nil, err
}
return utils.SliceConvert(objs, func(obj model.Obj) (model.Obj, error) {
thumb, ok := model.GetThumb(obj)
objRes := model.Object{
Name: obj.GetName(),
Size: obj.GetSize(),
Modified: obj.ModTime(),
IsFolder: obj.IsDir(),
}
if !ok {
return &objRes, nil
}
return &model.ObjThumb{
Object: objRes,
Thumbnail: model.Thumbnail{
Thumbnail: thumb,
},
}, nil
})
return d.pathMap[before], after
}
func (d *Alias) link(ctx context.Context, reqPath string, args model.LinkArgs) (*model.Link, model.Obj, error) {
@@ -101,60 +101,350 @@ func (d *Alias) link(ctx context.Context, reqPath string, args model.LinkArgs) (
if err != nil {
return nil, nil, err
}
// proxy || ftp,s3
if !args.Redirect || len(common.GetApiUrl(ctx)) == 0 {
return op.Link(ctx, storage, reqActualPath, args)
}
obj, err := fs.Get(ctx, reqPath, &fs.GetArgs{NoLog: true})
if err != nil {
return nil, nil, err
}
if common.ShouldProxy(storage, stdpath.Base(reqPath)) {
return nil, obj, nil
if args.Redirect && common.ShouldProxy(storage, stdpath.Base(reqPath)) {
return nil, nil, nil
}
return op.Link(ctx, storage, reqActualPath, args)
}
func (d *Alias) getReqPath(ctx context.Context, obj model.Obj, isParent bool) ([]*string, error) {
root, sub := d.getRootAndPath(obj.GetPath())
if sub == "" && !isParent {
return nil, errs.NotSupport
func isConsistent(a, b model.Obj) bool {
if a.GetSize() != b.GetSize() {
return false
}
dsts, ok := d.pathMap[root]
all := true
if !ok {
return nil, errs.ObjectNotFound
}
var reqPath []*string
for _, dst := range dsts {
path := stdpath.Join(dst, sub)
_, err := fs.Get(ctx, path, &fs.GetArgs{NoLog: true})
if err != nil {
all = false
if d.ProtectSameName && d.ParallelWrite && len(reqPath) >= 2 {
return nil, errs.NotImplement
}
continue
}
if !d.ProtectSameName && !d.ParallelWrite {
return []*string{&path}, nil
}
reqPath = append(reqPath, &path)
if d.ProtectSameName && !d.ParallelWrite && len(reqPath) >= 2 {
return nil, errs.NotImplement
}
if d.ProtectSameName && d.ParallelWrite && len(reqPath) >= 2 && !all {
return nil, errs.NotImplement
for ht, v := range a.GetHash().All() {
ah := b.GetHash().GetHash(ht)
if ah != "" && ah != v {
return false
}
}
if len(reqPath) == 0 {
return nil, errs.ObjectNotFound
}
return reqPath, nil
return true
}
func (d *Alias) getArchiveMeta(ctx context.Context, dst, sub string, args model.ArchiveArgs) (model.ArchiveMeta, error) {
reqPath := stdpath.Join(dst, sub)
func (d *Alias) getAllObjs(ctx context.Context, bObj model.Obj, ifContinue func(err error) (bool, error)) (BalancedObjs, error) {
objs := bObj.(BalancedObjs)
length := 0
for _, o := range objs {
var err error
var obj model.Obj
temp, isTemp := o.(*tempObj)
if isTemp {
obj, err = fs.Get(ctx, o.GetPath(), &fs.GetArgs{NoLog: true})
if err == nil {
if !bObj.IsDir() {
if obj.IsDir() {
err = errs.NotFile
} else if d.FileConsistencyCheck && !isConsistent(bObj, obj) {
err = errs.ObjectNotFound
}
} else if !obj.IsDir() {
err = errs.NotFolder
}
}
} else if o == nil {
err = errs.ObjectNotFound
}
cont, err := ifContinue(err)
if err != nil {
if cont {
continue
}
return nil, err
}
if isTemp {
objRes := temp.Object
// objRes.Name = obj.GetName()
// objRes.Size = obj.GetSize()
// objRes.Modified = obj.ModTime()
// objRes.HashInfo = obj.GetHash()
objs[length] = &objRes
} else {
objs[length] = o
}
length++
if !cont {
break
}
}
if length == 0 {
return nil, errs.ObjectNotFound
}
return objs[:length], nil
}
func (d *Alias) getBalancedPath(ctx context.Context, file model.Obj) string {
if d.ReadConflictPolicy == FirstRWP {
return file.GetPath()
}
files := file.(BalancedObjs)
if rand.Intn(len(files)) == 0 {
return file.GetPath()
}
files, _ = d.getAllObjs(ctx, file, getWriteAndPutFilterFunc(AllRWP))
return files[rand.Intn(len(files))].GetPath()
}
func getWriteAndPutFilterFunc(policy string) func(error) (bool, error) {
if policy == AllRWP {
return func(err error) (bool, error) {
return true, err
}
}
all := true
l := 0
return func(err error) (bool, error) {
if err != nil {
switch policy {
case AllStrictWP:
return false, ErrSamePathLeak
case DeterministicOrAllWP:
if l >= 2 {
return false, ErrSamePathLeak
}
}
all = false
} else {
switch policy {
case FirstRWP:
return false, nil
case DeterministicWP:
if l > 0 {
return false, ErrPathConflict
}
case DeterministicOrAllWP:
if l > 0 && !all {
return false, ErrSamePathLeak
}
}
l += 1
}
return true, err
}
}
func (d *Alias) getWriteObjs(ctx context.Context, obj model.Obj) (BalancedObjs, error) {
if d.WriteConflictPolicy == DisabledWP {
return nil, errs.PermissionDenied
}
return d.getAllObjs(ctx, obj, getWriteAndPutFilterFunc(d.WriteConflictPolicy))
}
func (d *Alias) getPutObjs(ctx context.Context, obj model.Obj) (BalancedObjs, error) {
if d.PutConflictPolicy == DisabledWP {
return nil, errs.PermissionDenied
}
objs, err := d.getAllObjs(ctx, obj, getWriteAndPutFilterFunc(d.PutConflictPolicy))
if err != nil {
return nil, err
}
strict := false
switch d.PutConflictPolicy {
case RandomBalancedRP:
ri := rand.Intn(len(objs))
return objs[ri : ri+1], nil
case BalancedByQuotaStrictP:
strict = true
fallthrough
case BalancedByQuotaP:
objs, ok := getRandomObjByQuotaBalanced(ctx, objs, strict, obj.GetSize())
if !ok {
return nil, ErrNoEnoughSpace
}
return objs, nil
default:
return objs, nil
}
}
func getRandomObjByQuotaBalanced(ctx context.Context, reqPath BalancedObjs, strict bool, objSize int64) (BalancedObjs, bool) {
// Get all space
details := make([]*model.StorageDetails, len(reqPath))
detailsChan := make(chan detailWithIndex, len(reqPath))
workerCount := 0
for i, p := range reqPath {
s, err := fs.GetStorage(p.GetPath(), &fs.GetStoragesArgs{})
if err != nil {
continue
}
if _, ok := s.(driver.WithDetails); !ok {
continue
}
workerCount++
go func(dri driver.Driver, i int) {
d, e := op.GetStorageDetails(ctx, dri)
if e != nil {
if !errors.Is(e, errs.NotImplement) && !errors.Is(e, errs.StorageNotInit) {
log.Errorf("failed get %s storage details: %+v", dri.GetStorage().MountPath, e)
}
}
detailsChan <- detailWithIndex{idx: i, val: d}
}(s, i)
}
for workerCount > 0 {
select {
case r := <-detailsChan:
details[r.idx] = r.val
workerCount--
case <-time.After(time.Second):
workerCount = 0
}
}
// Try select one that has space info
selected, ok := selectRandom(details, func(d *model.StorageDetails) uint64 {
if d == nil || d.FreeSpace() < objSize {
return 0
}
return uint64(d.FreeSpace())
})
if !ok {
if strict {
return nil, false
} else {
// No strict mode, return any of non-details ones
noDetails := make([]int, 0, len(details))
for i, d := range details {
if d == nil {
noDetails = append(noDetails, i)
}
}
if len(noDetails) == 0 {
return nil, false
}
selected = noDetails[rand.Intn(len(noDetails))]
}
}
return reqPath[selected : selected+1], true
}
func selectRandom[Item any](arr []Item, getWeight func(Item) uint64) (int, bool) {
var totalWeight uint64 = 0
for _, i := range arr {
totalWeight += getWeight(i)
}
if totalWeight == 0 {
return 0, false
}
r := rand.Uint64() % totalWeight
for i, item := range arr {
w := getWeight(item)
if r < w {
return i, true
}
r -= w
}
return 0, false
}
func (d *Alias) getCopyObjs(ctx context.Context, srcObj, dstDir model.Obj) (BalancedObjs, BalancedObjs, error) {
if d.PutConflictPolicy == DisabledWP {
return nil, nil, errs.PermissionDenied
}
dstObjs, err := d.getAllObjs(ctx, dstDir, getWriteAndPutFilterFunc(d.PutConflictPolicy))
if err != nil {
return nil, nil, err
}
dstStorageMap := make(map[string][]model.Obj)
allocatingDst := make(map[model.Obj]struct{})
for _, o := range dstObjs {
storage, e := fs.GetStorage(o.GetPath(), &fs.GetStoragesArgs{})
if e != nil {
return nil, nil, errors.WithMessagef(e, "cannot copy to virtual path [%s]", o.GetPath())
}
mp := storage.GetStorage().MountPath
dstStorageMap[mp] = append(dstStorageMap[mp], o)
allocatingDst[o] = struct{}{}
}
tmpSrcObjs, err := d.getAllObjs(ctx, srcObj, getWriteAndPutFilterFunc(AllRWP))
if err != nil {
return nil, nil, err
}
srcObjs := make(BalancedObjs, 0, len(dstObjs))
for _, src := range tmpSrcObjs {
storage, e := fs.GetStorage(src.GetPath(), &fs.GetStoragesArgs{})
if e != nil {
continue
}
mp := storage.GetStorage().MountPath
if tmp, ok := dstStorageMap[mp]; ok {
for _, dst := range tmp {
dstObjs[len(srcObjs)] = dst
srcObjs = append(srcObjs, src)
delete(allocatingDst, dst)
}
delete(dstStorageMap, mp)
}
}
dstObjs = dstObjs[:len(srcObjs)]
for dst := range allocatingDst {
src := tmpSrcObjs[0]
if d.ReadConflictPolicy == RandomBalancedRP || d.ReadConflictPolicy == AllRWP {
src = tmpSrcObjs[rand.Intn(len(tmpSrcObjs))]
}
srcObjs = append(srcObjs, src)
dstObjs = append(dstObjs, dst)
}
return srcObjs, dstObjs, nil
}
func (d *Alias) getMoveObjs(ctx context.Context, srcObj, dstDir model.Obj) (BalancedObjs, BalancedObjs, error) {
if d.PutConflictPolicy == DisabledWP {
return nil, nil, errs.PermissionDenied
}
dstObjs, err := d.getAllObjs(ctx, dstDir, getWriteAndPutFilterFunc(d.PutConflictPolicy))
if err != nil {
return nil, nil, err
}
tmpSrcObjs, err := d.getAllObjs(ctx, srcObj, getWriteAndPutFilterFunc(AllRWP))
if err != nil {
return nil, nil, err
}
if len(tmpSrcObjs) < len(dstObjs) {
return nil, nil, ErrNotEnoughSrcObjs
}
dstStorageMap := make(map[string][]model.Obj)
allocatingDst := make(map[model.Obj]struct{})
for _, o := range dstObjs {
storage, e := fs.GetStorage(o.GetPath(), &fs.GetStoragesArgs{})
if e != nil {
return nil, nil, errors.WithMessagef(e, "cannot move to virtual path [%s]", o.GetPath())
}
mp := storage.GetStorage().MountPath
dstStorageMap[mp] = append(dstStorageMap[mp], o)
allocatingDst[o] = struct{}{}
}
srcObjs := make(BalancedObjs, 0, len(tmpSrcObjs))
restSrcObjs := make(BalancedObjs, 0, len(tmpSrcObjs)-len(dstObjs))
for _, src := range tmpSrcObjs {
storage, e := fs.GetStorage(src.GetPath(), &fs.GetStoragesArgs{})
if e != nil {
continue
}
mp := storage.GetStorage().MountPath
if tmp, ok := dstStorageMap[mp]; ok {
dst := tmp[0]
if len(tmp) == 1 {
delete(dstStorageMap, mp)
} else {
dstStorageMap[mp] = tmp[1:]
}
dstObjs[len(srcObjs)] = dst
srcObjs = append(srcObjs, src)
delete(allocatingDst, dst)
} else {
restSrcObjs = append(restSrcObjs, src)
}
}
dstObjs = dstObjs[:len(srcObjs)]
// len(restSrcObjs) >= len(allocatingDst)
srcObjs = append(srcObjs, restSrcObjs...)
for dst := range allocatingDst {
dstObjs = append(dstObjs, dst)
}
return srcObjs, dstObjs, nil
}
func (d *Alias) getArchiveMeta(ctx context.Context, reqPath string, args model.ArchiveArgs) (model.ArchiveMeta, error) {
storage, reqActualPath, err := op.GetStorageAndActualPath(reqPath)
if err != nil {
return nil, err
@@ -168,8 +458,7 @@ func (d *Alias) getArchiveMeta(ctx context.Context, dst, sub string, args model.
return nil, errs.NotImplement
}
func (d *Alias) listArchive(ctx context.Context, dst, sub string, args model.ArchiveInnerArgs) ([]model.Obj, error) {
reqPath := stdpath.Join(dst, sub)
func (d *Alias) listArchive(ctx context.Context, reqPath string, args model.ArchiveInnerArgs) ([]model.Obj, error) {
storage, reqActualPath, err := op.GetStorageAndActualPath(reqPath)
if err != nil {
return nil, err
@@ -183,8 +472,7 @@ func (d *Alias) listArchive(ctx context.Context, dst, sub string, args model.Arc
return nil, errs.NotImplement
}
func (d *Alias) extract(ctx context.Context, dst, sub string, args model.ArchiveInnerArgs) (*model.Link, error) {
reqPath := stdpath.Join(dst, sub)
func (d *Alias) extract(ctx context.Context, reqPath string, args model.ArchiveInnerArgs) (*model.Link, error) {
storage, reqActualPath, err := op.GetStorageAndActualPath(reqPath)
if err != nil {
return nil, err
@@ -192,21 +480,53 @@ func (d *Alias) extract(ctx context.Context, dst, sub string, args model.Archive
if _, ok := storage.(driver.ArchiveReader); !ok {
return nil, errs.NotImplement
}
if args.Redirect && common.ShouldProxy(storage, stdpath.Base(sub)) {
_, err = fs.Get(ctx, reqPath, &fs.GetArgs{NoLog: true})
if err != nil {
if args.Redirect && common.ShouldProxy(storage, stdpath.Base(reqPath)) {
_, err := fs.Get(ctx, reqPath, &fs.GetArgs{NoLog: true})
if err == nil {
return nil, err
}
link := &model.Link{
URL: fmt.Sprintf("%s/ap%s?inner=%s&pass=%s&sign=%s",
common.GetApiUrl(ctx),
utils.EncodePath(reqPath, true),
utils.EncodePath(args.InnerPath, true),
url.QueryEscape(args.Password),
sign.SignArchive(reqPath)),
}
return link, nil
return nil, nil
}
link, _, err := op.DriverExtract(ctx, storage, reqActualPath, args)
return link, err
}
func getAllSort(dirs []model.Obj) model.Sort {
ret := model.Sort{}
noSort := false
noExtractFolder := false
for _, dir := range dirs {
if dir == nil {
continue
}
storage, err := fs.GetStorage(dir.GetPath(), &fs.GetStoragesArgs{})
if err != nil {
continue
}
if !noSort && storage.GetStorage().OrderBy != "" {
if ret.OrderBy == "" {
ret.OrderBy = storage.GetStorage().OrderBy
ret.OrderDirection = storage.GetStorage().OrderDirection
if ret.OrderDirection == "" {
ret.OrderDirection = "asc"
}
} else if ret.OrderBy != storage.GetStorage().OrderBy || ret.OrderDirection != storage.GetStorage().OrderDirection {
ret.OrderBy = ""
ret.OrderDirection = ""
noSort = true
}
}
if !noExtractFolder && storage.GetStorage().ExtractFolder != "" {
if ret.ExtractFolder == "" {
ret.ExtractFolder = storage.GetStorage().ExtractFolder
} else if ret.ExtractFolder != storage.GetStorage().ExtractFolder {
ret.ExtractFolder = ""
noExtractFolder = true
}
}
if noSort && noExtractFolder {
break
}
}
return ret
}
+379
View File
@@ -0,0 +1,379 @@
package alist_v3
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"path"
"strings"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/server/common"
"github.com/go-resty/resty/v2"
log "github.com/sirupsen/logrus"
)
type AListV3 struct {
model.Storage
Addition
}
func (d *AListV3) Config() driver.Config {
return config
}
func (d *AListV3) GetAddition() driver.Additional {
return &d.Addition
}
func (d *AListV3) Init(ctx context.Context) error {
d.Addition.Address = strings.TrimSuffix(d.Addition.Address, "/")
var resp common.Resp[MeResp]
_, _, err := d.request("/me", http.MethodGet, func(req *resty.Request) {
req.SetResult(&resp)
})
if err != nil {
return err
}
// if the username is not empty and the username is not the same as the current username, then login again
if d.Username != resp.Data.Username {
err = d.login()
if err != nil {
return err
}
}
// re-get the user info
_, _, err = d.request("/me", http.MethodGet, func(req *resty.Request) {
req.SetResult(&resp)
})
if err != nil {
return err
}
if utils.SliceContains(resp.Data.Role, model.GUEST) {
u := d.Address + "/api/public/settings"
res, err := base.RestyClient.R().Get(u)
if err != nil {
return err
}
allowMounted := utils.Json.Get(res.Body(), "data", conf.AllowMounted).ToString() == "true"
if !allowMounted {
return fmt.Errorf("the site does not allow mounted")
}
}
return err
}
func (d *AListV3) Drop(ctx context.Context) error {
return nil
}
func (d *AListV3) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
var resp common.Resp[FsListResp]
_, _, err := d.request("/fs/list", http.MethodPost, func(req *resty.Request) {
req.SetResult(&resp).SetBody(ListReq{
PageReq: model.PageReq{
Page: 1,
PerPage: 0,
},
Path: dir.GetPath(),
Password: d.MetaPassword,
Refresh: false,
})
})
if err != nil {
return nil, err
}
var files []model.Obj
for _, f := range resp.Data.Content {
file := model.ObjThumb{
Object: model.Object{
Name: f.Name,
Modified: f.Modified,
Ctime: f.Created,
Size: f.Size,
IsFolder: f.IsDir,
HashInfo: utils.FromString(f.HashInfo),
},
Thumbnail: model.Thumbnail{Thumbnail: f.Thumb},
}
files = append(files, &file)
}
return files, nil
}
func (d *AListV3) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
var resp common.Resp[FsGetResp]
headers := map[string]string{
"User-Agent": base.UserAgent,
}
// if PassUAToUpsteam is true, then pass the user-agent to the upstream
if d.PassUAToUpsteam {
userAgent := args.Header.Get("user-agent")
if userAgent != "" {
headers["User-Agent"] = userAgent
}
}
// if PassIPToUpsteam is true, then pass the ip address to the upstream
if d.PassIPToUpsteam {
ip := args.IP
if ip != "" {
headers["X-Forwarded-For"] = ip
headers["X-Real-Ip"] = ip
}
}
_, _, err := d.request("/fs/get", http.MethodPost, func(req *resty.Request) {
req.SetResult(&resp).SetBody(FsGetReq{
Path: file.GetPath(),
Password: d.MetaPassword,
}).SetHeaders(headers)
})
if err != nil {
return nil, err
}
return &model.Link{
URL: resp.Data.RawURL,
}, nil
}
func (d *AListV3) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) error {
_, _, err := d.request("/fs/mkdir", http.MethodPost, func(req *resty.Request) {
req.SetBody(MkdirOrLinkReq{
Path: path.Join(parentDir.GetPath(), dirName),
})
})
return err
}
func (d *AListV3) Move(ctx context.Context, srcObj, dstDir model.Obj) error {
_, _, err := d.request("/fs/move", http.MethodPost, func(req *resty.Request) {
req.SetBody(MoveCopyReq{
SrcDir: path.Dir(srcObj.GetPath()),
DstDir: dstDir.GetPath(),
Names: []string{srcObj.GetName()},
})
})
return err
}
func (d *AListV3) Rename(ctx context.Context, srcObj model.Obj, newName string) error {
_, _, err := d.request("/fs/rename", http.MethodPost, func(req *resty.Request) {
req.SetBody(RenameReq{
Path: srcObj.GetPath(),
Name: newName,
})
})
return err
}
func (d *AListV3) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
_, _, err := d.request("/fs/copy", http.MethodPost, func(req *resty.Request) {
req.SetBody(MoveCopyReq{
SrcDir: path.Dir(srcObj.GetPath()),
DstDir: dstDir.GetPath(),
Names: []string{srcObj.GetName()},
})
})
return err
}
func (d *AListV3) Remove(ctx context.Context, obj model.Obj) error {
_, _, err := d.request("/fs/remove", http.MethodPost, func(req *resty.Request) {
req.SetBody(RemoveReq{
Dir: path.Dir(obj.GetPath()),
Names: []string{obj.GetName()},
})
})
return err
}
func (d *AListV3) Put(ctx context.Context, dstDir model.Obj, s model.FileStreamer, up driver.UpdateProgress) error {
reader := driver.NewLimitedUploadStream(ctx, &driver.ReaderUpdatingProgress{
Reader: s,
UpdateProgress: up,
})
req, err := http.NewRequestWithContext(ctx, http.MethodPut, d.Address+"/api/fs/put", reader)
if err != nil {
return err
}
req.Header.Set("Authorization", d.Token)
req.Header.Set("File-Path", path.Join(dstDir.GetPath(), s.GetName()))
req.Header.Set("Password", d.MetaPassword)
if md5 := s.GetHash().GetHash(utils.MD5); len(md5) > 0 {
req.Header.Set("X-File-Md5", md5)
}
if sha1 := s.GetHash().GetHash(utils.SHA1); len(sha1) > 0 {
req.Header.Set("X-File-Sha1", sha1)
}
if sha256 := s.GetHash().GetHash(utils.SHA256); len(sha256) > 0 {
req.Header.Set("X-File-Sha256", sha256)
}
req.ContentLength = s.GetSize()
// client := base.NewHttpClient()
// client.Timeout = time.Hour * 6
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
bytes, err := io.ReadAll(res.Body)
if err != nil {
return err
}
log.Debugf("[openlist] response body: %s", string(bytes))
if res.StatusCode >= 400 {
return fmt.Errorf("request failed, status: %s", res.Status)
}
code := utils.Json.Get(bytes, "code").ToInt()
if code != 200 {
if code == 401 || code == 403 {
err = d.login()
if err != nil {
return err
}
}
return fmt.Errorf("request failed,code: %d, message: %s", code, utils.Json.Get(bytes, "message").ToString())
}
return nil
}
func (d *AListV3) GetArchiveMeta(ctx context.Context, obj model.Obj, args model.ArchiveArgs) (model.ArchiveMeta, error) {
if !d.ForwardArchiveReq {
return nil, errs.NotImplement
}
var resp common.Resp[ArchiveMetaResp]
_, code, err := d.request("/fs/archive/meta", http.MethodPost, func(req *resty.Request) {
req.SetResult(&resp).SetBody(ArchiveMetaReq{
ArchivePass: args.Password,
Password: d.MetaPassword,
Path: obj.GetPath(),
Refresh: false,
})
})
if code == 202 {
return nil, errs.WrongArchivePassword
}
if err != nil {
return nil, err
}
var tree []model.ObjTree
if resp.Data.Content != nil {
tree = make([]model.ObjTree, 0, len(resp.Data.Content))
for _, content := range resp.Data.Content {
tree = append(tree, &content)
}
}
return &model.ArchiveMetaInfo{
Comment: resp.Data.Comment,
Encrypted: resp.Data.Encrypted,
Tree: tree,
}, nil
}
func (d *AListV3) ListArchive(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) ([]model.Obj, error) {
if !d.ForwardArchiveReq {
return nil, errs.NotImplement
}
var resp common.Resp[ArchiveListResp]
_, code, err := d.request("/fs/archive/list", http.MethodPost, func(req *resty.Request) {
req.SetResult(&resp).SetBody(ArchiveListReq{
ArchiveMetaReq: ArchiveMetaReq{
ArchivePass: args.Password,
Password: d.MetaPassword,
Path: obj.GetPath(),
Refresh: false,
},
PageReq: model.PageReq{
Page: 1,
PerPage: 0,
},
InnerPath: args.InnerPath,
})
})
if code == 202 {
return nil, errs.WrongArchivePassword
}
if err != nil {
return nil, err
}
var files []model.Obj
for _, f := range resp.Data.Content {
file := model.ObjThumb{
Object: model.Object{
Name: f.Name,
Modified: f.Modified,
Ctime: f.Created,
Size: f.Size,
IsFolder: f.IsDir,
HashInfo: utils.FromString(f.HashInfo),
},
Thumbnail: model.Thumbnail{Thumbnail: f.Thumb},
}
files = append(files, &file)
}
return files, nil
}
func (d *AListV3) Extract(ctx context.Context, obj model.Obj, args model.ArchiveInnerArgs) (*model.Link, error) {
if !d.ForwardArchiveReq {
return nil, errs.NotSupport
}
var resp common.Resp[ArchiveMetaResp]
_, _, err := d.request("/fs/archive/meta", http.MethodPost, func(req *resty.Request) {
req.SetResult(&resp).SetBody(ArchiveMetaReq{
ArchivePass: args.Password,
Password: d.MetaPassword,
Path: obj.GetPath(),
Refresh: false,
})
})
if err != nil {
return nil, err
}
return &model.Link{
URL: fmt.Sprintf("%s?inner=%s&pass=%s&sign=%s",
resp.Data.RawURL,
utils.EncodePath(args.InnerPath, true),
url.QueryEscape(args.Password),
resp.Data.Sign),
}, nil
}
func (d *AListV3) ArchiveDecompress(ctx context.Context, srcObj, dstDir model.Obj, args model.ArchiveDecompressArgs) error {
if !d.ForwardArchiveReq {
return errs.NotImplement
}
dir, name := path.Split(srcObj.GetPath())
_, _, err := d.request("/fs/archive/decompress", http.MethodPost, func(req *resty.Request) {
req.SetBody(DecompressReq{
ArchivePass: args.Password,
CacheFull: args.CacheFull,
DstDir: dstDir.GetPath(),
InnerPath: args.InnerPath,
Name: []string{name},
PutIntoNewDir: args.PutIntoNewDir,
SrcDir: dir,
})
})
return err
}
func (d *AListV3) ResolveLinkCacheMode(_ string) driver.LinkCacheMode {
var mode driver.LinkCacheMode
if d.PassIPToUpsteam {
mode |= driver.LinkCacheIP
}
if d.PassUAToUpsteam {
mode |= driver.LinkCacheUA
}
return mode
}
var _ driver.Driver = (*AListV3)(nil)
+32
View File
@@ -0,0 +1,32 @@
package alist_v3
import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/op"
)
type Addition struct {
driver.RootPath
Address string `json:"url" required:"true"`
MetaPassword string `json:"meta_password"`
Username string `json:"username"`
Password string `json:"password"`
Token string `json:"token"`
PassIPToUpsteam bool `json:"pass_ip_to_upsteam" default:"true"`
PassUAToUpsteam bool `json:"pass_ua_to_upsteam" default:"true"`
ForwardArchiveReq bool `json:"forward_archive_requests" default:"true"`
}
var config = driver.Config{
Name: "AList V3",
LocalSort: true,
DefaultRoot: "/",
ProxyRangeOption: true,
LinkCacheMode: driver.LinkCacheAuto,
}
func init() {
op.RegisterDriver(func() driver.Driver {
return &AListV3{}
})
}
+182
View File
@@ -0,0 +1,182 @@
package alist_v3
import (
"encoding/json"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
)
type ListReq struct {
model.PageReq
Path string `json:"path" form:"path"`
Password string `json:"password" form:"password"`
Refresh bool `json:"refresh"`
}
type ObjResp struct {
Name string `json:"name"`
Size int64 `json:"size"`
IsDir bool `json:"is_dir"`
Modified time.Time `json:"modified"`
Created time.Time `json:"created"`
Sign string `json:"sign"`
Thumb string `json:"thumb"`
Type int `json:"type"`
HashInfo string `json:"hashinfo"`
}
type FsListResp struct {
Content []ObjResp `json:"content"`
Total int64 `json:"total"`
Readme string `json:"readme"`
Write bool `json:"write"`
Provider string `json:"provider"`
}
type FsGetReq struct {
Path string `json:"path" form:"path"`
Password string `json:"password" form:"password"`
}
type FsGetResp struct {
ObjResp
RawURL string `json:"raw_url"`
Readme string `json:"readme"`
Provider string `json:"provider"`
Related []ObjResp `json:"related"`
}
type MkdirOrLinkReq struct {
Path string `json:"path" form:"path"`
}
type MoveCopyReq struct {
SrcDir string `json:"src_dir"`
DstDir string `json:"dst_dir"`
Names []string `json:"names"`
}
type RenameReq struct {
Path string `json:"path"`
Name string `json:"name"`
}
type RemoveReq struct {
Dir string `json:"dir"`
Names []string `json:"names"`
}
type LoginResp struct {
Token string `json:"token"`
}
type MeResp struct {
Id int `json:"id"`
Username string `json:"username"`
Password string `json:"password"`
BasePath string `json:"base_path"`
Role IntSlice `json:"role"`
Disabled bool `json:"disabled"`
Permission int `json:"permission"`
SsoId string `json:"sso_id"`
Otp bool `json:"otp"`
}
type IntSlice []int
func (s *IntSlice) UnmarshalJSON(b []byte) error {
var i int
if json.Unmarshal(b, &i) == nil {
*s = []int{i}
return nil
}
return json.Unmarshal(b, (*[]int)(s))
}
type ArchiveMetaReq struct {
ArchivePass string `json:"archive_pass"`
Password string `json:"password"`
Path string `json:"path"`
Refresh bool `json:"refresh"`
}
type TreeResp struct {
ObjResp
Children []TreeResp `json:"children"`
hashCache *utils.HashInfo
}
func (t *TreeResp) GetSize() int64 {
return t.Size
}
func (t *TreeResp) GetName() string {
return t.Name
}
func (t *TreeResp) ModTime() time.Time {
return t.Modified
}
func (t *TreeResp) CreateTime() time.Time {
return t.Created
}
func (t *TreeResp) IsDir() bool {
return t.ObjResp.IsDir
}
func (t *TreeResp) GetHash() utils.HashInfo {
return utils.FromString(t.HashInfo)
}
func (t *TreeResp) GetID() string {
return ""
}
func (t *TreeResp) GetPath() string {
return ""
}
func (t *TreeResp) GetChildren() []model.ObjTree {
ret := make([]model.ObjTree, 0, len(t.Children))
for _, child := range t.Children {
ret = append(ret, &child)
}
return ret
}
func (t *TreeResp) Thumb() string {
return t.ObjResp.Thumb
}
type ArchiveMetaResp struct {
Comment string `json:"comment"`
Encrypted bool `json:"encrypted"`
Content []TreeResp `json:"content"`
RawURL string `json:"raw_url"`
Sign string `json:"sign"`
}
type ArchiveListReq struct {
model.PageReq
ArchiveMetaReq
InnerPath string `json:"inner_path"`
}
type ArchiveListResp struct {
Content []ObjResp `json:"content"`
Total int64 `json:"total"`
}
type DecompressReq struct {
ArchivePass string `json:"archive_pass"`
CacheFull bool `json:"cache_full"`
DstDir string `json:"dst_dir"`
InnerPath string `json:"inner_path"`
Name []string `json:"name"`
PutIntoNewDir bool `json:"put_into_new_dir"`
SrcDir string `json:"src_dir"`
}
+65
View File
@@ -0,0 +1,65 @@
package alist_v3
import (
"fmt"
"net/http"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/server/common"
"github.com/go-resty/resty/v2"
log "github.com/sirupsen/logrus"
)
func (d *AListV3) login() error {
if d.Username == "" {
return nil
}
var resp common.Resp[LoginResp]
_, _, err := d.request("/auth/login", http.MethodPost, func(req *resty.Request) {
req.SetResult(&resp).SetBody(base.Json{
"username": d.Username,
"password": d.Password,
})
})
if err != nil {
return err
}
d.Token = resp.Data.Token
op.MustSaveDriverStorage(d)
return nil
}
func (d *AListV3) request(api, method string, callback base.ReqCallback, retry ...bool) ([]byte, int, error) {
url := d.Address + "/api" + api
req := base.RestyClient.R()
req.SetHeader("Authorization", d.Token)
if callback != nil {
callback(req)
}
res, err := req.Execute(method, url)
if err != nil {
code := 0
if res != nil {
code = res.StatusCode()
}
return nil, code, err
}
log.Debugf("[openlist] response body: %s", res.String())
if res.StatusCode() >= 400 {
return nil, res.StatusCode(), fmt.Errorf("request failed, status: %s", res.Status())
}
code := utils.Json.Get(res.Body(), "code").ToInt()
if code != 200 {
if (code == 401 || code == 403) && !utils.IsBool(retry...) {
err = d.login()
if err != nil {
return nil, code, err
}
return d.request(api, method, callback, true)
}
return nil, code, fmt.Errorf("request failed,code: %d, message: %s", code, utils.Json.Get(res.Body(), "message").ToString())
}
return res.Body(), 200, nil
}
+20 -4
View File
@@ -45,7 +45,7 @@ func (d *AliDrive) GetAddition() driver.Additional {
func (d *AliDrive) Init(ctx context.Context) error {
// TODO login / refresh token
//op.MustSaveDriverStorage(d)
// op.MustSaveDriverStorage(d)
err := d.refreshToken()
if err != nil {
return err
@@ -171,7 +171,7 @@ func (d *AliDrive) Put(ctx context.Context, dstDir model.Obj, streamer model.Fil
Mimetype: streamer.GetMimetype(),
}
const DEFAULT int64 = 10485760
var count = int(math.Ceil(float64(streamer.GetSize()) / float64(DEFAULT)))
count := int(math.Ceil(float64(streamer.GetSize()) / float64(DEFAULT)))
partInfoList := make([]base.Json, 0, count)
for i := 1; i <= count; i++ {
@@ -297,11 +297,10 @@ func (d *AliDrive) Put(ctx context.Context, dstDir model.Obj, streamer model.Fil
if d.InternalUpload {
url = partInfo.InternalUploadUrl
}
req, err := http.NewRequest("PUT", url, io.LimitReader(rateLimited, DEFAULT))
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, io.LimitReader(rateLimited, DEFAULT))
if err != nil {
return err
}
req = req.WithContext(ctx)
res, err := base.HttpClient.Do(req)
if err != nil {
return err
@@ -328,6 +327,23 @@ func (d *AliDrive) Put(ctx context.Context, dstDir model.Obj, streamer model.Fil
return fmt.Errorf("%+v", resp2)
}
func (d *AliDrive) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
res, err, _ := d.request("https://api.aliyundrive.com/adrive/v1/user/driveCapacityDetails", http.MethodPost, func(req *resty.Request) {
req.SetContext(ctx)
}, nil)
if err != nil {
return nil, err
}
used := utils.Json.Get(res, "drive_used_size").ToInt64()
total := utils.Json.Get(res, "drive_total_size").ToInt64()
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: total,
UsedSpace: used,
},
}, nil
}
func (d *AliDrive) Other(ctx context.Context, args model.OtherArgs) (interface{}, error) {
var resp base.Json
var url string
+34 -33
View File
@@ -3,7 +3,6 @@ package aliyundrive_open
import (
"context"
"errors"
"fmt"
"net/http"
"path/filepath"
"time"
@@ -13,7 +12,6 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/rateg"
"github.com/go-resty/resty/v2"
log "github.com/sirupsen/logrus"
)
@@ -24,9 +22,8 @@ type AliyundriveOpen struct {
DriveId string
limitList func(ctx context.Context, data base.Json) (*Files, error)
limitLink func(ctx context.Context, file model.Obj) (*model.Link, error)
ref *AliyundriveOpen
limiter *limiter
ref *AliyundriveOpen
}
func (d *AliyundriveOpen) Config() driver.Config {
@@ -38,25 +35,23 @@ func (d *AliyundriveOpen) GetAddition() driver.Additional {
}
func (d *AliyundriveOpen) Init(ctx context.Context) error {
d.limiter = getLimiterForUser(globalLimiterUserID) // First create a globally shared limiter to limit the initial requests.
if d.LIVPDownloadFormat == "" {
d.LIVPDownloadFormat = "jpeg"
}
if d.DriveType == "" {
d.DriveType = "default"
}
res, err := d.request("/adrive/v1.0/user/getDriveInfo", http.MethodPost, nil)
res, err := d.request(ctx, limiterOther, "/adrive/v1.0/user/getDriveInfo", http.MethodPost, nil)
if err != nil {
d.limiter.free()
d.limiter = nil
return err
}
d.DriveId = utils.Json.Get(res, d.DriveType+"_drive_id").ToString()
d.limitList = rateg.LimitFnCtx(d.list, rateg.LimitFnOption{
Limit: 4,
Bucket: 1,
})
d.limitLink = rateg.LimitFnCtx(d.link, rateg.LimitFnOption{
Limit: 1,
Bucket: 1,
})
userid := utils.Json.Get(res, "user_id").ToString()
d.limiter.free()
d.limiter = getLimiterForUser(userid) // Allocate a corresponding limiter for each user.
return nil
}
@@ -70,6 +65,8 @@ func (d *AliyundriveOpen) InitReference(storage driver.Driver) error {
}
func (d *AliyundriveOpen) Drop(ctx context.Context) error {
d.limiter.free()
d.limiter = nil
d.ref = nil
return nil
}
@@ -80,16 +77,12 @@ func (d *AliyundriveOpen) GetRoot(ctx context.Context) (model.Obj, error) {
ID: d.RootFolderID,
Path: "/",
Name: "root",
Size: 0,
Modified: d.Modified,
IsFolder: true,
}, nil
}
func (d *AliyundriveOpen) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
if d.limitList == nil {
return nil, fmt.Errorf("driver not init")
}
files, err := d.getFiles(ctx, dir.GetID())
if err != nil {
return nil, err
@@ -107,8 +100,8 @@ func (d *AliyundriveOpen) List(ctx context.Context, dir model.Obj, args model.Li
return objs, err
}
func (d *AliyundriveOpen) link(ctx context.Context, file model.Obj) (*model.Link, error) {
res, err := d.request("/adrive/v1.0/openFile/getDownloadUrl", http.MethodPost, func(req *resty.Request) {
func (d *AliyundriveOpen) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
res, err := d.request(ctx, limiterLink, "/adrive/v1.0/openFile/getDownloadUrl", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": file.GetID(),
@@ -132,17 +125,10 @@ func (d *AliyundriveOpen) link(ctx context.Context, file model.Obj) (*model.Link
}, nil
}
func (d *AliyundriveOpen) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if d.limitLink == nil {
return nil, fmt.Errorf("driver not init")
}
return d.limitLink(ctx, file)
}
func (d *AliyundriveOpen) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) (model.Obj, error) {
nowTime, _ := getNowTime()
newDir := File{CreatedAt: nowTime, UpdatedAt: nowTime}
_, err := d.request("/adrive/v1.0/openFile/create", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, "/adrive/v1.0/openFile/create", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"parent_file_id": parentDir.GetID(),
@@ -168,7 +154,7 @@ func (d *AliyundriveOpen) MakeDir(ctx context.Context, parentDir model.Obj, dirN
func (d *AliyundriveOpen) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.Obj, error) {
var resp MoveOrCopyResp
_, err := d.request("/adrive/v1.0/openFile/move", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, "/adrive/v1.0/openFile/move", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": srcObj.GetID(),
@@ -198,7 +184,7 @@ func (d *AliyundriveOpen) Move(ctx context.Context, srcObj, dstDir model.Obj) (m
func (d *AliyundriveOpen) Rename(ctx context.Context, srcObj model.Obj, newName string) (model.Obj, error) {
var newFile File
_, err := d.request("/adrive/v1.0/openFile/update", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, "/adrive/v1.0/openFile/update", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": srcObj.GetID(),
@@ -230,7 +216,7 @@ func (d *AliyundriveOpen) Rename(ctx context.Context, srcObj model.Obj, newName
func (d *AliyundriveOpen) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
var resp MoveOrCopyResp
_, err := d.request("/adrive/v1.0/openFile/copy", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, "/adrive/v1.0/openFile/copy", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": srcObj.GetID(),
@@ -256,7 +242,7 @@ func (d *AliyundriveOpen) Remove(ctx context.Context, obj model.Obj) error {
if d.RemoveWay == "delete" {
uri = "/adrive/v1.0/openFile/delete"
}
_, err := d.request(uri, http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, uri, http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": obj.GetID(),
@@ -295,7 +281,7 @@ func (d *AliyundriveOpen) Other(ctx context.Context, args model.OtherArgs) (inte
default:
return nil, errs.NotSupport
}
_, err := d.request(uri, http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, uri, http.MethodPost, func(req *resty.Request) {
req.SetBody(data).SetResult(&resp)
})
if err != nil {
@@ -304,6 +290,21 @@ func (d *AliyundriveOpen) Other(ctx context.Context, args model.OtherArgs) (inte
return resp, nil
}
func (d *AliyundriveOpen) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
res, err := d.request(ctx, limiterOther, "/adrive/v1.0/user/getSpaceInfo", http.MethodPost, nil)
if err != nil {
return nil, err
}
total := utils.Json.Get(res, "personal_space_info", "total_size").ToInt64()
used := utils.Json.Get(res, "personal_space_info", "used_size").ToInt64()
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: total,
UsedSpace: used,
},
}, nil
}
var _ driver.Driver = (*AliyundriveOpen)(nil)
var _ driver.MkdirResult = (*AliyundriveOpen)(nil)
var _ driver.MoveResult = (*AliyundriveOpen)(nil)
+96
View File
@@ -0,0 +1,96 @@
package aliyundrive_open
import (
"context"
"fmt"
"sync"
"golang.org/x/time/rate"
)
// See document https://www.yuque.com/aliyundrive/zpfszx/mqocg38hlxzc5vcd
// See issue https://github.com/OpenListTeam/OpenList/issues/724
// We got limit per user per app, so the limiter should be global.
type limiterType int
const (
limiterList limiterType = iota
limiterLink
limiterOther
)
const (
listRateLimit = 3.9 // 4 per second in document, but we use 3.9 per second to be safe
linkRateLimit = 0.9 // 1 per second in document, but we use 0.9 per second to be safe
otherRateLimit = 14.9 // 15 per second in document, but we use 14.9 per second to be safe
globalLimiterUserID = "" // Global limiter user ID, used to limit the initial requests.
)
type limiter struct {
usedBy int
list *rate.Limiter
link *rate.Limiter
other *rate.Limiter
}
var limiters = make(map[string]*limiter)
var limitersLock = &sync.Mutex{}
func getLimiterForUser(userid string) *limiter {
limitersLock.Lock()
defer limitersLock.Unlock()
defer func() {
// Clean up limiters that are no longer used.
for id, lim := range limiters {
if lim.usedBy <= 0 && id != globalLimiterUserID { // Do not delete the global limiter.
delete(limiters, id)
}
}
}()
if lim, ok := limiters[userid]; ok {
lim.usedBy++
return lim
}
lim := &limiter{
usedBy: 1,
list: rate.NewLimiter(rate.Limit(listRateLimit), 1),
link: rate.NewLimiter(rate.Limit(linkRateLimit), 1),
other: rate.NewLimiter(rate.Limit(otherRateLimit), 1),
}
limiters[userid] = lim
return lim
}
func (l *limiter) wait(ctx context.Context, typ limiterType) error {
if l == nil {
return fmt.Errorf("driver not init")
}
switch typ {
case limiterList:
return l.list.Wait(ctx)
case limiterLink:
return l.link.Wait(ctx)
case limiterOther:
return l.other.Wait(ctx)
default:
return fmt.Errorf("unknown limiter type")
}
}
func (l *limiter) free() {
if l == nil {
return
}
limitersLock.Lock()
defer limitersLock.Unlock()
l.usedBy--
}
func (d *AliyundriveOpen) wait(ctx context.Context, typ limiterType) error {
if d == nil {
return fmt.Errorf("driver not init")
}
if d.ref != nil {
return d.ref.wait(ctx, typ) // If this is a reference driver, wait on the reference driver.
}
return d.limiter.wait(ctx, typ)
}
+23 -28
View File
@@ -50,10 +50,10 @@ func calPartSize(fileSize int64) int64 {
return partSize
}
func (d *AliyundriveOpen) getUploadUrl(count int, fileId, uploadId string) ([]PartInfo, error) {
func (d *AliyundriveOpen) getUploadUrl(ctx context.Context, count int, fileId, uploadId string) ([]PartInfo, error) {
partInfoList := makePartInfos(count)
var resp CreateResp
_, err := d.request("/adrive/v1.0/openFile/getUploadUrl", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, "/adrive/v1.0/openFile/getUploadUrl", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": fileId,
@@ -69,7 +69,7 @@ func (d *AliyundriveOpen) uploadPart(ctx context.Context, r io.Reader, partInfo
if d.InternalUpload {
uploadUrl = strings.ReplaceAll(uploadUrl, "https://cn-beijing-data.aliyundrive.net/", "http://ccp-bj29-bj-1592982087.oss-cn-beijing-internal.aliyuncs.com/")
}
req, err := http.NewRequestWithContext(ctx, "PUT", uploadUrl, r)
req, err := http.NewRequestWithContext(ctx, http.MethodPut, uploadUrl, r)
if err != nil {
return err
}
@@ -84,10 +84,10 @@ func (d *AliyundriveOpen) uploadPart(ctx context.Context, r io.Reader, partInfo
return nil
}
func (d *AliyundriveOpen) completeUpload(fileId, uploadId string) (model.Obj, error) {
func (d *AliyundriveOpen) completeUpload(ctx context.Context, fileId, uploadId string) (model.Obj, error) {
// 3. complete
var newFile File
_, err := d.request("/adrive/v1.0/openFile/complete", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, "/adrive/v1.0/openFile/complete", http.MethodPost, func(req *resty.Request) {
req.SetBody(base.Json{
"drive_id": d.DriveId,
"file_id": fileId,
@@ -137,11 +137,8 @@ func (d *AliyundriveOpen) calProofCode(stream model.FileStreamer) (string, error
}
buf := make([]byte, length)
n, err := io.ReadFull(reader, buf)
if err == io.ErrUnexpectedEOF {
return "", fmt.Errorf("can't read data, expected=%d, got=%d", len(buf), n)
}
if err != nil {
return "", err
if n != int(length) {
return "", fmt.Errorf("failed to read all data: (expect =%d, actual =%d) %w", length, n, err)
}
return base64.StdEncoding.EncodeToString(buf), nil
}
@@ -183,7 +180,7 @@ func (d *AliyundriveOpen) upload(ctx context.Context, dstDir model.Obj, stream m
createData["pre_hash"] = hash
}
var createResp CreateResp
_, err, e := d.requestReturnErrResp("/adrive/v1.0/openFile/create", http.MethodPost, func(req *resty.Request) {
_, err, e := d.requestReturnErrResp(ctx, limiterOther, "/adrive/v1.0/openFile/create", http.MethodPost, func(req *resty.Request) {
req.SetBody(createData).SetResult(&createResp)
})
if err != nil {
@@ -194,9 +191,7 @@ func (d *AliyundriveOpen) upload(ctx context.Context, dstDir model.Obj, stream m
hash := stream.GetHash().GetHash(utils.SHA1)
if len(hash) != utils.SHA1.Width {
cacheFileProgress := model.UpdateProgressWithRange(up, 0, 50)
up = model.UpdateProgressWithRange(up, 50, 100)
_, hash, err = streamPkg.CacheFullInTempFileAndHash(stream, cacheFileProgress, utils.SHA1)
_, hash, err = streamPkg.CacheFullAndHash(stream, &up, utils.SHA1)
if err != nil {
return nil, err
}
@@ -210,7 +205,7 @@ func (d *AliyundriveOpen) upload(ctx context.Context, dstDir model.Obj, stream m
if err != nil {
return nil, fmt.Errorf("cal proof code error: %s", err.Error())
}
_, err = d.request("/adrive/v1.0/openFile/create", http.MethodPost, func(req *resty.Request) {
_, err = d.request(ctx, limiterOther, "/adrive/v1.0/openFile/create", http.MethodPost, func(req *resty.Request) {
req.SetBody(createData).SetResult(&createResp)
})
if err != nil {
@@ -221,17 +216,20 @@ func (d *AliyundriveOpen) upload(ctx context.Context, dstDir model.Obj, stream m
if !createResp.RapidUpload {
// 2. normal upload
log.Debugf("[aliyundive_open] normal upload")
ss, err := streamPkg.NewStreamSectionReader(stream, int(partSize), &up)
if err != nil {
return nil, err
}
preTime := time.Now()
var offset, length int64 = 0, partSize
//var length
for i := 0; i < len(createResp.PartInfoList); i++ {
if utils.IsCanceled(ctx) {
return nil, ctx.Err()
}
// refresh upload url if 50 minutes passed
if time.Since(preTime) > 50*time.Minute {
createResp.PartInfoList, err = d.getUploadUrl(count, createResp.FileId, createResp.UploadId)
createResp.PartInfoList, err = d.getUploadUrl(ctx, count, createResp.FileId, createResp.UploadId)
if err != nil {
return nil, err
}
@@ -240,22 +238,19 @@ func (d *AliyundriveOpen) upload(ctx context.Context, dstDir model.Obj, stream m
if remain := stream.GetSize() - offset; length > remain {
length = remain
}
rd := utils.NewMultiReadable(io.LimitReader(stream, partSize))
if rapidUpload {
srd, err := stream.RangeRead(http_range.Range{Start: offset, Length: length})
if err != nil {
return nil, err
}
rd = utils.NewMultiReadable(srd)
rd, err := ss.GetSectionReader(offset, length)
if err != nil {
return nil, err
}
err = retry.Do(func() error {
_ = rd.Reset()
rateLimitedRd := driver.NewLimitedUploadStream(ctx, rd)
return d.uploadPart(ctx, rateLimitedRd, createResp.PartInfoList[i])
rd.Seek(0, io.SeekStart)
return d.uploadPart(ctx, driver.NewLimitedUploadStream(ctx, rd), createResp.PartInfoList[i])
},
retry.Context(ctx),
retry.Attempts(3),
retry.DelayType(retry.BackOffDelay),
retry.Delay(time.Second))
ss.FreeSectionReader(rd)
if err != nil {
return nil, err
}
@@ -268,5 +263,5 @@ func (d *AliyundriveOpen) upload(ctx context.Context, dstDir model.Obj, stream m
log.Debugf("[aliyundrive_open] create file success, resp: %+v", createResp)
// 3. complete
return d.completeUpload(createResp.FileId, createResp.UploadId)
return d.completeUpload(ctx, createResp.FileId, createResp.UploadId)
}
+26 -17
View File
@@ -19,7 +19,7 @@ import (
// do others that not defined in Driver interface
func (d *AliyundriveOpen) _refreshToken() (string, string, error) {
func (d *AliyundriveOpen) _refreshToken(ctx context.Context) (string, string, error) {
if d.UseOnlineAPI && d.APIAddress != "" {
u := d.APIAddress
var resp struct {
@@ -27,15 +27,17 @@ func (d *AliyundriveOpen) _refreshToken() (string, string, error) {
AccessToken string `json:"access_token"`
ErrorMessage string `json:"text"`
}
// 根据AlipanType选项设置driver_txt
driverTxt := "alicloud_qr"
if d.AlipanType == "alipanTV" {
driverTxt = "alicloud_tv"
}
_, err := base.RestyClient.R().
SetHeader("User-Agent", "Mozilla/5.0 (Macintosh; Apple macOS 15_5) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36 Chrome/138.0.0.0 Openlist/425.6.30").
err := d.wait(ctx, limiterOther)
if err != nil {
return "", "", err
}
_, err = base.RestyClient.R().
SetResult(&resp).
SetQueryParams(map[string]string{
"refresh_ui": d.RefreshToken,
@@ -54,11 +56,14 @@ func (d *AliyundriveOpen) _refreshToken() (string, string, error) {
}
return resp.RefreshToken, resp.AccessToken, nil
}
// 本地刷新逻辑,必须要求 client_id 和 client_secret
if d.ClientID == "" || d.ClientSecret == "" {
return "", "", fmt.Errorf("empty ClientID or ClientSecret")
}
err := d.wait(ctx, limiterOther)
if err != nil {
return "", "", err
}
url := API_URL + "/oauth/access_token"
//var resp base.TokenResp
var e ErrResp
@@ -110,18 +115,18 @@ func getSub(token string) (string, error) {
return utils.Json.Get(bs, "sub").ToString(), nil
}
func (d *AliyundriveOpen) refreshToken() error {
func (d *AliyundriveOpen) refreshToken(ctx context.Context) error {
if d.ref != nil {
return d.ref.refreshToken()
return d.ref.refreshToken(ctx)
}
refresh, access, err := d._refreshToken()
refresh, access, err := d._refreshToken(ctx)
for i := 0; i < 3; i++ {
if err == nil {
break
} else {
log.Errorf("[ali_open] failed to refresh token: %s", err)
}
refresh, access, err = d._refreshToken()
refresh, access, err = d._refreshToken(ctx)
}
if err != nil {
return err
@@ -132,12 +137,12 @@ func (d *AliyundriveOpen) refreshToken() error {
return nil
}
func (d *AliyundriveOpen) request(uri, method string, callback base.ReqCallback, retry ...bool) ([]byte, error) {
b, err, _ := d.requestReturnErrResp(uri, method, callback, retry...)
func (d *AliyundriveOpen) request(ctx context.Context, limitTy limiterType, uri, method string, callback base.ReqCallback, retry ...bool) ([]byte, error) {
b, err, _ := d.requestReturnErrResp(ctx, limitTy, uri, method, callback, retry...)
return b, err
}
func (d *AliyundriveOpen) requestReturnErrResp(uri, method string, callback base.ReqCallback, retry ...bool) ([]byte, error, *ErrResp) {
func (d *AliyundriveOpen) requestReturnErrResp(ctx context.Context, limitTy limiterType, uri, method string, callback base.ReqCallback, retry ...bool) ([]byte, error, *ErrResp) {
req := base.RestyClient.R()
// TODO check whether access_token is expired
req.SetHeader("Authorization", "Bearer "+d.getAccessToken())
@@ -149,6 +154,10 @@ func (d *AliyundriveOpen) requestReturnErrResp(uri, method string, callback base
}
var e ErrResp
req.SetError(&e)
err := d.wait(ctx, limitTy)
if err != nil {
return nil, err, nil
}
res, err := req.Execute(method, API_URL+uri)
if err != nil {
if res != nil {
@@ -159,11 +168,11 @@ func (d *AliyundriveOpen) requestReturnErrResp(uri, method string, callback base
isRetry := len(retry) > 0 && retry[0]
if e.Code != "" {
if !isRetry && (utils.SliceContains([]string{"AccessTokenInvalid", "AccessTokenExpired", "I400JD"}, e.Code) || d.getAccessToken() == "") {
err = d.refreshToken()
err = d.refreshToken(ctx)
if err != nil {
return nil, err, nil
}
return d.requestReturnErrResp(uri, method, callback, true)
return d.requestReturnErrResp(ctx, limitTy, uri, method, callback, true)
}
return nil, fmt.Errorf("%s:%s", e.Code, e.Message), &e
}
@@ -172,7 +181,7 @@ func (d *AliyundriveOpen) requestReturnErrResp(uri, method string, callback base
func (d *AliyundriveOpen) list(ctx context.Context, data base.Json) (*Files, error) {
var resp Files
_, err := d.request("/adrive/v1.0/openFile/list", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterList, "/adrive/v1.0/openFile/list", http.MethodPost, func(req *resty.Request) {
req.SetBody(data).SetResult(&resp)
})
if err != nil {
@@ -201,7 +210,7 @@ func (d *AliyundriveOpen) getFiles(ctx context.Context, fileId string) ([]File,
//"video_thumbnail_width": 480,
//"image_thumbnail_width": 480,
}
resp, err := d.limitList(ctx, data)
resp, err := d.list(ctx, data)
if err != nil {
return nil, err
}
+14 -32
View File
@@ -2,7 +2,6 @@ package aliyundrive_share
import (
"context"
"fmt"
"net/http"
"time"
@@ -12,7 +11,6 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/cron"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/rateg"
"github.com/go-resty/resty/v2"
log "github.com/sirupsen/logrus"
)
@@ -25,8 +23,7 @@ type AliyundriveShare struct {
DriveId string
cron *cron.Cron
limitList func(ctx context.Context, dir model.Obj) ([]model.Obj, error)
limitLink func(ctx context.Context, file model.Obj) (*model.Link, error)
limiter *limiter
}
func (d *AliyundriveShare) Config() driver.Config {
@@ -38,29 +35,26 @@ func (d *AliyundriveShare) GetAddition() driver.Additional {
}
func (d *AliyundriveShare) Init(ctx context.Context) error {
err := d.refreshToken()
d.limiter = getLimiter()
err := d.refreshToken(ctx)
if err != nil {
d.limiter.free()
d.limiter = nil
return err
}
err = d.getShareToken()
err = d.getShareToken(ctx)
if err != nil {
d.limiter.free()
d.limiter = nil
return err
}
d.cron = cron.NewCron(time.Hour * 2)
d.cron.Do(func() {
err := d.refreshToken()
err := d.refreshToken(ctx)
if err != nil {
log.Errorf("%+v", err)
}
})
d.limitList = rateg.LimitFnCtx(d.list, rateg.LimitFnOption{
Limit: 4,
Bucket: 1,
})
d.limitLink = rateg.LimitFnCtx(d.link, rateg.LimitFnOption{
Limit: 1,
Bucket: 1,
})
return nil
}
@@ -68,19 +62,14 @@ func (d *AliyundriveShare) Drop(ctx context.Context) error {
if d.cron != nil {
d.cron.Stop()
}
d.limiter.free()
d.limiter = nil
d.DriveId = ""
return nil
}
func (d *AliyundriveShare) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
if d.limitList == nil {
return nil, fmt.Errorf("driver not init")
}
return d.limitList(ctx, dir)
}
func (d *AliyundriveShare) list(ctx context.Context, dir model.Obj) ([]model.Obj, error) {
files, err := d.getFiles(dir.GetID())
files, err := d.getFiles(ctx, dir.GetID())
if err != nil {
return nil, err
}
@@ -90,13 +79,6 @@ func (d *AliyundriveShare) list(ctx context.Context, dir model.Obj) ([]model.Obj
}
func (d *AliyundriveShare) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if d.limitLink == nil {
return nil, fmt.Errorf("driver not init")
}
return d.limitLink(ctx, file)
}
func (d *AliyundriveShare) link(ctx context.Context, file model.Obj) (*model.Link, error) {
data := base.Json{
"drive_id": d.DriveId,
"file_id": file.GetID(),
@@ -105,7 +87,7 @@ func (d *AliyundriveShare) link(ctx context.Context, file model.Obj) (*model.Lin
"share_id": d.ShareId,
}
var resp ShareLinkResp
_, err := d.request("https://api.alipan.com/v2/file/get_share_link_download_url", http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterLink, "https://api.alipan.com/v2/file/get_share_link_download_url", http.MethodPost, func(req *resty.Request) {
req.SetHeader(CanaryHeaderKey, CanaryHeaderValue).SetBody(data).SetResult(&resp)
})
if err != nil {
@@ -135,7 +117,7 @@ func (d *AliyundriveShare) Other(ctx context.Context, args model.OtherArgs) (int
default:
return nil, errs.NotSupport
}
_, err := d.request(url, http.MethodPost, func(req *resty.Request) {
_, err := d.request(ctx, limiterOther, url, http.MethodPost, func(req *resty.Request) {
req.SetBody(data).SetResult(&resp)
})
if err != nil {
+67
View File
@@ -0,0 +1,67 @@
package aliyundrive_share
import (
"context"
"fmt"
"golang.org/x/time/rate"
)
// See issue https://github.com/OpenListTeam/OpenList/issues/724
// Seems there is no limit per user.
type limiterType int
const (
limiterList limiterType = iota
limiterLink
limiterOther
)
const (
listRateLimit = 3.9 // 4 per second in document, but we use 3.9 per second to be safe
linkRateLimit = 0.9 // 1 per second in document, but we use 0.9 per second to be safe
otherRateLimit = 14.9 // 15 per second in document, but we use 14.9 per second to be safe
)
type limiter struct {
list *rate.Limiter
link *rate.Limiter
other *rate.Limiter
}
func getLimiter() *limiter {
return &limiter{
list: rate.NewLimiter(rate.Limit(listRateLimit), 1),
link: rate.NewLimiter(rate.Limit(linkRateLimit), 1),
other: rate.NewLimiter(rate.Limit(otherRateLimit), 1),
}
}
func (l *limiter) wait(ctx context.Context, typ limiterType) error {
if l == nil {
return fmt.Errorf("driver not init")
}
switch typ {
case limiterList:
return l.list.Wait(ctx)
case limiterLink:
return l.link.Wait(ctx)
case limiterOther:
return l.other.Wait(ctx)
default:
return fmt.Errorf("unknown limiter type")
}
}
func (l *limiter) free() {
}
func (d *AliyundriveShare) wait(ctx context.Context, typ limiterType) error {
if d == nil {
return fmt.Errorf("driver not init")
}
//if d.ref != nil {
// return d.ref.wait(ctx, typ) // If this is a reference driver, wait on the reference driver.
//}
return d.limiter.wait(ctx, typ)
}
+28 -11
View File
@@ -1,6 +1,7 @@
package aliyundrive_share
import (
"context"
"errors"
"fmt"
@@ -15,11 +16,15 @@ const (
CanaryHeaderValue = "client=web,app=share,version=v2.3.1"
)
func (d *AliyundriveShare) refreshToken() error {
func (d *AliyundriveShare) refreshToken(ctx context.Context) error {
err := d.wait(ctx, limiterOther)
if err != nil {
return err
}
url := "https://auth.alipan.com/v2/account/token"
var resp base.TokenResp
var e ErrorResp
_, err := base.RestyClient.R().
_, err = base.RestyClient.R().
SetBody(base.Json{"refresh_token": d.RefreshToken, "grant_type": "refresh_token"}).
SetResult(&resp).
SetError(&e).
@@ -36,7 +41,11 @@ func (d *AliyundriveShare) refreshToken() error {
}
// do others that not defined in Driver interface
func (d *AliyundriveShare) getShareToken() error {
func (d *AliyundriveShare) getShareToken(ctx context.Context) error {
err := d.wait(ctx, limiterOther)
if err != nil {
return err
}
data := base.Json{
"share_id": d.ShareId,
}
@@ -45,7 +54,7 @@ func (d *AliyundriveShare) getShareToken() error {
}
var e ErrorResp
var resp ShareTokenResp
_, err := base.RestyClient.R().
_, err = base.RestyClient.R().
SetResult(&resp).SetError(&e).SetBody(data).
Post("https://api.alipan.com/v2/share_link/get_share_token")
if err != nil {
@@ -58,7 +67,7 @@ func (d *AliyundriveShare) getShareToken() error {
return nil
}
func (d *AliyundriveShare) request(url, method string, callback base.ReqCallback) ([]byte, error) {
func (d *AliyundriveShare) request(ctx context.Context, limitTy limiterType, url, method string, callback base.ReqCallback) ([]byte, error) {
var e ErrorResp
req := base.RestyClient.R().
SetError(&e).
@@ -71,6 +80,10 @@ func (d *AliyundriveShare) request(url, method string, callback base.ReqCallback
} else {
req.SetBody("{}")
}
err := d.wait(ctx, limitTy)
if err != nil {
return nil, err
}
resp, err := req.Execute(method, url)
if err != nil {
return nil, err
@@ -78,14 +91,14 @@ func (d *AliyundriveShare) request(url, method string, callback base.ReqCallback
if e.Code != "" {
if e.Code == "AccessTokenInvalid" || e.Code == "ShareLinkTokenInvalid" {
if e.Code == "AccessTokenInvalid" {
err = d.refreshToken()
err = d.refreshToken(ctx)
} else {
err = d.getShareToken()
err = d.getShareToken(ctx)
}
if err != nil {
return nil, err
}
return d.request(url, method, callback)
return d.request(ctx, limitTy, url, method, callback)
} else {
return nil, errors.New(e.Code + ": " + e.Message)
}
@@ -93,7 +106,7 @@ func (d *AliyundriveShare) request(url, method string, callback base.ReqCallback
return resp.Body(), nil
}
func (d *AliyundriveShare) getFiles(fileId string) ([]File, error) {
func (d *AliyundriveShare) getFiles(ctx context.Context, fileId string) ([]File, error) {
files := make([]File, 0)
data := base.Json{
"image_thumbnail_process": "image/resize,w_160/format,jpeg",
@@ -110,6 +123,10 @@ func (d *AliyundriveShare) getFiles(fileId string) ([]File, error) {
if data["marker"] == "first" {
data["marker"] = ""
}
err := d.wait(ctx, limiterList)
if err != nil {
return nil, err
}
var e ErrorResp
var resp ListResp
res, err := base.RestyClient.R().
@@ -123,11 +140,11 @@ func (d *AliyundriveShare) getFiles(fileId string) ([]File, error) {
log.Debugf("aliyundrive share get files: %s", res.String())
if e.Code != "" {
if e.Code == "AccessTokenInvalid" || e.Code == "ShareLinkTokenInvalid" {
err = d.getShareToken()
err = d.getShareToken(ctx)
if err != nil {
return nil, err
}
return d.getFiles(fileId)
return d.getFiles(ctx, fileId)
}
return nil, errors.New(e.Message)
}
+12 -1
View File
@@ -13,17 +13,23 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/189_tv"
_ "github.com/OpenListTeam/OpenList/v4/drivers/189pc"
_ "github.com/OpenListTeam/OpenList/v4/drivers/alias"
_ "github.com/OpenListTeam/OpenList/v4/drivers/alist_v3"
_ "github.com/OpenListTeam/OpenList/v4/drivers/aliyundrive"
_ "github.com/OpenListTeam/OpenList/v4/drivers/aliyundrive_open"
_ "github.com/OpenListTeam/OpenList/v4/drivers/aliyundrive_share"
_ "github.com/OpenListTeam/OpenList/v4/drivers/autoindex"
_ "github.com/OpenListTeam/OpenList/v4/drivers/azure_blob"
_ "github.com/OpenListTeam/OpenList/v4/drivers/baidu_netdisk"
_ "github.com/OpenListTeam/OpenList/v4/drivers/baidu_photo"
_ "github.com/OpenListTeam/OpenList/v4/drivers/chaoxing"
_ "github.com/OpenListTeam/OpenList/v4/drivers/chunk"
_ "github.com/OpenListTeam/OpenList/v4/drivers/cloudreve"
_ "github.com/OpenListTeam/OpenList/v4/drivers/cloudreve_v4"
_ "github.com/OpenListTeam/OpenList/v4/drivers/cnb_releases"
_ "github.com/OpenListTeam/OpenList/v4/drivers/crypt"
_ "github.com/OpenListTeam/OpenList/v4/drivers/degoo"
_ "github.com/OpenListTeam/OpenList/v4/drivers/doubao"
_ "github.com/OpenListTeam/OpenList/v4/drivers/doubao_new"
_ "github.com/OpenListTeam/OpenList/v4/drivers/doubao_share"
_ "github.com/OpenListTeam/OpenList/v4/drivers/dropbox"
_ "github.com/OpenListTeam/OpenList/v4/drivers/febbox"
@@ -33,12 +39,14 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/google_drive"
_ "github.com/OpenListTeam/OpenList/v4/drivers/google_photo"
_ "github.com/OpenListTeam/OpenList/v4/drivers/halalcloud"
_ "github.com/OpenListTeam/OpenList/v4/drivers/halalcloud_open"
_ "github.com/OpenListTeam/OpenList/v4/drivers/ilanzou"
_ "github.com/OpenListTeam/OpenList/v4/drivers/ipfs_api"
_ "github.com/OpenListTeam/OpenList/v4/drivers/kodbox"
_ "github.com/OpenListTeam/OpenList/v4/drivers/lanzou"
_ "github.com/OpenListTeam/OpenList/v4/drivers/lenovonas_share"
_ "github.com/OpenListTeam/OpenList/v4/drivers/local"
_ "github.com/OpenListTeam/OpenList/v4/drivers/mediafire"
_ "github.com/OpenListTeam/OpenList/v4/drivers/mediatrack"
_ "github.com/OpenListTeam/OpenList/v4/drivers/mega"
_ "github.com/OpenListTeam/OpenList/v4/drivers/misskey"
@@ -48,8 +56,10 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/onedrive_app"
_ "github.com/OpenListTeam/OpenList/v4/drivers/onedrive_sharelink"
_ "github.com/OpenListTeam/OpenList/v4/drivers/openlist"
_ "github.com/OpenListTeam/OpenList/v4/drivers/openlist_share"
_ "github.com/OpenListTeam/OpenList/v4/drivers/pikpak"
_ "github.com/OpenListTeam/OpenList/v4/drivers/pikpak_share"
_ "github.com/OpenListTeam/OpenList/v4/drivers/proton_drive"
_ "github.com/OpenListTeam/OpenList/v4/drivers/quark_open"
_ "github.com/OpenListTeam/OpenList/v4/drivers/quark_uc"
_ "github.com/OpenListTeam/OpenList/v4/drivers/quark_uc_tv"
@@ -59,6 +69,7 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/smb"
_ "github.com/OpenListTeam/OpenList/v4/drivers/strm"
_ "github.com/OpenListTeam/OpenList/v4/drivers/teambition"
_ "github.com/OpenListTeam/OpenList/v4/drivers/teldrive"
_ "github.com/OpenListTeam/OpenList/v4/drivers/terabox"
_ "github.com/OpenListTeam/OpenList/v4/drivers/thunder"
_ "github.com/OpenListTeam/OpenList/v4/drivers/thunder_browser"
@@ -69,11 +80,11 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/webdav"
_ "github.com/OpenListTeam/OpenList/v4/drivers/weiyun"
_ "github.com/OpenListTeam/OpenList/v4/drivers/wopan"
_ "github.com/OpenListTeam/OpenList/v4/drivers/wps"
_ "github.com/OpenListTeam/OpenList/v4/drivers/yandex_disk"
)
// All do nothing,just for import
// same as _ import
func All() {
}
+169
View File
@@ -0,0 +1,169 @@
package autoindex
import (
"context"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/antchfx/htmlquery"
"github.com/antchfx/xpath"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
)
type AutoIndex struct {
model.Storage
Addition
itemXPath *xpath.Expr
nameXPath *xpath.Expr
modifiedXPath *xpath.Expr
sizeXPath *xpath.Expr
ignores map[string]any
}
func (d *AutoIndex) Config() driver.Config {
return config
}
func (d *AutoIndex) GetAddition() driver.Additional {
return &d.Addition
}
func (d *AutoIndex) Init(ctx context.Context) error {
var err error
d.itemXPath, err = xpath.Compile(d.ItemXPath)
if err != nil {
return errors.WithMessage(err, "failed to compile Item XPath")
}
d.nameXPath, err = xpath.Compile(d.NameXPath)
if err != nil {
return errors.WithMessage(err, "failed to compile Name XPath")
}
if len(d.ModifiedXPath) > 0 {
d.modifiedXPath, err = xpath.Compile(d.ModifiedXPath)
if err != nil {
return errors.WithMessage(err, "failed to compile Modified XPath")
}
}
if len(d.SizeXPath) > 0 {
d.sizeXPath, err = xpath.Compile(d.SizeXPath)
if err != nil {
return errors.WithMessage(err, "failed to compile Size XPath")
}
}
ignores := strings.Split(d.IgnoreFileNames, "\n")
d.ignores = make(map[string]any, len(ignores))
for _, i := range ignores {
i = strings.TrimSpace(i)
if len(i) == 0 {
continue
}
d.ignores[i] = struct{}{}
}
hasScheme := strings.Contains(d.URL, "://")
hasSuffix := strings.HasSuffix(d.URL, "/")
if !hasScheme || !hasSuffix {
if !hasSuffix {
d.URL = d.URL + "/"
}
if !hasScheme {
d.URL = "https://" + d.URL
}
op.MustSaveDriverStorage(d)
}
return nil
}
func (d *AutoIndex) Drop(ctx context.Context) error {
return nil
}
func (d *AutoIndex) GetRoot(ctx context.Context) (model.Obj, error) {
return &model.Object{
Name: op.RootName,
Path: d.URL,
Modified: d.Modified,
Mask: model.Locked,
IsFolder: true,
}, nil
}
func (d *AutoIndex) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
res, err := base.RestyClient.R().
SetContext(ctx).
SetDoNotParseResponse(true).
Get(dir.GetPath())
if err != nil {
return nil, errors.WithMessagef(err, "failed to get url [%s]", dir.GetPath())
}
defer res.RawResponse.Body.Close()
doc, err := htmlquery.Parse(res.RawBody())
if err != nil {
return nil, errors.WithMessagef(err, "failed to parse [%s]", dir.GetPath())
}
itemsIter := d.itemXPath.Select(htmlquery.CreateXPathNavigator(doc))
var objs []model.Obj
for itemsIter.MoveNext() {
nameFull, err := parseString(d.nameXPath.Evaluate(itemsIter.Current().Copy()))
if err != nil {
log.Warnf("skip invalid name evaluating result: %v", err)
continue
}
nameFull = strings.TrimSpace(nameFull)
name, isDir := strings.CutSuffix(nameFull, "/")
if _, ok := d.ignores[name]; ok {
continue
}
var size int64 = 0
exact := false
modified := time.Now()
if d.sizeXPath != nil {
size, exact, err = parseSize(d.sizeXPath.Evaluate(itemsIter.Current().Copy()))
if err != nil {
log.Errorf("failed to parse size of %s: %v", name, err)
}
}
if d.modifiedXPath != nil {
modified, err = parseTime(d.modifiedXPath.Evaluate(itemsIter.Current().Copy()), d.ModifiedTimeFormat)
if err != nil {
log.Errorf("failed to parse modified time of %s: %v", name, err)
}
}
var o model.Obj = &model.Object{
Name: name,
IsFolder: isDir,
Path: dir.GetPath() + nameFull,
Modified: modified,
Size: size,
}
if exact {
o = &exactSizeObj{Obj: o}
}
objs = append(objs, o)
}
return objs, nil
}
func (d *AutoIndex) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if _, ok := file.(*exactSizeObj); ok || args.Redirect {
return &model.Link{URL: file.GetPath()}, nil
}
res, err := base.RestyClient.R().
SetContext(ctx).
SetDoNotParseResponse(true).
Head(file.GetPath())
if err != nil {
return nil, errors.WithMessagef(err, "failed to head [%s]", file.GetPath())
}
_ = res.RawResponse.Body.Close()
return &model.Link{
URL: file.GetPath(),
ContentLength: res.RawResponse.ContentLength,
}, nil
}
var _ driver.Driver = (*AutoIndex)(nil)
+29
View File
@@ -0,0 +1,29 @@
package autoindex
import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/op"
)
type Addition struct {
URL string `json:"url" required:"true"`
ItemXPath string `json:"item_xpath" required:"true"`
NameXPath string `json:"name_xpath" required:"true"`
ModifiedXPath string `json:"modified_xpath"`
SizeXPath string `json:"size_xpath"`
IgnoreFileNames string `json:"ignore_file_names" type:"text" default:".\n..\nParent Directory\nUp"`
ModifiedTimeFormat string `json:"modified_time_format" default:"02-Jan-2006 15:04" help:"Must be based on the time point Mon Jan 2 15:04:05 -0700 MST 2006"`
}
var config = driver.Config{
Name: "AutoIndex",
LocalSort: true,
CheckStatus: true,
NoUpload: true,
}
func init() {
op.RegisterDriver(func() driver.Driver {
return &AutoIndex{}
})
}
+13
View File
@@ -0,0 +1,13 @@
package autoindex
import (
"fmt"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
var (
errEmptyEvaluateResult = fmt.Errorf("empty result")
)
type exactSizeObj struct{ model.Obj }
+116
View File
@@ -0,0 +1,116 @@
package autoindex
import (
"fmt"
"strconv"
"strings"
"time"
"github.com/antchfx/xpath"
"github.com/pkg/errors"
)
var units = map[string]int64{
"": 1,
"b": 1,
"byte": 1,
"bytes": 1,
"k": 1 << 10,
"kb": 1 << 10,
"kib": 1 << 10,
"m": 1 << 20,
"mb": 1 << 20,
"mib": 1 << 20,
"g": 1 << 30,
"gb": 1 << 30,
"gib": 1 << 30,
"t": 1 << 40,
"tb": 1 << 40,
"tib": 1 << 40,
"p": 1 << 50,
"pb": 1 << 50,
"pib": 1 << 50,
}
func splitUnit(s string) (string, string) {
for i := len(s) - 1; i >= 0; i-- {
if s[i] >= '0' && s[i] <= '9' {
return strings.TrimSpace(s[:i+1]), strings.TrimSpace(s[i+1:])
}
}
return "", s
}
func parseSize(a any) (int64, bool, error) {
// 第二个返回值exact表示大小是否精确
if f, ok := a.(float64); ok {
return int64(f), false, nil
}
s, err := parseString(a)
if errors.Is(err, errEmptyEvaluateResult) {
// 可能是错误,也可能确实大小为0
// 如果确实大小为0,大概率不会下载,exact返回false也不会有什么性能损失
// 如果是错误,exact返回true会导致本地代理出错,综合来看返回false更好
return 0, false, nil
}
if err != nil {
return 0, false, err
}
s = strings.TrimSpace(s)
if s == "-" {
return 0, false, nil
}
nbs, unit := splitUnit(s)
mul, ok := units[strings.ToLower(unit)]
exact := mul == 1
if !ok {
mul = 1
// 推测无单位,exact应为false
}
nb, err := strconv.ParseInt(nbs, 10, 64)
if err != nil {
fnb, err := strconv.ParseFloat(nbs, 64)
if err != nil {
return 0, false, fmt.Errorf("failed to convert %s to number", nbs)
}
nb = int64(fnb * float64(mul))
exact = false
} else {
nb = nb * mul
}
return nb, exact, nil
}
func parseString(res any) (string, error) {
if r, ok := res.(string); ok {
if len(r) == 0 {
return "", errEmptyEvaluateResult
}
return r, nil
}
n, ok := res.(*xpath.NodeIterator)
if !ok {
return "", fmt.Errorf("unsupported evaluating result")
}
if !n.MoveNext() {
return "", fmt.Errorf("no matched nodes")
}
ns := n.Current().Value()
if len(ns) == 0 {
return "", errEmptyEvaluateResult
}
return ns, nil
}
func parseTime(res any, format string) (time.Time, error) {
s, err := parseString(res)
if err != nil {
return time.Now(), err
}
s = strings.TrimSpace(s)
t, err := time.Parse(format, s)
if err != nil {
return time.Now(), errors.WithMessagef(err, "failed to convert %s to time", s)
}
return t, nil
}
+49
View File
@@ -0,0 +1,49 @@
package autoindex
import (
"testing"
)
type wantType struct {
v int64
exact bool
error bool
}
func TestParseSize(t *testing.T) {
tests := []struct {
input string
want wantType
}{
{"100", wantType{100, true, false}},
{"1k", wantType{1024, false, false}},
{"1kb", wantType{1024, false, false}},
{"1K", wantType{1024, false, false}}, // case insensitive
{"1.5m", wantType{1572864, false, false}}, // 1.5 * 1024^2
{"500 bytes", wantType{500, true, false}},
{"-", wantType{0, false, false}},
{"", wantType{0, false, false}},
{"abc", wantType{0, false, true}},
{"1.5GB", wantType{1610612736, false, false}}, // 1.5 * 1024^3
{"2t", wantType{2199023255552, false, false}}, // 2 * 1024^4
{"1p", wantType{1125899906842624, false, false}}, // 1 * 1024^5
{"0", wantType{0, true, false}},
{" 100 ", wantType{100, true, false}}, // trimmed
{"100b", wantType{100, true, false}},
{"1gib", wantType{1073741824, false, false}}, // 1024^3
{"1z", wantType{1, false, false}}, // invalid unit, mul=1
{"1.5", wantType{1, false, false}}, // float without unit, truncated
{"2.7k", wantType{2764, false, false}}, // 2.7 * 1024 truncated
{"1.0g", wantType{1073741824, false, false}}, // 1.0 * 1024^3
{"invalid", wantType{0, false, true}},
{"123xyz", wantType{123, false, false}}, // unit not found, mul=1
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
got, exact, err := parseSize(tt.input)
if got != tt.want.v || exact != tt.want.exact || (err != nil) != tt.want.error {
t.Errorf("ParseSize(%q) = (%d, %t, %t), want (%d, %t, %t)", tt.input, got, exact, err != nil, tt.want.v, tt.want.exact, tt.want.error)
}
})
}
}
+9 -6
View File
@@ -25,12 +25,11 @@ type AzureBlob struct {
Addition
client *azblob.Client
containerClient *container.Client
config driver.Config
}
// Config returns the driver configuration.
func (d *AzureBlob) Config() driver.Config {
return d.config
return config
}
// GetAddition returns additional settings specific to Azure Blob Storage.
@@ -86,6 +85,9 @@ func (d *AzureBlob) Drop(ctx context.Context) error {
// List retrieves blobs and directories under the specified path.
func (d *AzureBlob) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
prefix := ensureTrailingSlash(dir.GetPath())
if prefix == "/" {
prefix = ""
}
pager := d.containerClient.NewListBlobsHierarchyPager("/", &container.ListBlobsHierarchyOptions{
Prefix: &prefix,
@@ -101,10 +103,11 @@ func (d *AzureBlob) List(ctx context.Context, dir model.Obj, args model.ListArgs
// Process directories
for _, blobPrefix := range page.Segment.BlobPrefixes {
objs = append(objs, &model.Object{
Name: path.Base(strings.TrimSuffix(*blobPrefix.Name, "/")),
Path: *blobPrefix.Name,
Modified: *blobPrefix.Properties.LastModified,
Ctime: *blobPrefix.Properties.CreationTime,
Name: path.Base(strings.TrimSuffix(*blobPrefix.Name, "/")),
Path: *blobPrefix.Name,
// Azure does not support properties now.
//Modified: *blobPrefix.Properties.LastModified,
//Ctime: *blobPrefix.Properties.CreationTime,
IsFolder: true,
})
}
+2 -8
View File
@@ -6,17 +6,13 @@ import (
)
type Addition struct {
driver.RootPath
Endpoint string `json:"endpoint" required:"true" default:"https://<accountname>.blob.core.windows.net/" help:"e.g. https://accountname.blob.core.windows.net/. The full endpoint URL for Azure Storage, including the unique storage account name (3 ~ 24 numbers and lowercase letters only)."`
AccessKey string `json:"access_key" required:"true" help:"The access key for Azure Storage, used for authentication. https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage"`
ContainerName string `json:"container_name" required:"true" help:"The name of the container in Azure Storage (created in the Azure portal). https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal"`
SignURLExpire int `json:"sign_url_expire" type:"number" default:"4" help:"The expiration time for SAS URLs, in hours."`
}
// implement GetRootId interface
func (r Addition) GetRootId() string {
return r.ContainerName
}
var config = driver.Config{
Name: "Azure Blob Storage",
LocalSort: true,
@@ -25,8 +21,6 @@ var config = driver.Config{
func init() {
op.RegisterDriver(func() driver.Driver {
return &AzureBlob{
config: config,
}
return &AzureBlob{}
})
}
+214 -80
View File
@@ -1,15 +1,19 @@
package baidu_netdisk
import (
"bytes"
"context"
"crypto/md5"
"encoding/hex"
"errors"
"io"
"mime/multipart"
"net/http"
"net/url"
"os"
stdpath "path"
"strconv"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
@@ -17,6 +21,7 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/net"
"github.com/OpenListTeam/OpenList/v4/pkg/errgroup"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/avast/retry-go"
@@ -31,6 +36,8 @@ type BaiduNetdisk struct {
vipType int // 会员类型,0普通用户(4G/4M)、1普通会员(10G/16M)、2超级会员(20G/32M)
}
var ErrUploadIDExpired = errors.New("uploadid expired")
func (d *BaiduNetdisk) Config() driver.Config {
return config
}
@@ -41,18 +48,20 @@ func (d *BaiduNetdisk) GetAddition() driver.Additional {
func (d *BaiduNetdisk) Init(ctx context.Context) error {
d.uploadThread, _ = strconv.Atoi(d.UploadThread)
if d.uploadThread < 1 || d.uploadThread > 32 {
d.uploadThread, d.UploadThread = 3, "3"
if d.uploadThread < 1 {
d.uploadThread, d.UploadThread = 1, "1"
} else if d.uploadThread > 32 {
d.uploadThread, d.UploadThread = 32, "32"
}
if _, err := url.Parse(d.UploadAPI); d.UploadAPI == "" || err != nil {
d.UploadAPI = "https://d.pcs.baidu.com"
d.UploadAPI = UPLOAD_FALLBACK_API
}
res, err := d.get("/xpan/nas", map[string]string{
"method": "uinfo",
}, nil)
log.Debugf("[baidu] get uinfo: %s", string(res))
log.Debugf("[baidu_netdisk] get uinfo: %s", string(res))
if err != nil {
return err
}
@@ -75,9 +84,10 @@ func (d *BaiduNetdisk) List(ctx context.Context, dir model.Obj, args model.ListA
}
func (d *BaiduNetdisk) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if d.DownloadAPI == "crack" {
switch d.DownloadAPI {
case "crack":
return d.linkCrack(file, args)
} else if d.DownloadAPI == "crack_video" {
case "crack_video":
return d.linkCrackVideo(file, args)
}
return d.linkOfficial(file, args)
@@ -179,6 +189,11 @@ func (d *BaiduNetdisk) PutRapid(ctx context.Context, dstDir model.Obj, stream mo
// **注意**: 截至 2024/04/20 百度云盘 api 接口返回的时间永远是当前时间,而不是文件时间。
// 而实际上云盘存储的时间是文件时间,所以此处需要覆盖时间,保证缓存与云盘的数据一致
func (d *BaiduNetdisk) Put(ctx context.Context, dstDir model.Obj, stream model.FileStreamer, up driver.UpdateProgress) (model.Obj, error) {
// 百度网盘不允许上传空文件
if stream.GetSize() < 1 {
return nil, ErrBaiduEmptyFilesNotAllowed
}
// rapid upload
if newObj, err := d.PutRapid(ctx, dstDir, stream); err == nil {
return newObj, nil
@@ -189,7 +204,7 @@ func (d *BaiduNetdisk) Put(ctx context.Context, dstDir model.Obj, stream model.F
tmpF *os.File
err error
)
if _, ok := cache.(io.ReaderAt); !ok {
if cache == nil {
tmpF, err = os.CreateTemp(conf.Conf.TempDir, "file-*")
if err != nil {
return nil, err
@@ -203,17 +218,17 @@ func (d *BaiduNetdisk) Put(ctx context.Context, dstDir model.Obj, stream model.F
streamSize := stream.GetSize()
sliceSize := d.getSliceSize(streamSize)
count := int(streamSize / sliceSize)
count := 1
if streamSize > sliceSize {
count = int((streamSize + sliceSize - 1) / sliceSize)
}
lastBlockSize := streamSize % sliceSize
if lastBlockSize > 0 {
count++
} else {
if lastBlockSize == 0 {
lastBlockSize = sliceSize
}
//cal md5 for first 256k data
// cal md5 for first 256k data
const SliceSize int64 = 256 * utils.KB
// cal md5
blockList := make([]string, 0, count)
byteSize := sliceSize
fileMd5H := md5.New()
@@ -243,7 +258,7 @@ func (d *BaiduNetdisk) Put(ctx context.Context, dstDir model.Obj, stream model.F
}
if tmpF != nil {
if written != streamSize {
return nil, errs.NewErr(err, "CreateTempFile failed, incoming stream actual size= %d, expect = %d ", written, streamSize)
return nil, errs.NewErr(err, "CreateTempFile failed, size mismatch: %d != %d ", written, streamSize)
}
_, err = tmpF.Seek(0, io.SeekStart)
if err != nil {
@@ -257,81 +272,111 @@ func (d *BaiduNetdisk) Put(ctx context.Context, dstDir model.Obj, stream model.F
mtime := stream.ModTime().Unix()
ctime := stream.CreateTime().Unix()
// step.1 预上传
// 尝试获取之前的进度
// step.1 尝试读取已保存进度
precreateResp, ok := base.GetUploadProgress[*PrecreateResp](d, d.AccessToken, contentMd5)
if !ok {
params := map[string]string{
"method": "precreate",
}
form := map[string]string{
"path": path,
"size": strconv.FormatInt(streamSize, 10),
"isdir": "0",
"autoinit": "1",
"rtype": "3",
"block_list": blockListStr,
"content-md5": contentMd5,
"slice-md5": sliceMd5,
}
joinTime(form, ctime, mtime)
log.Debugf("[baidu_netdisk] precreate data: %s", form)
_, err = d.postForm("/xpan/file", params, form, &precreateResp)
// 没有进度,走预上传
precreateResp, err = d.precreate(ctx, path, streamSize, blockListStr, contentMd5, sliceMd5, ctime, mtime)
if err != nil {
return nil, err
}
log.Debugf("%+v", precreateResp)
if precreateResp.ReturnType == 2 {
//rapid upload, since got md5 match from baidu server
// rapid upload, since got md5 match from baidu server
// 修复时间,具体原因见 Put 方法注释的 **注意**
precreateResp.File.Ctime = ctime
precreateResp.File.Mtime = mtime
return fileToObj(precreateResp.File), nil
}
}
// step.2 上传分片
threadG, upCtx := errgroup.NewGroupWithContext(ctx, d.uploadThread,
retry.Attempts(1),
retry.Delay(time.Second),
retry.DelayType(retry.BackOffDelay))
for i, partseq := range precreateResp.BlockList {
if utils.IsCanceled(upCtx) {
break
ensureUploadURL := func() {
if precreateResp.UploadURL != "" {
return
}
i, partseq, offset, byteSize := i, partseq, int64(partseq)*sliceSize, sliceSize
if partseq+1 == count {
byteSize = lastBlockSize
}
threadG.Go(func(ctx context.Context) error {
params := map[string]string{
"method": "upload",
"access_token": d.AccessToken,
"type": "tmpfile",
"path": path,
"uploadid": precreateResp.Uploadid,
"partseq": strconv.Itoa(partseq),
}
err := d.uploadSlice(ctx, params, stream.GetName(),
driver.NewLimitedUploadStream(ctx, io.NewSectionReader(cache, offset, byteSize)))
if err != nil {
return err
}
up(float64(threadG.Success()) * 100 / float64(len(precreateResp.BlockList)))
precreateResp.BlockList[i] = -1
return nil
})
precreateResp.UploadURL = d.getUploadUrl(path, precreateResp.Uploadid)
}
if err = threadG.Wait(); err != nil {
// 如果属于用户主动取消,则保存上传进度
// step.2 上传分片
uploadLoop:
for range 2 {
// 获取上传域名
ensureUploadURL()
// 并发上传
threadG, upCtx := errgroup.NewGroupWithContext(ctx, d.uploadThread,
retry.Attempts(UPLOAD_RETRY_COUNT),
retry.Delay(UPLOAD_RETRY_WAIT_TIME),
retry.MaxDelay(UPLOAD_RETRY_MAX_WAIT_TIME),
retry.DelayType(retry.BackOffDelay),
retry.RetryIf(func(err error) bool {
return !errors.Is(err, ErrUploadIDExpired)
}),
retry.LastErrorOnly(true))
totalParts := len(precreateResp.BlockList)
for i, partseq := range precreateResp.BlockList {
if utils.IsCanceled(upCtx) {
break
}
if partseq < 0 {
continue
}
i, partseq := i, partseq
offset, size := int64(partseq)*sliceSize, sliceSize
if partseq+1 == count {
size = lastBlockSize
}
threadG.Go(func(ctx context.Context) error {
params := map[string]string{
"method": "upload",
"access_token": d.AccessToken,
"type": "tmpfile",
"path": path,
"uploadid": precreateResp.Uploadid,
"partseq": strconv.Itoa(partseq),
}
section := io.NewSectionReader(cache, offset, size)
err := d.uploadSlice(ctx, precreateResp.UploadURL, params, stream.GetName(), section)
if err != nil {
return err
}
precreateResp.BlockList[i] = -1
progress := float64(threadG.Success()+1) * 100 / float64(totalParts+1)
up(progress)
return nil
})
}
err = threadG.Wait()
if err == nil {
break uploadLoop
}
// 保存进度(所有错误都会保存)
precreateResp.BlockList = utils.SliceFilter(precreateResp.BlockList, func(s int) bool { return s >= 0 })
base.SaveUploadProgress(d, precreateResp, d.AccessToken, contentMd5)
if errors.Is(err, context.Canceled) {
precreateResp.BlockList = utils.SliceFilter(precreateResp.BlockList, func(s int) bool { return s >= 0 })
return nil, err
}
if errors.Is(err, ErrUploadIDExpired) {
log.Warn("[baidu_netdisk] uploadid expired, will restart from scratch")
// 重新 precreate(所有分片都要重传)
newPre, err2 := d.precreate(ctx, path, streamSize, blockListStr, "", "", ctime, mtime)
if err2 != nil {
return nil, err2
}
if newPre.ReturnType == 2 {
return fileToObj(newPre.File), nil
}
precreateResp = newPre
precreateResp.UploadURL = ""
// 覆盖掉旧的进度
base.SaveUploadProgress(d, precreateResp, d.AccessToken, contentMd5)
continue uploadLoop
}
return nil, err
}
defer up(100)
// step.3 创建文件
var newFile File
@@ -342,25 +387,114 @@ func (d *BaiduNetdisk) Put(ctx context.Context, dstDir model.Obj, stream model.F
// 修复时间,具体原因见 Put 方法注释的 **注意**
newFile.Ctime = ctime
newFile.Mtime = mtime
// 上传成功清理进度
base.SaveUploadProgress(d, nil, d.AccessToken, contentMd5)
return fileToObj(newFile), nil
}
func (d *BaiduNetdisk) uploadSlice(ctx context.Context, params map[string]string, fileName string, file io.Reader) error {
res, err := base.RestyClient.R().
SetContext(ctx).
SetQueryParams(params).
SetFileReader("file", fileName, file).
Post(d.UploadAPI + "/rest/2.0/pcs/superfile2")
// precreate 执行预上传操作,支持首次上传和 uploadid 过期重试
func (d *BaiduNetdisk) precreate(ctx context.Context, path string, streamSize int64, blockListStr, contentMd5, sliceMd5 string, ctime, mtime int64) (*PrecreateResp, error) {
params := map[string]string{"method": "precreate"}
form := map[string]string{
"path": path,
"size": strconv.FormatInt(streamSize, 10),
"isdir": "0",
"autoinit": "1",
"rtype": "3",
"block_list": blockListStr,
}
// 只有在首次上传时才包含 content-md5 和 slice-md5
if contentMd5 != "" && sliceMd5 != "" {
form["content-md5"] = contentMd5
form["slice-md5"] = sliceMd5
}
joinTime(form, ctime, mtime)
var precreateResp PrecreateResp
_, err := d.postForm("/xpan/file", params, form, &precreateResp)
if err != nil {
return nil, err
}
// 修复时间,具体原因见 Put 方法注释的 **注意**
if precreateResp.ReturnType == 2 {
precreateResp.File.Ctime = ctime
precreateResp.File.Mtime = mtime
}
return &precreateResp, nil
}
func (d *BaiduNetdisk) uploadSlice(ctx context.Context, uploadUrl string, params map[string]string, fileName string, file *io.SectionReader) error {
b := bytes.NewBuffer(make([]byte, 0, bytes.MinRead))
mw := multipart.NewWriter(b)
_, err := mw.CreateFormFile("file", fileName)
if err != nil {
return err
}
log.Debugln(res.RawResponse.Status + res.String())
errCode := utils.Json.Get(res.Body(), "error_code").ToInt()
errNo := utils.Json.Get(res.Body(), "errno").ToInt()
headSize := b.Len()
err = mw.Close()
if err != nil {
return err
}
head := bytes.NewReader(b.Bytes()[:headSize])
tail := bytes.NewReader(b.Bytes()[headSize:])
rateLimitedRd := driver.NewLimitedUploadStream(ctx, io.MultiReader(head, file, tail))
req, err := http.NewRequestWithContext(ctx, http.MethodPost, uploadUrl+"/rest/2.0/pcs/superfile2", rateLimitedRd)
if err != nil {
return err
}
query := req.URL.Query()
for k, v := range params {
query.Set(k, v)
}
req.URL.RawQuery = query.Encode()
req.Header.Set("Content-Type", mw.FormDataContentType())
req.ContentLength = int64(b.Len()) + file.Size()
client := net.NewHttpClient()
if d.UploadSliceTimeout > 0 {
client.Timeout = time.Second * time.Duration(d.UploadSliceTimeout)
} else {
client.Timeout = DEFAULT_UPLOAD_SLICE_TIMEOUT
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
b.Reset()
_, err = b.ReadFrom(resp.Body)
if err != nil {
return err
}
body := b.Bytes()
respStr := string(body)
log.Debugln(respStr)
lower := strings.ToLower(respStr)
// 合并 uploadid 过期检测逻辑
if strings.Contains(lower, "uploadid") &&
(strings.Contains(lower, "invalid") || strings.Contains(lower, "expired") || strings.Contains(lower, "not found")) {
return ErrUploadIDExpired
}
errCode := utils.Json.Get(body, "error_code").ToInt()
errNo := utils.Json.Get(body, "errno").ToInt()
if errCode != 0 || errNo != 0 {
return errs.NewErr(errs.StreamIncomplete, "error in uploading to baidu, will retry. response=%s", res.String())
return errs.NewErr(errs.StreamIncomplete, "error uploading to baidu, response=%s", respStr)
}
return nil
}
func (d *BaiduNetdisk) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
du, err := d.quota(ctx)
if err != nil {
return nil, err
}
return &model.StorageDetails{DiskUsage: du}, nil
}
var _ driver.Driver = (*BaiduNetdisk)(nil)
+14
View File
@@ -1,6 +1,8 @@
package baidu_netdisk
import (
"time"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/op"
)
@@ -18,15 +20,27 @@ type Addition struct {
AccessToken string
RefreshToken string `json:"refresh_token" required:"true"`
UploadThread string `json:"upload_thread" default:"3" help:"1<=thread<=32"`
UploadSliceTimeout int `json:"upload_timeout" type:"number" default:"60" help:"per-slice upload timeout in seconds"`
UploadAPI string `json:"upload_api" default:"https://d.pcs.baidu.com"`
UseDynamicUploadAPI bool `json:"use_dynamic_upload_api" default:"true" help:"dynamically get upload api domain, when enabled, the 'Upload API' setting will be used as a fallback if failed to get"`
CustomUploadPartSize int64 `json:"custom_upload_part_size" type:"number" default:"0" help:"0 for auto"`
LowBandwithUploadMode bool `json:"low_bandwith_upload_mode" default:"false"`
OnlyListVideoFile bool `json:"only_list_video_file" default:"false"`
}
const (
UPLOAD_FALLBACK_API = "https://d.pcs.baidu.com" // 备用上传地址
UPLOAD_URL_EXPIRE_TIME = time.Minute * 60 // 上传地址有效期(分钟)
DEFAULT_UPLOAD_SLICE_TIMEOUT = time.Second * 60 // 上传分片请求默认超时时间
UPLOAD_RETRY_COUNT = 3
UPLOAD_RETRY_WAIT_TIME = time.Second * 1
UPLOAD_RETRY_MAX_WAIT_TIME = time.Second * 5
)
var config = driver.Config{
Name: "BaiduNetdisk",
DefaultRoot: "/",
PreferProxy: true,
}
func init() {
+41 -4
View File
@@ -1,12 +1,16 @@
package baidu_netdisk
import (
"errors"
"path"
"strconv"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
)
var (
ErrBaiduEmptyFilesNotAllowed = errors.New("empty files are not allowed by baidu netdisk")
)
type TokenErrResp struct {
@@ -71,9 +75,7 @@ func fileToObj(f File) *model.ObjThumb {
Modified: time.Unix(f.ServerMtime, 0),
Ctime: time.Unix(f.ServerCtime, 0),
IsFolder: f.Isdir == 1,
// 直接获取的MD5是错误的
HashInfo: utils.NewHashInfo(utils.MD5, DecryptMd5(f.Md5)),
// 百度API返回的MD5不可信,不使用HashInfo
},
Thumbnail: model.Thumbnail{Thumbnail: f.Thumbs.Url3},
}
@@ -188,4 +190,39 @@ type PrecreateResp struct {
// return_type=2
File File `json:"info"`
UploadURL string `json:"-"` // 保存断点续传对应的上传域名
}
type UploadServerResp struct {
BakServer []any `json:"bak_server"`
BakServers []struct {
Server string `json:"server"`
} `json:"bak_servers"`
ClientIP string `json:"client_ip"`
ErrorCode int `json:"error_code"`
ErrorMsg string `json:"error_msg"`
Expire int `json:"expire"`
Host string `json:"host"`
Newno string `json:"newno"`
QuicServer []any `json:"quic_server"`
QuicServers []struct {
Server string `json:"server"`
} `json:"quic_servers"`
RequestID int64 `json:"request_id"`
Server []any `json:"server"`
ServerTime int `json:"server_time"`
Servers []struct {
Server string `json:"server"`
} `json:"servers"`
Sl int `json:"sl"`
}
type QuotaResp struct {
Errno int `json:"errno"`
RequestId int64 `json:"request_id"`
Total int64 `json:"total"`
Used int64 `json:"used"`
//FreeSpace uint64 `json:"free"`
//Expire bool `json:"expire"`
}
+74 -13
View File
@@ -1,6 +1,7 @@
package baidu_netdisk
import (
"context"
"encoding/hex"
"errors"
"fmt"
@@ -40,7 +41,6 @@ func (d *BaiduNetdisk) _refreshToken() error {
ErrorMessage string `json:"text"`
}
_, err := base.RestyClient.R().
SetHeader("User-Agent", "Mozilla/5.0 (Macintosh; Apple macOS 15_5) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36 Chrome/138.0.0.0 Openlist/425.6.30").
SetResult(&resp).
SetQueryParams(map[string]string{
"refresh_ui": d.RefreshToken,
@@ -113,14 +113,14 @@ func (d *BaiduNetdisk) request(furl string, method string, callback base.ReqCall
errno := utils.Json.Get(res.Body(), "errno").ToInt()
if errno != 0 {
if utils.SliceContains([]int{111, -6}, errno) {
log.Info("refreshing baidu_netdisk token.")
log.Info("[baidu_netdisk] refreshing baidu_netdisk token.")
err2 := d.refreshToken()
if err2 != nil {
return retry.Unrecoverable(err2)
}
}
if 31023 == errno && d.DownloadAPI == "crack_video" {
if errno == 31023 && d.DownloadAPI == "crack_video" {
result = res.Body()
return nil
}
@@ -152,7 +152,7 @@ func (d *BaiduNetdisk) postForm(pathname string, params map[string]string, form
func (d *BaiduNetdisk) getFiles(dir string) ([]File, error) {
start := 0
limit := 200
limit := 1000
params := map[string]string{
"method": "list",
"dir": dir,
@@ -168,7 +168,6 @@ func (d *BaiduNetdisk) getFiles(dir string) ([]File, error) {
for {
params["start"] = strconv.Itoa(start)
params["limit"] = strconv.Itoa(limit)
start += limit
var resp ListResp
_, err := d.get("/xpan/file", params, &resp)
if err != nil {
@@ -187,6 +186,11 @@ func (d *BaiduNetdisk) getFiles(dir string) ([]File, error) {
} else {
res = append(res, resp.List...)
}
if len(resp.List) < limit {
break
}
start += limit
}
return res, nil
}
@@ -207,7 +211,7 @@ func (d *BaiduNetdisk) linkOfficial(file model.Obj, _ model.LinkArgs) (*model.Li
if err != nil {
return nil, err
}
//if res.StatusCode() == 302 {
// if res.StatusCode() == 302 {
u = res.Header().Get("location")
//}
@@ -245,7 +249,7 @@ func (d *BaiduNetdisk) linkCrack(file model.Obj, _ model.LinkArgs) (*model.Link,
func (d *BaiduNetdisk) linkCrackVideo(file model.Obj, _ model.LinkArgs) (*model.Link, error) {
param := map[string]string{
"type": "VideoURL",
"path": fmt.Sprintf("%s", file.GetPath()),
"path": file.GetPath(),
"fs_id": file.GetID(),
"devuid": "0%1",
"clienttype": "1",
@@ -324,10 +328,10 @@ func (d *BaiduNetdisk) getSliceSize(filesize int64) int64 {
// 非会员固定为 4MB
if d.vipType == 0 {
if d.CustomUploadPartSize != 0 {
log.Warnf("CustomUploadPartSize is not supported for non-vip user, use DefaultSliceSize")
log.Warnf("[baidu_netdisk] CustomUploadPartSize is not supported for non-vip user, use DefaultSliceSize")
}
if filesize > MaxSliceNum*DefaultSliceSize {
log.Warnf("File size(%d) is too large, may cause upload failure", filesize)
log.Warnf("[baidu_netdisk] File size(%d) is too large, may cause upload failure", filesize)
}
return DefaultSliceSize
@@ -335,17 +339,17 @@ func (d *BaiduNetdisk) getSliceSize(filesize int64) int64 {
if d.CustomUploadPartSize != 0 {
if d.CustomUploadPartSize < DefaultSliceSize {
log.Warnf("CustomUploadPartSize(%d) is less than DefaultSliceSize(%d), use DefaultSliceSize", d.CustomUploadPartSize, DefaultSliceSize)
log.Warnf("[baidu_netdisk] CustomUploadPartSize(%d) is less than DefaultSliceSize(%d), use DefaultSliceSize", d.CustomUploadPartSize, DefaultSliceSize)
return DefaultSliceSize
}
if d.vipType == 1 && d.CustomUploadPartSize > VipSliceSize {
log.Warnf("CustomUploadPartSize(%d) is greater than VipSliceSize(%d), use VipSliceSize", d.CustomUploadPartSize, VipSliceSize)
log.Warnf("[baidu_netdisk] CustomUploadPartSize(%d) is greater than VipSliceSize(%d), use VipSliceSize", d.CustomUploadPartSize, VipSliceSize)
return VipSliceSize
}
if d.vipType == 2 && d.CustomUploadPartSize > SVipSliceSize {
log.Warnf("CustomUploadPartSize(%d) is greater than SVipSliceSize(%d), use SVipSliceSize", d.CustomUploadPartSize, SVipSliceSize)
log.Warnf("[baidu_netdisk] CustomUploadPartSize(%d) is greater than SVipSliceSize(%d), use SVipSliceSize", d.CustomUploadPartSize, SVipSliceSize)
return SVipSliceSize
}
@@ -375,12 +379,69 @@ func (d *BaiduNetdisk) getSliceSize(filesize int64) int64 {
}
if filesize > MaxSliceNum*maxSliceSize {
log.Warnf("File size(%d) is too large, may cause upload failure", filesize)
log.Warnf("[baidu_netdisk] File size(%d) is too large, may cause upload failure", filesize)
}
return maxSliceSize
}
func (d *BaiduNetdisk) quota(ctx context.Context) (model.DiskUsage, error) {
var resp QuotaResp
_, err := d.request("https://pan.baidu.com/api/quota", http.MethodGet, func(req *resty.Request) {
req.SetContext(ctx)
}, &resp)
if err != nil {
return model.DiskUsage{}, err
}
return model.DiskUsage{TotalSpace: resp.Total, UsedSpace: resp.Used}, nil
}
// getUploadUrl 从开放平台获取上传域名/地址,并发请求会被合并,结果会在 uploadid 生命周期内复用。
// 如果获取失败,则返回 Upload API设置项。
func (d *BaiduNetdisk) getUploadUrl(path, uploadId string) string {
if !d.UseDynamicUploadAPI || uploadId == "" {
return d.UploadAPI
}
uploadUrl, err := d.requestForUploadUrl(path, uploadId)
if err != nil {
return d.UploadAPI
}
return uploadUrl
}
// requestForUploadUrl 请求获取上传地址。
// 实测此接口不需要认证,传method和upload_version就行,不过还是按文档规范调用。
// https://pan.baidu.com/union/doc/Mlvw5hfnr
func (d *BaiduNetdisk) requestForUploadUrl(path, uploadId string) (string, error) {
params := map[string]string{
"method": "locateupload",
"appid": "250528",
"path": path,
"uploadid": uploadId,
"upload_version": "2.0",
}
apiUrl := "https://d.pcs.baidu.com/rest/2.0/pcs/file"
var resp UploadServerResp
_, err := d.request(apiUrl, http.MethodGet, func(req *resty.Request) {
req.SetQueryParams(params)
}, &resp)
if err != nil {
return "", err
}
// 应该是https开头的一个地址
var uploadUrl string
if len(resp.Servers) > 0 {
uploadUrl = resp.Servers[0].Server
} else if len(resp.BakServers) > 0 {
uploadUrl = resp.BakServers[0].Server
}
if uploadUrl == "" {
return "", errors.New("upload URL is empty")
}
return uploadUrl, nil
}
// func encodeURIComponent(str string) string {
// r := url.QueryEscape(str)
// r = strings.ReplaceAll(r, "+", "%20")

Some files were not shown because too many files have changed in this diff Show More